Enhanced Prompt: Implement Strict Separation Between Unit Tests and Content-Based Tests¶

Context¶

You are tasked with implementing a comprehensive testing policy that enforces a strict separation between unit tests and content-based tests. This policy must ensure that unit tests always run quickly and reliably, while content-based tests only run when new content has been generated.

Objective¶

Create and implement a testing policy with the following requirements: 1. Unit tests must always run and must never include tests that validate actual content generation or content-dependent behavior 2. All content-based validation must live in a separate content-test suite 3. Content tests should run only when new content has been generated since the last successful content-test run 4. Do not ignore or filter tests; there must be no content-based tests inside the unit-test suite 5. Provide a complete, ready-to-use implementation with concrete examples

Deliverables¶

1. Testing Policy Document¶

Create docs/testing-policy.md that serves as the single source of truth with: - Clear definitions of unit tests vs content-based tests - Project structure requirements with explicit separation - Content change detection mechanisms (content digests, timestamps, etc.) - Test runner configuration guidelines - Validation checklist - Audit and adjustment process

2. Concrete Implementation Plan¶

Create docs/testing-policy-implementation.md with: - Current state analysis and audit process - Step-by-step refactoring instructions - Directory reorganization plan - Test marking and discovery configuration - Content change detection implementation - CI/CD integration examples - Local development workflow updates - Validation and verification procedures - Rollout plan with phases

3. Ready-to-Use Scripts and Configuration¶

Implement the following components:

Directory Structure¶

project/
├── src/                    # Source code
├── tests/                  # Test suite root
│   ├── unit/              # Unit tests (always run)
│   └── content/           # Content tests (run only on new content)
├── scripts/                # Automation scripts
└── docs/                   # Documentation

Content Change Detection¶

Create a robust mechanism to detect content changes using: - Content digests (SHA256 hashes of content files) - Timestamp tracking - State management for last successful test runs

Test Runner Configuration¶

Configure test runners for different ecosystems: - Python (pytest): Separate configurations for unit and content tests - JavaScript (Jest): Test path separation and test naming conventions - Java (Maven/Gradle): Profile-based test execution

CI/CD Integration¶

Implement CI/CD workflows that: - Always run unit tests on every build - Conditionally run content tests only when content changes - Fail if policy violations are detected - Report results separately for unit and content tests

4. Migration Example¶

Provide a minimal, runnable example showing: - How to migrate existing content-based tests out of the unit suite - How to map each content item to its tests - How to report results separately - Before/after code examples

5. Validation Checklist¶

Create an automated validation process that ensures: - Unit tests contain no content-based checks - Content tests are skipped unless there is new content - CI fails if the policy is violated - There is a documented process to audit and adjust the policy

Technical Requirements¶

Language and Framework Agnostic Approach¶

While providing concrete examples, ensure the solution is adaptable to: - Python with pytest - JavaScript with Jest - Java with Maven/Gradle - Other common testing frameworks

Cross-Platform Commands¶

Provide commands that work on Windows, macOS, and Linux: - Use portable shell scripting - Avoid platform-specific tools where possible - Document any platform-specific variations

Robust Change Detection¶

Implement a reliable mechanism for detecting content changes: - Content digests using cryptographic hashes - Per-content timestamps or content IDs - State management for tracking test runs - Graceful handling of edge cases

Implementation Steps¶

Phase 1: Policy Definition¶

1. Create docs/testing-policy.md with comprehensive policy
2. Define clear criteria for unit vs content tests
3. Document content change detection mechanisms
4. Establish validation requirements

Phase 2: Implementation Planning¶

1. Create docs/testing-policy-implementation.md
2. Audit current test structure
3. Plan directory reorganization
4. Design content change detection system

Phase 3: Code Implementation¶

1. Refactor directory structure
2. Move content tests to separate directory
3. Add test marking mechanisms
4. Implement content change detection scripts
5. Update CI/CD workflows
6. Create local development scripts

Phase 4: Validation and Documentation¶

1. Implement validation scripts
2. Update README and development documentation
3. Create migration examples
4. Test all workflows

Success Criteria¶

The implementation is successful when: - Unit tests run in < 5 minutes without content dependencies - Content tests only run when content actually changes - CI/CD pipelines clearly separate unit and content test results - No content-based tests exist in the unit test suite - The solution works cross-platform - Documentation is clear and comprehensive - Migration process is straightforward

Constraints¶

• Do not use test filtering or ignoring mechanisms
• Maintain complete separation of test suites
• Ensure backward compatibility during migration
• Keep implementation simple and maintainable
• Avoid complex external dependencies

Output Format¶

Provide the implementation as: 1. A complete policy document (docs/testing-policy.md) 2. A detailed implementation plan (docs/testing-policy-implementation.md) 3. Ready-to-use scripts and configuration files 4. Concrete examples and migration guides 5. Validation tools and checklists