Testing Policy Implementation Plan¶

1. Current State Analysis¶

1.1 Audit Existing Tests¶

First, we need to identify which tests are unit tests vs content-based tests:

# Find all test files
find tests/ -name "test_*.py" -type f

# Analyze test content to categorize
grep -r "def test_" tests/ | head -20

1.2 Current Test Structure¶

tests/
├── test_agents.py              # Unit tests
├── test_article_integrity.py   # CONTENT TESTS - Should move
├── test_categories.py          # Unit tests
├── test_new_articles.py        # CONTENT TESTS - Should move
├── test_missing_articles.py    # CONTENT TESTS - Should move
└── ...                         # Other unit tests

2. Project Structure Refactoring¶

2.1 Create New Directory Structure¶

mkdir -p tests/unit tests/content

2.2 Move Content Tests¶

# Move content-based tests to content directory
mv tests/test_article_integrity.py tests/content/
mv tests/test_new_articles.py tests/content/
mv tests/test_missing_articles.py tests/content/

# Keep unit tests in unit directory
mv tests/test_agents.py tests/unit/
mv tests/test_categories.py tests/unit/
# ... move other unit tests

2.3 Update Import Paths¶

After moving files, update import statements in content tests:

# In content test files, update imports if needed
# Example: Change relative imports to absolute imports

3. Test Marking and Discovery¶

3.1 Mark Content Tests¶

Add explicit markers to content tests:

tests/content/test_article_integrity.py:

import pytest

pytestmark = pytest.mark.content

class TestArticleLinkIntegrity:
    # ... existing tests

tests/content/test_new_articles.py:

import pytest

pytestmark = pytest.mark.content

# ... existing tests

3.2 Configure Test Runner¶

Create pytest configuration to exclude content tests from unit runs:

pyproject.toml:

[tool.pytest.ini_options]
markers = [
    "content: marks tests as content-based tests",
]

# Default to unit tests only
testpaths = ["tests/unit"]

# For content tests, use: pytest tests/content/ -v -m content

4. Content Change Detection Implementation¶

4.1 Create Content Digest Script¶

scripts/generate-content-digest.py:

#!/usr/bin/env python3
"""
Generate content digest for change detection.
"""
import hashlib
import os
import sys
from pathlib import Path

def generate_content_digest(content_dir="docs/trendek"):
    """Generate SHA256 digest of all content files."""
    content_path = Path(content_dir)
    if not content_path.exists():
        return ""

    hashes = []
    # Find all markdown files (content files)
    for file_path in content_path.rglob("*.md"):
        if file_path.is_file():
            with open(file_path, "rb") as f:
                file_hash = hashlib.sha256(f.read()).hexdigest()
                hashes.append(f"{file_hash}  {file_path}")

    # Sort to ensure consistent ordering
    hashes.sort()

    # Generate digest of all file hashes
    combined = "\n".join(hashes).encode("utf-8")
    return hashlib.sha256(combined).hexdigest()

if __name__ == "__main__":
    digest = generate_content_digest()
    print(digest)

    # Optionally save to file
    if len(sys.argv) > 1:
        with open(sys.argv[1], "w") as f:
            f.write(digest)

4.2 Create Content Test Runner Script¶

scripts/run-content-tests.sh:

#!/bin/bash
# Run content tests only when content has changed

set -e

CONTENT_DIGEST_FILE=".content-digest"
LAST_TESTED_DIGEST_FILE=".last-tested-digest"

echo "🔍 Checking for content changes..."

# Generate current content digest
python scripts/generate-content-digest.py $CONTENT_DIGEST_FILE

# Check if content has changed
if [ -f "$LAST_TESTED_DIGEST_FILE" ]; then
    CURRENT_DIGEST=$(cat $CONTENT_DIGEST_FILE)
    LAST_DIGEST=$(cat $LAST_TESTED_DIGEST_FILE)

    if [ "$CURRENT_DIGEST" = "$LAST_DIGEST" ]; then
        echo "✅ No content changes detected. Skipping content tests."
        exit 0
    else
        echo "📝 Content changes detected. Running content tests..."
    fi
else
    echo "📝 No previous test record found. Running content tests..."
fi

# Run content tests
if uv run pytest tests/content/ -v -m content; then
    # Update last tested digest on success
    cp $CONTENT_DIGEST_FILE $LAST_TESTED_DIGEST_FILE
    echo "✅ Content tests passed. Updated content digest."
else
    echo "❌ Content tests failed."
    exit 1
fi

5. CI/CD Configuration¶

5.1 Update GitHub Actions Workflow¶

.github/workflows/lint-and-test.yml:

name: Lint and Test

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4

    - name: Install uv
      uses: astral-sh/setup-uv@v5
      with:
        enable-cache: true

    - name: Set up Python
      run: uv python install 3.11

    - name: Install dependencies
      run: uv sync

    - name: Run Ruff linting
      run: uv run ruff check .

    - name: Run Ruff formatting check
      run: uv run ruff format --check .

    - name: Run unit tests only
      run: uv run pytest tests/unit/ -v

  content-test:
    runs-on: ubuntu-latest
    needs: lint-and-test
    # Only run on main branch or when content might have changed
    if: github.ref == 'refs/heads/main' || github.event_name == 'push'

    steps:
    - uses: actions/checkout@v4

    - name: Install uv
      uses: astral-sh/setup-uv@v5
      with:
        enable-cache: true

    - name: Set up Python
      run: uv python install 3.11

    - name: Install dependencies
      run: uv sync

    - name: Run content tests conditionally
      run: ./scripts/run-content-tests.sh

6. Local Development Workflow¶

6.1 Update Local Test Script¶

scripts/lint-and-test.sh:

#!/bin/bash
# Lint and test script for project - unit tests only

set -e

echo "🔍 Running Ruff linting..."
uv run ruff check .

echo ""
echo "✨ Running Ruff formatting check..."
uv run ruff format --check .

echo ""
echo "🧪 Running unit tests only..."
uv run pytest tests/unit/ -v

echo ""
echo "✅ All unit checks passed!"

6.2 Content Test Script for Local Use¶

scripts/test-content.sh:

#!/bin/bash
# Run content tests locally

set -e

echo "🔍 Running content tests..."
./scripts/run-content-tests.sh

echo ""
echo "✅ Content tests completed!"

7. Migration Process¶

7.1 Step-by-Step Migration¶

Create directory structure:
```
mkdir -p tests/unit tests/content
```

Move content tests:

mv tests/test_article_integrity.py tests/content/
mv tests/test_new_articles.py tests/content/
mv tests/test_missing_articles.py tests/content/

Move unit tests:

mv tests/test_*.py tests/unit/  # Move remaining tests

Add content markers: Add pytestmark = pytest.mark.content to each content test file
Update CI/CD configuration to separate unit and content tests
Update documentation to reflect new structure

7.2 Verification Commands¶

# Verify unit tests run without content tests
pytest tests/unit/ -v

# Verify content tests are properly marked
pytest tests/content/ -v -m content

# Verify no content tests are discovered in unit runs
pytest tests/unit/ -v -m "not content"

8. Validation Checklist Implementation¶

8.1 Automated Validation Script¶

scripts/validate-test-separation.py:

#!/usr/bin/env python3
"""
Validate test separation policy.
"""
import ast
import os
import sys
from pathlib import Path

def check_unit_tests_for_content_references():
    """Check that unit tests don't reference content files."""
    unit_tests_dir = Path("tests/unit")
    content_dirs = ["docs/trendek", "content"]

    violations = []

    for test_file in unit_tests_dir.rglob("test_*.py"):
        with open(test_file, "r") as f:
            content = f.read()

            # Check for content directory references
            for content_dir in content_dirs:
                if content_dir in content:
                    violations.append(f"Unit test {test_file} references content directory {content_dir}")

    return violations

def check_content_tests_are_marked():
    """Check that content tests have proper markers."""
    content_tests_dir = Path("tests/content")

    violations = []

    for test_file in content_tests_dir.rglob("test_*.py"):
        with open(test_file, "r") as f:
            content = f.read()

            # Check for content marker
            if "pytestmark = pytest.mark.content" not in content and "@pytest.mark.content" not in content:
                violations.append(f"Content test {test_file} is missing content marker")

    return violations

def main():
    print("🔍 Validating test separation policy...")

    violations = []
    violations.extend(check_unit_tests_for_content_references())
    violations.extend(check_content_tests_are_marked())

    if violations:
        print("❌ Policy violations found:")
        for violation in violations:
            print(f"  - {violation}")
        sys.exit(1)
    else:
        print("✅ All policy checks passed!")

if __name__ == "__main__":
    main()

9. Documentation Updates¶

9.1 Update README.md¶

Add sections about the new testing structure:

## Testing

This project maintains a strict separation between unit tests and content-based tests:

- `tests/unit/` - Unit tests that validate code logic (always run)
- `tests/content/` - Content tests that validate generated content (run conditionally)

See [docs/testing-policy.md](docs/testing-policy.md) for the complete testing policy.

9.2 Update Development Documentation¶

Update any development guides to reflect the new structure and commands.

10. Rollout Plan¶

Phase 1: Preparation (1 day)¶

Create policy document
Create implementation plan
Set up directory structure
Create content detection scripts

Phase 2: Migration (2 days)¶

Move content tests to content directory
Add content markers
Update CI/CD workflows
Update local scripts

Phase 3: Validation (1 day)¶

Run validation scripts
Verify CI/CD changes
Test local workflows
Update documentation

Phase 4: Monitoring (Ongoing)¶

Monitor CI/CD performance
Review test execution times
Audit test categorization monthly