Implementation Tasks: Execute Agent Against Test Cases¶

Branch: 006-agent-test-execution Spec: spec.md Plan: plan.md

TDD Approach Overview¶

This task list follows Test-Driven Development (TDD) methodology. Each feature is implemented using the Red-Green-Refactor cycle:

TDD Cycle¶

[TEST] Red: Write a failing test that defines the expected behavior
[CODE] Green: Write the minimal code to make the test pass
[VERIFY] Refactor: Run code quality checks (make format, make lint, make type-check)
[VERIFY] Validate: Run tests to ensure everything still passes

Task Labels¶

[TEST]: Write unit or integration tests (must fail initially)
[CODE]: Implement code to make tests pass
[VERIFY]: Run automated checks (make test-unit, make format, make lint, etc.)

Benefits of TDD for This Project¶

Clear Requirements: Tests document expected behavior before implementation
Regression Prevention: Tests catch breaking changes immediately
Better Design: Writing tests first encourages modular, testable code
Confidence: High test coverage from day one (targeting 80%+)
Faster Debugging: Failing tests pinpoint exact issues

Workflow Example¶

# Step 1: Write test (should fail)
# Edit tests/unit/models/test_config.py - write test for ExecutionConfig

# Step 2: Run test (verify it fails - RED)
make test-unit

# Step 3: Implement code (minimal to pass)
# Edit src/holodeck/models/config.py - implement ExecutionConfig

# Step 4: Run test again (should pass - GREEN)
make test-unit

# Step 5: Refactor and verify
make format
make lint
make type-check
make test-unit  # Ensure still passing

Task Summary¶

Total Tasks: 164 (TDD approach with TEST + CODE + VERIFY tasks)
Setup Tasks: 10 (T001-T010)
Foundational Tasks: 20 (T011-T030)
User Story 1 (P1 - Execute Basic Text-Only Tests): 49 tasks
T057-T060: Progress Indicators (4 tasks) ✅ COMPLETED
T061-T066: Executor Integration with Callback Pattern (6 NEW tasks)
T067-T074: CLI Command (8 tasks)
T075-T078: Integration Testing (4 tasks)
T043-T056, T031-T042: Evaluators, Agent Bridge, Test Executor (24 tasks) ✅ COMPLETED
T104-T122: Enhanced Progress Display & Callback Integration (19 tasks)
User Story 2 (P2 - Multimodal Files): 14 tasks (T079-T094)
User Story 3 (P3 - Per-Test Metrics): 9 tasks (T095-T103)
User Story 4 (P2 - Progress Display): 22 tasks (T104-T122, overlaps with US1)
User Story 5 (P3 - Report Files): 16 tasks (T123-T139)
Polish & QA Tasks: 16 tasks (T140-T164)

Implementation Strategy¶

MVP = User Story 1 (P1: Execute Basic Text-Only Test Cases)

TDD Workflow: For each feature, follow the Red-Green-Refactor cycle:

Red: Write failing tests that define expected behavior
Green: Write minimal code to make tests pass
Refactor: Clean up code while keeping tests green
Verify: Run make format, make lint, make type-check

Each user story is independently testable and deliverable. User stories follow priority order (P1, P2, P3) and dependency constraints.

Key TDD Principles Applied:

Write tests first, code second
Each [TEST] task followed by corresponding [CODE] task
Frequent verification with make test-unit and make test-integration
Code quality checks after each implementation group

Dependency Graph¶

Phase 1: Setup (T001-T010)
    ↓
Phase 2: Foundational (T011-T020) ← [BLOCKS ALL USER STORIES]
    ↓
    ├─→ Phase 3: US1 - Basic Text Execution (T021-T036) [P1 - MVP]
    │       ↓
    │       ├─→ Phase 4: US2 - Multimodal Files (T037-T043) [P2]
    │       ├─→ Phase 5: US3 - Per-Test Metrics (T044-T047) [P3]
    │       └─→ Phase 6: US4 - Progress Indicators (T048-T055) [P2]
    │
    └─→ Phase 7: US5 - Report Generation (T056-T062) [P3]
         ↓
Phase 8: Polish & QA (T063-T074)

Critical Path: Setup → Foundational → US1 → Polish

Parallel Opportunities:

US2, US3, US4 can start after US1 completes (independent of each other)
US5 requires only Foundational models (can run parallel to US1)

Phase 1: Setup & Infrastructure¶

Goal: Initialize project structure, install dependencies, configure development environment

TDD Approach: Set up testing infrastructure before implementation

Tasks¶

Phase 2: Foundational Components¶

Goal: Implement shared infrastructure needed by all user stories (blocking prerequisites)

TDD Approach: Write tests first for all models and core utilities, then implement to pass tests

IMPORTANT: All Phase 2 tasks must complete before ANY user story implementation begins.

Tasks¶

T011-T016: Data Models (Test-First)¶

T025-T028: File Processor (Test-First)¶

T025 [TEST] Write unit tests for file processor in tests/unit/lib/test_file_processor.py (test markitdown integration for PDF, images, Excel, Word, PowerPoint, CSV, HTML)
T026 [CODE] Implement file processor using markitdown in src/holodeck/lib/file_processor.py to pass T025 tests
T027 [TEST] Write unit tests for file caching in tests/unit/lib/test_file_processor.py (test .holodeck/cache/ directory creation, hash-based cache keys, cache hit/miss)
T028 [CODE] Implement file caching logic in src/holodeck/lib/file_processor.py to pass T027 tests
T029 [TEST] Write unit tests for remote URL download in tests/unit/lib/test_file_processor.py (test timeout, 3 retries with exponential backoff, error handling)
T030 [CODE] Implement remote URL download with timeout in src/holodeck/lib/file_processor.py to pass T029 tests
T031 [VERIFY] Run make test-unit to verify all Phase 2 tests pass
T032 [VERIFY] Run make format && make lint && make type-check

Phase 3: User Story 1 - Execute Basic Text-Only Test Cases (P1)¶

Story Goal: Execute text-based test cases against agents and display pass/fail status with metric scores

TDD Approach: Write tests for each component before implementation, following Red-Green-Refactor cycle

Independent Test: Create agent.yaml with 3 simple text test cases, run holodeck test agent.yaml, verify:

Sequential execution
Agent responses captured
Evaluation metrics calculated
Pass/fail results displayed

Dependencies: Requires Phase 2 (Foundational) completion

Tasks¶

T033-T042: Evaluators (Test-First)¶

T043-T050: Agent Bridge (Test-First)¶

T043 [TEST] Write unit tests for agent bridge in tests/unit/lib/test_runner/test_agent_bridge.py (test Semantic Kernel integration, Kernel creation, agent config loading, ChatHistory invocation, response capture, tool_calls capture)
T044 [CODE] Implement Semantic Kernel agent bridge in src/holodeck/lib/test_runner/agent_bridge.py to pass T043 tests
T045 [VERIFY] Run make test-unit to verify agent bridge tests pass
T046 [VERIFY] Run make format && make lint

T047-T056: Test Executor (Test-First)¶

T057-T060: Progress Indicators (Test-First)¶

T057 [TEST] Write unit tests for progress indicators in tests/unit/lib/test_runner/test_progress.py (test TTY detection with sys.stdout.isatty(), "Test X/Y" display, checkmarks/X marks, CI/CD plain text mode)
T058 [CODE] Implement progress indicators in src/holodeck/lib/test_runner/progress.py to pass T057 tests
T059 [VERIFY] Run make test-unit to verify progress tests pass
T060 [VERIFY] Run make format && make lint

T061-T066: Executor Integration with Callback Pattern (Test-First)¶

Design Pattern: Callback-based progress reporting

The progress indicator integrates with TestExecutor via a callback pattern for clean separation of concerns:

# CLI passes callback to executor
def progress_callback(result: TestResult) -> None:
    indicator.update(result)
    print(indicator.get_progress_line())

executor = TestExecutor(
    agent_config_path=config_path,
    progress_callback=progress_callback
)

# Executor calls callback after each test
for test_case in test_cases:
    result = await self._execute_single_test(test_case)
    test_results.append(result)
    if self.progress_callback:
        self.progress_callback(result)  # Notify CLI of completion

Benefits:

✅ Executor stays focused on orchestration (no display logic)
✅ CLI controls display (quiet, verbose, progress suppression)
✅ Testable without side effects (mock the callback)
✅ Flexible for future use cases (webhooks, logging, etc.)

Tasks:

T061 [TEST] Write unit tests for progress callback integration in tests/unit/lib/test_runner/test_executor.py (test callback invocation after each test, callback with None handling, multiple test execution flow)
T062 [CODE] Add callback support to TestExecutor in src/holodeck/lib/test_runner/executor.py (add progress_callback parameter to init, call callback after each test in execute_tests, handle None gracefully)
T063 [TEST] Write unit tests for CLI progress display in tests/unit/cli/commands/test_test.py (test ProgressIndicator initialization, callback function execution, progress line printing, final summary display)
T064 [CODE] Integrate ProgressIndicator in CLI command src/holodeck/cli/commands/test.py (create indicator, pass callback to executor, display summary after tests complete, respect --quiet/--verbose flags)
T065 [VERIFY] Run make test-unit to verify executor and CLI integration tests pass
T066 [VERIFY] Run make format && make lint && make type-check

T067-T074: CLI Command (Test-First)¶

T067 [TEST] Write unit tests for CLI command in tests/unit/cli/commands/test_test.py (test argument parsing for AGENT_CONFIG, option handling for --output, --format, --verbose, --quiet, --timeout flags)
T068 [CODE] Implement CLI command in src/holodeck/cli/commands/test.py to pass T067 tests
T069 [TEST] Write unit tests for exit code logic in tests/unit/cli/commands/test_test.py (test 0=success, 1=test failure, 2=config error, 3=execution error, 4=evaluation error)
T070 [CODE] Implement exit code logic in src/holodeck/cli/commands/test.py to pass T069 tests
T071 [VERIFY] Run make test-unit to verify CLI tests pass
T072 [VERIFY] Run make format && make lint
T073 [TEST] Write unit tests for report generation integration in tests/unit/cli/commands/test_test.py (test --output flag, JSON/Markdown format)
T074 [CODE] Implement report file generation in CLI command

T075-T078: Integration Testing¶

T075 [TEST] Create sample test agent.yaml in tests/fixtures/agents/test_agent.yaml with 3 simple text test cases
T076 [TEST] Write integration test for basic text execution in tests/integration/test_basic_execution.py (verify end-to-end test run with mocked LLM responses)
T077 [VERIFY] Run make test-integration to verify integration tests pass
T078 [VERIFY] Run make test to verify all tests pass

Phase 4: User Story 2 - Execute Multimodal Test Cases with Files (P2)¶

Story Goal: Execute test cases with attached files (PDF, images, Office documents) and provide file content to agent

TDD Approach: Write tests for file processing features, then implement to pass tests

Independent Test: Create test cases with PDF/image/Excel files, run tests, verify:

Files processed via markitdown
Content extracted and provided to agent
Agent responses reference file content
Tests execute successfully

Dependencies: Requires US1 (basic test execution infrastructure)

Tasks¶

T079-T088: File Processing Extensions (Test-First)¶

T079 [TEST] Write unit tests for page/sheet/range extraction in tests/unit/lib/test_file_processor.py (test FileInput.pages for PDF, FileInput.sheet for Excel, FileInput.range for PowerPoint, preprocessing before markitdown)
T080 [CODE] Implement page/sheet/range extraction in src/holodeck/lib/file_processor.py to pass T079 tests
T081 [TEST] Write unit tests for file size warnings in tests/unit/lib/test_file_processor.py (test >100MB file warning message, verify processing continues)
T082 [CODE] Implement file size warning logic in src/holodeck/lib/file_processor.py to pass T081 tests
T083 [TEST] Write unit tests for file processing error handling in tests/unit/lib/test_file_processor.py (test timeout, malformed files, ProcessedFileInput.error population, test continuation)
T084 [CODE] Implement file processing error handling in src/holodeck/lib/file_processor.py to pass T083 tests
T085 [VERIFY] Run make test-unit to verify file processor tests pass
T086 [VERIFY] Run make format && make lint

T087-T092: Executor Integration (Test-First)¶

T087 [TEST] Write unit tests for file processor integration in tests/unit/lib/test_runner/test_executor.py (test process files before agent invocation, verify ProcessedFileInput.markdown_content in agent context)
T088 [CODE] Integrate file processor with test executor in src/holodeck/lib/test_runner/executor.py to pass T087 tests
T089 [VERIFY] Run make test-unit to verify executor integration tests pass
T090 [VERIFY] Run make format && make lint

T091-T094: Integration Testing¶

T091 [TEST] Create sample multimodal test files in tests/fixtures/files/ (sample.pdf, sample.jpg, sample.xlsx, sample.docx, sample.pptx with known content)
T092 [TEST] Write integration test for multimodal execution in tests/integration/test_multimodal_execution.py (verify file processing and agent receives markdown content)
T093 [VERIFY] Run make test-integration to verify multimodal tests pass
T094 [VERIFY] Run make test to verify all tests pass

Phase 5: User Story 3 - Execute Tests with Per-Test Metric Configuration (P3)¶

Story Goal: Allow test cases to specify their own evaluation metrics, overriding global defaults

TDD Approach: Write tests for metric resolution logic before implementation

Independent Test: Create test cases with varying metric configurations, verify:

Per-test metrics override global metrics
Tests without metrics use global defaults
Invalid metric references raise errors

Dependencies: Requires US1 (evaluation infrastructure)

Tasks¶

T095-T100: Per-Test Metrics (Test-First)¶

T095 [TEST] Write unit tests for per-test metric resolution in tests/unit/lib/test_runner/test_executor.py (test TestCaseModel.evaluations override, test fallback to AgentConfig.evaluations.metrics)
T096 [CODE] Implement per-test metric resolution logic in src/holodeck/lib/test_runner/executor.py to pass T095 tests
T097 [TEST] Write unit tests for metric validation in tests/unit/lib/test_runner/test_executor.py (test undefined metric raises ConfigError, test valid metrics pass)
T098 [CODE] Implement metric validation in src/holodeck/lib/test_runner/executor.py to pass T097 tests
T099 [VERIFY] Run make test-unit to verify metric resolution tests pass
T100 [VERIFY] Run make format && make lint

T101-T104: Integration Testing¶

T101 [TEST] Write integration test for per-test metrics in tests/integration/test_evaluation_metrics.py (verify metric override behavior, test global defaults)
T102 [VERIFY] Run make test-integration to verify metric tests pass
T103 [VERIFY] Run make test to verify all tests pass

Phase 6: User Story 4 - Display Test Results with Enhanced Progress Indicators (P2)¶

Story Goal: Show real-time progress with visual indicators during test execution

TDD Approach: Write tests for display features before implementation

Independent Test: Run 10 test cases, observe console output, verify:

Progress indicators update in real-time
Checkmarks/X marks appear for pass/fail
Final summary displayed
CI/CD compatibility (no interactive elements in non-TTY)

Dependencies: Requires US1 (basic execution)

Tasks¶

T104-T122: Enhanced Progress Display and Spinner (Test-First)¶

Note: Core progress indicator (T057-T058) is already implemented. These tasks add optional spinner animation and color enhancements built on top of the callback integration.

Phase 7: User Story 5 - Generate Test Report Files (P3)¶

Story Goal: Generate detailed test reports in JSON or Markdown format

TDD Approach: Write tests for report generation before implementation

Independent Test: Run tests with --output flag, verify:

JSON file created with complete test data
Markdown file created with human-readable format
Report structure matches expected schema

Dependencies: Requires US1 (test results data structure)

Tasks¶

T123-T133: Report Generation (Test-First)¶

T134-T139: CLI Format Detection & Report Output (Test-First)¶

T134 [TEST] Write unit tests for format auto-detection in tests/unit/cli/commands/test_test.py (test .json vs .md extension detection, test --format flag override)
T135 [CODE] Implement format auto-detection in src/holodeck/cli/commands/test.py to pass T134 tests
T136 [TEST] Write unit tests for report file output in CLI in tests/unit/cli/commands/test_test.py (test --output flag, verify file creation)
T137 [CODE] Implement report file output in CLI to pass T136 tests
T138 [VERIFY] Run make test-unit to verify format detection and output tests pass
T139 [VERIFY] Run make test to verify all tests pass

Phase 8: Polish & Cross-Cutting Concerns¶

Goal: Final refinements, error handling, documentation, and production readiness

TDD Approach: Write tests for error handling and edge cases, then implement

Tasks¶

T140-T148: Error Handling (Test-First)¶

T140 [TEST] Write unit tests for structured error messages in tests/unit/lib/test_runner/test_executor.py (test error format: "ERROR: {summary}\n Cause: {cause}\n Suggestion: {action}")
T141 [CODE] Implement structured error messages in src/holodeck/lib/test_runner/executor.py to pass T140 tests
T142 [TEST] Write unit tests for large file warnings in tests/unit/lib/test_file_processor.py (test warning when file >100MB before processing)
T143 [CODE] Implement warning messages for large files in src/holodeck/lib/file_processor.py to pass T142 tests
T144 [TEST] Write unit tests for timeout error handling in tests/unit/lib/test_runner/test_executor.py (test graceful timeout handling, test error messages)
T145 [CODE] Improve timeout error handling in executor to pass T144 tests
T146 [VERIFY] Run make test-unit to verify error handling tests pass
T147 [VERIFY] Run make format && make lint
T148 [VERIFY] Run make security to verify no security issues

T149-T152: Configuration Defaults (Test-First)¶

T149 [TEST] Write unit tests for configuration defaults in tests/unit/config/test_defaults.py (test file_timeout=30, llm_timeout=60, download_timeout=30, cache_dir=".holodeck/cache", cache_enabled=True)
T150 [CODE] Add configuration defaults to src/holodeck/config/defaults.py to pass T149 tests
T151 [VERIFY] Run make test-unit to verify defaults tests pass
T152 [VERIFY] Run make format && make lint

T153-T164: Documentation & Quality Assurance¶

Parallel Execution Opportunities (TDD Context)¶

Note: In TDD, [TEST] tasks must complete before their corresponding [CODE] tasks, but different test/code pairs can run in parallel.

Phase 1 (Setup)¶

All tasks can run in parallel - different files being created

Suggested groups:

Group A: T001-T005 (dependencies and directory structure)
Group B: T006-T009 (CLI and model stubs)
Group C: T010 (verification)

Phase 2 (Foundational)¶

Test/code pairs for different models can run in parallel:

Group A [P]: T011-T024 (model test/code pairs, ensure T011→T012, T013→T014, etc.)
Group B [P]: T025-T030 (file processor test/code pairs, ensure T025→T026, T027→T028, etc.)
Sequential: T031-T032 (verification after all implementations)

TDD Workflow: Write all model tests first (T011, T013, T015, T017, T019, T021, T023), then implement models (T012, T014, T016, T018, T020, T022, T024)

Phase 3 (US1)¶

Test/code pairs for different components can run in parallel:

Group A [P]: T033-T042 (evaluator test/code pairs)
Group B [P]: T043-T046 (agent bridge test/code pairs)
Group C [P]: T047-T056 (executor test/code pairs)
Group D [P]: T057-T060 (progress test/code pairs)
Group E [P]: T061-T066 (CLI test/code pairs)
Sequential: T067-T070 (integration tests and final verification)

TDD Workflow: Within each group, write tests before implementation (e.g., T033→T034, T035→T036)

Phase 4 (US2)¶

Test/code pairs for file processing:

Group A [P]: T071-T078 (file processing test/code pairs)
Group B [P]: T079-T082 (executor integration test/code pairs)
Sequential: T083-T086 (integration tests and verification)

Phase 5 (US3)¶

Test/code pairs for metric resolution:

Group A [P]: T087-T092 (metric resolution test/code pairs)
Sequential: T093-T095 (integration tests and verification)

Phase 6 (US4)¶

Test/code pairs for progress display:

Group A [P]: T096-T111 (all progress feature test/code pairs)

TDD Workflow: Write all progress tests first (T096, T098, T100, T102, T104, T106, T108), then implement (T097, T099, T101, T103, T105, T107, T109)

Phase 7 (US5)¶

Test/code pairs for report generation:

Group A [P]: T112-T120 (reporter test/code pairs)
Group B [P]: T121-T124 (CLI format detection test/code pairs)
Sequential: T125-T127 (integration tests and verification)

Phase 8 (Polish)¶

Test/code pairs for error handling and defaults:

Group A [P]: T128-T133 (error handling test/code pairs)
Group B [P]: T134-T137 (configuration defaults test/code pairs)
Sequential: T138-T148 (documentation and quality assurance)

Validation Checklist¶

For each user story phase:

Story goal clearly stated
Independent test criteria defined
All required components identified (models, services, CLI, tests)
Dependencies on other stories documented
Tasks include exact file paths
Parallel opportunities marked with [P]
Story labels ([US1], [US2], etc.) applied correctly
Total task count and breakdown by phase provided
Exit codes defined (0=success, 1=test failure, 2=config error, 3=execution error, 4=evaluation error)

Testing Strategy¶

Unit Tests (34 test files expected)¶

Models:

tests/unit/models/test_test_result.py (ProcessedFileInput, MetricResult, TestResult, ReportSummary, TestReport)
tests/unit/models/test_config.py (ExecutionConfig validation)

Lib Components:

tests/unit/lib/test_file_processor.py (markitdown integration, caching, range extraction)
tests/unit/lib/test_runner/test_executor.py (config resolution, test execution flow, timeout handling)
tests/unit/lib/test_runner/test_agent_bridge.py (Semantic Kernel integration)
tests/unit/lib/test_runner/test_progress.py (TTY detection, output formatting)
tests/unit/lib/test_runner/test_reporter.py (JSON/Markdown generation)
tests/unit/lib/evaluators/test_azure_ai.py (Azure AI Evaluation SDK)
tests/unit/lib/evaluators/test_nlp_metrics.py (BLEU, ROUGE, METEOR, F1)

CLI:

tests/unit/cli/commands/test_test.py (CLI argument parsing, option handling)

Integration Tests (4 test files)¶

tests/integration/test_basic_execution.py (US1: end-to-end text test execution)
tests/integration/test_multimodal_execution.py (US2: file processing and agent context)
tests/integration/test_evaluation_metrics.py (US3: per-test metric resolution)
tests/integration/test_report_generation.py (US5: JSON and Markdown output)

Success Criteria¶

From spec.md:

Configuration Hierarchy Reminder¶

CLI Flags  >  agent.yaml execution  >  Environment Variables  >  Built-in Defaults
(highest priority)                                              (lowest priority)

Example:

Built-in default: file_timeout=30s
Environment variable: HOLODECK_FILE_TIMEOUT=45
agent.yaml: execution.file_timeout: 60
CLI flag: --file-timeout 90
Resolved: 90s (CLI wins)

File Paths Reference¶

New Files (Core) - Phase 1 & 2:

src/holodeck/models/test_result.py (T002)
src/holodeck/models/config.py (T003, T011)
src/holodeck/lib/file_processor.py (T005, T025-T030)
src/holodeck/lib/test_runner/executor.py (T006, T047-T056, T062)
src/holodeck/lib/test_runner/agent_bridge.py (T006, T043-T045)
src/holodeck/lib/evaluators/base.py (T007, T033-T034)
src/holodeck/lib/evaluators/azure_ai.py (T007, T035-T038)
src/holodeck/lib/evaluators/nlp_metrics.py (T007, T039-T040)

New Files (Progress & CLI) - Phase 3A & 3B:

src/holodeck/lib/test_runner/progress.py (T057-T058, T104-T122)
src/holodeck/lib/test_runner/reporter.py (T123-T139)
src/holodeck/cli/commands/test.py (T061-T066 callback integration, T067-T074 CLI, T110-T111 callback display)

Modified Files:

pyproject.toml (T001, dependencies)
src/holodeck/models/agent.py (T023-T024 execution field)
src/holodeck/cli/main.py (T007 register test command)
src/holodeck/config/defaults.py (T149-T150)
CLAUDE.md (documentation updates)

Test Files:

34+ test files across unit and integration directories

Notes¶

Semantic Kernel: Agent execution uses semantic_kernel.agents.Agent with ChatHistory for context
markitdown: Unified file processor handles all file types with .convert() method
Azure AI Evaluation: Per-metric model configuration for cost optimization (GPT-4o for critical metrics, GPT-4o-mini for general)
Retry Logic: 3 attempts with exponential backoff (2s/4s/8s for LLM, 1s/2s/4s for downloads)
Caching: Remote files cached in .holodeck/cache/ with hash-based keys
Exit Codes: 0=success, 1=test failure, 2=config error, 3=execution error, 4=evaluation error
TTY Detection: Use sys.stdout.isatty() for progress indicator behavior
Configuration Merge: CLI > agent.yaml > env vars > defaults
Callback Pattern (T061-T066, T110-T111): Progress display is decoupled from executor via callback function. CLI creates ProgressIndicator, passes callback to TestExecutor, receives TestResult updates after each test. Keeps executor focused on orchestration, allows CLI to handle display modes (quiet/verbose).
Progress Indicator (T057-T058): TTY-aware progress display with real-time updates, pass/fail symbols (✓/✗ for TTY, PASS/FAIL for CI/CD), summary statistics, and execution timing. Built-in support for quiet and verbose modes.

End of Tasks Document