Skip to content

Logging plan

Plan: Implement Best Practice Python Logging for HoloDeck

Overview

Implement comprehensive, production-ready logging following Python best practices across the test command execution path.

Key Findings

  • Current State: Only 4 files have minimal logging (config/loader.py, agent_factory.py, schema.py, merge.py)
  • Critical Gaps: test.py, executor.py, file_processor.py, evaluators have NO logging
  • No Global Config: No centralized logging setup or standardization

Implementation Plan

Phase 1: Centralized Logging Infrastructure

  1. Create logging configuration module (src/holodeck/lib/logging_config.py)

  2. Structured logging format with timestamps, levels, module names

  3. Support for console and file handlers
  4. Integration with ExecutionConfig (verbose/quiet modes)
  5. Environment variable support (HOLODECK_LOG_LEVEL, HOLODECK_LOG_FILE)

  6. Log rotation for production use

  7. Initialize logging in package entry point (src/holodeck/__init__.py)

  8. Call setup_logging() on package import
  9. Ensure consistent configuration across all modules

Phase 2: Add Logging to Critical Paths

  1. Test Command (cli/commands/test.py)

  2. Log command invocation with parameters

  3. Log config loading success/failure
  4. Log test execution start/completion with timing
  5. Log report generation and file saving

  6. Enhanced error logging for all exception handlers

  7. Test Executor (lib/test_runner/executor.py)

  8. Log component initialization (FileProcessor, AgentFactory, Evaluators)

  9. Log test loop progress at DEBUG level
  10. Log individual test execution with timing
  11. Log file processing errors
  12. Log agent invocation timeouts and errors
  13. Log tool validation results
  14. Log evaluation execution and failures

  15. Log pass/fail determination with reasoning

  16. File Processor (lib/file_processor.py)

  17. Log file type detection (local vs remote)

  18. Log large file warnings
  19. Log download attempts and retries
  20. Log cache hits/misses

  21. Log conversion errors

  22. Evaluators (lib/evaluators/)

  23. Log evaluator initialization
  24. Log metric execution start/completion
  25. Log metric errors (currently caught silently)
  26. Log threshold comparisons

Phase 3: Enhancement & Testing

  1. Enhance existing logging in agent_factory.py

  2. Add timing information to retry logs

  3. Add structured context (attempt numbers, delays)

  4. Add logging utilities for common patterns

  5. Context managers for operation timing

  6. Structured context helpers (test_id, metric_name, etc.)

  7. Update documentation

  8. Add logging section to README.md
  9. Document log levels and environment variables
  10. Add troubleshooting guide with log examples

Log Levels Strategy

  • DEBUG: Detailed execution flow, variable values, all decisions
  • INFO: Test progress, component initialization, successful operations
  • WARNING: Retries, fallbacks, large files, cache issues
  • ERROR: Failed operations, exceptions, validation errors
  • CRITICAL: Fatal errors preventing execution

Configuration Integration

  • Respect --verbose flag: Set DEBUG level
  • Respect --quiet flag: Set ERROR level only
  • Default: INFO level
  • File output optional via env var or config

Testing Strategy

  • Unit tests for logging configuration
  • Integration tests verifying logs are generated
  • Test log output format and content
  • Test verbose/quiet mode integration