Skip to content

Research Documentation

This directory contains research documents for HoloDeck project technical decisions and integrations.

Available Research Documents

Test Execution Framework Integration Research

File: test-execution-integration-research.md Date: November 1, 2025 Size: ~7,200 words

Comprehensive research on integrating four key technologies into HoloDeck's Python CLI test execution framework:

  1. Semantic Kernel Agents - Agent execution, tool calling, and orchestration
  2. markitdown - Document conversion (PDF, Office, images, etc.)
  3. Azure AI Evaluation SDK - AI-assisted quality metrics (groundedness, relevance, coherence)
  4. NLP Metrics Libraries - Traditional metrics (BLEU, ROUGE, METEOR, F1)

Key Sections: - Installation and setup requirements - API usage patterns with code examples - Error handling and best practices - Integration architecture recommendations - Decision matrix with rationale - Performance and cost considerations

Quick Decisions: - Agent Framework: Semantic Kernel (over LangChain) - Document Conversion: markitdown (over textract/apache-tika) - AI Evaluation: Azure AI Evaluation SDK (over custom prompts) - NLP Metrics: Hugging Face evaluate library (over NLTK)

How to Use This Research

For Implementation

  1. Start with Section 5 (Integration Architecture) for overall design
  2. Refer to specific technology sections for detailed implementation
  3. Use code examples as starting templates
  4. Follow best practices and common pitfalls sections

For Decision Making

  1. Review Section 6 (Decision Matrix) for technology choices
  2. Check alternatives considered and rationale
  3. Understand risk mitigation strategies
  4. Consider performance and cost implications

For Development

  1. Install dependencies from recommended versions
  2. Follow API usage patterns
  3. Implement error handling as documented
  4. Use integration examples for HoloDeck-specific code

Research Process

Research was conducted via: - Microsoft Learn documentation - GitHub repositories and issue trackers - PyPI package information - Recent blog posts and articles (2025) - Community discussions and comparisons

All recommendations are based on: - Current best practices (as of November 2025) - Azure ecosystem alignment - Production readiness - Maintenance and support - Cost-effectiveness

Contributing

When adding new research documents: 1. Use descriptive filenames: topic-research.md 2. Include date and author in frontmatter 3. Structure with clear sections and TOC 4. Provide code examples and references 5. Update this README with summary 6. Include decision rationale and alternatives

Last Updated: November 1, 2025