Quickstart: HierarchicalDocumentTool¶

Overview¶

The HierarchicalDocumentTool provides advanced document search capabilities combining: - Semantic search (dense embeddings for conceptual queries) - Keyword search (BM25 for precise terminology) - Exact match (section numbers, identifiers) - Hybrid fusion (Reciprocal Rank Fusion to combine all modalities)

It preserves document structure during parsing, so search results include the full hierarchical context (e.g., "Title I > Chapter 2 > Section 203").

Quick Start (3 minutes)¶

1. Add to your agent.yaml¶

name: my-agent
description: Agent with document search

model:
  provider: openai
  name: gpt-4o

instructions:
  inline: |
    You are a helpful assistant with access to a knowledge base.
    Use the docs_search tool to find relevant information.

tools:
  - name: docs_search
    type: hierarchical_document
    description: "Search company documentation"
    source: "./docs/"

2. Add your documents¶

Place your documents in the ./docs/ directory: - Markdown (.md) - PDF (.pdf) - Word (.docx) - Text (.txt)

3. Run your agent¶

holodeck chat agent.yaml

The tool automatically: 1. Converts documents to markdown (via markitdown) 2. Parses hierarchical structure from headings 3. Chunks by structure (max 800 tokens) 4. Generates LLM context for each chunk (Anthropic approach - calls Claude Haiku) 5. Generates embeddings from contextualized text 6. Builds BM25 (on contextualized text) and exact match indices

Configuration Examples¶

Semantic-Only Search¶

Best for conceptual questions without specific terminology.

tools:
  - name: concept_search
    type: hierarchical_document
    description: "Search by concepts"
    source: "./docs/"
    search_mode: semantic
    contextual_embeddings: true

Keyword-Heavy Search¶

Best for technical documents with specific terminology.

tools:
  - name: tech_search
    type: hierarchical_document
    description: "Technical documentation search"
    source: "./technical/"
    search_mode: hybrid
    semantic_weight: 0.3
    keyword_weight: 0.5
    exact_weight: 0.2

Legal/Regulatory Documents¶

Optimized for section references and definitions.

tools:
  - name: legal_search
    type: hierarchical_document
    description: "Search legal documents"
    source: "./legal/"
    search_mode: hybrid
    extract_definitions: true
    extract_cross_references: true
    exact_weight: 0.3

Custom Context Model¶

Specify a different LLM for context generation (defaults to Claude Haiku).

tools:
  - name: docs_search
    type: hierarchical_document
    description: "Search with custom context model"
    source: "./docs/"

    # LLM context generation settings
    contextual_embeddings: true
    context_model:
      provider: anthropic
      name: claude-3-haiku-20240307
      temperature: 0.0
    context_max_tokens: 100
    context_concurrency: 10  # Parallel LLM calls

Large Document Corpus with Persistence¶

tools:
  - name: knowledge_base
    type: hierarchical_document
    description: "Persistent knowledge base"
    source: "./knowledge/"
    database:
      provider: postgres
      connection_string: "${POSTGRES_URL}"
    top_k: 20

Keyword Index: In-Memory BM25 (Default)¶

By default, keyword search uses an in-memory BM25 index via rank_bm25. No extra configuration needed -- this is the default when keyword_index is omitted or set to in-memory.

tools:
  - name: docs_search
    type: hierarchical_document
    description: "Search with in-memory BM25 keyword index"
    source: "./docs/"
    search_mode: hybrid
    keyword_index:
      provider: in-memory  # Optional -- this is the default

Keyword Index: OpenSearch (Production)¶

For production workloads, use an external OpenSearch cluster for the keyword index. This offloads BM25 scoring to a dedicated search engine, suitable for large corpora and multi-instance deployments.

tools:
  - name: docs_search
    type: hierarchical_document
    description: "Search with OpenSearch keyword backend"
    source: "./docs/"
    search_mode: hybrid
    keyword_index:
      provider: opensearch
      endpoint: "https://search.example.com:9200"
      index_name: "my-keyword-index"
      username: "${OPENSEARCH_USERNAME}"
      password: "${OPENSEARCH_PASSWORD}"
      verify_certs: true
      timeout_seconds: 10

Docker quickstart: To run OpenSearch locally via Docker:
docker run -d -p 9200:9200 -e "discovery.type=single-node" \
  -e "DISABLE_SECURITY_PLUGIN=true" opensearchproject/opensearch:latest
Then use endpoint: "http://localhost:9200" with no auth.

MongoDB Atlas with Native Hybrid Search¶

tools:
  - name: atlas_search
    type: hierarchical_document
    description: "MongoDB Atlas with native hybrid search"
    source: "./docs/"
    database:
      provider: mongodb  # MongoDB Atlas - supports native hybrid_search()
      connection_string: "${MONGODB_ATLAS_URL}"

Important: The mongodb provider refers to MongoDB Atlas (native MongoDB), which supports native hybrid search via Semantic Kernel's MongoDBAtlasStore. Azure Cosmos DB's MongoDB API (azure-cosmos-mongo) is NOT supported for hybrid search - it requires a different integration pattern with separate vector + text search APIs. If you need hybrid search with Azure Cosmos DB, use azure-cosmos-nosql which supports native hybrid search.

Search Modes¶

Mode	Use Case	Example Query
`semantic`	Conceptual questions	"What are the compliance requirements?"
`keyword`	Technical terms	"authentication timeout error"
`exact`	Section references	"Section 203(a)(1)"
`hybrid`	General purpose	"reporting requirements in Section 5"

How It Works¶

Preprocessing Pipeline (Anthropic Contextual Retrieval)¶

┌─────────────────────────────────────────────────────────────────────────┐
│                         INGESTION (per document)                         │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  document.pdf  →  [markitdown]  →  markdown text                         │
│                                                                          │
│  markdown text →  [StructuredChunker]  →  chunks with parent_chain       │
│                                                                          │
│  For each chunk:                                                         │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │ [LLM Context Generation] - Claude Haiku                           │  │
│  │                                                                    │  │
│  │ Input:  Whole document + chunk                                     │  │
│  │ Output: "This chunk describes the reporting requirements under     │  │
│  │          Section 203 of the Environmental Protection Act..."       │  │
│  │                                                                    │  │
│  │ Cost: ~$0.03 per 100-page document                                 │  │
│  └───────────────────────────────────────────────────────────────────┘  │
│                                                                          │
│  contextualized = "{LLM context}\n\n{original chunk}"                    │
│                                                                          │
│  contextualized  →  [Embedding]  →  dense vector                         │
│  contextualized  →  [BM25/Native] →  keyword index                       │
│  section_id      →  [Dict]        →  exact match index                   │
│                                                                          │
│  Result: 49% better retrieval vs standard RAG (Anthropic research)       │
└─────────────────────────────────────────────────────────────────────────┘

Hybrid Search Flow¶

Query Analysis: Detects exact match patterns (section numbers, quoted phrases)
Parallel Search: Runs semantic, keyword, and exact match searches
RRF Fusion: Combines ranked results using Reciprocal Rank Fusion (k=60)
Enrichment: Adds relevant definitions if terms are defined in documents

Query: "reporting requirements in Section 403"
                    │
    ┌───────────────┼───────────────┐
    ▼               ▼               ▼
[Semantic]      [BM25]          [Exact]
contextualized  contextualized  section_id
embeddings      text search     lookup
    │               │               │
    └───────────────┼───────────────┘
                    ▼
              [RRF Fusion]
              k=60, weights
                    │
                    ▼
            [Top 10 Results]

Result Format¶

Each search result includes:

[1] Score: 0.847 | Source: docs/compliance.md
Location: Chapter 5 > Section 5.3 > Reporting Requirements
Section: sec_5_3

Annual reporting must be submitted within 60 days of the fiscal year end.
All covered entities must include revenue breakdowns by category.

Relevant definitions:
  • Covered entity: Any organization with annual revenue exceeding...

Performance Tips¶

Chunk size: Default 800 tokens works well. Increase for dense technical content, decrease for granular retrieval.
Weights tuning:
More semantic: Increase semantic_weight for conceptual queries
More precise: Increase keyword_weight for technical terminology
Section-heavy: Increase exact_weight for regulatory documents
Persistence: For large corpora (>1000 documents), use a persistent database to avoid re-indexing on restart.
Contextual embeddings: Keep enabled (default) for 49% better retrieval accuracy per Anthropic research.
Context generation cost: ~$0.03 per 100-page document using Claude Haiku. For cost-sensitive applications, increase context_concurrency to reduce ingestion time.
Large documents: For very large documents that exceed context limits, the tool automatically truncates the document context while preserving the chunk content.

Testing Your Configuration¶

# Run with verbose output to see search details
holodeck test agent.yaml --verbose

# Or use chat mode to test interactively
holodeck chat agent.yaml

Test queries: - Conceptual: "What are the main requirements?" - Keyword: "authentication error handling" - Exact: "Section 5.2.1" - Mixed: "how to handle errors in Section 3"