Agent Configuration Guide¶

This guide explains how to define AI agents using HoloDeck's agent.yaml configuration file.

Overview¶

An agent configuration file defines a single AI agent with everything it needs:

Model settings: Which LLM provider to use (OpenAI, Azure, Anthropic)
Instructions: System prompt (from a file or inline)
Tools: Capabilities the agent can use (search, functions, APIs, templates)
Evaluations: Metrics to measure quality
Test cases: Scenarios to validate the agent

All configuration is declarative—no code required.

Basic Structure¶

name: my-agent                    # Required: Agent name
description: Agent description    # Optional: What this agent does
author: "Your Name"               # Optional: Who created this agent

model:                            # Required: LLM configuration
  provider: anthropic             # Required: openai|azure_openai|anthropic|ollama
  name: claude-sonnet-4-20250514  # Required: Model identifier
  auth_provider: oauth_token      # Optional: Uses CLAUDE_CODE_OAUTH_TOKEN env var
  temperature: 0.7                # Optional: 0.0-2.0
  max_tokens: 2000                # Optional: Maximum generation tokens

instructions:                     # Required: System prompt
  inline: "You are a helpful..."  # Option 1: Inline text
  # OR
  # file: instructions.txt        # Option 2: External file

tools: []                         # Optional: Agent capabilities
evaluations:                      # Optional: Quality metrics
  metrics: []
test_cases: []                    # Optional: Test scenarios

Backend auto-selection: HoloDeck picks the execution backend from the provider field. anthropic routes to the Claude Backend; everything else (openai, azure_openai, ollama) routes to the Semantic Kernel Backend. See those guides for backend-specific configuration.

Agent Name¶

Required: Yes
Type: String
Format: 1-100 characters, alphanumeric + hyphens, must start with letter
Examples: customer-support, code-reviewer, data-analyzer

name: customer-support

Agent Description¶

Required: No
Type: String
Constraints: Max 500 characters
Purpose: Describe what this agent does for documentation

description: Handles customer support queries with ticket creation

Agent Author¶

Required: No
Type: String
Constraints: Max 256 characters
Purpose: Document who created or maintains this agent

author: "Alice Johnson"

This field is useful for: - Attribution and credit in multi-team environments - Understanding who to contact for questions about the agent - Tracking agent ownership and maintenance responsibility

Model Configuration¶

Defines which LLM provider and model to use.

Provider Field¶

Required: Yes
Type: String (Enum)
Options:
openai - OpenAI API (GPT-4o, GPT-4o-mini, etc.)
azure_openai - Azure OpenAI Service
anthropic - Anthropic Claude (native Claude Agent SDK backend)
ollama - Local models via Ollama

model:
  provider: anthropic

Model Name¶

Required: Yes
Type: String
Purpose: Identifies specific model within provider
Examples by Provider:
OpenAI: gpt-4o, gpt-4o-mini, gpt-4-turbo
Azure: gpt-4, gpt-4-32k
Anthropic: claude-sonnet-4-20250514, claude-opus-4-20250514, claude-3-5-haiku-20241022
Ollama: llama3.2, mistral, codellama

model:
  name: gpt-4o

Temperature¶

Optional: Yes
Type: Float
Range: 0.0 to 2.0
Default: 0.7 (if not specified)
Meaning:
0.0 - Deterministic, focused responses
0.7 - Balanced randomness
1.5+ - Very creative, random responses

model:
  temperature: 0.8  # More creative

Max Tokens¶

Optional: Yes
Type: Integer
Constraint: Must be positive
Purpose: Limit maximum length of generated responses

model:
  max_tokens: 4000

Top P¶

Optional: Yes
Type: Float
Range: 0.0 to 1.0
Purpose: Nucleus sampling (alternative to temperature)

model:
  top_p: 0.9

Authentication Provider¶

Optional: Yes (defaults to api_key)
Type: String (Enum)
Applies to: anthropic provider only (ignored for other providers)
Purpose: Select the authentication method for Claude Agent SDK

Method	Env Variable	Use Case
`api_key`	`ANTHROPIC_API_KEY`	Direct API access (default)
`oauth_token`	`CLAUDE_CODE_OAUTH_TOKEN`	Claude Code OAuth (recommended)
`bedrock`	AWS credentials (`AWS_ACCESS_KEY_ID`, etc.)	AWS Bedrock
`vertex`	GCP credentials (`GOOGLE_APPLICATION_CREDENTIALS`)	Google Vertex AI
`foundry`	Foundry credentials	Foundry platform

model:
  provider: anthropic
  name: claude-sonnet-4-20250514
  auth_provider: oauth_token  # Uses CLAUDE_CODE_OAUTH_TOKEN env var

Embedding Provider¶

When using model.provider: anthropic with vectorstore or hierarchical_document tools, you must define an embedding_provider at the agent level. This is because Anthropic does not provide embedding models — a separate provider is needed to generate embeddings for semantic search.

Required: When provider: anthropic AND using vectorstore/hierarchical_document tools
Type: LLM provider configuration object

# Using Ollama for embeddings (local, free)
embedding_provider:
  provider: ollama
  name: nomic-embed-text:latest
  endpoint: http://localhost:11434

# Or using OpenAI for embeddings
embedding_provider:
  provider: openai
  name: text-embedding-3-small

Note: embedding_provider is ignored when using openai, azure_openai, or ollama as the main provider, since those providers can generate embeddings natively.

Claude Agent SDK Settings¶

When using model.provider: anthropic, the agent.yaml accepts a top-level claude: section for backend-specific capabilities (permission modes, extended thinking, web search, subagents, bash, file_system, working_directory, max_turns, allowed_tools).

See Claude Backend for the full reference and examples. The claude block is ignored for non-Anthropic providers.

Instructions¶

Defines the system prompt that guides agent behavior.

Inline Instructions¶

Embed the prompt directly in agent.yaml:

instructions:
  inline: |
    You are a customer support specialist.

    Guidelines:
    - Be polite and professional
    - Provide accurate information
    - Escalate complex issues to supervisors

File-Based Instructions¶

Reference an external file (path relative to agent.yaml):

instructions:
  file: system_prompt.txt

File at system_prompt.txt:

You are a customer support specialist.

Guidelines:
- Be polite and professional
- Provide accurate information
- Escalate complex issues to supervisors

Rules¶

Exactly one required: Either inline OR file, not both
Max length (inline): 5000 characters
File path: Relative to agent.yaml directory (see File References guide)

Prompt Versioning (YAML Frontmatter)¶

When you use instructions.file, HoloDeck also parses optional YAML frontmatter at the top of the file to version and label the prompt. The frontmatter block is stripped before the body reaches the LLM — your prompt text is unchanged.

---
version: v1.2.0
author: your-name
description: Short note describing what this prompt version does.
tags:
  - rag
  - customer-support
---

# System Prompt

You are a customer support specialist.
...

Recognised keys (all optional):

Key	Type	Notes
`version`	string	Your manual version identifier. If omitted, HoloDeck derives `auto-<sha256[:8]>` from the prompt body so every run still has a stable id.
`author`	string	Free-form — who owns this version of the prompt.
`description`	string	One-liner shown in the dashboard.
`tags`	list of strings	Free-form tags surfaced as filter chips in the dashboard.

Any other keys are preserved under extra on the persisted run record — unknown keys don't error, they just don't populate the typed fields.

How it's used:

Every holodeck test run records the resolved PromptVersion on the eval-run JSON, including body_hash (full SHA-256 of the frontmatter-stripped body) for exact reproducibility.
The dashboard groups runs by version, draws vertical markers on the pass-rate chart whenever version changes between consecutive runs, and surfaces tags as filter chips.
Malformed YAML frontmatter fails the test run with a clear ConfigError — you cannot silently ship an unparseable prompt header.
Inline instructions (instructions.inline) skip frontmatter entirely and always resolve to version: auto-<sha256[:8]> with source: inline.

Response Format¶

Defines the expected structure of the agent's responses (agent-level only).

The response_format field constrains the LLM to generate structured output following a JSON Schema. This is useful for:

Ensuring consistent response structure
Integrating with downstream systems that expect specific formats
Validating response quality programmatically
Guiding the LLM toward well-formatted outputs

Inline Response Format¶

Define the schema directly in agent.yaml:

response_format:
  type: object
  properties:
    answer:
      type: string
      description: The answer to the user's question
    confidence:
      type: number
      description: Confidence score 0.0-1.0
    sources:
      type: array
      items:
        type: string
      description: References used in the answer
  required:
    - answer
    - confidence
  additionalProperties: false

File-Based Response Format¶

Reference an external JSON Schema file (path relative to agent.yaml):

response_format: schemas/response.json

File at schemas/response.json:

{
  "type": "object",
  "properties": {
    "answer": {
      "type": "string",
      "description": "The answer to the user's question"
    },
    "confidence": {
      "type": "number",
      "description": "Confidence score 0.0-1.0"
    },
    "sources": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "References used in the answer"
    }
  },
  "required": ["answer", "confidence"],
  "additionalProperties": false
}

Supported JSON Schema Keywords¶

Response format uses a Basic JSON Schema subset supporting:

type - Data type (string, number, integer, boolean, object, array, null)
properties - Object properties and their schemas
required - Array of required property names
items - Schema for array items
enum - List of allowed values
description - Documentation string
minimum - Minimum numeric value
maximum - Maximum numeric value
additionalProperties - Whether to allow extra properties (true|false)

Unsupported Keywords¶

The following keywords are not supported and will cause validation errors:

$ref - Schema references
anyOf - Multiple schema options
oneOf - Exactly one schema match
allOf - All schemas must match
patternProperties - Regex-based properties
minLength, maxLength - String length constraints
minItems, maxItems - Array length constraints
And other JSON Schema draft keywords

Rules¶

Optional: Response format is not required
Agent-level only: Not inherited from global configuration
Validated at load time: Invalid schemas cause errors with clear messages
File path: Relative to agent.yaml directory

Validation¶

Invalid response formats will produce clear error messages:

Error: Invalid JSON in response_format field
File: agent.yaml
Line: 42
Details: Missing required property: 'answer'

Error: Unknown JSON Schema keyword: '$ref'
File: schemas/response.json
Details: Keyword '$ref' is not supported. Use basic JSON Schema keywords only.

Tools¶

Define capabilities the agent can use. See the Tools Reference Guide for detailed documentation.

tools:
  - name: search-docs
    description: Search company documentation
    type: vectorstore
    source: docs/

  - name: get-user
    description: Retrieve user information
    type: function
    file: tools/user_tools.py
    function: get_user

  - name: file-system
    description: File system access
    type: mcp
    command: npx
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/data"]
    config:
      allowed_directories: ["/data", "/tmp"]

  - name: summarize
    description: Summarize text
    type: prompt
    template: "Summarize this in 2-3 sentences: {{text}}"
    parameters:
      text:
        type: string
        description: Text to summarize

Tool Constraints¶

Max tools: 50 per agent
Tool names: Must be unique, alphanumeric + underscores
Required fields: name, description, type

Evaluations¶

Defines metrics to measure agent quality. See the Evaluations Guide for details.

evaluations:
  model:  # Optional: default model for all metrics
    provider: openai
    name: gpt-4o

  metrics:
    - metric: groundedness
      threshold: 0.8
      enabled: true

    - metric: relevance
      threshold: 0.75

Test Cases¶

Defines scenarios to validate agent behavior.

test_cases:
  - name: "Support request"
    input: "How do I reset my password?"
    expected_tools: [search-docs]
    ground_truth: "Instructions for password reset"

  - input: "What are your hours?"
    expected_tools: []

Test Case Fields¶

name: Test identifier (optional)
input: User query (required, max 5000 chars)
expected_tools: Tools that should be called (optional)
ground_truth: Expected response for comparison (optional)
files: Multimodal inputs like images, PDFs (optional, max 10 per test)

Constraints¶

Max test cases: 100 per agent
Test names: Must be unique if provided

Complete Example¶

name: support-agent
description: Handles customer support queries with knowledge base search

model:
  provider: azure_openai
  name: gpt-4o
  temperature: 0.7
  max_tokens: 2000

instructions:
  file: system_prompt.txt

response_format:
  type: object
  properties:
    answer:
      type: string
      description: The support response
    confidence:
      type: number
      description: How confident we are in this answer
    escalation_needed:
      type: boolean
      description: Whether the issue should be escalated
  required:
    - answer
    - confidence

tools:
  - name: search-kb
    description: Search knowledge base
    type: vectorstore
    source: knowledge_base.json
    chunk_size: 500

  - name: create-ticket
    description: Create support ticket
    type: function
    file: tools/support.py
    function: create_ticket
    parameters:
      title:
        type: string
        description: Ticket title
      priority:
        type: string
        description: low|medium|high

evaluations:
  model:
    provider: azure_openai
    name: gpt-4o
    temperature: 0.0

  metrics:
    - metric: f1_score
      threshold: 0.8
    - metric: bleu
      threshold: 0.75

test_cases:
  - name: "Password reset"
    input: "How do I reset my password?"
    expected_tools: [search-kb]
    ground_truth: "Step-by-step password reset instructions"
    evaluations:
      - f1_score
      - bleu

  - name: "Open ticket"
    input: "I need help with my account"
    expected_tools: [search-kb, create-ticket]
    ground_truth: "Ticket created and knowledge base searched"
    evaluations:
      - f1_score

Validation Rules¶

Required Fields¶

name: Must be provided
model.provider: Must be provided and valid
model.name: Must be provided
instructions: Must have either inline or file

Mutual Exclusivity¶

Instructions: Either inline OR file, not both
Prompt tools: Either template OR file, not both

Ranges¶

Temperature: 0.0 to 2.0
Max tokens: Must be > 0
Tool limit: Max 50 per agent
Test cases: Max 100 per agent

Response Format Validation¶

response_format: Optional
Must be valid JSON Schema (if inline) or valid JSON file (if external)
Only Basic JSON Schema keywords supported
Invalid schemas produce clear error messages with file location and details
Not inherited from global configuration (agent-level only)

File References¶

Paths are relative to agent.yaml directory
Files must exist (checked during loading)
Absolute paths are supported

Environment Variables¶

Replace sensitive values with environment variables:

model:
  provider: openai
  name: gpt-4o
  # API key from environment, see global config guide

See the Global Configuration Guide for environment variable interpolation details.

Common Patterns¶

Minimal Agent (Inline Instructions)¶

name: simple-agent

model:
  provider: openai
  name: gpt-4o

instructions:
  inline: "You are a helpful assistant."

Agent with Structured Output (Response Format)¶

name: structured-agent
description: Returns structured responses

model:
  provider: openai
  name: gpt-4o

instructions:
  inline: "Answer questions in the specified format."

response_format:
  type: object
  properties:
    answer:
      type: string
    confidence:
      type: number
      minimum: 0
      maximum: 1
  required:
    - answer
    - confidence

Agent with File References¶

name: documented-agent

model:
  provider: azure_openai
  name: gpt-4

instructions:
  file: prompts/system.txt

response_format: schemas/output.json

tools:
  - name: search
    type: vectorstore
    source: data/docs.json

Full-Featured Agent¶

name: enterprise-agent
description: Production-ready support agent

model:
  provider: azure_openai
  name: gpt-4o
  temperature: 0.6
  max_tokens: 4096

instructions:
  file: system_prompt.txt

response_format:
  type: object
  properties:
    response:
      type: string
    sources:
      type: array
      items:
        type: string
    confidence:
      type: number
      minimum: 0
      maximum: 1
  required:
    - response

tools:
  - name: knowledge-base
    type: vectorstore
    source: kb/
  - name: system-check
    type: function
    file: tools/system.py
    function: check_status
  - name: external-api
    type: mcp
    command: npx
    args: ["-y", "custom-mcp-server"]

evaluations:
  model:
    provider: azure_openai
    name: gpt-4o
    temperature: 0.0

  metrics:
    - metric: f1_score
      threshold: 0.85
    - metric: bleu
      threshold: 0.8

test_cases:
  - name: "Basic query"
    input: "Hello, can you help?"
    expected_tools: [knowledge-base]
    ground_truth: "Yes, I'm here to help with your support request"
    evaluations:
      - f1_score
      - bleu

Troubleshooting¶

Error: "name is required"¶

Add name field at top level

Error: "instructions: Either file or inline must be provided"¶

Ensure instructions section has either inline or file field

Error: "instruction file not found"¶

Check file path is correct and relative to agent.yaml directory
Use absolute paths if needed

Error: "Invalid model provider"¶

Use valid provider: openai, azure_openai, anthropic, or ollama

Error: "Tool name must be unique"¶

Each tool must have a unique name field

Error: "Invalid JSON in response_format field"¶

Ensure response_format is valid JSON (if inline) or references a valid JSON file
Check for missing quotes, commas, or brackets in JSON

Error: "response_format file not found"¶

Check the file path is correct and relative to agent.yaml directory
File must exist and be readable

Error: "Unknown JSON Schema keyword: '$ref'"¶

Remove unsupported JSON Schema keywords ($ref, anyOf, oneOf, allOf, patternProperties, etc.)
Use only Basic JSON Schema keywords: type, properties, required, items, enum, description, minimum, maximum, additionalProperties

Next Steps¶

See Tools Reference for tool configuration details
See Evaluations Guide for quality metrics
See Global Configuration for shared settings
See File References for path resolution