Skip to content

Data Models API Reference

HoloDeck uses Pydantic models for all configuration validation. This section documents the complete data model hierarchy used throughout the platform.

Agent Models

Core agent configuration and instruction models.

Agent

Bases: BaseModel

Agent configuration entity.

Root configuration for a single AI agent instance, defining model, instructions, tools, evaluations, and test cases.

name = Field(default=..., description='Agent identifier') class-attribute instance-attribute

description = Field(default=None, description='Human-readable description') class-attribute instance-attribute

instructions = Field(default=..., description='System instructions (file or inline)') class-attribute instance-attribute

model = Field(default=..., description='LLM provider configuration') class-attribute instance-attribute

tools = Field(default=None, description='Agent tools (vectorstore, function, mcp, prompt)') class-attribute instance-attribute

evaluations = Field(default=None, description='Evaluation configuration') class-attribute instance-attribute

test_cases = Field(default=None, description='Test scenarios') class-attribute instance-attribute

embedding_provider = Field(default=None, description='Provider for embedding generation. Required when provider: anthropic + vectorstore tools.') class-attribute instance-attribute

claude = Field(default=None, description='Claude Agent SDK-specific settings. Applicable only when model.provider: anthropic.') class-attribute instance-attribute

tool_filtering = Field(default=None, description='Automatic tool filtering configuration. When enabled, dynamically filters tools per request based on semantic similarity to reduce token usage.') class-attribute instance-attribute

Instructions

Bases: BaseModel

System instructions specification (file or inline).

Represents the system prompt for an agent, supporting both file references and inline text.

model_post_init(__context)

Validate file and inline mutual exclusivity.

Source code in src/holodeck/models/agent.py 
50
51
52
53
54
55
def model_post_init(self, __context: Any) -> None:
    """Validate file and inline mutual exclusivity."""
    if self.file and self.inline:
        raise ValueError("Cannot provide both 'file' and 'inline'")
    if not self.file and not self.inline:
        raise ValueError("Must provide either 'file' or 'inline'")

validate_file(v) classmethod

Validate file is not empty if provided.

Source code in src/holodeck/models/agent.py 
34
35
36
37
38
39
40
@field_validator("file")
@classmethod
def validate_file(cls, v: str | None) -> str | None:
    """Validate file is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("file must be non-empty if provided")
    return v

validate_inline(v) classmethod

Validate inline is not empty if provided.

Source code in src/holodeck/models/agent.py 
42
43
44
45
46
47
48
@field_validator("inline")
@classmethod
def validate_inline(cls, v: str | None) -> str | None:
    """Validate inline is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("inline must be non-empty if provided")
    return v

LLM Provider Models

Language model provider configuration for OpenAI, Azure OpenAI, and Anthropic.

ProviderEnum

Bases: str, Enum

Supported LLM providers.

LLMProvider

Bases: BaseModel

LLM provider configuration.

Specifies which LLM provider and model to use, along with model parameters.

provider = Field(..., description='LLM provider') class-attribute instance-attribute

temperature = Field(default=0.3, description='Temperature (0.0-2.0)') class-attribute instance-attribute

max_tokens = Field(default=1000, description='Maximum tokens to generate') class-attribute instance-attribute

top_p = Field(default=None, description='Nucleus sampling parameter') class-attribute instance-attribute

Claude Agent SDK Models

Configuration models for the Claude Agent SDK integration.

AuthProvider

Bases: str, Enum

Authentication method for Anthropic provider.

PermissionMode

Bases: str, Enum

Level of autonomous action for Claude Agent SDK.

ClaudeConfig

Bases: BaseModel

Claude Agent SDK-specific settings.

All fields optional. All capabilities default to disabled (least-privilege).

ExtendedThinkingConfig

Bases: BaseModel

Extended reasoning (deep thinking) configuration.

BashConfig

Bases: BaseModel

Shell command execution settings.

FileSystemConfig

Bases: BaseModel

File read/write/edit access settings.

SubagentConfig

Bases: BaseModel

Parallel sub-agent execution settings.

Backend Abstraction Models

Protocol types for the multi-backend architecture.

AgentBackend

Bases: Protocol

Provider backend factory.

Each backend encapsulates provider-specific initialisation logic and exposes a uniform surface for single-turn invocations (invoke_once) and stateful sessions (create_session). Callers must call initialize before any other method and teardown when done.

create_session() async

Create a new stateful multi-turn session.

Returns:

Type Description
AgentSession

A fresh AgentSession instance bound to this backend.

Raises:

Type Description
BackendInitError

If the backend was not initialised before calling.
BackendSessionError

If the session cannot be created.
Source code in src/holodeck/lib/backends/base.py 
123
124
125
126
127
128
129
130
131
132
133
async def create_session(self) -> AgentSession:
    """Create a new stateful multi-turn session.

    Returns:
        A fresh AgentSession instance bound to this backend.

    Raises:
        BackendInitError: If the backend was not initialised before calling.
        BackendSessionError: If the session cannot be created.
    """
    ...

initialize() async

Prepare the backend for use.

Raises:

Type Description
BackendInitError

If the backend cannot be initialised (e.g. missing API key, unavailable subprocess).
Source code in src/holodeck/lib/backends/base.py 
 94
 95
 96
 97
 98
 99
100
101
async def initialize(self) -> None:
    """Prepare the backend for use.

    Raises:
        BackendInitError: If the backend cannot be initialised (e.g.
            missing API key, unavailable subprocess).
    """
    ...

invoke_once(message, context=None) async

Execute a single stateless agent turn.

Parameters:

Name Type Description Default
message str

The user message to send to the agent.

 required
context list[dict[str, Any]] | None

Optional list of prior conversation turns.

 None

Returns:

Type Description
ExecutionResult

ExecutionResult containing the agent response and metadata.

Raises:

Type Description
BackendSessionError

If the invocation fails at runtime.
BackendTimeoutError

If the invocation exceeds configured timeout.
Source code in src/holodeck/lib/backends/base.py 
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
async def invoke_once(
    self,
    message: str,
    context: list[dict[str, Any]] | None = None,
) -> ExecutionResult:
    """Execute a single stateless agent turn.

    Args:
        message: The user message to send to the agent.
        context: Optional list of prior conversation turns.

    Returns:
        ExecutionResult containing the agent response and metadata.

    Raises:
        BackendSessionError: If the invocation fails at runtime.
        BackendTimeoutError: If the invocation exceeds configured timeout.
    """
    ...

teardown() async

Release all backend resources.

Source code in src/holodeck/lib/backends/base.py 
135
136
137
async def teardown(self) -> None:
    """Release all backend resources."""
    ...

AgentSession

Bases: Protocol

Stateful multi-turn conversation session.

Implementations maintain conversation history across multiple send calls. Callers must invoke close when the session is no longer needed to release any held resources (connections, subprocesses, etc.).

close() async

Release session resources (connections, subprocesses, etc.).

Source code in src/holodeck/lib/backends/base.py 
79
80
81
async def close(self) -> None:
    """Release session resources (connections, subprocesses, etc.)."""
    ...

send(message) async

Send a message and receive a single-turn result.

Parameters:

Name Type Description Default
message str

The user message to send to the agent.

 required

Returns:

Type Description
ExecutionResult

ExecutionResult containing the agent response and metadata.
Source code in src/holodeck/lib/backends/base.py 
56
57
58
59
60
61
62
63
64
65
async def send(self, message: str) -> ExecutionResult:
    """Send a message and receive a single-turn result.

    Args:
        message: The user message to send to the agent.

    Returns:
        ExecutionResult containing the agent response and metadata.
    """
    ...

send_streaming(message) async

Send a message and stream the agent response token by token.

Parameters:

Name Type Description Default
message str

The user message to send to the agent.

 required

Yields:

Type Description
AsyncGenerator[str, None]

Successive string chunks of the agent response.
Source code in src/holodeck/lib/backends/base.py 
67
68
69
70
71
72
73
74
75
76
77
async def send_streaming(self, message: str) -> AsyncGenerator[str, None]:
    """Send a message and stream the agent response token by token.

    Args:
        message: The user message to send to the agent.

    Yields:
        Successive string chunks of the agent response.
    """
    # Protocol stub — concrete implementations use `yield`
    yield ""  # pragma: no cover

ExecutionResult(response, tool_calls=list(), tool_results=list(), token_usage=TokenUsage.zero(), structured_output=None, num_turns=1, is_error=False, error_reason=None) dataclass

Provider-agnostic result of a single agent turn.

Attributes:

Name Type Description
response str

The text response from the agent.
tool_calls list[dict[str, Any]]

List of tool call records made during execution.
tool_results list[dict[str, Any]]

List of tool result records returned during execution.
token_usage TokenUsage

Token consumption metadata for this turn.
structured_output Any | None

Optional structured output from the agent.
num_turns int

Number of turns taken to produce this result.
is_error bool

Whether the execution ended in an error state.
error_reason str | None

Human-readable reason for the error, if any.

Tool Models

Six types of tools are supported: vectorstore, hierarchical document, function, MCP, prompt, and plugins.

Tool

Bases: BaseModel

Base tool model with discriminated union for subtypes.

VectorstoreTool

Bases: BaseModel

Vectorstore tool for semantic search over documents.

validate_chunk_overlap(v) classmethod

Validate chunk_overlap is non-negative if provided.

Source code in src/holodeck/models/tool.py 
352
353
354
355
356
357
358
@field_validator("chunk_overlap")
@classmethod
def validate_chunk_overlap(cls, v: int | None) -> int | None:
    """Validate chunk_overlap is non-negative if provided."""
    if v is not None and v < 0:
        raise ValueError("chunk_overlap must be non-negative")
    return v

validate_chunk_size(v) classmethod

Validate chunk_size is positive if provided.

Source code in src/holodeck/models/tool.py 
344
345
346
347
348
349
350
@field_validator("chunk_size")
@classmethod
def validate_chunk_size(cls, v: int | None) -> int | None:
    """Validate chunk_size is positive if provided."""
    if v is not None and v <= 0:
        raise ValueError("chunk_size must be positive")
    return v

validate_database(v) classmethod

Validate database is not empty string if provided as string.

Source code in src/holodeck/models/tool.py 
334
335
336
337
338
339
340
341
342
@field_validator("database")
@classmethod
def validate_database(
    cls, v: DatabaseConfig | str | None
) -> DatabaseConfig | str | None:
    """Validate database is not empty string if provided as string."""
    if isinstance(v, str) and not v.strip():
        raise ValueError("database reference must be a non-empty string")
    return v

validate_embedding_dimensions(v) classmethod

Validate embedding_dimensions is positive and reasonable if provided.

Source code in src/holodeck/models/tool.py 
378
379
380
381
382
383
384
385
386
387
@field_validator("embedding_dimensions")
@classmethod
def validate_embedding_dimensions(cls, v: int | None) -> int | None:
    """Validate embedding_dimensions is positive and reasonable if provided."""
    if v is not None:
        if v <= 0:
            raise ValueError("embedding_dimensions must be positive")
        if v > 10000:
            raise ValueError("embedding_dimensions unreasonably large (max 10000)")
    return v

validate_min_similarity_score(v) classmethod

Validate min_similarity_score is between 0.0 and 1.0 if provided.

Source code in src/holodeck/models/tool.py 
370
371
372
373
374
375
376
@field_validator("min_similarity_score")
@classmethod
def validate_min_similarity_score(cls, v: float | None) -> float | None:
    """Validate min_similarity_score is between 0.0 and 1.0 if provided."""
    if v is not None and not (0.0 <= v <= 1.0):
        raise ValueError("min_similarity_score must be between 0.0 and 1.0")
    return v

validate_source(v) classmethod

Validate source is not empty.

Source code in src/holodeck/models/tool.py 
326
327
328
329
330
331
332
@field_validator("source")
@classmethod
def validate_source(cls, v: str) -> str:
    """Validate source is not empty."""
    if not v or not v.strip():
        raise ValueError("source must be a non-empty path")
    return v

validate_structured_config()

Validate structured data configuration.

When vector_field is set (structured data mode), id_field becomes required to enable record identification for upserts and deduplication.

Source code in src/holodeck/models/tool.py 
312
313
314
315
316
317
318
319
320
321
322
323
324
@model_validator(mode="after")
def validate_structured_config(self) -> "VectorstoreTool":
    """Validate structured data configuration.

    When vector_field is set (structured data mode), id_field becomes required
    to enable record identification for upserts and deduplication.
    """
    if self.vector_field is not None and self.id_field is None:
        raise ValueError(
            "id_field is required when vector_field is specified "
            "(structured data mode)"
        )
    return self

validate_top_k(v) classmethod

Validate top_k is a positive integer.

Source code in src/holodeck/models/tool.py 
360
361
362
363
364
365
366
367
368
@field_validator("top_k")
@classmethod
def validate_top_k(cls, v: int) -> int:
    """Validate top_k is a positive integer."""
    if v <= 0:
        raise ValueError("top_k must be a positive integer")
    if v > 100:
        raise ValueError("top_k should not exceed 100")
    return v

FunctionTool

Bases: BaseModel

Function tool for calling Python functions.

validate_file(v) classmethod

Validate file is not empty.

Source code in src/holodeck/models/tool.py 
411
412
413
414
415
416
417
@field_validator("file")
@classmethod
def validate_file(cls, v: str) -> str:
    """Validate file is not empty."""
    if not v or not v.strip():
        raise ValueError("file must be a non-empty path")
    return v

validate_function(v) classmethod

Validate function is not empty.

Source code in src/holodeck/models/tool.py 
419
420
421
422
423
424
425
@field_validator("function")
@classmethod
def validate_function(cls, v: str) -> str:
    """Validate function is not empty."""
    if not v or not v.strip():
        raise ValueError("function must be a non-empty identifier")
    return v

MCPTool

Bases: BaseModel

MCP (Model Context Protocol) tool for standardized integrations.

Supports four transport types: - stdio (default): Local MCP servers via subprocess - sse: Remote servers via Server-Sent Events - websocket: Bidirectional WebSocket communication - http: Streamable HTTP transport

For stdio transport, only npx, uvx, or docker commands are allowed for security reasons.

validate_request_timeout(v) classmethod

Validate request_timeout is positive.

Source code in src/holodeck/models/tool.py 
528
529
530
531
532
533
534
@field_validator("request_timeout")
@classmethod
def validate_request_timeout(cls, v: int) -> int:
    """Validate request_timeout is positive."""
    if v <= 0:
        raise ValueError("request_timeout must be positive")
    return v

validate_transport_fields()

Validate transport-specific required fields.

  • stdio transport requires 'command'
  • sse, websocket, http transports require 'url'
Source code in src/holodeck/models/tool.py 
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
@model_validator(mode="after")
def validate_transport_fields(self) -> "MCPTool":
    """Validate transport-specific required fields.

    - stdio transport requires 'command'
    - sse, websocket, http transports require 'url'
    """
    if self.transport == TransportType.STDIO:
        if self.command is None:
            raise ValueError("'command' is required for stdio transport")
    elif (
        self.transport
        in (TransportType.SSE, TransportType.WEBSOCKET, TransportType.HTTP)
        and self.url is None
    ):
        raise ValueError(f"'url' is required for {self.transport.value} transport")
    return self

validate_url_scheme(v) classmethod

Validate URL scheme for HTTP-based transports.

Allows http:// only for localhost, requires https:// for remote URLs. WebSocket URLs can use wss:// or ws://.

Source code in src/holodeck/models/tool.py 
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
@field_validator("url")
@classmethod
def validate_url_scheme(cls, v: str | None) -> str | None:
    """Validate URL scheme for HTTP-based transports.

    Allows http:// only for localhost, requires https:// for remote URLs.
    WebSocket URLs can use wss:// or ws://.
    """
    if v is None:
        return v
    # Allow http:// for localhost, require https:// otherwise
    if v.startswith("http://"):
        localhost_prefixes = (
            "http://localhost",
            "http://127.0.0.1",
            "http://[::1]",
        )
        if not any(v.startswith(prefix) for prefix in localhost_prefixes):
            raise ValueError("'url' must use https:// (or http:// for localhost)")
    elif not v.startswith(("https://", "wss://", "ws://")):
        raise ValueError("'url' must use https://, wss://, or ws:// scheme")
    return v

PromptTool

Bases: BaseModel

Prompt-based tool for AI-powered semantic functions.

check_template_or_file(v, info) classmethod

Validate that exactly one of template or file is provided.

Source code in src/holodeck/models/tool.py 
603
604
605
606
607
608
609
610
611
612
613
614
615
616
@field_validator("parameters", mode="before")
@classmethod
def check_template_or_file(cls, v: Any, info: Any) -> Any:
    """Validate that exactly one of template or file is provided."""
    data = info.data
    template = data.get("template")
    file_path = data.get("file")

    if not template and not file_path:
        raise ValueError("Exactly one of 'template' or 'file' must be provided")
    if template and file_path:
        raise ValueError("Cannot provide both 'template' and 'file'")

    return v

validate_file(v) classmethod

Validate file is not empty if provided.

Source code in src/holodeck/models/tool.py 
585
586
587
588
589
590
591
@field_validator("file")
@classmethod
def validate_file(cls, v: str | None) -> str | None:
    """Validate file is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("file must be non-empty if provided")
    return v

validate_parameters(v) classmethod

Validate parameters is not empty.

Source code in src/holodeck/models/tool.py 
593
594
595
596
597
598
599
600
601
@field_validator("parameters")
@classmethod
def validate_parameters(
    cls, v: dict[str, dict[str, Any]]
) -> dict[str, dict[str, Any]]:
    """Validate parameters is not empty."""
    if not v:
        raise ValueError("parameters must have at least one parameter")
    return v

validate_template(v) classmethod

Validate template is not empty if provided.

Source code in src/holodeck/models/tool.py 
577
578
579
580
581
582
583
@field_validator("template")
@classmethod
def validate_template(cls, v: str | None) -> str | None:
    """Validate template is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("template must be non-empty if provided")
    return v

Evaluation Models

Metrics and evaluation framework configuration.

EvaluationMetric

Bases: BaseModel

Evaluation metric configuration.

Represents a single evaluation metric with flexible model configuration, including per-metric LLM model overrides.

validate_custom_prompt(v) classmethod

Validate custom_prompt is not empty if provided.

Source code in src/holodeck/models/evaluation.py 
121
122
123
124
125
126
127
@field_validator("custom_prompt")
@classmethod
def validate_custom_prompt(cls, v: str | None) -> str | None:
    """Validate custom_prompt is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("custom_prompt must be non-empty if provided")
    return v

validate_enabled(v) classmethod

Validate enabled is boolean.

Source code in src/holodeck/models/evaluation.py 
81
82
83
84
85
86
87
@field_validator("enabled")
@classmethod
def validate_enabled(cls, v: bool) -> bool:
    """Validate enabled is boolean."""
    if not isinstance(v, bool):
        raise ValueError("enabled must be boolean")
    return v

validate_fail_on_error(v) classmethod

Validate fail_on_error is boolean.

Source code in src/holodeck/models/evaluation.py 
89
90
91
92
93
94
95
@field_validator("fail_on_error")
@classmethod
def validate_fail_on_error(cls, v: bool) -> bool:
    """Validate fail_on_error is boolean."""
    if not isinstance(v, bool):
        raise ValueError("fail_on_error must be boolean")
    return v

validate_metric(v) classmethod

Validate metric is not empty.

Source code in src/holodeck/models/evaluation.py 
65
66
67
68
69
70
71
@field_validator("metric")
@classmethod
def validate_metric(cls, v: str) -> str:
    """Validate metric is not empty."""
    if not v or not v.strip():
        raise ValueError("metric must be a non-empty string")
    return v

validate_retry_on_failure(v) classmethod

Validate retry_on_failure is in valid range.

Source code in src/holodeck/models/evaluation.py 
 97
 98
 99
100
101
102
103
@field_validator("retry_on_failure")
@classmethod
def validate_retry_on_failure(cls, v: int | None) -> int | None:
    """Validate retry_on_failure is in valid range."""
    if v is not None and (v < 1 or v > 3):
        raise ValueError("retry_on_failure must be between 1 and 3")
    return v

validate_scale(v) classmethod

Validate scale is positive.

Source code in src/holodeck/models/evaluation.py 
113
114
115
116
117
118
119
@field_validator("scale")
@classmethod
def validate_scale(cls, v: int | None) -> int | None:
    """Validate scale is positive."""
    if v is not None and v <= 0:
        raise ValueError("scale must be positive")
    return v

validate_threshold(v) classmethod

Validate threshold is numeric if provided.

Source code in src/holodeck/models/evaluation.py 
73
74
75
76
77
78
79
@field_validator("threshold")
@classmethod
def validate_threshold(cls, v: float | None) -> float | None:
    """Validate threshold is numeric if provided."""
    if v is not None and not isinstance(v, int | float):
        raise ValueError("threshold must be numeric")
    return v

validate_timeout_ms(v) classmethod

Validate timeout_ms is positive.

Source code in src/holodeck/models/evaluation.py 
105
106
107
108
109
110
111
@field_validator("timeout_ms")
@classmethod
def validate_timeout_ms(cls, v: int | None) -> int | None:
    """Validate timeout_ms is positive."""
    if v is not None and v <= 0:
        raise ValueError("timeout_ms must be positive")
    return v

EvaluationConfig

Bases: BaseModel

Evaluation framework configuration.

Container for evaluation metrics with optional default model configuration. Supports standard EvaluationMetric, GEvalMetric (custom criteria), and RAGMetric (RAG pipeline evaluation).

model = Field(None, description='Default LLM model for all metrics') class-attribute instance-attribute

metrics = Field(..., description='List of metrics to evaluate (standard, GEval, or RAG)') class-attribute instance-attribute

Test Case Models

Test case definitions with multimodal file input support.

FileInput

Bases: BaseModel

File input for multimodal test cases.

Represents a single file reference for test case inputs, supporting both local files and remote URLs with optional extraction parameters.

check_path_or_url(v, info) classmethod

Validate that exactly one of path or url is provided.

Source code in src/holodeck/models/test_case.py 
38
39
40
41
42
43
@field_validator("path", "url", mode="before")
@classmethod
def check_path_or_url(cls, v: Any, info: Any) -> Any:
    """Validate that exactly one of path or url is provided."""
    # This runs before validation, so we check in root_validator
    return v

model_post_init(__context)

Validate path and url mutual exclusivity after initialization.

Source code in src/holodeck/models/test_case.py 
62
63
64
65
66
67
def model_post_init(self, __context: Any) -> None:
    """Validate path and url mutual exclusivity after initialization."""
    if self.path and self.url:
        raise ValueError("Cannot provide both 'path' and 'url'")
    if not self.path and not self.url:
        raise ValueError("Must provide either 'path' or 'url'")

validate_pages(v) classmethod

Validate pages are positive integers.

Source code in src/holodeck/models/test_case.py 
54
55
56
57
58
59
60
@field_validator("pages")
@classmethod
def validate_pages(cls, v: list[int] | None) -> list[int] | None:
    """Validate pages are positive integers."""
    if v is not None and not all(isinstance(p, int) and p > 0 for p in v):
        raise ValueError("pages must be positive integers")
    return v

validate_type(v) classmethod

Validate file type is supported.

Source code in src/holodeck/models/test_case.py 
45
46
47
48
49
50
51
52
@field_validator("type")
@classmethod
def validate_type(cls, v: str) -> str:
    """Validate file type is supported."""
    valid_types = {"image", "pdf", "text", "excel", "word", "powerpoint", "csv"}
    if v not in valid_types:
        raise ValueError(f"type must be one of {valid_types}, got {v}")
    return v

TestCaseModel

Bases: BaseModel

Test case for agent evaluation.

Represents a single test scenario with input, optional expected output, expected tool usage, multimodal file inputs, and RAG context.

validate_files(v) classmethod

Validate files list is not empty if provided.

Source code in src/holodeck/models/test_case.py 
117
118
119
120
121
122
123
@field_validator("files")
@classmethod
def validate_files(cls, v: list[FileInput] | None) -> list[FileInput] | None:
    """Validate files list is not empty if provided."""
    if v is not None and len(v) > 10:
        raise ValueError("Maximum 10 files per test case")
    return v

validate_ground_truth(v) classmethod

Validate ground_truth is not empty if provided.

Source code in src/holodeck/models/test_case.py 
109
110
111
112
113
114
115
@field_validator("ground_truth")
@classmethod
def validate_ground_truth(cls, v: str | None) -> str | None:
    """Validate ground_truth is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("ground_truth must be non-empty if provided")
    return v

validate_input(v) classmethod

Validate input is not empty.

Source code in src/holodeck/models/test_case.py 
93
94
95
96
97
98
99
@field_validator("input")
@classmethod
def validate_input(cls, v: str) -> str:
    """Validate input is not empty."""
    if not v or not v.strip():
        raise ValueError("input must be a non-empty string")
    return v

validate_name(v) classmethod

Validate name is not empty if provided.

Source code in src/holodeck/models/test_case.py 
101
102
103
104
105
106
107
@field_validator("name")
@classmethod
def validate_name(cls, v: str | None) -> str | None:
    """Validate name is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("name must be non-empty if provided")
    return v

TestCase = TestCaseModel module-attribute

Global Configuration Models

Project-wide settings for vectorstore, deployment, and execution.

VectorstoreConfig

Bases: BaseModel

Vectorstore configuration for global defaults.

Specifies connection details and options for a specific vectorstore backend.

validate_connection_string(v) classmethod

Validate connection_string is not empty.

Source code in src/holodeck/models/config.py 
42
43
44
45
46
47
48
@field_validator("connection_string")
@classmethod
def validate_connection_string(cls, v: str) -> str:
    """Validate connection_string is not empty."""
    if not v or not v.strip():
        raise ValueError("connection_string must be a non-empty string")
    return v

validate_provider(v) classmethod

Validate provider is not empty.

Source code in src/holodeck/models/config.py 
34
35
36
37
38
39
40
@field_validator("provider")
@classmethod
def validate_provider(cls, v: str) -> str:
    """Validate provider is not empty."""
    if not v or not v.strip():
        raise ValueError("provider must be a non-empty string")
    return v

DeploymentConfig

Bases: BaseModel

Main deployment configuration model.

Attributes:

Name Type Description
runtime RuntimeType

Runtime type (currently only container)
registry RegistryConfig

Container registry configuration
target CloudTargetConfig

Cloud deployment target configuration
protocol ProtocolType

API protocol type
port Annotated[int, Field(ge=1, le=65535)]

Container port to expose
health_check_path str

HTTP path for health checks (e.g., /health, /healthz)
environment dict[str, str]

Environment variables for the container
platform str

Target platform for container image (e.g., linux/amd64, linux/arm64)

GlobalConfig

Bases: BaseModel

Global configuration entity.

Configuration stored in ~/.holodeck/config.yaml for sharing defaults across multiple agents, including LLM providers, vectorstores, execution, and deployment settings.

Test Result Models

Models for representing test execution results and reports.

TestResult

Bases: BaseModel

Result of executing a single test case.

Contains the test input, agent response, tool calls, metric results, and overall pass/fail status along with any errors encountered.

TestReport

Bases: BaseModel

Complete test execution report.

Contains all test results, summary statistics, and metadata about the test run including agent name, version, and environment.

validate_results_count()

Validate that summary total_tests matches results count.

Source code in src/holodeck/models/test_result.py 
178
179
180
181
182
183
184
185
186
@model_validator(mode="after")
def validate_results_count(self) -> "TestReport":
    """Validate that summary total_tests matches results count."""
    if self.summary.total_tests != len(self.results):
        raise ValueError(
            f"summary.total_tests ({self.summary.total_tests}) "
            f"must match number of results ({len(self.results)})"
        )
    return self

Error Models

HoloDeck exception hierarchy for error handling.

HoloDeckError

Bases: Exception

Base exception for all HoloDeck errors.

All HoloDeck-specific exceptions inherit from this class, enabling centralized exception handling and error tracking.

ConfigError(field, message)

Bases: HoloDeckError

Exception raised for configuration errors.

This exception is raised when configuration loading or parsing fails. It includes field-specific information to help users identify and fix configuration issues.

Attributes:

Name Type Description
field

The configuration field that caused the error
message

Human-readable error message describing the issue

Initialize ConfigError with field and message.

Parameters:

Name Type Description Default
field str

Configuration field name where error occurred

 required
message str

Descriptive error message

 required
Source code in src/holodeck/lib/errors.py 
26
27
28
29
30
31
32
33
34
35
def __init__(self, field: str, message: str) -> None:
    """Initialize ConfigError with field and message.

    Args:
        field: Configuration field name where error occurred
        message: Descriptive error message
    """
    self.field = field
    self.message = message
    super().__init__(f"Configuration error in '{field}': {message}")

ValidationError(field, message, expected, actual)

Bases: HoloDeckError

Exception raised for validation errors during configuration parsing.

Provides detailed information about what was expected versus what was received, enabling users to quickly understand and fix validation issues.

Attributes:

Name Type Description
field

The field that failed validation
message

Description of the validation failure
expected

Human description of expected value/type
actual

The actual value that failed validation

Initialize ValidationError with detailed information.

Parameters:

Name Type Description Default
field str

Field that failed validation (can use dot notation for nested fields)

 required
message str

Description of what went wrong

 required
expected str

Human-readable description of expected value

 required
actual str

The actual value that failed

 required
Source code in src/holodeck/lib/errors.py 
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
def __init__(
    self,
    field: str,
    message: str,
    expected: str,
    actual: str,
) -> None:
    """Initialize ValidationError with detailed information.

    Args:
        field: Field that failed validation (can use dot notation for nested fields)
        message: Description of what went wrong
        expected: Human-readable description of expected value
        actual: The actual value that failed
    """
    self.field = field
    self.message = message
    self.expected = expected
    self.actual = actual
    full_message = (
        f"Validation error in '{field}': {message}\n"
        f"  Expected: {expected}\n"
        f"  Got: {actual}"
    )
    super().__init__(full_message)

FileNotFoundError(path, message)

Bases: HoloDeckError

Exception raised when a configuration file is not found.

Includes the file path and helpful suggestions for resolving the issue.

Attributes:

Name Type Description
path

Path to the file that was not found
message

Human-readable error message

Initialize FileNotFoundError with path and message.

Parameters:

Name Type Description Default
path str

Path to the file that was not found

 required
message str

Descriptive error message, optionally with suggestions

 required
Source code in src/holodeck/lib/errors.py 
88
89
90
91
92
93
94
95
96
97
def __init__(self, path: str, message: str) -> None:
    """Initialize FileNotFoundError with path and message.

    Args:
        path: Path to the file that was not found
        message: Descriptive error message, optionally with suggestions
    """
    self.path = path
    self.message = message
    super().__init__(f"File not found: {path}\n{message}")

BackendError

Bases: HoloDeckError

Base exception for all backend errors.

Catch this to handle any backend-related failure without needing to know the specific subtype.