Data Models API Reference

HoloDeck uses Pydantic models for all configuration validation. This section documents the complete data model hierarchy used throughout the platform.

Agent Models

Core agent configuration and instruction models.

Agent

Bases: BaseModel

Agent configuration entity.

Root configuration for a single AI agent instance, defining model, instructions, tools, evaluations, and test cases.

name = Field(default=..., description='Agent identifier') class-attribute instance-attribute

description = Field(default=None, description='Human-readable description') class-attribute instance-attribute

instructions = Field(default=..., description='System instructions (file or inline)') class-attribute instance-attribute

model = Field(default=..., description='LLM provider configuration') class-attribute instance-attribute

tools = Field(default=None, description='Agent tools (vectorstore, function, mcp, prompt)') class-attribute instance-attribute

evaluations = Field(default=None, description='Evaluation configuration') class-attribute instance-attribute

test_cases = Field(default=None, description='Test scenarios') class-attribute instance-attribute

Instructions

Bases: BaseModel

System instructions specification (file or inline).

Represents the system prompt for an agent, supporting both file references and inline text.

model_post_init(__context)

Validate file and inline mutual exclusivity.

Source code in src/holodeck/models/agent.py

def model_post_init(self, __context: Any) -> None:
    """Validate file and inline mutual exclusivity."""
    if self.file and self.inline:
        raise ValueError("Cannot provide both 'file' and 'inline'")
    if not self.file and not self.inline:
        raise ValueError("Must provide either 'file' or 'inline'")

validate_file(v) classmethod

Validate file is not empty if provided.

Source code in src/holodeck/models/agent.py

@field_validator("file")
@classmethod
def validate_file(cls, v: str | None) -> str | None:
    """Validate file is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("file must be non-empty if provided")
    return v

validate_inline(v) classmethod

Validate inline is not empty if provided.

Source code in src/holodeck/models/agent.py

@field_validator("inline")
@classmethod
def validate_inline(cls, v: str | None) -> str | None:
    """Validate inline is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("inline must be non-empty if provided")
    return v

LLM Provider Models

Language model provider configuration for OpenAI, Azure OpenAI, and Anthropic.

ProviderEnum

Bases: str, Enum

Supported LLM providers.

LLMProvider

Bases: BaseModel

LLM provider configuration.

Specifies which LLM provider and model to use, along with model parameters.

provider = Field(..., description='LLM provider') class-attribute instance-attribute

temperature = Field(default=0.3, description='Temperature (0.0-2.0)') class-attribute instance-attribute

max_tokens = Field(default=1000, description='Maximum tokens to generate') class-attribute instance-attribute

top_p = Field(default=None, description='Nucleus sampling parameter') class-attribute instance-attribute

Tool Models

Five types of tools are supported: vectorstore, function, MCP, prompt, and plugins.

Tool

Bases: BaseModel

Base tool model with discriminated union for subtypes.

VectorstoreTool

Bases: BaseModel

Vectorstore tool for semantic search over documents.

validate_chunk_overlap(v) classmethod

Validate chunk_overlap is non-negative if provided.

Source code in src/holodeck/models/tool.py

@field_validator("chunk_overlap")
@classmethod
def validate_chunk_overlap(cls, v: int | None) -> int | None:
    """Validate chunk_overlap is non-negative if provided."""
    if v is not None and v < 0:
        raise ValueError("chunk_overlap must be non-negative")
    return v

validate_chunk_size(v) classmethod

Validate chunk_size is positive if provided.

Source code in src/holodeck/models/tool.py

@field_validator("chunk_size")
@classmethod
def validate_chunk_size(cls, v: int | None) -> int | None:
    """Validate chunk_size is positive if provided."""
    if v is not None and v <= 0:
        raise ValueError("chunk_size must be positive")
    return v

validate_database(v) classmethod

Validate database is not empty string if provided as string.

Source code in src/holodeck/models/tool.py

@field_validator("database")
@classmethod
def validate_database(
    cls, v: DatabaseConfig | str | None
) -> DatabaseConfig | str | None:
    """Validate database is not empty string if provided as string."""
    if isinstance(v, str) and not v.strip():
        raise ValueError("database reference must be a non-empty string")
    return v

validate_embedding_dimensions(v) classmethod

Validate embedding_dimensions is positive and reasonable if provided.

Source code in src/holodeck/models/tool.py

@field_validator("embedding_dimensions")
@classmethod
def validate_embedding_dimensions(cls, v: int | None) -> int | None:
    """Validate embedding_dimensions is positive and reasonable if provided."""
    if v is not None:
        if v <= 0:
            raise ValueError("embedding_dimensions must be positive")
        if v > 10000:
            raise ValueError("embedding_dimensions unreasonably large (max 10000)")
    return v

validate_min_similarity_score(v) classmethod

Validate min_similarity_score is between 0.0 and 1.0 if provided.

Source code in src/holodeck/models/tool.py

@field_validator("min_similarity_score")
@classmethod
def validate_min_similarity_score(cls, v: float | None) -> float | None:
    """Validate min_similarity_score is between 0.0 and 1.0 if provided."""
    if v is not None and not (0.0 <= v <= 1.0):
        raise ValueError("min_similarity_score must be between 0.0 and 1.0")
    return v

validate_source(v) classmethod

Validate source is not empty.

Source code in src/holodeck/models/tool.py

@field_validator("source")
@classmethod
def validate_source(cls, v: str) -> str:
    """Validate source is not empty."""
    if not v or not v.strip():
        raise ValueError("source must be a non-empty path")
    return v

validate_top_k(v) classmethod

Validate top_k is a positive integer.

Source code in src/holodeck/models/tool.py

@field_validator("top_k")
@classmethod
def validate_top_k(cls, v: int) -> int:
    """Validate top_k is a positive integer."""
    if v <= 0:
        raise ValueError("top_k must be a positive integer")
    if v > 100:
        raise ValueError("top_k should not exceed 100")
    return v

FunctionTool

Bases: BaseModel

Function tool for calling Python functions.

validate_file(v) classmethod

Validate file is not empty.

Source code in src/holodeck/models/tool.py

@field_validator("file")
@classmethod
def validate_file(cls, v: str) -> str:
    """Validate file is not empty."""
    if not v or not v.strip():
        raise ValueError("file must be a non-empty path")
    return v

validate_function(v) classmethod

Validate function is not empty.

Source code in src/holodeck/models/tool.py

@field_validator("function")
@classmethod
def validate_function(cls, v: str) -> str:
    """Validate function is not empty."""
    if not v or not v.strip():
        raise ValueError("function must be a non-empty identifier")
    return v

MCPTool

Bases: BaseModel

MCP (Model Context Protocol) tool for standardized integrations.

Supports four transport types: - stdio (default): Local MCP servers via subprocess - sse: Remote servers via Server-Sent Events - websocket: Bidirectional WebSocket communication - http: Streamable HTTP transport

For stdio transport, only npx, uvx, or docker commands are allowed for security reasons.

validate_request_timeout(v) classmethod

Validate request_timeout is positive.

Source code in src/holodeck/models/tool.py

@field_validator("request_timeout")
@classmethod
def validate_request_timeout(cls, v: int) -> int:
    """Validate request_timeout is positive."""
    if v <= 0:
        raise ValueError("request_timeout must be positive")
    return v

validate_transport_fields()

Validate transport-specific required fields.

• stdio transport requires 'command'
• sse, websocket, http transports require 'url'

Source code in src/holodeck/models/tool.py

@model_validator(mode="after")
def validate_transport_fields(self) -> "MCPTool":
    """Validate transport-specific required fields.

    - stdio transport requires 'command'
    - sse, websocket, http transports require 'url'
    """
    if self.transport == TransportType.STDIO:
        if self.command is None:
            raise ValueError("'command' is required for stdio transport")
    elif (
        self.transport
        in (TransportType.SSE, TransportType.WEBSOCKET, TransportType.HTTP)
        and self.url is None
    ):
        raise ValueError(f"'url' is required for {self.transport.value} transport")
    return self

validate_url_scheme(v) classmethod

Validate URL scheme for HTTP-based transports.

Allows http:// only for localhost, requires https:// for remote URLs. WebSocket URLs can use wss:// or ws://.

Source code in src/holodeck/models/tool.py

@field_validator("url")
@classmethod
def validate_url_scheme(cls, v: str | None) -> str | None:
    """Validate URL scheme for HTTP-based transports.

    Allows http:// only for localhost, requires https:// for remote URLs.
    WebSocket URLs can use wss:// or ws://.
    """
    if v is None:
        return v
    # Allow http:// for localhost, require https:// otherwise
    if v.startswith("http://"):
        localhost_prefixes = (
            "http://localhost",
            "http://127.0.0.1",
            "http://[::1]",
        )
        if not any(v.startswith(prefix) for prefix in localhost_prefixes):
            raise ValueError("'url' must use https:// (or http:// for localhost)")
    elif not v.startswith(("https://", "wss://", "ws://")):
        raise ValueError("'url' must use https://, wss://, or ws:// scheme")
    return v

PromptTool

Bases: BaseModel

Prompt-based tool for AI-powered semantic functions.

check_template_or_file(v, info) classmethod

Validate that exactly one of template or file is provided.

Source code in src/holodeck/models/tool.py

@field_validator("parameters", mode="before")
@classmethod
def check_template_or_file(cls, v: Any, info: Any) -> Any:
    """Validate that exactly one of template or file is provided."""
    data = info.data
    template = data.get("template")
    file_path = data.get("file")

    if not template and not file_path:
        raise ValueError("Exactly one of 'template' or 'file' must be provided")
    if template and file_path:
        raise ValueError("Cannot provide both 'template' and 'file'")

    return v

validate_file(v) classmethod

Validate file is not empty if provided.

Source code in src/holodeck/models/tool.py

@field_validator("file")
@classmethod
def validate_file(cls, v: str | None) -> str | None:
    """Validate file is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("file must be non-empty if provided")
    return v

validate_parameters(v) classmethod

Validate parameters is not empty.

Source code in src/holodeck/models/tool.py

@field_validator("parameters")
@classmethod
def validate_parameters(
    cls, v: dict[str, dict[str, Any]]
) -> dict[str, dict[str, Any]]:
    """Validate parameters is not empty."""
    if not v:
        raise ValueError("parameters must have at least one parameter")
    return v

validate_template(v) classmethod

Validate template is not empty if provided.

Source code in src/holodeck/models/tool.py

@field_validator("template")
@classmethod
def validate_template(cls, v: str | None) -> str | None:
    """Validate template is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("template must be non-empty if provided")
    return v

Evaluation Models

Metrics and evaluation framework configuration.

EvaluationMetric

Bases: BaseModel

Evaluation metric configuration.

Represents a single evaluation metric with flexible model configuration, including per-metric LLM model overrides.

validate_custom_prompt(v) classmethod

Validate custom_prompt is not empty if provided.

Source code in src/holodeck/models/evaluation.py

@field_validator("custom_prompt")
@classmethod
def validate_custom_prompt(cls, v: str | None) -> str | None:
    """Validate custom_prompt is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("custom_prompt must be non-empty if provided")
    return v

validate_enabled(v) classmethod

Validate enabled is boolean.

Source code in src/holodeck/models/evaluation.py

@field_validator("enabled")
@classmethod
def validate_enabled(cls, v: bool) -> bool:
    """Validate enabled is boolean."""
    if not isinstance(v, bool):
        raise ValueError("enabled must be boolean")
    return v

validate_fail_on_error(v) classmethod

Validate fail_on_error is boolean.

Source code in src/holodeck/models/evaluation.py

@field_validator("fail_on_error")
@classmethod
def validate_fail_on_error(cls, v: bool) -> bool:
    """Validate fail_on_error is boolean."""
    if not isinstance(v, bool):
        raise ValueError("fail_on_error must be boolean")
    return v

validate_metric(v) classmethod

Validate metric is not empty.

Source code in src/holodeck/models/evaluation.py

@field_validator("metric")
@classmethod
def validate_metric(cls, v: str) -> str:
    """Validate metric is not empty."""
    if not v or not v.strip():
        raise ValueError("metric must be a non-empty string")
    return v

validate_retry_on_failure(v) classmethod

Validate retry_on_failure is in valid range.

Source code in src/holodeck/models/evaluation.py

@field_validator("retry_on_failure")
@classmethod
def validate_retry_on_failure(cls, v: int | None) -> int | None:
    """Validate retry_on_failure is in valid range."""
    if v is not None and (v < 1 or v > 3):
        raise ValueError("retry_on_failure must be between 1 and 3")
    return v

validate_scale(v) classmethod

Validate scale is positive.

Source code in src/holodeck/models/evaluation.py

@field_validator("scale")
@classmethod
def validate_scale(cls, v: int | None) -> int | None:
    """Validate scale is positive."""
    if v is not None and v <= 0:
        raise ValueError("scale must be positive")
    return v

validate_threshold(v) classmethod

Validate threshold is numeric if provided.

Source code in src/holodeck/models/evaluation.py

@field_validator("threshold")
@classmethod
def validate_threshold(cls, v: float | None) -> float | None:
    """Validate threshold is numeric if provided."""
    if v is not None and not isinstance(v, int | float):
        raise ValueError("threshold must be numeric")
    return v

validate_timeout_ms(v) classmethod

Validate timeout_ms is positive.

Source code in src/holodeck/models/evaluation.py

@field_validator("timeout_ms")
@classmethod
def validate_timeout_ms(cls, v: int | None) -> int | None:
    """Validate timeout_ms is positive."""
    if v is not None and v <= 0:
        raise ValueError("timeout_ms must be positive")
    return v

EvaluationConfig

Bases: BaseModel

Evaluation framework configuration.

Container for evaluation metrics with optional default model configuration. Supports standard EvaluationMetric, GEvalMetric (custom criteria), and RAGMetric (RAG pipeline evaluation).

model = Field(None, description='Default LLM model for all metrics') class-attribute instance-attribute

metrics = Field(..., description='List of metrics to evaluate (standard, GEval, or RAG)') class-attribute instance-attribute

Test Case Models

Test case definitions with multimodal file input support.

FileInput

Bases: BaseModel

File input for multimodal test cases.

Represents a single file reference for test case inputs, supporting both local files and remote URLs with optional extraction parameters.

check_path_or_url(v, info) classmethod

Validate that exactly one of path or url is provided.

Source code in src/holodeck/models/test_case.py

@field_validator("path", "url", mode="before")
@classmethod
def check_path_or_url(cls, v: Any, info: Any) -> Any:
    """Validate that exactly one of path or url is provided."""
    # This runs before validation, so we check in root_validator
    return v

model_post_init(__context)

Validate path and url mutual exclusivity after initialization.

Source code in src/holodeck/models/test_case.py

def model_post_init(self, __context: Any) -> None:
    """Validate path and url mutual exclusivity after initialization."""
    if self.path and self.url:
        raise ValueError("Cannot provide both 'path' and 'url'")
    if not self.path and not self.url:
        raise ValueError("Must provide either 'path' or 'url'")

validate_pages(v) classmethod

Validate pages are positive integers.

Source code in src/holodeck/models/test_case.py

@field_validator("pages")
@classmethod
def validate_pages(cls, v: list[int] | None) -> list[int] | None:
    """Validate pages are positive integers."""
    if v is not None and not all(isinstance(p, int) and p > 0 for p in v):
        raise ValueError("pages must be positive integers")
    return v

validate_type(v) classmethod

Validate file type is supported.

Source code in src/holodeck/models/test_case.py

@field_validator("type")
@classmethod
def validate_type(cls, v: str) -> str:
    """Validate file type is supported."""
    valid_types = {"image", "pdf", "text", "excel", "word", "powerpoint", "csv"}
    if v not in valid_types:
        raise ValueError(f"type must be one of {valid_types}, got {v}")
    return v

TestCaseModel

Bases: BaseModel

Test case for agent evaluation.

Represents a single test scenario with input, optional expected output, expected tool usage, multimodal file inputs, and RAG context.

validate_files(v) classmethod

Validate files list is not empty if provided.

Source code in src/holodeck/models/test_case.py

@field_validator("files")
@classmethod
def validate_files(cls, v: list[FileInput] | None) -> list[FileInput] | None:
    """Validate files list is not empty if provided."""
    if v is not None and len(v) > 10:
        raise ValueError("Maximum 10 files per test case")
    return v

validate_ground_truth(v) classmethod

Validate ground_truth is not empty if provided.

Source code in src/holodeck/models/test_case.py

@field_validator("ground_truth")
@classmethod
def validate_ground_truth(cls, v: str | None) -> str | None:
    """Validate ground_truth is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("ground_truth must be non-empty if provided")
    return v

validate_input(v) classmethod

Validate input is not empty.

Source code in src/holodeck/models/test_case.py

@field_validator("input")
@classmethod
def validate_input(cls, v: str) -> str:
    """Validate input is not empty."""
    if not v or not v.strip():
        raise ValueError("input must be a non-empty string")
    return v

validate_name(v) classmethod

Validate name is not empty if provided.

Source code in src/holodeck/models/test_case.py

@field_validator("name")
@classmethod
def validate_name(cls, v: str | None) -> str | None:
    """Validate name is not empty if provided."""
    if v is not None and (not v or not v.strip()):
        raise ValueError("name must be non-empty if provided")
    return v

TestCase = TestCaseModel module-attribute

Global Configuration Models

Project-wide settings for vectorstore, deployment, and execution.

VectorstoreConfig

Bases: BaseModel

Vectorstore configuration for global defaults.

Specifies connection details and options for a specific vectorstore backend.

validate_connection_string(v) classmethod

Validate connection_string is not empty.

Source code in src/holodeck/models/config.py

@field_validator("connection_string")
@classmethod
def validate_connection_string(cls, v: str) -> str:
    """Validate connection_string is not empty."""
    if not v or not v.strip():
        raise ValueError("connection_string must be a non-empty string")
    return v

validate_provider(v) classmethod

Validate provider is not empty.

Source code in src/holodeck/models/config.py

@field_validator("provider")
@classmethod
def validate_provider(cls, v: str) -> str:
    """Validate provider is not empty."""
    if not v or not v.strip():
        raise ValueError("provider must be a non-empty string")
    return v

DeploymentConfig

Bases: BaseModel

Deployment configuration for global defaults.

Specifies deployment platform and settings for deploying agents.

validate_type(v) classmethod

Validate type is not empty.

Source code in src/holodeck/models/config.py

@field_validator("type")
@classmethod
def validate_type(cls, v: str) -> str:
    """Validate type is not empty."""
    if not v or not v.strip():
        raise ValueError("type must be a non-empty string")
    return v

GlobalConfig

Bases: BaseModel

Global configuration entity.

Configuration stored in ~/.holodeck/config.yaml for sharing defaults across multiple agents, including LLM providers, vectorstores, execution, and deployment settings.

Test Result Models

Models for representing test execution results and reports.

TestResult

Bases: BaseModel

Result of executing a single test case.

Contains the test input, agent response, tool calls, metric results, and overall pass/fail status along with any errors encountered.

TestReport

Bases: BaseModel

Complete test execution report.

Contains all test results, summary statistics, and metadata about the test run including agent name, version, and environment.

validate_results_count()

Validate that summary total_tests matches results count.

Source code in src/holodeck/models/test_result.py

@model_validator(mode="after")
def validate_results_count(self) -> "TestReport":
    """Validate that summary total_tests matches results count."""
    if self.summary.total_tests != len(self.results):
        raise ValueError(
            f"summary.total_tests ({self.summary.total_tests}) "
            f"must match number of results ({len(self.results)})"
        )
    return self

Error Models

HoloDeck exception hierarchy for error handling.

HoloDeckError

Bases: Exception

Base exception for all HoloDeck errors.

All HoloDeck-specific exceptions inherit from this class, enabling centralized exception handling and error tracking.

ConfigError(field, message)

Bases: HoloDeckError

Exception raised for configuration errors.

This exception is raised when configuration loading or parsing fails. It includes field-specific information to help users identify and fix configuration issues.

Attributes:

Name	Type	Description
field		The configuration field that caused the error
message		Human-readable error message describing the issue

Initialize ConfigError with field and message.

Parameters:

Name	Type	Description	Default
field	str	Configuration field name where error occurred	required
message	str	Descriptive error message	required

Source code in src/holodeck/lib/errors.py

def __init__(self, field: str, message: str) -> None:
    """Initialize ConfigError with field and message.

    Args:
        field: Configuration field name where error occurred
        message: Descriptive error message
    """
    self.field = field
    self.message = message
    super().__init__(f"Configuration error in '{field}': {message}")

ValidationError(field, message, expected, actual)

Bases: HoloDeckError

Exception raised for validation errors during configuration parsing.

Provides detailed information about what was expected versus what was received, enabling users to quickly understand and fix validation issues.

Attributes:

Name	Type	Description
field		The field that failed validation
message		Description of the validation failure
expected		Human description of expected value/type
actual		The actual value that failed validation

Initialize ValidationError with detailed information.

Parameters:

Name	Type	Description	Default
field	str	Field that failed validation (can use dot notation for nested fields)	required
message	str	Description of what went wrong	required
expected	str	Human-readable description of expected value	required
actual	str	The actual value that failed	required

Source code in src/holodeck/lib/errors.py

def __init__(
    self,
    field: str,
    message: str,
    expected: str,
    actual: str,
) -> None:
    """Initialize ValidationError with detailed information.

    Args:
        field: Field that failed validation (can use dot notation for nested fields)
        message: Description of what went wrong
        expected: Human-readable description of expected value
        actual: The actual value that failed
    """
    self.field = field
    self.message = message
    self.expected = expected
    self.actual = actual
    full_message = (
        f"Validation error in '{field}': {message}\n"
        f"  Expected: {expected}\n"
        f"  Got: {actual}"
    )
    super().__init__(full_message)

FileNotFoundError(path, message)

Bases: HoloDeckError

Exception raised when a configuration file is not found.

Includes the file path and helpful suggestions for resolving the issue.

Attributes:

Name	Type	Description
path		Path to the file that was not found
message		Human-readable error message

Initialize FileNotFoundError with path and message.

Parameters:

Name	Type	Description	Default
path	str	Path to the file that was not found	required
message	str	Descriptive error message, optionally with suggestions	required

Source code in src/holodeck/lib/errors.py

def __init__(self, path: str, message: str) -> None:
    """Initialize FileNotFoundError with path and message.

    Args:
        path: Path to the file that was not found
        message: Descriptive error message, optionally with suggestions
    """
    self.path = path
    self.message = message
    super().__init__(f"File not found: {path}\n{message}")

Related Documentation

• Configuration Loading: How to load and validate configurations
• Test Runner: Test execution framework using these models
• Evaluation Framework: Evaluation system using these models