CLI API Reference

HoloDeck provides a command-line interface for project initialization, agent testing, and configuration management. This section documents the programmatic CLI API.

Main CLI

Entry point for the HoloDeck CLI application using Click.

main(ctx)

HoloDeck - Experimentation platform for AI agents.

Commands

init Initialize a new agent project test Run agent test cases chat Interactive chat session with an agent

Initialize and manage AI agent projects with YAML configuration.

Source code in src/holodeck/cli/main.py

@click.group(invoke_without_command=True)
@click.version_option(version=__version__, prog_name="holodeck")
@click.pass_context
def main(ctx: click.Context) -> None:
    """HoloDeck - Experimentation platform for AI agents.

    Commands:
        init   Initialize a new agent project
        test   Run agent test cases
        chat   Interactive chat session with an agent

    Initialize and manage AI agent projects with YAML configuration.
    """
    # Show help if no command is provided
    if ctx.invoked_subcommand is None:
        click.echo(ctx.get_help())

CLI Commands

Init Command

Initialize a new HoloDeck project with bundled templates.

init(project_name, template, description, author, force, llm, vectorstore, evals_arg, mcp_arg, non_interactive)

Initialize a new HoloDeck agent project.

Creates a new project directory with all required configuration files, example instructions, tools templates, test cases, and data files.

The generated project includes agent.yaml (main configuration), instructions/ (system prompts), tools/ (custom function templates), data/ (sample datasets), and tests/ (evaluation test cases).

TEMPLATES:

conversational  - General-purpose conversational agent (default)
research        - Research/analysis agent with vector search examples
customer-support - Customer support agent with function tools

INTERACTIVE MODE (default):

When run without --non-interactive, the wizard prompts for:
- Agent name
- LLM provider (Ollama, OpenAI, Azure OpenAI, Anthropic)
- Vector store (ChromaDB, Qdrant, In-Memory)
- Evaluation metrics
- MCP servers

NON-INTERACTIVE MODE:

Use --non-interactive with --name to skip prompts and use defaults:

    holodeck init --name my-agent --non-interactive

Or override specific values:

    holodeck init --name my-agent --llm openai --vectorstore qdrant

EXAMPLES:

Basic project with interactive wizard:

    holodeck init

Quick setup with defaults (no prompts):

    holodeck init --name my-agent --non-interactive

Custom LLM and vector store:

    holodeck init --name my-agent --llm openai --vectorstore qdrant

Full customization without prompts:

    holodeck init --name my-agent --llm anthropic \
        --vectorstore chromadb --evals rag-faithfulness,rag-answer_relevancy \
        --mcp brave-search,memory --non-interactive

For more information, see: https://useholodeck.ai/docs/getting-started

Source code in src/holodeck/cli/commands/init.py

@click.command(name="init")
@click.option(
    "--name",
    "project_name",
    default=None,
    help="Agent/project name (required in non-interactive mode)",
)
@click.option(
    "--template",
    default="conversational",
    type=str,
    callback=validate_template,
    help="Project template: conversational (default), research, or customer-support",
)
@click.option(
    "--description",
    default=None,
    help="Brief description of what the agent does",
)
@click.option(
    "--author",
    default=None,
    help="Name of the project creator or organization",
)
@click.option(
    "--force",
    is_flag=True,
    help="Overwrite existing project directory without prompting",
)
@click.option(
    "--llm",
    type=click.Choice(sorted(VALID_LLM_PROVIDERS)),
    default=None,
    help="LLM provider (skips interactive prompt)",
)
@click.option(
    "--vectorstore",
    type=click.Choice(sorted(VALID_VECTOR_STORES)),
    default=None,
    help="Vector store (skips interactive prompt)",
)
@click.option(
    "--evals",
    "evals_arg",
    default=None,
    help="Comma-separated evaluation metrics (skips interactive prompt)",
)
@click.option(
    "--mcp",
    "mcp_arg",
    default=None,
    help="Comma-separated MCP servers (skips interactive prompt)",
)
@click.option(
    "--non-interactive",
    is_flag=True,
    help="Skip all interactive prompts (use defaults or flag values)",
)
def init(
    project_name: str | None,
    template: str,
    description: str | None,
    author: str | None,
    force: bool,
    llm: str | None,
    vectorstore: str | None,
    evals_arg: str | None,
    mcp_arg: str | None,
    non_interactive: bool,
) -> None:
    """Initialize a new HoloDeck agent project.

    Creates a new project directory with all required configuration files,
    example instructions, tools templates, test cases, and data files.

    The generated project includes agent.yaml (main configuration), instructions/
    (system prompts), tools/ (custom function templates), data/ (sample datasets),
    and tests/ (evaluation test cases).

    TEMPLATES:

        conversational  - General-purpose conversational agent (default)
        research        - Research/analysis agent with vector search examples
        customer-support - Customer support agent with function tools

    INTERACTIVE MODE (default):

        When run without --non-interactive, the wizard prompts for:
        - Agent name
        - LLM provider (Ollama, OpenAI, Azure OpenAI, Anthropic)
        - Vector store (ChromaDB, Qdrant, In-Memory)
        - Evaluation metrics
        - MCP servers

    NON-INTERACTIVE MODE:

        Use --non-interactive with --name to skip prompts and use defaults:

            holodeck init --name my-agent --non-interactive

        Or override specific values:

            holodeck init --name my-agent --llm openai --vectorstore qdrant

    EXAMPLES:

        Basic project with interactive wizard:

            holodeck init

        Quick setup with defaults (no prompts):

            holodeck init --name my-agent --non-interactive

        Custom LLM and vector store:

            holodeck init --name my-agent --llm openai --vectorstore qdrant

        Full customization without prompts:

            holodeck init --name my-agent --llm anthropic \\
                --vectorstore chromadb --evals rag-faithfulness,rag-answer_relevancy \\
                --mcp brave-search,memory --non-interactive

    For more information, see: https://useholodeck.ai/docs/getting-started
    """
    try:
        # Get current working directory as output directory
        output_dir = Path.cwd()

        # Parse comma-separated arguments
        evals_list = _parse_comma_arg(evals_arg)
        mcp_list = _parse_comma_arg(mcp_arg)

        # Validate evals if provided
        if evals_list:
            invalid_evals = [e for e in evals_list if e not in VALID_EVALS]
            if invalid_evals:
                valid = ", ".join(sorted(VALID_EVALS))
                invalid_str = ", ".join(invalid_evals)
                click.secho(
                    f"Warning: Invalid eval(s): {invalid_str}. Valid: {valid}",
                    fg="yellow",
                )
                evals_list = [e for e in evals_list if e in VALID_EVALS]

        # Validate MCP servers if provided
        if mcp_list:
            invalid_mcp = [s for s in mcp_list if s not in VALID_MCP_SERVERS]
            if invalid_mcp:
                valid = ", ".join(sorted(VALID_MCP_SERVERS))
                click.secho(
                    f"Warning: Invalid MCP server(s): {', '.join(invalid_mcp)}. "
                    f"Valid options: {valid}",
                    fg="yellow",
                )
                mcp_list = [s for s in mcp_list if s in VALID_MCP_SERVERS]

        # Determine if we should run wizard
        if non_interactive or not is_interactive():
            # Non-interactive mode: --name is required
            if not project_name:
                click.secho(
                    "Error: --name is required in non-interactive mode",
                    fg="red",
                )
                raise click.Abort()

            # Use defaults or flag values
            selected_llm = llm or "ollama"

            # Create provider config for providers that require endpoint
            provider_config = None
            if selected_llm == "azure_openai":
                # Use env var placeholders for Azure OpenAI
                provider_config = ProviderConfig(
                    endpoint="${AZURE_OPENAI_ENDPOINT}",
                )

            wizard_result = WizardResult(
                agent_name=project_name,
                template=template,
                llm_provider=selected_llm,
                provider_config=provider_config,
                vector_store=vectorstore or "chromadb",
                evals=evals_list if evals_list else get_default_evals(),
                mcp_servers=mcp_list if mcp_list else get_default_mcp_servers(),
            )
        else:
            # Interactive mode: run wizard
            # Skip template prompt if --template was provided (not default)
            wizard_result = run_wizard(
                skip_agent_name=project_name is not None,
                skip_template=template != "conversational",
                skip_llm=llm is not None,
                skip_vectorstore=vectorstore is not None,
                skip_evals=evals_arg is not None,
                skip_mcp=mcp_arg is not None,
                agent_name_default=project_name,
                template_default=template,
                llm_default=llm or "ollama",
                vectorstore_default=vectorstore or "chromadb",
                evals_defaults=evals_list if evals_list else None,
                mcp_defaults=mcp_list if mcp_list else None,
            )

        # Use agent_name from wizard result as project name
        final_project_name = wizard_result.agent_name

        # Check if project directory already exists (unless force)
        project_dir = output_dir / final_project_name
        if project_dir.exists() and not force:
            # Prompt user for confirmation
            if click.confirm(
                f"Project directory '{final_project_name}' already exists. "
                "Do you want to overwrite it?",
                default=False,
            ):
                force = True
            else:
                click.echo("Initialization cancelled.")
                return

        # Create project initialization input
        init_input = ProjectInitInput(
            project_name=final_project_name,
            template=wizard_result.template,
            description=description,
            author=author,
            output_dir=str(output_dir),
            overwrite=force,
            agent_name=wizard_result.agent_name,
            llm_provider=wizard_result.llm_provider,
            provider_config=wizard_result.provider_config,
            vector_store=wizard_result.vector_store,
            evals=wizard_result.evals,
            mcp_servers=wizard_result.mcp_servers,
        )

        # Initialize project
        initializer = ProjectInitializer()
        result = initializer.initialize(init_input)

        # Handle result
        if result.success:
            # Display success message
            click.echo()  # Blank line for readability
            click.secho("Project initialized successfully!", fg="green", bold=True)
            click.echo()
            click.echo(f"Project: {result.project_name}")
            click.echo(f"Location: {result.project_path}")
            click.echo(f"Template: {result.template_used}")
            click.echo()
            click.echo("Configuration:")
            click.echo(f"  Agent Name: {wizard_result.agent_name}")
            click.echo(f"  Template: {wizard_result.template}")
            click.echo(f"  LLM Provider: {wizard_result.llm_provider}")
            click.echo(f"  Vector Store: {wizard_result.vector_store}")
            click.echo(f"  Evals: {', '.join(wizard_result.evals) or 'none'}")
            click.echo(
                f"  MCP Servers: {', '.join(wizard_result.mcp_servers) or 'none'}"
            )
            click.echo()
            click.echo(f"Time: {result.duration_seconds:.2f}s")

            # Show created files (first 10, then summary)
            if result.files_created:
                click.echo()
                click.echo("Files created:")
                # Show key files first (config, instructions, tools, data)
                key_files = [
                    f
                    for f in result.files_created
                    if "agent.yaml" in f
                    or "system-prompt" in f
                    or "tools" in f
                    or "data" in f
                ]
                for file_path in key_files[:5]:
                    click.echo(f"  - {file_path}")
                if len(result.files_created) > 5:
                    remaining = len(result.files_created) - 5
                    click.echo(f"  ... and {remaining} more file(s)")

            click.echo()
            click.echo("Next steps:")
            click.echo(f"  1. cd {result.project_name}")
            click.echo("  2. Edit agent.yaml to configure your agent")
            click.echo("  3. Edit instructions/system-prompt.md to customize behavior")
            click.echo("  4. Add tools in tools/ directory")
            click.echo("  5. Update test_cases in agent.yaml")
            click.echo("  6. Run tests with: holodeck test agent.yaml")
            click.echo()
        else:
            # Display error message
            click.secho("Project initialization failed", fg="red", bold=True)
            click.echo()
            for error in result.errors:
                click.secho(f"Error: {error}", fg="red")
            click.echo()
            raise click.Abort()

    except WizardCancelledError as e:
        # Handle wizard cancellation gracefully
        click.echo()
        click.secho("Wizard cancelled.", fg="yellow")
        raise click.Abort() from e

    except KeyboardInterrupt as e:
        # Handle Ctrl+C gracefully with cleanup
        click.echo()
        click.secho("Initialization cancelled by user.", fg="yellow")
        raise click.Abort() from e

    except (ValidationError, InitError) as e:
        # Handle known errors
        click.secho(f"Error: {str(e)}", fg="red")
        raise click.Abort() from e

    except Exception as e:
        # Handle unexpected errors
        click.secho(f"Unexpected error: {str(e)}", fg="red")
        raise click.Abort() from e

Test Command

Run tests for a HoloDeck agent with evaluation and reporting.

test(agent_config, output, format, verbose, quiet, timeout, force_ingest)

Execute agent test cases with evaluation metrics.

Runs test cases defined in the agent configuration file and displays pass/fail status with evaluation metric scores.

AGENT_CONFIG is the path to the agent.yaml configuration file.

Source code in src/holodeck/cli/commands/test.py

@click.command()
@click.argument("agent_config", type=click.Path(exists=True))
@click.option(
    "--output",
    type=click.Path(),
    default=None,
    help="Path to save test report file (JSON or Markdown)",
)
@click.option(
    "--format",
    type=click.Choice(["json", "markdown"]),
    default=None,
    help="Report format (auto-detect from extension if not specified)",
)
@click.option(
    "--verbose",
    "-v",
    is_flag=True,
    help="Enable verbose output with debug information",
)
@click.option(
    "--quiet",
    "-q",
    is_flag=True,
    help="Suppress progress output (summary still shown)",
)
@click.option(
    "--timeout",
    type=int,
    default=None,
    help="LLM execution timeout in seconds",
)
@click.option(
    "--force-ingest",
    "-f",
    is_flag=True,
    help="Force re-ingestion of all vector store source files",
)
def test(
    agent_config: str,
    output: str | None,
    format: str | None,
    verbose: bool,
    quiet: bool,
    timeout: int | None,
    force_ingest: bool,
) -> None:
    """Execute agent test cases with evaluation metrics.

    Runs test cases defined in the agent configuration file and displays
    pass/fail status with evaluation metric scores.

    AGENT_CONFIG is the path to the agent.yaml configuration file.
    """
    # Reconfigure logging based on CLI flags
    # If verbose is enabled, it overrides quiet mode
    effective_quiet = quiet and not verbose
    setup_logging(verbose=verbose, quiet=effective_quiet)

    logger.info(
        f"Test command invoked: config={agent_config}, "
        f"verbose={verbose}, quiet={quiet}, timeout={timeout}, "
        f"force_ingest={force_ingest}"
    )

    start_time = time.time()

    try:
        # Create execution config from CLI options
        cli_config = None
        if timeout is not None or verbose or quiet:
            cli_config = ExecutionConfig(
                llm_timeout=timeout,
                file_timeout=None,
                download_timeout=None,
                cache_enabled=None,
                cache_dir=None,
                verbose=verbose or None,
                quiet=quiet or None,
            )

        # Load agent config to get test count for progress indicator
        from holodeck.config.context import agent_base_dir
        from holodeck.config.defaults import DEFAULT_EXECUTION_CONFIG
        from holodeck.config.loader import ConfigLoader

        logger.debug(f"Loading agent configuration from {agent_config}")
        loader = ConfigLoader()
        agent = loader.load_agent_yaml(agent_config)
        logger.info(f"Agent configuration loaded successfully: {agent.name}")

        # Set the base directory context for resolving relative paths in tools
        agent_dir = str(Path(agent_config).parent.resolve())
        agent_base_dir.set(agent_dir)
        logger.debug(f"Set agent_base_dir context: {agent_base_dir.get()}")

        # Resolve execution config once (CLI > agent.yaml > project > user > defaults)
        project_config = loader.load_project_config(agent_dir)
        project_execution = project_config.execution if project_config else None
        user_config = loader.load_global_config()
        user_execution = user_config.execution if user_config else None

        resolved_config = loader.resolve_execution_config(
            cli_config=cli_config,
            yaml_config=agent.execution,
            project_config=project_execution,
            user_config=user_execution,
            defaults=DEFAULT_EXECUTION_CONFIG,
        )
        logger.debug(
            f"Resolved execution config: verbose={resolved_config.verbose}, "
            f"quiet={resolved_config.quiet}, llm_timeout={resolved_config.llm_timeout}"
        )

        # Get total test count
        total_tests = len(agent.test_cases) if agent.test_cases else 0
        logger.info(f"Found {total_tests} test cases to execute")

        # Initialize progress indicator
        progress = ProgressIndicator(
            total_tests=total_tests, quiet=quiet, verbose=verbose
        )

        # Initialize spinner thread
        spinner_thread = SpinnerThread(progress)

        # Define callbacks
        def on_test_start(test_case: TestCaseModel) -> None:
            """Start spinner for new test."""
            progress.start_test(test_case.name or "Test")
            if not spinner_thread.is_alive() and not quiet and sys.stdout.isatty():
                # Start thread if not running (it handles its own loop)
                # Note: We can't restart a thread, so we might need a new instance
                # or just keep it running and use events.
                # Simpler approach: Start it once before execution and use flags.
                pass

        def progress_callback(result: TestResult) -> None:
            """Update progress indicator and display progress line."""
            # Temporarily stop spinner to print result
            # In a real implementation with threading, we might need a lock
            # But here we just clear the line in the main thread (via update)

            # Update progress state
            progress.update(result)

            # Get formatted result line
            progress_line = progress.get_progress_line()

            if progress_line:  # Only print if not empty (respects quiet mode)
                # Clear current line (handled by \r in spinner)
                sys.stdout.write("\r" + " " * 60 + "\r")
                click.echo(progress_line)

        # Initialize executor with pre-loaded configs to avoid duplicate I/O
        logger.debug("Initializing test executor")
        executor = TestExecutor(
            agent_config_path=agent_config,
            progress_callback=progress_callback,
            on_test_start=on_test_start,
            force_ingest=force_ingest,
            agent_config=agent,
            resolved_execution_config=resolved_config,
        )

        # Run tests asynchronously
        logger.info("Starting test execution")

        # Start spinner thread
        if not quiet and sys.stdout.isatty():
            spinner_thread.start()

        async def run_tests_with_cleanup() -> TestReport:
            """Execute tests and ensure proper cleanup of MCP plugins."""
            try:
                return await executor.execute_tests()
            finally:
                await executor.shutdown()

        try:
            report = asyncio.run(run_tests_with_cleanup())
        finally:
            # Ensure spinner stops
            if spinner_thread.is_alive():
                spinner_thread.stop()
                spinner_thread.join()

        elapsed_time = time.time() - start_time
        logger.info(
            f"Test execution completed in {elapsed_time:.2f}s - "
            f"{report.summary.passed} passed, {report.summary.failed} failed"
        )

        # Display summary (always shown, even in quiet mode)
        summary_text = progress.get_summary()
        click.echo(summary_text)

        # Save report if output specified
        if output:
            logger.debug(f"Saving report to {output} (format={format})")
            _save_report(report, output, format)
            logger.info(f"Report saved successfully to {output}")

        # Exit with appropriate code
        if report.summary.failed > 0:
            logger.info("Exiting with failure status (failed tests)")
            sys.exit(1)
        else:
            logger.info("Exiting with success status (all tests passed)")
            sys.exit(0)

    except ConfigError as e:
        logger.error(f"Configuration error: {e}", exc_info=True)
        click.echo(f"Configuration Error: {e}", err=True)
        sys.exit(2)
    except ExecutionError as e:
        logger.error(f"Execution error: {e}", exc_info=True)
        click.echo(f"Execution Error: {e}", err=True)
        sys.exit(3)
    except EvaluationError as e:
        logger.error(f"Evaluation error: {e}", exc_info=True)
        click.echo(f"Evaluation Error: {e}", err=True)
        sys.exit(4)
    except Exception as e:
        logger.error(f"Unexpected error: {e}", exc_info=True)
        click.echo(f"Error: {str(e)}", err=True)
        sys.exit(3)

CLI Utilities

Project initialization and scaffolding utilities.

ProjectInitializer()

Handles project initialization logic.

Provides methods to: - Validate user inputs (project name, template, permissions) - Load and validate template manifests - Initialize new agent projects with all required files

Initialize the ProjectInitializer.

Source code in src/holodeck/cli/utils/project_init.py

def __init__(self) -> None:
    """Initialize the ProjectInitializer."""
    self.template_renderer = TemplateRenderer()
    # Get available templates from discovery function
    self.available_templates = set(TemplateRenderer.list_available_templates())

initialize(input_data)

Initialize a new agent project.

Creates a new project directory with all required files and templates. Follows all-or-nothing semantics: either the entire project is created successfully, or no files are created and the directory is cleaned up.

Parameters:

Name	Type	Description	Default
input_data	ProjectInitInput	ProjectInitInput with validated user inputs	required

Returns:

Name	Type	Description
ProjectInitResult	ProjectInitResult	Result of initialization with status and metadata

Raises:

Type	Description
InitError	If initialization fails (will attempt cleanup)

Source code in src/holodeck/cli/utils/project_init.py

def initialize(self, input_data: ProjectInitInput) -> ProjectInitResult:
    """Initialize a new agent project.

    Creates a new project directory with all required files and templates.
    Follows all-or-nothing semantics: either the entire project is created
    successfully, or no files are created and the directory is cleaned up.

    Args:
        input_data: ProjectInitInput with validated user inputs

    Returns:
        ProjectInitResult: Result of initialization with status and metadata

    Raises:
        InitError: If initialization fails (will attempt cleanup)
    """
    start_time = time.time()
    project_name = input_data.project_name.strip()
    output_dir = Path(input_data.output_dir)
    project_dir = output_dir / project_name

    files_created = []

    try:
        # Validate inputs first
        self.validate_inputs(input_data)

        # Load template manifest
        template = self.load_template(input_data.template)

        # Create project directory
        if project_dir.exists() and input_data.overwrite:
            # Remove existing directory if force flag is set
            shutil.rmtree(project_dir)

        project_dir.mkdir(parents=True, exist_ok=False)
        files_created.append(str(project_dir))

        # Prepare provider-specific config
        provider_config = input_data.provider_config
        endpoint_env_var = get_provider_endpoint_env_var(input_data.llm_provider)

        # Determine endpoint value
        llm_endpoint = None
        if provider_config and provider_config.endpoint:
            llm_endpoint = provider_config.endpoint
        elif endpoint_env_var:
            # Use environment variable placeholder as default
            llm_endpoint = f"${{{endpoint_env_var}}}"

        # Prepare template variables
        template_vars = {
            "project_name": project_name,
            "description": input_data.description or "TODO: Add agent description",
            "author": input_data.author or "",
            # Wizard configuration fields
            "agent_name": input_data.agent_name,
            "llm_provider": input_data.llm_provider,
            "llm_model": get_model_for_provider(input_data.llm_provider),
            "llm_endpoint": llm_endpoint,
            "llm_api_key_env_var": get_provider_api_key_env_var(
                input_data.llm_provider
            ),
            "vector_store": input_data.vector_store,
            "vector_store_endpoint": get_vectorstore_endpoint(
                input_data.vector_store
            ),
            "evals": input_data.evals,
            "mcp_servers": [
                get_mcp_server_config(s) for s in input_data.mcp_servers
            ],
        }

        # Add template-specific defaults from manifest
        if template.defaults:
            template_vars.update(template.defaults)

        # Create files from template
        template_dir = (
            Path(__file__).parent.parent.parent / "templates" / input_data.template
        )

        # Process each file in the template manifest
        if template.files:
            for file_spec in template.files.values():
                if not file_spec.required:
                    continue

                file_path = project_dir / file_spec.path
                file_path.parent.mkdir(parents=True, exist_ok=True)

                if file_spec.template:
                    # Render Jinja2 template
                    template_file = template_dir / f"{file_spec.path}.j2"
                    if not template_file.exists():
                        # Try without .j2 extension
                        template_file = template_dir / file_spec.path

                    if file_path.suffix == ".yaml" or file_path.suffix == ".yml":
                        # Validate YAML files against schema
                        content = self.template_renderer.render_and_validate(
                            str(template_file), template_vars
                        )
                    else:
                        # Render non-YAML files normally
                        content = self.template_renderer.render_template(
                            str(template_file), template_vars
                        )

                    file_path.write_text(content)
                else:
                    # Copy static files directly
                    source_file = template_dir / file_spec.path
                    if source_file.exists():
                        shutil.copy2(source_file, file_path)

                files_created.append(str(file_path.relative_to(output_dir)))

        # Also copy .gitignore if it exists
        gitignore_src = template_dir / ".gitignore"
        if gitignore_src.exists():
            gitignore_dst = project_dir / ".gitignore"
            shutil.copy2(gitignore_src, gitignore_dst)
            files_created.append(str(gitignore_dst.relative_to(output_dir)))

        duration = time.time() - start_time

        return ProjectInitResult(
            success=True,
            project_name=project_name,
            project_path=str(project_dir),
            template_used=input_data.template,
            files_created=files_created,
            warnings=[],
            errors=[],
            duration_seconds=duration,
        )

    except (ValidationError, InitError) as e:
        # Clean up partial directory on error
        if project_dir.exists():
            with contextlib.suppress(Exception):
                shutil.rmtree(project_dir)

        duration = time.time() - start_time

        return ProjectInitResult(
            success=False,
            project_name=project_name,
            project_path=str(project_dir),
            template_used=input_data.template,
            files_created=[],
            warnings=[],
            errors=[str(e)],
            duration_seconds=duration,
        )

    except Exception as e:
        # Clean up partial directory on unexpected error
        if project_dir.exists():
            with contextlib.suppress(Exception):
                shutil.rmtree(project_dir)

        duration = time.time() - start_time

        return ProjectInitResult(
            success=False,
            project_name=project_name,
            project_path=str(project_dir),
            template_used=input_data.template,
            files_created=[],
            warnings=[],
            errors=[f"Unexpected error: {str(e)}"],
            duration_seconds=duration,
        )

load_template(template_name)

Load and validate a template manifest.

Loads the manifest.yaml file from a template directory and validates it against the TemplateManifest schema.

Parameters:

Name	Type	Description	Default
template_name	str	Name of the template (e.g., 'conversational')	required

Returns:

Name	Type	Description
TemplateManifest	TemplateManifest	Parsed and validated template manifest

Raises:

Type	Description
FileNotFoundError	If template or manifest file not found
InitError	If manifest cannot be parsed or validated

Source code in src/holodeck/cli/utils/project_init.py

def load_template(self, template_name: str) -> TemplateManifest:
    """Load and validate a template manifest.

    Loads the manifest.yaml file from a template directory and validates
    it against the TemplateManifest schema.

    Args:
        template_name: Name of the template (e.g., 'conversational')

    Returns:
        TemplateManifest: Parsed and validated template manifest

    Raises:
        FileNotFoundError: If template or manifest file not found
        InitError: If manifest cannot be parsed or validated
    """
    # Get template directory
    # Templates are bundled in src/holodeck/templates/
    template_dir = Path(__file__).parent.parent.parent / "templates" / template_name

    if not template_dir.exists():
        raise FileNotFoundError(f"Template directory not found: {template_dir}")

    manifest_path = template_dir / "manifest.yaml"

    if not manifest_path.exists():
        raise FileNotFoundError(f"Template manifest not found: {manifest_path}")

    try:
        with open(manifest_path) as f:
            manifest_data = yaml.safe_load(f)

        if not manifest_data:
            raise InitError(f"Template manifest is empty: {manifest_path}")

        # Validate against TemplateManifest schema
        manifest = TemplateManifest.model_validate(manifest_data)
        return manifest

    except yaml.YAMLError as e:
        raise InitError(f"Template manifest contains invalid YAML: {e}") from e
    except Exception as e:
        if isinstance(e, ValidationError | InitError):
            raise
        raise InitError(f"Failed to load template manifest: {e}") from e

validate_inputs(input_data)

Validate user inputs for project initialization.

Checks: - Project name format (alphanumeric, hyphens, underscores, no leading digits) - Project name is not empty and within length limits - Template exists in available templates - Output directory is writable - Project directory doesn't already exist (unless overwrite is True)

Parameters:

Name	Type	Description	Default
input_data	ProjectInitInput	ProjectInitInput with user-provided values	required

Raises:

Type	Description
ValidationError	If any validation checks fail

Source code in src/holodeck/cli/utils/project_init.py

def validate_inputs(self, input_data: ProjectInitInput) -> None:
    """Validate user inputs for project initialization.

    Checks:
    - Project name format (alphanumeric, hyphens, underscores, no leading digits)
    - Project name is not empty and within length limits
    - Template exists in available templates
    - Output directory is writable
    - Project directory doesn't already exist (unless overwrite is True)

    Args:
        input_data: ProjectInitInput with user-provided values

    Raises:
        ValidationError: If any validation checks fail
    """
    project_name = input_data.project_name.strip()

    # Check project name is not empty
    if not project_name:
        raise ValidationError("Project name cannot be empty")

    # Check project name length
    if len(project_name) > self.MAX_PROJECT_NAME_LENGTH:
        raise ValidationError(
            f"Project name cannot exceed {self.MAX_PROJECT_NAME_LENGTH} characters"
        )

    # Check project name format
    if not re.match(self.PROJECT_NAME_PATTERN, project_name):
        raise ValidationError(
            f"Invalid project name: '{project_name}'. "
            "Project names must start with a letter or underscore, "
            "and contain only alphanumeric characters, hyphens, and underscores."
        )

    # Check template exists
    if input_data.template not in self.available_templates:
        templates_list = ", ".join(sorted(self.available_templates))
        raise ValidationError(
            f"Unknown template: '{input_data.template}'. "
            f"Available templates: {templates_list}"
        )

    # Check output directory is writable
    output_dir = Path(input_data.output_dir)
    if not output_dir.exists():
        raise ValidationError(f"Output directory does not exist: {output_dir}")

    if not output_dir.is_dir():
        raise ValidationError(f"Output path is not a directory: {output_dir}")

    try:
        # Test write permissions by attempting to check access
        if not os.access(str(output_dir), os.W_OK):
            raise ValidationError(f"Output directory is not writable: {output_dir}")
    except OSError as e:
        raise ValidationError(f"Cannot access output directory: {e}") from e

    # Check project directory doesn't already exist (unless force)
    project_dir = output_dir / project_name
    if project_dir.exists() and not input_data.overwrite:
        raise ValidationError(
            f"Project directory already exists: {project_dir}. "
            "Use --force to overwrite."
        )

CLI Exceptions

CLI-specific exception handling.

CLIError

Bases: Exception

Base exception for all CLI errors.

This is the parent class for all exceptions raised by the CLI module. Users can catch this to handle any CLI error generically.

ValidationError

Bases: CLIError

Raised when user input validation fails.

This exception is raised when: - Project name is invalid (special characters, leading digits, etc.) - Template choice doesn't exist - Directory permissions are insufficient - Input constraints are violated

Attributes:

Name	Type	Description
message		Description of the validation failure

InitError

Bases: CLIError

Raised when project initialization fails.

This exception is raised when: - Directory creation fails - File writing fails - Cleanup fails after partial creation - Unexpected errors occur during initialization

Attributes:

Name	Type	Description
message		Description of the initialization failure

TemplateError

Bases: CLIError

Raised when template processing fails.

This exception is raised when: - Template manifest is malformed or missing - Jinja2 rendering fails - Generated YAML doesn't validate against schema - Template variables are missing or invalid

Attributes:

Name	Type	Description
message		Description of the template failure

Usage from Python

You can invoke CLI commands programmatically:

from holodeck.cli.main import main
from click.testing import CliRunner

runner = CliRunner()

# Initialize a new project
result = runner.invoke(main, ['init', '--template', 'conversational', '--name', 'my-agent'])
print(result.output)

# Run tests
result = runner.invoke(main, ['test', 'path/to/agent.yaml'])
print(result.output)

CLI Entry Point

The CLI is registered as the holodeck command via pyproject.toml:

[project.scripts]
holodeck = "holodeck.cli.main:main"

After installation, use from terminal:

holodeck init --template conversational --name my-agent
holodeck test agent.yaml

Related Documentation

• Project Templates: Available templates
• Configuration Loading: Configuration system
• Test Runner: Test execution