Tasks: Unstructured Vector Ingestion and Search¶

Input: Design documents from /docs/product-specs/008-unstructured-vector-ingestion-search/ Prerequisites: plan.md, spec.md, research.md, data-model.md, contracts/

Tests: Tests are NOT explicitly requested in the feature specification. Test tasks are included for validation but can be removed if not desired.

Organization: Tasks are grouped by user story to enable independent implementation and testing of each story.

Format: `[ID] [P?] [Story] Description`¶

[P]: Can run in parallel (different files, no dependencies)
[Story]: Which user story this task belongs to (e.g., US1, US2, US3)
Include exact file paths in descriptions

Path Conventions¶

Single project: src/holodeck/, tests/ at repository root
Existing project structure is extended

Phase 1: Setup (Shared Infrastructure)¶

Purpose: Add dependencies and prepare infrastructure for vectorstore tools

T001 Add Semantic Kernel dependency to pyproject.toml (vector stores, text chunking, embeddings)
T002 [P] Add redis-py async dependency to pyproject.toml for Redis vector store support
T003 [P] Update pyproject.toml with tiktoken dependency for token-based text chunking
T004 Install all new dependencies via poetry and verify no conflicts

Phase 2: Foundational (Blocking Prerequisites)¶

Purpose: Core abstractions and utilities that ALL user stories depend on

⚠️ CRITICAL: No user story work can begin until this phase is complete

Checkpoint: Foundation ready - user story implementation can now begin in parallel

Phase 3: User Story 1 - Semantic Search Over Unstructured Data (Priority: P1) 🎯 MVP¶

Goal: Enable agents to perform semantic search over single files or directories with automatic ingestion, embedding generation, and relevance-ranked results

Independent Test: Configure a vectorstore tool pointing to either a single markdown file or a directory of mixed file types, run queries, and verify semantically relevant results are returned with source references

Unit Tests for User Story 1¶

NOTE: Write these tests FIRST, ensure they FAIL before implementation

T016 [P] [US1] Unit test for VectorStoreTool initialization with valid config in tests/unit/test_vectorstore_tool.py
T017 [P] [US1] Unit test for VectorStoreTool initialization with missing source path in tests/unit/test_vectorstore_tool.py
T018 [P] [US1] Unit test for VectorStoreTool file discovery (single file) in tests/unit/test_vectorstore_tool.py
T019 [P] [US1] Unit test for VectorStoreTool file discovery (directory with nested subdirectories) in tests/unit/test_vectorstore_tool.py
T020 [P] [US1] Unit test for VectorStoreTool search result formatting in tests/unit/test_vectorstore_tool.py
T021 [P] [US1] Unit test for text chunking with markdown content in tests/unit/test_text_chunker.py
T022 [P] [US1] Unit test for VectorStore upsert and get operations in tests/unit/test_vector_store.py
T023 [P] [US1] Unit test for VectorStore search with top_k filtering in tests/unit/test_vector_store.py

Implementation for User Story 1¶

Agent Integration for User Story 1¶

Purpose: Integrate VectorStoreTool with AgentFactory so agents can use vectorstore tools during execution

T037a [US1] Add TextEmbedding service registration method to AgentFactory in src/holodeck/lib/test_runner/agent_factory.py (OpenAI/Azure text-embedding-3-small) following Semantic Kernel pattern (depends on T024)
T037b [US1] Create KernelFunction for VectorStoreTool.search using kernel.add_function() in AgentFactory (NOT decorator) in src/holodeck/lib/test_runner/agent_factory.py (depends on T033)
T037c [US1] Add _register_vectorstore_tools method to AgentFactory to discover vectorstore tools from agent config, initialize them, and register search functions via kernel.add_function() in src/holodeck/lib/test_runner/agent_factory.py (depends on T037a, T037b)
T037d [US1] Update AgentFactory._create_kernel to call embedding service registration when vectorstore tools are configured in src/holodeck/lib/test_runner/agent_factory.py (depends on T037a)
T037e [US1] Update AgentFactory._create_agent to call _register_vectorstore_tools before agent creation in src/holodeck/lib/test_runner/agent_factory.py (depends on T037c)
T037f [P] [US1] Unit test for AgentFactory embedding service registration in tests/unit/test_agent_factory.py
T037g [P] [US1] Unit test for VectorStoreTool kernel function registration via add_function in tests/unit/test_agent_factory.py
T037h [P] [US1] Unit test for AgentFactory vectorstore tool discovery and initialization in tests/unit/test_agent_factory.py

Integration Tests for User Story 1¶

T038 [P] [US1] Integration test for end-to-end ingestion and search with single markdown file in tests/integration/test_vectorstore_integration.py
T039 [P] [US1] Integration test for end-to-end ingestion and search with directory of mixed file types in tests/integration/test_vectorstore_integration.py
T040 [P] [US1] Integration test for nested subdirectory traversal in tests/integration/test_vectorstore_integration.py
T041 [P] [US1] Integration test for search result ordering by relevance score in tests/integration/test_vectorstore_integration.py

Checkpoint: At this point, User Story 1 should be fully functional and testable independently - agents can perform semantic search over files and directories

Phase 4: User Story 2 - Custom Embedding Model Configuration (Priority: P2)¶

Goal: Allow developers to specify custom embedding models for optimization (domain-specific embeddings, cost, performance)

Independent Test: Configure embedding_model parameter in vectorstore tool, verify embeddings are generated using the specified model, compare against default model behavior

Unit Tests for User Story 2¶

T042 [P] [US2] Unit test for embedding model parameter parsing from config in tests/unit/test_vectorstore_tool.py
T043 [P] [US2] Unit test for default embedding model selection based on LLM provider in tests/unit/test_vectorstore_tool.py
T044 [P] [US2] Unit test for custom embedding model initialization in tests/unit/test_vectorstore_tool.py

Implementation for User Story 2¶

T045 [P] [US2] Implement embedding model resolution logic in VectorStoreTool.init (custom model vs provider default) in src/holodeck/tools/vectorstore_tool.py (depends on T025)
T046 [US2] Implement TextEmbedding service initialization with configurable model in VectorStoreTool.init in src/holodeck/tools/vectorstore_tool.py (depends on T045)
T047 [US2] Add provider-specific default embedding model mapping (OpenAI → text-embedding-3-small, Azure → text-embedding-ada-002) in src/holodeck/tools/vectorstore_tool.py (depends on T046)
T048 [US2] Add validation for embedding model availability based on configured LLM provider in src/holodeck/config/validator.py (depends on T015)
T049 [US2] Update embedding dimension detection to match configured model in DocumentRecord creation in src/holodeck/tools/vectorstore_tool.py (depends on T031, T046)

Integration Tests for User Story 2¶

T050 [P] [US2] Integration test for custom embedding model usage (text-embedding-3-large) in tests/integration/test_vectorstore_integration.py
T051 [P] [US2] Integration test for default embedding model when no explicit model specified in tests/integration/test_vectorstore_integration.py

Checkpoint: At this point, User Stories 1 AND 2 should both work independently - custom embedding models are supported

Phase 5: User Story 3 - Vector Database Persistence (Priority: P3)¶

Goal: Enable persistent vector storage using Redis for production deployments with large knowledge bases and multi-session persistence

Independent Test: Configure Redis vector database connection, ingest test data, verify data persists across agent restarts, measure search performance with large datasets

Unit Tests for User Story 3¶

T052 [P] [US3] Unit test for file modification timestamp tracking in tests/unit/test_vectorstore_tool.py
T053 [P] [US3] Unit test for mtime comparison logic (needs re-ingestion vs up-to-date) in tests/unit/test_vectorstore_tool.py
T054 [P] [US3] Unit test for --force-ingest flag bypassing mtime checks in tests/unit/test_vectorstore_tool.py
T055 [P] [US3] Unit test for Redis connection fallback to in-memory on connection failure in tests/unit/test_vector_store.py

Implementation for User Story 3¶

Integration Tests for User Story 3¶

T067 [P] [US3] Integration test for Redis vector store persistence across agent restarts in tests/integration/test_vectorstore_integration.py
T068 [P] [US3] Integration test for automatic re-ingestion when source file modified in tests/integration/test_vectorstore_integration.py
T069 [P] [US3] Integration test for --force-ingest flag forcing full re-ingestion in tests/integration/test_vectorstore_integration.py
T070 [P] [US3] Integration test for Redis connection failure fallback to in-memory in tests/integration/test_vectorstore_integration.py

Checkpoint: All user stories should now be independently functional - Redis persistence and modification tracking complete

Phase 6: Polish & Cross-Cutting Concerns¶

Purpose: Improvements that affect multiple user stories

Dependencies & Execution Order¶

Phase Dependencies¶

Setup (Phase 1): No dependencies - can start immediately
Foundational (Phase 2): Depends on Setup completion (T001-T004) - BLOCKS all user stories
User Stories (Phase 3-5): All depend on Foundational phase completion (T005-T015)
User stories can then proceed in parallel (if staffed)
Or sequentially in priority order (P1 → P2 → P3)
Polish (Phase 6): Depends on all desired user stories being complete

User Story Dependencies¶

User Story 1 (P1): Can start after Foundational (Phase 2) - No dependencies on other stories
User Story 2 (P2): Can start after Foundational (Phase 2) - Extends US1 but independently testable (depends on T025, T031)
User Story 3 (P3): Can start after Foundational (Phase 2) - Extends US1 but independently testable (depends on T028, T032, T033)

Within Each User Story¶

Tests MUST be written and FAIL before implementation
Data models before tool implementation
Core functionality before edge case handling
Implementation before integration tests
Story complete before moving to next priority

Parallel Opportunities¶

Phase 1 (Setup):

T002, T003 can run in parallel

Phase 2 (Foundational):

T005, T006, T007, T008, T009, T013, T015 can all run in parallel (different files, no dependencies)

User Story 1:

All unit tests (T016-T023) can run in parallel
T024 and T025 must complete before parallel work begins
T028, T030 can run in parallel after T025
Agent integration: T037a-T037e must complete sequentially (depends on T033)
Agent integration unit tests (T037f, T037g, T037h) can run in parallel
All integration tests (T038-T041) can run in parallel after implementation and agent integration

User Story 2:

All unit tests (T042-T044) can run in parallel
T045 and T046 must complete before T047 and T049
All integration tests (T050-T051) can run in parallel after implementation

User Story 3:

All unit tests (T052-T055) can run in parallel
T056 and T057 must complete before parallel work begins
T062, T063, T066 can run in parallel
All integration tests (T067-T070) can run in parallel after implementation

Phase 6 (Polish):

T071, T072, T073, T074, T075, T076, T077, T082, T083, T084 can all run in parallel

Parallel Example: User Story 1¶

# Launch all unit tests for User Story 1 together:
Task T016: "Unit test for VectorStoreTool initialization with valid config"
Task T017: "Unit test for VectorStoreTool initialization with missing source path"
Task T018: "Unit test for VectorStoreTool file discovery (single file)"
Task T019: "Unit test for VectorStoreTool file discovery (directory with nested subdirectories)"
Task T020: "Unit test for VectorStoreTool search result formatting"
Task T021: "Unit test for text chunking with markdown content"
Task T022: "Unit test for VectorStore upsert and get operations"
Task T023: "Unit test for VectorStore search with top_k filtering"

# Launch foundational abstractions together (after Phase 2):
Task T007: "Create DocumentRecord dataclass"
Task T008: "Create SourceFile dataclass"
Task T009: "Create QueryResult dataclass"
Task T013: "Create TextChunker wrapper class"

Implementation Strategy¶

MVP First (User Story 1 Only)¶

Complete Phase 1: Setup (T001-T004)
Complete Phase 2: Foundational (T005-T015) - CRITICAL
Complete Phase 3: User Story 1 (T016-T041)
STOP and VALIDATE: Test semantic search over single files and directories
Deploy/demo basic vectorstore functionality

Deliverable: Agents can perform semantic search over documentation files with in-memory storage

Incremental Delivery¶

Complete Setup + Foundational → Foundation ready (T001-T015)
Add User Story 1 → Test independently → Deploy/Demo (T016-T041) - MVP!
Add User Story 2 → Test independently → Deploy/Demo (T042-T051) - Custom embedding models
Add User Story 3 → Test independently → Deploy/Demo (T052-T070) - Redis persistence
Polish → Run validation (T071-T086)

Each story adds value without breaking previous stories.

Parallel Team Strategy¶

With multiple developers:

Team completes Setup + Foundational together (T001-T015)
Once Foundational is done:
Developer A: User Story 1 (T016-T041 + T037a-T037h agent integration)
Developer B: User Story 2 (T042-T051) - Can start in parallel
Developer C: User Story 3 (T052-T070) - Can start in parallel
Stories complete and integrate independently

Summary¶

Total Tasks: 94 tasks

Phase 1 (Setup): 4 tasks
Phase 2 (Foundational): 11 tasks (BLOCKS all user stories)
Phase 3 (User Story 1): 34 tasks (8 unit tests + 14 implementation + 8 agent integration + 4 integration tests)
Phase 4 (User Story 2): 10 tasks (3 unit tests + 5 implementation + 2 integration tests)
Phase 5 (User Story 3): 19 tasks (4 unit tests + 11 implementation + 4 integration tests)
Phase 6 (Polish): 16 tasks

Parallel Opportunities: 41 tasks marked with [P] can run in parallel within their phases

MVP Scope: Phases 1-3 (T001-T041 + T037a-T037h) - 49 tasks for basic semantic search functionality with agent integration

Independent Test Criteria:

US1: Configure vectorstore with file/directory, run queries, verify relevant results with source attribution
US2: Configure custom embedding model, verify model is used, compare results to default
US3: Configure Redis, verify persistence across restarts, test modification tracking with --force-ingest

Suggested Implementation Order: P1 (US1) → P2 (US2) → P3 (US3) → Polish

Notes¶

[P] tasks = different files, no dependencies within phase
[Story] label maps task to specific user story for traceability
Each user story should be independently completable and testable
Verify tests fail before implementing (TDD approach)
Run code quality commands (format, lint, type-check) after each phase
Commit after each task or logical group
Stop at any checkpoint to validate story independently
Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence