Quality score¶
Review date: 2026-09-06; rows for retrieval, evaluation and install profiles updated 2026-11-10 for spec 042. Owner: maintainers of each listed area. This score describes available enforcement and evidence, not a release certification.
Grades: A has named checks run for this audit. B has named checks located but not rerun here. C has a documented gap without sufficient mechanical enforcement. Update a row after relevant changes and record the command and result.
| Area / layer | Grade | Evidence and verification route | Gap |
|---|---|---|---|
| Repository harness | A | make harness-check, 15 checker and publishing tests passed. Strict MkDocs build, Black, Ruff, and focused MyPy passed. |
Semantic freshness still requires code review |
| Configuration and schemas | B | Schema sync, config tests, make schema-check |
Full suite not run for this documentation audit |
| Runtime selection and contracts | B | Backend tests, especially test_selector.py and test_base.py |
Provider parity still has open spec tasks |
| Tools and retrieval | B | Tool tests, adapter unit tests, integration tests | Live stores require service-specific verification |
| Retrieval adapter contract | B | Contract suite: one parametrized suite for all eight native adapters, with capability errors checked before ingestion | Pinecone ran on Pinecone Local (end of life) with a fake control plane, Azure AI Search on a fake service with the real SDK over a stub transport, Cosmos DB hybrid search on a fake only. No hosted-service run for Pinecone, Azure AI Search or Cosmos DB |
| Stored-data compatibility | B | Legacy-layout fixtures (Qdrant, Chroma, Postgres layouts written by the removed connectors), Qdrant, Chroma and Postgres compatibility tests, migration tests | Manifest and fingerprint not run on Pinecone, Azure AI Search, Cosmos DB or Weaviate. Concurrent writers are narrowed, not eliminated, on stores without compare-and-set |
| Golden fixtures | B | Goldens and their tests: retrieval ranking, parser output, plain chunks and NLP scores. A change to a golden needs a stated reason | The goldens freeze behavior, not relevance quality |
| Provider verify matrix | B | make verify-providers and the provider cases of make verify-live: real Qdrant, Chroma, Postgres, Weaviate and Cosmos emulator, with a check that the store holds records. See Reliability |
Pinecone and Azure AI Search cases skip without credentials. One flaky case: the OpenAI model sometimes declines to repeat a code word in an AG-UI turn |
| Install profiles | B | Profile tests, missing-extra tests, no-Semantic-Kernel guard; make verify-profiles and the profiles CI workflow install the built wheel for 16 profiles |
The hosted workflow's first run was on PR #442. The Node.js install in the image template has no provenance review (FR-029) |
| Evaluations and optimizer | B | Evaluator tests, retired-metric tests, optimizer tests | Model quality needs representative evaluation runs. METEOR tests skip when the NLTK data is absent |
| Serving and deployment | B | Serve tests, deploy tests | Cloud behavior requires authorized end-to-end validation |
| Temporal deterministic boundary | B | Import purity, sandbox safety, optional import guard | Replay/service integration needs the corresponding environment |
| Dashboard | B | Dashboard tests, frontend guidance | Visual changes require rendered inspection |
| Global dependency layering | C | Architecture exceptions | No repository-wide import-layer enforcement |
| Spec and guide freshness | C | Spec inventory, harness links | Historical task/status drift and docsite warnings require manual reconciliation |
| Published template schemas | C | make schema-check covers schemas/agent.schema.json |
The copies under src/holodeck/templates/*/schemas/ are hand-kept and stale; make schema does not regenerate them |
Verification policy¶
For a behavior change, run its regression tests and affected contracts with -n auto.
For shared schema or protocol changes, broaden to their consumers.
For Python changes, run focused formatting, lint, and type checks.
Keep required commit and CI checks. See Contributing.
The constitution sets an 80% coverage target. Measure it with explicit coverage options. Do not infer coverage from test counts or from the presence of tests. Document environmental blockers and skipped live checks separately from failures.
Maintenance loop¶
When a check fails repeatedly, identify the missing contract, fixture, diagnostic, or instruction. Fix that cause with a focused change and a remediation message. Review this map when a module or its test strategy changes. Track unresolved gaps in technical debt.