Holmes LabAgentic Software Engineering Architect

Tests, review and completion evidence

Connect failure reproduction, regression scope and actual observations instead of reporting only passed.

Guide verification basis: Holmes-Kit 0.40.0 · 2026-11-05 · Installation target: latest at execution time

Prerequisites, evidence and installation tradeoffs

Before you start

  • Your own workspace and connected MCP session after the basics. Define HOLMES_CLI or PowerShell HolmesCli from the installation step in this terminal and verify its file and version.
  • An approved A-SPEC and a real test environment

Produce a review package separating implemented, tested and runtime-observed behavior.

Documented for 0.40.0 · release contracts reviewed 2026-11-04. Native evidence is the historical 0.39.0 run of 2026-11-01. Read-only binding matched for Codex and Claude in an independent macOS workspace. Claude hook events were observed but routing is unverified; Codex hooks were not observed. Native revalidation on 0.40.0 and RED/GREEN acceptance of a small governed change remain incomplete. Windows/Linux instructions have no native acceptance run.

1. Separate user folder
Keeps project dependencies and global npm installation unchanged; separate versioned folder. Manage its path and updates separately; retain the folder while workspaces use it.
2. Project local
Project/team versioning through a lockfile; recommended for the practice project. Changes package.json and lockfile; may introduce npm metadata to a non-Node project.
3. Global npm
Convenient shared installation across projects. Check write permissions and update impact on all projects sharing this installation.

Package location and client settings are separate. Even project-local Codex wiring may change user-level settings.

Step 1 / 4

Reproduce a behavioral failure

Choose a failing situation for each success criterion.

Paste into your AI CLIShared

Ask your agent

Verify the current approved A-SPEC and persist maintenance_analyze. Anchor the test first line with its actual ID, then record assertion RED with test_run.

Expected result — inspect your actual output

  • The failure directly identifies the missing required behavior.
A manual learning note, not automatic verification of your workspace.
If the result differs

A package cannot load or a fixture is invalid.

Fix the test environment and rerun the behavioral check; do not use an environment failure as RED evidence.

What the agent does / terms

After approved scope and maintenance analysis, record assertion RED with official test_run.

MCP
The connection through which your AI calls Holmes tools.
REQ → H-SPEC → A-SPEC → T-SPEC
Requirement → high-level design → implementation design → test specification. Use actual IDs returned by the tools.
Hook / binding
A pre/post execution check / matching this session to the intended project and server.

Step 2 / 4

Verify the improvement with the same test

Review the T-SPEC and implementation scope.

Paste into your AI CLIShared

Ask your agent

Implement minimally under the approved T-SPEC and rerun the same test. If an assertion changes, explain why and how it still checks the original requirement.

Expected result — inspect your actual output

  • The same behavior test is GREEN and edits remain within approved scope.
A manual learning note, not automatic verification of your workspace.
If the result differs

The test passes but another input fails.

Add boundary and failure inputs, fix the implementation and record new results.

What the agent does / terms

After mirroring T-SPEC approval, implement the minimal change and rerun the RED test.

MCP
The connection through which your AI calls Holmes tools.
REQ → H-SPEC → A-SPEC → T-SPEC
Requirement → high-level design → implementation design → test specification. Use actual IDs returned by the tools.
Hook / binding
A pre/post execution check / matching this session to the intended project and server.

Step 3 / 4

Check regressions and usage paths

Choose real user paths and existing behavior to preserve.

Paste into your AI CLIShared

Ask your agent

Find actual verification commands in project scripts or test configuration and run them. For UI changes, inspect the real screen and direct URLs, and state what each check does not establish.

Expected result — inspect your actual output

  • Per-command results and actual screen observations are separated.
A manual learning note, not automatic verification of your workspace.
If the result differs

Environment limits prevent browser or service execution.

Label the path unverified and record reproduction conditions; do not turn build success into operational success.

What the agent does / terms

Run relevant regressions, type checks and build; inspect UI changes in a browser.

MCP
The connection through which your AI calls Holmes tools.
REQ → H-SPEC → A-SPEC → T-SPEC
Requirement → high-level design → implementation design → test specification. Use actual IDs returned by the tools.
Hook / binding
A pre/post execution check / matching this session to the intended project and server.

Step 4 / 4

Close independent review and remaining findings

Review important open findings and the deployment decision.

Paste into your AI CLIShared

Ask your agent

Assemble the final diff, RED/GREEN, regressions, build, runtime observations and unverified items for review. Separate independent review results and open critical/important findings; keep deployment separate.

Expected result — inspect your actual output

  • Every completion claim has actual evidence and open findings remain visible.
A manual learning note, not automatic verification of your workspace.
If the result differs

The diff changes after review and no longer matches the evidence.

Refresh affected verification and review, and state the scope of earlier results.

What the agent does / terms

Provide requirement, diff and logs to a reviewer; never record self-review as independent review.

MCP
The connection through which your AI calls Holmes tools.
REQ → H-SPEC → A-SPEC → T-SPEC
Requirement → high-level design → implementation design → test specification. Use actual IDs returned by the tools.
Hook / binding
A pre/post execution check / matching this session to the intended project and server.

Recorded Holmes Lab change

A-SPEC-101.21 restored three hidden primary links on home—Technical journey, Work and Practice—removed graph fallback JSON links and expanded Practice content. RED, regression and build records in docs/evidence/engineering-perspective are historical evidence. Running these instructions in your project is a reproduction, not that historical execution.

Related case ↗

Continue with your project

Use Adoption for existing code without specs, or Requirements when specifications already exist. Other workflows require their stated prerequisites.