Wiki › Engineering with agents
Agent Verification Loops
Evidence-driven coding-agent loops with real-system execution, protected checks, commit-bound reviews, bounded repair and incident-based verifier evaluations.
An agent verification loop gives a coding agent executable evidence about its changes, feeds failures back into implementation, and routes unresolved or consequential issues to a human. The X Article “How the Claude Code team thinks about verification,” captured from Addy Osmani, describes this as infrastructure for scaling delegation rather than simply asking the model to be more careful. It connects Harness Engineering to Production LLM Reliability.
Three complementary kinds of evidence
| Check | What it establishes | Important limit |
| --- | --- | --- |
| End-to-end execution | A real user flow works in the running app, service or simulator. | Coverage is limited to exercised paths and observed outcomes. |
| Property-based tests | Generated inputs probe an explicit invariant: round trips, idempotency, equivalence or domain constraints. | A wrong or incomplete property remains a wrong or incomplete specification. |
| Differential tests | Old and new implementations agree on shared inputs during a migration or rewrite. | The old system can contain bugs; intentional differences need an approved list. |
The source favors E2E when prioritizing one investment and recommends giving the agent tools to start, drive and inspect the actual system. A repository-local verify skill can encode startup, smoke flows and evidence capture. This is situated practitioner advice, not a universal replacement for unit tests or a proof that E2E cannot be fooled.
Protect the judge from the change
- Read review instructions from the base/default branch, not the candidate PR: a change must not silently weaken the rules that judge it.
- Bind review evidence to a particular commit. New pushes invalidate the assumption that the reviewed artifact is the current one.
- Run cheap local tests, builds and lint before spending shared CI and review capacity.
- Let code owners define checks and explanations that reflect the problems they actually care about.
- Apply a reliability checklist by diff triggers rather than running every historical rule on every change. The article reports 39 items, commonly 3–10 relevant to a diff.
These controls complement Critical Thinking with Agents: independent standards and meaningful acceptance criteria matter more than a larger number of green checks.
Bounded repair and staged enforcement
The reported team workflow lets the agent attempt three repair rounds before involving a human. Review rounds reuse earlier findings and failure context; tuning one reviewer can be preferable to stacking more reviewers. These are examples of bounded iteration, not defaults adopted automatically by this wiki or its agent.
New commit gates start in observe mode to expose false positives before enforce mode. Verification stamps include none so missing evidence remains queryable. The described gate allows an explained exception and an escape after repeated blocks; it is not a hard guarantee that all commits are verified.
The source rejects ignoring flaky tests, adding arbitrary sleeps, or rerunning until green. It also reports deleting an unresolved flaky test with an explanation. That last practice can reduce coverage: deletion should not be mistaken for evidence that the underlying behavior is correct.
Evaluate the verifier and prune its rules
Replay historical incidents and reverted changes against the repository state at review time; include stable merged changes as controls against over-flagging. Keep tuning examples separate from a fixed evaluation set. Measure findings before and after verification and final filtering, since later stages can discard a correctly detected bug. Separate must-fix issues from nits and pre-existing problems.
Turn recurring human review comments and incidents into candidate rules only when the pattern is observable in a diff. The source describes requiring examples from at least two PRs, attempts to disprove candidate rules, and human approval. Periodically prune obsolete checks and instructions so the verification layer stays fast and relevant. This is the failure-to-scaffolding ratchet of Harness Engineering, applied to the judge itself.
Evidence and scope
The article reports roughly 90% of code on the Claude Code team being written and reviewed by Claude and up to a 10-point task-success improvement from a separate rubric grader in structured-file generation. These are source-reported internal results, not independently verified measurements or an established effect size for arbitrary software projects. The captured text uses a team first-person voice; its presence on Osmani's account does not establish that Osmani personally performed those team activities.
The practical synthesis is to make “done” executable, protect acceptance rules, attach evidence to the artifact, bound repair loops and evaluate the evaluator. None of this removes the whole-system invariant or human-ownership problem.
Mentioned in 3 pages
Engineering with agents
People & organisations