Andrey Ashgaliyev

Wiki › Engineering with agents

Agent Verification Loops

Evidence-driven coding-agent loops with real-system execution, protected checks, commit-bound reviews, bounded repair and incident-based verifier evaluations.

IdeaUpdated November 17, 20261 source3 min read

An agent verification loop gives a coding agent executable evidence about its changes, feeds failures back into implementation, and routes unresolved or consequential issues to a human. The X Article “How the Claude Code team thinks about verification,” captured from Addy Osmani, describes this as infrastructure for scaling delegation rather than simply asking the model to be more careful. It connects Harness Engineering to Production LLM Reliability.

Three complementary kinds of evidence

| Check | What it establishes | Important limit |

| --- | --- | --- |

| End-to-end execution | A real user flow works in the running app, service or simulator. | Coverage is limited to exercised paths and observed outcomes. |

| Property-based tests | Generated inputs probe an explicit invariant: round trips, idempotency, equivalence or domain constraints. | A wrong or incomplete property remains a wrong or incomplete specification. |

| Differential tests | Old and new implementations agree on shared inputs during a migration or rewrite. | The old system can contain bugs; intentional differences need an approved list. |

The source favors E2E when prioritizing one investment and recommends giving the agent tools to start, drive and inspect the actual system. A repository-local verify skill can encode startup, smoke flows and evidence capture. This is situated practitioner advice, not a universal replacement for unit tests or a proof that E2E cannot be fooled.

Protect the judge from the change

These controls complement Critical Thinking with Agents: independent standards and meaningful acceptance criteria matter more than a larger number of green checks.

Bounded repair and staged enforcement

The reported team workflow lets the agent attempt three repair rounds before involving a human. Review rounds reuse earlier findings and failure context; tuning one reviewer can be preferable to stacking more reviewers. These are examples of bounded iteration, not defaults adopted automatically by this wiki or its agent.

New commit gates start in observe mode to expose false positives before enforce mode. Verification stamps include none so missing evidence remains queryable. The described gate allows an explained exception and an escape after repeated blocks; it is not a hard guarantee that all commits are verified.

The source rejects ignoring flaky tests, adding arbitrary sleeps, or rerunning until green. It also reports deleting an unresolved flaky test with an explanation. That last practice can reduce coverage: deletion should not be mistaken for evidence that the underlying behavior is correct.

Evaluate the verifier and prune its rules

Replay historical incidents and reverted changes against the repository state at review time; include stable merged changes as controls against over-flagging. Keep tuning examples separate from a fixed evaluation set. Measure findings before and after verification and final filtering, since later stages can discard a correctly detected bug. Separate must-fix issues from nits and pre-existing problems.

Turn recurring human review comments and incidents into candidate rules only when the pattern is observable in a diff. The source describes requiring examples from at least two PRs, attempts to disprove candidate rules, and human approval. Periodically prune obsolete checks and instructions so the verification layer stays fast and relevant. This is the failure-to-scaffolding ratchet of Harness Engineering, applied to the judge itself.

Evidence and scope

The article reports roughly 90% of code on the Claude Code team being written and reviewed by Claude and up to a 10-point task-success improvement from a separate rubric grader in structured-file generation. These are source-reported internal results, not independently verified measurements or an established effect size for arbitrary software projects. The captured text uses a team first-person voice; its presence on Osmani's account does not establish that Osmani personally performed those team activities.

The practical synthesis is to make “done” executable, protect acceptance rules, attach evidence to the artifact, bound repair loops and evaluate the evaluator. None of this removes the whole-system invariant or human-ownership problem.

Mentioned in 3 pages

Engineering with agents

People & organisations

Sources

  1. How the Claude Code team thinks about verification