Accepted Is Not Measured

Capturing an intent, approving it, and measuring its outcome answer different questions.

On September 7 a pull request in the Pathmode monorepo merged, and 42.5 seconds later the intent behind it flipped to shipped without anyone touching it. The merge webhook pulled the merged diff, graded it against the spec, scored 88, and carried four human review resolutions forward with zero unresolved failures and one item still unclear. The model assessed the diff; code validated citation locations and calculated the final score and pass decision. Those checks made the assessment inspectable. They did not establish that every judgment was correct.

That result was useful: an assessment of the implementation against the spec, with one uncertainty still visible. But it could not tell me whether the shipped change would produce the product outcome we wanted. That required a different kind of evidence.

Two metrics before implementation

Anthropic's AI-native SDLC playbook is now a fourteen-lesson course on Claude Academy. Lesson two, the first real move after the introduction, is capturing an idea as intent.md: the originator talks to Claude, Claude drafts the file from an organization template, the originator corrects it, and it gets committed with an author and a timestamp.

Two metrics come with that stage. The leading one is speed, measured from the first conversation to the committed file, and expected to fall from weeks to hours. The lagging one is survival rate, which the course defines precisely as "the share of intent.md files that the product owner accepts" into design rather than closes.

Both are measurable with an explicit recording convention. Time to capture needs the start of the conversation as well as the commit timestamp; Git alone cannot establish when an unrecorded conversation began. Acceptance share needs a consistent distinction between accepted and rejected proposals, excluding reviews still pending. Neither metric establishes whether a shipped change worked.

Acceptance rate describes selection at review. Reviewers may use substantial customer evidence, and changes in the rate can prompt useful questions about proposal quality or review standards. A high or low rate is not inherently good or bad. It does not, by itself, tell you what happened to customers after implementation.

Production signals need a link back to the decision

The maintain lesson describes monitoring production and process signals, detecting breaches, and routing findings for action. A finding can become a new intent.md and enter the pipeline. Its lagging indicators include findings that become merged fixes and repeat incidents by class.

That is a useful feedback loop. The narrower gap is that the lesson does not specify a systematic review of each shipped intent against its proposed product outcomes. A team could add that practice to the workflow, but detecting a new problem does not automatically answer whether an earlier change achieved its purpose.

An intent accepted in March might propose reducing checkout abandonment, ship in April, and leave abandonment unchanged. The system can remain operationally healthy throughout. Someone needs to connect that result to the original decision and ask what it means for the next one.

What checking an accepted intent actually takes

Pathmode calls it closed-loop coverage: the percentage of shipped intents that have source evidence, an exact implementation artifact, an adjudicated verdict on what was built, and post-ship outcome evidence. Four things chained together, from the evidence that provoked the decision to the result observed afterward. Coverage measures whether that record is complete, not whether every outcome succeeded. A well-documented result showing no improvement still closes a loop.

Keeping that chain useful takes more than agreeing on a file format.

The intent needs an identity that outlives its text. Files get rewritten. If the unit is a path in the repo, you cannot tell a spec that survived six revisions from a different spec that inherited the filename. Pathmode's intent.md carries id, version and specVersion in frontmatter for this. The id survives; the version changes.

The outcome has to have been stated before the build, in a form you can check. An intent that promises to improve things cannot be graded. One that names a baseline, a direction and a window can be. Agreeing on the check before implementation makes it harder to redefine success around whatever happened. The result may still be inconclusive, and an observed improvement does not by itself prove the change caused it.

Somebody has to come back. Name an owner and a review date, or configure an explicit automated follow-up. The owner can be the original reviewer, supported by an agent. Merging a pull request does not by itself make that later review happen.

A decision can record what would reopen it. Not every decision needs this and Pathmode leaves the field optional, because some calls are genuinely irreversible. But when a decision rests on evidence, and that evidence later changes or disappears, the record can say so. Pathmode flags exactly that case and puts the flag in front of the agent as well as the human.

What we have demonstrated so far

As of September 9, 2026, our recorded outcome measurements come entirely from internal use and demos. We have not yet demonstrated this loop with an external team. The mechanism exists; customer proof is still ahead of us.

Pathmode supports recording outcome measurements and surfaces unmeasured shipped outcomes for follow-up. That establishes a capability, not evidence that teams will use it. The next proof is a team returning to a shipped intent, recording what happened, and using that result in a subsequent decision.

Revisit one shipped intent

The course provides useful measures for capture and acceptance. Add a separate practice for the result after shipping. Readiness checks before implementation and outcome reviews afterward serve different purposes; neither substitutes for the other.

Pick one change shipped last month. Find the outcome it proposed, the evidence behind it, and the implementation that shipped. Record what you can observe now against the intended baseline and review window. If the data is missing or the result is inconclusive, record that explicitly and assign the next check.

Then make the product decision: keep the approach, investigate further, or revise it. Attach the reasoning to the same intent so the next person or agent can see what changed your mind.

The 42.5-second webhook result answered an implementation question. Whether the change achieved its product outcome remains a separate measurement, taken later, by someone who comes back on purpose.

Accepted is not measured.