A Failure Taxonomy Is a Product Feature
Why a recovery workflow needs to distinguish missing evidence, disagreement, and infrastructure trouble.
Scope: Engineering or methodology note based on a September 17, 2026 snapshot. Source-level behavior is not a public availability or universal compatibility claim. Any scenarios are illustrative, not customer outcomes.
When a recovery comparison does not pass, the first question should be what kind of event occurred. Did the candidate behave differently? Did the observer lack a required capability? Did the browser fail before the comparison could be made?
Those situations can produce similar-looking logs and require very different responses. A product that preserves the distinction helps an engineer investigate the right problem instead of repeatedly editing a candidate that was never fairly evaluated.
A comparison needs two valid observations
Imagine an unchanged candidate and donor whose pages both require a resource that the measurement environment cannot observe. A refusal may be the correct result. Calling it a candidate regression would imply evidence the system does not have.
Now imagine that both observations are valid and a declared text landmark differs. That is a measured disagreement. The report should point to the assertion and the bound inputs, not dilute the result into a generic instability warning.
Retrying has a limited role
A retry can help investigate whether an environment is unstable. It should not automatically erase a reproducible counterexample. The frozen Ushman equivalence research explicitly treats stability analysis and candidate divergence as separate questions.
This distinction also helps avoid a bad incentive: making the dashboard greener by increasing retries until a failure disappears. A tool that produces evidence should preserve the meaningful failure and explain its frequency or conditions rather than vote it away.
Carry the category through the handoff
An actionable report can identify the failed measurement stage, the observed disagreement, or the unavailable evidence owner. That category should survive into product output and any later automation. Scraping an ambiguous log line should not be necessary to decide whether to recapture, repair the environment, narrow the scope, or investigate the candidate.
Kalu’s current source uses explicit blocked, unsupported, partial, and failed outcomes in its bounded commands. This is an architectural direction and a source-level observation, not a claim that every historical research diagnostic has already been harvested.
The product benefit is practical: fewer false accusations against the application, fewer accidental passes, and a clearer next step. Good failure reporting is not merely a developer convenience. It is part of what the tool is promising to know.