Agent Benchmark Execution Honesty Gate Automation: A Score of 85-100, Guaranteed, Whether or Not the Agent Ran

A benchmark score exists specifically to answer a consequential question: is this version of the agent good enough, and did the most recent change make it better or worse? Every decision built on top of that score — setting a baseline, flagging a regression, deciding whether a change is safe to release — inherits whatever the score actually measured. A benchmark score computed independently of the agent's real behavior doesn't just fail to help answer that question; it actively produces a confident-looking wrong answer, because nothing about a random number in a fixed range signals that it isn't connected to anything real.

Building a genuine benchmark-execution engine for an AI agent means orchestrating the agent's actual real pipelines — each with its own real side effects, its own real failure modes, its own real latency — and then comparing genuinely produced output against a defined expectation. That is substantial, multi-system engineering work. A placeholder that instead draws a high, plausible success rate at random and marks outputs as matching accordingly requires none of that, produces a complete-looking result immediately, and — critically — tends to look BETTER than a genuinely working agent would, since a random draw between 85 and 100 percent has no ceiling imposed by the agent's actual, imperfect real-world performance. That combination — easy to build, and flatteringly high-scoring — is exactly what lets a placeholder survive well past the point a real engine was supposed to replace it.

This gap surfaced from the same internal audit that fixed the Outcome Simulator's randomly-generated scores and Axiom's two fabricated AI-recommendation failure paths, both covered on this pillar's companion pages — the same underlying pattern, applied here to an internal engineering tool rather than a customer-facing feature. The prior code's own header comment was direct about what it was: "this does NOT invoke the Axiom agent. It fabricates outputs from Math.random() to drive the scoring/UI demo," with the Benchmark screen showing a matching "execução simulada" disclaimer. But, exactly as with the Outcome Simulator, the disclaimer stayed on the screen while the number it described did not: the fabricated score became a `baseline`, and every subsequent run's equally fabricated score was compared against that baseline to decide whether a "regression" had occurred.

The mechanics of the fabrication, read directly from the removed code: a success rate was drawn once per run, uniformly between 85% and 100% — `0.85 + Math.random() * 0.15` — and then every boolean output the benchmark case expected was independently marked true or false by comparing a fresh random draw against that rate. The consequence is exact and unconditional: because the success rate itself was never allowed to fall below 85%, EVERY benchmark run scored somewhere between 85 and 100, regardless of whether the agent behind it was performing well, performing badly, or — as was actually the case — never being invoked in the first place. A baseline set from one random run and a later score from a second, independent random run then fed a genuine-looking regression calculation: two random numbers being compared could produce "regression detected" or "no regression" by pure chance, in either direction, with zero relationship to whether the agent had actually changed at all.

Rather than treating the fix as an opportunity to quickly wire up something resembling real execution, the team made a deliberate choice recorded directly in the fix's own reasoning: building genuine benchmark execution requires orchestrating the agent's actual pipelines — template application, sandbox execution, guided generation — each with its own real side effects, and comparing genuinely produced output against the case's defined expectation. Attempting that as a fast follow-on to this specific fix, without doing it properly, would — in the fix's own words — "produce another layer of lies, just harder to find" than the one being removed. So the function that used to fabricate outputs now does exactly one thing: it throws an explicit error stating plainly that real benchmark execution is not yet implemented, and that previous scores came from randomly generated outputs that never measured the agent at all.

The surrounding run-recording logic, unchanged by this fix and read in full for context, already had a genuine try/catch around the execution step: when the execution function throws, the run is recorded with a `failed` status and the real error message is stored directly in that run's own result record — meaning a benchmark attempt today shows up honestly in the run history as a failure with a specific, truthful explanation, rather than silently disappearing or continuing to display a fabricated score. Because that failure is thrown before any score is calculated, the regression check further downstream — the exact mechanism a randomly generated baseline used to corrupt — is never reached at all for a failed run, so no fabricated regression verdict can be produced from it either.

A change to the agent's pipeline introduces a real regression — a step that used to work now fails outright. Before this fix, running the benchmark against that broken pipeline would still very likely land somewhere in the 85-to-100 range, since the score never depended on the agent actually being called. After the fix, running the same benchmark honestly fails, recorded as a failed run with a specific reason — rather than reporting a healthy score for a pipeline that no longer works.

A benchmark score, when one is eventually available, will only ever come from watching the agent do something real — not from a random draw disguised as a demo. In the meantime, a team looking at benchmark run history sees an honest, specific statement that execution isn't implemented yet, rather than a healthy-looking number in the 85-to-100 range that never meant anything and could not have caught a real regression even by accident.

This fix removes a fabricated scoring mechanism and replaces it with an honest refusal — it does not, itself, deliver a working benchmark-execution engine. A team specifically needing a functioning agent-performance benchmark today will need that built as its own, separate, properly-scoped piece of work; what exists now is the honest absence of a score, not a substitute measurement.

Replacing one fabricated scoring mechanism with a second, hastily built approximation risks producing a number that looks more credible while still not measuring the real thing — a subtler, harder-to-catch version of the same underlying problem. Centriu Axiom's fix deliberately avoids that trade: rather than building a new simulation to replace the old one, it states the true current limitation directly, leaving the door open for a properly scoped real implementation later instead of a rushed one now.