AI Action-Impact Estimation Integrity Automation: A Test Suite That Passed Because the Numbers Were Invented

Labeling an estimate as "illustrative" on the screen where it first appears is honest and genuinely useful — it tells the person looking at it, in that moment, not to treat the number as a guarantee. What that label cannot do is follow the number to every other place it subsequently travels: into a history table, into an audit log, into a summary read aloud to a client on a call. Each of those destinations receives the number itself, not the caveat that accompanied it on the original screen, and a number sitting alone in an audit record or a client conversation reads as a measurement whether or not it started life as a randomly generated placeholder.

A test that checks "does this value exist, and does it fall between 0 and 100" is a perfectly reasonable, standard test to write for a scoring feature — it catches missing fields, wildly out-of-range values, and type errors. The problem is that this exact test provides zero information about whether the score reflects anything real, because a function that does nothing but call a random-number generator inside that same 0-to-100 range will pass it every single time, with total reliability. A test suite built entirely around "is the shape right" rather than "is the content right" can report full, green, passing coverage for a feature that has never once measured the thing it claims to measure — and nothing about a passing test run signals that gap to anyone looking at the results.

This gap surfaced from the same internal audit that rewrote Axiom's proof-of-action mechanism and closed its Trust Center, Skill Library, and Policy Copilot gaps (covered on this pillar's companion pages) — a feature not yet fully built presenting itself as complete. For the Outcome Simulator specifically, the prior code's own header comment was unusually candid about the defect it described without fixing: "the impact dimensions and scenarios below are HEURISTIC ESTIMATES... numeric scores are randomized within risk-weighted ranges... so the same input yields different scores on each run." The comment was accurate and the disclaimer it described was genuinely shown on screen — but neither fixed the underlying number, and the audit found that the score still flowed, unlabeled, into the simulation's own history, into an audit trail entry recording an `overall_score` as if it meant something, and into whatever a person told a client based on it.

The fix replaces the six-dimension, three-scenario random generator with a real request to the same AI proxy already used elsewhere in the product. The action's actual context — its type, description, declared risk level, whether it requires approval, its rationale, and its full payload — is assembled into a structured prompt asking the model to score the identical six dimensions and produce the identical three named scenarios (optimistic, base, pessimistic), explicitly instructed to justify every score against what is actually in the context rather than inventing anything not provided. The request is made with `temperature: 0`, specifically so the same action, submitted twice, produces the same reading — the exact property the prior random-based version could never offer, and the property a genuine measurement is expected to have.

The response is then validated field by field before anything is trusted: every one of the six required dimensions must be present by name with a numeric score between 0 and 100, or the function throws naming the specific missing dimension; every scenario must carry a valid probability between 0 and 1 and a valid impact score between 0 and 100, or the function throws naming exactly which value was invalid. None of this validation existed as meaningfully as it needed to before, because a random-number generator can't produce a malformed number — the entire category of "what if the response is wrong" barely applied to code that made up its own numbers rather than asking anything external for them. If the AI proxy itself fails to respond at all, the function throws immediately, with an explicit comment recording the reasoning directly: falling back to a randomly generated placeholder inside the catch block "would recreate the exact defect exactly where nobody is looking." No simulation record is created in any of these failure cases.

The accompanying test file makes the defect and its fix unusually explicit for anyone reviewing this feature's history: a new comment block states plainly that the ORIGINAL test suite "only checked that 6 dimensions existed with a number between 0 and 100 — which a random generator satisfies by construction," meaning the old tests were passing BECAUSE the values were fabricated within range, not despite it. The new tests specifically check the opposite of what the old ones checked: that the service genuinely refuses (and records nothing) when the AI doesn't respond, when its response isn't valid JSON, when a required dimension is missing, and when a score falls outside the valid range — plus two dedicated determinism tests confirming the exact same action produces the exact same overall score across two separate runs, and that a deliberately low-scored mock response and a deliberately high-scored one produce correspondingly low and high results, rather than results clustering near a fixed range regardless of input.

A high-risk campaign action is simulated for impact, producing an overall score. Before the fix, simulating that identical action again moments later could produce a meaningfully different score purely by chance, with no way to tell whether the difference meant anything. After the fix, simulating the same, unchanged action again produces the identical score — and a genuinely different action, such as one carrying a materially different risk level or payload, produces a genuinely different one.

An impact estimate presented before an action executes now reflects an actual model reading of that action's real context, not a randomly drawn number inside a fixed band — and it carries that same integrity into the simulation history and audit trail, not only on the original screen where a disclaimer happened to be visible. A person relying on the score to decide whether to proceed, or repeating it to a client, is repeating a genuine estimate rather than a number invented to fill a UI element.

This mechanism remains, honestly, an AI-generated heuristic estimate of a single action's likely impact before it runs — it is not a prediction trained on the organization's own historical outcome data, and the interface's "illustrative estimate" framing remains accurate and necessary even after this fix. A team specifically wanting a statistically trained forecast based on past campaign results will need a different mechanism; what changed here is that the estimate now genuinely responds to the action being evaluated, not that it became a historical-data-driven prediction.

A test verifying only that a score exists and falls in a valid range will pass with total reliability for a feature that has never once performed a real evaluation, since a random-number generator inside that range can never fail it. Centriu Axiom's fix pairs the real estimation rewrite with tests built around the two properties a fabricated number can never honestly have — genuine refusal when there is nothing real to report, and genuine determinism when there is.