AI Skill Test-Execution Integrity Automation: A Coin Flip Decided Pass or Fail, Then Copied the Answer Key

A test suite with a green pass badge next to a version number is one of the most trusted signals in software — it's specifically the kind of evidence a team relies on to decide whether something is safe to promote or ship. That trust is only as sound as what the code labeled "test" actually does the moment it runs: nothing about a badge, a percentage, or a run-history entry can, by itself, distinguish a genuine comparison of a real output against a real expectation from a function that simply decided the outcome in advance and recorded a result to match. The entire value of a test suite lives in the gap between "looks like a test ran" and "a test genuinely ran," and that gap is invisible from every normal vantage point except reading the function itself.

Building a complete testing feature — defining test cases, storing run history, rendering a pass/fail badge, computing an aggregate success rate — is substantial, genuinely necessary work, and a placeholder decision function that just returns a plausible pass/fail ratio lets every one of those other pieces look and behave correctly from the very first demo. The run-history table fills in with rows, the badge renders, the percentage looks sensible — because nothing about how those downstream pieces work depends on whether the underlying decision came from a real evaluation or a coin flip; they only depend on receiving a record shaped like one. That shape-only dependency is exactly what allows a placeholder to survive well past the point where a real evaluation was supposed to replace it, since nothing anywhere in the pipeline ever has a reason to ask.

This gap surfaced from the same internal audit that rewrote Axiom's proof-of-action mechanism and its safety-scanner enforcement (both covered on this pillar's companion pages) — a feature not yet fully built presenting itself as complete. For the Skill Library specifically, the prior code inside `runTest` was, quoted directly: `const passed = Math.random() > 0.2` — an unconditional 80% pass rate with no relationship whatsoever to the skill's actual prompt, its version, or the specific test case being run. Worse, when that random roll happened to land on "pass," the code copied the test's own `expected_json` — the answer key — directly into `actual_json`, the field specifically meant to record what the skill actually produced. The "obtained" result displayed to anyone reviewing that test run was, by construction, always identical to the expected result, because it was never anything else. And this didn't stay confined to the screen: every run — genuine or not — was logged to the audit trail as a `skill_test_run` event carrying `passed: true`, a permanent record asserting that an evaluation had occurred and succeeded, when nothing had been evaluated at all. Anyone deciding whether to promote a skill version to production based on that run history would have been deciding based on statistical noise dressed up as measurement.

The fix replaces the entire decision with a real execution path. `runTest` now looks up the actual skill version being tested and its real `prompt_template`; if that version doesn't exist, it fails immediately with an explicit message that nothing was executed, rather than reporting any pass/fail result at all. If the version exists but its prompt template is empty, the function explicitly refuses — "there is nothing to execute; define the prompt before testing" — precisely because reporting a pass or fail for a test with no defined behavior would recreate the exact failure mode this fix exists to eliminate. When a real prompt does exist, it's rendered by substituting each double-brace placeholder name with the corresponding value from the test's own `input_json`, then sent to the same AI proxy the rest of the product already uses, explicitly configured for a temperature of zero and a JSON response format — deterministic, real generation, not a simulated one. If the AI backend itself fails to respond, that's recorded as an infrastructure failure with its own specific message, honestly distinguished from a skill genuinely failing its test, since a temporary outage in the underlying AI proxy says nothing about whether the skill's own prompt is any good. If the response comes back but isn't valid JSON, the function fails with a message saying so and still records the first 500 characters of what was actually returned, for debugging — rather than discarding it. Only once a real, parsed JSON response exists does the function compare it against the expected output, using a canonical, key-order-independent comparison that walks every expected field and reports the specific ones that are missing or mismatched, rather than a single undifferentiated "didn't match."

Critically, the recorded `actual_json` now always holds the real response the model produced — including on a pass, which is the exact case the old code fabricated by copying the answer key instead. The audit log entry logged for every run now reflects a genuine outcome: a real pass only when a real comparison found no divergence, a real failure with the real, specific reason otherwise. Confirmed directly against the current source: `Math.random()` is gone from the function entirely, `proxyGemini` is genuinely imported and called at temperature zero, and a real, wired button in `SkillLibrary.tsx` (`handleRunAllTests`) still calls this exact path today, running every defined test case for a skill version in sequence.

A skill's test expects a JSON object with three specific fields. Before the fix, running that test would report passing roughly four times out of five no matter how the prompt was worded, because nothing had ever actually asked a model to produce anything. After the fix, the same test genuinely sends the rendered prompt to the AI; if the response is missing one of the three expected fields, the test fails and the run's error message names exactly which field was absent, rather than a generic "didn't match."

A pass recorded against a skill version now means a real prompt was rendered, sent to a real model, and its real response genuinely matched the expected output field by field — not that a random number happened to clear a fixed threshold. Anyone reviewing a skill's test history or deciding whether to trust a version in production is now reading a record of what actually happened, including the model's real output on every run, not a copy of the answer key standing in for it.

This mechanism verifies that a skill version's output matches an exact expected JSON shape for the specific test cases someone has explicitly defined — it does not evaluate open-ended qualities like tone, creativity, or persuasiveness beyond whatever the expected JSON encodes, and it says nothing about how the skill behaves on inputs nobody has written a test case for. A team wanting broader, subjective quality evaluation of a skill's output will need a separate mechanism for that; what this fix guarantees is that the specific, defined comparisons it runs are genuinely run.

A pass/fail decision generated by a fixed-probability random function will always trend toward that same fixed percentage regardless of how good or bad the underlying prompt actually is — it carries zero information about quality, because it was never connected to quality in the first place. Centriu Axiom's fix replaces that with a real execution and a real comparison, so a skill's pass rate can now genuinely move in response to a prompt actually getting better or worse, because it's finally measuring the thing it always claimed to measure.