Shared-Machine Contention False RBAC Failures Automation: A Working Test, Answering Wrong Under Load

A continuous-integration test suite is meant to answer one question cleanly: does the logic under test behave correctly? That clean answer depends on an assumption easy to take for granted — that the TEST ENVIRONMENT itself has enough resources to run reliably. A CI runner sharing its physical machine with a live, running production system inherits whatever resource pressure production happens to be under at that exact moment, on top of its own testing workload — and a resource-starved environment can produce a test failure that looks, on its surface, identical to a genuine logic defect, while actually reflecting nothing more than an environment that briefly did not have enough memory or CPU time available to answer a query promptly.

Running continuous-integration checks on the same physical machine as a live production system is a genuinely reasonable, cost-effective infrastructure choice, especially for a system whose CI workload is not large enough to justify a fully separate, dedicated build machine on its own. The specific risk that choice introduces is that CI's own resource needs and production's own live resource consumption are no longer independent of each other — a moment of unusually high production activity, or an unusually resource-heavy combination of concurrent CI steps, can starve a specific query of the memory or attention it needs to return an answer within its expected time, producing a failure that looks, from the test result alone, exactly like a genuine defect in the logic being tested rather than a transient symptom of shared-resource contention.

Two real, dated test failures, at specific measured timestamps, both reported an alarming result: a specific administrator's role-based access, correctly and continuously in place for weeks according to independent records, was being read back by a real-database access-control test as entirely absent. Both failures occurred specifically INSIDE actual CI executions. The CI runner responsible genuinely shares its physical machine with Centriu Helix's own live production environment, and during the specific window each failure occurred, package installation, type-checking, and linting were all running concurrently, competing for a measured, small pool of free memory. Under that specific contention, the test's own database access-check query simply did not return an answer within its expected time.

The actual cause was confirmed directly: reproducing the identical check on the identical runner machine, but OUTSIDE the resource-constrained window of a live CI job, passed correctly every single time — direct, reproducible confirmation that nothing was ever wrong with the administrator's actual access. A timeout, not a denial, had occurred, and the test's own logic had no way to tell the two apart.

The fix teaches the underlying access-control tests to retry specifically and only under one precise condition: when the module's own access-check function reports its dedicated, distinct "could not verify" outcome — a specific signal the module had only just gained the ability to report as part of the same broader production audit that found this issue. A genuine, confirmed denial is deliberately never retried: a real "no" is valid and final on its very first answer, and retrying a genuine denial would risk masking an actual, real access-control defect behind a comforting-looking retry loop. If three attempts in a row still receive no definitive answer at all, the test fails loudly and explicitly, stating directly that the likely cause is network or resource contention rather than quietly passing, or — the specific failure mode this fix directly targets — quietly blaming the wrong system for a defect that was never actually there.

As the fix's own reasoning states directly: this is deliberately not the same thing as simply tolerating a flaky, occasionally-wrong test. It is a refusal to let a test fail with the WRONG accusation — the specific accusation that had already sent a real investigation looking for a role-based-access defect in a system where none actually existed.

Running the identical access-control check during a live CI job's own resource-constrained window, competing with package installation, type-checking, and linting for a small pool of free memory, produced a false "no access" result for a genuinely intact administrator role. Running the exact same check, on the exact same machine, outside that specific resource window, correctly and consistently confirmed the access was intact — direct proof the failure had nothing to do with the access-control logic itself.

Centriu Helix's real-database access-control tests now correctly distinguish a genuine access denial from a resource-driven timeout on their shared CI/production machine, retrying specifically the latter and never the former — closing a gap where two real, dated test failures had falsely accused a genuinely intact administrator role of being broken, and had sent a real investigation looking for a defect that was never actually there.

This automation covers specifically a working CI runner giving an intermittently WRONG answer under real resource contention on its shared machine — it is a distinct, independent finding from this pillar's separate companion page on Helix's CI runner being completely DEAD for hours the same broader sprint, where no answer of any kind was being produced at all. A reader looking for that different, infrastructure-availability finding should see that companion page directly; this page is about a runner that was working the whole time, but occasionally answering incorrectly under load.

Retrying any test failure indiscriminately is the simplest possible response to intermittent CI flakiness, and it is specifically the wrong answer for a test whose entire purpose is catching a genuine access-control defect — an indiscriminate retry would just as readily mask a real, confirmed "access denied" result as it would a genuine timeout, defeating the test's actual purpose in the process. Retrying only a precisely identified, distinct condition (a dedicated "could not verify" signal, built specifically to be distinguishable from a real denial) is more work to build, and is the only version of a retry that fixes the environment problem without simultaneously weakening the test's ability to catch the exact category of defect it exists to catch.