Job Lock Reclaim Automation: A Crashed Worker That Held the Key Forever

A locking mechanism meant to prevent two workers from claiming the identical job at the same time needs to answer one question correctly: is this specific lock still legitimately held? The simplest, most direct way to answer that is to check whether the lock has ever been explicitly released — and for the overwhelmingly common case, where every worker eventually finishes and releases its own lock cleanly, that simple check is completely sufficient. Its blind spot is narrow but absolute: it has no way to distinguish a lock that is still legitimately, actively held from one whose owner simply stopped existing mid-job, taking with it the only process that would ever have called release. Both look structurally identical to a check asking only 'has this been released' — the answer is 'no' in both cases, for entirely different, and very differently urgent, reasons.

Building a lock's uniqueness guarantee around "has this been explicitly released" is the natural, minimal implementation for the ordinary, successful case — every worker that finishes its job takes the deliberate final step of releasing its own lock, and the system behaves exactly as intended for as long as that always happens. The gap exists specifically for the class of failure that never gets a chance to reach that final, deliberate step at all: a crash, an unhandled exception outside any cleanup path, a forced process termination. None of those failures are rare or exotic in a real, long-running production system — they are the ordinary cost of running software at scale — and a locking mechanism that has no separate answer for them will eventually encounter one, at which point the affected resource becomes stuck until a person notices and intervenes by hand.

Centriu Helix's own background-job locking mechanism enforces, through a partial unique database index, that only one active lock can exist at a time for any given (organization, job type) combination — the guarantee that stops two workers from ever executing the identical job simultaneously. That index was scoped specifically around whether a lock's own `released_at` field had been set; it carried no independent logic checking a lock's own recorded expiration time during the acquisition path. A worker that crashed, was forcibly restarted, or otherwise ended abruptly while holding a lock — before ever reaching its own explicit release step — left that lock permanently marked as held, with the specific (organization, job type) pairing it represented unreachable by any future legitimate attempt until a person manually cleared the stale row.

The fix replaces the direct insert-based acquisition with a single, atomic database function, `helix_acquire_job_lock`. In one uninterruptible database operation, it checks whether an existing lock for the requested pairing has already passed its own recorded expiration; if so, that lock is released and the new one is granted as part of the SAME statement, rather than as two separate steps a crash or a race between two simultaneous callers could land between. If a lock exists and has NOT yet expired, the function correctly reports a genuine conflict (surfaced to the caller as the same `23505` code the original direct-insert approach already used, preserving the calling code's existing error-handling path without any change needed there). Because the function returns a set rather than a single row (a structural requirement of the atomic reclaim-or-insert logic), the calling repository code was updated to correctly read the first row of that set, with an explicit guard against the shape being empty.

Before the fix, a worker that crashed mid-job while holding a lock left that lock permanently marked as held — every future attempt at the identical (organization, job type) pairing was refused indefinitely, exactly as if a legitimate job were still genuinely running, with no way to distinguish the two short of a person manually inspecting and clearing the stale row. After the fix, the identical crash still leaves the lock behind, but the moment its own recorded expiration passes, the very next legitimate attempt reclaims it automatically, in the same atomic step that grants the new claim.

Centriu Helix's background-job locking mechanism now automatically reclaims a lock left behind by a crashed or abruptly-terminated worker the moment that lock's own recorded expiration passes, closing a gap where a single crash could permanently block every future attempt at a specific (organization, job type) pairing until someone noticed and intervened by hand — with the lock's original protection against two genuinely concurrent workers left completely intact.

This automation covers specifically Centriu Helix's own background-job lock reclaim mechanism. It is a distinct, differently-shaped concurrency fix from this pillar's separate wave 82 page on a CI gate silently disabling itself without a credential — that page concerns a testing/verification gap, not a runtime resource-locking mechanism.

A lock's uniqueness guarantee scoped only around explicit release is simpler to build and reason about, and it is entirely correct for as long as every holder always reaches its own release step. The moment any holder can instead end abruptly without ever reaching that step — an ordinary, expected failure mode in any real production system running at scale — that same simplicity becomes a permanent liability with no built-in recovery path. Checking a lock's own recorded expiration IN THE SAME ATOMIC OPERATION as attempting to acquire it is the only version of the two that both closes the crashed-worker gap and avoids reopening a race condition between two callers who might otherwise both see an expired lock as available at the same moment.