Atlas Durable Production Resume-Queue Automation: "I Had to Ask Four Times"

Making a specific failure rare is real, valuable engineering — and it is a completely different achievement from making a system able to recover automatically the next time that failure happens anyway, because rare is not the same as impossible. As long as finishing an interrupted job requires a person to notice it happened and manually intervene, the client experience during that gap is identical whether the crash occurs once a year or once a week: work goes out incomplete, and someone has to ask, and ask again, before it's actually finished.

Every piece of a multi-image production is already saved to the database the moment it finishes generating, and the existing interface already calculates exactly which pieces of a given production are still missing — the parts needed to finish a job that stopped partway through had always existed. What was missing was something running that same calculation from the server side, on a schedule, without waiting for a person to open the item and click through it manually.

The fix adds a claim function that looks specifically for a production sitting in an in-progress or failed state, with at least one image genuinely missing, untouched for more than 12 minutes — deliberately long enough that a production still actively generating (which saves each finished piece immediately) is never mistaken for a stalled one. A claimed production is marked with a lease: a timestamp saying how long this specific attempt owns the job, so if the reconciling process itself dies partway through — the same class of failure this whole fix exists to survive — the lease simply expires and the next scheduled cycle picks the job back up, rather than leaving it locked forever by a process that no longer exists. `FOR UPDATE SKIP LOCKED` at the database level ensures two overlapping cycles never claim the same production at once. An attempt counter caps retries at three, with a 15-minute wait built into the same lease mechanism between attempts, so a production failing for a genuine reason — the image provider refusing a specific prompt, for instance — cannot loop indefinitely and burn image-generation credit chasing a failure that retrying will never fix.

The age ceiling was not part of the original design — it was added the same day, before the mechanism was ever switched on, specifically because checking what it would have acted on first showed something the original design hadn't accounted for: all 8 real incomplete productions sitting in the database at the time were 13 to 22 DAYS old, not minutes or hours. Resuming work that stale is not a rescue — it's regenerating images against a brief the client may have already changed, on the client's own account, burning real credit for output nobody is still waiting on. The shipped version only claims a production interrupted within the last 6 hours; anything older is left alone entirely, on the reasoning that resuming only makes sense for work someone is still genuinely expecting.

When a claimed job runs, each missing image is regenerated one at a time — deliberately not in parallel, because generating an entire batch simultaneously is what exhausted the server's memory limit in the first place and started this whole chain of problems. The regeneration call itself only ever targets a card with no saved output, which makes the whole mechanism idempotent by construction: resuming a production can never overwrite art that already exists, and can never charge twice for the same image. Every attempt — whether it fully finishes the job, partially finishes it, or gives up after the third try — writes a plain-language entry to that item's own visible history, naming exactly what happened, so nobody looking at a piece of finished art later has to wonder why it appeared without a person having done anything.

A client requests a three-image carousel. The server process handling the request is interrupted after the first image finishes and saves correctly; the second and third never generate. Before this fix, that carousel simply sits incomplete until the client notices, asks, and someone opens the item to manually retry it — the real, quoted sequence that took roughly four requests to resolve. After the fix, the next scheduled cycle — within minutes — finds the same production sitting idle with two images missing, regenerates exactly those two, and the carousel is complete before anyone has to ask even once.

A production interrupted mid-batch — by a deploy, a restart, or any other cause — now finishes on its own within minutes for the vast majority of real cases, closing a gap where the only previous path to completion was a client asking, sometimes repeatedly, before a person manually intervened.

This automation resumes a production that stopped partway through with real, salvageable partial output — it does not retry indefinitely against a genuine, repeated production failure, and it deliberately does not touch anything abandoned for more than 6 hours, on the assumption that stale work is no longer what a client is waiting on.

Removing the specific cause of a crash is the right first move and a genuinely different achievement from building the capability to recover when some other, unforeseen cause interrupts the same job next month — no single fix to one cause makes a system durable against every future cause of the same category of failure. Centriu's approach treats "finish what already has real partial output" as its own capability, independent of why the interruption happened, which is what actually stops a client from ever needing to ask twice, let alone four times.