Stuck-Generating-Status Self-Healing Watchdog Automation: The Bar That Froze at 95%

A progress bar that intentionally approaches but never touches 100% — because the real completion signal comes from the backend, not from elapsed time — is a reasonable design for representing genuine uncertainty about exactly how long a generation step takes. It becomes indistinguishable from a permanently broken process the moment the backend step it's waiting on dies without ever sending that completion signal: the bar has no way to know the difference between "still legitimately working" and "will never finish," because both look, from where it's sitting, like the identical state of simply not being done yet.

The specific incident traced back to one real item that had previously been produced as a Reel-format video, which — as part of that earlier production — had its own dedicated video-cover project created and stored, tagged with a distinct project kind. When that same item was later re-produced in its current format, the lookup meant to find "does this item already have a card project" queried without filtering by kind at all, and pulled back the leftover video-cover project instead of creating or finding a genuine one for the new production run. Generation proceeded against that stale project, and the step responsible for marking a plan item as produced recognized the video-cover kind specifically and exited early — a shortcut that made sense for its original purpose but, applied to the wrong project, meant the actual plan item's own status was never touched. It sat in "generating" indefinitely, with the interface's asymptotic progress bar having no way to distinguish that from a process still legitimately running.

A second, independently damaging defect sat in the same lookup: it used a fetch expecting at most a single matching row, without checking whether that fetch returned an error. With two card projects already existing under the same item — a state that existed for real, for two separate items, at the time of the fix — that strict single-row fetch failed with an error nobody was reading, the calling code interpreted the failed lookup as "no existing project found," and created a brand new one on every single click, multiplying the underlying conflict every time someone tried to recover by clicking produce again.

The fix corrects the lookup itself: it now explicitly filters out the video-cover kind, orders by recency and takes the most recent genuine match, and checks for a query error explicitly rather than assuming an empty result means nothing exists. But the fix doesn't stop at this one traced cause. A migration adds a scheduled watchdog, running every five minutes, that finds any item that has been sitting in a generating state for more than 20 minutes — for ANY reason, not specifically this one — and returns it to its prior approved state automatically, writing that correction into the item's own history so a person checking later can see exactly when and why the system stepped in, rather than the item simply appearing to have quietly fixed itself. This exists specifically because the underlying CLASS of root cause remains structurally possible: there is no dedicated worker queue behind this production step, the route fires an internal request without waiting on its own process's lifetime to guarantee delivery, and a deploy or restart landing mid-flight can kill the work silently — two such restarts happened on the very day this was fixed. Rather than treating each specific way that can happen as something to patch one at a time, the watchdog treats the SYMPTOM — stuck in generating too long — as the thing worth guarding against directly, regardless of which specific cause produces it next.

An item that was once produced as a video gets re-produced later in a different format. Before the fix, the production route quietly reuses the old video-cover project left over from that earlier run, generation proceeds against it, and the actual item is never marked complete — the progress bar sits at 95% indefinitely, and refreshing the page only restarts the identical, doomed process. After the fix, the correct, current project is found and produced against directly, and even if some other unrelated failure left an item stuck, the watchdog would return it to a workable state within 20 minutes on its own.

A Centriu Atlas content item can no longer get permanently stuck at 95% because of a stale leftover project from an earlier production run, and any item that does get stuck generating for any other reason is automatically returned to a workable state within 20 minutes, with the correction logged rather than left as an unexplained mystery.

The watchdog corrects an item's STATUS when it's been stuck too long — it does not diagnose or fix whatever underlying process actually failed to complete; a person still needs to decide whether to re-attempt production once an item is returned to its prior state.

Fixing only the specific stale-project lookup that caused this one traced incident would have left every OTHER way a generation step can silently die — a mid-flight deploy, a fire-and-forget request that never completes, a future bug nobody has found yet — free to produce the identical stuck-forever symptom again. Centriu's fix does both: correct the specific traced cause, and add a time-based watchdog that treats the symptom itself as worth guarding against, independent of which cause produces it.