Deal Lifecycle Automation Rule Execution and Crash-Safe Retry Automation: Every Run Ends Completed, Failed or Skipped — Never Lost

A CRM automation rule that "usually" works is worse than no automation at all, because a team stops manually checking the thing it was built to replace. The real engineering question behind any such rule is never just "can it evaluate a condition and take an action" — it is what happens when two background passes run close together, what happens when the worker crashes mid-action, and what happens when the deal the rule was about to touch was deleted a second before it got there. Centriu Orbit's automation-rule engine is built around those specific failure modes, not just the happy path.

A basic version of "if X happens, do Y" is a modest afternoon of work. A SAFE version — one where a crashed worker cannot cause a duplicate execution, where a race between two worker passes cannot let the same queued run be claimed twice, where a rule change is respected even for work already sitting in the queue, and where a partial failure is reported clearly instead of leaving mixed state — is a meaningfully larger investment most CRM add-ons never make. It is also easy to get subtly wrong in ways that only surface under real production load: an optimistic "select the next item, then update it" claim pattern looks correct in testing and then quietly double-processes a run the first time two worker instances happen to run within the same second.

A rule is defined against one of four triggers (deal created, deal stage changed, deal stale past a configured number of days, or activity overdue past a configured number of hours), an optional set of condition groups (an "all" group requiring every condition to match, an "any" group requiring at least one — each condition compares a deal field like value, priority or status against an operator and a value), and an ordered list of actions from a fixed set of four: create a linked follow-up activity, log a system note on the deal's own timeline, change the deal's priority, or reassign its owner.

When a trigger fires, a run is queued rather than executed immediately. A background worker then claims a batch of queued runs atomically — using a database-level locking mechanism that guarantees no two worker passes can ever claim the same run, replacing an earlier design that briefly allowed exactly that race. For each claimed run, the worker independently re-verifies that the automation still belongs to the same organization as the run (a defense-in-depth check that does not simply trust the join), skips (rather than fails) the run if the rule has since been paused or the target deal or activity no longer exists, and re-evaluates the rule's conditions AT THIS MOMENT against the entity's current state — not the state it was in when the run was originally queued. Only if conditions still match does it execute every action in the run in order; if any single action fails, the entire run is marked failed with that specific error preserved, rather than reporting success for the actions that happened to work.

A run that gets stuck mid-execution — a worker instance that crashed or timed out partway through — is automatically detected once it has sat in the "running" state for more than five minutes and re-queued for the next pass. This can happen up to three total attempts; beyond that, the run is marked permanently failed with a specific, stated reason, so a genuinely broken rule surfaces clearly instead of silently retrying forever or disappearing from view.

An admin sets up a rule: when a deal in the "Proposal Sent" stage goes 5 days without an update AND its value is at or above a configured threshold, create a follow-up activity due the next day and log a note on the deal's timeline. A qualifying deal crosses the 5-day mark and a run is queued.

A background worker instance claims a small batch of queued runs, including this one, using the atomic claim step — guaranteeing no other worker pass could have claimed the same run at the same moment. Partway through executing this specific run's actions, the worker instance crashes. Five minutes later, the reaper detects the run still marked "running" with no completion recorded, and re-queues it for attempt number two. The next worker pass claims it again, re-checks that the deal still exists and still meets the stale-and-value conditions (it does — nothing changed in the interim), and this time completes both actions: the follow-up activity is created, and the timeline note is logged. The deal owner sees both changes reflected on the deal with no indication anything went wrong, because from their side, nothing did.

A stale deal, an overdue activity, a stage change or a new deal can each trigger a real, logged action without anyone remembering to check for it manually — and a worker crash mid-execution results in exactly one clean execution once recovery completes, never a silent miss and never a duplicate. Every fired rule leaves a visible trace on the deal's own timeline, so nobody has to trust that "the automation probably ran."

A team that needs an automation to reach beyond deals and activities — updating a contact record, sending an external message, calling a third-party API — will not find that here: the action set is deliberately fixed to four CRM-native operations, not an open-ended or custom-code automation surface. A team looking for cross-channel, multi-system no-code automation (marketing sequences, transactional messaging, cross-product workflows) wants Centriu Flow instead — this feature is scoped specifically to what happens to a deal or activity inside Orbit itself.

Checking for stale deals or overdue activities by eye depends entirely on someone remembering to look, and a background script without real safeguards can just as easily double-fire on a race condition or silently stop after a crash as it can help. Centriu Orbit's automation-rule engine is built around the specific failure modes that make background automation risky — an atomic claim preventing double execution, conditions re-checked at the moment of action, and a bounded, visible retry-then-fail path for anything that gets stuck — so the team's trust in "the rule ran" does not depend on nothing ever going wrong underneath it.