Finds the runs that stopped without saying so and puts them right — the
run-level twin of Oban.Plugins.Lifeline (which only rescues the Oban row).
Cron, every five minutes (mix phoenix_kit.update adds the entry to existing
hosts).
The sweeper only lists candidates (unfinished runs untouched for
@grace_seconds). Whether one needs help is decided by
PhoenixKit.Jobs.Engine.recover/4 under the run's row lock, from the run as it
is at that moment — a run a live worker advanced, rescued or claimed since the
listing is left alone, so a slow sweep can never undo a newer dispatch
(dev_docs/plans/2026-10-03-job-runs.md, §16 F1).
It reads each run's own Oban job (oban_job_id, checked to be this run's
RunWorker job of this generation), through Oban's configured repo and prefix
(PhoenixKit.Jobs.ObanStore), never "is there any job" and never a stale
heartbeat alone (§14 R5). When Oban cannot be asked — no instance runs in this
VM — every run that depends on a dispatch is left, one that never had a job id
included, so a sweep from a script cannot spend a rescue on a dispatch nothing
here could have made:
| the run | what it has | action |
|---|---|---|
| holds a queue claim | its job is executing | leave it (Lifeline owns a genuinely dead one) |
| holds a queue claim | its job is anything else | the batch died: release the claim |
| holds an inline claim | a lease still running | leave it |
| holds an inline claim | a lease that ran out (Run.lease_expired?/2) | the script died: release the claim |
| inline, between two batches (waiting out a delay) | a lease still running | leave it: the script will be back at wake_at |
| inline, between two batches | a lease that ran out | rescue — the dispatch carries the rest of the delay |
| waiting | available, scheduled, retryable, executing | leave it |
| waiting | discarded | its attempts are spent: the run fails |
| waiting | cancelled outside the engine | the run fails |
| waiting | completed, or gone, or of another generation | a lost dispatch: rescue |
A script (Jobs.run_inline/3) has no Oban job, so a missing job proves nothing
about it; its claim carries a lease (an hour without a Jobs.heartbeat/1) which
also covers the waits between its batches (wake_at plus the lease).
The limitation is deliberate: a crashed script's run stays held until its lease
runs out, and a new script may take it over then.
A rescue gives the run a new generation and dispatches it, and is counted on the
row (rescues), so the budget survives restarts; at @rescue_limit the run fails
instead of restarting for ever. A run changed in the last @grace_seconds is
left alone: its dispatch may be in flight.
Each pass stamps job_runs_last_sweep_at, which the Jobs page reads to warn when
no sweeper has been seen. The stamp is written without the settings history
(history: false): it moves on every pass, and a permanent entry per pass
would bury the Activity feed.
Summary
Functions
The setting the last pass stamps, an ISO 8601 timestamp.
One pass. Returns how many claims were released, runs failed and dispatches rescued. Public so a test, a Mix task or the Jobs page can run it now.
Functions
The setting the last pass stamps, an ISO 8601 timestamp.
@spec sweep(DateTime.t(), keyword()) :: %{ released: integer(), failed: integer(), rescued: integer() }
One pass. Returns how many claims were released, runs failed and dispatches rescued. Public so a test, a Mix task or the Jobs page can run it now.
:after_listing (a function, for tests) is called once the candidates are
listed and before the first is recovered, to let a test move the world in
between.