#10 Runner robustness: hung/killed jobs wedge the runner; add timeout, heartbeat, orphan requeue

closed

Opened by atgreen

Problem

A hung or killed job leaves the runner wedged and the work stuck, with no recovery:

Hung job never fails. When a job's step hangs (see the make build hang issue) or its container is force-removed, the runner does not detect the failure — the job stays running indefinitely and the runner stops taking new work. The workflow-completion hook never fires (so e.g. speculative dependency-fix builds never resolve).
No per-job timeout/heartbeat. Jobs carry a timeout field but a wedged step isn't reaped. There's no liveness heartbeat to detect a dead/stuck job.
Runner death orphans jobs. If a runner dies mid-job (e.g. pod restart), its assigned/running jobs are left stuck — they aren't requeued. (requeue-workflow-job exists for failed delivery, but not for runner death.)

Proposed

Enforce the per-job timeout on the runner (kill the step container + report failure → completion hook fires).
Runner→server heartbeat per active job; server reaps jobs whose runner is offline / heartbeat stale and requeues them (status → queued, runner_id NULL).
Make the runner robust to its step container disappearing (treat as step failure, not a wedge).

Comments (2)

atgreen3991818635

Status reconciliation (current main)

Re-checked this against the code. Two of the three concerns shipped after this issue was filed; the third is now implemented on branch fix-runner-job-timeout (commit 81b03ba).

① Per-job timeout enforcement — was the real gap; now fixed (81b03ba)

Per-step timeout: was enforced (main.lisp step deadline → terminate-process → exit 124 → failure), but the per-job timeout: was plumbed end-to-end (workflow.lisp → cave_workflow_jobs.timeout_seconds → TaskEvent.timeout_seconds) and then ignored by the runner — only the global 120-min reaper cap bounded a wedged, step-timeout-less job.

Fix:

Runner now reads task.timeout_seconds and bounds the whole step sequence with a job deadline. Each step's kill deadline is min(step, job); a job-deadline kill fails the job (overriding continue-on-error) and skips later steps. Reported status stays failure (the job-status CHECK has no timed_out); the log says "Job timed out after Ns".
Server stops coercing an unset job timeout to a low default (was 300s on WatchTasks, 60s on FetchTask) and sends 0 = unset, so enabling enforcement does not silently cap every timeout-less job (e.g. the cave build) at 5 min. Unset ⇒ no runner cap; the stale-job reaper remains the global backstop.

② Heartbeat + reaping of dead/stuck jobs — already done (e834df6, 188526d)

Runner heartbeat every 3s (update-runner-heartbeat), dead-idle-runner liveness probe (e834df6), cleanup-offline-runners every 30s, and reap-stale-workflow-jobs every 5 min requeueing assigned/running orphans with bounded attempts. Liveness is per-runner (connection-driven), not a per-job heartbeat RPC — a silently-wedged running job is caught at ~3-min stale window + reaper tick rather than instantly. Also folded in: the scheduler now ticks every 60s (was 300s) so the reaper's 120s claim interval is honored.

③ Requeue on runner death / orphaned jobs — already done (188526d)

requeue-workflow-job on delivery failure, plus the two reapers covering both assigned and running. Requeues with bounded attempts (cap 3) and fails the unrecoverable ones — slightly better than the original proposal (no poison-job loop).

Remaining

Nothing blocking. Once 81b03ba lands, ① is closed and this issue can be closed. Possible future polish (not tracked here): a true per-job liveness signal to cut the ~5-min worst-case reap latency for a silently-wedged running job.

atgreen3991835097

Deployed to production

The per-job timeout fix (commit 81b03ba, PR #19) is live: cave.moxielogic.com now runs 162562c (footer Cave 0.1.0-g162562c+dirty), and both the cave and cave-runner images were rebuilt at sha-162562c.

Status of all three concerns:

① Per-job timeout enforcement — implemented + deployed. Runner now bounds the step sequence by task.timeout_seconds; server sends 0 = unset (so timeout-less jobs are unaffected and the stale-job reaper stays the backstop). Of the current workflows, only .cave/workflows/release.yml sets a job timeout: (1800s), which is now enforced.
② Heartbeat + reaping — already shipped (e834df6, 188526d); scheduler tick tightened to 60s so the 120s reap interval is honored.
③ Orphan requeue — already shipped (188526d).

Closing as done. Note: the runner-side kill path is compile-verified and mirrors the proven step-timeout path but has not yet been exercised by a real timed-out job in prod — a deliberate timeout:+hanging-step test is the remaining confidence check (separate from this issue). Unrelated: the self-hosted CI hang that blocked the merge gate is issue #9 (build-step OOM), not addressed here.