#9 CI `make build` hangs in the cave-runner pod (nested podman)

open

Opened by atgreen

Symptom

make build in the CI workflow (.cave/workflows/ci.yml, step "Build Cave") hangs in the on-host cave-runner pod's nested container. Observed two instances on a fix-branch speculative build: one ran ~45 min, an earlier one ~5 h, neither completing. The "Install dependencies" (ocicl install) and ls steps succeed; the build step never finishes. This also explains why a CI backlog never drained — every CI run wedged on this step.

Context

Runs inside cave-runner (privileged, podman-in-podman, User=0) → nested fedora-cave-builder container.
make build produces 4 artifacts: cave-server + cave (SBCL save-lisp-and-die), cavectl + zoekt-git-index (Go).
podman top on the job container only shows the sleep infinity keepalive (steps run via podman exec), so the actual hung process wasn't visible that way.

Likely suspects

SBCL image dump is memory-heavy; the nested container may be memory-starved/thrashing.
A Go build fetching modules with no network/proxy in the nested env (less likely — clone/pull worked).
A build step waiting on stdin.

Next steps

Reproduce by running the CI step manually in the builder image on the host and watching where it stalls; add resource limits/visibility; consider a build memory floor or ulimit/cgroup tuning for the runner pod.

Comments (3)

atgreen3990996443

Dig-in findings (root cause):

It is not the build — every target compiles fine in isolation in the nested builder image: cave-server (SBCL save-lisp-and-die), cave, cavectl, zoekt-git-index all exit 0 (cave-server in <6 min). Network/DNS/Go-proxy all reachable from the nested container.

Two real causes make the CI job hang in practice:

fuse-overlayfs workdir I/O is pathologically slow. The job runs make build against -v workdir:/workspace, and the runner's workdir sits on fuse-overlayfs in the nested pod. An I/O-heavy build (SBCL + Go compile thousands of files) crawls — observed ~47 min and counting vs ~6–10 min on the container's native overlay. Fix options: put the runner workdir on tmpfs or a native-overlay volume, or build in the container FS and copy results.
Concurrent builds + zombie containers exhaust the 2-CPU/limited-RAM host. The original "5-hour hang" was contention: a cancelled job's container (cave-job-110) kept running 5 h after cancel, alongside a 26-job backlog of concurrent heavy SBCL builds → thrash. (See the runner-robustness issue for the cancel-doesn't-kill-container and orphan problems.)

So the immediate hang is the slow workdir FS; the catastrophic multi-hour hangs were resource contention from un-reaped concurrent/zombie builds.

atgreen3990999125

Update: an uninterrupted single build also wedges — log frozen ~40 min on the same line: ; compiling file ".../sly-.../slynk/slynk.lisp", with no further output. FASLs write to /root/.cache (fast container FS); source is read from /workspace (fuse-overlayfs). So the freeze is specifically while compiling the sly/slynk dep in the job environment, even though the identical build completes in an isolated podman run (native overlay, <6 min).

Prime suspects: (a) memory pressure by the time the dep tree reaches slynk → GC thrash/stall on a 2-CPU/limited-RAM host; (b) a load/compile-time form in slynk interacting badly with the mounted FS. Worth: cap/observe container memory, persist the ocicl FASL cache across jobs (avoid recompiling the whole tree), and move the workdir off fuse-overlayfs. Also: is sly even needed in the runtime dep set, or can it be dropped from the build?

atgreen3991005259

FASL-cache persistence is implemented + deployed, but insufficient on its own — here's why.

I mounted a persistent /root/.cache/common-lisp into job containers (confirmed live) and seeded it (76 MB, /workspace-keyed). It still recompiles the whole tree. Two compounding reasons:

ocicl/ is gitignored, so each job's fresh clone has no deps → ocicl install re-downloads/extracts them with fresh mtimes every build.
ASDF invalidates FASLs by source mtime. New dep mtimes > cached FASL mtimes → recompile, despite the cache being present and correctly keyed.

So the persisted FASL cache is necessary but not sufficient. The real fix needs dep persistence + mtime stability, e.g. one of:

Persistent per-repo workspace: git fetch && checkout into a reused workdir instead of fresh-cloning each job → ocicl/ persists, only changed files get new mtimes, FASL cache stays valid. (Cleanest; biggest win; also speeds clones.)
Bake deps + precompiled FASLs into fedora-cave-builder (image layer = stable mtimes) → every build starts warm.
Persist the ocicl systems dir as a volume and normalize dep mtimes after install (e.g. touch -d @1 ocicl -R) so cached FASLs win.

Net: the FASL-cache mount stays (it's required), but the effective fix is persisting the deps too — recommend the persistent-workspace approach.