#254 GC never returns pages to the OS and has no RSS ceiling — runaway loads OOM the whole machine

closed

Opened by atgreen

Two host-level OOM kills on 2026-07-08 during the asdf-operate quote-pool debugging campaign:

16:21 funcl pid 772001 killed at anon-rss 18.7 GiB (total-vm 28.5 GiB)
15:49 gdb pid 746295 killed at anon-rss 19.3 GiB (it was digesting that same funcl heap)

Host is 30 GiB RAM + 8 GiB swap, so a single runaway funcl takes the desktop down with it.

Two contributing runtime gaps (both distinct from whatever bug makes the corrupted load allocate without bound):

No page release. src/cold/gc.lisp has no madvise(MADV_DONTNEED)/munmap anywhere — once a page in the 24 GiB MAP_NORESERVE arena (cave#214, elf-tiny.lisp +gc-collectable-heap-size+) is touched, RSS never comes back down. Reclaimed/demoted regions stay resident, so RSS is a permanent high-water mark of cumulative frontier travel, not live data. The start map (1 byte per 8 heap bytes) adds ~1/8 more on top.
No self-imposed RSS ceiling. The only backstop is the kernel OOM killer at machine scale. A touched-pages budget (env var, e.g. FUNCL_MAX_HEAP) that routes to the existing OOM trampoline (gc.lisp gc-emit-oom path) would turn "machine dies" into "funcl reports heap-exhausted", which is also strictly better for debugging the runaway itself.

Interim workflow containment (verified working): run repros and gdb under systemd-run --user --scope -p MemoryMax=6G -p MemorySwapMax=0 -- build/funcl ... so the cgroup kills only the repro at 6 GiB.

Comments (3)

atgreen3992534024

FUNCL_MAX_HEAP ceiling IMPLEMENTED (2026-07-08, working tree; suite: all 8 shards green).

Mechanism:

gc-emit-init-linux captures &envp[0] (rsp + 8*argc + 16, while RSP still points at argc) into new fixed slot +344 and zeroes the cap slot +352 every exec — a saved image cannot carry the build machine's setting. user-defvar/argv base moved 344 -> 360 (all references were symbolic).
gc-max-heap-cap (gc.lisp) lazily parses FUNCL_MAX_HEAP on first collection — allocation-free (fixnum byte compares via %mem-ref-u8, runs in the collector's no-alloc zone). Accepts decimal bytes with optional K/M/G binary suffix, upper or lower case; absent/0/garbage/over-24GiB = unlimited. Overflow-clamped per suffix.
gc-install-window clamps PRISTINE windows (lo >= hw: full-sweep virgin tail and the minor fallback's past-old base) to the cap vaddr so the slow path re-enters the collector AT the ceiling. Below-hw reclaimed-run windows are exempt: already-touched pages (no RSS growth) and shrinking one would leave bare zeroed map bytes — the cave#238b corruption class.
gc-collect post-cycle check errors when hw > cap OR the installed window could not be clamped under it (request cannot fit below the ceiling): "FUNCL_MAX_HEAP heap ceiling exceeded - Storage exhausted", static string per the no-format rule.

Verified: 32M cap kills a 48MB live-list load with the message and exit 1 (RSS bounded); unset/1G caps run the same load to completion; parse probe table exact for 32K/512m/1073741824/banana/huge/0; --hosted exit-42 gate passes; make test-fast-sharded ALL 8 SHARDS PASSED.

Known bounds:

The asm boot window is still the 2 GiB cave#238 nursery, so enforcement begins at the FIRST collection — worst-case overshoot ~2 GiB + side-map overhead before the cap binds. Sufficient for the machine-OOM protection goal; an asm-side boot clamp would need env parsing before Lisp is up.
The error is process-fatal (raise from collect-stub context escapes handler-case) — matches gc-vp invariant-error semantics.
--hosted binaries segfault on ANY collection (pre-existing, filed as #255), so the ceiling cannot protect them until that is fixed.

Page release (madvise on reclaimed regions) remains open — this issue's item 1.

atgreen3992535730

v2 LANDED (63fee53) — the v1 hw-based ceiling was WRONG and would have false-killed healthy loads.

Measured on the asdf.lisp load: the heap high-water mark legitimately spans 20+ GiB of sparse MAP_NORESERVE address space (thousands of mostly-untouched compile buffers) while actual RSS peaks at 1.5-2 GiB. hw over-counts memory >10x, so a hw ceiling is unusable.

v2 bounds what actually kills machines: PEAK PROCESS RSS, read via a new %sys-maxrss primitive (getrusage(2) RUSAGE_SELF into fixed scratch at heap+1MiB, allocation-free) and checked after every collection. The v1 window clamp is reverted — allocation behavior is byte-identical to before the feature.

Semantics: FUNCL_MAX_HEAP = peak-RSS byte budget (K/M/G suffixes, 1 TiB clamp, absent/0/garbage = unlimited). Enforcement points are collections, i.e. every <= 2 GiB of cumulative allocation — budgets under ~2.5 GiB trip at the first collect. ru_maxrss is a lifetime peak: once tripped, every later collect re-trips (kill switch, not back-pressure).

Verified: asdf.lisp load completes under 8G (peak 2.0 GiB; v1 killed it at 554 MB); consing runaway under 1G dies at 2.3 GiB peak, in-process, exit 1. Suite 8/8 green.

atgreen3992785502

PRIMARY RISK FIXED: FUNCL_MAX_HEAP peak-RSS ceiling landed (63fee53, v2 getrusage-based; runtime-layout.lisp +352). Runaway loads now die at the ceiling instead of OOMing the machine. Remaining nicety — returning pages to the OS (madvise) — is a perf item, not a safety one; reopen or file fresh if it matters later. Note cave#255 (--hosted a.outs lack the ceiling) remains open separately.