#214 Heap capped at 4 GB by disp32-encoded side-map biases in the write-barrier fast paths

open
Opened by atgreen

The collectable heap is pinned to #x50000000 + 2.75 GiB (ends exactly at the 4 GiB boundary). It cannot grow past that by bumping the constants: the allocation fast path's 3-instruction publish sequence (object-start map) and the post-write card barrier both compute their side-map byte address with a disp32 displacement — +gc-start-map-bias+ and +gc-card-bias+ must fit in a signed 32-bit immediate, and the whole address layout in src/cold/elf-tiny.lisp (start map #x20000000, mark stack #x38000000, region dir #x3C000000, card table #x3D000000, heap #x50000000) is arranged around that ±2 GiB reach.

Lifting the cap means reworking the barrier emission (gc-emit-publish-start and the card-barrier shape in src/cold/gc.lisp): movabs the bias into a scratch register (or keep it in a pinned callee-saved register) — one extra instruction on every allocation site, plus re-planning the virtual layout so heap, start map (heap/8), and card table (heap/512) scale together above 4 GiB. All the side regions are NORESERVE virtual reservations, so a 32-64 GiB heap costs no resident memory.

Motivation (2026-07-03, cave#205 campaign): asdf-scale loads allocate multiple GiB of short-lived compile buffers; when cumulative allocation exceeds the 2.75 GiB virgin tail the collector starts full mark-sweep cycles per reclaimed window and load times balloon. Churn reduction (the 8 MiB-per-form DWARF buffer fix) mostly hides this today, but a big heap is the durable fix the owner has asked for — 'we have lots of memory; give it a huge heap.'

Related: the GC pacing policy in gc-sweep-finish (virgin-tail preference) and the abandoned hole free-list experiment (corrupted at scale — see the cave#205 thread for the census 20/23 postmortems).

Comments (2)

atgreen3992079017

PRAGMATIC LIFT LANDED (bdc2e02). Heap moved to 24 GiB at 0x800000000; disp32 side-map biases preserved by construction (start map at heap>>3 => bias 0; card table at heap>>9 + 128MiB => bias 0x8000000). NORESERVE so no resident cost. Census 19min -> 4min, ~zero collections. Full movabs-barrier rework for >27 GiB still open here, but this unblocks the census decisively.

atgreen3992785829

ASSESSMENT 2026-07-11: unchanged — heap capped at 4 GB by disp32-encoded side-map biases in the write-barrier fast paths. With FUNCL_MAX_HEAP defaulting workloads well below 4 GB this is not currently limiting anything real (today's full asdf loads peak ~1.5-4 GB RSS). Becomes relevant for genuinely large heaps; the fix (imm64 biases or a register-based base) costs a few bytes per barrier. Defer until a workload actually needs >4 GB.