#238 Moving young collector corrupts heap when it runs; bounding the nursery unlocks ~22x asdf-load speedup (currently minor GC never fires)

Localized: it's a free-block bookkeeping bug in the window-CAPPING path (not stack descriptors, not the moving collector)

Two facts corrected the earlier hypothesis:

1. The moving young collector never runs anyway. (defvar *gc-minor-enabled* 0) — minor is disabled by default, and even enabled it's gated by gc-audit-stack (requires a provably-precise end-to-end stack walk), which fails during compilation. So minor=0 and every collection is a full non-moving mark-sweep. The nursery-cap corruption was therefore NOT the moving collector.

2. gc-verify-heap pinpoints the actual fault: free-block length /= extent (gc.lisp:447). A sealed free block's stored length (word at its base) disagrees with its actual extent (distance to the next published start), i.e. a stale start-map byte sits inside a free block. Downstream, the mutator allocates into / walks that bad block → the "not a byte-vector" crash.

Fast repro (no rebuild, seconds — errors ~254 forms into the asdf load)

(gc-stress 268435456)   ; 256 MiB window cap — also turns on gc-verify after each collection
(gc)                    ; install the capped window
(load "asdf.lisp")      ; => gc-verify-heap: free-block length /= extent

So the EXISTING gc-stress window-capping already trips this on a heavy compile workload — it's a latent bug in the window-cap path, exposed because gc-stress has apparently never been run against a big compile.

Where it is

gc-install-window (gc.lisp ~1809), when it clamps a window to a cap, seals the remainder [lo+cap, hi] via gc-close-run (line 1836 → 1759): sets the map byte at lo+cap to 3 and stores (- hi (lo+cap)) as the length, but does NOT clear interior start-map bytes. On reasoning, the reclaimed run it caps should have a clean interior (gc-sweep-walk clears dead-object start bytes to 0 as it merges runs), so the straightforward hole-cap path looks consistent — which means the bug is a subtler edge case (an off-by-one, an alignment, the virgin-tail-vs-hole branch, or the need>cap widening). Needs the exact bad base/stored-length/extent to nail.

Next step to fix

Capture the exact bad block: add 3 debug heap slots (past the argv slots) or reuse the heal-log; on the first free-block length /= extent, stash base/stored-len/extent and set (gc-stress 0) so the load finishes uncapped/correct, then read the slots. Then it's a targeted fix in gc-close-run / the cap-seal / sweep. Once fixed, the nursery cap (JVM Eden) should give the ~22x speedup (12:24 -> ~0:34) safely.

Prize

Baseline: 579 full collections, ~50% of wall in the collector, 12:24, 566 MB RSS. Target (nursery cap once the free-block bug is fixed): ~0:34 and ~102 MB — ~22x faster loads (and every asdf probe ~20x faster).

Captured the exact bad free block (via a temp diagnostic: on first mismatch, stash base/stored/extent into argv heap slots 344/352/360, disable stress so the load finishes uncapped, read the slots)

bad-base      = 34775637704   (heap_start + 396.633 MiB)
stored-length = 5,600,496     (5.341 MiB)   <- what gc-close-run wrote in the block
extent        = 8,387,912     (7.999 MiB)   <- walker's distance to the next OBJECT start
diff          = -2,787,416    (block UNDER-reports by 2.66 MiB)
base mod 8 = 0 (granule-aligned)

So: it is a SINGLE free region of ~8 MiB (all interior/kind-0 bytes after the block start), but the free-block header at base records only 5.34 MiB. There is no stray start byte at base+5.34 MiB (extent skips kind-0 to the next real start at base+8 MiB). i.e. the sealed block's length is short of the actual contiguous free extent by 2.66 MiB — an un-coalesced / stale length.

Analysis (why this is subtle)

Only two callers write free blocks:

• gc-sweep-walk run-close: (gc-close-run run-start g) where g is a SURVIVOR start → length is exact.
• gc-install-window cap-seal: (gc-close-run (+ lo cap) hi) where hi = best_start + best_len from gc-sweep-finish, and best_len is a CLOSED run (ends at a survivor) → also exact.

On paper both produce a length equal to the extent, yet a block ends up 2.66 MiB short. So the length went stale AFTER sealing (adjacent object freed by a later collection without the block's length being re-extended), OR gc-sweep-finish handed a window hi that wasn't the run's true end in some branch (virgin-vs-hole selection, the need>cap widening, or the trailing-run/hw path at 1781-1784). The sweep advances over a free block by TRUSTING its stored length (gc-sweep-extent-end line 1770 = base + stored_len), so a short stored length makes the sweep skip the merge of the adjacent freed space — the length never gets corrected.

Reproduce + capture (fast, seconds to the first mismatch)

(gc-stress 268435456)(gc)(load "asdf.lisp")   ; gc-verify-heap: free-block length /= extent

To capture: temporarily change gc-verify-object's kind-3 mismatch branch to stash base / (%mem-ref-u64 base) / extent into slots 344/352/360 and (gc-stress 0), then read the slots after the (uncapped) load completes.

Fix direction

Make the free-block length authoritative-by-scan rather than trusted: either (a) gc-sweep-extent-end for a kind-3 block should scan to the next real start (gc-next-start) instead of trusting the stored length, so a stale short length can't make the sweep skip the merge; or (b) on cap-seal, extend hi to the run's true end (gc-next-start) before storing the length. (a) is likely the robust fix and localizes to one function. Then re-enable the nursery cap for the ~22x win. Self-host-critical — gate with full test-fast-sharded + the gc-stress verify repro loading clean.

ROOT CAUSE FOUND (via gdb) — a GC-rooting gap for symbol-pool name byte-vectors

The nursery cap and the free-block-seal were both red herrings for the real crash. gdb on the reproducing build (nursery cap on) caught the "not a byte-vector" and gave a walkable stack + the bad datum:

#0 AG-SYMBOL-POOL-FIND-OFFSET     ← loads a "name" byte-vector = 0x8feb3cf90 (NOT a bv)
   AG-RUNTIME-POOLED-SYMBOL-PTR
   AG-CANON-HOSTED-SYM            ← canonicalising a symbol during macro-expansion
   AG-MACRO-APPLY-HOSTED … AG-REPL-EVAL-EACH … AG-LOAD-SOURCE-ACTION

Inspecting the datum:

rsi = 0x8feb3cf90   (lowtag 0 = untagged)
high-water = 0x87f51b748   →  datum is ABOVE hw   (heap ~2.1 GiB in; hw ~2.05 GiB)
mem[0x8feb3cf90] = 0a 00 00 00 00 00 00 00 | 41 40 b3 fe 08 00 00 00 | 07 ...
                   (word0=10, word1=0x8feb34041 a cons ptr, word2=7 — a REUSED non-bv object)

So a live name byte-vector was reclaimed: a nursery-boundary full collection swept its region and lowered the high-water mark below it (the trailing-run rejoin, gc-sweep-walk ~1781), then the mutator re-allocated that space for another object. ag-symbol-pool-find-offset then follows the stale reference and finds a non-bv → "not a byte-vector".

Why marking didn't keep it alive

ag-runtime-pooled-symbol-ptr (reader.lisp:769) resolves pooled symbols into the arena (*ag-pool-emit-base-vaddr* + off + 3), but it searches on a name byte-vector via ag-symbol-pool-find-offset(name, *ag-symbol-pool-offsets*) — and those name-bvs live in the collectable heap. That name-bv was not marked as reachable, so the collection freed it. The reference is effectively an untagged/raw path that neither precise nor conservative marking traces (both only recognise lowtag 1/3), which is also why forcing conservative full-marking did NOT fix it.

The baseline (no nursery cap) survives only because its single collection fires at ~24 GiB, after everything has settled; frequent collections (nursery cap) hit the transient window during hosted macro-expansion.

Fix direction (fresh, careful GC-rooting work)

Ensure the symbol-pool name byte-vectors are GC roots — check whether *ag-symbol-pool-offsets* (and the transient name in ag-canon-hosted-sym) is in the mark root set (gc-mark-roots marks repl-defvar-slots + registry + pool ranges + static roots; a compiler global like *ag-symbol-pool-offsets* may be missing). If its name-bvs are rooted, they won't be reclaimed and the nursery cap becomes safe → the ~22x load speedup unlocks. Verify with the fast repro: (gc-stress 268435456)(gc)(load "asdf.lisp") must load clean and gc-verify-heap must not fire.

Fixes attempted that did NOT work (recorded so they aren't re-tried)

1. Removing the window-cap free-block seal (holes used whole) — free-block verify error persisted (it's the sweep's own gc-close-run going stale, and it's a benign/secondary symptom).
2. Forcing the non-moving full mark through the conservative superset — didn't help, because the missed reference is an untagged raw pointer, not a tagged stack root.

Both reverted; tree is back to the correct uncapped baseline.

CORRECTION — my "untagged datum 0x8feb3cf90" was unreliable

emit-bv-type-check (runtime.lisp:2157) is a LOWTAG-only guard: mov rax,rbx; and rax,7; cmp rax, other-ptr(3); jz ok; else gc-emit-raise-static-bv. gc-emit-raise-static-bv then overwrites RBX with the message bv, so at the trampoline the failing operand is already gone. The RSI value I read (0x8feb3cf90) was NOT the operand — so the "untagged reference" and "reclaimed-above-hw" conclusions drawn from it are not trustworthy. What IS solid:

• The corruption reproduces in seconds with the nursery cap (or gc-stress + verify).
• Walkable bt: AG-SYMBOL-POOL-FIND-OFFSET <- AG-RUNTIME-POOLED-SYMBOL-PTR <- AG-CANON-HOSTED-SYM <- AG-MACRO-APPLY-HOSTED <- … <- AG-LOAD-SOURCE-ACTION.
• ag-symbol-pool-find-offset calls ag-bv-equal(entry-name, name) and one operand fails the lowtag==3 check → a non-byte-vector reaches ag-bv-equal during hosted macro-expansion, right after a collection.

Fixes attempted and FAILED (all reverted — do not re-try blindly)

1. Cap only virgin windows / use holes whole (remove the gc-install-window free-block seal).
2. Force the non-moving full mark through the conservative superset (gc-mark-stack visit-mode 0 → gc-mark-stack-words). Confirmed *gc-visit-mode*=0 during full mark, so it DID run — still corrupted.
3. gc-mark-pool-range DOES follow symbol value cells (+8..+32), so *ag-symbol-pool-offsets*'s alist and its entry name-bvs ARE rooted — ruling out "global not a root".
4. (Not shipped) lowtag-0 conservative marking — dropped once I realized the datum it was based on was unreliable.

RELIABLE next step

Conditional breakpoint at the ACCESS site (the operand is live there, unlike at the trampoline): rbreak ^AG-SYMBOL-POOL-FIND-OFFSET$ (or ^AG-BV-EQUAL$), Python stop() that reads the two args and fires only when one has & 7 != 3, then dump that operand + its memory + which arg (entry-name vs input name). That distinguishes: (a) a fixnum/cons wrongly in the offsets alist (compiler bug), (b) a genuinely reclaimed bv (GC-rooting), or (c) a mis-tagged pointer. Only then is the fix determinate.

Tree is at the correct committed baseline (asdf:operate runs code = the goal; cave#237/#237b landed). The ~22x nursery-cap speedup remains blocked on this one bug.

FIXED (da8ac1a) — root cause was a stale virgin-window-remainder seal; nursery cap now safe. Remaining: the minor collector itself.

The bug (one line): gc-close-current-window sealed the unused remainder of any window with r13 != heap_end. A nursery/stress-capped VIRGIN window therefore got its remainder sealed — a stale kind-3 byte + length ABOVE the high-water mark. The next virgin window (installed at the post-sweep hw, below that seal) handed the space back to the mutator, which rewrites memory but not interior map bytes (virgin invariant), so the stale seal byte survived INSIDE a live object. A later sweep's gc-next-start took it for an object start, read a garbage "length", walked into live data, and merged LIVE objects into free runs — gdb showed 34 symbol-pool alist pairs sealed as free-block heads (mapb=3, car = small block lengths) and hw lowered 2 GiB below the live alist head. Fix: the virgin test under a cap is r13 > hw (hole windows always end ≤ hw) — above-hw remainder rejoins virgin, no seal, same as the old heap-end case.

Also landed: +gc-nursery-bytes+ = 2 GiB; gc-install-window caps virgin-tail windows (holes installed whole, never clamped), boot window = one nursery.

Validation: full asdf.lisp load completes to ASDF/FOOTER with 0 corruption under the cap; ~15 min of gc-stress(256MiB) with per-collection gc-verify-heap/gc-verify-cards found 0 inconsistencies; --hosted 42; test-fast-sharded ALL 8 SHARDS PASSED (nursery active during the self-host build).

Honest perf: 12:24 → 10:28 (~15%). The earlier "0:34 = 22x" was the corrupted run dying early — retracted. All 735 collections are still FULL mark-sweeps; the real speedup requires the minor (moving) collector.

Remaining item (the real prize): (gc-minor-enable 1) + the asdf load = minors engage and die: Storage exhausted ~3/4 through, then SIGSEGV (status 139). Deterministic repro:

printf '(gc-minor-enable 1)\n(load "src/mini-clos.lisp")\n(load "asdf.lisp")\n' | build/funcl-stage2

Debug fresh with gdb (rbreak ^GC-COLLECT-MINOR$, watch evacuation/reclaim accounting — storage exhausted suggests young reclaim isn't returning space, or evac-top runaway). Minor stays disabled by default until then.

Groundwork landed (9d8d012); the remaining minor bug is heal-mode symbol corruption

Two space-accounting gaps fixed (verified inert in the default config; gates green):

1. Empty OLD-region demotion — OLD was a one-way ratchet (promoted-then-dead regions permanently unusable). Now, between mark and sweep of a full collection, OLD regions containing no marked entity demote to YOUNG. Live-old regions stay OLD (old->old refs have no card marks — never expose them to evacuation).
2. First-fit window search (gc-window-fit-from) — the minor's post-collection window install clamped to the first gap above hw; with OLD regions sprinkled low, a large request that missed the first thin gap OOM'd with ~22 GiB virgin above.

The remaining bug (minors stay disabled)

With minors force-enabled, the asdf load still dies at the same form cluster (%refresh-component-inline-methods area, ~3395 output lines, ~5:30): a symbol NAME prints with binary garbage (%COMPUTE-+?4F...-VALUE) immediately before Storage exhausted + SIGSEGV. So minor heal-mode misses a reference in the runtime symbol machinery: a symbol-name byte-vector (or a length within it) is clobbered/moved without healing, and the OOM is downstream fallout from a corrupted size.

Suspects for the next campaign:

• Runtime-interned symbol records (the loader interns thousands during asdf) whose slots live OUTSIDE the baked pool range — are their ranges all registered in the root-ranges registry, and does heal mode visit them?
• The below-pool-range records observed earlier (record 0x6e9f0b < range start 0x6f7698).
• Transients held in registers/frames during the hosted expander at that form cluster (heal requires the precise walk — audit-gated — but a passed audit with an incomplete frame map would heal-miss silently).

Fast repro: printf '(gc-minor-enable 1)\n(load "src/mini-clos.lisp")\n(load "asdf.lisp")\n' | build/funcl-stage2 → garbage symbol name then OOM+SEGV ~5:30 in. Also note: gc-verify's walk can false-positive on wholesale-reclaimed young space at heap_start ("walk reached a non-start granule") — the verifier assumes an entity start at heap_start; benign, but fix it before using verify in minor debugging.

Defer fix landed (af2f31e) — emission-window corruption class ELIMINATED; one more raw-reference class remains

Root cause of the ~3395-line symbol corruption: ag-emit-quote-pool embeds RAW absolute pointers to heap name byte-vectors inside the code buffer (opaque bytes — unhealable) until the range is registered. A minor moving a name bv in that window leaves the embedded pointer stale. The full collector was safe there (bvs alive+unmoved via *ag-symbol-pool-offsets*), which is why only minors died.

Fix (GC-unsafe-region pattern): *gc-minor-defer* set at ag-emit-quote-pool entry (raw value-cell write), cleared by gc-register-pool-emissions / gc-patch-pool-slots; the dispatcher degenerates to the non-moving full while set. Unregistered-path failure mode = minors stay deferred (safe, slower).

Result: minors-enabled asdf load runs FAR past the old death point — the garbage-symbol-name class is gone. A different class remains: later in the load a cascade of The value N is not a LIST (car/cdr) errors begins (thousands; load limps to ~4253 lines, ends at ASDF/COMPONENT) — i.e. at least one more site holds raw untraced pointers to young heap objects that a moving collection invalidates (conses this time, not name bvs).

Candidates for the next site (same hunt recipe: repro + gdb trampoline filter on the first error + stack walk):

• other code-buffer embeddings that reference HEAP (not arena) objects — e.g. baked quote structures, macro-table conses, fn-table entries;
• %fixnum-bits-as-fn-ptr / %fn-ptr-as-fixnum stashes of heap addresses (grep for callers holding results across allocation);
• dynamic-frame / return-stack entries encoding heap pointers as fixnums (cave#148 machinery);
• the register-quoted walk (ag-repl-register-one) reading record fields mid-window.

Perf signal: even limping, the minors run paced the load at 2:31 vs 10:28 shipped — the prize (~4x+) is real once the remaining site(s) are traced or deferred.

Groundwork now landed (all inert while *gc-minor-enabled* = 0): empty-OLD demotion + first-fit window search (9d8d012), emission defer (af2f31e), seal fix + 2 GiB nursery (da8ac1a).

Campaign checkpoint (7bc9e32): conservative minors implemented; ONE residual dangle class before default-on

Landed (all default-inert/hardening; gates green): window map hygiene (u8 span clear + covering-block truncation + card pre-dirtying per codex review), pending-pool heal (replaces the defer, which starved 465/474 collections), REPL root-range registration (codex finding, empirically confirmed — installed pool records were never registered), SGen-style conservative minors (audit-fail → pin young regions from a conservative stack scan; pin-fixpoint heals; reclaim promotes), dispatcher gate counters + pool validators.

Independent review: OpenAI codex reviewed the full design (brief + repo read). Verdict: conservative minors sound under the "only tagged starts are references" invariant; flagged (1) old-window unbarriered init stores (fixed: card pre-dirty), (2) missing REPL range registration (fixed), (3) the derived/raw-address invariant as the open risk — and recommends the endgame full be EVACUATING (reuse minor heal/forwarding; pin ambiguous regions) rather than sliding mark-compact.

Residual (blocks default-on): minors force-enabled on the asdf load now die later with ~431 x "the value is not a byte-vector" starting at DEFUN SAFE-READ-FROM-STRING (~line 617 of output) — a raw byte-vector reference some path holds unhealed across an evacuating minor (codex risk 3 class). All previous classes (garbage names, car/cdr cascades, OOM, SEGV) are gone. Deterministic.

Next probe (established recipe): hardware watchpoint via the raise-condition trampoline filtered on 'byte-vector', dump the operand's address, then watch its map byte / birth writes to identify the holder that missed healing. Candidate holders: JIT fn-table entries, macro-table (ag-macros) name bvs, dwarf/line-pair buffers, ag-string-pool pending list interactions, %fn-ptr-as-fixnum stashes held across allocation.

Measurement criteria for default-on (per codex): repeated asdf loads with zero stale roots; bounded pinned bytes/minor; full-GC wall share well below ~50%; counters: minor/degen/gate-*, pinned regions per minor, evacuated bytes, old-window allocation bytes, RSS/hw trend.

GOAL RESULT (2380b9c, local — push pending server access): asdf load 10:28 -> 4:12, ONE collection — the collector no longer dominates

The twist ending: with every GC bug fixed, the remaining minors-enabled cascade traced into AG-DWARF-ADD-STR-OFFSETS / AG-JIT-REGISTER-BATCH. Skipping JIT debug-info batches once a minor has run (they walk post-evacuation state + their swallow-unwind poisoned per-form state) gave:

• load completes CLEAN (only the pre-existing benign WILD-PATH error)
• wall 4:12 vs 10:28 fulls-only (2.5x) — and ONE collection total (minor=1, degen=0), RSS 683 MB

Reframe: the collector was never the dominant cost — the JIT debug-metadata builder generated the overwhelming majority of allocation churn (the "~24 GiB cumulative" was mostly its per-batch buffers). With that churn gone, the 14k-line load barely needs collecting.

Config semantics: default (minors off) unchanged — dwarf batches + gdb symbols intact; gates green (hosted 42, sharded 8/8). (gc-minor-enable 1) = fast mode, trading gdb symbols for REPL-emitted code.

Follow-ups:

1. Fix the DWARF batch builder: (a) stop holding heap references across collections (make it re-read via defvars / copy names eagerly), (b) slash its allocation (same class as the old 8MiB info-buf fix) — this gives the DEFAULT config the same win with metadata intact. Highest-value next item.
2. Burn-in conservative minors (repeated loads, pinned-bytes bounds, validator-clean) then consider default-on.
3. Codex-recommended endgame: evacuating fulls reusing the minor heal machinery.

GOAL MET IN DEFAULT CONFIG (942507c, local — push pending cave-host recovery)

The proper DWARF fix: ag-dwarf-build-line-buf allocated an 8 MiB scratch per top-level form on the JIT batch path (+256 KiB eh-frame) — ~2,900 batches = the entire ~24 GiB churn that made GC look dominant. Fixed by size-classing (64 KiB literal buffer for normal batches, 8 MiB fallback for giants; literal caps per the elisp-host constraint).

Default config (minors off, gdb metadata fully intact): 10:28 → 4:14, collect-count = 0 for the whole 14k-line load. The collector cannot dominate compilation time when the load never triggers it. Gates green (hosted 42, sharded 8/8).

Scoreboard: old default 10:28/735 fulls · new default 4:14/ZERO · minors+skip opt-in 4:12/one.

NOTE: commits 2380b9c + 942507c are local-only — the cave git host is fork-exhausted ('Resource temporarily unavailable' post-auth); needs a kick. Remaining GC roadmap (burn-in minors; evacuating fulls per codex; DWARF builder GC-safety for minors-with-metadata) stays tracked here.

ASSESSMENT 2026-07-11: still disabled (minor GC never fires); full asdf+alexandria loads now run to completion on the non-moving path, so correctness is unaffected — this is the single biggest PERF item (bounding the nursery measured ~22x on asdf loads). Blocked conceptually on the evacuation-correctness work (ADR-0001 fwd-header machinery is in place; the corruption repro needs a bisect under gc-stress with the young path forced on). Note two GC dispatch gaps got fixed today that would have bitten evacuation testing: ratio + closure records missing from trace/heal/stale/verify (cave#282, e17d39b).