#260 alexandria-scale asdf:load-system fails heap-state-dependently mid-plan (not-a-function / SIGSEGV at ~10th action) — GC under nested load suspected

closed
Opened by atgreen

asdf:load-system now works (c639698: source-load compile model; single-file foo and multi-file :serial bar systems load and run). Scaling to the vendored Alexandria (17 :serial files, compat/alexandria/alexandria.asd) fails NON-DETERMINISTICALLY:

Same files load clean via plain (load "compat/alexandria/load-all.lisp") — the content is fine. The delta under asdf: a much deeper call context (operate -> perform-plan -> perform -> load* -> nested loader) and a 2GB+ heap with collections firing mid-nested-load. Heap-state-dependent failure point + error-type variance says memory corruption, not a logic bug — cave#238's "moving young collector corrupts heap when it runs" family, or conservative-scan gaps for the loader's deep frames (the load's compile machinery holding raw emission state across a collection?).

Repro: cat scratchpad/alx3.lisp-style input | build/funcl — load asdf.lisp, load compat/alexandria/alexandria.asd (ABSOLUTE path — relative .asd load has its own undefined-function bug, unfiled), (asdf:load-system "alexandria").

Also noted: (type-of ) prints T (type-of gap for mc instances in some contexts), and the relative-path .asd load failure deserves its own look.

Comments (5)

atgreen3992563260

ASDF FULLY EXONERATED — this is a pure core GC bug with a minimal deterministic repro and a named corruption signature (2026-07-09 forensics):

REPRO (no asdf involved): (dotimes (i 140000000) (cons i i)) (gc) (dotimes (i 140000000) (cons i i)) (gc) (load "compat/alexandria/load-all.lisp") => "; Error: the value is not a byte-vector | in: DEFUN" — and EVERY subsequent defun fails identically (compiler state permanently damaged).

Discriminators:

  • K1: after just (load "asdf.lisp"), a PLAIN load of alexandria corrupts the same way — no plan machinery needed.
  • K2: a small 2-file system under (gc-stress 262144) — collections every 256KB — loads FINE. Mid-load collection alone is not fatal; SCALE (multi-GB churned heap) is required.
  • K3: pure cons churn + (gc), then plain alexandria load → corrupt. Fully asdf-free.

AUTOPSY (post-corruption validators, K4):

  • gc-validate-pools: STATIC pools OK.
  • ag-string-pool-offsets: HEAD entry's car is not a byte-vector (most-recently-pushed entry's name bv reclaimed/overwritten).
  • ag-macros: same — head entry's car corrupted.
  • (ag-fn-lookup "GC-COLLECT") => -1 — the runtime fn-table walk broken even for a baked cold function (possibly via the cave#188 fence-index buckets reading the same trashed region).

Signature = live conses whose CAR byte-vectors were reclaimed and later overwritten: the cave#238b class (a stale/garbage extent makes the sweep jump over live marked objects and merge them into free runs; the space is then recycled as an allocation window and clobbered). Non-moving full mark-sweep only — minors are disabled — so this is the sweep/map path, not the moving collector.

Next forensic step queued: (gc-stress 8M) arms gc-verify-heap before EVERY collection — should name the invariant and the guilty cycle (run in flight as K5).

atgreen3992563360

K5 result (discriminator, not a dead end): with (gc-stress 8M) arming gc-verify-heap + gc-verify-cards before EVERY collection, the corruption still occurs and the VERIFIER NEVER RAISES. The heap walk (start map, extents, descriptors) stays self-consistent while live data is reclaimed — so this is NOT the cave#238b sweep-extent/stale-seal class. The sweep is honestly reclaiming objects the MARK PHASE NEVER MARKED: a missed root.

Sharpened hypothesis: entries pushed onto compiler-state alists DURING a load (ag-string-pool-offsets, ag-macros — the corrupted heads are the most-recently-pushed entries; older tails survive) are reachable only through a root the mark misses at scale — e.g. a cold defvar slot outside the static-root table's scanned range, or a stale slot value read at mark time. Small loads survive (K2: 2-file system under 256KB stress) because their pushes happen before/after collections differently or fit windows below the reuse boundary.

Fix-campaign starter kit:

  • Minimal repro: churn 2x2GB + (gc) + plain alexandria load (~10 min, deterministic).
  • Post-corruption validators: k4-walk-bvs over the two alists + (ag-fn-lookup "GC-COLLECT") => -1.
  • Technique per the cave#238b root-fix history: map-byte/address WATCHPOINTS — take the address of a soon-to-die bv (%fn-ptr-as-fixnum right after its push), gdb watch that word across the guilty collection, and see whether it is (a) never marked (mark-phase miss => root audit) or (b) marked then clobbered by a window (window-placement bug).
  • Audit list for the missed root: the static-root table's coverage of COLD defvar slots (which ranges exactly?), gc-register-pool-emissions timing vs alist pushes, and the +14MiB bundle defvar arena boundary.
atgreen3992568464

cave#260's GC-corruption theory is DISPROVEN and the real root is cave#261 (hosted expander early-binding; the types.lisp failure chain is fully documented there). Keeping this issue open only for the one unexplained residue: the asdf-context runs died at DIFFERENT files (functions/hash-tables) with one SIGSEGV, while plain-load deaths were deterministic at types.lisp — that variance is unaccounted for and may still hide a secondary heap/nesting sensitivity. Re-examine after cave#261 lands.

atgreen3992591546

Update from the asdf/alexandria north-star campaign (2026-07-09): the alexandria-load 'not a byte-vector in DEFUN' symptom that looked like this GC bug is NOT GC. Loading F1-7 + types.lisp with an 8GiB no-GC window (no collection fires) gives IDENTICAL 36 errors to the default-GC run. Root = deterministic types.lisp frob-macrolet expander bailing to the tree-walker (format-symbol → non-bv defun names). Tracked in cave#265. This GC issue may still be real for OTHER symptoms, but the alexandria-scale failure is NOT it.

atgreen3992785526

FIXED — the full plan (17+ file actions) executes repeatedly with GC active, no heap-state-dependent mid-plan failure. The 'not-a-function at ~10th action' was the closure-cons designator class (cave#281), not GC. Evidence 2026-07-11: (push asdf:central-registry) + (asdf:load-system "alexandria") completes a REAL multi-file component load on build/funcl; (alexandria:flatten '(1 (2) 3)) => (1 2 3); 8/8 breadth probes incl. closure-returning curry/compose. Fix chain: ADR-0028 closure records e17d39b + eql specializers 822a554 + macro-aware eval 0f0eed8 + REPL robustness 21f15f2.