#314 save-image from an image-restored session loses/corrupts package side tables — blocks the CDS-style asdf image (1.17s boot vs 10s fasl replay)

Progress in cbfbec0 — three of the four layers fixed and verified inside a freshly baked image:

1. FIXED: use-list intern-table bleed. Root cause of the "4518-entry use list": %pkg-listify treated a package CONS as a LIST of designators, so uiop's (use-package p) walked the package's own intern alist and pkg-add-use recorded every (name . sym) entry as a used package. ASDF/INTERFACE now has 23 uses. Bonus: every find-symbol uses-walk was iterating that junk — the fasl replay boot dropped another 26G instructions (160.5G now vs 244G at the 7766bcb baseline).
2. FIXED: package-nicknames persists (container index 8). (find-package "ASDF") resolves in the baked image; minimal bake round-trips nicknames.
3. FIXED: reader qualified door minted twins for names read through a REEXPORTING package. It now routes through the ADR-0031 accessibility hook (find-symbol first) like the loader. In the baked image: (eq 'asdf:asdf-version 'asdf/upgrade:asdf-version) = T, fboundp = T.
4. STILL OPEN: calls into restored asdf code return garbage. (asdf:asdf-version) errors "the value is not a byte-vector" INSIDE the call (even under stringp, so it is not a printer issue) despite fboundp T and the value cell reading a proper string via symbol-value. (asdf:load-system ...) dies with mini-clos "no accessor". So at least one restore gap below the symbol layer: suspects are fn-table/code-offset skew in second-generation bakes (the recipe's (gc) before save-image?), the fn-cell install pass wiring the wrong record's cell, or mc accessor-table restoration. Note find-symbol still returns a record non-eq to the reader quotes post-restore (:FS-EQ NIL) — the own-alist vs pkgsym-map reconciliation (ag-pkg-own-publish) may need the restore-side treatment too.

Current numbers: baked-image boot 2.3s / 100MB (once layer 4 lands) vs fasl-replay ~9s / 160.5G instructions. build-asdf now bakes FROM the auto-loaded fasl (also fixes its double-load after the boot auto-load landed).

Layer-4 lead: the dyn-sym arena (top 1 MiB of the fixed 32 MiB RWX mmap, non-moving records, compiled code embeds their FIXED vaddrs) is saved/restored as the cave#209 top-slice — but a GEN-2 bake mints asdf's records ON TOP of a restored slice (ag-image-restore-dyn-sym-cursor repoints the bump past it), and the observed symptoms say the second save does not carry the stacked slice correctly: compiled asdf code reading through arena addresses gets garbage ("not a byte-vector" inside (asdf:asdf-version)) while every heap-copied door works (symbol-value of the same special reads a proper string; fboundp via own-alist records = T). Next session: read ag-image-restore-dyn-sym-cursor + the top-slice capture in ag-save-image-action and check whether the gen-2 save's slice extent covers restored-base..new-cursor or only the session's own mints. The mc 'no accessor' on load-system is likely the same arena story (accessor closures referencing arena records), so one fix may clear both.

LAYER 4 ROOT-CAUSED, and the framing was WRONG — it is NOT dyn-sym top-slice stacking. Corrected diagnosis (fresh gen-2 bake with layers 1-3 fixes, probed):

The failing symbol asdf-version lives at a HEAP address (~1.1 billion, far above the 32 MiB arena at 4198400), NOT in the dyn-sym top-slice. So the "gen-2 mints on top of a restored slice / slice extent" theory is refuted — it is a heap symbol record.

The real bug is a SYMBOL DOOR SPLIT surviving the gen-2 bake. Post-restore probe (build/funcl-asdf): (:SAME-RECORD NIL :QUOTE-BOUND NIL :FS-BOUND T) i.e. (quote asdf/upgrade::asdf-version) and (find-symbol "ASDF-VERSION" (find-package "ASDF/UPGRADE")) resolve to DIFFERENT records. The QUOTE's record is UNBOUND; the FIND-SYMBOL/package-table record is BOUND (holds "3.3.7"). asdf:asdf-version's compiled code reads the QUOTE's record (unbound twin) and then, on the failure path, tries to print the symbol whose NAME is a widetag-0x0D object — "the value is not a byte-vector".

Two distinct facts, don't conflate:

1. THE CRASH DRIVER: the image restore rebuilds the reader's symbol-pool-offsets door AND the package-own-alist door, but for a FASL-REPLAYED symbol they converge on DIFFERENT records — the compiled quote embeds the unbound twin while the bound value lives on the package-table record. Same class as cave#304/#307/#309 (ag-pkg-own-publish reconciliation) but at the IMAGE-RESTORE boundary. Pre-save (build/funcl, fasl auto-loaded, NOT baked) the quote resolves to the BOUND record (val-strp T) and asdf:asdf-version WORKS — so the split is INTRODUCED by save+restore, not by the fasl replay itself.
2. BENIGN PRE-EXISTING: asdf symbol NAMES are widetag-0x0D objects (strings), not 0x01 byte-vectors — true in the pre-save session too. The printer's %out-sym-name assumes a byte-vector and cannot print a 0x0D-named symbol. Latent printer gap, only surfaces on the error path; fix separately (make %out-sym-name / byte-vector-p paths accept string-widetag names).

FIX DIRECTION: at image restore, reconcile the reader-door record and the package-table record to ONE record per (pkg,name) for fasl-minted symbols — publish the bound record onto the quote's pool-offset entry, or make ag-image-rebuild-symbol-pool-offsets and ag-pkg-own-publish share the canonical record (the cave#304 door-home pattern, applied post-restore). This is the ADR-0026 reconciliation class; deep + risky (image subsystem) so scoped as a focused effort. Stage C's image-reinstall-shrink depends on it; the pkgsym-arm Stage C increment already shipped separately (145.7G).

Title should be retitled away from "top-slice stacking".

SECOND OPINION (codex, adversarial — asked it to BREAK the door-split theory given our history of wrongly blaming ADR-0026). It sharpened the diagnosis and its decisive base-address probe (which I had failed to run — I only had eq=NIL, never the two bases) settles it:

DECISIVE PROBE (bases + values of the SAME name via quote / find-symbol:ASDF/UPGRADE / find-symbol:ASDF): PRE-SAVE (build/funcl, fasl loaded, NOT baked): eq-qu=T eq-qa=T eq-ua=T ; qb=ub=ab=1112579576 ; value=1081258499 (string, bound) => ONE physical record; all doors agree; asdf:asdf-version WORKS. POST-BAKE (build/funcl-asdf): eq-qu=NIL eq-qa=NIL eq-ua=T ; qb=1085467480 (UNBOUND, value=23) ; ub=ab=34398222960 (BOUND, value=34397912795 the string) => TWO physical records. The find-symbol door (both packages agree) keeps the BOUND record at the high restored-heap address (3.4e10); the QUOTE door resolves to a SEPARATE UNBOUND record at 1.08e9.

CONCLUSIONS (codex refuted the weak alternatives):

• NOT a pre-existing fasl twin (alt c): pre-save all three bases are identical — the fasl replay mints ONE record. save+restore INTRODUCES the second record.
• NOT a lost value slot in restore (alt b): ag-image-encode-heap-slot encodes a MISSING/uncollectable object as NIL(7), and restore only skips value-install when the encoded slot is exactly unbound(23). So value=23 means the quote-record was GENUINELY unbound at save — it is a distinct, never-bound record, not a corrupted copy of the bound one.
• NOT the 0x0D string name (alt d): strings share byte-vector layout; byte-vector-byte/length/ag-bv-equal lowtag-check (not widetag-check), so string-widetag names compare/print fine. The 'not a byte-vector' is an unbound/immediate value flowing into a byte op on the ERROR path, not the name.
• IS a real physical split, precisely: the restored PACKAGE-TABLE door (find-symbol) owns the bound heap record (3.4e10); the READER/QUOTE door mints/keeps a FRESH UNBOUND record (1.08e9). Compiled asdf code embeds the quote-door record.

So the fault is NOT vague 'ADR-0026 dragon' and NOT top-slice stacking. It is: restore reconciles heap symbols only to baked static/keyword records (ag-image-restore-* canonicalizer), NOT to real package-homed records — so a package-homed symbol reached via the reader/quote door does not converge with the same symbol reached via the restored package own-alist. Codex: 'heap-restored symbol clone not reconciled with the package/accessibility canonical record.'

FIX LOCATION (now concrete): the reader's qualified intern post-restore (ag-reader-intern-qualified-nonkw -> %pkg-accessible-intern -> find-symbol-first) is NOT finding the restored package record (it mints fresh at 1.08e9) even though a top-level find-symbol DOES find it (3.4e10). Either ag-image-rebuild-symbol-pool-offsets seeds the quote door from a different record set than the restored package tables, or the reader's find-symbol runs before/against a table that lacks the restored record. Next: probe WHY find-symbol inside the reader intern misses the record that top-level find-symbol hits (same function, different result => a table/timing/canonicalization gap at restore). Retitle away from 'top-slice stacking'.

PROBE CHAIN COMPLETE — corrected diagnosis (and it is NOT the reader/ADR-0026 door split I claimed; the skepticism was right). Every hypothesis falsified in turn, ending at a precise, different root:

HEALTHY (ruled out): the package/reader layer resolves the BOUND record correctly via EVERY door and query type: find-symbol("ASDF-VERSION", ASDF/UPGRADE) = 3.4398e10 BOUND find-symbol-in-pkg (own table) = 3.4398e10 BOUND %pkg-accessible-intern (the reader's hook) = 3.4398e10 BOUND find-symbol with a 0x01 byte-vector query = 3.4398e10 BOUND (bv01-equal-stored=T) ONE package cons (find-package deterministic), symbol in its own table exactly once, hooks NON-nil. Falsified along the way: pre-existing fasl twin (pre-save bases identical), value-lost-in-restore (missing obj encodes NIL=7 not unbound=23), 0x0D-string-name mismatch (0x01 query finds 0x0D name fine), nil reader hooks, package twin, query-widetag mismatch. My REPL (quote asdf/upgrade::asdf-version) resolving to a fresh 1.08e9 record was a PROBE ARTIFACT (lowercase REPL input / reader mint), NOT the failing path.

THE ACTUAL BUG (decisive test): set the package-door record's value to "SENTINEL-9" via (set (find-symbol ...) "SENTINEL-9") — reads back "SENTINEL-9" fine — then (asdf:asdf-version) STILL fails with 'not a byte-vector' and does NOT see the sentinel. So asdf's COMPILED CODE reads a DIFFERENT record than the package door. The package layer is healthy; the compiled asdf code (restored code-mmap) holds a baked asdf-version value-cell reference that, after restore's pool-slot/KIND-2 relocation, points at a SEPARATE UNBOUND record rather than the live (bound) package record.

FIX TARGET (narrowed, and OFF the reader/package doors): the restore-side relocation of the compiled code-mmap's baked symbol/value-cell references (ag-image-relocate-restored-code-mmap-pool-slots / the KIND-2 value-cell path) must reconcile to the SAME record the package tables restore — i.e. the compiled code's value-cell slot should land on the bound package record, not a distinct unbound clone. Next: dump the exact vaddr the compiled asdf-version reads (gdb break on the value-cell load, or disasm the restored asdf-version fn) and compare to the bound record's base 3.4398e10 — that names the mis-relocated slot. This is a code-mmap relocation fix, not a symbol-unification fix.

GDB ROUND — two new facts, but the exact wrong-vaddr defeated static gdb on the restored image (dynamic layout). The decisive proof (SENTINEL test, prior comment) stands without it.

NEW FACTS:

1. The error is 'in: PRIN1', NOT in asdf-version. So (asdf-version) is effectively (defun asdf-version () asdf-version) — it READS the value cell and RETURNS it raw; the returned value is the UNBOUND marker (23), and PRIN1 then chokes ('the value is not a byte-vector'). So the failing read is a plain special-var reference returning unbound, confirming: asdf-version's code reads a record whose value cell = 23.
2. symbol-function(asdf/upgrade:asdf-version) is a RAW FN-PTR (lowtag 010) whose entry is at ~1.08e9 — a HEAP address, FAR above the 32 MiB code-mmap (top ~0x2401000/37.7e6). So the asdf function body lives in a heap-resident executable region, NOT the restored code-mmap. That is why 'compiled asdf code' reads a distinct record: the function that reads asdf-version is itself heap-allocated, and its embedded value-cell reference points at an unbound record disconnected from the restored package record.

GDB LIMITATIONS HIT (documented so next session skips them): (a) break *0x40006f (gc-raise-condition-vaddr) never fired — the raise routes through a HEAP-restored vaddr slot, not the static symbol address, so breakpoints on the cold raise entry miss on a restored image. (b) %fn-ptr-as-fixnum + static disassembly kept mis-classifying the fn (closure record header read as code = 0xC3 ret byte; raw-ptr vs closure lowtag) because the restored heap layout is dynamic.

NEXT STEP (better than static gdb): a Lisp probe that already HAS the addresses sets a hardware watchpoint, OR instrument the value-cell load at emit time for this one symbol. But the fix target is unchanged and confirmed: at image restore, the record referenced by heap-resident asdf function code for a package-homed special var must reconcile to the SAME record the package tables restore (currently a distinct unbound clone). This is symbol-record reconciliation for compiled/heap-code references at restore — NOT the reader/package doors (proven healthy) and NOT ADR-0026 unification.

PINPOINTED (find-symbol instrumentation, reverted). The bug is a SINGLE mis-relocated keyword constant — decisively localized, far from any door/ADR-0026 theory.

Instrumented find-symbol to log every call RETURNING an unbound record, then ran (asdf:asdf-version). Between call-markers, the ONE failing lookup is: FS316U via name=[ASDF-VERSION] len=12 name-widetag=1 pkg=ASDF/INTERFACE res-base=34398244376 The query name is "ASDF-VERSION" (len 12, widetag 1 = byte-vector), NOT "ASDF-VERSION" (len 14, widetag 13 = string). asdf-version's source is (symbol-value (find-symbol (string :asdf-version) :asdf)): so (string :asdf-version) is yielding the byte-vector "ASDF-VERSION" (earmuffs stripped, wrong widetag). find-symbol then hits the ASDF-VERSION function symbol (value cell unbound=23), (or 23 ...) returns 23, PRIN1 chokes -> 'not a byte-vector'. The manual REPL (string :asdf-version) = the string "ASDF-VERSION" (widetag 13) and works, so ONLY asdf-version's BAKED keyword constant is corrupt.

MECHANISM: asdf-version's baked keyword :asdf-version has its NAME SLOT pointing, post-gen-2-restore, at the byte-vector "ASDF-VERSION" (widetag 1) — almost certainly the ASDF-VERSION function symbol's OWN name object (a relocation COLLISION), instead of the string "ASDF-VERSION". Not name-editing; wrong-object relocation. Pre-save it is correct (asdf:asdf-version works on build/funcl), so the gen-2 SAVE+RESTORE mis-relocates this keyword constant's name reference.

STRONGEST HYPOTHESIS for WHY (unconfirmed, gates the fix): asdf-version's compiled code lives at ~1.08e9 — a fasl/heap region ABOVE the 32 MiB base code-mmap (top ~37.7e6), NOT the base code-mmap. The code-mmap pool-slot relocation (ag-image-relocate-restored-code-mmap-pool-slots, keyword records register under +gc-root-range-kind-symbols+ so they ARE in scope IF in the code-mmap) is BOUNDED to the base code-mmap extent — so keyword constants embedded in the higher fasl/heap code region are relocated by a different (or no) pass and collide. NOT YET CONFIRMED that asdf code is in a separate region and how its constants relocate.

FIX (targeted, but risk-gated on confirming the above): ensure keyword/symbol name-slot references embedded in ALL restored executable regions (base code-mmap AND the fasl/heap code region asdf lives in) relocate to the correct heap objects. Next cheap step before writing image code: confirm asdf-version's code region (base code-mmap vs a separate fasl/heap mmap) and dump the raw keyword-constant slot the compiled code reads to see the exact wrong vaddr it relocated to. I stopped short of writing the relocation fix — image-restore surgery on an unconfirmed collision mechanism risks silent heap corruption; the precise localization above makes it a focused next step.

SECOND OPINION (codex, adversarial) — REFUTED my mechanism and found the real bug. Value of asking: I was about to write the WRONG fix (extend relocation to a phantom code region).

(a) MY 'separate out-of-scope code region' hypothesis is FALSE. FASL COs install by APPENDING into the base code-mmap byte-vector (ag-fasl-install-co, repl.lisp:1267); asdf boot passes the same runtime code buffer (repl.lisp:1189); save-image saves that ONE code-mmap (repl.lisp:801), restored at +elf-mmap-vaddr+ (image.lisp:2738). So ~1.08e9 is a HEAP object/record/closure pointer, NOT a CO code entry — I mis-inferred an executable mapping from a pointer value. (ag-fn-table stores mmap OFFSETS, not heap addresses, confirming it.)

(c) THE REAL BUG = a ROOT-COLLECTION / ENCODE asymmetry, not relocation bounds. Save ROOT collection is bounded at the LIVE code cursor (ag-image-roots-with-code-mmap-pool-roots, image.lisp:1458 — root-len is set to 0 at save per repl.lisp:825, so it falls to byte-vector-length = code-hi), EXCLUDING the widened top-slice. But the slot ENCODER and restore RELOCATOR use the widened file extent (image.lisp:1547, 3089). The image.lisp:1609 comment justifies the narrow root bound with 'top-slice targets are reachable via funcl-packages' — codex proved that is FALSE for dynamically-minted KEYWORDS: ag-runtime-mint-keyword (compiler.lisp:618) registers keyword MAPS + a root range, NOT a package cdr. So a top-slice keyword's NAME object is never collected as a heap root, never earns a relocatable index, and its reference dangles/collides post-restore — landing asdf-version's :asdf-version keyword's name on the ASDF-VERSION function symbol's name bv.

(d) MINIMAL SAFE FIX (codex): do NOT extend relocation to a new region (none exists). Instead align ROOT collection coverage with the already-widened encode/relocate coverage — add a ag-image-code-mmap-file-len branch to the root-collection bound (image.lisp:1458). Risk codex flagged: larger root set / slower save (but bounded by registered root ranges, not a raw 32MiB scan). I additionally flag the cave#209 note that widening root collection previously 'GC-thrashed at asdf scale' — so I will apply it and MEASURE the save; if it thrashes I'll make it surgical (collect only keyword records' name targets). Applying + testing (asdf:asdf-version must work post-bake).

FIXED (partial) — committed 23800a2. codex's root-collection diagnosis was correct and the fix WORKS: (asdf:asdf-version) now returns "3.3.7" on the baked gen-2 image (was 'not a byte-vector').

Fix: added a ag-image-code-mmap-file-len branch to the root-collection bound in ag-image-roots-with-code-mmap-pool-roots (image.lisp:1458), so save-side root collection covers the dynamic-symbol top-slice — matching the already-widened encoder/relocator — and top-slice KEYWORD name objects now earn a relocatable heap index.

Verified: (asdf:asdf-version) => 3.3.7 on rebuilt build/funcl-asdf; gen-2 boot ~2.8s; build-asdf save 6:22 with NO cave#209 thrash (peak RSS 699MB, so the 'GC-thrash at asdf scale' fear did not materialize — the root ranges bound the walk, not a raw 32MiB scan); regression-clean: 8/8 sharded suite, smoke, spike 16/16, package edge probes, alexandria-via-fasl (the 24 %DEFTYPE-REGISTER warnings are PRE-EXISTING, byte-identical in this session's pre-change logs).

STILL OPEN on this issue (separate mechanism, NOT the keyword bug): (asdf:load-system "alexandria") on the baked gen-2 image errors 'no accessor | in: LOAD-SYSTEM' — a mini-clos accessor-table restore gap. asdf-version (keyword-driven) is fixed; load-system (mini-clos-driven) needs its own diagnosis. Keeping this issue open for that; the keyword/root-collection half is done.

KEYWORD EQ-IDENTITY split (the 'no accessor' load-system failure) — codex second opinion + fix in build.

Diagnosis (probed): a baked mini-clos instance (vector :%mc-instance% class slotvec) has slot0 = a keyword named %MC-INSTANCE% at heap base 3.4e10, but the runtime/reader/compiled-code :%mc-instance% is a DIFFERENT keyword record at 1.08e9. eq=NIL, so mc-instance-p fails, class-of returns SIMPLE-VECTOR, slot-value raises 'no accessor'. FRESH make-instance works (uses the canonical keyword). Root: ag-image-heap-needs-collection-p (image.lisp:841) returns 1 for every symbol incl keywords, so a keyword reached from a heap slot is HEAP-COPIED and restores as a fresh duplicate; the keyword-pool rebuild only canonicalizes ARENA keywords, and the existing pre-pass2 canonicalizer (cave#155) only unifies keywords with a BAKED STATIC record — :%mc-instance% (minted, no static) has none. This bites EVERY eq/member/case-on-keyword against baked data.

codex refuted my encode-side proposal (new wire tag + deferred fixups across all decode writers = too much surgery) and my 'make all keywords arena-resident' idea (reader/runtime allocation semantics change = high risk). Also rejected 'adopt heap keywords into the pool rebuild' (fixes the door not the existing baked data/code split; arbitrary identity with multiple duplicates).

RECOMMENDED (lowest risk, in build): RESTORE-SIDE canonical reference rewrite. After ag-image-rebuild-keyword-pool-offsets (while ag-image-loaded-heap-master is live), walk the restored heap and rewrite every keyword REFERENCE (cons car/cdr, simple-vector slots) to the canonical keyword of the same name — static (ag-runtime-static-keyword-ptr) else pooled (ag-runtime-pooled-keyword-ptr) else minted (ag-runtime-mint-keyword, idempotent/pool-backed). Identity-preserving (same name → one identity), no image-format change, repairs even the mini-clos instance tags in place. Implemented ag-image-canonicalize-heap-keywords (image.lisp) + the call in repl.lisp restore path. Building + testing (asdf-version AND load-system alexandria must work on the gen-2 image; measure restore time for the heap walk).

FIXED — committed 49f858c. The keyword EQ-identity canonicalization works: (asdf:load-system "alexandria") now runs to completion on the baked gen-2 image (build/funcl-asdf), (alexandria:curry #'+ 1) 41) => 42, 'no accessor' gone.

Fix: restore-side ag-image-canonicalize-heap-keywords (image.lisp) — after the keyword pool rebuild, while the heap master is live, rewrites every keyword REFERENCE (cons car/cdr, simple-vector slots) to the canonical keyword of the same name (static/pooled/minted). Identity-preserving. Gated on the symbol-pool-range readiness so the host-built stage1 skips it (an ungated first cut SIGSEGV'd the stage1 save-image-restart roundtrip — keywordp/ag-runtime-* are prelude fns that crash via the not-yet-ready fn-table during a stage1 image-load; stage1 has no keywords to canonicalize anyway).

Verified: gen-2 image asdf-version => 3.3.7 AND load-system alexandria => 42; gen-2 boot ~2.86s (heap walk adds no measurable cost); 8/8 sharded suite (incl. the stage1 save-image-restart test), smoke, spike 16/16, package edge probes, alexandria-via-fasl on build/funcl all green.

CLOSING cave#314: layer 4 is fully resolved across both halves — (1) the keyword-NAME root-collection gap (23800a2, asdf-version) and (2) the keyword EQ-IDENTITY split (49f858c, load-system). The CDS-style baked-asdf fast-boot image now works end to end: build/funcl-asdf boots ~2.8s with asdf ready and load-system functional, vs ~10s for the fasl replay. Earlier layers 1-3 (nicknames, uses intern-bleed, reader qualified door) landed in cbfbec0. Retitle/close as resolved.