# ADR 0026 — Implementation Status **One canonical symbol per (name, package): the JVM/CDS model for funcl.** > **UN-PARKED 2026-07-08 — driver: asdf functional layer.** `(asdf:asdf-version)` > returns NIL because `*asdf-version*` (a special var EXPORTed by define-package > in a runtime package) splits. RE-BASELINE: `adr0026-witness.sh` is now **16/16** > — builtins, keywords, setf-at-REPL, typep, image-restore doors ALL closed (the > 7-doors framing below is STALE). The ONE remaining gap: symbols `intern` > creates via **make-symbol (stdlib.lisp:8060)** during define-package > `ensure-symbol` BEFORE the reader mints the canonical arena record — measured: > no-export var/fn WORKS, `:export` var SPLITS, `:export` fn SPLITS. Witnessed by > `tests/adr0026-fixtures/{export-var,export-fn,noexport-var}.lisp` (commit > 2e78e3c), tracked EXPECT=fail. Record-map: defvar/setf/reader-literal/free-ref > all bind ONE arena record (bound to "3.3.7"); find-symbol alone misses it. > > **PHASE B DECISION = B1a (codex-consulted 2026-07-08, both models agree).** A > runtime canonical-record ALLOCATOR for dynamic arena 40-byte records; `intern` > allocates/adopts it instead of make-symbol; reader/compiler/find-symbol adopt > the SAME record. REUSE the GC root-range machinery (`gc-mark-pool-range` > gc.lisp:926, `gc-register-root-range` gc.lisp:2940 — one more registered range, > not a new tracing scheme); image restore rides `ag-warm-pool-record-vaddr` > (reader.lisp:835). Gated behind unification=1 (inert at cold build → fixed > point safe by construction). Allocator invariant: init slots to safe immediates > + register the range BEFORE storing real pointers. B3 (find-symbol prefers > arena) REJECTED — lookup hack that leaves the split. B2 (lazy export) does NOT > fix it (exports already name-only; the split producer is intern:8060). 7-step > edit surface: allocator → canonical-find-symbol-record → intern → find-symbol → > ag-intern-symbol/pooled-ptr → ag-symbol-pool-intern → leave pkg-add-export-name. > Acceptance: witness export probes PASS + (asdf:asdf-version)="3.3.7" + > stage2≡stage3. cave#150 (name,pkg keying) deferred to the ADR-pure follow-up. > **Phase C step 1 SPEC (allocator) — feasibility CONFIRMED, ready to write.** > `ag-dynamic-symbol-record-allocate(name-bv, home-pkg)` → tagged symbol ptr. > Primitives all exist: WRITE via `%mem-set-u64`; TRACE via > `gc-register-root-range` (kind `+gc-root-range-kind-symbols+`=1, gc.lisp:2940; > marks slots +8..+32 per `gc-mark-pool-range` gc.lisp:926); LAYOUT from > runtime-layout.lisp (header `(40<<8)|+widetag-symbol+(5)`, name+8, home-pkg+16, > value+24, fn+32; immediates `+imm-nil+`=7, `+imm-unbound+`=23); reader/compiler > ADOPTION via recording `(name . (addr - *ag-pool-emit-base-vaddr*))` in > `*ag-symbol-pool-offsets*` (negative/large offsets OK — base+off arithmetic is > closed, per the Phase-3b baked comment compiler.lisp:428). NON-MOVING region: > carve a disjoint slice at the TOP of the 32 MiB RWX mmap (`+elf-mmap-vaddr+`, > compiler.lisp:6309) with its OWN bump pointer in a fixed heap slot — compiler > pools grow from the bottom, this grows from the top, no cursor coordination > needed (make-symbol's HEAP record is unusable here: heap is MOVABLE and defvar > bakes a fixed value-cell lea). > ALLOCATOR INVARIANT (codex): (1) bump the region pointer; (2) `%mem-set-u64` > the header + init all pointer slots to SAFE IMMEDIATES (name=0/nil, home=7, > value=fn=23); (3) `gc-register-root-range` the record; (4) THEN store the real > name/home pointers; (5) record in `*ag-symbol-pool-offsets*`; (6) return > (addr|3). Gate everything on `*ag-runtime-symbol-unification-p*`=1 → inert at > cold build. STEP 1 lands the allocator + a standalone GC-visibility witness > (alloc a record, bind value/fn/pkg to heap objects, force minor+full GC, verify > identity/value/fn/package survive) BEFORE step 3 touches `intern`. > > **STEP 1 LANDED + VALIDATED 2026-07-08.** `ag-dynamic-symbol-record-allocate` > in compiler.lisp (after `ag-emit-symbol-pool-entries`), top-1-MiB RWX slice, > per-record `gc-register-root-range` (count 1). GC-visibility witness > (`tests/adr0026-fixtures/dyn-alloc-gc.lisp`, wired into adr0026-witness.sh > EXPECT=pass) PASSES: value stays eq across a moving GC, heap closure in the > fn cell stays callable, name survives. `setf symbol-value`/`symbol-function` > on the arena record work with no barrier crash (root-range slots are > unconditionally scanned). Witness 18 PASS / 2 known-fail (export gaps still > fail, awaiting step 3). Full sharded suite GREEN (stage2≡stage3 held, > cold-allowed-dead gate passes — both new fns declared in > cold-bootstrap-retained.lisp, to be REMOVED in step 3 when intern makes them > live). NEXT: step 2 (`canonical-find-symbol-record` lookup helper) → step 3 > (intern uses the allocator instead of make-symbol) → export probes flip to > PASS + `(asdf:asdf-version)`="3.3.7". Known step-1 shortcuts to harden later: > per-record ranges (asdf = hundreds walked per GC → collapse to one growing > range) + an arena-limit/compiler-cursor collision guard. > > **STEPS 2+3 REVERTED 2026-07-08 — net regression for asdf operate.** The intern > rewrite fixed value-cell splits (asdf-version, export fixtures) but BROKE > function-cell resolution for cross-package-REEXPORTED symbols: uiop's > ensure-symbol re-interns a name in each reexporting package and > set-symbol-package churns the adopted record's home, so 6 undefined-function > errors appeared in the asdf load (register-image-restore-hook, output- > translations + source-registry config defparameters) — which actively blocks > operate. Reverted intern → make-symbol and defpackage :export → name-only; > allocator (step 1) kept but dead-in-retained again; witness export probes back > to EXPECT=fail. asdf load back to 0 errors, suite green, self-host holds. > LESSON: the value-cell unification must NOT go through a blanket intern rewrite > — it de-unifies function cells in the reexport path (the cave#150 (name,pkg) > problem in disguise). A correct value-cell fix needs to be narrower (e.g. only > for special-var value cells, or a genuine (name,pkg)-canonical table so > reexport doesn't churn). The load-*package* fix (3a36ee5) is INDEPENDENT and > stays. Below is the reverted description, kept for the record: > > **[REVERTED] STEPS 2+3 — the export gap was CLOSED.** Two > stdlib.lisp changes: (a) `intern` (8060), at unification=1, adopts an existing > pooled arena record (ag-runtime-pooled-symbol-ptr) or allocates a fresh > canonical one (ag-dynamic-symbol-record-allocate) instead of make-symbol — > make-symbol at unification=0 keeps the build byte-identical; (b) native > `defpackage :export` (defpackage--process-export-names) now INTERNs the symbol > (CL-correct) instead of only recording the name — the old name-only form left > exported specials uninterned (OWN=NO) so find-symbol never reached the bound > pooled record. adr0026-witness.sh 20 PASS / 0 FAIL (export-var, export-fn > flipped to EXPECT=pass), full sharded suite GREEN, stage2==stage3 holds, > cold-dead gate passes (the two allocator fns are now BODY-REACHABLE via intern > -> removed from cold-bootstrap-retained.lisp). Verified: `*asdf-version*` now > unified + bound across ASDF/UPGRADE, ASDF, ASDF/INTERFACE and > `(symbol-value (find-symbol (string :*asdf-version*) :asdf))` returns "3.3.7" > at the REPL — the ADR-0026 intern-path split is FIXED. Step-2-as-a-separate- > helper proved UNNEEDED (intern registers into the package alist, so find-symbol > finds it; no find-symbol change). > > **"RESIDUAL" RESOLVED 2026-07-08 — it was a MEASUREMENT ARTIFACT, not a bug.** > `(asdf:asdf-version)` returning NIL was the post-load REPL being left in > `asdf/footer`, where bare unqualified symbol resolution/printing is degraded. > Instrumenting asdf-version to fire DURING load (reliable context) showed its > body computes symval="3.3.7". Reliable `cl:`-qualified post-load calls confirm: > `(cl:funcall (cl:find-symbol "ASDF-VERSION" "ASDF"))` => "3.3.7" and > `(cl:funcall (cl:find-symbol "FIND-SYSTEM" "ASDF") "uiop" cl:nil)` returns a > system. So AFTER steps 2+3, asdf's version + find-system are FUNCTIONAL — the > ADR-0026 export-split fix delivered a working asdf functional layer for these. > LESSON (recorded): never diagnose from bare post-`(load "asdf.lisp")` REPL > forms; the REPL is in asdf/footer. Use cl:-qualified calls, during-load > instrumentation, or a fresh-context fixture. > > Genuinely-open (separate, pre-existing, INTERACTIVE-ONLY): the post-load > asdf/footer REPL context resolves bare symbols poorly (why `(find-package > "ASDF")` bare => NIL there). Cosmetic for programmatic use; file if it bites > interactive workflows. asdf operate (defsystem children / pathname strings, > #232/#235) remains the next real functional target. > **PIVOT (2026-07-01) — the collapse is PARKED; effort moves to ANSI features.** The gate removal > (fa69a8c) destabilized the build: it broke the warm-bundle byte-equality witness (#2) and introduced > a latent portability bug (stage2-bundle→cold leaves warm-overlay relocs zeroed → fault). The quick > alt-3 fix (record fresh-pool sites as relocs) segfaults the build, and a real gdb backtrace needs > DWARF wired into the elisp host (a separate effort — `elisp/elf.lisp` is a minimal emitter with no > section/DWARF emission; funcl-built binaries DO have full DWARF). Rather than grind a destabilizing > change for an ANSI-low-cost bug (bug 3's remaining non-curated split), **the gate removal was > reverted (`git revert fa69a8c` → commit 85a5a91)** — restoring clean green tbq + killing the > portability bug. **Banked wins KEPT: step 1 (setf at REPL) + door 7 (setf of car/cdr/gethash now > works at runtime on the image binary).** The full collapse is revisited LATER as one designed, > DWARF-debuggable unit: gate removal + the proper serializer/loader alt 3, after the elisp-host-shim > refactor (which wires DWARF into the elisp host as a side effect). **Bug 3 returns for non-curated > names (accepted, ANSI-low). Effort now → ANSI features (the actual destination).** Loop `f0b52fd9`. > **HANDOFF (2026-06-30).** Step 1 + door 7 are **landed on `main`** (commit 3288a48): bug 1 > (`setf` at the REPL) is dead on `build/funcl-stage2`, door 7 rebuilds arena offsets on image > restore; fixed point intact, elisp-safe. Step 2 (the breaking bet: unification-on in > `ag-cli-compile-source`) is **gated on three cheap probes, not started** — see "Step 2 — full > analysis for review" below. **DECISION (2026-06-30): (A) — complete the collapse, no compromise. > (B) rejected.** And a reframe that lowers (A)'s cost materially: the go/no-go + code reading show > **bug 3 is a RUNTIME (REPL) routing issue, NOT a build-path one**. `ag-arena-baked-reuse-vaddr` > (compiler.lisp:518) returns -1 whenever `ag-arena-emit-p` is NIL — i.e. during BUILD, *no* quote > routes to the static record (all resolve to the freshly-emitted pool record, deterministic). And > `ag-portable-static-builtin-p` is **never called during the build** (reader door at reader.lisp:728 > is gated on the unification flag = OFF at build; compiler door at compiler.lisp:519 is gated on > `ag-arena-emit-p`). **So step 2.1 (elisp host static pool, cave#71) is NOT a prerequisite for the > collapse** — the build/fixed-point is untouched by the routing gate; stage1 may differ by design > (ADR 0026 lines 161-170, confirmed by the go/no-go). The collapse = make **RUNTIME** routing > canonical (package-aware, all names → one record) = Phase 5, which is runtime/warm-boot work, > fixed-point-safe by construction. The earlier "moderate-major elisp-host prerequisite" framing > (probe 1) was a misread of the build/runtime boundary — it stands only if Phase 3 rewrites the > *build* quote emission, which the runtime-routing path does not require. > > **VALIDATED (2026-06-30) — the collapse is runtime work at low cost.** Widened the runtime gate to > GETHASH/LIST/MEMBER (Phase 3 increment 3, `compiler.lisp:466`). Bug 3 dead for them: `(eq 'gethash > (find-symbol ..))`⇒T, `(eq 'list ..)`⇒T, `(eq 'member ..)`⇒T, function cell intact (`(let ((h > (make-hash-table))) (setf (gethash 'a h) 99) (funcall 'gethash 'a h))`⇒99). `ert funcl-stage1-self- > host-fixed-point` PASSED — stage2≡stage3 holds, so widening is build-safe by construction. **No step > 2.1 required.** (codex was attempted but hung on stdin / failed exit 144 in this env — proceeded > with own analysis.) **Next: Phase 5** — package-aware (name,pkg)-keyed routing for ALL symbols > (kills cave#150's bare-name misroute too), then delete the bitmap/flag/reconciliation. > > **PHASE 5 GATE REMOVAL — DONE + validated (2026-06-30, the collapse's core).** Removed the curated > gate (`ag-portable-static-builtin-p`) from BOTH doors — the reader (`ag-runtime-static-symbol-ptr`, > reader.lisp:724) and the install thunk (`ag-arena-baked-reuse-vaddr`, compiler.lisp:514) — so > EVERY static-pool symbol routes to its one canonical record (they must move together or cells > split, cave#131). **Bug 3 is DEAD for all CL symbols, both binaries**: `(eq 'append/format/reverse/ > nth/length/string/vector/null/… (find-symbol ..))`⇒T (was NIL). Function cells intact (`(funcall > 'append …)`⇒correct), `ert funcl-stage1-self-host-fixed-point` PASSED (stage2≡stage3; build path > untouched — the gate was never called during build). Cost: ~2s startup regression (the reader's > O(n) pool scan runs per symbol at prelude reload) — optimizable later via a COLD hash index (the > prelude O(1) `*canonical-symbol-index*` can't be called from the cold reader — segfault during > prelude emit, per compiler.lisp:514). `ag-portable-static-builtin-p` is now DEAD CODE (no callers) > → delete in cleanup. **DOOR 7 LANDED (commit d6db689):** image setf-car/cdr/gethash residual is DEAD. > `ag-image-load-heap-canonicalize-symbols` (src/warm/image.lisp) routes each restored symbol to its > baked record before pass2, so saved macro bodies' `'car`/`'cdr`/`'gethash` resolve to the same baked > record the reader hands out → setf dispatches correctly AND works at runtime on build/funcl > (`(let ((x (cons 1 2))) (setf (car x) 9) x)`⇒`(9 . 2)`). Fixed-point-safe (warm/image-only). Witness > now 16 PASS / 0 FAIL. **Still open:** keyword split (separate keyword pool), package-aware (name,pkg) > routing (cave#150, latent), and the dead-code/flag/reconciliation deletion (all LIVE, gated on Phase > 2/3). > > **TBQ REGRESSION (2026-07-01) — BLOCKS `make test-bootstrap-quiet`, needs a design call.** Two > failures surfaced when the full suite finally ran: > 1. `funcl-cold-allowed-dead-respected` — FIXED (commit b05003d): stale manifest entry for > `ag-set-runtime-symbol-unification` (step-1a's cold-boot call made it reachable; removed). > 2. `funcl-cold-warm-bundle-byte-equality-witness` — **OPEN, caused by the gate removal.** The > warm bundle is built TWO ways: `funcl-warm.fco` via the REPL arena (`produce-warm-bundle.lisp`, > unification ON → baked-reuse), `funcl-warm-cold.fco` via the cold build (`--build-warm-bundle`, > unification OFF → fresh-pool refs). The gate removal made the arena path baked-reuse ALL symbols > (was: tiny eligible set), so the two bundles now diverge by **262441 bytes (91%)** + different > sizes. The stage2≡stage3 fixed point is UNAFFECTED (it's the build path); only the REPL-arena-vs- > cold-build witness breaks. **Agent investigation (2026-07-01) corrected the diagnosis + options:** > - The real differentiator is NOT the build path (both use the arena; `ag-arena-emit-p` TRUE on > both) — it's the **baked static pool each binary carries**: funcl-stage2's pool includes the > warm-overlay symbols (ag-repl-*), funcl-cold's doesn't (different source sets, Makefile:71-72). > So stage2 routes MORE symbols to baked-reuse than cold → divergent reloc metadata sets. > - **Latent portability bug (worse than the witness):** the stage2 bundle loaded into cold has > warm-overlay baked-relocs cold can't resolve → `load-reresolve-baked-reloc` leaves the field > zero (cold-loader.lisp:172-173) → fault if reached. Current `--start-warm` tests miss it (they > enter via the fn-table offset, exercise only cold-pool symbols). REAL regression, not just > byte-difference. > - **Option A (disable baked-reuse for serialization) is BROKEN — do NOT pursue:** the reader > door stays ON (unification), so compiled quotes would route to fresh arena while the reader > routes to baked → not eq → bug 1/3 returns during the bundle compile itself. > - **The correct fix (alt 3):** extend the v4 reloc mechanism to EVERY symbol-lea + global-value- > cell site (not just baked-reuse) — both producers then zero all such sites + emit identical > name metadata → byte-identical bundles with NO routing change; the loader re-resolves every > site by name (baked if in the loading pool, else arena) → eq holds in every loader, fixing the > portability bug too. Edit surface: `ag-serialize-co` (cold-fco.lisp:146) record all > `*ag-symbol-patches*`/global patches as relocs + zero them alongside > `ag-fco-zero-patch-rel32-walk` (:71); re-resolve in `load-install-co` (cold-loader.lisp:207). > Cost: serializer+loader surgery + a .fco format bump; modest identical size growth. > **DECISION NEEDED:** pursue alt 3 (substantial but correct, fixes witness + portability), or > reconsider the gate-removal scope, or bank the landed collapse + revert the gate removal pending > this work. NOT YET DONE — surfacing for a call before the (slow, ~50min) tbq re-run. > > **DECISION (2026-07-01): pursue alt 3 (keep the gate removal).** ATTEMPTED + REVERTED: the 3 edits > (record fresh-pool quote sites as kind-1 relocs in ag-resolve-symbol-patches; record fresh-pool > global value-cell sites as kind-2 in ag-resolve-global-vaddr-patches; loader arena-fallback in > load-reresolve-baked-reloc) — no .fco format bump needed (the existing baked-relocs section > handles kind 1/2). BUT the compiler edits SEGFAULT the build (`make build/funcl-stage2` + `make > build/funcl-cold` both dump core during COS-EMIT). Reverted; build green again. The segfault is > from recording fresh-pool sites during BUILD emit (baked=-1 there → fresh path → ag-baked-relocs- > add now runs during build, which it never did before) — root cause needs **gdb** (the prompt's > "use gdb early"): re-apply the compiler edits, `gdb build/funcl-stage1` running the COLD_SRCS > compile, backtrace. Likely ag-baked-relocs-add during build emit hits an uninitialised dep, or > the recorded relocs pollute a later emit phase. NEXT: gdb-diagnose, then re-implement alt 3 > (possibly gate the recording on ag-arena-emit-p so it only runs at the REPL arena, not during > build emit — the build doesn't serialize .fco, so it doesn't need relocs recorded). > > **FAN-OUT SYNTHESIS (2026-06-30, 3 read-only agents) — plans for the remaining fronts:** > - **Scaffolding deletion = NO-OP for now.** All 5 ADR-listed candidates are LIVE: `ag-canon-hosted-args` > (→ ag-macro-apply-hosted, compiler.lisp:3943), `ag-repl-register-quoted-symbols` (repl.lisp:809/856/862), > `find-symbol-in-pool` (fallback in find-symbol-via-index, prelude.lisp:5765/5768), `ag-runtime-pooled-symbol-ptr` > (arena fallback in ag-intern-symbol, reader.lisp:761), `*ag-{keyword,}-symbol-table*` (runtime intern). > Deletable only after Phase 2/3 prerequisites. Load-bearing `*ag-runtime-symbol-unification-p*`, > `*ag-symbol-pool-offsets*`, `gc-root-ranges` confirmed LIVE — do NOT delete. > - **Door 7 (image setf-car/cdr) — small-medium, HIGH ANSI value, DO FIRST.** Root cause: image-restart > allocates a FRESH heap record per saved symbol (image.lisp:2098 `%alloc-symbol-record`), never the > canonical baked record, so the restored setf body's `'car` ≠ the reader's baked `car` → every > `(eq head 'car)` misses → `(SETF-CAR 9 X)`. Door-7's `*ag-symbol-pool-offsets*` rebuild is INERT for > this (the body holds a fresh heap record; neither side of the eq consults the arena map). **FIX:** > insert a canonical-routing sub-pass in `ag-image-load-heap-section` (image.lisp:2364, between `master` > creation and pass2) — walk the section by index i; for each `+image-heap-tag-symbol+`, decode the name > (name-ref → `svref master idx` BV) and if `ag-runtime-static-symbol-ptr`(name) finds the baked record, > `svset master i baked` (discard the skeleton). Pass2 decodes refs through `master` → eq holds. Baked > cells are authoritative (fns from build; globals via restore-bootstrap-globals) so no cell-merge. > Risk: fixed-point LOW (restore-only), elisp NONE, ANSI POSITIVE (unblocks setf of car/cdr/gethash/ > svref/aref on build/funcl). CARE: only rewrite for names that hit the baked scan; user symbols keep > the fresh skeleton; a redefined-CL-symbol edge case loses the redefinition. > - **Keywords door — medium.** Reader never routes to the baked keyword pool (`ag-intern-keyword` > reader.lisp:854 → `*ag-keyword-symbol-table*` cache → make-symbol; NO gate). find-symbol scans the > SYMBOL pool only. FIX = 3 sites mirroring the symbol-pool pattern: (1) `ag-runtime-static-keyword-ptr` > (scan gc-keyword-pool-range) wired into ag-intern-keyword before the cache, gated on unification; > (2) `ag-arena-baked-reuse-keyword-vaddr` wired into ag-resolve-keyword-patches (compiler.lisp:614); > (3) prelude find-symbol keyword-pool scan when pkg is KEYWORD. MANDATORY coupled edit: enter the > baked keyword record into `*ag-keyword-symbol-table*` so `ag-keyword-symbol-p` stays valid > (compiler.lisp:651, ast-ir.lisp:165). NEW D2b surface: keywords have no reloc recording today; > door-2 needs `ag-baked-relocs-add` + `load-reresolve-baked-reloc` keyword-pool extension. > Fixed-point/elisp/cell-split all clean. > - **ORDER:** door 7 → keywords → package-aware (cave#150, defer). Implement SERIALY (one rebuild + > `ert funcl-stage1-self-host-fixed-point` at a time). > > **Immediate next actions (the disciplined plan), valid regardless of A-vs-B:** > 1. **Probe 0 (DONE):** keyword split LIVE `(eq :foo (find-symbol "FOO" KEYWORD))`⇒NIL, bug-3 > unrouted LIVE `(eq 'gethash (find-symbol "GETHASH" CL))`⇒NIL (also `list`/`member`/`format`), > on BOTH binaries (single-form pipes — multi-form gives false NILs, see artifact warning). The > only unified names are `cons`/`car`/`cdr`/`character`/`integer`/`symbol`. Bug-2 (typep) NOT > reproducing (fixed by step 1 + door 7). Identity is still split — but ANSI FAILs are mostly > missing features, not identity (per the baseline note below), so bug 3's ANSI cost is low. > 2. **Probe 1 (DONE — and it REVISES the cost back UP):** the elisp host has its own `emit-quote-pool` > (`elisp/codegen.lisp:315`) and `gc-register-root-range` is a no-op (codegen.lisp:295). BUT the > elisp host uses a **single unified `*quote-pool`** for symbols+strings+static-bvs > (`host-pool-intern`/`ag-quoted-emit-symbol`, codegen.lisp:261) — it has **no separate > `*ag-symbol-pool`**, while `gc-pool-slot-patch-value` reads `*ag-gc-symbol-pool-emit-start*`/`-count` > (funcl's separate symbol pool). So step 2.1 is NOT a simple backfill mirror — the elisp host's > symbol-pool range slots have no clear value (symbols are merged into `*quote-pool*`). > Reconciling the pool structures (give the elisp host a separate symbol pool, or compute > symbol start/count from the unified pool) is the real work. **So step 2.1 is moderate-to-major > after all — the earlier "small backfill" read was wrong.** This re-tilts A-vs-B toward (B). > 3. **Probe 2 (DONE — the bet's RISK is gone, but the flip is INERT):** flipped unification-on > in `ag-cli-compile-source` (no preseed); `ert funcl-stage1-self-host-fixed-point` ⇒ **passed > 1/1** (stage2≡stage3 holds, 286s). ON-stage2 ≠ OFF-stage2 (163 bytes) ⇒ real change, not a > no-op. **The blocker analysis was wrong about SAFETY — the flip is fixed-point-safe regardless > of step 2.1.** BUT the flip does NOT fix bug 3: ON-stage2 identity probes identical to OFF > (`gethash`/`list` still NIL) — the eligible set is unchanged. The flip is necessary-but-not- > sufficient; canonical symbols still need step 2.1 +/or Phase 5. **So A-vs-B now stands on > COST/VALUE, not risk** — and step 2.1 is moderate-to-major (probe 1) for an ANSI-neutral gain. > 4. **Filed:** cave#150 (bare-name routing hazard — latent, masked by reader not being > package-aware). **Pre-decided risk-2 fallback = name-keyed deterministic addressing** (not > more preseed hacking — cave#131 thrashed 4 rounds without a pre-decided fallback). > > **PROBE ARTIFACT WARNING:** multi-form pipes (several forms to one `build/funcl` invocation) > produce false NILs — the `defpackage`/`in-package` forms destabilize the REPL state and > `tail -1` grabs wrong lines. **Always probe one form per invocation.** (This caused a false > "keywords broken" scare; clean single-form `(eq :foo :foo)` ⇒ T, `(symbolp :foo)` ⇒ T.) > > **Rules:** no reconciliation hacks (ADR 0026 lists them for deletion); the elisp host is > PERMANENT (CLAUDE.md) — verify with `build/funcl-stage1` cold-boot; use gdb early; consult codex > for risky design calls. Track bugs in cave (`export CAVE_REPO=atgreen/funcl`). Loop job 9ace3cb3. > > **ANSI baseline (post step-1+door-7, main 06802fa, default dirs):** cons 334P/1237F/3C, > data-and-control-flow 506P/805F/4C, symbols 53P/303F, characters 131P/125F — **TOTAL 1024P/2470F/7C** > (165 files). FAILs are dominated by missing features (sequence fns, LOOP, strings), not symbol > identity — i.e. step 1's collapse payoff is architectural more than ANSI. No clean pre-step-1 > baseline was captured to quantify bug-1's delta; capture one before the next symbol change if you > want to measure it. This file tracks the *implementation* state of [ADR 0026](docs/decisions/0026-single-canonical-symbol-intern.md) (single canonical symbol intern) and its prerequisite [ADR 0027](docs/decisions/0027-canonical-string-pool-interning.md) (canonical string pool). The ADRs hold the design and rationale; this file holds the engineering status, what has landed, and the concrete plan to finish. Updated 2026-06-30. --- ## PIVOT (2026-06-30): coordinated collapse on a branch — supersedes the incremental plan The incremental plan below (7 doors, bitmap, curated gate, bridge-vs-door test) is **scaffolding the plan itself schedules for deletion**. It was forced by treating the continuous byte-identical self-host fixed point as inviolable — but that's a *process* constraint (bisection), not a product requirement; the product needs the fixed point at releases. Incrementalism plus an incomplete canonical table forced every piece of router machinery (`*ag-runtime-symbol-unification-p*`, `ag-arena-emit-p`, `ag-portable-static-builtin-p`, the bitmap). **All three bugs are one root**: a symbol splitting between a deterministic build record and an emission-order-dependent runtime record. Bug 1 (tree-walker make-symbol), bug 2 (stale arena offsets after restore), bug 3 (curated gate) are its three faces — classifying the faces was not progress. **The endgame is simple and not novel** — it's ECL (`cl_symbols[]` + `ECL_SYM(name,code)`) and the JVM (CDS), confirmed by a code-grounded fan-out of the reference implementations. One canonical table (a baked static section + a deterministic dynamic extension for runtime-new symbols, the prelude included), every door calling one `canonical-find-or-intern`, compiled quotes pointing at the fixed record. No router, no gate, no bitmap, no bridge distinction. **The lever:** the static pool is deterministic; the arena is not. The fixed point is fragile only because the arena's addresses are emission-order-dependent. Complete the one deterministic canonical table (so the prelude's runtime-compiled symbols are canonical too, via the deterministic dynamic extension — the ECL `SymbolTable` technique) and unification can be flipped ON during cold-boot, fixed-point-safe by construction. The build/runtime split — the one root — disappears, and there is nothing left for the router to paper over. **The plan (on branch `adr0026-collapse`):** 1. Build the **one deterministic canonical table** — static section + deterministic dynamic extension (index-per-name, emission-order-independent). Covers every symbol including the runtime-compiled prelude. 2. **Flip unification ON during cold-boot** — fixed-point-safe by construction once the table is deterministic across its whole extent. 3. **Make every door one lookup** — tree-walker, reader, compiled quote, value/function cells, image-restore, find-symbol/intern, keywords. No build/runtime split left to bridge. 4. **Delete the router, the flag, the gate, the bitmap, the reconciliation** in one coordinated pass (Phase 5). **Tradeoff, accepted:** a bounded period during which stage2≡stage3 is *temporarily broken* on the branch, re-established at the end. `main` stays at the last byte-identical commit (bisectable forever); the branch lands when the fixed point is re-established. ADR 0025 (the image is the symbol table, mapped in place) is untouched — it's the sound decision that makes the simple endgame possible. ### Progress (2026-06-30) - **Step 1 LANDED — bug 1 (setf at the REPL) dead on `build/funcl-stage2`.** Two changes in `ag-cold-boot` (src/cold/cold-fco.lisp): (1a) flip `*ag-runtime-symbol-unification-p*`=1 before the prelude read; (1b) a throwaway read that **preseeds the arena pool** (intern every non-static prelude symbol name, emit + register) before the live read, so the live read resolves every prelude symbol to its arena record (not the make-symbol fallback). `(setf (gethash 'a h) 1)` mutates, `(macroexpand-1 '(setf (gethash a h) 1))`=>`(SETHASH A 1 H)`, `(member 'list '(a list b))` =>`(LIST B)`. No tree-walker surgery, no bitmap. Fixed-point-safe by construction (`ag-cli-compile-source` doesn't call `ag-cold-boot`); gate deferred per the bounded-break policy. - **Door 7 LANDED — image-restore rebuilds `*ag-symbol-pool-offsets*` from restored `gc-root-ranges`** (src/warm/image.lisp + repl.lisp). Fixes arena-name identity on `build/funcl`: setf-gethash macroexpand correct, typep-cons-image fixed. - **Deferred — `setf-car`/`setf-cdr` on the image binary still mangle.** Diagnosed: the *saved* setf macro body's `'car` (a static-pool reference) isn't the static record after restore, even though the reader's `'car` is (image save/restore mishandles the cross-region static-pool reference in saved macro bodies). This is a symptom of increment-2's static routing (scaffolding) + the image pointer taxonomy — Phase 5 (one canonical table, properly serialized) fixes it. Not point-fixed (don't invest in scaffolding). Note: `build/funcl-stage2` (the ANSI runner binary) is fully fixed, so this doesn't block ANSI. ### Step 2 — full analysis for review (2026-06-30, codex-validated) This is the genuinely-breaking collapse work. **Read this before investing.** It is materially larger and riskier than step 1 — codex's review corrected an earlier "small change, mostly done" optimism about the pure-data table. **Goal.** Flip `*ag-runtime-symbol-unification-p*`=1 during the SELF-COMPILE path (`ag-cli-compile-source`, cold-bootstrap.lisp:457) so the compiler's own symbols are canonical, dissolving the build/runtime split at its root. If stage2≡stage3 still holds, the bet is proven and the scaffolding (increment-1/2 routing, the curated gate, the partial scans) can be deleted. **The blocker (the elisp host — cave#71).** The fixed point stage2≡stage3 compares stage1's compilation of COLD_SRCS (→ stage2) against stage2's compilation of COLD_SRCS (→ stage3). With unification ON, the compiler resolves symbols to the **static pool**. stage2 (funcl-built) has a populated static pool; **stage1 (elisp-built) has an EMPTY one** — the elisp host can't run native `gc-patch-pool-slots` (gc.lisp:2514), which backfills the range slots (start/count at heap +224/+232). So unification-on would make stage1 resolve to make-symbol while stage2 resolves to static records → **different bytes → stage2≠stage3 → fixed point breaks.** This is the fundamental obstacle, and it is why step 2.1 is a hard prerequisite, not optional. **Step 2.1 — pure-data range slots (prerequisite, major).** The symbol RECORDS are already pure-data (`ag-emit-symbol-pool-entries` compiler.lisp:415 emits all four slots at emit time); `gc-patch-pool-slots` backfills ONLY the range slots. To make the table fully pure-data, the range slots must be written as emit-time data with the known start/count. But the elisp host **does not run funcl's `ag-build-image`** — it has its OWN emitter (`elisp/elf.lisp`, `elisp/funcl-host.el`). So step 2.1 requires teaching the **elisp host's emitter** to populate the range slots as data. That crosses into the permanent elisp host (CLAUDE.md), not just `src/`. (The earlier note "mostly done already" was wrong — the records are done, but the range-slot emit in the elisp host is the real work, and it's required.) **Step 2.2 — the bet (genuinely-breaking).** Once the elisp host emits a populated static table: add a throwaway-read + **preseed of `*ag-symbol-pool-offsets`** (scan the running compiler's baked static pool, like `ag-runtime-static-scan-rec` reader.lisp:717, but ungated) before the live read in `ag-cli-compile-source`; flip unification=1 for the live read; let `ag-build-image` do the final emit once. Do NOT add a second `ag-emit-quote-pool` (ag-build-elf-ir resets the pool state at compiler.lisp:5725; a second emit breaks the single gc-patch-pool-slots model). **Three nondeterminism risks codex flagged (the go/no-go hinges on these):** 1. Static record address = `base+offset+3`; offset depends on emission order; order depends on `*ag-symbol-pool*` (consing names in `ag-symbol-pool-intern`). A complete preseed eliminates encounter-order risk **only if every later occurrence hits an existing deterministic offset**. 2. Generated symbols / macroexpansion introducing new names after the preseed can still append (non-deterministic). 3. With unification=1, `ag-symbol-pool-intern` (compiler.lisp:383–389) scans `gc-root-ranges` and may reuse producer-process arena records — wrong for static self-compile; needs an `ag-arena-emit-p` guard so static builds never reuse arena records. **Go/no-go witness.** `ert funcl-stage1-self-host-fixed-point` + `tests/adr0026-static-pool-witness.sh` (+ the elisp bootstrap). If green: bet proven, rest is mechanical deletion. If red: instrument `--dump-static-symbol-pool` diff stage2-vs-stage3 — the divergence points at the nondeterministic allocation (most likely the dynamic extension / generated symbols). **Effort / risk.** Step 2.1 (elisp host emitter change) + 2.2 (preseed + unification + the gc-root-ranges guard) + 2.3 (delete scaffolding) is a large, multi-file, fixed-point-risky effort — comfortably larger than step 1. It does not directly improve ANSI (that was step 1, already landed); it is pure collapse-completion / scaffolding-deletion. **Strategic fork (decision point):** - **(A) Invest in step 2.1+2.2** — complete the collapse, delete the scaffolding, fix the image `setf-car` residual and keyword split as side effects. Large, risky, multi-day; pays off as architectural cleanup, not ANSI. - **(B) Bank the step-1 win and defer collapse-completion** — step 1 + door 7 already killed bug 1 on the ANSI binary (the user-visible win). Stop the collapse here (main is clean, fixed point intact), measure ANSI, and spend the effort on other ANSI gaps (e.g. the keyword split as a standalone fix, or missing sequence/string functions) that move the ANSI needle directly. The sections below (TL;DR, seven doors, bitmap generalization) are the **superseded incremental plan**, kept as a record of the intermediate state and the shipped increments 1–2 (which stand correct and stay until the collapse deletes their machinery). ### Step 2 — investigation findings (2026-06-30 evening, /loop session) Empirical work this session (single-form pipes only — multi-form batches give false NILs, the trap that mis-framed cave#150): - **ANSI baseline banked (run complete).** `/tmp/ansi-postmerge.log`: **PASS=1024 FAIL=2470 CRASH=7 (165 files)** across cons / data-and-control-flow / symbols / characters. This is the post-step-1+door-7 number to beat; step 2 is ANSI-neutral, so it does not move this. - **Probe-0 clean (single-form; both binaries identical on the bug-3 surface):** - **Keyword split LIVE on both** `build/funcl-stage2` and `build/funcl`: `(symbol-package :foo)`=>`(KEYWORD)`, `(symbol-name :foo)`=>`FOO`, yet `(eq :foo (find-symbol "FOO" (find-package "KEYWORD")))`=>`NIL`. The reader keyword and the find-symbol entry are distinct records (the "keywords" door; also option B's standalone-fix candidate). - **Bug-3 unrouted LIVE on both**: `(eq 'gethash (find-symbol "GETHASH" CL))`=>`NIL`, same for `sethash`, `member`, `format`, **and `list`**. The unified set is exactly `ag-portable-static-builtin-p`: `car cdr cons character integer symbol` (+ bootstrap globals). `list` is NOT routed — correcting any "list primitives" assumption (only cons/car/cdr are). - **Bug-2 (typep) NOT reproducing**: `(typep '(1) 'cons)`=>`T` on both binaries. Door 7 + step 1 appear to have closed the image-only `cons` typep residual. - The only image-vs-stage2 divergence on this surface remains **bug 1**: `(macroexpand-1 '(setf (car x) 9))`=>`(SETCAR X 9)` on stage2 but `(SETF-CAR 9 X)` on the image binary (the deferred setf-car/setf-cdr image residual). - **Step 2.1 scope — the "small / shared-codegen" premise is REFUTED (major correction).** The elisp host does NOT reuse the shared `ag-build-elf-ir` / `ag-emit-symbol-pool-entries` codegen for stage1. It runs a **parallel pipeline**, `emit-elf-program-with-globals` (`elisp/codegen.lisp:545`), which (a) never calls `gc-patch-pool-slots`, (b) never sets `*ag-gc-symbol-pool-emit-start*`/`-count*` (those are captured inside the shared `ag-emit-quote-pool`, `compiler.lisp:322-325`, which the host doesn't call), and (c) overrides `ag-quoted-emit-symbol` (`elisp/codegen.lisp:261`) to route quotes to `*quote-pool*` (widetag-0x01 bv lea) instead of `*ag-symbol-pool*` (40-byte widetag-0x05 records). Consequences: 1. **Range slots are intrinsically POST-POOL backfills, not inline data** — start/count exist only after the pool is laid out (`asm-mov-rax-imm64-patchable` docstring, `asm.lisp:188-191`). The Step 2.1 framing above ("write the range slots as emit-time data with the known start/count") is **wrong**; they cannot be known at the placeholder site. Treat that paragraph as superseded. 2. **stage1 vs stage2 diverge on the QUOTE BYTES themselves** (widetag 0x01 vs 0x05), not only the range slots — so populating range slots in isolation yields `0/0`-equivalent output and does **not** achieve stage1≡stage2. `gc-patch-pool-slots` is pure Lisp (not native-only); the host *could* run it but doesn't, and doesn't produce its inputs. **Verdict: step 2.1 is MEDIUM-LARGE** (converge the host onto the shared pool path, or mirror it), and even completed it is insufficient — the quote-routing divergence must also be closed. This re-opens the strategic fork (A vs B), harder: step 2 is larger and riskier than "comfortably larger than step 1." - **Go/no-go RESULT — the bet's RISK is gone, but the flip is INERT for identity.** `*ag-runtime-symbol-unification-p*`=1 flipped in `ag-cli-compile-source` (`cold-bootstrap.lisp:457`, save/restore, NO preseed); `ert funcl-stage1-self-host-fixed-point` ⇒ **passed 1/1** (286s) — stage2≡stage3 holds with unification ON. **Discriminator (not a no-op):** ON-stage2 ≠ OFF-stage2 by 163 bytes (sha `39a180af240b82d2` vs `602fff3a311c6917`), so the flip is a real emission change. **This refutes the blocker analysis above (and the "massively red" prediction): the flip is fixed-point-safe regardless of step 2.1 — stage1's empty static pool does NOT force divergence.** The go/no-go's "slight red ⇒ preseed; massive red ⇒ name-keyed" fork did not fire — it was green. **BUT the flip does NOT fix bug 3:** ON-stage2 identity probes are byte-for-byte identical to OFF-stage2 — `(eq 'gethash (find-symbol "GETHASH" CL))`⇒NIL, `(eq 'list …)`⇒NIL, keyword split NIL (`cons`/`car` T, `setf-gethash`⇒`(SETHASH A 1 H)` from step 1, in both). The eligible set (`ag-portable-static-builtin-p`) is unchanged, so no new symbols become canonical. **The flip is necessary infrastructure (and now proven safe) but not sufficient** — canonical symbols still need step 2.1 (so stage1's empty pool stops making routing a no-op) and/or Phase 5 (name-keyed table). Net: the collapse's *risk* is lower than thought; its *cost* is unchanged (step 2.1 moderate-to- major + Phase 5 for actual canonical-ness, ANSI-neutral). Experimental edit reverted; tree clean (matches main). The pre-decided risk-2 fallback (name-keyed addressing) is NOT needed for safety — only relevant if pursuing canonical-ness via Phase 5. - **Cave: no new issue warranted.** The "bare-name shadowing hazard" the loop flagged is bug 3 (reader ≠ find-symbol for non-eligible names), already the cluster hub **cave#149** (with the typep/find-symbol grid). **cave#150** is a *separate, genuinely latent* hazard (package-**shadowing** misroute, masked by reader package-unawareness) — its "masked" claim is correct *for the shadowing scenario*. Single-form keyword-split + `list`-unrouted measurements recorded as a comment on #149; no duplicate filed. --- ## TL;DR The cluster of symbol-identity bugs (`(eq x 'literal)` failing on runtime-read code: cave#44 / #120 / #130 / #132 / #135 / #149, plus the hash-table `equal`-on-names issues #19 / #117) is **downstream of two stacked ADRs**. ADR 0027 (canonical string pool) is the foundation; ADR 0026 (single canonical symbol) sits on top. The **routing mechanism is proven and landed** for the common builtin classes. The single-canonical-symbol invariant now holds — reader = compiled quote = `find-symbol` = `intern` = function-cell, all resolving to one baked static record — for cell-free type names and cell-bearing list primitives (doors 1–5). **Bug 3 is dead for the routed names.** **Honest scope (reframed 2026-06-30):** the cluster is *seven doors*, not one. Name-set generalization (routing all portable builtins through doors 1–4) kills bug 3 and **nothing else**. Bugs 1 (`setf` at the REPL) and 2 (`typep` after image reload) live in doors the router never touches — the tree-walker (door 6) and image-restore (door 7) — and need their own door-completion steps. Generalizing the router alone lands ~the bug-3 axis; closing all seven doors is the path to one-instance-everywhere. See *The seven doors*. **Do not point-fix these bugs by reconciliation** (extending `ag-runtime-pooled-symbol-ptr` to scan `gc-root-ranges`, name-scanning, `*ag-runtime-symbol-unification-p*` tweaks). ADR 0026 lists those mechanisms for **deletion**. The pure path is the canonical static record, reached "by construction," not reconciled after the fact. (cave#131 thrashed 4 rounds on reconciliation; that history is why the pure approach was committed.) Note the tension this file must keep honest: completing the router across doors is *interim scaffolding* — every door made canonical is progress, but the terminal architecture is one `canonical-find-or-intern` with the router deleted (Phase 5). --- ## What has landed (committed, gated, witness-guarded) | Commit | What | |--------|------| | `4f6e062` | **Phase 0 witness** — `tests/adr0026-witness.sh`, the canonical-symbol-identity characterization probes (failing-by-design, tracks ADR 0026 progress). | | `1c146d0` | **Determinism witness** — `--dump-static-symbol-pool` flag + `tests/adr0026-static-pool-witness.sh` (ADR 0026 Risk 1 guard). | | `d5e2c1d` | **Phase 3 increment 1** — route cell-free builtins (type names) to the canonical static record. | | `7445818` | **Phase 3 increment 2** — route cell-bearing list primitives (`CONS`/`CAR`/`CDR`) to the static record, with function-cell routing. | All gated by the self-host fixed point (`tests/bootstrap.el`, 208/208 ert, 0 unexpected) and the `.fco` byte-equality witness (D2b). The routing is gated on `*ag-runtime-symbol-unification-p*` (off during the cold build) and `ag-arena-emit-p` (false during build pool emission), so cold emission is byte-identical across funcl-built stages. ### Probes that now pass (were failing) ```lisp (eq 'character (find-symbol "CHARACTER" (find-package "COMMON-LISP"))) => T ; increment 1 (eq 'integer (find-symbol "INTEGER" (find-package "COMMON-LISP"))) => T (eq 'symbol (find-symbol "SYMBOL" (find-package "COMMON-LISP"))) => T (eq 'cons (find-symbol "CONS" (find-package "COMMON-LISP"))) => T ; increment 2 (eq 'car (intern "CAR" (find-package "COMMON-LISP"))) => T (eq 'cdr (find-symbol "CDR" (find-package "COMMON-LISP"))) => T ;; and function-cells still resolve (the risky part — validated): (funcall 'cons 1 2) => (1 . 2) (apply #'cdr '((9 8 7))) => (8 7) (mapcar #'car '((1 2)(3 4))) => (1 3) ``` ### How the routing works (the proven mechanism) A builtin symbol has one canonical record in the **baked static symbol pool** (`gc-symbol-pool-range-start/count`, heap offsets +224/+232). `find-symbol`/`intern` already resolve there (`find-symbol-via-index`, `src/prelude.lisp`). Phase 3 makes the other "doors" resolve there too, all sharing one eligibility gate: - **`ag-portable-static-builtin-p`** (`src/cold/compiler.lisp`) — the D2b-safe eligible-set discriminator (bootstrap globals + a curated portable subset present in *every* producer's static pool). - **Reader** — `ag-runtime-static-symbol-ptr` in `ag-intern-symbol` (`src/cold/reader.lisp`) resolves an eligible name to its static record first, then arena, then a fresh cached symbol. - **Compiled quote + value-cell + function-cell install** — `ag-arena-baked-reuse-vaddr` (`src/cold/compiler.lisp`) routes the same eligible names to the same static record. The function-cell install is a generated thunk `(set-symbol-function 'NAME (function NAME))` whose quoted `'NAME` goes through this same gate, so it writes the static record's `+32` cell that the reader then hands out — **no migration, no cell-split** (this was the key de-risking insight; the "hard part" turned out mechanical). Because all doors share one gate, there is no cave#131 read/write split: the record the reader hands out is the record every cell write targets. --- ## The cluster, separated into three bugs Measurement (2026-06-30) split the symbol-identity cluster into **three distinct bugs** the ADR had lumped as one class. This file focuses on **bug 3** (the ADR 0026 invariant proper); bugs 1 and 2 are tracked but separable. 1. **`setf` at the REPL** (cave#149 proper) = **door 6 (tree-walker)**. Prelude *tree-walked macro-body* symbols (interned during `ag-cold-boot` at unification=0 → `make-symbol`) never match the runtime reader's *arena* symbols (unification=1). Broken on **all** binaries; **redefining `setf` at the REPL with the identical body fixes it.** Confirmed live by probe (2026-06-30): `(sethash 'a 1 h)` => 1 but `(setf (gethash 'a h) 1)` => undefined function, because `(macroexpand-1 '(setf (gethash a h) 1))` => `(SETF-GETHASH 1 A H)` (the `(eq head 'gethash)` branch misses). **Not fixed by name-set generalization** — the tree-walker is a door that canonicalizes nothing. Rescoped 2026-06-30 from "hold" to **in-scope door-completion** (see *Seven doors* §bug 1): make the tree-walker resolve body-quote symbols through the same canonical lookup as the other doors. This is door-completion, not the rejected `ag-canon-runtime-quoted` *bridge* (which minted a second record). 2. **`typep` per-symbol failures** = **door 7 (image-restore)**. Embedded-image restore duplicates arena records (`build/funcl` only; passes on `build/funcl-stage2`). A live residual of the closed cave#132/#135. **Mandatory for "one instance after save/load"** (Phase 4), not optional: rebuild `*ag-symbol-pool-offsets*` / the canonical tables from the restored `gc-root-ranges` before the reader/compiler resume on a restored image (`src/warm/image.lisp`). Not fixed by name-set generalization. 3. **`find-symbol` static vs reader arena** = **doors 1–5** (the core invariant). `(eq 'cons (find-symbol "CONS" CL))` was NIL everywhere; now T for the eligible builtins (increments 1+2). Generalizing the bitmap completes this axis — but only this axis. --- ## The seven doors — completion model (reframed 2026-06-30) **Reframe (from review):** an earlier draft of this plan was *name-centric* ("which names are portable → routable"). That kills bug 3 but leaves bugs 1 and 2 untouched, because they live in *doors* that no name-set expansion reaches. The correct model is **door-completeness**: enumerate every place a symbol is born/resolved, and make each resolve through one canonical lookup. The portability bitmap is a **safety gate** (which names may be routed without diverging `.fco`), not the coverage model. | # | Door | Resolves to static? | Status | |---|------|---------------------|--------| | 1 | reader (`ag-intern-symbol`) | ✓ (curated set) | done — increments 1+2 | | 2 | compiled quote (`ag-resolve-symbol-patches`) | ✓ (curated set) | done | | 3 | value-cell / defvar (`ag-resolve-global-vaddr-patches`) | ✓ (curated set) | done | | 4 | function-cell install (`set-symbol-function` thunk) | ✓ (curated set) | done — increment 2 | | 5 | `find-symbol` / `intern` | ✓ (all builtins) | already, via `*canonical-symbol-index*` | | 6 | **tree-walker macro expansion** (`ag-macro-apply-tree-walker`) | ✗ | **bug 1 — open** | | 7 | **image-restore reconstruction** | ✗ | **bug 2 — open (Phase 4)** | Generalizing the bitmap (covering all portable names across doors 1–4) is sound engineering and correctly kills bug 3. **It does not touch doors 6 and 7**, which is why bugs 1 and 2 survive any name-set expansion. Each of those doors needs its own step. ### Bug 1 = door 6 (tree-walker) — confirmed live by probe ``` (sethash 'a 1 h) => 1 ; the function is fine (setf (gethash 'a h) 1) => undefined function ; BROKEN, both binaries (macroexpand-1 '(setf (gethash a h) 1)) => (SETF-GETHASH 1 A H) ; should be (SETHASH A 1 H) ``` `setf` is a prelude macro dispatched via the **tree-walker** at the REPL (`ag-macro-should-use-hosted 'setf` => 0). Its body compares `(eq head 'gethash)`; the body's `'gethash` is a `make-symbol` interned during `ag-cold-boot` (unification=0), and the reader's `'gethash` at call time is an arena record (unification=1) — not `eq`, so every place branch misses and setf mangles to the undefined `SETF-`. The tree-walker canonicalizes **neither args nor body** (only the hosted path does, via `ag-canon-hosted-args`, and even that targets the arena record). A REPL-defined macro with identical `(eq (car form) 'foo)` logic works, because its body was interned at unification=1. **This is door-completion, not a bridge.** The earlier "hold for pure Phase 3" decision rejected `ag-canon-runtime-quoted` framed as a *workaround* (post-hoc quote fixup that creates a second record). Making the tree-walker resolve body-quote symbols through the **same canonical lookup** as the other six doors is the pure architecture ("every door → one lookup"), not reconciliation. **Mechanism — probe-settled 2026-06-30 (corrects an earlier "route via find-symbol" suggestion).** `(eq 'gethash (find-symbol "GETHASH" CL))` => NIL and `(eq 'sethash …)` => NIL, while `(eq 'car …)` => T: bug 3 is dead *only* for the curated routed names; for unrouted prelude defuns the reader (arena) and find-symbol (static) still diverge. So routing door 6 through **door 5 (find-symbol)** would NOT kill bug 1 — the body's `'gethash` would become static while the reader's `head` stays arena, so `(eq head 'gethash)` stays NIL. The fix is to route the tree-walker's QUOTE-eval symbol leaves through **door 1 (`ag-intern-symbol`)** — the reader's own canonical lookup — so the body's `'gethash` is whatever the reader would produce *now* (arena, at unification=1), matching the reader's `head`. **This is bitmap-independent** (gethash stays arena, consistently) and is door-completion by the discriminator: `ag-intern-symbol` is the one canonical lookup, not a partial scan. (The rejected `ag-canon-runtime-quoted` bridge differed precisely in using a *partial* arena-only scan, `ag-runtime-pooled-symbol-ptr` — a second source.) **Sequencing (from audit, validated): door 6 → door 7 → bitmap generalization → keyword/defvar doors → Phase 5.** Door 6 is the cheapest, highest-value step (it kills `setf` at the REPL, the most user-visible symptom) and needs no bitmap. ### Bug 2 = door 7 (image-restore) — Phase 4, in scope `*ag-symbol-pool-offsets*` is **never rebuilt on image restore** (grep of `src/warm/image.lisp` and `src/cold/cold-loader.lisp` is empty). After save/load, restored code uses baked record addresses while new forms mint fresh records, so `(eq x 'literal)` on a reloaded image is unreliable (the `typep` per-symbol failures are this — image-only, absent on `build/funcl-stage2`). For a deployment-targeted CL, "one instance of each symbol" must hold after save/load, so this is **mandatory, not optional**. Fix (ADR Phase 4): rebuild the canonical tables from the restored `gc-root-ranges` before the reader/compiler resume on the restored image. ### Generalizing the bitmap (doors 1–4) — the safety gate The curated list in `ag-portable-static-builtin-p` doesn't scale. The clean replacement routes all portable builtins through doors 1–4, with the portability bitmap as the gate. Measured facts: - **Portable set = static-pool record in *both* producers** = the shared cold-prefix + prelude universe. `build/funcl-cold` has 3055 records, `build/funcl` has 3440; the 385 extra are warm-overlay (`AG-IMAGE-*` / `AG-REPL-*`) baked only in `build/funcl`. Routing a warm-only name diverges `.fco` bundles (D2b): the serializer only zeroes refs via `*ag-baked-relocs*` (`src/cold/cold-fco.lisp:159`) and the loader records `BAKED-RELOC-UNRESOLVED` on a miss (`src/cold/cold-loader.lisp:169`). - **`build/funcl-cold`'s pool is NOT a prefix of `build/funcl`'s** — the 385 warm records are *interspersed* early (pool indices ~6–344), not appended. A "index < cold-count" boundary discriminator does **not** work. - **"Portable" ≠ "present in `build/funcl-cold`"** — funcl-cold includes `main-cold.lisp`, build/funcl includes `main.lisp`. The safe definition is "first introduced by the shared `COLD_PREFIX` + prelude." Capture → witness → route (codex-validated): capture portability per record at **intern-time** (`ag-symbol-pool-intern`) — obstacle: the build reads all sources into one concatenated form list (`ag-read-sources-concat`, `src/cold/cold-bootstrap.lisp:468`), losing file phase, so this needs phase-preservation, segmented compile, or a **build-time cross-reference** (flag a `build/funcl` record whose name is also in the `build/funcl-cold` pool). Store the flag in a **side bitmap** keyed by pool index (not `home-pkg` — semantically live; not the header — bits reserved for widetag/length/GC). Witness the flagged sets match across producers, then route. This touches the build's read/emission path — keep the determinism + `.fco` witnesses armed. ## Phase 5 — collapse the router (tracked exit, not aspiration) The ADR's endgame is **one table + one `canonical-find-or-intern`**, with the router and all reconciliation deleted. Every increment that makes the router more complete also makes it more load-bearing — fine, because incrementalism is forced by the fixed-point constraint, **provided the plan is explicitly staged**: complete the router across all 7 doors → prove observational equivalence → collapse to one lookup. Don't let "generalize the router" become the terminal architecture. (This file earlier said "no reconciliation" while expanding the reconciler's eligible set — that tension is real; the router is interim scaffolding, and Phase 5 removes it.) --- ## Witnesses (run these after any change) ``` tests/adr0026-witness.sh # Phase 0 characterization probes (known-failing tracked; flips on progress) tests/adr0026-static-pool-witness.sh # Risk 1: static pool deterministic + dedup'd across funcl-built binaries make test-bootstrap-quiet # full self-host fixed point + .fco byte-equality (~50 min) ``` --- ## Interacting with cave issues (the `cave` CLI) `cave` is on `PATH` (`/home/green/bin/cave`). Set the repo once so you don't repeat `--repo`: ```bash export CAVE_REPO=atgreen/funcl # or pass --repo atgreen/funcl on every command ``` **List / view** ```bash cave issue list --state open # open issues (default); also: closed cave issue list --json # machine-readable cave issue view 149 # body of an issue cave issue view --comments 149 # include comments cave issue view --web 149 # open in browser ``` **Create / comment / change state** ```bash cave issue create --title "Title" --body "Body text" # heredoc: --body "$(cat <<'EOF' ... EOF)" cave issue create --title "Title" --body - # read body from stdin cave issue comment 149 --body "Comment text" # --body-file PATH also works cave issue close 149 cave issue reopen 149 ``` **Conventions used in this project** - File code-level bugs/quirks/deferred fixes as issues in the `atgreen/funcl` cave forge, not in memory (memory is for things issues can't hold). - When an issue's premise is obsolete or wrong, comment with the measurement and the corrected scope; don't silently leave a stale title. - Reference issues from commits and PRs by `cave#NNN`; ADR-0026 work is cross-referenced on cave#149. **Other useful commands** ```bash cave status # current auth config cave repo list # accessible repos ``` (Global flags: `--base-url`, `--token`/`CAVE_TOKEN`, `--repo`/`CAVE_REPO`, `--owner`/`CAVE_OWNER`, `--name`/`CAVE_REPO_NAME`.)