#261 Hosted macro-expander compiles EARLY-BIND general names (ensure-package class); the %SETF-SYMBOL bail accidentally shields it — expander calls need late binding

2026-07-09 progress — one brick landed, mechanism cleared, blocker sharpened.

LANDED (26b5b2f): (function COLD-STATIC) now resolves via the runtime-fn-addr cross-region fallback (was a wild rip+0 pointer). Verified: (funcall (function ag-sym-concat-2) "AA" "BB") => "AABB". Also fixes the 3 UNRESOLVED-FN-VALUE wild-jump sites from cave#259. Suite 8/8; load-system fixtures green. This is prerequisite infrastructure for late-binding (funcall (function NAME)) to work when NAME is a cold static.

MECHANISM CLEARED: funcl's notinline late-binding WORKS across an arity-changing redefinition — a caller compiled against a 1-arg (declaim notinline)'d fn correctly reaches the later (name &rest keys) redefinition ((MANY-ARG A (B C))). So late-binding itself is not the gap.

SHARPENED DIAGNOSIS of the two experimental failures:

• types.lisp bail is caused SOLELY by %SETF-SYMBOL (setf/push in the macrolet body) — an INTERNAL STABLE name resolvable via runtime-fn-addr but not ag-fn-lookup, so the cave#161 bail predicate counts it unresolved and sends the whole expander (incl. its format calls) to the closed tree-walker.
• The %/AG- whitelist / broadened-bail attempts made alexandria WORK (flatten correct!) but regressed asdf with 'ENSURE-PACKAGE expected 1, got 25'. Root: stdlib's 1-arg ensure-package (stdlib.lisp:7862) is superseded by uiop's &key version (asdf.lisp:725). Since notinline late-binding demonstrably works, the regression is a declaim-TIMING interaction: un-bailing uiop's define-package expander changes compile ORDER so a consumer of ensure-package early-binds before the notinline declaim is processed.

CANDIDATE TARGETED FIX (untested, needs a build+both-suites cycle): register ONLY the handful of stable tree-walker-primitive helpers (%SETF-SYMBOL, AG-SYM-CONCAT-2/3, AG-MAKE-SETF-SYMBOL) into the RUNTIME ag-fn-table so ag-fn-lookup finds them — un-bails types.lisp WITHOUT the broad whitelist that pulled in uiop's define-package expander. Risk: may still un-bail some asdf expander; must verify asdf.lisp load + load-system fixtures stay green before landing. The declaim-timing bug is the deeper item and should be understood independently.

DECISIVE NEGATIVE RESULT (2026-07-09) — the bail-predicate approach is dead; the fix must be hosted-path late-binding.

Tested the narrowest possible bail-narrowing: an EXACT four-name allowlist (%SETF-SYMBOL, AG-MAKE-SETF-SYMBOL, AG-SYM-CONCAT-2/3) exempted from ag-count-unresolved-patches / ag-co-has-unresolved-patches-p — far narrower than the %/AG- prefix whitelist that failed before. Results:

• Fresh alexandria: un-bailed types.lisp (good — no more tree-walker format garbage) but then 'call to undefined function: %SETF-SYMBOL' at runtime, because %SETF-SYMBOL is ONLY a tree-walker primitive (eval.lisp dispatch) with NO compiled function — the hosted call has no target. (Fixable by adding (defun %setf-symbol (n) (ag-make-setf-symbol n)), which my earlier attempt showed makes alexandria's flatten work.)
• asdf.lisp full load: REGRESSED with 'ENSURE-PACKAGE expected 1, got 25' + 'G44 expected 2, got 4' — the SAME cascade as the broad whitelist, from just the four-name exemption. Some asdf expander's sole unresolved name is one of the four, so exempting it un-bails that expander and exposes consumer-side early-binding of ensure-package (stdlib 1-arg superseded by uiop &key @asdf.lisp:725).

CONCLUSION: the bail-to-tree-walker is LOAD-BEARING accidental late-binding. The tree-walker resolves calls by name at eval time (inherently late-bound); the hosted compile early-binds. ANY un-bailing exposes the early-binding of redefined library functions. So no bail-predicate change can be safe.

THE ONLY CORRECT FIX (scoped ir-lower-call campaign): make hosted-compiled expander/consumer code LATE-BIND calls to redefinable (non-primitive) names — the same late-binding the notinline path already does (verified working across arity changes) — applied to the hosted-compile path generally, not gated on an explicit declaim. Prereqs in place: brick A (26b5b2f) makes (function COLD-STATIC) resolve, and (defun %setf-symbol ...) closes the primitive gap. The open design question is the discriminator: late-bind library/user names but keep true primitives (car/cons/+, which have no (function ...) value) inline. That is next session's work.

Tree reverted to brick-A HEAD; asdf load-system baseline intact.

ir-lower-call late-binding campaign (2026-07-09) — MAJOR VALIDATION + a precise remaining spec.

Implemented the principled fix: a flag (ag-expander-late-bind-p, default 0) set only by ag-compile-lambda-impl around a macro-expander body compile; ir-lower-call then late-binds every general named call via (funcall (function NAME) . args) — the proven notinline rewrite — with a narrow exclusion for non-funcallable heads (%-prefix internals + FUNCTION). Self-host held (stage2 self-compiled fine, build byte-consistent).

RESULT — the historical blocker is GONE: the 'ENSURE-PACKAGE expected 1, got 25' arity error that killed every prior bail-narrowing attempt DID NOT OCCUR. Late-binding correctly resolves the uiop 25-arg ensure-package that supersedes the stdlib 1-arg one at call time. This validates the whole hypothesis: the fast-path early-binding of redefined library functions IS the root cause, and late-binding is the fix.

NEW failure (well-understood, not the old class): 'attempt to call an object that is not a function | in: WITH-UPGRADABILITY' + an alexandria SIGSEGV. Cause: the BLANKET late-bind is too aggressive — (function NAME) is only valid when NAME is a defined FUNCTION at expander-compile time. For names that are forward-referenced, undefined, or macro-names, (function NAME) yields a wild/NIL value and the funcall crashes. (Normal early-binding handles forward refs via end-of-compile patches; late-binding resolves (function NAME) mid-compile.)

PRECISE SPEC FOR NEXT ATTEMPT: gate the late-bind on 'NAME resolves to a function RIGHT NOW' (ag-fn-lookup(name) != -1 OR ag-runtime-fn-addr-lookup(name) != -1). Resolvable names (ensure-package, format) late-bind → correct; unresolvable names fall through to the normal early-bind/forward-patch path → no new breakage. ARCHITECTURE CONSTRAINT: this guard needs ag-fn-lookup, which is NOT in the ast-ir smoke-test subset (reader+buffer+asm+elf-tiny+ast-ir) — so it CANNOT live in ir-lower-call. Move the late-binding decision to emit-ir-fn-call instead (compiler.lisp, has ag-fn-lookup, and is the exact discrimination point — only genuine general calls reach it, primitives are dispatched earlier by emit-ir-call). Emit the (function NAME)+indirect-call sequence there when the flag is set AND the name resolves. Note: this still won't fix types.lisp's %SETF-SYMBOL (unresolvable, no defun) — that separately needs (defun %setf-symbol (n) (ag-make-setf-symbol n)) so it becomes resolvable. Two independent pieces; both needed for alexandria.

Reverted to brick-A HEAD (baseline green).

CAMPAIGN CONCLUSION (2026-07-09) — the true final blocker is identified and it is DEEP.

Implemented the full principled fix WITH the resolvable-now guard: flag ag-expander-late-bind-p set by ag-compile-lambda-impl; ir-lower-call late-binds a general call only when (funcall (function NAME)) is valid — NAME funcallable (not %-prefix / FUNCTION) AND defined right now (ag-fn-lookup OR ag-runtime-fn-addr resolves). ast-ir self-host test PASSED (the ag-fn-lookup forward-ref is a non-issue — dead branch under flag=0). Self-host held.

CONFIRMED WORKING: late-binding eliminates the ensure-package arity error — asdf ran clean under late-binding alone. The redefinition class is genuinely fixed by this.

BUT the alexandria fix is BLOCKED by a coupling that cannot be broken without a deeper fix:

• types.lisp bails because %SETF-SYMBOL is unresolved (tree-walker primitive, no compiled fn). Making it resolvable — EITHER by a warm stdlib defun (so ag-fn-lookup finds it) OR by broadening the bail predicate to consult ag-runtime-fn-addr — un-bails types.lisp.
• BUT %SETF-SYMBOL is ALSO the sole blocking unresolved name in uiop's define-package expander. Resolving it un-bails define-package TOO. Its EXPANDER then runs hosted and emits CONSUMER code calling (ensure-package …25 args…). That consumer code is compiled via the NORMAL path (not ag-compile-lambda-impl), so the expander late-bind flag is OFF → it EARLY-BINDS ensure-package to the stdlib 1-arg version (asdf.lisp:725's 25-arg &key version isn't current yet) → 'expected 1, got 25' (32-39x).
• Verified BOTH un-bail methods (warm defun AND bail-broadening) reintroduce the ensure-package regression identically. The expander-body late-binding does NOT cover the code an expander GENERATES.

THE TRUE FINAL BLOCKER: consumer-side early-binding of redefined library functions (declaim-timing). asdf declaims ensure-package notinline (with-upgradability) precisely to force late binding, but the define-package consumer compiles BEFORE that declaim is processed. The bail-to-tree-walker accidentally works because the tree-walker resolves the consumer's calls by name at eval time (late). To un-bail define-package safely, the consumer code must late-bind ensure-package — which needs either the notinline declaim to be in effect at consumer-compile time (source/timing fix in asdf.lisp) or general consumer-side late-binding of redefinable names.

LANDABLE from this campaign: the late-binding infrastructure itself is correct and safe (asdf clean, self-host held) but provides no user-visible unblock alone, so NOT landed speculatively. Reverted to brick-A HEAD (26b5b2f). NEXT: tackle the declaim-timing consumer-early-binding directly — e.g. make asdf's ensure-package notinline declaim effective before its first define-package consumer, or a consumer-side late-bind for names known-redefined. Also: alexandria has a SECOND independent gap, SETF-VALUES (multiple-value setf, types.lisp:50) — 9 bails, needs (setf (values …)) support regardless.

CODEX CONSULT (2026-07-09) — sharp guidance, changes the plan. Key correction to my approach:

CODEX'S CORE INSIGHT: the fix is NOT to make %SETF-SYMBOL selectively resolvable (that's "a diagnostic crutch, not a compiler design" — types.lisp and uiop define-package share the same blocker, can't un-bail one without the other). The real bug is that COMPILED MACRO OUTPUT is allowed to early-bind a function ASDF intends to be upgradeable. My expander-body late-binding covered the expander BODY but not the code the expander GENERATES — and that generated consumer code (the 25-arg ensure-package call) is where the early-binding happens.

THE FIX (a compilation POLICY that propagates expander→output):

1. Keep expander-body late-binding.
2. Make %SETF-SYMBOL resolvable normally (warm defun).
3. Add a narrow compiler policy: when compiling the RESULT of a macro expander that was itself compiled under hosted late-binding, compile that expansion result under a "late-bind notinline names" policy. Implement via a DYNAMIC compiler context var (NOT cons metadata — "smuggling metadata through cons identity will become brittle immediately"): (let ((compiling-macro-expansion t)) (compile-form expanded-form)) and ir-lower-call checks it.
4. Scope it initially to names already declaimed notinline OR a tiny bootstrap override set { ensure-package }. asdf declaims ensure-package notinline via with-upgradability, so this is principled — it makes the consumer call honor the notinline the source already asked for.
5. Broaden from ensure-package → "all notinline-declared names in the current compiler env" once verified.

DO NOT hoist the notinline declaim earlier in asdf source — "source-order cheating, not a principled compiler fix." A bootstrap-notinline-overrides seed table is an acceptable TEMPORARY hack "but call it what it is."

LONG-TERM (codex agrees funcl's default is inverted vs CL): global named calls should late-bind through the symbol fn-cell by default; early direct calls are the OPTIMIZATION (local/sealed/same-unit/inline). Incremental path: keep early-bind for cold/compiler-internal; add per-compilation policy :early-default vs :late-default; use :late-default for loaded library + macro-generated code; recover perf with direct-call opt for provably-safe cases; eventually make notinline "boring instead of load-bearing."

ORDERING codex recommends: (1) %SETF-SYMBOL resolvable, (2) narrow late-bind policy for macro-expanded consumer code protecting ensure-package, (3) verify asdf 0 errors, (4) broaden to all-notinline, (5) later consider late-by-default for non-core loaded code.

NEXT SESSION: implement the compiling-macro-expansion policy var + have ag-compile-lambda-impl bind it while the expander's OUTPUT is compiled (not just the expander body), and ir-lower-call late-bind notinline names under it. This is the missing coverage — the generated consumer code — that expander-body late-binding didn't reach. Codex session 019f4618-ad27-7422-a336-171002d371a3 saved for follow-up.

CRITICAL REFRAME (2026-07-09, baseline probe) — two of my theories are now DISPROVEN; the ensure-package regression mechanism is genuinely NOT yet understood. Recording facts vs theories to stop the thrash.

ESTABLISHED FACTS (verified, not theories):

1. CL is late-by-default: ALL production CLs (SBCL/CMUCL/CCL/ECL/CLASP/CLISP) late-bind global calls through an fdefn/symbol-fcell indirection; direct call is a proven-safe optimization (inline / same-compilation-unit / standard-fn). funcl is the lone outlier (early-by-default). [Explore survey, file:line evidence — solid, worth acting on regardless.]
2. asdf's define-package expands to (funcall 'ensure-package ,@args) (asdf.lisp:937) — funcall of a SYMBOL, which is funcl's ALREADY-late-bound idiom (verified: (funcall 'f) sees a redefinition).
3. On the BASELINE binary after loading asdf: NO symbol split for ensure-package. (%fn-ptr-as-fixnum 'asdf/package::ensure-package) == (%fn-ptr-as-fixnum (find-symbol "ENSURE-PACKAGE" "ASDF/PACKAGE")) == 7424083 (SAME record). Both (funcall 'ensure-package "X" :use '("CL")) AND (funcall #'ensure-package ...) return cleanly with 25 args — NO 'expected 1, got 25'. So the 25-arg ensure-package IS correctly installed and callable via the symbol on baseline.

DISPROVEN THEORIES (do NOT pursue):

• GC-corruption (cave#260): fresh control failed identically → not GC.
• Symbol-split of ensure-package (cave#257 for THIS symbol): same record, funcall works → not a split.
• Expander-BODY late-binding as the fix: made asdf run 1600 lines clean but didn't cover the consumer; the consumer is (funcall 'symbol) which is ALREADY late → so expander-body late-binding wasn't even the relevant lever for ensure-package.

STILL UNEXPLAINED: the 'ENSURE-PACKAGE expected 1, got 25' (32-39x) appears ONLY in the un-bailed config (late-binding + %setf-symbol resolvable), with 'in: LET' / 'in: WITH-UPGRADABILITY' context. Since define-package's own call is (funcall 'ensure-package) [late, works on baseline], the failing call must be a DIFFERENT, DIRECT (ensure-package ...) call site that un-bailing exposes — location UNIDENTIFIED. Possibly a hosted-compiled expander that generates a direct call, or parse-define-package-form internals, or a with-upgradability-generated site.

DISCIPLINED NEXT STEP (instrumentation-first, NO more theories): rebuild the un-bailed config, use the P7 named traps + gdb break on the arity-error raise to capture the EXACT function + call site that early-binds a 1-arg ensure-package. Do NOT implement any fix until that call site is identified. The late-by-default inversion (stage 1-3, reference memory) remains the correct long-term direction independent of this specific bug, but is a separate architecture effort (should be an ADR).

Tree at brick-A baseline 26b5b2f (clean). codex session 019f4618 available.

ROOT-CAUSE REFRAME (2026-07-09, instrumentation-driven) — the cave#261 blocker is NOT late-binding. It is a &key ARITY MISCOMPILE.

gdb on the arity raise failed MECHANICALLY (funcl relocates calls away from the static ELF symbols, so breakpoints on MAKE-PROGRAM-ERROR / FUNCL-RAISE-ARITY-ERROR never fire despite 39 hits). But source analysis + REPL isolation cracked it:

FACTS ESTABLISHED:

1. asdf has only 3 ensure-package sites; the ONLY call is (funcall 'ensure-package ,@args) at asdf.lisp:937 — the already-late idiom. There is NO direct (ensure-package ...) call anywhere. So 'expected 1, got 25' is about the CALLEE, not a call site.
2. ensure-package is (defun ensure-package (name &key <13 keys>) ...). On the un-bailed binary AFTER load, (funcall #'asdf/package::ensure-package "T" :use '("CL")) => 'ENSURE-PACKAGE expected 1, got 3'. So the ensure-package FUNCTION compiled as a STRICT 1-ARG function — the 13 &key params were DROPPED.
3. The identical construct works everywhere else: a 13-key defun at the bare REPL on the un-bailed binary => correct (1 2 NIL 4 ... 13); (with-upgradability () (defun mykf (name &key a b) ...)) at the REPL after load => correct (1 2 3); plain &key defun => correct. And ensure-package works on the BASELINE (bailed) binary (my earlier probe: funcall with :use returned (K1TEST)).
4. So the miscompile is SPECIFIC to: ensure-package compiled DURING the un-bailed load (at line 725), where the macro/expander state is partial. NOT a many-keys bug, NOT with-upgradability structure (ensure-function-notinline splices the defun verbatim), NOT global &key corruption.

DISPROVEN this session: GC-corruption (cave#260), ensure-package symbol-split (same record, funcall works), late-binding-coverage (the consumer is already (funcall 'sym), late).

MECHANISM (hypothesis, needs 1 more instrumented run): un-bailing (%setf-symbol resolvable) flips whether with-upgradability / the &key-rewriter's helpers compile HOSTED vs TREE-WALK at line 725 during the load. One of those paths mangles ensure-package's &key lambda-list (drops the keys), producing a 1-arg function. NEXT: instrument the &key lambda-list reconstruction (compiler.lisp &key rewriter / emit-ir-defun) + log hosted-vs-treewalk for the defun at line 725 under the un-bailed load; find where the 13 keys are dropped.

IMPLICATION: the entire late-binding sub-campaign was the wrong layer. Late-by-default remains correct long-term CL architecture (reference_late_binding_cl_impls, separate ADR) but is NOT what unblocks Alexandria. The unblock is fixing this &key miscompile. Tree restored to clean baseline 26b5b2f.

&key MISCOMPILE — deep localization (2026-07-09, ~10 instrumented build cycles). Not yet root-caused, but tightly bounded and reproducible-in-context.

CONFIRMED: ensure-package (name &key <13 keys>) compiles as STRICT 1-ARG during the un-bailed asdf load — its &key + 13 keys are DROPPED to just (name). Verified via 4 compiler instrumentation points, ALL of which fired for the control ensure-pathname (n=29 havekey=1, works) but NONE for ensure-package:

• ag-macro-expand-call DEFUN dispatch (name-matched, unconditional): NO fire.
• ag-macro-expand-binding-form (param-count >=10): NO fire.
• emit-ir-defun (param-count >=10): NO fire.
• ag-rewrite-key-defun (name-matched): NO fire. So ensure-package's defun reaches every compiler defun-processing path already truncated to (name) (1 param, below the >=10 thresholds), i.e. the drop is UPSTREAM of all defun processing — in the loader/macroexpand splice path during the un-bailed load.

RULED OUT (all reproduce CORRECTLY in isolation on the un-bailed binary, no drop):

• eval-when + &key defun (small)
• eval-when + multiple sibling defuns + &key
• LARGE eval-when (240 filler defuns) + 13-key defun
• with-upgradability + &key defun
• plain 13-key defun at REPL
• many keys (13) at REPL So the trigger is NOT eval-when per se, NOT large body, NOT many keys, NOT with-upgradability — it needs the SPECIFIC asdf-load macro/expander STATE around line 725 that no isolation reproduces.

ALSO RULED OUT this session: late-binding (the consumer is (funcall 'ensure-package), already late), ensure-package symbol-split (same record, funcall works on baseline), GC-corruption (cave#260).

STRUCTURAL CLUE: ensure-package is inside a big (eval-when (:load-toplevel :compile-toplevel :execute) ...) at asdf.lisp:470; ensure-pathname (works) is inside (with-upgradability ()). gdb-on-static-symbol is useless (funcl relocates calls away from ELF symbols).

PRECISE NEXT STEP (one uninstrumented seam): instrument the REPL LOADER path — src/warm/repl.lisp ag-repl-eval-form / ag-repl-eval-form-dispatch defun branch / ag-repl-process-form — to dump ensure-package's params AS the loader receives each top-level form during the un-bailed load. That path (not the file-compile ag-macro-expand-call pass I instrumented) is what actually processes the eval-when's progn body forms during a runtime load. The drop is between the eval-when expansion and the defun's individual processing there.

Baseline restored clean 26b5b2f. Diagnostic %setf-symbol + all compiler instrumentation reverted.

RESOLVED — NOT A BUG. Instrumented the exact emit sites (ag-rewrite-key-defun, emit-fixed-arity-check, emit-min-arity-check) on a CLEAN baseline (26b5b2f + emit probes only) and loaded asdf.lisp. Result:

• RKD ensure-package have-key=1 nparams=15 → &key IS detected and rewritten
• EMAC ensure-package min-expected=1 → compiles as a variadic (min-arity) function, the CORRECT path
• EFAC=0 → NO fixed-arity-1 check emitted
• arity errors during clean asdf load = 0

ensure-package was never broken. The 'expected 1, got 25' symptom only appeared in earlier sessions WITH the un-bail diagnostic (%setf-symbol in stdlib + define-package expander un-bail) applied — i.e. it was an ARTIFACT of the instrumentation, not a real defect. Closing.

Real remaining blocker for the north-star (load alexandria via asdf): asdf:load-system reaches completion but raises MISSING-COMPONENT — find-system cannot locate alexandria.asd via central-registry directory scan. Tracking that separately.