#316 some uiop/package cell callable is widetag-0x08 but its entry does not take trailing caps — blocks the ADR-0032 closure-cell stub

open
Opened by atgreen

Found implementing ADR-0032 Stage B.2. The shared closure-cell stub (call-site CALLs it with RCX = non-raw cell value, RAX = nargs; it verifies widetag 0x08, pops the site return address into R11, pushes the ncaps captures per the ADR-0028 trailing-param convention, re-pushes the return address, tail-jumps the entry) is CORRECT for genuine closures — probe-verified for 0-cap, 1-cap, and 2-user+2-cap shapes, including across forced GC.

But (compile-file "asdf.lisp" ...) dies in UIOP/PACKAGE with: invalid number of arguments: G63 expected 2, got 4 i.e. a cell-resident widetag-0x08 record whose header says ncaps=2 while its entry (G63, an early lifted lambda of the compile session) expects exactly 2 args TOTAL — it does NOT follow the trailing-caps convention. got 4 = user 2 + the stub's 2 pushed "caps".

Open questions:

  1. What constructs this record? Candidates: a non-ag-build-closure-form closure constructor, a mini-clos wrapper, or some record class sharing widetag 0x08 whose slot 1 happens to hold a raw fn ptr. uiop/package is the first system in asdf.lisp, so it reproduces within seconds of compile-file.
  2. How does the DESIGNATOR funcall path treat the same object today? Its inline dispatch does the identical unpack, so either this object never flows through designator funcall, or funcall would break on it too if it did.

Two prior traps burned in the same campaign, recorded here so the next attempt skips them:

Current state (shipped): the stub is emitted at REPL-ready on BOTH .fco witness producers (bytes are name-independent, keeps cave#315's drift unchanged) but is DORMANT — both arms' non-raw paths kept the B.1 name-keyed fallback until this species is identified. Bonus repro of an adjacent quirk: (funcall (function seventh) ...) after (set-symbol-function 'seventh ) resolved the ORIGINAL stdlib seventh, not the cell value — the (function NAME) hybrid reads a different door than the cell; worth folding into the ADR-0031/0032 door audit.

Comments (5)

atgreen3993062882

Codex consult (verified in-repo): there is NO second native widetag-0x08 layout — ADR-0028's [header|entry|caps...] is the only one, and funcall/apply already push trailing caps exactly like the stub, so "special-case the stub" would be wrong even if it worked. Ranked hypotheses: (1) a corrupt/miswired closure record — header claims ncaps=2 but slot-1 entry points at a function compiled for user-args only, i.e. producer/relocation/name-resolution corruption; (2) a (setf symbol-function) path installing an object that violates the callable protocol; (3) [low] the evaluator's cons-shaped (%CLOSURE params body . env) representation (eval.lisp:150) leaking somewhere it shouldn't — not 0x08, should be rejected loudly.

IMPORTANT cross-link: codex rates this plausibly DOWNSTREAM of cave#317 (wrong-cell linkage via twin records can fill cells through the wrong record). Sequence: fix #317, then re-run this diagnostic; only chase a closure-producer bug if G63 persists.

Decisive diagnostic at the failure point: decode the cell — (list :sym name :cell cell :header header :payload (ash header -8) :ncaps (1- (ash header -8)) :entry entry :entry-name :entry-params ). entry-params=2 with ncaps=2 and 2 user args = invalid record; then chase the writer.

Fix design: keep stub semantics identical to funcall for 0x08; do NOT normalize/discard captures in set-symbol-function; before re-enabling the stub broadly, add a debug validation that closure-in-cell entry arity matches user-nargs+ncaps (or variadic) and fail with the decoded species instead of silently falling back.

atgreen3993069001

SHARPENED (post cave#317 fix, clean retest matrix): G63 is REAL and INDEPENDENT of #317 — an earlier "resolved downstream" conclusion was an artifact of a mangled build whose baked arm had silently kept the fallback. Discriminating facts now established:

  • pkgsym arm -> stub routing: GREEN (alexandria loads, spikes 16/0, closure matrix 101/77/101 incl. across GC).
  • baked (CL-homed) arm -> stub routing: reproducibly dies in uiop/package with "G63 expected 2, got 4" during (compile-file "asdf.lisp").
  • So the offending cell value lives in a CL-HOMED/baked name's fn-cell, stored during uiop/package processing.

Promoted hypothesis (was codex's P2/low): the EVALUATOR's cons-shaped closure — eval-when bodies at compile-file run hosted EVAL, and eval.lisp:150's (%CLOSURE params body . env) representation may reach a fn-cell for a CL-homed helper uiop/package redefines (the cave#301 ensure-package family are CL-homed non-ANSI stdlib names — exactly baked-arm territory). A cons is lowtag 001, so the arm's non-raw test routes it to the stub, whose header read on a cons base dereferences the CAR — if that leads to the closure widetag check passing by coincidence, the unpack garbage matches the observed arity error. The stub should REJECT lowtag!=011 explicitly before the widetag read (it tests lowtag first — verify that arm's encoding) and the %CLOSURE species needs either rejection-with-decode or a real funcall route.

Landed state: pkgsym arm fully on B.2 (bind + stub); baked arm's non-raw keeps the B.1 fallback with a comment pointing here. Next session: the decode diagnostic at the failure point (fields per the earlier comment), starting with (compile-file "asdf.lisp") + a bind-style print of the cell's lowtag/header before the stub call.

atgreen3993071505

SESSION FINDINGS (localized, not yet fixed — handing off with a precise repro recipe):

  1. SPECIES: genuine ADR-0028 closure record (lowtag 011 + widetag 0x08). PROVEN by the failure being an ARITY raise ("expected 2, got 4"), not not-a-function — the stub's lowtag and widetag gates both pass. This REFUTES the cons-%CLOSURE hypothesis (a cons is lowtag 001 → would raise not-a-function).

  2. SHAPE: header claims ncaps=2 (stub pushes 2 caps → got 4), entry is a gensym G63 that expects 2 args TOTAL. So either the entry pointer is wrong (points at a 2-arg function that isn't this closure's 4-arg lifted body — cave#309 G113 gensym-collision class) or the header ncaps is inconsistent with the entry (producer bug).

  3. LOCATION: prefix-bisected to recycle-symbol (asdf.lisp:623-643) — its dolist + multiple-value-bind body captures recycled and foundp (exactly 2 captures). The offending closure is reached through SOME baked/CL-homed NAME's fn-cell (the baked arm fired for it); the stub raise names the ENTRY (G63), not that cell's name — identifying the cell name is the open question.

  4. WHY THE FALLBACK IS ROBUST: the B.1 name-keyed rel32 fallback on the baked arm's non-raw path IGNORES the cell's closure and calls the name's real fn-table entry directly — so it never touches the bad closure. That is why the fallback build compiles asdf CLEAN and the stub build dies. Corollary: the bug is a MIS-INSTALLED CELL (a baked name whose cell wrongly holds this closure), latent until the stub actually reads the cell.

REPRO RECIPE for the decode (must use a STUB build — the fallback build does NOT reproduce): in src/cold/compiler.lisp the baked arm's fb-nonraw path, swap the name-keyed fallback for the ag-cell-closure-stub-offset cond (both arms' pkgsym version shows the shape); rebuild; then gdb break on MAKE-PROGRAM-ERROR (stdlib.lisp:2727 is the SOLE source of "invalid number of arguments" — confirmed by grep) with run < compile-form.lisp, bt to see whether the caller is a JIT prologue (runtime, real bad closure) or a compiler frame (emit-time miscount). Breakpoints on the baked FUNCL-RAISE-ARITY-ERROR / MAKE-PROGRAM-ERROR did NOT hit on the fallback build simply because it doesn't reproduce — untested on the stub build.

Landed state unchanged: baked arm on B.1 fallback (green); pkgsym arm fully on stub+bind (green). Only the baked arm's non-raw→stub routing is withheld.

atgreen3993072057

CODEX CONSULT (evidence-based, verified against the repo):

TOP HYPOTHESIS (codex, and I confirmed the mechanism): WRONG ENTRY POINTER via G gensym-name collision — the cave#309 G113 class resurfacing at compile-file time, NOT a closure producer miscount and NOT "baked cells must never hold closures" (codex corrected that framing: set-symbol-function legitimately stores closures; ADR-0028 closures are functionp, and asdf uses (setf fdefinition)).

Verified collision surface:

  • ag-fn-add (src/cold/stage1.lisp:804) has NO uniqueness guard: blind (cons (name.offset) table), and ag-fn-lookup returns NEWEST match.
  • ag-macro-gensym (compiler.lisp:4547) mints G from the global ag-gensym-counter.
  • ag-gensym-counter is RESET TO 0 in the REPL reset paths (repl.lisp:2156, 2413) — so a compile-file session can re-mint a G63 that shadows an earlier same-named lifted lambda.
  • (function G63) is resolved BY NAME through the fn-value patch / ag-fn-lookup path → newest-wins → a closure allocated at recycle-symbol's 2-capture site (header ncaps=2, correct) gets its entry slot bound to a DIFFERENT, 2-arg G63. Exactly the observed "header 2 caps + entry expects 2, got 4".
  • ag-gensym-counter-absorb exists but only bumps where explicitly called; it doesn't prevent an in-session reset+remint collision.

CODEX'S DECISIVE DIAGNOSTIC (better than my gdb-on-make-program-error plan — that confirms the CALLEE, not the owning cell; the stub tail-jumps so the owner is unrecoverable from the arity frame): on a STUB build, in the baked non-raw branch BEFORE the stub call, call a temp debug helper (reloading RCX/RAX after) that prints: (:owner OWNER :site-nargs N :cell CELL :header HEADER :ncaps NCAPS :entry ENTRY :entry-offset OFFSET :entry-name ENTRY-NAME :entry-params-exact PARAMS :same-name-fn-table-entries DUPES) CRITICAL: entry-params-exact must be keyed by ENTRY OFFSET, not ag-fn-params-lookup (that is ALSO name-keyed and will lie under duplicate G63). Add a temp (offset . (name . params)) table at emit-ir-defun right after ag-fn-add. One run answers everything: params=2 → wrong G63 (collision confirmed); params=4 but raise says 2 → entry-decode/prologue-metadata bug; owner unrelated → chase the cell write.

FIX DESIGN (codex): if collision confirmed, fix GENSYM LINKAGE not the stub — make lifted G names globally collision-proof before they enter ag-fn-table / fn-value patches (ag-macro-gensym skips any G already present in ag-fn-table + restored + pending-loaded names), and make ag-fn-add a HARD ERROR on a duplicate G with different params. Keep a stub-side entry-arity==site-nargs+ncaps check only as a diagnostic guard, never as the semantic fix (it would just convert a wrong binding into a silent fallback and preserve the corruption).

This also retro-explains why the pkgsym arm's stub route is GREEN: pkgsym names aren't gensyms, so no newest-wins collision on their entries.

atgreen3993074457

DIAGNOSTIC RESULTS (ran codex's decode; both leading hypotheses REFUTED):

  1. NO gensym fn-table collision (codex hypothesis #1, REFUTED). Instrumented ag-fn-add to warn on any duplicate G added while a same-named entry exists. ZERO GENSYM-DUP across the entire asdf compile. So (function G63) is NOT resolving to a newest-wins wrong entry — G63 is a unique fn-table entry.

  2. NOT reached through either cell-first stub arm (REFUTED my own "baked cell" framing). Added an offset-independent cell decoder (ag-cell-diag316: owner name + closure ncaps + entry-raw) fired from BOTH arms' fb-nonraw path before the stub. Emission verified working (a probe closure installed on EIGHTH printed "CELL316 owner=EIGHTH lowtag=3 CLOSURE ncaps=1"). During (compile-file "asdf.lisp"): ZERO CELL316 lines from either arm, yet G63 still crashes. => NO cell-first call site encounters a non-raw cell during the asdf compile. The failing call reaches G63 via the ORDINARY FUNCALL path (emit-ir-funcall's closure arm, runtime.lisp ~1290), which pushes captures exactly like the stub — untouched by ADR-0032.

  3. Reinterpreted arithmetic: G63 expects 2 args TOTAL, header ncaps=2 => the lifted body has 0 user params. A 2-capture closure over a 0-user-arg body, funcall'd with 2 USER args => 2 user + 2 caps = 4 = "got 4". So the call SITE passes 2 user args to a function whose real user-arity is 0. Candidate: recycle-symbol's dolist RESULT form (values recycled foundp) — a 0-arg body capturing recycled+foundp — being called where a 2-arg function is expected.

  4. POSITION-SENSITIVE. Shipped 22b6efa (baked arm = name-keyed FALLBACK, pkgsym arm = stub) compiles asdf CLEAN (make asdf-fasl produced the 5.85MB fasl; alexandria loads). Routing the baked arm to the stub — even though its fb-nonraw is NEVER executed during the compile (zero CELL316) — introduces G63. The only effect is EMISSION SHAPE: the baked arm emits the stub-cond instead of the fallback, shifting downstream offsets/gensym timing. So G63 is a LATENT closure-arity miscompile MASKED by the shipped layout and EXPOSED by the shifted one — the cave#309 position/gensym-coincidence class, not an ADR-0032 stub bug.

CONCLUSION: cave#316 is NOT a blocker for the shipped state (baked arm on fallback is correct and green). Activating the baked arm's stub is safe to defer; the real bug is a pre-existing latent closure miscompile. NEXT STEP: instrument emit-ir-funcall's closure arm (not the stub) to decode+catch the G63 closure at its funcall site on a build that reproduces (baked->stub layout), OR find the site that funcalls a 0-user-arg (values recycled foundp) closure with 2 args. Reverted all diagnostics; tree back at ca46784 (green).