#311 warm reloc pipeline: incremental patch drain breaks redefinition healing (full re-walk is load-bearing)

open

Opened by atgreen

Finding from the 2026-07-13 fasl-boot perf campaign (session killed mid-way; follow-up session verified and reverted the drain).

Attempted change: replace the per-form ag-patches-apply + ag-fn-value-patches-apply full walks with an incremental drain (ag-warm-relocs-drain): NIL the globals each drain, apply only fresh entries, stash double-miss forward references on pending ledgers healed via cheap ag-fn-lookup probes. Motivation: the full lists only ever grow, so the warm pipeline was O(n^2) over a session — an asdf-sized fasl replay burned ~23s in these walks.

Why it is WRONG: the full re-walk is not just waste — it is the warm pipeline's redefinition-healing mechanism. Warm direct calls are EARLY-BOUND rel32s; when a name is re-registered (new CO, same name), the next form's full walk re-resolves every historical call site to the new target. mini-clos leans on this constantly: each defmethod rebuilds its generic's dispatcher under the same name, so under the drain, call sites emitted before a defmethod keep calling the stale dispatcher.

Observed failure: (asdf:load-system "alexandria") fails with check-type failed: #<UNDEFINED-SYSTEM> is not of type (OR NULL COMPONENT) (component-find-path, via print-object) — typep/CLOS dispatch silently wrong after asdf's load-time defmethods. Reproduces on BOTH the source-load and fasl-replay paths. Bisect: committed HEAD (cbb941f) passes; drain-only tree fails; mint-index-only unverified-suspect-cleared (drain-only build had mint cluster reverted and still failed).

Fix direction for a correct incremental drain: keep the pending ledgers for forward refs, but add a name→sites reverse index (cave#188-style buckets); on re-registration of an existing name, re-patch exactly that name's historical sites. Cost O(sites-of-redefined-names) per drain instead of O(all-sites). Alternatively this is one more argument for the staged late-binding inversion (fdefn-style cells for warm calls) — see the late-binding notes: all prod CLs late-bind global calls, and this bug class disappears there.

Note for the eventual implementation: the sharded ert suite does NOT catch this (all 8 shards green with the broken drain) — an asdf/alexandria load-system probe, or any defmethod-after-call-site test, is the regression gate to add.

Comments (3)

atgreen3992990189

The reverted drain implementation, for reference when implementing the reverse-index fix (this exact code is what broke redefinition healing — the heal/stash logic is reusable, it just needs the name→sites re-patch added on re-registration):

(defvar *ag-warm-pending-calls* nil)
(defvar *ag-warm-pending-fn-values* nil)

(defun ag-warm-pending-calls-heal (buf entries)
  (cond
    ((null entries) nil)
    (t (let ((p (car entries)))
         (let ((target (ag-fn-lookup (cdr p))))
           (cond
             ((= target -1)
              (setq *ag-warm-pending-calls*
                    (cons p *ag-warm-pending-calls*)))
             (t (patch-rel32 buf (car p) target)))))
       (ag-warm-pending-calls-heal buf (cdr entries)))))

;; (ag-warm-pending-fn-values-heal identical shape, over *ag-warm-pending-fn-values*)

(defun ag-warm-calls-stash-misses (patches)
  (cond
    ((null patches) nil)
    (t (let ((p (car patches)))
         (cond
           ((and (= (ag-fn-lookup (cdr p)) -1)
                 (= (ag-runtime-fn-addr-lookup (cdr p)) -1))
            (setq *ag-warm-pending-calls*
                  (cons p *ag-warm-pending-calls*)))
           (t nil)))
       (ag-warm-calls-stash-misses (cdr patches)))))

(defun ag-warm-relocs-drain (buf)
  (let ((pend *ag-warm-pending-calls*))
    (setq *ag-warm-pending-calls* nil)
    (ag-warm-pending-calls-heal buf pend))
  (let ((pend *ag-warm-pending-fn-values*))
    (setq *ag-warm-pending-fn-values* nil)
    (ag-warm-pending-fn-values-heal buf pend))
  (let ((pending *ag-patches*))
    (setq *ag-patches* nil)
    (ag-patches-apply-walk buf pending)
    (ag-warm-calls-stash-misses pending))
  (let ((pending *ag-fn-value-patches*))
    (setq *ag-fn-value-patches* nil)
    (ag-fn-value-patches-apply-walk buf pending)
    (ag-warm-fn-values-stash-misses pending)))

It replaced every (ag-patches-apply code)(ag-fn-value-patches-apply code) pair in src/warm/repl.lisp (9 sites) + src/warm/image.lisp (1 site), and the two REPL reset paths cleared the pending ledgers. Missing piece: on ag-repl-emit-and-register (or wherever a name is RE-registered into ag-fn-table), collect the re-registered names and re-patch their historical sites from a name→sites index built as patches are first applied. Measured stakes: boot-time asdf fasl replay is ~18s with the full walks, was ~5s with this drain.

Also note: the fn-value heal above patches rel32 via ag-fn-lookup only — the cross-region (rt-addr) vaddr-space fixup from ag-fn-value-patch-resolve's fallback branch needs to be mirrored if pending fn-values can resolve to cold statics.

atgreen3993010751

perf evidence (2026-07-14, perf record -F 400 --call-graph dwarf on a build/funcl boot with matching build/asdf.funclfa, ~14s wall, 5.2K cpu_core samples; JIT frames symbolized via (save-perf-map)):

~66% — the per-CO full reloc re-walks (this issue):

AG-FN-IDX-HASH-WALK 27.2% (ag-fn-lookup's hash-index bucket walker — cheap per call, called ~n² times)
name compares under lookup (AG-BV-EQUAL-LOOP 11.3 / AG-BV-EQUAL 4.1 / AG-SYM-EQ-NAME 2.2 / AG-NORM-TO-NAME 3.6) ≈ 21%
per-package walk + pk discriminator (AG-FN-LOOKUP-WALK-PK/-MTF, AG-FN-PK, AG-FN-IDX-NORM-MISCP) ≈ 6%
patch writes (PATCH-U32-AT 3.9, PATCH-REL32) ≈ 4%; fn-value resolve/walks ≈ 2%; rt-addr fallback walk 1.5%; pool-find ≈ 2.4%

~27% — runtime package/intern machinery in JIT code (replay's tag-6 pkgsym reloc resolution through the accessibility door + asdf's replayed define-package CO-RUNs):

PKG-EXPORTS-ENTRY 6.9%, FIND-PACKAGE-WALK 5.4%, BV-EQUAL(+LOOP) 9.4%, STRING-CMP2/STRING=/PKG-IN-LIST/%KEY-ARG/%STRING-DESIGNATOR/PKG-NAME-IN-LIST ≈ 5%
These are flat linear scans (package list, export lists) — a cave#188 fence-index candidate INDEPENDENT of this issue; fixing the re-walk alone leaves a ~4-5s package-scan floor.

~7% — everything else (image restore ~10ms, build-id sha ~0.2s after the 4-KiB sample fix, unnamed CO-RUN thunk bodies).

Per-second leaf shares: reloc-walk machinery holds 35-55% steadily across the replay (consistent with n² spread over ~5k COs), with dips where package wiring / top-level thunk execution dominates. So the reverse-index drain fix addresses two-thirds of the boot; the package-scan class is the next third.

atgreen3993015422

Package-scan share addressed in eaa0c57: find-package fence index + cl-package-memo (external-symbol-p's CL check was 10M find-package calls with two string= probes each). asdf fasl boot: 244G -> 186.7G instructions, ~18s -> ~10s wall, 438MB -> 209MB peak RSS. The remaining boot is now ~2/3 the reloc re-walks (this issue) + a linear-exports tail whose indexed replacement is withheld pending the cave#313 mystery. Numbers in this issue's earlier profile comment are superseded accordingly.