#186 Tree-walker macro evaluator silently produces garbage on unknown heads — should fail loudly (or defer)

open
Opened by atgreen

ag-macro-eval-call (compiler.lisp ~3404) is a closed dispatch (quote cons list append car cdr if progn when unless cond null consp eq gensym let let* + hand-wired %-primitives). When a PRELUDE macro body calls a helper DEFUN or uses an unsupported shape, the walker substitutes structurally — vars replaced by their init forms, splices unevaluated — yielding a garbage expansion that compiles into 'attempt to call an undefined function' at runtime, with no hint the MACRO was the problem. The identical body works as a REPL-defined macro (hosted path), which misdirects diagnosis. Cost: two failed labels implementations before the constraint was identified (cave#181, commit b3b0988's log has the story).

Suggestions (from how neighbors handle their restricted bootstrap evaluators):

  1. LOUD FAIL: unknown head in a macro-body position => a static report naming the head and the macro (like ag-set-hosted-bail-loud but on by default for the walker) instead of structural substitution. Cheap, kills the whole silent-garbage class. NB signal with a static string, not make-type-error/format (tree-shake).
  2. Optional, bigger: post-boot re-registration of prelude macro expanders through the hosted compile path (ag-use-hosted-macro-eval infra half-exists, stage1.lisp:114), lifting the op-set constraint from prelude macros entirely. Preliminary-then-redefine is a proven bootstrap pattern.

Until then the rule is documented in the prelude labels comment: prelude macro bodies must stay inside the walker's op set; computation requires recursive helper MACROS (psetq/with-slots-1/%labels-* pattern).

Comments (3)

atgreen3991982501

Owner discussion 2026-07-02 — scope upgrade path:

The tree-walker is NOT an interpreter — it's a closed-dispatch macro-body expander (~150 lines, no user-defun calls, no fn values, no loops). Three escalation tiers for this issue:

  1. NOW (this issue as filed): loud-fail on unknown heads instead of silent structural substitution.
  2. NEXT: post-boot re-registration of prelude expanders through the hosted path (the CLISP preliminary-then-redefine pattern; ag-use-hosted-macro-eval infra half-exists).
  3. DESTINATION (owner: the HotSpot north star requires this anyway): ONE bootstrap-eval.lisp — a full interpreter in the portable subset (~500-800 lines: environments, all special forms, application through the existing fn-table so every compiled primitive is free). Serves three roles with one artifact: bootstrap macro-body evaluation (kills this bug class by construction), honest EVAL/REPL semantics, and the permanent tier-0 of the tiered-execution story (profiling hooks + deopt landing pad get added to it later, not built from scratch). HotSpot never deleted its interpreter — it is load-bearing forever; funcl today is 'C2 with no interpreter', which is exactly why the tree-walker stopgap exists.

Recommend: do (1) cheaply now; open a design issue/ADR for (3) when the 170+ queue clears.

atgreen3991988393

Tier 1 landed in bd5f8af: the unknown-head branch now prints MACROEVAL-PASSTHRU (static bvs, unification-gated => build/boot silent, byte-identical emission) before the unchanged pass-through. Zero reports across the standard battery; a deliberately-bad prelude macro names its offending helper exactly. Tiers 2 (hosted re-registration) and 3 (bootstrap-eval.lisp interpreter / tier-0) remain open here.

atgreen3992786005

IMPACT SHRANK 2026-07-11: the macro-aware EVAL fix (0f0eed8) routes runtime (eval …) through the HOSTED expander, so the tree-walker now only serves the COLD compile path. The fail-loudly proposal stands for that path (MACROEVAL-PASSTHRU ghost-bug class cost multiple debugging campaigns, incl. ~13 build cycles on disproven cave#261). Suggested shape: passthru on an unknown head PRINTS the head to stderr (already partially done) AND, under a strict flag settable in CI builds, hard-errors.