# funcl — project context for Claude ## What this is A Common Lisp implementation. The actual compiler lives in `src/` and is written in funcl's own language. During v0.x development, an Emacs Lisp host in `elisp/` produces the binaries that compile `src/` until self-host is achieved (ADR 0006). (A legacy SBCL host in `host-sbcl-legacy/` bootstrapped the earliest work; it has since been DELETED — the elisp path now produces a working `build/funcl`.) v1.0 targets: Linux x86-64 and Windows x86-64, with Go-style cross-compilation. Authoritative scope and contracts live in [PRD.md](PRD.md). Deferred items in [PRD-v1.1.md](PRD-v1.1.md) and [PRD-v2.0.md](PRD-v2.0.md). ## Architecture commitments (do not change without an ADR) - **The implementation is the linker.** Direct ELF emission on Linux, PE/COFF on Windows. No GNU ld, lld, MSVC link, MinGW, or any system linker is invoked. See `docs/decisions/0002-linking-model.md`. - **The runtime is built on the OS's stable boundary, not libc.** - Linux: raw syscalls (`clone`, `futex`, `mmap`, `epoll_*`, `io_uring`). - Windows: PE imports against `kernel32.dll`, `ntdll.dll`, `ws2_32.dll`, `advapi32.dll` (the OS-supplied DLLs). No msvcrt, no ucrtbase, no VC redistributable. - **GC**: Shenandoah-generational-style at v1.0; ZGC-generational-style as a v1.1 swap-in behind the same interface. The interface is designed against both collectors from day one. See `docs/decisions/0001-gc-algorithm.md`. - **Bordeaux-Threads compatibility**: `bt:make-thread` is 1:1 to a native OS thread (`clone(2)` on Linux, `CreateThread` on Windows). Fibers require explicit `(fiber:spawn …)`. Do not transparently fiberize existing CL code. - **Host**: Emacs Lisp + cl-lib in `elisp/`. The host is permanent per the user's directional preference — even after self-host closes, `elisp/` stays as an always-runnable cold-start path. Nothing about `src/` is elisp-specific; the same files compile under the elisp host and (eventually) under stage1 itself. Self-host (`funcl-compile` produces a byte-identical `funcl-compile-2`) is the ADR 0006 milestone 2 goal. ## How to work in this repo - **`src/`** holds the funcl-source compiler — the v1.0 product, the bootstrap target, the thing that ships. Files here are written in a careful subset of CL that is valid in BOTH the Emacs Lisp host (via cl-lib) AND eventually the funcl bootstrap compiler itself. - **`elisp/`** is the **Emacs Lisp host**. `elisp/funcl-host.el` is the entry point invoked by `emacs --batch`; `elisp/compat.el` is the shim that supplies funcl's primitive operations (byte buffers, file write) using Emacs unibyte buffers, plus a few aliases for forms that differ between Emacs Lisp and CL (`defstruct` → `cl-defstruct`, `rem` → `%`, etc.). `elisp/{codegen,elf,pe}.lisp` are funcl-source shims that only the elisp host drives; they live here (not under `src/`) to mark that ownership. The host stays after self-host as the always- runnable cold-start path. - **`host-sbcl-legacy/`** (the previous SBCL-hosted compiler and its FiveAM suite) has been **DELETED** — the elisp path now produces a working `build/funcl`, so the reference contract is retired. There is no `funcl.asd`; funcl is built by the **Makefile**, not ASDF. - Tests live in `tests/` and run via the Makefile (`make test-fast-sharded`, `make test-bootstrap`, `tests/ansi-run.sh`, …) — ERT (`tests/bootstrap.el`) + shell drivers (`tests/*.sh`), not ASDF/FiveAM. - Decisions go in `docs/decisions/` as numbered ADRs. - Benchmarks go in `bench/`. Compatibility fixtures in `compat/`. ## Fast iteration (the make/elisp path — learned the hard way) The live v0.x workflow is make-based against the elisp host: - `make build` → `build/funcl` (stage1 via Emacs, then self-compiled). - `make test-fast-sharded` → the suite fanned across 8 emacs workers with per-shard logs (build/test-shard-N.log) that NAME failing tests with backtraces. ~10–15 min; prefer this as the pre-push gate. - `make test-bootstrap-quiet` → the same suite, serial (~50 min on a cold tree; the old ~25–30 min figure predates the current test count). Failure detail is filtered but preserved (FAILED/summary lines) — still, the sharded target is strictly more informative. - `tests/ansi-run.sh [DIRS|all]` → ANSI regression coverage. `FUNCL=` overrides the binary under test (e.g. a worktree's stage2), letting baseline-vs-change comparisons run against one checkout's test tree. - `make elisp-bytecode` after ANY edit to `src/cold/*.lisp` or `elisp/*` — funcl-host-load prefers fresh `.elc` (5–10× on suites and stage1). A stale `.elc` is skipped by the newer-than check, so forgetting costs speed, never correctness. The emacs GC is already tuned (512 MB nursery in funcl-host.el) — don't re-tune it. Tips that save the most time: - **Two binaries, two stdlib stories (materializer Phase 2 changed this).** `build/funcl-stage2` still opens `./src/stdlib.lisp` from CWD at startup and compiles it in-process — stdlib edits take effect by just re-running it (run from the repo root so the relative path resolves). `build/funcl` is the IMAGE-BAKED binary: stdlib + mini-clos are snapshotted at build time, so stdlib edits need `make build` to reach it. **Probe stdlib work against `build/funcl-stage2`; verify against `build/funcl` after a rebuild.** A stdlib edit silently invisible in `build/funcl` is this, not a bug. CAVEAT (cave#251): stage2's reload only refreshes EXISTING prelude macro names — a NEW defmacro in stdlib.lisp is silently dropped and its uses compile to undefined-function calls. Probe new prelude MACROS against `build/funcl` after `make build`, never against stage2. - **`--hosted` emits `./a.out`; run THAT for the exit-code gate.** `build/funcl --hosted f.lisp && ./a.out` — the a.out exit code is the last form's value (dbl→42, helper→105). The compile step itself exits 0. - **Pure stdlib additions carry zero self-host risk.** The binary is built from `COLD_SRCS` (which excludes `stdlib.lisp`), so a *new* prelude defun cannot change the binary or break the self-host fixed point. Validate additive functions with probes + the tree-shake gate only; reserve the full bootstrap for changes to EXISTING functions reached during compilation (the printer, `typep`, the `loop` macros, anything under `src/cold/`). - **Probe by piping forms through funcl:** `printf '(form)\n' | build/funcl` — the REPL echoes each result. Only NIL is false (0 is truthy). - **Every new prelude defun with no cold caller must be declared dead** in `src/cold-bootstrap-retained.lisp`, or `funcl-cold-allowed-dead-respected` fails. Fast check (no rebuild): `build/funcl-cold --tree-shake-report "$COLON" out`, where `$COLON` is `funcl-build-cold-srcs` (from `build/build-manifest.el`) joined by `:`; then diff the `(BODY-UNREACHABLE …)` set in `build/cold-tree-shake.sexp` against the union of the two manifests — add UNEXPECTED names, keep STALE empty. Macros expand away and never appear; only functions do. - **Don't let a tree-shake-live function reach `format`.** The arith cluster (`%generic-add` &c.) is statically reachable, so anything it calls is too — routing a guard through `make-type-error` (which calls `format`) drags the whole printer into the cold live set and breaks tree-shaking. Signal with a static report string instead. - **Paren-check in `lisp-mode`**, not the default fundamental-mode: `emacs --batch --eval '(with-temp-buffer (insert-file-contents F) (lisp-mode) (check-parens))'`. Fundamental-mode miscounts parens in `;` comments and `"` strings and sends you on a false hunt. - **The ANSI runner understates progress.** Many `.lsp` files abort early on an unsupported `loop` clause or a missing fixture, so per-file PASS counts hide functions that are actually correct — confirm by probing the function directly. Debugging lessons that each cost a campaign (2026-07-02): - **Never diagnose from single-pattern-grep'd REPL output.** The REPL and loader echo each form's VALUE, which lands on the same or next line as your payload — `grep "a:"` sees `a:NIL` (payload + echo) and misses the real `Y` on the unmatched next line. Three ghost bugs got filed this way. Use `write-string` markers and read unfiltered `tail`; re-run the original probe unfiltered before filing anything. - **`MACROEVAL-PASSTHRU ` on stderr names your bug.** Prelude macros expand via a CLOSED-dispatch tree-walker (quote cons list append car cdr if progn when unless cond null consp eq gensym let let* + a few `%` primitives). Helper *defuns* in a prelude macro body silently expand to garbage — use recursive helper MACROS (the `%labels-*` / psetq chain-peeling pattern). REPL/loaded macros go through the richer hosted path and don't have this limit. - **Boot-compiled prelude macro bodies can't compare keyword literals against macro-arg keywords** (identity split; runtime/hosted macros are fixed). Expand structurally instead — `eval-when` returns PROGN unconditionally for exactly this reason. - **No boot-populated heap tables.** A top-level prelude form that builds an alist/hash at boot gets snapshotted by `save-image` and dangles on restore (segfault). Express builtin tables as FUNCTIONS over compiled quotes; runtime-built state is fine. - **Symbol-identity bugs: get addresses first.** `(%fn-ptr-as-fixnum obj)` returns any object's tagged address — comparing the same name across doors (compiled quote / reader / find-symbol / macro arg) locates the split in one probe. First-use-works-then-fails identity = a canon hit swapping which record an expansion embeds. - **Load Alexandria only via `compat/alexandria/load-all.lisp`.** Order matters (io.lisp needs macros.lisp's `once-only` at expansion time); ad-hoc orders produced 5 phantom "undefined function" errors on consumer defuns — a forward-referenced macro fails at the CONSUMER, naming neither the macro nor the order. Budget ~36 s for the load in probe timeouts, and run long probe batteries in chunked sessions. - **`--hosted` convention:** the produced binary's exit code is the LAST FORM's value — `(dbl 21)` → exit 42. There is no `exit-with-code`. - **defstruct constructors are POSITIONAL** — `(make-pt 7 8)`, not `(make-pt :x 7 :y 8)` (cave#159's phantom). Debugging lessons (2026-07-11 session — each cost real time): - **The `write-string` marker trap wastes hours — read DOUBLED values, not markers.** `(write-string "X=")(prin1 v)` at the REPL prints `X=NIL` because the REPL echoes `write-string`'s NIL return right after the marker, with `v`'s real value on the NEXT line. This made me mis-read `symbol-name`, `eq`, and the unification flag as BROKEN when they WORK — five times in one session. Instead use plain `(prin1 v)(terpri)` and read the doubled value (`(1 2)(1 2)` = prin1 output + REPL echo). The FIRST form's value merges into the `CL-USER>` prompt line, so lead a probe with a bare `(terpri)` or expect the first result hidden. - **Reading a compiler-internal global at the REPL is UNRELIABLE — measure BEHAVIOR.** `*ag-runtime-symbol-unification-p*`, `mini-clos::*mc-ctors*`, &c. read NIL/0 from the REPL (the symbol split) regardless of the real compiled value. Call the function and check the effect (`intern-pkg-scoped-p`, `ag-quote-pkgsym-p`, actual fn-cell/return values), never the flag. - **Tee full output to a file; never grep-filter a slow run live.** A ~15-min asdf run's `UNDEFINED-DESIGNATOR: ` line (which names a failing designator) got filtered away, forcing a full re-run. `CMD 2>&1 | tee full.log` then grep the FILE, so you can re-grep without re-running. `df -h /tmp` first — an unfiltered asdf load is ~800 MB; `rm` the log when done. - **`build/funcl foo.lisp` COMPILES the file (prints BUILD-START/EMIT), it does NOT run it.** To RUN forms, pipe to stdin (`printf '…' | build/funcl` or `build/funcl < file` — that's the REPL). `--hosted foo.lisp` emits `./a.out` whose exit code is the last form's value. Confusing these three costs a wasted run every time. ## Style and conventions - This is a CL project that will become a CL implementation. Idiomatic Common Lisp throughout. Macros where they pay; CLOS where it pays. - Per-subsystem packages (`funcl.reader`, `funcl.compiler`, etc.), re-exported through the top-level `funcl` package as a façade. - No external dependencies until v0.x ships something runnable. Bootstrap stage is pure CL hosted on SBCL. ## License GPL-2.0-only WITH Classpath-exception-2.0. Copyright © 2026 Anthony Green . Every new source file should include the GPLv2 with Classpath exception header (see existing files).