#322 B2: raw-region dump/load at preferred addresses, behind a flag (Axis B / ADR 0025 D2)
openPart of #318 (epic). Axis B, ADR 0025 D2. Depends on B1.
Goal
Add a raw-region dump/load path behind a flag: heap + code regions written as bytes and mmap'd back at their preferred (save) base — ADR 0025 D2 raw dump. No per-object reconstruction on this path.
Approach
- New save path: dump regions verbatim + a section table (region → file offset, size, save vaddr).
- New load path (flagged):
mmapeach region at its save base (delta 0). - Runs alongside the old serializer; old path stays default until B4 proves the ledger.
Acceptance gate
At preferred base, a dump/load round-trip reproduces the heap with no allocate-and-decode pass. Suite-green on the flagged path incl. asdf load. (Correctness under a DIFFERENT base is B4's job — do not claim it here.)
Comments (5)
B2 part 1 landed — commit d2b3f8f
Unwired, flag-gated raw-dump primitives (default off = zero behavior change):
+image-section-heap-raw+ (id 9), *ag-image-raw-heap-dump-p*,
ag-image-emit-heap-raw-payload (dumps [collectable-base, high-water) via
%mem-ref-u8), ag-image-load-heap-raw-section (restores via %mem-set-u8),
ag-image-find-section-entry.
Verified additive: paren-balanced, make build clean (6.672M->6.680M), image
boot smoke correct. Part 2 (round-trip wiring into ag-image-save-to-buf +
load dispatch) is gated behind B3 (#323) bitmap + B4 (#324) non-zero-delta
verifier, so no reconstruction path is replaced until relocation is proven
complete.
Bound fix (via 8bd5c24)
ag-image-emit-heap-raw-payload used ag-image-collectable-heap-hi, which read
only the lagging GC high-water slot — so on a never-collected image the raw dump
would have emitted 0 bytes. Fixed in 8bd5c24 (B3) to max(8*%heap-top,
high-water). B2 part 1 primitives are now bound-correct; still unwired pending
the round-trip commit.
B2 part 2 (round-trip wiring) — plan + risk, from reading the save path
Save path is ag-image-save-to-buf-with-code-mmap (image.lisp:2243): fixed
4-section layout METADATA+FN-TABLE+CODE-MMAP+HEAP, section-count hardcoded 4,
offsets computed linearly. Wiring under *ag-image-raw-heap-dump-p* = a
5-section variant (…+HEAP-RAW+RELOC) replacing the object-walk HEAP.
The real difficulty (why this is its own careful cycle, not a tail change): a raw heap dump is NOT self-contained. The collectable heap holds cross-region pointers (code pointers in fn-cells / closure entries) and the current restore does heavy fixup the object-walk format depends on — pool-slot relocation, symbol/keyword/pkgsym pool rebuilds, mc-tables, cond-tables. A raw round-trip that actually BOOTS must either (a) keep those fixups and only swap the heap transport, or (b) wait for B5 authoritative adoption. Delta-0 "copy bytes back" is necessary but not sufficient for a bootable image.
Recommend: do B2 part 2 as save+load transport ONLY (flag-gated), verified by a save->reload->boot round-trip test, with the existing fixup steps still running; then B4 (#324) forces non-zero delta to prove the B3 bitmap; only then B5/B6. This is a full build+suite+asdf cycle and a real round-trip harness — a fresh careful session, not a rushed increment.
B2 part 2 SAVE SIDE working (transport-only) — flag-gated 6-section image
ag-image-save-to-buf-with-code-mmap-raw: METADATA + FN-TABLE + CODE-MMAP(slot 2)
- HEAP(slot 3) + HEAP-RAW + RELOC, appended so the hardcoded CODE-MMAP slot-2 assumption holds (per codex). Verified: flag-on save produces a valid 6-section 49.3MB image (HEAP-RAW=47.8MB @0x800000000, RELOC=93KB bitmap, mem_vaddr=root container 0x802d2b6a9); default (flag-off) save unchanged at 4 sections + boots.
Two real bugs found + fixed while implementing (functional testing earned its keep):
%alloc-bvcannot allocate a single object larger than the 1 MiB region size → "Storage exhausted" on a ~6 MiB snapshot bv. Fixed by emitting the raw payload DIRECTLY into the image buffer (ag-image-emit-heap-raw-bytes-walk, the B2p1 primitive), no intermediate bv.make-byte-vector(used for the 256 MiB save buf) takes a different large-object path;%alloc-bvis region-limited.- The 256 MiB save buffer lands in the collectable heap, so
ag-image-collectable-heap-hifolded it into the raw span (measured 322 MiB vs a ~48 MiB real heap) → emitted 322 MiB into a 256 MiB buf → hang/overflow. Fixed:ag-image-savesnapshots the high-water into*ag-image-raw-heap-hi*BEFORE allocating the output buffer; the raw save uses that.
Perf note for step 2: the raw payload is emitted byte-by-byte (~48 MiB) via
byte-vector-push; the save takes ~1-2 min. A bulk mem->buf primitive would
help but is not needed for transport. Also: the save-time heap is ~48 MiB (vs
~6 MiB fresh) because ag-image-runtime-roots compiles the mc/cond bridge
thunks. Sharded suite running; commit pending green.
Implementation surface (mapped against current code)
Save orchestration:
ag-image-save-to-buf(image.lisp:2176) →ag-image-save-to-buf-with-code-mmap(:2242) for real save-image use. Sections today: METADATA + FN-TABLE + CODE-MMAP + HEAP.Key asymmetry to exploit: CODE-MMAP is ALREADY a raw byte-copy of a region mapped back at a fixed vaddr (
+elf-mmap-vaddr+) — the raw-region precedent B2 needs. The HEAP section, by contrast, is object-serialized (ag-image-emit-heap-objects-walkover a canonical-sorted table, withag-image-encode-heap-slotindex-encoding every inter-object pointer,:1636).B2 = give the HEAP the CODE-MMAP treatment, behind a flag:
+image-section-heap-raw+(next free id; ADR 0025 D2 raw dump).*ag-image-raw-heap-dump-p*(default NIL = todays object-walk).+elf-heap-vaddr+, INSTEAD of the object-walk. Pointers stay absolute.mmap/copy the raw bytes back to+elf-heap-vaddr+(delta 0) — no pass-1/pass-2 decode. Gate the restore on the raw sections presence.Boundaries / risks found:
gc-slot-ref +heap-gc-high-water-offset+(gc.lisp).Depends on nothing to START (flag-gated, additive); B4 (#324) consumes it.