#322 B2: raw-region dump/load at preferred addresses, behind a flag (Axis B / ADR 0025 D2)

open
Opened by atgreen

Part of #318 (epic). Axis B, ADR 0025 D2. Depends on B1.

Goal

Add a raw-region dump/load path behind a flag: heap + code regions written as bytes and mmap'd back at their preferred (save) base — ADR 0025 D2 raw dump. No per-object reconstruction on this path.

Approach

Acceptance gate

At preferred base, a dump/load round-trip reproduces the heap with no allocate-and-decode pass. Suite-green on the flagged path incl. asdf load. (Correctness under a DIFFERENT base is B4's job — do not claim it here.)

Comments (5)

atgreen3993112492

Implementation surface (mapped against current code)

Save orchestration: ag-image-save-to-buf (image.lisp:2176) → ag-image-save-to-buf-with-code-mmap (:2242) for real save-image use. Sections today: METADATA + FN-TABLE + CODE-MMAP + HEAP.

Key asymmetry to exploit: CODE-MMAP is ALREADY a raw byte-copy of a region mapped back at a fixed vaddr (+elf-mmap-vaddr+) — the raw-region precedent B2 needs. The HEAP section, by contrast, is object-serialized (ag-image-emit-heap-objects-walk over a canonical-sorted table, with ag-image-encode-heap-slot index-encoding every inter-object pointer, :1636).

B2 = give the HEAP the CODE-MMAP treatment, behind a flag:

  • New section type +image-section-heap-raw+ (next free id; ADR 0025 D2 raw dump).
  • New defvar *ag-image-raw-heap-dump-p* (default NIL = todays object-walk).
  • Save: when the flag is set, dump the collectable-heap arena bytes [heap-base, high-water) verbatim into the raw section with mem-vaddr = +elf-heap-vaddr+, INSTEAD of the object-walk. Pointers stay absolute.
  • Load: mmap/copy the raw bytes back to +elf-heap-vaddr+ (delta 0) — no pass-1/pass-2 decode. Gate the restore on the raw sections presence.

Boundaries / risks found:

  • The heap high-water is gc-slot-ref +heap-gc-high-water-offset+ (gc.lisp).
  • The dynamic canonical-symbol arena top-slice (cave#209) currently rides in CODE-MMAPs widened file-sz; a raw heap dump must not double-count it.
  • Delta-0 only proves round-trip, NOT relocation completeness — that is B4s forced-non-zero-delta gate, and needs B3s bitmap first.

Depends on nothing to START (flag-gated, additive); B4 (#324) consumes it.

atgreen3993112918

B2 part 1 landed — commit d2b3f8f

Unwired, flag-gated raw-dump primitives (default off = zero behavior change): +image-section-heap-raw+ (id 9), *ag-image-raw-heap-dump-p*, ag-image-emit-heap-raw-payload (dumps [collectable-base, high-water) via %mem-ref-u8), ag-image-load-heap-raw-section (restores via %mem-set-u8), ag-image-find-section-entry.

Verified additive: paren-balanced, make build clean (6.672M->6.680M), image boot smoke correct. Part 2 (round-trip wiring into ag-image-save-to-buf + load dispatch) is gated behind B3 (#323) bitmap + B4 (#324) non-zero-delta verifier, so no reconstruction path is replaced until relocation is proven complete.

atgreen3993113540

Bound fix (via 8bd5c24)

ag-image-emit-heap-raw-payload used ag-image-collectable-heap-hi, which read only the lagging GC high-water slot — so on a never-collected image the raw dump would have emitted 0 bytes. Fixed in 8bd5c24 (B3) to max(8*%heap-top, high-water). B2 part 1 primitives are now bound-correct; still unwired pending the round-trip commit.

atgreen3993113607

B2 part 2 (round-trip wiring) — plan + risk, from reading the save path

Save path is ag-image-save-to-buf-with-code-mmap (image.lisp:2243): fixed 4-section layout METADATA+FN-TABLE+CODE-MMAP+HEAP, section-count hardcoded 4, offsets computed linearly. Wiring under *ag-image-raw-heap-dump-p* = a 5-section variant (…+HEAP-RAW+RELOC) replacing the object-walk HEAP.

The real difficulty (why this is its own careful cycle, not a tail change): a raw heap dump is NOT self-contained. The collectable heap holds cross-region pointers (code pointers in fn-cells / closure entries) and the current restore does heavy fixup the object-walk format depends on — pool-slot relocation, symbol/keyword/pkgsym pool rebuilds, mc-tables, cond-tables. A raw round-trip that actually BOOTS must either (a) keep those fixups and only swap the heap transport, or (b) wait for B5 authoritative adoption. Delta-0 "copy bytes back" is necessary but not sufficient for a bootable image.

Recommend: do B2 part 2 as save+load transport ONLY (flag-gated), verified by a save->reload->boot round-trip test, with the existing fixup steps still running; then B4 (#324) forces non-zero delta to prove the B3 bitmap; only then B5/B6. This is a full build+suite+asdf cycle and a real round-trip harness — a fresh careful session, not a rushed increment.

atgreen3993130647

B2 part 2 SAVE SIDE working (transport-only) — flag-gated 6-section image

ag-image-save-to-buf-with-code-mmap-raw: METADATA + FN-TABLE + CODE-MMAP(slot 2)

  • HEAP(slot 3) + HEAP-RAW + RELOC, appended so the hardcoded CODE-MMAP slot-2 assumption holds (per codex). Verified: flag-on save produces a valid 6-section 49.3MB image (HEAP-RAW=47.8MB @0x800000000, RELOC=93KB bitmap, mem_vaddr=root container 0x802d2b6a9); default (flag-off) save unchanged at 4 sections + boots.

Two real bugs found + fixed while implementing (functional testing earned its keep):

  1. %alloc-bv cannot allocate a single object larger than the 1 MiB region size → "Storage exhausted" on a ~6 MiB snapshot bv. Fixed by emitting the raw payload DIRECTLY into the image buffer (ag-image-emit-heap-raw-bytes-walk, the B2p1 primitive), no intermediate bv. make-byte-vector (used for the 256 MiB save buf) takes a different large-object path; %alloc-bv is region-limited.
  2. The 256 MiB save buffer lands in the collectable heap, so ag-image-collectable-heap-hi folded it into the raw span (measured 322 MiB vs a ~48 MiB real heap) → emitted 322 MiB into a 256 MiB buf → hang/overflow. Fixed: ag-image-save snapshots the high-water into *ag-image-raw-heap-hi* BEFORE allocating the output buffer; the raw save uses that.

Perf note for step 2: the raw payload is emitted byte-by-byte (~48 MiB) via byte-vector-push; the save takes ~1-2 min. A bulk mem->buf primitive would help but is not needed for transport. Also: the save-time heap is ~48 MiB (vs ~6 MiB fresh) because ag-image-runtime-roots compiles the mc/cond bridge thunks. Sharded suite running; commit pending green.