Skip to content

fix(os-linux): only forward exact whole-region munmaps to kfree - #36

Merged
zouguangxian merged 1 commit into
LayerZero-Labs:mainfrom
Acentelles:fix/munmap-region-accounting
Sep 2, 2026
Merged

fix(os-linux): only forward exact whole-region munmaps to kfree#36
zouguangxian merged 1 commit into
LayerZero-Labs:mainfrom
Acentelles:fix/munmap-region-accounting

Conversation

@zouguangxian

Copy link
Copy Markdown
Collaborator

Summary

  • sys_munmap currently forwards any guest (addr, len) straight to kfree. The kernel heap only supports freeing the exact (pointer, layout) pairs it handed out, so a POSIX-legal partial, interior, spanning, or repeated unmap corrupts the heap free list or trips the allocator's bad-free assert.
  • Adds a fixed-capacity region table (crates/zeroos-os-linux/src/handlers/memory.rs) that sys_mmap records into and sys_munmap consults: kfree is only called when (addr, len) exactly matches a live region. Every other unmap shape keeps the pages mapped (a bounded leak, since a kmalloc-backed region can't be partially freed) and returns success, per POSIX.
  • Table storage uses GlobalCell (unsynchronized), not a lock: ZeroOS runs single-core with no interrupts and no preemption, and scheduling is cooperative, so the table is never accessed concurrently — same reasoning already backing the scheduler's own GlobalOption state.

Context

Upstream ask from a16z/jolt#1822, which hit this via a guarded workaround in the Jolt SDK. Original patch and validation by @Acentelles; iterated during review to drop an initial spin::Mutex in favor of the GlobalCell idiom already used elsewhere in this kernel.

Test plan

  • cargo clippy -p zeroos-os-linux --target riscv64gc-unknown-linux-musl --no-default-features --features memory,vfs,scheduler,random -- -D warnings
  • Unit tests: exact-match removal, suppression of partial/interior/spanning/unknown unmaps, full-table failure, slot reuse

sys_munmap forwarded any (addr, len) straight to kfree, but the kernel
heap only supports freeing the exact (pointer, layout) pairs it handed
out, so POSIX-legal repeated, partial, or interior munmaps corrupted
the free list or tripped the allocator's bad-free assert inside the
trap handler, aborting the guest. musl std guests reach these paths
through raw libc::munmap users and reservation trimming.

Track the regions sys_mmap hands out in a fixed-capacity GlobalCell
table (single-core, no interrupts, cooperative scheduling; same
soundness argument as the scheduler's own state) and forward only a
munmap whose base and page-rounded length exactly match a live entry;
every other munmap keeps the pages mapped (a bounded leak, since a
kmalloc-backed region cannot be partially returned) and reports
success per POSIX. If the table fills, sys_mmap frees the fresh
region and fails with -ENOMEM rather than handing out a mapping whose
unmap could never be validated.
@cursor

cursor Bot commented Sep 2, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes guest-visible memory syscall behavior and can leak kmalloc-backed pages on non-exact unmaps, but avoids critical heap corruption; cap of 1024 concurrent mmaps may surface as ENOMEM under heavy mapping load.

Overview
Fixes heap corruption from POSIX-shaped munmap calls by tracking each sys_mmap allocation in a fixed 1024-entry static table (GlobalCell, no heap for bookkeeping).

sys_mmap now records (base, page-rounded size) after kmalloc; if the table is full it rolls back with kfree and returns ENOMEM. sys_munmap only calls kfree when the request exactly matches a live entry; partial, interior, spanning, repeated, or unknown unmaps no longer touch the allocator—they leave pages mapped (bounded leak) and still return success, matching POSIX where the heap cannot split regions.

Adds unit tests for record_region / take_exact_region (exact unmap once, non-exact shapes rejected, full table, slot reuse).

Reviewed by Cursor Bugbot for commit bf25245. Bugbot is set up for automated code reviews on this repo. Configure here.

@zouguangxian zouguangxian reopened this Sep 2, 2026
@zouguangxian
zouguangxian merged commit c572b88 into LayerZero-Labs:main Sep 2, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants