Skip to content

vmm: Do not mark the copy-on-write restore mapping MADV_HUGEPAGE - #8

Closed
CMGS wants to merge 4 commits into
devfrom
cow-overlay-no-thp
Closed

CMGS wants to merge 4 commits into
devfrom
cow-overlay-no-thp

Conversation

@CMGS

@CMGS CMGS commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Drops MADV_HUGEPAGE from the copy-on-write restore overlay (do_mmap_cow_saved_regions).

A page the guest writes becomes a private anonymous copy of one host base page (4 KiB on x86_64) either way. Read faults on a large-folio filesystem can map PMD-sized file folios read-only, but a read fault that misses the page cache can then request readahead of two PMD-sized folios (about 4 MiB), including the faults taken by the restore path itself. For a CoW-restored VM, thp=on and thp=off differ only in this hint: the anonymous region's MADV_HUGEPAGE is replaced along with its VMA by the MAP_FIXED overlay.

Evidence (test2, 16 CoW clones of a 2 GiB guest, thp=on vs thp=off on the current dev build):

  • idle-host restore p50/p99: 9.9/11.3 ms → 5.8/6.4 ms
  • restore under contention: 49.9/97.6 ms → 9.4/14.3 ms
  • memory-file reads during the run: -26% (with 2x major faults)

Separate from #7.

Read-mostly guard (test2, kernel 6.12; prod runs 7.0). 16 CoW clones of a 2 GiB guest under a light workload, dev 750aa2b vs a build with #7 and this PR, benchmarks in the first and the last clone:

750aa2b with this PR
idle restore p50 / p99 8.5 / 9.4 ms 5.6 / 6.2 ms
16-clone restore p50 47.8 ms 8.9 ms
sha256 loop over a cached file 1553 MiB/s 1552-1554 MiB/s
random reads 3.89M / 3.85M per s 3.75M / 3.73M per s (about -3.5%)
CPU loop 13.0M / 13.5M per s 13.7M / 13.7M per s
FilePmdMapped per clone at the end about 448 MB 0
major faults (slice) 3871 34033

So on a large-folio filesystem the hint does map 2 MiB file folios, and dropping it costs about 3.5% of random-read throughput in a read-mostly guest, while streaming reads and CPU are unchanged and restore gets 1.5x faster idle and about 5x faster with 16 clones restoring.

Gates (node1): fmt, typos, 4 clippy variants, fuzz check, vmm memory_manager 11 tests, virtio-devices 168 tests, release build on the code commit; typos on the two docs commits.

Squash message:

vmm: Do not mark the copy-on-write restore mapping MADV_HUGEPAGE

A copy-on-write restore maps the snapshot file MAP_PRIVATE over guest
RAM, and marked that mapping MADV_HUGEPAGE when thp is on. A page the
guest writes becomes a private anonymous copy of one host base page
(4 KiB on x86_64) with or without the hint. On a filesystem with large
folios, the hint lets read faults map PMD-sized file folios read-only,
but a read fault that misses the page cache can then request readahead
of two PMD-sized folios, about 4 MiB, including the faults the restore
path takes.

Measured on x86_64 with Linux 6.12 and XFS, 16 copy-on-write clones of
a 2 GiB guest, with and without the hint:
- a restore on an idle host took p50 5.6 ms instead of 8.5 ms, and
  p50 8.9 ms instead of 47.8 ms with 16 clones restoring at once;
- random reads of read-mostly guest memory ran about 3.5% slower,
  since the file-backed PMD mappings (about 448 MB per clone) are
  gone; streaming reads and CPU-bound work were unchanged.

Restore latency sits on the clone path, so drop the hint from the
overlay. The reserve policy is still applied, and the docs state the
trade-off.

Signed-off-by: CMGS <ilskdw@gmail.com>

CMGS added 4 commits November 10, 2026 17:30
A copy-on-write restore maps the snapshot file MAP_PRIVATE over guest
RAM, and marked that mapping MADV_HUGEPAGE when thp is on. A private
copy of a file page is always 4 KiB, so the hint buys no huge pages,
but it makes each fault on the mapping read 2 MiB around the faulting
page, including the faults the restore path itself takes.

Measured with 16 copy-on-write clones of a 2 GiB guest, comparing
thp=on with thp=off, which differ only in this hint for such a VM:
- a restore on an idle host took p50 5.8 ms / p99 6.4 ms instead of
  9.9 / 11.3 ms;
- a restore while other clones fault in their memory took p50 9.4 ms
  / p99 14.3 ms instead of 49.9 / 97.6 ms;
- reads from the snapshot memory files dropped by about 26%, at twice
  the number of major faults.

Drop the hint from the overlay. The reserve policy is still applied.

Signed-off-by: CMGS <ilskdw@gmail.com>
memory.md says guest memory is labelled MADV_HUGEPAGE when thp is on,
which no longer holds for memory restored with
memory_restore_mode=copyonwrite.

Signed-off-by: CMGS <ilskdw@gmail.com>
Distinguish the private anonymous copies a guest write creates from the
read-only PMD-sized file folios that MADV_HUGEPAGE allows, record what
dropping the hint costs and saves as measured, and qualify the base
page size and the readahead amount.

Signed-off-by: CMGS <ilskdw@gmail.com>
@CMGS

CMGS commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Squashed into dev as 21a6219 with the squash message above, on top of #7 (e456597). The memory_manager.rs conflict with #7 is resolved by dropping the thp argument and keeping cow_restored.

@CMGS CMGS closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant