feat(aarch64): add GDB debugging for Linux KVM by cshung · Pull Request #2016 · hyperlight-dev/hyperlight

cshung · 2026-11-03T00:34:05Z

Summary

ARM64 Linux/KVM guests support GDB register and memory access, two hardware breakpoints, BRK software breakpoints, and hardware single-step.

KVM reports debug stops through KVM_EXIT_DEBUG. Guest address translation uses the existing page-table reader. Debugger code writes synchronize the instruction cache. Debug state survives vCPU reset, and register access respects the active stack pointer.

Depends on #1907. The two KVM commits sit above its updated head, 00b72128. Shared snapshot, cancellation, and crash-detach recovery fixes belong to that foundation.

KVM-only comparison.

Validation

Windows ARM64: all six GDB client tests and 14 backend/recovery tests pass after rebase. GDB-enabled Clippy passes.
The full standard test suite, build, formatting, and standard Clippy pass locally. The combined source tree is byte-identical before and after the stack rebase.
Earlier Raspberry Pi 4 validation: 16 targeted native unit tests and six GDB integration tests passed. Linux runtime tests were not repeated for this update.

Pi validation uses a separate, uncommitted ARMv8.0 allocator fix and guests built with RUSTFLAGS="-C target-feature=-lse". Pi compatibility changes are outside these commits.

Open review finding

Debugger memory reads and writes still need guest-page-sized splitting on ARM64 Linux hosts with 16 KiB or 64 KiB pages.

Copilot

🟡 Changes recommended

Unresolved critical flaws allow guest modification of executable trampoline code and debugger memory corruption, while breakpoint cleanup and snapshot-ordering issues can leave sandboxes unrecoverable.

5 open issues

What changed in this PR

Adds ARM64 guest debugging on Linux/KVM, alongside the shared debugger infrastructure and Windows/WHP support from #1907.

Changes:

Supports ARM64 register and memory access, software and hardware breakpoints, and hardware single-step.
Adds instruction-cache synchronization, debug-state preservation across reset, and snapshot safeguards.
Adds debugger tests and documents platform capabilities.

File	Description
`src/hyperlight_host/src/sandbox/uninitialized_evolve.rs`	Initializes ARM64 debug vectors during VM setup.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Exposes backing ranges for cache synchronization.
`src/hyperlight_host/src/sandbox/initialized.rs`	Guards snapshots and restores against active software breakpoints.
`src/hyperlight_host/src/mem/mgr.rs`	Exposes checked host memory ranges to the debugger.
`src/hyperlight_host/src/mem/exe.rs`	Carries debug-vector addresses in load metadata.
`src/hyperlight_host/src/mem/elf.rs`	Finds and relocates the debug-vector symbol.
`src/hyperlight_host/src/hypervisor/virtual_machine/whp/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/whp/aarch64.rs`	Implements WHP debugging, cache synchronization, and tests.
`src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs`	Adds ARM64 debug exits and initialization errors.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/aarch64.rs`	Implements KVM debugging and active-stack-pointer handling.
`src/hyperlight_host/src/hypervisor/regs/aarch64/whp_reg.rs`	Defines WHP debug register identifiers.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Uses shared debugger handling and breakpoint storage.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Integrates ARM64 stops, stepping, and mapping safeguards.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/debug.rs`	Provides shared requests and software-breakpoint lifecycle handling.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Connects ARM64 VM setup and execution to GDB.
`src/hyperlight_host/src/hypervisor/gdb/target.rs`	Adds ARM64 register conversion to the shared target.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Adds page-table translation and cache-flush support.
`src/hyperlight_host/src/hypervisor/gdb/event_loop.rs`	Reports ARM64 initial stops to GDB.
`src/hyperlight_host/src/hypervisor/gdb/arch/x86_64.rs`	Houses x86 breakpoint constants and stop classification.
`src/hyperlight_host/src/hypervisor/gdb/arch/aarch64.rs`	Defines ARM64 software breakpoints and alignment checks.
`src/hyperlight_host/src/hypervisor/gdb/arch.rs`	Selects architecture-specific debugger helpers.
`src/hyperlight_host/examples/guest-debugging/main.rs`	Adds ARM64 breakpoint, stepping, and code-write integration tests.
`src/hyperlight_host/build.rs`	Enables GDB for supported ARM64 platforms.
`src/hyperlight_guest_bin/src/arch/aarch64/exception/entry.rs`	Exports debug vectors and forwards synchronous exceptions.
`src/hyperlight_common/src/arch/aarch64/exn.rs`	Defines the shared debug-exception HVC immediate.
`docs/how-to-debug-a-hyperlight-guest.md`	Documents ARM64 capabilities and guest-symbol requirements.
`CHANGELOG.md`	Records Linux/KVM and Windows/WHP ARM64 debugging support.

_{Review effort: Balanced}

Adds GDB stub support for guests running on the ARM64 WHP backend. * Split the x86_64 debug code out of `hyperlight_vm/x86_64.rs` into a shared `hyperlight_vm/debug.rs`. * Rename `gdb/x86_64_target.rs` to `gdb/target.rs` and add per-architecture register handling in `gdb/arch/`. * Give `DebuggableVm` default methods so backends without hardware breakpoints or single step report `Unsupported`. * Implement `DebuggableVm` for the aarch64 WHP VM, using the batched register helpers. * Flush the host instruction cache after guest memory writes so software breakpoints take effect. * Document ARM64 debugging in `docs/how-to-debug-a-hyperlight-guest.md` and `CHANGELOG.md`. On Windows ARM64 (WHP): * `just build`, `just clippy`, `just guests`, `just test` and `just fmt-check` pass. --------- Signed-off-by: cshung <3410332+cshung@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Changes * Run the full disable path when the debugger detaches after a crash, so software breakpoints and pending steps are cleaned up. * Cancel pending software steps for any cancellation source, not only debugger interrupts. * Reject `map_region` requests that overlap the hidden cache sync page at GPA 0x1000, and add a test. * Restore the `≡` and `×` characters in `hyperlight_vm/x86_64.rs` comments. ## Verification On Windows ARM64 (WHP): * `just clippy`, `just fmt-check` and the `hypervisor::` and `map_region` lib tests with the `gdb` feature pass. --------- Signed-off-by: cshung <3410332+cshung@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Guest synchronous exception vectors forward debug stops through HVC. Hardware single-step supports branches, returns, and self-loops. Two hardware breakpoint slots and BRK software breakpoints are available. Interrupted register context and breakpoint configuration survive cache synchronization and vCPU reset. Asynchronous vectors retain their normal exception paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

Poisoned sandbox restore discards software breakpoint state. Crash detach disables backend debugging even when breakpoint memory is unavailable. Snapshot checkpoints reject installed software breakpoints before replacing memory. The WHP cache-sync page permits guest reads and execution. Exception-vector tests use a separate writable stack. Cache flushing resolves layered snapshot memory through the guest memory view. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

ARM64 KVM debugging uses KVM_SET_GUEST_DEBUG for hardware single-step and two hardware breakpoints. Software breakpoints use BRK. Guest virtual addresses use the existing page-table reader. Debugger code writes synchronize the host instruction cache. Register access respects the active stack pointer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

ARM64 KVM debug jobs run the GDB suite before the general Rust tests. Install GDB when the runner lacks it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

hyperlight-gh-bot · 2026-11-04T17:21:23Z

Benchmark Results

Measured commit: 11361ca19d5c
Baseline commit: 6c17f7cd4e78

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`372.81 µs` (➖ 1.00x slower)
`byte_chunks`		`757.14 ns` (➖ 1.01x slower)
`vec_bytes`		`577.26 ns` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`141.22 ns` (➖ 1.03x faster)
`262144`	`526.01 ns` (➖ 1.17x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.01 ms` (➖ 1.04x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.71 ns` (➖ 1.00x slower)	`7.68 ns` (➖ 1.00x faster)	`7.67 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`97.63 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`1.99 µs` (➖ 1.09x faster)	`2.04 µs` (➖ 1.06x slower)
`262144`	`7.03 µs` (➖ 1.02x faster)	`7.14 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.03 µs` (➖ 1.01x slower)	`1.13 µs` (➖ 1.10x slower)
`65536`	`6.15 µs` (➖ 1.03x faster)	`6.25 µs` (➖ 1.02x faster)
`262144`	`26.84 µs` (➖ 1.02x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`695.70 µs` (➖ 1.01x slower)
`byte_chunks`		`701.37 ns` (➖ 1.03x faster)
`vec_bytes`		`545.92 ns` (➖ 1.04x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`162.08 ns` (➖ 1.00x slower)
`262144`	`539.80 ns` (➖ 1.07x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.89 ms` (➖ 1.09x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.22 ns` (➖ 1.05x slower)	`7.24 ns` (➖ 1.05x slower)	`7.20 ns` (➖ 1.05x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`55.00 µs` (➖ 1.11x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.21 µs` (➖ 1.02x slower)	`2.21 µs` (➖ 1.04x slower)
`262144`	`7.81 µs` (➖ 1.02x slower)	`7.80 µs` (➖ 1.03x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`741.00 ns` (➖ 1.01x faster)	`749.36 ns` (➖ 1.01x slower)
`65536`	`7.32 µs` (➖ 1.01x faster)	`7.45 µs` (➖ 1.01x slower)
`262144`	`30.96 µs` (➖ 1.01x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`320.81 µs` (➖ 1.05x faster)
`byte_chunks`		`930.48 ns` (➖ 1.00x slower)
`vec_bytes`		`704.97 ns` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`182.48 ns` (➖ 1.01x faster)
`262144`	`672.43 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`68.80 ms` (➖ 1.02x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.88 ns` (➖ 1.00x faster)	`9.87 ns` (➖ 1.00x faster)	`9.87 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`107.76 µs` (➖ 1.02x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.45 µs` (➖ 1.00x faster)	`2.62 µs` (➖ 1.02x faster)
`262144`	`8.98 µs` (➖ 1.03x faster)	`9.30 µs` (➖ 1.04x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.31 µs` (➖ 1.05x slower)	`1.28 µs` (➖ 1.08x faster)
`65536`	`7.98 µs` (➖ 1.01x faster)	`8.22 µs` (➖ 1.02x slower)
`262144`	`34.80 µs` (➖ 1.05x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`861.24 µs` (➖ 1.19x slower)
`byte_chunks`		`977.72 ns` (➖ 1.03x slower)
`vec_bytes`		`687.46 ns` (➖ 1.02x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`254.51 ns` (➖ 1.01x faster)
`262144`	`673.25 ns` (➖ 1.03x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`85.60 ms` (➖ 1.09x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.72 ns` (➖ 1.00x slower)	`8.70 ns` (➖ 1.01x faster)	`8.74 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`61.59 µs` (➖ 1.12x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.30 µs` (➖ 1.00x faster)	`2.29 µs` (➖ 1.01x faster)
`262144`	`7.97 µs` (➖ 1.00x slower)	`7.98 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`904.45 ns` (➖ 1.01x faster)	`888.73 ns` (➖ 1.03x faster)
`65536`	`7.45 µs` (➖ 1.01x faster)	`7.53 µs` (➖ 1.00x faster)
`262144`	`38.43 µs` (➖ 1.16x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`1.87 ms` (➖ 1.01x slower)
`byte_chunks`		`1.22 µs` (➖ 1.07x slower)
`vec_bytes`		`762.55 ns` (➖ 1.02x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`229.76 ns` (➖ 1.01x faster)
`262144`	`794.70 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`10.11 ms` (➖ 1.02x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.60 ns` (➖ 1.02x faster)	`10.77 ns` (➖ 1.02x slower)	`10.58 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`601.94 µs` (➖ 1.18x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.53 µs` (➖ 1.10x faster)	`2.52 µs` (➖ 1.00x faster)
`262144`	`8.47 µs` (➖ 1.11x faster)	`8.61 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.32 µs` (➖ 1.11x faster)	`1.37 µs` (➖ 1.02x slower)
`65536`	`8.76 µs` (➖ 1.06x faster)	`9.33 µs` (➖ 1.07x slower)
`262144`	`37.81 µs` (➖ 1.34x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`3.14 ms` (➖ 1.04x slower)
`byte_chunks`		`1.22 µs` (➖ 1.02x slower)
`vec_bytes`		`775.88 ns` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`208.38 ns` (➖ 1.01x faster)
`262144`	`739.21 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.82 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.26 ns` (➖ 1.02x slower)	`9.73 ns` (➖ 1.01x faster)	`9.65 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`603.39 µs` (➖ 1.07x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.43 µs` (➖ 1.05x slower)	`2.31 µs` (➖ 1.01x slower)
`262144`	`7.49 µs` (➖ 1.02x faster)	`7.65 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.08 µs` (➖ 1.06x slower)	`1.20 µs` (➖ 1.15x slower)
`65536`	`8.31 µs` (➖ 1.00x slower)	`8.22 µs` (➖ 1.00x slower)
`262144`	`40.56 µs` (➖ 1.19x faster)

_{Reported by cargo ci bench-report --candidate run:41869275808 --config-file bench_report.toml.}

cshung added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. ready-for-review PR is ready for (re-)review labels Nov 3, 2026

Copilot AI balanced review requested due to automatic review settings November 3, 2026 00:34

cshung requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 3, 2026 00:34

cshung marked this pull request as draft November 3, 2026 00:36

cshung removed the ready-for-review PR is ready for (re-)review label Nov 3, 2026

Copilot started reviewing on behalf of cshung November 3, 2026 00:37

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

cshung force-pushed the cshung/kvm-aarch64-gdb branch from a60be9b to 55da94f Compare November 3, 2026 01:58

This comment has been minimized.

Sign in to view

cshung force-pushed the cshung/kvm-aarch64-gdb branch from 55da94f to 2418379 Compare November 3, 2026 19:28

This comment has been minimized.

Sign in to view

cshung and others added 7 commits November 4, 2026 08:36

ci(aarch64): run KVM GDB tests

741e3e2

ARM64 KVM debug jobs run the GDB suite before the general Rust tests. Install GDB when the runner lacks it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

fix(gdb): split memory access at guest page boundaries

11361ca

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

cshung force-pushed the cshung/kvm-aarch64-gdb branch from a26c2d9 to 11361ca Compare November 4, 2026 16:37

Conversation

cshung commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Open review finding

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Nov 4, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

cshung commented Nov 3, 2026 •

edited

Loading