fix(gc): route RS4GC through the in-process LLVM backend

[proggeramlug]

Closes #7327.

The problem

PERRY_RS4GC=1 failed on any stock toolchain with:

error: unterminated attribute group

RS4GC ran as an external opt subprocess and handed its output to clang. On a
Mac that pairing is Homebrew's LLVM 22 opt feeding Apple's clang 21, and the
newer opt emits attributes the older clang cannot parse. So RS4GC was
reachable only with PERRY_LLVM_CLANG pointed at a version-matched LLVM 22 —
which the existing CI arm does, and which no user does.

That mattered more than a knob normally would, because RS4GC is the only
backend that can root an invoke, and since #7302 every call inside a try is
an invoke. The explicit bridge refuses them outright (#7330). 128 of 479 gap
tests (26%) contain try {}. So there was no working statepoint path for a
quarter of the suite on a default toolchain.

The fix

Run the pass in-process, where #7301 already pins LLVM 22 and no IR crosses a
toolchain boundary. The pass itself was already known to schedule there —
rs4gc_schedules_in_process has been asserting it. Two gaps had to be closed to
get from "schedules" to "produces a linkable object":

1. inprocess.rs ignored -S. It fell into the catch-all that discards
   -c, so the statepoint backends asked for assembly and were handed an
   object. Native-frame GC roots via LLVM statepoints, opt-in (#7173, #7174) #7314's compact-map rewriter rewrites .llvm_stackmaps in assembly
   text — that is where LLVM prints function addresses as symbol names — so it
   needs the real thing.

2. Nothing assembled the result. The returned assembly went straight into a
   .o and the link died with ld: unknown file type. This now mirrors the
   external path: write to plan.asm_path, run compact_and_assemble, return
   the object. The assembler is resolved via find_clang() because plan.clang
   is the literal (in-process) placeholder on this path — and using the system
   clang for it is sound, since the skew was an IR parse failure and by this
   point the IR is gone.

Result

All 9 gc-ratchet probes compile under PERRY_RS4GC=1 PERRY_LLVM_INPROCESS=1
with no PERRY_LLVM_* pinning, including probe 09, which the bridge cannot
compile at all:

probe	checksum vs shadow stack	objects copied	__perry_gcmap
01_nursery_churn	identical	17,060	605 B
02_survivor_promotion	identical	90,275	669 B
03_cross_gen_writes	identical	41,129	650 B
04_dead_after_deep_stack	identical	16,510	853 B
05_closure_capture	identical	11,169	730 B
06_string_retention	identical	16,229	595 B
07_array_grow_evacuate	identical	16,134	640 B
08_map_set_sidetables	identical	5,946	622 B
09_try_catch_roots	identical	10,892	1,995 B

9/9 byte-identical to the shadow-stack control, every one under
PERRY_CONSERVATIVE_STACK_SCAN=off so the native map is doing the rooting.
--statepoint-report reports backend rs4gc: 9 function(s) on probe 09 — no
per-function bail to the bridge.

Gating

Existing RS4GC steps all pin PERRY_LLVM_OPT + PERRY_LLVM_CLANG, so none of
them can observe that the pinning is no longer needed. The new step is the only
arm that unsets both. It asserts four things, each of which is a way a GC gate
in this repo has previously gone green while measuring nothing:

• __perry_gcmap exists (the map was emitted)
• __llvm_stackmaps is gone (the compact rewrite ran)
• a copying minor actually copied, via gc_evacuation_liveness_assert.py
  (gc: the native-roots forced-evacuation arm is vacuous — nothing moves, and 5 of 8 probes match zero stack-map records #7336/ci(gc): make the forced-evacuation arm actually evacuate, and assert it (#7336) #7338 — the arm that justified statepoints was reporting
  copied_objects = 0 on all 8 probes)
• --only-backend rs4gc (the subject was RS4GC, not a silent bridge fallback)

Job timeout goes 90 → 120 because the step builds a second time with the
llvm-inprocess feature, which cargo cannot share with the build above it.

No silent-fallback hole

The concern worth stating, since it is the #7332 shape: if the in-process
compile failed and fell back to external clang after maybe_rs4gc_preprocess
had already skipped the external opt, the result would be a binary with no
statepoints at all — correct-looking until a collection freed something live.
Checked: both the failure branch and the missing-feature stub bail!, so there
is no path from "asked for in-process" to "served the text path".

Tests

• dash_s_requests_assembly_and_dash_c_does_not — the parse gap directly.
  Verified it fails when "-S" => emit_asm = true is reverted to "-S" => {}.
• assembly_emission_is_text_not_an_object — asserts the two FileTypes do not
  return identical bytes, so a future regression cannot silently re-swallow
  -S.

Scope

This does not change any default. llvm-inprocess remains a non-default cargo
feature, so PERRY_RS4GC=1 on a stock release build still takes the external
path and still fails loudly there. Making RS4GC a default is a separate decision
that depends on #7301's feature becoming default, and on #7333 (the x86-64
walker) for non-aarch64 hosts.

Summary by CodeRabbit

• New Features

  • RS4GC can now run fully in-process with the pinned LLVM toolchain.
  • Assembly output is supported when using the in-process compiler.
  • invoke roots and compact GC-map generation are supported without external LLVM tool incompatibilities.

• Bug Fixes

  • Prevented duplicate garbage-collection rewriting during compilation.
  • Improved handling of compacted stack maps and generated assembly/object output.

• Tests

  • Added validation covering evacuation, copying, GC-map rewriting, and RS4GC lowering.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f60b9350-148e-467d-a4f8-743eea27bf35

📥 Commits

Reviewing files that changed from the base of the PR and between 480aab9 and f398e23.

📒 Files selected for processing (4)

• .github/workflows/gc-native-roots.yml
• changelog.d/7339-rs4gc-in-process.md
• crates/perry-codegen/src/inprocess.rs
• crates/perry-codegen/src/linker.rs

---

📝 Walkthrough

Walkthrough

The in-process LLVM backend now performs RS4GC rewriting, supports assembly emission, and integrates compact GC-map assembly. CI validates try/catch roots, evacuation, and RS4GC lowering on ARM64.

Changes

In-process LLVM RS4GC pipeline

Layer / File(s)	Summary
Compiler output and RS4GC passes crates/perry-codegen/src/inprocess.rs	The compiler tracks -S, preserves object output for -c, runs RS4GC passes in-process, selects assembly or object emission, and tests both formats.
GC-map assembly integration crates/perry-codegen/src/linker.rs	The linker avoids duplicate external RS4GC rewriting and assembles compacted LLVM output through a system clang when assembly is emitted.
In-process RS4GC validation .github/workflows/gc-native-roots.yml, changelog.d/7339-rs4gc-in-process.md	The ARM64 workflow validates try/catch roots, compact-map replacement, evacuation, object movement, and the rs4gc backend. The changelog records the new behavior and coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant gc_native_roots as gc-native-roots workflow
  participant inprocess_rs as inprocess.rs
  participant linker_rs as linker.rs
  participant clang
  gc_native_roots->>inprocess_rs: build with llvm-inprocess
  inprocess_rs->>inprocess_rs: run RS4GC statepoint rewriting
  inprocess_rs-->>linker_rs: emit assembly
  linker_rs->>linker_rs: compact GC maps
  linker_rs->>clang: assemble compacted assembly
  clang-->>linker_rs: return object
  linker_rs-->>gc_native_roots: run evacuation and backend checks

Loading

Possibly related issues

• exp: in-process LLVM backend — stop shipping IR as text (Phase 0: feasible, byte-identical objects) #7241: Extends the in-process LLVM backend with the RS4GC rewriting and assembly/object emission planned by this issue.

Possibly related PRs

• PerryTS/perry#7301: Extends the in-process LLVM backend in the same compilation paths.
• PerryTS/perry#7322: Introduces the native-roots workflow that this change expands with in-process RS4GC validation.
• PerryTS/perry#7330: Covers the RS4GC invoke try/catch root path that this change enables in-process.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states that RS4GC now uses the in-process LLVM backend.
Description check	✅ Passed	The description explains the problem, implementation, tests, CI coverage, linked issue, and unchanged defaults, although it does not use the exact template headings.
Linked Issues check	✅ Passed	The changes address issue #7327 by enabling RS4GC to root invoke instructions in try/catch code through the in-process LLVM backend.
Out of Scope Changes check	✅ Passed	The code, tests, changelog entry, and CI changes directly support the RS4GC in-process backend objective.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/7327-rs4gc-in-process

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[proggeramlug]

Full gap suite, two arms, 479/479 — zero new regressions, zero refusals

Ran every test_gap_*.ts twice: once with the default shadow-stack build as a
control, once under PERRY_RS4GC=1 PERRY_LLVM_INPROCESS=1, both diffed against
the pinned Node 26.5.1 oracle.

pass -> pass              447
diff -> diff               19    pre-existing, unchanged by the backend
node_fail -> node_fail     13    oracle cannot run the test
────────────────────────────
NEW REGRESSIONS             0
RS4GC refusals              0
RS4GC compile failures      0

Zero refusals is the number that matters here, more than zero regressions.
128 of the 479 tests contain try {}, so every call in them is an invoke,
which the explicit bridge refuses outright (#7330). RS4GC compiled every test in
the suite.

For context: the earlier soak that returned "13 regressions, do not flip" was
measured against the bridge, before #7329/#7330, on a backend that
structurally cannot compile a quarter of the suite. That verdict should not be
carried forward, and docs/engine-plan.md now says so.

Scope, stated plainly

This makes RS4GC-in-process viable as a default on aarch64. It does not make
it one, and two things still gate that — neither of which is correctness:

1. llvm-inprocess is a non-default cargo feature (In-process LLVM backend: native construction via the C API, opt-in (#7241, engine-plan layer 0) #7301's scope).
2. x86-64 is blocked on gc: the platform-unwinder stack-map walker segfaults on Linux/ELF — and on x86-64 it is the only walker #7333. I measured the mechanism while this was running:
   _Unwind_GetGR(ctx, 7) does not return an unreliable value, it segfaults
   — RBX/RBP/RIP return fine, RAX and RSP both SIGSEGV, because libgcc tracks
   only the columns CFI restores and RSP is derived from the CFA. Every x86-64
   root is Indirect [RSP + off], so that call faults 100% of the time on the
   only register those roots use. Comment with a reproducer is on gc: the platform-unwinder stack-map walker segfaults on Linux/ELF — and on x86-64 it is the only walker #7333.

The plain in-process path (PERRY_LLVM_INPROCESS=1 with no statepoints, i.e.
#7301's normal use) was regression-checked separately and is unchanged.