gh-149584: `_remote_debugging`: Map remote pages with `mach_vm_remap` on macOS instead of copying them by maurycy · Pull Request #161011 · python/cpython

maurycy · 2026-10-31T15:43:39Z

The PR adds vm_remap support to profiling.sampling (Tachyon) on macOS, instead of always using mach_vm_read_overwrite().

In plain English :-) instead of copying remote proces memory, just map the same page and read from it directly. Direct memcpy() is much cheaper than a syscall. The trick works especially for typically stably mapped objects, hence explicit syscall read in #160888 for data stack.

The performance gain is relatively good:

Metric	Baseline (`main` b84e0b7 + #160888)	The PR	Change
Work rate	315524.5 calls/sec	981934.2 calls/sec	3.11x faster
Sample rate	306917.8 samples/sec	901670.6 samples/sec	2.94x faster
Avg call time	3.17 µs	1.02 µs	-67.8%
Total attempts	920,754	2,705,013	2.94x more
Successful samples	920,754	2,705,013	-
Failed samples	0	0	-
Success rate	100.00%	100.00%	-
Work efficiency	97.3%	91.8%	-

On top of this, there's little loss in accuracy:

Oracle, this PR (12 cases, 3 runs, 0.5 s each)

3.16.0a1+dev (heads/main-dirty:b84e0b7ce05, Oct 31 2026, 13:09:14) [Clang 21.0.0 (clang-2100.3.34.2)]
cases=async_running_task,asyncio,basic,code_object_reuse,deep_recursion,deep_static,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf,threads runs=3 duration=0.5 rate_khz=100.0 warmup=0.7 poisson_sampling=False

async_running_task (get_async_stack_trace) ref_keys=14
mode                 samples    stacks     empty           µs      errors% impossible  impossible%
blocking-nocache      143429    286858      5930    7.82±0.02    0.00±0.00          0    0.00±0.00
live-nocache          141384    282732      8513    6.53±0.08    0.05±0.01        949    0.34±0.01

asyncio (get_async_stack_trace) ref_keys=45
mode                 samples    stacks     empty           µs      errors%
blocking-nocache      143796    268858      6097    8.22±0.03    0.00±0.00
live-nocache          146382    177393      3232    6.91±0.44    0.19±0.03

basic (get_stack_trace) ref_keys=56
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      146864    6.63±0.22    0.00±0.00          ref          ref          -
blocking-cache        149984    3.13±0.31    0.00±0.00  0.423±0.075  0.413±0.074      0.010
live-cache            150000    1.30±0.07    0.00±0.00  0.565±0.058  0.555±0.058      0.010
live-nocache          150000    3.86±0.04    0.00±0.00  0.566±0.058  0.556±0.057      0.010

code_object_reuse (get_stack_trace) ref_keys=10
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    6.91±0.03    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    4.93±0.19    0.00±0.00          0    0.00±0.00  0.017±0.014  0.014±0.014      0.003
live-cache            150000    1.56±0.08    0.00±0.00          0    0.00±0.00  0.176±0.011  0.172±0.011      0.003
live-nocache          150000    4.62±0.06    0.00±0.00          0    0.00±0.00  0.018±0.011  0.015±0.011      0.003

deep_recursion (get_stack_trace) ref_keys=1004
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       60313   10.43±0.22    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache         35380   16.07±0.30    0.00±0.00          0    0.00±0.00  0.478±0.014  0.383±0.016      0.094
live-cache             28327   35.09±2.83    4.51±0.22       4076   14.44±1.20  0.573±0.007  0.465±0.010      0.107
live-nocache           27174   38.28±2.42    5.25±0.24       4659   17.18±1.11  0.555±0.003  0.444±0.007      0.108

deep_static (get_stack_trace) ref_keys=1
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       11189   27.64±0.81    0.00±0.00          ref          ref          -
blocking-cache         13213    6.10±0.14    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-cache             13508    5.13±0.03    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-nocache           11402   26.43±0.81    0.00±0.00  0.000±0.000  0.000±0.000      0.000

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      149998    6.94±0.02    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    6.73±0.07    0.00±0.00          0    0.00±0.00  0.016±0.004  0.011±0.004      0.004
live-cache            150000    5.26±0.03    0.00±0.00       1003    0.67±0.09  0.189±0.016  0.184±0.016      0.004
live-nocache          150000    4.57±0.16    0.00±0.00        918    0.61±0.13  0.176±0.026  0.172±0.026      0.004

gen_alternating (get_stack_trace) ref_keys=18
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      149998    7.58±0.01    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    7.14±0.07    0.00±0.00          0    0.00±0.00  0.105±0.008  0.096±0.008      0.008
live-cache            125168    5.87±0.14   16.55±0.04      31674   25.30±3.30  0.307±0.008  0.297±0.008      0.010
live-nocache          125917    5.11±0.13   16.06±0.04      32144   25.53±1.52  0.308±0.011  0.299±0.011      0.010

nested_alternating (get_stack_trace) ref_keys=11
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    6.90±0.07    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    5.55±0.09    0.00±0.00          0    0.00±0.00  0.016±0.008  0.011±0.008      0.005
live-cache            150000    1.83±0.01    0.00±0.00         30    0.02±0.03  0.131±0.013  0.127±0.013      0.005
live-nocache          150000    4.96±0.05    0.00±0.00          0    0.00±0.00  0.154±0.015  0.149±0.015      0.005

oversized_chunk (get_stack_trace) ref_keys=15
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    6.90±0.06    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    5.59±0.13    0.00±0.00          0    0.00±0.00  0.028±0.003  0.023±0.003      0.005
live-cache            150000    3.19±0.05    0.00±0.00         29    0.02±0.03  0.080±0.004  0.074±0.004      0.005
live-nocache          150000    4.86±0.07    0.00±0.00          1    0.00±0.00  0.077±0.005  0.072±0.005      0.005

shared_leaf (get_stack_trace) ref_keys=6
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      149999    6.95±0.03    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    5.11±0.20    0.00±0.00          0    0.00±0.00  0.008±0.005  0.005±0.005      0.003
live-cache            150000    1.62±0.03    0.00±0.00      14955    9.97±0.39  0.119±0.011  0.116±0.011      0.003
live-nocache          150000    4.85±0.02    0.00±0.00          0    0.00±0.00  0.123±0.009  0.120±0.009      0.003

threads (get_stack_trace) ref_keys=20
mode                 samples    stacks           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       29141    268166  51.10±27.24    0.00±0.00          ref          ref          -
blocking-cache         80117    703951   10.94±2.78    0.00±0.00  0.045±0.018  0.044±0.018      0.002
live-cache            117956    814566    6.48±0.93    0.00±0.00  0.080±0.036  0.078±0.036      0.002
live-nocache           43251    283134   28.99±5.00    0.00±0.00  0.067±0.010  0.065±0.010      0.002

v.

Oracle, baseline (12 cases, 3 runs, 0.5 s each)

3.16.0a1+dev (heads/main-dirty:b84e0b7ce05, Oct 31 2026, 13:09:14) [Clang 21.0.0 (clang-2100.3.34.2)]
cases=async_running_task,asyncio,basic,code_object_reuse,deep_recursion,deep_static,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf,threads runs=3 duration=0.5 rate_khz=100.0 warmup=0.7 poisson_sampling=False

async_running_task (get_async_stack_trace) ref_keys=6
mode                 samples    stacks     empty           µs      errors% impossible  impossible%
blocking-nocache       29187     58374     74960  17.27±11.39    0.00±0.00          0    0.00±0.00
live-nocache           49526     99028      6270   24.57±0.74    1.28±0.23       4024    4.06±0.07

asyncio (get_async_stack_trace) ref_keys=15
mode                 samples    stacks     empty           µs      errors%
blocking-nocache       54424     68208     22235   19.08±4.01    0.00±0.00
live-nocache           62230     73032      2005   21.83±0.34    0.59±0.19

basic (get_stack_trace) ref_keys=12
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      134801    9.70±0.13    0.00±0.00          ref          ref          -
blocking-cache        149943    5.92±0.34    0.00±0.00  0.696±0.033  0.689±0.034      0.007
live-cache            150000    3.56±0.03    0.00±0.00  0.712±0.032  0.705±0.032      0.007
live-nocache          150000    6.98±0.08    0.00±0.00  0.711±0.032  0.704±0.032      0.007

code_object_reuse (get_stack_trace) ref_keys=11
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      116286   11.02±1.05    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    7.14±0.36    0.00±0.00          0    0.00±0.00  0.272±0.242  0.269±0.244      0.004
live-cache            149831    5.03±0.54    0.00±0.00          0    0.00±0.00  0.702±0.140  0.700±0.142      0.004
live-nocache          147628    8.31±0.15    0.00±0.00          0    0.00±0.00  0.251±0.223  0.248±0.225      0.004

deep_recursion (get_stack_trace) ref_keys=769
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       88546   11.31±0.65    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache         45389   14.91±1.06    0.00±0.00          0    0.00±0.00  0.637±0.087  0.583±0.096      0.057
live-cache             28321   24.43±0.92   42.63±1.50       7654   27.03±0.52  0.840±0.042  0.768±0.054      0.074
live-nocache           26788   23.46±0.99   48.45±4.13       7824   29.23±1.53  0.832±0.044  0.758±0.056      0.077

deep_static (get_stack_trace) ref_keys=1
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       11276   30.12±0.74    0.00±0.00          ref          ref          -
blocking-cache         13407    8.39±0.21    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-cache             13949    7.34±0.11    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-nocache           11506   28.81±0.41    0.00±0.00  0.000±0.000  0.000±0.000      0.000

flat_alternating (get_stack_trace) ref_keys=5
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      130888    9.78±0.07    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        149999    7.35±0.04    0.00±0.00          0    0.00±0.00  0.101±0.047  0.096±0.047      0.004
live-cache            148202    8.46±0.24    0.00±0.00        896    0.60±0.06  0.148±0.053  0.144±0.053      0.004
live-nocache          149984    7.93±0.11    0.00±0.00        860    0.57±0.09  0.143±0.049  0.139±0.049      0.004

gen_alternating (get_stack_trace) ref_keys=17
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      119399   11.13±0.12    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        149999    7.98±0.03    0.00±0.00          0    0.00±0.00  0.250±0.031  0.241±0.031      0.009
live-cache            106653   10.68±0.15   16.37±0.15      22137   20.75±0.64  0.333±0.029  0.322±0.029      0.010
live-nocache          113135   10.00±0.23   16.27±0.22      23105   20.41±1.15  0.325±0.025  0.315±0.025      0.010

nested_alternating (get_stack_trace) ref_keys=6
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      123268    9.71±0.19    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    6.67±0.09    0.00±0.00          0    0.00±0.00  0.130±0.074  0.126±0.075      0.004
live-cache            149999    4.77±0.03    0.00±0.00         30    0.02±0.03  0.147±0.058  0.142±0.059      0.004
live-nocache          136534    8.50±0.15    0.00±0.00          0    0.00±0.00  0.161±0.049  0.156±0.049      0.005

oversized_chunk (get_stack_trace) ref_keys=8
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      131287    9.56±0.20    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        149998    6.84±0.03    0.00±0.00          0    0.00±0.00  0.111±0.068  0.105±0.068      0.005
live-cache            149985    6.72±0.11    0.00±0.00         58    0.04±0.01  0.142±0.025  0.136±0.026      0.005
live-nocache          147499    8.24±0.10    0.00±0.00          0    0.00±0.00  0.176±0.045  0.170±0.045      0.005

shared_leaf (get_stack_trace) ref_keys=4
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      132620    9.61±0.11    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    6.80±0.04    0.00±0.00          0    0.00±0.00  0.031±0.012  0.028±0.012      0.003
live-cache            150000    4.59±0.10    0.00±0.00          0    0.00±0.00  0.108±0.022  0.105±0.021      0.003
live-nocache          146627    8.47±0.15    0.00±0.00          0    0.00±0.00  0.107±0.020  0.104±0.019      0.003

threads (get_stack_trace) ref_keys=7
mode                 samples    stacks           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       31783    200455   41.02±3.12    0.00±0.00          ref          ref          -
blocking-cache         47157    413580   25.29±7.73    0.00±0.00  0.090±0.049  0.088±0.049      0.002
live-cache             56879    455878   20.91±7.51    0.01±0.01  0.095±0.013  0.092±0.013      0.002
live-nocache           26561    223023  55.50±26.02    0.00±0.00  0.103±0.046  0.101±0.046      0.003

(this is where #153804 shines in my opinion.)

To make things even better, the CPU usage is also relatively better:

case / frame cache	base user µs	base sys µs	base CPU %	alias (PR) user µs	alias sys µs	alias CPU %
async_running_task/cache	2.67	10.53	13.2	2.46	5.12	7.6
async_running_task/nocache	3.44	13.05	16.5	3.35	7.17	10.5
asyncio/cache	4.63	15.06	19.7	5.32	10.43	15.8
asyncio/nocache	7.75	22.67	30.4	5.48	10.61	16.1
basic/cache	1.59	5.73	7.3	1.61	2.92	4.5
basic/nocache	2.41	8.87	11.3	2.38	5.33	7.7
code_object_reuse/cache	2.36	8.88	11.2	2.30	3.72	6.0
code_object_reuse/nocache	3.80	14.22	18.0	2.91	5.99	8.9
deep_recursion/cache	12.46	37.53	50.0	10.78	25.61	36.4
deep_recursion/nocache	10.24	32.43	42.7	11.66	29.89	41.6
deep_static/cache	5.77	5.33	11.1	5.85	2.60	8.4
deep_static/nocache	14.49	18.95	33.4	14.82	16.02	30.8
flat_alternating/cache	2.69	9.70	12.4	3.50	7.65	11.2
flat_alternating/nocache	2.68	9.55	12.2	2.79	5.75	8.5
gen_alternating/cache	3.12	11.89	15.0	3.19	6.79	10.0
gen_alternating/nocache	3.01	11.36	14.4	3.15	6.29	9.4
nested_alternating/cache	2.38	8.18	10.6	2.62	4.96	7.6
nested_alternating/nocache	2.78	9.75	12.5	3.03	6.13	9.2
oversized_chunk/cache	2.89	10.24	13.1	3.08	6.73	9.8
oversized_chunk/nocache	2.79	9.80	12.6	2.91	5.95	8.9
shared_leaf/cache	1.96	6.69	8.6	2.02	3.61	5.6
shared_leaf/nocache	2.69	9.63	12.3	2.86	5.99	8.9
threads/cache	4.76	25.05	29.8	5.65	15.28	20.9
threads/nocache	14.06	60.64	74.6	11.55	38.28	49.8

(everything on PGO+LTO builds.)

Some considerations:

vm_remap is a Mach 3, so it's at least 1992? The XNU kernel is extremely helpful, but there's also https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/osfmk/man/vm_remap.html etc. I used Fable 5.1, and previous Claude models, a lot when asking questions about the kernel. I recommend doing the same.
I feel that _Py_RemoteDebug_ReadRemoteMemory and _Py_RemoteDebug_PagedReadRemoteMemory are increasingly leaky abstractions; the optimal way of reading depends on what (simply: datastacks have different volatility / are less stable than interpreters etc.), the operating system; this is visible more since gh-149584: Fix excessive overhead in the Tachyon profiler regarding the cache behavior #149649 We can reorganize this as a follow up

Issue: _remote_debugging: reading whole pages over and over #149584

maurycy · 2026-10-31T15:55:15Z

+}
+
+static int
+alias_maybe_probe_entry(proc_handle_t *handle, alias_page_entry_t *entry)


mach_vm_region_recurse is expensive, hence adaptive probing here, as the exact churn depends on the target

see #3

c00f7ef

maurycy requested a review from pablogsal as a code owner October 31, 2026 15:43

bedevere-app Bot added the awaiting review label Oct 31, 2026

bedevere-app Bot mentioned this pull request Oct 31, 2026

_remote_debugging: reading whole pages over and over #149584

Open

maurycy mentioned this pull request Oct 31, 2026

Use mach_vm_remap in Tachyon on macOS maurycy/cpython#3

Closed

news

99d1803

maurycy commented Oct 31, 2026

View reviewed changes

Navigation Menu

Repository navigation

gh-149584: `_remote_debugging`: Map remote pages with `mach_vm_remap` on macOS instead of copying them - #161011

gh-149584: `_remote_debugging`: Map remote pages with `mach_vm_remap` on macOS instead of copying them#161011
maurycy wants to merge 2 commits into
python:mainfrom
maurycy:tachyon-vmremap

maurycy commented Oct 31, 2026 •

edited

Loading

Uh oh!

maurycy Oct 31, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

maurycy commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maurycy Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

maurycy commented Oct 31, 2026 •

edited

Loading

maurycy Oct 31, 2026 •

edited

Loading