Skip to content

Commit 4ceb3fa

Browse files
pablogsalmiss-islingtonmaurycyclaudevstinner
authored
[3.15] Restore deferred backports for 3.15.1, including GH-158874 (#158882)
* [3.15] gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) (#158813) gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) * let me declare single limit * use our new limit in process_frame_chain() * add it in parse_async_frame_chain() * parse_coro_chain() * NEWS * async in the message? * test * no race * process_task_awaited_by * process_task_awaited_by limit test * NEWS * MAX_TASK_WAITER_CHAIN_DEPTH * TASK_WAITER_CHAIN_DEPTH in test * TASK_WAITER_CHAIN_DEPTH 256 * prevent the drift with the comment * better naming, better style * MAX_TASK_WAITER_CHAIN_DEPTH comment * task-waiter iterative bfs walk * iterative coro-walk * nicer news * 1 << 14 * comment * unused read_Py_ssize_t * fix tombstones * simplify * correct msg * better test * news for tombstones * left-over from when testing buggy version * redundant new line (cherry picked from commit e0861c6) Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> (cherry picked from commit 9e401cf) * [3.15] gh-155811: Add a seqcount to `gc_stats` to prevent torn reads (GH-155828) (#158829) * update_seq * no need for XCHGL, MOVL is enough? * gh-155811: Retry an inconsistent GC snapshot once --------- (cherry picked from commit 5fecd44) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 0281240) * [3.15] gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) (#158830) gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) * the kolektor * test * better test * news * =Q, move const to the base, not self.running * gh-151292: Track binary writer finalization state --------- (cherry picked from commit f839c06) Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 8a7c23f) * [3.15] gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (#158691) gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (cherry picked from commit 9d22a53) Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit 4f7af46) * [3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) (#158831) * gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195) * degrade gracefully * news * better NEWS wording * do not raise on MAX_REMOTE_STR_READ * bye MAX_REMOTE_STR_READ * fix -m asyncio ps|pstree * test truncation and linetable sentinel * simpler * simpler * redundant now * respect #157790 in the news --------- Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 7d25916) * Preserve the stable ABI when creating fallback frame names --------- Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> (cherry picked from commit c27f494) * [3.15] Add MSan to CI (GH-158625) (#158832) Run a job that the test suite with MSan to the CI (#158625) * Run the test suite with MSan in CI * Additional fixes * Add `_Py_MSAN_UNPOISON_STRING` * Apply Victor's suggestions * Apply Victor's suggestions --------- (cherry picked from commit b93fb19) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com> (cherry picked from commit 1bc78de) * [3.15] gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (#156814) gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (cherry picked from commit c3706f4) Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> (cherry picked from commit fce28da) * [3.15] gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) (#158845) gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) * gh-158552: Wait for Windows threads to suspend before blocking sampling * Use a named Windows thread enumeration status constant (cherry picked from commit 1643525) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 31288da) * [3.15] gh-152721: Fix quadratic RLE replay time in the profiling binary reader (GH-152722) (#158850) Backport of GH-152722. Co-authored-by: tonghuaroot <tonghuaroot@gmail.com> (cherry picked from commit ebaca2a) * [3.15] gh-156545: Fix flamegraph export RecursionError on deeply recursive programs (GH-156546) (#158851) Backport of GH-156546. Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> (cherry picked from commit 0ac7217) * [3.15] gh-158540: Add the profiled script's directory to sys.path (GH-158548) (#158844) gh-158540: Add the profiled script's directory to sys.path (GH-158548) * gh-158540: Add the profiled script's directory to sys.path When a script is profiled with ``python -m profiling.sampling run script.py`` from another directory, the script cannot import modules placed next to it, because ``_sync_coordinator._execute_script()`` executes it with the working directory (added by ``_setup_environment()`` for the module case) as ``sys.path[0]`` instead of the script's own directory. Make the script's directory importable in ``_execute_script()``, matching the behavior of ``python script.py``. Add a regression test that runs the coordinator on a script importing a sibling module. * Update _sync_coordinator.py Comment simplified * gh-158540: Resolve symlinks when adding the script directory to sys.path ``python script.py`` resolves symlinks when computing ``sys.path[0]``, so a script reached through a symlink (``link.py -> sub/where.py``) imports modules from the real script's directory. Apply ``os.path.realpath()`` before taking the directory name, and make sure the result is placed at the front of ``sys.path`` even if it was already listed. Add a regression test for a symlinked script. --------- (cherry picked from commit 3f02aab) Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com> Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com> (cherry picked from commit f52d831) * [3.15] gh-153838: Skip non-regular source files in the heatmap exporter (GH-153839) (#158853) Backport of GH-153839. Co-authored-by: tonghuaroot <tonghuaroot@gmail.com> (cherry picked from commit 48998df) * [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) (#158852) * [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) Backport of GH-158581. Co-authored-by: LucasZhou <donghao.zhou@outlook.com> * [3.15] gh-158539: Use portable static assertion messages * [3.15] gh-158539: Keep layout assertions with debug-offset validation * [3.15] gh-158539: Use the platform guard for in-process inspection tests --------- Co-authored-by: LucasZhou <donghao.zhou@outlook.com> (cherry picked from commit d625ecb) * [3.15] gh-158522: Fix truncated stack for a task whose coroutine recurses (GH-158526) (#158870) Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com> (cherry picked from commit a904b39) * gh-156545, gh-158539: Fix deep flamegraph export on small C stacks and macOS runtime lookup (#158874) (cherry picked from commit 114de19) Include the C-stack test helper from main, introduced by ce5ae29, which the regression test requires but 3.15 does not yet provide. --------- Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com> Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com> Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com> Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com> Co-authored-by: LucasZhou <donghao.zhou@outlook.com> Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
1 parent 8b02517 commit 4ceb3fa

56 files changed

Lines changed: 1830 additions & 390 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/build.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -605,6 +605,9 @@ jobs:
605605
- check-name: Undefined behavior
606606
sanitizer: UBSan
607607
free-threading: false
608+
- check-name: Memory
609+
sanitizer: MSan
610+
free-threading: false
608611
uses: ./.github/workflows/reusable-san.yml
609612
with:
610613
sanitizer: ${{ matrix.sanitizer }}

‎.github/workflows/reusable-san.yml‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ jobs:
6060
|| ''
6161
}}
6262
- name: UBSan option setup
63-
if: inputs.sanitizer != 'TSan'
63+
if: inputs.sanitizer == 'UBSan'
6464
run: >-
6565
echo
6666
"UBSAN_OPTIONS=${SAN_LOG_OPTION}
@@ -69,6 +69,20 @@ jobs:
6969
>> "$GITHUB_ENV"
7070
env:
7171
SAN_LOG_OPTION: log_path=${{ github.workspace }}/san_log
72+
- name: MSan option setup
73+
if: inputs.sanitizer == 'MSan'
74+
run: |
75+
echo "MSAN_OPTIONS=${SAN_LOG_OPTION} allocator_may_return_null=1 handle_segv=0" >> "$GITHUB_ENV"
76+
# MSan reports false positives for memory initialized by libraries
77+
# that are not built with MSan, so disable modules that use them.
78+
# _remote_debugging links to libzstd directly, but we unpoision the memory.
79+
{
80+
echo '*disabled*'
81+
echo '_bz2 _ctypes _curses _curses_panel _dbm _decimal _gdbm _hashlib'
82+
echo '_lzma _sqlite3 _ssl _tkinter _uuid _zstd readline zlib'
83+
} > Modules/Setup.local
84+
env:
85+
SAN_LOG_OPTION: log_path=${{ github.workspace }}/san_log
7286
- name: Add ccache to PATH
7387
run: |
7488
echo "PATH=/usr/lib/ccache:$PATH" >> "$GITHUB_ENV"
@@ -93,6 +107,8 @@ jobs:
93107
# gh-157958: -O2 instead of the pydebug default -Og to avoid a clang 21
94108
# compile-time blowup on some interpreter files.
95109
# (https://github.com/llvm/llvm-project/issues/179695)
110+
# MSan uses --with-assertions instead of --with-pydebug because its
111+
# hooks on the Python memory allocators hide uninitialized reads.
96112
- name: Configure CPython
97113
run: >-
98114
./configure
@@ -101,9 +117,11 @@ jobs:
101117
${{
102118
inputs.sanitizer == 'TSan'
103119
&& '--with-thread-sanitizer'
120+
|| inputs.sanitizer == 'MSan'
121+
&& '--with-memory-sanitizer'
104122
|| '--with-undefined-behavior-sanitizer --with-strict-overflow'
105123
}}
106-
--with-pydebug
124+
${{ inputs.sanitizer == 'MSan' && '--with-assertions' || '--with-pydebug' }}
107125
${{ inputs.sanitizer == 'TSan' && '--with-openssl="$OPENSSL_DIR" --with-openssl-rpath=auto' || '' }}
108126
${{ inputs.free-threading && '--disable-gil' || '' }}
109127
- name: Build CPython

‎Doc/using/configure.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1015,6 +1015,10 @@ Debug options
10151015

10161016
Enable MemorySanitizer allocation error detector, ``msan`` (default is no).
10171017

1018+
MSan reports false positives for memory initialized by libraries that are
1019+
not built with MSan, so either build all dependencies with MSan or disable
1020+
the extension modules that use them in :file:`Modules/Setup.local`.
1021+
10181022
.. versionadded:: 3.6
10191023

10201024
.. option:: --with-undefined-behavior-sanitizer

‎Include/internal/pycore_interp_structs.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,7 @@ struct gc_old_stats_buffer {
219219
struct gc_stats {
220220
struct gc_young_stats_buffer young;
221221
struct gc_old_stats_buffer old[2];
222+
uint32_t update_seq;
222223
};
223224

224225
struct _gc_runtime_state {

‎Include/pyport.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -558,6 +558,7 @@ extern "C" {
558558
# define _Py_MEMORY_SANITIZER
559559
# define _Py_NO_SANITIZE_MEMORY __attribute__((no_sanitize_memory))
560560
# define _Py_MSAN_UNPOISON(PTR, SIZE) (__msan_unpoison(PTR, SIZE))
561+
# define _Py_MSAN_UNPOISON_STRING(STR) (__msan_unpoison_string(STR))
561562
# endif
562563
# endif
563564
# if __has_feature(address_sanitizer)
@@ -599,6 +600,9 @@ extern "C" {
599600
#ifndef _Py_MSAN_UNPOISON
600601
# define _Py_MSAN_UNPOISON(PTR, SIZE)
601602
#endif
603+
#ifndef _Py_MSAN_UNPOISON_STRING
604+
# define _Py_MSAN_UNPOISON_STRING(STR)
605+
#endif
602606

603607
/* AIX has __bool__ redefined in it's system header file. */
604608
#if defined(_AIX) && defined(__bool__)

‎Lib/asyncio/tools.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@ def __init__(
2727
# ─── indexing helpers ───────────────────────────────────────────
2828
def _format_stack_entry(elem: str|FrameInfo) -> str:
2929
if not isinstance(elem, str):
30+
if elem.location is None:
31+
if elem.filename in ("", "~"):
32+
return f"{elem.funcname}"
33+
return f"{elem.funcname} {elem.filename}"
3034
if elem.location.lineno == 0 and elem.filename == "":
3135
return f"{elem.funcname}"
3236
else:
@@ -190,8 +194,7 @@ def build_task_table(result):
190194
# Build coroutine stack string
191195
frames = [frame for coro in task_info.coroutine_stack
192196
for frame in coro.call_stack]
193-
coro_stack = " -> ".join(_format_stack_entry(x).split(" ")[0]
194-
for x in frames)
197+
coro_stack = " -> ".join(x.funcname for x in frames)
195198

196199
# Handle tasks with no awaiters
197200
if not task_info.awaited_by:
@@ -202,8 +205,7 @@ def build_task_table(result):
202205
# Handle tasks with awaiters
203206
for coro_info in task_info.awaited_by:
204207
parent_id = coro_info.task_name
205-
awaiter_frames = [_format_stack_entry(x).split(" ")[0]
206-
for x in coro_info.call_stack]
208+
awaiter_frames = [x.funcname for x in coro_info.call_stack]
207209
awaiter_chain = " -> ".join(awaiter_frames)
208210
awaiter_name = id2name.get(parent_id, "Unknown")
209211
parent_id_str = (hex(parent_id) if isinstance(parent_id, int)

‎Lib/profiling/sampling/_sync_coordinator.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,11 @@ def _execute_script(script_path: str, script_args: List[str], cwd: str) -> None:
168168
if not os.path.isfile(script_path):
169169
raise TargetError(f"Script not found: {script_path}")
170170

171+
script_dir = os.path.dirname(os.path.realpath(script_path))
172+
if script_dir in sys.path:
173+
sys.path.remove(script_dir)
174+
sys.path.insert(0, script_dir)
175+
171176
# Replace sys.argv to match original script call
172177
sys.argv = [script_path] + script_args
173178

‎Lib/profiling/sampling/binary_collector.py‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
"""Thin Python wrapper around C binary writer for profiling data."""
22

3+
import sys
34
import time
45

56
import _remote_debugging
@@ -81,6 +82,7 @@ def __init__(self, filename, sample_interval_usec, *, skip_idle=False,
8182
self.filename = filename
8283
self.sample_interval_usec = sample_interval_usec
8384
self.skip_idle = skip_idle
85+
self.running = True
8486

8587
compression_type = _resolve_compression(compression)
8688
start_time_us = int(time.monotonic() * 1_000_000)
@@ -102,9 +104,19 @@ def collect(self, stack_frames, timestamp_us=None):
102104
timestamp_us: Optional timestamp in microseconds. If not provided,
103105
uses time.monotonic() to generate one.
104106
"""
107+
if not self.running:
108+
return
105109
if timestamp_us is None:
106110
timestamp_us = int(time.monotonic() * 1_000_000)
107-
self._writer.write_sample(stack_frames, timestamp_us)
111+
try:
112+
self._writer.write_sample(stack_frames, timestamp_us)
113+
except OverflowError as e:
114+
if not self._writer.limit_reached:
115+
raise
116+
self.running = False
117+
print(f"Warning: {e}; stopping early and keeping the data "
118+
"collected so far.",
119+
file=sys.stderr)
108120

109121
def collect_failed_sample(self):
110122
"""Record a failed sample attempt (no-op for binary format)."""
@@ -143,9 +155,5 @@ def __enter__(self):
143155
return self
144156

145157
def __exit__(self, exc_type, exc_val, exc_tb):
146-
"""Context manager exit - finalize unless there was an error."""
147-
if exc_type is None:
148-
self._writer.finalize()
149-
else:
150-
self._writer.close()
151-
return False
158+
"""Finalize if the writer can still produce a valid file."""
159+
return self._writer.__exit__(exc_type, exc_val, exc_tb)

‎Lib/profiling/sampling/heatmap_collector.py‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -785,14 +785,14 @@ def _generate_file_html(self, output_path: Path, filename: str,
785785
line_counts: Dict[int, int], self_counts: Dict[int, int],
786786
file_stat: FileStats):
787787
"""Generate HTML for a single source file with heatmap coloring."""
788-
# Read source file
788+
source_lines = [f"# Source file not available: {filename}"]
789789
try:
790-
source_lines = Path(filename).read_text(encoding='utf-8', errors='replace').splitlines()
791-
except (IOError, OSError) as e:
792-
if not (filename.startswith('<') or filename.startswith('[') or
793-
filename in ('~', '...', '.') or len(filename) < 2):
794-
print(f"Warning: Could not read source file {filename}: {e}")
795-
source_lines = [f"# Source file not available: {filename}"]
790+
path = Path(filename)
791+
if path.is_file():
792+
source_lines = path.read_text(
793+
encoding='utf-8', errors='replace').splitlines()
794+
except (IOError, OSError):
795+
pass
796796

797797
# Generate HTML for each line
798798
max_samples = max(line_counts.values()) if line_counts else 1

‎Lib/profiling/sampling/stack_collector.py‎

Lines changed: 42 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -60,13 +60,18 @@ def export(self, filename):
6060

6161
lines.sort(key=lambda x: (-x[1], x[0]))
6262

63-
with open(filename, "w") as f:
63+
with open(filename, "w",
64+
encoding="utf-8", errors="surrogatepass") as f:
6465
for stack, count in lines:
6566
f.write(f"{stack} {count}\n")
6667
print(f"Collapsed stack output written to {filename}")
6768
return True
6869

6970

71+
# Allow for tree conversion and the dict/list frames in the Python JSON encoder.
72+
_FLAMEGRAPH_RECURSION_MARGIN = 6000
73+
74+
7075
class FlamegraphCollector(StackTraceCollector):
7176
def __init__(self, *args, **kwargs):
7277
super().__init__(*args, **kwargs)
@@ -165,34 +170,41 @@ def set_mode(self, mode):
165170
self.stats["mode"] = mode
166171

167172
def export(self, filename):
168-
flamegraph_data = self._convert_to_flamegraph_format()
169-
170-
# Debug output with string table statistics
171-
num_functions = len(flamegraph_data.get("children", []))
172-
total_time = flamegraph_data.get("value", 0)
173-
string_count = len(self._string_table)
174-
s1 = "" if num_functions == 1 else "s"
175-
s2 = "" if total_time == 1 else "s"
176-
s3 = "" if string_count == 1 else "s"
177-
print(
178-
f"Flamegraph data: {num_functions} root function{s1}, "
179-
f"{total_time} total sample{s2}, "
180-
f"{string_count} unique string{s3}"
181-
)
182-
183-
if num_functions == 0:
173+
# Converting the call tree recurses to the sampled stack depth.
174+
old_limit = sys.getrecursionlimit()
175+
sys.setrecursionlimit(old_limit + _FLAMEGRAPH_RECURSION_MARGIN)
176+
try:
177+
flamegraph_data = self._convert_to_flamegraph_format()
178+
179+
# Debug output with string table statistics
180+
num_functions = len(flamegraph_data.get("children", []))
181+
total_time = flamegraph_data.get("value", 0)
182+
string_count = len(self._string_table)
183+
s1 = "" if num_functions == 1 else "s"
184+
s2 = "" if total_time == 1 else "s"
185+
s3 = "" if string_count == 1 else "s"
184186
print(
185-
"Warning: No functions found in profiling data. Check if sampling captured any data."
187+
f"Flamegraph data: {num_functions} root function{s1}, "
188+
f"{total_time} total sample{s2}, "
189+
f"{string_count} unique string{s3}"
186190
)
187-
return False
188191

189-
html_content = self._create_flamegraph_html(flamegraph_data)
192+
if num_functions == 0:
193+
print(
194+
"Warning: No functions found in profiling data. "
195+
"Check if sampling captured any data."
196+
)
197+
return False
190198

191-
with open(filename, "w", encoding="utf-8") as f:
192-
f.write(html_content)
199+
html_content = self._create_flamegraph_html(flamegraph_data)
193200

194-
print(f"Flamegraph saved to: {filename}")
195-
return True
201+
with open(filename, "w", encoding="utf-8") as f:
202+
f.write(html_content)
203+
204+
print(f"Flamegraph saved to: {filename}")
205+
return True
206+
finally:
207+
sys.setrecursionlimit(old_limit)
196208

197209
@staticmethod
198210
@functools.lru_cache(maxsize=None)
@@ -486,7 +498,12 @@ def _get_source_lines(self, func):
486498
return None
487499

488500
def _create_flamegraph_html(self, data):
489-
data_json = json.dumps(data)
501+
try:
502+
data_json = json.dumps(data)
503+
except RecursionError:
504+
# The C encoder can exhaust the C stack independently of the
505+
# Python recursion limit. iterencode() uses the Python encoder.
506+
data_json = "".join(json.JSONEncoder().iterencode(data))
490507

491508
template_dir = importlib.resources.files(__package__)
492509
vendor_dir = template_dir / "_vendor"

0 commit comments

Comments
 (0)