Repository navigation
ci: upload test reports when a watchdog-dumped job times out - #880
Conversation
A test JVM that hangs runs the job into its 180-minute timeout, which ends the job as cancelled, not failed. The gdb watchdog's thread dump is then copied into test-reports by prepare_reports.sh (an always() step), but the "Upload test reports" step only runs on failure(), so the dump is lost in exactly the case the watchdog exists for. Also upload the test reports on cancellation when the watchdog left a dump (build/logs/gdb-watchdog.log), in the two jobs that have a watchdog. Runs cancelled for other reasons (e.g. superseded by a newer push) still upload nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
More details
The upload conditions preserve failure uploads and allow cancelled jobs to upload reports when a watchdog dump exists. Report preparation already runs after cancellation, so the collected dump remains available for upload.
🤖 Bits Code Review · Commit 80f1cde
CI Test ResultsRun: #39615128483 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 31 | Failed: 0 | Cancelled: 1 Updated: 2026-10-21 17:17:20 UTC |
The watchdog's gdb has never produced a dump on the GitHub runners: Ubuntu's yama ptrace_scope=1 only allows attaching to descendants, so every attach failed with "ptrace: Inappropriate ioctl for device". Run it with sudo, as the runners allow without a password. Before attaching, send the JVM SIGQUIT as well, for the Java-level picture (stacks, monitors and their owners): HotSpot prints it to stdout, J9 writes a javacore file, which prepare_reports.sh already collects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
The
Upload test reportsstep in the glibc amd64 and glibc aarch64 test jobs now also runs when the job was cancelled and the gdb watchdog left a thread dump (build/logs/gdb-watchdog.log):Why
The gdb watchdog (#760) dumps the native threads of a hung test JVM after 20 minutes, so that the dump "lands in the test-reports artifact" before the 180-minute job timeout. But when the timeout fires, GitHub ends the job as cancelled, not failed.
prepare_reports.sh(analways()step) still copies the dump intotest-reports/, but the upload step isif: failure(), so it is skipped and the dump is lost. That is exactly the case the watchdog exists for.Seen on #868:
test-linux-glibc-amd64 (8-ibm, debug, regular)hung in the regular suite, most likely the IBM SDK 8 JVMTI deadlock from #760, which so far had been seen only in the slow suite. The watchdog fired at 20 minutes, the job timed out at 180 minutes, and no artifact was uploaded. The job log confirms thatalways()steps run after the timeout:Prepare reportsran, andUpload CI outcomeran but found no file.The watchdog itself could not attach
This PR's own CI run hit the same 8-ibm hang, and the new upload worked. But the uploaded
gdb-watchdog.logheld onlyptrace: Inappropriate ioctl for devicefor both JVMs. Ubuntu's yamaptrace_scope=1only lets a process attach to its own descendants, so the watchdog has never produced a dump on these runners. The second commit fixes that:sudo, which these runners allow without a password (the ASan path already usessudo sysctl).SIGQUITfor Java-level stacks and lock owners. HotSpot prints them to stdout; J9 writes a javacore file, whichprepare_reports.shalready collects.What the uploaded
test-raw.logdid show: the hang is inDatadogSettingsTest.testRecordDatadogSetting(cpu=1ms, IBM SDK 8, CPU engineasgct), between the profiler start (6 libraries patched) and the stop'sRestore libraries. That's the same test that hung on #868.Scope
ci.yml'scancel-in-progress, from uploading anything, unless the watchdog had already fired.🤖 Generated with Claude Code