Skip to content

fix(daemon): recover stale service PID files safely - #1469

Merged
wizzomafizzo merged 5 commits into
mainfrom
fix/watchdog-pid-reuse
Sep 10, 2026
Merged

wizzomafizzo merged 5 commits into
mainfrom
fix/watchdog-pid-reuse

Conversation

@wizzomafizzo

@wizzomafizzo wizzomafizzo commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

  • Recover a confirmed foreign/reused PID during service Start, under the existing start gate, without signaling that process or changing PID-file format.
  • Keep unknown/inaccessible process identity fail-closed. Stop and Restart still refuse unrelated PIDs.
  • Distinguish cached service executables and shell script arguments from unrelated processes merely mentioning a service file; retain deleted-executable identity across binary replacement.
  • Cover stale-PID recovery with competing starts, healthy-service idempotence, unrelated-process survival and unknown identity.

Ref ZaparooProject/Main_MiSTer#24. This is the Core-owned cleanup companion to Main's watchdog identity check; Core should ship before relying on automatic PID-reuse recovery.

Validated with task test, task lint, task deadlock, task cross-lint:all and task vulncheck. Original code failed the recovery regression. Vulnerability scan reports no called vulnerable code; it notes uncalled dependency advisories. No device service, deployment or release changes. Work isolated from the owner's active checkout.

Summary by CodeRabbit

  • Bug Fixes
    • Daemon startup now recovers from stale PID files that reference unrelated running processes.
    • Unrelated processes are no longer signaled during PID-file recovery.
    • Process identity checks are stricter, improving reliability when validating service processes.
    • Repeated or concurrent starts continue to maintain a single service process.

@coderabbitai

coderabbitai Bot commented Sep 9, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 3bff6838-0243-42e8-9073-b324193d6944

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: fca89297-245d-4ca2-aa3c-75340286b7ac

📥 Commits

Reviewing files that changed from the base of the PR and between d346a2e and 3cf1dfa.

📒 Files selected for processing (2)
  • pkg/service/daemon/daemon.go
  • pkg/service/daemon/daemon_test.go

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


📝 Walkthrough

Walkthrough

The daemon now uses typed PID mismatch errors and explicit process identity checks. Startup removes a confirmed stale PID file without signaling the unrelated process. Tests cover concurrent recovery, shell identity validation, and unavailable process identities.

Changes

Daemon PID handling

Layer / File(s) Summary
Process identity validation
pkg/service/daemon/daemon.go, pkg/service/daemon/daemon_test.go
Process identity checks now return errors, handle deleted executables, and restrict shell-backed matches to valid service interpreter arguments. Tests cover rejected arguments and unavailable identities.
Stale PID recovery during startup
pkg/service/daemon/daemon.go, pkg/service/daemon/daemon_test.go
Start removes an unchanged PID file after confirming that its PID belongs to another process. Concurrent starts publish one service PID and do not signal the unrelated process.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Start
  participant Running
  participant serviceProcessIdentity
  participant PIDFile
  Start->>Running: Check recorded PID
  Running->>serviceProcessIdentity: Identify recorded PID
  serviceProcessIdentity-->>Running: Confirm foreign process
  Running-->>Start: Return servicePIDMismatchError
  Start->>PIDFile: Re-read and remove unchanged stale PID
  Start-->>Start: Launch service
Loading

Merge Risk: ⚪ Minimal · up to 3cf1d

Daemon startup now safely replaces confirmed stale PID files without signaling unrelated processes, while preserving fail-closed handling for unknown identities. The covered recovery and idempotence behavior is ready to merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: safely recovering stale service PID files during daemon startup.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.59574% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/service/daemon/daemon.go 83.72% 7 Missing ⚠️
cmd/batocera/main.go 0.00% 1 Missing ⚠️
cmd/libreelec/main.go 0.00% 1 Missing ⚠️
cmd/mister/main.go 0.00% 1 Missing ⚠️
cmd/mistex/main.go 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

The kernel places the interpreted script after any shebang option, so
"#!/bin/sh -e" puts the option at argv[1] and the script at argv[2].
Reading only argv[1] reported a live shell-backed service as foreign,
and Start now removes a foreign PID file, so the wrapper would have
started a second service against the running one.

Scan for the first argument that is not a flag instead, and keep -c and
-s rejected so a path in a command string stays data rather than
identity. Resolved interpreters on buildroot images are also ash, ksh or
zsh rather than a busybox symlink, so accept those.
Every platform wrapper checks Running before auto-starting and returns
on any error, so a reused PID failed with "error checking service
status" and Start was never called. The recovery this branch adds was
unreachable from the only entry point users have.

Treat the PID-file conflict as not running and let Start clear it under
the start gate. Other Running errors, including unknown process
identity, still abort as before.
All four platform wrappers repeated the same "is it running, and is this
the recoverable stale PID file" predicate inline, in package main where
nothing tests it. Move it behind RunningForAutoStart so the behaviour is
covered once, in the package that owns it.

Running still reports the conflict, so Stop and Restart keep refusing to
touch an unrelated process.
RunningForAutoStart tolerates exactly one error, the recoverable stale
PID conflict. Nothing asserted that it still surfaces the others, and
reporting an unreadable PID file as "not running" would start a second
service on top of whatever that file described.
@wizzomafizzo
wizzomafizzo merged commit 20c691f into main Sep 10, 2026
16 of 17 checks passed
@wizzomafizzo
wizzomafizzo deleted the fix/watchdog-pid-reuse branch September 10, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant