Skip to content

feat(diagnostics, ticket-136): detect host anomalies beyond cache cleanup - #179

Merged
tom-sapletta-com merged 1 commit into
mainfrom
ticket/136-release-2-4-0-host-anomalies
Nov 16, 2026
Merged

tom-sapletta-com merged 1 commit into
mainfrom
ticket/136-release-2-4-0-host-anomalies

Conversation

@tom-sapletta-com

Copy link
Copy Markdown
Contributor

Why

On 2026-11-16 a manual clean-up of a shared build host found causes of load and disk pressure that fixos auto and fixos cleanup did not report: a service restarting in a loop (16 881 restarts), Docker container logs without rotation (6 GiB), 26 GiB of untouched backups in ~/.local/state, a 22 GiB JetBrains index with no IDE running, about 420 zombie processes and 470 leaked identical helper processes.

What

New command fixos anomalies backed by fixos/diagnostics/host_anomalies.py (stdlib only, command runner and /proc root injectable):

Detector Reports
restart-loop system and user units with a high NRestarts, flagged high when in auto-restart
container-logs missing log-opts.max-size in daemon.json, and json logs above 100 MiB
stale-state top-most directory under ~/.local/state / ~/.local/share above the size threshold with nothing modified for N days
ide-index JetBrains index directories; protected while that IDE runs
processes zombies per parent; groups of long-idle processes with an identical command line

System-level remedies stay suggested commands. --remove PATH and --remove-safe delete only stale-state and idle ide-index paths that a fresh scan still reports, inside the home directory, never through a symlink or a protected directory, and write a receipt.

Verification

  • python -m pytest tests/unit/test_host_anomalies.py -q: 10 passed.
  • ruff check and ruff format --check pass for the three new files.
  • ./project/governance-check.sh: GOV-PASS.
  • Live host run found three restart loops that the manual pass had missed and the four zombie parents. A first version reported docker-proxy and containerd-shim as leaked groups; grouping by the complete command line removed those false positives (covered by a test).
  • Not verified here: the full test suite, and the stale-state detector's live output (its du pass over a 138 GiB state store was still running when this PR was opened).

Scope note

Reuses the unassigned ticket-136 allocation that blocked new allocation. No version bump: VERSION and pyproject.toml are outside the application workstream, so releasing this (and the already merged artifact reclaim) as 2.4.0 needs an integration ticket.

🤖 Generated with Claude Code

…anup

Add 'fixos anomalies': read-only detection of systemd restart loops,
container logs without rotation, large stale directories in tool state
stores, regenerable IDE indexes, and zombie or leaked identical processes.
Removal is limited to stale-state and idle IDE-index findings confirmed by
a fresh scan, inside the user's home, with a receipt.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@tom-sapletta-com
tom-sapletta-com merged commit 6d38f9b into main Nov 16, 2026
12 checks passed
@tom-sapletta-com
tom-sapletta-com deleted the ticket/136-release-2-4-0-host-anomalies branch November 16, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant