Skip to content

List endpoints collapse under a single 25s poller: 1000-row operational view builds saturate the API (supports #334) #335

Open

Description

@tom-sapletta-com

Evidence (2026-11-02 15:0x CET)

Container subactor-planfile-coding-recovery-v2 (serving 127.0.0.1:18769): 204% CPU, 8.3 GiB; GET /tickets?...limit=1000&view=operational every ~25s from one consumer (subactor/platform hr-control, see subactor/platform#690) drives list endpoints into 6s+ timeouts for EVERYONE — fleet campaigns fail planning (planfile_unavailable:TimeoutError), ticket creation returns 500 intermittently.

Restart only partially helped (2-6s responses resume immediately): the load is chronic, the view rebuild is O(thousands) per request.

Ask (supports open PR #334 'serve the last complete index')

Activity

  1. tom-sapletta-com commented on Nov 2, 2026

    @tom-sapletta-com
    ContributorAuthor

    Additional evidence (2026-11-02 15:2x): this is a memory accumulation, not just CPU. Container ran 46h: 8.3 GiB → 12.8 GiB RSS with list endpoints at 6s timeouts and 204% CPU. After restart: 3.2 GiB and 0.3-3s responses — degradation resumes within minutes under the same load (hr-control polling unchanged, see subactor/platform#690). Also observed a full ASGI traceback with 500 on POST /tickets during load. Mitigation in place: a host watchdog restarts the container when list probes time out twice (every 10 min) — removed once a real fix lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions