Skip to content

fix(controller): fill configured local runner capacity - #624

Closed
Bluu (Bluuok) wants to merge 1 commit into
microsoft:mainfrom
Bluuok:fix/local-pool-capacity
Closed

Bluu (Bluuok) wants to merge 1 commit into
microsoft:mainfrom
Bluuok:fix/local-pool-capacity

Conversation

@Bluuok

@Bluuok Bluu (Bluuok) commented Oct 3, 2026 •

Copy link
Copy Markdown

With local_runner.maximum_size=60 or 128, the local controller still requests only 50 queuing/running rollouts. If those first 50 stay running, the remaining configured worker slots cannot be used and later queued work remains invisible.

Size the query limit to at least the configured pool capacity, preserving the existing minimum/default of 50.

Validation on Ubuntu 24.04 / Python 3.12 with locked CPU dependencies:

  • python -m pytest -q tests/controller: 28 passed.
  • Parameterized two-cycle regressions cover capacities 25, 50, 60, and 128. Unchanged main fails the 60/128 cases with only 50 active records; the patch passes all four. API responses honor the requested limit and process spawning is mocked.
  • Ruff/format, Pyright, copyright headers, pre-commit, and diff whitespace passed.

No GPU training was run. This change is independent of terminal-process history cleanup in #618.

AI assistance was used to investigate, implement, and validate the fix. The final diff and regression behavior were reviewed before submission. The CLA check has passed; fork CI is awaiting maintainer approval.

@Bluuok
Bluu (Bluuok) marked this pull request as ready for review October 3, 2026 14:32
Copilot AI balanced review requested due to automatic review settings October 3, 2026 14:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The focused implementation correctly addresses capacity underutilization and includes representative regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Updates local reconciliation to query enough active rollouts to utilize the configured worker pool.

Changes:

  • Sets the query limit to at least the configured pool size while preserving the minimum of 50.
  • Adds regression coverage for pool sizes 25, 50, 60, and 128.
File Description
agentlightning/​controller/​local_reconciler.py Scales the rollout query limit with local capacity.
tests/​controller/​test_local_reconciler.py Verifies query limits and pool utilization.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants