Followup for #1119 - #1126
Open
spirali wants to merge 3 commits into
Open
Conversation
The scheduler's per-worker placement solve (run_scheduling_solver) was unbounded: with many distinct task resource shapes across many workers, HiGHS's search can take minutes to hours before returning, freezing the single-threaded server for the entire duration (no heartbeats, no RPCs, no other scheduling). See AI-QChem/QE-NO#6. Add HighsSolver::solve_bounded(), used only by the scheduler's placement solve: - mip_rel_gap (default 10%, HQ_SCHEDULER_MIP_REL_GAP): accept a solution once HiGHS has proven it within this fraction of optimal. Task resource requests are themselves estimates, so exact optimality is false precision; benchmarking showed this alone takes a pathological many-shape instance from a 20s+ timeout to under 2s. - time_limit (default 5s, HQ_SCHEDULER_MIP_TIME_LIMIT_MS): a hard wall-clock backstop. A ReachedTimeLimit result is still dispatched if HiGHS reports a real feasible incumbent (never violates a constraint, just not proven within the gap); otherwise the pass is skipped and the ~20ms scheduler debounce retries. The existing exact solve() is kept unchanged and is still what worker/resources/groups.rs's NUMA/socket resource allocator uses -- it shares the same underlying LpSolver but needs a guaranteed-exact feasible allocation, not a good-enough one. Discovered via a regression: applying mip_rel_gap globally broke test_complex_coupling1, an allocator test with no relation to the scheduler, because both call sites shared one solve() prior to this change. Unit tests default (cfg(test) in config.rs) to exact, unhurried solving, matching every existing test's exact-count assertions; new tests opt into the tuned config via a thread-local override (with_test_solver_config) rather than process-global env vars, so they can't race with unrelated tests running concurrently on other threads. New tests: - test_schedule_many_distinct_shapes_stays_bounded: regression test for the original hang, at the shape/worker scale that reproduces it. - test_schedule_bounded_dispatches_feasible_incumbent_on_time_limit: a too-short time limit still dispatches a partial feasible solution. - test_schedule_bounded_infeasible_returns_none_safely: a genuinely infeasible request stays unassigned, not a panic. - test_allocator_stays_exact_regardless_of_scheduler_gap_tuning: guards the allocator/scheduler solve split found above.
Per review on PR #1119: bound the scheduler's MILP solve by wall-clock time only, not by relative gap, since some downstream logic assumes an optimal placement. The time limit moves from a thread-local/env-var read into SchedulerConfig::mip_time_limit, threaded from `hq server start --scheduler-time-limit` through ServerConfig, server_start, CoreRef, and Core down to the solve call site. solve_bounded now returns (Solution, is_optimal) instead of (Solution, objective_value). SchedulingSolution gains an is_optimal field so downstream consumers (e.g. proactive filling) can react to a non-optimal placement.
3 tasks
spirali
force-pushed
the
scheduler-solve-timeout
branch
from
July 29, 2026 10:41
7f8d871 to
1371faa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is followup for #1119 (I do not have permission to update the @MauriceDHanisch branch).
At the end we do not need to block prefill when non optimal solution is not find but have to block autoalloc query.