Repository navigation
fix(broker): a lease on several cards waits for its window with no deadline - #292
Merged
Merged
Conversation
A night job split across two cards and started in the evening gave up after berthd's default wait, while the same job on one card waits for its hours. One test drives AcquireAll with a pinned clock, the other runs berth run against a two-card berthd. Both fail here.
…adline Acquire and AcquireAll each worked out how long a request waits, and only Acquire knew that a window with no --wait means no deadline. waitFor now holds the rule, and both call it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A lease window (#290) lets a job wait for set hours. On one card, a windowed run with no
--waitwaits for its window with no deadline. On several cards (berth run --gpu gpu0,gpu1 --window 23:00-07:00), the same run gave up after berthd's default wait of 120 s and exited refused.Cause
AcquireandAcquireAlleach worked out the wait. OnlyAcquirehad the rule that a window with no--waitmeans no deadline.Fix
Broker.waitForholds the rule, and bothAcquireandAcquireAllcall it. No other code readsDefaultWait.Proof
TestWindowWaiterOnSeveralCardsOutlastsTheDefaultWait(broker): a lease on two cards with a pinned evening clock and a 50 ms default wait is still waiting after 200 ms, and it gets both cards once the clock moves inside the window.TestRunOnSeveralCardsWaitsPastTheDefaultForItsWindow(runner):berth runongpu0,gpu1against a two-card berthd with a 200 ms default wait is still queued on both cards after 1 s.nextwithout the fix. The runner test's output before the fix:refused: gave up after 200ms: outside its window 09:48-10:48; opens in 1h.go test ./..., the Windows build and vet, golangci-lint. The new tests also pass under-race -count=3.