Characterize stalled backlog reads after reload threshold changes

[taylan-oai]

Summary

Characterize a missed priority-reader wakeup when matching.getTasksReloadAt changes above the loaded-task count, so a follow-up fix has a deterministic reproduction.

Problem

priTaskReader.completeTask requests another persistence read only when the remaining loaded count equals the current reload threshold. The pump can discard its previous notification while the buffer exceeds the old threshold. Raising the setting above the remaining count then allows successful completions to drain the buffer without requesting another read.

For example, the test persists six tasks, loads four with threshold one, and completes two. Changing the threshold to three and completing the remaining loaded tasks leaves two persisted tasks unread, with an empty notification channel. Removing a temporary override to restore three produces the same result. Worker polls do not signal this pump, and the periodic metadata sync does not read tasks.

Approach

Add a comment at the equality check and a passing characterization test using testing/synctest, the real dynamic-config getter, active reader loops, and taskQueueDB backed by the existing in-memory test manager. It covers unchanged, increased, restored, and decreased thresholds for both priority and fair readers. The stalled priority cases also demonstrate recovery through a subsequent persisted write and its reader notification.

These assertions document current behavior; a future fix should change the stalled cases to require progress without another write. Production behavior is unchanged.

Two contained options for a follow-up, neither implemented or validated here:

1. Signal on completions at or below the threshold, retaining the one-slot notification channel. This handles completions after a transition, but needs an assessment of extra empty reads and does not react to a config change by itself.
2. Subscribe to reload-threshold changes and notify the affected readers. This can react without another completion, but needs subscription cleanup and synchronization with reader creation and teardown.

Validation

• go test -p 4 -tags test_dep ./service/matching -run '^TestReaderReloadThresholdChange$' -count=100 passed all eight cases on each run.
• Initial focused verbose run passed.
• go test -p 4 -tags test_dep ./service/matching -run '^TestReaderReloadThresholdChange$' -race -count=20 passed.
• Related priority/fair backlog-manager suites passed.
• make lint-code-fast passed with the repository lint config, GOLANGCI_LINT_BASE_REV=1f7c7b0faba0270ea5cbc3798bbd574215d4a43b, and GOLANGCI_LINT_FIX=false; no new issues.

The test exercises reader scheduling and the persistence wrapper; it does not use an external datastore, worker RPCs, or a production configuration client.

References

PR #10091 changes cold-start priority ordering. Its current diff leaves the completion-time equality check unchanged. PR #7429 introduced the dynamic reload setting.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}