Repository navigation
feat(stovepipe): define queue status API and RFC #854
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
25f1cac
feat(stovepipe): [DRAFT] define queue status API and RFC
mnoah1 3b1176b
docs(stovepipe): explain queue status and commit boundaries
mnoah1 320d233
docs(stovepipe): distinguish enablement from future pause controls
mnoah1 c17f573
docs(stovepipe): simplify queue status contract comments
mnoah1 393d55f
refactor(stovepipe): name queue enablement status explicitly
mnoah1 8c55c7e
docs(stovepipe): match unknown enablement enum comment
mnoah1 91ac1ae
fix(stovepipe): regenerate TypeScript queue status bindings
mnoah1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| # Queue enablement and status | ||
|
|
||
| Stovepipe needs to answer two questions: is a queue enabled now, and was it enabled for a particular commit? Each answer includes the commit at which that state took effect. | ||
|
|
||
| ## Why we need this | ||
|
|
||
| A client needs to know whether Stovepipe validation is expected for a commit. Today's setting is not enough: a queue can be disabled now even though an earlier commit was covered, or enabled before its first validation finishes. | ||
|
|
||
| The existing information cannot answer this reliably: | ||
|
|
||
| - A configuration flag describes the desired setting, and the current queue row holds the latest applied state. Neither alone retains earlier commit intervals. | ||
| - Request history describes validation attempts. Batching leaves intermediate commits without their own requests, so a missing request does not imply disabled. | ||
| - Last-green and validation results describe outcomes, rather than whether validation was expected. | ||
|
|
||
| A dedicated lookup needs durable enablement history with commit boundaries, independent of validation work. | ||
|
|
||
| ## Enablement and pause | ||
|
|
||
| Enablement defines which commits are covered by Stovepipe. Pause is a temporary operational control over whether a queue accepts or advances work. An enabled queue can be paused and still owe validation results for its covered commits; enabled does not promise that work is progressing or that results are green. | ||
|
|
||
| For example, if a queue is enabled at A and later paused while validating C, the status for C remains enabled from A. Unpausing resumes work without creating a new enablement boundary. Disabling at D instead changes coverage from D onward, while preserving the answers for earlier commits. | ||
|
|
||
| Future pause management should follow the [queue pause RFC](../submitqueue/queue-pause-and-unpause.md), which distinguishes refusing new requests from freezing accepted work. That RFC covers SubmitQueue; Stovepipe's ingestion and processing behavior still needs its own design. | ||
|
|
||
| Pause can use the same internal-state, summary and history pattern described below, with separate operational fields and history events. Pause and unpause must not advance the enablement version or change commit boundaries. Keep pause out of the enablement enum, and expose it as current operational status rather than mixing it into a commit-specific enablement answer. A future administration API can manage both controls independently; this lookup remains read-only. | ||
|
|
||
| ## What the endpoint returns | ||
|
|
||
| `GetQueueStatus` takes a queue name and an optional `change_uri`. Omitting the commit selects the latest queue summary; supplying it selects the enablement covering that commit. The response contains an `enablement` object of type `QueueEnablementStatus`: | ||
|
|
||
| | Field | Meaning | | ||
| |---|---| | ||
| | `state` | `ENABLED` or `DISABLED`. | | ||
| | `effective_from_commit_uri` | First covered commit, inclusive. | | ||
| | `version` | Sequence number that increases for each enablement change. | | ||
| | `changed_at_ms` | Recorded change timestamp in Unix milliseconds, distinct from commit time. | | ||
|
|
||
| The boundary, version and timestamp describe enablement only. Future operational fields, such as pause status, belong beside this object and can change independently. | ||
|
|
||
| See the [proto](../../../api/stovepipe/proto/stovepipe.proto) for the wire contract. | ||
|
|
||
| Suppose commits arrive in order A through G, with enablement changes at A, D and G: | ||
|
|
||
| | Lookup | State | Effective from | | ||
| |---|---|---| | ||
| | No commit supplied | Enabled | G | | ||
| | C | Enabled | A | | ||
| | D or F | Disabled | D | | ||
| | G | Enabled | G | | ||
|
|
||
| C and F need no validation requests. Their ancestry relative to the boundaries determines the answer. | ||
|
|
||
| Registration starts a queue as disabled version 1, with no boundary; this covers commits before its first enablement. Missing state is an error, not disabled. If changes share a boundary, the highest version wins. Versions order changes; timestamps do not. | ||
|
|
||
| ## Internal state, summary and history | ||
|
|
||
| Follow the model used for request summaries and history, while keeping the internal queue authoritative: | ||
|
|
||
| | Record | Purpose | | ||
| |---|---| | ||
| | Existing `queue` row | Current enablement alongside pipeline coordination fields. | | ||
| | `queue_log` | Immutable applied changes, keyed by queue and enablement version. | | ||
| | `queue_summary` | Current read view derived from those changes. | | ||
|
|
||
| The queue row and summary each have a storage version for optimistic locking. The API exposes a separate enablement version, which advances only for enabled/disabled changes. Routine pipeline updates create no enablement history entries or gaps. | ||
|
|
||
| The transition controller captures the queue's tip and a timestamp once when preparing a change. Retries keep those values; requesting the existing state is a no-op. Application first finishes recording the previous state, then conditionally updates the queue and records the new state through `queuelog.Materializer.PersistLog`. | ||
|
|
||
| History is written before its summary. If either write fails after the queue update, the queue retains the missing change. Initialization and transition retries repair the read models before another enablement change can replace it. Repeating an identical change is safe, and older changes cannot move the summary backward. Concurrent pipeline updates preserve their coordination fields; competing enablement decisions are rejected as stale. | ||
|
|
||
| ## Commit lookup and consistency | ||
|
|
||
| Lookup reads one summary and its matching history entry, and uses that version throughout the call. It never writes or repairs state. “Current” therefore means the latest state in the read models, which may lag the internal queue until a writer retries. | ||
|
|
||
| For a supplied commit, the existing source-control integration verifies that the commit and latest boundary belong to the queue's current lineage. The controller walks changes newest-first and selects the first boundary that is an ancestor of, or equal to, the commit. Before walking backward, it verifies that the commit precedes the boundary; incomparable commits and unsupported history rewrites fail instead of becoming disabled. This resolves intermediate commits without a commit index or local checkout. Current-only lookup needs no source-control call. | ||
|
|
||
| | Failure | RPC error | | ||
| |---|---| | ||
| | Malformed selector | `INVALID_ARGUMENT` | | ||
| | No queue summary | `NOT_FOUND` | | ||
| | Unresolved commit or unsupported lineage | `FAILED_PRECONDITION` | | ||
| | Missing or inconsistent history | `INTERNAL` | | ||
|
|
||
| Dependency and cancellation errors retain their transport classification. An unresolved commit fails the whole call. | ||
|
|
||
| ## Scope | ||
|
|
||
| This stack records enablement and exposes status lookup. It does not enforce ingestion or execution controls. Configuration freshness, administration, pause enforcement and public history access remain separate work. Validation results and last-green stay in their existing APIs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| // Copyright (c) 2026 Uber Technologies, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package handler | ||
|
|
||
| import ( | ||
| "context" | ||
| "testing" | ||
|
|
||
| "github.com/stretchr/testify/require" | ||
| pb "github.com/uber/submitqueue/api/stovepipe/protopb" | ||
| "google.golang.org/grpc/codes" | ||
| "google.golang.org/grpc/status" | ||
| ) | ||
|
|
||
| func TestQueueStatusContractIsUnimplemented(t *testing.T) { | ||
| server := &StovepipeServer{} | ||
| response, err := server.GetQueueStatus(context.Background(), &pb.GetQueueStatusRequest{Queue: "repo/main"}) | ||
| require.Nil(t, response) | ||
| require.Equal(t, codes.Unimplemented, status.Code(err)) | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this can be any commit and not just the ones we processed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes that's correct.