feat: declare Serialized in the 004 and 008 bench specs - #2262
Merged
Conversation
The harnesses emit provider="serialized" but the leaderboards are driven by the explicit providers block in each spec, so the series never rendered. Adds the block for both benches, same query shape as the incumbents. Validated with SpecSchema.safeParse across all 218 specs; every formula stays under the 240-char cap that silently drops a spec. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2261. The harnesses emit
provider="serialized"and Prometheus carries the series, but both leaderboards are driven by the explicitproviders:block in the spec YAML, so nothing rendered.Adds the block for both benches with the same query shape as the incumbents.
Live check against Prom on
ocb-par-main:Validated with
SpecSchema.safeParseacross all 218 specs:ALL SPECS VALID, and every formula stays under the 240-char cap that silently drops a whole spec.Bench 004 will stay empty for Serialized until the metadata-coverage harness is producing again — see the deploy note below.
Separate pre-existing outage, not introduced here
Bench 004 has been dark for 4 to 7 days (
increase(metadata_coverage_checks_total[96h]) == 0for mobula, codex and jupiter alike). Root cause is the webshare proxy the container routes through: it answers000after a 15s timeout for every target, while the same request direct returns200in 55ms. Two other containers (ocb-perp-cohort-stats,ocb-gas-estimation) sit behind the same dead proxy.Mitigated on the box by adding
pulse-v2-api.mobula.ioandapi.mobula.ioto that container'sNO_PROXYso the token feeder and the Mobula REST calls dial direct. The feeder now subscribes, but no launchpad tokens have been queued yet, so the bench is not confirmed recovered. The proxy subscription itself still needs a decision: renew it, or drop the proxy from these three containers.🤖 Generated with Claude Code
https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA