Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions benchmarks/aggregator-head-lag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,13 @@ abstract: |
same event.

methodology:
- "Aggregators measured: Mobula, Codex, GeckoTerminal."
- "Aggregators measured: Mobula, Codex, GeckoTerminal, Serialized."
- "Chains: Base, BNB Chain, Solana, Robinhood Chain."
- "Regions: us-east, eu-west, sgp. Cross-region median reported in the headline."
- "Reference: archive nodes per chain, validated against block hashes."
- "Metric: gauge `head_lag_seconds`, sampled every 15 seconds. Aggregated over the 24-hour window using `quantile_over_time`."
- "Success rate: presence ratio. Share of expected sampling slots where a value was actually emitted (5,760 expected per provider per day at the 15s cadence). 100% means the aggregator's feed was reachable for the full window."
- "Cardinality: 3 aggregators × 4 chains × 3 regions = 36 active series."
- "Cardinality: 4 aggregators × 4 chains × 3 regions = 48 active series."

findings:
- "Cross-chain average puts {{best_name}} on top at {{best_p50}} (p50, 24h) across {{count}} providers. Per-chain leaders differ: Base {{best_name:chain:base}} at {{best_p50:chain:base}}, Solana {{best_name:chain:solana}} at {{best_p50:chain:solana}}, BNB {{best_name:chain:bnb}} at {{best_p50:chain:bnb}}, Robinhood {{best_name:chain:robinhood}} at {{best_p50:chain:robinhood}}. Chains differ in block cadence (Solana 400ms vs Base/BNB 2-3s), which shifts the floor independently of provider speed."
Expand Down Expand Up @@ -217,6 +217,30 @@ providers:
p50: avg by (aggregator) (quantile_over_time(0.50, head_lag_seconds{aggregator="codex", region="sgp"}[24h])) * 1000
series: avg_over_time(head_lag_seconds{aggregator="codex", region="sgp"}[1h]) * 1000 unless (changes(head_lag_seconds{aggregator="codex", region="sgp"}[15m]) == 0 and count_over_time(head_lag_seconds{aggregator="codex", region="sgp"}[15m]) > 5 and on(chain, region) sum by (chain, region) (changes(head_lag_seconds{region="sgp"}[15m])) > 5 and on(aggregator, region) avg_over_time(ws_connected{aggregator="codex", region="sgp", chain=""}[15m]) > 0.8)

- slug: serialized
name: Serialized
tag: WebSocket trades stream
formula: "Median seconds between an on-chain swap on the bench pools and the same trade arriving on Serialized's WebSocket trades stream (subscribed per pool by its token side), sampled every 15s over 24h."
queries:
p50: avg by (aggregator) (quantile_over_time(0.50, head_lag_seconds{aggregator="serialized"}[24h])) * 1000
p90: avg by (aggregator) (quantile_over_time(0.90, head_lag_seconds{aggregator="serialized"}[24h])) * 1000
p99: avg by (aggregator) (quantile_over_time(0.99, head_lag_seconds{aggregator="serialized"}[24h])) * 1000
mean: avg by (aggregator) (avg_over_time(head_lag_seconds{aggregator="serialized"}[24h])) * 1000
success: clamp_max(avg by (aggregator) (count_over_time(head_lag_seconds{aggregator="serialized"}[24h]) / 5760), 1)
sample_size: sum(count_over_time(head_lag_seconds{aggregator="serialized"}[24h]))
series: avg_over_time(head_lag_seconds{aggregator="serialized"}[1h]) * 1000 unless (changes(head_lag_seconds{aggregator="serialized"}[15m]) == 0 and count_over_time(head_lag_seconds{aggregator="serialized"}[15m]) > 5 and on(chain, region) sum by (chain, region) (changes(head_lag_seconds{}[15m])) > 5 and on(aggregator, region) avg_over_time(ws_connected{aggregator="serialized", chain=""}[15m]) > 0.8)
live_activity: sum(changes(head_lag_seconds{aggregator="serialized"}[15m]))
regions:
- region: us-east
p50: avg by (aggregator) (quantile_over_time(0.50, head_lag_seconds{aggregator="serialized", region="us-east"}[24h])) * 1000
series: avg_over_time(head_lag_seconds{aggregator="serialized", region="us-east"}[1h]) * 1000 unless (changes(head_lag_seconds{aggregator="serialized", region="us-east"}[15m]) == 0 and count_over_time(head_lag_seconds{aggregator="serialized", region="us-east"}[15m]) > 5 and on(chain, region) sum by (chain, region) (changes(head_lag_seconds{region="us-east"}[15m])) > 5 and on(aggregator, region) avg_over_time(ws_connected{aggregator="serialized", region="us-east", chain=""}[15m]) > 0.8)
- region: eu-west
p50: avg by (aggregator) (quantile_over_time(0.50, head_lag_seconds{aggregator="serialized", region="eu-west"}[24h])) * 1000
series: avg_over_time(head_lag_seconds{aggregator="serialized", region="eu-west"}[1h]) * 1000 unless (changes(head_lag_seconds{aggregator="serialized", region="eu-west"}[15m]) == 0 and count_over_time(head_lag_seconds{aggregator="serialized", region="eu-west"}[15m]) > 5 and on(chain, region) sum by (chain, region) (changes(head_lag_seconds{region="eu-west"}[15m])) > 5 and on(aggregator, region) avg_over_time(ws_connected{aggregator="serialized", region="eu-west", chain=""}[15m]) > 0.8)
- region: ap-southeast
p50: avg by (aggregator) (quantile_over_time(0.50, head_lag_seconds{aggregator="serialized", region="sgp"}[24h])) * 1000
series: avg_over_time(head_lag_seconds{aggregator="serialized", region="sgp"}[1h]) * 1000 unless (changes(head_lag_seconds{aggregator="serialized", region="sgp"}[15m]) == 0 and count_over_time(head_lag_seconds{aggregator="serialized", region="sgp"}[15m]) > 5 and on(chain, region) sum by (chain, region) (changes(head_lag_seconds{region="sgp"}[15m])) > 5 and on(aggregator, region) avg_over_time(ws_connected{aggregator="serialized", region="sgp", chain=""}[15m]) > 0.8)

- slug: geckoterminal
name: GeckoTerminal
tag: REST feed
Expand Down
34 changes: 33 additions & 1 deletion docs/methodology/serialized-onboarding-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
> **Pre-onboarding evaluation.** Run before Serialized is wired into any live harness, so the
> decision to include or exclude them on each bench is documented and reproducible.
>
> **Version:** v1.2, 2026-09-05 (§8 corrected in v1.1; §16.3 root cause corrected and §17 added in v1.2). Author: internal. Key used: tenant `OpenChainBench`,
> **Version:** v1.3, 2026-09-06 (§8 corrected in v1.1; §16.3 root cause corrected and §17 added in v1.2; §18 added in v1.3: Serialized wired into bench 001). Author: internal. Key used: tenant `OpenChainBench`,
> plan `starter`, keyId `d5511a080aaa`, issued 2026-09-04.

---
Expand Down Expand Up @@ -501,3 +501,35 @@ Layer 2: coverage against canonical fields we define, with the per-vendor mappin
Layer 3, the real bench: snapshot every provider's verdict at mint, resolve on-chain at T+7d
(liquidity below 5% of peak, or LP pulled), publish recall and false-positive rate per provider.
Snapshot `top10HoldersPct` and `bundlersHoldingsPct` as the primary signals per §17.5.


## 18. Bench 001: Serialized is in (v1.3, 2026-09-06)

Earlier sections called 001 "blocked on a policy decision". That was the wrong framing, and it hid a
practical question nobody had tested: does Serialized's stream cover the four bench pools at all?

Their trades stream is keyed by **token** with an optional `pools` filter, while the bench is keyed
by **pool**. Subscribing by the pool's native side (SOL, WETH, WBNB) acknowledges and delivers
nothing, consistent with their REST 404 on `So111...112`: the chain native is a quote asset to them,
never a token. Their own `GET /v1/pool` names the other side under `token` (USDC on Solana and Base,
BUSD on BNB, USDG on Robinhood). Subscribing by that address with `pools=<bench pool>` delivers the
tape for exactly that market.

| Chain | pool | subscribe by | events with `txHash` |
|---|---|---|---|
| solana | 7qbRF6... | USDC `EPjF...` | 4 in 75 s (1,694 token-wide) |
| base | 0xd0b5... | USDC `0x8335...` | 10 in 90 s (173 token-wide) |
| bnb | 0x58f8... | BUSD `0xe9e7...` | 2 in 90 s |
| robinhood | 0x69bf... | USDG `0x5fc5...` | 79 in 90 s (3,888 token-wide) |

One constraint from their official docs shaped the implementation: **5 concurrent connections per
key**. The harness runs in three regions off one key, so the monitor opens one connection per
process and multiplexes the four pools as subscriptions. A first test that opened eight connections
was refused with close code 1008 ("connection limit (5 per key)"), which is also why an earlier
Solana attempt looked like a failure.

`harnesses/aggregator-head-lag/cmd/script/serialized_head_lag_monitor.go` records both series:
`head_lag_seconds` from their own `at` (same treatment as Mobula and Codex, same negative filter), and
`head_lag_ref_seconds` against the node reference clock matched by `txHash`, which is the one that can
rank providers. Every event carries `txHash`, `block`, `poolAddress` and a `preconfirmed` boolean per
their docs, so Base flashblocks are visible rather than inferred.
2 changes: 2 additions & 0 deletions harnesses/aggregator-head-lag/cmd/script/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
type Config struct {
CoinGeckoAPIKey string
MobulaAPIKey string
SerializedAPIKey string
DefinedSessionCookie string
MonitorRegion string // Deployment region: us-west, us-east, singapore, etc.
MobulaWSURL string // Mobula fast-trade WebSocket endpoint (allows staging to use EU-specific cluster)
Expand All @@ -21,6 +22,7 @@ func loadEnv() (*Config, error) {
// First, try to load from environment variables (for production/Railway)
config.CoinGeckoAPIKey = strings.TrimSpace(os.Getenv("COINGECKO_API_KEY"))
config.MobulaAPIKey = strings.TrimSpace(os.Getenv("MOBULA_API_KEY"))
config.SerializedAPIKey = strings.TrimSpace(os.Getenv("SERIALIZED_API_KEY"))
config.DefinedSessionCookie = strings.TrimSpace(os.Getenv("DEFINED_SESSION_COOKIE"))
config.MonitorRegion = strings.TrimSpace(os.Getenv("MONITOR_REGION"))
config.MobulaWSURL = strings.TrimSpace(os.Getenv("MOBULA_WS_URL"))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -811,6 +811,9 @@ func runHeadLagMonitor(config *Config, stopChan <-chan struct{}) {
wg.Add(1)
go runGeckoTerminalHeadLagMonitor(config, stopChan, &wg)

wg.Add(1)
go runSerializedHeadLagMonitor(config, stopChan, &wg)

// Wait for all to finish
wg.Wait()
fmt.Println("[HEAD-LAG] All monitors stopped")
Expand Down
Loading
Loading