Skip to content

docs: correct the head-lag claim in the Serialized audit, add follow-ups - #2264

Merged
Flotapponnier merged 1 commit into
devfrom
docs/serialized-audit-v11
Sep 5, 2026
Merged

docs: correct the head-lag claim in the Serialized audit, add follow-ups#2264
Flotapponnier merged 1 commit into
devfrom
docs/serialized-audit-v11

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

The correction

v1.0 of the audit claimed bench 001 references archive nodes and was therefore unaffected by providers disagreeing about on-chain time. That was wrong. It repeated benchmarks/aggregator-head-lag.yml instead of reading harnesses/aggregator-head-lag.

There is no archive-node reference in that harness. grep -rl "archive|eth_getBlockByNumber|getBlockTime|blockTimestamp" over its cmd/ returns nothing. The gauge behind the leaderboard is:

// head_lag_monitor.go:211  (Mobula)
onChainTime := time.UnixMilli(trade.Date)     // Mobula's own field
// head_lag_monitor.go:707  (Codex)
onChainTime := time.Unix(event.Timestamp, 0)  // Codex's own field

The spec claims otherwise in three places: methodology[7], the FAQ, and the per-chain explainers. Measured impact: for the same transaction hash, serialized.at minus mobula.date is −707 ms on Solana and +1000 ms on Base, so the published numbers partly measure clock placement rather than pipeline speed.

Plus head_lag_monitor.go:219 drops negative lags silently, which would discard 100% of Serialized's Base preconfirmed sample.

This is a documentation-versus-code defect on a live, publicly cited bench, independent of any provider. Not fixed here, only documented.

New results recorded

  • Bench 067 now conclusive. The earlier inconclusive run used the wrong probe address; the harness pins canonical ones in registry.go. Re-run verbatim: listed 19, verified 5, 0 errors. Would rank last (CoinStats 127, Mobula 50, Zerion 42, Moralis 15).
  • Negative capabilities probe-confirmed, not assumed: 404 on /v1/wallet/nfts, /v1/nft/collection, /v1/nfts, /v1/swap/quote, /v1/quote, /v1/route, /v1/bridge/quote.
  • A real pricing defect. Serialized prices BONK at 6.33e-07 while Mobula, DexScreener (Orca, $306k liquidity) and GeckoTerminal all agree at ~3.31e-06. 5.2x low, $55.7M market cap against a real ~$290M. Root cause visible in their own /v1/token/pools: rank 1 is a pool with liquidity 4,339 and the deep Orca pool is absent from the list, while their own ranks 2 and 3 disagree with rank 1 by the same 5x.
  • A cross-API price-accuracy bench is not proven. Two basket constructions, both recorded as failures so nobody repeats them. It cannot use another aggregator as reference; the reference has to be computed from on-chain reserves over an RPC we control.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA

v1.0 claimed bench 001 references archive nodes and was therefore
unaffected by provider clock disagreement. That repeated the spec
instead of reading the harness: there is no archive-node reference in
aggregator-head-lag at all, and the leaderboard gauge is computed from
each provider's own timestamp. The spec says the opposite in three
places.

Also records the now-conclusive bench 067 result (verified 5 of 19
chains), probe-confirmed negative capabilities, and a verified 5.2x
BONK mispricing traced to pool discovery missing the main market.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA
@Flotapponnier
Flotapponnier merged commit 7ad99d5 into dev Sep 5, 2026
1 check failed
@Flotapponnier
Flotapponnier deleted the docs/serialized-audit-v11 branch September 5, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant