From 24712ae19ceb1e273ad24303395f46dfd023d8d3 Mon Sep 17 00:00:00 2001 From: only-cli Date: Fri, 4 Sep 2026 16:00:32 -0400 Subject: [PATCH] docs: sync the Benchmarks section with the Reddit feed rerun The page-view suite now reads Reddit through the Atom feeds, so the section carries the 2026-09-04 numbers on oc 0.5.3: real content on all fifteen pages, 118x fewer tokens than raw HTML across the fourteen both could read, and the two feeds at 459 and 493 tokens. The access paragraph describes Reddit's metering the way the rerun measured it. --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index d09bb9d..efb8f9c 100644 --- a/README.md +++ b/README.md @@ -107,7 +107,7 @@ Works on any mostly-static site with no per-site setup: news sites, blogs, docum A shortcut only ever resolves to a URL and then takes the same path `oc open` does, so it changes nothing about what a page costs or how it reads. The last argument takes every word after it, so `oc ddg search claude code cli` and `oc aws search s3 lifecycle rules` need no quoting, and a path argument keeps its slashes, so `oc learn doc azure/aks/what-is-aks` reaches that page. -A few of these (X, Reddit, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question ` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. Reddit goes through the Atom feeds on www.reddit.com: old.reddit.com has sent logged-out readers to a login page since June 2026 and the `.json` views answer 403 without an OAuth token, so a reddit.com page URL handed to `oc open` still meets that wall, where `oc reddit post `, or the same URL with `/.rss` on the end, reads the post and its comments. oc asks reddit.com with its Firefox fingerprint first, because Reddit's edge refuses the Chrome one more often than not. The feeds carry titles, authors, dates, and bodies but no scores or comment counts, and anonymous reddit.com allows roughly ten requests a minute per address, so a burst of Reddit shortcuts ends in a 429 that takes minutes to clear. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript and sites with hard bot challenges that expose no feed. Sites that genuinely require your account can be reached with `oc login` (bring your own cookies). +A few of these (X, Reddit, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question ` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. Reddit goes through the Atom feeds on www.reddit.com: old.reddit.com has sent logged-out readers to a login page since June 2026 and the `.json` views answer 403 without an OAuth token, so a reddit.com page URL handed to `oc open` still meets that wall, where `oc reddit post `, or the same URL with `/.rss` on the end, reads the post and its comments. oc asks reddit.com with its Firefox fingerprint first, because Reddit's edge refuses the Chrome one more often than not. The feeds carry titles, authors, dates, and bodies but no scores or comment counts, and anonymous reddit.com meters each client tightly: a second request within half a minute of the first has come back 429 in testing, and a burst of Reddit shortcuts ends in refusals that take minutes to clear, so space them out. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript and sites with hard bot challenges that expose no feed. Sites that genuinely require your account can be reached with `oc login` (bring your own cookies). Want a website on that list? Open a pull request, or an issue naming the site; see [CONTRIBUTING.md](CONTRIBUTING.md). @@ -168,26 +168,26 @@ Private and internal addresses are refused whether or not a proxy is set. With a ## Benchmarks -Full methodology, per-task rows, and the Codex runs live in [only-cli/benchmarks](https://github.com/only-cli/benchmarks). Where things stand (oc 0.5.1, September 2026, live sites): +Full methodology, per-task rows, and the Codex runs live in [only-cli/benchmarks](https://github.com/only-cli/benchmarks). Where things stand (oc 0.5.1 to 0.5.3, September 2026, live sites): -- **125x fewer tokens than raw HTML** across the twelve real pages both could read: 8,519 against 1,064,474. 15x fewer than Jina Reader, 59x fewer than Playwright MCP's accessibility snapshot. -- **Real content on every page it could reach, and an honest failure on the two it could not.** Reddit now sends logged-out readers to a login wall, and every other tool returned that wall, or a 403 block page, as a success. The suite hands every tool the old.reddit.com page URLs, which still end at that wall; since 0.5.2 the `oc reddit` shortcuts read the same threads through Reddit's Atom feeds, a route the suite does not measure yet. Yahoo Finance refuses plain fetch outright and DuckDuckGo still blocks lynx; oc's Chrome impersonation read both. +- **118x fewer tokens than raw HTML** across the fourteen real pages both could read: 9,466 against 1,119,003. 17x fewer than Jina Reader, 54x fewer than Playwright MCP's accessibility snapshot. +- **Real content on all fifteen pages, Reddit included.** Reddit's pages now sit behind a login wall, so the suite reads its thread and subreddit through the Atom feeds that still answer anonymously, the same route the `oc reddit` shortcuts take: 459 and 493 tokens through oc against 10,452 and 21,155 for the feed XML, while the browser-based tools and Jina Reader got the block pages Reddit serves their fingerprints. Yahoo Finance refuses plain fetch outright and DuckDuckGo still blocks lynx; oc's Chrome impersonation read both. - **Half the cost of Claude Code's built-in `WebSearch`** on Wikipedia lookups: $0.23 against $0.45 for five questions, both 5/5 correct, on 25x less fresh input. - **21% cheaper than `WebFetch` and 34% cheaper than `WebSearch`** on eleven language docs lookups, at equal or better accuracy. - **10% cheaper than `WebFetch` and 49% cheaper than `WebSearch`** on twelve dependency lookups across GitHub, npm, PyPI, RubyGems, crates.io, Docker Hub, Stack Overflow and an RFC, 12/12 correct with no tuned shortcut for most of those sites. The tables behind those numbers: -**Tokens per page, no model in the loop.** Fifteen real pages: a news front page, a Reddit discussion, search results, a stock quote, three cloud CLI references, the Python, MDN, and Node.js references, and more. +**Tokens per page, no model in the loop.** Fifteen real pages: a news front page, a Reddit thread and subreddit through their Atom feeds, search results, a stock quote, three cloud CLI references, the Python, MDN, and Node.js references, and more. | method | tokens for 15 pages | notes | | --- | ---: | --- | -| `oc open` | 8,971 | real content on 13 of 15 pages; the two Reddit pages are behind a login wall, and oc is the only reader that reported that instead of returning the wall | -| Jina Reader | 105,198 | both Reddit results are block pages; failed LinkedIn and the Node.js `fs` page outright | -| Playwright MCP | 531,303 | accessibility snapshots; both Reddit snapshots are the login wall | -| raw HTML fetch | 1,240,669 | both Reddit results are the login wall, 88,000 tokens of it each; Yahoo Finance refused the connection | +| `oc open` | 9,913 | real content on all 15 pages; the two Reddit feeds cost 459 and 493 tokens | +| Jina Reader | 145,679 | both Reddit results are block pages; failed LinkedIn outright | +| Playwright MCP | 535,908 | accessibility snapshots; both Reddit snapshots are block pages, since Reddit refuses the Chrome fingerprint | +| raw HTML fetch | 1,119,003 | the two Reddit feeds are 10,452 and 21,155 tokens of XML; Yahoo Finance refused the connection | -oc's budget keeps every page near 500 tokens however much it weighs: the YouTube watch page is 345,487 tokens raw and 688 through oc, Node's `fs` reference 275,425 against 479. On the twelve pages both could read, raw HTML costs 125x what oc does: 1,064,474 against 8,519. +oc's budget keeps every page near 500 tokens however much it weighs: the YouTube watch page is 363,516 tokens raw and 688 through oc, Node's `fs` reference 275,425 against 479. On the fourteen pages both could read, raw HTML costs 118x what oc does: 1,119,003 against 9,466. Reddit meters anonymous feed readers, so the suite spaces its Reddit requests a minute apart; a second request from the same client inside thirty seconds came back 429 in every faster pass. **Whole tasks against the agent's built-in web tools.** Read cost is one thing, what an agent actually spends is another, so a second set of suites runs full lookups end to end in Claude Code (`claude-sonnet-5`), one tool per run, and grades every answer. Five Wikipedia lookups, eleven language documentation lookups, and twelve lookups on the pages around a dependency, where oc has shortcuts only for GitHub and Stack Overflow and renders the rest generically: