feat(fetch): add --raw to web fetch for selector discovery - #419
Open
ankitranjan7 wants to merge 1 commit into
Open
feat(fetch): add --raw to web fetch for selector discovery#419ankitranjan7 wants to merge 1 commit into
ankitranjan7 wants to merge 1 commit into
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
🟠 Maintainer review suggested — low confidenceThe automated review could not reach a fully supported conclusion. Limitations
This review is advisory and does not block merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
webcmd web fetchcould only ever return readability-extracted article text, so an agent hunting for a CSS selector, a meta tag, or an inline<script>payload had no way to see the HTML — two recent eval scenarios abandoned webcmd here and fell back tocurl | grep -P, which does not exist on macOS, losing the task.What changed
--raw(boolean, default false) onweb fetch: returns the response body verbatim — no readability, no markdown conversion.--max-charswith a hard cut at the limit plus the existing[webcmd: truncated at N characters; rerun with --max-chars 0]marker, andtruncated: true. Never a silent cut.bytes(body byte length) alongside the existingstatus,finalUrl,contentType,truncatedfields; markdown output gainsBytes:/Truncated:lines. Non-raw results are byte-identical to before — no new fields.examplenow shows--raw;cli-manifest.jsonregenerated.--allow-privategating, timeouts, the challenge ladder,FETCH_BLOCKED,FETCH_REQUIRES_BROWSER, and the 10 MiB body cap.Before / After
Before — no way to get HTML:
After:
Default path, unchanged:
Truncation is announced, not silent:
Tests
npm run typecheckclean.npx vitest run --project unit src/fetch src/hosted/manifest.test.ts src/check-hosted-contract.test.ts— 82 passed. New cases: raw returns the unprocessed body with metadata; non-raw still extracts and omitsbytes; raw truncation is reported; a Cloudflare challenge with--rawstill ends inFETCH_BLOCKED;--rawreaches the client and markdown metadata renders.Full
--project unitrun: 123 failures with this change vs 122 onorigin/main— the one delta was the hosted-contract byte check, fixed by regeneratingcli-manifest.json(now committed); the remaining 122 are pre-existing on main in this environment (hosted lifecycle, playwright sandbox provenance, plugin manifest) and untouched here.🤖 Generated with Claude Code