Skip to content

web fetch: https://example.com returns r.jina.ai's stale cached snapshot, not the page - #1915

Draft
ZeroPoint95 wants to merge 1 commit into
devfrom
zeropoint95/1771-web-fetch-https-example-com-returns-r-ji
Draft

ZeroPoint95 wants to merge 1 commit into
devfrom
zeropoint95/1771-web-fetch-https-example-com-returns-r-ji

Conversation

@ZeroPoint95

Copy link
Copy Markdown
Collaborator

Changes

  • codeaf web fetch now sends r.jina.ai's no-cache header (X-No-Cache: true) on every fetch, keyed or not, so the reader cannot answer from its 3600s page cache by default.
  • When the reader still serves its cache (X-Cache: HIT), the fetch discards the snapshot and GETs the target directly — codeaf User-Agent, 8 MiB bounded read, HTML stripped to text — and returns an error naming the fallback instead of the snapshot when the direct fetch fails or comes back empty.
  • Adds unit tests for the no-cache ask (keyed and unkeyed), the HIT-fallback matrix (stripped direct text, 500 names the fallback, empty body is an error), and a change entry invalidating the belief that codeaf web fetch https://example.com returned the real page rather than the stale cache.
  • Final change: 3 files, +238/−0.

Closes #1771.

Validation

  • Passed locally: cd . && go test -count=1 ./internal/search/.
  • Passed locally: cd . && go test -count=1 -run 'TestJina' -v ./internal/search/ 2>&1 | grep -E '^(=== RUN|--- (PASS|FAIL)|PASS|FAIL|ok)' ; echo "PIPE_EXIT:${PIPESTATUS[0]}".
  • Passed locally: cd . && go test -count=1 -race ./internal/search/.
  • Proof · 8 of 8 claims shown locally. GitHub CI is checked after publishing.
  • ✓ The search package's tests pass fresh (-count=1), including all 9 jina tests: the 4 pre-existing (TestJinaFetch, TestJinaFetchRejectsNonURLs, TestJinaFetchErrorStatus, TestJinaFetchIsCappedByTheRenderer) whose bodies are unedited in the diff (additions only), and the 4 new ones for the change plus TestJinaSearchResultsParsesBlocks.
  • ✓ The search package is race-clean with the new fallback tests' shared serve() base-URL pattern.
  • ✓ Vet, gofmt, the changelog gate and the name/manual laws stay green: the change leaves the tree buildable and well-formed.

by codeaf factory

Factory-Approval: evt_b7b18ebb @ZeroPoint95 7a4c722

…ectly

r.jina.ai serves its 3600s page cache by default, so codeaf web fetch
returned Jina's stale snapshot of a page — with its own cached-snapshot
warning — instead of the page. The fetch now sends the reader's no-cache
header and, when the answer is still marked a cache hit (X-Cache: HIT),
discards it and GETs the target directly, stripping the HTML to text.

Fixes #1771

Assisted-by: CodeAF (deepseek-v4-flash-latest)
Co-Authored-By: CodeAF <267109073+agentfield-bot@users.noreply.github.com>
@santoshkumarradha
santoshkumarradha force-pushed the zeropoint95/1771-web-fetch-https-example-com-returns-r-ji branch from 7a4c722 to eceb873 Compare October 24, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

web fetch: https://example.com returns r.jina.ai's stale cached snapshot, not the page

1 participant