Repository navigation
Crawlers: 10 s fetch, 30 s render, 30-day recheck for slow or broken targets - #442
Merged
Merged
Conversation
…targets Fleet crawler policy in one module (lib/crawl/policy.ts): CRAWL_FETCH_TIMEOUT_MS 10 s (Tor keeps a separate 30 s budget), CRAWL_BROWSER_TIMEOUT_MS 30 s with a <= 2 s bounded settle, and CRAWL_RECHECK_MS 30 days for a target that timed out, failed DNS/TLS/connect or answered 5xx (429 honours Retry-After, capped). - promote ingest: 15 s -> 10 s with the body inside the budget; slow/broken feeds park 30 days, 4xx keeps the backoff until 8 in a row, then parks. - social feed autopost: 12 s -> 10 s; new next_check_at/failed_at park a dead feed 30 days (4xx: daily) instead of refetching every 15 minutes. - lx feed sources: given_up is no longer forever; one recheck every 30 days. - lx sitemaps: root failures are errors (not "empty"), errors get sitemap_next_check_at +30 days and an hourly worker sweep runs the recheck. - audit fetch 15 s -> 10 s, outreach page/PDF fetch -> 10 s, feed validation -> 10 s. - audit render: load + bounded settle instead of networkidle; a failed render is recorded as renderError, not an empty page. Outreach render: one 30 s deadline across load, settle and scrolling. Exempt and untouched: uptime monitors, ad link-health, own/vendor services, worker/pdf.ts and worker/frames.ts (render our own pages, already bounded). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThreatCrush Security Scan50 finding(s) HIGH/CRITICAL: 2 | MEDIUM: 30 | LOW: 18
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fleet crawler rule (2026-11-08): third-party fetches get 10 s, headless renders 30 s, and a slow or broken target is flagged and left alone for 30 days, then rechecked once.
lib/crawl/policy.ts: the constants (CRAWL_FETCH_TIMEOUT_MS10 s,TOR_FETCH_TIMEOUT_MS30 s,CRAWL_BROWSER_TIMEOUT_MS30 s, settle 2 s,CRAWL_RECHECK_MS30 d) plus pure failure classification (timeout / DNS / TLS / connect / 5xx = slow_or_broken, 429 = Retry-After capped at 30 d).next_check_at/failed_atpark a dead feed 30 d (4xx: daily) instead of every 15 min. Saving the config clears the flag.given_upgets one recheck every 30 d (5 per sweep).sitemap_next_check_at+30 d and an hourly worker sweep runs the recheck.load+ bounded settle instead ofnetworkidle; a failed render is recorded asrenderErrorinstead of an empty rendered page. Outreach render: one 30 s deadline across load, settle and scrolling.Exempt, untouched: uptime monitors, ad link-health, own/vendor calls,
heroBackfill,worker/pdf.tsandworker/frames.ts(render our own pages, already bounded).Migration
20261108170000_crawl_recheck_30d.sqlis additive and already applied to prod (9 errored sitemaps backfilled).🤖 Generated with Claude Code