Repository navigation
Crawler rule: 10 s fetch, 30 s render, slow/broken sources flagged 30 days - #274
Merged
Merged
Conversation
… days - core/crawl.js: CRAWL_FETCH_TIMEOUT_MS 10 s, CRAWL_BROWSER_TIMEOUT_MS 30 s, BULK_DOWNLOAD_TIMEOUT_MS 1 h, FLAG_DAYS 30, failureClass() - http.request default 30 s -> 10 s; download keeps an hour; thrown errors marked upstream, HTTP errors carry status - opensite TIMEOUT_MS 15 s -> 10 s; storefront/buyvps/statuspage/openvuln page and API reads drop their 15-30 s literals for the default - renderWithObscura: one 30 s budget, waitUntil load + bounded settle instead of networkidle 60 s, rejects on timeout, always closes the tab - ingest: a timeout, DNS/TLS/connect error or 5xx sets next_run_at 30 days out with sources.flag/flagged_at (migration 0044); boot's rescheduleFailedSources skips flagged rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sh1pt-actions-fleet
Bot
force-pushed
the
crawl-10s-30s-30d
branch
from
November 8, 2026 17:19
dcc0862 to
e478343
Compare
ralyodio
added a commit
that referenced
this pull request
Nov 8, 2026
…ces (#275) The deploy of #274 never applied 0044: ALTER TABLE sources needs ACCESS EXCLUSIVE, and the web's long collection queries hold ACCESS SHARE on sources for minutes, so the migration hit its 10 s lock_timeout on every retry and the health check rolled the deploy back (the old container kept serving). A new table takes no lock on sources. 0044 never applied anywhere, so the file is rewritten in place. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fleet crawler rule (2026-11-08).
packages/core/src/crawl.js:CRAWL_FETCH_TIMEOUT_MS10 s,CRAWL_BROWSER_TIMEOUT_MS30 s,BULK_DOWNLOAD_TIMEOUT_MS1 h,FLAG_DAYS30,failureClass(err)(timeout = slow; DNS/TLS/connect/5xx = broken; 4xx, parse, EAI_AGAIN and our own Postgres errors = null).http.js:requestdefault 30 s -> 10 s;downloadkeeps the hour; errors markedupstream, HTTP errors carrystatus. Adapters that pass an explicit longer budget for a big document keep it.opensite.jsTIMEOUT_MS15 s -> 10 s. storefront/buyvps/statuspage/openvuln page and API reads use the default.renderWithObscura: one 30 s budget,waitUntil: 'load'+ bounded settle (<= 2 s) instead ofnetworkidle60 s; a timeout rejects; the tab is always closed.next_run_at30 days out andsources.flag/flagged_at(migration 0044, nullable columns); any other outcome clears them. Boot'srescheduleFailedSourcesskips flagged rows.test/crawl-rule.test.js. (installmap + news-directories fail on main already.)🤖 Generated with Claude Code