Repository navigation
fix(crawl): stop one host starving a tier; show header follows the feed (0.6.1) - #89
Merged
Merged
Conversation
…ed (0.6.1) The Tech Universe showed "13d ago" over an episode list that ran to 2d ago: its next read was due Oct 30 and never happened. 321,844 of 389,547 shows were overdue. dueFeeds returned strictly oldest-first, the daemon over-reads only a few dozen rows, and 58 of the first 60 rows of the slow tier were feeds.redcircle.com, whose per-host cap the hot tier had already used. So active and slow read nothing on every tick and their slots fell to dormant. - dueFeeds takes turns between hosts inside the 1000 oldest due rows (21 ms on the production table via feed_state_due_tier). - The tick goes from 5 s to 2 s: the crawler ceiling was 17k reads an hour against about 15k of steady-state demand, so it could never clear a backlog. - The show header takes the newest date and largest count from the feed the page already reads, so it cannot contradict the list under it. - CI runs Postgres, so the crawler SQL tests stop skipping there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug: https://p0dcasters.com/podcast/the-tech-universe showed "43 episodes, 13d ago" above a list reaching Episode 53, 2d ago.
Cause: the crawler was starving, this is not one show. 321,844 of 389,547 shows were overdue, the oldest since Oct 29.
dueFeedswas strictly oldest-first and the daemon over-reads only a few dozen rows; 58 of the first 60slowrows and 46 of the first 60activerows werefeeds.redcircle.com, whose per-host cap thehottier had already used. Since the last restartactiveandslowhad read 0 feeds whiledormantread 3,214.Changes
dueFeedstakes turns between hosts inside the 1000 oldest due rows. 21 ms on the production table, via the existing index. Regression test confirmed failing on the old query.TEST_DATABASE_URLtests were skipping there.🤖 Generated with Claude Code