Repository navigation
feat(feed): find more LinkedIn posts and import links in bulk - #83
Merged
Merged
Conversation
/feed had stalled at about 119 posts: 5 searches a day, 10 results each, from a fixed list of 20 queries that mostly found posts we already had. - Ask Olostep for 25 results per search (the API maximum), limited to linkedin.com - Grow the search list to about 99: 40 program, topic and year searches, plus one search per known member, so every member found leads to their other Dev Weekends posts - Remember where the collector stopped (FeedCollectorState), so the daily cron and admin runs work through the whole list; the cursor only moves when some searches succeed - Add a "Deep search" admin button (40 searches, about 200 credits, with a confirmation) for a one-time backfill, capped at 60 per request - Add bulk import: admins paste LinkedIn post links and they go live Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/feedhad stalled at about 119 posts. The collector ran 5 searches a day with 10 results each, from a fixed list of 20 queries that mostly returned posts we already had. This makes discovery cover far more ground and adds a manual way in.include_domains.FeedCollectorState), so the daily cron and admin runs work through the whole list instead of repeating searches. The cursor only moves when some searches succeed, so a bad key or an outage doesn't skip part of the list.Limits worth knowing
Search can only find posts that are public and indexed by search engines. Posts visible only to connections, or never indexed, can't be found by any tool without logging in to LinkedIn, which LinkedIn's terms don't allow for automated collection. That's why bulk import and the submit form matter for reaching the full set.
Test plan
tscandnext lintclean for the changed files🤖 Generated with Claude Code