feat(lifecycle): integrate self-hosted Firecrawl browser scraper - #1019
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
blove
enabled auto-merge (squash)
September 5, 2026 21:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use Firecrawl's actual open-source standalone Playwright scraper for company homepage capture, without a hosted Firecrawl account or full crawler stack. Package a pinned upstream revision as one browser container and replace the hosted API client with an authenticated call to our own service.
The upstream patch adds requested/final URL provenance and a narrow homepage-only interface. Capture has one active request, a ten-second deadline, disconnect cleanup, and a 2 MiB response cap. DOMContentLoaded avoids waiting for stalled page assets. The client retains the existing bounded evidence schema and separate email delivery flow. Direct capture remains the default until the service is deployed and verified.
Validation: lifecycle and growth-research test/check/lint/build passed; 61 client/selector tests and 10 handler tests passed. Docker built successfully; actual Chromium probes blocked private subresources and redirects. A stalled-image regression timed out before the document-ready change and returned evidence afterward. Six real company homepages produced schema-valid evidence through the actual client and authenticated local container, including Neon and LangChain redirects. No model calls, database changes, or emails.
Production activation awaits valid container-host access: the existing Railway credential was rejected. No hosted API fallback, paid account requirement, queue, or new agent workflow is introduced. The browser service's DNS checks are documented separately from direct-fetch DNS pinning.