Repository navigation
feat(firecrawl): traceAI-firecrawl instrumentor for TH-8325 - #206
Merged
Merged
Conversation
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103
…/th-8325-firecrawl Refs: TH-8103
The real firecrawl-py 4.46.2 v2 client calls a loopback fake of the Firecrawl API (Firecrawl(api_url=...)): scrape, search, crawl including its internal status poll, and a scrape that gets HTTP 500. Spans leave through the real fi_instrumentation register() exporter into harness.Receiver (3eaadc8). Asserts the collector path, both auth headers, project_name/project_type=observe, one TOOL span per call (the crawl poll adds none), page count, limit, credits and host on the crawl, query and count on search, ERROR status plus exception event on the 500, and that the key, URL path, page body and result titles never leave. Mutation check: recording the full URL instead of the host fails both contract tests. 5 passed (3 unit + 2 contract). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
…rt_crawl firecrawl-py 4.46.2 AsyncFirecrawlClient.crawl awaits self.start_crawl, which is wrapped too, so an async blocking crawl exported a sibling firecrawl.start_crawl span next to firecrawl.crawl. A context variable now marks a traced Firecrawl call as active; nested wrapped calls in the same context run untraced. Adds a loopback fake of the v2 API, real-client tests for sync and async crawl, and an async contract test through the shared harness. R1 (TH-8325 installed review). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
get_crawl_status(job_id) and cancel_crawl(crawl_id) take the id as the first positional argument. cancel_crawl returns a bool, so its span never carried firecrawl.job_id, and a failed status call had none either. The start attributes now read args[0] for those two methods, so start, status and cancel share the job id (AC-03), sync and async, including on errors. Also shortens the fake server's shutdown poll. R2 (TH-8325 installed review). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
firecrawl-py returns failed and cancelled crawl jobs without raising, so a failed blocking crawl was exported as an OK span. crawl and get_crawl_status now record firecrawl.status from the job. A failed job sets the span to ERROR "failed"; a cancelled job sets ERROR "cancelled" and firecrawl.cancelled=true. Tested sync and async for completed, failed and cancelled jobs. R3 (TH-8325 installed review). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
Attribute extraction ran before the vendor call and outside any guard, so
scrape("http://[::1/page") raised urlsplit's ValueError to the caller
before the SDK ran. Reading the result and recording an exception were
not guarded either: a result attribute that raised replaced the vendor's
return value, and an exception whose str() failed replaced the vendor's
exception. Each step is now isolated and logged at debug level, the span
always ends exactly once, and a tracer that cannot start a span falls back
to the plain vendor call.
R5 (TH-8325 installed review).
approved: Nikhil 2026-10-03 blanket
Refs: TH-8325
The wrappers started the span but never made it current, so HTTP client spans and anything else the SDK traces became siblings of the TOOL span instead of its children. The vendor call now runs inside trace.use_span(span, end_on_exit=False) with exception recording left to the wrapper, so an error is recorded once. Tests pin HTTP span parentage for sync and async, that the context is restored afterwards, and that a vendor error carries exactly one exception event. R6 (TH-8325 installed review). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
…ror codes - AC-07: a cancelled asyncio task running a blocking crawl ends the span ERROR "cancelled" with firecrawl.cancelled=true. cancel_crawl records firecrawl.cancelled from the API's answer. - J1: scrape, search, crawl and start_crawl record firecrawl.formats, the requested format names from formats= or scrape_options.formats. A JSON format's prompt and schema stay off the span. - J5: a firecrawl-py FirecrawlError records firecrawl.error.status_code and firecrawl.error.code (for example 429 and the vendor's rate-limit code). R7 (TH-8325 installed review). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
_redact read instance.api_key, which firecrawl-py 4.46.2 clients do not have: the key lives on http_client, async_http_client and config. A search query that contained the key was exported verbatim. The job id path also passed the span instead of the client. Redaction now reads every key location (including a key resolved from FIRECRAWL_API_KEY) and the result path passes the client instance. R8 (TH-8325 installed review). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
… Pythons The exact firecrawl-py==4.46.2 runtime pin made BaseInstrumentor.instrument() log an error and wrap nothing on any other release, and forced that exact version on users. pyproject and _instruments now declare >=4.46.2,<5; the tests stay pinned to 4.46.2. fi-instrumentation-otel's floor moves to >=1.1.0, the release the tests import, and the Python range and classifiers list only 3.10, 3.11 and 3.13, the versions the suite runs on. Tests pin the range, that instrument() wraps on an in-range release and skips an out-of-range one, and the declared metadata. R11, N1 (TH-8325 installed review). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
…g and exit flush Adds the tests the installed review listed as missing, all against the real firecrawl-py 4.46.2 clients and a loopback fake: - map emits one TOOL span with the host only, sync and async; - AC-08: the async twins of all seven methods export the same span names, attributes and status as the sync client; - uninstrument() restores every wrapped method on both clients and new clients emit no spans afterwards; - R4: the Firecrawl facade binds methods at construction, so a client built before instrument() is untraced and one built while instrumented keeps tracing after uninstrument() (pins the documented behaviour); - AC-09: a script that exits without force_flush() still exports its scrape span through register()'s batch processor; an os._exit control exports nothing. The uninstrument and facade tests fail when _uninstrument skips the restore (mutation run locally, then reverted). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
…d limits - R10: drop "content capture is off by default"; there is no such switch. State what is never recorded and that error messages are kept as raised. - R9: firecrawl-py defaults a crawl job's credits_used to 0 when the API omits it, so a recorded 0 is ambiguous. Documented, behaviour unchanged. - R4: Firecrawl/AsyncFirecrawl bind methods at construction; build them after instrument(), and a client built while instrumented keeps tracing after uninstrument(). - N2: the sync Watcher polls from its own thread, so each poll is a root get_crawl_status span; AsyncWatcher polls are parented. - Lists every attribute with the methods that set it, the job-status and cancellation rules (R3, R7), the supported firecrawl-py range and the tested Pythons (R11, N1). approved: Nikhil 2026-10-03 blanket Refs: TH-8325
No behaviour change. approved: Nikhil 2026-10-03 blanket Refs: TH-8325
…ob's status A get_crawl_status poll that got an answer succeeded, whatever the job's state; marking it ERROR made healthy status polls (and every watcher poll of a failed job) look like failures. It now records firecrawl.status and stays OK; crawl() keeps ERROR for failed/cancelled jobs and firecrawl.cancelled. Coordinator decision on the open question from the fix round. Test first: the parametrised job-status test failed 4 cases (failed and cancelled, sync and async) on 8c1e79e. approved: Nikhil 2026-10-03 blanket Refs: TH-8325
Keep error telemetry to safe metadata and preserve original SDK exceptions. Use FITracer and explicit query privacy controls; canonicalize format labels and mirror SDK options precedence; count all supported search groups. Correct Observe registration and published-core exit provenance. Add real SDK/exporter regression coverage and admitted Python 3.12 metadata. Verified test-first bounded correction; original review and failed probes are retained in ticket evidence. No shared packages or vendor surfaces changed.
The search query was only written to fi.retrieval.query, which the Future AGI backend does not map, so the Observe span panel showed no input for firecrawl.search. Write the same redacted, capped query to input.value as the sibling tool tracers do. Hidden inputs still omit it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Record the search query in gen_ai.retrieval.query (the repo constant the backend reads) and the count in firecrawl.result_count instead of the unmapped fi.retrieval.* keys. - gen_ai.span.kind: RETRIEVER for search; scrape, map and the crawl calls stay TOOL. - Wrap only methods defined on each class (vars). - uninstrument() now also stops facades built while instrumented (shared enabled flag), so a Firecrawl() built before uninstrument() stops tracing. - Add a CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status 2026-10-21: merged
devinto the branch (cbe84b0, no conflicts), so dev's Python tests workflow ran on this PR: Plan, fi_instrumentation, package (firecrawl), Python tests all green. Local run: 168 passed (Python 3.11, firecrawl-py 4.53.2). The harness files are byte-identical to #203 and to the other TH-8103 PRs, so any merge order works.traceAI-firecrawl0.1.0 is not on PyPI yet. After this merges, publish 0.1.0, then merge docs#998 (it now installs from PyPI).Merge order: this PR carries the shared OTLP harness from #203 (TH-8103, files identical). Merge #203 first, or together with this PR. It also merges cleanly with Exa's #205 in either order.
Summary
Adds
traceAI-firecrawlfor the Firecrawl v2 sync and async clients. Wrapsscrape,search,map,crawl,start_crawl,get_crawl_statusandcancel_crawlwith TOOL spans. Does not patch v1FirecrawlAppor OpenLIT.Current head:
e983153cf327a24f1e57e5559bb52de25aee8e83.Verification
Pinned
firecrawl-py==4.46.2. On the clean committed head:fi-instrumentation-otel==1.1.0All eight JUnits: zero failures, errors or skips; source-versus-wheel import provenance verified. Two initial Python 3.13 cells were interrupted by a long-command tool timeout; only those cells were rerun, with separate retained logs and actual exit 0. Interrupted attempts are not counted as passes.
RED method: archive prior reviewed
8a29398source/core/harness, copy the final 168 tests and execute from that snapshot: 113 failed / 55 passed, exit 1, zero errors/skips. Extra validation controls passing the baseline are not labelled RED regressions.Wheel and sdist built successfully; emitted Python 3.10–3.13 metadata agrees with the package range. A separate fresh installed-wheel/Python 3.11 smoke ran 168 tests successfully, exit 0: the resolver selected firecrawl-py 4.49.1 and published core 1.1.0. The imported wheel bytes match the built artifact. Its exit subtest still imports repository instrumentor source (same published core), so this is not full wheel-only exit proof. This extra resolved-version smoke is not a guarantee for all 4.x versions; distribution-metadata simulations remain version guards.
Independent round-2 static review and delta verification cleared the five P2 delivery findings and original F6/V1 triggers. Separate review contexts used the same effective model/provider fallback; no cross-vendor proof is claimed. The delta report retains CHANGES_REQUESTED for the nonblocking P3 below. No GitHub CI checks are present; local checks are not CI-green evidence.
Known limitations
scrape_options={'formats': {'markdown': False, 'html': True}}, is accepted/coerced by the SDK but omits span format metadata. Independently reproduced for sync/async search, start-crawl and blocking crawl. SDK results and tested privacy remain intact. Use typedScrapeOptions/ScrapeFormatsfor demonstrated format telemetry; not every mapping representation is covered.Video demo
Recorded build: exact PR head
e983153cf327a24f1e57e5559bb52de25aee8e83. The source tree was clean before and after recording. The historical before-baseline is8a29398(reviewed, not released).Video: a 7 min 35 s narrated real-terminal walkthrough (1920x1080, H.264/AAC, 10 chapters). It is attached privately to TH-8325 together with captions (SRT), a transcript, chapter lists and a before/after preview. The media lives only in Future AGI's Linear workspace and is not shared publicly; sign-in is required. MP4 sha256 is
37f61c82c10b62262e5045bda8bffb3f5156c45d6803620e0f6d88f0ed5fb790(15,712,638 bytes).Chapters:
How it was made:
Media verification:
Not shown: live Firecrawl API, collector, backend, tenancy or trace view. This PR has no GitHub CI checks.
No merge, deploy or package release performed.
Verification sweep (2026-10-12)
dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).firecrawl.searchis now a RETRIEVER span withgen_ai.retrieval.query; the other calls stay TOOL spans; the kind key isgen_ai.span.kind.Re-verification (2026-10-16)
Head tested:
8c1289f2eaa2d0012e9c11e79e24eae90995aed9. No code or test changes were needed.localhost:14338, uncompressed). The Firecrawl API itself is a loopback stub of the documented v2 API (STANDIN, no liveapi.firecrawl.dev). Run: AGENT parent, scrape, search, map, crawl, start_crawl, get_crawl_status, cancel_crawl, one scrape returning HTTP 500 (ERROR), one async search.035026fb-5939-478e-990a-f2e140374011):[agent] research-agentwith children[tool] firecrawl.scrape/map/crawl/start_crawl/get_crawl_status/cancel_crawl,[retriever] firecrawl.searchx2, and the failing scrapestatus=ERROR:InternalServerError. Same tree on 4.46.2. UI trace view checked by eye: tree, ERROR badge and agent graph correct.spanscolumns scanned): default mode, the marker appears only in the search query (gen_ai.retrieval.query,input), by design; page body, title, URL path/query, error text, API key, username, hostname,/Users/and127.0.0.1: 0 hits. WithFI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true: 0 hits for the marker and everything else,inputempty. Same on both versions. Error event carries only the exception name.Live vendor verification (2026-10-17)
Real Firecrawl API (
api.firecrawl.dev, no base-URL override), run as the README tells a customer, exported to local Future AGI. Versions: firecrawl-py 4.50.0, fi-instrumentation-otel 1.1.0, traceAI-firecrawl 0.1.0, Python 3.12. Spend: about 15 free-plan credits.fc0dbe7b-cf70-494e-a5ed-d3b06b91e4df(project th8103-rick-firecrawl-live),fi_verifyPASS, 9 spans: AGENT research-agent > TOOL scrape, RETRIEVER search, TOOL map, start_crawl, get_crawl_status, cancel_crawl, crawl, and a wrong-key scrape ending ERROR UnauthorizedError (firecrawl.error.status_code=401).server.address=example.com,firecrawl.formats=["markdown"],firecrawl.result_count=2, sharedfirecrawl.job_id, crawlstatus=completed,page_count=1,credits_used=1,limit=1.start_crawlERRORRateLimitError, status code 429, vendor text not copied (trace0e189c43-63d1-4420-b7c1-42bdb92dbca5).gen_ai.retrieval.query/input.value; page body, URL path/query, wrong and real keys absent. WithFI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=truethe query marker is absent.watcher(), failed/cancelled crawl job statuses, and Python 3.10, 3.11, 3.13 end-to-end (suite-only). Pinned 4.46.2 was not run live.🤖 Generated with Claude Code