Haystack components for live web search, webpage and YouTube transcript retrieval, and bounded website crawling with Context.dev.
pip install context-dev-haystackCreate an API key in the Context.dev dashboard, then export it:
export CONTEXT_API_KEY="your-api-key"| Component | Purpose | Import |
|---|---|---|
ContextWebSearch |
Search the live web and return ranked Haystack Documents and source links | haystack_integrations.components.websearch.context |
ContextFetcher |
Fetch webpages or YouTube videos as clean Markdown Documents | haystack_integrations.components.fetchers.context |
ContextCrawler |
Crawl websites into Documents with explicit page and depth limits | haystack_integrations.components.fetchers.context |
All components support both run() and run_async(), Haystack serialization, custom timeouts, and retry configuration.
from haystack_integrations.components.websearch.context import ContextWebSearch
search = ContextWebSearch(top_k=5, include_markdown=True)
result = search.run(query="Recent advances in retrieval-augmented generation")
documents = result["documents"]
links = result["links"]Use include_domains, exclude_domains, freshness, and country to constrain results. Extra Context.dev Search API fields can be supplied through search_params.
from haystack_integrations.components.fetchers.context import ContextFetcher
fetcher = ContextFetcher()
result = fetcher.run(
urls=[
"https://haystack.deepset.ai",
"https://www.youtube.com/watch?v=UF8uR6Z6KLc",
]
)
documents = result["documents"]Each URL becomes a Haystack Document. Webpages contain clean Markdown and page metadata; supported YouTube URLs return timestamped transcript Markdown.
from haystack_integrations.components.fetchers.context import ContextCrawler
crawler = ContextCrawler(crawl_params={"maxPages": 25, "maxDepth": 2})
result = crawler.run(urls=["https://docs.haystack.deepset.ai"])
documents = result["documents"]ContextCrawler defaults to one page to prevent accidental credit consumption. Set maxPages explicitly for larger crawls.
result = await search.run_async(query="Haystack agents")
documents = result["documents"]The fetcher and crawler process multiple input URLs concurrently in their async methods.
See CONTRIBUTING.md for the Hatch-based development and release workflow.
Apache-2.0. See LICENSE.