diff --git a/README.md b/README.md index f586dd8..aa061f2 100644 --- a/README.md +++ b/README.md @@ -1,457 +1,446 @@ -# SerpApi Python Library & Package +# SerpApi Python Library [![Package](https://img.shields.io/pypi/v/serpapi?color=green)](https://pypi.org/project/serpapi) [![serpapi-python](https://github.com/serpapi/serpapi-python/actions/workflows/ci.yml/badge.svg)](https://github.com/serpapi/serpapi-python/actions/workflows/ci.yml) -Integrate search data into your AI workflow, RAG / fine-tuning, or Python application using this official wrapper for [SerpApi](https://serpapi.com). +Integrate search data into your AI workflow, RAG / fine-tuning, or Python application using this official [SerpApi Python SDK](https://serpapi.com/integrations/python). -SerpApi supports Google, Google Maps, Google Shopping, Baidu, Yandex, Yahoo, eBay, App Stores, and [more](https://serpapi.com). +SerpApi supports Google, Google Maps, Google Shopping, Baidu, Yandex, Yahoo, eBay, app stores, and [more](https://serpapi.com). -Query a vast range of data at scale, including web search results, flight schedules, stock market data, news headlines, and [more](https://serpapi.com). +> See the [full Python SDK documentation](https://serpapi-python.readthedocs.io/en/latest/) for setup, usage guides, and API reference. ## Installation -To install the `serpapi` package, simply run the following command: +Python 3.6 or later is required. + +To install the `serpapi` package, simply run the following command inside your Python virtual environment: ```bash -$ pip install serpapi +pip install serpapi ``` -Please note that this package is separate from the legacy `serpapi` module, which is available on PyPi as `google-search-results`. This package is maintained by SerpApi, and is the recommended way to access the SerpApi service from Python. +## Quickstart + +[Create a SerpApi account](https://serpapi.com/users/sign_up?plan=free) to get your API key, then store it in an environment variable: + +```bash +export SERPAPI_KEY="your_api_key" +``` -## Quick start +Environment variables keep secrets out of your source code. Read the API key with `os.environ["SERPAPI_KEY"]`. Python raises `KeyError` if the variable is missing. -Let's start by searching for Coffee on Google: +Run a Google search and access the structured results: ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("SERPAPI_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - "engine": "google", - "q": "coffee" + "engine": "google", + "q": "coffee" }) print(results) ``` -The `results` variable now contains a `SerpResults` object, which acts just like a standard dictionary, with some convenient functions added on top. - -This example runs a search for "coffee" on Google. It then returns the results as a regular Python Hash. - See the [playground](https://serpapi.com/playground) to generate your own code. - -The SerpApi key can be obtained from [serpapi.com/signup](https://serpapi.com/users/sign_up?plan=free). +The `results` variable contains a dict-like `SerpResults` object. Read fields with `results["organic_results"]` or `results.get("organic_results", [])`. -Environment variables are a secure, safe, and easy way to manage secrets. - Set `export SERPAPI_KEY=` in your shell. - Python accesses these variables from `os.environ["SERPAPI_KEY"]`. +This example runs a search for "coffee" on Google. See the [Playground](https://serpapi.com/playground) to generate your own code. -### Extract content from a URL +For Windows setup and instructions to save and run the example, see the [getting-started guide](https://serpapi-python.readthedocs.io/en/latest/user_guide/getting-started.html). -The [Extract API](https://serpapi.com/extract-api) supports single-page scraping, URL discovery, and crawling. Call `client.extract()` with a parameter dictionary or keyword arguments. The client sets `engine="extract"` automatically. +### Error handling -Set `output="md"` to get the page's main content as a Markdown string: +HTTP failures raise `serpapi.HTTPError`. Connection failures raise `serpapi.HTTPConnectionError`, and read timeouts raise `serpapi.TimeoutError`. See the [status and error codes documentation](https://serpapi.com/api-status-and-error-codes) for the meaning of each HTTP status. ```python import os + import serpapi -client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60) -markdown = client.extract(url="https://serpapi.com/", output="md") -print(markdown) +# A default timeout can be set here. +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=10) + +try: + results = client.search({ + "engine": "google", + "q": "coffee", + }) +except serpapi.TimeoutError: + raise RuntimeError("The request timed out.") from None +except serpapi.HTTPConnectionError: + raise RuntimeError("Could not connect to SerpApi.") from None +except serpapi.HTTPError as e: + raise RuntimeError(f"HTTP {e.status_code}: {e.error or 'Request failed'}") from None + +if "error" in results: + raise RuntimeError(results["error"]) ``` -JSON is the default output format. It returns a dictionary-like `SerpResults` object with page information, structured data, links, and Markdown: +A successful HTTP response can still contain an `error` field, such as when a search returns no results. Check it before reading result fields. See [Errors and timeouts](https://serpapi-python.readthedocs.io/en/latest/user_guide/errors-and-timeouts.html) for more examples. + +## Features + +- **Asynchronous searches** for submitting jobs and retrieving completed results from the Search Archive API. +- **Persistent connections and connection pooling** for reusing HTTP connections across searches. +- Search results as a dict-like **SerpResults** object by default, token-efficient **Markdown** with `"output": "md"`, or raw **search-engine HTML** with `"output": "html"`. +- SDK methods for the [Image API](https://serpapi.com/image-api), [Location API](https://serpapi.com/locations-api), [Search Archive API](https://serpapi.com/search-archive-api), and [Account API](https://serpapi.com/account-api). +- **Content extraction** with `client.extract()` for reading pages, discovering URLs, and crawling websites. +- **Configurable HTTP timeouts** established by setting `timeout` in the client. + +## Migrating from google-search-results + +If you use the legacy `google-search-results` package, follow the [migration guide](https://serpapi-python.readthedocs.io/en/latest/user_guide/migrating-from-google-search-results.html) to update your code. Both packages use `import serpapi`, so install only one in your environment. + +## Response formats + +| Output | Use it for | Returns | +|--------|------------|---------| +| `json` (default) | Reading fields, filtering, or storing results | Dict-like `SerpResults` | +| `md` | Passing content directly to an LLM | Markdown `str` | +| `html` | Inspecting or processing raw HTML | HTML `str` | + +Omit `output` or set `output="json"` for JSON: ```python -results = client.extract(url="https://serpapi.com/") -print(results["page_information"]["title"]) -print(results["markdown"]) +results = client.search({ + "engine": "google", + "q": "coffee" +}) ``` -Use `output="html"` to get the retrieved HTML as a string: +Use `results.as_dict()` when another library needs a plain `dict`, including before calling `json.dumps()`: ```python -html = client.extract(url="https://serpapi.com/", output="html") +data = results.as_dict() ``` -Use `mode="map"` to discover URLs or `mode="crawl"` to extract multiple pages: +Set `output="md"` for Markdown: ```python -urls = client.extract(url="https://serpapi.com/", mode="map", include_sitemap=False) -pages = client.extract( - url="https://serpapi.com/", mode="crawl", max_pages=2, max_depth=1, timeout=180 -) +markdown = client.search({ + "engine": "google", + "q": "coffee", + "output": "md" +}) ``` -Crawl mode can consume one search credit per successfully extracted page, including the starting page. Set `max_pages` to limit the number of pages. +Set `output="html"` for HTML: -See the [Extract API guide](https://github.com/serpapi/serpapi-python/blob/master/docs/user_guide/extract.md) for response examples and more options. - -### Error handling +```python +raw_html = client.search({ + "engine": "google", + "q": "coffee", + "output": "html" +}) +``` -Unsuccessful requests raise `serpapi.HTTPError` or `serpapi.TimeoutError` exceptions. The returned status code will reflect the sort of error that occurred, please refer to [Status and Error Codes Documentation](https://serpapi.com/api-status-and-error-codes) for more details. +Archived results are also available as Markdown with +`client.search_archive(search_id=search_id, output="md")`. -```python -import os -import serpapi +Learn more about [SerpApi Markdown Output](https://serpapi.com/markdown-output). -# A default timeout can be set here. -client = serpapi.Client(api_key=os.getenv("API_KEY"), timeout=10) +## Configuration -try: - results = client.search({ - 'engine': 'google', - 'q': 'coffee', - }) -except serpapi.HTTPError as e: - if e.status_code == 401: # Invalid API key - print(e.error) # "Invalid API key. Your API key should be here: https://serpapi.com/manage-api-key" - elif e.status_code == 400: # Missing required parameter - pass - elif e.status_code == 429: # Exceeds the hourly throughput limit OR account run out of searches - pass -except serpapi.TimeoutError as e: - # Handle timeout - print(f"The request timed out: {e}") -``` +| Option | Default | Description | +|--------|---------|-------------| +| `api_key` | `None` | Your SerpApi API key. | +| `timeout` | `None` | Connection and read timeout in seconds. No timeout is set by default. | -## Documentation +### Asynchronous searches -Documentation is [available on Read the Docs](https://serpapi-python.readthedocs.io/en/latest/). +Searches wait for their results by default. Use `{"async": True}` to submit a search job on SerpApi and retrieve its results later from the Search Archive API. -Change history is [available on GitHub](https://github.com/serpapi/serpapi-python/blob/master/HISTORY.md). +SerpApi's `async` parameter is separate from Python `asyncio`. The SDK methods are synchronous: each call blocks until its HTTP response arrives and cannot be used with `await`. -### Markdown output for AI agents +> **Note:** `async` is a reserved keyword in Python 3.7+, so pass it inside the parameters dictionary (`{"async": True}`) rather than as a keyword argument. -For AI agents and LLM workflows, use `output="md"`. It returns search results with clean headings, links, and tables while using roughly half the tokens of JSON on average. +Submit a search and check its status later: ```python -markdown = client.search({ +submitted = client.search({ "engine": "google", "q": "coffee", - "output": "md", + "async": True, }) +search_id = submitted["search_metadata"]["id"] + +# Retrieve later; the search may still be processing. +results = client.search_archive(search_id=search_id) +print(results["search_metadata"]["status"]) ``` -Markdown works across SerpApi APIs and is returned as a plain Python string. See [Markdown Output for AI Agents](https://serpapi.com/markdown-output.md) for details. +If the search is still processing, retrieve it again later. See the [async search guide](https://serpapi-python.readthedocs.io/en/latest/user_guide/async-search-archive.html) for polling with HTTP timeouts and a deadline. -Raw HTML is also available with `output="html"` and is returned as a plain Python string. +## Examples -## Basic examples +Here are some examples of our most popular APIs. You can find the full list of supported engines and parameters in our [documentation](https://serpapi.com/search-engine-apis). -### Search Bing -```python -import os -import serpapi +### Google Shopping -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'bing', - 'q': 'coffee' -}) -``` -- API Documentation: [serpapi.com/bing-search-api](https://serpapi.com/bing-search-api) +Scrape Google Shopping results with product names, prices, ratings, and merchant information. -### Search Baidu ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'baidu', - 'q': 'coffee', + "engine": "google_shopping", + "q": "Macbook M4" }) +shopping_results = results["shopping_results"] +print(shopping_results) ``` -- API Documentation: [serpapi.com/baidu-search-api](https://serpapi.com/baidu-search-api) -### Search Yahoo -```python -import os -import serpapi +[See documentation](https://serpapi.com/google-shopping-api) -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'yahoo', - 'p': 'coffee', -}) -``` -- API Documentation: [serpapi.com/yahoo-search-api](https://serpapi.com/yahoo-search-api) +**Google Shopping Light** -### Search YouTube -```python -import os -import serpapi +A [light variant](https://serpapi.com/google-shopping-light-api) engine called `google_shopping_light` is also available for faster, lower-cost shopping searches. -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'youtube', - 'search_query': 'coffee', -}) -``` -- API Documentation: [serpapi.com/youtube-search-api](https://serpapi.com/youtube-search-api) +### Google Images + +Scrape Google Images search results, including image URLs, thumbnails, titles, and source pages. -### Search Walmart ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'walmart', - 'query': 'coffee', + "engine": "google_images", + "q": "coffee", }) +print(results) ``` -- API Documentation: [serpapi.com/walmart-search-api](https://serpapi.com/walmart-search-api) -### Search eBay -```python -import os -import serpapi +[See documentation](https://serpapi.com/google-images-api) -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'ebay', - '_nkw': 'coffee', -}) -``` -- API Documentation: [serpapi.com/ebay-search-api](https://serpapi.com/ebay-search-api) +**Google Images Light** -### Search Naver -```python -import os -import serpapi +A [light variant](https://serpapi.com/google-images-light-api) engine called `google_images_light` is also available for faster, lower-cost image searches. -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'naver', - 'query': 'coffee', -}) -``` -- API Documentation: [serpapi.com/naver-search-api](https://serpapi.com/naver-search-api) +### Extract API + +Extract page content from a public URL as Markdown for an LLM: -### Search Home Depot ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'home_depot', - 'q': 'chair', -}) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60) +markdown = client.extract(url="https://serpapi.com/", output="md") +print(markdown) ``` -- API Documentation: [serpapi.com/home-depot-search-api](https://serpapi.com/home-depot-search-api) -### Search Apple App Store -```python -import os -import serpapi +Omit `output` or set `output="json"` for a dict-like `SerpResults` object with structured page data and Markdown. Use `output="html"` for HTML as a string. -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'apple_app_store', - 'term': 'coffee', -}) -``` -- API Documentation: [serpapi.com/apple-app-store](https://serpapi.com/apple-app-store) +For URL discovery and crawling, see [Map Mode](https://github.com/serpapi/serpapi-python/blob/master/docs/user_guide/extract.md#discover-urls) and [Crawl Mode](https://github.com/serpapi/serpapi-python/blob/master/docs/user_guide/extract.md#crawl-multiple-pages) in the Python guide. -### Search DuckDuckGo -```python -import os -import serpapi +[See documentation](https://serpapi.com/extract-api) -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'duckduckgo', - 'q': 'coffee', -}) -``` -- API Documentation: [serpapi.com/duckduckgo-search-api](https://serpapi.com/duckduckgo-search-api) +### Google Lens with image URL or upload + +Google Lens accepts either a publicly accessible image URL or an uploaded +image. To search by URL, pass the URL directly: -### Search Google ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'google', - 'q': 'coffee' + "engine": "google_lens", + "url": "https://i.imgur.com/HBrB8p0.png", }) +print(results) ``` -- API Documentation: [serpapi.com/search-api](https://serpapi.com/search-api) -### Search Google Scholar -```python -import os -import serpapi +To search a local image, place a file named `image.png` in your working directory, upload it, and pass its temporary `image_id` to Google Lens: -client = serpapi.Client(api_key=os.getenv("API_KEY")) +```python +upload = client.upload_image("image.png") results = client.search({ - 'engine': 'google_scholar', - 'q': 'coffee', + "engine": "google_lens", + "image_id": upload["image_id"], }) ``` -- API Documentation: [serpapi.com/google-scholar-api](https://serpapi.com/google-scholar-api) -### Search Google Autocomplete +Uploaded images can be JPG/JPEG, PNG, or WebP files up to 500 KB. The returned +`image_id` expires after 10 minutes. + +[See Image API documentation](https://serpapi.com/image-api) · [See Google Lens image upload documentation](https://serpapi.com/google-lens-upload-an-image) + +### Google Trends + +Track search interest over time and compare the popularity of search terms. + ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'google_autocomplete', - 'q': 'coffee', + "engine": "google_trends", + "q": "coffee", + "data_type": "TIMESERIES" }) +interest_over_time = results["interest_over_time"] +print(interest_over_time) ``` -- API Documentation: [serpapi.com/google-autocomplete-api](https://serpapi.com/google-autocomplete-api) -### Search Google Immersive Product +[See documentation](https://serpapi.com/google-trends-api) -Get a current `page_token` from Google Shopping before requesting product details: +### Google Flights -```python -import os -import serpapi +Search flight routes, schedules, prices, and booking options. -client = serpapi.Client(api_key=os.getenv("API_KEY")) -shopping = client.search( - engine="google_shopping", - q="lg tv", - gl="us", - hl="en", - no_cache=True, -) -page_token = next( - product["immersive_product_page_token"] - for product in shopping["shopping_results"] - if product.get("immersive_product_page_token") -) -results = client.search( - engine="google_immersive_product", - page_token=page_token, -) -``` -- API Documentation: [serpapi.com/google-immersive-product-api](https://serpapi.com/google-immersive-product-api) +> Note: The `google_flights` engine does not use `q`. +> Specify route and date parameters such as `departure_id`, `arrival_id`, `outbound_date`, and `return_date`. -### Search Google Reverse Image ```python import os +from datetime import date, timedelta + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) +outbounding = date.today() + timedelta(days=30) +returning = outbounding + timedelta(weeks=2) + results = client.search({ - 'engine': 'google_reverse_image', - 'image_url': 'https://upload.wikimedia.org/wikipedia/commons/6/63/Wikipedia-logo.png', - 'max_results': '1', + "engine": "google_flights", + "departure_id": "LAX", + "arrival_id": "AUS", + "outbound_date": outbounding.isoformat(), + "return_date": returning.isoformat() }) +flights = results.get("best_flights") or results.get("other_flights", []) +print(flights) ``` -- API Documentation: [serpapi.com/google-reverse-image](https://serpapi.com/google-reverse-image) -### Search Google Lens by image URL or upload +[See documentation](https://serpapi.com/google-flights-api) -Google Lens accepts either a publicly accessible image URL or an uploaded -image. To search by URL, pass the URL directly: +### Google AI Mode API + +The Google AI Mode API returns AI-generated answers with structured text blocks, references, images, products, and more. ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("SERPAPI_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - "engine": "google_lens", - "url": "https://i.imgur.com/HBrB8p0.png", + "engine": "google_ai_mode", + "q": "best coffee maker" }) +text_blocks = results["text_blocks"] +print(text_blocks) ``` -To search a local image, place a file named `image.png` in your working directory, upload it, and pass its temporary `image_id` to Google Lens: +[See documentation](https://serpapi.com/google-ai-mode-api) -```python -upload = client.upload_image("image.png") -results = client.search({ - "engine": "google_lens", - "image_id": upload["image_id"], -}) -``` +### Bing Search -Uploaded images can be JPG/JPEG, PNG, or WebP files up to 500 KB. The returned -`image_id` expires after 10 minutes. - -- API Documentation: [Google Lens image uploads](https://serpapi.com/google-lens-upload-an-image), [Image API](https://serpapi.com/image-api) +Scrape Bing web search results, including organic results, ads, related searches, and more. -### Search Google Local Services ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'google_local_services', - 'q': 'electrician', - 'data_cid': '6745062158417646970', + "engine": "bing", + "q": "coffee" }) +print(results) ``` -- API Documentation: [serpapi.com/google-local-services-api](https://serpapi.com/google-local-services-api) -### Search Google Maps -```python -import os -import serpapi +[See documentation](https://serpapi.com/bing-search-api) +### DuckDuckGo Search -client = serpapi.Client(api_key=os.getenv("API_KEY")) -results = client.search({ - 'engine': 'google_maps', - 'q': 'pizza', - 'll': '@40.7455096,-74.0083012,15.1z', - 'type': 'search', -}) -``` -- API Documentation: [serpapi.com/google-maps-api](https://serpapi.com/google-maps-api) +Scrape DuckDuckGo search results, including organic results, ads, knowledge graphs, and related searches. -### Search Google Jobs ```python import os -import serpapi +import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'google_jobs', - 'q': 'software engineer', + "engine": "duckduckgo", + "q": "coffee", }) +print(results) ``` -- API Documentation: [serpapi.com/google-jobs-api](https://serpapi.com/google-jobs-api) -### Search Google Play +[See documentation](https://serpapi.com/duckduckgo-search-api) + +### Baidu Search + +Scrape Baidu search results, including organic results, answer boxes, and related searches. + ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'google_play', - 'q': 'kite', - 'store': 'apps', + "engine": "baidu", + "q": "coffee", }) +print(results) ``` -- API Documentation: [serpapi.com/google-play-api](https://serpapi.com/google-play-api) -### Search Google Images +[See documentation](https://serpapi.com/baidu-search-api) + +### Amazon Search + +Scrape Amazon product search results, including product names, prices, ratings, reviews, and availability. + +> Note: The `amazon` engine uses the `k` parameter for a keyword search, not `q`. + ```python import os + import serpapi -client = serpapi.Client(api_key=os.getenv("API_KEY")) +client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"]) results = client.search({ - 'engine': 'google_images', - 'tbm': 'isch', - 'q': 'coffee', + "engine": "amazon", + "k": "Coffee", + "amazon_domain": "amazon.com" }) +organic_results = results["organic_results"] +print(organic_results) ``` -- API Documentation: [serpapi.com/google-images-api](https://serpapi.com/google-images-api) + +[See documentation](https://serpapi.com/amazon-search-api) + +## More resources + +- [Python SDK documentation](https://serpapi-python.readthedocs.io/en/latest/) for setup, usage, and examples +- [SerpApi documentation](https://serpapi.com/search-api) for supported APIs and parameters +- [SerpApi status](https://serpapi.com/api-metrics) for API success rates and response times + +### Playground + +Build and test your request in the **[Playground](https://serpapi.com/playground)**, then generate Python code. + +### For agents + +- [AI agent guide](https://serpapi-python.readthedocs.io/en/latest/ai-agents.html) for using the Python SDK with AI agents +- [SerpApi llms.txt](https://serpapi.com/llms.txt) for links to each API's Markdown parameter reference ## License @@ -459,4 +448,6 @@ MIT License. ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing, and publishing instructions. +Contributions are welcome. Make sure to read our [contributing guide](CONTRIBUTING.md). + +© 2026 [SerpApi](https://serpapi.com/) diff --git a/docs/user_guide/async-search-archive.md b/docs/user_guide/async-search-archive.md index f669653..206cbd8 100644 --- a/docs/user_guide/async-search-archive.md +++ b/docs/user_guide/async-search-archive.md @@ -7,9 +7,9 @@ description: "Submit searches now and retrieve their results later with the Sear An asynchronous search lets you submit a request now and collect the results later. Set the API's `async` parameter to `true` to receive a search ID, then use that ID with the Search Archive API. Each Python client call still waits for an HTTP response. These methods do not use Python's `asyncio` or `await`. -Create a client as shown in [Client Usage](client-usage.md#create-a-client) before running the examples. +Create a client as shown in the [client usage guide](client-usage.md#create-a-client) before running the examples. -## Submit an Async Search +## Submit an async search `async` is a reserved word in Python. Put it in a dictionary and use `**` to pass that dictionary's entries as keyword arguments: @@ -19,40 +19,53 @@ submitted = client.search( q="coffee", location="Austin, Texas", **{"async": True}, + timeout=30, ) search_id = submitted["search_metadata"]["id"] print(search_id) ``` -## Retrieve the Search +## Retrieve the search ```python -archived = client.search_archive(search_id=search_id) +archived = client.search_archive(search_id=search_id, timeout=30) status = archived.get("search_metadata", {}).get("status") print(status) ``` -Check the status before reading the results. If the search is still processing, wait and call `search_archive()` again. This loop checks every two seconds until the search succeeds or reports an error: +## Poll for completion + +Check the status before reading the results. If the search is still processing, wait and call `search_archive()` again. This example checks every two seconds, with an HTTP timeout of up to 30 seconds per request and a 120-second polling deadline: ```python import time +deadline = time.monotonic() + 120 + while True: - archived = client.search_archive(search_id=search_id) + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError("Polling timed out before the search completed.") + + archived = client.search_archive( + search_id=search_id, timeout=min(30, remaining) + ) status = archived.get("search_metadata", {}).get("status") - if status == "Success": + if status in ("Success", "Cached"): break if status == "Error": raise RuntimeError(archived.get("error", "Search failed")) - time.sleep(2) + time.sleep(min(2, max(0, deadline - time.monotonic()))) ``` -## When to Use Async Searches +The deadline is checked between requests. If the search remains unfinished, the loop raises Python's built-in `TimeoutError`. The HTTP timeout limits connection and read waits for each request; it does not stop repeated successful responses that say the search is still processing. + +## When to use async searches Use asynchronous searches when one part of your program submits searches and another collects the results later. For example, a scheduled job could submit a batch of searches for another worker to process.