You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7c24765
Browse filesBrowse the repository at this point in the historyBrowse files
Reuse HTTPS connections and negotiate HTTP/2 per IPData instance
Replace the per-instance requests.Session with one httpx.Client so that
every IPData instance keeps a persistent connection pool (20 connections,
60 s keep-alive) for its whole life and multiplexes requests over HTTP/2
when api.ipdata.co negotiates it, falling back to HTTP/1.1 keep-alive
otherwise. A first request to the API costs a TCP + TLS 1.3 handshake
(roughly 660 to 980 ms); a request on a reused connection costs about a
third of that, so clients should be created once and reused.
- IPData.__init__ keeps its signature and gains http2=True and
transport=None keyword arguments; the latter injects any
httpx.BaseTransport (httpx.MockTransport in the new offline tests).
When no transport is injected httpx builds its default one, which
honours HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE like requests did and
applies the pool limits; an explicit HTTPTransport would have ignored
both, so the connection-level `retries` knob is not used and connect
failures are retried by the retry loop below instead.
- retry_limit and retry_backoff_factor were dead: the retrying adapter
was mounted on the prefix "http" which requests never selects. They
now work as the docstring always described: GET and POST retry on
500, 502, 503 and 504 and on httpx.TransportError (reset, EOF or
timeout on a stale pooled connection) after sleeping
retry_backoff_factor * 2 ** (attempt - 2) seconds, i.e. 0.5, 1, 2, 4,
8, 16 and 32 s by default. 429 is retried only when the response has
a Retry-After header (sleep min(Retry-After, 60 s)); otherwise it is
returned at once so an exhausted quota fails fast.
- bulk() now uses the client timeout; it sent its POST without one.
- Add close(), __enter__ and __exit__ to release the pool.
- The User-Agent becomes ipdata-python/<version> using
importlib.metadata (fallback "unknown") and requests send
Accept: application/json.
- geofeeds.py downloads with httpx.get(..., follow_redirects=True), as
requests followed redirects by default.
- Dependency: requests -> httpx[http2]>=0.28,<1 in setup.cfg and
requirements.txt. Version 4.0.9 -> 4.1.0.
- CI: the workflow gains a pull_request trigger and a Python 3.9-3.13
matrix running `python -m pytest -m "not api"`; the existing job
(full suite with the API key secret, build, tag-gated publish) keeps
running on pushes to master and tags only.
- Tests: offline tests through httpx.MockTransport cover lookup, bulk
(timeout present), the retry schedule with a patched sleep, 429
fast-fail and Retry-After, close/context manager, user-agent and
api-key parameters, transport reuse across calls and the module-level
default client. The four live tests stay marked `api`.
- README: new "Connection reuse and HTTP/2" section and offline test
instructions.
Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@@ -367,6 +368,24 @@ NOTE: Alternatively it is much simpler to process bulk lookups using the `ipdata
367
368
```
368
369
</details>
369
370
371
+
### Connection reuse and HTTP/2
372
+
373
+
An `IPData` client keeps a persistent HTTPS connection pool for as long as it lives. Create one client per process (or one per API key or endpoint), keep it, and reuse it for every lookup: each new client pays a fresh TLS handshake. Requests to api.ipdata.co are multiplexed over HTTP/2 when the platform negotiates it, and fall back to HTTP/1.1 keep-alive otherwise.
374
+
375
+
The client is built on [httpx](https://www.python-httpx.org/) and is thread-safe: one instance can serve every thread of a process at once (the CLI's `batch` command shares a single client across its thread pool). Idle connections are kept open for 60 seconds and up to 20 connections are pooled. Call `close()` at shutdown, or use the client as a context manager, to release the pool. Pass `http2=False` to stay on HTTP/1.1 keep-alive, or `transport=` to inject your own `httpx.BaseTransport` (for example an `httpx.MockTransport` in tests).
376
+
377
+
```python
378
+
from ipdata import IPData
379
+
380
+
with IPData("<YOUR API KEY>") as client: # one client, many lookups
381
+
for ip in ["8.8.8.8", "1.1.1.1"]:
382
+
print(client.lookup(ip).country_name)
383
+
```
384
+
385
+
The module-level helpers `ipdata.lookup()` and `ipdata.bulk()` share one process-wide client, `ipdata.default_client`, which is created on first use and lives until the process exits. Connections must not be shared across `os.fork()` or `multiprocessing` boundaries: create your `IPData` instance (or make the first module-level call) in the child process, after forking.
386
+
387
+
Failed requests are retried up to `retry_limit` times (default 7). GET and POST requests that fail with a transport error (connection reset, timeout) or with HTTP 500, 502, 503 or 504 are retried with exponential backoff: 0.5, 1, 2, 4, 8, 16 and 32 seconds for the default `retry_backoff_factor=1`. A 429 response is retried only when it carries a `Retry-After` header (waiting at most 60 seconds); without one it is returned immediately, so an exhausted quota fails fast. The `HTTPS_PROXY` and `NO_PROXY` environment variables are honoured; a custom CA bundle is read from `SSL_CERT_FILE` (httpx does not read `REQUESTS_CA_BUNDLE`).
388
+
370
389
## Using the ipdata CLI
371
390
372
391
@@ -754,3 +773,9 @@ To run all tests
754
773
```shell
755
774
pytest
756
775
```
776
+
777
+
The tests marked `api` call the live API and need an `IPDATA_API_KEY` environment variable (or a `src/ipdata/.env` file). To run only the offline tests, as CI does for pull requests:
:class:`~.IPData` is the primary class for making API requests.
19
19
"""
20
20
21
-
importos
21
+
importemail.utils
22
+
importimportlib.metadata
22
23
importipaddress
23
-
importrequests
24
24
importlogging
25
-
importurllib3
26
-
importfunctools
25
+
importos
26
+
importtime
27
+
fromdatetimeimportdatetime, timezone
27
28
28
-
fromrequests.adaptersimportHTTPAdapter, Retry
29
+
importhttpx
29
30
30
31
logger=logging.getLogger(__name__)
31
32
logger.addHandler(logging.NullHandler())
32
33
34
+
try:
35
+
__version__=importlib.metadata.version("ipdata")
36
+
exceptimportlib.metadata.PackageNotFoundError: # running from a source checkout
37
+
__version__="unknown"
38
+
39
+
# HTTP statuses that are retried, for GET and POST requests alike
40
+
RETRY_STATUSES=frozenset({500, 502, 503, 504})
41
+
# Longest pause honoured from a 429 Retry-After header, in seconds
42
+
MAX_RETRY_AFTER=60.0
43
+
44
+
# Seam for the tests, which replace it to check the retry schedule without sleeping
45
+
_sleep=time.sleep
46
+
33
47
34
48
classIPDataException(Exception):
35
49
pass
@@ -57,12 +71,26 @@ class IPData(object):
57
71
"""
58
72
Instances of IPData are used to make API requests. To make requests to the EU endpoint, set "endpoint" to "https://eu-api.ipdata.co".
59
73
74
+
Every instance owns one HTTPS connection pool that is kept alive and reused for all of its requests
75
+
(multiplexed over HTTP/2 when the server negotiates it, HTTP/1.1 keep-alive otherwise), so create one
76
+
client per process (or per API key or endpoint), reuse it for every lookup and call :meth:`close` (or use
77
+
the instance as a context manager) when you are done with it. Instances are thread-safe. A client must
78
+
be created after forking: never share one across os.fork() or multiprocessing boundaries.
79
+
80
+
Failed requests are retried up to 'retry_limit' times. GET and POST requests that fail with a transport
81
+
error (connection reset, timeout, ...) or with HTTP 500, 502, 503 or 504 are retried after sleeping
82
+
retry_backoff_factor * 2 ** (attempt - 2) seconds, i.e. 0.5, 1, 2, 4, 8, 16 and 32 seconds with the
83
+
defaults. HTTP 429 is retried only when the response carries a Retry-After header (sleeping at most 60
84
+
seconds); without one the response is returned immediately so that an exhausted quota fails fast.
85
+
60
86
:param api_key: A valid IPData API key
61
87
:param endpoint: The API endpoint to send requests to
62
-
:param timeout: The default requests timeout
88
+
:param timeout: The default requests timeout in seconds, applied to every request including bulk lookups
63
89
:param retry_limit: The maximum number of retries in case of failure
64
90
:param retry_backoff_factor: The number of seconds to sleep in between retries. With a 1 second backoff calls with be retried up to 7 times at the following intervals: 0.5, 1, 2, 4, 8, 16, 32
65
91
:param debug: A boolean used to set the log level. Set to True when debugging.
92
+
:param http2: Negotiate HTTP/2 with automatic fallback to HTTP/1.1. Set to False to use HTTP/1.1 keep-alive only.
93
+
:param transport: An optional httpx.BaseTransport used instead of the default HTTPS transport, for example an httpx.MockTransport in tests
0 commit comments