Skip to content

Commit 7c24765

Browse files
committed
Reuse HTTPS connections and negotiate HTTP/2 per IPData instance
Replace the per-instance requests.Session with one httpx.Client so that every IPData instance keeps a persistent connection pool (20 connections, 60 s keep-alive) for its whole life and multiplexes requests over HTTP/2 when api.ipdata.co negotiates it, falling back to HTTP/1.1 keep-alive otherwise. A first request to the API costs a TCP + TLS 1.3 handshake (roughly 660 to 980 ms); a request on a reused connection costs about a third of that, so clients should be created once and reused. - IPData.__init__ keeps its signature and gains http2=True and transport=None keyword arguments; the latter injects any httpx.BaseTransport (httpx.MockTransport in the new offline tests). When no transport is injected httpx builds its default one, which honours HTTPS_PROXY/NO_PROXY and SSL_CERT_FILE like requests did and applies the pool limits; an explicit HTTPTransport would have ignored both, so the connection-level `retries` knob is not used and connect failures are retried by the retry loop below instead. - retry_limit and retry_backoff_factor were dead: the retrying adapter was mounted on the prefix "http" which requests never selects. They now work as the docstring always described: GET and POST retry on 500, 502, 503 and 504 and on httpx.TransportError (reset, EOF or timeout on a stale pooled connection) after sleeping retry_backoff_factor * 2 ** (attempt - 2) seconds, i.e. 0.5, 1, 2, 4, 8, 16 and 32 s by default. 429 is retried only when the response has a Retry-After header (sleep min(Retry-After, 60 s)); otherwise it is returned at once so an exhausted quota fails fast. - bulk() now uses the client timeout; it sent its POST without one. - Add close(), __enter__ and __exit__ to release the pool. - The User-Agent becomes ipdata-python/<version> using importlib.metadata (fallback "unknown") and requests send Accept: application/json. - geofeeds.py downloads with httpx.get(..., follow_redirects=True), as requests followed redirects by default. - Dependency: requests -> httpx[http2]>=0.28,<1 in setup.cfg and requirements.txt. Version 4.0.9 -> 4.1.0. - CI: the workflow gains a pull_request trigger and a Python 3.9-3.13 matrix running `python -m pytest -m "not api"`; the existing job (full suite with the API key secret, build, tag-gated publish) keeps running on pushes to master and tags only. - Tests: offline tests through httpx.MockTransport cover lookup, bulk (timeout present), the retry schedule with a patched sleep, 429 fast-fail and Retry-After, close/context manager, user-agent and api-key parameters, transport reuse across calls and the module-level default client. The four live tests stay marked `api`. - README: new "Connection reuse and HTTP/2" section and offline test instructions. Claude-Session: https://claude.ai/code/session_01VQiRp4zEJKUXHrKkFaeUke Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 4d57900 commit 7c24765

8 files changed

Lines changed: 660 additions & 43 deletions

File tree

‎.github/workflows/python-publish.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,33 @@ on:
66
- master
77
tags:
88
- '*'
9+
pull_request:
910

1011
jobs:
12+
test:
13+
name: Test (Python ${{ matrix.python-version }})
14+
runs-on: ubuntu-latest
15+
strategy:
16+
fail-fast: false
17+
matrix:
18+
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
19+
steps:
20+
- uses: actions/checkout@v4
21+
- name: Set up Python ${{ matrix.python-version }}
22+
uses: actions/setup-python@v5
23+
with:
24+
python-version: ${{ matrix.python-version }}
25+
- name: Install dependencies
26+
run: |
27+
python -m pip install --upgrade pip
28+
python -m pip install -r requirements.txt
29+
- name: Test with pytest (offline tests only, no API key needed)
30+
run: |
31+
python -m pytest -m "not api"
32+
1133
build-n-publish:
1234
name: Build and publish to PyPI
35+
if: github.event_name == 'push'
1336
runs-on: ubuntu-latest
1437
steps:
1538
- uses: actions/checkout@v4

‎README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Visit our [Documentation](https://docs.ipdata.co/) for more examples and tutoria
2121
- [Getting only one field](#getting-only-one-field)
2222
- [Getting a number of specific fields](#getting-a-number-of-specific-fields)
2323
- [Bulk Lookups](#bulk-lookups)
24+
- [Connection reuse and HTTP/2](#connection-reuse-and-http2)
2425
- [Using the ipdata CLI](#using-the-ipdata-cli)
2526
- [Windows Installation Notes](#windows-installation-notes)
2627
- [Available commands](#available-commands)
@@ -367,6 +368,24 @@ NOTE: Alternatively it is much simpler to process bulk lookups using the `ipdata
367368
```
368369
</details>
369370

371+
### Connection reuse and HTTP/2
372+
373+
An `IPData` client keeps a persistent HTTPS connection pool for as long as it lives. Create one client per process (or one per API key or endpoint), keep it, and reuse it for every lookup: each new client pays a fresh TLS handshake. Requests to api.ipdata.co are multiplexed over HTTP/2 when the platform negotiates it, and fall back to HTTP/1.1 keep-alive otherwise.
374+
375+
The client is built on [httpx](https://www.python-httpx.org/) and is thread-safe: one instance can serve every thread of a process at once (the CLI's `batch` command shares a single client across its thread pool). Idle connections are kept open for 60 seconds and up to 20 connections are pooled. Call `close()` at shutdown, or use the client as a context manager, to release the pool. Pass `http2=False` to stay on HTTP/1.1 keep-alive, or `transport=` to inject your own `httpx.BaseTransport` (for example an `httpx.MockTransport` in tests).
376+
377+
```python
378+
from ipdata import IPData
379+
380+
with IPData("<YOUR API KEY>") as client: # one client, many lookups
381+
for ip in ["8.8.8.8", "1.1.1.1"]:
382+
print(client.lookup(ip).country_name)
383+
```
384+
385+
The module-level helpers `ipdata.lookup()` and `ipdata.bulk()` share one process-wide client, `ipdata.default_client`, which is created on first use and lives until the process exits. Connections must not be shared across `os.fork()` or `multiprocessing` boundaries: create your `IPData` instance (or make the first module-level call) in the child process, after forking.
386+
387+
Failed requests are retried up to `retry_limit` times (default 7). GET and POST requests that fail with a transport error (connection reset, timeout) or with HTTP 500, 502, 503 or 504 are retried with exponential backoff: 0.5, 1, 2, 4, 8, 16 and 32 seconds for the default `retry_backoff_factor=1`. A 429 response is retried only when it carries a `Retry-After` header (waiting at most 60 seconds); without one it is returned immediately, so an exhausted quota fails fast. The `HTTPS_PROXY` and `NO_PROXY` environment variables are honoured; a custom CA bundle is read from `SSL_CERT_FILE` (httpx does not read `REQUESTS_CA_BUNDLE`).
388+
370389
## Using the ipdata CLI
371390

372391

@@ -754,3 +773,9 @@ To run all tests
754773
```shell
755774
pytest
756775
```
776+
777+
The tests marked `api` call the live API and need an `IPDATA_API_KEY` environment variable (or a `src/ipdata/.env` file). To run only the offline tests, as CI does for pull requests:
778+
779+
```shell
780+
pytest -m "not api"
781+
```

‎requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
requests
1+
httpx[http2]>=0.28,<1
22
rich
33
click
44
click_default_group

‎setup.cfg‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[metadata]
22
name = ipdata
3-
version = 4.0.9
3+
version = 4.1.0
44
author = Jonathan Kosgei
55
author_email = jonathan@ipdata.co
66
description = This is the official IPData client library for Python
@@ -24,7 +24,7 @@ package_dir =
2424
packages = find:
2525
python_requires = >=3.9
2626
install_requires =
27-
requests
27+
httpx[http2]>=0.28,<1
2828
rich
2929
click
3030
click_default_group

‎src/ipdata/geofeeds.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
import random
99
from pathlib import Path
1010

11-
import requests
11+
import httpx
1212

1313
from .codes import COUNTRIES, REGION_CODES
1414

@@ -49,7 +49,7 @@ def _download(self):
4949
cache_path = f"{self.dir}/{random_name}.csv"
5050

5151
try:
52-
response = requests.get(self.source, timeout=60)
52+
response = httpx.get(self.source, timeout=60, follow_redirects=True)
5353
text = response.text
5454
with open(cache_path, "w") as f:
5555
f.write(

‎src/ipdata/ipdata.py‎

Lines changed: 143 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -18,18 +18,32 @@
1818
:class:`~.IPData` is the primary class for making API requests.
1919
"""
2020

21-
import os
21+
import email.utils
22+
import importlib.metadata
2223
import ipaddress
23-
import requests
2424
import logging
25-
import urllib3
26-
import functools
25+
import os
26+
import time
27+
from datetime import datetime, timezone
2728

28-
from requests.adapters import HTTPAdapter, Retry
29+
import httpx
2930

3031
logger = logging.getLogger(__name__)
3132
logger.addHandler(logging.NullHandler())
3233

34+
try:
35+
__version__ = importlib.metadata.version("ipdata")
36+
except importlib.metadata.PackageNotFoundError: # running from a source checkout
37+
__version__ = "unknown"
38+
39+
# HTTP statuses that are retried, for GET and POST requests alike
40+
RETRY_STATUSES = frozenset({500, 502, 503, 504})
41+
# Longest pause honoured from a 429 Retry-After header, in seconds
42+
MAX_RETRY_AFTER = 60.0
43+
44+
# Seam for the tests, which replace it to check the retry schedule without sleeping
45+
_sleep = time.sleep
46+
3347

3448
class IPDataException(Exception):
3549
pass
@@ -57,12 +71,26 @@ class IPData(object):
5771
"""
5872
Instances of IPData are used to make API requests. To make requests to the EU endpoint, set "endpoint" to "https://eu-api.ipdata.co".
5973
74+
Every instance owns one HTTPS connection pool that is kept alive and reused for all of its requests
75+
(multiplexed over HTTP/2 when the server negotiates it, HTTP/1.1 keep-alive otherwise), so create one
76+
client per process (or per API key or endpoint), reuse it for every lookup and call :meth:`close` (or use
77+
the instance as a context manager) when you are done with it. Instances are thread-safe. A client must
78+
be created after forking: never share one across os.fork() or multiprocessing boundaries.
79+
80+
Failed requests are retried up to 'retry_limit' times. GET and POST requests that fail with a transport
81+
error (connection reset, timeout, ...) or with HTTP 500, 502, 503 or 504 are retried after sleeping
82+
retry_backoff_factor * 2 ** (attempt - 2) seconds, i.e. 0.5, 1, 2, 4, 8, 16 and 32 seconds with the
83+
defaults. HTTP 429 is retried only when the response carries a Retry-After header (sleeping at most 60
84+
seconds); without one the response is returned immediately so that an exhausted quota fails fast.
85+
6086
:param api_key: A valid IPData API key
6187
:param endpoint: The API endpoint to send requests to
62-
:param timeout: The default requests timeout
88+
:param timeout: The default requests timeout in seconds, applied to every request including bulk lookups
6389
:param retry_limit: The maximum number of retries in case of failure
6490
:param retry_backoff_factor: The number of seconds to sleep in between retries. With a 1 second backoff calls with be retried up to 7 times at the following intervals: 0.5, 1, 2, 4, 8, 16, 32
6591
:param debug: A boolean used to set the log level. Set to True when debugging.
92+
:param http2: Negotiate HTTP/2 with automatic fallback to HTTP/1.1. Set to False to use HTTP/1.1 keep-alive only.
93+
:param transport: An optional httpx.BaseTransport used instead of the default HTTPS transport, for example an httpx.MockTransport in tests
6694
"""
6795

6896
log = logging.getLogger(__name__)
@@ -103,6 +131,8 @@ def __init__(
103131
retry_limit=7,
104132
retry_backoff_factor=1,
105133
debug=False,
134+
http2=True,
135+
transport=None,
106136
):
107137
if api_key is None:
108138
api_key = os.environ.get("IPDATA_API_KEY")
@@ -112,7 +142,12 @@ def __init__(
112142
self.api_key = api_key
113143
self.endpoint = endpoint.rstrip("/") # remove trailing /
114144
self._timeout = timeout
115-
self._headers = {"user-agent": "ipdata-python"} # set default UserAgent
145+
self._retry_limit = retry_limit
146+
self._retry_backoff_factor = retry_backoff_factor
147+
self._headers = {
148+
"user-agent": f"ipdata-python/{__version__}", # set default UserAgent
149+
"accept": "application/json",
150+
}
116151
self._query_params = {"api-key": self.api_key}
117152

118153
# Enable debugging
@@ -126,24 +161,37 @@ def __init__(
126161
handler.setFormatter(logging.Formatter("%(message)s", datefmt="[%X]"))
127162
self.log.addHandler(handler)
128163

129-
# Work around renamed argument in urllib3.
130-
if hasattr(urllib3.util.Retry.DEFAULT, "allowed_methods"):
131-
methods_arg = "allowed_methods"
132-
else:
133-
methods_arg = "method_whitelist"
134-
135-
# Retry settings
136-
retry_args = {
137-
"total": retry_limit,
138-
"backoff_factor": retry_backoff_factor,
139-
"status_forcelist": [429, 500, 502, 503, 504],
140-
methods_arg: {"POST"},
141-
}
164+
# One connection pool per client instance. Connections are kept alive for 60 seconds and reused
165+
# across requests (multiplexed over HTTP/2 when the server negotiates it, HTTP/1.1 keep-alive
166+
# otherwise) instead of paying a TCP + TLS handshake per call. When no transport is injected
167+
# httpx builds its default one, which honours HTTPS_PROXY / NO_PROXY and the SSL_CERT_* variables
168+
# like requests did; with an injected transport httpx uses it as-is and ignores http2 and limits.
169+
self._client = httpx.Client(
170+
base_url=self.endpoint,
171+
http2=http2,
172+
timeout=httpx.Timeout(timeout),
173+
headers=self._headers,
174+
params=self._query_params,
175+
limits=httpx.Limits(
176+
max_connections=20, max_keepalive_connections=20, keepalive_expiry=60.0
177+
),
178+
transport=transport,
179+
)
180+
181+
def close(self):
182+
"""
183+
Closes the connection pool. The instance can not be used afterwards. Call this at shutdown, or use the instance as a context manager:
184+
185+
>>> with IPData("YOUR API KEY") as ipdata:
186+
... ipdata.lookup("8.8.8.8")
187+
"""
188+
self._client.close()
142189

143-
adapter = HTTPAdapter(max_retries=urllib3.Retry(**retry_args))
190+
def __enter__(self):
191+
return self
144192

145-
self._session = requests.Session()
146-
self._session.mount("http", adapter)
193+
def __exit__(self, exc_type, exc_value, traceback):
194+
self.close()
147195

148196
def _validate_fields(self, fields):
149197
"""
@@ -173,6 +221,74 @@ def _validate_ip_address(self, ip):
173221
if request_ip.is_private or request_ip.is_reserved or request_ip.is_multicast:
174222
raise ValueError(f"{ip} is a reserved IP Address")
175223

224+
def _backoff(self, attempt):
225+
"""
226+
Seconds to sleep before retry number 'attempt' (1-based): 0.5, 1, 2, 4, 8, 16, 32 with the default backoff factor.
227+
"""
228+
return self._retry_backoff_factor * 2 ** (attempt - 2)
229+
230+
@staticmethod
231+
def _retry_after(value):
232+
"""
233+
Parses a Retry-After header, given either as delay-seconds or as an HTTP-date.
234+
235+
:returns: The number of seconds to wait, or None if the value can not be parsed
236+
"""
237+
value = value.strip()
238+
if value.isdigit():
239+
return float(value)
240+
try:
241+
when = email.utils.parsedate_to_datetime(value)
242+
except (TypeError, ValueError):
243+
return None
244+
if when.tzinfo is None:
245+
when = when.replace(tzinfo=timezone.utc)
246+
return max((when - datetime.now(timezone.utc)).total_seconds(), 0.0)
247+
248+
def _request(self, method, path, **kwargs):
249+
"""
250+
Sends a request over the shared connection pool, retrying it as described in the class docstring.
251+
252+
:returns: The last httpx.Response received
253+
:raises httpx.TransportError: if the request still fails to get a response once the retries are exhausted
254+
"""
255+
attempt = 0
256+
while True:
257+
attempt += 1
258+
try:
259+
response = self._client.request(method, path, **kwargs)
260+
except httpx.UnsupportedProtocol:
261+
raise # a configuration error that no retry can fix
262+
except httpx.TransportError as e:
263+
if attempt > self._retry_limit:
264+
raise
265+
delay = self._backoff(attempt)
266+
reason = f"{type(e).__name__}: {e}"
267+
else:
268+
status_code = response.status_code
269+
if attempt > self._retry_limit:
270+
return response
271+
if status_code in RETRY_STATUSES:
272+
delay = self._backoff(attempt)
273+
elif status_code == 429 and "retry-after" in response.headers:
274+
delay = self._retry_after(response.headers["retry-after"])
275+
if delay is None:
276+
delay = self._backoff(attempt)
277+
delay = min(delay, MAX_RETRY_AFTER)
278+
else:
279+
return response
280+
reason = f"HTTP {status_code}"
281+
self.log.debug(
282+
"Retrying %s %s in %.1fs (retry %d of %d) after %s",
283+
method,
284+
path,
285+
delay,
286+
attempt,
287+
self._retry_limit,
288+
reason,
289+
)
290+
_sleep(delay)
291+
176292
def lookup(self, resource="", fields=[], select_field=None):
177293
"""
178294
Makes a GET request to the IPData API for the specified 'resource' and the given 'fields'.
@@ -204,16 +320,11 @@ def lookup(self, resource="", fields=[], select_field=None):
204320
fields = [select_field]
205321

206322
self._validate_fields(fields)
207-
query_params = self._query_params | {"fields": ",".join(fields)}
323+
query_params = {"fields": ",".join(fields)}
208324

209325
# Make the request
210326
try:
211-
response = self._session.get(
212-
f"{self.endpoint}/{resource}",
213-
headers=self._headers,
214-
params=query_params,
215-
timeout=self._timeout,
216-
)
327+
response = self._request("GET", f"/{resource}", params=query_params)
217328
except Exception as e:
218329
raise IPDataException(e)
219330

@@ -253,16 +364,11 @@ def bulk(self, resources, fields=[]):
253364
raise ValueError("Bulk lookups must contain at least 1 resource.")
254365

255366
self._validate_fields(fields)
256-
query_params = self._query_params | {"fields": ",".join(fields)}
367+
query_params = {"fields": ",".join(fields)}
257368

258369
# Make the requests
259370
try:
260-
response = self._session.post(
261-
f"{self.endpoint}/bulk",
262-
headers=self._headers,
263-
params=query_params,
264-
json=resources,
265-
)
371+
response = self._request("POST", "/bulk", params=query_params, json=resources)
266372
except Exception as e:
267373
raise IPDataException(f"Error when looking up {resources} - {e}")
268374

0 commit comments

Comments
 (0)