Repository navigation
Conversation
no_proxy=fe11::/16 python -c 'import httpx; c = httpx.Client()' Closes: encode#3221
|
Hi, thanks for this PR. I found this httpx issue 6 months ago and wanted to address it since then. Your change only makes httpx not crash when an IPv6 range is in no_proxy but it is not properly handled. I build upon your work and addressed the issue with extending
EDIT: PR is here: p3ck#1 It is also not yet properly tested as I didn't manage to run all tests. I only managed to run tests I modified. |
da3dda0 to
71c6448
Compare
|
@ofilip Thank you for the great work. Can you check the changes I made to get it to pass linting and testing? Thanks |
I reviewed your commits and everything seems in place. Thx a lot for review, tests and improvements. |
httpx <= 0.28.1 wraps NO_PROXY IPv6 entries including the CIDR mask inside brackets (e.g. `[::1/128]`), which its URL parser rejects as `Invalid port: ':1'`. This is triggered by OrbStack/Docker Desktop injecting `NO_PROXY=localhost,127.0.0.0/8,::1/128` into containers. Strip the `/prefix` from IPv6 CIDR entries before initializing AsyncOpenAI. This is safe because httpx doesn't support CIDR range matching anyway — it only does exact-host comparison. Upstream fix: encode/httpx#3741 (pending merge)
Setting no_proxy patterns matched by hostname string comparison. So 192.168.0.1 would not match all://192.168.0.0/16. Add IPNetPattern, backed by ipaddress.ip_network, and a build_url_pattern() factory that returns IPNetPattern for all:// patterns containing a slash and WildcardURLPattern otherwise. Introduce a Pattern protocol so Client and AsyncClient can hold either kind in their _mounts dict. URLPattern is kept as a backward-compatible alias for WildcardURLPattern. Together with pydantic#967 will close - pydantic#829 - pydantic#899 Ported from encode/httpx#3741. Co-Authored-By: Ondrej Filip <ondrej.filip@firma.seznam.cz> Co-Authored-By: Bill Peck <bpeck@redhat.com>
Setting no_proxy patterns matched by hostname string comparison. So 192.168.0.1 would not match all://192.168.0.0/16. Add IPNetPattern, backed by ipaddress.ip_network, and a build_url_pattern() factory that returns IPNetPattern for all:// patterns containing a slash and WildcardURLPattern otherwise. Introduce a Pattern protocol so Client and AsyncClient can hold either kind in their _mounts dict. URLPattern is kept as a backward-compatible alias for WildcardURLPattern. Together with pydantic#967 will close - pydantic#829 - pydantic#899 Ported from encode/httpx#3741. Co-Authored-By: Ondrej Filip <ondrej.filip@firma.seznam.cz> Co-Authored-By: Bill Peck <bpeck@redhat.com>
OrbStack injects a no_proxy/NO_PROXY into every container that lists its internal IPv6 ULA range (e.g. ::1, fd07:...::/64). httpx cannot parse a bare IPv6 entry in no_proxy -- it splits on ':' and reads the address bytes as a port, raising InvalidURL at Client construction -- so any task that builds an httpx client (notably huggingface_hub v1.x via datasets) crashes before its first request and silently falls back to the offline cache. The injection is regenerated on every OrbStack restart, so a host-side edit does not stick. Probe the container's injected no_proxy once and pin a cleaned override (IPv6 entries dropped, everything else preserved) into the exec env so every agent action and the in-container verifier inherit it. Best-effort and a no-op on hosts that inject no IPv6. Refs: encode/httpx#3741, psf/requests#6313, orbstack/orbstack#2449
* verifier env: share context across exp, route through caches, restore apt-get shim Three container-environment fixes for the separate verifier, grouped: - share a built verifier context across runs of the same task to avoid rebuilding it every trial - route the separate-verifier container through the apt/pip caches with the same retry/timeout policy the agent container already gets, so a flaky mirror cannot stall the grade - restore the real apt-get on images that ship a hermetic no-op `apt-get` shim (keeping the real binary at apt-get.real), so offline verifies can install packages the image did not pre-bake (e.g. git-multibranch's `expect`). No-op on images without the shim. * env: strip httpx-breaking IPv6 entries from injected no_proxy OrbStack injects a no_proxy/NO_PROXY into every container that lists its internal IPv6 ULA range (e.g. ::1, fd07:...::/64). httpx cannot parse a bare IPv6 entry in no_proxy -- it splits on ':' and reads the address bytes as a port, raising InvalidURL at Client construction -- so any task that builds an httpx client (notably huggingface_hub v1.x via datasets) crashes before its first request and silently falls back to the offline cache. The injection is regenerated on every OrbStack restart, so a host-side edit does not stick. Probe the container's injected no_proxy once and pin a cleaned override (IPv6 entries dropped, everything else preserved) into the exec env so every agent action and the in-container verifier inherit it. Best-effort and a no-op on hosts that inject no IPv6. Refs: encode/httpx#3741, psf/requests#6313, orbstack/orbstack#2449 * supervisor: rerun a fresh baseline when the active one was interrupted A baseline run killed mid-flight leaves state.json pointing the active baseline at a record that never finalized as a keep. _ensure_baseline_at_head trusted that pointer on commit+panels alone, so on restart it short-circuited (returned False) and let the loop compare candidates against partial, ungraded evidence instead of measuring a fresh baseline. Discard a non-keep/unconcluded active baseline before the short-circuit: finalize the stale record as a crash and fall back to its parent keep (or seed fresh when there is none), so an interrupted baseline reliably restarts fresh. Adds a regression test reproducing the short-circuit. * config: document and expand the shared-verifier task list Annotate shared_verifier_task_names with the concrete reason each task needs to reuse the agent's live container instead of a fresh separate verifier (running daemon the verifier connects to / build artifacts or installed deps the agent produced / git over the agent's working tree / local-only image), and add the tasks that fail for those same reasons: git-multibranch, mailman, adaptive-rejection-sampler, git-leak-recovery, merge-diff-arc-agi-task, sanitize-git-repo, reshard-c4-data. * agent: raise default turn timeout to 3600s The previous 600s ceiling truncated long-running build/verify steps. Raise the default per-turn timeout so slow tasks are bounded by the trial budget rather than cut off mid-action. * exclude reshard-c4-data from the eval panel Drop reshard-c4-data from the train panel and add it to a contamination_risk excluded group instead of carrying a local task override for it. Running the task offline (to avoid re-downloading the held-out HF c4 shard every trial) would require an override that commits the canonical solution and verifier oracle into this repo, which the self-improvement loop could read. Excluding it removes that risk; the task also exceeds the train task_timeout_sec.
Setting no_proxy patterns matched by hostname string comparison. So 192.168.0.1 would not match all://192.168.0.0/16. Add IPNetPattern, backed by ipaddress.ip_network, and a build_url_pattern() factory that returns IPNetPattern for all:// patterns containing a slash and WildcardURLPattern otherwise. Introduce a Pattern protocol so Client and AsyncClient can hold either kind in their _mounts dict. URLPattern is kept as a backward-compatible alias for WildcardURLPattern. Together with pydantic#967 will close - pydantic#829 - pydantic#899 Ported from encode/httpx#3741. Co-Authored-By: Ondrej Filip <ondrej.filip@firma.seznam.cz> Co-Authored-By: Bill Peck <bpeck@redhat.com>
|
@lovelydinosaur Any chance this could be included in a bugfix release for the current code before 1.0? |
ege-arhan
left a comment
There was a problem hiding this comment.
In IPNetPattern.__init__:
self.net = ipaddress.ip_network(ip_net)ipaddress.ip_network defaults to strict=True. If an environment sets a CIDR entry containing non-zero host bits (e.g. NO_PROXY=192.168.1.5/24 or NO_PROXY=[fe80::1]/64), ipaddress.ip_network raises ValueError: ... has host bits set.
In build_url_pattern:
def build_url_pattern(pattern: str) -> Pattern:
try:
proto, rest = pattern.split("://", 1)
if proto == "all" and "/" in rest:
return IPNetPattern(rest)
except ValueError:
pass
return WildcardURLPattern(pattern)When IPNetPattern raises ValueError, build_url_pattern silently falls back to WildcardURLPattern("all://192.168.1.5/24"), which cannot match host 192.168.1.5 because it treats /24 as a URL path component.
Passing strict=False in IPNetPattern:
self.net = ipaddress.ip_network(ip_net, strict=False)masks host bits and allows valid CIDR subnet matching even when a specific IP within that subnet is provided in NO_PROXY.
|
Tested this branch against a bracketed, non-CIDR IPv6 literal, which is not hypothetical: a desktop app I use appends Installed head
$ no_proxy='[::1]' python -c 'import httpx; httpx.Client()'
httpx.InvalidURL: Invalid port: ':1]'The bracket stripping added in if addr[0] == "[" and addr[-1] == "]":
addr = addr[1:-1]
ip_net = f"{addr}/{range}"so a bracketed literal without a prefix never reaches it and still fails. Would you consider normalizing a bare bracketed literal (strip Happy to open a separate issue with the full traceback if you'd rather keep this PR focused on CIDR. |
OKAN-SOFTWARE
left a comment
There was a problem hiding this comment.
I noticed a potential ordering issue with overlapping CIDR patterns.
IPNetPattern.priority currently returns the same (-1, 0, 0) value for every network. Since mounts are sorted by priority and the first matching pattern determines the transport, overlapping CIDR entries are resolved by insertion order rather than network specificity.
For example:
mounts = {
"all://10.0.0.0/8": broad_transport,
"all://10.1.0.0/16": specific_transport,
}Both patterns match http://10.1.2.3/, but the /8 entry takes precedence when listed first.
Could we incorporate self.net.prefixlen into the priority so the longest matching prefix wins? A regression test with overlapping networks in both insertion orders would help ensure consistent routing behavior.
The existing tests cover individual CIDR matches but don't appear to exercise this case.
Summary
Fixes no_proxy to support IPV6 CIDR format.
no_proxy=fe11::/16 python -c 'import httpx; c = httpx.Client()'
Closes: #3221
Checklist