Skip to content

fix: normalize_url preserves path/query casing, lowercasing only scheme and host - #2241

Open
enriquealonso01 wants to merge 1 commit into
apify:masterfrom
enriquealonso01:fix/normalize-url-preserve-path-case
Open

enriquealonso01 wants to merge 1 commit into
apify:masterfrom
enriquealonso01:fix/normalize-url-preserve-path-case

Conversation

@enriquealonso01

Copy link
Copy Markdown

Summary

normalize_url lowercased the entire URL via str(yarl_new_url).lower(). Since only the scheme and host are case-insensitive (RFC 3986 §6.2.2.1), this silently merged distinct URLs during request deduplication — e.g. https://example.com/Product/ABC and https://example.com/product/abc collapse into one request key, and case-significant query tokens (signed URLs, API tokens) were mangled.

Changes

  • normalize_url now returns str(yarl_new_url) unchanged: yarl already lowercases scheme and host during parsing, while path/query/fragment keep their original casing
  • Docstring updated to state the case-sensitivity contract

Closes #2008

Test plan

  • Updated the existing HTTPS://EXAMPLE.COM/?KEY=VALUE case: scheme/host lowercase, query casing preserved
  • Added regression cases: path casing preserved (/Product/ABC), query token casing preserved (?token=SeCrEt)
  • pytest tests/unit/_utils/test_requests.py — 17 passed

Breaking change note

As flagged in #2008 (comment) (this is planned for v2 as breaking): anyone who relied on full-lowercase unique keys will see different keys for case-distinct URLs. Targeting master per the current default branch — happy to rebase onto a v2 branch if one exists.

Submitted as part of Hacktoberfest-adjacent open-source contributions by @enriquealonso01.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

normalize_url lowercases the entire URL, silently deduplicating case-distinct pages

2 participants