fix(html): keep a URL from being cut at a space or a stray parenthesis - #2519
Open
Lukas (L4XB) wants to merge 1 commit into
Open
Lukas (L4XB) wants to merge 1 commit into
Lukas (L4XB) wants to merge 1 commit into
Conversation
A bare Markdown link destination ends at the first whitespace and at an
unbalanced closing parenthesis. Both are legal in a query string or a fragment,
which this converter deliberately does not re-encode, so a URL carrying either
was written out in a form that reads back truncated:
<a href="https://example.com/s?q=a b"> -> [spaced](https://example.com/s?q=a b)
<a href="https://example.com/s?q=a)b"> -> [result](https://example.com/s?q=a)b)
Wrap such a destination in angle brackets, which is what CommonMark provides
for the case and which leaves the URL itself byte for byte as it was.
An image destination is parsed exactly like a link destination, and convert_img
was not escaping its src at all -- not even the path quoting convert_a has
applied for a while. It now shares both. A data URI is left alone, since its
payload is not a path to quote.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A bare Markdown link destination ends at the first whitespace and at an unbalanced closing parenthesis. Both are legal in a query string or a fragment, and
_CustomMarkdownifydeliberately does not re-encode those components — there is a test pinning exactly that,test_html_href_does_not_quote_query_or_fragment, because percent-encoding a query would rewrite sub-delimiters the server may be reading.The result is a destination that no longer round-trips:
Reading that Markdown back gives a truncated URL, silently — the output is still valid Markdown, just pointing somewhere else.
convert_imghas the same problem and one more: it was not escapingsrcat all, not even the path quotingconvert_aapplies.An image destination is parsed by exactly the same rule as a link destination, so both truncate.
How
_escape_uri(url)— the path quotingconvert_aalready did, extracted soconvert_imgcan share it.convert_a's behaviour is unchanged by the extraction._format_destination(url)— wraps the destination in<...>when it holds whitespace, a parenthesis or an angle bracket. That is CommonMark's own mechanism for the case, and it leaves the URL byte for byte as it was, which is the property the existing test is protecting. An angle bracket inside is percent-encoded, since it cannot appear in an angle-bracket destination.convert_imggets_escape_uri+_format_destination. Adata:URI skips the quoting: its payload is not a path, and quoting it would turndata:image/png;base64,...intodata:image/png%3Bbase64,....A balanced pair of parentheses in a query would parse fine bare, and is wrapped anyway. Telling balanced from unbalanced is more machinery than this warrants, and
<...>is always valid.Test
packages/markitdown/tests/test_html_converter.py— five tests added, one changed.maintests/test_html_converter.pyThe changed one, so you do not have to find it:
test_html_href_does_not_quote_query_or_fragmentnow expects[example](<https://example.com/a%20path?query=a b%20c#fragment with spaces>)instead of the same string without the angle brackets. The URL inside is the identicalexpected_hrefthe test already declared — nothing is re-encoded, so what the test is named for still holds. Only the delimiters are added, and a comment in the test says why.The new ones cover an unbalanced
)in an href, an imagesrcwith a space, an imagesrcwith an unbalanced), plus two guards: an ordinary URL stays bare (no angle brackets), and a data URI is left exactly as it was.Full suite: 897 passed, 14 skipped.
black(the pinned 23.7.0 from.pre-commit-config.yaml) reports both files unchanged.One thing this does not touch, in case you want it in the same change: the
alttext of an image is still taken raw from the attribute, so analtholding an unbalanced]breaksthe same way. That is a separate escape (text, not destination) and I left it out rather than widen this.