Skip to content

Replace auto-linked URLs in a single pass - #41

Open
arpitjain099 wants to merge 1 commit into
dcwatson:mainfrom
arpitjain099:linear-link-replacement
Open

Replace auto-linked URLs in a single pass#41
arpitjain099 wants to merge 1 commit into
dcwatson:mainfrom
arpitjain099:linear-link-replacement

Conversation

@arpitjain099

Copy link
Copy Markdown

This is the patch you offered to look at.

_transform did the link replacement in two halves and both were quadratic. The search loop pulled one match at a time and spliced the token in with data = data[:start] + token + data[end:], rebuilding the whole string for every URL, and the restore step at the end ran a separate data.replace(token, replacement) for every token, each of which scans the whole string again. Neither is noticeable with a handful of links, but a long post made of many short ones pays O(n^2).

Both are now a single re.sub. The token format is unchanged, so it still survives the escape and cosmetic passes the same way, and _link_replace is called from the callback exactly as before.

Timings through render_html on "http://a.co/x " repeated, Python 3.12:

  URLs    bytes     before     after
  2,000   28,000     0.100s    0.004s
  4,000   56,000     0.382s    0.008s
  8,000  112,000     1.521s    0.017s
 16,000  224,000     5.742s    0.033s
 32,000  448,000    22.188s    0.071s

The shape matters more than the ratio: the new column doubles as the input doubles, the old one roughly quadruples.

On not changing behaviour, I ran the old and new modules side by side over 4,008 generated inputs across 4 option combinations, 16,032 comparisons, and the output is byte identical. The generator included the awkward cases I could think of: text that already looks like a placeholder token, links in parentheses, links followed by punctuation, -- and ... next to links, [url] tags, and replace_links=False.

Two things worth flagging rather than hiding.

The two tests I added are not regression tests for the slowness, and they pass on both the old and new code. They cover the behaviour this refactor touches, many links all getting replaced and a token-looking string being left alone, so a future change to this area has something to trip over. I left out a timing assertion deliberately since those go flaky in CI.

There is one behavioural nuance. The old restore loop ran str.replace per token in sequence, so each pass could see the text produced by the previous one. The single re.sub does not re-scan its own output. I could not construct an input where that differs, because _url_re cannot match a token, but a custom linker that returned text shaped like {{ bbcode-link-N }} would behave differently. That seems like the better behaviour to me, though it is your call.

Both halves of the link replacement in _transform rebuilt the whole string
once per URL: the search loop spliced a token in with a slice concatenation
for every match, and the restore step ran a separate str.replace for every
token. A post with many links therefore cost time quadratic in its length.

Use re.sub for both, so each is one linear pass. Output is unchanged.

On 448 KB of text made of 32,000 short links, render_html goes from 22.2s
to 0.071s, and timings now double with the input rather than quadrupling.

Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant