fix(html): keep a pipe in a table cell from adding a column - #2520
Open
Lukas (L4XB) wants to merge 1 commit into
Open
Lukas (L4XB) wants to merge 1 commit into
Lukas (L4XB) wants to merge 1 commit into
Conversation
A Markdown table row is split on every unescaped pipe, and markdownify writes
a cell's text through unchanged. A part number, a shell command or a regex
alternation in a cell therefore pushes the rest of the row into columns the
header does not have:
| Product | Spec |
| --- | --- |
| Cable | USB-A|USB-C |
Escape the pipes in a cell before markdownify lays the row out. A backslash
already in the cell is doubled first, so it cannot consume the escape.
This runs for every format that reaches Markdown through the HTML converter,
including .html, .docx (mammoth) and .xlsx (sheet_to_html).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
A Markdown table row is split on every unescaped pipe.
_CustomMarkdownifyinherits markdownify'sconvert_td/convert_th, which write the cell's text through unchanged, so a cell that holds a pipe silently adds a column.On
main(eb31b5c), an.xlsxwith a part number in it:The header has 2 columns and the
Cablerow now has 3. Read back, the row isCable / USB-A / USB-C—4 portsis still in column 2 whileUSB-Chas moved to a column that does not exist, so the table no longer lines up from that row on.It is not an exotic cell value. A pipe shows up in part numbers, shell commands (
grep -E 'a|b'), regex alternations, "either/or" notes, and anything pasted out of a terminal.This reaches Markdown through the shared HTML converter, so every format that routes through it is affected:
.html.docx_CustomMarkdownify.xlsx/.xlssheet_to_html→_CustomMarkdownify.epub,.ipynbHTML output, RSSThe fix
convert_td/convert_thescape the pipes in the cell's text before markdownify lays the row out:The regex matches the run of backslashes in front of the pipe rather than the pipe alone. That is the part that matters: a cell already containing
a\|bwould otherwise come out asa\\|b, where the cell's own backslash escapes the one we added and the pipe splits the row anyway. Doubling the run first makesa\|bintoa\\\|b, which reads back as the literala\|b.A pipe outside a table is untouched — only the two cell converters call it.
Tests
packages/markitdown/tests/test_table_cell_pipe.py. The helper reads the produced table back the way a Markdown reader does (split on unescaped pipes, then resolve backslash escapes) and asserts on the cells, not on a fixed string, so it tests the property rather than the formatting.9 cases: a part number, a shell command, a doubled
||, a cell with its owna\|b, a header cell, an.xlsxend to end, plus 3 guards —C:\path, a plain cell, and a pipe in a paragraph outside any table.Measured:
Full suite:
851 passed, 14 skipped(unchanged).blackclean.