Skip to content

fix(xlsx): a blank cell should not rewrite the rest of its column - #2516

Open
Lukas (L4XB) wants to merge 5 commits into
microsoft:mainfrom
L4XB:fix/xlsx-blank-cell-nan
Open

Lukas (L4XB) wants to merge 5 commits into
microsoft:mainfrom
L4XB:fix/xlsx-blank-cell-nan

Conversation

@L4XB

@L4XB Lukas (L4XB) commented Sep 16, 2026

Copy link
Copy Markdown

What

An XLSX sheet is rendered with DataFrame.to_html(index=False). Two pandas defaults meet there:

  • to_html writes the string NaN into a cell that is empty (NaT for a date, None for a missing value),
  • and read_excel upcasts a column that holds one — an int column becomes float64, a bool column becomes float64.

So a single blank cell rewrites every other cell in its column. Given this sheet —

Shipped Units Due Id
TRUE 12 2026-01-05 1
2
FALSE 7 2026-02-09 14:30 3

markitdown produced:

| Shipped | Units | Due | Id |
| --- | --- | --- | --- |
| 1.0 | 12.0 | 2026-01-05 00:00:00 | 1 |
| NaN | NaN | NaT | 2 |
| 0.0 | 7.0 | 2026-02-09 14:30:00 | 3 |

A blank cell is extremely common — the very first thing in a real spreadsheet — and every one of those differences is wrong in a way a reader cannot undo: NaN reads as a value, 12.0 is not what the sheet says, 1.0 has lost that the column is a yes/no, and 2026-01-05 00:00:00 invents a time.

How

  • pd.read_excel(..., dtype=object) in both calls in _read_xlsx_sheets and in XlsConverter, so a blank no longer changes the type of its neighbours. pandas' Excel readers already hand back each cell's own Python type (int, float, bool, datetime), so nothing is being guessed here — it is pandas' re-inference that is dropped.
  • One renderer, sheet_to_html(sheet), with na_rep="" and a per-cell formatter. pandas applies na_rep to the blanks and the formatter to everything else, so the column is never re-typed on the way out. A fractional number is the exception: with index=False, pandas renders a float in an object column with its own display precision and skips the formatter, so 1e-07 came out as 0.0 (1.000000e-07 on main). The table passes float_format for that, which keeps the 15 significant digits Excel shows.
  • The formatter exists for one case: openpyxl returns a datetime for a date cell, and str() on it appends a midnight time. A date renders as 2026-01-05, a datetime keeps its time.

After:

| Shipped | Units | Due | Id |
| --- | --- | --- | --- |
| True | 12 | 2026-01-05 | 1 |
|  |  |  | 2 |
| False | 7 | 2026-02-09 14:30:00 | 3 |

XlsConverter gets the same two changes. xlrd stores every number as a double, but pandas turns a whole-number double back into an int, so a blank upcast an .xls column just like an .xlsx one. With pandas' own inference a blank date cell also makes the column datetime64, and the date formatter cannot render its NaT, so such a file would fail with NaTType does not support time. test.xls renders byte-identical with dtype=object.

Since #2506, markitdown-ocr's XlsxConverterWithOCR subclasses XlsxConverter and renders its tables through it, so it gets the fix without a change to its source. Its tests pinned the old output: the test_xlsx_converter.py snapshots expected | NaN | NaN | rows, and test_inherited_repairs_and_native_table_fixes_reach_ocr pinned the exact read_excel arguments. Both now expect the fixed behaviour, and a new test converts a sheet with an image and a blank row through the OCR converter.

Test

packages/markitdown/tests/test_xlsx_blank_cells.py — eight tests, pinning the exact markdown. The .xls one reads a small fixture, tests/test_files/test_blank_cells.xls, with a blank row between an int, a bool and a date column.

main this branch
tests/test_xlsx_blank_cells.py 6 failed, 2 passed 8 passed
markitdown-ocr test_xlsx_converter.py + test_xlsx_inheritance.py 6 failed, 15 passed 21 passed
# on main
FAILED test_blank_cell_stays_blank
FAILED test_a_blank_cell_does_not_turn_the_column_into_floats
  assert '## Sheet1\n| Units | Year |\n| --- | --- |\n| 12.0 | 2026 |\n| NaN | 2025 |' == ...
FAILED test_a_blank_cell_does_not_turn_booleans_into_numbers
FAILED test_a_date_cell_carries_no_time_and_a_blank_one_is_empty
FAILED test_a_fractional_number_keeps_its_value
FAILED test_an_xls_blank_cell_is_read_the_same_way

The two that pass on both sides are the guards that must not move: a sheet with no blank cell in it renders exactly as before, and a cell holding HTML still round-trips exactly as before.

The existing fixtures are unchanged — test.xlsx and test.xls produce byte-identical markdown on both sides (they contain no blank cells), and both suites are green when run the way CI runs them:

markitdown      pytest   1003 passed, 14 skipped   (main: 995 passed, 14 skipped)
markitdown-ocr  pytest    109 passed               (main: 108 passed)

black (the pinned 23.7.0 from .pre-commit-config.yaml) reports every tracked Python file unchanged.

Not in this PR

A cell whose text is N/A, NULL, NaN, None or #N/A is still read as missing, because of pandas' default na_values. It rendered as NaN on main and renders blank here. Keeping the text needs keep_default_na=False, which is left for a separate change; #2499 is also in that area.

`DataFrame.to_html` writes the string `NaN` into an empty cell, and pandas
upcasts any column that holds one. A spreadsheet with a single blank cell was
therefore converted with `NaN` where the blank is, `12.0` where the sheet says
`12`, `1.0` where it says `TRUE`, and `NaT` where a date is missing.

Read the sheets with `dtype=object` so a blank no longer changes the type of
its neighbours, render the blanks with `na_rep=""`, and pass a per-cell
formatter so the remaining values are rendered as themselves - including a
date cell, which openpyxl returns as a `datetime` whose `str()` would append a
midnight time the spreadsheet does not have.

`.xls` keeps pandas' own inference, since xlrd stores every number as a double
and there is no integer to preserve; it shares the renderer, so its blanks stop
reading as `NaN` too. The same two lines in `markitdown-ocr`'s XLSX converter
now go through that renderer as well.
microsoft#2506 turned `_read_xlsx_sheets` into a context manager and made
`XlsxConverterWithOCR` a subclass of `XlsxConverter`. The fix now sits on
that structure: both `read_excel` calls in `_read_xlsx_sheets` pass
`dtype=object`, and the sheet loop in `XlsxConverter.convert` renders
through `sheet_to_html`.

`_xlsx_converter_with_ocr.py` is taken from main as is. It no longer reads
or renders a sheet itself, so the fix reaches it through the parent.
The OCR converter renders its tables through `XlsxConverter` now, so the
blank-cell fix applies to it. Its snapshots pinned the old `NaN` cells,
and the inheritance test pinned the exact `read_excel` arguments. Both now
expect the fixed behaviour.

A new test converts a sheet with an image and a blank row between an int
and a bool column. It checks that the OCR output keeps `12` and `True`
and leaves the blank cells empty.
`.xls` kept pandas' own type inference on the grounds that xlrd stores
every number as a double. But pandas turns a whole-number double back into
an int, so a blank cell still re-typed its column (`12.0`, `1.0`). It also
turned a date column into `datetime64`, and the shared date formatter
raised on its `NaT`: "NaTType does not support time". An `.xls` file with a
blank date cell failed to convert at all.

Read `.xls` with `dtype=object`, like `.xlsx`. `test.xls` renders the same
as before.
With `index=False`, pandas renders a float in an object column through its
own `float_format` and never reaches `formatters`, so `1e-07` came out as
`0.0` once the sheets were read with `dtype=object`. On main, which reads
with inference, the same cell rendered as `1.000000e-07`.

The table now passes `float_format`, which formats with the 15 significant
digits Excel shows: `1e-07`, `3.14159265358979`, and `0.1 + 0.2` as `0.3`.
Both the .xlsx and the .xls path render through `sheet_to_html`, so both are
covered.
@L4XB

Copy link
Copy Markdown
Author

Merged main into this branch (no rebase, so the commit that was already here is unchanged) and moved the fix onto the #2506 structure:

  • _read_xlsx_sheets is a context manager now. Both of its read_excel calls pass dtype=object, and the sheet loop in XlsxConverter.convert renders through sheet_to_html.
  • XlsxConverterWithOCR subclasses XlsxConverter since Refactor Office OCR converters to reuse core conversion pipelines #2506 and no longer reads or renders sheets itself, so this PR no longer touches its source. It gets the fix through the parent.
  • Two OCR tests pinned the old behaviour and are updated: the test_xlsx_converter.py snapshots expected | NaN | NaN | rows, and test_inherited_repairs_and_native_table_fixes_reach_ocr pinned the exact read_excel arguments. A new test converts a sheet with an image and a blank row between an int and a bool column.

Two corrections to my own earlier claims, both found while re-checking:

  • .xls. I wrote that xlrd has no integers to preserve and that dtype=object would turn 89 into 89.0. Both are wrong: pandas turns a whole-number double back into an int, so a blank upcast an .xls column just like an .xlsx one. Worse, with pandas' inference a blank date cell makes the column datetime64, and the date formatter cannot render its NaT, so such a file failed to convert at all (NaTType does not support time). .xls is now read with dtype=object too; test.xls renders byte-identical, and a small test_blank_cells.xls fixture pins the case.
  • Fractional numbers. With index=False, pandas renders a float in an object column with its own display precision and skips formatters entirely, so 1e-07 came out as 0.0 (1.000000e-07 on main). The table now passes float_format with 15 significant digits, the precision Excel shows: 1e-07, 3.14159265358979, 0.1 + 0.2 as 0.3, 1234567.891 unchanged, and 1e-07 | 3.14159265358979 | 4e-06 for the same values in an .xls. test_a_fractional_number_keeps_its_value pins it.

Not covered here: a cell whose text is N/A, NULL, NaN or None is still read as missing, because of pandas' default na_values. It rendered as NaN on main and renders blank now — the text is lost either way. Keeping it needs keep_default_na=False, which I would leave for a separate change; #2499 is in that area too.

Measured at 06ded9e:

Python 3.10, pandas 2.3.3 Python 3.12, pandas 3.0.5
packages/markitdown pytest 1003 passed, 14 skipped 1003 passed, 14 skipped
packages/markitdown-ocr pytest 109 passed 109 passed

Both runs import the package from this worktree (checked by printing markitdown.__file__). With only _xlsx_converter.py swapped back to main's version, tests/test_xlsx_blank_cells.py is 6 failed, 2 passed against the 8 tests here. black 23.7.0, the pinned version from .pre-commit-config.yaml, leaves every tracked file unchanged.

I also mutated the change in ten ways — each of the three dtype=object reads, both sheet_to_html call sites, na_rep, formatters, float_format, and the two branches of the date formatter. Each mutation is asserted to change the file exactly once, and every one of the ten fails at least one test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant