fix(xlsx): preserve blank cells and literal NA-like strings

[shiyuanyeming-hub]

Summary

pd.read_excel applies pandas' default missing-value handling, so the values that reach the Markdown renderer are not the values stored in the sheet:

• literal strings such as NA, NULL or None are read as missing values, and
• a column that contains a missing value is promoted to float64, so whole numbers render as 2.0 and blank cells render as the text NaN.

Both the .xlsx and the .xls readers now pass keep_default_na=False, so the converter reproduces what is actually stored in the file.

Fixes #2484
Fixes #2498

Before / after

tests/test_files/test_na_values.xls, added in this PR.

before

## Sheet1
| count | value |
| --- | --- |
| 1.0 | NaN |
| NaN | NaN |
| 2.0 | NaN |
| 3.0 | NaN |

after

## Sheet1
| count | value |
| --- | --- |
| 1 | NA |
|  | NULL |
| 2 | None |
| 3 | n/a |

Why keep_default_na=False and not dtype=object

dtype=object also fixes #2484, but it is not safe:

cell	before	keep_default_na=False	dtype=object
2024-01-05	2024-01-05	2024-01-05	2024-01-05 00:00:00
1.23456789012345e-7	1.234568e-07	1.234568e-07	0.0
2 in a column with a blank	2.0	2	2
literal NA	NaN	NA	NaN

dtype=object appends a time component to dates and collapses small floats to 0.0, and it does not fix #2498 at all.

Tests

New file packages/markitdown/tests/test_excel_na_values.py:

• a blank cell does not promote whole numbers to float (XLSX/XLS: whole numbers render as "1.0" when the column contains a blank cell #2484)
• literal NA / NULL / None / n/a / null / nan survive verbatim (Excel conversion replaces literal NA/NULL strings and blank cells with NaN #2498)
• a blank cell renders as an empty cell rather than the text NaN (Excel conversion replaces literal NA/NULL strings and blank cells with NaN #2498)
• the .xls (xlrd) reader behaves the same way, using the new tests/test_files/test_na_values.xls fixture
• dates and small floats are this is the guard for the dtype=object alternative above

I checked that 9 of the 10 tests fail on the current main, and that the tenth fails if the fix is changed to dtype=object.

Full suite locally: 1004 passed, 14 skipped. The one unrelated failure, test_speech_transcription, is a missing ffprobe binary on this machine. black --check passes.

Note on the behaviour change

This changes the default output for sheets containing blank cells or literal NA-like strings, which is what both issues ask for. If you would rather keep the current default and make it opt-in, I am happy to rework it.

[shiyuanyeming-hub]

@microsoft-github-policy-service agree