Skip to content

fix(ipynb): read the notebook's own language and a string source - #2518

Open
Lukas (L4XB) wants to merge 1 commit into
microsoft:mainfrom
L4XB:fix/ipynb-language-and-source
Open

Lukas (L4XB) wants to merge 1 commit into
microsoft:mainfrom
L4XB:fix/ipynb-language-and-source

Conversation

@L4XB

Copy link
Copy Markdown

What

IpynbConverter assumes two shapes that nbformat does not guarantee.

1. Every code cell is fenced as Python.

md_output.append(f"```python\n{''.join(source_lines)}\n```")

An R notebook comes out as:

```python
summary(cars)

The notebook says what it is — `metadata.language_info.name`, and on older notebooks `metadata.kernelspec.language` — and neither was read. A reader, human or model, is told the cell is Python when it is not.

**2. A `source` that is one string loses the title.**

nbformat types `source` as a `multiline_string`: *either* a list of lines or a single string. Both appear in the wild — `nbformat.write` emits lists, but plenty of notebooks are generated by other tooling, and `nbformat.read` accepts both.

`"".join(source)` happens to be correct for both, which is why this went unnoticed. The title scan is not:

```python
for line in source_lines:
    if line.startswith("# "):

Over a string, that iterates one character at a time, and "#".startswith("# ") is never true. Same notebook, two spellings of the same cell:

# source as a list
{"cell_type": "markdown", "source": ["# My Title\n", "\n", "some text"]}  -> title: "My Title"

# source as a string
{"cell_type": "markdown", "source": "# My Title\n\nsome text"}            -> title: None

How

  • _fence_language(notebook_content)metadata.language_info.name, then metadata.kernelspec.language, then "python" as before. A value that is not a usable fence info string (empty, containing a space, containing a backtick) falls through to the default rather than producing a broken fence.
  • _source_lines(cell) — splits a string source into lines with splitlines(keepends=True), passes a list through unchanged, and yields nothing for any other type. Joining is unaffected; the title scan now sees lines in both cases.

Raw cells keep their untagged fence — only code cells carry a language.

Test

packages/markitdown/tests/test_ipynb_converter.py — 13 tests.

main this branch
tests/test_ipynb_converter.py 4 failed, 9 passed 13 passed
# on main
FAILED test_code_cells_use_the_notebooks_own_language[metadata0-R]
FAILED test_code_cells_use_the_notebooks_own_language[metadata1-julia]
FAILED test_a_string_source_is_read_as_lines
FAILED test_a_source_that_is_neither_a_string_nor_a_list_is_skipped

The nine that pass on both sides are the guards: a notebook with no language metadata still fences as python, a list source renders byte-identically, metadata.title still wins over a heading, raw cells stay untagged, and the three malformed-metadata cases fall back rather than emitting a broken fence.

Full suite: 905 passed, 14 skipped. black (the pinned 23.7.0 from .pre-commit-config.yaml) reports both files unchanged.

Two shapes nbformat does not guarantee were assumed.

Code cells were always fenced as ```python, whatever the notebook is written
in, so an R or Julia notebook came out labelled as Python. The language is in
the notebook: `metadata.language_info.name`, or `metadata.kernelspec.language`
on older ones. A notebook that records neither keeps the previous default.

`source` is an nbformat `multiline_string` -- a list of lines or a single
string. Joining is the same either way, which is why this went unnoticed, but
iterating is not: the title scan saw one character per step and never matched a
`# ` heading, so a notebook whose source is a string got no title at all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant