Repository navigation
Fix UnicodeMapper's trimming of some whitespace that should be preserved #248
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
stesee
merged 2 commits into
Codeuctivity:main
from
opendocx:fix/unicode-xml-whitespace-upstream
Oct 24, 2026
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| <?xml version="1.0" encoding="UTF-8" standalone="yes"?> | ||
| <w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"> | ||
| <w:body><w:p><w:r><w:t>NBSP leading xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t> foo</w:t></w:r></w:p><w:p><w:r><w:t>NBSP trailing xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t>foo </w:t></w:r></w:p><w:p><w:r><w:t>NBSP only xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t> </w:t></w:r></w:p><w:p><w:r><w:t>NBSP mixed xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>NBSP split field xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t>{[elseif</w:t></w:r><w:r><w:t> TakingTitle == "Unequal"]}</w:t></w:r></w:p><w:p><w:r><w:t>NBSP leading xml:space default</w:t></w:r></w:p><w:p><w:r><w:t xml:space="default"> foo</w:t></w:r></w:p><w:p><w:r><w:t>NBSP trailing xml:space default</w:t></w:r></w:p><w:p><w:r><w:t xml:space="default">foo </w:t></w:r></w:p><w:p><w:r><w:t>NBSP only xml:space default</w:t></w:r></w:p><w:p><w:r><w:t xml:space="default"> </w:t></w:r></w:p><w:p><w:r><w:t>NBSP mixed xml:space default</w:t></w:r></w:p><w:p><w:r><w:t xml:space="default"> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>NBSP split field xml:space default</w:t></w:r></w:p><w:p><w:r><w:t>{[elseif</w:t></w:r><w:r><w:t xml:space="default"> TakingTitle == "Unequal"]}</w:t></w:r></w:p><w:p><w:r><w:t>NBSP leading xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve"> foo</w:t></w:r></w:p><w:p><w:r><w:t>NBSP trailing xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve">foo </w:t></w:r></w:p><w:p><w:r><w:t>NBSP only xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve"> </w:t></w:r></w:p><w:p><w:r><w:t>NBSP mixed xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve"> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>NBSP split field xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t>{[elseif</w:t></w:r><w:r><w:t xml:space="preserve"> TakingTitle == "Unequal"]}</w:t></w:r></w:p><w:p><w:r><w:t>NBSP trailing keyword split field</w:t></w:r></w:p><w:p><w:r><w:t>{[elseif </w:t></w:r><w:r><w:t>TakingTitle == "Unequal"]}</w:t></w:r></w:p><w:p><w:r><w:t>NBSP standalone separator run</w:t></w:r></w:p><w:p><w:r><w:t>{[elseif</w:t></w:r><w:r><w:t> </w:t></w:r><w:r><w:t>TakingTitle == "Unequal"]}</w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+0085 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+1680 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2000 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2001 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2002 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2003 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2004 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2005 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2006 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2007 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2008 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2009 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+200A mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2028 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+2029 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+202F mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+205F mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>Unicode U+3000 mixed boundaries</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>XML whitespace edges xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>referenced CR xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t>foo bar</w:t></w:r></w:p><w:p><w:r><w:t>referenced CRLF xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t>foo bar</w:t></w:r></w:p><w:p><w:r><w:t>internal tab xml:space absent</w:t></w:r></w:p><w:p><w:r><w:t>foo	bar</w:t></w:r></w:p><w:p><w:r><w:t>XML whitespace edges xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve"> 	 foo 	 </w:t></w:r></w:p><w:p><w:r><w:t>referenced CR xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve">foo bar</w:t></w:r></w:p><w:p><w:r><w:t>referenced CRLF xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve">foo bar</w:t></w:r></w:p><w:p><w:r><w:t>internal tab xml:space preserve</w:t></w:r></w:p><w:p><w:r><w:t xml:space="preserve">foo	bar</w:t></w:r></w:p><w:p/></w:body> | ||
| </w:document> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # UnicodeMapper whitespace fixtures | ||
|
|
||
| `UnicodeMapperTests.TreatsXmlSpaceLikeWord` compares each crafted input paragraph | ||
| with the literal text and tabs in the corresponding paragraph saved by Microsoft | ||
| Word. Each case has a name paragraph followed by a content paragraph. Each document | ||
| ends with one blank paragraph. The tests check paragraph counts and matching names | ||
| so a missing or reordered case cannot silently pass. | ||
|
|
||
| The original `UM-Whitespace-test.docx` and `UM-Whitespace-Word-saved.docx` contain | ||
| 107 cases from the January 2026 investigation. They are unchanged. | ||
|
|
||
| The supplementary `UM-Whitespace-Unicode-test.docx` and | ||
| `UM-Whitespace-Unicode-Word-saved.docx` contain 43 cases. The expected document was | ||
| opened and saved without content edits using Microsoft Word for Mac 16.114 on | ||
| October 21, 2026. These are synthetic documents; no customer document was used. | ||
|
|
||
| The supplementary cases cover: | ||
|
|
||
| - Leading, trailing, standalone, and mixed-boundary NBSP with `xml:space` absent, | ||
| `default`, and `preserve`. | ||
| - An NBSP separator at either side of a run boundary and in a standalone run. | ||
| - All other XML-valid characters recognized as whitespace by .NET: U+0085, | ||
| U+1680, U+2000 through U+200A, U+2028, U+2029, U+202F, U+205F, and U+3000. | ||
| Each is surrounded by ordinary XML whitespace to distinguish trimming the XML | ||
| whitespace from trimming the Unicode character. | ||
| - Ordinary spaces, tabs, CR, and CRLF, with preservation absent and present. | ||
| CR and LF use character references so the XML parser does not normalize them | ||
| before Word or UnicodeMapper sees them. Literal XML line endings in the original | ||
| fixture are normalized by XML parsing and do not make that distinction. | ||
|
|
||
| Word preserves the tested Unicode characters, trims only boundary space/tab/CR/LF | ||
| when preservation is absent or `default`, converts internal line endings to spaces, | ||
| and retains tabs when preservation is enabled. These observations cover the listed | ||
| inputs and Word version, not every possible whitespace or formatting context. | ||
|
|
||
| `UM-Whitespace-Unicode-test.xml` is the exact `word/document.xml` payload in the | ||
| supplementary input DOCX, retained for review and reproduction. To regenerate the | ||
| input from the repository root using Python's standard library: | ||
|
|
||
| ```python | ||
| from pathlib import Path | ||
| from zipfile import ZipFile | ||
|
|
||
| root = Path("TestFiles") | ||
| xml = (root / "UM-Whitespace-Unicode-test.xml").read_bytes() | ||
| with ZipFile(root / "UM-Whitespace-test.docx") as source: | ||
| with ZipFile(root / "UM-Whitespace-Unicode-test.docx", "w") as target: | ||
| for entry in source.infolist(): | ||
| data = xml if entry.filename == "word/document.xml" else source.read(entry.filename) | ||
| target.writestr(entry, data) | ||
| ``` | ||
|
|
||
| Keep the crafted input unchanged. Open it in Word and use Save As to create the | ||
| separate `UM-Whitespace-Unicode-Word-saved.docx`. Do not manufacture expected output | ||
| with UnicodeMapper or another word processor. Review any changes in the Word-saved | ||
| text before updating expectations. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you thought about other spots that may need the same fix?