[ryan-kipawa]

Closes #23

What and why

Opening or creating a dfs file fails when its path contains characters outside cp1252, for example a Chinese folder name on a Chinese Windows machine (#23). The cause: DfsFile.Open and DfsBuilder.CreateFile encoded the path as cp1252 before handing it to the native library's char* open. Windows reads that char* in the machine's ANSI code page (Python's mbcs), and Linux passes the bytes to the filesystem, which expects UTF-8, so on Linux any non-ASCII path failed, Danish included. A new DfsDLLUtil.EncodeFileName encodes with mbcs on Windows and os.fsencode elsewhere, and turns a name the ANSI code page cannot represent into a ValueError that names the code page and the offending character. DfsFile.Open encodes before any file system check, so such a name fails before any I/O. The Linux library setup paths (EUM.xml, CARTSETUPLINUX) were ascii-encoded the same way, so importing mikecore from a non-ASCII install directory failed; they now use os.fsencode.

Commits

1. 1fd0f92 Encode dfs file names for the platform, not as cp1252: EncodeFileName and its two call sites, plus tests/test_filename_encoding.py. The test is in the same commit as the fix, so checking out a test-only commit to see it fail is not possible; the Before evidence below runs it against master instead.
2. b09e84f Report the ANSI code page correctly; fsencode Linux lib paths: the error message asks Windows for the code page (getpreferredencoding says utf-8 in UTF-8 mode), and eum.py / Projections.py use os.fsencode for the library directory. The message is built with an f-string, as ruff on master requires.
3. 323690f Address review of the file name encoding: locale.getencoding() replaces ctypes.windll.kernel32.GetACP() (it returns the ANSI code page even in UTF-8 mode, and drops the pyrefly warning); filename: str | os.PathLike[str], with the os.fspath result in a new name: str instead of rebinding the parameter; the UnicodeEncodeError is chained (from e) and the message names the offending character; Open encodes before os.path.isfile/os.access; DfsDLLUtil's docstring now covers marshalling; the tests close files in finally and skip based on locale.getencoding() instead of copying the encoder.
4. 31e7977 Drop the Windows settings advice from the file name error: it named a Windows UI label that can change, and the setting was never verified. The message now states the code page, the unsupported character and the path.

Code Review Guide

Glossary

• ANSI code page (ACP): the legacy 8-bit/DBCS code page Windows uses for char* file APIs; cp1252 in Western Europe, cp936 in China. Python's mbcs codec encodes to it.
• os.fsencode: encodes a path the way the OS filesystem expects; UTF-8 (with surrogateescape) on Linux.
• char* open: the native ufs library's dfsFileRead / dfsFileEdit / dfsFileAppend / dfsFileCreate take a byte string, not wchar_t*, which is why a Windows path outside the ACP cannot be opened at all.

Where the change sits

flowchart LR
  U["user / mikeio"] --> DO["DfsuFile.Open<br/>DfsuFile.py:322"]
  U --> FF["DfsFileFactory.DfsuFileOpen<br/>DfsFileFactory.py:64"]
  DO --> O["DfsFile.Open<br/>DfsFile.py:786"]
  FF --> O
  U --> CF["DfsBuilder.CreateFile<br/>DfsBuilder.py:317"]
  O --> E["DfsDLLUtil.EncodeFileName<br/>DfsFile.py:1576"]
  CF --> E
  E --> N["ufs native<br/>dfsFileRead / dfsFileCreate (char*)"]
  I["import mikecore"] --> EU["eumDLL.Init (Linux)<br/>eum.py:1432"]
  I --> MZ["MzCartDLL.Init (Linux)<br/>Projections.py:62"]
  EU --> EN["eumSetupLoadLinux (char*)"]
  MZ --> MN["CARTSETUPLINUX (char*)"]
  classDef changed fill:#2da44e,stroke:#1a7f37,color:#fff
  class O,CF,E,EU,MZ changed

Loading

Every dfs open and create goes through the two green call sites into one encoder; the import path changes only run on Linux. Line numbers are at the PR head 323690f.

Conventions followed

Existing code	This PR
Platform split on os.name == "nt"
DfsDLL.py:62 if os.name == "nt": lib = ctypes.CDLL(... "ufs.dll") else: lib = ctypes.CDLL(... "libufs.so")	DfsFile.py:1579 if os.name != "nt": return os.fsencode(filename) name: str = os.fspath(filename) try: return name.encode("mbcs")
Native-marshalling helpers are static methods on DfsDLLUtil (extend the existing class, no new module)
DfsFile.py:1592 @staticmethod def ToEumAxisUnit(eumUnitInt: int): ...	DfsFile.py:1575 @staticmethod def EncodeFileName(filename: str | os.PathLike[str]) -> bytes: ...
Tests use the public API, real testdata/ files and pytest's tmp_path
test_dfsu_builder.py:31 (master) def test_node_ids_set_after_nodes_are_written(tmp_path): filename = str(tmp_path / "nodeids_after.dfsu")	test_filename_encoding.py:30 def test_open_non_ascii_path(tmp_path, name): ... dfs = DfsuFile.Open(str(filename)) try: assert dfs.NumberOfElements == 3636 finally: dfs.Close()

Deviations

None. Review asked for two that the first version had (ctypes.windll for the code page, and an untyped filename), and commit 3 removes both.

Evidence

Linux runs below are on head 31e7977 (rebased onto master 8c0502b), unless marked "master" or "Before". The Windows runs marked "(earlier)" were made on the first two commits merged locally onto ae7ef82, before the review fixes; I couldn't reach a Windows machine for the new head, so Windows on 323690f is covered by CI's windows-latest (cp1252) jobs only.

Before (Linux, master code + this PR's test file): uv run pytest -q -rfs tests/test_filename_encoding.py

FAILED tests/test_filename_encoding.py::test_open_non_ascii_path[\xd8resund \xe6\xf8\xe5]
FAILED tests/test_filename_encoding.py::test_open_non_ascii_path[\u9879\u76ee \u5408\u754c\u6570\u6a21]
FAILED tests/test_filename_encoding.py::test_create_non_ascii_path[\xd8resund \xe6\xf8\xe5]
FAILED tests/test_filename_encoding.py::test_create_non_ascii_path[\u9879\u76ee \u5408\u754c\u6570\u6a21]
SKIPPED [1] tests/test_filename_encoding.py:56: Windows ANSI code page only
4 failed, 1 skipped in 0.40s

The Danish open fails with Exception: Could not load file .../Øresund æøå/Øresund æøå.dfsu (Error code 2003), and the Danish create writes a file under the cp1252-mangled name so filename.exists() is false. This confirms the Linux failure that the original PR description could only infer.

After (Linux):

tests/test_filename_encoding.py::test_open_non_ascii_path[\xd8resund \xe6\xf8\xe5] PASSED
tests/test_filename_encoding.py::test_open_non_ascii_path[\u9879\u76ee \u5408\u754c\u6570\u6a21] PASSED
tests/test_filename_encoding.py::test_create_non_ascii_path[\xd8resund \xe6\xf8\xe5] PASSED
tests/test_filename_encoding.py::test_create_non_ascii_path[\u9879\u76ee \u5408\u754c\u6570\u6a21] PASSED
tests/test_filename_encoding.py::test_unrepresentable_path_gives_clear_error SKIPPED
4 passed, 1 skipped in 0.29s

Windows (cp1252 machine), opening a dfsu in a 项目 合界数模 folder (earlier) (script in Verify):

• master: UnicodeEncodeError: "'charmap' codec can't encode characters in position 45-46: character maps to <undefined>"
• this PR (before the review fixes; the message now names the unsupported character and no longer gives Windows settings advice): ValueError: "File name can not be represented in the Windows ANSI code page (cp1252): C:\\...\\tmp18w1kvn7\\\u9879\u76ee \u5408\u754c\u6570\u6a21\\OresundHD.dfsu. Rename the file or folder, or enable 'Beta: Use Unicode UTF-8 for worldwide language support' in Windows region settings."

Windows pytest -q -rs tests/test_filename_encoding.py (earlier): 3 passed, 2 skipped (the two Chinese cases skip as not representable in cp1252; the error-message test runs and passes).

Linux import from a non-ASCII install directory (mikecore/ copied under .../søren/):

• master: UnicodeEncodeError: 'ascii' codec can't encode character '\xf8' in position 100 at eumFilePath.encode("ascii")
• this PR: imported .../after/søren/mikecore/__init__.py

Repo checks (AGENTS.md), on head 31e7977:

Check	Result
uv run pytest (Linux)	183 passed, 1 skipped, 8 xfailed, 8 subtests passed
uv run pytest (Windows, cp1252, earlier)	151 passed, 2 skipped, 8 xfailed, 8 subtests passed
uv run pyrefly check	0 errors; warnings 328, same as master 8c0502b (ceiling 330): this PR adds none
uvx pre-commit run --all-files	ruff check, ruff format and mixed line ending pass

CI on head 323690f (before the rebase; CI re-runs on 31e7977): ruff, pyrefly, test (ubuntu-latest / windows-latest, 3.11 / 3.14) all pass. On both Windows jobs tests\test_filename_encoding.py .s.s. (181 passed, 2 skipped, 8 xfailed): the Chinese cases skip on cp1252 and test_unrepresentable_path_gives_clear_error runs and passes, exercising the new locale.getencoding() message path.

MIKE IO downstream (test_mikeio.yml, MIKE IO main), branch 64db468, before the rebase onto 8c0502b vs master a5b84a1: the same failures on all 4 jobs, both already failing on master and unrelated to paths: test_eum.py::test_eum_conversion (all jobs) and test_dfs2.py::test_select_area_rotated_UTM_2 (Linux, last-digit float). Counts are identical: ubuntu 2 failed, 978 passed, windows 1 failed, 978 passed, 1 skipped. Each job's log shows the checkout's mikecore 0.3.0a1 installed.

Verify it yourself

gh pr checkout 63
uv sync
uv run pytest -v tests/test_filename_encoding.py   # After
uvx pre-commit run --all-files
uv run pyrefly check --output-format=json --min-severity=warn --summary=none | grep -c '"severity": "warn"'
# Before: master's code with only the new test
git checkout --detach origin/master
git checkout fix/non-ascii-filenames -- tests/test_filename_encoding.py
uv run pytest -q -rfs tests/test_filename_encoding.py
git reset --hard && git checkout fix/non-ascii-filenames

Windows Chinese-folder script

Run from a checkout's root with uv run python cn.py .:

import os, shutil, sys, tempfile
sys.path.insert(0, sys.argv[1])
from mikecore.DfsuFile import DfsuFile

folder = os.path.join(tempfile.mkdtemp(), "项目 合界数模")
os.mkdir(folder)
fn = os.path.join(folder, "OresundHD.dfsu")
shutil.copy(os.path.join(sys.argv[1], "testdata", "OresundHD.dfsu"), fn)
try:
    DfsuFile.Open(fn)
except Exception as e:
    print(type(e).__name__ + ":", ascii(str(e)))

Linux non-ASCII install directory

mkdir -p "/tmp/x/søren" && cp -r mikecore "/tmp/x/søren/"
.venv/bin/python -c "import sys; sys.path.insert(0, '/tmp/x/søren'); import mikecore; print('imported', mikecore.__file__)"

Risk

• Windows paths that worked before still work only if mbcs encodes everything cp1252 did. That rests on the ACP being cp1252 on those machines: then mbcs and cp1252 produce the same bytes for every representable name. Proved on a cp1252 machine by the Danish cases and the full suite (151 passed on Windows, before the review fixes). On a machine whose ACP is not cp1252, a name with e.g. ø used to be sent as cp1252 bytes the native library would misread; now it is sent in the ACP or rejected with the ValueError. Not tested on a non-cp1252 Windows machine (unproven; the Chinese cases would run there instead of skipping).
• Linux: os.fsencode gives UTF-8, which is what the filesystem stores; ASCII paths encode to the same bytes as before. Proved by the Linux suite and the Before/After above.
• Full CJK support on cp1252 Windows is still impossible: the native open is char*. Users get a clear error instead of a crash in cp1252.py.
• Exception type: on Windows an unrepresentable name now raises ValueError instead of UnicodeEncodeError. UnicodeEncodeError is itself a ValueError, so except ValueError callers are unaffected; only a caller catching UnicodeEncodeError specifically would miss it (the MIKE IO suite shows no new failures, see Evidence).
• Unproven assumption: that the native ufs library opens its char* path with the ANSI file APIs, so that with Windows' UTF-8 setting on, the mbcs (= UTF-8) bytes open correctly. Not tested here, and the error message does not suggest it.
• No MIKE Zero check: the change is in how a path reaches the native library, not in what is read or written, so the GUI would show the same file content.

Not in this PR

• File titles, application titles and item names are still encoded as cp1252 (DfsBuilder.py:349-350, 511); those are strings stored inside the file, not paths.
• Opening paths outside the ANSI code page on Windows would need a wchar_t* open in the native ufs library.
• An 8.3 short-path fallback (GetShortPathNameW) could open existing files outside the ANSI code page without the OS setting; not attempted.
• The review's to-do items (drop windll, type filename, finally in the tests, the DfsDLLUtil docstring) are done in commit 3.

🤖 Generated with Claude Code