Skip to content

流式导入 - #63

Merged
chsword merged 3 commits into
mainfrom
feat/streaming-import
Sep 16, 2026
Merged

chsword merged 3 commits into
mainfrom
feat/streaming-import

Conversation

@chsword

@chsword chsword commented Sep 16, 2026

Copy link
Copy Markdown
Owner

做了什么

新增 ExcelStreamToObject:逐行读出 .xlsx 的一张工作表,不把整个工作簿建进内存。上一版让导出不再受行数牵制,这一版补上读的一侧。

using var file = File.OpenRead("orders.xlsx");
foreach (var order in ExcelHelper.ExcelStreamToObject<Order>(file))
    Handle(order);

本机实测(五列,含文本、金额与日期,net8.0):

行数 整份读入 流式导入
5 万 393 MB / 3.1 秒 103 MB / 1.1 秒
20 万 1265 MB / 8.4 秒 194 MB / 2.9 秒

两条路读出的数据逐字段一致(实测中以金额求和校验,两者相同)。快出一截是因为不必先为每一格建一个对象。流式导入的内存并非常数:共享字符串表会整个留在内存里,省掉的是「每一格一个对象」那一份。

怎么做的

读的是工作表那份 XML(XSSFReader 取出各部件,XmlReader 逐行解析),共享字符串表与样式表仍走 NPOI 的公开入口。一行解出来即交给调用方,随后便可回收。

导入的类型转换原本直接写在 ICell 之上,而逐行读出拿到的是原始 XML,并没有单元格对象。故把两者都归到内部的 CellData(一格读到的东西)与 IImportRow(一行)上:转换那一套因而只有一份,两条路的行为不会各自漂移。这一步是纯重构,改完既有的 392 个用例原样通过。

三处不同

  • 公式格读的是文件中存着的上一次计算结果,而非当场求值。Excel 存盘时会写下这个结果,故 Excel 保存过的文件照常读出;本库导出的文件里公式没有这个结果(要等 Excel 打开时才算出),此时该格读作空白。整份读入那条路仍当场求值,行为不变。
  • 只有 .xlsx 能够逐行读出,.xls 的数据并非顺序存放,传入时照旧整份读入,结果一致。
  • 工作表在一开始就定位,表名写错当场抛出 Excel2ObjectException,不必等到取第一行。

行为变更

均对既有入口同样生效,属读取行为的收敛:字典形式的导入改为惰性给出;标题行中同名的标题以最左一列为准(此前 ToDictionary 会抛 ArgumentException);非文本的标题不再中断导入;日期序列号超出 Excel 日历时上报的异常改为本库抛出并写明该序列号。

测试

新增 12 个用例(StreamingImportTest):与整份读入逐字段比对、共享字符串与行内字符串两种写法、空白与缺列、按表名定位、表名写错当场报错、TitleSkipLine、公式有/无缓存值、读取失败的上报位置(D4)、惰性取用、.xls 入口、字典形式。

全量 404 个用例在 net8.0 与 net10.0 通过,库在 7 个目标框架上 0 警告 0 错误。

版本 2.11.0,文档见 docs/versions/v2.11.0.md。

🤖 Generated with Claude Code

ExcelStreamToObject 逐行读出 .xlsx 的一张工作表,不把整个工作簿建进内存:读的是工作表那份 XML,一行解出来即交给调用方。实测二十万行五列:整份读入峰值 1265 MB / 8.4 秒,流式 194 MB / 2.9 秒,两条路读出的数据逐字段一致。取到的序列是惰性的,Take 一类的操作真的能少读。

为此把导入的类型转换与 NPOI 的单元格解耦:一格读到的东西归到 CellData,一行归到 IImportRow,整份读入与逐行读出因而共用同一套转换,不会各自漂移。

三处不同写在文档里:公式格读的是文件中存着的上一次计算结果而非当场求值(本库导出的文件没有这个结果,读作空白);只有 .xlsx 能逐行读出,.xls 照旧整份读入;工作表在一开始就定位,表名写错当场报错。

行为变更:字典形式的导入改为惰性给出;标题行中同名的标题以最左一列为准(此前抛 ArgumentException);非文本的标题不再中断导入;日期序列号越界时上报的异常改为本库抛出并写明该序列号。

版本 2.11.0。

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Copilot AI lite review requested due to automatic review settings September 16, 2026 02:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical XML parsing and multiple moderate correctness and resource-cleanup issues remain.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds lazy, row-by-row .xlsx importing through ExcelStreamToObject, while unifying existing conversion logic.

Changes:

  • Added XML-based streaming worksheet parsing.
  • Refactored shared cell, row, and error handling.
  • Added tests, documentation, and version 2.11.0 updates.
File summaries
File Summary Findings
README.md Chinese feature and release documentation Nit: incorrect additional v2.10.0 link.
README_EN.md English feature and release documentation Nit: incorrect additional v2.10.0 link.
docs/versions/v2.11.0.md Version documentation and API examples No findings.
docs/README.md Documentation index update No findings.
Chsword.Excel2Object/Internal/XlsxRowReader.cs Streaming XLSX XML parser Critical namespace parsing issue; moderate empty-row, row-number, and uncached-formula handling issues.
Chsword.Excel2Object/Internal/ImportRow.cs Shared import-row abstraction No findings.
Chsword.Excel2Object/Internal/ImportContext.cs Import context and error positioning No findings.
Chsword.Excel2Object/Internal/CellData.cs Shared cell representation No findings.
Chsword.Excel2Object/ExcelImporter.cs Streaming API and conversion refactor Moderate resource cleanup, option callback, duplicate-header, and non-seekable .xls issues; nit on nullable cast.
Chsword.Excel2Object/ExcelHelper.cs Static streaming API facade No findings.
Chsword.Excel2Object/Chsword.Excel2Object.csproj Version update to 2.11.0 No findings.
Chsword.Excel2Object.Tests/StreamingImportTest.cs Streaming import test coverage No findings.
Review details

Suppressed comments (3)

Chsword.Excel2Object/ExcelImporter.cs:204

  • Duplicate header titles are still not resolved to the leftmost column for model imports: ContainsKey(cell.Key) only checks the current column, so both occurrences are added and PopulateModelFromRow assigns the property twice, leaving the rightmost value. De-duplicate by the matched property/title, as the dictionary path does, so the first matching column is retained.
                if (prop.Key != null && !dictColumns.ContainsKey(cell.Key))

Chsword.Excel2Object/ExcelImporter.cs:118

  • For a seekable .xls stream this branch falls back to the existing importer, but for a non-seekable .xls stream LooksLikeXlsx returns true and OPCPackage.Open rejects it as an .xlsx. Thus the documented .xls fallback only works for seekable streams even though this public API accepts any Stream; either preserve the prefix while detecting the format or document and validate this restriction.
        if (!input.CanSeek) return true;

        var position = input.Position;
        var head = new byte[2];
        var read = input.Read(head, 0, 2);
        input.Position = position;
        return read == 2 && head[0] == (byte) 'P' && head[1] == (byte) 'K';

Chsword.Excel2Object/ExcelImporter.cs:133

  • This null-forgiving cast suppresses nullable analysis in newly added code. The type test makes the runtime cast safe, but the repository's nullable guidance requires avoiding ! for this pattern; return a typed lazy projection (for example, Cast<TModel>()) instead of hiding the warning.
        if (typeof(TModel) == typeof(Dictionary<string, object>))
            return (InternalExcelToDictionary(rows, context) as IEnumerable<TModel>)!;
  • Files reviewed: 12/12 changed files
  • Comments generated: 8
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

{
if (xml.NodeType != XmlNodeType.Element) continue;

switch (xml.Name)
Comment thread Chsword.Excel2Object/ExcelImporter.cs Outdated
optionAction?.Invoke(options);

// .xls 的格式无从逐行读出,整份读入后结果与既有方式一致
if (!LooksLikeXlsx(input)) return ExcelToObject<TModel>(ReadAll(input), optionAction);
Comment on lines +129 to +136
private static IEnumerable<TModel> ToModels<TModel>(IEnumerator<IImportRow>? rows, ImportContext context)
where TModel : class, new()
{
if (typeof(TModel) == typeof(Dictionary<string, object>))
return (InternalExcelToDictionary(rows, context) as IEnumerable<TModel>)!;

return InternalExcelToObject<TModel>(rows, context);
}
if (xml.NodeType == XmlNodeType.Element && xml.Name == "row")
{
// 行号自 1 计起;缺失时按上一行顺延
rowIndex = Number(xml.GetAttribute("r")) - 1;
if (rowIndex < 0) rowIndex = 0;
cells = new Dictionary<int, CellData>();
column = -1;
if (!xml.IsEmptyElement) continue;
Comment on lines +214 to +218
if (text == null) return CellData.Blank();

var format = styles.Of(style);
return CellData.OfNumber(double.Parse(text, NumberStyles.Float, CultureInfo.InvariantCulture),
format.Index, format.Text, date1904);
Comment thread README.md Outdated
- [x] 合并单元格:按列合并连续相同值、指定区域 ✅ **v2.9.1 新增**
- [x] 大文件流式导出:内存不随行数增长 ✅ **v2.10.0 新增** - 查看 [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md)
- [x] 大文件流式导出:内存不随行数增长 ✅ **v2.10.0 新增**
- [x] 大文件流式导入:逐行读出,不把整个工作簿建进内存 ✅ **v2.11.0 新增** - 查看 [docs/versions/v2.11.0.md](docs/versions/v2.11.0.md) - 查看 [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md)
Comment thread README_EN.md Outdated
- [x] Merged cells: consecutive equal values in a column, or a range of your own ✅ **New in v2.9.1**
- [x] Streaming export for large files: a memory footprint that does not grow with the row count ✅ **New in v2.10.0** - See [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md)
- [x] Streaming export for large files: a memory footprint that does not grow with the row count ✅ **New in v2.10.0**
- [x] Streaming import for large files: read row by row instead of building the whole workbook ✅ **New in v2.11.0** - See [docs/versions/v2.11.0.md](docs/versions/v2.11.0.md) - See [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md)
chsword and others added 2 commits September 16, 2026 03:10
元素名改用 LocalName:带前缀的写法(<x:row>)同样合法,此前会把这类文件读成没有数据;行的 r 属性缺失时按上一行顺延,此前一律归到第 0 行,错误位置因而报错;自闭合的空行就地给出,否则 TitleSkipLine 会把它数漏。

没有算出结果的公式常写成一个空的 <v/>,此前会交给 double.Parse 而成为「读取失败」,与文档所说的「读作空白」不符。现空内容一律读作空白,并在用例中确认没有上报。空白格落在非可空数值列上仍以转换失败中止,与整份读入一致,另有用例把二者钉在一起。

.xls 的回退此前会把 optionAction 调用两次,有状态的回调因而生效两次;改为先判断格式再建立选项。取行的枚举器改由转换那一步持有并释放:流式导入由它持有着打开的工作簿,Take 或 break 中途放弃时也须关上。

另有 README 特性表中错接到 v2.10.0 的链接,以及运算优先级、ReadCell 拆分、去掉与其余入口不一致的判空。

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
流式导入的用例要拆看 .xlsx 里的部件,用到 ZipArchive,而 SDK 风格的项目在 .NET Framework 下并不默认引用该程序集。

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
@chsword
chsword merged commit b2a62cc into main Sep 16, 2026
9 checks passed
@chsword
chsword deleted the feat/streaming-import branch September 16, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants