Repository navigation
流式导入 - #63
Merged
Merged
流式导入#63
Conversation
ExcelStreamToObject 逐行读出 .xlsx 的一张工作表,不把整个工作簿建进内存:读的是工作表那份 XML,一行解出来即交给调用方。实测二十万行五列:整份读入峰值 1265 MB / 8.4 秒,流式 194 MB / 2.9 秒,两条路读出的数据逐字段一致。取到的序列是惰性的,Take 一类的操作真的能少读。 为此把导入的类型转换与 NPOI 的单元格解耦:一格读到的东西归到 CellData,一行归到 IImportRow,整份读入与逐行读出因而共用同一套转换,不会各自漂移。 三处不同写在文档里:公式格读的是文件中存着的上一次计算结果而非当场求值(本库导出的文件没有这个结果,读作空白);只有 .xlsx 能逐行读出,.xls 照旧整份读入;工作表在一开始就定位,表名写错当场报错。 行为变更:字典形式的导入改为惰性给出;标题行中同名的标题以最左一列为准(此前抛 ArgumentException);非文本的标题不再中断导入;日期序列号越界时上报的异常改为本库抛出并写明该序列号。 版本 2.11.0。 Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical XML parsing and multiple moderate correctness and resource-cleanup issues remain.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds lazy, row-by-row .xlsx importing through ExcelStreamToObject, while unifying existing conversion logic.
Changes:
- Added XML-based streaming worksheet parsing.
- Refactored shared cell, row, and error handling.
- Added tests, documentation, and version 2.11.0 updates.
File summaries
| File | Summary | Findings |
|---|---|---|
README.md |
Chinese feature and release documentation | Nit: incorrect additional v2.10.0 link. |
README_EN.md |
English feature and release documentation | Nit: incorrect additional v2.10.0 link. |
docs/versions/v2.11.0.md |
Version documentation and API examples | No findings. |
docs/README.md |
Documentation index update | No findings. |
Chsword.Excel2Object/Internal/XlsxRowReader.cs |
Streaming XLSX XML parser | Critical namespace parsing issue; moderate empty-row, row-number, and uncached-formula handling issues. |
Chsword.Excel2Object/Internal/ImportRow.cs |
Shared import-row abstraction | No findings. |
Chsword.Excel2Object/Internal/ImportContext.cs |
Import context and error positioning | No findings. |
Chsword.Excel2Object/Internal/CellData.cs |
Shared cell representation | No findings. |
Chsword.Excel2Object/ExcelImporter.cs |
Streaming API and conversion refactor | Moderate resource cleanup, option callback, duplicate-header, and non-seekable .xls issues; nit on nullable cast. |
Chsword.Excel2Object/ExcelHelper.cs |
Static streaming API facade | No findings. |
Chsword.Excel2Object/Chsword.Excel2Object.csproj |
Version update to 2.11.0 | No findings. |
Chsword.Excel2Object.Tests/StreamingImportTest.cs |
Streaming import test coverage | No findings. |
Review details
Suppressed comments (3)
Chsword.Excel2Object/ExcelImporter.cs:204
- Duplicate header titles are still not resolved to the leftmost column for model imports:
ContainsKey(cell.Key)only checks the current column, so both occurrences are added andPopulateModelFromRowassigns the property twice, leaving the rightmost value. De-duplicate by the matched property/title, as the dictionary path does, so the first matching column is retained.
if (prop.Key != null && !dictColumns.ContainsKey(cell.Key))
Chsword.Excel2Object/ExcelImporter.cs:118
- For a seekable
.xlsstream this branch falls back to the existing importer, but for a non-seekable.xlsstreamLooksLikeXlsxreturnstrueandOPCPackage.Openrejects it as an.xlsx. Thus the documented.xlsfallback only works for seekable streams even though this public API accepts anyStream; either preserve the prefix while detecting the format or document and validate this restriction.
if (!input.CanSeek) return true;
var position = input.Position;
var head = new byte[2];
var read = input.Read(head, 0, 2);
input.Position = position;
return read == 2 && head[0] == (byte) 'P' && head[1] == (byte) 'K';
Chsword.Excel2Object/ExcelImporter.cs:133
- This null-forgiving cast suppresses nullable analysis in newly added code. The type test makes the runtime cast safe, but the repository's nullable guidance requires avoiding
!for this pattern; return a typed lazy projection (for example,Cast<TModel>()) instead of hiding the warning.
if (typeof(TModel) == typeof(Dictionary<string, object>))
return (InternalExcelToDictionary(rows, context) as IEnumerable<TModel>)!;
- Files reviewed: 12/12 changed files
- Comments generated: 8
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| { | ||
| if (xml.NodeType != XmlNodeType.Element) continue; | ||
|
|
||
| switch (xml.Name) |
| optionAction?.Invoke(options); | ||
|
|
||
| // .xls 的格式无从逐行读出,整份读入后结果与既有方式一致 | ||
| if (!LooksLikeXlsx(input)) return ExcelToObject<TModel>(ReadAll(input), optionAction); |
Comment on lines
+129
to
+136
| private static IEnumerable<TModel> ToModels<TModel>(IEnumerator<IImportRow>? rows, ImportContext context) | ||
| where TModel : class, new() | ||
| { | ||
| if (typeof(TModel) == typeof(Dictionary<string, object>)) | ||
| return (InternalExcelToDictionary(rows, context) as IEnumerable<TModel>)!; | ||
|
|
||
| return InternalExcelToObject<TModel>(rows, context); | ||
| } |
| if (xml.NodeType == XmlNodeType.Element && xml.Name == "row") | ||
| { | ||
| // 行号自 1 计起;缺失时按上一行顺延 | ||
| rowIndex = Number(xml.GetAttribute("r")) - 1; |
| if (rowIndex < 0) rowIndex = 0; | ||
| cells = new Dictionary<int, CellData>(); | ||
| column = -1; | ||
| if (!xml.IsEmptyElement) continue; |
Comment on lines
+214
to
+218
| if (text == null) return CellData.Blank(); | ||
|
|
||
| var format = styles.Of(style); | ||
| return CellData.OfNumber(double.Parse(text, NumberStyles.Float, CultureInfo.InvariantCulture), | ||
| format.Index, format.Text, date1904); |
| - [x] 合并单元格:按列合并连续相同值、指定区域 ✅ **v2.9.1 新增** | ||
| - [x] 大文件流式导出:内存不随行数增长 ✅ **v2.10.0 新增** - 查看 [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md) | ||
| - [x] 大文件流式导出:内存不随行数增长 ✅ **v2.10.0 新增** | ||
| - [x] 大文件流式导入:逐行读出,不把整个工作簿建进内存 ✅ **v2.11.0 新增** - 查看 [docs/versions/v2.11.0.md](docs/versions/v2.11.0.md) - 查看 [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md) |
| - [x] Merged cells: consecutive equal values in a column, or a range of your own ✅ **New in v2.9.1** | ||
| - [x] Streaming export for large files: a memory footprint that does not grow with the row count ✅ **New in v2.10.0** - See [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md) | ||
| - [x] Streaming export for large files: a memory footprint that does not grow with the row count ✅ **New in v2.10.0** | ||
| - [x] Streaming import for large files: read row by row instead of building the whole workbook ✅ **New in v2.11.0** - See [docs/versions/v2.11.0.md](docs/versions/v2.11.0.md) - See [docs/versions/v2.10.0.md](docs/versions/v2.10.0.md) |
元素名改用 LocalName:带前缀的写法(<x:row>)同样合法,此前会把这类文件读成没有数据;行的 r 属性缺失时按上一行顺延,此前一律归到第 0 行,错误位置因而报错;自闭合的空行就地给出,否则 TitleSkipLine 会把它数漏。 没有算出结果的公式常写成一个空的 <v/>,此前会交给 double.Parse 而成为「读取失败」,与文档所说的「读作空白」不符。现空内容一律读作空白,并在用例中确认没有上报。空白格落在非可空数值列上仍以转换失败中止,与整份读入一致,另有用例把二者钉在一起。 .xls 的回退此前会把 optionAction 调用两次,有状态的回调因而生效两次;改为先判断格式再建立选项。取行的枚举器改由转换那一步持有并释放:流式导入由它持有着打开的工作簿,Take 或 break 中途放弃时也须关上。 另有 README 特性表中错接到 v2.10.0 的链接,以及运算优先级、ReadCell 拆分、去掉与其余入口不一致的判空。 Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
流式导入的用例要拆看 .xlsx 里的部件,用到 ZipArchive,而 SDK 风格的项目在 .NET Framework 下并不默认引用该程序集。 Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
做了什么
新增
ExcelStreamToObject:逐行读出.xlsx的一张工作表,不把整个工作簿建进内存。上一版让导出不再受行数牵制,这一版补上读的一侧。本机实测(五列,含文本、金额与日期,net8.0):
两条路读出的数据逐字段一致(实测中以金额求和校验,两者相同)。快出一截是因为不必先为每一格建一个对象。流式导入的内存并非常数:共享字符串表会整个留在内存里,省掉的是「每一格一个对象」那一份。
怎么做的
读的是工作表那份 XML(
XSSFReader取出各部件,XmlReader逐行解析),共享字符串表与样式表仍走 NPOI 的公开入口。一行解出来即交给调用方,随后便可回收。导入的类型转换原本直接写在
ICell之上,而逐行读出拿到的是原始 XML,并没有单元格对象。故把两者都归到内部的CellData(一格读到的东西)与IImportRow(一行)上:转换那一套因而只有一份,两条路的行为不会各自漂移。这一步是纯重构,改完既有的 392 个用例原样通过。三处不同
.xlsx能够逐行读出,.xls的数据并非顺序存放,传入时照旧整份读入,结果一致。Excel2ObjectException,不必等到取第一行。行为变更
均对既有入口同样生效,属读取行为的收敛:字典形式的导入改为惰性给出;标题行中同名的标题以最左一列为准(此前
ToDictionary会抛ArgumentException);非文本的标题不再中断导入;日期序列号超出 Excel 日历时上报的异常改为本库抛出并写明该序列号。测试
新增 12 个用例(
StreamingImportTest):与整份读入逐字段比对、共享字符串与行内字符串两种写法、空白与缺列、按表名定位、表名写错当场报错、TitleSkipLine、公式有/无缓存值、读取失败的上报位置(D4)、惰性取用、.xls入口、字典形式。全量 404 个用例在 net8.0 与 net10.0 通过,库在 7 个目标框架上 0 警告 0 错误。
版本 2.11.0,文档见
docs/versions/v2.11.0.md。🤖 Generated with Claude Code