Repository navigation
perf: cut DOCX, spreadsheet and HTML memory; refresh bench results - #49
Merged
Merged
Conversation
Measures per-case latency, ops/s, throughput and heap allocations (count, bytes, peak) plus process peak RSS over a generated HTML/DOCX/XLSX/ODS corpus. just bench saves JSON + Markdown reports to crates/markkit-bench/results/, mirroring bashkit-bench. Criterion moves to just bench-criterion.
…duplication html5ever's tree builder is quadratic in nesting depth: 687 KiB of nested divs took 12.6 s and 292 KiB of unclosed nested tables ran past the 60 s timeout. A pre-scan (html::nesting::cap_nesting) now drops tags nested past max_nesting_depth + 8 before html5ever runs, keeping text. Tables took rows via descendant queries, copying nested-table rows into every enclosing table (4x output per level). Rows now come from direct structure only. Text flattened at max_nesting_depth leaked script/style/template bodies; flat_text skips non-content elements. New invariant S17.
# Conflicts: # knowledge/security/threat-model.md
docx_report: peak heap 18.5 MiB -> 3.9 MiB, allocations 265k -> 137k. Body blocks are converted while document.xml is parsed and freed after; XML names and attribute values are interned; attribute and child lists are exact-size; run text stays borrowed until it becomes IR. Markdown escaping and output normalization skip per-run/per-line allocations, which helps every format.
xlsx_large (150k cells): allocations 768k -> 513k, ods_medium 86k -> 72k. XLSX cells are formatted straight from calamine's DataRef instead of cloning into Data first, cell text is trimmed in place, and table layout sizes each grid row once.
html_docs_page peak 7.5 -> 4.7 MiB, html_tables 8.6 -> 5.1 MiB; allocations roughly halved. The scraper node arena is sized from a clamped byte scan instead of growing by doubling, collect_inline sizes its node list from the child count, text is collapsed without a copy, and nested renderers no longer reserve 4 KiB each.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
Fixes the remaining
markkit-benchfindings (the nesting one shipped in #43). Output is unchanged; memory and allocations drop.document.xmlis converted block by block while it is parsed, and each block's XML subtree is freed right after (xml::parse_split). The XML tree is compact: element names and attribute values are interned, attribute and child lists are exact-size boxed slices, and run text stays borrowed until it becomes IR (sharedooxmlinline code, so PPTX benefits too). ZIP entries are read into a buffer pre-sized from the declared size (clamped tomax_archive_entry_sizeand 16 MiB; limits still enforced on real bytes).DataRef(no clone intoDataand second copy), and cell text is trimmed in place.Why
Bench findings from #22: DOCX peaked at 18.5 MiB for a 9 KiB file, XLSX made ~5 allocations per cell, HTML peaked at ~25x input.
Before / After
just bench, allocations and peak heap per extraction. Before ismainat 16b3014 + #43, after is this branch:DOCX report latency 52.6 → ~20 ms. Other timings are within noise. Re-checked after syncing with #44/#45/#47/#48: same numbers.
Risk
w:document, now:body) still surface (docx_without_document_or_body_is_error), and node and depth limits still apply while streaming (docx_streamed_body_keeps_node_and_depth_limits).Checklist
fmt_ref=fmt_data,trim_owned, node estimate clamps, DOCX structure/limit tests)🤖 Generated with Claude Code