Repository navigation
scan: look archives up by their first member's CRC first - #19937
Merged
LibretroAdmin merged 5 commits intoOct 28, 2026
Merged
Conversation
A zip, 7z, apk, rar, zst or bz2 was hashed whole before its first member's CRC was fetched. Cartridge databases key on ROM CRCs, so a one-ROM-per-zip set always matches by the member: the whole read, the main cost of scanning zips on a slow share, was wasted. The databases are now walked first for the member's CRC alone. Where it is stored (zip, 7z, RAR with CRC32) that costs one directory read; a zst, bz2 or BLAKE2 RAR member is decoded for it, as before. Only when that misses is the archive read whole, or a reader's whole result taken, and the databases walked again for its own CRC, which the arcade databases key on. The archive is expanded only after both miss. Each walk's size check uses its own size: the member's, then the archive's. By default the core gate looks in a database only when a core claims the file's extension. Stock cartridge cores do not list zip, so a cartridge zip was read whole and matched only once expanded. The first walk now also takes the databases claimed for the first member's extension. A hit there is held while the walk finishes the databases claimed for the archive's name: a match in one of those is written for the archive; otherwise the archive is expanded without the whole read. With "Scan Inside Archives" off nothing expands, so the first walk keeps to the archive's name. The parallel readers likewise start an archive with its first member's CRC, size and extension, and hand them to the scan. They hash archives whole only as far as the scan grants: a member-CRC miss grants one whole read ahead, each further miss in a row twice the last, up to 16 at a time, and a member-CRC match takes back what is left. A lone unknown cartridge then costs one wasted read, while an arcade folder soon has the readers 16 ahead. While the scan first walks a database's index in a member pass mid streak, the readers may keep 16 ahead, so the walk does not leave them idle. A grant goes first to the nearest archive ahead that has only its member CRC. Without a reader's whole result the scan hashes an archive it needs whole itself, in steps, like a file over the cutoff. An archive with no CRC for its first entry, such as a zip or 7z with a directory first, skipped every database, so its own CRC was never compared. It now goes straight to the whole walk, and readers always hash it whole. In first-match mode the walk's early exit leads to the whole walk too, while that has not run. The CRCs, sizes and walk are cleared at each file's start. A file that errored left them to the next, and an archive after it was looked up by the stale CRC. The serial is cleared with them: an archive matched after a file that errored in its serial lookup was written under that file's serial. Every database's member CRC is now checked before any whole CRC; each database used to check the whole CRC first. That differs only when a zip's first member matches a cartridge record and the whole zip an arcade record. The cartridge record now wins, written for the zip itself where the core gate lets the zip's name into the cartridge database, otherwise through the expansion. With "Scan Inside Archives" off nothing expands, so in that second case the arcade record still wins.
A record count from 128 up is written as a msgpack uint32. The read-ahead lanes size their sets from the reader window, 64 files per reader, so they write 131 to 259 records; a fixint byte past 127 decodes as a map or a bool and the database is rejected on two- and three-core hosts.
A database whose crc index build is refused is flagged, and neither the size probe nor the CRC lookup starts another build for it. The index budget only shrinks, so a retry could only walk the database again to the same refusal, once per content file. The database scan test gains a cap on index builds and a lane that scans plain files with every build refused: one query walk each.
An archive whose header lists more than one entry, an arcade set as a rule, is read whole first, and each database takes its own CRC or its first member's. A set whose first member is a cartridge ROM is written for its arcade record rather than expanded into the cartridge playlist, and an arcade folder walks no database claimed only for a member's extension. Readers read such an archive whole without a grant. The member pass stays for one-member archives, where a cartridge zip needs no whole read. file_archive_get_file_crc32_size_and_ext() reports the entry count the backends already take from the archive's header. A database claimed only for the member's extension joins the member pass only while it has or may get a crc index; unindexed, the lookup is a full walk the expansion repeats. A reader's member CRC survives a whole read that fails or is over the cutoff, so the scan reads only the archive itself. Lanes: an arcade set with a cartridge ROM first, an arcade folder beside a cartridge database, sets whose reader reads fail, and cartridge zips against an unindexed database.
When the archive is mapped, zip_parse_file_init() finds the end of central directory record, reads the ZIP64 records and walks the central directory where they lie; only the context is allocated. Unmapped archives read them through the stream. The scan's member pass opens every zip for its directory, so this is its hot path. archive_zip64_test counts reads through the stream and checks that, mapped, a plain zip with a comment and a ZIP64 one give their first member's CRC without any.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Guidelines
C89_BUILD=1Description
A follow-up to #19920. Archives were hashed whole before their first member's CRC was looked up. Cartridge databases key on ROM CRCs, so for one-ROM zips that read decided nothing; on a network share it was most of the scan.
Changes
Evidence
Wall seconds, base -> this commit, cold-run medians;
*: beyond the base runs' range. R=0 has no readers, R=4 four. cart: one-ROM cartridge zips (NES; 15 systems on Android); fbneo-tz: 596 TorrentZipped FBNeo parents. Playlists equal base's in every run.NFS 4.2, Linux, against d2db43b:
†: 10 alternating rounds; both within spread. fbneo-tz's ~0.05-0.09 s is the one-time walk of the extra cart rdbs (see the notes below).
Measured before the rebase, against ad2e9e0 (#19920 as merged):
Notes for review
.rdbfiles (39 MB), each indexed once per scan in steps (Android index time 7-28 -> 100-137 ms), then about 24 µs of lookups per archive..rdbper lookup, now per arcade archive for each such database, a main-thread stall per archive without threads.x.zipin any database order, where base's result depended on it; (b) otherwise the zip expands into the cartridge playlist, where base wrotex.zipto the arcade one.scan_begin_budget_test.exeexits 53 under wine, at base too;tools/stack_budget.pypasses.Tested
On this commit's code:
database_scan_test(199 checks, 76 new) anddatabase_index_testunder ASan+UBSan, TSan,HAVE_THREADS=0, readers off and a read-ahead cap of 2;C89_BUILD=1objects, 0 warnings; default and--disable-threadsbuilds; CI samples-tasks viatools/ci_local.py; mingw-w64 tests under wine; a RAR 5, .bz2 and .zst fixture scan (playlists as base's). Before the rebase also: the scan test under 32 busy loops,tools/stack_budget.py, the tree-wide checks.The full GitHub CI passed on this code in my fork: 151 jobs, Apple jobs included.
Related Issues
[Any issues this pull request may be addressing]
Related Pull Requests
Reviewers
[If possible @mention all the people that should review your pull request]