Repository navigation
scan: read an archive whole only when a database may hold its size - #19944
Merged
Merged
Conversation
The scan fetched an archive's first member's CRC, size and extension through a helper that stops at the first entry and reports how many the header counts, and its readers did the same. One walk now reads the whole directory, in the readers as in the scan: every member's stored CRC, size and name, kept for each of the archive's passes. Its transfer state is on the heap, out of the readers' frames. The walk replaces the helper in the archive passes; the helper stays for its other users. The walk decodes the first member for a CRC where the archive stores none, a zstd stream or a RAR 5 member hashed with BLAKE2, so it is looked up as before. Later members are not decoded and keep a CRC of 0. A .bz2 is decoded to be listed at all. Folders are not members: an archive of a folder and one file has one member, that file, and is looked up by its CRC first.
A zip whose end-of-central-directory comment is TORRENTZIPPED- and eight hex digits was written by TorrentZip, as arcade sets are. The zip reader already reads that record to find the directory, so it now says so in the transfer state, for a caller that wants the hint. The CRC in the comment is not checked: nothing depends on it.
The crc index build walks each database once, off the main thread. In the same walk it now also collects the size of every record whose rom_name ends in an archive extension (zip, 7z, apk, rar, zst), kept in one sorted array keyed by size and type, so a caller can ask whether a database may hold an archive of a given type and size. A record of that type with no size, or an index built without the sizes, answers yes. The sizes count against the index budget. The stepped database scan reports one string field for this: libretrodb_scan_begin() and its callback gain a name field. A record's rom_name can also be extracted on request.
An archive whose first member missed was read whole and, when that missed too, expanded: each member went through the databases again, by the stored CRC the directory already held. An archive of several members was read whole before any lookup, even where no database could hold it, so disc zips and console sets were read whole again. One member: its stored CRC is looked up first, as before. A hit in a database claimed only for the member's type is held, and every member's CRC is looked up next. On a miss the archive is read whole only when a database whose cores open its type may hold it by its exact size; its own CRC is then looked up there, and where that is not asked or misses, every member's. Several members: the same size question comes first. Each database whose cores open the type is then asked for the first member's CRC, and for the archive's own CRC only when the size let it be read whole, so an arcade record still wins and no database claimed only for a member's type is walked before it. Where neither matches, every member's stored CRC is looked up. A database may hold the archive when it has a record whose rom_name is of the archive's type and size, one of that type with no size, or no sizes at all. Every member is looked up with the gates and size checks the expansion uses. A hit names the archive itself only when it is the one record hit, that record is named for this archive and its database's cores open the type; other hits are written as the expansion wrote them, archive#member. The expansion stays for scans that keep unmatched files, for archives whose directory cannot be read, and where nothing hit and a member stores no CRC, which only the expansion unpacks for one. Readers read an archive whole ahead only after a whole match, twice as many after each such match up to the cap; any other verdict takes the rest back. A TorrentZip set of several members is still read whole by a reader with no grant while some database opens zip. An archive over the readers' cutoff is handed back before a grant is spent on it. This narrows "several members are read whole first" to "read whole when a database may hold it by its size". A record whose crc is an archive's own CRC but whose rom_name is not of the archive's type, or whose size is not the file's, is no longer matched; each of FBNeo's 596 sets carries its zip's size. A zip of a folder and one ROM takes the one-member order.
A RAR 4 archive of stored entries, written by hand as the zips are, takes the passes zip and 7z take: a first member's hit where a core opens rar names the archive, a later member's elsewhere is held as archive#member, and the archive is read whole only while a database whose cores open rar holds a rar record of its size.
Davey-Hughes
force-pushed
the
scan-size-gate
branch
from
October 30, 2026 01:27
d0562ba to
5159259
Compare
The flag was cleared only by the zip reader, so a transfer reused for a 7z, a rar or an archive with no backend kept what the last zip said. It is now cleared before any backend opens the archive.
Readers read every TorrentZip set of several members whole while some database opens zip, but where no database may hold a set by its size the scan drops that CRC, so a folder of sets no database holds was read whole for nothing. Each TorrentZip set the scan rules on now says whether its own CRC was asked, and the readers read the next ones whole only while it was.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A follow-up to #19937. As merged, #19937 includes LibretroAdmin's rule that an archive of several members, usually an arcade set, is read whole before any lookup, so that each database can take either the archive's own CRC or its first member's. That suits arcade sets. Disc zips, console sets and merged arcade sets also hold several files, though, and they were read whole even when no database could hold them. On a NAS, 10 PS1 disc zips took 82 s to scan; with this PR they take 78 ms.
This PR keeps the rule wherever it can match and narrows it with an exact-size gate. An archive of several members is read whole only when a database whose cores open its type holds a record named for that archive type with the file's exact size, or holds no sizes. Otherwise each member is looked up by the CRC its directory already stores.
Commits 1-3 prepare: one directory walk per archive, a TorrentZip hint from the zip reader, and archive record sizes beside the crc index. Commit 4 makes the change; 5 and 6 are tests.
What changes
Numbers
A Linux desktop scanning an NFS share on a NAS, stock databases. Master here is 40a5f4e (#19937 as merged), compared with this PR applied to it. One run per build and set, master first, page cache dropped before each. All runs were under load (1-minute load 6.7-24.2 on 32 cores), and the NAS's own cache can favour the second run, so the wall times are rough; the whole-read counts and playlists are the firm result.
What the size gate can lose
A record keyed on an archive's own CRC is no longer matched when its
rom_nameis not of the archive's type or its size is not the file's. An audit of the 146 stock.rdbfiles found none:rom_name, and all 106,134 archive-named records in the 146 databases have a size, so the no-sizes fallback never applies to stock data.Where playlists can differ
x.zip#dir/rom; now it isx.zipwhen a core of the ROM's database opens zip.x.zip, where master's expansion wrotex.zip#member.None of these came up in the sets above.
Notes for review
rearm,ahead_lone,ahead_double,ahead_streak): streaks now count whole matches, not member misses; labels and fixture CRCs follow.reader_fail_arcade's zip is now a TorrentZip, and the arcade recordarc_cart_firstshares gains its zip'srom_name. Their assertions are unchanged.Testing
database_scan_test(328 checks; 214 on master) anddatabase_index_test(38; 24 on master) pass, also under ASan, TSan,HAVE_THREADS=0(312 checks), readers off (313), and in 5 runs beside 12 busy loops. New cases cover the size gate on zip, 7z and RAR 4, the every-member lookup, folder-first archives and TorrentZip read-ahead.archive_zip_testand the libretro-db parser test pass.archive_zip_testpasses.C89_BUILD=1compiles the 7 touched objects without warnings; the default and--disable-threadsbuilds link.A later PR will match arcade sets by a fingerprint that does not depend on how they were packed; it needs changes to libretro-database.