Parse multi-row headers in the Wikipedia GPU ingest; stop guessing threads - #93
Merged
Merged
Conversation
The GPU list pages put units in stacked header rows ("Memory / Size (MiB)",
"Core clock (MHz)") and leave body cells as bare numbers, so the old
single-row matcher dropped almost every row. Read the whole header block,
apply the header unit, map code names to microarchitectures, keep the full
bus string, file pre-Oct-2010 AMD cards under ATI, and skip IGP, dual-GPU
and idle-TDP columns. Abbreviated month names now parse too.
Wikipedia EPYC tables write "10 Oct, 2024", which fell through to the year-only fallback (2024-01-01).
RDNA and Polaris rows put game and boost clocks in one "Core / Clock" cell, and some list only the game clock. Leave base unknown when a cell stacks two clocks, and only copy base into boost for pre-2012 boards.
parse_cores_threads("16") returned (16, 16). The Atom and Xeon list pages
have a bare "Cores" column, so Hyper-Threaded parts (Tunnel Creek,
Centerton, Broadwell-EX...) came out with half their real thread count.
Return (16, None) so such rows stay drafts instead of guessing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Wikipedia GPU list pages use two-row
<th>headers with units in the header (Memory / Size (MiB),Core clock (MHz)) and bare numbers in the body, so the old single-row matcher produced almost no complete rows.wikitable.parse_table_header_block: joins the stacked header texts per column; drops all<sup>note markers.wikipedia_gpu: columns classified from the joined label, header unit applied; code name → microarchitecture (NV2x → Kelvin, RV8xx/Cypress → TeraScale 2, GK → Kepler, ...); full bus string kept (PCIe 2.1 x16,AGP 8x); AMD cards launched before 2010-10 filed underati; IGP/nForce, dual-GPU (2× 1024) and idle-TDP columns skipped; pre-boost boards store boost == base.normalize: abbreviated months (Mar 11, 2010,Sept. 2019); FirePro/NVS/RTX PRO → enterprise.Dry run against the current dataset gives 78 complete GPUs not yet in
data/gpu(spot-checked against the source rows).Tests:
pytestgreen apart fromtest_shipped_seed_data_is_valid, which reads the sibling TechAPI checkout that has unrelated in-progress edits. mypy and ruff clean.Refs GetTechAPI/TechAPI#1
Reopened as a new PR: #91 was closed without a comment while its
testjob was failing on unrelated seed data (fixed in #92), and the branch has since been rebased ontomain. The follow-up commits since #91:parse_cores_threads("16")now returns threads = None instead of copying the core count. Hyper-Threaded parts (Atom, Xeon) otherwise came out with half their real thread count; Fix thread counts on 23 Hyper-Threaded Atoms from #254 TechAPI#264 corrected the records the old behavior produced.