Parse multi-row headers in the Wikipedia GPU ingest - #91
Closed
Seungpyo1007 wants to merge 4 commits into
Closed
Seungpyo1007 wants to merge 4 commits into
Seungpyo1007 wants to merge 4 commits into
Conversation
The GPU list pages put units in stacked header rows ("Memory / Size (MiB)",
"Core clock (MHz)") and leave body cells as bare numbers, so the old
single-row matcher dropped almost every row. Read the whole header block,
apply the header unit, map code names to microarchitectures, keep the full
bus string, file pre-Oct-2010 AMD cards under ATI, and skip IGP, dual-GPU
and idle-TDP columns. Abbreviated month names now parse too.
Wikipedia EPYC tables write "10 Oct, 2024", which fell through to the year-only fallback (2024-01-01).
RDNA and Polaris rows put game and boost clocks in one "Core / Clock" cell, and some list only the game clock. Leave base unknown when a cell stacks two clocks, and only copy base into boost for pre-2012 boards.
parse_cores_threads("16") returned (16, 16). The Atom and Xeon list pages
have a bare "Cores" column, so Hyper-Threaded parts (Tunnel Creek,
Centerton, Broadwell-EX...) came out with half their real thread count.
Return (16, None) so such rows stay drafts instead of guessing.
This was referenced Sep 25, 2026
This was referenced Sep 25, 2026
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Wikipedia GPU list pages use two-row
<th>headers with units in the header (Memory / Size (MiB),Core clock (MHz)) and bare numbers in the body, so the old single-row matcher produced almost no complete rows.wikitable.parse_table_header_block: joins the stacked header texts per column; drops all<sup>note markers.wikipedia_gpu: columns classified from the joined label, header unit applied; code name → microarchitecture (NV2x → Kelvin, RV8xx/Cypress → TeraScale 2, GK → Kepler, ...); full bus string kept (PCIe 2.1 x16,AGP 8x); AMD cards launched before 2010-10 filed underati; IGP/nForce, dual-GPU (2× 1024) and idle-TDP columns skipped; pre-boost boards store boost == base.normalize: abbreviated months (Mar 11, 2010,Sept. 2019); FirePro/NVS/RTX PRO → enterprise.Dry run against the current dataset gives 78 complete GPUs not yet in
data/gpu(spot-checked against the source rows).Tests:
pytestgreen apart fromtest_shipped_seed_data_is_valid, which reads the sibling TechAPI checkout that has unrelated in-progress edits. mypy and ruff clean.Refs GetTechAPI/TechAPI#1