Repository navigation
fix(data): remove 18 fabricated placeholder values from PDA records - #344
Merged
Merged
Conversation
Post-dedup scan of all 112 pda records for the fabricated-default fingerprint
(one fake-precise value repeated across many unrelated devices), independent of
whether a record had a duplicate twin.
Clusters found:
- ram_gb=1 on 83 of 112 records. Definitively fabricated: no PDA from 2000-2011
had 1 GB of RAM (Dell Axim X5 = 32/64 MB, Palm IIIc = 8 MB, etc.). This is the
Kaggle/phonedb bulk-import placeholder.
- battery_mah=1500 on 33 records (mostly the classic Palm line), and
weight_g=190 on 9 - see "left as-is" below.
Fixed 18 ram_gb=1 values to the real, sourced figure (ram_gb is a required field,
so these are repaired to a cited value, not nulled). No value invented - each
comes from an authoritative source added to source_urls:
- 13 Dell Axim (Wikipedia Dell Axim specification table): X5/X3/X30 low-end
("Basic") = 32 MB = 0.03125; every other Axim model (X3/X3i Advanced, X30
Advanced, and the whole X50/X51 line) = 64 MB = 0.0625.
- 5 Fujitsu-Siemens Pocket Loox N520/N560 = 64 MB = 0.0625 (Fujitsu N520 user
manual + clubic N560 launch spec).
Left as-is, documented rather than guessed or nulled:
- Classic Palm line (Palm IIIc/IIIxe/VIIx = 8 MB, m100 = 2 MB, m105/m125/m130/
m500/m505 = 8 MB, m515 = 16 MB, Tungsten/Zire/Treo/Centro, etc.): their real
RAM is below the schema's ram_gb>=0.016 (16 MB) floor, so it is not
representable - almost certainly why they were stamped with the ram=1
placeholder in the first place. Can't set the real value (fails validate),
can't null a required field. Reported on #296.
- The remaining ram_gb=1 records (i-mate PDA phones, HP iPAQ Glisten, Mitac Mio
728, later Palm Treo/webOS) need individual per-device sourcing; left documented.
- battery_mah=1500: the classic Palms ran on AAA/AA cells (m100/m105/m125) or
small internal Li-ions - clearly not a uniform 1500 mAh - but there is no clean
per-device mAh to cite without inventing, and the field is required (can't null).
Reported on #296.
- ppi=114 (26 records) and the 3.5"/240x320 combo are NOT fabricated: 240x320
QVGA on a 3.5" panel really is ~114 ppi, legitimately shared by that whole
common screen class. weight_g=190 (9) is real chassis-sharing (iPAQ h63xx
family, i-mate HTC rebrands). Left untouched.
Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict`
(0 hard anomalies).
Refs #296
Re-dumped the pdas collection from the corrected data tree (fresh scratch DB, TECHAPI_DATA_DIR on this worktree). Only the 18 Dell Axim / Fujitsu-Siemens detail pages whose ram_gb was repaired change; no other collection touched.
github-actions
Bot
added
bug
Member
🔎 Data verification — Tier 0 (offline existence/trust)Scored by 12/12 categories ? 190,468 records ? diff base develop@e174584 Changed records in this PR18 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 18
"Yellow" : 0
"Red" : 0
Full-dataset baseline190468 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 33024
"Yellow" : 156391
"Red" : 1053
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
Member
TechEngine change review: PASS12/12 categories ? 190,468 records ? diff base develop@e174584
Changed data
Changed record examplespda modified
Heuristic review
|
Member
TechEngine validation stats: PASS12/12 categories ? 190,468 records ? diff base develop@e174584
Data summary
Warning Tracked verified coverage is below 50% for monitor 0.0% (0/231), software 0.8% (358/42493), laptop 5.3% (105/1966), tablet 8.6% (285/3305), soc 13.4% (284/2117), all 16.7% (31901/190468), smartphone 18.6% (17235/92824), website 19.0% (7635/40084), and 1 more. Validation notes
Key output:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to the PDA dedup pass (#335). That pass only caught fabricated import
placeholders inside duplicate pairs (where comparing the two sides revealed the
fake one). This scans all 112 post-dedup pda records for the same
fabricated-default fingerprint — a single fake-precise value repeated across many
unrelated devices — regardless of whether a record had a twin.
Fixed 18 records; documented the rest that can't be repaired without inventing.
Both gates pass:
python -m app.validate→ passedintegrity_check.py <data> --strict→ 0 hard anomaliesClusters found
ram_gb1battery_mah1500display.ppi114display3.5" / 240x320weight_g190Spot-checked against sources: Dell Axim X5 = 32/64 MB (Wikipedia, PCMag,
DigitalTrends), Palm IIIc = 8 MB (Wikipedia, TidBITS, Computing History) — all
flatly contradict the stamped
ram_gb=1.Fixed (18 records, all cited —
ram_gbis required so repaired, not nulled)("Basic") = 32 MB (
0.03125); X3/X3i Advanced, X30 Advanced, and the entireX50/X51 line = 64 MB (
0.0625).0.0625) — Fujitsu N520user manual + clubic N560 launch spec.
No value invented — every figure comes from an authoritative source added to
source_urls.Left as-is, documented (not guessed, not nulled)
m500/m505 = 8 MB, m515 = 16 MB, Tungsten/Zire/Treo/Centro …): their real RAM is
below the schema's
ram_gb >= 0.016(16 MB) floor, so it cannot berepresented — almost certainly why they were stamped with
ram=1on import.Can't set the true value (fails validate), can't null a required field.
ram_gb=1records (i-mate PDA phones, HP iPAQ Glisten, Mitac Mio 728,later Palm Treo/webOS) need individual per-device sourcing; left documented
rather than risk inventing.
battery_mah=1500: the classic Palms ran on AAA/AA cells (m100/m105/m125 —no mAh rating) or small internal Li-ions; there is no clean per-device mAh to
cite without inventing, and the field is required (can't null).
ppi=114, the 3.5"/240x320 combo, andweight_g=190are genuinely sharedreal values, not fabricated defaults — left untouched.
Ownership / dump
data/pda/only. Dump refreshed as the last commit (fresh scratch DB,TECHAPI_DATA_DIRon this worktree): only the 18 repaired detail pages change,no other collection touched.
Closes #296 — this removes a sourced slice of the pda fabricated-placeholder values;
the required-field-but-unrepresentable/unsourceable remainder stays open and is
documented above.