Skip to content

fix(data): merge 27 duplicate SoCs, add 20 Commons images and 8 tablet Wikipedia URLs - #251

Merged
Seungpyo1007 merged 11 commits into
developfrom
data/0925-soc-dedup-images-tablets
Sep 25, 2026
Merged

Seungpyo1007 merged 11 commits into
developfrom
data/0925-soc-dedup-images-tablets

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

  • SoC dedup: 25 groups / 27 records that were the same chip and the same clock under different spellings (e.g. hisilicon-kirin-910t / hisilicon-kirin910t, marvell-pxa270-520mhz / -520-mhz / 520mhz-…) merged into the most-referenced record. 83 device soc references repointed; two PhoneDB source URLs carried over. Same-chip, different-clock records were left separate on purpose. The 27 stale dump pages were removed.
  • Commons images, pass 2: 20 more records got a real, freely-licensed image (records whose image_url key was absent entirely). Same gates as pass 1 (TechEngine fix(site): History timeline reads as growth, not repeated commit names #88).
  • Tablet Wikipedia URLs: 8 script-confirmed citations (150 checked).
  • Last commit refreshes the public dump; it also picks up the 146 promotions from data(verify): verified promotions (2026-09-25) #250.

Closes #1

Test plan

  • python -m app.validate passed

…P, Ativ Tab, Mi Pad 4, Galaxy Note 8.0, and Oppo Pad Neo

Refs #1
… the same chip and clock

Merges groups like intel-xscale-pxa272-520-mhz / intel-xscale-pxa272-520mhz and
hisilicon-kirin-910t / hisilicon-kirin910t into one record each. Different clock
SKUs of the same chip family are left separate. 83 device soc references were
repointed, and two PhoneDB source URLs from the removed records were kept.

Refs #1
@TechEngineBot

Copy link
Copy Markdown
Member

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate and authoritative for merge.

Changed records in this PR

259 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 186
    "Yellow" : 72
    "Red" : 1
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
brand 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 98 98 0 0 100.0%
soc 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨🟨 12 10 2 0 83.3%
smartphone 🟩🟩🟩🟩🟨🟨🟨🟨🟨🟨🟨🟥 103 36 66 1 35.0%
tablet 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨 35 34 1 0 97.1%
pda 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 3 0 3 0 0.0%
gpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 7 7 0 0 100.0%
cpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 1 1 0 0 100.0%
All 🟩🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟥 259 186 72 1 71.8%

Full-dataset baseline

105773 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 23540
    "Yellow" : 81137
    "Red" : 1096
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
brand 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 207 122 85 0 58.9%
soc 🟩🟨🟨🟨🟨🟨🟥🟥🟥🟥🟥🟥 2077 231 824 1022 11.1%
smartphone 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 93396 17909 75419 68 19.2%
tablet 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 3455 284 3170 1 8.2%
watch 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 433 39 390 4 9.0%
pda 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 140 83 57 0 59.3%
gpu 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 2030 1211 819 0 59.7%
cpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨🟥 4035 3661 373 1 90.7%
All 🟩🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟥 105773 23540 81137 1096 22.3%

Hard violations (forced red):

Count Check
1 !boost_ge_base

green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.

@Seungpyo1007
Seungpyo1007 merged commit 6c089d6 into develop Sep 25, 2026
5 checks passed
@Seungpyo1007
Seungpyo1007 deleted the data/0925-soc-dedup-images-tablets branch September 25, 2026 02:26
@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine change review: PASS

Check Result
python -m app.validate PASS
New hard integrity anomalies vs PR base PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 3 27 0 0 0
smartphone 0 98 0 0 0 0
tablet 0 10 0 0 0 0
watch 0 0 0 0 0 0
pda 0 3 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0

Changed record examples

soc modified

  • soc/hisilicon/2014/hisilicon-kirin-910t.json - HiSilicon Kirin 910T
  • soc/qualcomm/2019/qualcomm-snapdragon-sip-1.json - Qualcomm Snapdragon SiP 1
  • soc/samsung/2008/samsung-s3c2443.json - Samsung S3C2443

soc deleted

  • soc/arm/2008/arm-11-369-mhz.json - ARM 11 369 MHz
  • soc/arm/2011/1-0-ghz-arm-11.json - 1.0 GHz ARM 11
  • soc/arm/2011/800mhz.json - 800MHz
  • soc/arm/2016/1-ghz.json - 1 GHz
  • soc/arm/2018/1-2ghz.json - 1.2GHz
  • soc/hisilicon/2014/hisilicon-kirin910t.json - HiSilicon KIRIN910T
  • soc/intel/2004/intel-pxa-262-200-mhz.json - Intel PXA 262 200 MHz
  • soc/intel/2005/intel-xscale-312mhz.json - Intel XScale 312MHz
  • soc/intel/2006/intel-pxa-272-416-mhz.json - Intel PXA 272 416 MHz
  • soc/intel/2006/intel-pxa270-416-mhz.json - Intel PXA270 416 MHz
  • soc/intel/2006/intel-xscale-pxa-270-416-mhz.json - Intel XScale PXA 270 416 MHz
  • soc/intel/2006/intel-xscale-pxa272-520mhz.json - Intel XScale PXA272 520MHz
  • soc/intel/2007/intel-xscale-pxa270-520-mhz.json - Intel XScale PXA270 520 MHz
  • soc/intel/2007/intel-xscale-pxa270-520mhz.json - Intel XScale PXA270 520MHz
  • soc/intel/2009/520-mhz-intel-pxa270.json - 520 MHz Intel PXA270
  • ... 12 more

smartphone modified

  • smartphone/acer/2009/dx650.json - DX650
  • smartphone/amazon/2014/fire-phone-at-t/amazon-fire-phone-at-t-2gb-32gb-4g-lte.json - Amazon Fire Phone (AT&T)
  • smartphone/amazon/2014/fire-phone.json - Fire Phone
  • smartphone/amazon/2014/fire-phone/amazon-fire-phone-beridzeg45-4601-2gb.json - Amazon Fire Phone
  • smartphone/amazon/2014/fire-phone/amazon-fire-phone-scrapegsma-2300.json - Amazon Fire Phone
  • smartphone/asus/2006/p505.json - P505
  • smartphone/asus/2019/asus-zenfone-max-plus-m2-dual-sim-lte-a-br-32gb-zb634kl.json - Asus ZenFone Max Plus M2 Dual SIM LTE-A BR 32GB ZB634KL
  • smartphone/asus/2019/asus-zenfone-max-shot-dual-sim-lte-a-br-32gb-zb634kl.json - Asus ZenFone Max Shot Dual SIM LTE-A BR 32GB ZB634KL
  • smartphone/asus/2019/asus-zenfone-max-shot-dual-sim-lte-a-br-64gb-zb634kl.json - Asus ZenFone Max Shot Dual SIM LTE-A BR 64GB ZB634KL
  • smartphone/blackberry/2009/bold-9700.json - Bold 9700
  • smartphone/blackberry/2009/curve-8520.json - Curve 8520
  • smartphone/blackberry/2011/curve-9360.json - Curve 9360
  • smartphone/blackberry/2011/curve-9360/blackberry-curve-9360-aitoolbuzz-1312-1gb.json - Curve 9360
  • smartphone/blackberry/2018/storm3.json - Storm3
  • smartphone/blackberry/2018/storm3/blackberry-storm3-aitoolbuzz-1345-1gb.json - Storm3
  • ... 83 more

tablet modified

  • tablet/blu/2011/touch-book-7-0/blu-touch-book-7-0-beridzeg45-814-0-5gb.json - BLU Touch Book 7.0
  • tablet/blu/2011/touch-book-7-0/blu-touch-book-7-0.json - Blu Touch Book 7.0
  • tablet/lg/2014/g-pad-10-1/lg-g-pad-10-1-scrapegsma-5048.json - LG G Pad 10.1
  • tablet/oppo/2024/pad-neo/oppo-pad-neo-8gb-128gb-4g-lte.json - Oppo Pad Neo
  • tablet/oppo/2024/pad-neo/oppo-pad-neo-aitoolbuzz-7686-8gb-128gb-4g-lte.json - Oppo Pad Neo
  • tablet/oppo/2024/pad-neo/oppo-pad-neo-allphones-5881-8gb-128gb-4g-lte.json - Oppo Pad Neo
  • tablet/samsung/2012/ativ-tab-p8510/samsung-ativ-tab-p8510-scrapegsma-8180.json - Samsung Ativ Tab P8510
  • tablet/samsung/2013/galaxy-note-8-0/samsung-galaxy-note-8-0-beridzeg45-2132-2gb.json - Samsung Galaxy Note 8.0
  • tablet/sony/2012/tablet-p/sony-tablet-p-allphones-7467-1gb-4gb-2g.json - Sony Tablet P
  • tablet/xiaomi/2018/mi-pad-4/xiaomi-mi-pad-4-allphones-8253-4gb-32gb-4g-lte.json - Xiaomi Mi Pad 4

pda modified

  • pda/hp/2006/ipaq-rw6818/ipaq-rw6818.json - iPAQ rw6818
  • pda/hp/2006/ipaq-rw6828/ipaq-rw6828.json - iPAQ rw6828
  • pda/hp/2007/ipaq-610c/ipaq-610c.json - iPAQ 610c

Heuristic review

  • Heuristic warnings: none found.

@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 207 122 85 0 207 58.9%
soc 2077 193 1884 0 2077 9.3%
smartphone 93396 17501 75895 0 93396 18.7%
tablet 3455 273 3182 0 3455 7.9%
watch 433 39 394 0 433 9.0%
pda 140 74 66 0 140 52.9%
gpu 2030 1028 1002 0 2030 50.6%
cpu 4035 2869 1166 0 4035 71.1%
all 105773 22099 83674 0 105773 20.9%

Warning

Tracked verified coverage is below 50% for tablet 7.9% (273/3455), watch 9.0% (39/433), soc 9.3% (193/2077), smartphone 18.7% (17501/93396), all 20.9% (22099/105773).
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • The integrity gate blocks only hard anomalies introduced by this PR relative to its merge base.
  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py (PR head compared with PR base)
loaded CPU=4035 GPU=2030
Integrity section Flagged lines
structural 18
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 155
GPU cross-source ratio outliers + sanity 18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants