Skip to content

Fix verified cross-reference data contradictions - #253

Merged
Seungpyo1007 merged 12 commits into
developfrom
fix/crossref-contradictions
Sep 25, 2026
Merged

Seungpyo1007 merged 12 commits into
developfrom
fix/crossref-contradictions

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Cross-reference contradiction review

Reviewed the last append-only entry per record in the crossref, smartphone, tablet, watch, and PDA caches against the matched Wikipedia or Wikidata page. Changed a value only where the page unambiguously describes that same device; preserved all verified flags. The Pixel 2 XL record moved from the 2007 directory to the 2017 directory because the repository validator requires the path year to match release_date.

Cache Corrected (a) Wrong match or variant (b) Unclear or source conflict (c)
crossref 0 8 0
smartphone 5 28 35
tablet 7 14 16
watch 4 0 11
PDA 1 0 4
Total 17 50 66

Unchanged (b) verdicts include Wikidata homonyms (for example MX as Tokyo MX), model collisions (Sendo M550 versus Eten M550), and family pages or memory variants. Unchanged (c) verdicts include SoC aliases, source disagreements for older Palm display sizes, and Wikipedia table parsing that treats storage as RAM (for example Samsung Gear's 4GB/512 MB RAM).

Record Field Old New Source
google-pixel-2-xl-4gb-64gb-4g release_date 2007-10-10 2017-10-19 Pixel phones
chat-3g battery_mah 1150 1500 INQ Chat 3G
inq-chat-3g-aitoolbuzz-4199-1gb battery_mah 1150 1500 INQ Chat 3G
butterfly soc qualcomm-apq8064-snapdragon-600 qualcomm-apq8064-snapdragon-s4-pro HTC Butterfly
iqoo-vivo-iqoo-3-8gb-128gb-4g-lte soc snapdragon-865-plus snapdragon-865 iQOO 3
amazon-kindle-fire-aitoolbuzz-689-1gb ram_gb 1 0.5 Amazon Fire tablet, 1st generation
amazon-kindle-fire ram_gb 1 0.5 Amazon Fire tablet, 1st generation
kindle-fire ram_gb; variant.memory.ram_gb 1 0.5 Amazon Fire tablet, 1st generation
lg-optimus-pad-v900-aitoolbuzz-5290-32gb-32gb-3g ram_gb; variant.memory.ram_gb 32 1 LG Optimus Pad
lg-optimus-pad-v900-allphones-4324-32gb-32gb-3g ram_gb; variant.memory.ram_gb 32 1 LG Optimus Pad
htc-jetstream-1gb-1gb-4g-lte display.resolution 800x1280 pixels 768x1280 pixels HTC Jetstream
htc-jetstream-aitoolbuzz-3272-1gb-32gb-4g-lte display.resolution 800x1280 pixels 768x1280 pixels HTC Jetstream
gear-s ram_gb; variant.memory.ram_gb 1 0.5 Samsung Gear, Gear S row
gear-s2 ram_gb; variant.memory.ram_gb 1 0.5 Samsung Gear, Gear S2 row
lg-watch-sport-aitoolbuzz-5425-0-75gb-4gb-4g-lte weight_g 150 86 LG Watch Sport
watch-sport weight_g 150 86 LG Watch Sport
palm-pre-2 battery_mah 1500 1150 Palm Pre 2

Validation: PYTHONIOENCODING=utf-8 python -m app.validate passed. The dump was not run.

Closes #1

@Seungpyo1007 Seungpyo1007 added enhancement New feature or request data Dataset changes labels Sep 25, 2026
@github-actions github-actions Bot added the bug Something isn't working label Sep 25, 2026
@Seungpyo1007 Seungpyo1007 removed the bug Something isn't working label Sep 25, 2026
@TechEngineBot

TechEngineBot commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate and authoritative for merge.

Changed records in this PR

13 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 12
    "Yellow" : 1
    "Red" : 0
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
smartphone 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 3 3 0 0 100.0%
tablet 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨🟨 5 4 1 0 80.0%
watch 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 4 4 0 0 100.0%
pda 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 1 1 0 0 100.0%
All 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨 13 12 1 0 92.3%

Full-dataset baseline

105773 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 23551
    "Yellow" : 81126
    "Red" : 1096
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
brand 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 207 122 85 0 58.9%
soc 🟩🟨🟨🟨🟨🟨🟥🟥🟥🟥🟥🟥 2077 231 824 1022 11.1%
smartphone 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 93396 17912 75416 68 19.2%
tablet 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 3455 288 3166 1 8.3%
watch 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 433 43 386 4 9.9%
pda 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 140 83 57 0 59.3%
gpu 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨 2030 1211 819 0 59.7%
cpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟨🟥 4035 3661 373 1 90.7%
All 🟩🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟥 105773 23551 81126 1096 22.3%

Hard violations (forced red):

Count Check
1 !boost_ge_base

green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

Reverted two of the 17 corrections (INQ Chat 3G battery 1150→1500, HTC Jetstream resolution 1280x800→1280x768). In both cases the Wikipedia infobox disagrees with the record's original source (GSMArena-derived data: 1150 mAh, 1280x800), and HTC's own spec for the Jetstream is WXGA 1280x800. Since the sources conflict, the original values stay. The other 15 corrections are unambiguous (e.g. storage recorded as RAM, a 2007 release year on a 2017 phone) and remain.

@github-actions github-actions Bot added the bug Something isn't working label Sep 25, 2026
@Seungpyo1007
Seungpyo1007 merged commit 34271a0 into develop Sep 25, 2026
5 checks passed
@Seungpyo1007
Seungpyo1007 deleted the fix/crossref-contradictions branch September 25, 2026 04:41
@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine change review: PASS

Check Result
python -m app.validate PASS
New hard integrity anomalies vs PR base PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 0 0 0 0 0
smartphone 1 2 1 0 1 1
tablet 0 5 0 0 0 0
watch 0 4 0 0 0 0
pda 0 1 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0

Changed record examples

smartphone added

  • smartphone/google/2017/pixel-2-xl/google-pixel-2-xl-4gb-64gb-4g.json - Google Pixel 2 XL

smartphone modified

  • smartphone/htc/2012/butterfly.json - Butterfly
  • smartphone/iqoo/2020/vivo-iqoo-3/iqoo-vivo-iqoo-3-8gb-128gb-4g-lte.json - Vivo iQOO 3

smartphone deleted

  • smartphone/google/2007/pixel-2-xl/google-pixel-2-xl-4gb-64gb-4g.json - Google Pixel 2 XL

tablet modified

  • tablet/amazon/2011/kindle-fire/amazon-kindle-fire-aitoolbuzz-689-1gb.json - Kindle Fire
  • tablet/amazon/2011/kindle-fire/amazon-kindle-fire.json - Amazon Kindle Fire
  • tablet/amazon/2011/kindle-fire/kindle-fire.json - Kindle Fire
  • tablet/lg/2011/optimus-pad-v900/lg-optimus-pad-v900-aitoolbuzz-5290-32gb-32gb-3g.json - LG Optimus Pad V900
  • tablet/lg/2011/optimus-pad-v900/lg-optimus-pad-v900-allphones-4324-32gb-32gb-3g.json - LG Optimus Pad V900

watch modified

  • watch/lg/2017/watch-sport/lg-watch-sport-aitoolbuzz-5425-0-75gb-4gb-4g-lte.json - LG Watch Sport
  • watch/lg/2017/watch-sport/watch-sport.json - Watch Sport
  • watch/samsung/2014/gear-s/gear-s.json - Gear S
  • watch/samsung/2015/gear-s2/gear-s2.json - Gear S2

pda modified

  • pda/palm/2011/palm-pre-2/palm-pre-2.json - Palm Pre 2

Heuristic review

  • Added records by manufacturer/brand: google: 1
  • Added records by source class: kaggle: 1
  • Heuristic warnings: none found.

@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 207 122 85 0 207 58.9%
soc 2077 193 1884 0 2077 9.3%
smartphone 93396 17501 75895 0 93396 18.7%
tablet 3455 273 3182 0 3455 7.9%
watch 433 39 394 0 433 9.0%
pda 140 74 66 0 140 52.9%
gpu 2030 1028 1002 0 2030 50.6%
cpu 4035 2869 1166 0 4035 71.1%
all 105773 22099 83674 0 105773 20.9%

Warning

Tracked verified coverage is below 50% for tablet 7.9% (273/3455), watch 9.0% (39/433), soc 9.3% (193/2077), smartphone 18.7% (17501/93396), all 20.9% (22099/105773).
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • The integrity gate blocks only hard anomalies introduced by this PR relative to its merge base.
  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py (PR head compared with PR base)
loaded CPU=4035 GPU=2030
Integrity section Flagged lines
structural 18
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 155
GPU cross-source ratio outliers + sanity 18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants