Skip to content

Commit b4ea1bd

Browse files
committed
fix(data): merge 28 confirmed PDA near-duplicate records
Scanned the pda category (140 records, reviewed every match by hand) for the three duplicate patterns the tablet/watch (#324) and smartphone (#329/#330) passes turned up. Merged 28 redundant files into 21 keepers (140 -> 112), keeping the better-sourced side field by field and unioning source_urls. Pattern 1 - identical scrape imported twice under different row-id slugs (100% identical specs, differ only by source_row): - motorola/2002/accompli-388: kept ...-scrapegsma-344, deleted ...-1227. - mitac/2005/mio-a701: kept ...-scrapegsma-278, deleted ...-1161. Pattern 2 - same device under a brand-prefixed slug and a bare slug in sibling dirs. Same convention as the #330 GSMArena/phonedb merge: kept the bare GSMArena/Kaggle record (real specs + raw_gsmarena, sometimes Wikipedia), upgraded name to the brand-inclusive form, unioned source_urls to retain the phonedb reference, kept the keeper's soc/release_date, left verified as-is, deleted the phonedb twin: - treo-680 <- palm-treo-680-palm-lowrider ("Palm Treo 680") - treo-750v <- palm-treo-750v-htc-cheetah ("Palm Treo 750v") - treo-750 <- palm-treo-750-htc-cheetah ("Palm Treo 750") - treo-500v <- palm-treo-500v-palm-otto ("Palm Treo 500v") Treo 750 vs 750v and Treo 500 vs 500v were kept as distinct SKUs (only the matching-model twins were merged). palm-treo-500-palm-otto (2008 Treo 500) is a different device and was left alone. Pattern 3 - cross-source duplicate: the same PDA imported as a clean short-slug record and as a cigarplug/scrape-gsma row in the same base dir. Kept the clean record (correct soc FK, canonical slug), deleted the scrape twin, and repaired the keeper's ram_gb from the twin's cited memory_internal (the clean side carried the fabricated ram_gb=1 placeholder; the scrape side had the real value). The github.com/cigarplug/scrape-gsma URL was unioned so the repaired RAM is sourced: - 16 HP iPAQ pairs: h6310/h6315/h6320/h6325/hw6510/hw6515 -> 0.0625 GB (64 MB); hw6910/hw6915/rw6815/rw6818/rw6828/514 -> 0.0625; 610c/910c/data-messenger/ voice-messenger -> 0.125 GB (128 MB). - i-mate pda2 (two identical scrape twins, -1004 and -121) -> 0.125 GB. - i-mate pda2k -> 0.125 GB; i-mate pdal -> 0.062 GB. - palm treo-pro (3-way: bare + scrape-gsma-7066 + phonedb palm-treo-pro): kept the bare Wikipedia-cited record, ram 1 -> 0.125 GB (128 MB), name -> "Palm Treo Pro", unioned all four source_urls. Left unmerged and documented (ambiguous, per "leave ambiguous unmerged"): the GSM webOS palm-pre-scrapegsma-7065 and palm-pre-plus-scrapegsma-7063 overlap a region-split family (palm-pre[-plus]-palm-castle-c CDMA and palm-pre[-plus]-gsm-*-palm-castle-g EU/NA) but do not map 1:1, so no merge. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm, pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone. No value was invented: every ram_gb repair took the real figure already present in the deleted twin's cited memory_internal. verified flags unchanged. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296
1 parent ee4f2fe commit b4ea1bd

52 files changed

Lines changed: 74 additions & 1247 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎data/pda/hp/2004/ipaq-h6310/hp-ipaq-h6310-scrapegsma-3824.json‎

Lines changed: 0 additions & 45 deletions
This file was deleted.

‎data/pda/hp/2004/ipaq-h6310/ipaq-h6310.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,14 @@
44
"brand": "hp",
55
"soc": "ti-omap-1510",
66
"release_date": "2004-01-01",
7-
"ram_gb": 1,
7+
"ram_gb": 0.0625,
88
"battery_mah": 1800,
99
"weight_g": 190.0,
1010
"os": "Windows Mobile",
1111
"verified": false,
1212
"source_urls": [
13-
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices"
13+
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices",
14+
"https://github.com/cigarplug/scrape-gsma"
1415
],
1516
"display": {
1617
"size_inch": 3.5,

‎data/pda/hp/2004/ipaq-h6315/hp-ipaq-h6315-scrapegsma-3823.json‎

Lines changed: 0 additions & 45 deletions
This file was deleted.

‎data/pda/hp/2004/ipaq-h6315/ipaq-h6315.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,14 @@
44
"brand": "hp",
55
"soc": "ti-omap-1510",
66
"release_date": "2004-01-01",
7-
"ram_gb": 1,
7+
"ram_gb": 0.0625,
88
"battery_mah": 1800,
99
"weight_g": 190.0,
1010
"os": "Windows Mobile",
1111
"verified": false,
1212
"source_urls": [
13-
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices"
13+
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices",
14+
"https://github.com/cigarplug/scrape-gsma"
1415
],
1516
"display": {
1617
"size_inch": 3.5,

‎data/pda/hp/2004/ipaq-h6320/hp-ipaq-h6320-scrapegsma-3822.json‎

Lines changed: 0 additions & 45 deletions
This file was deleted.

‎data/pda/hp/2004/ipaq-h6320/ipaq-h6320.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,14 @@
44
"brand": "hp",
55
"soc": "ti-omap-1510",
66
"release_date": "2004-01-01",
7-
"ram_gb": 1,
7+
"ram_gb": 0.0625,
88
"battery_mah": 1800,
99
"weight_g": 190.0,
1010
"os": "Windows Mobile",
1111
"verified": false,
1212
"source_urls": [
13-
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices"
13+
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices",
14+
"https://github.com/cigarplug/scrape-gsma"
1415
],
1516
"display": {
1617
"size_inch": 3.5,

‎data/pda/hp/2004/ipaq-h6325/hp-ipaq-h6325-scrapegsma-3821.json‎

Lines changed: 0 additions & 45 deletions
This file was deleted.

‎data/pda/hp/2004/ipaq-h6325/ipaq-h6325.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,14 @@
44
"brand": "hp",
55
"soc": "ti-omap-1510",
66
"release_date": "2004-01-01",
7-
"ram_gb": 1,
7+
"ram_gb": 0.0625,
88
"battery_mah": 1800,
99
"weight_g": 190.0,
1010
"os": "Windows Mobile",
1111
"verified": false,
1212
"source_urls": [
13-
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices"
13+
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices",
14+
"https://github.com/cigarplug/scrape-gsma"
1415
],
1516
"display": {
1617
"size_inch": 3.5,

‎data/pda/hp/2005/ipaq-hw6510/hp-ipaq-hw6510-scrapegsma-3820.json‎

Lines changed: 0 additions & 45 deletions
This file was deleted.

‎data/pda/hp/2005/ipaq-hw6510/ipaq-hw6510.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,14 @@
44
"brand": "hp",
55
"soc": "intel-pxa270-312-mhz",
66
"release_date": "2005-01-01",
7-
"ram_gb": 1,
7+
"ram_gb": 0.0625,
88
"battery_mah": 1200,
99
"weight_g": 165.0,
1010
"os": "Windows Mobile",
1111
"verified": false,
1212
"source_urls": [
13-
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices"
13+
"https://www.kaggle.com/datasets/msainani/gsmarena-mobile-devices",
14+
"https://github.com/cigarplug/scrape-gsma"
1415
],
1516
"display": {
1617
"size_inch": 3.0,

0 commit comments

Comments
 (0)