Skip to content

feat(verify): score software and websites with batched Wikidata liveness - #103

Merged
Seungpyo1007 merged 1 commit into
mainfrom
feat/software-website-scoring
Sep 27, 2026
Merged

Seungpyo1007 merged 1 commit into
mainfrom
feat/software-website-scoring

Conversation

@Seungpyo1007

Copy link
Copy Markdown
Member

Software and website records lacked domain rules and could not earn green. Add completeness fields and plausibility checks using the actual imports, plus a check-wikidata command that batches 50 QIDs per request into the existing promotion URL cache.

The command uses an identified User-Agent, maxlag=5, sequential requests spaced by one second, incremental cache saves, freshness filtering before the QID cap (older generic HTTP verdicts are rechecked), and a default maximum of 20,000 IDs. Generic URL checks also route Wikidata citations through the entity batcher so HTTP redirects cannot overwrite dead entity verdicts. Existing entities are alive; missing and redirect entities are dead; transport, maxlag and malformed-response failures remain uncached for retry. The network workflow restores the URL cache between runs so later runs advance through the frontier.

Missing optional fields are neutral in consistency and reflected in completeness. Malformed calendar dates hard-fail; early/future dates, malformed lists, missing QID citations and malformed homepage URLs are soft flags. Website imports use launch_date; 290 pre-Web dates include publisher founding dates (e.g. AccuWeather, 1962), so these ambiguities do not force red. Software dates are checked against 1950 and the current year; website launch/release against 1989, founded dates against 1800.

Entity liveness does not confirm record claims. Label/date/homepage cross-reference is deferred because multilingual aliases and imported inception/release semantics need separate rules; the existing contradiction veto remains intact.

Field coverage (filled means neither null nor an empty string/list/object; false booleans count as populated):

Category Field Present Filled Fill rate
software developers 10,014 10,014 23.57%
software genres 1,222 1,222 2.88%
software licenses 13,916 13,916 32.75%
software name 42,493 42,493 100.00%
software operating_systems 12,395 12,395 29.17%
software programming_languages 5,279 5,279 12.42%
software publishers 1,027 1,027 2.42%
software release_date 4,514 4,514 10.62%
software slug 42,493 42,493 100.00%
software source_urls 42,493 42,493 100.00%
software verified 42,493 42,493 100.00%
website homepage_url 34,409 34,409 85.84%
website languages 11,913 11,913 29.72%
website launch_date 13,833 13,833 34.51%
website name 40,084 40,084 100.00%
website owners 5,149 5,149 12.85%
website slug 40,084 40,084 100.00%
website source_urls 40,084 40,084 100.00%
website verified 40,084 40,084 100.00%

Local status results (2026-09-27):

Category Total Green Yellow Red
software 42,493 216 42,277 0
website 40,084 7,623 32,461 0

Flag counts: software has zero flags; website has 290 launch_date_plausible and 1 homepage_http_url; neither category has hard failures or missing-QID flags. Existing T2 weighting remains unchanged: sparse Wikidata-only records remain yellow, and green still needs a live cited T1/T2 source for promotion. No records were promoted by this change.

Validation: 481 tests passed in the complete pytest suite with an external data checkout and explicit basetemp; 19 focused cache/coverage tests passed after the final routing update; mypy passed over all 107 app files; ruff passed over app/tests. Mock HTTP tests cover 50-ID batches, deduplication, missing/redirect entities, rate spacing, maxlag failures, shared cache persistence and promotion decisions.

Refs #98
Refs GetTechAPI/TechAPI#297

API contract: wbgetentities documentation, MediaWiki API etiquette.

@Seungpyo1007 Seungpyo1007 added the enhancement New feature or request label Sep 27, 2026
@Seungpyo1007 Seungpyo1007 self-assigned this Sep 27, 2026
@Seungpyo1007
Seungpyo1007 merged commit d6bc8c6 into main Sep 27, 2026
1 check passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in TechAPI-Project Sep 27, 2026
@Seungpyo1007
Seungpyo1007 deleted the feat/software-website-scoring branch September 30, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant