feat(verify): score software and websites with batched Wikidata liveness - #103
Merged
Merged
Conversation
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Software and website records lacked domain rules and could not earn green. Add completeness fields and plausibility checks using the actual imports, plus a
check-wikidatacommand that batches 50 QIDs per request into the existing promotion URL cache.The command uses an identified User-Agent, maxlag=5, sequential requests spaced by one second, incremental cache saves, freshness filtering before the QID cap (older generic HTTP verdicts are rechecked), and a default maximum of 20,000 IDs. Generic URL checks also route Wikidata citations through the entity batcher so HTTP redirects cannot overwrite dead entity verdicts. Existing entities are alive; missing and redirect entities are dead; transport, maxlag and malformed-response failures remain uncached for retry. The network workflow restores the URL cache between runs so later runs advance through the frontier.
Missing optional fields are neutral in consistency and reflected in completeness. Malformed calendar dates hard-fail; early/future dates, malformed lists, missing QID citations and malformed homepage URLs are soft flags. Website imports use
launch_date; 290 pre-Web dates include publisher founding dates (e.g. AccuWeather, 1962), so these ambiguities do not force red. Software dates are checked against 1950 and the current year; website launch/release against 1989, founded dates against 1800.Entity liveness does not confirm record claims. Label/date/homepage cross-reference is deferred because multilingual aliases and imported inception/release semantics need separate rules; the existing contradiction veto remains intact.
Field coverage (filled means neither null nor an empty string/list/object; false booleans count as populated):
Local status results (2026-09-27):
Flag counts: software has zero flags; website has 290
launch_date_plausibleand 1homepage_http_url; neither category has hard failures or missing-QID flags. Existing T2 weighting remains unchanged: sparse Wikidata-only records remain yellow, and green still needs a live cited T1/T2 source for promotion. No records were promoted by this change.Validation: 481 tests passed in the complete pytest suite with an external data checkout and explicit basetemp; 19 focused cache/coverage tests passed after the final routing update; mypy passed over all 107 app files; ruff passed over app/tests. Mock HTTP tests cover 50-ID batches, deduplication, missing/redirect entities, rate spacing, maxlag failures, shared cache persistence and promotion decisions.
Refs #98
Refs GetTechAPI/TechAPI#297
API contract: wbgetentities documentation, MediaWiki API etiquette.