Repository navigation
feat(web-search): add Yuanbao and Bocha AI output comparisons - #275
Open
lanyue-llk wants to merge 98 commits into
Open
lanyue-llk wants to merge 98 commits into
lanyue-llk wants to merge 98 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Tencent Yuanbao Standard/Lite and Bocha AI Search through the existing adapter registry. Evaluate result lists, additional source excerpts and generated answers in separate pools with their own primary metrics, participants and evidence.
Preserve native result slots and complete physical judge receipts. Collection retries default to zero. Missing Yuanbao success envelopes stop collection with unknown cost. Citation grading uses an HTML5 tree with original answer spans. It follows inline CSS validity, disclosure visibility, implied tag closures and image alternatives; deeply nested answers use an iterative traversal. Targets come from the parsed link. Explicit native citation IDs retain their own delivered source fields, including distinct excerpts for a repeated URL. Repeated copies of one ID retain the first-ranked record; URL-only links fall back to the first matching URL. Malformed capture arrays remain unavailable evidence and do not abort resumable collection or L2 admission. Reference capture retains body bindings in both modes, recovers legacy bodies from verified native responses and carries prior evidence into isolated paths when retrying.
Reference matching now verifies every candidate for each SimpleQA question from hash-bound full captures. Titles must match; when the original contains the reference answer, the mirror must contain it too. Otherwise both full articles must meet the 80% block-overlap threshold. Dataset builds and report checks replay the complete audit, and only verified mirrors enter source scores. The revised dataset retains all 831 questions; the factual quality comparison retains its 773 valid references. Search requests, timings and costs remain unchanged.
The maintained report explains the two benchmarks and three output levels before the rankings. Tables default to their primary metric and support sorting and scrolling. L1 factual results also show publisher-only scores and unverified mirror counts; the dataset page includes verification coverage for all 831 questions. Failed requests are visible beside affected methods at every output level. Release preparation avoids duplicate delivery validation and revision replay.
Scoring and dataset impact: these changes affect source-retrieval scores. The data lock selects SimpleQA revision
570dde53db18031d7cafe9806f6465e8de1ed0c1and Web Search revision7297b3a7a665cb7f79fd7986e78d19e72f962645. Corrections carry source hashes and paired regression controls. Software and certified-edition catalogs retain their released identities.Validation: the control-label corrections pass 384 targeted citation, answer-judge, presentation and public scoring tests, together with 23 documentation checks and repository lint. Browser comparisons cover visibility, disclosure state, duplicate attributes, target escaping and computed display applicability and initial select-control, input and textarea labels. Paired controls run through the production scoring boundary. All 980 saved answer packets retain identical grading inputs, and the maintained report passed full replay and hosted interaction checks. CI checks cover both Python shards, installation, packaging and rendered references. Report instances, full captures and measured evidence remain outside Git.