Repository navigation
feat(scoring): 六维评分行为锚点标尺(中/英同标尺,试点) - #115
Merged
Merged
Conversation
scores 块六维此前只有一句名称提示,LLM 打分跨样本跨语种不可比——可解释 打分的前提是同一个 85 分在任何一天意味着同一件事。scoring_caliber 新增 ANCHOR_BANDS 与中英两份锚点表(每维四档可判定行为描述),中/英 prompt 从 它渲染锚点段落,论文版按裁决不并入首期;漂移测试扩展为锚点逐字一致、 档位逐维提取比对、论文版排除断言。附 scripts/scoring_anchor_ab.py 对照 跑分工具:剥离锚点段可逐字节还原 main 版 prompt(独立复核已验证),全量 启用前先跑对照确认分布迁移幅度。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
六维评分字段(quality/hot/freshness/creator/viral/risk)此前在 prompt 里只有一句名称提示,LLM 打分跨样本、跨语种不可比;TopicEye 的卖点是可解释打分,可解释的前提是「同一个 85 分在任何一天意味着同一件事」。按 Easel 调研 C3 的圆桌裁决(4/4 trial,严格排在 #110 之后),以试点口径落地:
scoring_caliber新增六维 × 四档行为锚点(0-30/40-60/70-85/90-100,每档一句可判定的行为描述),中/英 prompt 从同一份常量渲染锚点段落,并附对照跑分工具供全量前确认分布迁移幅度。论文版按裁决不并入首期(它是独立的四维精读体系)。Fixes #114
Area
Verification
pytest tests/test_scoring_caliber.py tests/test_scoring_anchor_ab.py→ 17 passed(含锚点逐字一致、逐维档位提取、论文版排除、剥离还原、分布汇总与门槛跨越)tests/test_scoring_engine.py tests/test_today_picks.py tests/test_paper_analysis.py tests/test_scoring_flow.py tests/test_analyses_api.py同批全绿make test-backend全量 → exit 0ruff check+ruff format --check+check_layering.py→ 全过Verifier verdict
pass——独立复核(非实现过程)六条全 confirmed,关键两条以实跑验证:① 对照基线还原做到字节级一致(剥离锚点段的中/英 prompt 与 main 版逐字符 cmp 相同,A/B 的「旧臂」就是历史 prompt);② 漂移锁变异模拟全红(手改任一档描述/档位边界/论文版误植/表缺档,对应测试全部判红)。两个措辞级 note:en 第四档漏译「深度复盘」已在提交前修复;risk=70 单点缝隙判可忽略。
Checklist
feat(scoring): …)issue-114-scoring-behavior-anchors,正文写Fixes #114Notes for reviewer
cd backend && uv run python scripts/scoring_anchor_ab.py --limit 30,对照报告贴回 六维评分行为锚点标尺:中/英 prompt 从一句名称提示升级为四档可判定描述(试点) #114;线上在 admin/scoring-dashboard 按analyzed_at分界观察。样本不足时放宽--hours。DIMENSION_ANCHORS_*即可,prompt 与测试自动跟随(漂移锁防的是绕过 caliber 手改 prompt)。