集成 Langfuse 遥测与离线评测 - #8
Open
hifizz wants to merge 1 commit into
Open
Conversation
调研结论见 docs/observability/01-调研报告.md（AI SDK v7 遥测重构、 Langfuse vs LangSmith 对比与选型），设计取舍见 02-设计方案.md。 遥测： - instrumentation.ts 启动注册 NodeTracerProvider + LangfuseSpanProcessor + AI SDK v7 全局遥测集成（@langfuse/vercel-ai-sdk） - chat route 以 chat-turn 根观测包装：sessionId=threadId、 普通/深度研究 tag、onEnd/onError/onAbort 收尾、after() 冲刷 - PDF 洞察与 RAG embedding 自动追踪并标 functionId - 未配置 LANGFUSE_* 时整体停用、零开销（与 R2/search 降级惯例一致） 评测： - assistant 消息 id 即该轮 traceId（服务端 generateMessageId 下发）， 👍/👎 经 /api/feedback 以幂等 score 回写对应 trace - pnpm eval 离线评测：Langfuse experiment run，规则断言 + LLM-as-a-judge，支持内置样例集或远端 dataset Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AQcZQKQtg6kbdRiFTEQJdG
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
集成 Langfuse 作为可观测与评测后端，实现对话链路的完整追踪、用户反馈打分、以及离线数据集评测能力。遥测为可选降级功能，未配置
LANGFUSE_*时零开销停用。关键变更
遥测基础设施
instrumentation.ts：Next.js 启动钩子，注册 OTel NodeTracerProvider 与 AI SDK v7 全局遥测集成lib/observability/register.ts：OTel provider 与 Langfuse 集成的注册逻辑（含重复注册防护）lib/observability/langfuse.ts：Langfuse 客户端与 span processor 的全局单例管理、配置检测、traceId 校验constants/observability.ts：trace 名、score 名、functionId 等共享命名的集中定义聊天链路集成
app/api/chat/route.ts：startActiveObservation("chat-turn")）与会话分组（sessionId = threadId）generateMessageId），实现消息与 trace 的一一对应onEnd/onError/onAbort回调收尾根观测after()冲刷 span 批次（serverless 部署保障）functionId（遥测分组）app/api/feedback/route.ts：点赞/点踩 → Langfuse score 回写，幂等 id 防重复计数app/page.tsx：FeedbackAdapter 实现，fire-and-forget POST/api/feedback离线评测
scripts/eval.ts：LANGFUSE_EVAL_DATASET环境变量）代码格式化
components/examples/base.tsx：统一改用 Prettier 推荐的行尾无分号风格（与项目 prettier 配置对齐）依赖与配置
package.json：新增pnpm eval脚本、Langfuse 相关依赖（@langfuse/vercel-ai-sdk、@langfuse/client、@opentelemetry/sdk-trace-node）.env.example：补充LANGFUSE_PUBLIC_KEY、LANGFUSE_SECRET_KEY、LANGFUSE_BASE_URL配置说明lib/ai/embeddings.ts、lib/attachments/insights.ts：为 embedding 与 PDF 摘要生成调用标记functionId文档
docs/observability/01-调研报告.md：AI SDK v7 遥测体系、Langfuse vs LangSmith 对比、选型依据docs/observability/02-设计方案.md：数据https://claude.ai/code/session_01AQcZQKQtg6kbdRiFTEQJdG