feat: attach tool definitions to generations - #26
Open
wierdbytes wants to merge 2 commits into
Open
wierdbytes wants to merge 2 commits into
wierdbytes wants to merge 2 commits into
Conversation
Closes langfuse#23. Attach the definitions of the tools the model is offered — `getAllTools()` narrowed to `getActiveTools()`, which is exactly the set pi puts in the request — as `tools` on the first message of every generation's input. That is the ChatML shape the Langfuse UI builds its Tools section from.
Unit tests pin the two helpers: attaching to a single message or the head of an array without mutating the caller's data, and narrowing the registry to the active names while dropping non-ChatML fields. The integration tests check the exported input against pi's default tool set and, with a fixture that registers two tools and deactivates one, that extension tools arrive with their schema and deactivated ones stay out.
|
|
wierdbytes
marked this pull request as ready for review
September 3, 2026 19:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #23.
Change
The extension recorded which tools the model called —
tool_callson the generation output,Tool: <name>spans — but not which tools it was offered, so a trace could not tell "not offered" from "not chosen", and the Langfuse playground had nothing to replay a generation with.Each generation's input now carries the definitions of the tools in the request as
toolson its first message. The set ispi.getAllTools()narrowed topi.getActiveTools(): the first covers built-in, extension and MCP tools, the second names the subset on the agent, which is exactly what pi puts in the request. Both accessors are optional-chained and the helper never throws, so a pi without them degrades to no tools.No conversion is needed.
getAllTools()already returns{ name, description, parameters }, and that is the schema the Langfuse UI reads frommessage.toolson the ChatML input. The UI dedupes by name across messages, so one copy on the first message is enough; repeating it on every message would only bloat the export.attachToolDefinitionshandles both a single message and a message array without mutating the caller's data, so it composes with an input that another change turns into an array.Before:
After:
Tests
pnpm typecheckis clean.pnpm testgives 71 of 71 passed, up from 62 onmain.promptGuidelinesandsourceInfo, and returning nothing on a pi without the accessors or with throwing ones.["read", "bash", "edit", "write"]with their JSON schemas; a run with a fixture extension that registers two tools and deactivates one viasetActiveToolsmust carry the active one with its exact schema and not the deactivated one. Onmainboth fail withgeneration 0 must carry tools on its first message.