Skip to content

feat: attach tool definitions to generations - #26

Open
wierdbytes wants to merge 2 commits into
langfuse:mainfrom
wierdbytes:feat/tool-definitions
Open

wierdbytes wants to merge 2 commits into
langfuse:mainfrom
wierdbytes:feat/tool-definitions

Conversation

@wierdbytes

@wierdbytes wierdbytes commented Sep 3, 2026

Copy link
Copy Markdown

Closes #23.

Change

The extension recorded which tools the model called — tool_calls on the generation output, Tool: <name> spans — but not which tools it was offered, so a trace could not tell "not offered" from "not chosen", and the Langfuse playground had nothing to replay a generation with.

Each generation's input now carries the definitions of the tools in the request as tools on its first message. The set is pi.getAllTools() narrowed to pi.getActiveTools(): the first covers built-in, extension and MCP tools, the second names the subset on the agent, which is exactly what pi puts in the request. Both accessors are optional-chained and the helper never throws, so a pi without them degrades to no tools.

No conversion is needed. getAllTools() already returns { name, description, parameters }, and that is the schema the Langfuse UI reads from message.tools on the ChatML input. The UI dedupes by name across messages, so one copy on the first message is enough; repeating it on every message would only bloat the export.

attachToolDefinitions handles both a single message and a message array without mutating the caller's data, so it composes with an input that another change turns into an array.

Before:

screen-2026-09-03-21 10 55

After:

screen-2026-09-03-21 20 41

Tests

  • pnpm typecheck is clean.
  • pnpm test gives 71 of 71 passed, up from 62 on main.
  • Seven unit tests cover the two helpers: attaching to a single message or the head of an array, passing through inputs with nothing to attach to, narrowing the registry to the active names in registry order while dropping promptGuidelines and sourceInfo, and returning nothing on a pi without the accessors or with throwing ones.
  • Two integration runs read the tools off every generation's first message: the plain sandbox run must list ["read", "bash", "edit", "write"] with their JSON schemas; a run with a fixture extension that registers two tools and deactivates one via setActiveTools must carry the active one with its exact schema and not the deactivated one. On main both fail with generation 0 must carry tools on its first message.

Closes langfuse#23.

Attach the definitions of the tools the model is offered — `getAllTools()`
narrowed to `getActiveTools()`, which is exactly the set pi puts in the
request — as `tools` on the first message of every generation's input.
That is the ChatML shape the Langfuse UI builds its Tools section from.
Unit tests pin the two helpers: attaching to a single message or the head
of an array without mutating the caller's data, and narrowing the registry
to the active names while dropping non-ChatML fields. The integration tests
check the exported input against pi's default tool set and, with a fixture
that registers two tools and deactivates one, that extension tools arrive
with their schema and deactivated ones stay out.
@CLAassistant

CLAassistant commented Sep 3, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Attach tool definitions to generations

2 participants