feat: capture the system prompt in traces - #24
Open
wierdbytes wants to merge 2 commits into
Open
wierdbytes wants to merge 2 commits into
wierdbytes wants to merge 2 commits into
Conversation
Closes langfuse#21. Read the effective system prompt with `ctx.getSystemPrompt()` in `agent_start` — by then the session has applied the final override, whereas `before_agent_start` handlers still see it mid-chain — and attach it redacted to the turn root as `metadata.system_prompt`. Each generation's input is also headed with a `{ role: "system" }` message so the Langfuse UI renders it as the `> system` block there as well.
A fixture extension loaded after ours rewrites the prompt in before_agent_start, so the tests pin the agent_start timing from langfuse#21: the root metadata and every generation's leading system message must carry the override, in full, and Langfuse keys planted in the prompt must be redacted before export.
|
|
wierdbytes
marked this pull request as ready for review
September 3, 2026 19:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #21.
Change
The extension never recorded the system prompt: the turn root's input was
{ role: "user", content }and each generation's input the same or{ role: "tool", tool_results }, so a trace could not show what the model was instructed with.The prompt is now read with
ctx.getSystemPrompt()inagent_startand attached, redacted, to the turn root asmetadata.system_prompt. Each generation's input is headed with a{ role: "system" }message carrying the same text, which the Langfuse UI renders as the> systemblock.agent_startrather thanbefore_agent_start, as suggested in the issue: the runner hands eachbefore_agent_starthandler the prompt as it stands mid-chain, so an extension registered after this one can still rewrite it. Byagent_startthe session has applied the final override andgetSystemPrompt()returns what is actually sent. The call is optional-chained and wrapped, so a pi without it degrades to no capture.The prompt is stored in full, not through
truncateText. User and tool outputs are unbounded and can be megabytes; the prompt is the debugging subject itself, and a truncated copy would be the one thing a reader cannot reconstruct. Redaction still applies, in the handler and again via the processor mask on export.Before:
After:
Tests
pnpm typecheckis clean.pnpm testgives 64 of 64 passed, up from 62 onmain.test/system-prompt.test.tsruns pi with a fixture extension loaded after ours that rewrites the prompt inbefore_agent_start. The root metadata and the leading system message of all three generations must end with the fixture's marker and still hold pi's base prompt in front of it. Reading the prompt inbefore_agent_startinstead ofagent_startmakes this test fail; onmainit fails withroot metadata must carry system_prompt.sk-lf-…key in the prompt and asserts it appears nowhere in the export, replaced by the redaction mark, with the same text on the root and on every generation.