Skip to content

feat: capture the system prompt in traces - #24

Open
wierdbytes wants to merge 2 commits into
langfuse:mainfrom
wierdbytes:feat/system-prompt
Open

wierdbytes wants to merge 2 commits into
langfuse:mainfrom
wierdbytes:feat/system-prompt

Conversation

@wierdbytes

@wierdbytes wierdbytes commented Sep 3, 2026

Copy link
Copy Markdown

Closes #21.

Change

The extension never recorded the system prompt: the turn root's input was { role: "user", content } and each generation's input the same or { role: "tool", tool_results }, so a trace could not show what the model was instructed with.

The prompt is now read with ctx.getSystemPrompt() in agent_start and attached, redacted, to the turn root as metadata.system_prompt. Each generation's input is headed with a { role: "system" } message carrying the same text, which the Langfuse UI renders as the > system block.

agent_start rather than before_agent_start, as suggested in the issue: the runner hands each before_agent_start handler the prompt as it stands mid-chain, so an extension registered after this one can still rewrite it. By agent_start the session has applied the final override and getSystemPrompt() returns what is actually sent. The call is optional-chained and wrapped, so a pi without it degrades to no capture.

The prompt is stored in full, not through truncateText. User and tool outputs are unbounded and can be megabytes; the prompt is the debugging subject itself, and a truncated copy would be the one thing a reader cannot reconstruct. Redaction still applies, in the handler and again via the processor mask on export.

Before:

screen-2026-09-03-21 10 55

After:

screen-2026-09-03-21 11 55

Tests

  • pnpm typecheck is clean.
  • pnpm test gives 64 of 64 passed, up from 62 on main.
  • test/system-prompt.test.ts runs pi with a fixture extension loaded after ours that rewrites the prompt in before_agent_start. The root metadata and the leading system message of all three generations must end with the fixture's marker and still hold pi's base prompt in front of it. Reading the prompt in before_agent_start instead of agent_start makes this test fail; on main it fails with root metadata must carry system_prompt.
  • A second run plants an sk-lf-… key in the prompt and asserts it appears nowhere in the export, replaced by the redaction mark, with the same text on the root and on every generation.

Closes langfuse#21.

Read the effective system prompt with `ctx.getSystemPrompt()` in
`agent_start` — by then the session has applied the final override, whereas
`before_agent_start` handlers still see it mid-chain — and attach it
redacted to the turn root as `metadata.system_prompt`.

Each generation's input is also headed with a `{ role: "system" }` message
so the Langfuse UI renders it as the `> system` block there as well.
A fixture extension loaded after ours rewrites the prompt in
before_agent_start, so the tests pin the agent_start timing from langfuse#21: the
root metadata and every generation's leading system message must carry the
override, in full, and Langfuse keys planted in the prompt must be redacted
before export.
@CLAassistant

CLAassistant commented Sep 3, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@wierdbytes
wierdbytes marked this pull request as ready for review September 3, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Capture system prompt in traces

2 participants