Repository navigation
feat(openwebui): Future AGI filter function (TH-8232) - #207
Merged
Merged
Conversation
A standalone Filter an admin pastes into Open WebUI. It exports one span per chat turn to /tracer/v1/traces with X-Api-Key and X-Secret-Key and a project_name resource attribute. outlet never raises, so a dead collector does not break the chat. Content is omitted when redact is set, and email is hashed only when include_email_hash is on. 7 tests passed without Open WebUI. No traceAI runtime dependency, because Open WebUI installs the frontmatter requirements into its own environment.
Contributor
Author
|
Validation on unchanged head 95885ad, not a merge claim.
Not run: an Open WebUI process. The filter is the plugin entry point Open WebUI calls. Loopback only. No CI check run on this head. Reviewer requested: NVJKKartik. This comment is not a GitHub approval. |
…0.11.4
Run in a real Open WebUI 0.11.4 (pip, native), the filter as written sent
nothing: three independent defects each made it a silent no-op.
- Valves was a module-level class. Open WebUI only applies valves when the
Filter instance has a Valves attribute, so self.valves stayed None, the
admin UI showed no valves (spec null, update 401) and outlet never
exported. Valves is now Filter.Valves, every field has a default and
__init__ sets self.valves.
- The frontmatter opened with '"""title:' on one line. Open WebUI parses
it only when the first line is '"""' alone, so the requirements were
never installed; Open WebUI 0.11.4 does not ship the OTLP/HTTP exporter,
so the export import would have failed (and been swallowed).
- SecretStr valves: Open WebUI stores model_dump(), which turned the keys
into "**********" and stored nothing ({} read back, checked live). The
keys now serialize by value, stay masked in repr, and render as
password inputs (json_schema_extra input.type=password).
The outlet also read fields Open WebUI does not send (body output, usage,
user_id, email): the reply, its usage and the user now come from the last
assistant message and __user__. Usage reads prompt/completion tokens and
sums them; Open WebUI's stored total_tokens doubles when a provider repeats
usage in two stream chunks (seen live: 17690 for 8838+7).
Per the TH-8232 PRD (R-02, R-04): each turn is now a CHAIN root span with an
LLM child, timed from inlet to outlet via the request's __metadata__ (the
old inlet kept an unbounded per-chat dict that outlet never read), and
outlet only queues spans: a cached provider per valves set exports through
a BatchSpanProcessor with a bounded queue instead of building a provider
and exporting synchronously on Open WebUI's event loop per turn. With the
collector unreachable chats still answer in about a second.
README rewritten: install steps Open WebUI needs (turn on, Global or per
model), valves table, span shape, privacy, and the limit that failed turns
get no outlet and so no span. Tests rewritten against Open WebUI 0.11.4's
hook arguments (21, was 7).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open WebUI serves many users, so the redact valve now defaults to true: with the filter installed and only the keys set, no prompt or reply leaves the instance. Turns still carry chat and user ids, model and token counts, so the trace tree, sessions and usage show in Observe with Input and Output empty. The README says plainly to set redact=false to see prompts and replies. Tests use a neutral model id. Checked live in Open WebUI 0.11.4 against a local Future AGI stack with a real OpenAI-compatible endpoint: one chat turn with default valves arrived with no message text; one with redact=false arrived with the prompt and the reply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Document that stock Open WebUI 0.11.4 bypasses Function filters for plain JSON stream errors, retain the observable decoded SSE regression, and describe the upstream hook needed for the bypassed path. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch carried an older copy of the shared OTLP test harness. The copy in #203 (and in every sibling integration PR) is a superset: it also accepts chunked request bodies and records request headers and resource attributes. Using the same bytes avoids an add/add conflict on python/tests/harness whichever of these PRs merges first. Harness tests: 6 passed; filter tests: 37 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a standalone Future AGI filter function for Open WebUI 0.11.4. The filter exports one detached
openwebui.chatchain span with anopenwebui.llmchild for each completed chat turn, using the endpoint and credentials configured in its valves.redact=true); administrators can opt in withredact=false.Verification sweep (2026-10-12)
add(17, 25)and returned42; cumulative usage was recorded asinput_tokens=18369,output_tokens=69,total_tokens=18438.Tests
pytest integrations/openwebui/tests/test_filter.py --noconftest -q: 31 passed.The branch is pushed at
cba16eaand is mergeable. Merge order: merge #203 first with a merge commit, and the README catalog PR #250 last; the other integrations are order-independent.Re-verification (2026-10-16)
No code change was needed; the filter at head
20707ccis unchanged. Tests:pytest integrations/openwebui/tests/test_filter.py --noconftest -qgives 37 passed.openwebui.chat(chain) with childopenwebui.llm(llm), modelgpt-6-luna, session = chat id. Native tool turn (add 17+25 -> 42): 0.12.3 tokens 7326/26, 0.11.4 tokens about 7100/26 (cumulative over the tool loop). Plain turn 3613/5. Traces e.g. 0.12.33a9eb000-565b-4adb-4e11-5d4887124f86,7b735782-9031-dffb-e5be-042cdf269c63; content-on81ea4f15-a264-a1ab-37d1-88a30e841f70; 0.11.437eb8659-1b1c-b430-a02e-1218e115dfb9.redact=true: unique marker in the prompt, 0 of 4 spans (0.12.3) and 0 of 8 spans (0.11.4) contain it in any stored column (scan oftoString(tuple(*))over the spans table, includes resource attrs and events); also 0 hits for the OS user name,/Users/, the account email andTraceback. Withredact=falsethe marker is ininput.value/output.valueof both spans (4 of 12 spans on 0.12.3 contained it, the two chain + two llm of the marker turns).provider returned a failed response; neither the stub's error text nor a plantedSECRETSTACK /Users/...string nor the prompt marker is stored. A provider HTTP 500 yields no span (host bypass, already documented). Same result on 0.11.4 and 0.12.3.model,messageswithid/role/content/usage,chat_id). Observation: an API client that starts a chat with nochat_id(parent_id: null) produced no filter hooks in my test on 0.12.3; web-UI-created chats (existing chat ids) are traced. Documented as a limitation.Live model verification (2026-10-19)
Re-run against the real OpenAI API (
https://api.openai.com/v1,gpt-4o-mini) as the Open WebUI model connection, instead of a local stand-in. Open WebUI 0.11.4 (pip install open-webui==0.11.4, Python 3.12), filterfutureagi_filter.pyversion 0.3.0 installed through the Open WebUI API as in the install steps, local Future AGI as projectth8103-livefill-TH-8232. 0.12.3 was not re-run.redact=true)0ed539cf-1c03-5a63-e9c0-24a2b04964bcopenwebui.chat(chain) with childopenwebui.llm, modelgpt-4o-mini, 3189 in / 2 out tokens, session = chat id, no contentadd)43ac29ee-713e-368e-a601-e6cd784c29ccredact=false6d2b79b4-ab6e-214a-8abe-4fb776d1ef11input53 B /output2 B on the chain span)Privacy: with
redact=truethe unique marker (in the prompt of both turns) was in none of the 4 stored rows, in any column, and the OpenAI key was not either. Withredact=falsethe marker was in the 2 rows of that turn.Unknown model id: a model id that Open WebUI does not list is rejected by Open WebUI itself with HTTP 400
Model not foundbefore any filter runs, so there is no trace. With the id listed in the connection (gpt-4o-mini-not-a-model), OpenAI returned HTTP 404model_not_found; Open WebUI logged the failure and the filter exported no span. This is the stock-0.11.4 limitation already described above (a plain provider error is not passed to the filter), now confirmed against the real API: an upstream error does not produce an ERROR span.