Skip to content

feat(trace): Future AGI tenant export (TH-8233) - #1

Closed
nik13 wants to merge 14 commits into
mainfrom
feat/th-8233-futureagi-export
Closed

nik13 wants to merge 14 commits into
mainfrom
feat/th-8233-futureagi-export

Conversation

@nik13

@nik13 nik13 commented Oct 4, 2026 •

Copy link
Copy Markdown

Ready for review

The blocking image/deployment qualification is complete. The fork builds and runs with real MySQL and Infinity, real browser settings onboarding, document ingestion/retrieval, and a real model chat whose OTLP payload was accepted by Future AGI Cloud. Required GitHub review is still outstanding; no merge, registry publication or ticket closure has been performed.

  • Fork head: 60492deba70ea7431e13c23de271ab10c3aa0aea.
  • Base main: 2400ca8eb51432b1d304266d0de9ca232d9b53fa, confirmed an ancestor.
  • Customer guide: docs PR #1024, head 1a9e1964871b252022415fba24e2a28e11074f4d.

What this integration provides

Per-tenant Future AGI export for RAGFlow's Go chat pipeline, configured beside Langfuse in User settings → API. A retrieved-document chat emits ragflow.chat with ragflow.retrieval and ragflow.llm children. Settings reads mask credentials; content can be redacted; a bounded asynchronous queue avoids blocking chat. Credentials remain plaintext database columns, matching the existing integration pattern, so database access and backups need protection. Token counts are RAGFlow tokenizer estimates, not provider billing totals.

Stock RAGFlow v1.0.0 does not contain this exporter. Read-only inspection of that release and upstream main 3c6d2255af06164f0818d0959af10de0afd8b8fa did not establish a supported fork-free OTLP route: no exporter/provider initialization was found, and Langfuse uses its own endpoints and Basic authentication. Nothing was opened, commented on or pushed upstream.

Build fixes

  • Pin the native builder and model/tokenizer asset images by digest. This is not a claim that every external input is hermetic.
  • Use an overridable Go proxy chain with | fallback on transport/server errors and the official proxy first. Checksum verification remains enabled.
  • Keep Go module/build-cache environment variables in effect for compilation, not just for the preceding git config command.
  • Run native recovery with uv run --script, which installs the downloader's declared dependencies. Remove the stale archive from its actual ragflow_deps/ path. Keep the required office_oxide 0.1.12 pin rather than accepting the builder's incompatible preseeded 0.1.16 library.
  • Add regression checks for matching/stale caches, wrong downloaded versions, missing headers, downloader failure and Docker acquisition settings. Update all recovery help text to the working command.

Decisive production qualification

E2B run th8233-rf1024-live-1109e, 2026-11-09 03:09–03:27 UTC, 4 vCPU / 8192 MB, exit 0:

  • Exact checkout 60492deba, tree cdd580eb1990b95684b9fce60e9cfde988ff9ce6.
  • Production image sha256:93b4cda46d76fb5eba6bc142001b8586427da687394b79302c64f4e48ccd7b56, 769,489,952 bytes.
  • Image version v1.0.0-rc1-102-g60492de, derived from actual release ancestry. Development mode disabled; database version checks passed for API, admin and ingestor.
  • Both shell build-regression suites passed. build.sh --test -run FutureAGI ./internal/service ./internal/handler passed with CGO/native libraries, as did the native unit suites invoked by the build harness.
  • Two successive migrations against real MySQL 8.0.40 passed. tenant_futureagi contains the expected credential, endpoint, project, redact and timestamp columns.
  • Production entrypoint started API/admin/ingestor/nginx against Infinity 0.7.3, MinIO, Kvrocks and NATS.
  • Playwright drove real signup/login and the Future AGI configuration dialog against the running server and MySQL. Save succeeded and subsequent reads returned masked keys. No settings API was mocked.
  • A real text document was uploaded and ingested. Infinity retrieval returned its content. OpenRouter chat using openai/gpt-4.1-nano answered ORCHID-42, the code in that document; embeddings used openai/text-embedding-3-small.
  • Captured OTLP contains one trace with the chat root and correctly parented retrieval/LLM spans, retrieved document content/score and model/token attributes. Provider-key absence was checked in-process in E2B and again locally before forwarding.

Cloud acceptance: the approved local forward_otlp.py helper returned trace-1.json 200 and forwarded 1/1 to project ragflow-th8233-qualification. Trace ID: 96f1a57535864125c2d81b3843d1e80b. Cloud credentials never entered E2B. This records collector acceptance of the inspected real payload; no Cloud browser/UI verification is claimed.

The run ended with 6,198,460,416 bytes of RAM and 14,752,927,744 bytes of disk available; every container reported oom=false. No resource exhaustion was observed. The staged provider credential was removed immediately afterward, and sandbox deletion was confirmed by 404.

Fresh-reader and CI verification

  • E2B th8233-rf1024-reader-1109f, exit 0: extracted and ran the exact Bash block from the rewritten customer page in a clean sandbox. It built sha256:3c73454ed390d29890c49283ab61240cd837665edd8f22584aab14a947130524 from the same pinned checkout, with the correct release-based version. No credentials were staged; sandbox deletion confirmed by 404.
  • Final-head fork regression CI passed: PR run, push run.
  • All eight checks passed on the rewritten docs head: PR Checks and rendered-page regression, FastMCP, Temporal. The actual rendered RAGFlow guide was inspected.
  • The inherited self-hosted sep-tests jobs were skipped for the draft/label policy. They are not being called passed; targeted native/backend testing and the real-stack qualification above ran in E2B instead.

All builds/tests ran in E2B or GitHub Actions, not on the local workstation. No rebase, force-push or branch rename occurred.

Receipts and scope

  • /Volumes/nik-ext-1/tmp/traceai-release-20261104/e2b/runs/th8233-rf1024-live-1109e/
  • /Volumes/nik-ext-1/tmp/traceai-release-20261104/e2b/runs/th8233-rf1024-reader-1109f/
  • Inspected OTLP, retrieval, answer, browser and database evidence: /Volumes/nik-ext-1/tmp/agent-plugins-1106/media/rf1024-live-1109e-evidence/evidence/
  • Qualification scripts: /Volumes/nik-ext-1/tmp/agent-plugins-1106/media/rf1024-live-r3-stage/ and /Volumes/nik-ext-1/tmp/agent-plugins-1106/media/rf1024-reader-stage/.
  • Rendered guide: /Volumes/nik-ext-1/tmp/agent-plugins-1106/media/rf1024-docs-render-1109f/ragflow-customer-guide.png.

Earlier runs th8233-rf1024-fix-1109a, th8233-rf1024-stack-1109b, and th8233-rf1024-live-1109c/d remain retained in the same runs directory. They established build/bootstrap progress and exposed harness assumptions (--help exits 1, two animated login cards, and insufficient Git ancestry). They are superseded by the successful final runs, not hidden or characterized as application/resource failures.

Qualification covers this pinned fork on a fresh MySQL/Infinity deployment and the Go chat pipeline. It does not establish upgrades of existing customer databases, stock-image support, Elasticsearch parity, agent-canvas tracing or capacity limits. The guide preserves those boundaries. Merge/review and any subsequent release/publication remain separate actions; TH-8233 has not been closed by this work.

🤖 Generated with Claude Code

nik13 added 2 commits October 4, 2026 12:54
A second tenant exporter beside Langfuse, not a replacement. RAGFlow's
own OTel config has no header field and no wired exporter, so a recipe
cannot reach fi-collector. This client posts OTLP/HTTP to
/tracer/v1/traces with X-Api-Key and X-Secret-Key, and sets project_name
from the tenant row. A full channel drops the span. The chat path never
waits. Redact omits prompt and answer text.

Keys at rest are plain strings, matching tenant_langfuse. Retrieval
chunk ids are not in the Langfuse call sites and are not invented here.
@nik13
nik13 requested a review from NVJKKartik October 10, 2026 08:59
@nik13

nik13 commented Oct 10, 2026

Copy link
Copy Markdown
Author

Validation on cffab59, not a merge claim.

RED: go test of the client failed to compile. 0x9e3779b97f4a7c15 overflowed int64 in futureAGISpan. GREEN: the same package tests passed after the constant is XOR'd as uint64, including the existing collector-header test and a chat-path test that posts two tenants and drops content when redact is on.

Not run: a RAGFlow server, tenant DB lookup, or retrieval chunk attributes. Loopback only. Internal fixture ingester, not production. No key values recorded. Reviewer requested: NVJKKartik. This comment is not a GitHub approval.

nik13 and others added 8 commits October 11, 2026 21:43
- decorateAnswer called futureAGISpan with chat.ID and modelName, but chat
  is not in scope there and modelName was local to the Langfuse block, so
  ragflow/internal/service did not compile ("undefined: chat",
  "undefined: modelName"). The model name is now read once for both
  exporters, and the chat id comes from the per-request client, as it
  does for Langfuse.
- The span started and ended at the same instant. It now starts when the
  request timer started.
- A failed model call produced no span. Both pipeline error branches now
  send one with ERROR status and the error message.
- Trace ids were derived from the clock. They are now random.
- futureagi_test.go redeclared contains(), already defined in
  kb_prompt_test.go, so the package tests did not compile either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chat span is queued just before the request returns and calls
Shutdown. With both the stop and the events channel ready, the worker's
select dropped the span about half the time (9 to 13 of 20 in the new
test). The worker now drains what is queued before it exits, as the
Langfuse client already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exporter only ran on the OpenAI-compatible endpoint and only for
dialogs with a knowledge base: chats from the RAGFlow UI and REST
sessions, embedded chatbots and dialogs without datasets sent nothing.
Each turn was one span without a span kind, the retrieved chunks were
not exported, the raw error text (with model hosts) went into the
status, and every request waited up to 2 s on a per-request shutdown.

- AsyncChat is now a thin wrapper: when the chat's tenant has a
  tenant_futureagi row it starts a turn, runs the pipeline unchanged and
  forwards its results. Every caller (chat sessions, the OpenAI-
  compatible endpoint, chatbots) goes through it.
- A turn is a CHAIN span (question, answer, session = conversation id
  or chat id, user.id = tenant id) with a RETRIEVER child (query, chunk
  ids, similarity scores, chunk text, document names, top_n/top_k,
  threshold) and an LLM child (model, provider, input messages, answer,
  RAGFlow's own cl100k token counts). Agentic and SQL answers keep the
  CHAIN span.
- Failed turns are ERROR. Error text has URLs, IPs, e-mail addresses,
  paths and the server's user/host names masked; with redact on it is
  a fixed line. Redact drops every prompt, message, chunk text,
  document name and answer, and keeps ids, scores, model and tokens.
- One cached client per tenant credential set; the queue is bounded
  (1024), a full queue or a collector error drops the trace with at
  most one warning a minute, and the chat path never waits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was no way to configure the export short of writing the
tenant_futureagi row by hand. Add the same API shape as
/langfuse/api-key: POST/PUT stores the API key, secret key, Observe
project name, endpoint (default https://api.futureagi.com) and redact
flag after an empty OTLP export to the collector accepted the keys
(401/403 fails the save); GET returns the row with masked keys; DELETE
removes it. Saving or deleting retires the tenant's cached client.

Keys are stored as plain columns, the same as tenant_langfuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clients now live for the process, so flush them (2 s cap) after the
HTTP server shut down instead of per request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or in-flight turns

- futureagi_live_test.go (build tag manual, opt-in with FUTUREAGI_LIVE=1)
  drives RAGFlow's own model provider, chat assistant, chat session
  completion and OpenAI-compatible services against a real collector and
  model, with a bad-key save, a broken-model failure and an optional
  collector outage.
- Shutdown now waits (bounded) for turns whose pipeline is still closing
  its channel: the OpenAI-compatible handler returns on the final result,
  so its turn finishes a moment after the request.
- With redact on, web-search chunk ids (built from the page URL) are
  exported as a stable hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AsyncChatSolo resolved the model into a local target in the pinned tree;
v1.0.0-rc2 builds the chat model through ModelFactory and has no such
variable, so the hook did not compile there. Both trees have the
llmModelConfig the main path already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…quest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a Future AGI card next to the Langfuse card (same placement, same
hook/service/dialog structure) for /api/v1/futureagi/api-key: API key,
secret key, Observe project, endpoint and a redact switch, with delete.
The server returns keys masked, so the form never prefills them.

Checked with oxfmt and oxlint (the versions lefthook pins); the dialog was
not rendered and the TypeScript was not type-checked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@nik13
nik13 marked this pull request as draft November 9, 2026 01:34
nik13 and others added 3 commits November 8, 2026 17:54
Pin builder and asset images, fall back on Go proxy transport errors, retain the build cache for compilation, and honor downloader script metadata when replacing incompatible office_oxide archives.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Use the tested uv script invocation in all build help and failure messages so manual recovery also installs the downloader dependencies.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 marked this pull request as ready for review November 9, 2026 03:55
@nik13

nik13 commented Nov 9, 2026

Copy link
Copy Markdown
Author

Superseded by the upstream contribution infiniflow#22183 (branch contrib/futureagi-tracing, rebuilt on current RAGFlow main). It also adds the shared outbound host policy and keeps provider error bodies out of traces. This branch stays for reference.

@nik13 nik13 closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant