Repository navigation
feat(examples): MLflow OTLP trace-export recipe and contract test (TH-8334) - #233
Merged
Merged
Conversation
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103
No traceAI package: MLflow 3.16.1 already exports its spans over OTLP when OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. src/app.py makes two nested mlflow.start_span calls (a CHAIN root and a CHAT_MODEL child with gen_ai.* and mlflow.llm.* model/token keys) around a canned stand-in model client, so it makes no model call. check_environment() exits with a message on the settings that make MLflow export nothing without an error: no full traces URL (MLflow does not append /v1/traces, utils/otlp.py:98-99), a protocol other than http/protobuf (the default is gRPC, otlp.py:124-136), OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_METRICS_ENDPOINT (they turn on gRPC metrics, which drop every span), and OTEL_RESOURCE_ATTRIBUTES without project_name or with an item MLflow cannot parse (provider.py:716-731). --drop-content registers a span processor through mlflow.tracing.configure; MLflow has no content-off switch. requirements.txt pins mlflow==3.16.1 and adds opentelemetry-exporter-otlp-proto-http==1.45.0, which mlflow does not install. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8334
- _guarded_run.py blocks and logs every non-loopback connect, connect_ex and getaddrinfo before the script runs (same guard as the OpenLLMetry recipe); _guard_probe.py is its positive control against TEST-NET-1, the IPv6 documentation prefix and .invalid. - bare_span.py runs the app's spans without check_environment(), so the test can record what MLflow itself does with the settings the app refuses. - set_destination_app.py calls mlflow.tracing.set_destination before the app, for the precedence test. - fixtures/gen_ai_span.otlp.json is a hand-built OTLP request with two gen_ai.* spans for post_otlp(): the shape MLflow sends with MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true and its default JSON-encoded shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8334
Runs src/app.py as written with the real mlflow 3.16.1 in a subprocess, under the loopback guard, with a fresh temp dir as cwd and HOME so every file MLflow writes is checked. Spans go to the harness Receiver; path and catalog checks use a loopback catch-all. Covered: /tracer/v1/traces and /v1/traces both 200 and MLflow does not append /v1/traces (origin, origin/ and /tracer get a 404); no protocol means silent no-export; a base or metrics endpoint drops every span; percent-encoded x-api-key/x-secret-key with ',' and '=' arrive intact; project_name/project_type on the resource, lost on a trailing comma; MLflow JSON-encodes every attribute value (gen_ai.request.model arrives with its quotes); no experiment id or session on the export, and an experiment id passed in OTEL_RESOURCE_ATTRIBUTES arrives as a resource attribute; content in mlflow.spanInputs/spanOutputs and the drop-content processor; dual export unset/false/true with and without a tracking URI (./mlflow.db checked by trace id) and its model-catalog fetch; set_destination silences OTLP entirely, and with dual export sends both; the telemetry switch; MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true output; the hand-built gen_ai.* fixture via post_otlp() and its match with live runs. Opt-in (FI_COLLECTOR_SRC): fi-collector's alias tables include the gen_ai.* keys and no mlflow.* key, and the README's collector claims and span-type table match adapter.go/converter.go and MLflow's translator.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8334
Live OTLP export only: states that historical import is not this integration and that an experiment id is a resource attribute, not a Future AGI session. Publishes one endpoint form, <collector origin>/tracer/v1/traces (PRD O-2: MLflow uses OTEL_EXPORTER_OTLP_TRACES_ENDPOINT as is), with the tested path table. Documents dual export both ways (no tracking URI means a local ./mlflow.db, not OTLP-only), set_destination precedence, the model-catalog fetch, telemetry, MLflow's JSON encoding of attribute values, the key inventory with what fi-collector reads (adapter.go/converter.go at 4af5338, source reading), the optional MLFLOW_ENABLE_OTEL_GENAI_SEMCONV switch and its costs, content capture and the span-processor hook, and the SF-1 gap list. Marks what was tested and what was read from source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8334
The model-catalog test asserts GET /catalog/openai.json requests, not a count; the count of four came from exploration runs. Say only what the test checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8334
…ns and headers Tests first for review-r1/verify-r1 (R1, R3, R4, V1, V2). On the unchanged app and README at 997ac28 these fail with assertion failures (15 on Python 3.12): - check_environment() must refuse, before the first span and with zero requests to a loopback catch-all that also stands in for the model catalog and DATABRICKS_HOST: MLFLOW_TRACKING_URI=databricks and databricks://profile, MLFLOW_TRACING_DESTINATION=0 (dual export unset or false), a trailing-slash traces endpoint, a general OTEL_EXPORTER_OTLP_HEADERS, unset or empty TRACES headers, and TRACES headers without x-secret-key. No header value may appear in the output. - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf with the TRACES protocol unset must export normally. - The README Run block's own export/unset lines, run by bash over a stale environment (MLFLOW_TRACING_DESTINATION=0, a general other-vendor header, a catalog URI on a loopback recorder) and with no recipe variable pre-set, must deliver both spans with the FI keys to the intended receiver, leave no stale value, write no file and fetch no catalog. All those lines must come before the app line. - The README header line must stop, in a script and when pasted into an interactive bash, when FI_API_KEY is unset or FI_SECRET_KEY is empty. Vendor-behaviour tests that pass already: MLflow posts a trailing-slash endpoint verbatim (404); MLFLOW_TRACING_DESTINATION silences OTLP like a set_destination call and only logs INFO lines; with dual export it sends both; with MLFLOW_TRACKING_URI=databricks and dual export off MLflow still fetches /catalog/openai.json and, with an experiment id, GETs /api/2.0/mlflow/experiments/get, all on loopback with the guard log empty; upper-case header names arrive; TRACES grpc over a general http/protobuf and a general grpc are refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8334
check_environment() now also exits, before the first span and with one
message each (decision D-8334a):
- MLFLOW_TRACKING_URI=databricks or databricks://... with dual export off
(R1, V2): MLflow then reads the experiment from the workspace on the
first span, fetches its model catalog (github.com by default) to price
the model span, and replaces OTLP with Unity Catalog for a linked
experiment.
- MLFLOW_TRACING_DESTINATION set with dual export off (V1): it replaces
the OTLP export with no message.
- A trailing slash after /v1/traces (R3): MLflow posts it verbatim and
fi-collector's exact patterns answer 404. The rstrip("/") is gone.
- OTEL_EXPORTER_OTLP_HEADERS set, or OTEL_EXPORTER_OTLP_TRACES_HEADERS
without both x-api-key and x-secret-key (R4): the exporter falls back to
the general headers when the TRACES ones are empty. Names are matched
case-insensitively as the exporter parses them; values are never
printed.
The protocol is now read as MLflow reads it (TRACES, then
OTEL_EXPORTER_OTLP_PROTOCOL, default grpc), so a general http/protobuf is
accepted (R3). Dual export is parsed as MLflow parses booleans (true/1).
The README Run block exports MLFLOW_MODEL_CATALOG_URI="" and unsets
OTEL_EXPORTER_OTLP_HEADERS and MLFLOW_TRACING_DESTINATION, and its header
line stops behind ': "${FI_API_KEY:?}" "${FI_SECRET_KEY:?}" &&' when a key
is empty, instead of exporting an empty value. The app docstring states the
tested condition, and the comment on the gen_ai.* keys says the quoted
model is not priced (R2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
…and header refusals
README prose for review-r1/verify-r1, following D-8334a:
- R1: the catalog paragraph now states the true condition. With dual
export off and a non-Databricks tracking URI MLflow does not fetch the
catalog (tested: no connection attempt with the default URI); with a
Databricks tracking URI it does (tested on loopback). The recipe table
lists MLFLOW_MODEL_CATALOG_URI empty and the variables that must be
unset, and says which of them src/app.py checks.
- V2: a "Databricks tracking URI" section: the client-side pricing and
catalog fetch, the experiment read on the first span (both tested with a
loopback workspace and catalog), the Unity Catalog replacement of OTLP
(source reading), the refusal, and dual export with Databricks as out of
scope and untested, with the calls it makes from source. The intro now
states the tested condition for "contacts only the OTLP endpoint".
- V1: MLFLOW_TRACING_DESTINATION next to set_destination (tested silence
and dual export), and that a set_destination call in code cannot be
seen by check_environment().
- R3: the trailing-slash row in the endpoint table, and the
OTEL_EXPORTER_OTLP_PROTOCOL fallback.
- R4: the general-headers fallback, the refusal, and why the header line
starts with ': "${FI_API_KEY:?}" "${FI_SECRET_KEY:?}" &&'.
- R2: the quoted model gets no token price (cost 0); "cost not priced" in
the SF-1 JSON-decoding bullet.
- R5: mlflow.traceTag.<key> in the SF-1 list and in what the drop-content
processor does not cover.
- Troubleshooting and the Tests section follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
A live run with mlflow.openai.autolog() against fi-collector (future-agi dev ea12cc3) showed that, under the Run block (GenAI semconv switch off), autolog spans arrive as a tree of unknown spans with no model, tokens or cost: autolog writes only mlflow.llm.model and mlflow.chat.tokenUsage, which fi-collector does not read. The gen_ai.* keys the recipe app sets by hand hide this. With MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true the model calls are llm spans with model and tokens, tools are tool spans and chains chain spans. Add that result and tell autolog users to turn the switch on, and mark the recipe app's collector behaviour as seen live, not only read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true, the live autolog run's tool and agent spans were renamed execute_tool and invoke_agent and carried no gen_ai.tool.name or gen_ai.agent.name, so the function names reached Future AGI nowhere. Say so next to the advice to turn the switch on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decision (Nikhil, 2026-10-11): the Run block now exports MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true. With it, fi-collector types MLflow's spans (chat -> llm, execute_tool -> tool, CHAIN -> chain) and reads a plain model name, token counts and cost; without it every span is unknown and an autolog model call carries no model or tokens. The cost, renamed spans and lost function names, is documented with how to turn the switch off. README: "What is exported" now leads with the switch-on export, moves the JSON-encoded inventory under "Turning the switch off", and records the live run of the Run block (recipe app and real autolog) against fi-collector. Tests: the recipe environment includes the switch; recipe_run is the translated export (CHAIN > chat gpt-4o-mini) and a switch_off_run fixture keeps the JSON-encoded key checks. The Run block test starts from a stale MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=false and requires the block to set it. MLflow's trace id is now derived from the OTLP trace id, since the switch drops mlflow.traceRequestId; the switch-off run checks the two agree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ecipe export (TH-8334 re-verification) Plain MLflow exports an exception event with the stack trace (local paths) and the exception text as the span status message. src/app.py now always drops the exception events; --drop-content also clears the status message and keeps ERROR. Contract test no longer hard-codes the mlflow version (passes on 3.16.1 and 3.17.1); new test with a positive control. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds an MLflow → Future AGI trace-export recipe at
python/examples/mlflow/(Linear TH-8334, decision D-8334). It is not a package and adds no instrumentor: MLflow 3.16.1 already exports its own spans over OTLP, so the recipe points that export at fi-collector with environment variables and adds the Future AGI project as a resource attribute. A wrapper aroundmlflow.start_spanwould double every span. Historical import from a tracking server, Databricks or Unity Catalog is out of scope.src/app.py: two nestedmlflow.start_spancalls (answer_questionCHAIN root,chat_modelCHAT_MODEL child) around a canned stand-in model client, so it makes no model call.check_environment()exits with a message on every setting the tests found to export nothing silently.--drop-contentregisters a span processor throughmlflow.tracing.configure.requirements.txt:mlflow==3.16.1plusopentelemetry-exporter-otlp-proto-http==1.45.0; mlflow installs no OTLP exporter.README.md: run block, tested endpoint table, headers, resource/experiment, telemetry, dual export,set_destination, key inventory with the fi-collector column, the optional GenAI semconv switch, span types, content, SF-1 gap list, troubleshooting and the exact test command.tests/: contract test (44 functions, 74 collected cases with the collector source; 71 + 3 opt-in skips without), a loopback network guard with a positive control, two app fixtures and a hand-built OTLP key fixture forpost_otlp().What the tests established (mlflow 3.16.1, loopback receiver)
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTunchanged to the exporter (mlflow/tracing/utils/otlp.py:98-99); it does not append/v1/traces./tracer/v1/tracesand/v1/tracesreach the receiver; the origin, origin/and/tracerpost to the wrong path (404, spans lost). The README publishes/tracer/v1/traces, the path traceAI's own exporter uses.http/protobuf(MLflow defaults to gRPC; with no gRPC exporter it hands out no-op spans and logs only at debug level);OTEL_EXPORTER_OTLP_ENDPOINTor..._METRICS_ENDPOINTset (turns on gRPC metrics and every span fails atend());OTEL_RESOURCE_ATTRIBUTESwithoutproject_nameor with an item lacking=(MLflow re-parses it to{}). Added at7b47514(D-8334a): a trailing-slash endpoint (404, spans lost); an inherited generalOTEL_EXPORTER_OTLP_HEADERS(sent instead of empty TRACES headers, so another vendor's key could reach Future AGI); TRACES headers missing thex-api-keyorx-secret-keyname;MLFLOW_TRACING_DESTINATIONwithout dual export (traces go to that store instead, silently); a Databricks tracking URI without dual export (MLflow calls Databricks, fetches its model catalog from github.com to price model spans, and a UC-linked experiment replaces the OTLP export). The protocol is read the way MLflow reads it (TRACES, then generalOTEL_EXPORTER_OTLP_PROTOCOL).x-api-key/x-secret-keyarrive intact, including a secret containing,and=; names arrive lower-case (fi-collector reads them case-insensitively,pkg/auth/middleware.go:45-46).project_name,project_type=observe,service.name; nosession.idoruser.id.gen_ai.request.modelarrives as"\"gpt-4o-mini\"", tokens as"12".mlflow.spanInputs/mlflow.spanOutputs;--drop-contentremoves every marker in both semconv modes../mlflow.dbwith the same trace (a PRD assumption said OTLP-only; documented as a failure mode).set_destination(MlflowExperimentLocation("0"))→ zero OTLP requests.$HOME/.config/mlflow/telemetry.json;MLFLOW_DISABLE_TELEMETRY=true(added to the recipe) writes nothing. No telemetry network attempt was observed; the phone-home path is source reading.MLFLOW_MODEL_CATALOG_URIempty) the recipe contacts only the OTLP endpoint (tested).What fi-collector does with these spans (source reading at future-agi
main4af5338)mlflow.spanType,mlflow.llm.model,mlflow.chat.tokenUsage,mlflow.spanInputs/Outputs) are read; kind isunknownby default (converter.go:79-131).gen_ai.*keys the app sets are read, but the model value is copied verbatim with JSON quotes (adapter.go:317-323); token strings are parsed (:339-342).MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=trueMLflow emits plaingen_ai.*andgen_ai.operation.name, but drops everymlflow.*key and renames spans; the recipe leaves it off and the README states the trade-off.mlflow.spanTypekind alias,mlflow.llm.*model/provider aliases,mlflow.chat.tokenUsage,mlflow.spanInputs/Outputs, JSON-decoding of MLflowgen_ai.*values,invoke_agentsynonym.Deviations from the spec (approved: Nikhil 2026-10-03 blanket; self-approved by Rick)
processor/otel.py:95-105); the recipe documentsmlflow.experiment_id=<id>inOTEL_RESOURCE_ATTRIBUTES(tested) and says it is not a Future AGI session../mlflow.db(provider.py:884-907), contrary to PRD J3/AC-05.gen_ai.*/llm.*keys; model/token promotion is not asserted (SF-1).MLFLOW_DISABLE_TELEMETRY=trueand setMLFLOW_MODEL_CATALOG_URI=""in the Run block (no github.com catalog fetch). The Run block's header line stops ifFI_API_KEYorFI_SECRET_KEYis empty, and it unsetsOTEL_EXPORTER_OTLP_HEADERSandMLFLOW_TRACING_DESTINATION.Review round 1 (pr-reviewer t_588813db and pr-verifier t_d06c1b98 at
997ac28: CHANGES_REQUESTED) → fixed at7b47514Decision D-8334a (approved: Nikhil 2026-10-03 blanket): the supported mode is OTLP-only export to Future AGI with a non-Databricks tracking URI;
check_environment()refuses the environment settings listed in the README that would silently send nothing, send traces or keys elsewhere, or contact an outside service. It cannot see code-levelset_destination()/set_tracking_uri()calls (documented), and it does not yet refuse invalid MLflow boolean values (V3 below). Rejected: supporting Databricks or dual export in this round (untested surface with real external calls) — documented as untested instead.MLFLOW_MODEL_CATALOG_URI.MLFLOW_TRACING_DESTINATIONsilently routed every trace away from Future AGI. Now refused unless dual export is on, and unset in the Run block.OTEL_EXPORTER_OTLP_PROTOCOLis accepted the way MLflow reads it.mlflow.traceTag.<key>added to the SF-1 gap list.c66da1e): on the997ac28app/README the new tests give 15 failures (all assertion failures), 56 passed, 3 skipped. They include a README-setup regression that runs only the Run block's ownexport/unsetlines on top of a stale environment (stale destination, another vendor's header, a catalog recorder) without pre-seeding the variables under test.Review round 2 (pr-verifier t_cfed1ab8 at
7b47514: VERIFIED, no blocking findings)R1–R5, V1 and V2 resolved. New P3s, recorded as post-merge follow-ups (no third fix round on P3s alone; head stays
7b47514):MLFLOW_TRACE_ENABLE_OTLP_DUAL_EXPORT=''/yes/off, or an invalidMLFLOW_ENABLE_OTLP_EXPORTER) makes MLflow raise while building its provider;start_spanlogs at debug and hands out no-op spans, so nothing is exported, exit 0.check_environment()treats those values as "off". Follow-up: refuse values outsidetrue/false/1/0.unset OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_METRICS_ENDPOINTline (the app refuses those variables, but deleting that README line would not fail a test). Follow-up: seed both in the stale environment.Tests
Head
7b47514. Coordinator rerun from the repo root on a clean tree with the README command; actual exit codes and JUnit captured per cell (tickets/TH-8334/exact-head-7b47514-checked/):python/examples/mlflow/, 1,800 added lines.3eaadc8..997ac28): 0 hits; fix-round evidence and recipe at7b47514: 0 hits. FI keys in tests are placeholders.Limits
@mlflow.tracewere not run. gRPC transport not tested. Only SQLite tracking store (not./mlruns). macOS arm64 only. Python 3.14 not run.drop_contentdoes not touchexceptionevents on failing spans (source reading).Video demo
Recorded build: head
7b4751480f10d407ad4afe25eb2cbc69f13e6bae(this PR); 9 real terminal chapters of the demo driver from a clean read-only worktree plus intro and review cards; 8:20, 11 chapters, narrated with captions.The video (SHA-256
f71b02b4e52a7f8ea677efa8fbfdde266e411e176e6adfbb665faa99bb449d6e), captions (SRT), transcript, chapter list, preview and both media-verification notes are private attachments on Linear issue TH-8334 (Future AGI workspace; sign-in required). They are not linked here as raw upload URLs.What it shows: the README Run block's own 12 export/unset lines run in a bare bash, then the app exports 2 spans to
/tracer/v1/traceswith header names only and the project resource, the model arriving JSON-quoted ("gpt-4o-mini"), guard 0, no files, no catalog fetch; MLflow on its own exits 0 and loses the spans with default gRPC (0 requests, empty stderr) or a wrong/trailing-slash endpoint (404), and the app refuses each; six D-8334a refusals with 0 recorder requests, the general protocol accepted, and MLflow alone with a Databricks URI calling the workspace and the model catalog (loopback recorder); the main proof: the README Run block over a stale environment — the new README exports 2 spans with the FI keys and contacts nothing else, the old one (997ac28) exports 0 spans, writes./mlflow.dband fetches the catalog 4 times; with the FI API key unset the new README stops at the header line and the old one sends the other vendor's header; dual export writes./mlflow.dbandset_destination()in code sends 0 OTLP; collector pricing as source reading (quoted model → cost 0); live Python 3.11 suite 71 passed + 3 skipped next to the recorded matrix and RED log; V3 reproduced live (an invalid MLflow boolean exports nothing, exit 0). Key, header and placeholder values are never printed.Independent media checks (Rick): full audio/video decode exit 0; 11 embedded chapters match the list; the four pauses of 1.8–4.6 s sit inside live-command scenes (the longest checked frame by frame: output still printing); all chapter-end frames, the preview, the live test result and the pause frames inspected (legible, nothing under the captions, no key or header values, local paths or ports); 7 delivered files scanned with the driver's 15 fixture needles plus path, port, header and credential checks: zero hits. Driver provenance: the first driver worker stopped on a model quota limit after writing two scripts; Rick completed the runner, scanner and outcome assertions and ran all nine chapters twice on the final bytes (exit 0, byte-identical, clean source) before recording. Disclosed edits: static waiting cut (largest 90.5 s of live pytest), nothing sped up, long output paged from the same run's saved output. Not shown: a live model, Databricks, a running fi-collector, auth, storage or the trace view.
Stacked on #203 (shared harness,
3eaadc8), basedev. Not merged.Linear: TH-8334. approved: Nikhil 2026-10-03 blanket
Verification sweep (2026-10-12)
MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=trueby default (types, model, tokens and cost appear in Future AGI); the README explains how to turn it off and what that trades.dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).Re-verification (2026-10-16)
Versions. mlflow 3.17.1 (latest stable on PyPI, 2026-10-16) and the pinned 3.16.1;
opentelemetry-exporter-otlp-proto-http1.45.1 / 1.45.0; Python 3.12; realopenaiclient; local Future AGI (OTLP at/tracer/v1/traces, uncompressed). Run block used as written (only origin, keys, project substituted).Real E2E (mlflow 3.17.1,
mlflow.openai.autolog()+@mlflow.traceAGENT/TOOL, model via an OpenAI-compatible proxy, gpt-6-luna).fi_verifyPASS, projectth8103-rick-mlflow-real:Recipe
src/app.pyitself (projectth8103-rick-mlflow-trace-export, trace 698333f4-851f-1824-40ea-e4b2d8874815):CHAINoverchat gpt-4o-mini(llm, model gpt-4o-mini, tokens 12/7). UI screenshot checked: trace tree, agent graph, tool and model spans, tokens. Agent spans stay kindunknown(existing SF-1 gap,invoke_agentsynonym);input/outputcolumns stay empty (existing SF-1 gap).Contract test. Pinned 3.16.1: 75 passed, 3 skipped (opt-in fi-collector source). Latest 3.17.1: 75 passed, 3 skipped. Before my change 3.17.1 had 1 failure (test hard-coded
telemetry.sdk.version == "3.16.1"); 3.16.1 had 72 passed.Privacy (found and fixed). Plain MLflow exports, for a failed span, an
exceptionevent whoseexception.stacktraceholds local file paths (so the user name) and the exception message, plus the exception text as the span status message. The README only listed this as "not covered". Fixed at780ca7d:drop_exception_eventsspan processor, always registered bysrc/app.py(removes the exception events; status stays ERROR);--drop-content/drop_contentnow also clears the status message (keeps ERROR);tests/failing_app.py+ testtest_error_text_and_stack_trace_are_dropped_and_status_stays_error(3 modes, with a positive control that plain MLflow exportsexception.stacktrace,Traceback, the marker and the local path); the contract test version assertion now reads the installed mlflow version.Stored-column scan (every column of every span row of the project, ClickHouse): with defaults the unique prompt markers
REALMARK_OK_8334/REALMARK_FAIL_8334are present (attrs_string); with the processors on,HIDEMARK3_OK_8334,HIDEMARK3_FAIL_8334,Traceback,/Users/, the user name, the hostname, the provider error text and the Future AGI key/secret have 0 hits (6 rows, projectth8103-rick-mlflow-hide3, failure trace included). Autolog's tool schemas (tools,gen_ai.tool.definitions) are still exported; they are developer code, documented. MLflow has no content switch: content is on by default and removed only by the processor (README "Content").Docs. future-agi/docs#1005 (draft, base main, page
src/pages/docs/integrations/mlflow.mdx, no navigation.ts; suggested nav group "Integrations", title "MLflow").Unchanged / not proven. Databricks, dual export, tracking-server backfill remain untested by design.
requirements.txtstays on 3.16.1.🤖 Generated with Claude Code