Skip to content

feat(examples): MLflow OTLP trace-export recipe and contract test (TH-8334) - #233

Merged
nik13 merged 16 commits into
devfrom
feat/th-8334-mlflow
Oct 26, 2026
Merged

nik13 merged 16 commits into
devfrom
feat/th-8334-mlflow

Conversation

@nik13

@nik13 nik13 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds an MLflow → Future AGI trace-export recipe at python/examples/mlflow/ (Linear TH-8334, decision D-8334). It is not a package and adds no instrumentor: MLflow 3.16.1 already exports its own spans over OTLP, so the recipe points that export at fi-collector with environment variables and adds the Future AGI project as a resource attribute. A wrapper around mlflow.start_span would double every span. Historical import from a tracking server, Databricks or Unity Catalog is out of scope.

  • src/app.py: two nested mlflow.start_span calls (answer_question CHAIN root, chat_model CHAT_MODEL child) around a canned stand-in model client, so it makes no model call. check_environment() exits with a message on every setting the tests found to export nothing silently. --drop-content registers a span processor through mlflow.tracing.configure.
  • requirements.txt: mlflow==3.16.1 plus opentelemetry-exporter-otlp-proto-http==1.45.0; mlflow installs no OTLP exporter.
  • README.md: run block, tested endpoint table, headers, resource/experiment, telemetry, dual export, set_destination, key inventory with the fi-collector column, the optional GenAI semconv switch, span types, content, SF-1 gap list, troubleshooting and the exact test command.
  • tests/: contract test (44 functions, 74 collected cases with the collector source; 71 + 3 opt-in skips without), a loopback network guard with a positive control, two app fixtures and a hand-built OTLP key fixture for post_otlp().

What the tests established (mlflow 3.16.1, loopback receiver)

  • Endpoint: MLflow passes OTEL_EXPORTER_OTLP_TRACES_ENDPOINT unchanged to the exporter (mlflow/tracing/utils/otlp.py:98-99); it does not append /v1/traces. /tracer/v1/traces and /v1/traces reach the receiver; the origin, origin/ and /tracer post to the wrong path (404, spans lost). The README publishes /tracer/v1/traces, the path traceAI's own exporter uses.
  • Silent failures the app refuses: protocol not http/protobuf (MLflow defaults to gRPC; with no gRPC exporter it hands out no-op spans and logs only at debug level); OTEL_EXPORTER_OTLP_ENDPOINT or ..._METRICS_ENDPOINT set (turns on gRPC metrics and every span fails at end()); OTEL_RESOURCE_ATTRIBUTES without project_name or with an item lacking = (MLflow re-parses it to {}). Added at 7b47514 (D-8334a): a trailing-slash endpoint (404, spans lost); an inherited general OTEL_EXPORTER_OTLP_HEADERS (sent instead of empty TRACES headers, so another vendor's key could reach Future AGI); TRACES headers missing the x-api-key or x-secret-key name; MLFLOW_TRACING_DESTINATION without dual export (traces go to that store instead, silently); a Databricks tracking URI without dual export (MLflow calls Databricks, fetches its model catalog from github.com to price model spans, and a UC-linked experiment replaces the OTLP export). The protocol is read the way MLflow reads it (TRACES, then general OTEL_EXPORTER_OTLP_PROTOCOL).
  • Headers: percent-encoded x-api-key/x-secret-key arrive intact, including a secret containing , and =; names arrive lower-case (fi-collector reads them case-insensitively, pkg/auth/middleware.go:45-46).
  • Resource: project_name, project_type=observe, service.name; no session.id or user.id.
  • Spans: one trace, CHAIN root + CHAT_MODEL child, status OK. Every attribute value is a JSON-encoded string (MLflow behaviour): gen_ai.request.model arrives as "\"gpt-4o-mini\"", tokens as "12".
  • Content: MLflow has no content switch. Inputs/outputs land in mlflow.spanInputs/mlflow.spanOutputs; --drop-content removes every marker in both semconv modes.
  • Dual export: off → OTLP only. On with no tracking URI → OTLP and a local ./mlflow.db with the same trace (a PRD assumption said OTLP-only; documented as a failure mode). set_destination(MlflowExperimentLocation("0")) → zero OTLP requests.
  • Telemetry: importing mlflow writes $HOME/.config/mlflow/telemetry.json; MLFLOW_DISABLE_TELEMETRY=true (added to the recipe) writes nothing. No telemetry network attempt was observed; the phone-home path is source reading.
  • Network: every child process runs under the guard; zero non-loopback attempts in the committed suite. The model-catalog fetch MLflow does when writing to a store, or with a Databricks tracking URI, is pointed at a loopback recorder or disabled; the Databricks refusal tests point the workspace at a loopback recorder too. With the Run block (OTLP only, non-Databricks tracking URI, MLFLOW_MODEL_CATALOG_URI empty) the recipe contacts only the OTLP endpoint (tested).

What fi-collector does with these spans (source reading at future-agi main 4af5338)

  • None of MLflow's own keys (mlflow.spanType, mlflow.llm.model, mlflow.chat.tokenUsage, mlflow.spanInputs/Outputs) are read; kind is unknown by default (converter.go:79-131).
  • The gen_ai.* keys the app sets are read, but the model value is copied verbatim with JSON quotes (adapter.go:317-323); token strings are parsed (:339-342).
  • With MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true MLflow emits plain gen_ai.* and gen_ai.operation.name, but drops every mlflow.* key and renames spans; the recipe leaves it off and the README states the trade-off.
  • SF-1 gap list (no code here): mlflow.spanType kind alias, mlflow.llm.* model/provider aliases, mlflow.chat.tokenUsage, mlflow.spanInputs/Outputs, JSON-decoding of MLflow gen_ai.* values, invoke_agent synonym.

Deviations from the spec (approved: Nikhil 2026-10-03 blanket; self-approved by Rick)

  • Experiment id is not on the export in 3.16.1 (processor/otel.py:95-105); the recipe documents mlflow.experiment_id=<id> in OTEL_RESOURCE_ATTRIBUTES (tested) and says it is not a Future AGI session.
  • Dual export with no tracking URI writes ./mlflow.db (provider.py:884-907), contrary to PRD J3/AC-05.
  • MLflow's default export carries no gen_ai.*/llm.* keys; model/token promotion is not asserted (SF-1).
  • Autolog (PRD J2) not run; the architecture's test list wins. Autolog keys are cited from source.
  • Only mlflow 3.16.1 (no previous minor); Python 3.10–3.13 all run.
  • Added MLFLOW_DISABLE_TELEMETRY=true and set MLFLOW_MODEL_CATALOG_URI="" in the Run block (no github.com catalog fetch). The Run block's header line stops if FI_API_KEY or FI_SECRET_KEY is empty, and it unsets OTEL_EXPORTER_OTLP_HEADERS and MLFLOW_TRACING_DESTINATION.

Review round 1 (pr-reviewer t_588813db and pr-verifier t_d06c1b98 at 997ac28: CHANGES_REQUESTED) → fixed at 7b47514

Decision D-8334a (approved: Nikhil 2026-10-03 blanket): the supported mode is OTLP-only export to Future AGI with a non-Databricks tracking URI; check_environment() refuses the environment settings listed in the README that would silently send nothing, send traces or keys elsewhere, or contact an outside service. It cannot see code-level set_destination()/set_tracking_uri() calls (documented), and it does not yet refuse invalid MLflow boolean values (V3 below). Rejected: supporting Databricks or dual export in this round (untested surface with real external calls) — documented as untested instead.

  • R1 (P2): README said OTLP-only export never fetches the model catalog; with a Databricks tracking URI MLflow prices model spans client-side and fetches it on every run. Now refused, and the Run block empties MLFLOW_MODEL_CATALOG_URI.
  • V1 (P2): an inherited MLFLOW_TRACING_DESTINATION silently routed every trace away from Future AGI. Now refused unless dual export is on, and unset in the Run block.
  • V2 (P3): the "nothing here talks to Databricks" claim was false with a Databricks tracking URI (REST call on first span; a UC-linked experiment replaces OTLP). Now refused and the README/docstring state the tested condition.
  • R3 (P3): trailing-slash endpoint refused; the general OTEL_EXPORTER_OTLP_PROTOCOL is accepted the way MLflow reads it.
  • R4 (P3): inherited general headers refused; TRACES headers must name both FI keys (names only checked, values never printed).
  • R2 (P3): README states the quoted model string defeats fi-collector pricing (cost 0) with the semconv switch off. R5 (P3): mlflow.traceTag.<key> added to the SF-1 gap list.
  • RED (tests first, c66da1e): on the 997ac28 app/README the new tests give 15 failures (all assertion failures), 56 passed, 3 skipped. They include a README-setup regression that runs only the Run block's own export/unset lines on top of a stale environment (stale destination, another vendor's header, a catalog recorder) without pre-seeding the variables under test.

Review round 2 (pr-verifier t_cfed1ab8 at 7b47514: VERIFIED, no blocking findings)

R1–R5, V1 and V2 resolved. New P3s, recorded as post-merge follow-ups (no third fix round on P3s alone; head stays 7b47514):

  • V3: an invalid MLflow boolean (MLFLOW_TRACE_ENABLE_OTLP_DUAL_EXPORT=''/yes/off, or an invalid MLFLOW_ENABLE_OTLP_EXPORTER) makes MLflow raise while building its provider; start_span logs at debug and hands out no-op spans, so nothing is exported, exit 0. check_environment() treats those values as "off". Follow-up: refuse values outside true/false/1/0.
  • V4: the README stale-environment test does not cover the unset OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_METRICS_ENDPOINT line (the app refuses those variables, but deleting that README line would not fail a test). Follow-up: seed both in the stale environment.
  • V5 (this body): stale test counts and a broader refusal claim than the code — fixed above.
  • Residuals noted by the verifier: Databricks model-serving environment not documented; exporter header-value edge cases.

Tests

Head 7b47514. Coordinator rerun from the repo root on a clean tree with the README command; actual exit codes and JUnit captured per cell (tickets/TH-8334/exact-head-7b47514-checked/):

env -u PYTHONPATH PYTHONPATH="python:python/tests" uv run --no-project --python <py> \
  --with pytest --with protobuf --with opentelemetry-proto --with wrapt --with jsonschema \
  --with 'mlflow==3.16.1' --with 'opentelemetry-exporter-otlp-proto-http==1.45.0' \
  pytest python/examples/mlflow/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs
              README command        FI_COLLECTOR_SRC=<fi-collector @ 4af5338>
Python 3.10   71 passed, 3 skipped  74 passed
Python 3.11   71 passed, 3 skipped  74 passed
Python 3.12   71 passed, 3 skipped  74 passed
Python 3.13   71 passed, 3 skipped  74 passed
All eight commands: actual exit 0, no failures, collection errors or timeouts; tree clean before and after.
  • 9 files, all under python/examples/mlflow/, 1,800 added lines.
  • Credential-value scan of the recipe, build evidence and branch history (3eaadc8..997ac28): 0 hits; fix-round evidence and recipe at 7b47514: 0 hits. FI keys in tests are placeholders.

Limits

  • Databricks tracking URIs and dual export with Databricks are refused/untested (source reading for what MLflow would call).
  • No fi-collector was run: 200/400/401, project stamping, alias promotion and storage are source reading. The harness receiver accepts anything on the two paths.
  • Autolog and @mlflow.trace were not run. gRPC transport not tested. Only SQLite tracking store (not ./mlruns). macOS arm64 only. Python 3.14 not run.
  • drop_content does not touch exception events on failing spans (source reading).
  • No CI job runs this example; the opt-in collector tests are run by hand.

Video demo

Recorded build: head 7b4751480f10d407ad4afe25eb2cbc69f13e6bae (this PR); 9 real terminal chapters of the demo driver from a clean read-only worktree plus intro and review cards; 8:20, 11 chapters, narrated with captions.

The video (SHA-256 f71b02b4e52a7f8ea677efa8fbfdde266e411e176e6adfbb665faa99bb449d6e), captions (SRT), transcript, chapter list, preview and both media-verification notes are private attachments on Linear issue TH-8334 (Future AGI workspace; sign-in required). They are not linked here as raw upload URLs.

Time Chapter
00:00 Problem: MLflow OTLP export to Future AGI
00:50 head: PR 233 at 7b47514
01:25 run: the README Run block then the app
02:24 silent: what the check prevents
03:11 refusals: D-8334a
04:03 stale-env: main before and after proof
05:11 destinations: writes beside OTLP
05:41 collector: source reading of fi-collector 4af5338
06:35 tests: live suite and recorded matrix
07:08 limits
08:02 Review status

What it shows: the README Run block's own 12 export/unset lines run in a bare bash, then the app exports 2 spans to /tracer/v1/traces with header names only and the project resource, the model arriving JSON-quoted ("gpt-4o-mini"), guard 0, no files, no catalog fetch; MLflow on its own exits 0 and loses the spans with default gRPC (0 requests, empty stderr) or a wrong/trailing-slash endpoint (404), and the app refuses each; six D-8334a refusals with 0 recorder requests, the general protocol accepted, and MLflow alone with a Databricks URI calling the workspace and the model catalog (loopback recorder); the main proof: the README Run block over a stale environment — the new README exports 2 spans with the FI keys and contacts nothing else, the old one (997ac28) exports 0 spans, writes ./mlflow.db and fetches the catalog 4 times; with the FI API key unset the new README stops at the header line and the old one sends the other vendor's header; dual export writes ./mlflow.db and set_destination() in code sends 0 OTLP; collector pricing as source reading (quoted model → cost 0); live Python 3.11 suite 71 passed + 3 skipped next to the recorded matrix and RED log; V3 reproduced live (an invalid MLflow boolean exports nothing, exit 0). Key, header and placeholder values are never printed.

Independent media checks (Rick): full audio/video decode exit 0; 11 embedded chapters match the list; the four pauses of 1.8–4.6 s sit inside live-command scenes (the longest checked frame by frame: output still printing); all chapter-end frames, the preview, the live test result and the pause frames inspected (legible, nothing under the captions, no key or header values, local paths or ports); 7 delivered files scanned with the driver's 15 fixture needles plus path, port, header and credential checks: zero hits. Driver provenance: the first driver worker stopped on a model quota limit after writing two scripts; Rick completed the runner, scanner and outcome assertions and ran all nine chapters twice on the final bytes (exit 0, byte-identical, clean source) before recording. Disclosed edits: static waiting cut (largest 90.5 s of live pytest), nothing sped up, long output paged from the same run's saved output. Not shown: a live model, Databricks, a running fi-collector, auth, storage or the trace view.

Stacked on #203 (shared harness, 3eaadc8), base dev. Not merged.

Linear: TH-8334. approved: Nikhil 2026-10-03 blanket

Verification sweep (2026-10-12)

  • The recipe's Run block now turns on MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true by default (types, model, tokens and cost appear in Future AGI); the README explains how to turn it off and what that trades.
  • Verified end to end: the real library made real calls, traces were exported to a local Future AGI stack built from dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).
  • Commits added in this sweep:
    • b93c37c docs(examples): say what MLflow autolog traces look like in Future AGI
    • 5ed53a6 docs(examples): note that the MLflow GenAI switch drops function names
    • 59483b8 feat(examples): turn on MLflow's OTel GenAI keys in the MLflow recipe
  • Merge order: merge test(harness): shared OTLP harness for TH-8103 contract tests #203 first (merge commit), and the README catalog PR docs: list the TH-8103 integrations in the README catalogs #250 last. Everything else is order-independent.

Re-verification (2026-10-16)

Versions. mlflow 3.17.1 (latest stable on PyPI, 2026-10-16) and the pinned 3.16.1; opentelemetry-exporter-otlp-proto-http 1.45.1 / 1.45.0; Python 3.12; real openai client; local Future AGI (OTLP at /tracer/v1/traces, uncompressed). Run block used as written (only origin, keys, project substituted).

Real E2E (mlflow 3.17.1, mlflow.openai.autolog() + @mlflow.trace AGENT/TOOL, model via an OpenAI-compatible proxy, gpt-6-luna). fi_verify PASS, project th8103-rick-mlflow-real:

trace 70c3e3df-8d25-eab4-b645-f27a8edfc081 (4 spans)
- [unknown] invoke_agent refund_agent
  - [llm] chat gpt-6-luna model=gpt-6-luna tok=360/21
  - [tool] execute_tool get_refund_window
  - [llm] chat gpt-6-luna model=gpt-6-luna tok=365/17
trace db841083-8a48-7fe4-3b6a-8c4debcf3614 (2 spans)   # failure case, model "no-such-model"
- [unknown] invoke_agent refund_agent  status=ERROR
  - [llm] chat  status=ERROR

Recipe src/app.py itself (project th8103-rick-mlflow-trace-export, trace 698333f4-851f-1824-40ea-e4b2d8874815): CHAIN over chat gpt-4o-mini (llm, model gpt-4o-mini, tokens 12/7). UI screenshot checked: trace tree, agent graph, tool and model spans, tokens. Agent spans stay kind unknown (existing SF-1 gap, invoke_agent synonym); input/output columns stay empty (existing SF-1 gap).

Contract test. Pinned 3.16.1: 75 passed, 3 skipped (opt-in fi-collector source). Latest 3.17.1: 75 passed, 3 skipped. Before my change 3.17.1 had 1 failure (test hard-coded telemetry.sdk.version == "3.16.1"); 3.16.1 had 72 passed.

Privacy (found and fixed). Plain MLflow exports, for a failed span, an exception event whose exception.stacktrace holds local file paths (so the user name) and the exception message, plus the exception text as the span status message. The README only listed this as "not covered". Fixed at 780ca7d:

  • drop_exception_events span processor, always registered by src/app.py (removes the exception events; status stays ERROR);
  • --drop-content / drop_content now also clears the status message (keeps ERROR);
  • new tests/failing_app.py + test test_error_text_and_stack_trace_are_dropped_and_status_stays_error (3 modes, with a positive control that plain MLflow exports exception.stacktrace, Traceback, the marker and the local path); the contract test version assertion now reads the installed mlflow version.
    Stored-column scan (every column of every span row of the project, ClickHouse): with defaults the unique prompt markers REALMARK_OK_8334 / REALMARK_FAIL_8334 are present (attrs_string); with the processors on, HIDEMARK3_OK_8334, HIDEMARK3_FAIL_8334, Traceback, /Users/, the user name, the hostname, the provider error text and the Future AGI key/secret have 0 hits (6 rows, project th8103-rick-mlflow-hide3, failure trace included). Autolog's tool schemas (tools, gen_ai.tool.definitions) are still exported; they are developer code, documented. MLflow has no content switch: content is on by default and removed only by the processor (README "Content").

Docs. future-agi/docs#1005 (draft, base main, page src/pages/docs/integrations/mlflow.mdx, no navigation.ts; suggested nav group "Integrations", title "MLflow").

Unchanged / not proven. Databricks, dual export, tracking-server backfill remain untested by design. requirements.txt stays on 3.16.1.

🤖 Generated with Claude Code

nik13 and others added 11 commits October 3, 2026 18:58
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends
no Content-Length. The Receiver read Content-Length only, so every Node
export came back 400 with zero spans; two TH-8103 children had to put a
de-chunking relay in front of it. Receiver now accepts both framings.

It also records one entry per accepted export (path, lower-cased
headers, flattened resource attributes) via requests(), so a contract
test can assert X-Api-Key/X-Secret-Key and project_name through the
shared harness instead of a private recorder.

Verified: 6 harness tests pass, and the TanStack example's real Node
exporter delivered 4 chunked exports (4 spans, collector path, both
auth headers, project_name/project_type) with no relay.

Refs: TH-8339, TH-8103
No traceAI package: MLflow 3.16.1 already exports its spans over OTLP when
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. src/app.py makes two nested
mlflow.start_span calls (a CHAIN root and a CHAT_MODEL child with gen_ai.*
and mlflow.llm.* model/token keys) around a canned stand-in model client,
so it makes no model call.

check_environment() exits with a message on the settings that make MLflow
export nothing without an error: no full traces URL (MLflow does not append
/v1/traces, utils/otlp.py:98-99), a protocol other than http/protobuf (the
default is gRPC, otlp.py:124-136), OTEL_EXPORTER_OTLP_ENDPOINT or
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT (they turn on gRPC metrics, which drop
every span), and OTEL_RESOURCE_ATTRIBUTES without project_name or with an
item MLflow cannot parse (provider.py:716-731).

--drop-content registers a span processor through
mlflow.tracing.configure; MLflow has no content-off switch.

requirements.txt pins mlflow==3.16.1 and adds
opentelemetry-exporter-otlp-proto-http==1.45.0, which mlflow does not
install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
- _guarded_run.py blocks and logs every non-loopback connect, connect_ex
  and getaddrinfo before the script runs (same guard as the OpenLLMetry
  recipe); _guard_probe.py is its positive control against TEST-NET-1,
  the IPv6 documentation prefix and .invalid.
- bare_span.py runs the app's spans without check_environment(), so the
  test can record what MLflow itself does with the settings the app
  refuses.
- set_destination_app.py calls mlflow.tracing.set_destination before the
  app, for the precedence test.
- fixtures/gen_ai_span.otlp.json is a hand-built OTLP request with two
  gen_ai.* spans for post_otlp(): the shape MLflow sends with
  MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true and its default JSON-encoded shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
Runs src/app.py as written with the real mlflow 3.16.1 in a subprocess,
under the loopback guard, with a fresh temp dir as cwd and HOME so every
file MLflow writes is checked. Spans go to the harness Receiver; path and
catalog checks use a loopback catch-all.

Covered: /tracer/v1/traces and /v1/traces both 200 and MLflow does not
append /v1/traces (origin, origin/ and /tracer get a 404); no protocol
means silent no-export; a base or metrics endpoint drops every span;
percent-encoded x-api-key/x-secret-key with ',' and '=' arrive intact;
project_name/project_type on the resource, lost on a trailing comma;
MLflow JSON-encodes every attribute value (gen_ai.request.model arrives
with its quotes); no experiment id or session on the export, and an
experiment id passed in OTEL_RESOURCE_ATTRIBUTES arrives as a resource
attribute; content in mlflow.spanInputs/spanOutputs and the drop-content
processor; dual export unset/false/true with and without a tracking URI
(./mlflow.db checked by trace id) and its model-catalog fetch;
set_destination silences OTLP entirely, and with dual export sends both;
the telemetry switch; MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true output; the
hand-built gen_ai.* fixture via post_otlp() and its match with live runs.

Opt-in (FI_COLLECTOR_SRC): fi-collector's alias tables include the gen_ai.*
keys and no mlflow.* key, and the README's collector claims and span-type
table match adapter.go/converter.go and MLflow's translator.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
Live OTLP export only: states that historical import is not this
integration and that an experiment id is a resource attribute, not a
Future AGI session. Publishes one endpoint form,
<collector origin>/tracer/v1/traces (PRD O-2: MLflow uses
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT as is), with the tested path table.

Documents dual export both ways (no tracking URI means a local ./mlflow.db,
not OTLP-only), set_destination precedence, the model-catalog fetch,
telemetry, MLflow's JSON encoding of attribute values, the key inventory
with what fi-collector reads (adapter.go/converter.go at 4af5338, source
reading), the optional MLFLOW_ENABLE_OTEL_GENAI_SEMCONV switch and its
costs, content capture and the span-processor hook, and the SF-1 gap list.
Marks what was tested and what was read from source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
The model-catalog test asserts GET /catalog/openai.json requests, not a
count; the count of four came from exploration runs. Say only what the
test checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
…ns and headers

Tests first for review-r1/verify-r1 (R1, R3, R4, V1, V2). On the unchanged
app and README at 997ac28 these fail with assertion failures (15 on
Python 3.12):

- check_environment() must refuse, before the first span and with zero
  requests to a loopback catch-all that also stands in for the model
  catalog and DATABRICKS_HOST: MLFLOW_TRACKING_URI=databricks and
  databricks://profile, MLFLOW_TRACING_DESTINATION=0 (dual export unset or
  false), a trailing-slash traces endpoint, a general
  OTEL_EXPORTER_OTLP_HEADERS, unset or empty TRACES headers, and TRACES
  headers without x-secret-key. No header value may appear in the output.
- OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf with the TRACES protocol unset
  must export normally.
- The README Run block's own export/unset lines, run by bash over a stale
  environment (MLFLOW_TRACING_DESTINATION=0, a general other-vendor header,
  a catalog URI on a loopback recorder) and with no recipe variable
  pre-set, must deliver both spans with the FI keys to the intended
  receiver, leave no stale value, write no file and fetch no catalog. All
  those lines must come before the app line.
- The README header line must stop, in a script and when pasted into an
  interactive bash, when FI_API_KEY is unset or FI_SECRET_KEY is empty.

Vendor-behaviour tests that pass already: MLflow posts a trailing-slash
endpoint verbatim (404); MLFLOW_TRACING_DESTINATION silences OTLP like a
set_destination call and only logs INFO lines; with dual export it sends
both; with MLFLOW_TRACKING_URI=databricks and dual export off MLflow still
fetches /catalog/openai.json and, with an experiment id, GETs
/api/2.0/mlflow/experiments/get, all on loopback with the guard log empty;
upper-case header names arrive; TRACES grpc over a general http/protobuf
and a general grpc are refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
check_environment() now also exits, before the first span and with one
message each (decision D-8334a):

- MLFLOW_TRACKING_URI=databricks or databricks://... with dual export off
  (R1, V2): MLflow then reads the experiment from the workspace on the
  first span, fetches its model catalog (github.com by default) to price
  the model span, and replaces OTLP with Unity Catalog for a linked
  experiment.
- MLFLOW_TRACING_DESTINATION set with dual export off (V1): it replaces
  the OTLP export with no message.
- A trailing slash after /v1/traces (R3): MLflow posts it verbatim and
  fi-collector's exact patterns answer 404. The rstrip("/") is gone.
- OTEL_EXPORTER_OTLP_HEADERS set, or OTEL_EXPORTER_OTLP_TRACES_HEADERS
  without both x-api-key and x-secret-key (R4): the exporter falls back to
  the general headers when the TRACES ones are empty. Names are matched
  case-insensitively as the exporter parses them; values are never
  printed.

The protocol is now read as MLflow reads it (TRACES, then
OTEL_EXPORTER_OTLP_PROTOCOL, default grpc), so a general http/protobuf is
accepted (R3). Dual export is parsed as MLflow parses booleans (true/1).

The README Run block exports MLFLOW_MODEL_CATALOG_URI="" and unsets
OTEL_EXPORTER_OTLP_HEADERS and MLFLOW_TRACING_DESTINATION, and its header
line stops behind ': "${FI_API_KEY:?}" "${FI_SECRET_KEY:?}" &&' when a key
is empty, instead of exporting an empty value. The app docstring states the
tested condition, and the comment on the gen_ai.* keys says the quoted
model is not priced (R2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
…and header refusals

README prose for review-r1/verify-r1, following D-8334a:

- R1: the catalog paragraph now states the true condition. With dual
  export off and a non-Databricks tracking URI MLflow does not fetch the
  catalog (tested: no connection attempt with the default URI); with a
  Databricks tracking URI it does (tested on loopback). The recipe table
  lists MLFLOW_MODEL_CATALOG_URI empty and the variables that must be
  unset, and says which of them src/app.py checks.
- V2: a "Databricks tracking URI" section: the client-side pricing and
  catalog fetch, the experiment read on the first span (both tested with a
  loopback workspace and catalog), the Unity Catalog replacement of OTLP
  (source reading), the refusal, and dual export with Databricks as out of
  scope and untested, with the calls it makes from source. The intro now
  states the tested condition for "contacts only the OTLP endpoint".
- V1: MLFLOW_TRACING_DESTINATION next to set_destination (tested silence
  and dual export), and that a set_destination call in code cannot be
  seen by check_environment().
- R3: the trailing-slash row in the endpoint table, and the
  OTEL_EXPORTER_OTLP_PROTOCOL fallback.
- R4: the general-headers fallback, the refusal, and why the header line
  starts with ': "${FI_API_KEY:?}" "${FI_SECRET_KEY:?}" &&'.
- R2: the quoted model gets no token price (cost 0); "cost not priced" in
  the SF-1 JSON-decoding bullet.
- R5: mlflow.traceTag.<key> in the SF-1 list and in what the drop-content
  processor does not cover.
- Troubleshooting and the Tests section follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8334
@nik13
nik13 requested a review from NVJKKartik October 10, 2026 08:22
nik13 and others added 4 commits October 11, 2026 20:04
A live run with mlflow.openai.autolog() against fi-collector (future-agi
dev ea12cc3) showed that, under the Run block (GenAI semconv switch off),
autolog spans arrive as a tree of unknown spans with no model, tokens or
cost: autolog writes only mlflow.llm.model and mlflow.chat.tokenUsage,
which fi-collector does not read. The gen_ai.* keys the recipe app sets
by hand hide this. With MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true the model
calls are llm spans with model and tokens, tools are tool spans and
chains chain spans.

Add that result and tell autolog users to turn the switch on, and mark
the recipe app's collector behaviour as seen live, not only read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true, the live autolog run's tool
and agent spans were renamed execute_tool and invoke_agent and carried no
gen_ai.tool.name or gen_ai.agent.name, so the function names reached
Future AGI nowhere. Say so next to the advice to turn the switch on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decision (Nikhil, 2026-10-11): the Run block now exports
MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=true. With it, fi-collector types MLflow's
spans (chat -> llm, execute_tool -> tool, CHAIN -> chain) and reads a plain
model name, token counts and cost; without it every span is unknown and an
autolog model call carries no model or tokens. The cost, renamed spans and
lost function names, is documented with how to turn the switch off.

README: "What is exported" now leads with the switch-on export, moves the
JSON-encoded inventory under "Turning the switch off", and records the live
run of the Run block (recipe app and real autolog) against fi-collector.

Tests: the recipe environment includes the switch; recipe_run is the
translated export (CHAIN > chat gpt-4o-mini) and a switch_off_run fixture
keeps the JSON-encoded key checks. The Run block test starts from a stale
MLFLOW_ENABLE_OTEL_GENAI_SEMCONV=false and requires the block to set it.
MLflow's trace id is now derived from the OTLP trace id, since the switch
drops mlflow.traceRequestId; the switch-off run checks the two agree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ecipe export (TH-8334 re-verification)

Plain MLflow exports an exception event with the stack trace (local paths)
and the exception text as the span status message. src/app.py now always
drops the exception events; --drop-content also clears the status message
and keeps ERROR. Contract test no longer hard-codes the mlflow version
(passes on 3.16.1 and 3.17.1); new test with a positive control.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13
nik13 merged commit 2ab5418 into dev Oct 26, 2026
4 checks passed
@nik13
nik13 deleted the feat/th-8334-mlflow branch October 26, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant