Repository navigation
feat(examples): EverOS native OTel tracing recipe and contract test (TH-8329) - #232
Merged
Merged
Conversation
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103
tests/loopback_guard/sitecustomize.py is the Cognee recipe's guard: loaded through PYTHONPATH in every process a scenario starts (the scenario and the `everos init` it runs), it refuses and logs any connect/connect_ex/ getaddrinfo to a host other than 127.0.0.1, ::1 or localhost. tests/_guard_probe.py is its positive control (TEST-NET-1, 2001:db8::1, example.invalid). tests/_fake_openai.py answers the chat and embeddings calls EverOS 1.4.1 makes through everalgo in one add/flush/search run: memcell boundary detection, episode extraction and atomic-fact extraction, with 1024-wide embeddings (EverOS's LanceDB width). One switch makes the atomic-fact reply malformed, for the error-text test. approved: Nikhil 2026-10-03 blanket Refs: TH-8329
tests/everos_session.py runs `everos init`, then drives EverOS's own app (create_app(), what `everos server start` serves) in-process with Starlette's TestClient: no port is bound, no server or HTTP wrapper is started, and the app's own lifespan installs EverOS's tracer from the [observability] settings. It adds three messages, flushes, waits for the episode to be indexed through /api/v2/memory/get (which opens no span) and runs one hybrid search. everalgo counts tokens with tiktoken's o200k_base, which tiktoken downloads from openaipublic.blob.core.windows.net on first use (the guard caught it); the script registers a byte-level stand-in under that name. tests/everos_config_probe.py exports one everos.memory.search span through everos.config.load_settings and init_tracing/shutdown_tracing, to test the endpoint, header and resource settings without the whole app. approved: Nikhil 2026-10-03 blanket Refs: TH-8329
EverOS (EverMind-AI/EverOS, PyPI everos; not EverMemOS Cloud or the legacy Claude Code plugin) exports its own OTLP spans. The recipe pins everos[otel]==1.4.1 (latest on PyPI 2026-10-05) and sets EverOS's own [observability] keys (settings.py:624-668): enabled, endpoint (used as given, so the full /tracer/v1/traces path) and headers (a toml table or a JSON EVEROS_OBSERVABILITY__HEADERS), plus OTEL_RESOURCE_ATTRIBUTES for project_name/project_type, since EverOS has no resource setting. Content capture stays off. The README records the spans 1.4.1 emits (no architecture name renamed), that fi-collector derives kind "unknown" for all of them (converter.go:79-131) so the architecture's kind table is an SF-1 gap, why no processor is written, model/token/cost behaviour, and what content capture does and does not cover. requirements.txt pins the tested OpenTelemetry 1.45.0. approved: Nikhil 2026-10-03 blanket Refs: TH-8329
The fixtures are EverOS 1.4.1's own spans from the live scenario, as the harness Receiver decoded them: capture off, capture on, and the capture-off run's resource. Fixture tests (no EverOS needed) post each with post_otlp(), check it with compare(), and assert: span names and tree, langfuse.observation.type per span, no fi-collector kind key and no cost key, gen_ai model/token keys only on generation and embedding spans, no query or other content with capture off, the query on everos.memory.search with capture on, project_name and project_type=observe on the resource, and no keys or host paths. Live tests (when everos is installed) run the recipe, capture-on and a rejected-model-reply scenario under the loopback guard and match them to the fixtures, check the export path, both auth headers and the resource, and probe the toml block, the default-off switch, the OTel header and endpoint variables, an endpoint without its path and a missing OTEL_RESOURCE_ATTRIBUTES. Re-record with EVEROS_RECORD_FIXTURE=1. 33 passed on Python 3.12.10 and 3.13.15; without everos, 17 passed and 16 skipped (3.12). approved: Nikhil 2026-10-03 blanket Refs: TH-8329
…ummary verify-r1 V1: with capture off, EverOS 1.4.1 exports a provider's error body as span status and exception text. The loopback OpenAI fake can now answer an embeddings request that contains a given text with HTTP 400 and an OpenAI-style error body quoting the rejected input, followed by a marker at character 4200 of the message. A fourth live scenario, embedding_error, runs the recipe's add/flush/hybrid search with the fake rejecting the query's embedding. The test pins what EverOS does today: the search answers 422; everos.embedding, everos.search.recall and everos.memory.search carry ERROR status descriptions and exception events (message and chained stack trace) holding the query and the late marker, uncut past 4096 characters; neither marker is in any span attribute or the resource, and no span has a content key. It passes on the current code by design. README tests, failing until the README changes: - the error-text caveat must name the channels, those spans and everos.ome.extract_atomic_facts, state that capture_content = false, set_redactor and the 4096 cap do not apply (attributes.py:61-77), cite the embedding and LLM provider error paths and both error tests, and recommend keeping tracing off for sensitive inputs; the capture-off and never-exported bullets must link to it. - review R1: the collector summary's Model and Tokens bullets must cover every span the capture-off fixture gives a model and tokens, naming the agent-typed everos.ome.extract_atomic_facts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8329
…ge summary verify-r1 V1: the Privacy section named only an extraction error quoting a model reply. It now has an "Error text is not covered by the switch" subsection: OpenTelemetry records an exception's message as the ERROR status description and in an exception event (message and chained stack trace, with installation paths) on the failing span and every EverOS span it propagates out of; capture_content = false, set_redactor and the 4096-character cap act only on the two content attributes (attributes.py:61-77), and the server's own provider has no filter setting. Provider error bodies kept by EverOS (embedding openai_provider.py:39-49, 121-134; LLM openai_provider.py:106-109) are the main case: the embedding_error test shows a quoted query and a marker 4200 characters in on everos.embedding, everos.search.recall and everos.memory.search with capture off. The section recommends keeping tracing off for sensitive inputs until EverOS or a controlled collector strips exception events and status, and says this recipe adds no filter. The capture-off and never-exported bullets are limited to the tested runs and link to it; the setup text no longer implies the switch is enough. review R1: the Model and Tokens bullets said generation and embedding spans only. They now cover the span current at each recorded call, including the agent-typed everos.ome.extract_atomic_facts (extract_atomic_facts.py:38-50, runner.py:229-251, _usage_client.py:44-57). The Tests section lists the embedding_error run and the new totals: 37 tests, 20 fixture and 17 live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8329
…pans
The README left the collector side as source reading ("whether that
adapter runs ... was not established") and said nothing about Input,
Output, sessions or users. A live run (everos server start configured as
in Set up, real model and embeddings, fi-collector from future-agi dev)
settles them:
- every span is stored as unknown; the backend Langfuse adapter does not
run on fi-collector-ingested spans;
- Input/Output stay empty in both capture settings (captured content is
only in langfuse.observation.* attributes, which fi-collector does not
read);
- langfuse.session.id / langfuse.user.id do not reach Sessions/Users;
- a failed chat call's span carries no model or tokens, and embedding
spans carry tokens only when the provider reports usage;
- unknown-model cost is stored as 0; no project_name -> 400, bad keys
-> 401, endpoint without its path -> 404.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ture AGI run Re-verification 2026-10-16: 1.4.2 (latest on PyPI) exports the same spans as 1.4.1; live tests now accept 1.4.1 and 1.4.2 and fail on anything else. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status 2026-10-21: merged
devinto the branch (4880866, no conflicts). CI: Plan, fi_instrumentation and Python tests are green. This is a recipe under python/examples, not a package, so no package job runs. Local recipe suite with the README command (Python 3.12, everos 1.4.2): 37 passed. Nothing to publish. Note: everos 1.5.0 was released on 2026-10-21 and is untested. The recipe and docs#1004 pin 1.4.2, and the live tests accept only 1.4.1/1.4.2.Summary
This PR adds an EverOS → Future AGI tracing recipe at
python/examples/everos/(Linear TH-8329, decision D-8329). There is notraceai-everospackage. EverOS (EverMind-AI/EverOS, PyPIeveros, Apache-2.0) has exported its own OpenTelemetry spans since 1.2.0, off by default. This recipe only turns that export on and points it at fi-collector. It adds no instrumentor, no span processor and no HTTP wrapper.Recipe. The settings go in EverOS's own
[observability]keys, as env vars or in~/.everos/everos.toml(config/settings.py:624-668):EVEROS_OBSERVABILITY__ENABLED=trueEVEROS_OBSERVABILITY__ENDPOINT=https://api.futureagi.com/tracer/v1/traces. The full path is required because EverOS uses the value verbatim (provider.py:134-138); a bare origin posts to/and gets a 404 (tested).EVEROS_OBSERVABILITY__HEADERSas JSON withX-Api-Key/X-Secret-Key. Values containing,,=or spaces need no encoding (tested).OTEL_RESOURCE_ATTRIBUTES=project_name=…,project_type=observe. EverOS has no resource setting, so this is the only way to set the project (provider.py:127-129).Tests and fixtures. The fixtures are a live recording of EverOS 1.4.1's own spans, not a hand-built body.
tests/everos_session.pyrunseveros init, then EverOS's owncreate_app()(whateveros server startserves) through Starlette's in-processTestClient:EVEROS_OBSERVABILITY__*;The recordings, one with capture off and one with capture on, are posted with the harness
post_otlp()and checked withcompare().What arrives, and what fi-collector does with it (source reading at future-agi
main4af5338)everos.memory.add,.flush,everos.memcell.boundary,everos.extract,everos.persist.markdownservice/memorize.py:211-260,user_memory.py:117-178unknowneveros.memory.search,everos.search.recall,everos.search.rankmemory/search/manager.py:192-676unknowneveros.embedding,everos.ome.<strategy>openai_provider.py:121-123,ome/_dispatch/runner.py:229-238unknownunknownfor every span. fi-collector readsfi.span.kind,gen_ai.span.kind,llm.request.typeandopeninference.span.kind, then falls back togen_ai.operation.name(exporter/clickhouse25exporter/converter.go:79-131). EverOS sets none of them; the tests assert this. See D2 below.gen_ai.request.modelandgen_ai.usage.input_tokens/output_tokensare on the alias lists (pkg/adapter/adapter.go:279-310).component/llm/client.py:111-114).everos.ome.extract_atomic_factsspan, because the usage wrapper writes to whatever span is current (_usage_client.py:53-57).converter.go:285-294). That was not tested against a collector.Privacy
langfuse.observation.input/output(core/observability/tracing/attributes.py:68-77).d602c40). OTel records any exception as an ERROR status description and anexceptionevent (message plus chained stack trace) on the failing span and every EverOS span it propagates out of.capture_content,set_redactorand the 4096-character cap don't apply to those channels. Two cases: provider error bodies (OpenAI's client puts the whole body in the message and EverOS keeps it,component/embedding/openai_provider.py:39-49, 121-134,component/llm/openai_provider.py:106-109; tested for embeddings: a 400 that quotes the query plus a marker 4,200 characters in lands in status and events ofeveros.embedding,everos.search.recallandeveros.memory.search, in no attribute; the chat path is source reading), and a model reply quoted in an extraction error (tested). The README advises keeping tracing off for inputs that must not leave the process.set_redactor(attributes.py:46, 55-58). The recipe cannot attach a span processor: EverOS builds a module-local provider and processor, not the OTel global (provider.py:130-139,tracing_lifespan.py:42). So the README documents these exceptions instead of filtering them.Deviations from the spec (approved: Nikhil 2026-10-03 blanket)
post_otlp()+compare()are still used.langfuse.observation.type(attributes.py:126). Mapped the way the backend Langfuse adapter maps it (futureagi/tracer/utils/adapters/langfuse.py:40-51), it would give the architecture's kinds. The exception iseveros.search.rank, which EverOS typesspan(CHAIN), not RETRIEVER.everos.memory.flush,everos.persist.markdownandeveros.ome.<strategy>.everos.memory.bufferandeveros.reflect.consolidateexist in source only.OTEL_EXPORTER_OTLP_*also works; it posts to<base>/v1/tracesand merges per header (tested). The docs draft is corrected in company-brain Dev #63.==1.4.1, the only version tested; the 1.2.0 floor is stated as untested. Python 3.12 and 3.13 only, because EverOS requires >=3.12 (METADATA:22).Review round 1 (pr-reviewer t_8be03933 APPROVE; pr-verifier t_049d8fb2 CHANGES_REQUESTED at
62b2d18)V1 (P2), fixed at
d602c40(D-8329a, approved: Nikhil 2026-10-03 blanket): the capture-off caveat named only extraction failures; provider error bodies are a wider channel (see Privacy). Decision: document and pin, no filter. Rejected: a recipe-local export filter around EverOS's own provider (EverOS builds a module-local provider/processor, so it would be a new instrumentation component beyond D-8329, untested against EverOS internals) — recorded as a follow-up option.R1 (P3), fixed: the README said tokens are on generation and embedding spans only; the agent-typed
everos.ome.extract_atomic_factsalso carries the call's model and tokens. A fixture-driven README test now checks the Model/Tokens bullets name every span that carries them.RED (tests before docs): on the
62b2d18README the three new README tests fail (Model, Tokens, error-text caveat); 34 passed. The newembedding_errorlive run passes before and after by design: it pins existing EverOS behaviour, which the README now documents.Round 2: pr-verifier t_32c56149 VERIFIED at
d602c40, no blocking findings; V1 and R1 resolved. V2 (P3, this body's capture-off wording) fixed here. Optional follow-ups: check the two error runs for the secret/message markers too (source tracing shows neither fake error path quotes them); the recipe-local status/event filter stays a rejected follow-up option.Review round 2 (pr-verifier t_32c56149 at
d602c40: VERIFIED)V1 resolved: the README states that error text (provider error bodies, extraction replies) is exported in ERROR status,
exception.messageand stack traces in every capture setting and past the 4,096-character cap; tests pin it on EverOS 1.4.1 / OTel 1.45.0, and the README test fails 17/17 on62b2d18. R1 resolved. No blocking findings. V2 (P3, wording) fixed in this body's Privacy section. Optional follow-ups retained: the two error runs are not separately scanned for the placeholder keys or message marker (source tracing shows neither fake error path quotes them), and a recipe-local status/event filter stays a rejected option under D-8329a. Companion docs and architecture corrected to this head in company-brain #63 (bdfcea9, open).Tests
Head
d602c40. Coordinator rerun from the repo root on a clean tree, in a scrubbed environment, with real process exit codes and JUnit (tickets/TH-8329/exact-head-d602c40-checked/results.json):python/examples/everos/(the fix round touched README, tests and the OpenAI fake). 0 secret-like added lines, and 0 host paths in the tracked fixtures.o200k_base. The guard caught tiktoken's download fromopenaipublic.blob.core.windows.net, and token counts only feed everalgo's boundary limits, never a span.Limits
unknownkind, pricing, 401, project stamping and storage are source reading.everos.ome.*spans are timing-dependent, so the live-vs-fixture check tolerates missing ones. Request-path spans must match exactly.TestClient, not uvicorn./api/v1alias;OTEL_EXPORTER_OTLP_TRACES_ENDPOINT;langfuse_*settings andsample_rate;otelextra.Video demo
Recorded build: head
d602c400b46dc3d2fe08b2b10a247c606240856f(this PR); 10 real terminal chapters of the demo driver from a clean read-only worktree plus intro and review cards; 6:54, 12 chapters, narrated with captions.The video (SHA-256
838bf2a73fac2f62715835156abf523ef7fbba41b9d9ebc826ca42cb4b9814be), captions (SRT), transcript, chapter list, preview and both media-verification notes are private attachments on Linear issue TH-8329 (Future AGI workspace; sign-in required). They are not linked here as raw upload URLs.What it shows: real EverOS 1.4.1 in-process against the recipe's loopback OpenAI fake, exporting its own spans to
/tracer/v1/traceswith header names only;everos.ome.extract_atomic_factscarries model and tokens (R1); the fixture compare passes and a one-value change fails; every span's kind isunknownuntil SF-1 (source reading); capture off exports no content in a run where nothing fails, capture on adds exactly the four documented attributes; the main D-8329a proof: an embedding 400 that echoes the query puts the query marker and a tail marker at character 4200 (past EverOS's 4,096 cap) in the ERROR status,exception.messageand stack trace ofeveros.embedding,everos.search.recallandeveros.memory.search, and in no attribute or resource; the rejected extraction reply lands onextract_atomic_facts; the README section is present atd602c40and its test fails 17/17 on62b2d18. Marker text and header values are never printed, only letters, counts and key names.Independent media checks (Rick): full audio/video decode exit 0; 12 embedded chapters (one sidecar time corrected 03:47 → 03:48 to match the MP4); no audio gap of 1.5 s or more anywhere; every chapter-end frame and the preview inspected (legible, nothing under the captions, no key or header values, local paths or ports); 7 delivered files scanned with the driver's 19 fixture needles plus path, port, header and credential checks: zero hits. Disclosed edits: only static waiting shortened and final real frames held; long output paged from the same live run's saved output (labelled). The recorder started on Claude Opus 5.5 and finished on the configured fallback (GPT-6.1 Sol via Codex). The closing card says the company-brain #63 docs correction is in progress; it has since been verified and pushed (
bdfcea9), and no product code changed after recording. Not shown: a live model or embedding provider, EverMind cloud, a running fi-collector, auth, storage or the trace view.Stacked on #203 (shared harness,
3eaadc8), basedev. Not merged.Linear: TH-8329. approved: Nikhil 2026-10-03 blanket
Verification sweep (2026-10-12)
dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).Re-verification (2026-10-16)
Head
a817a6d(new commit on top of730afe9; nothing rewritten). Docs page: future-agi/docs#1004 (draft, base main, no navigation.ts; suggested nav: Integrations > traceAI > "EverOS").Versions. Latest stable
everoson PyPI is 1.4.2 (released 2026-10-14); previously pinned 1.4.1. OpenTelemetry 1.45.0. Both versions run on Python 3.12; 1.4.2 also on 3.13.What was wrong. Only the pin: 1.4.2 existed, and on it
test_recipe_resource_is_an_observe_projectfailed (it hard-codedservice.version == 1.4.1). The wheels'core/observability/**andtracing_lifespan.pysources are identical between 1.4.1 and 1.4.2; 1.4.2 changed persistence (LanceDB/index) and the embedding provider. No span renamed.What changed (commit a817a6d).
requirements.txtpinseveros[otel]==1.4.2; live tests accept exactly 1.4.1 and 1.4.2 and assert the resource'sservice.versionequals the installed version (any other version fails loudly); README updated (versions, a "Real Future AGI run" note). No change to recipe settings.Tests.
pytest python/examples/everos/tests: 37 passed on 3.12 with 1.4.2; 37 passed on 3.12 with 1.4.1; 37 passed on 3.13 with 1.4.2; without everos installed 20 passed, 17 skipped. Before the test change: 1.4.2 36 passed, 1 failed (above).Real E2E against local Future AGI (recipe exactly as the README:
EVEROS_OBSERVABILITY__*env, endpointhttp://localhost:14338/tracer/v1/traces, project viaOTEL_RESOURCE_ATTRIBUTES;everos server start; real chat model gpt-6-luna via a local OpenAI-compatible proxy; real embeddings gemini-embedding-001; one add, flush, search over HTTP):1.4.2 capture-off trace ids: add
1ed90493-51dd-2a53-6dc9-d643ccb6a779, flush7527966d-e14c-26da-966b-61010f125aed, searchcadc9e02-5d76-a275-01c4-49c1fed4a8d3. Failure search trace8a6cd544-5115-2cf3-f342-24b7e1009475.fi_verify excerpt (1.4.2, off):
everos.memory.flush>everos.memcell.boundary model=gpt-6-luna tok=1509/114,everos.extract model=gpt-6-luna tok=2051/295,everos.persist.markdown,everos.ome.extract_atomic_facts tok=1531/230,everos.ome.trigger_profile_clustering>everos.embedding model=gemini-embedding-001; search:everos.memory.search>everos.search.recall>everos.embedding,everos.search.rank. Hierarchy correct, model and tokens filled where EverOS records them. Failure:everos.embedding,everos.search.recall,everos.memory.searchstored ERROR withEmbeddingInputError: embedding input rejected (400).... fi_verify PASS on all four. All spans stored with kindunknown(as documented; SF-1).Privacy proof (unique marker in the message and the query; every ClickHouse
spanscolumn scanned, incl.resource_attrs,attrs_string,span_events): capture off on 1.4.2 and 1.4.1: 0 hits for the marker and for the message text. Capture on (1.4.2): marker inattrs_stringofeveros.memory.searchandeveros.extractonly (the four documented attributes). No FI keys, model keys, home paths or username in any stored row (all four runs); the error run's marker count is 0 (its rejected input was not quoted by that provider). The error-text caveat in the README (error status/event text is exported in every capture setting) is unchanged and still true.UI. The Observe UI trace page never finished rendering on the shared stack (trace-detail API answered 503 "temporarily unavailable" under host load average ~34; ClickHouse hit its 1.46 GiB cap on a wide scan); screenshots show only loading states, so the UI check is NOT proven. ClickHouse rows are the evidence.
Not covered.
/api/v1alias,OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, library use without the server, 1.2.0 to 1.3.x, a Python 3.10/3.11 run (EverOS needs 3.12+), a hosted Future AGI endpoint.Verdict: Rick's recipe was correct; fixes were the version pin and the version-pinned test.