Skip to content

ci: run each touched Python package's tests on every PR, and fix fi_instrumentation's tests (TH-8673) - #274

Merged
khushalsonawat merged 12 commits into
mainfrom
ci/TH-8673-python-tests
Oct 19, 2026
Merged

khushalsonawat merged 12 commits into
mainfrom
ci/TH-8673-python-tests

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

What does this PR do?

It gives traceAI CI for its Python packages.

On every PR into main (and on every push to it), a workflow runs fi_instrumentation's tests and the tests of each Python package the PR touches. Each package runs on its own dev dependencies. One check, Python tests, sums up the result; it is the one to mark required.

The workflow is also set to run for dev. GitHub runs the workflow found in a PR's merge commit (or in the pushed commit), and dev has no workflow today. So until main is merged into dev after this PR merges, a PR into dev runs it only if its own branch carries the file, and a push to dev runs nothing (see "Branch protection" below).

Packages whose tests already fail on main are named, each with its reason, in a visible list, .github/python-tests-known-broken.txt. They still run and show a warning, but don't fail the check. Every fix needed to get there is in this PR, as described below.

Why?

Linear: TH-8673. traceAI had no CI, so PRs merged with no checks.

What runs

  • fi_instrumentation (always): poetry install --all-extras in python/, then pytest tests. The 20 tests/test_framework_*.py files are not run here; each runs in its package's job, and the job log says so.
  • Each touched package (python/frameworks/<name>/, plus the traceai SDK in python/traceai/):
    • Install: poetry install from its lock, or pip install -e ".[dev]" for the 4 Hatch packages. pytest, pytest-asyncio and pytest-mock are added, at pinned versions, only where the package doesn't have them. Adding the pinned pytest-asyncio moves an existing pytest 9 to 8.4.2 in 5 packages (agno, beeai, milvus, redis-vector, strands), still inside what they declare.
    • Run: its own tests/, plus its file in python/tests/ if it has one. That file runs with python -P from python/, so the package is tested against its own fi-instrumentation-otel, not the repo's source.
    • A package with no tests says so in a notice; it is never skipped silently.
  • What counts as touched:
    • A change under python/frameworks/<name>/ or python/traceai/, or to python/tests/test_framework_<name>.py. A moved file counts for both the package it left and the one it entered (the diff runs with --no-renames, and -z keeps names with tabs, quotes or backslashes intact).
    • A change to python/fi_instrumentation/, python/pyproject.toml or python/poetry.lock runs every package, each with the repo's fi_instrumentation installed over its own. A change to python/tests/conftest.py or python/tests/pytest.ini runs every package that has a core test file.
    • A change to the workflow, to anything under .github/scripts/ (a new helper included) or to the list runs every package.
    • If a PR's base commit is missing from the clone, it is fetched; if that fails too, every package runs.
  • An empty plan cannot pass by accident. A skipped package job counts as green, so the planner fails (and Python tests with it) when the plan cannot be right: a PR or push range with no commits (wrong base or head SHA), --all with no files, a CI or core change that finds 0 packages, or a change inside a python/frameworks/<dir>/ that has no pyproject.toml. Commits that net to no change (an empty commit, a change and its revert) plan nothing, which is fine. After the planner, the Python tests job checks the package jobs against the planned matrix: if packages were planned they must have succeeded, so skipped jobs never read as green; and each package job fails if a test set it was given did not run. A change that touches no package is not an error: docs, other languages and the end-to-end folders rightly plan nothing, and fi_instrumentation's job always runs.
  • The planner and the verdict are tested first. The Plan job runs .github/scripts/test_python_test_plan.py (standard-library unittest, table-driven) before planning. It covers the planner's cases (a package change, a core change, a shared conftest change, a test_framework file, a move between packages, odd file names, bad list lines) and the known-broken verdict (.github/scripts/known_broken_verdict.sh) under every outcome. The verdict fails on any Tests outcome it does not know (for example a renamed step id), and a text check ties its inputs to the workflow's step ids. If it fails, nothing runs and Python tests fails.
  • Settings: Python 3.11, which every package allows. Poetry 2.1.2, installing each lock exactly as resolved. FI_BASE_URL and FI_GRPC_URL point at a closed local port: tests that call register() export at shutdown and would otherwise reach the real collector. No secrets are used, and checkouts drop their credentials. The core tests point the collector at the same local port in every test (an autouse fixture in python/tests/conftest.py), so none reaches the real collector even when run locally with the variables unset.
  • When it runs again: on every push to a PR, and on an edit to the PR (edited), so retargeting a PR re-runs it. A title or description edit re-runs it too, which costs a full run each time (49 package jobs on a PR that touches the CI files); skipping those is not safe, because the skipped run would cancel a real one and could count as green. A newer run of the same PR cancels the older one.

The known-broken list (30 packages)

Each entry gives a package and the reason its tests fail today:

agno, autogen, bedrock, cerebras, cohere, deepseek, dspy, fireworks, groq, guardrails, haystack, huggingface, instructor, langchain, litellm, llama_index, mcp, mistralai, ollama, openai-agents, pinecone, portkey, pydantic-ai, qdrant, smolagents, together, traceai, vertexai, vllm, xai.

  • A listed package still runs, shows a warning, and does not fail Python tests.
  • If a listed package's tests all pass, CI fails until its line is removed. So a PR that fixes a listed package must remove its line in the same PR. This applies to the open LangChain changes in feat(python): hide error details with the content hide flags #272 and fix(langchain): Deep Agents compatibility, cookbook and thread_id as session.id (TH-8245) #214: if either makes LangChain's tests pass, CI fails on that PR until langchain comes off the list.
  • Known limit: the list is per package, so for a listed package the check covers nothing. A new failure there only shows as the same warning. Some listed packages' test files don't load at all: openai-agents and smolagents (their own), guardrails (its core file). Follow-up tickets remove entries one by one.

Fixes made so packages can be tested

  • Python ranges, 23 packages. They declared ranges their dependencies can't meet, so Poetry couldn't resolve them.
    • Each now ends at <3.14, like fi-instrumentation-otel. Where the range started at 3.9 it now starts at >3.9, because fi-instrumentation-otel excludes 3.9.0.
    • autogen and claude-agent-sdk start at 3.10, which their own dependencies need. claude-agent-sdk drops its 3.9 classifier.
    • These changes show on PyPI at each package's next release.
    • Packages: autogen, bedrock, cerebras, chromadb, claude-agent-sdk, cohere, deepseek, fireworks, google-genai, groq, lancedb, milvus, minimax, mistralai, mongodb-vector, pgvector, pinecone, qdrant, redis-vector, together, vertexai, weaviate, xai.
    • The traceai SDK (python/traceai) gets the same fix: >=3.9,<3.13 becomes >3.9,<3.13. Its pyproject also named a README.md that does not exist, which made every install fail; that line is removed. (The name traceai on PyPI belongs to an unrelated project, so this SDK isn't published under it.)
  • Re-locks, 8 packages whose lock was older than their pyproject: autogen, bedrock, google-genai, groq, guardrails, instructor, mistralai, vertexai.
  • fi-instrumentation-otel 1.1.0 in 21 locks. These locks pinned 0.1.11 or 0.1.14, while the code and tests use attribute names that exist only from 1.0. They now resolve 1.1.0, the version a fresh install gets. This is dev-only: the declared floors are unchanged, and raising them is TH-8681. The locks shrink because 1.1.0 made its evaluation libraries optional. livekit's lock is left to fix(livekit): keep the Future AGI exporter when LiveKit adds its privacy filter (TH-8665) #273.
    • Packages: anthropic, autogen, bedrock, crewai, dspy, google-adk, google-genai, groq, guardrails, haystack, instructor, litellm, llama_index, mcp, mistralai, openai-agents, openai, pipecat, portkey, smolagents, vertexai.
  • fi_instrumentation's tests now pass: 198 passed, 0 failed. On main, run the way this CI runs it (Python 3.11, framework files excluded), the result is 144 passed, 9 failed and 1 error. With the framework files included on Python 3.10, it is 82 passed, 8 failed and 84 errors.
    • test_otel.py imported SESSION_NAME, which otel.py no longer has.
    • test_init.py::test_lazy_imports deleted fi_instrumentation from sys.modules and never put it back. That broke every later test that patches fi_instrumentation.otel.
    • Several tests were brought up to date with the current code: the context-attribute tests (the GenAI names), the headers test (batching is the default), the gRPC default (no port), and the observe and experiment tests (they now check the project type).

How was it tested?

  • CI on this PR runs every package, because the PR touches the CI files. On 5f8e90a (run 39270411388), all 52 jobs were green: the planner's own tests (49, in Plan), fi_instrumentation (199 passed), every unlisted package, and every listed package warning (traceai among them: 18 passed, 1 failed, the listed test). Run 39251750583 (on 6db150e) shows the rule working: anthropic, crewai, google-adk and openai had started passing while still listed, so the check failed until their lines were removed.
  • What the required check covers today: fi_instrumentation, plus the unlisted packages a PR touches. 16 of those have tests; google-genai, livekit and pipecat have none and say so. The end-to-end suites are out of scope: python/e2e_tests needs a live backend and API keys; python/frameworks/tests_e2e is not a package; its two pytest files (chromadb, lancedb) run embedded and could join CI later, and its docker-compose.yml databases are for run_real_tests.py.
  • The verdict step, run in bash under every outcome:
    • tests pass: fails, "remove its line" (also when an install step failed, which it names);
    • tests fail: warns with the listed reason;
    • an install fails and the tests fail: warns that the results may not reflect the listed reason;
    • setup fails before the tests: fails;
    • no tests at all: fails.
  • Bugs planted, then reverted. Caught:
    • a listed package that passes no longer failing;
    • a pass hidden when an install step failed;
    • a failed install making a listed package fail;
    • a failed fi_instrumentation install not named;
    • a setup failure treated as a pass;
    • the SESSION_NAME import put back;
    • the module deletion put back;
    • a list entry naming a missing package, or a duplicate entry;
    • in the planner's tests: a changed package file no longer planned, a typo in the core path, "succes" in the verdict's pass check, rename detection turned back on, -z dropped, a core change back to a subset, edited dropped, the test step removed from the Plan job, the Tests step id renamed, the verdict's Tests input dropped, the verdict passing an unknown outcome, --all planning from an empty list, PR_BASE_SHA wired to the head SHA, the planner tests step made continue-on-error, the verdict's if: negated, has_packages using > 1, a core_tests typo in join(), .github/scripts/ no longer counting as CI, the PR-base fetch removed, each empty-plan guard removed, net-empty commits failing again, the plan outputs or the package job's if: miswired, continue-on-error on the Plan job or the Tests step, the result job accepting skipped packages, a test failure swallowed, and a planned test set not run;
    • clean_env removing the collector URLs again, or the collector fixture no longer autouse (the end-to-end registration test checks the exporter points at the local port; a settings test checks the fixture runs).
  • Nikhil's feat(python): hide error details with the content hide flags #272 (fi_instrumentation error details) merged locally on top of this PR: the core tests give 336 passed. feat(python): hide error details with the content hide flags #272 alone can't collect, because main's test_otel.py is broken. The two merge in either order without conflicts. Note that feat(python): hide error details with the content hide flags #272 targets dev, not main.

Checklist

  • Branch is off main
  • Conventional Commits
  • No TODOs or commented-out code left in
  • No secrets
  • VOCABULARY check (N/A: docs/VOCABULARY.md is not in this repo)

Notes for reviewers

  • Packaging bug, not fixed here. 33 packages use attribute names that only exist from fi-instrumentation-otel 1.0 but declare floors from >=0.1.7 to >=0.1.14. A fresh install works. An environment that already has an older fi-instrumentation-otel gets broken spans. Raising those floors is TH-8681, after this PR.
  • Unlocked packages drift. 25 framework packages have no lock (21 Poetry, 4 Hatch), and neither does the traceai SDK, so each run picks up the latest releases of their dependencies, and main can go red with no PR. pinecone and qdrant already show this.
  • Removing a package. A PR that deletes a package's pyproject.toml but keeps other files in its folder fails the planner on purpose. Removing the whole folder is fine (and a listed package's line must come off the list).
  • A core change runs all 49 package jobs. This PR has no core change, so the core install into the 27 packages without core test files has not run in CI yet; their locks or ranges already resolve fi-instrumentation-otel 1.1.0, which the repo's core matches. To try it before requiring the check, a throwaway draft PR that touches a comment in python/fi_instrumentation/ would run it.
  • Separate follow-up: fi-instrumentation-otel's own <3.14 cap now also sits in these 23 packages' ranges.
  • Branch protection. Once this merges, require Python tests on main. For dev: first merge main into dev (a clean merge, since dev is behind main; its PR into dev runs the workflow), and only then require Python tests on dev. Requiring it on dev earlier would leave every PR into dev waiting on a check that never reports. Once a branch has the file, open PRs into it pick up the check on their next push (or a close and reopen); until then a required check shows as waiting. The same goes for the open PRs into main.

🤖 Generated with Claude Code

…nstrumentation's tests (TH-8673)

A workflow runs fi_instrumentation's tests on every PR into main, plus the
tests of each Python package the PR touches, each on its own dev dependencies
(Python 3.11, Poetry or Hatch) with pytest added. A change to the CI files
runs every package. One "Python tests" job sums up the result, to be marked
required. Tests are kept off the real collector.

fi_instrumentation's tests (198 now pass, none fail): test_otel.py no longer
imports the removed SESSION_NAME; test_init.py restores the module it deletes,
which had broken every later test that patches fi_instrumentation.otel; the
context-attribute, headers and gRPC-default tests follow the current code.

Draft: not yet reviewed; paused for TH-8665.
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…-8673)

.github/python-tests-known-broken.txt names each package whose tests fail on
main, with the reason. Listed packages still run and show a warning, but don't
fail the "Python tests" check; a listed package whose tests pass fails CI
until its line is removed.

23 packages declared a Python range their dependencies can't meet, so Poetry
could not resolve them: each now ends at <3.14 like fi-instrumentation-otel,
starts at >3.9 where it started at 3.9 (fi-instrumentation-otel excludes
3.9.0), and autogen and claude-agent-sdk start at 3.10, which their own
dependencies need. 8 lock files older than their pyproject are re-locked.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…idy the workflow (TH-8673)

The 23 packages made installable in the previous commit now reach their tests;
15 of them fail on main and are added to the known-broken list with their
reasons (29 entries). The list says a PR that makes a listed package pass must
remove its line.

Workflow: core test files run with python -P, so a package is tested against
its own fi-instrumentation-otel, not the repo's source; Poetry installs the
lock as resolved; push runs on main are no longer cancelled; checkouts drop
their credentials; the test tools are pinned; clearer verdict messages. The
plan script also reacts to python/tests/conftest.py and non-ASCII paths, and
says so when the list file is missing.

Tests: test_register_observe_mode checks the project type; the experiment-mode
test, which only passed because of a removed argument, now checks the
experiment project type. claude-agent-sdk drops its Python 3.9 classifier.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…the names they use (TH-8673)

anthropic, crewai, google-adk and openai lock fi-instrumentation-otel 0.1.11 or
0.1.14, but their code and core tests use attribute names that exist from 1.0.
They passed before only because the core tests ran against the repo's own
fi_instrumentation source; with python -P they run against the package's own.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…and run core tests on core changes (TH-8673)

21 package locks still pinned fi-instrumentation-otel 0.1.11 or 0.1.14, while
their code and core tests use attribute names that exist from 1.0; with the
core tests now importing each package's own fi-instrumentation-otel, they
could not even load. Their locks now resolve 1.1.0, the version a fresh
install gets (livekit's lock is left to #273). The locks shrink because 1.1.0
made its evaluation libraries optional.

Workflow: pytest, pytest-asyncio and pytest-mock are added only where a
package's environment lacks them; a change to fi_instrumentation (or
python/tests/pytest.ini) runs every package with core tests; a redundant line
in the verdict step is gone.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…n list (TH-8673)

With their locks on fi-instrumentation-otel 1.1.0 their tests pass, so the
verdict step failed them until their lines were removed, as designed.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…TH-8673)

python/traceai, the one-line setup SDK, sits outside python/frameworks/ and
was never tested. The plan now treats it as a package; its Python range gets
the same fix as the others (>3.9, which fi-instrumentation-otel needs), and its
one failing test (how SemanticConvention prints on Python 3.11+) is listed.

Reasons for smolagents, groq and instructor now match the CI log; the
test-toolset comments say that adding a pinned pytest-asyncio can move an
existing pytest.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…-8673)

Most open traceAI PRs target dev, so the workflow now runs on pull requests
and pushes for dev as well as main; the job name stays "Python tests".

python/traceai's pyproject named a README.md that does not exist, so its
install failed in CI and none of its tests ran; the line is removed. A
listed package whose install fails now gets its own warning instead of its
listed reason, and a core change also runs the traceai SDK, which wraps it.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… the check (TH-8673)

The known-broken verdict now looks at the Tests step first. Tests that
pass fail the job ("remove its line"), even after a failed install,
which the message names. Tests that fail after a failed install get a
warning that the results may not reflect the listed reason, instead of
claiming the tests did not run.

The header now says the workflow runs for main and dev, and that the
check can be required on dev only once this file is on dev.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… cancelled job (TH-8673)

When more than one install step fails, the verdict now lists them with
commas. A listed package whose tests were skipped says they did not run
because an earlier step failed or the job was cancelled, instead of
blaming setup alone.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat
khushalsonawat marked this pull request as ready for review October 19, 2026 13:31

@khushalsonawat khushalsonawat left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three Mediums, all about a green check that hides a real failure: moved files, core changes, and the planner having no tests. The rest holds: no secrets or write token on fork PRs, no injection through the plan, nothing reaches a futureagi host in the tip's run, and the range and lock changes are dev-only.

Comment thread .github/workflows/python-tests.yml Resolved
Comment thread .github/scripts/python_test_plan.py Resolved
Comment thread .github/scripts/python_test_plan.py Resolved
…TH-8673)

The planner now runs git itself (--diff RANGE or --all) with
--no-renames and -z: a moved file lists both paths, so the package it
left is planned too, and names with tabs, quotes or backslashes come
through intact.

A change to python/fi_instrumentation/, python/pyproject.toml or
python/poetry.lock now plans every package, each tested against the
repo's core, instead of only those with core test files.

The known-broken verdict moves to .github/scripts/known_broken_verdict.sh
and fails on a Tests outcome it does not know. The Plan job runs
.github/scripts/test_python_test_plan.py first: table-driven unittest
cases for the planner, git paths, the verdict and the workflow wiring.

Also: clean_env points the collector URLs at a closed local port
instead of removing them, the concurrency comment is corrected, and an
edited PR (such as a retargeted one) runs again.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

@khushalsonawat khushalsonawat left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three earlier findings hold, and I've resolved them. One new Medium: the --all path and the PR base/head wiring have no test, so a break there plans nothing and the check stays green.

Comment thread .github/scripts/python_test_plan.py Resolved
…TH-8673)

The planner fails when its input cannot be right: a range with no
commits (the base or head SHA is wrong), --all with no files, a CI or
core change that finds no packages, or a change in a framework folder
with no pyproject.toml. Commits that net to no change plan nothing.

The Python tests job now checks the package jobs against the planned
matrix, so a miswired output or condition that skips them all fails
instead of reading as green. Each package job fails if a test set it was
given did not run.

Everything under .github/scripts/ now counts as CI. A PR base missing
from the clone is fetched, or every package runs. The core tests point
the collector at a closed local port in every test (autouse). The
planner's tests cover --all, the SHA wiring and these checks, and no
longer run the user's git hooks.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat khushalsonawat self-assigned this Oct 19, 2026
@khushalsonawat
khushalsonawat merged commit b529fd8 into main Oct 19, 2026
104 checks passed
@khushalsonawat
khushalsonawat deleted the ci/TH-8673-python-tests branch October 19, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant