Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 130 additions & 1 deletion python/frameworks/crewai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,138 @@ import os

os.environ["FI_API_KEY"] = FI_API_KEY
os.environ["FI_SECRET_KEY"] = FI_SECRET_KEY

# Set these before importing CrewAI to opt out of its separate native telemetry.
os.environ["CREWAI_DISABLE_TELEMETRY"] = "true"
os.environ["CREWAI_DISABLE_TRACKING"] = "true"
```

### Privacy and credentials

#### Structural guarantee: framework configuration is not content

The instrumentor never exports an LLM, Agent, Crew, Task, Tool or client object
itself, and never calls its `repr`, `str`, `model_dump`, `dict`, `vars` or generic
JSON serializer to capture configuration. LLM API keys, base URLs (including
URLs with userinfo), configuration headers and credentials are not selected.
This does not depend on finding a secret's spelling or value.

The complete wrapper inventory is:

| Wrapper | Explicit input projection | Explicit result projection |
| --- | --- | --- |
| `Task._execute_core` | Task description, expected output and application context; no agent or tool argument object | `TaskOutput.json_dict` when it is a plain dictionary, otherwise `TaskOutput.raw`; plain application results |
| `Crew.kickoff` | Application `inputs`; named agent/task business metadata below | `CrewOutput.json_dict` when it is a plain dictionary, otherwise `CrewOutput.raw`; prompt/completion/total usage counts |
| `ToolUsage._use` | `ToolCalling.tool_name`, `ToolCalling.arguments`; tool name only from the runtime tool | Plain application result; no tool configuration |

Crew metadata selects only crew/task keys and UUIDs; agent role, goal, backstory,
key, UUID, verbosity, iteration/rate limits, delegation flag and tool names;
and task description, expected output, UUID, execution/human-input flags,
assigned agent role/key, context task descriptions and tool names. Task description
and expected output are also emitted as formatted aliases when `share_crew=True`.
`Agent.execute_task` and LLM calls have no additional wrappers in this package:
they execute inside the task span and their arguments/results are not serialized
independently. Separately installed LLM instrumentors have their own policies.

Opaque runtime objects nested in application data are replaced with
`[omitted runtime object]`, without inspecting their attributes. The guarantee is
about framework configuration objects, not credentials deliberately copied into
application prose or a plain dictionary. Instrumentation preserves application
arguments, results and raised exceptions.

#### Application content: a field-name policy, not a universal secret detector

With content capture on, task/tool inputs and outputs are application content.
The following policy applies to plain dictionaries, lists and tuples (copied
without mutation), and complete JSON object/array strings. It also applies to
context and span/event attributes, including structured `log.body` attributes.
It does not filter independent application logging pipelines or console output.

Names are case-insensitive; an initial `X-`/`X_` prefix and punctuation/separators
are normalized away. Dot/bracket field paths are inspected component by
component, including nested dictionary paths. A field and all its descendants
are replaced by `__REDACTED__` if the normalized component contains any of these
markers:

- `apikey`, `accesskey`, `privatekey`, `clientkey`, `signingkey`,
`subscriptionkey`, `applicationkey`, `masterkey`;
- `token`, `secret`, `password`, `passwd`, `passphrase`, `credential`, `auth`,
`cookie`, `session`, `signature`, `bearer`, `jwt`, `otp`.

`pin` matches a whole normalized component only. Adjacent components are also
combined for the eight `*key` markers above, so `client.api.key` and
`client[private][key]` match their single-field equivalents.

Header interpretation requires an explicit field whose normalized name ends in
`headers` (for example `headers`, `request_headers`, `rawHeaders`):

- Dictionaries with credential-named header keys;
- Two-item name/value pairs and collections of pairs;
- Alternating name/value lists such as Node's `rawHeaders`;
- Records with a `name`, `key` or `header` selector and a `value` field;
- Flattened dot/bracket records with those selectors, or `0`/`1` selectors,
including descendants of list-valued `value` fields in one attribute batch.

A credential-named header selector preserves its name but redacts the entire
paired value. Outside a named headers field, pairs and `{name, value}` records
are ordinary business data, never inferred to be headers. For example,
`topics=['authentication', 'billing']`, business tables and product records keep
their values.

The exact normalized metadata exceptions below remain visible (children are
still inspected). These cover standard token counters, generation limits,
tokenizer metadata, author metadata and non-secret session/service metadata:

`acceptedpredictiontokens`, `apisecretid`, `audiotokens`, `author`, `authoremail`;
`authorid`, `authority`, `authorname`, `authors`, `authorship`;
`authorurl`, `cachecreationinputtokens`, `cachedcontenttokencount`, `cachedinputtokens`, `cachedtokencount`;
`cachedtokens`, `cachereadinputtokens`, `cachereadtokens`, `cachewriteinputtokens`, `cachewritetokens`;
`candidatestokencount`, `completiontokencount`, `completiontokens`, `completiontokensdetails`, `consultokenttl`;
`inputtokencount`, `inputtokens`, `inputtokensdetails`, `maxcompletiontokens`, `maxinputtokens`;
`maxnewtokens`, `maxoutputtokens`, `maxtokens`, `numtokens`, `outputtokencount`;
`outputtokens`, `outputtokensdetails`, `promptcachehittokens`, `promptcachemisstokens`, `prompttokencount`;
`prompttokens`, `prompttokensdetails`, `reasoningtokencount`, `reasoningtokens`, `rejectedpredictiontokens`;
`securitytokenid`, `servicetokencount`, `sessioncount`, `sessionduration`, `sessiondurationms`;
`subjecttokentype`, `thoughtstokencount`, `tokencount`, `tokenid`, `tokenizer`;
`tokenizername`, `tokenizerversion`, `tokensused`, `tokenusage`, `tooluseprompttokencount`;
`totaltokencount`, `totaltokens`, `vaulttokencount`.

These are field-name rules, not value inspection. Unknown credential names
(for example an application-specific `pwd`, `sig` or vendor key header not
matching the markers), novel shapes, arbitrary free text and encodings other
than complete JSON objects/arrays are outside this policy. Do not put credentials
in content fields or metadata. Dictionaries beyond depth 20 are omitted, as are
structured strings larger than 1,000,000 characters or unparseable structured
strings; opaque objects are not dumped as a fallback.

To disable content capture, set `FI_HIDE_INPUTS=true` and `FI_HIDE_OUTPUTS=true`
before instrumentation, or pass `TraceConfig(hide_inputs=True, hide_outputs=True)`.
Directional text/message hiding also covers CrewAI's content aliases. Wrapper
failures record ERROR status without copying exception messages or stacktraces.
For manually recorded error events, use `FI_HIDE_ERROR_DETAILS=true`.

### CrewAI's separate native telemetry

CrewAI can export through its own provider to `telemetry.crewai.com`, independently
of traceAI. **This instrumentor does not change environment variables, patch
CrewAI's telemetry classes, disable its provider or restore/change your settings
on `uninstrument()`.** Choose the native telemetry policy explicitly before
importing CrewAI.

In CrewAI 0.152.0 and 1.15.33, `CREWAI_DISABLE_TELEMETRY=true` or
`CREWAI_DISABLE_TRACKING=true` disables native telemetry while leaving traceAI
available. `OTEL_SDK_DISABLED=true` is also honored, but disables the OTel SDK
and traceAI spans too; do not use it when you want traceAI tracing.
`CREWAI_TELEMETRY_OPT_OUT` and `DO_NOT_TRACK` are not native disable flags.

CrewAI 0.41.1 has no native telemetry opt-out flag: its constructor creates an
exporter for `https://telemetry.crewai.com:4319/v1/traces` and ignores the flags
above. If you need native telemetry disabled, upgrade to a version that honors
`CREWAI_DISABLE_TELEMETRY` before running sensitive workloads. For legacy
workloads, enforce a network egress policy; traceAI's projections cannot protect
an independent provider's exports. `share_crew=False` is not a native telemetry
disable switch.

## Quickstart

### Register Tracer Provider
Expand Down Expand Up @@ -57,7 +187,6 @@ Set up your Crewai client with built-in observability.

```python
from crewai import LLM, Agent, Crew, Process, Task
from crewai_tools import SerperDevTool

def story_example():
llm = LLM(
Expand Down
40 changes: 32 additions & 8 deletions python/frameworks/crewai/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 4 additions & 2 deletions python/frameworks/crewai/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,10 @@ classifiers = [

[tool.poetry.dependencies]
python = ">=3.10, <3.13"
crewai = ">0.19.0"
fi-instrumentation-otel = ">=0.1.11"
crewai = ">=0.41.1"
fi-instrumentation-otel = ">=1.2.2"
# CrewAI 0.41.1 imports pkg_resources, removed in setuptools 81.
setuptools = ">=78.1.1,<81"


[tool.poetry.urls]
Expand Down
114 changes: 114 additions & 0 deletions python/frameworks/crewai/tests/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
"""Local synthetic tests only; never disable the OTel SDK's positive controls."""

import os

import pytest

# Must precede CrewAI imports, which can initialize a telemetry singleton.
os.environ["CREWAI_DISABLE_TELEMETRY"] = "true"
os.environ["CREWAI_DISABLE_TRACKING"] = "true"
os.environ["OTEL_SDK_DISABLED"] = "false"


@pytest.fixture(autouse=True)
def native_telemetry_off(request, monkeypatch):
from importlib.metadata import version
from crewai.telemetry import Telemetry

if request.node.name.startswith("test_native_telemetry_"):
yield
return
if version("crewai") == "0.41.1":
# Test-only isolation: legacy CrewAI does not honor native disable flags.
# Production instrumentation must never patch the user's telemetry.
def disabled(instance):
instance.ready = False
instance.trace_set = False

monkeypatch.setattr(Telemetry, "__init__", disabled)
else:
assert Telemetry._is_telemetry_disabled()
yield


@pytest.fixture
def service_credentials():
return {
name: f"FAKE_SERVICE_CREDENTIAL_{index}_7f9b2a"
for index, name in enumerate(
(
"X-Amz-Security-Token",
"X-Vault-Token",
"X-Consul-Token",
"X-Subject-Token",
"X-Api-Secret",
"X-Service-Token",
"security_token",
"X-Token",
"PRIVATE-TOKEN",
"private_token",
"X-Gitlab-Token",
"Circle-Token",
"Ocp-Apim-Subscription-Key",
"subscription_key",
"DD-APPLICATION-KEY",
"application_key",
"X-Parse-Master-Key",
"master_key",
"app_secret",
"X-Auth",
"Authorization",
"X-API-Key",
"vendor-api.key",
"client[private][key]",
"sessionid",
"passphrase",
"signature",
"bearer",
"jwt",
"otp",
"otp_code",
"totp",
"pin",
)
)
}


@pytest.fixture
def native_tool_usage():
"""Construct the real version-specific ToolUsage around a local function."""
from importlib.metadata import version
from types import SimpleNamespace

from crewai.tools.tool_usage import ToolUsage

def create(function):
if version("crewai") == "0.41.1":
from crewai.agents.tools_handler import ToolsHandler
from langchain_core.tools import StructuredTool

tool = StructuredTool.from_function(function)
usage = ToolUsage(
tools_handler=ToolsHandler(),
tools=[tool],
original_tools=[],
tools_description=function.__name__,
tools_names=function.__name__,
task=SimpleNamespace(used_tools=0),
function_calling_llm=None,
agent=SimpleNamespace(tools_results=[]),
action=None,
)
usage._printer.print = lambda **kwargs: None
return usage
from crewai.tools.structured_tool import CrewStructuredTool

return ToolUsage(
tools_handler=None,
tools=[CrewStructuredTool.from_function(function)],
task=None,
function_calling_llm=None,
)

return create
Loading
Loading