diff --git a/python/frameworks/discoveryengine/README.md b/python/frameworks/discoveryengine/README.md index 653ff7779..610a2e59f 100644 --- a/python/frameworks/discoveryengine/README.md +++ b/python/frameworks/discoveryengine/README.md @@ -1,52 +1,27 @@ # traceAI-discoveryengine -OpenTelemetry instrumentation for the search and answer methods of the -[`google-cloud-discoveryengine`](https://pypi.org/project/google-cloud-discoveryengine/) -client (Google Discovery Engine, also sold as Vertex AI Search). - -It traces the v1 `search`, `search_lite` and `answer_query` calls, sync and -async. Conversation and session management (`create_conversation`, -`converse_conversation` and the rest), `stream_answer_query`, document -ingestion, data store and engine administration produce no spans from this -package. It does not instrument `google.genai`; for the Gemini client use -`traceAI-google-genai`. - -Status: alpha. Verified against the live `discoveryengine.googleapis.com` -on 2026-10-18 (see [Live vendor verification](#live-vendor-verification-2026-10-18)): -`search` succeeded on the gRPC, REST and gRPC asyncio transports, and the -error paths of `search`, `search_lite` and `answer_query` were recorded from -real server errors. The success paths of `search_lite` and `answer_query` -were not available on that setup and are covered by the loopback tests only. - -## Why a wrapper - -The client does not trace itself. With `google-cloud-discoveryengine` 0.20.5 -an uninstrumented `search`, `search_lite` or `answer_query` emits no span, -including with `GOOGLE_SDK_EXPERIMENTAL_PYTHON_TRACING_ENABLED=true` -(`google-api-core` traces only methods whose transport passes `method_name` -to `wrap_method`, and this client does not). -`tests/test_discoveryengine_stock_spans.py` measures this before any wrapper -is installed. A gRPC client instrumentation you add yourself creates -transport spans, which carry no query, result count or retriever kind. - -## Installation +Trace Google Discovery Engine (Vertex AI Search) retrieval, answers, +conversation turns, ranking and grounding checks in Future AGI with the +[official Python client](https://pypi.org/project/google-cloud-discoveryengine/). + +## Install ```bash pip install traceAI-discoveryengine ``` -To install from a checkout of this repository instead: - -```bash -pip install "fi-instrumentation-otel>=1.1.0" "google-cloud-discoveryengine>=0.20.5,<1" \ - ./python/frameworks/discoveryengine -``` +Supports Python 3.10, 3.11, 3.12 and 3.13. Requires +`google-cloud-discoveryengine>=0.20.5,<1` and +`fi-instrumentation-otel>=1.1.0`. Use the **v1** clients explicitly: +`google.cloud.discoveryengine` re-exports v1beta clients at +`google-cloud-discoveryengine` 0.20.5 and is not instrumented. -It accepts `google-cloud-discoveryengine>=0.20.5,<1` and -`fi-instrumentation-otel>=1.1.0`. The suite runs with -`google-cloud-discoveryengine` 0.20.5 on Python 3.10, 3.11, 3.12 and 3.13. +## Configure and search -## Usage +Set `FI_API_KEY` and `FI_SECRET_KEY` in your environment for Future AGI. +Configure [Google Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc) +for your Google client, or pass it credentials as usual. The integration +does not change how the client authenticates. ```python from fi_instrumentation import register @@ -54,334 +29,167 @@ from fi_instrumentation.fi_types import ProjectType from google.cloud import discoveryengine_v1 as discoveryengine from traceai_discoveryengine import DiscoveryEngineInstrumentor -tracer_provider = register(project_type=ProjectType.OBSERVE, project_name="my-search-app") -DiscoveryEngineInstrumentor().instrument(tracer_provider=tracer_provider) +provider = register(project_type=ProjectType.OBSERVE, project_name="support-search") +DiscoveryEngineInstrumentor().instrument(tracer_provider=provider) serving_config = ( "projects/my-gcp-project/locations/global/collections/default_collection/" "engines/my-engine/servingConfigs/default_search" ) -client = discoveryengine.SearchServiceClient() # Application Default Credentials -pager = client.search(request={"serving_config": serving_config, "query": "open telemetry"}) - -answers = discoveryengine.ConversationalSearchServiceClient() -answers.answer_query(request={"serving_config": serving_config, "query": {"text": "What is OTLP?"}}) +search = discoveryengine.SearchServiceClient() +pager = search.search(request={"serving_config": serving_config, "query": "rotate an API key"}) +print(len(pager.results)) +provider.force_flush() ``` -Import the clients from `google.cloud.discoveryengine_v1`, as above. -**`from google.cloud import discoveryengine` produces no spans**: at 0.20.5 -that unversioned module re-exports the v1beta clients, which are not traced -(nor are v1alpha clients). +Replace the serving config with your engine's resource name. The Future AGI +project (`support-search`) is separate from your GCP project. For a regional +engine, pass the appropriate endpoint to the Google client, for example +`client_options={"api_endpoint": "eu-discoveryengine.googleapis.com"}`. + +Open **Observe** in Future AGI and select your project. The call appears as a +search span with its query, duration, status and first-page result count. +Inside an active application span, Discovery Engine calls appear as children. +Existing clients and clients created after `instrument()` are both traced. -The methods are wrapped on the client classes, so clients built before or -after `instrument()` are both traced. See -[`examples/search_and_answer.py`](examples/search_and_answer.py). +## Supported operations -### Google credentials and projects +All rows support the synchronous `Client` and asynchronous `AsyncClient`. +Span names use the `discoveryengine.` prefix followed by the method name. -The Google clients keep reading Application Default Credentials (ADC) or the -credentials you pass them, exactly as without this package; it never reads -ADC or adds a credential of its own. `register()` reads `FI_API_KEY` and -`FI_SECRET_KEY` for Future AGI. +| Service | Method | Span kind | +|---|---|---| +| `SearchService` | `search`, `search_lite` | `RETRIEVER` | +| `ConversationalSearchService` | `answer_query`, `stream_answer_query`, `converse_conversation` | `CHAIN` | +| `RankService` | `rank` | `RERANKER` | +| `GroundedGenerationService` | `check_grounding` | `EVALUATOR` | + +Answers are retrieval-and-generation operations, not raw model calls. The +integration does not invent a model name, token usage or cost. Google API +retries remain inside one logical span. -The Future AGI project (`project_name` above) is not your GCP project. The -GCP project appears only inside resource names such as the serving config. +Search spans end when the initial pager is returned. Later pages are not +consumed by the instrumentor and do not add spans; the result count covers +only the first page. Administrative operations (including conversation/session +creation, ingestion and engine management) and v1alpha/v1beta clients are not +instrumented. Use `traceAI-google-genai` for the separate Gemini client. -An engine in a multi-region location needs its regional endpoint: +## Stream an answer ```python -client = discoveryengine.SearchServiceClient( - client_options={"api_endpoint": "eu-discoveryengine.googleapis.com"} +answers = discoveryengine.ConversationalSearchServiceClient() +stream = answers.stream_answer_query( + request={"serving_config": serving_config, "query": {"text": "How do I rotate an API key?"}} ) +try: + for response in stream: + print(response.answer.state.name) +finally: + stream.close() +provider.force_flush() ``` -## Instrumented calls +For an async client, obtain the stream with +`stream = await answers.stream_answer_query(request=...)`, consume it with +`async for`, and call `await stream.aclose()` when stopping early. -| Call | Span name | -|---|---| -| `SearchServiceClient.search`, `SearchServiceAsyncClient.search` | `discoveryengine.search` | -| `SearchServiceClient.search_lite`, `SearchServiceAsyncClient.search_lite` | `discoveryengine.search_lite` | -| `ConversationalSearchServiceClient.answer_query`, `ConversationalSearchServiceAsyncClient.answer_query` | `discoveryengine.answer_query` | - -What the server allows depends on your app, not on this package: - -- `search_lite` is meant for public website search with an API key - (`client_options={"api_key": ...}`). Against any other data store the - server answers `FAILED_PRECONDITION` "API key is only supported for public - website search." (seen live with OAuth credentials and an unstructured - data store). -- `answer_query` needs an app with the Large Language Model add-on - (Enterprise edition). On a Standard edition app the server answers - `FAILED_PRECONDITION` "This feature is only available when Large Language - Model add-on is enabled." (seen live). - -Both errors are traced like any other error (status ERROR, -`discoveryengine.error.status` = `FAILED_PRECONDITION`, code `400`). - -Each call produces one span with `gen_ai.span.kind` = `RETRIEVER`. Retries made -by `google-api-core` inside one call (when you pass `retry=`; at 0.20.5 these -methods have no default retry) stay in that one span. The span is a child of the active span -when there is one, and a root span otherwise. It is current while the -transport sends the request, so a span from a gRPC client instrumentation -nests under it. - -`search` and `search_lite` return a pager. Iterating past the first page -makes one more RPC of the same method per page, directly through the -transport and not through `search()` or `search_lite()`, so later pages add -no span: the span covers the call that created the pager, and -`discoveryengine.result_count` counts the first page only. - -## Span attributes - -| Attribute | Calls | Value | -|---|---|---| -| `gen_ai.span.kind` | all | `RETRIEVER` | -| `discoveryengine.serving_config` | requests with `serving_config` | The serving config resource name from the request, cut to 1 KB. | -| `discoveryengine.result_count` | successful calls | `search`, `search_lite`: number of results in the returned (first) page. `answer_query`: number of `references` on the answer (the grounding sources). Omitted when the response has no answer. | -| `discoveryengine.answer.length` | successful `answer_query` with an answer | Length of `answer_text` in characters. The text itself is never recorded. | -| `discoveryengine.answer.state` | successful `answer_query` with an answer | The answer state name, for example `SUCCEEDED` or `FAILED`. | -| `discoveryengine.session` | `answer_query` with a session | The session resource name from the request and then from the response. A request for a new session (`.../sessions/-`) records the name the response returns. | -| `gen_ai.retrieval.query`, `input.value` | requests with a query | `SearchRequest.query` or `AnswerQueryRequest.query.text`, recorded by default. See [Privacy](#privacy). | -| `discoveryengine.error.status`, `discoveryengine.error.code` | failed calls | The gRPC status name (for example `PERMISSION_DENIED`) and the HTTP-style code `google-api-core` maps it to (for example `403`). On the REST transport, where `google-api-core` raises by HTTP code (403 is `Forbidden`, without a gRPC status), the status name is the canonical `status` of the JSON error body. For a `RetryError`, the last attempt's. | -| `discoveryengine.cancelled` | cancelled async calls | `true`; see [Errors and cancellation](#errors-and-cancellation). | - -No model name, token count or cost is recorded. At 0.20.5 -`AnswerQueryResponse` and `Answer` carry no model field, and the model -version a request asks for (`answer_generation_spec.model_spec`) is not the -model that answered, so none is reported. Discovery Engine is billed in GCP, -not per span. - -Spans come from an `fi_instrumentation.FITracer`, so the context attributes -of `using_session`, `using_user`, `using_metadata`, `using_tags` and -`using_attributes` are stamped on every Discovery Engine span started inside -them (`session.id`, `user.id`, `metadata`, `tag.tags`). -`discoveryengine.session` is Discovery Engine's own session and is a -separate attribute. - -## Privacy - -The query follows `TraceConfig`, like the input of every traceAI -instrumentor: it is recorded by default (`gen_ai.retrieval.query` and -`input.value`, with credentials removed and cut to 1 KB of UTF-8 on a -character boundary), and `FI_HIDE_INPUTS=true` or -`TraceConfig(hide_inputs=True)` hides it everywhere, including server error -text. There is no package-specific switch. - -Also recorded: counts, the answer state, the serving config and session -resource names, and error text (status name, code and message). - -Never recorded: Google credentials, search results (titles, snippets, -documents, summaries), answer text, citations and reference content, -request metadata, stack traces and local file paths. There is no setting -that adds them. - -To keep the query off the trace: +The span stays open until the stream is exhausted, closed or cancelled. The +wrapper does not prefetch or retain response chunks. Only the final complete +answer supplies answer text, length and reference counts; Google's terminal +snapshot is not added to its preceding text deltas. A stream that ends without +a `SUCCEEDED` or `FAILED` answer is marked incomplete. Closing early or +cancelling marks it cancelled. Exhaust or explicitly close streams to finish +traces promptly; a plain `break` does not close an iterator. -```python -from fi_instrumentation import TraceConfig +## Control captured content +By default, queries are captured but response content is not. To include a +bounded projection of answer text, citations, reference chunks, search titles +and snippets, conversation summaries, ranked records and grounded claims: + +```python DiscoveryEngineInstrumentor().instrument( - tracer_provider=tracer_provider, - config=TraceConfig(hide_inputs=True), # or FI_HIDE_INPUTS=true + tracer_provider=provider, + capture_content=True, ) ``` -While the query is hidden, each occurrence of it in server-written text -(the error status description and the `exception` event's message) becomes -`__REDACTED__`. That -covers the verbatim text and the escaped copies the client libraries write: -protobuf text format (a status detail that quotes the query, which -`google-api-core` appends to the error message with `'`, `"` and `\` -backslash-escaped, `\n`, `\r` and `\t` as short escapes, and other ASCII -control characters and DEL as 3-digit octal such as `\013`), the hex -escapes of older gRPC cores (the chained gRPC -error's `debug_error_string`, with non-ASCII bytes as `\xHH`; seen with -grpcio 1.62, not with 1.82 or 1.84), and the Python repr and JSON string -forms. A short query is removed wherever it occurs, including inside other -words. A copy escaped or reformatted any other way is not matched. -If the query cannot be read, the error message is recorded as -`__REDACTED__`. Other request fields that a server error may quote (a -`filter`, `user_pseudo_id`) are not removed. - -Credentials are removed from every recorded value, including error -messages, and replaced with `[redacted]`. The package looks for them -where the client keeps them: the transport's google.auth credentials -(`token`, `refresh_token`, `client_secret`; an API key credential's -`token`), `client_options.api_key`, and per-call `metadata` entries named -`authorization`, `proxy-authorization`, `x-goog-api-key` or -`x-goog-iam-authorization-token` (with and without the `Bearer` prefix). -It reads them before the call and again when it records an error, so a -token set on those credentials during the call (on first use or by a -refresh) is removed too. -The package reads `metadata` once and passes the client a tuple of the same -pairs, so metadata given as a generator still reaches the server; if -reading it raises, the client raises the same error, as without the package. -Text shaped like a Google credential is replaced too, even if the client -never held it: `ya29.` access tokens, `AIza` API keys, `1//` refresh tokens, -JWTs (`eyJ...`: three base64url parts, such as a service account's -self-signed token), and, in server-written text only, the value after -`Bearer`. If the place the client keeps credentials (or the call's -`metadata`) cannot be read, before or after the call, the call is traced -without the query and with -`[not recorded: the client credentials could not be read]` in place of the -error message. - -`TraceConfig` settings apply. Pass `config=TraceConfig(...)` or set the -environment variables before calling `instrument()`: - -| Setting | Effect | -|---|---| -| `hide_inputs` / `FI_HIDE_INPUTS=true` | Records `input.value` as `__REDACTED__` and drops `gen_ai.retrieval.query`. The query is also removed from error text as described above. Counts and resource names stay. | -| `hide_outputs` / `FI_HIDE_OUTPUTS=true` | No effect: no output text is recorded. | -| `pii_redaction` / `FI_PII_REDACTION=true` | Replaces emails, phone numbers, SSNs, card numbers, IPv4 addresses and `sk-`/`pk-` style keys with tokens such as `` in every recorded text: attributes, the error status and the `exception` event. It runs after credentials (and the hidden query) are removed and before the size caps, so a cut cannot leave part of an email. The patterns also match ids: in a resource name with a GCP project number, the last ten digits are replaced (`projects/123456789012/...` is recorded as `projects/12/...`). | - -`config` must be a `fi_instrumentation.TraceConfig`; anything else raises -`TypeError` and nothing is wrapped. - -## Errors and cancellation - -An error raised by the client (a `google.api_core.exceptions` subclass such -as `PermissionDenied`, `InvalidArgument` or `ServiceUnavailable`, or a -`RetryError` when retries run out) sets the span status to ERROR, records -`discoveryengine.error.status` and `discoveryengine.error.code`, and records -one `exception` event. Credentials are removed from both, and the query when -it is not recorded (see [Privacy](#privacy)). The exception is re-raised -unchanged, so your code still sees the server's original message. - -The error text is cleaned in this order: credentials, then the hidden query, -then PII with `pii_redaction`. It is then cut on a UTF-8 character boundary: -the status description's message and `exception.message` to 1 KB. The -`exception` event carries `exception.type` and `exception.message` only: -`exception.stacktrace` is never written, because traceback frames contain -local file paths (home directory, user name, virtualenv). - -An `answer_query` that returns an answer with state `FAILED` is not an -exception: the span gets status ERROR with description `answer state FAILED` -and no `exception` event. - -Cancelling an async call (`asyncio.CancelledError`) ends the span with -status ERROR, description `cancelled`, and `discoveryengine.cancelled` = -`true`, without an exception event. - -If the instrumentation itself fails (reading a request, a response or the -credentials, starting a span, recording an error), it logs at debug level and -your call proceeds with its own result or exception. -`fi_instrumentation.suppress_tracing()` skips the span. - -## Limits - -- Recorded text is cut on a UTF-8 character boundary after it is cleaned: - 1 KB for the query and each resource name, 1 KB for `exception.message` - and the message in the error status. No stack trace is recorded. - With `pii_redaction`, `FITracer` runs its own PII pass on attributes after - the cut, and a new match can make a value a few bytes longer; that pass - only redacts more. -- A token carried by a gRPC channel you build yourself (call credentials on - a `channel=` you pass to the transport) is not visible to the package; - only the credential shapes above are removed from text. -- When it builds the gRPC channel, `google-api-core` gives it a scoped copy - of credentials that require scopes, such as service-account credentials - without scopes. A token minted on that copy during the call (a self-signed - JWT or an access token) is never on the credentials the client holds, so - only its shape (`eyJ...` or `ya29.`) removes it. -- Only v1 is traced. `google.cloud.discoveryengine` (v1beta at 0.20.5), - `discoveryengine_v1beta` and `discoveryengine_v1alpha` clients produce no - spans. -- `stream_answer_query` and `converse_conversation` are not traced. -- The tests cover the gRPC, gRPC asyncio and REST (`transport="rest"`) - transports. The async REST transport is not exercised. -- `instrument()` checks the installed `google-cloud-discoveryengine` - version. Outside `>=0.20.5,<1` it logs an error and wraps nothing. -- Do not combine this package with another instrumentor that wraps the same - client methods, or calls will produce duplicate spans. A gRPC - instrumentation adds transport spans under these spans; it does not - duplicate them. - -## Live vendor verification (2026-10-18) - -Run against the real `discoveryengine.googleapis.com` (v1, `global`) in a -staging GCP project, with the smallest app: one unstructured data store with -three inline `text/plain` documents and one Standard edition generic search -engine (no LLM add-on). The engine and data store were deleted after the run. -The code followed [Usage](#usage); the only change was -`credentials=google.oauth2.credentials.Credentials(token=..., quota_project_id=...)` -on each client, because the machine had no ADC. - -Versions: `google-cloud-discoveryengine` 0.20.5 (latest on PyPI), -`fi-instrumentation-otel` 1.1.0, google-api-core 2.42.0, google-auth -2.62.0, grpcio 1.84.0, protobuf 7.36.2, Python 3.12. Spans were exported -over OTLP/HTTP to a local Future AGI and checked in its store. - -| Call | Transport | Live result | Span | -|---|---|---|---| -| `search` | gRPC, REST, gRPC asyncio | 2 results | OK, `discoveryengine.result_count` = 2, query, serving config | -| `search` on a serving config that does not exist | gRPC, REST | `NotFound` 404 | ERROR, `NOT_FOUND` / 404 | -| `search_lite` (OAuth, not a website data store) | gRPC | `FailedPrecondition` 400 | ERROR, `FAILED_PRECONDITION` / 400 | -| `answer_query` (Standard edition app) | gRPC | `FailedPrecondition` 400 | ERROR, `FAILED_PRECONDITION` / 400 | -| `answer_query` (Standard edition app) | REST | `BadRequest` 400 | ERROR, `FAILED_PRECONDITION` (from the JSON error body) / 400 | - -With the defaults the query was stored on every Discovery Engine span; with -`FI_HIDE_INPUTS=true` it was in none of the stored columns (`input.value` = -`__REDACTED__`, no `gen_ai.retrieval.query`). In both modes the Discovery -Engine spans held no result content (titles, snippets), and no stored column -held a credential, the access token, a local path, the user name, the host -name, the account's e-mail address or a stack trace. The real error messages -seen did not quote the query. - -Not verified live: a successful `search_lite` (needs a public website data -store and an API key), a successful `answer_query` and its answer -attributes (`discoveryengine.answer.*`, `discoveryengine.session`; needs the -LLM add-on), pagination, a regional endpoint, the async REST transport, and -removal of a hidden query from a server error that quotes it. The loopback -tests cover these. - -## Tests - -The three error bodies the live API returned on 2026-10-18 are replayed in -`tests/test_discoveryengine_rest.py` (`test_live_rest_error_bodies_record_status_and_code`). - -The tests drive the real `google-cloud-discoveryengine` clients over their -own gRPC transports against a gRPC fake on 127.0.0.1 -(`tests/_discoveryengine_support.py`), and over the REST transport against a -loopback HTTP server that answers the documented v1 REST routes -(`tests/test_discoveryengine_rest.py`). They use no Google credentials, no -ADC and no GCP project: credentials are placeholders, and the serving config -is a placeholder resource name. The contract tests export through the real -`register()` OTLP exporter into the shared `harness.Receiver`, and post -captured spans with `harness.post_otlp()`. - -From the repository root: - -```bash -env -u PYTHONPATH PYTHONPATH="python/frameworks/discoveryengine:python:python/tests" \ - uv run --no-project --python 3.11 \ - --with pytest --with pytest-asyncio --with opentelemetry-api --with opentelemetry-sdk \ - --with opentelemetry-instrumentation --with opentelemetry-exporter-otlp-proto-http \ - --with wrapt --with requests --with jsonschema --with protobuf --with opentelemetry-proto \ - --with 'google-cloud-discoveryengine==0.20.5' \ - pytest python/frameworks/discoveryengine/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs -``` - -To run against the published `fi-instrumentation-otel` wheel instead of the -in-repo source, drop `python:` from `PYTHONPATH` and add -`--with 'fi-instrumentation-otel==1.1.0'`. +Choose these options in your initial `instrument()` call, not a second call +on an already instrumented process. Content appears as JSON in `output.value`. +Only named content fields are projected; the full vendor response is never +dumped. Each text field is capped at 1 KB, each collection at 20 items and the +complete JSON value at 16 KB. Larger outputs are partial and are not a copy of +the complete response. Streaming text deltas are never captured individually. -Replace `3.11` with `3.10`, `3.12` or `3.13` for the other supported -Pythons. +Use `TraceConfig` or the corresponding environment variables to restrict data: -The stock-span measurement (`tests/test_discoveryengine_stock_spans.py`) -also runs with `opentelemetry-instrumentation-grpc` installed. Then -`GOOGLE_SDK_EXPERIMENTAL_PYTHON_TRACING_ENABLED=true` turns -`google-api-core`'s experimental tracing on, and the file checks that the -unwrapped client still emits no span: +```python +from fi_instrumentation import TraceConfig -```bash -env -u PYTHONPATH PYTHONPATH="python/frameworks/discoveryengine:python:python/tests" \ - uv run --no-project --python 3.11 \ - --with pytest --with pytest-asyncio --with opentelemetry-api --with opentelemetry-sdk \ - --with opentelemetry-instrumentation --with opentelemetry-exporter-otlp-proto-http \ - --with wrapt --with requests --with jsonschema --with protobuf --with opentelemetry-proto \ - --with 'google-cloud-discoveryengine==0.20.5' --with opentelemetry-instrumentation-grpc \ - pytest python/frameworks/discoveryengine/tests/test_discoveryengine_stock_spans.py \ - -q -p no:cacheprovider --noconftest -o addopts= -rfEs +DiscoveryEngineInstrumentor().instrument( + tracer_provider=provider, + config=TraceConfig(hide_inputs=True, hide_outputs=True), +) ``` + +- `hide_inputs=True` / `FI_HIDE_INPUTS=true` hides the query and disables + response-content capture, since an output can repeat an input. +- `hide_outputs=True` / `FI_HIDE_OUTPUTS=true` disables response-content + capture even when `capture_content=True`. +- `pii_redaction=True` / `FI_PII_REDACTION=true` applies the core PII patterns + before text is truncated. Counts and resource names remain; PII patterns + can also mask numeric identifiers in resource names. + +Credentials visible on the client's transport, client options or auth metadata +are removed before text is capped, including credentials refreshed during a +call. Google token shapes and bearer values in server text are also removed. +If credential lookup fails, no response content is recorded. Tokens attached +to a custom gRPC channel or a private credential copy are not readable by the +instrumentor; only recognized token shapes can be removed in those cases. +Avoid putting credentials in application content. + +Queries and resource names are capped at 1 KB. Error text is capped at 1 KB; +`exception.stacktrace` is never written. On search/answer errors, hidden query +text and its common escaped forms are removed. Other fields echoed by a +server, such as filters, are not matched by that query-redaction step. For +streaming, conversation, ranking and grounding errors, server-written messages are +omitted/redacted entirely because they can contain records, facts or history. +Core `TraceConfig` error-detail controls continue to apply. + +## Attributes and errors + +| Attribute | Meaning | +|---|---| +| `input.value` | Query text, or the grounding answer candidate; hidden with `hide_inputs`. | +| `gen_ai.retrieval.query` | Search, answer and conversation query. | +| `gen_ai.reranker.query` | Ranking query. | +| `discoveryengine.serving_config` | Serving config resource name. | +| `discoveryengine.ranking_config`, `discoveryengine.grounding_config` | Operation resource names. | +| `discoveryengine.result_count` | First-page search results, answer references, conversation search results, ranked records or cited grounding facts. | +| `discoveryengine.answer.length`, `discoveryengine.answer.state` | Answer character count and state; final snapshot for a stream. | +| `discoveryengine.session` | Answer session resource name, when present. | +| `discoveryengine.grounding.support_score` | Grounding support score returned by Google. | +| `discoveryengine.stream.chunk_count` | Number of response chunks consumed. | +| `discoveryengine.error.status`, `discoveryengine.error.code` | Google error status and HTTP-style code. | +| `discoveryengine.cancelled` | `true` for async cancellation or early stream closure/cancellation. | + +Raised exceptions are re-raised unchanged and produce an ERROR span. A returned +answer in state `FAILED` also produces ERROR, without inventing an exception. +Cancellation ends the span without an exception event. Instrumentation failures +do not replace your SDK result or exception. `suppress_tracing()` skips tracing, +and `DiscoveryEngineInstrumentor().uninstrument()` restores the client methods. +Context from `using_session`, `using_user`, `using_metadata`, `using_tags` and +`using_attributes` is preserved; the Google session is separate from `session.id`. + +## Troubleshooting + +- **No spans:** import `discoveryengine_v1`, call `register()` before + `instrument()`, and flush the provider before a short-lived process exits. +- **`FAILED_PRECONDITION`:** `search_lite` requires public website search with + an API key. Answer generation requires an app with the appropriate LLM + add-on; check your engine's configuration. +- **Duplicate spans:** do not install another wrapper for the same Google + client methods. Generic gRPC tracing supplies transport spans, not these + semantic spans; it can run underneath this integration. +- **Missing streamed answer:** consume the stream through its terminal answer. + Closing early records cancellation rather than a complete answer. diff --git a/python/frameworks/discoveryengine/pyproject.toml b/python/frameworks/discoveryengine/pyproject.toml index ea997167b..91f37d5f1 100644 --- a/python/frameworks/discoveryengine/pyproject.toml +++ b/python/frameworks/discoveryengine/pyproject.toml @@ -1,7 +1,7 @@ [tool.poetry] name = "traceAI-discoveryengine" version = "0.1.0" -description = "OpenTelemetry instrumentation for Google Discovery Engine (Vertex AI Search) search and answer calls" +description = "OpenTelemetry instrumentation for Google Discovery Engine (Vertex AI Search) semantic operations" authors = ["Future AGI "] readme = "README.md" packages = [ diff --git a/python/frameworks/discoveryengine/tests/_discoveryengine_support.py b/python/frameworks/discoveryengine/tests/_discoveryengine_support.py index 791aac4d6..93fa9675d 100644 --- a/python/frameworks/discoveryengine/tests/_discoveryengine_support.py +++ b/python/frameworks/discoveryengine/tests/_discoveryengine_support.py @@ -228,12 +228,17 @@ def __init__(self) -> None: ), ) ) - self.port = self._server.add_insecure_port("127.0.0.1:0") + # macOS sandbox-exec permits native loopback, but not gRPC's IPv4- + # mapped IPv6 address. Use native ::1 there without relaxing the sandbox. + import sys + + host = "[::1]" if sys.platform == "darwin" else "127.0.0.1" + self.port = self._server.add_insecure_port(host + ":0") self.secure_port = self._server.add_secure_port( - "127.0.0.1:0", grpc.local_server_credentials(grpc.LocalConnectionType.LOCAL_TCP) + host + ":0", grpc.local_server_credentials(grpc.LocalConnectionType.LOCAL_TCP) ) - self.target = "127.0.0.1:{0}".format(self.port) - self.secure_target = "127.0.0.1:{0}".format(self.secure_port) + self.target = "{0}:{1}".format(host, self.port) + self.secure_target = "{0}:{1}".format(host, self.secure_port) self._server.start() # -- recording ----------------------------------------------------------- diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_answer.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_answer.py index 1a2e306bd..c5464199b 100644 --- a/python/frameworks/discoveryengine/tests/test_discoveryengine_answer.py +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_answer.py @@ -1,4 +1,4 @@ -"""J3, AC-04: ConversationalSearchServiceClient.answer_query is one RETRIEVER span.""" +"""J3, AC-04: ConversationalSearchServiceClient.answer_query is one CHAIN span.""" from __future__ import annotations @@ -31,7 +31,7 @@ def fake(): yield server -def test_answer_query_is_one_retriever_span_with_counts_not_the_answer(fake): +def test_answer_query_is_one_chain_span_with_counts_not_the_answer(fake): with instrumented() as traced: response = answer_client(fake).answer_query(request=answer_request(QUERY)) @@ -39,7 +39,7 @@ def test_answer_query_is_one_retriever_span_with_counts_not_the_answer(fake): span = traced.one() assert span.name == "discoveryengine.answer_query" assert attrs(span) == { - "gen_ai.span.kind": "RETRIEVER", + "gen_ai.span.kind": "CHAIN", "discoveryengine.serving_config": SERVING_CONFIG, "discoveryengine.result_count": ANSWER_REFERENCES, "discoveryengine.answer.length": len(ANSWER_TEXT), diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_content.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_content.py new file mode 100644 index 000000000..1a6e24ef8 --- /dev/null +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_content.py @@ -0,0 +1,195 @@ +"""Content controls and actual OTLP/HTTP round trips with synthetic credentials.""" + +import asyncio +import json +import sys +from pathlib import Path + +import pytest +from _discoveryengine_support import ( + ANSWER_TEXT, + FakeDiscoveryEngine, + answer_client, + answer_request, + attrs, + instrumented, + search_client, + search_request, +) +from fi_instrumentation import TraceConfig +from google.api_core.exceptions import PermissionDenied +from google.cloud import discoveryengine_v1 as de +from google.protobuf.json_format import MessageToDict +from opentelemetry.exporter.otlp.proto.common.trace_encoder import encode_spans +from opentelemetry.trace import StatusCode +from test_discoveryengine_operations import CASES, client_with_response +from test_discoveryengine_streaming import chunk, mock_client + +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "tests")) +from harness import Receiver, _flatten_attributes, post_otlp # noqa: E402 + + +@pytest.mark.parametrize( + "capture,hide_inputs,hide_outputs,visible", + [ + (False, False, False, False), + (True, False, False, True), + (True, True, False, False), + (True, False, True, False), + ], +) +def test_otlp_capture_is_opt_in_and_hide_flags_win(capture, hide_inputs, hide_outputs, visible): + config = TraceConfig(hide_inputs=hide_inputs, hide_outputs=hide_outputs) + with ( + FakeDiscoveryEngine() as fake, + instrumented(config=config, capture_content=capture) as traced, + ): + search = search_client(fake) + answer = answer_client(fake) + try: + search.search(request=search_request()) + answer.answer_query(request=answer_request()) + list(answer.stream_answer_query(request=answer_request())) + finally: + search.transport.close() + answer.transport.close() + for service, operation, payload, response, _ in CASES: + client, _ = client_with_response(service, operation, response) + try: + getattr(client, operation)(request=payload) + finally: + client.transport.close() + # Real OTLP protobuf serialization, HTTP collector reception and decoding. + payload = MessageToDict(encode_spans(traced.spans())) + with Receiver() as receiver: + assert post_otlp(payload, receiver.endpoint) == 200 + spans = receiver.spans() + assert len(spans) == 6 + wire = json.dumps(spans) + for marker in (ANSWER_TEXT, "private record", "private fact", "private reply"): + assert (marker in wire) is visible + for span in spans: + values = _flatten_attributes(span["attributes"]) + assert ("output.value" in values) is visible + if visible: + json.loads(values["output.value"]) + if hide_inputs: + assert "query marker" not in wire + + +def test_content_is_bounded_valid_json_and_credentials_and_pii_are_scrubbed(): + token = "synthetic-secret-marker" + record = {"id": "1", "title": "person@example.invalid " + token, "content": "long " * 5000} + response = de.RankResponse(records=[record] * 100) + with instrumented(capture_content=True, config=TraceConfig(pii_redaction=True)) as traced: + client, _ = client_with_response("RankService", "rank", response) + try: + client.rank( + request={ + "query": "question", + "ranking_config": "projects/test/locations/global/rankingConfigs/default", + }, + metadata=(("x-goog-api-key", token),), + ) + finally: + client.transport.close() + output = attrs(traced.one())["output.value"] + assert len(output.encode()) <= 16 * 1024 + parsed = json.loads(output) + assert len(parsed["records"]) <= 20 + assert token not in output + assert "person@example.invalid" not in output + assert "" in output + assert "[redacted]" in output + + +def test_stream_deltas_are_not_captured_and_terminal_snapshot_is_not_duplicated(): + token = "synthetic-secret-marker" + with instrumented(capture_content=True) as traced: + client, _ = mock_client( + iter( + [ + chunk("synthetic-"), + chunk("secret-marker"), + chunk(token + " safe answer", de.Answer.State.SUCCEEDED), + ] + ) + ) + try: + list( + client.stream_answer_query( + request=answer_request(), metadata=(("x-goog-api-key", token),) + ) + ) + finally: + client.transport.close() + output = attrs(traced.one())["output.value"] + assert token not in output + assert "synthetic-" not in output + assert json.loads(output)["answer_text"] == "[redacted] safe answer" + + +@pytest.mark.parametrize("service,operation,payload,response,kind", CASES) +@pytest.mark.parametrize("is_async", [False, True]) +def test_new_operations_hide_arbitrary_echoed_payloads_and_preserve_errors( + service, operation, payload, response, kind, is_async +): + error = PermissionDenied("private fact; private record; private history; query marker") + + async def run(): + client, mock = client_with_response(service, operation, response, True) + mock.side_effect = error + try: + with pytest.raises(PermissionDenied) as raised: + await getattr(client, operation)(request=payload) + assert raised.value is error + finally: + await client.transport.close() + + with instrumented(config=TraceConfig(hide_inputs=True, hide_outputs=True)) as traced: + if is_async: + asyncio.run(run()) + else: + client, mock = client_with_response(service, operation, response) + mock.side_effect = error + try: + with pytest.raises(PermissionDenied) as raised: + getattr(client, operation)(request=payload) + assert raised.value is error + finally: + client.transport.close() + span = traced.one() + assert span.status.status_code is StatusCode.ERROR + assert attrs(span)["discoveryengine.error.status"] == "PERMISSION_DENIED" + wire = encode_spans(traced.spans()).SerializeToString() + for marker in (b"private fact", b"private record", b"private history", b"query marker"): + assert marker not in wire + + +def test_flattened_conversation_query_and_grounding_metrics(): + service, operation, payload, response, _ = CASES[2] + with instrumented() as traced: + client, _ = client_with_response(service, operation, response) + try: + client.converse_conversation(name=payload["name"], query=payload["query"]) + finally: + client.transport.close() + service, operation, payload, response, _ = CASES[1] + client, _ = client_with_response(service, operation, response) + try: + client.check_grounding(payload) + finally: + client.transport.close() + conversation, grounding = traced.spans() + assert attrs(conversation)["input.value"] == "query marker" + assert attrs(grounding)["discoveryengine.grounding.support_score"] == 0.75 + assert attrs(grounding)["discoveryengine.result_count"] == 1 + + +def test_bad_capture_option_does_not_patch_clients(): + from traceai_discoveryengine import DiscoveryEngineInstrumentor + + original = de.RankServiceClient.rank + with pytest.raises(TypeError, match="capture_content"): + DiscoveryEngineInstrumentor().instrument(capture_content="false") + assert de.RankServiceClient.rank is original diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_contract.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_contract.py index bdedcfc65..8c59486d8 100644 --- a/python/frameworks/discoveryengine/tests/test_discoveryengine_contract.py +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_contract.py @@ -148,7 +148,7 @@ def test_real_client_calls_reach_the_collector_contract(monkeypatch): assert span["status"]["code"] == "STATUS_CODE_OK" values = _flatten_attributes(answer["attributes"]) - assert values["gen_ai.span.kind"] == "RETRIEVER" + assert values["gen_ai.span.kind"] == "CHAIN" assert int(values["discoveryengine.result_count"]) == ANSWER_REFERENCES assert int(values["discoveryengine.answer.length"]) == len(ANSWER_TEXT) assert values["discoveryengine.answer.state"] == "SUCCEEDED" diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_example.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_example.py index 0936aa26b..3872e95b3 100644 --- a/python/frameworks/discoveryengine/tests/test_discoveryengine_example.py +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_example.py @@ -70,7 +70,8 @@ def test_example_exports_one_search_and_one_answer_span(monkeypatch, capsys): "discoveryengine.answer_query", ] search, answer = (_flatten_attributes(span["attributes"]) for span in spans) - assert search["gen_ai.span.kind"] == answer["gen_ai.span.kind"] == "RETRIEVER" + assert search["gen_ai.span.kind"] == "RETRIEVER" + assert answer["gen_ai.span.kind"] == "CHAIN" assert search["discoveryengine.serving_config"] == SERVING_CONFIG assert int(search["discoveryengine.result_count"]) == SEARCH_RESULTS assert answer["discoveryengine.answer.state"] == "SUCCEEDED" diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_operations.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_operations.py new file mode 100644 index 000000000..eb7b1f900 --- /dev/null +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_operations.py @@ -0,0 +1,130 @@ +"""Offline v1 coverage through real GAPIC clients and synthetic transport responses.""" + +import asyncio +from unittest.mock import AsyncMock, Mock + +import pytest +from _discoveryengine_support import ( + FakeDiscoveryEngine, + answer_client, + answer_request, + attrs, + instrumented, +) +from google.auth.credentials import AnonymousCredentials +from google.cloud import discoveryengine_v1 as de +from opentelemetry.trace import StatusCode + +CASES = [ + ( + "RankService", + "rank", + { + "ranking_config": "projects/test/locations/global/rankingConfigs/default_ranking_config", + "query": "query marker", + "records": [{"id": "1", "content": "private record"}], + }, + de.RankResponse(records=[{"id": "1", "content": "private record", "score": 0.9}]), + "RERANKER", + ), + ( + "GroundedGenerationService", + "check_grounding", + { + "grounding_config": "projects/test/locations/global/groundingConfigs/default_grounding_config", + "answer_candidate": "query marker", + "facts": [{"fact_text": "private fact"}], + }, + de.CheckGroundingResponse(support_score=0.75, cited_facts=[{"chunk_text": "private fact"}]), + "EVALUATOR", + ), + ( + "ConversationalSearchService", + "converse_conversation", + { + "name": "projects/test/locations/global/dataStores/test/conversations/1", + "query": {"input": "query marker"}, + }, + de.ConverseConversationResponse( + reply={"summary": {"summary_text": "private reply"}}, search_results=[{"id": "1"}] + ), + "CHAIN", + ), +] + + +def client_with_response(service, operation, response, is_async=False): + cls = getattr(de, service + ("AsyncClient" if is_async else "Client")) + client = cls(credentials=AnonymousCredentials()) + rpc = getattr(client.transport, operation) + mock = AsyncMock(return_value=response) if is_async else Mock(return_value=response) + client.transport._wrapped_methods[rpc] = mock + return client, mock + + +@pytest.mark.parametrize("service,operation,payload,response,kind", CASES) +@pytest.mark.parametrize("is_async", [False, True]) +def test_v1_semantic_operations(service, operation, payload, response, kind, is_async): + async def run_async(): + client, mock = client_with_response(service, operation, response, True) + try: + result = await getattr(client, operation)(request=payload) + assert result is response + assert mock.call_count == 1 + finally: + await client.transport.close() + + with instrumented() as traced: + if is_async: + asyncio.run(run_async()) + else: + client, mock = client_with_response(service, operation, response) + try: + assert getattr(client, operation)(request=payload) is response + assert mock.call_count == 1 + finally: + client.transport.close() + span = traced.one() + assert span.name == "discoveryengine." + operation + assert attrs(span)["gen_ai.span.kind"] == kind + assert attrs(span)["input.value"] == "query marker" + assert span.status.status_code is StatusCode.OK + for marker in ("private record", "private fact", "private reply"): + assert marker not in traced.wire() + + +def test_stream_lifetime_covers_consumption(): + with FakeDiscoveryEngine() as fake, instrumented() as traced: + client = answer_client(fake) + try: + stream = client.stream_answer_query(request=answer_request()) + assert traced.spans() == [] + chunks = list(stream) + finally: + client.transport.close() + span = traced.one() + assert span.name == "discoveryengine.stream_answer_query" + assert len(chunks) == 2 + assert attrs(span)["discoveryengine.answer.state"] == "SUCCEEDED" + assert attrs(span)["discoveryengine.answer.length"] == len(chunks[-1].answer.answer_text) + assert attrs(span)["discoveryengine.stream.chunk_count"] == 2 + assert span.status.status_code is StatusCode.OK + + +@pytest.mark.parametrize("service,operation,payload,response,kind", CASES) +def test_async_operation_cancellation(service, operation, payload, response, kind): + async def run(): + client, mock = client_with_response(service, operation, response, True) + mock.side_effect = asyncio.CancelledError() + try: + with pytest.raises(asyncio.CancelledError): + await getattr(client, operation)(request=payload) + finally: + await client.transport.close() + + with instrumented() as traced: + asyncio.run(run()) + span = traced.one() + assert attrs(span)["discoveryengine.cancelled"] is True + assert span.status.status_code is StatusCode.ERROR + assert not span.events diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_packaging.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_packaging.py index a6736c0e9..250449f31 100644 --- a/python/frameworks/discoveryengine/tests/test_discoveryengine_packaging.py +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_packaging.py @@ -130,5 +130,7 @@ def test_the_readme_states_the_caps_the_code_applies(): assert "1 KB" in readme # No stack trace is recorded, so there is no stack trace cap. assert not hasattr(_wrappers, "MAX_STACKTRACE_BYTES") - assert "16 KB" not in readme + assert _wrappers.MAX_CONTENT_BYTES == 16 * 1024 + assert "16 KB" in readme + assert "capture_content=True" in readme assert "`exception.stacktrace` is never written" in readme diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_rest.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_rest.py index 8c1584217..7d40c5281 100644 --- a/python/frameworks/discoveryengine/tests/test_discoveryengine_rest.py +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_rest.py @@ -214,7 +214,7 @@ def test_rest_calls_get_the_same_spans_as_grpc(endpoint): } assert span.status.status_code is StatusCode.OK assert attrs(answer_span) == { - "gen_ai.span.kind": "RETRIEVER", + "gen_ai.span.kind": "CHAIN", "discoveryengine.serving_config": SERVING_CONFIG, "discoveryengine.result_count": ANSWER_REFERENCES, "discoveryengine.answer.length": len(ANSWER_TEXT), diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_scope.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_scope.py index 1046175eb..40df6f2db 100644 --- a/python/frameworks/discoveryengine/tests/test_discoveryengine_scope.py +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_scope.py @@ -1,4 +1,4 @@ -"""Scope: only v1 search, search_lite and answer_query (sync and async) are traced. +"""Scope: only the seven semantic v1 operations (sync and async) are traced. AC-05: ``create_conversation`` and the other admin RPCs emit no span. AC-09: instrumenting Discovery Engine does not import or instrument @@ -31,6 +31,12 @@ ("discoveryengine_v1", "SearchServiceAsyncClient", "search_lite"), ("discoveryengine_v1", "ConversationalSearchServiceClient", "answer_query"), ("discoveryengine_v1", "ConversationalSearchServiceAsyncClient", "answer_query"), + *{("discoveryengine_v1", cls + suffix, method) + for cls, methods in ( + ("ConversationalSearchService", ("stream_answer_query", "converse_conversation")), + ("RankService", ("rank",)), + ("GroundedGenerationService", ("check_grounding",)), + ) for suffix in ("Client", "AsyncClient") for method in methods}, } @@ -57,7 +63,7 @@ def _client_methods(): return methods -def test_only_the_six_v1_methods_are_wrapped(): +def test_only_the_fourteen_v1_methods_are_wrapped(): before = _client_methods() with instrumented(): during = _client_methods() @@ -104,14 +110,13 @@ def test_create_conversation_emits_no_span(fake): assert traced.spans() == [] -def test_stream_answer_query_is_not_traced(fake): - # The architecture lists search, search_lite and answer_query only. +def test_stream_answer_query_is_traced(fake): with instrumented() as traced: responses = list(answer_client(fake).stream_answer_query(request=answer_request())) assert [response.answer.state.name for response in responses] == ["STREAMING", "SUCCEEDED"] assert fake.methods() == ["StreamAnswerQuery"] - assert traced.spans() == [] + assert traced.names() == ["discoveryengine.stream_answer_query"] def test_instrumenting_does_not_import_or_instrument_google_genai(): diff --git a/python/frameworks/discoveryengine/tests/test_discoveryengine_streaming.py b/python/frameworks/discoveryengine/tests/test_discoveryengine_streaming.py new file mode 100644 index 000000000..1aeea31dc --- /dev/null +++ b/python/frameworks/discoveryengine/tests/test_discoveryengine_streaming.py @@ -0,0 +1,215 @@ +"""Stream lifecycle, identity and context checks; no cloud endpoints or ADC.""" + +import asyncio +import gc +from unittest.mock import AsyncMock, Mock + +import pytest +from _discoveryengine_support import ( + FakeDiscoveryEngine, + answer_request, + async_answer_client, + attrs, + instrumented, +) +from fi_instrumentation import suppress_tracing +from google.api_core.exceptions import PermissionDenied +from google.auth.credentials import AnonymousCredentials +from google.cloud import discoveryengine_v1 as de +from opentelemetry import trace +from opentelemetry.trace import StatusCode +from traceai_discoveryengine import DiscoveryEngineInstrumentor + + +def chunk(text="", state=de.Answer.State.STREAMING): + return de.AnswerQueryResponse(answer={"answer_text": text, "state": state}) + + +def mock_client(stream, is_async=False): + cls = ( + de.ConversationalSearchServiceAsyncClient + if is_async + else de.ConversationalSearchServiceClient + ) + client = cls(credentials=AnonymousCredentials()) + # The async GAPIC method returns the wrapped transport's awaitable. + rpc = AsyncMock(return_value=stream) if is_async else Mock(return_value=stream) + client.transport._wrapped_methods[client.transport.stream_answer_query] = rpc + return client, rpc + + +def test_chunks_are_lazy_identity_preserved_and_context_does_not_leak(): + pieces = [chunk("part"), chunk("ial"), chunk("partial", de.Answer.State.SUCCEEDED)] + reads = [] + with instrumented(capture_content=True) as traced: + tracer = traced.provider.get_tracer("app") + + def generate(): + for piece in pieces: + reads.append(trace.get_current_span().get_span_context().span_id) + yield piece + + client, rpc = mock_client(generate()) + try: + with tracer.start_as_current_span("parent") as parent: + stream = client.stream_answer_query(request=answer_request()) + assert not reads + assert rpc.call_count == 1 + assert traced.spans() == [] + results = [] + for piece in stream: + assert trace.get_current_span() is parent + results.append(piece) + span, parent_span = traced.spans() + assert all(a is b for a, b in zip(results, pieces)) + assert span.parent.span_id == parent_span.context.span_id + assert reads == [span.context.span_id] * 3 + assert attrs(span)["discoveryengine.answer.length"] == 7 + assert attrs(span)["output.value"].count("partial") == 1 + finally: + client.transport.close() + + +@pytest.mark.parametrize("ending", ["close", "drop", "error", "incomplete", "failed"]) +def test_sync_stream_abandonment_and_failures_end_once(ending): + error = PermissionDenied("hidden query echoed") + + def generate(): + yield chunk("partial") + if ending == "error": + raise error + if ending == "failed": + yield chunk(state=de.Answer.State.FAILED) + + with instrumented() as traced: + client, _ = mock_client(generate()) + try: + stream = client.stream_answer_query(request=answer_request()) + next(stream) + if ending == "close": + stream.close() + stream.close() + elif ending == "drop": + del stream + gc.collect() + elif ending == "error": + with pytest.raises(PermissionDenied) as raised: + list(stream) + assert raised.value is error + stream.close() + else: + list(stream) + span = traced.one() + assert span.status.status_code is StatusCode.ERROR + if ending in ("close", "drop"): + assert attrs(span)["discoveryengine.cancelled"] is True + assert not span.events + elif ending == "error": + assert len(span.events) == 1 + elif ending == "failed": + assert attrs(span)["discoveryengine.answer.state"] == "FAILED" + else: + assert span.status.description == "incomplete answer stream" + finally: + client.transport.close() + + +def test_cancel_forwards_result_and_finishes_even_before_first_read(): + vendor = Mock() + vendor.cancel.return_value = True + with instrumented() as traced: + client, _ = mock_client(vendor) + try: + stream = client.stream_answer_query(request=answer_request()) + assert stream.cancel() is True + assert stream.cancel() is True + finally: + client.transport.close() + assert vendor.cancel.call_count == 2 + assert attrs(traced.one())["discoveryengine.cancelled"] is True + + +@pytest.mark.parametrize("ending", ["ok", "error", "cancel", "close", "incomplete", "failed"]) +def test_async_stream_lifecycle(ending): + error = PermissionDenied("query marker") + + async def run(): + async def generate(): + yield chunk("partial") + if ending == "error": + raise error + if ending == "cancel": + raise asyncio.CancelledError() + if ending == "ok": + yield chunk("partial", de.Answer.State.SUCCEEDED) + if ending == "failed": + yield chunk(state=de.Answer.State.FAILED) + + client, rpc = mock_client(generate(), True) + with instrumented() as traced: + try: + stream = await client.stream_answer_query(request=answer_request()) + assert rpc.call_count == 1 + assert traced.spans() == [] + first = await anext(stream) + assert first.answer.answer_text == "partial" + assert not trace.get_current_span().get_span_context().is_valid + if ending == "close": + await stream.aclose() + await stream.aclose() + elif ending in ("error", "cancel"): + with pytest.raises( + PermissionDenied if ending == "error" else asyncio.CancelledError + ) as raised: + await anext(stream) + if ending == "error": + assert raised.value is error + else: + async for _ in stream: + pass + span = traced.one() + assert span.status.status_code is ( + StatusCode.OK if ending == "ok" else StatusCode.ERROR + ) + if ending in ("close", "cancel"): + assert attrs(span)["discoveryengine.cancelled"] is True + assert not span.events + finally: + await client.transport.close() + + asyncio.run(run()) + + +def test_async_real_grpc_stream(): + async def run(fake): + client = async_answer_client(fake) + try: + stream = await client.stream_answer_query(request=answer_request()) + return [response async for response in stream] + finally: + await client.transport.close() + + with FakeDiscoveryEngine() as fake, instrumented() as traced: + chunks = asyncio.run(run(fake)) + assert len(chunks) == 2 + assert traced.one().status.status_code is StatusCode.OK + assert attrs(traced.one())["discoveryengine.stream.chunk_count"] == 2 + + +@pytest.mark.parametrize("mode", ["suppressed", "uninstrumented_bound"]) +def test_stream_suppression_and_uninstrumentation(mode): + with instrumented() as traced: + vendor = iter([chunk("done", de.Answer.State.SUCCEEDED)]) + client, _ = mock_client(vendor) + try: + bound = client.stream_answer_query + if mode == "suppressed": + with suppress_tracing(): + assert bound(request=answer_request()) is vendor + else: + DiscoveryEngineInstrumentor().uninstrument() + assert bound(request=answer_request()) is vendor + assert len(list(vendor)) == 1 + finally: + client.transport.close() + assert traced.spans() == [] diff --git a/python/frameworks/discoveryengine/traceai_discoveryengine/__init__.py b/python/frameworks/discoveryengine/traceai_discoveryengine/__init__.py index f0d907f29..f0c542190 100644 --- a/python/frameworks/discoveryengine/traceai_discoveryengine/__init__.py +++ b/python/frameworks/discoveryengine/traceai_discoveryengine/__init__.py @@ -1,11 +1,8 @@ """OpenTelemetry instrumentation for Google Discovery Engine (google-cloud-discoveryengine). -Wraps the v1 ``SearchServiceClient.search``, ``SearchServiceClient.search_lite``, -``ConversationalSearchServiceClient.answer_query`` and their async twins. -Conversation and session CRUD, ``stream_answer_query``, document ingestion -and the other admin RPCs are not traced. The unversioned -``google.cloud.discoveryengine`` alias re-exports the v1beta clients and is -not traced either. +Wraps v1 search, answers (including streaming), conversation turns, ranking +and grounding, for synchronous and asynchronous clients. Administrative RPCs +and the unversioned v1beta alias are intentionally not patched. """ import logging @@ -19,8 +16,12 @@ from traceai_discoveryengine._wrappers import ( ANSWER_QUERY, + CHECK_GROUNDING, + CONVERSE_CONVERSATION, + RANK, SEARCH, SEARCH_LITE, + STREAM_ANSWER_QUERY, AsyncOperationWrapper, OperationWrapper, Options, @@ -46,19 +47,33 @@ _SERVICES + "conversational_search_service.client", "ConversationalSearchServiceClient", False, - (ANSWER_QUERY,), + (ANSWER_QUERY, STREAM_ANSWER_QUERY, CONVERSE_CONVERSATION), ), ( _SERVICES + "conversational_search_service.async_client", "ConversationalSearchServiceAsyncClient", True, - (ANSWER_QUERY,), + (ANSWER_QUERY, STREAM_ANSWER_QUERY, CONVERSE_CONVERSATION), + ), + (_SERVICES + "rank_service.client", "RankServiceClient", False, (RANK,)), + (_SERVICES + "rank_service.async_client", "RankServiceAsyncClient", True, (RANK,)), + ( + _SERVICES + "grounded_generation_service.client", + "GroundedGenerationServiceClient", + False, + (CHECK_GROUNDING,), + ), + ( + _SERVICES + "grounded_generation_service.async_client", + "GroundedGenerationServiceAsyncClient", + True, + (CHECK_GROUNDING,), ), ) class DiscoveryEngineInstrumentor(BaseInstrumentor): # type: ignore[misc] - """Instrument Discovery Engine v1 search and answer client methods. + """Instrument Discovery Engine v1 semantic client methods. ``instrument()`` accepts ``tracer_provider`` and ``config`` (a ``fi_instrumentation.TraceConfig``, read from the ``FI_HIDE_*`` and @@ -67,7 +82,9 @@ class DiscoveryEngineInstrumentor(BaseInstrumentor): # type: ignore[misc] ``hide_inputs`` / ``FI_HIDE_INPUTS`` hides it everywhere, including server error text. Spans come from an ``FITracer``, so ``using_session`` / ``using_user`` / ``using_metadata`` / ``using_tags`` / ``using_attributes`` - context attributes are stamped on them. + context attributes are stamped on them. ``capture_content=True`` opts in + to bounded response content; either ``hide_inputs`` or ``hide_outputs`` + overrides it. The default is counts and states only, without output text. """ def instrumentation_dependencies(self) -> Collection[str]: @@ -89,7 +106,11 @@ def _instrument(self, **kwargs: Any) -> None: "capture_query is ignored: the query is recorded unless " "TraceConfig(hide_inputs=True) or FI_HIDE_INPUTS=true is set" ) + capture_content = kwargs.get("capture_content", False) + if not isinstance(capture_content, bool): + raise TypeError("capture_content must be a bool") options = Options( + capture_content=capture_content, hide_inputs=bool(config.hide_inputs), hide_outputs=bool(config.hide_outputs), pii_redaction=bool(config.pii_redaction), diff --git a/python/frameworks/discoveryengine/traceai_discoveryengine/_wrappers.py b/python/frameworks/discoveryengine/traceai_discoveryengine/_wrappers.py index 49cd25589..8f3b9a1e0 100644 --- a/python/frameworks/discoveryengine/traceai_discoveryengine/_wrappers.py +++ b/python/frameworks/discoveryengine/traceai_discoveryengine/_wrappers.py @@ -1,4 +1,4 @@ -"""wrapt wrappers for Discovery Engine search, search_lite and answer_query (sync and async).""" +"""Semantic wrappers for Discovery Engine v1 client methods (sync and async).""" from __future__ import annotations @@ -32,10 +32,29 @@ ERROR_STATUS = "discoveryengine.error.status" ERROR_CODE = "discoveryengine.error.code" CANCELLED = "discoveryengine.cancelled" +STREAM_CHUNKS = "discoveryengine.stream.chunk_count" +GROUNDING_SCORE = "discoveryengine.grounding.support_score" +RANKING_CONFIG = "discoveryengine.ranking_config" +GROUNDING_CONFIG = "discoveryengine.grounding_config" SEARCH = "search" SEARCH_LITE = "search_lite" ANSWER_QUERY = "answer_query" +STREAM_ANSWER_QUERY = "stream_answer_query" +CONVERSE_CONVERSATION = "converse_conversation" +RANK = "rank" +CHECK_GROUNDING = "check_grounding" +ANSWER_OPERATIONS = (ANSWER_QUERY, STREAM_ANSWER_QUERY) +_KINDS = { + ANSWER_QUERY: "CHAIN", + STREAM_ANSWER_QUERY: "CHAIN", + CONVERSE_CONVERSATION: "CHAIN", + RANK: "RERANKER", + CHECK_GROUNDING: "EVALUATOR", +} +OUTPUT_VALUE = "output.value" +MAX_CONTENT_BYTES = 16 * 1024 +MAX_ITEMS = 20 REDACTED = "[redacted]" # Recorded instead of server-written text when the credentials the client # holds could not be read, so they cannot be removed from it. @@ -79,14 +98,14 @@ class Options: """How a span's text is cleaned, from the ``TraceConfig`` that ``FITracer`` also applies. The query is recorded unless ``hide_inputs`` (``FI_HIDE_INPUTS``) is set, - as in every traceAI instrumentor. Result and answer text are never - recorded (counts and lengths only), so ``hide_outputs`` has nothing to - hide. + as in every traceAI instrumentor. Output content is opt-in and still + obeys ``hide_outputs``. Only complete streaming answers are captured. """ hide_inputs: bool = False hide_outputs: bool = False pii_redaction: bool = False + capture_content: bool = False class _State: @@ -164,7 +183,7 @@ def _field(message: Any, name: str) -> Any: if message is None: return None if isinstance(message, Mapping): - return message.get(name) + return message[name] if name in message else None return getattr(message, name, None) @@ -173,7 +192,7 @@ def _has(message: Any, name: str) -> bool: if message is None: return False if isinstance(message, Mapping): - return message.get(name) is not None + return message[name] if name in message else None is not None try: return name in message except TypeError: @@ -265,14 +284,22 @@ def add(value: Any) -> None: def _request(args: Tuple[Any, ...], kwargs: Mapping[str, Any]) -> Any: - return args[0] if args else kwargs.get("request") + return ( + args[0] + if args + else (kwargs.get("request") if kwargs.get("request") is not None else kwargs) + ) def _query_text(operation: str, request: Any) -> Optional[str]: """``SearchRequest.query`` or ``AnswerQueryRequest.query.text``; None when empty.""" query = _field(request, "query") - if operation == ANSWER_QUERY: + if operation in ANSWER_OPERATIONS: query = _field(query, "text") + elif operation == CONVERSE_CONVERSATION: + query = _field(query, "input") + elif operation == CHECK_GROUNDING: + query = _field(request, "answer_candidate") return query if isinstance(query, str) and query else None @@ -286,11 +313,11 @@ def _request_attributes( operation: str, request: Any, secrets: Optional[Sequence[str]], options: Options ) -> Dict[str, Any]: pii = options.pii_redaction - attributes: Dict[str, Any] = {SPAN_KIND: RETRIEVER} + attributes: Dict[str, Any] = {SPAN_KIND: _KINDS.get(operation, RETRIEVER)} serving_config = _resource_name(_field(request, "serving_config"), secrets or (), pii) if serving_config is not None: attributes[SERVING_CONFIG] = serving_config - if operation == ANSWER_QUERY: + if operation in ANSWER_OPERATIONS: # "sessions/-" asks the server to start a session; the response names it. session = _resource_name(_field(request, "session"), secrets or (), pii) if session is not None: @@ -303,9 +330,14 @@ def _request_attributes( attributes[INPUT_VALUE] = REDACTED_VALUE elif query and secrets is not None: text = _clean(query, secrets, pii=pii) - attributes[RETRIEVAL_QUERY] = text + if operation != CHECK_GROUNDING: + attributes["gen_ai.reranker.query" if operation == RANK else RETRIEVAL_QUERY] = text # The backend input panel reads input.value; keep both keys equal. attributes[INPUT_VALUE] = text + for field, key in (("ranking_config", RANKING_CONFIG), ("grounding_config", GROUNDING_CONFIG)): + value = _resource_name(_field(request, field), secrets or (), pii) + if value is not None: + attributes[key] = value return attributes @@ -350,7 +382,7 @@ def _c_hex_escaped(text: str) -> str: def _hidden_inputs( operation: str, request: Any, secrets: Optional[Sequence[str]], options: Options -) -> Tuple[str, ...]: +) -> Hidden: """The query text to remove from server-written text, as it reads after scrubbing. Besides the verbatim text, the copies the client libraries escape: the @@ -360,6 +392,11 @@ def _hidden_inputs( so a longer copy wins where two match at the same place. Empty when the query is recorded (``hide_inputs`` off, the default) or empty. """ + # New calls accept records, facts and history; streaming errors can echo + # generated text. Fail closed rather than trying + # to reverse every possible server-side formatting of an entire request. + if operation in (RANK, CHECK_GROUNDING, CONVERSE_CONVERSATION, STREAM_ANSWER_QUERY): + return None if not _hides_query(options): return () query = _query_text(operation, request) @@ -385,11 +422,20 @@ def _response_attributes( error status description. """ attributes: Dict[str, Any] = {} - if operation != ANSWER_QUERY: - # The pager reads through to the first SearchResponse. - results = _count(_field(result, "results")) + if operation not in ANSWER_OPERATIONS: + # The pager reads through to the first SearchResponse; never iterate it. + result_field = { + RANK: "records", + CONVERSE_CONVERSATION: "search_results", + CHECK_GROUNDING: "cited_facts", + }.get(operation, "results") + results = _count(_field(result, result_field)) if results is not None: attributes[RESULT_COUNT] = results + if operation == CHECK_GROUNDING and _has(result, "support_score"): + score = _field(result, "support_score") + if isinstance(score, (int, float)): + attributes[GROUNDING_SCORE] = score return attributes, None failure = None @@ -415,6 +461,97 @@ def _response_attributes( return attributes, failure +def _content_attributes( + operation: str, result: Any, secrets: Sequence[str], options: Options +) -> Dict[str, Any]: + """A bounded, explicit output projection, never a dump of a vendor response. + + Hiding inputs also disables content capture: an answer or retrieved document + can repeat the query, records, history or grounding facts. Streaming callers + invoke this only for the final, complete answer, never for text fragments + (which might split a credential across chunks). + """ + if not options.capture_content or options.hide_outputs or options.hide_inputs: + return {} + + def text(value: Any) -> Optional[str]: + if isinstance(value, str) and value: + return _clean(value, secrets, pii=options.pii_redaction, server=True) + return None + + def fields(value: Any, names: Sequence[str]) -> Dict[str, str]: + return { + name: cleaned for name in names if (cleaned := text(_field(value, name))) is not None + } + + def take(value: Any) -> Sequence[Any]: + # Only repeated fields on the response, never a pager or stream. + count = _count(value) + return [value[index] for index in range(min(count, MAX_ITEMS))] if count is not None else () + + output: Dict[str, Any] = {} + if operation in ANSWER_OPERATIONS: + answer = _field(result, "answer") + output.update(fields(answer, ("answer_text",))) + output["references"] = [ + fields(_field(ref, "chunk_info"), ("content",)) + for ref in take(_field(answer, "references")) + ] + output["citations"] = [ + { + "start_index": _field(citation, "start_index"), + "end_index": _field(citation, "end_index"), + "sources": [ + fields(source, ("reference_id",)) + for source in take(_field(citation, "sources")) + ], + } + for citation in take(_field(answer, "citations")) + ] + elif operation == RANK: + output["records"] = [ + dict(fields(record, ("id", "title", "content")), score=_field(record, "score")) + for record in take(_field(result, "records")) + ] + elif operation == CHECK_GROUNDING: + output["cited_facts"] = [ + fields(fact, ("chunk_text",)) for fact in take(_field(result, "cited_facts")) + ] + output["claims"] = [ + fields(claim, ("claim_text",)) for claim in take(_field(result, "claims")) + ] + else: + if operation == CONVERSE_CONVERSATION: + summary = _field(_field(result, "reply"), "summary") + results = _field(result, "search_results") + else: + summary = _field(result, "summary") + results = _field(result, "results") + output.update(fields(summary, ("summary_text",))) + output["results"] = [] + for item in take(results): + document = _field(item, "document") + derived = _field(document, "derived_struct_data") + record = fields(document, ("id",)) + record.update(fields(derived, ("title",))) + record["snippets"] = [ + fields(snippet, ("snippet",)) for snippet in take(_field(derived, "snippets")) + ] + record.update(fields(_field(item, "chunk"), ("content",))) + output["results"].append(record) + # Keep valid JSON even at the cap; remove complete collection items rather + # than cutting a serialized string in the middle of a JSON escape. + while True: + encoded = json.dumps(output, ensure_ascii=False) + if len(encoded.encode("utf-8", "replace")) <= MAX_CONTENT_BYTES: + return {OUTPUT_VALUE: encoded, "output.mime_type": "application/json"} + collections = [value for value in output.values() if isinstance(value, list) and value] + if not collections: + return {} + max(collections, key=len).pop() + output["truncated"] = True + + def _describe(error: BaseException) -> str: try: return str(error) @@ -473,7 +610,10 @@ def _error_attributes(error: BaseException) -> Dict[str, Any]: A REST-transport error has no gRPC status; its JSON body's status is used. """ source: Any = error - if getattr(error, "grpc_status_code", None) is None and getattr(error, "cause", None) is not None: + if ( + getattr(error, "grpc_status_code", None) is None + and getattr(error, "cause", None) is not None + ): source = error.cause # type: ignore[attr-defined] attributes: Dict[str, Any] = {} status = getattr(getattr(source, "grpc_status_code", None), "name", None) @@ -572,13 +712,21 @@ def __init__( self.hidden = hidden self.instance = instance self.metadata = metadata + self.ended = False def ok(self, result: Any) -> None: + if self.ended: + return failure = None try: + secrets = self._secrets_after_call() attributes, failure = _response_attributes( - self.operation, result, self.secrets or [], self.options + self.operation, result, secrets or [], self.options ) + if secrets is not None: + attributes.update( + _content_attributes(self.operation, result, secrets, self.options) + ) for key, value in attributes.items(): self.span.set_attribute(key, value) except Exception: @@ -611,6 +759,8 @@ def _secrets_after_call(self) -> Optional[List[str]]: return sorted(dict.fromkeys(self.secrets + current), key=len, reverse=True) def error(self, error: BaseException) -> None: + if self.ended: + return # No result count: nothing was returned, so the count is unknown. secrets = self._secrets_after_call() try: @@ -644,6 +794,8 @@ def error(self, error: BaseException) -> None: self._end() def cancelled(self) -> None: + if self.ended: + return # Cancellation is not an exception: no event and no result count. try: self.span.set_attribute(CANCELLED, True) @@ -653,12 +805,173 @@ def cancelled(self) -> None: self._end() def _end(self) -> None: + if self.ended: + return + self.ended = True try: self.span.end() except Exception: logger.debug("Could not end the Discovery Engine span", exc_info=True) +class _StreamBase: + """Lazy stream proxy: keep no chunks and attach context only during reads. + + Google streams text deltas followed by a full SUCCEEDED answer. Only that + terminal snapshot is captured, so its text and references are not doubled. + See https://cloud.google.com/generative-ai-app-builder/docs/stream-answer. + """ + + def __init__(self, stream: Any, call: _Call) -> None: + self._stream = stream + self._call = call + self._chunks = 0 + self._terminal = False + self._failure: Optional[str] = None + + def __getattr__(self, name: str) -> Any: + return getattr(self._stream, name) + + def _observe(self, result: Any) -> None: + if self._call.ended: + return + self._chunks += 1 + try: + self._call.span.set_attribute(STREAM_CHUNKS, self._chunks) + answer = _field(result, "answer") + state = getattr(_field(answer, "state"), "name", None) + secrets = self._call._secrets_after_call() + if _has(result, "session"): + session = _resource_name( + _field(_field(result, "session"), "name"), + secrets or [], + self._call.options.pii_redaction, + ) + if session is not None: + self._call.span.set_attribute(SESSION, session) + if state not in ("SUCCEEDED", "FAILED"): + return + self._terminal = True + attributes, failure = _response_attributes( + STREAM_ANSWER_QUERY, result, secrets or [], self._call.options + ) + # A FAILED snapshot must not be undone by later status-only chunks. + self._failure = self._failure or failure + if secrets is not None: + attributes.update( + _content_attributes(STREAM_ANSWER_QUERY, result, secrets, self._call.options) + ) + for key, value in attributes.items(): + self._call.span.set_attribute(key, value) + except Exception: + logger.debug("Could not read the Discovery Engine stream chunk", exc_info=True) + + def _finish(self) -> None: + if self._call.ended: + return + try: + failure = self._failure or (None if self._terminal else "incomplete answer stream") + self._call.span.set_status( + Status(StatusCode.ERROR, failure) if failure else Status(StatusCode.OK) + ) + except Exception: + logger.debug("Could not set the stream status", exc_info=True) + self._call._end() + + def cancel(self) -> Any: + try: + return self._stream.cancel() + finally: + self._call.cancelled() + + def __del__(self) -> None: + # A dropped, partly consumed stream must not leave a recording span. + # Do not invoke transport methods during interpreter shutdown. + call = self.__dict__.get("_call") + if call is not None and not call.ended: + call.cancelled() + + +class _Stream(_StreamBase): + def __init__(self, stream: Any, call: _Call) -> None: + super().__init__(stream, call) + self._iterator = None + + def __iter__(self) -> "_Stream": + return self + + def __next__(self) -> Any: + try: + with _current(self._call.span): + if self._iterator is None: + self._iterator = iter(self._stream) + result = next(self._iterator) + except StopIteration: + self._finish() + raise + except BaseException as error: + self._call.error(error) + raise + self._observe(result) + return result + + def close(self) -> Any: + try: + close = getattr(self._stream, "close", None) or getattr(self._stream, "cancel", None) + return close() if close is not None else None + finally: + self._call.cancelled() + + def __enter__(self) -> "_Stream": + return self + + def __exit__(self, *_: Any) -> None: + self.close() + + +class _AsyncStream(_StreamBase): + def __init__(self, stream: Any, call: _Call) -> None: + super().__init__(stream, call) + self._iterator = None + + def __aiter__(self) -> "_AsyncStream": + return self + + async def __anext__(self) -> Any: + try: + with _current(self._call.span): + if self._iterator is None: + self._iterator = self._stream.__aiter__() + result = await self._iterator.__anext__() + except StopAsyncIteration: + self._finish() + raise + except asyncio.CancelledError: + self._call.cancelled() + raise + except BaseException as error: + self._call.error(error) + raise + self._observe(result) + return result + + async def aclose(self) -> Any: + try: + close = getattr(self._stream, "aclose", None) + if close is not None: + return await close() + cancel = getattr(self._stream, "cancel", None) + return cancel() if cancel is not None else None + finally: + self._call.cancelled() + + async def __aenter__(self) -> "_AsyncStream": + return self + + async def __aexit__(self, *_: Any) -> None: + await self.aclose() + + class _BaseWrapper: def __init__( self, @@ -698,7 +1011,7 @@ def _start( attributes = _request_attributes(self._operation, request, secrets, self._options) except Exception: # an attribute must never break the user's call logger.debug("Could not read the Discovery Engine request", exc_info=True) - attributes = {SPAN_KIND: RETRIEVER} + attributes = {SPAN_KIND: _KINDS.get(self._operation, RETRIEVER)} hidden: Hidden = () try: hidden = _hidden_inputs(self._operation, request, secrets, self._options) @@ -734,6 +1047,8 @@ def __call__( except BaseException as error: call.error(error) raise + if self._operation == STREAM_ANSWER_QUERY: + return _Stream(result, call) call.ok(result) return result @@ -760,5 +1075,7 @@ async def __call__( except BaseException as error: call.error(error) raise + if self._operation == STREAM_ANSWER_QUERY: + return _AsyncStream(result, call) call.ok(result) return result