diff --git a/AGENTS.md b/AGENTS.md index 52ce6361..d1a30aef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -386,7 +386,7 @@ Returns a `ConfigInstance` with: **Behavior when `.invoke()` is called:** 1. Fetches and validates the `AiConfigRep` variation from LaunchDarkly using `key` and the supplied `context`. Raises if the variation is disabled or invalid. -2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, (b) wildcard `['*', mode]` fallback for multi-provider adapters (e.g. LangChain). Raises if no matching handler is found. +2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, ignoring any `providers` list; (b) a wildcard `['*', mode]` whose `providers` list contains the name, shortest list first and earlier registration on a tie; (c) an unscoped wildcard `['*', mode]`. Raises if no matching handler is found. 3. Invokes the selected handler with the config, user input, tool handlers, variables, and history. The `context` passed to `.invoke()` is automatically merged into `variables` under the key `ldContext`, so templates can reference `{{ldContext.key}}`, `{{ldContext.email}}`, etc. If `history` is provided, it is passed to the handler as the 5th positional argument — messages-mode handlers splice it into the messages array; agent-mode handlers append it to the system prompt. 4. Emits LaunchDarkly telemetry events: duration (`$ld:ai:duration:total`), outcome (`$ld:ai:generation:success` / `$ld:ai:generation:error`), and token counts (`$ld:ai:tokens:*`). 5. If `judgeConfiguration` is present: @@ -421,7 +421,7 @@ registry = Registry( ) ``` -`.register(handlers=[], tools={})` can be called multiple times to add more handlers or tools. Duplicate `provides_for` keys or tool names produce a warning and the last registration wins. +`.register(handlers=[], tools={})` can be called multiple times to add more handlers or tools. Handler identity is `provides_for` plus the provider set: two wildcards of the same mode with different `providers` lists both stay registered, while the same set (order-independent; repeated names collapse) or two unscoped wildcards warn and the last registration wins. Duplicate tool names do the same. `global_registry` is a pre-constructed singleton `Registry` instance. @@ -472,7 +472,7 @@ async def handler( handler.provides_for = [provider_name: str, mode: Literal["agent", "messages"]] ``` -The `provides_for` list is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. **Always attach `provides_for` using `create_handler` rather than direct attribute assignment.** +The `provides_for` list is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. Pass `providers=["Bedrock"]` to `create_handler` to limit which `config.provider.name` values that wildcard accepts; names must match exactly, and an empty list or a blank name is rejected. Omit `providers` to accept every provider. **Always attach `provides_for` using `create_handler` rather than direct attribute assignment.** ### Factory Function diff --git a/packages/client/README.md b/packages/client/README.md index 924aefdf..f25b3f63 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -192,7 +192,7 @@ Two limits keep a trajectory from spending the judge's context window: at most 5 A tool result is now judge-prompt input. It stays literal for the same reason the generated output does: the judge config is handed to the handler unrendered and the handler makes exactly one template pass, so a `{{...}}` sequence coming back from a tool is never expanded into the judge prompt. -**Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`. Each judge selects its handler from `handler` or `handlers` by its provider and mode, with the same rule as generation. There is no mode fallback: a messages-mode judge never runs on an agent handler. A judge's prompt must also match its handler's mode. Judges get no tools: the harness passes an empty tool map and removes `tools` from the judge config. +**Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`. Each judge selects its handler from `handler` or `handlers` by its provider and mode, with the same rule as generation. There is no mode fallback: a messages-mode judge never runs on an agent handler. A judge's prompt must also match its handler's mode. Judges get no tools: the harness passes an empty tool map and removes `tools` from the judge config. A wildcard's `providers` list, when set, accepts only those names: an exact provider match wins, then the shortest list that contains the name, then an unscoped `*`. Judges are resolved through flag delivery, and handlers are matched to them, **before** any evaluation records are created — a missing judge, a judge that no handler covers, or a judge whose prompt does not match its handler fails the run up front rather than after the generation spend. One error lists every judge with a problem. After that point a criterion failure never aborts the run: an unparseable judge response, an out-of-range score, a raising handler or scorer, and a row whose generation errored each become a per-criterion `ERROR` event with a cause code (`invalid_judge_output`, `invalid_score`, `handler_raised`, `scorer_raised`, `generation_incomplete`) and a top-level `errorMessage`. Event *delivery* is different: the backend needs one result per `(row, criterion)` to finish row accounting, so if tracking a criterion event fails, every remaining result is still attempted and flushed and then `run()` raises — rather than polling to its timeout with the cause hidden. diff --git a/packages/client/api-report/launchdarkly_ai_server.txt b/packages/client/api-report/launchdarkly_ai_server.txt index d1792118..27bcf113 100644 --- a/packages/client/api-report/launchdarkly_ai_server.txt +++ b/packages/client/api-report/launchdarkly_ai_server.txt @@ -18,7 +18,7 @@ ConversationIdSpanProcessor(_SpanProcessorBase) [class] on_end(span: Any) -> None on_start(span: Span, parent_context: otel_context.Context | None = None) -> None shutdown() -> None -create_handler(provides_for: tuple[str, Literal['agent', 'messages']], fn: _HandlerFn, stream_fn: _StreamFn | None = None, capture_content: bool | None = None) -> ProviderHandler +create_handler(provides_for: tuple[str, Literal['agent', 'messages']], fn: _HandlerFn, stream_fn: _StreamFn | None = None, capture_content: bool | None = None, *, providers: Sequence[str] | None = None) -> ProviderHandler create_run_usage() -> RunUsage end_span_once(span: Any, tracker: set[int], abandoned: bool = False, cancelled: bool = False) -> None end_unfinished_spans(*spans: Any) -> None @@ -151,10 +151,11 @@ ProviderGraphResponse [class, @dataclass] response: str usage: UsageDict ProviderHandler [class] - __init__(fn: _HandlerFn, provides_for: tuple[str, Literal['agent', 'messages']] | None = None, stream_fn: _StreamFn | None = None, capture_content: bool | None = None) + __init__(fn: _HandlerFn, provides_for: tuple[str, Literal['agent', 'messages']] | None = None, stream_fn: _StreamFn | None = None, capture_content: bool | None = None, *, providers: Sequence[str] | None = None) __call__(config: AiConfigRep, user_input: str | None = None, tool_handlers: dict[str, Any] | None = None, variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None) -> HandlerResult [async] capture_content: bool [property] has_stream: bool [property] + providers: tuple[str, ...] | None = normalize_providers(providers) provides_for: tuple[str, Literal['agent', 'messages']] | None = provides_for stream(config: AiConfigRep, user_input: str | None = None, tool_handlers: dict[str, Any] | None = None, variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None) -> AsyncGenerator[dict[str, Any], None] [async] ProviderResponse(Generic[T]) [class, @dataclass] diff --git a/packages/client/src/launchdarkly_ai_server/evaluations/module.py b/packages/client/src/launchdarkly_ai_server/evaluations/module.py index d73e8191..4be7f356 100644 --- a/packages/client/src/launchdarkly_ai_server/evaluations/module.py +++ b/packages/client/src/launchdarkly_ai_server/evaluations/module.py @@ -11,7 +11,7 @@ from typing import Any, Literal, cast, overload from ..lifecycle import get_client, init_client -from ..utils import normalize_mode +from ..utils import normalize_mode, registration_key from .api import ( DEFAULT_BASE_URI, EvaluationsError, @@ -654,7 +654,7 @@ def _validate_handlers( named = [ (f"handlers[{index}]", item) for index, item in enumerate(handlers) ] - seen: dict[tuple[str, str], str] = {} + seen: dict[Any, str] = {} for name, candidate in named: if not callable(candidate): raise EvaluationsError(f"{name} must be callable") @@ -665,12 +665,14 @@ def _validate_handlers( "create_handler() or a provider package's create_*_handler() " "so that it can be matched to a config's provider and mode." ) - if provides_for in seen: + # Two scoped wildcards of one mode are distinct when their provider lists differ. + identity = registration_key(candidate) + if identity in seen: raise EvaluationsError( - f"{seen[provides_for]} and {name} both provide for " + f"{seen[identity]} and {name} both provide for " f"{provides_for!r}. Pass one handler for each provider and mode." ) - seen[provides_for] = name + seen[identity] = name return [candidate for _, candidate in named] @staticmethod diff --git a/packages/client/src/launchdarkly_ai_server/judges.py b/packages/client/src/launchdarkly_ai_server/judges.py index d46b9c0a..55189d7b 100644 --- a/packages/client/src/launchdarkly_ai_server/judges.py +++ b/packages/client/src/launchdarkly_ai_server/judges.py @@ -30,15 +30,21 @@ from .utils import ( handler_key, normalize_mode, + select_mode_handler, to_ld_context, to_usage_dict, + wildcard_covers, ) def _provider_matches(handler: ProviderHandler, provider: str | None) -> bool: - """Returns True when the handler covers the given provider or is a wildcard.""" + """True when the handler names the provider or its wildcard allowlist contains it.""" key = handler_key(handler) - return key is not None and (key[0] == provider or key[0] == "*") + if key is None: + return False + if provider is not None and key[0] == provider: + return True + return wildcard_covers(handler, provider) def _mode_matches(handler: ProviderHandler, mode: str) -> bool: @@ -51,35 +57,6 @@ def _mode_matches(handler: ProviderHandler, mode: str) -> bool: return key is not None and key[1] == mode -def _handler_for( - handlers: list[ProviderHandler], provider: str | None, mode: str -) -> ProviderHandler | None: - """Exact ``(provider, mode)`` match, then a same-mode wildcard. - - Modes are compared after normalization, so a ``completion`` tag matches - a ``messages`` judge. A wildcard such as LangChain's ``('*', 'messages')`` - does not hide a handler registered for the judge's own provider. - """ - exact = next( - ( - handler - for handler in handlers - if (key := handler_key(handler)) is not None and key == (provider, mode) - ), - None, - ) - if exact is not None: - return exact - return next( - ( - handler - for handler in handlers - if (key := handler_key(handler)) is not None and key == ("*", mode) - ), - None, - ) - - logger = logging.getLogger(__name__) @@ -149,11 +126,11 @@ async def run_judges( # config, collapse messages into a single instructions block. judge_handler: ProviderHandler = handler collapse_messages = False - if handlers: - exact = _handler_for(handlers, judge_provider, judge_mode) + if handlers is not None: + exact = select_mode_handler(handlers, judge_provider, judge_mode) agent_fallback = ( - _handler_for(handlers, judge_provider, "agent") - if not exact and judge_mode == "messages" + select_mode_handler(handlers, judge_provider, "agent") + if exact is None and judge_mode == "messages" else None ) if exact: @@ -167,8 +144,10 @@ async def run_judges( handler, "agent" ) else: + # No compatible handler — skip rather than calling a provider + # whose allowlist does not include this judge. logger.warning( - "Judge '%s' skipped: no handler provides for provider %r", + "Judge %r skipped: no handler covers provider %r", judge_key, judge_provider, ) @@ -347,11 +326,11 @@ async def _build_judge_tasks( ) collapse_messages = False - if handlers: - exact = _handler_for(handlers, judge_provider, judge_mode) + if handlers is not None: + exact = select_mode_handler(handlers, judge_provider, judge_mode) agent_fallback = ( - _handler_for(handlers, judge_provider, "agent") - if not exact and judge_mode == "messages" + select_mode_handler(handlers, judge_provider, "agent") + if exact is None and judge_mode == "messages" else None ) if exact: @@ -364,6 +343,11 @@ async def _build_judge_tasks( ) else: # No compatible handler — skip, same as run_judges. + logger.warning( + "Judge %r skipped: no handler covers provider %r", + judge_key, + judge_provider, + ) continue evaluation_metric_key = ( @@ -415,15 +399,20 @@ async def run_judge( report_usage("client.runJudge") from .tracking import execute_and_track - exact = _handler_for(handlers, task.judge_provider, task.judge_mode) + exact = select_mode_handler(handlers, task.judge_provider, task.judge_mode) agent_fallback = ( - _handler_for(handlers, task.judge_provider, "agent") - if task.judge_mode == "messages" and not exact + select_mode_handler(handlers, task.judge_provider, "agent") + if task.judge_mode == "messages" and exact is None else None ) judge_handler = exact or agent_fallback if judge_handler is None: + logger.warning( + "Judge %r skipped: no handler covers provider %r", + task.config_key, + task.judge_provider, + ) return None # Defensive: a task serialized before outputFormat was stripped at build time diff --git a/packages/client/src/launchdarkly_ai_server/registry.py b/packages/client/src/launchdarkly_ai_server/registry.py index 3fcede91..9e5ff6d1 100644 --- a/packages/client/src/launchdarkly_ai_server/registry.py +++ b/packages/client/src/launchdarkly_ai_server/registry.py @@ -5,7 +5,7 @@ from typing import Any from .types import NativeTool, ProviderHandler -from .utils import handler_key +from .utils import registration_key logger = logging.getLogger(__name__) @@ -42,15 +42,15 @@ def register( ) -> None: if handlers: for handler in handlers: - key = handler_key(handler) + key = registration_key(handler) if key is not None: - # Same identity as select_handler / evaluations: provider plus - # normalized mode (completion and messages collide). + # Identity is provides_for plus the provider set. Two scoped + # wildcards of the same mode stay registered when their lists differ. existing_idx = next( ( i for i, h in enumerate(self._handlers) - if handler_key(h) == key + if registration_key(h) == key ), None, ) diff --git a/packages/client/src/launchdarkly_ai_server/types.py b/packages/client/src/launchdarkly_ai_server/types.py index 45139325..d2a1e7b4 100644 --- a/packages/client/src/launchdarkly_ai_server/types.py +++ b/packages/client/src/launchdarkly_ai_server/types.py @@ -1,6 +1,6 @@ from __future__ import annotations -from collections.abc import AsyncGenerator, Awaitable, Callable +from collections.abc import AsyncGenerator, Awaitable, Callable, Sequence from dataclasses import dataclass, field from typing import Any, Generic, Literal, TypeVar @@ -148,6 +148,31 @@ class HandlerStreamEvent: ] +def normalize_providers(providers: Sequence[str] | None) -> tuple[str, ...] | None: + """ + Validates an optional wildcard allowlist. + + ``None`` means the handler is unscoped. A list must contain at least one + name and is stored as a tuple in first-seen order, with repeats dropped. + Names are compared to ``config.provider.name`` exactly, so a name with + surrounding whitespace is rejected rather than trimmed. + """ + if providers is None: + return None + if ( + isinstance(providers, (str, bytes)) + or len(providers) == 0 + or any( + not isinstance(name, str) or name == "" or name != name.strip() + for name in providers + ) + ): + raise ValueError( + "providers must be a non-empty list of provider names without surrounding whitespace" + ) + return tuple(dict.fromkeys(providers)) + + class ProviderHandler: """ Callable handler with provider metadata. @@ -156,6 +181,7 @@ class ProviderHandler: - ``__call__`` — blocking invocation - ``stream`` — optional async-generator streaming (may be ``None``) - ``provides_for`` — ``(provider_name, mode)`` tuple or ``None`` + - ``providers`` — optional allowlist consulted only when ``provides_for`` names ``"*"`` - ``capture_content`` — the effective content-capture decision for this handler ``capture_content`` is declared here so the client core can apply the handler's own content @@ -166,6 +192,7 @@ class ProviderHandler: """ provides_for: tuple[str, Literal["agent", "messages"]] | None + providers: tuple[str, ...] | None def __init__( self, @@ -173,11 +200,14 @@ def __init__( provides_for: tuple[str, Literal["agent", "messages"]] | None = None, stream_fn: _StreamFn | None = None, capture_content: bool | None = None, + *, + providers: Sequence[str] | None = None, ) -> None: self._fn = fn self.provides_for = provides_for self._stream_fn = stream_fn self._capture_content_override = capture_content + self.providers = normalize_providers(providers) @property def capture_content(self) -> bool: diff --git a/packages/client/src/launchdarkly_ai_server/utils.py b/packages/client/src/launchdarkly_ai_server/utils.py index 5e1fabba..1d5c4747 100644 --- a/packages/client/src/launchdarkly_ai_server/utils.py +++ b/packages/client/src/launchdarkly_ai_server/utils.py @@ -2,9 +2,9 @@ import json import re -from collections.abc import Mapping +from collections.abc import Mapping, Sequence from dataclasses import dataclass, field -from typing import Any, Literal +from typing import Any, Literal, cast from .sdk_usage import report_usage from .types import ( @@ -24,6 +24,8 @@ def create_handler( fn: _HandlerFn, stream_fn: _StreamFn | None = None, capture_content: bool | None = None, + *, + providers: Sequence[str] | None = None, ) -> ProviderHandler: """ Wraps a plain async callable in a :class:`ProviderHandler` with the given @@ -32,9 +34,14 @@ def create_handler( ``capture_content`` is this handler's override. ``None`` reads the process default from :func:`configure` when the value is used. ``True`` or ``False`` overrides that default. + + ``providers`` is consulted only when ``provides_for`` starts with ``"*"``. + It limits which ``config.provider.name`` values that wildcard accepts. """ report_usage("client.createHandler") - return _create_handler(provides_for, fn, stream_fn, capture_content) + return _create_handler( + provides_for, fn, stream_fn, capture_content, providers=providers + ) def _create_handler( @@ -42,6 +49,8 @@ def _create_handler( fn: _HandlerFn, stream_fn: _StreamFn | None = None, capture_content: bool | None = None, + *, + providers: Sequence[str] | None = None, ) -> ProviderHandler: """Non-reporting :func:`create_handler`, used by the package factories.""" return ProviderHandler( @@ -49,6 +58,7 @@ def _create_handler( provides_for=provides_for, stream_fn=stream_fn, capture_content=capture_content, + providers=providers, ) @@ -72,6 +82,77 @@ def model_parameters(config: AiConfigRep) -> dict[str, Any]: return dict(parameters) if isinstance(parameters, dict) else {} +def wildcard_covers(handler: Any, provider: str | None) -> bool: + """Whether a ``"*"`` handler accepts *provider*. An allowlist is a filter; an absent one accepts every name.""" + key = handler_key(handler) + if key is None or key[0] != "*": + return False + allow = getattr(handler, "providers", None) + if allow is None: + return True + return provider is not None and provider in tuple(allow) + + +def best_wildcard(handlers: list[Any], provider: str | None, mode: str) -> Any | None: + """ + The ``"*"`` handler of *mode* that accepts *provider*. + + A scoped list beats an unscoped wildcard. Among scoped lists that contain + the name, the shorter list wins, and equal lengths keep the earlier + registration. + """ + scoped: list[tuple[int, int, Any]] = [] + unscoped: Any | None = None + for index, handler in enumerate(handlers): + key = handler_key(handler) + if key is None or key[1] != mode: + continue + if not wildcard_covers(handler, provider): + continue + allow = getattr(handler, "providers", None) + if allow is None: + if unscoped is None: + unscoped = handler + continue + scoped.append((len(tuple(allow)), index, handler)) + if scoped: + scoped.sort(key=lambda item: (item[0], item[1])) + return scoped[0][2] + return unscoped + + +def select_mode_handler( + handlers: list[Any], provider: str | None, mode: str +) -> Any | None: + """Exact ``(provider, mode)`` first, then the best wildcard of that mode. + + *mode* is compared after :func:`normalize_mode`, through :func:`handler_key`. + """ + mode = normalize_mode(mode) + if provider: + exact = next( + ( + handler + for handler in handlers + if handler_key(handler) == (provider, mode) + ), + None, + ) + if exact is not None: + return exact + return best_wildcard(handlers, provider, mode) + + +def covers_provider_name(handler: Any, provider: str | None) -> bool: + """True when the handler names *provider* or its wildcard allowlist contains it, in any mode.""" + key = handler_key(handler) + if key is None or provider is None: + return False + if key[0] == provider: + return True + return wildcard_covers(handler, provider) + + def collapse_messages_to_instructions(config: AiConfigRep) -> AiConfigRep: """ When only an agent handler is available for a messages-mode config, collapse @@ -116,6 +197,26 @@ def handler_key( return None +def registration_key( + handler: Any, +) -> tuple[tuple[str, str], tuple[str, ...] | None] | None: + """ + Handler identity for :class:`Registry` and the evaluations duplicate check. + + :func:`handler_key` (provider plus normalized mode, so ``completion`` and + ``messages`` collide), plus the allowlist as a set on a ``"*"`` handler. + Order and repeated names do not make a new identity. Routing ignores the + list on an exact handler, so it is not part of that handler's identity. + """ + key = handler_key(handler) + if key is None: + return None + providers = getattr(handler, "providers", None) + if key[0] != "*" or providers is None: + return (key, None) + return (key, tuple(sorted(set(providers)))) + + def number_or_zero(value: Any) -> int: """Coerces a provider-reported token count to a finite number, defaulting to 0. @@ -540,15 +641,18 @@ def select_handler( Selects a handler from *handlers* based on the provider and mode in *config*/*meta*. Resolution order: - 1. Exact ``(provider, mode)`` match. - 2. Wildcard ``("*", mode)`` match — for multi-provider adapters like LangChain. - 3. (Non-strict only) Provider-only match, then single-handler fallback. + 1. Exact ``(provider, mode)`` match. A ``providers`` list on an exact handler is ignored. + 2. Wildcard ``("*", mode)`` whose ``providers`` list contains the name. The + shortest list wins; equal lengths keep the earlier registration. + 3. Unscoped wildcard ``("*", mode)`` (``providers`` is ``None``). + 4. (Non-strict only) Provider-only match, then single-handler fallback. + A sole handler whose allowlist omits the name is not used. - When *strict* is ``True`` (default, used by ``config()``), steps 1–2 are tried - and a descriptive error is raised if neither matches. + When *strict* is ``True`` (default, used by ``config()``), steps 1–3 are tried + and a descriptive error is raised if none match. When *strict* is ``False`` (used by ``graph()``), resolution falls back - progressively: exact match → wildcard match → provider-only match → single-handler fallback. + progressively through the provider-only match and the single-handler fallback. """ provider = ( (config.get("provider") or {}).get("name") if isinstance(config, dict) else None @@ -558,19 +662,13 @@ def select_handler( mode = normalize_mode(meta.get("mode") if isinstance(meta, dict) else None) - exact = next((h for h in handlers if handler_key(h) == (provider, mode)), None) - if exact: - return exact - - wildcard = next((h for h in handlers if handler_key(h) == ("*", mode)), None) - if wildcard: - return wildcard + chosen = select_mode_handler(handlers, provider, mode) + if chosen is not None: + return cast(ProviderHandler, chosen) if strict: has_coverage = any( - (key := handler_key(h)) is not None - and (key[0] == provider or key[0] == "*") - for h in handlers + covers_provider_name(handler, provider) for handler in handlers ) if not has_coverage: raise ValueError(f"Handler for provider {provider} not found") @@ -588,7 +686,11 @@ def select_handler( return by_provider if len(handlers) == 1: - return handlers[0] + only = handlers[0] + allow = getattr(only, "providers", None) + if allow is not None and provider not in tuple(allow): + raise ValueError(f"Handler for provider {provider} not found") + return only raise ValueError(f"Handler for provider {provider} not found") diff --git a/packages/client/tests/test_client.py b/packages/client/tests/test_client.py index 24833549..e6cdb248 100644 --- a/packages/client/tests/test_client.py +++ b/packages/client/tests/test_client.py @@ -416,11 +416,33 @@ async def test_parse_failure_returns_raw_string_when_output_format_set( # --------------------------------------------------------------------------- -def _handler_for(provider: str, mode: str, response: str = "ok") -> ProviderHandler: +def _handler_for( + provider: str, + mode: str, + response: str = "ok", + providers: list[str] | None = None, +) -> ProviderHandler: async def fn(cfg, user_input, tool_handlers, variables, history=None) -> dict: # type: ignore[override] return {"output": response, "usage": {"input_tokens": 1, "output_tokens": 1}} - return ProviderHandler(fn=fn, provides_for=(provider, mode)) # type: ignore[arg-type] + handler = ProviderHandler(fn=fn, provides_for=(provider, mode)) # type: ignore[arg-type] + if providers is not None: + handler.providers = tuple(providers) # type: ignore[attr-defined] + return handler + + +def _variation(provider: str, mode: str = "messages") -> dict[str, Any]: + return { + "model": {"name": "gpt-4"}, + "provider": {"name": provider}, + "instructions": "hi", + "_ldMeta": { + "enabled": True, + "variationKey": "v1", + "version": 1, + "mode": mode, + }, + } class TestConfigMultiHandler: @@ -539,6 +561,116 @@ async def test_wildcard_handler_does_not_match_different_mode( with pytest.raises((ValueError, RuntimeError)): await rm.invoke("hi", CONTEXT) + async def test_scoped_wildcard_matches_a_listed_provider( + self, mock_ld_client: MagicMock + ) -> None: + mock_ld_client.variation = AsyncMock(return_value=_variation("Bedrock")) + h = _handler_for("*", "messages", "from-bedrock", providers=["Bedrock"]) + result = await config(key="flag", handler=[h]).invoke("hi", CONTEXT) + assert result.response == "from-bedrock" + + async def test_scoped_wildcard_rejects_an_unlisted_provider( + self, mock_ld_client: MagicMock + ) -> None: + mock_ld_client.variation = AsyncMock(return_value=_variation("OpenAI")) + h = _handler_for("*", "messages", "from-bedrock", providers=["Bedrock"]) + with pytest.raises( + ValueError, match=r"^Handler for provider OpenAI not found$" + ): + await config(key="flag", handler=[h]).invoke("hi", CONTEXT) + + async def test_scoped_wildcard_beats_unscoped_for_a_listed_provider( + self, mock_ld_client: MagicMock + ) -> None: + mock_ld_client.variation = AsyncMock(return_value=_variation("Bedrock")) + unscoped = _handler_for("*", "messages", "any") + scoped = _handler_for("*", "messages", "bedrock", providers=["Bedrock"]) + result = await config(key="flag", handler=[unscoped, scoped]).invoke( + "hi", CONTEXT + ) + assert result.response == "bedrock" + + async def test_duck_typed_wildcard_handler_is_selected( + self, mock_ld_client: MagicMock + ) -> None: + mock_ld_client.variation = AsyncMock(return_value=_variation("OpenAI")) + + class DuckHandler: + provides_for = ("*", "messages") + providers = None + capture_content = False + stream = None + + async def __call__(self, *args: Any, **kwargs: Any) -> dict[str, Any]: + return { + "output": "duck", + "usage": {"input_tokens": 1, "output_tokens": 1}, + } + + result = await config(key="flag", handler=[DuckHandler()]).invoke("hi", CONTEXT) # type: ignore[list-item] + assert result.response == "duck" + + async def test_two_scoped_wildcards_divide_providers( + self, mock_ld_client: MagicMock + ) -> None: + bedrock = _handler_for("*", "messages", "bedrock", providers=["Bedrock"]) + anthropic = _handler_for("*", "messages", "anthropic", providers=["Anthropic"]) + mock_ld_client.variation = AsyncMock(return_value=_variation("Anthropic")) + result = await config(key="flag", handler=[bedrock, anthropic]).invoke( + "hi", CONTEXT + ) + assert result.response == "anthropic" + + async def test_exact_handler_wins_over_scoped_wildcard( + self, mock_ld_client: MagicMock + ) -> None: + scoped = _handler_for("*", "messages", "scoped", providers=["TestProvider"]) + exact = _handler_for("TestProvider", "messages", "exact") + result = await config(key="flag", handler=[scoped, exact]).invoke("hi", CONTEXT) + assert result.response == "exact" + + async def test_unscoped_wildcard_is_the_fallback( + self, mock_ld_client: MagicMock + ) -> None: + scoped = _handler_for("*", "messages", "bedrock", providers=["Bedrock"]) + unscoped = _handler_for("*", "messages", "any") + result = await config(key="flag", handler=[scoped, unscoped]).invoke( + "hi", CONTEXT + ) + assert result.response == "any" + + async def test_shorter_provider_list_wins(self, mock_ld_client: MagicMock) -> None: + mock_ld_client.variation = AsyncMock(return_value=_variation("Bedrock")) + wide = _handler_for("*", "messages", "wide", providers=["Bedrock", "Anthropic"]) + narrow = _handler_for("*", "messages", "narrow", providers=["Bedrock"]) + result = await config(key="flag", handler=[wide, narrow]).invoke("hi", CONTEXT) + assert result.response == "narrow" + + async def test_equal_length_overlap_keeps_the_earlier_handler( + self, mock_ld_client: MagicMock + ) -> None: + mock_ld_client.variation = AsyncMock(return_value=_variation("Bedrock")) + earlier = _handler_for( + "*", "messages", "earlier", providers=["Bedrock", "Gemini"] + ) + later = _handler_for( + "*", "messages", "later", providers=["Bedrock", "Anthropic"] + ) + result = await config(key="flag", handler=[earlier, later]).invoke( + "hi", CONTEXT + ) + assert result.response == "earlier" + + async def test_scoped_wildcard_does_not_match_a_different_mode( + self, mock_ld_client: MagicMock + ) -> None: + mock_ld_client.variation = AsyncMock( + return_value=_variation("Bedrock", mode="agent") + ) + h = _handler_for("*", "messages", "bedrock", providers=["Bedrock"]) + with pytest.raises(ValueError, match="Bedrock"): + await config(key="flag", handler=[h]).invoke("hi", CONTEXT) + async def test_resolves_handlers_from_registry( self, mock_ld_client: MagicMock ) -> None: diff --git a/packages/client/tests/test_evaluations_run.py b/packages/client/tests/test_evaluations_run.py index 435d0f4c..49fda8d3 100644 --- a/packages/client/tests/test_evaluations_run.py +++ b/packages/client/tests/test_evaluations_run.py @@ -2695,6 +2695,144 @@ async def wildcard_judge( assert [config["provider"]["name"] for config in judged] == ["Anthropic"] +@pytest.mark.asyncio +async def test_scoped_wildcard_does_not_run_a_judge_outside_its_list( + monkeypatch: pytest.MonkeyPatch, +) -> None: + transport = judge_run_transport() + judge_variation(monkeypatch, provider="Anthropic") + evals = init_evaluations( + project_key="proj", api_key="token", sdk_key="sdk-key", transport=transport + ) + + async def bedrock_judge( + config: dict[str, Any], + user_input: str | None = None, + tool_handlers: dict[str, Callable[..., Any]] | None = None, + variables: dict[str, Any] | None = None, + history: list[dict[str, Any]] | None = None, + ) -> dict[str, Any]: + return {"output": '{"score": 1, "reasoning": "ok"}'} + + scoped = create_handler(("*", "messages"), bedrock_judge, providers=["Bedrock"]) + + with pytest.raises(EvaluationsError, match="Anthropic"): + await evals.run( + key="support-qa", + dataset="golden", + generation={"provider": "OpenAI", "model": "gpt-4o", "mode": "completion"}, + criteria=[Judge(key="$ld:ai:judge:accuracy")], + handlers=[create_handler(("OpenAI", "messages"), _generation_only), scoped], + ) + + +@pytest.mark.asyncio +async def test_two_scoped_judge_handlers_divide_providers( + monkeypatch: pytest.MonkeyPatch, + stub_sdk_client: MagicMock, +) -> None: + transport = judge_run_transport() + judge_variation(monkeypatch, provider="Anthropic") + evals = init_evaluations( + project_key="proj", api_key="token", sdk_key="sdk-key", transport=transport + ) + chosen: list[str] = [] + + def judge_handler(name: str) -> Any: + async def run( + config: dict[str, Any], + user_input: str | None = None, + tool_handlers: dict[str, Callable[..., Any]] | None = None, + variables: dict[str, Any] | None = None, + history: list[dict[str, Any]] | None = None, + ) -> dict[str, Any]: + chosen.append(name) + return {"output": '{"score": 1, "reasoning": "ok"}'} + + return run + + bedrock = create_handler( + ("*", "messages"), judge_handler("bedrock"), providers=["Bedrock"] + ) + anthropic = create_handler( + ("*", "messages"), judge_handler("anthropic"), providers=["Anthropic"] + ) + + result = await evals.run( + key="support-qa", + dataset="golden", + generation={"provider": "OpenAI", "model": "gpt-4o", "mode": "completion"}, + criteria=[Judge(key="$ld:ai:judge:accuracy")], + handlers=[ + create_handler(("OpenAI", "messages"), _generation_only), + bedrock, + anthropic, + ], + ) + + assert result.passed is True + assert chosen == ["anthropic"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("listed", "expected"), + [ + (["unscoped", "wide", "narrow"], "narrow"), + (["unscoped", "other"], "unscoped"), + ], +) +async def test_judge_handler_prefers_shortest_allowlist_then_unscoped( + monkeypatch: pytest.MonkeyPatch, + stub_sdk_client: MagicMock, + listed: list[str], + expected: str, +) -> None: + transport = judge_run_transport() + judge_variation(monkeypatch, provider="Anthropic") + evals = init_evaluations( + project_key="proj", api_key="token", sdk_key="sdk-key", transport=transport + ) + chosen: list[str] = [] + + def judge_handler(name: str) -> Any: + async def run( + config: dict[str, Any], + user_input: str | None = None, + tool_handlers: dict[str, Callable[..., Any]] | None = None, + variables: dict[str, Any] | None = None, + history: list[dict[str, Any]] | None = None, + ) -> dict[str, Any]: + chosen.append(name) + return {"output": '{"score": 1, "reasoning": "ok"}'} + + return run + + allowlists: dict[str, list[str] | None] = { + "unscoped": None, + "wide": ["Bedrock", "Anthropic", "Gemini"], + "narrow": ["Anthropic", "Bedrock"], + "other": ["Bedrock"], + } + handlers = [ + create_handler( + ("*", "messages"), judge_handler(name), providers=allowlists[name] + ) + for name in listed + ] + + result = await evals.run( + key="support-qa", + dataset="golden", + generation={"provider": "OpenAI", "model": "gpt-4o", "mode": "completion"}, + criteria=[Judge(key="$ld:ai:judge:accuracy")], + handlers=[create_handler(("OpenAI", "messages"), _generation_only), *handlers], + ) + + assert result.passed is True + assert chosen == [expected] + + @pytest.mark.asyncio async def test_a_messages_mode_judge_never_runs_on_an_agent_handler( monkeypatch: pytest.MonkeyPatch, @@ -4497,6 +4635,20 @@ async def _echo(*args: Any, **kwargs: Any) -> dict[str, Any]: }, "both provide for", ), + ( + # The same provider set in another order is the same identity. + { + "handlers": [ + create_handler( + ("*", "messages"), _echo, providers=["Bedrock", "Gemini"] + ), + create_handler( + ("*", "messages"), _echo, providers=["Gemini", "Bedrock"] + ), + ] + }, + r"handlers\[0\] and handlers\[1\] both provide for", + ), ], ) async def test_handler_arguments_are_validated_before_network_io( diff --git a/packages/client/tests/test_graph.py b/packages/client/tests/test_graph.py index 24155844..b3d5d17e 100644 --- a/packages/client/tests/test_graph.py +++ b/packages/client/tests/test_graph.py @@ -170,6 +170,18 @@ async def failing_variation(key: str, ctx: dict, default: Any) -> Any: # --------------------------------------------------------------------------- +class TestScopedWildcard: + async def test_sole_scoped_handler_does_not_run_an_unlisted_provider( + self, mock_ld_client: MagicMock + ) -> None: + handler = _make_handler() + handler.provides_for = ("*", "messages") # type: ignore[assignment] + handler.providers = ("Bedrock",) # type: ignore[attr-defined] + g = graph("graph-key", handlers=[handler]) + with pytest.raises(ValueError, match="TestProvider"): + await g.invoke("hi", CONTEXT) + + class TestGraphInvoke: async def test_throws_when_graph_disabled(self, mock_ld_client: MagicMock) -> None: mock_ld_client.variation = AsyncMock(return_value={"edges": {}}) diff --git a/packages/client/tests/test_judges.py b/packages/client/tests/test_judges.py index 6e53b4ee..8b19ae95 100644 --- a/packages/client/tests/test_judges.py +++ b/packages/client/tests/test_judges.py @@ -4,6 +4,7 @@ """ import json +import logging from dataclasses import asdict from typing import Any, ClassVar from unittest.mock import AsyncMock, MagicMock, patch @@ -249,6 +250,113 @@ async def fn( ) assert seen == ["typesafe"] + async def test_scoped_wildcard_skips_a_judge_outside_its_list( + self, mock_ld_client: MagicMock + ) -> None: + called = False + + async def recording_fn( + config, user_input, tool_handlers, variables, history=None + ) -> dict: # type: ignore[override] + nonlocal called + called = True + return {"output": '{"score": 0.8, "reasoning": "ok"}', "usage": {}} + + scoped = ProviderHandler( + fn=recording_fn, + provides_for=("*", "agent"), # type: ignore[arg-type] + providers=["Bedrock"], + ) + parent = ProviderHandler( + fn=recording_fn, + provides_for=("OpenAI", "messages"), # type: ignore[arg-type] + ) + mock_ld_client.variation = AsyncMock( + return_value={ + "model": {"name": "claude-3-5-sonnet"}, + "provider": {"name": "Anthropic"}, + "messages": [{"role": "user", "content": "Evaluate this."}], + "_ldMeta": { + "enabled": True, + "variationKey": "j1", + "version": 1, + "mode": "judge", + }, + } + ) + config = { + "model": {"name": "gpt-4"}, + "provider": {"name": "OpenAI"}, + "instructions": "hi", + "judgeConfiguration": {"judges": [{"key": "judge-1", "samplingRate": 1.0}]}, + } + import random + + with patch.object(random, "random", return_value=0.0): + await run_judges( + config=config, + user_context=CONTEXT, + handler=parent, + handlers=[scoped], + user_input="q", + llm_response="response", + base_track_data={"runId": "x"}, + ) + assert called is False + + @pytest.mark.parametrize( + ("judge_provider", "runs"), [("Bedrock", True), ("Anthropic", False)] + ) + async def test_scoped_parent_runs_only_a_listed_judge_provider( + self, mock_ld_client: MagicMock, judge_provider: str, runs: bool + ) -> None: + called = False + + async def recording_fn( + config, user_input, tool_handlers, variables, history=None + ) -> dict: # type: ignore[override] + nonlocal called + called = True + return {"output": '{"score": 0.8, "reasoning": "ok"}', "usage": {}} + + parent = ProviderHandler( + fn=recording_fn, + provides_for=("*", "messages"), # type: ignore[arg-type] + providers=["Bedrock"], + ) + mock_ld_client.variation = AsyncMock( + return_value={ + "model": {"name": "m"}, + "provider": {"name": judge_provider}, + "messages": [{"role": "user", "content": "Evaluate this."}], + "_ldMeta": { + "enabled": True, + "variationKey": "j1", + "version": 1, + "mode": "judge", + }, + } + ) + config = { + "model": {"name": "m"}, + "provider": {"name": "Bedrock"}, + "instructions": "hi", + "judgeConfiguration": {"judges": [{"key": "judge-1", "samplingRate": 1.0}]}, + } + import random + + with patch.object(random, "random", return_value=0.0): + await run_judges( + config=config, + user_context=CONTEXT, + handler=parent, + handlers=[], + user_input="q", + llm_response="response", + base_track_data={"runId": "x"}, + ) + assert called is runs + async def test_exact_agent_handler_fallback_collapses_messages( self, mock_ld_client: MagicMock ) -> None: @@ -599,7 +707,7 @@ async def completion_fn( @pytest.mark.asyncio async def test_returns_none_when_only_unrelated_mode( - self, monkeypatch: pytest.MonkeyPatch + self, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture ) -> None: from launchdarkly_ai_server import JudgeTask, run_judge @@ -628,7 +736,13 @@ async def agent_fn( parent_track_data={"runId": "run-1"}, ) - assert await run_judge(task, [handler]) is None + with caplog.at_level(logging.WARNING, logger="launchdarkly_ai_server.judges"): + assert await run_judge(task, [handler]) is None + # The same warning run_judges logs when it skips a judge. + assert ( + "Judge 'judge-key' skipped: no handler covers provider 'TestProvider'" + in caplog.messages + ) class TestRunJudgeScoreReporting: diff --git a/packages/client/tests/test_registry.py b/packages/client/tests/test_registry.py index 407ed96e..de7da9de 100644 --- a/packages/client/tests/test_registry.py +++ b/packages/client/tests/test_registry.py @@ -19,11 +19,15 @@ def _make_handler( provides_for: tuple[str, str] | None = ("Test", "messages"), + providers: list[str] | None = None, ) -> ProviderHandler: async def fn(config, user_input, tool_handlers, variables): # type: ignore[override] return {"output": "ok"} - return ProviderHandler(fn=fn, provides_for=provides_for) # type: ignore[arg-type] + handler = ProviderHandler(fn=fn, provides_for=provides_for) # type: ignore[arg-type] + if providers is not None: + handler.providers = tuple(providers) # type: ignore[attr-defined] + return handler # --------------------------------------------------------------------------- @@ -63,6 +67,82 @@ def test_duplicate_handler_warns_and_replaces( assert r.handlers[0] is h2 assert any("already registered" in m for m in caplog.messages) + def test_scoped_wildcards_with_different_lists_both_stay(self) -> None: + r = Registry() + bedrock = _make_handler(("*", "agent"), providers=["Bedrock"]) + anthropic = _make_handler(("*", "agent"), providers=["Anthropic"]) + r.register(handlers=[bedrock, anthropic]) + assert r.handlers == [bedrock, anthropic] + + def test_unscoped_wildcard_stays_beside_a_scoped_one(self) -> None: + r = Registry() + unscoped = _make_handler(("*", "agent")) + scoped = _make_handler(("*", "agent"), providers=["Bedrock"]) + r.register(handlers=[unscoped, scoped]) + assert r.handlers == [unscoped, scoped] + + def test_same_provider_set_warns_and_replaces_regardless_of_order( + self, caplog: pytest.LogCaptureFixture + ) -> None: + r = Registry() + first = _make_handler(("*", "agent"), providers=["Anthropic", "Bedrock"]) + second = _make_handler(("*", "agent"), providers=["Bedrock", "Anthropic"]) + with caplog.at_level(logging.WARNING): + r.register(handlers=[first]) + r.register(handlers=[second]) + assert r.handlers == [second] + assert any("already registered" in message for message in caplog.messages) + + def test_repeated_provider_name_is_the_same_identity( + self, caplog: pytest.LogCaptureFixture + ) -> None: + r = Registry() + first = _make_handler(("*", "agent"), providers=["Bedrock"]) + second = _make_handler(("*", "agent"), providers=["Bedrock", "Bedrock"]) + with caplog.at_level(logging.WARNING): + r.register(handlers=[first, second]) + assert r.handlers == [second] + assert any("already registered" in message for message in caplog.messages) + + def test_re_registered_handler_keeps_its_slot( + self, caplog: pytest.LogCaptureFixture + ) -> None: + r = Registry() + bg = _make_handler(("*", "messages"), providers=["Bedrock", "Gemini"]) + ba = _make_handler(("*", "messages"), providers=["Bedrock", "Anthropic"]) + bg_again = _make_handler(("*", "messages"), providers=["Gemini", "Bedrock"]) + with caplog.at_level(logging.WARNING): + r.register(handlers=[bg, ba]) + r.register(handlers=[bg_again]) + assert r.handlers == [bg_again, ba] + + def test_exact_handlers_collide_whatever_their_provider_list( + self, caplog: pytest.LogCaptureFixture + ) -> None: + r = Registry() + plain = _make_handler(("OpenAI", "messages")) + listed = _make_handler(("OpenAI", "messages"), providers=["OpenAI"]) + with caplog.at_level(logging.WARNING): + r.register(handlers=[plain, listed]) + assert r.handlers == [listed] + + def test_duplicate_warning_lists_names_in_sorted_order( + self, caplog: pytest.LogCaptureFixture + ) -> None: + r = Registry() + with caplog.at_level(logging.WARNING): + r.register( + handlers=[ + _make_handler(("*", "messages"), providers=["Gemini", "Bedrock"]) + ] + ) + r.register( + handlers=[ + _make_handler(("*", "messages"), providers=["Bedrock", "Gemini"]) + ] + ) + assert any("('Bedrock', 'Gemini')" in message for message in caplog.messages) + def test_register_replaces_across_mode_aliases( self, caplog: pytest.LogCaptureFixture ) -> None: @@ -141,6 +221,18 @@ def test_b_overrides_a_on_handler_conflict(self) -> None: assert len(result.handlers) == 1 assert result.handlers[0] is h_b + def test_compose_keeps_scoped_wildcards_with_different_lists(self) -> None: + bedrock = _make_handler(("*", "agent"), providers=["Bedrock"]) + anthropic = _make_handler(("*", "agent"), providers=["Anthropic"]) + result = compose(Registry(handlers=[bedrock]), Registry(handlers=[anthropic])) + assert result.handlers == [bedrock, anthropic] + + def test_compose_replaces_the_same_provider_set(self) -> None: + first = _make_handler(("*", "agent"), providers=["Bedrock", "Anthropic"]) + second = _make_handler(("*", "agent"), providers=["Anthropic", "Bedrock"]) + result = compose(Registry(handlers=[first]), Registry(handlers=[second])) + assert result.handlers == [second] + def test_b_overrides_a_across_mode_aliases(self) -> None: h_a = _make_handler(("OpenAI", "messages")) h_b = _make_handler(("OpenAI", "completion")) diff --git a/packages/client/tests/test_utils.py b/packages/client/tests/test_utils.py index ca8b38fd..c40cbcf4 100644 --- a/packages/client/tests/test_utils.py +++ b/packages/client/tests/test_utils.py @@ -9,6 +9,7 @@ import pytest from launchdarkly_ai_server import ( + ProviderHandler, create_handler, make_track_data, model_stamps_from_meta, @@ -196,6 +197,103 @@ async def fn( h = create_handler(("MyProvider", "messages"), fn) assert h.provides_for == ("MyProvider", "messages") + def test_providers_defaults_to_none(self) -> None: + async def fn( + config: object, + user_input: object, + tool_handlers: object, + variables: object, + history: object = None, + ) -> dict: # type: ignore[override] + return {"output": "ok"} + + h = create_handler(("*", "messages"), fn) + assert h.providers is None + + def test_attaches_providers(self) -> None: + async def fn( + config: object, + user_input: object, + tool_handlers: object, + variables: object, + history: object = None, + ) -> dict: # type: ignore[override] + return {"output": "ok"} + + h = create_handler(("*", "messages"), fn, providers=["Bedrock", "Anthropic"]) + assert h.provides_for == ("*", "messages") + assert h.providers == ("Bedrock", "Anthropic") + + def test_rejects_an_empty_provider_list(self) -> None: + async def fn( + config: object, + user_input: object, + tool_handlers: object, + variables: object, + history: object = None, + ) -> dict: # type: ignore[override] + return {"output": "ok"} + + with pytest.raises(ValueError, match="providers"): + create_handler(("*", "messages"), fn, providers=[]) + + def test_rejects_a_blank_provider_name(self) -> None: + async def fn( + config: object, + user_input: object, + tool_handlers: object, + variables: object, + history: object = None, + ) -> dict: # type: ignore[override] + return {"output": "ok"} + + with pytest.raises(ValueError, match="providers"): + create_handler(("*", "messages"), fn, providers=["Bedrock", ""]) + + @pytest.mark.parametrize("name", [" Bedrock", "Bedrock ", "\tBedrock"]) + def test_rejects_a_provider_name_with_surrounding_whitespace( + self, name: str + ) -> None: + async def fn(*args: object, **kwargs: object) -> dict: + return {"output": "ok"} + + with pytest.raises(ValueError, match="providers"): + create_handler(("*", "messages"), fn, providers=[name]) + + def test_rejects_a_bare_string(self) -> None: + async def fn(*args: object, **kwargs: object) -> dict: + return {"output": "ok"} + + with pytest.raises(ValueError, match="providers"): + create_handler(("*", "messages"), fn, providers="Bedrock") + + def test_drops_repeated_names_keeping_first_seen_order(self) -> None: + async def fn(*args: object, **kwargs: object) -> dict: + return {"output": "ok"} + + h = create_handler( + ("*", "messages"), fn, providers=["Bedrock", "Anthropic", "Bedrock"] + ) + assert h.providers == ("Bedrock", "Anthropic") + + def test_provider_handler_validates_and_copies_providers(self) -> None: + async def fn(*args: object, **kwargs: object) -> dict: + return {"output": "ok"} + + with pytest.raises(ValueError, match="providers"): + ProviderHandler(fn, ("*", "messages"), providers=[]) + names = ["Bedrock"] + h = ProviderHandler(fn, ("*", "messages"), providers=names) + names.append("OpenAI") + assert h.providers == ("Bedrock",) + + def test_provider_handler_takes_providers_by_keyword_only(self) -> None: + async def fn(*args: object, **kwargs: object) -> dict: + return {"output": "ok"} + + with pytest.raises(TypeError): + ProviderHandler(fn, ("*", "messages"), None, None, ["Bedrock"]) # type: ignore[misc] + def test_returns_callable(self) -> None: async def fn( config: object, diff --git a/packages/langchain-agents/README.md b/packages/langchain-agents/README.md index b6eaf9c5..f44f698d 100644 --- a/packages/langchain-agents/README.md +++ b/packages/langchain-agents/README.md @@ -4,6 +4,8 @@ LangChain handler for `launchdarkly-ai-server` using **LangGraph's `StateGraph`* **`provides_for`:** `['*', 'agent']` — matches any flag variation where `meta.mode` is `"agent"` and no more-specific handler is registered. LangChain is a framework adapter, not a provider: it routes through `langchain-anthropic`, `langchain-openai`, and others at runtime based on `config.provider.name`. Use `'*'` so that flags configured with `provider.name = "Anthropic"` or `"OpenAI"` are automatically handled without requiring a separate native handler. +Pass `providers=["Bedrock"]` to `create_langchain_agents_handler`, `langchain_agents`, or `langchain_graph` to accept only those `config.provider.name` values. Names must match the config exactly. `langchain_graph` forwards `providers` to the handler and not to `graph()`. Several scoped handlers can be registered together; the shortest list that contains the name wins. Omit `providers` to keep matching every provider. + ## Installation ```bash diff --git a/packages/langchain-agents/api-report/launchdarkly_ai_langchain_agents.txt b/packages/langchain-agents/api-report/launchdarkly_ai_langchain_agents.txt index 4a23ab21..f6097771 100644 --- a/packages/langchain-agents/api-report/launchdarkly_ai_langchain_agents.txt +++ b/packages/langchain-agents/api-report/launchdarkly_ai_langchain_agents.txt @@ -1,5 +1,5 @@ # Public API of launchdarkly_ai_langchain_agents. Generated by `make api-report`; do not edit. -create_langchain_agents_handler(llm: Any = None, *, capture_content: bool | None = None) -> ProviderHandler +create_langchain_agents_handler(llm: Any = None, *, providers: Sequence[str] | None = None, capture_content: bool | None = None) -> ProviderHandler langchain_agents(config_key: str, user_input: str, context: LDContext, **kwargs: Any) -> Any langchain_graph(key: str, llm: Any = None, **options: Any) -> Any to_lang_graph(def_promise: Any, opts: dict[str, Any] | None = None) -> Any diff --git a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/graph.py b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/graph.py index 4c83f00d..ad896476 100644 --- a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/graph.py +++ b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/graph.py @@ -16,6 +16,14 @@ def langchain_graph(key: str, llm: Any = None, **options: Any) -> Any: Equivalent to ``graph(key, handlers=[create_langchain_agents_handler(llm)], **options)``. Use the base ``graph()`` directly for multi-provider graphs. + + ``providers`` is forwarded to the handler, not to ``graph()``. It limits which + ``config.provider.name`` values the wildcard accepts. """ report_usage("langchain-agents.langchainGraph", PACKAGE_NAME, __version__) - return _graph(key, handlers=[_create_langchain_agents_handler(llm)], **options) + providers = options.pop("providers", None) + return _graph( + key, + handlers=[_create_langchain_agents_handler(llm, providers=providers)], + **options, + ) diff --git a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py index 5346a412..0e32a4bd 100644 --- a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py +++ b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py @@ -8,7 +8,7 @@ import asyncio import inspect import json -from collections.abc import AsyncGenerator, Mapping +from collections.abc import AsyncGenerator, Mapping, Sequence from typing import Any from launchdarkly_ai_server import ( @@ -632,7 +632,10 @@ def _run_usage_from_messages(messages: list[Any]) -> Any: def create_langchain_agents_handler( - llm: Any = None, *, capture_content: bool | None = None + llm: Any = None, + *, + providers: Sequence[str] | None = None, + capture_content: bool | None = None, ) -> ProviderHandler: """Creates a ``ProviderHandler`` for LangChain via ``create_react_agent``. @@ -644,15 +647,24 @@ def create_langchain_agents_handler( which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default for this handler. Conversation content is PII, so a run emits only metadata, meaning models, token counts, timings and tool names, until capture is turned on. + + *providers* limits which ``config.provider.name`` values this wildcard accepts. + Omit it to match every provider. Names must match the config exactly, for example + ``["Bedrock"]``. """ report_usage( "langchain-agents.createLangChainAgentsHandler", PACKAGE_NAME, __version__ ) - return _create_langchain_agents_handler(llm, capture_content=capture_content) + return _create_langchain_agents_handler( + llm, providers=providers, capture_content=capture_content + ) def _create_langchain_agents_handler( - llm: Any = None, *, capture_content: bool | None = None + llm: Any = None, + *, + providers: Sequence[str] | None = None, + capture_content: bool | None = None, ) -> ProviderHandler: """Non-reporting :func:`create_langchain_agents_handler`, used by this package's wrappers.""" content_override = capture_content @@ -810,6 +822,7 @@ def _stream_impl( _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] capture_content=content_override, + providers=providers, ) @@ -991,14 +1004,17 @@ def langchain_agents( **kwargs: Any, ) -> Any: """Convenience wrapper: creates a handler and calls config(...).invoke().""" - # Both are lifted out of kwargs: capture_content configures the handler, variables belong to - # the invocation. Leaving either in would pass it to config(), which takes neither, so a caller - # asking for content on spans got a TypeError instead of content. + # Lifted out of kwargs: capture_content and providers configure the handler, variables belong + # to the invocation. Leaving them in would pass them to config(), which takes none of them, so a + # caller asking for content on spans got a TypeError instead of content. report_usage("langchain-agents.langchainAgents", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) capture_content = kwargs.pop("capture_content", None) + providers = kwargs.pop("providers", None) return _config( key=config_key, - handler=_create_langchain_agents_handler(capture_content=capture_content), + handler=_create_langchain_agents_handler( + providers=providers, capture_content=capture_content + ), **kwargs, ).invoke(user_input, context, variables=variables) diff --git a/packages/langchain-agents/tests/test_graph.py b/packages/langchain-agents/tests/test_graph.py index e6ea186f..7a621198 100644 --- a/packages/langchain-agents/tests/test_graph.py +++ b/packages/langchain-agents/tests/test_graph.py @@ -40,6 +40,15 @@ def test_user_cannot_override_handlers(self) -> None: assert "handlers" in kw assert len(kw["handlers"]) == 1 + def test_providers_forwarded_to_the_handler(self) -> None: + with patch("launchdarkly_ai_langchain_agents.graph._graph") as mock_graph: + mock_graph.return_value = MagicMock() + langchain_graph("key", providers=["Bedrock"]) + kw = mock_graph.call_args[1] + assert "providers" not in kw + assert kw["handlers"][0].providers == ("Bedrock",) + assert kw["handlers"][0].provides_for == ("*", "agent") + def test_llm_option_forwarded(self) -> None: mock_llm = MagicMock() with patch("launchdarkly_ai_langchain_agents.graph._graph") as mock_graph: diff --git a/packages/langchain-agents/tests/test_handler.py b/packages/langchain-agents/tests/test_handler.py index a68c42a4..9bd7f537 100644 --- a/packages/langchain-agents/tests/test_handler.py +++ b/packages/langchain-agents/tests/test_handler.py @@ -189,6 +189,19 @@ def test_provides_for_values_are_correct(self) -> None: h = create_langchain_agents_handler() assert h.provides_for == ("*", "agent") + def test_providers_is_unset_by_default(self) -> None: + h = create_langchain_agents_handler() + assert h.providers is None + + def test_providers_scopes_the_wildcard(self) -> None: + h = create_langchain_agents_handler(providers=["Bedrock"]) + assert h.provides_for == ("*", "agent") + assert h.providers == ("Bedrock",) + + def test_empty_providers_is_rejected(self) -> None: + with pytest.raises(ValueError, match="providers"): + create_langchain_agents_handler(providers=[]) + def test_multiple_calls_return_independent_instances(self) -> None: h1 = create_langchain_agents_handler() h2 = create_langchain_agents_handler() @@ -1354,6 +1367,21 @@ def test_config_key_forwarded_as_key(self) -> None: "hello", ctx, variables=None ) + def test_providers_is_forwarded_to_the_handler(self) -> None: + mock_config_instance = MagicMock() + mock_config_fn = MagicMock(return_value=mock_config_instance) + mock_config_instance.invoke = MagicMock(return_value="result") + + with patch.object(handler_mod, "_config", mock_config_fn): + from launchdarkly_ai_langchain_agents.handler import langchain_agents + + ctx = {"kind": "user", "key": "u1"} + langchain_agents("my-flag", "hello", ctx, providers=["Bedrock"]) + + call_kwargs = mock_config_fn.call_args.kwargs + assert "providers" not in call_kwargs + assert call_kwargs["handler"].providers == ("Bedrock",) + def test_callable_without_extra_kwargs(self) -> None: mock_config_instance = MagicMock() diff --git a/packages/langchain-messages/README.md b/packages/langchain-messages/README.md index abe5e2b1..724ef0e9 100644 --- a/packages/langchain-messages/README.md +++ b/packages/langchain-messages/README.md @@ -4,6 +4,8 @@ LangChain handler for `launchdarkly-ai-server` using **LangChain chat models** ( **`provides_for`:** `['*', 'messages']` — matches any flag variation where `meta.mode` is `"messages"` and no more-specific handler is registered. LangChain is a framework adapter, not a provider: it routes through `langchain-anthropic`, `langchain-openai`, and others at runtime based on `config.provider.name`. Use `'*'` so that flags configured with `provider.name = "Anthropic"` or `"OpenAI"` are automatically handled without requiring a separate native handler. +Pass `providers=["Bedrock"]` to `create_langchain_messages_handler` (and to `langchain_messages`) to accept only those `config.provider.name` values. Names must match the config exactly. Several scoped handlers can be registered together; the shortest list that contains the name wins. Omit `providers` to keep matching every provider. + ## Installation ```bash diff --git a/packages/langchain-messages/api-report/launchdarkly_ai_langchain_messages.txt b/packages/langchain-messages/api-report/launchdarkly_ai_langchain_messages.txt index c9602d48..25b2a6d1 100644 --- a/packages/langchain-messages/api-report/launchdarkly_ai_langchain_messages.txt +++ b/packages/langchain-messages/api-report/launchdarkly_ai_langchain_messages.txt @@ -1,3 +1,3 @@ # Public API of launchdarkly_ai_langchain_messages. Generated by `make api-report`; do not edit. -create_langchain_messages_handler(llm: Any = None, *, capture_content: bool | None = None) -> ProviderHandler +create_langchain_messages_handler(llm: Any = None, *, providers: Sequence[str] | None = None, capture_content: bool | None = None) -> ProviderHandler langchain_messages(config_key: str, user_input: str, context: LDContext, llm: Any = None, **kwargs: Any) -> Any diff --git a/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py b/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py index 78f36622..e5d53ac1 100644 --- a/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py +++ b/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py @@ -2,7 +2,7 @@ import asyncio import json -from collections.abc import AsyncGenerator, Mapping +from collections.abc import AsyncGenerator, Mapping, Sequence from types import SimpleNamespace from typing import Any @@ -762,7 +762,10 @@ async def _run_structured_turn( def create_langchain_messages_handler( - llm: Any = None, *, capture_content: bool | None = None + llm: Any = None, + *, + providers: Sequence[str] | None = None, + capture_content: bool | None = None, ) -> ProviderHandler: """ Creates a ``ProviderHandler`` for LangChain (chat models). @@ -777,13 +780,22 @@ def create_langchain_messages_handler( which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default for this handler. Conversation content is PII, so a run emits only metadata, meaning models, token counts, timings and tool names, until capture is turned on. + + *providers* limits which ``config.provider.name`` values this wildcard accepts. + Omit it to match every provider. Names must match the config exactly, for example + ``["Bedrock"]``. """ report_usage("langchain-messages.createLangChainHandler", PACKAGE_NAME, __version__) - return _create_langchain_messages_handler(llm, capture_content=capture_content) + return _create_langchain_messages_handler( + llm, providers=providers, capture_content=capture_content + ) def _create_langchain_messages_handler( - llm: Any = None, *, capture_content: bool | None = None + llm: Any = None, + *, + providers: Sequence[str] | None = None, + capture_content: bool | None = None, ) -> ProviderHandler: """Non-reporting :func:`create_langchain_messages_handler`, used by this package's wrappers.""" content_override = capture_content @@ -1104,6 +1116,7 @@ def _stream_impl( _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] capture_content=content_override, + providers=providers, ) @@ -1501,16 +1514,17 @@ def langchain_messages( **kwargs: Any, ) -> Any: """Convenience wrapper: creates a handler and calls config(...).invoke().""" - # Both are lifted out of kwargs: capture_content configures the handler, variables belong to - # the invocation. Leaving either in would pass it to config(), which takes neither, so a caller - # asking for content on spans got a TypeError instead of content. + # Lifted out of kwargs: capture_content and providers configure the handler, variables belong + # to the invocation. Leaving them in would pass them to config(), which takes none of them, so a + # caller asking for content on spans got a TypeError instead of content. report_usage("langchain-messages.langchainMessages", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) capture_content = kwargs.pop("capture_content", None) + providers = kwargs.pop("providers", None) return _config( key=config_key, handler=_create_langchain_messages_handler( - llm, capture_content=capture_content + llm, providers=providers, capture_content=capture_content ), **kwargs, ).invoke(user_input, context, variables=variables) diff --git a/packages/langchain-messages/tests/test_handler.py b/packages/langchain-messages/tests/test_handler.py index d18aed5f..9281355d 100644 --- a/packages/langchain-messages/tests/test_handler.py +++ b/packages/langchain-messages/tests/test_handler.py @@ -197,6 +197,27 @@ def test_provides_for_is_the_wildcard_provider(self) -> None: h = create_langchain_messages_handler(llm=_make_llm()) assert h.provides_for == ("*", "messages") + def test_providers_is_unset_by_default(self) -> None: + from launchdarkly_ai_langchain_messages import create_langchain_messages_handler + + h = create_langchain_messages_handler(llm=_make_llm()) + assert h.providers is None + + def test_providers_scopes_the_wildcard(self) -> None: + from launchdarkly_ai_langchain_messages import create_langchain_messages_handler + + h = create_langchain_messages_handler( + llm=_make_llm(), providers=["Bedrock", "Anthropic"] + ) + assert h.provides_for == ("*", "messages") + assert h.providers == ("Bedrock", "Anthropic") + + def test_empty_providers_is_rejected(self) -> None: + from launchdarkly_ai_langchain_messages import create_langchain_messages_handler + + with pytest.raises(ValueError, match="providers"): + create_langchain_messages_handler(llm=_make_llm(), providers=[]) + def test_multiple_calls_return_independent_instances(self) -> None: from launchdarkly_ai_langchain_messages import create_langchain_messages_handler @@ -1136,6 +1157,24 @@ def test_calls_through_to_model_call(self) -> None: "hello", ctx, variables=None ) + def test_providers_is_forwarded_to_the_handler(self) -> None: + import launchdarkly_ai_langchain_messages.handler as handler_mod + + mock_config_instance = MagicMock() + mock_config_fn = MagicMock(return_value=mock_config_instance) + mock_config_instance.invoke = MagicMock(return_value="result") + + with patch.object(handler_mod, "_config", mock_config_fn): + from launchdarkly_ai_langchain_messages.handler import langchain_messages + + ctx = {"kind": "user", "key": "u1"} + langchain_messages("my-flag", "hello", ctx, providers=["Bedrock"]) + + call_kwargs = mock_config_fn.call_args.kwargs + assert "providers" not in call_kwargs + assert call_kwargs["handler"].providers == ("Bedrock",) + assert call_kwargs["handler"].provides_for == ("*", "messages") + def test_callable_without_extra_kwargs(self) -> None: import launchdarkly_ai_langchain_messages.handler as handler_mod