Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion doc/code/registry/0_registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,12 @@ reconstruction:

Use `source.params` for overrides; do not also supply `params`. The backend
keeps the source's other constructor inputs, including authentication, on the
server. It does not change the source. Missing, changed, or ambiguous sources
server. It does not change the source. A copy also keeps the settings that
choose where requests go and how they sign in, such as `endpoint`, `auth_mode`,
`api_key` and `headers` for OpenAI targets, and the region, key and resource ID
for the Azure Speech converters; overriding one of these is an error. Such
parameters report `copy_locked: true`, and components declare them in
`COPY_LOCKED_PARAMETER_NAMES`. Missing, changed, or ambiguous sources
Comment on lines +134 to +139

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add this rule to the restrictions in the backend README's Input Validation and Trust Boundary section, and state exactly what it guarantees? A copy keeps its source's connection, but creating a new target can still send server keys to an endpoint the caller chooses.

produce an error. If a source was renamed, a unique matching hash can resolve it.
An optional `effective_hash` checks the reconstructed object's identity.
Registered descriptors report `reconstructable` and, for targets,
Expand Down
76 changes: 76 additions & 0 deletions frontend/src/components/Registry/CreateConverterDialog.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,82 @@ describe('CreateConverterDialog', () => {
expect(onTemporary.mock.calls[0][1].hash).toBe('private-hash')
})

it('keeps a copy on its source connection settings', async () => {
const user = userEvent.setup()
const onTemporary = jest.fn()
mockConverterParameters([
{ name: 'azure_speech_region', type_name: 'str', required: false, default: null, copy_locked: true },
{ name: 'synthesis_voice_name', type_name: 'str', required: false, default: 'en-US-AvaNeural' },
], 'AzureSpeechTextToAudioConverter')
const converter = {
converter_id: 'speech',
identifier: {
class_name: 'AzureSpeechTextToAudioConverter', class_module: 'pyrit.converter',
pyrit_version: 'test', hash: 'source-hash',
},
}
mockedConvertersApi.buildConverter.mockResolvedValue({ identifier: { ...converter.identifier, hash: 'copy-hash' } })
renderDialog({ editing: { converter }, onTemporary })

const voice = await screen.findByRole('textbox', { name: /synthesis_voice_name/i })
expect(screen.queryByRole('textbox', { name: /azure_speech_region/i })).not.toBeInTheDocument()
expect(screen.getByText(/connection settings come from the source converter/i)).toBeInTheDocument()
await user.clear(voice)
await user.type(voice, 'en-US-JennyNeural')
await user.click(screen.getByRole('button', { name: 'Apply Settings' }))

await waitFor(() => expect(onTemporary).toHaveBeenCalledTimes(1))
expect(mockedConvertersApi.buildConverter).toHaveBeenCalledWith('AzureSpeechTextToAudioConverter', {
source_name: 'speech', source_hash: 'source-hash', params: { synthesis_voice_name: 'en-US-JennyNeural' },
})
})

it('drops a connection setting carried over from earlier temporary settings', async () => {
const user = userEvent.setup()
const onTemporary = jest.fn()
mockConverterParameters([
{ name: 'azure_speech_region', type_name: 'str', required: false, default: null, copy_locked: true },
{ name: 'synthesis_voice_name', type_name: 'str', required: false, default: 'en-US-AvaNeural' },
{ name: 'output_format', type_name: 'str', required: false, default: 'wav' },
], 'AzureSpeechTextToAudioConverter')
const converter = {
converter_id: 'speech',
identifier: {
class_name: 'AzureSpeechTextToAudioConverter', class_module: 'pyrit.converter',
pyrit_version: 'test', hash: 'source-hash',
},
}
const spec = {
source_name: 'speech', source_hash: 'source-hash',
params: { azure_speech_region: 'westus2', output_format: 'mp3' },
}
mockedConvertersApi.buildConverter.mockResolvedValue({ identifier: { ...converter.identifier, hash: 'copy-hash' } })
renderDialog({ editing: { converter, spec }, onTemporary })

const voice = await screen.findByRole('textbox', { name: /synthesis_voice_name/i })
await user.clear(voice)
await user.type(voice, 'en-US-JennyNeural')
await user.click(screen.getByRole('button', { name: 'Apply Settings' }))

await waitFor(() => expect(onTemporary).toHaveBeenCalledTimes(1))
expect(mockedConvertersApi.buildConverter).toHaveBeenCalledWith('AzureSpeechTextToAudioConverter', {
source_name: 'speech',
source_hash: 'source-hash',
params: { output_format: 'mp3', synthesis_voice_name: 'en-US-JennyNeural' },
})
})

it('offers connection settings when creating a converter', async () => {
mockConverterParameters([
{ name: 'azure_speech_region', type_name: 'str', required: false, default: null, copy_locked: true },
], 'AzureSpeechTextToAudioConverter')
renderDialog()
await selectConverterType('AzureSpeechTextToAudioConverter')

expect(await screen.findByRole('textbox', { name: /azure_speech_region/i })).toBeInTheDocument()
expect(screen.queryByText(/connection settings come from the source converter/i)).not.toBeInTheDocument()
})

it('locks scalar inputs while Apply Settings is building so a newer edit cannot be lost', async () => {
const user = userEvent.setup()
const onTemporary = jest.fn()
Expand Down
24 changes: 20 additions & 4 deletions frontend/src/components/Registry/CreateConverterDialog.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,18 @@ export default function CreateConverterDialog({
() => converterTypes.find((item) => item.converter_type === selectedType),
[converterTypes, selectedType],
)
// Temporary settings can't move a copy to a different endpoint or credential.
const lockedParameterNames = useMemo(
() => new Set((editing ? selectedConverterType?.parameters ?? [] : [])
.filter((parameter) => parameter.copy_locked)
.map((parameter) => parameter.name)),
[selectedConverterType, editing],
)
const editableParameters = useMemo(
() => (selectedConverterType?.parameters ?? []).filter((parameter) => !lockedParameterNames.has(parameter.name)),
[selectedConverterType, lockedParameterNames],
)
const keepsSourceConnection = lockedParameterNames.size > 0
const groupedConverterTypes = useMemo(() => {
const groups = new Map<string, ConverterTypeEntry[]>()
for (const converterType of converterTypes) {
Expand Down Expand Up @@ -594,7 +606,7 @@ export default function CreateConverterDialog({
}

const submit = async () => {
const missingParameters = (selectedConverterType?.parameters ?? []).some(
const missingParameters = editableParameters.some(
(parameter) => parameter.required
&& !parameter.default
&& !acceptsEmptySearchReplaceValue(selectedType, parameter.name)
Expand All @@ -605,7 +617,7 @@ export default function CreateConverterDialog({
return
}

const parameters = (selectedConverterType?.parameters ?? []).filter(
const parameters = editableParameters.filter(
(parameter) => !editing || changedParameters.has(parameter.name),
)
const params: Record<string, unknown> = Object.fromEntries(
Expand Down Expand Up @@ -640,8 +652,9 @@ export default function CreateConverterDialog({
source_name: editing.spec?.source_name ?? editing.converter.converter_id,
source_hash: editing.spec?.source_hash ?? editing.converter.identifier.hash,
params: {
// Settings applied before copies kept their source's connection would now be refused.
...Object.fromEntries(Object.entries(editing.spec?.params ?? {})
.filter(([name]: [string, unknown]) => !changedParameters.has(name))),
.filter(([name]: [string, unknown]) => !changedParameters.has(name) && !lockedParameterNames.has(name))),
...params,
},
}
Expand Down Expand Up @@ -805,8 +818,11 @@ export default function CreateConverterDialog({
/>
</HelpField>}
{editing && <Text>Changes apply only to this stage. The registered converter stays unchanged.</Text>}
{keepsSourceConnection && (
<Text>Connection settings come from the source converter and can&apos;t be changed here.</Text>
)}
<div className={styles.parameterGrid}>
{selectedConverterType?.parameters.map((parameter) => (
{editableParameters.map((parameter) => (
<div key={`${selectedType}:${parameter.name}`} className={styles.parameterRow}>
{parameter.variants || parameter.multiline ? (
<ParameterField
Expand Down
2 changes: 2 additions & 0 deletions frontend/src/types/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -728,6 +728,8 @@ export interface Parameter {
multiline?: boolean
/** Whether the value must be omitted when identity-based authentication is selected. */
identity_conflicting?: boolean
/** Whether a private copy keeps its source's value, because it sets where requests go or how they sign in. */
copy_locked?: boolean
description?: string | null
}

Expand Down
7 changes: 7 additions & 0 deletions pyrit/backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -397,6 +397,13 @@ uses them:
- `HTTPXAPITarget`, which uploads local files, can be created through the API only when
`target_upload_directory` is set in `.pyrit_conf`; it then uploads files only from that
folder, and callers cannot supply their own.
- A copy of a registered target or converter, made from its `source`, keeps the source's
connection: it can't change the parameters its type reports as `copy_locked` (for OpenAI
targets the endpoint, `auth_mode`, key, headers, and HTTP client settings; for the Azure
Speech converters the region, key, and resource ID), and trying returns **400**. So a copy
can't take its source's credentials, or a token for an identity its source was set up with,
to another endpoint or resource. This rule covers copies only: creating a new target can
still send this server's key to an endpoint the caller chooses, as described below.

Targets that need a setting the API does not allow can be registered in Python or with an
initializer.
Expand Down
10 changes: 8 additions & 2 deletions pyrit/backend/models/targets.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,14 @@ class CreateTargetRequest(BaseModel):

@model_validator(mode="after")
def _validate_source(self) -> "CreateTargetRequest":
if self.source is not None and (self.register or self.params or "auth_mode" in self.model_fields_set):
raise ValueError("source requires register=false and overrides, including auth_mode, in source.params")
if self.source is None:
return self
if self.register:
raise ValueError("source requires register=false")
if self.params:
raise ValueError("a copy takes its overrides in source.params")
if "auth_mode" in self.model_fields_set:
raise ValueError("a copy uses its source's auth mode, so auth_mode can't be set")
return self


Expand Down
2 changes: 1 addition & 1 deletion pyrit/backend/services/converter_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ def _build_instance_from_object(self, *, converter_id: str, converter_obj: Any)
from pyrit.registry.resolution import project_parameter_value

for parameter in metadata.parameters if metadata else []:
if not parameter.is_external_input or parameter.sensitive:
if not parameter.is_external_input or parameter.sensitive or parameter.copy_locked:
continue
try:
value = project_parameter_value(
Expand Down
6 changes: 5 additions & 1 deletion pyrit/converter/azure_speech_audio_to_text_converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import logging
import time
from collections.abc import Awaitable, Callable
from typing import TYPE_CHECKING, Any
from typing import TYPE_CHECKING, Any, ClassVar

if TYPE_CHECKING:
import azure.cognitiveservices.speech as speechsdk
Expand Down Expand Up @@ -44,6 +44,10 @@ class AzureSpeechAudioToTextConverter(Converter):
#: The resource ID for accessing the service when using Entra ID auth.
AZURE_SPEECH_RESOURCE_ID_ENVIRONMENT_VARIABLE: str = "AZURE_SPEECH_RESOURCE_ID"

COPY_LOCKED_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset(
{"azure_speech_region", "azure_speech_key", "azure_speech_resource_id"}
)

def __init__(
self,
*,
Expand Down
6 changes: 5 additions & 1 deletion pyrit/converter/azure_speech_text_to_audio_converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

import logging
from collections.abc import Awaitable, Callable
from typing import TYPE_CHECKING, Literal
from typing import TYPE_CHECKING, ClassVar, Literal

if TYPE_CHECKING:
import azure.cognitiveservices.speech as speechsdk # noqa: F401
Expand Down Expand Up @@ -42,6 +42,10 @@ class AzureSpeechTextToAudioConverter(Converter):
#: The resource ID for accessing the service when using Entra ID auth.
AZURE_SPEECH_RESOURCE_ID_ENVIRONMENT_VARIABLE: str = "AZURE_SPEECH_RESOURCE_ID"

COPY_LOCKED_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset(
{"azure_speech_region", "azure_speech_key", "azure_speech_resource_id"}
)

#: Supported audio formats for output.
AzureSpeechAudioFormat = Literal["wav", "mp3"]

Expand Down
4 changes: 4 additions & 0 deletions pyrit/converter/converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ class Converter(Identifiable):
#: Constructor parameters whose controls must preserve line breaks.
MULTILINE_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset()

#: Constructor parameters that choose where requests go or how they authenticate. A private
#: copy built from API input keeps its source's values for these.
COPY_LOCKED_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset()
Comment on lines +67 to +69

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every converter can be copied automatically (constructor capture, base get_reconstruction_parameters), but this list starts empty. A future converter with an endpoint and a key would let copies change the endpoint unless its author remembers to declare it, and nothing in CI would catch that.

Could we add one test: every copyable class with a parameter from get_sensitive_parameter_names() must list it in COPY_LOCKED_PARAMETER_NAMES? It passes today.


#: Capability requirements placed on the converter's target (if any).
#: Subclasses that use a target should override this and pass the target to
#: ``super().__init__(converter_target=...)`` so the base class can validate it.
Expand Down
11 changes: 9 additions & 2 deletions pyrit/models/parameter.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,8 @@ class Parameter(BaseModel):
value's live Python type and its allowed set (a ``Literal[...]`` or ``Enum``
*is* the allowed set) and drives ``coerce_value`` / ``validate``; it is **not**
serialized. Serialization instead projects the type into the display fields
``type_name``, ``choices``, ``is_list``, ``sensitive``, ``multiline``, and
``identity_conflicting`` (plus ``required`` from the ``REQUIRED_VALUE``
``type_name``, ``choices``, ``is_list``, ``sensitive``, ``multiline``,
``identity_conflicting``, and ``copy_locked`` (plus ``required`` from the ``REQUIRED_VALUE``
sentinel), so a consumer can rebuild a usable contract from the registry
without the live type travelling on the wire.

Expand Down Expand Up @@ -150,6 +150,13 @@ class Parameter(BaseModel):
default=False,
description="Whether this parameter must be omitted when identity-based authentication is selected.",
)
copy_locked: bool = Field(
default=False,
description=(
"Whether a private copy keeps its source's value, because the parameter chooses where requests go "
"or how they authenticate."
),
)
destination: ParameterDestination = Field(
default=ParameterDestination.CONSTRUCTOR,
exclude=True,
Expand Down
5 changes: 5 additions & 0 deletions pyrit/prompt_target/common/prompt_target.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,11 @@ class PromptTarget(Identifiable):
# unaffected.
server_credential_parameter: ClassVar[str | None] = None

# Constructor parameters that choose where requests go or how they authenticate. A private
# copy built from API input keeps its source's values for these, so a credential stays
# with the endpoint it was configured for.
COPY_LOCKED_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset()

def __init_subclass__(cls, **kwargs: object) -> None:
"""
Validate that subclasses follow the keyword-only ``__init__`` contract.
Expand Down
4 changes: 4 additions & 0 deletions pyrit/prompt_target/openai/openai_target.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ class OpenAITarget(PromptTarget):
# such as Ollama or vLLM may need no key at all.
supported_auth_modes: ClassVar[tuple[AuthMode, ...]] = ("api_key", "identity", "none")

COPY_LOCKED_PARAMETER_NAMES: ClassVar[frozenset[str]] = frozenset(
{"endpoint", "auth_mode", "api_key", "headers", "httpx_client_kwargs"}
)
Comment on lines +67 to +69

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that target copies with these overrides are only built for a dry run, does the target half belong in this PR? The only target copies that send are temperature bindings, and the open #3506 stops creating them in favor of a per-request setting (it keeps only replay of saved ones). Unless something that sends target copies with these overrides is planned, I'd leave the target lock out, along with its test changes (including the #3809 update), and add it with that consumer.


model_name_environment_variable: str
endpoint_environment_variable: str
api_key_environment_variable: str
Expand Down
11 changes: 10 additions & 1 deletion pyrit/registry/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -982,7 +982,8 @@ def recreate_instance(
InstanceT: The new component, owned by the caller.

Raises:
ValueError: If overrides are unknown or the source class is unavailable.
ValueError: If overrides are unknown, external overrides set a connection setting the
source's class keeps for copies, or the source class is unavailable.
"""
inputs = self.get_reconstruction_parameters(source)
raw_params = dict(params)
Expand Down Expand Up @@ -1011,6 +1012,14 @@ def recreate_instance(
seam_base_type=self._seam_base_type(),
allowed_path_params=allowed_path_params,
)
# A copy keeps where its source sends requests and how it signs in, so a credential
# can't be pointed at a different endpoint. Any value is refused, even the source's own.
locked = sorted(name for name in params if by_name[name].copy_locked)
if locked:
raise ValueError(
f"A copy keeps its source's connection settings, so it can't set {', '.join(locked)}. "
"Create a new one to use different settings."
)
from pyrit.models import SeedPrompt

for name, value in raw_params.items():
Expand Down
10 changes: 10 additions & 0 deletions pyrit/registry/resolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,7 @@ def _parameters_from_signature(
sensitive_parameter_names: frozenset[str],
multiline_parameter_names: frozenset[str],
identity_conflicting_parameter_names: frozenset[str],
copy_locked_parameter_names: frozenset[str],
) -> list[Parameter]:
"""
Build parameters declared by one constructor signature.
Expand All @@ -280,6 +281,8 @@ def _parameters_from_signature(
names whose values require multiline controls.
identity_conflicting_parameter_names (frozenset[str]): Identifier-declared
names that must be omitted for identity-based authentication.
copy_locked_parameter_names (frozenset[str]): Component-declared names whose
values a private copy keeps from its source.

Returns:
list[Parameter]: Parameters declared by the constructor.
Expand Down Expand Up @@ -315,6 +318,7 @@ def _parameters_from_signature(
sensitive=name in sensitive_parameter_names,
multiline=name in multiline_parameter_names,
identity_conflicting=name in identity_conflicting_parameter_names,
copy_locked=name in copy_locked_parameter_names,
)
if parameter.is_seed_prompt:
parameter = parameter.model_copy(update={"multiline": True})
Expand Down Expand Up @@ -342,6 +346,10 @@ def derive_parameters(
of constructor parameters whose controls must preserve line breaks. These
names supplement the identifier's shared multiline names for that component only.

A component can also declare ``COPY_LOCKED_PARAMETER_NAMES``: the constructor
parameters that choose where its requests go or how they authenticate. A private
copy built from external input keeps the source's values for these.

A non-required reference parameter declared by ``seam_base_type`` is a
construction seam, not a build input of every subclass: the base accepts it so
subclasses can forward a value (e.g. for validation) through one construction
Expand Down Expand Up @@ -377,6 +385,7 @@ def derive_parameters(
identity_conflicting_parameter_names = (
identifier_type.get_identity_conflicting_parameter_names() if identifier_type is not None else frozenset[str]()
)
copy_locked_parameter_names = getattr(cls, "COPY_LOCKED_PARAMETER_NAMES", frozenset[str]())
parameters: list[Parameter] = []
seen: set[str] = set()
for owner, signature in _constructor_sources(cls):
Expand All @@ -387,6 +396,7 @@ def derive_parameters(
sensitive_parameter_names=sensitive_parameter_names,
multiline_parameter_names=multiline_parameter_names,
identity_conflicting_parameter_names=identity_conflicting_parameter_names,
copy_locked_parameter_names=copy_locked_parameter_names,
):
if parameter.name in seen:
continue
Expand Down
Loading
Loading