[azure-ai-projects] Add Voice Agents realtime client, samples, and tests aligned with .beta structure - #48965
xitzhang (xitzhang) wants to merge 9 commits into
Conversation
…ta structure feature/azure-ai-projects/vnext already has the generated Voice Agents surface (.beta.agent_endpoint_conversations, .beta.agent_telephony, .beta.agents telephony binding/call/generate methods, .beta.voice_agent_web_socket) via PR #48950. This change adds the missing hand-written realtime WebSocket client and the surrounding samples/tests/docs to make Voice Agents a complete, usable feature, all under the .beta naming convention: - azure/ai/projects/_realtime.py and aio/_realtime.py: hand-written realtime WebSocket client (Realtime/AsyncRealtime, RealtimeConnection(Manager)), exposed as client.beta.realtime / async_client.beta.realtime via a new property on BetaOperations. Exported from azure.ai.projects.operations / azure.ai.projects.aio.operations (not the top-level package), matching how other .beta-only classes are exported. - 11 samples under samples/agents/voice/: CRUD lifecycle, guided generation, versioning, tools, live text/audio conversations, function-tool calling, and reading back persisted conversations/audio. - 14 test files under tests/agents/ and tests/foundry_features_header/: CRUD, telephony, telephony campaigns, conversations, realtime client (mocked + live), and telephony protocol tests. - Supporting test infra: conftest.py Foundry-Features/Accept sanitizers, test_base.py foundry_voice_model_name, and a NON_OPERATION_BETA_ATTRIBUTES exclusion in the generic .beta header-injection test (realtime is a hand-written WebSocket entry point, not a generated REST operation, so it can't be exercised by that generic mechanism -- it has its own dedicated header test in test_realtime_client.py). - PostEmitter.ps1: regression guard so a future tsp-client update can't silently overwrite _realtime.py's SDK client-identification fix (PR #48848) without failing the emit step. - pyproject.toml: optional ealtime extra (websockets / aiohttp). - CHANGELOG.md, README.md, .env.template, dev_requirements.txt updated. Renames applied while porting (vnext's TypeSpec commit is newer than the source branch this was ported from, and renamed several methods beyond just .beta nesting -- verified against vnext's own generated code and docs/public-methods.md): - agent_endpoint_conversations: dropped the "agent_conversation" infix and renamed *_content methods to download_* (e.g. get_agent_conversation_item -> get_item, get_agent_conversation_item_audio_content -> download_item_audio). - agent_telephony: dropped the redundant "telephony_" infix (e.g. create_telephony_call_job -> create_call_job, begin_validate_telephony_campaign -> begin_validate_campaign). - agents.generate_agent -> agents.generate, and telephony binding/call methods moved from top-level client.agents to client.beta.agents. Validated: black/pylint/mypy/pyright all clean. Full test suite: 219 passed/74 skipped/0 failed outside voice; voice+header suite 732 passed/49 skipped/8 failed (all 8 are brand-new tests with no recording yet, not a regression). Live-tested all 11 samples end-to-end against a real Foundry project with a real gpt-realtime deployment: real HTTP status codes, real multi-turn realtime conversations with byte-accurate audio sizing, real function-tool invocation, and real conversation/audio persistence+readback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
This comment has been minimized.
This comment has been minimized.
| model=model, | ||
| instructions="You are a friendly voice assistant. Keep replies short and natural.", | ||
| audio=VoiceAgentAudioConfig( | ||
| output=VoiceAgentAudioOutputConfig(voice="en-US-AvaNeural", voice_type=VoiceType.AZURE_STANDARD), |
There was a problem hiding this comment.
Regarding voice="en-US-AvaNeural" I expected to see a selection from an enum instead of writing a string. Do we defined all voices as a union in TypeSpec? If not, why?
There was a problem hiding this comment.
I checked the generated model: voice is intentionally Optional[str], while voice_type is already an extensible VoiceType enum. I think service added this is also to avoid unnecessary API version changes when voices are added or removed. Since this comes from TypeSpec, we can revisit it at the spec level if needed.
| DefaultAzureCredential() as credential, | ||
| AIProjectClient(endpoint=endpoint, credential=credential, allow_preview=True) as project_client, | ||
| ): | ||
| agent = project_client.beta.agents.generate(GenerateVoiceAgentRequest(kind=AgentKind.VOICE, name=agent_name)) |
There was a problem hiding this comment.
".agents.generate" is a too generic name. We'll see where it lands in our sub-client discussions. Depending on what sub-client it's on, we may or may not need a more descriptive name. If it's ".beta.voice-agent.generate" then the name is probably okay.
There was a problem hiding this comment.
Also can we eliminate kind as parameter?
There was a problem hiding this comment.
Makes sense -- this one's out of my hands in this PR though: .beta.agents.generate is already the shape that landed in the generated code this branch picked up (from #48950). I’ll summarize the fixes related to the TypeSpec.
There was a problem hiding this comment.
Also can we eliminate kind as parameter?
Given the design goal of generate, Dan’s view is that it shouldn’t be limited to voice and may support broader generative scenarios in the future. Since generate is intended as a simple way to deploy different kinds of agents, it may make sense to keep kind here.
| DefaultAzureCredential() as credential, | ||
| AIProjectClient(endpoint=endpoint, credential=credential, allow_preview=True) as project_client, | ||
| ): | ||
| agent = project_client.beta.agents.generate(GenerateVoiceAgentRequest(kind=AgentKind.VOICE, name=agent_name)) |
There was a problem hiding this comment.
Also can we eliminate kind as parameter?
| # definition as-is (instead of reconstructing a new one from a few fields) so audio, | ||
| # greeting, tools, and any other service-selected settings are preserved. | ||
| definition.store = True # type: ignore[attr-defined] | ||
| project_client.agents.create_version( |
There was a problem hiding this comment.
should create_version internally calls generate, and make generate private?
There was a problem hiding this comment.
Good question! I'd keep them separate — generate is for AI-assisted scaffolding, create_version needs to stay generic across all agent kinds. Also I will use create agent api here, not generate and change version. Will update samples
| api_version: Optional[str] = None, | ||
| credential_scopes: Optional[List[str]] = None, | ||
| extra_query: Optional[Mapping[str, str]] = None, | ||
| extra_headers: Optional[Mapping[str, str]] = None, |
There was a problem hiding this comment.
Darren Cohen (@dargilco) should extra headers and extra query part of kwargs for our standard?
Review feedback fixes:
- CSpell: add "redef" to sdk/ai/cspell.yaml (fixes CI failure on the
type: ignore[no-redef] comment in
sample_voice_agent_live_audio_conversation_async.py).
- PostEmitter.ps1: remove the _realtime.py regression guard -- verified
empirically (ran a real tsp-client update against the pinned commit; the
hand-written files came back byte-identical) that tsp-client update does
not touch files it doesn't generate, so the guard was unnecessary.
- pyproject.toml: rename the "realtime" optional-dependency extra to
"voice" (matches how it'll be documented for voice agents); update the
2 samples that referenced [realtime] to [voice].
- Samples: pin azure-ai-projects>=2.7.0 (drop the b1/--pre prerelease
pins -- this package no longer ships beta releases) across all 11 voice
samples; use the [voice] extra instead of manually listing aiohttp in
the 2 samples that need it.
- _realtime.py / aio/_realtime.py:
- Remove the foundry_features parameter from
RealtimeConnectionManager/Realtime.connect/AsyncRealtime.connect --
the realtime route is voice-agent-specific, so the header value is
always the same and callers should not need (or be able) to override
it. The value is now a hardcoded module constant.
- Fix structured_inputs' type: it was Optional[str], silently requiring
callers to pre-serialize their own JSON string with no documentation
of the expected shape. Retyped to Optional[Mapping[str, Any]],
matching the analogous generated model field
(CreateTelephonyCallJobRequest.structured_inputs: dict[str, any]),
and now serialized internally via SdkJSONEncoder.
- Updated test_realtime_client.py/_async.py's _make_manager() helpers
to match (foundry_features removed; the header-value assertions still
pass since it's now a module constant rather than a per-call override).
Test recordings:
- Generated live recordings for the 8 new voice-agent tests that had none
yet (test_voice_agent_crud(_async).py x6, test_read_conversation(_async)
x2), against a real Foundry project with a real gpt-realtime deployment.
Verified sanitization (endpoint/auth stripped) and re-verified end-to-end
in playback mode after a fresh assets restore from the new tag.
- Pushed the new recordings to Azure/azure-sdk-assets (tag
python/ai/azure-ai-projects_59c7584f68) via the GitHub Git Data REST API,
since a direct git push to that repo hangs in this environment; updated
assets.json accordingly.
Validated: black/pylint/mypy clean; full existing test suite unaffected
(740 passed, 49 skipped, 0 failed, up from 732 passed/8 failed).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Glenn Harper (glharper)
left a comment
Glenn Harper (glharper)
left a comment
There was a problem hiding this comment.
Comments 2-6 from the local review, checked against the current PR revision.
| except TimeoutError: | ||
| print("Timed out waiting for the agent's reply.") | ||
| conn.response.cancel() | ||
| return |
There was a problem hiding this comment.
5. [P2] Drain a cancelled response before accepting the next text turn
After a reply times out, the samples send response.cancel() and return to prompting without consuming the cancelled response's terminal event. The next turn's event pump returns on any RealtimeServerEventResponseDone, without matching the response ID. A late completion for the cancelled turn therefore ends the next turn's pump before the new reply is displayed, leaving the stream out of step with the user. The same issue exists in the async text sample.
Evidence: Deterministic sync and async runs supplied a first-turn timeout, the first response's cancelled response.done, and the second response's transcript/completion. Both issued two response requests but failed to display the second reply; the async run left its transcript and completion unread.
Suggested fix: Consume cancellation through the matching terminal event before reusing the stream, and correlate completion events with the active response ID. If cancellation cannot be confirmed within a bounded wait, end the session rather than accept another turn on an unsynchronized stream. Apply the same handling to the async greeting-timeout path.
There was a problem hiding this comment.
Fixed in both the sync and async samples (including the async greeting-timeout path). Each turn now tracks its response id from response.created, and after a timeout+cancel, a new bounded drain step consumes that response's terminal event (correlated by id) before the next turn starts — so a late completion can no longer be mistaken for the next turn's reply. If it can't be confirmed in time, the sample now ends the session cleanly instead of continuing on a stream it can't trust.
| except HttpResponseError as e: | ||
| if e.status_code == 404: | ||
| continue |
There was a problem hiding this comment.
6. [P2] Require a successful per-item audio retrieval in the conversation tests
Both tests tolerate every per-item audio request returning 404 without requiring a single successful audio retrieval. Their setup requests a stored audio response and requires a completed conversation with a merged recording, but a broken per-item audio route can still pass the test without ever reaching download_item_audio(). Recording that behavior would preserve the false-positive result in playback. The same issue exists in the async test.
Evidence: Running both unchanged test bodies with mocked completed conversations containing user and assistant-audio items passed despite two item-audio 404s and zero item-audio download calls in each run. This reproduction bypassed live setup and recording decorators; it did not contact the service.
Suggested fix: Continue tolerating 404 for genuinely non-audio items, but require at least one successful audio-item metadata retrieval and a nonempty download for Foundry-managed storage.
There was a problem hiding this comment.
Fixed in both the sync and async test — now tracks whether any item successfully retrieved audio and asserts that after the loop, so a fully-broken get_item_audio/download_item_audio route can no longer pass silently.
…nt samples to use create_version directly - _realtime.py / aio/_realtime.py: send structured_inputs as the documented structured_input query parameter instead of a custom header; distinguish graceful WebSocket closure from abnormal closure/transport errors so `for event in conn:` no longer silently swallows real failures. - sample_voice_agent_live_audio_conversation_async.py: always clear buffered local playback audio on barge-in, independent of whether a server response is still active. - sample_voice_agent_live_text_conversation.py / _async.py: drain a cancelled response's terminal event (correlated by response id) before starting the next turn, so a late completion can't be mistaken for the next reply. - test_voice_agent_conversations.py / _async.py: require at least one successful per-item audio retrieval instead of tolerating an all-404 run. - test_realtime_client.py / _async.py: add regression tests for the structured_input query parameter and the graceful-vs-abnormal closure distinction. - sample_voice_agent_live_audio_conversation_async.py, sample_voice_agent_live_text_conversation.py, sample_voice_agent_live_text_conversation_async.py: replace the generate()-then-patch-store two-step pattern with a single direct VoiceAgentDefinition(..., store=True) + create_version() call, matching sample_voice_agent_basic.py. Live-tested all three end-to-end. - GeneratePublicMethods.ps1 / docs/public-methods.md / api.md / api.metadata.yml: fix the generator to recognize the hand-written beta.realtime sub-client (a plain property, not a generated *Operations group) and regenerate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… into xitzhang/voice-agents-beta
A parallel merge from feature/azure-ai-projects/vnext brought in a docstring bullet-list join fix (for a Sphinx warning) that pushed one line over pylint's 120-char limit. Re-wrap with correct RST continuation indentation; verified clean with docutils (no Sphinx warnings) and pylint (10.00/10). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… into xitzhang/voice-agents-beta Realigns hand-written realtime client code with the new voice_agents sub-client hierarchy merged from vnext (PR #49003): beta.realtime -> beta.voice_agents.realtime, beta.agent_endpoint_conversations -> beta.voice_agents.conversations, beta.agent_telephony/agents.*_telephony_* -> beta.voice_agents.telephony, beta.agents.generate -> beta.agents.create_from_prompt. Removes the now-dropped agent_version_override handshake parameter, updates all samples/tests/docs to match, and regenerates docs/public-methods.md and api.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Realtime/AsyncRealtime and their _ConfigProvider docstrings incorrectly said they could be constructed from either the top-level client or .beta.voice_agents; in practice only BetaVoiceAgentsOperations ever constructs them. Corrected the docstrings/type hints to match. - Reformatted 4 telephony test files with black (eng/black-pyproject.toml) after the longer beta.voice_agents.telephony identifier pushed some lines past 120 chars. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Renames the 6 hand-written voice-agent realtime classes to align with the generated Beta<Group>Operations naming convention used throughout the rest of the .beta.voice_agents.* hierarchy (BetaVoiceAgentsOperations, BetaVoiceAgentsConversationsOperations, BetaVoiceAgentsTelephonyOperations): - Realtime -> BetaRealtime - RealtimeConnection -> BetaRealtimeConnection - RealtimeConnectionManager -> BetaRealtimeConnectionManager - AsyncRealtime -> AsyncBetaRealtime - AsyncRealtimeConnection -> AsyncBetaRealtimeConnection - AsyncRealtimeConnectionManager -> AsyncBetaRealtimeConnectionManager Updated everywhere these are referenced: class definitions and __all__ in _realtime.py/aio/_realtime.py, imports/type-hints/__all__ in operations/_patch.py and its async twin, the hand-written-subclient detection set in docs/GeneratePublicMethodsDoc.ps1, samples that import/type-hint the connection classes, the realtime unit test suites, a docstring comment in foundry_features_header_test_base.py, and CHANGELOG.md. Regenerated docs/public-methods.md and api.md/api.metadata.yml to match. Generated model files (_enums.py/_models.py) were left untouched -- their "Realtime" mentions are generic OpenAI Realtime API prose, not references to these classes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
[Pilot] PR Pipeline Failure AnalysisWhat failedAzure Pipeline build 6835391 (
Relevant pipeline outputRecommended next steps
Automated fix: Fix found, view and apply fix
|
4 participants
Summary
Adds Voice Agents to �zure-ai-projects, aligned with the .beta naming/structure already established on this branch ( eature/azure-ai-projects/vnext picked up the generated Voice Agents surface via #48950). This PR adds the pieces needed to make it a complete, usable feature: the hand-written realtime WebSocket client, samples, tests, and docs.
Related to #47803 (tracking �next -> main). cc Darren Cohen (@dargilco)
What's included
ealtime is a hand-written WebSocket entry point, not a generated REST operation, so it needs its own dedicated header test instead -- see est_realtime_client.py).
ealtime extra (websockets / �iohttp).
Renames applied while porting
This branch's TypeSpec commit turned out to be newer than the source content this was ported from, and had renamed several methods beyond just .beta nesting (verified against this branch's own generated code and docs/public-methods.md):
Validation