[Bug]: Support Foundry OpenAI v1 and Responses API endpoints in live evaluations

[aprilgittens]

Pre-flight checklist

• I have searched existing issues and this is not a duplicate.
• I have read the FAQ.
• I have removed any API keys, endpoints, or other secrets from logs and config snippets I paste below.

Summary

Live evaluations returned 404 Resource not found for every Model Router and baseline request, despite valid keys and deployments.

The repository configured all endpoints using AsyncAzureOpenAI and hard-coded chat.completions.create(). However, current Foundry endpoints use the OpenAI v1 format: https://<resource>.services.ai.azure.com/openai/v1

Model Router uses /chat/completions, while some baseline and judge deployments use /responses. Supplying the full portal Target URI did not work because the SDK appended another operation path. Removing part of the resource hostname caused APIConnectionError because the resulting hostname did not exist.

A secondary error occurred after all requests failed: AttributeError: 'NoneType' object has no attribute 'get'

The verifier assumed cost and latency metrics were dictionaries, but all-error runs serialize them as null.

I used GitHub Copilot to resolve the issue and I'm sharing it's summarized proposed fix below:

1. Add an endpoint api_mode setting supporting: chat_completions and responses
2. Use AsyncOpenAI with the Foundry /openai/v1 base URL.
3. Map request and token fields appropriately for each API.
4. Configure Model Router for Chat Completions and baseline/judge models for Responses.
5. Reject URLs ending in /chat/completions or /responses with an actionable configuration error.
6. Update .env.example, presets, documentation, and SDK dependency requirements.
7. Treat null cost and latency metrics as missing data instead of crashing verification.

When validating this fix, all three configured live deployments succeeded with minimal smoke requests. The non-live suite passed with 181 tests, along with lint, dependency checks, and configuration dry-run validation.

Steps to reproduce

1. Update .env baseline model to gpt-5.6-sol and the judge model to gpt-5.5
2. Set pricing region to westus3
3. In live_demo.yaml replace dataset with a custom dataset.
4. Run .\scripts\demo.ps1 -Live

Expected behaviour

Evaluation should happen, including the generated report.

Actual behaviour

Model Router errors: 10/10 [nwt_001] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_002] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_003] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_004] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_006] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_005] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_007] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_008] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_010] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_009] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} Baseline errors: 10/10 [nwt_001] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_002] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_003] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_004] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_006] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_005] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_007] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_008] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_010] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} [nwt_009] NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} Traceback (most recent call last): File "C:\Users\apspeigh\Documents\Model-Router-Auto-Evaluation\scripts\run_eval.py", line 167, in <module> main()

File "C:\Users\apspeigh\Documents\Model-Router-Auto-Evaluation\scripts\run_eval.py", line 160, in main
vr = verify_local_eval(config.output_directory)
File "C:\Users\apspeigh\Documents\Model-Router-Auto-Evaluation\src\verify.py", line 102, in verify_local_eval
if cost.get("estimated_cost_usd") is not None:
^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
Exception: C:\Users\apspeigh\Documents\Model-Router-Auto-Evaluation\scripts\demo.ps1:65:13
Line |
65 | throw "Live evaluation failed."
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Live evaluation failed.

### Which part of the pipeline is affected?

Other / not sure

### Python version

3.13.15

### Operating system

Windows 11

### Repo commit or release

_No response_

### Relevant configuration

```yaml

```

### Logs and screenshots

```shell

```

### Additional context

I have the updated files w/ the GitHub Copilot fix if you'd like it.

[leestott]

April Gittens (@aprilgittens) can you pr the fix Sanjeev Jagtap (@sanjeev3)