diff --git a/README.md b/README.md index 98f578c..683811c 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,8 @@ Guided setup deploys **one regional endpoint** using JavaScript, C#, or Python. *Easy Auth checks the Microsoft caller before the Function decrypts the request. Live success confirms provider acceptance. Confirm delivery separately. Evaluation validates the request without sending a message. +The diagram is a topology overview; current packages read API-key secrets for each live request, +while Azure Identity owns OAuth token caching. See [architecture details](TECHNICAL.md#single-region-architecture). [Open the full-size diagram](docs/images/single-region-architecture.png).* @@ -74,7 +76,7 @@ Before live use, review [production readiness](TECHNICAL.md#production-readiness ### Advanced topics - [Multi-region routing](docs/FRONTDOOR.md) -- [Provider fallback](docs/MULTI-PROVIDER-IMPLEMENTATION.md) +- [Secondary-provider design](docs/MULTI-PROVIDER-IMPLEMENTATION.md) ## Getting help diff --git a/TECHNICAL.md b/TECHNICAL.md index 1cd726f..08fcaa9 100644 --- a/TECHNICAL.md +++ b/TECHNICAL.md @@ -36,7 +36,7 @@ Authentication failures return **401 or 403**. ## Resiliency and business continuity The [Azure Front Door design](docs/FRONTDOOR.md) adds regional origins behind one public URL. -It covers capacity, shared keys, dependencies, readiness, failover drills, and rollback. +It covers capacity, shared keys, dependencies, readiness, regional recovery drills, and rollback. ## Implementations @@ -159,12 +159,16 @@ provider-neutral delivery object (`OtpDelivery.otp` in JavaScript/Python, `OtpDe It is not converted to a number or extracted from `message`, and is not logged or returned in the HTTP response. Existing built-in provider message and voice mappings remain unchanged. -The optional `riskContext` must be a JSON object or `null`. It is passed unchanged to provider -adapters as `OtpDelivery.riskContext` in JavaScript, `OtpDelivery.RiskContext` +The optional `riskContext` must be a JSON object or `null`. It can contain signals such as a +scenario name or IP address. It is passed unchanged to provider adapters as +`OtpDelivery.riskContext` in JavaScript, `OtpDelivery.RiskContext` (`Dictionary`) in C#, and `OtpDelivery.risk_context` -(`dict[str, object]`) in Python. No individual risk fields are required or interpreted by the -shared handler. Risk data is not logged or returned in HTTP responses, and built-in adapters -do not forward it to providers. Outbound risk-field mapping must be explicit and provider-specific. +(`dict[str, object]`) in Python. The shared handler does not require or interpret individual +fields. Risk data is not logged or returned in HTTP responses, and built-in adapters do not +forward it to providers. A custom adapter must explicitly map each field it uses. + +Current runtimes do not read an incoming `textToVoice` property. The Soprano adapters build their +provider `text2voice` object from `message` and `locale`. Validation reasons include: @@ -199,7 +203,8 @@ The shared outcomes are `Continue`, `Block`, and `Fail`. | Provider `429` | `429` | | Provider `401` or `403` | `401` | | Other provider `4xx` | `400` | -| Missing configuration, network failure, or unknown response | `502` | +| Missing or unregistered provider selection | `400` | +| Missing or invalid endpoint, authentication-mode mismatch, credential or network failure, or unknown response | `502` | | Provider timeout | `504` | Unknown or incomplete provider responses fail closed. Check a timed-out live request before @@ -247,13 +252,13 @@ The provider timeout defaults to `1500` and is capped at `2500` milliseconds. Each live request uses the endpoint for its selected channel. Endpoints must be absolute HTTPS URLs with a hostname, valid port, no user information, and no fragment. Redirects are not followed. -Each channel uses only its configured endpoint. Provider failover requires the -[advanced fallback design](docs/MULTI-PROVIDER-IMPLEMENTATION.md). +Each channel uses only its configured endpoint. A secondary provider requires the +[advanced secondary-provider design](docs/MULTI-PROVIDER-IMPLEMENTATION.md). All runtimes acquire provider credentials on demand for each live request: -- **API keys:** read the required secret values from Key Vault each time. Failed reads never - fall back to a previous secret bundle. +- **API keys:** read the required secret values from Key Vault each time. A failed read does not + reuse a previous secret bundle. - **OAuth:** reuse Azure Identity SDK credential instances and call their token API for each request. The SDK owns token caching and renewal; a usable cached token does not require another token exchange. Managed-identity assertions are obtained only when the SDK needs them. @@ -365,7 +370,8 @@ Before live use: **Easy Auth is the caller-authentication gate before the Function runs.** Follow the [exact trust checklist](docs/ONBOARDING.md#manual-deployment-trust-checklist) for HTTPS, issuer, audience, caller allowlist, and unauthenticated rejection. Never disable it or exempt SendOtp. -There is no handler token validator or function-key fallback. +The handler does not validate bearer tokens itself and does not accept a function key as an +alternative. JWE encryption and the nonce do not identify the caller. Anyone with the public key can encrypt. Provider credentials authenticate outbound calls, not incoming requests. Vault access uses managed identity. diff --git a/docs/APPLICATION-INSIGHTS.md b/docs/APPLICATION-INSIGHTS.md index a0eda48..acdbc29 100644 --- a/docs/APPLICATION-INSIGHTS.md +++ b/docs/APPLICATION-INSIGHTS.md @@ -119,7 +119,7 @@ and [Microsoft Entra-authenticated ingestion](https://learn.microsoft.com/azure/ ## Multiple regions Each region has its own Function App and identity. Validate telemetry for every origin. -Application Insights does not provide application routing or failover. +Application Insights does not route traffic or implement regional recovery. | Layout | Use | | --- | --- | diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index 666c187..d11d05e 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -47,7 +47,7 @@ Authorization: Bearer | --- | --- | | `Authorization` | Required bearer token. Easy Auth consumes it. | | `Content-Type` | Required. Use `application/json`. | -| `x-ms-correlation-id` | Optional correlation fallback. | +| `x-ms-correlation-id` | Optional correlation value. | | `x-ms-client-request-id` | Optional request identifier. | The Function does not forward the bearer token to the phone provider. @@ -57,17 +57,30 @@ The Function does not forward the bearer token to the phone provider. -- **`type`** — Required string. Use `microsoft.mfa.otpDeliver.v1`. -- **`channel`** — Required integer or string. `1` or `"sms"` for SMS. `2` or `"voice"` for voice. -- **`mode`** — Required integer or string. `1` or `"live"` to send. `2` or `"evaluation"` - for no-send validation. -- **`encryptedDeliveryContext`** — Required compact JWE string. -- **`tenantId`** — Optional routing metadata. -- **`correlationId`** — Optional sign-in correlation value. -- **`ttlSeconds`** — Optional positive integer up to `2147483647`. +```json +{ + "type": "microsoft.mfa.otpDeliver.v1", + "channel": 1, + "mode": 1, + "encryptedDeliveryContext": "", + "tenantId": "", + "correlationId": "", + "ttlSeconds": 30 +} +``` -Send integer channel, mode, and TTL values in their documented forms for consistent behavior across -all runtimes. See [request validation and runtime differences](../TECHNICAL.md#request-validation-and-runtime-differences). +| Field | Required | Description | +| --- | --- | --- | +| `type` | Yes | Use `microsoft.mfa.otpDeliver.v1`. | +| `channel` | Yes | `1` or `"sms"` for SMS. `2` or `"voice"` for voice. | +| `mode` | Yes | `1` or `"live"` to send. `2` or `"evaluation"` for no-send validation. | +| `encryptedDeliveryContext` | Yes | Compact JWE containing the delivery data. | +| `tenantId` | No | Tenant metadata supplied by Microsoft Entra ID. The shared handler does not use it to select a provider. | +| `correlationId` | No | Sign-in correlation value. | +| `ttlSeconds` | No | Positive integer up to `2147483647`. The sample validates this value but does not enforce passcode expiry or a delivery deadline. | + +Use the documented channel and mode values. When supplying `ttlSeconds`, use an integer. See +[request validation and runtime differences](../TECHNICAL.md#request-validation-and-runtime-differences). ### Encrypted delivery context @@ -75,21 +88,37 @@ all runtimes. See [request validation and runtime differences](../TECHNICAL.md#r The JWE uses `RSA-OAEP-256` and `A256GCM`. After decryption, the JSON contains: -- **`nonce`** — Required. Returned unchanged after evaluation or provider acceptance. -- **`phoneNumber`** — Required phone number. -- **`message`** — Required localized message containing the passcode. -- **`otp`** — Optional string containing the separate six-digit passcode. Preserve leading zeros. -- **`locale`** — Optional voice locale. -- **`extension`** — Optional office-voice data. -- **`riskContext`** — Optional JSON object containing arbitrary risk fields; may be `null`. -- **`textToVoice`** — Optional legacy speech data. +```json +{ + "nonce": "", + "phoneNumber": "+15551234567", + "message": "Your verification code is 012345.", + "otp": "012345", + "locale": "en-US", + "riskContext": { + "scenario": "signIn", + "ipAddress": "203.0.113.10" + } +} +``` + +| Field | Required | Description | +| --- | --- | --- | +| `nonce` | Yes | Nonblank string returned unchanged after evaluation or provider acceptance. | +| `phoneNumber` | Yes | Nonblank recipient phone number. | +| `message` | Yes | Nonblank localized message containing the passcode. | +| `otp` | No | Passcode supplied separately as a string. Leading zeros are preserved. | +| `locale` | No | Voice locale. | +| `extension` | No | Legacy office-voice data. The shared handler does not forward it to built-in providers. | +| `riskContext` | No | JSON key-value object containing risk signals; may be `null`. | Microsoft Entra ID supplies `otp` for SMS and voice, with a generated test code in evaluation mode. It belongs inside the encrypted delivery context, not the outer request body. Requests that omit it or set it to `null` remain supported. -`riskContext` has no fixed field schema. Unknown keys, nested objects and arrays, and values such -as `false`, `0`, and `null` are preserved. Missing fields are not filled with defaults. +`riskContext` can include signals such as a scenario name or IP address. The shared handler +preserves the supplied object but does not interpret, log, return, or forward its fields to the +built-in providers. A custom adapter must explicitly map any fields it needs. See [JWE validation and key handling](../TECHNICAL.md#jwe-validation-and-key-handling) for limits, algorithms, key settings, and runtime behavior. @@ -130,7 +159,7 @@ Content-Type: application/json | Field | Description | | --- | --- | | `nonce` | Original decrypted nonce. | -| `correlationId` | Request correlation value or generated fallback. | +| `correlationId` | Request correlation value or generated request ID. | | `providerStatus` | `"accepted"` for evaluation or recognized live acceptance. | Failures omit `nonce` and `providerStatus`. @@ -142,15 +171,89 @@ Failures omit `nonce` and `providerStatus`. | Status | Description | | --- | --- | | `200 OK` | Evaluation completed or the provider accepted the request. | -| `400 Bad Request` | Invalid request or encrypted content. | +| `400 Bad Request` | Invalid request or encrypted content, a missing or unrecognized provider setting, or a provider `4xx` response that maps to `400`. Provider authentication failures and throttling map to the statuses listed separately below. | | `401 Unauthorized` | Caller or provider authentication failed. | | `403 Forbidden` | Caller access was denied or the provider blocked the request. | | `429 Too Many Requests` | Provider throttling. | -| `502 Bad Gateway` | Provider configuration, network, or response failure. | +| `500 Internal Server Error` | Unexpected Function or provider response-processing failure. | +| `502 Bad Gateway` | Missing or invalid provider endpoint, mismatched authentication mode, or failure to obtain credentials, reach the provider, or process its response. | | `504 Gateway Timeout` | Provider request timed out. | Check a timed-out live request before retrying. The provider might already have accepted it. +## Error response + +The Function returns JSON for application errors: + +```json +{ + "error": "provider_delivery_failed", + "requestId": "", + "correlationId": "" +} +``` + +| Field | Description | +| --- | --- | +| `error` | Stable public error category. | +| `requestId` | Request identifier for safe log correlation. | +| `reason` | Validation detail when it is safe to return. Usually present only for `400` responses. | +| `correlationId` | Correlation value when available. | + +Error categories are: + +- `bad_request` for invalid request data or an incomplete decrypted delivery context. +- `decryption_failed` when the encrypted delivery context cannot be decrypted or parsed. +- `provider_delivery_failed` for provider selection, configuration, credentials, transport, or + mapped provider-response failures. +- `delivery_failed` for an unexpected handler failure. + +These JSON errors come from the SendOtp handler. Easy Auth can reject a request before the handler +runs. The .NET Functions host can also reject a request while binding its input, before the handler +runs. Those responses can have a different format. When `reason` is present, it is a fixed +validation message rather than an exception message. + +Unavailable provider credentials, for example, return `502 Bad Gateway`. The safe diagnostic +detail is recorded as +`failureStage=provider_credentials` and `failureReason=credential_unavailable` in application +logs, not exposed in the response. + +### Diagnose a credential failure + +When a live request cannot obtain provider credentials, current packages emit the following events: + +| Signal | Meaning | +| --- | --- | +| `request_received` | The Function received the CYOT request. | +| `envelope_validated` or `payload_validated` | The outer request passed validation. | +| `delivery_context_decrypted` | The encrypted delivery context was decrypted. | +| `provider_selected` | The configured provider was selected. | +| `provider_credential_resolution_started` | The Function started resolving provider credentials. | +| `request_failed` with credential stage, unavailable reason, and HTTP `502` | Credential resolution failed before the provider request started. | +| `response_prepared`, then `request_completed` with HTTP `502` | The handler prepared a `provider_delivery_failed` response with HTTP `502`. These events do not confirm that the caller received it. | + +This pattern points to provider authentication or credential configuration, not envelope validation, +decryption, or provider delivery. Check the selected provider's Key Vault secrets or OAuth managed +identity, federation, tenant, scope, and permissions. + +Field names and export shapes vary by runtime. JavaScript uses fields such as `failureStage`, +`failureReason`, and `httpStatus`; .NET commonly exports `FailureStage`, `FailureReason`, and +`HttpStatus`; Python logging exporters can place the same values in extra properties. Confirm the +deployed shape through [runtime-specific log discovery](MONITORING.md#runtime-specific-log-discovery) +and the [application log reference](LOGGING.md). + +Current packages resolve credentials during live requests and do not emit +`credential_refresh_failed`. Older packages can emit different events. Confirm the deployed +package version and use its matching documentation before comparing sequences; upgrade only +through the normal reviewed deployment process. + +Use the `ResultCode` field on the workspace `AppRequests` row together with correlated +`request_failed` and `request_completed` application events. The request telemetry records the +observed HTTP transport result; application events record handler progress. Filters, sampling, and +ingestion can omit an event, so a missing trace does not prove that the handler skipped that stage. +See [provider credential acquisition fails](../setup/docs/Troubleshooting.md#provider-credential-acquisition-fails) +for corrective steps. + ## Related guidance diff --git a/docs/COST-CALCULATOR.md b/docs/COST-CALCULATOR.md index abaaf26..4125833 100644 --- a/docs/COST-CALCULATOR.md +++ b/docs/COST-CALCULATOR.md @@ -13,8 +13,8 @@ The guide separates the estimate into two parts: ## 1. Estimate telecom-provider costs > [!IMPORTANT] -> This is not a billing report. Confirm retries, fallback, message segments, destinations, taxes, -> fees, and pricing with the provider. +> This is not a billing report. Confirm retries, secondary-provider attempts, message segments, +> destinations, taxes, fees, and pricing with the provider. | Step | Task | | --- | --- | @@ -202,7 +202,7 @@ Estimated SMS/voice requests = 10,000 Apply the provider's destination-specific rates to those estimated requests. Use separate rows when rates differ by destination, sender type, route, or message category. Include a -contingency for growth, retries, fallback behavior, and seasonal peaks. +contingency for growth, retries, approved secondary-provider behavior, and seasonal peaks. ### Confirm provider rates @@ -265,7 +265,7 @@ Include these resources: | Function App | Hosts the application and its HTTPS endpoint. | There is no separate base charge; compute is charged through the selected Functions hosting plan. | | Application Insights and Log Analytics | Provides operational monitoring, request diagnostics, provider latency, failure data, and troubleshooting telemetry. | Telemetry volume and retention. | | Storage account | Stores Function host information and the deployment package. | Stored capacity, transactions, and LRS versus ZRS replication. Zone-redundant deployments require ZRS storage. | -| Key Vault | Protects provider credentials, secrets, and the encryption certificate. | Secret and certificate operations. This is normally a small cost because credentials are cached. | +| Key Vault | Protects provider credentials, secrets, and the encryption certificate. | Secret and certificate operations. Current packages read API-key secrets on each live request; OAuth token caching is owned by the Azure Identity SDK. | | Outbound bandwidth | Carries API requests and responses between Azure Functions and the provider. | Outbound data after the applicable Azure allowance. This is usually small for provider API payloads. | | Traffic Manager or equivalent global routing | Routes traffic in an optional multi-region disaster-recovery deployment. | DNS queries or requests, depending on the selected routing service. | @@ -274,7 +274,7 @@ Collect these inputs before creating your estimate: - Monthly SMS and voice authentication requests, including authentication, enrollment, recovery, and test traffic. - Average Function execution duration, including provider API latency. -- Expected retry and failover allowance. +- Expected retry and regional-recovery allowance. - Function memory allocation and operational logging generated per request. - Required region, availability-zone resilience, and regional disaster recovery. - Your Azure pricing, discounts, free grants, taxes, and currency. @@ -282,7 +282,7 @@ Collect these inputs before creating your estimate: Use adjusted request volume consistently across the estimate: ```text -Adjusted monthly requests = monthly authentication requests x retry/failover multiplier +Adjusted monthly requests = monthly authentication requests x retry/recovery multiplier Execution GB-seconds = adjusted monthly requests x execution seconds x memory in GB Monitoring data in GB = adjusted monthly requests x logging KB per request / 1,048,576 Estimated Azure total = Functions + monitoring + storage + Key Vault @@ -312,5 +312,5 @@ the Function cold-start portion. Zone redundancy protects against a datacenter-z failure within one region; it does not provide regional disaster recovery. For a multi-region design, also account for globally unique idempotency, provider-aware -failover, shared fraud and spend limits, and an emergency stop. Autoscaling Azure compute +regional recovery, shared fraud and spend limits, and an emergency stop. Autoscaling Azure compute must not create unlimited telecom spend. diff --git a/docs/FRONTDOOR.md b/docs/FRONTDOOR.md index 6863b34..d62c332 100644 --- a/docs/FRONTDOOR.md +++ b/docs/FRONTDOOR.md @@ -71,7 +71,7 @@ Build and test the regional design outside live traffic. Review the Follow this regional procedure rather than rerunning `Setup-Epp.ps1` against the same application. The setup script creates one region. The steps below coordinate Front Door, shared -key material, regional registrations, and failover without replacing an existing endpoint's +key material, regional registrations, and recovery routing without replacing an existing endpoint's application configuration or certificate unexpectedly. @@ -88,7 +88,7 @@ Record the workload mix, offered and completed request rates, concurrency, paylo duration, per-region latency percentiles, errors/timeouts, resource saturation, and dependency throttling. Verify hosting/subscription quotas, available warm capacity and scale-out delay, storage and vault limits, identity/token-service capacity, and provider account/rate limits. -Use measured load results in both failover directions. Autoscale limits, instance counts, and +Use measured load results in both recovery directions. Autoscale limits, instance counts, and approved quotas are inputs, while sustained request rate, latency, errors, and saturation show whether the design meets the agreed capacity target. Validate the target deployment with its own measured load result. @@ -174,7 +174,7 @@ Readiness confirms that the handler and decryption key are usable. Validate prov and handset delivery separately. Point probes to the implemented readiness handler, return a live result rather than a cached static `200`, and keep Easy Auth enabled for SendOtp. Front Door treats only a `200` probe response as healthy. -Review [failover behavior](#failover-behavior) before setting recovery expectations. +Review [regional recovery behavior](#regional-recovery-behavior) before setting expectations. ## 4. Configure Front Door manually @@ -253,25 +253,25 @@ Approve synthetic data, bounded load, and the nonproduction. Agree load, headroom, recovery, error, latency, soak, and restoration targets first. Use an authorized harness, **not a real Microsoft Entra ID sign-in flow**. Evaluation suppresses -sends only inside this endpoint. Caller fallback or retries can still deliver. Confirm caller behavior -with Microsoft and obtain separate approval for sign-in or delivery tests. +sends only inside this endpoint. Caller retries or other delivery paths can still deliver. Confirm +caller behavior with Microsoft and obtain separate approval for sign-in or delivery tests. | What to test | How and where to test safely | Expected outcome / acceptance | Evidence to keep | |---|---|---|---| | Baseline encrypted evaluation | Send synthetic encrypted `mode: 2` requests through Front Door with both origins healthy. Confirm each origin serves requests. | Matching nonce checked locally, no delivery, and agreed baseline error/latency limits met. | Offered/completed rates, latency/errors, safe correlation IDs, and serving region. Exclude nonce values and request bodies. | | Bad caller credentials, unauthorized callers, and direct-origin access | From the test harness, try missing/invalid/wrong-audience tokens, an unauthorized app, and direct-origin requests including a spoofed `X-Azure-FDID`. Send valid evaluations before and after. | Invalid calls are rejected while valid evaluations still work. Authentication and ingress restrictions remain enabled. | Response status and sanitized access diagnostics identifying the rejection, plus successful surrounding evaluations. | -| Failover from A to B | Follow the recovery test: isolate only approved test origin A while leaving it **enabled in Front Door**. Keep B healthy and evaluation traffic bounded. | Health-based routing to B meets agreed recovery and transition-error limits, without changing the public URL, trust, or encryption key. | Fault/probe/client timestamps, every failure/timeout, and B's request/health signals. | -| Failover from B to A | After A is fully restored and stable, repeat with B isolated but **enabled in Front Door**. | A meets the same agreed criteria. Both directions must pass. | The same evidence, identifying A as the survivor and confirming B's subsequent restoration. | +| Recovery from A to B | Follow the recovery test: isolate only approved test origin A while leaving it **enabled in Front Door**. Keep B healthy and evaluation traffic bounded. | Health-based routing to B meets agreed recovery and transition-error limits, without changing the public URL, trust, or encryption key. | Fault/probe/client timestamps, every failure/timeout, and B's request/health signals. | +| Recovery from B to A | After A is fully restored and stable, repeat with B isolated but **enabled in Front Door**. | A meets the same agreed criteria. Both directions must pass. | The same evidence, identifying A as the survivor and confirming B's subsequent restoration. | | Sustained survivor capacity | In each direction, maintain the [combined peak offered load plus planned headroom](#capacity-gate-each-region-must-support-the-combined-peak-load) for the agreed soak. | **Each region handles at least double its normal share in the balanced two-region case, plus headroom**, within agreed error/latency and dependency limits. Evaluation proves only the exercised path. | Offered/completed rates, concurrency, latency percentiles, errors/timeouts, saturation/throttling, quotas and available capacity per region. | | Restart, cold start, and credential acquisition without the failed region | In a separately approved dependency-isolation test environment, make the failed region's storage/vault paths unavailable and start or restart the survivor stack. Exercise key resolution and request-time credential acquisition using the non-delivering provider setup below. Do not add a second fault to the basic recovery test. | The survivor starts and serves using its own dependencies. SDK token caches do not hide cross-region access. Missing failed-region telemetry is not treated as success. | Dependency configuration/access evidence, startup/key/credential-acquisition results, client outcomes, and timestamps captured outside the failed region. | | Complete live-path capacity and provider credential failures | Separately use a provider-approved, production-representative **non-delivering sandbox/stub** exercising credential acquisition, SDK token renewal, and provider latency/quota behavior. Test valid and invalid/expired test credentials without production accounts or recipients. | Both survivor regions meet the capacity criteria on this path. Credential failures are surfaced, not reported as successful sends. Evaluation-only throughput cannot pass this test. | Credential/provider timing, failures/throttling, load results, and provider capacity/quota evidence for behavior the sandbox/stub cannot reproduce. | -| Restoration and conservative failback | Restore the original approved state even after a failed test. Verify the restored origin's own readiness and successful evaluations, then observe traffic returning for the agreed window. | Both origins are healthy and serving within agreed limits, with original security settings intact, before any next fault. | Restoration confirmation, origin-specific readiness/request evidence and failback latency/errors. | +| Restoration and controlled traffic return | Restore the original approved state even after a failed test. Verify the restored origin's own readiness and successful evaluations, then observe traffic returning for the agreed window. | Both origins are healthy and serving within agreed limits, with original security settings intact, before any next fault. | Restoration confirmation, origin-specific readiness/request evidence, and traffic-return latency/errors. | For every row, record **pass, fail, or not tested**, the approved target, observed outcome, and evidence location. Assign an owner and retest date to gaps. A successful routing test is useful, but is not a substitute for the separate capacity and dependency tests. - + ## 6. Validate recovery Run this test in an approved nonproduction environment built for recovery validation. It checks @@ -287,8 +287,8 @@ thresholds before starting. verify a restoration timer or backup operator. 3. **Set the baseline and gates.** Save the approved routing, authentication, ingress, package, key, and dependency state. Confirm both origins are healthy and serving. Agree survivor load, - soak time, recovery target, allowed failures/timeouts, latency/resource limits, failback window, - and abort thresholds before injecting a fault. + soak time, recovery target, allowed failures/timeouts, latency/resource limits, + traffic-return window, and abort thresholds before injecting a fault. 4. **Isolate one origin without disabling it in Front Door.** Apply an approved reversible fault, such as stopping the test Function App, and timestamp request and confirmation separately. Keep the survivor unchanged. Do not weaken Easy Auth or ingress. Probe health is not a security boundary. @@ -299,9 +299,9 @@ thresholds before starting. 6. **Restore on completion or abort.** Stop unsafe load and restore the saved state at the deadline, any abort threshold, or completion. Cleanup must run even if assertions or the test runner fail. Escalate immediately if restoration cannot be confirmed. -7. **Verify failback, then reverse direction.** Check the restored origin's dependencies, key and - credential resolution, readiness, and evaluation responses. Wait through the agreed stable - observation window before testing the other origin. Both directions must pass. +7. **Verify restored traffic, then reverse direction.** Check the restored origin's dependencies, + key and credential resolution, readiness, and evaluation responses. Wait through the agreed + stable observation window before testing the other origin. Both directions must pass. 8. **Record results and gaps.** Keep sanitized approvals, configuration/package identifiers, workload limits, timestamps, rates, latency/errors, dependency evidence, aborts, and restoration. Mark each criterion pass/fail/not tested with an owner and retest date. Never attach secrets, @@ -319,8 +319,8 @@ that cannot be tested safely and assign an owner and follow-up date. Use separate approved checks for real Microsoft Entra ID caller timing, provider acceptance, and handset delivery. Define retry and availability targets with the provider and onboarding owner. - -## Failover behavior + +## Regional recovery behavior Front Door selects from eligible healthy origins using priority, latency, and weight. Health status uses a sliding sample of probe results, and only HTTP `200` counts as healthy. Detection and routing diff --git a/docs/LOGGING.md b/docs/LOGGING.md index 773d43f..4eae66d 100644 --- a/docs/LOGGING.md +++ b/docs/LOGGING.md @@ -94,7 +94,7 @@ Microsoft IDs are absent or change. Log names deliberately distinguish these sou | `functionName` | Runtime function name: `SendOtp` in JavaScript/.NET, `send_otp` in Python. The HTTP route remains `/api/SendOtp` in every runtime. | | `functionRequestId` | Generated by this handler. Matches `requestId` in failure responses. It is not Microsoft's request ID. | | `functionInvocationId` | Azure Functions host invocation ID. Null for direct handler calls without a host context. | -| `x-ms-client-request-id` | Raw Microsoft per-attempt ID from the header of the same name, not the generated fallback used by dispatch. | +| `x-ms-client-request-id` | Raw Microsoft per-attempt ID from the header of the same name, not the generated Function request ID used by dispatch. | | `x-ms-correlation-id` | Raw selected Microsoft correlation: envelope `correlationId` first, then the header of the same name. The existing precedence is unchanged, and this never contains a generated Function ID. | | `msCorrelationIdSource` | `envelope`, `header`, or `none`. Identifies the selected value when the envelope and header differ. | | `providerMessageId` | Raw provider-normalized message/reference ID returned by the provider, including Telesign `reference_id`, for support lookup. Not filled from delivery or Function IDs, although a provider may echo an ID it received. | diff --git a/docs/MULTI-PROVIDER-IMPLEMENTATION.md b/docs/MULTI-PROVIDER-IMPLEMENTATION.md index d8d4e88..fca9a89 100644 --- a/docs/MULTI-PROVIDER-IMPLEMENTATION.md +++ b/docs/MULTI-PROVIDER-IMPLEMENTATION.md @@ -1,10 +1,12 @@ -# Provider fallback +# Secondary-provider design - + [Documentation](README.md) | [Normal deployment](GETTING-STARTED.md) | [Provider interface](../TECHNICAL.md#provider-result-mapping) -This advanced design adds controlled primary-to-secondary fallback around `/api/SendOtp`. +The sample does not implement automatic secondary-provider attempts. This advanced design shows +how a customer-owned implementation can add one controlled primary-to-secondary attempt around +`/api/SendOtp`. Use the secondary **only after proof that the primary did not accept the request**. Timeouts, lost responses, and generic errors are uncertain and must not trigger an automatic resend. @@ -15,7 +17,7 @@ lost responses, and generic errors are uncertain and must not trigger an automat | 2 | [Configure both accounts](#2-configure-and-prevalidate-both-accounts) | | 3 | [Isolate credentials](#3-credential-isolation) | | 4 | [Add durable operation state](#4-operation-state) | -| 5 | [Implement guarded fallback](#5-fallback-flow) | +| 5 | [Implement the guarded secondary attempt](#5-secondary-provider-flow) | | 6 | [Apply one deadline](#6-use-one-end-to-end-deadline) | | 7 | [Test and deploy](#7-validation-and-deployment) | @@ -33,20 +35,20 @@ const providerOrder = Object.freeze({ ``` Reject unsupported channels and request-supplied provider configuration. Do not use round-robin, -fan-out, recursive fallback, or primary retries. Attempt the secondary at most once. +fan-out, recursive provider switching, or primary retries. Attempt the secondary at most once. -Keep **routing order** separate from **fallback eligibility**. Use this policy: +Keep **routing order** separate from **secondary-attempt eligibility**. Use this policy: | Primary result | Action | | --- | --- | | Accepted, successful, or `PENDING` | Stop. Do not contact the secondary. Acceptance is not handset delivery. | | Block/fraud denial, bad recipient or payload, invalid inbound authentication | Stop. Never use another provider to bypass the denial or validation. | | Unknown status, malformed/lost response, timeout, connection/read error, or cancellation after dispatch | Record a terminal or uncertain outcome. Do not automatically contact the secondary. | -| Generic 4xx/5xx, `Fail`, or `provider_rejected` | No fallback based on that classification alone. | +| Generic 4xx/5xx, `Fail`, or `provider_rejected` | Do not try the secondary based on that classification alone. | | Primary-only credential acquisition fails before transport is entered | Eligible only if this pre-dispatch fact is recorded, the account policy explicitly permits it, the durable operation guard allows the transition, and the request budget is sufficient. | | Provider-specific, documented proof that the primary did not accept the operation | Eligible only through a reviewed nonacceptance classifier, explicit account policy, the durable guard and sufficient budget. | -Enable fallback only after the provider confirms a safe nonacceptance signal. Caller +Enable a secondary attempt only after the provider confirms a safe nonacceptance signal. Caller authentication and provider controls still apply. ## 2. Configure and prevalidate both accounts @@ -83,7 +85,8 @@ The following customer-owned format is not loaded by the sample. Replace each ``` This SMS policy requires `EPP_PROVIDER_SMS_ENDPOINT` for each account. Voice uses -`EPP_PROVIDER_VOICE_ENDPOINT`. There is no shared endpoint fallback. +`EPP_PROVIDER_VOICE_ENDPOINT`. Each channel requires its own setting; the handler does not use +the other channel's setting when one is missing. At startup, validate both account configurations before enabling the endpoint. Check for unique IDs, the required channel endpoint, matching authentication mode, approved URLs, credential @@ -153,7 +156,7 @@ An evaluation request must not start credential acquisition. Rebuild contexts on controlled restart when configuration changes. Never repurpose a live service for another account. - + ## 4. Operation state Add a **durable attempt store and idempotency contract** before allowing a second submission. @@ -185,12 +188,11 @@ Record operation state, safe provider context ID and per-attempt classifications request logs. Do not store/log bodies, OTPs or credentials as attempt evidence. Retention and key reuse rules must cover the caller's replay window. A store guard cannot atomically commit an external provider send: it is **not provider-side idempotency or true exactly-once delivery**. -Microsoft Entra ID retries or native fallback outside this endpoint can still produce duplicates. -Confirm that behavior with the owner rather than promising the local guard prevents it. +Retries or separate delivery paths outside this endpoint can still produce duplicates. Confirm +caller behavior with the owner rather than promising the local guard prevents it. - - -## 5. Fallback flow + +## 5. Secondary-provider flow In [SendOtp.js](../javascript/src/functions/SendOtp.js), preserve authentication at the platform boundary, [envelope validation](../javascript/src/functions/entraPayload.js), @@ -207,8 +209,8 @@ returns a structured category. Use `providerConfig` for outbound settings and th The existing credential catch immediately returns 502, and transport/result failures immediately return errors. The attempt helper reports the stage to the main flow instead of hiding -it in a catch that calls the secondary. Preserve their final HTTP classifications when fallback -is denied. This is **policy pseudocode**, not executable glue or a complete handler: +it in a catch that calls the secondary. Preserve their final HTTP classifications when the +secondary attempt is not allowed. This is **policy pseudocode**, not executable glue or a complete handler: ```javascript // Runs after existing validation/decryption/evaluation handling. @@ -241,7 +243,7 @@ result fails after intent, leave that attempt non-reclaimable. Do not send throu **Do not use `result.httpStatus >= 400` or `catch -> sendSecondary`.** [providerResult.js](../javascript/src/functions/providerResult.js) exposes `Continue`, `Fail` -and `Block`, not a safe-to-fallback receipt. Preserve Block as terminal and recognized +and `Block`, not proof that a secondary attempt is safe. Preserve Block as terminal and recognized accepted/`PENDING` as accepted. A customer classifier must require provider-specific, documented nonacceptance for this exact attempt and account policy approval. It must not reinterpret generic `Fail` or a broad status range as proof. @@ -270,10 +272,10 @@ The current credential service has its own acquisition bound and the transport i `parseProviderTimeout` normalizes a single provider timeout, **not** an aggregate deadline. Add explicit remaining-budget checks and cancellation propagation in your customer integration. Do not use `Promise.race` to return while a send continues in the background. If ownership, -deadline or cancellation changes after dispatch, record uncertainty and never fall back. +deadline or cancellation changes after dispatch, record uncertainty and do not try the secondary. Reserve enough time for the complete secondary attempt and final response, and check the remaining time again before sending. Skip the secondary when the remaining time cannot cover that work, -even when the primary result would otherwise allow fallback. +even when the primary result would otherwise allow it. ## 7. Validation and deployment @@ -305,20 +307,21 @@ store faults. Block external network. Verify: decryption failures remain errors. Credentials, endpoints, and wire responses stay isolated between accounts. Fake classifiers and stores cannot prove provider nonacceptance or distributed durability. -Routing and isolation tests alone are **not fallback tests**. No production classifier is supplied. -Keep response-based fallback disabled until reviewed. +Routing and isolation tests alone do **not validate secondary-provider behavior**. No production +classifier is supplied. Keep response-based secondary attempts disabled until reviewed. Deploy one app/package at `/api/SendOtp` for code-owned policy, orchestrator, classifier, or store changes. Validate startup configuration on controlled restart. Verify request-time acquisition after rotation and update pinned references explicitly. Unchanged approved tests or Markdown edits need no redeployment. Preserve inbound authentication and origin/network restrictions, including any Front Door -forwarding. Regional failover is separate from provider fallback. Before live use, review -account permissions/consent, sender/recipient entitlement, durable-store guarantees and caller -retry/native-fallback behavior. Rotate any exposed credentials before testing. Any Microsoft Entra ID sign-in or -provider test needs separate authorization, an approved recipient, explicit attempt limits and -stop conditions for uncertainty. HTTP 200/`PENDING` is not handset receipt. Nothing here authorizes -a send or proves Microsoft Entra ID integration, capacity, regional resilience or live delivery. +forwarding. Regional recovery is separate from secondary-provider attempts. Before live use, +review account permissions/consent, sender/recipient entitlement, durable-store guarantees, and +caller retry behavior. Rotate any exposed credentials before testing. Any Microsoft Entra ID +sign-in or provider test needs separate authorization, an approved recipient, explicit attempt +limits, and stop conditions for uncertainty. HTTP 200/`PENDING` is not handset receipt. Nothing +here authorizes a send or proves Microsoft Entra ID integration, capacity, regional resilience, +or live delivery. ## .NET and Python integration pointers @@ -337,6 +340,6 @@ the JavaScript behavior transfers unchanged. Pass explicit account contexts instead of rereading process settings. Use a separate [CredentialTokenService](../python/src/credentials.py) and [SecretResolver(env)](../python/src/secrets.py) per account. Close each credential service at shutdown. -Both runtimes still need customer-owned durable operation state, deadline enforcement and -reviewed fallback classification. These pointers are not evidence that fallback is implemented -or validated in either runtime. +Both runtimes still need customer-owned durable operation state, deadline enforcement, and a +reviewed secondary-attempt classification. These pointers are not evidence that provider +switching is implemented or validated in either runtime. diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 4d9d369..4d7989a 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -199,6 +199,13 @@ Evaluation validates the authorized encrypted request path without sending a mes validate provider credentials, provider acceptance, or recipient delivery. For live requests, `200` means provider acceptance. Confirm delivery separately and do not blindly retry timeouts. +Evaluation also does not test a production migration percentage. That percentage is controlled by +Microsoft Entra ID through the approved rollout procedure; the Function does not read or apply it. +Before increasing production traffic, complete the authorized evaluation and controlled live tests +for each intended channel. At every approved percentage, verify Entra routing evidence, Function +telemetry, provider acceptance, recipient delivery, and rollback thresholds for the agreed +observation window. See [evaluation and percentage rollout](../setup/docs/README.md#evaluation-and-percentage-rollout). + Repeat deployed checks after package, authentication, credential, key, or routing changes. @@ -254,8 +261,8 @@ the deployed validation checks. | `identityProviders.azureActiveDirectory.validation.defaultAuthorizationPolicy.allowedApplications` | Nonempty approved Microsoft caller app-ID allowlist | | `globalValidation.excludedPaths` | No `SendOtp` or alternate-ingress exemption | -The handler has no fallback token validator. Never use `common` or `organizations` as the issuer, -derive trust from request fields, or exempt `SendOtp`. +The handler does not validate bearer tokens itself. Never use `common` or `organizations` as the +issuer, derive trust from request fields, or exempt `SendOtp`. Keep the app registration's `tokenEncryptionKeyId` null. Microsoft Entra ID bearer-token encryption is separate from the JWE delivery payload and is not supported by this sample. diff --git a/docs/README.md b/docs/README.md index 42300f3..d1e1531 100644 --- a/docs/README.md +++ b/docs/README.md @@ -42,7 +42,7 @@ Use the guide that matches your task. ## Advanced topics - [Multi-region routing](FRONTDOOR.md) -- [Provider fallback](MULTI-PROVIDER-IMPLEMENTATION.md) +- [Secondary-provider design](MULTI-PROVIDER-IMPLEMENTATION.md) ## Support diff --git a/setup/docs/README.md b/setup/docs/README.md index 0520aa8..6743157 100644 --- a/setup/docs/README.md +++ b/setup/docs/README.md @@ -31,7 +31,7 @@ Both channels are configured. Profile objects `deployment.routes.global` and values. Setup writes the endpoints to `EPP_PROVIDER_SMS_ENDPOINT` and `EPP_PROVIDER_VOICE_ENDPOINT`; Soprano also supplies its API app ID and scope. -Use a package that supports channel-specific settings; neither channel falls back to the other URL. +Use a package that supports channel-specific settings; neither channel uses the other channel's URL. The current profiles use identical Global/EU URLs, and Soprano uses the same app ID/scope. The route label is not a residency or Azure-region selection. Infobip and Sinch are bundled @@ -447,7 +447,7 @@ configuration metadata, not provider credentials or private-key bytes. Keep it a | `tenantId`, `subscriptionId`, `applicationId` | Verify the intended customer directory, Azure subscription, and endpoint **client ID**. | | `resources.functionApp`, `resources.keyVault`, `resources.applicationInsights`, `resources.logAnalytics` | Find the exact Azure resources; do not guess names or use a different environment's vault. | | `provider`, `endpointRegion`, `providerTenantId` | Confirm provider routing, separate from the customer's directory and Azure location. Both SMS and voice use this configuration. | -| `allowedProviderTenantIds` | Review the complete preserved provider-tenant allowlist, including the selected provider. This does not enable runtime fallback between providers. | +| `allowedProviderTenantIds` | Review the complete preserved provider-tenant allowlist, including the selected provider. This does not enable runtime switching between providers. | | `providerSmsEndpoint`, `providerVoiceEndpoint` | Confirm the distinct or identical provider URLs for SMS and voice. These are outbound provider URLs, not the Function's `endpointUrl`. | | `endpointUrl`, `identifierUri` | Exact SendOtp URL and registered app audience information for the authorized test/policy operator. | | `encryptionKeyId`, `certificateThumbprint`, `certificateId`, `certificateSecretId`, `certificateExpiresUtc` | Public-key identification and version/renewal inventory. A secret **ID** is not its value; do not fetch its private-key contents. | @@ -602,6 +602,46 @@ Once that procedure is available: 5. If validation fails, execute the agreed rollback below. Do not leave a broken channel activated while deleting its resources or changing authentication to debug it. + +### Evaluation and percentage rollout + +Evaluation and migration percentage answer different questions: + +- **Evaluation** checks that Microsoft Entra ID can call the endpoint and that the Function can + authenticate, validate, and decrypt the request. The request uses evaluation mode and does not + read provider credentials or send an SMS or voice call. +- **Migration percentage** is a Microsoft Entra rollout control for eligible production + authentication traffic. It is not a request field, Function setting, provider setting, or + percentage implemented by this sample. + +Microsoft's public [telephony provider FAQ](https://learn.microsoft.com/entra/identity/authentication/phone-providers-faq#migration-and-evaluation) +describes evaluation as validation before production traffic is gradually enabled. +Use the exact percentage control, eligible population, and channel scope defined by Microsoft's +current approved procedure. Public Microsoft Graph SMS and voice resource definitions do not +document the EPP field placement or traffic-selection algorithm. Do not invent a Graph property or +assume a percentage guarantees an exact request count or a stable set of users. + +Plan the rollout in these stages: + +| Stage | Test | Evidence before continuing | +| --- | --- | --- | +| Evaluation | For each intended channel, have the authorized Microsoft caller send a valid encrypted evaluation request. | `200`, matching nonce, correlated evaluation events, and no provider credential or send events. | +| Controlled live test | Before increasing production traffic, send separately approved live requests to controlled recipients. | Provider acceptance, recipient delivery, correlated telemetry, and no unresolved timeout or duplicate-send concern. | +| Initial percentage | Apply the smallest percentage and population approved for the rollout. Hold the setting long enough to observe representative traffic. | Expected Entra routing, Function success/error/latency, provider acceptance, recipient delivery, support signals, and no rollback trigger. | +| Percentage increase | Change one rollout control at a time. Repeat the same checks at every approved increment. | Results remain within the agreed thresholds for the full observation window. | +| Full rollout | Move to the approved final percentage only after every intended channel and population passes. | Policy readback, sustained telemetry, provider capacity, operational ownership, and tested rollback readiness. | + +Evaluation success is necessary but does not validate provider credentials, provider capacity, +production percentage selection, provider acceptance, or recipient delivery. A configured +percentage is also not proof that the expected requests reached this endpoint. Correlate Microsoft +Entra rollout evidence, Function telemetry, provider records, and controlled recipient results. + +Before each increase, record the current policy readback, percentage, eligible population, +channels, UTC start time, observation window, owners, success thresholds, and rollback triggers. +If a threshold is breached, stop increasing traffic and restore the reviewed prior policy through +the approved procedure. Changing the percentage does not roll back the Function package, +credentials, or Azure resources. + ### Rollback and decommissioning Policy rollback, package/configuration rollback, and resource deletion are different operations. diff --git a/setup/docs/Troubleshooting.md b/setup/docs/Troubleshooting.md index 9c7e4be..571e356 100644 --- a/setup/docs/Troubleshooting.md +++ b/setup/docs/Troubleshooting.md @@ -83,10 +83,10 @@ Follow the [Front Door guide](../../docs/FRONTDOOR.md), including readiness and - A 401/403/404 probe is unhealthy. Check the readiness path, origin host header, and profile-pinned network restriction. Never exempt SendOtp or disable Easy Auth. -- An authenticated SendOtp 403 is a failure, not proof of failover. Check origin, ingress, and auth. +- An authenticated SendOtp 403 is a failure, not proof of regional recovery. Check origin, ingress, and auth. - A shared-URL success does not prove every origin works; verify each with safe regional telemetry. - Distinguish delayed aggregate metrics from event timestamps and client observations. Review - [failover behavior](../../docs/FRONTDOOR.md#failover-behavior) before setting recovery expectations. + [regional recovery behavior](../../docs/FRONTDOOR.md#regional-recovery-behavior) before setting recovery expectations. ## Deployment fails with SubscriptionIsOverQuotaForSku @@ -146,12 +146,14 @@ and HTTP `502`. The request has not reached the provider. provider secret names. For OAuth, verify the attached outbound managed identity, application federation, provider tenant/scope, and provider permissions using the [authentication handoff](../../docs/ONBOARDING.md#complete-provider-authentication). -Removing a refresher does not correct missing permissions or invalid identity configuration. +Match diagnostics to the deployed package version. Regardless of event names, verify permissions +and identity configuration. Current packages acquire credentials only during live requests, retain SDK token caching, and -have no custom failure cooldown. The next live request can retry acquisition. Deploy the updated -package to remove older refresher behavior; obsolete cache switches have no effect on current -packages. Restart after configuration changes. See +have no custom failure cooldown. The next live request can retry acquisition. Confirm the deployed +package version before comparing event sequences. If moving to current behavior, upgrade through +the normal reviewed deployment process; obsolete cache switches have no effect on current packages. +Restart after configuration changes. See [credential acquisition](../../TECHNICAL.md#provider-settings-and-credential-acquisition). Evaluation does not prove provider credentials work. @@ -340,7 +342,7 @@ Setup closes public ingress after persistent publication failure. Keep Easy Auth enabled and app `tokenEncryptionKeyId` null; bearer validation and payload JWE are separate. 2. **After invocation:** correlate [runtime events](../../docs/MONITORING.md#runtime-specific-log-discovery). Confirm `EPP_PROVIDER_SMS_ENDPOINT` or `EPP_PROVIDER_VOICE_ENDPOINT` and account access. - The selected URL is required and has no shared/other-channel fallback; deploy a supporting package. + The selected URL is required and the handler does not use the other channel's URL; deploy a supporting package. 3. Complete [provider authentication](../../docs/ONBOARDING.md#complete-provider-authentication): exact vault secrets for Telesign, provider-side authorization for Soprano. Arrange [authorized evaluation, then controlled live checks](../../docs/ONBOARDING.md#validate-the-deployed-endpoint).