Auth0 WIP #2: fail-fast config gate, JWKS rotation/outage handling, version-controlled Post-Login Action

[jcschaff]

Summary

Auth0 WIP #2 — hardens the authentication path landed in #96 so that an
identity-provider hiccup, a rotated signing key, or a half-configured cluster
each produce a correct and legible outcome instead of a silent one.

Stacked on #97 (chore/keycloak-tests) — this PR's base is that branch, so
the diff here is the four commits on top of it. Merge #97 first; GitHub will
retarget this to main automatically.

What changed

1. Startup gate instead of a startup warning (e799d9a)

_warn_if_auth0_misconfigured() logged a warning and let the pod start, so a
cluster with a missing AUTH0_DOMAIN/AUTH0_AUDIENCE reported healthy and then
failed every authenticated request. The warning also misdescribed the failure
(it promised a 401, which was never what happened).

• Auth0Settings.configuration_errors() (config.py) — pure, side-effect-free
  enumeration of every reason the settings could not verify a token. Accepts
  both valid shapes: a bare AUTH0_DOMAIN, or explicit AUTH0_ISSUER and
  AUTH0_JWKS_URI overrides (how a non-Auth0 OIDC provider, e.g. the Keycloak
  test realm, is configured). Half of the override pair is reported as an error.
• _validate_auth0_configuration() (api/main.py) raises out of lifespan, so
  uvicorn exits non-zero and Kubernetes shows CrashLoopBackOff with the reason
  in kubectl logs.
• New AUTH_REQUIRED setting (default true) is the escape hatch: set it false
  to run a deployment deliberately without an identity provider — the API then
  starts, logs what is missing, and every authenticated endpoint returns 503.
• kustomize/config/biosim-{local,rke}/api.env get AUTH0_DOMAIN +
  AUTH0_AUDIENCE (biosim-gke already had them from Auth0-Integration #96), so those overlays do
  not start crash-looping the moment the gate lands.

2. JWKS handling: rotation, outages, and malformed key sets (e3db45b, de31e3e)

_get_jwks() previously refetched on every miss, raised on any failure, and
indexed k["kty"]/["kid"]/["use"]/["n"]/["e"] directly — an entry missing the
RFC 7517-optional use field raised KeyError → HTTP 500.

• Unknown kid forces one refresh (cooldown-guarded, 60s) before rejecting
  the token. Auth0 rotates signing keys without notice; this turns a rotation
  from an hour-long outage into one slow request. The cooldown is load-bearing:
  without it a flood of bogus kids is an amplification vector against the IdP.
• Stale-while-revalidate: a cached document past its 1h TTL is still served
  for up to 24h while refreshes fail, then refused. Well inside Auth0's rotation
  overlap, so a key cached in that window is still a key the tenant published.
• Negative cache (10s) after a failed fetch: one outbound request per window
  per process, not one per inbound request.
• Single-flight refresh via asyncio.Lock with the double-checked pattern
  already used in auth0_management.py. The lock is held across the fetch only,
  never across jwt.decode, so validation stays parallel.
• _select_rsa_key() guards every field access and defaults a missing use to
  "sig".

3. Error responses and diagnostics (95c3383)

• When no usable key set exists, the response is 503 + Retry-After
  ("Authentication temporarily unavailable"), not a 401 — the caller's token was
  never the problem. Detail text names no URL, no exception, no token material.
• get_optional_user no longer swallows the 503. Downgrading an
  authenticated caller to anonymous during an Auth0 outage silently changes the
  authorization outcome (ownership checks, role gates). 401s stay swallowed —
  a bad token on an optional-auth endpoint is still just "not authenticated".
• The rejected-kid log line deliberately does not echo the kid; it comes
  from an unverified, attacker-controlled header.
• _warn_roles_claim_absent() — rate-limited (5 min) runtime assertion that the
  Post-Login Action is live. Without it, an absent Action means every
  require_roles endpoint 403s and no admin exists, presenting as a permissions
  bug with no signal anywhere.

4. The Auth0 Action is now version-controlled (e799d9a)

auth0/actions/post-login.js + auth0/README.md. #96 depended on a Post-Login
Action that existed only as dashboard state; this is the reviewed source of
truth the dashboard is expected to match. The README documents the required
Roles, the M2M application and its exact scopes (read:roles,
create:role_members — kept separate from the update:users/delete:users
application /api/v1/me will need), the Action secrets, the auth0 dependency,
the flow binding, and a post-deploy smoke check. Nothing in auth0/ is
deployed by CI or kubectl — applying it is a dashboard action.

Backend CLAUDE.md and .env.example gain matching Authentication sections.

Tests

All new, all against real tokens or real HTTP behavior — no mocked JWT
verification:

File	Covers
tests/api/test_startup_auth_config.py	the startup gate, both AUTH_REQUIRED modes, each malformed-config shape
tests/common/test_auth0_jwks.py	TTL, stale-while-revalidate bound, negative cache, single-flight, malformed key sets
tests/common/test_auth0_reliability.py	rotation recovery, forced-refresh cooldown, outage behavior
tests/common/test_auth0_roles_claim.py	roles-claim absent / empty / wrong-type
tests/api/test_auth_error_responses.py	401 vs 503 status/headers/detail, get_optional_user propagation
tests/fixtures/jwks_fixtures.py	locally generated RSA key sets for the JWKS tests

Review notes

Two things worth a look before merging:

• .mcp.json is committed at the repo root and points at a machine-local
  PyCharm MCP endpoint (http://127.0.0.1:64462/stream). That port is specific
  to one developer's IDE session — it probably belongs in .gitignore (as
  .vscode/.cursor are in this same commit) rather than in the repo.
• _warn_if_auth0_misconfigured() is commented out rather than deleted —
  wrapped in a """ block in api/main.py, with the call site left as a
  comment. It is fully replaced by _validate_auth0_configuration(); worth
  deleting outright.