Collect server-side identity provider EIDs in a single page-load write

.cargo/config.toml

@@ -28,8 +28,8 @@ test_details = "test --target aarch64-apple-darwin"
 # (native) are simply not listed.
 build-fastly = "build -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1"
 check-fastly = "check -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1"
-clippy-fastly = "clippy -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-js -p trusted-server-openrtb --all-targets --all-features --target wasm32-wasip1 -- -D warnings"
-test-fastly = "test -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1"
+clippy-fastly = "clippy -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-identity-providers -p trusted-server-js -p trusted-server-openrtb --all-targets --all-features --target wasm32-wasip1 -- -D warnings"
+test-fastly = "test -p trusted-server-core -p trusted-server-adapter-fastly -p trusted-server-identity-providers -p trusted-server-js -p trusted-server-openrtb --target wasm32-wasip1"
 # Feature-on counterpart of `test-fastly`. `test-fastly` does NOT pass
 # --all-features, so without this the reusable-sandbox code is linted by
 # `clippy-fastly` but its tests never execute.

AGENTS.md

@@ -21,6 +21,7 @@ crates/
   trusted-server-adapter-cloudflare/    # Cloudflare Workers entry point (wasm32-unknown-unknown binary)
   trusted-server-adapter-spin/          # Fermyon Spin entry point (wasm32-wasip1 component)
   trusted-server-cli/                   # Host-target `ts` operator CLI
+  trusted-server-identity-providers/    # Server-side identity provider kinds (used by the Fastly adapter)
   trusted-server-js/                    # TypeScript/JS build — per-integration IIFE bundles
     lib/         # TS source, Vitest tests, esbuild pipeline
 ```

CHANGELOG.md

@@ -22,10 +22,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   measured from dispatch, including delayed headers and trickling bodies on all
   adapters. Complete available responses survive a slow publisher origin;
   unfinished providers time out and mediation is skipped after budget exhaustion.
-  Provider bids that are already complete at the final nonblocking snapshot remain
+  Provider bids that are already complete at the final snapshot remain
   eligible even after `tmax`; mediation starts only with remaining budget and
   rejects responses collected after its cutoff.
   Fastly abandons unfinished requests; Axum, Cloudflare, and Spin cancel owned work.
+  On Fastly the final snapshot may wait up to 1 ms per partially received body so
+  the host can surface bytes that already arrived, and reads on into the body of
+  a response whose headers arrived by the cutoff instead of abandoning it.
+- Added server-side identity providers (`[[ec.identity_providers]]`, Fastly only). Enabled providers look up a visitor's IDs on publisher navigations within `ec.identity_budget_ms` (1–1000, default 200), or after the response within `ec.identity_after_response_budget_ms` (1–5000, default 2000) for `collect = "after_response"`. A provider owns its `sources`: batch sync rejects them with the new `provider_owned` reason, pull sync skips them, and browser IDs never fill them. Set `tcf_vendor_id` to require TCF vendor consent for lookups and for every ID of the provider's sources in bidstream, `x-ts-eids` and `/identify`, browser-sent IDs included; a disabled provider keeps that gate. See [Identity providers](docs/guide/configuration.md#identity-providers).
+- Batch sync reports a new `row_full` item reason when a mapping would push the EC row past its new 16 KiB size limit, and continues with the remaining groups.
+- An EC row over 16 KiB written by an earlier release loses its largest partner records on its next write that adds an ID, so the added ID still lands.
+- **Breaking:** An EC is now minted, and the EC row written, only on a top-level publisher navigation: a `GET` with `Sec-Fetch-Dest: document`, `Sec-Fetch-Mode: navigate` and no `prefetch` in `Sec-Purpose`. Clients that send no Fetch Metadata get no EC, a navigation answered with a 301, 302, 303, 307 or 308 redirect mints none, integration proxies no longer mint one, and prefetches and prerenders never mint or write. Each navigation makes at most one conditional page-load write; a consent withdrawal's tombstone is still written on any request that carries the EC cookie.
+- **Breaking:** Browser EIDs from the `/auction` request body are no longer stored, and `/auction` and `/_ts/page-bids` no longer write browser EIDs from `ts-eids` or `sharedId`. Those cookies are ingested on the next top-level navigation.
+- **Breaking:** An EC minted on the current navigation is no longer sent as `user.id` to bidders, that navigation's auction does not carry the browser's `ts-eids`, and identity providers are not called for it, until a later navigation finds its row. Identity providers are also not called for an EC cookie whose row does not exist.
+- This release registers no identity provider kind, so `ts config push` rejects an enabled `[[ec.identity_providers]]` entry and every runtime refuses to start with one; configure providers with `enabled = false` until a kind ships.
+- A consent withdrawal keeps the `ts-ec` cookie while its tombstone cannot be written (after bounded CAS retries, or when the row cannot be read), so the next request carrying the cookie retries. Previously the cookie was expired first.
+- **Breaking:** Axum, Cloudflare and Spin refuse to start when an enabled identity provider is configured, and Fastly refuses to start when an enabled provider names a `kind` this build does not include.
+- **Breaking (rollback):** Stored provider UIDs may be up to 1024 bytes (browser, batch and pull ingest stay at 512). A binary from before this release cannot read an EC row holding a longer UID, so roll back only after such rows expire or are rewritten. An older binary also ignores the full-fidelity `eids` and the 30-day provider `opt_outs` and drops them when it rewrites a row (for example in batch or pull sync), so opted-out visitors are looked up again after rolling forward. A config blob that sets `ec.identity_providers` or a non-default identity budget cannot be loaded by an older binary; restore the defaults and re-push before rolling back.
 - Explicit TS Console activation with `?ts_console=1` now sets a 30-minute diagnostics cookie instead of a browser-session cookie, independently of the mobile trace flag. Ordinary requests do not refresh it; end and explicitly enable diagnostics again to adopt the lifetime for an existing session.
 - Publisher documents with diagnostics active now use private, no-store responses and strip conditional and range request headers before fetching the origin. An unexpected origin `304` becomes a private `502` because it cannot supply an instrumented document, including when mobile tracing is disabled.
 - **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER__GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header.

Cargo.toml

@@ -7,6 +7,7 @@ members = [
     "crates/trusted-server-adapter-spin",
     "crates/trusted-server-cli",
     "crates/trusted-server-core",
+    "crates/trusted-server-identity-providers",
     "crates/trusted-server-integration-tests",
     "crates/trusted-server-js",
     "crates/trusted-server-openrtb",
@@ -121,6 +122,7 @@ toml_edit = "0.23.10"
 tower = "0.4"
 tracing = "0.1"
 trusted-server-core = { path = "crates/trusted-server-core" }
+trusted-server-identity-providers = { path = "crates/trusted-server-identity-providers" }
 trusted-server-js = { path = "crates/trusted-server-js" }
 trusted-server-openrtb = { path = "crates/trusted-server-openrtb" }
 url = "2.5.8"

crates/trusted-server-adapter-axum/src/app.rs

@@ -91,6 +91,7 @@ fn build_state_with_services(
     settings: Settings,
     services: Option<RuntimeServices>,
 ) -> Result<Arc<AppState>, Report<TrustedServerError>> {
+    trusted_server_core::ec::provider::reject_enabled_providers(&settings.ec, "Axum")?;
     let plan = Arc::new(compile_auction_plan(&settings)?);
     plan.validate_for_target(trusted_server_core::platform::AuctionTargetId::Axum)?;
     let orchestrator = build_orchestrator_with_plan(Arc::clone(&plan), &settings)?;
@@ -224,15 +225,12 @@ async fn dispatch_fallback(
     }
 
     if state.registry.has_route(&method, &path) {
-        let mut ec_context = EcContext::default();
         return state
             .registry
             .handle_proxy(ProxyDispatchInput {
                 method: &method,
                 path: &path,
                 settings: &state.settings,
-                kv: None,
-                ec_context: &mut ec_context,
                 services,
                 req,
             })
@@ -499,8 +497,7 @@ fn named_route_handler(
                         Ok(admin_ec_lookup_not_supported())
                     }
                     NamedRouteHandler::AdminEidsLookup => {
-                        let partner_registry =
-                            PartnerRegistry::from_config(&state.settings.ec.partners)?;
+                        let partner_registry = PartnerRegistry::from_ec(&state.settings.ec)?;
                         handle_admin_eids_lookup(&partner_registry, &req)
                     }
                     NamedRouteHandler::LegacyAdminDenied => Ok(legacy_admin_alias_denied()),
@@ -746,14 +743,33 @@ fn trace_metadata(state: &AppState, request: &Request) -> TraceMetadata {
         geo: None,
     }
 }
+
 #[cfg(test)]
 mod trace_dispatch_tests {
     use super::*;
+
     use futures::executor::block_on;
     use trusted_server_core::redacted::Redacted;
     use trusted_server_core::settings::TrustedClientIpConfig;
     use trusted_server_core::trace::TraceTerminalResponse;
 
+    #[test]
+    fn enabled_identity_provider_fails_startup() {
+        let mut settings = trace_settings(false);
+        settings.ec.identity_providers =
+            vec![trusted_server_core::test_support::fixtures::enabled_identity_provider()];
+
+        let Err(error) = build_state_with_settings(settings) else {
+            panic!("should refuse to start with an enabled identity provider");
+        };
+
+        let message = format!("{error:?}");
+        assert!(
+            message.contains("example-provider") && message.contains("Axum"),
+            "should name the provider and the adapter: {message}"
+        );
+    }
+
     fn trace_settings(enabled: bool) -> Settings {
         Settings::from_toml(&format!(
             r#"

crates/trusted-server-adapter-cloudflare/src/app.rs

@@ -204,6 +204,7 @@ fn build_state_with_services(
     settings: Settings,
     services: Option<RuntimeServices>,
 ) -> Result<Arc<AppState>, Report<TrustedServerError>> {
+    trusted_server_core::ec::provider::reject_enabled_providers(&settings.ec, "Cloudflare")?;
     let plan = Arc::new(compile_auction_plan(&settings)?);
     plan.validate_for_target(trusted_server_core::platform::AuctionTargetId::Cloudflare)?;
     let orchestrator = build_orchestrator_with_plan(Arc::clone(&plan), &settings)?;
@@ -535,15 +536,12 @@ fn build_router(state: &Arc<AppState>) -> RouterService {
                     EdgeCacheHeader::CloudflareCdnCacheControl,
                 )
             } else if state.registry.has_route(&method, &path) {
-                let mut ec_context = EcContext::default();
                 state
                     .registry
                     .handle_proxy(ProxyDispatchInput {
                         method: &method,
                         path: &path,
                         settings: &state.settings,
-                        kv: None,
-                        ec_context: &mut ec_context,
                         services: &services,
                         req,
                     })
@@ -655,7 +653,7 @@ fn build_router(state: &Arc<AppState>) -> RouterService {
             .get(
                 "/_ts/admin/eids",
                 make_handler(Arc::clone(&state), |s, _services, req| async move {
-                    let partner_registry = PartnerRegistry::from_config(&s.settings.ec.partners)?;
+                    let partner_registry = PartnerRegistry::from_ec(&s.settings.ec)?;
                     handle_admin_eids_lookup(&partner_registry, &req)
                 }),
             )
@@ -825,6 +823,23 @@ fn trace_metadata(state: &AppState, request: &Request) -> TraceMetadata {
 mod tests {
     use super::*;
 
+    #[test]
+    fn enabled_identity_provider_fails_startup() {
+        let mut settings = aps_profile_settings();
+        settings.ec.identity_providers =
+            vec![trusted_server_core::test_support::fixtures::enabled_identity_provider()];
+
+        let Err(error) = build_state_with_settings(settings) else {
+            panic!("should refuse to start with an enabled identity provider");
+        };
+
+        let message = format!("{error:?}");
+        assert!(
+            message.contains("example-provider") && message.contains("Cloudflare"),
+            "should name the provider and the adapter: {message}"
+        );
+    }
+
     fn aps_profile_settings() -> Settings {
         let mut settings = Settings::from_toml(
             r#"

crates/trusted-server-adapter-fastly/Cargo.toml

@@ -37,6 +37,7 @@ log-fastly = { workspace = true }
 serde = { workspace = true }
 serde_json = { workspace = true }
 trusted-server-core = { workspace = true }
+trusted-server-identity-providers = { workspace = true }
 url = { workspace = true }
 urlencoding = { workspace = true }
 web-time = { workspace = true }