Mirror provider EIDs to a cookie and run pull sync as an identity provider

CHANGELOG.md

@@ -29,7 +29,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   On Fastly the final snapshot may wait up to 1 ms per partially received body so
   the host can surface bytes that already arrived, and reads on into the body of
   a response whose headers arrived by the cutoff instead of abandoning it.
-- Added server-side identity providers (`[[ec.identity_providers]]`, Fastly only). Enabled providers look up a visitor's IDs on publisher navigations within `ec.identity_budget_ms` (1–1000, default 200), or after the response within `ec.identity_after_response_budget_ms` (1–5000, default 2000) for `collect = "after_response"`. A provider owns its `sources`: batch sync rejects them with the new `provider_owned` reason, pull sync skips them, and browser IDs never fill them. Set `tcf_vendor_id` to require TCF vendor consent for lookups and for every ID of the provider's sources in bidstream, `x-ts-eids` and `/identify`, browser-sent IDs included; a disabled provider keeps that gate. See [Identity providers](docs/guide/configuration.md#identity-providers).
+- Added server-side identity providers (`[[ec.identity_providers]]`, Fastly only). Enabled providers look up a visitor's IDs on publisher navigations within `ec.identity_budget_ms` (1–1000, default 200), or after the response within `ec.identity_after_response_budget_ms` (1–5000, default 2000) for `collect = "after_response"`. A provider owns its `sources`: batch sync rejects them with the new `provider_owned` reason, and browser IDs never fill them. Set `tcf_vendor_id` to require TCF vendor consent for lookups and for every ID of the provider's sources in bidstream, `x-ts-eids` and `/identify`, browser-sent IDs included; a disabled provider keeps that gate. See [Identity providers](docs/guide/configuration.md#identity-providers).
+- Pull sync now runs as a `pull-sync` identity provider built from the existing `[[ec.partners]]` settings, after the response on top-level publisher navigations from recognized browsers, within `ec.identity_after_response_budget_ms` shared with other after-response providers. It no longer runs after `/auction` responses, on prefetches or on navigations without Fetch Metadata, and a partner slower than the budget is not filled, where each pull used to wait up to 15 seconds. `ec.pull_sync_concurrency` now caps the partners pulled per navigation (`0` means no limit; it used to mean one at a time), partner order rotates hourly, and a partner past the cap logs `skipped_concurrency_cap`. The `ts-ec-pull-complete` marker is retired and leftover cookies are expired; without it, a returning navigation reads the EC row before the origin response even when nothing else needs it. `kind = "pull-sync"` in `[[ec.identity_providers]]` is rejected, as is a configured provider that claims a pull-sync partner's source.
+- Added `mirror_enabled` on `[[ec.partners]]` (default `false`; requires `bidstream_enabled`). The partner's provider-written UIDs are mirrored to the JavaScript-readable `__Host-ts-eids-mirror` cookie, leaving out a source's UIDs while any provider claiming it lacks TCF vendor consent. See [EID Mirror Cookie](docs/guide/edge-cookies.md#eid-mirror-cookie).
+- **Breaking:** For a partner with pull sync enabled, a resolved `ts_pull_token` that cannot form an `Authorization: Bearer` header, such as one ending in a newline, now fails config validation and stops startup. Before, only that partner's pull sync failed, after the page was sent. Check pull-token secrets before deploying.
 - Batch sync reports a new `row_full` item reason when a mapping would push the EC row past its new 16 KiB size limit, and continues with the remaining groups.
 - An EC row over 16 KiB written by an earlier release loses its largest partner records on its next write that adds an ID, so the added ID still lands.
 - **Breaking:** An EC is now minted, and the EC row written, only on a top-level publisher navigation: a `GET` with `Sec-Fetch-Dest: document`, `Sec-Fetch-Mode: navigate` and no `prefetch` in `Sec-Purpose`. Clients that send no Fetch Metadata get no EC, a navigation answered with a 301, 302, 303, 307 or 308 redirect mints none, integration proxies no longer mint one, and prefetches and prerenders never mint or write. Each navigation makes at most one conditional page-load write; a consent withdrawal's tombstone is still written on any request that carries the EC cookie.

crates/trusted-server-adapter-fastly/src/app.rs

@@ -373,7 +373,6 @@ pub(crate) struct EcFinalizeState {
     pub(crate) use_finalize_kv: bool,
     pub(crate) eids_cookie: Option<String>,
     pub(crate) sharedid_cookie: Option<String>,
-    pub(crate) is_real_browser: bool,
     /// Per-request services carried to the entry point so the pull-sync
     /// dispatcher can reuse the same platform HTTP client.
     pub(crate) services: RuntimeServices,
@@ -461,14 +460,20 @@ fn joined_cookie_header(req: &Request) -> String {
 /// Captures the request signals for after-response identity lookups on a
 /// permitted publisher navigation.
 ///
-/// Returns `None` when no after-response provider is enabled, the request has
+/// Returns `None` when no after-response provider (including a pull-sync
+/// partner) is enabled, the client failed the browser check, the request has
 /// no EC, or the identity graph is unavailable.
 fn prepare_after_response_lookups(
     state: &AppState,
     req: &Request,
     ec: &EcRequestState,
 ) -> Option<AfterResponseLookups> {
     state.after_response_identity.as_ref()?;
+    // Bot gate: pull sync and other after-response lookups disclose the EC ID
+    // to partners, so they run only for recognized browsers.
+    if !ec.is_real_browser {
+        return None;
+    }
     ec.kv_graph.as_ref()?;
     let ec_id = ec.ec_context.ec_value()?.to_owned();
     let signals = ProviderRequestSignals::read(req, ec);
@@ -560,7 +565,6 @@ impl EcRequestState {
             use_finalize_kv: self.finalize_kv_graph.is_some(),
             eids_cookie: self.eids_cookie,
             sharedid_cookie: self.sharedid_cookie,
-            is_real_browser: self.is_real_browser,
             services: self.services,
         }
     }
@@ -935,8 +939,6 @@ async fn run_named_route(
 /// graph + partner registry + rate limiter, with a default EC context for
 /// response finalization.
 fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> Response {
-    let device_signals = device_signals_for(&req);
-    let is_real_browser = device_signals.looks_like_browser();
     let eids_cookie = crate::extract_cookie_value(&req, COOKIE_TS_EIDS);
     let sharedid_cookie = crate::extract_cookie_value(&req, COOKIE_SHAREDID);
 
@@ -954,7 +956,6 @@ fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) ->
         use_finalize_kv: false,
         eids_cookie,
         sharedid_cookie,
-        is_real_browser,
         services: services.clone(),
     });
     response
@@ -3474,11 +3475,68 @@ mod tests {
             .get::<super::EcFinalizeState>()
             .expect("fallback response should carry EcFinalizeState");
         assert!(
-            finalize.is_real_browser,
+            finalize
+                .ec_context
+                .device_signals()
+                .is_some_and(DeviceSignals::looks_like_browser),
             "browser-shaped JA4/H2 signals from the request extension must mark the request as a real browser"
         );
     }
 
+    #[test]
+    fn after_response_lookups_run_only_for_real_browsers() {
+        let mut settings = test_settings();
+        settings.ec.ec_store = Some("ec_identity_store".to_owned());
+        settings.ec.partners = vec![
+            serde_json::from_value(serde_json::json!({
+                "name": "Pull partner",
+                "source_domain": "partner.example.com",
+                "pull_sync_enabled": true,
+                "pull_sync_url": "https://sync.example.com/pull",
+                "pull_sync_allowed_domains": ["sync.example.com"],
+                "ts_pull_token": "example-pull-token-32-bytes-minimum",
+            }))
+            .expect("should build pull-sync partner"),
+        ];
+        let state = build_state_from_settings(settings).expect("should build app state");
+        let browser_request = || {
+            let mut req = request_builder()
+                .method(Method::GET)
+                .uri("https://www.example.com/article")
+                .header(header::COOKIE, format!("ts-ec={}.test01", "a".repeat(64)))
+                .body(Body::empty())
+                .expect("should build request");
+            req.extensions_mut().insert(DeviceSignals::derive(
+                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 \
+                 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36",
+                Some("t13d1516h2_8daaf6152771_b186095e22b6"),
+                Some("1:65536;2:0;4:6291456;6:262144"),
+            ));
+            req
+        };
+        let context = RequestContext::new(browser_request(), PathParams::default());
+        let services = build_per_request_services(&state, &context);
+        let req = browser_request();
+        let mut ec = super::build_ec_request_state(&state.settings, &services, &req);
+        assert!(
+            ec.is_real_browser,
+            "should classify the request as a browser"
+        );
+
+        let browser = super::prepare_after_response_lookups(&state, &req, &ec);
+        ec.is_real_browser = false;
+        let bot = super::prepare_after_response_lookups(&state, &req, &ec);
+
+        assert!(
+            browser.is_some(),
+            "should prepare pull sync for a real browser"
+        );
+        assert!(
+            bot.is_none(),
+            "should never prepare pull sync for a client that fails the browser check"
+        );
+    }
+
     #[test]
     fn missing_device_signals_extension_classifies_as_non_browser() {
         // Without the entry-point-captured signals, the reconstructed request
@@ -3493,7 +3551,10 @@ mod tests {
             .get::<super::EcFinalizeState>()
             .expect("fallback response should carry EcFinalizeState");
         assert!(
-            !finalize.is_real_browser,
+            !finalize
+                .ec_context
+                .device_signals()
+                .is_some_and(DeviceSignals::looks_like_browser),
             "a request without captured device signals should not be classified as a real browser"
         );
     }

crates/trusted-server-adapter-fastly/src/main.rs

@@ -26,9 +26,6 @@ use trusted_server_core::ec::finalize::ec_finalize_response;
 use trusted_server_core::ec::kv::KvIdentityGraph;
 use trusted_server_core::ec::page_load::NavigationPermit;
 use trusted_server_core::ec::provider::{ClientSignals, RequestCookies};
-use trusted_server_core::ec::pull_sync::{
-    PullSyncContext, build_pull_sync_context, dispatch_pull_sync,
-};
 use trusted_server_core::ec::registry::PartnerRegistry;
 use trusted_server_core::error::TrustedServerError;
 use trusted_server_core::integrations::RequestFilterEffects;
@@ -570,13 +567,12 @@ fn edgezero_main(
                 identity,
                 &mut response,
             ) {
-                Ok(partner_registry) => {
+                Ok(_) => {
                     send_edgezero_response(
                         response,
                         request_filter_effects.as_ref(),
                         counters.as_ref(),
                     );
-                    run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state);
                     if let Some(state) = app_state.as_deref() {
                         run_after_response_lookups(state, &mut ec_state, after_response);
                     }
@@ -598,17 +594,12 @@ fn edgezero_main(
                         identity,
                         &mut response,
                     ) {
-                        Ok(partner_registry) => {
+                        Ok(_) => {
                             send_edgezero_response(
                                 response,
                                 request_filter_effects.as_ref(),
                                 counters.as_ref(),
                             );
-                            run_edgezero_pull_sync_after_send(
-                                &settings,
-                                &partner_registry,
-                                &ec_state,
-                            );
                             return;
                         }
                         Err(e) => {
@@ -720,33 +711,6 @@ fn apply_edgezero_ec_finalize(
     Ok(partner_registry)
 }
 
-fn run_edgezero_pull_sync_after_send(
-    settings: &Settings,
-    partner_registry: &PartnerRegistry,
-    ec_state: &EcFinalizeState,
-) {
-    if !ec_state.is_real_browser {
-        return;
-    }
-
-    let prepared_context = build_pull_sync_context(&ec_state.ec_context, partner_registry);
-    let Some((context, kv)) =
-        prepare_pull_sync_after_send(prepared_context, || require_identity_graph(settings))
-    else {
-        return;
-    };
-
-    let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME);
-    dispatch_pull_sync(
-        settings,
-        &kv,
-        partner_registry,
-        &limiter,
-        &context,
-        &ec_state.services,
-    );
-}
-
 /// Runs after-response identity lookups and applies their follow-up write.
 ///
 /// Called once the response has been sent, so provider latency never delays
@@ -775,6 +739,15 @@ pub(crate) fn run_after_response_lookups(
         return;
     };
     let settings = &*state.settings;
+    // The request snapshot holds the row as finalize left it. A source held
+    // there is still held in the live row unless consent was withdrawn, which
+    // blocks the lookup anyway, so the live read is skipped when nothing is due.
+    let now = current_timestamp();
+    let request_entry = ec_state.ec_context.kv_snapshot().entry_for(&lookups.ec_id);
+    if !coordinator.any_due(request_entry, UnixSeconds::new(now)) {
+        log::debug!("After-response lookups skipped: nothing due");
+        return;
+    }
     let Some(graph) = maybe_identity_graph(settings) else {
         return;
     };
@@ -789,7 +762,6 @@ pub(crate) fn run_after_response_lookups(
     }
     let transport = PlatformLookupTransport::new(&ec_state.services);
     let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME);
-    let now = current_timestamp();
     let pending = futures::executor::block_on(coordinator.dispatch(
         &transport,
         &LookupContext {
@@ -820,24 +792,6 @@ pub(crate) fn run_after_response_lookups(
     log::info!("After-response lookups: write outcome={outcome}");
 }
 
-fn prepare_pull_sync_after_send<F>(
-    context: Option<PullSyncContext>,
-    graph_factory: F,
-) -> Option<(PullSyncContext, KvIdentityGraph)>
-where
-    F: FnOnce() -> Result<KvIdentityGraph, Report<TrustedServerError>>,
-{
-    let context = context?;
-    let kv = match graph_factory() {
-        Ok(kv) => kv,
-        Err(err) => {
-            log::debug!("Pull sync: identity graph unavailable, skipping: {err:?}");
-            return None;
-        }
-    };
-    Some((context, kv))
-}
-
 /// Sends a finalized `EdgeZero` response to the client.
 ///
 /// Streaming `EdgeZero` bodies commit headers first, then pipe chunks to Fastly's
@@ -1280,23 +1234,6 @@ mod tests {
         }
     }
 
-    #[test]
-    fn pull_sync_noop_states_skip_post_send_graph_factory() {
-        let calls = std::cell::Cell::new(0);
-        let result = prepare_pull_sync_after_send(None, || {
-            calls.set(calls.get() + 1);
-            Err(Report::new(TrustedServerError::KvStore {
-                store_name: "unexpected".to_owned(),
-                message: "graph factory should not run".to_owned(),
-            }))
-        });
-        assert!(
-            result.is_none(),
-            "a skipped pull-sync plan should return none"
-        );
-        assert_eq!(calls.get(), 0, "should not invoke the graph factory");
-    }
-
     #[test]
     fn trace_path_classification_survives_core_request_conversion() {
         // `main` skips client-IP sanitization when the native path is a trace

crates/trusted-server-core/src/auction/endpoints.rs

@@ -955,6 +955,7 @@ mod tests {
             source_domain: source_domain.to_owned(),
             openrtb_atype: crate::settings::EcPartner::default_openrtb_atype(),
             bidstream_enabled: true,
+            mirror_enabled: false,
             api_token: Some(crate::redacted::Redacted::new(format!(
                 "token-{source_domain}-32-bytes-minimum-value"
             ))),