Skip to content

JITSU-227: Google Data Manager adapter and scoped OAuth/recovery - #1523

Merged
absorbb merged 2 commits into
newjitsufrom
feat/jitsu-227-google-data-manager
Sep 17, 2026
Merged

absorbb merged 2 commits into
newjitsufrom
feat/jitsu-227-google-data-manager

Conversation

@absorbb

@absorbb absorbb commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

JITSU-227 — Google Data Manager adapter, scoped OAuth and independently scheduled provider recovery.

  • Add builtin.reverse.google-ads / audience beside the existing event implementation. Support existing-audience additions and explicit tombstone removals; event conversion delivery is unchanged.
  • Support raw/pre-hashed email and phone. Normalize/hash before the journal; require explicit consent and Customer Match terms; removals need no granted consent.
  • Stage ingest/remove request IDs in durable per-batch receipts. Recover by polling without querying the source or resending requests. Accept only verified whole-request success; stop on partial/warned/unknown outcomes without guessing row acceptance.
  • Resolve Google OAuth through a service-authenticated, workspace/sync/revision-scoped console endpoint. Enforce destination-owned connection and fixed integration, reread admission after Nango retrieval. Runner gets only a short-lived access token cached in memory, not the Nango secret or refresh token.
  • Pending provider processing now ends the worker as WAITING, not FAILED. The console shows processing and the next check time; workers exit cleanly and syncctl cleans up their Pods.
  • Persist the recovery schedule in existing source_task.metrics. Syncctl independently launches one-shot recovery Pods: first check after 30 minutes, 1.3x backoff capped at one hour, fixed 24-hour polling window. This does not depend on the model CronJob and also works for manual-only syncs.
  • Bind each check to its waiting parent, logical run, workspace and revision. Kubernetes leases and fresh console admission remain mandatory. Cancel WAITING without a live Pod; queued/stale recovery workers cannot revive cancelled/superseded work or start fresh extraction. Prior waiting attempts become RESUMED when another attempt takes over.

Deliberate boundaries

  • No live audience writes, Google Cloud provisioning, service deployment, billing or database migrations.
  • Reverse sync UI creation remains gated. Google mirror remains disabled until the next audience-provisioning/baseline and unchanged-member-expiry-refresh slice.
  • No blind POST retry, including 401/429/5xx/timeouts. Lost request ID or unattributable partial results require operator reconciliation; no override/reset UI in this PR.
  • OAuth failure before any Google call is a definite non-submission: persist rejected outcomes and fail the attempt. Independent requests already in flight must still settle before cleanup.
  • Real failures remain FAILED and do not automatically retry indefinitely. If the final polling check is still pending, stop automatic checks with an explicit timeout and retain recovery state. Scheduling delays may postpone that final check; manual/cron recovery later does not reset the same run's polling deadline.
  • Cancelling a waiting attempt stops its automatic recovery chain, not already-submitted provider work or the model's regular CronJob.
  • In-memory token caching is bounded; disabling a sync blocks fresh issuance, not instant revocation of cached/in-flight tokens.

Validation

  • Runner: 189 tests passed; one optional scale test skipped. Includes real PostgreSQL recovery, stale/duplicate/cancelled parent rejection, fixed deadline propagation, permanent rejection, and no source replay.
  • Destination-functions: 171 tests passed. Credential-gated live tests were not exercised.
  • Console: 30 integration tests passed across sync-service and OAuth/export/admission, including scoped reverse-vs-connector cancellation dispatch.
  • Syncctl: tests passed, including disposable PostgreSQL due-check selection, terminal cleanup, cancellation scope and failed pre-start child handling; fake Kubernetes tests cover deterministic Pod creation, distinct check attempts, and revision binding.
  • Runner and console typechecks, runner bundle, formatting and diff checks passed.
  • Go tests run with -vet=off: ordinary go test encounters a pre-existing non-constant fmt.Errorf(stderr.String()) vet error in job_runner.go outside this change.
  • Local risky-tier review covers the incremental changes; the cancellation race found in the first checkpoint was fixed and regression-tested.

Deployment

Owner approved deploy:console for beta console deployment on merge. Runner and syncctl require separate releases together so the controller recognizes WAITING/RESUMED and schedules recovery. Syncctl's DB role needs SELECT on reverse_sync_control in addition to its existing task permissions. Fresh Google OAuth project/Data Manager API enablement and scopes are documented, not provisioned here.

See libs/destination-functions/src/functions/google-ads-reverse/README.md for the provider contract and services/retl-runner/src/runtime.md for scheduling, cancellation and deployment details.

JITSU-227: deliver existing-audience additions and explicit removals with durable request polling. Keep mirror and UI creation gated pending audience provisioning and membership refresh.
@absorbb absorbb added the deploy:console Auto-deploy console to beta when this PR merges (JITSU-68) label Sep 16, 2026
jitsu-code-review[bot]
jitsu-code-review Bot previously approved these changes Sep 16, 2026

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Google Data Manager adapter, durable request reconciliation, and scoped runner OAuth flow. No actionable correctness, security, or user-visible regression findings identified.

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Google Data Manager adapter, scoped OAuth endpoint/client, and reverse-delivery recovery lifecycle (runner, syncctl, and console task handling). No actionable correctness, security, or user-visible regression findings in this range.

@absorbb absorbb removed the deploy:console Auto-deploy console to beta when this PR merges (JITSU-68) label Sep 17, 2026
@absorbb
absorbb merged commit e032341 into newjitsu Sep 17, 2026
12 checks passed
@absorbb
absorbb deleted the feat/jitsu-227-google-data-manager branch September 17, 2026 08:29
absorbb added a commit that referenced this pull request Sep 17, 2026
…1524)

## Summary

JITSU-227 — **stacked on #1523** (`feat/jitsu-227-google-data-manager`).
Merge that prerequisite first; this PR's reviewable diff is the
managed-audience/expiry-refresh slice only.

- Add explicit, workspace-authorized Data Manager audience provisioning
with a durable internal creation intent. CAS allows one POST; uncertain
results reconcile by a persisted server-random marker, never blindly
recreate or adopt an arbitrary existing list.
- Bind Jitsu-created audiences to one intended sync/account/OAuth
configuration. Export strips caller-supplied proof and supplies
validated internal evidence; the runner verifies remote
identity/type/ownership/marker before mirror execution.
- Enable core snapshot mirroring for those managed audiences. Existing
audiences remain additions/explicit removals only. Persist
provider-ready hashed payloads and reuse them across recovery.
- Apply the owner-approved **540-day membership / 30-day refresh**
policy. Unchanged fresh members are skipped; due refreshes must be
accepted before removals. Two additive Prisma-managed timestamps store
member acceptance time and an immutable generation cutoff.

## Scope / deployment

- Reverse sync editor/creation rollout remains gated and is the next
slice. This is not a production availability announcement.
- **Manual deployment** requested: no `deploy:console` label. Runner
deployment remains separate. Apply the Prisma schema before deploying
the updated runner; existing table grants cover the new columns.
- No billing, worker ownership layer, application encryption, live
advertising writes or infrastructure provisioning.
- Exclusive audience management is an operator agreement, not an
API-enforced Google lock. Other writers must not modify managed
audiences. An empty estimated size never establishes a baseline.
- Missing/ambiguous creation evidence stays unresolved; no automatic
reset/delete or second create request. Paused/failed/infrequent syncs
can still let members expire.

## Validation

- Destination-functions: 186 tests passed (live credential-gated
integrations not exercised).
- Node runner: 193 passed, 1 optional scale test skipped; runner build
passed.
- Console: 130 unit tests passed; provisioning/OAuth integration tests
passed against disposable PostgreSQL and mocked HTTP.
- Destination-functions, runner and console typechecks passed; changed
console files passed ESLint; diff/format checks passed.
- Local risky-tier finder/verifier/fresh final review before push. Fixed
predictable creation correlation with a server-random nonce and added
regression coverage.

Implementation decisions and API/Hightouch research:
`services/retl-runner/src/google-audience-design.md`; provider
setup/recovery:
`libs/destination-functions/src/functions/google-ads-reverse/README.md`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant