feat(firestore-bigquery-export): reinstate the Cloud Tasks write buffer

kits/firestore-bigquery-export/CHANGELOG.md

@@ -1,3 +1,4 @@
+- feat: reinstate the extension's Cloud Tasks write buffer. A failed inline BigQuery write now enqueues onto a new `syncBigQuery` task queue (5 attempts, 60s minimum backoff, throttled by the restored `MAX_DISPATCHES_PER_SECOND` param, default 100) instead of replaying the Firestore event through Eventarc redelivery for up to 24 hours; `MAX_ENQUEUE_ATTEMPTS` (default 3) is also back. The `onSuccess` event returns with the queue handler. Two behavior changes against earlier release candidates: a row that exhausts the queue is dropped unless `BACKUP_COLLECTION` is set (extension parity - the tracker backs the row up on every terminal insert failure, so configure a backup collection), and deleting or moving the functions can leave the Cloud Tasks queue behind. A failed enqueue is logged at error level, published as an `onError` event, and dropped, as in the extension; the trigger no longer declares `retry: true`, so nothing is redelivered through Eventarc. Export the new `syncBigQuery` function from your codebase entry, and deploy with Firebase CLI 15.28.0+ so the trigger can address its own queue (`FIREBASE_KIT_INSTANCE_ID`); requires firebase-admin 14.2.0+.
 - fix: restore explicit function placement from `DATABASE_REGION`, now with the Firestore-location-to-Cloud-Run-region mapping. The `DATABASE_REGION` parameter is back and all three functions deploy to the region derived from it: regional locations pass through unchanged, and the multi-region locations map to a Cloud Run region (`nam5`/`nam7` to `us-central1`, `eur3` to `europe-west1`) instead of failing the deploy. With the parameter unset the functions still declare no region and the CLI falls back as before (`us-central1` by default, `FIREBASE_FUNCTIONS_DEFAULT_REGION` to override). Placement requires firebase-tools >= 15.28.0 (older CLIs do not load `.env` at discovery and keep the fallback). If your `.env` already carries `DATABASE_REGION` from an extension migration, upgrading to this version moves the functions to the mapped region on your next deploy, which deletes and recreates them.
 - fix: stop deploying functions to the `DATABASE_REGION` value. Firestore multi-region locations (`eur3`, `nam5`, `nam7`) are not Cloud Run regions, so any multi-region database made every deploy fail. The `DATABASE_REGION` parameter is removed; the functions now declare no region and deploy to `us-central1` by default (set `FIREBASE_FUNCTIONS_DEFAULT_REGION` when deploying to choose another region), while the Firestore trigger is always pinned to the database's own region. `ExportConfig.location` is removed from the library surface.
 - Initial release of kit, see README for differences between the legacy extension and this kit

kits/firestore-bigquery-export/README.md

@@ -5,9 +5,10 @@ BigQuery Firebase Extension as an npm package you add to your own Firebase
 Functions codebase and deploy.
 
 It listens for document writes on a collection, serializes each change, and
-writes it to a BigQuery changelog table. Failed writes are retried through a
-Firebase Functions runtime retry policy. The functions run in your own Firebase
-project; there is no hosted version, so you deploy them yourself.
+writes it to a BigQuery changelog table. Failed writes buffer through a Cloud
+Tasks queue (`syncBigQuery`), which retries them on its own throttled schedule.
+The functions run in your own Firebase project; there is no hosted version, so
+you deploy them yourself.
 
 ## Install
 
@@ -30,6 +31,7 @@ conflicts with that automatic setup.
 | `roles/datastore.user`         | write failed-row records back to Firestore (only if you configure a backup collection) |
 | `roles/eventarc.eventReceiver` | receive Gen2 Firestore trigger events                                                  |
 | `roles/run.invoker`            | allow Eventarc to invoke the Gen2 Cloud Run service                                    |
+| `roles/cloudtasks.enqueuer`    | enqueue failed writes onto the kit's own `syncBigQuery` task queue                     |
 | `bigquery.googleapis.com`      | mirror Firestore collection changes in BigQuery                                        |
 
 If the dataset lives in a different project (`BIGQUERY_PROJECT_ID`), grant the
@@ -38,12 +40,13 @@ CMEK dataset, also grant the BigQuery service account access to your KMS key.
 
 ## Usage
 
-Export the three functions from your functions codebase entry:
+Export the four functions from your functions codebase entry:
 
 ```ts
 // functions/src/index.ts
 export {
   fsexportbigquery,
+  syncBigQuery,
   initBigQuerySync,
   setupBigQuerySync,
 } from "@firebase-function-kits/firestore-bigquery-export";
@@ -59,6 +62,7 @@ DATABASE_REGION=europe-west2
 ```
 
 - `fsexportbigquery` is the Firestore trigger.
+- `syncBigQuery` is the write-buffer task queue that retries failed writes.
 - `initBigQuerySync` is the first-deploy provisioning lifecycle task.
 - `setupBigQuerySync` is the reconfigure provisioning lifecycle task.
 
@@ -87,8 +91,15 @@ later, behind the `kits` experiment):
 `instances` maps each instance id to the directory (relative to
 `firebase.json`) holding that instance's `.env`. The CLI prefixes every
 function and task queue name with `kit-<instance id>-`, so the functions above
-deploy as `kit-default-fsexportbigquery`, `kit-default-initBigQuerySync`, and
-`kit-default-setupBigQuerySync`.
+deploy as `kit-default-fsexportbigquery`, `kit-default-syncBigQuery`,
+`kit-default-initBigQuerySync`, and `kit-default-setupBigQuerySync`.
+
+Deploy with Firebase CLI 15.28.0 or later: it sets the
+`FIREBASE_KIT_INSTANCE_ID` env var on the deployed functions, which the trigger
+needs to address its own `syncBigQuery` queue. On functions deployed with an
+older CLI, enqueues fail (logged at error level and published as an `onError`
+event; the event is dropped, as in the extension) until you redeploy with a
+newer CLI.
 
 ```sh
 firebase experiments:enable kits
@@ -112,7 +123,9 @@ loads them at deploy time and prompts for any required values that are missing.
 | `datasetLocation`                | `DATASET_LOCATION`                  | no       | `us`               | BigQuery dataset location                                          |
 | `database`                       | `DATABASE`                          | no       | `(default)`        | Firestore database id                                              |
 | `bigqueryProjectId`              | `BIGQUERY_PROJECT_ID`               | no       | project id         | Dataset project, if different                                      |
-| `backupCollection`               | `BACKUP_COLLECTION`                 | no       | (empty)            | Firestore collection for failed rows                               |
+| `backupCollection`               | `BACKUP_COLLECTION`                 | no       | (empty)            | Strongly recommended: collection for rows the queue gave up on     |
+| `maxDispatchesPerSecond`         | `MAX_DISPATCHES_PER_SECOND`         | no       | `100`              | `syncBigQuery` queue dispatch rate (1-500)                         |
+| `maxEnqueueAttempts`             | `MAX_ENQUEUE_ATTEMPTS`              | no       | `3`                | In-process enqueue attempts before giving up (1-10)                |
 | `transformFunction`              | `TRANSFORM_FUNCTION`                | no       | (empty)            | Optional transform Cloud Function                                  |
 | `tablePartitioning`              | `TABLE_PARTITIONING`                | no       | `NONE`             | Table partitioning strategy                                        |
 | `timePartitioningField`          | `TIME_PARTITIONING_FIELD`           | no       | (empty)            | Time-partitioning column name                                      |
@@ -154,11 +167,11 @@ the instances cannot collide.
 
 ## Events
 
-When `EVENTARC_CHANNEL` is configured, the function publishes `onStart` and
-`onError` lifecycle events under
-`firebase.extensions.firestore-bigquery-export.v1.*`. The extension's
-`onSuccess` event is not published; see the events entry under
-"Differences from the Stream Firestore to BigQuery extension" below.
+When `EVENTARC_CHANNEL` is configured, the functions publish lifecycle events
+under `firebase.extensions.firestore-bigquery-export.v1.*`: `onStart` and
+`onError` from the write path, and `onSuccess` from the `syncBigQuery` task
+when a buffered write lands (matching the extension, which only emitted
+`onSuccess` from its queue handler).
 
 ## Provisioning
 
@@ -216,10 +229,61 @@ curl -fsS -X POST -H "Content-Type: application/json" -d '{"data":{}}' \
 ```
 
 The Firestore write path never provisions on the hot path. If resources are
-missing when a write arrives, the inline write fails, the handler calls
-`ensureInitialized()` once as a self-heal and retries the write, and a remaining
-failure is surfaced to the function runtime retry policy (`retry: true` on
-`fsexportbigquery`).
+missing when a write arrives, the inline write fails and the change buffers
+through the `syncBigQuery` queue, which re-attempts the write on Cloud Tasks'
+schedule. The queue handler does not provision, as in the extension: if the
+resources are still missing the retries fail and the row lands in
+`BACKUP_COLLECTION`; run the lifecycle task (redeploy) to recreate them.
+
+## Failure handling
+
+The write path mirrors the extension's Cloud Tasks buffer:
+
+1. The trigger attempts the BigQuery insert inline. On success, done.
+2. On failure, it enqueues the serialized change onto the `syncBigQuery` queue
+   (up to `MAX_ENQUEUE_ATTEMPTS` in-process attempts with backoff, keyed by
+   event id so a retried enqueue cannot buffer the same event twice) and the
+   execution succeeds. The trigger declares no retry policy, as in the
+   extension: a failure _before_ the write is attempted (serializing the
+   change, publishing the `onStart` event) fails the execution once and the
+   event is not redelivered.
+3. `syncBigQuery` re-attempts the write on the queue's schedule: 5 attempts,
+   60 seconds minimum backoff, throttled to `MAX_DISPATCHES_PER_SECOND`
+   dispatches per second (500 concurrent max, and the function allows 500
+   instances so that ceiling is reachable; gen2 would otherwise cap at 100).
+4. On every terminal insert failure the tracker writes the row to
+   `BACKUP_COLLECTION` (when configured), keyed by the event id, before the
+   task fails. After the fifth attempt the task is dropped. **Without a backup collection, the row is dropped with it** -
+   configure `BACKUP_COLLECTION`.
+5. If the enqueue itself fails (BigQuery AND Cloud Tasks both failing), the
+   trigger logs at error level, publishes an `onError` event, and the
+   execution succeeds: the event is dropped, exactly as the extension did in
+   this window.
+
+### Recovering parked rows
+
+Rows in `BACKUP_COLLECTION` are changelog-shaped documents, not plain document
+snapshots, so `fs-bq-import-collection` cannot consume them. Treat them as
+"possibly failed": a transient failure that later succeeded on retry also
+leaves one behind, and nothing cleans them up. To recover after an outage,
+load the backup docs' fields into a temp table and `MERGE` them into the
+changelog table with a `WHEN NOT MATCHED` condition on `event_id` (the
+anti-join is mandatory because of those stale rows). See
+[firebase/extensions#3031](https://github.com/firebase/extensions/issues/3031)
+for the full recipe.
+
+### Known limits
+
+- A task queue is a project-level resource created for each task function.
+  Deleting the functions (or moving them to another region) can leave the old
+  queue behind; delete it in the Cloud Tasks console if it lingers.
+- Rows that exhaust the queue with no `BACKUP_COLLECTION` configured are gone.
+  This matches the extension; it is the reason the backup collection is
+  strongly recommended.
+- `BACKUP_COLLECTION` captures rows whose BigQuery insert fails. A failure
+  earlier in the tracker, such as a `TRANSFORM_FUNCTION` endpoint that is down
+  or returns malformed JSON, throws before the insert and is not backed up.
+  Same as the extension.
 
 ## Differences from the Stream Firestore to BigQuery extension
 
@@ -234,23 +298,25 @@ boolean params, and only the literal string `true` enables them. The extension
 used `yes` / `no` for the last two, so copying an old config across leaves them
 silently disabled. Change any `yes` to `true` in your `.env`.
 
-### Failed writes retry differently
+### Failed writes: same buffer
 
-The extension pushed a failed BigQuery write onto a Cloud Tasks queue
-(`syncBigQuery`) and retried it from there. This kit has no task queue on the
-write path. A failed write is retried once in place, and anything still failing
-is handed to the Cloud Functions runtime retry policy, which redelivers the
-Firestore event.
+The kit keeps the extension's write-path architecture: a failed BigQuery write
+buffers through the `syncBigQuery` Cloud Tasks queue, with the same shape (5
+attempts, 60s minimum backoff, `MAX_DISPATCHES_PER_SECOND` throttling) and the
+same knobs (`MAX_DISPATCHES_PER_SECOND`, `MAX_ENQUEUE_ATTEMPTS`) - your
+migrated `.env` values carry over unchanged.
 
-The practical effects: retries no longer show up as a separate function or
-queue in the console, and the two knobs that tuned that queue,
-`MAX_DISPATCHES_PER_SECOND` and `MAX_ENQUEUE_ATTEMPTS`, no longer exist.
+When the enqueue itself fails, the kit does what the extension does: logs at
+error level, publishes an `onError` event, and drops the event. The trigger
+declares no retry policy, so nothing is redelivered through Eventarc.
 
-### Events
+Earlier release candidates of this kit had no queue: they retried every failed
+write through Eventarc redelivery for up to 24 hours and never lost a row
+inside that window. That property is gone by design - a row that exhausts the
+queue without a configured `BACKUP_COLLECTION` is dropped, exactly as in the
+extension. Set `BACKUP_COLLECTION`.
 
-`onSuccess` is no longer published. The extension emitted it from the task
-queue handler, which is gone, so the kit publishes `onStart` and `onError`
-only.
+### Events
 
 Events are published under `firebase.extensions.firestore-bigquery-export.v1.*`
 only. The extension also published a duplicate copy of every event under
@@ -321,14 +387,16 @@ BigQuery changelog table, so they still work against data this kit writes.
 ## API surface
 
 - **Main entry** (`@firebase-function-kits/firestore-bigquery-export`): exports
-  `fsexportbigquery`, `initBigQuerySync`, and `setupBigQuerySync`, and
-  registers the first-deploy / redeploy provisioning hooks. Runtime config is
-  resolved lazily on first invocation. Use this entry from Firebase
-  deploy/emulator/runtime. For your own triggers, import from `./lib` instead.
-- **Library entry** (`./lib`): `handleDocumentWrite`, the raw handler for owning
-  trigger registration yourself, plus the config types and helpers
-  (`ExportConfig`, `resolveExportConfig`, `toTrackerConfig`) for building its
-  injected `HandlerContext`. Safe to import anywhere.
+  `fsexportbigquery`, `syncBigQuery`, `initBigQuerySync`, and
+  `setupBigQuerySync`, and registers the first-deploy / redeploy provisioning
+  hooks. Runtime config is resolved lazily on first invocation. Use this entry
+  from Firebase deploy/emulator/runtime. For your own triggers, import from
+  `./lib` instead.
+- **Library entry** (`./lib`): `handleDocumentWrite` and
+  `handleSyncBigQueryTask`, the raw handlers for owning trigger registration
+  yourself, plus the config types and helpers (`ExportConfig`,
+  `resolveExportConfig`, `toTrackerConfig`, `SerializedDocumentChange`) for
+  building their injected `HandlerContext`. Safe to import anywhere.
 
 The change-tracker engine is an internal dependency and is not exported.