fix(firestore-vector-search): restore batched backfill

kits/firestore-vector-search/CHANGELOG.md

@@ -1 +1,5 @@
 - Initial release of kit, see README for differences between the legacy extension and this kit
+- Restored the extension's batched backfill: the backfill and update triggers enumerate the collection by document reference instead of loading it into memory, chunk it into 50 document ids per Cloud Task, run one task at a time, and embed each chunk in provider-sized batches with a single API call per batch. Documents are marked `BACKFILLED` or `FAILED_BACKFILL` in one batched write, and a failed batch no longer fails the task.
+- Restored the index metadata gate at `_<instance id>/index`: a backfill or update pass runs only when the embedding provider, the vector dimension or the input/output field names differ from the last recorded pass, so a redeploy that changes none of them no longer re-embeds the whole collection. Unlike the extension, the progress counters are merged into that document rather than replacing it, so the gate survives its own first pass.
+- Raised the OpenAI embedding client's batch size from 1 to the extension's 16, so a backfill chunk is embedded in one request per 16 documents.
+- With both `DO_BACKFILL` and `UPDATE_ON_CONFIGURE` set, only the backfill pass is enqueued. The two passes share one task thread on `_<instance id>/index`, and the backfill pass covers every document the update pass would.

kits/firestore-vector-search/README.md

@@ -156,9 +156,9 @@ This kit is version 0.1.3 of the extension repackaged as an npm package, and it
 the least literal of the ports. The seven functions, the Firestore vector index,
 the query document collection and the callable all survive with their names and
 settings intact, so a `.env` copied from your installed instance needs no value
-changes. The embedding providers, the backfill, and the shape of the status field
-written onto your documents all changed, so read this before you point the kit at
-a collection an installed instance has already embedded.
+changes. The embedding providers and the shape of the status field written onto
+your documents both changed, so read this before you point the kit at a
+collection an installed instance has already embedded.
 
 ### `EMBEDDING_PROVIDER: multimodal` is not implemented
 
@@ -182,8 +182,9 @@ delete the old vector index first if it was created with a different dimension.
 ### You set `INSTANCE_ID` yourself, and it names the query collection
 
 The extension derived its instance id at install and used it for the query
-collection (`_<instance id>/index/queries`), the index metadata document
-(`_<instance id>/index`) and its task queues. Here `INSTANCE_ID` is a setting you
+collection (`_<instance id>/index/queries`), the index metadata and backfill
+progress document (`_<instance id>/index`, with its `enqueues` subcollection)
+and its task queues. Here `INSTANCE_ID` is a setting you
 provide, and it must match this instance's key in the `instances` map in
 `firebase.json`. To keep serving the query documents your clients already write
 to, set it to your installed instance's id. The four task queue names can also be
@@ -201,35 +202,40 @@ whatever `EMBEDDING_PROVIDER` is set to. If either does not exist, `firebase
 deploy` prompts you for a value and fails outright when running
 non-interactively (CI). Create the one you do not need with a placeholder value.
 
-### `UPDATE_ON_CONFIGURE` now re-embeds on every deploy
-
-This setting was declared by the extension but never read. Reconfiguring an
-installed instance re-embedded documents only when the provider, the vector
-dimension or the input/output field names had actually changed, which the
-extension tracked in its index metadata document.
-
-The kit keeps no such metadata and does no comparison. `UPDATE_ON_CONFIGURE: true`
-enqueues a full re-embed of every document that already has an embedding after
-*every* `firebase deploy`, whether anything relevant changed or not, and
-`DO_BACKFILL: true` embeds the whole collection after the first deploy. On a large
-collection that is a large Vertex AI or OpenAI bill per deploy. Set
-`UPDATE_ON_CONFIGURE: false` and re-embed deliberately when you change providers.
-
-### Backfill is one task per document, and reads the collection in one go
-
-The extension chunked the collection into batches sized to the provider (16
-documents per OpenAI call), embedded each batch in a single API call, and tracked
-progress in its metadata document. The kit reads the entire collection with one
-`get()` and enqueues one Cloud Task per document, each of which embeds one
-document with one API call.
-
-Two consequences. A collection large enough that a single `get()` does not fit in
-the trigger's 512 MiB will fail the backfill outright, and there is no
-resume-from-progress. Backfilling *n* documents now costs *n* task invocations and
-*n* embedding calls rather than *n*/batch size.
-
-There is also no install-time progress reporting, since there is no extension
-install UI to report into. Watch the function logs instead.
+### `UPDATE_ON_CONFIGURE` is read, and the backfill gate is stricter than the extension's
+
+This setting was declared by the extension but never read: its update pass was
+gated on `DO_BACKFILL` instead. The kit reads `UPDATE_ON_CONFIGURE`, so the two
+passes are controlled independently — `DO_BACKFILL` after the first deploy,
+`UPDATE_ON_CONFIGURE` after every redeploy.
+
+Both passes are then gated on the index metadata document at
+`_<instance id>/index`, as the extension's were: a pass runs only when the
+embedding provider, the vector dimension or the input/output field names differ
+from what the last pass recorded there. Redeploying without changing any of them
+enqueues nothing and costs nothing.
+
+The two passes share that document as their task thread, so only one of them
+runs per deploy: with both settings on, the backfill pass runs, which covers
+every document the update pass would have (the update pass is the same
+eligibility rule plus "and already has an embedding").
+
+The extension's gate did not survive its own first pass, because the progress
+counters it wrote to the same document replaced the recorded configuration. The
+kit merges instead, so the comparison fields persist and the gate holds on every
+later deploy. To force a full re-embed without changing any setting, delete the
+`_<instance id>/index` document; its `queries` subcollection is untouched, so
+the query documents your clients write to survive.
+
+### There is no install-time progress reporting
+
+The extension reported backfill progress and failures through the extension
+install UI (`setProcessingState`). There is no such surface for a kit, so
+progress is visible in the function logs and in the progress fields on
+`_<instance id>/index` (`backfillJobsTotal`, `backfillJobsProcessed`,
+`backfillJobsSkipped`, `backfillJobsFailed`, `backfillStatus`) instead. One
+document per chunk is written under `_<instance id>/index/enqueues`, as the
+extension did, each carrying its chunk of document ids and its own status.
 
 ### The `status` field on your documents is a different shape
 
@@ -246,10 +252,12 @@ status: { state: "COMPLETED" }
 status: { state: "ERROR", message: "<error message>" }
 ```
 
-The states themselves are narrower too: `PROCESSING` and `BACKFILLED` are no
-longer written, only `COMPLETED` and `ERROR`. Anything reading
-`status.<instance id>.state`, or a security rule or index keyed to it, needs
-updating. The field name is still `STATUS_FIELD_NAME`, defaulting to `status`.
+The states themselves are narrower too: `PROCESSING` is no longer written. The
+write triggers write `COMPLETED` or `ERROR`, and the backfill and update passes
+write `BACKFILLED` or `FAILED_BACKFILL` alongside a `completeTime`, as the
+extension did. Anything reading `status.<instance id>.state`, or a security rule
+or index keyed to it, needs updating. The field name is still
+`STATUS_FIELD_NAME`, defaulting to `status`.
 
 Query documents no longer get a status field at all. They previously carried
 `status.textQuery`, so if you were waiting on that to know a query had finished,
@@ -302,6 +310,13 @@ for; the Firebase CLI grants these for you.
 
 ### Unchanged
 
+- The backfill and update passes still enumerate the collection by document
+  reference, chunk it into 50 document ids per Cloud Task, run one task at a
+  time, and embed each chunk in provider-sized batches with a single API call per
+  batch (16 documents per OpenAI call). A document whose input is not a string is
+  skipped, as is one whose status is already set to anything other than
+  `BACKFILLED`. A failed batch marks its documents `FAILED_BACKFILL` and the task
+  still succeeds.
 - The indexed collection is still `COLLECTION_NAME` (default `products`), the
   input, output and status fields still default to `input`, `embedding` and
   `status`, and embeddings are still written as native Firestore vectors.