Skip to content

feat(bigtable): Make write batch size and concurrency configurable - #6847

Merged
ntkathole merged 3 commits into
feast-dev:masterfrom
Reactor11:feat/bigtable-configurable-write-batching
Sep 21, 2026
Merged

ntkathole merged 3 commits into
feast-dev:masterfrom
Reactor11:feat/bigtable-configurable-write-batching

Conversation

@Reactor11

Copy link
Copy Markdown
Contributor

What this PR does / why we need it

The Bigtable online store hardcodes two values that govern its write path:

  • MUTATIONS_PER_OP = 50_000 — target mutations per MutateRows request
  • BIGTABLE_CLIENT_CONNECTION_POOL_SIZE = 10 — the ThreadPoolExecutor size used to parallelize writes in online_write_batch

On a shared Bigtable instance this is a problem: a large materialization fans its writes out across the thread pool with no way to tune the request size or concurrency, issuing an unthrottled write burst that can saturate the instance and inflate read-path tail latency for other workloads sharing it. Today the only way to soften that burst is to fork the online store.

This PR exposes both as optional BigtableOnlineStoreConfig fields:

Field Default Controls
mutations_per_write 50000 mutations per Bigtable write request
write_concurrency 10 concurrent write requests (thread-pool size) per worker

Both default to the existing module constants, so behavior is unchanged unless explicitly configured. Operators running against a shared instance can now lower either value to reduce the write load a materialization places on Bigtable, trading materialization speed for lower peak write pressure. Both are validated as PositiveInt, and online_write_batch now floors rows-per-request at 1 so a very wide feature view combined with a small mutations_per_write can't produce a zero-sized batch.

Example:

online_store:
  type: bigtable
  instance: my-instance
  mutations_per_write: 10000
  write_concurrency: 2

Which issue(s) this PR fixes

N/A — backward-compatible enhancement.

Misc

  • Adds sdk/python/tests/unit/infra/online_store/test_bigtable_online_store.py covering defaults matching the legacy constants, positive-int validation, batch chunking, the wide-feature-view floor, and the configurable thread-pool size.

The Bigtable online store hardcodes the mutations-per-write batch size
(MUTATIONS_PER_OP = 50_000) and the write thread-pool size
(BIGTABLE_CLIENT_CONNECTION_POOL_SIZE = 10). On a shared Bigtable
instance, a large materialization issues its writes as an unthrottled
burst that can saturate the instance and inflate read-path tail latency
for other workloads sharing it.

Expose both as optional BigtableOnlineStoreConfig fields,
mutations_per_write and write_concurrency, defaulting to the existing
constants so behavior is unchanged. Operators can lower either to reduce
the write load a materialization places on the instance, at the cost of
longer materialization time.

Add unit tests covering the defaults, positive-int validation, batch
chunking, the one-row-per-request floor for very wide feature views, and
the configurable thread-pool size.

Signed-off-by: Manas Bhardwaj <manas1109bhardwaj@gmail.com>
@Reactor11
Reactor11 requested a review from a team as a code owner September 18, 2026 08:40
@Reactor11

Copy link
Copy Markdown
Contributor Author

Hi @franciscojavierarceo please review this.

At our org, we want to control the number of writes and concurrency. As of now it is hard coded.

@shuchu shuchu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ntkathole ntkathole changed the title feat(bigtable): make write batch size and concurrency configurable feat(bigtable): Make write batch size and concurrency configurable Sep 21, 2026
@codecov-commenter

codecov-commenter commented Sep 21, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.59%. Comparing base (f8883d0) to head (d2db6c6).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6847      +/-   ##
==========================================
+ Coverage   47.48%   47.59%   +0.10%     
==========================================
  Files         422      422              
  Lines       52324    52328       +4     
  Branches     7591     7591              
==========================================
+ Hits        24847    24906      +59     
+ Misses      25703    25646      -57     
- Partials     1774     1776       +2     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.93% <100.00%> (+0.11%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/infra/online_stores/bigtable.py 41.37% <100.00%> (+41.37%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f8883d0...d2db6c6. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Reactor11

Copy link
Copy Markdown
Contributor Author

Hi @ntkathole - please help in merging this.

@ntkathole
ntkathole merged commit 5388522 into feast-dev:master Sep 21, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants