Skip to content

feat(bigquery-jdbc): add aggregation logic - #14202

Open
Neenu1995 wants to merge 4 commits into
jdbc-telemetry-pr8-implfrom
jdbc-telemetry-pr8-impl-aggregator
Open

feat(bigquery-jdbc): add aggregation logic#14202
Neenu1995 wants to merge 4 commits into
jdbc-telemetry-pr8-implfrom
jdbc-telemetry-pr8-impl-aggregator

Conversation

@Neenu1995

@Neenu1995 Neenu1995 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor
  • In-Memory Metric Aggregation (TelemetryBatcher):

    • Added raw event queue (LinkedBlockingQueue) with dedicated in-memory aggregation maps using composite keys (ConnectionAttemptKey, StatementExecutionKey, ErrorMetricKey, FeatureUsageKey).
    • Aggregates event frequencies and latency histograms (sum, total count, and bucket distributions) directly upon ingestion, significantly reducing memory footprint and object allocations.
    • Implemented atomic map swapping on flush with merge-back capabilities (requeue) on transport failures.
  • Streamlined Recording Pipeline (TelemetryManager):

    • Updated recordConnectionAttempt, recordStatementExecution, and recordFeatureUsage to pass raw parameters directly to TelemetryBatcher without allocating intermediate Protobuf messages.
    • Added calculateBucketIndex to compute latency histogram bucket indexes directly during statement recording.

@Neenu1995
Neenu1995 requested review from a team as code owners August 27, 2026 19:20

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the telemetry batching mechanism in TelemetryBatcher and TelemetryManager to use lock-free, in-memory accumulators (ConcurrentHashMap and LongAdder) instead of a blocking queue of Protobuf messages. This optimization eliminates object allocation and GC overhead during telemetry recording. The reviewer identified a potential issue in the refactored reschedule method, which is missing an isClosed guard, potentially leading to a RejectedExecutionException or scheduling tasks on a closed batcher during shutdown.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant