Server performs six executor submissions per unary RPC; dispatch count is the executor-independent cost floor

[bpalermo]

What

On master, a unary server RPC performs six submissions to the per-stream
SerializingExecutor:

1. MethodLookup — ServerImpl.streamCreatedInternal
2. HandleServerCall — queued back-to-back with the above
3. OnReady
4. MessagesAvailable
5. HalfClosed (where the application handler actually runs, for unary)
6. Closed

After SerializingExecutor's CAS coalescing this typically works out to 2–3 real
cross-thread handoffs per RPC, plus the hop back to the event loop for WriteQueue's
coalesced flush.

This is related to, but distinct from, #2118: the LinkedBlockingQueue contention
profiled there is one symptom of the underlying constant — the number of submissions
per RPC — which no choice of underlying executor eliminates. With a
virtual-thread-per-task executor (a reasonable modern default, supported per #11726)
there is no shared LinkedBlockingQueue at all, and the cost is still there, now as
per-hop unpark/wakeup latency and scheduler pressure rather than queue contention.
Fixing the executor doesn't fix this; the dispatch count is structural.

Measurements

Setup: arm64 Kubernetes cluster, 1-CPU/1-Gi Guaranteed QoS pods (one server pod per
4-core worker), tiny unary echo, grpc-java 1.83.1 on non-shaded Netty 4.2.16 (dispatch
shape verified unchanged on master). Two workloads: a fixed-rate grid at k6
constant-arrival 200 req/s per arm (the table below), and saturation ramps (200→2400 rps
offered in 2-minute steps; later tables). For the fixed-rate grid, each
configuration was run both on HotSpot JDK 21 and as a GraalVM CE 21.0.2 native image —
the twin results isolate the effect to gRPC's dispatch mechanics rather than JIT/runtime
behavior. An untouched REST control arm ran in the same cluster during every window to
rule out environmental drift. Multiple 25-minute windows per configuration.

Server executor	p50	p99	CPU (JVM)	CPU (native)
default (virtual-thread-per-task)	~3.1 ms	65–72 ms	~220 m	~390 m
directExecutor()	~2.9 ms (−5%)	115–137 ms (2×)	145 m (−35%)	260 m (−35%)

Two observations worth separating:

1. The dispatch machinery costs ~0.4–0.65 ms CPU per RPC (the delta between the two
rows, attributable to the serialized submissions above — an echo handler does essentially
nothing else). An echo workload maximally inflates the relative overhead, which is why
we're not leading with percentages — but the absolute per-RPC machinery cost is
executor-independent and survives into real workloads as capacity loss. Saturation ramps
(200→2400 rps offered in 2-minute steps, same pods) put numbers on that:

Arm (per 1-CPU pod)	plateau goodput	knee
grpc-netty JVM, directExecutor()	~2,140 rps	~2,200
grpc-netty JVM, VT executor	~2,060 rps	~2,000
grpc-netty native-image, VT executor	~1,550–1,600 rps	~1,200

Both gRPC arms degraded gracefully at saturation (HTTP/2 flow-control backpressure, zero
restarts) — for context, an HTTP/1.1 REST arm in the same cluster collapsed outright past
its ~960 rps plateau.

2. directExecutor() halves CPU but, at low load, doubles p99. The mechanism is
run-to-completion
convoying with deferred-flush amplification: the whole pipeline runs in the event loop's
IO phase, but WriteQueue's flush is scheduled "onto the tail of the event loop"
(WriteQueue.later) — a task, which runs only after the IO phase completes. In a burst
of N ready requests, even the first response is flushed only after all N are handled, so
any stall's cost is structurally amplified. Notably, WriteQueue already has an
immediate-drain escape hatch — drainNow() — but it is only invoked on the client
(NettyClientHandler, GOAWAY processing). The server has no equivalent; a server-side
drain at the end of the read batch (channelReadComplete) bounds the amplification.

Importantly, the executor trade-off is load-dependent. The 2× p99 penalty above is
at ~10% utilization. As offered load rises, the picture inverts — p99 (ms) vs offered
(rps) on the JVM arms, [VT / direct / direct+drainNow-prototype]:

offered	VT	direct	direct + drainNow
400	12.8	9.6	9.6
800	44.5	38.3	32.4
1200	86.8	86.6	66.6
1600	238.7	222.5	222.0
1800	324.9	289.3	230.7
2000	661.8	381.9	391.6
2200	972.5	649.7	498.1
2400	2069.8	1993.9	717.1

At low load, VT dispatch protects tails (bursts are spread across carrier threads); above
~75% utilization, directExecutor() wins both goodput and tails — the per-dispatch
overhead is what costs the VT arm its capacity and its high-load tails (scheduler
pressure). Users are currently forced to pick which regime to sacrifice.

On-cluster replication of the drainNow prototype

The third column above is a rebuild of the direct arm with only grpc-netty's classes
replaced by the drain-at-channelReadComplete prototype (same node, same jar otherwise,
matched offered load, zero restarts in both runs). p99 deltas vs. stock direct: −23% at
1200, −20% at 1800, −23% at 2200, and −64% at 2400 (1994 ms → 717 ms); roughly equal
at 1600/2000; goodput unchanged (2,137 vs 2,116 rps peak). The benefit concentrates
exactly where the convoy mechanism predicts: the deep-saturation regime where read
batches are largest.

Caveats: single run per configuration; first-step p99 samples on JVM arms include JIT
warmup; the native-image arm's two restarts during its own run were liveness-probe
starvation at saturation (an operational artifact, not a gRPC issue).

Streaming vs. unary: the per-call machinery priced end-to-end

A follow-up measurement isolates how much of the above is per-call overhead. Same
1-CPU JVM pods, persistent bidi echo streams (20 or 40), open-loop driver with
absolute scheduling and bounded in-flight, ramps to 16,000 msg/s aggregate:

• The pod that plateaus at ~2,060–2,140 unary calls/s sustains ~15,000–16,000 streamed
  msg/s — ~7.5× — on identical hardware. On a long-lived stream the per-call
  submissions (items 1, 2, 6 above, and OnReady) amortize away across messages, so
  this ratio is an end-to-end price of the per-call dispatch machinery.
• What still separates executors on streams is the per-message MessagesAvailable
  submission, and it only becomes visible near saturation — p99 (ms), VT vs.
  directExecutor(): 6k: 27/15 · 10k: 91/50 · 12k: 165/67 · 14k: 406/165 ·
  16k: 1350/741. At 16k offered, the VT arm sheds 625 msg/s (deferred) against direct's
  38 while direct delivers 15,962 msg/s. Below ~5k msg/s the executors are
  indistinguishable on streams.

Same caveats (single run per configuration, JIT warmup steps excluded). Raw per-step
tables for all runs:
unary capacity (runs 1–5),
streaming (S1/S2/E1/E2);
narrative summary
here.

Prior art

• Excess contention in ExecutorService #2118 (2016, open): profiled the shared-queue contention symptom; stalled on
  ForkJoinPool having no safe default sizing (and a Java 6 floor that no longer exists).
• Change Default Thread Executor of the Server #7381: changing the default executor buys little and is hard to do safely — taken as
  given here; none of the proposals below touch executor defaults.
• Add the ability to have a separate thread pool/executor for each endpoint #7874 / ServerCallExecutorSupplier: any change must compose with the mid-flight
  SerializingExecutor.setExecutor() switch (the proposals below do).
• PR core: don't wrap sequential executors in ClientCallImpl and ServerImpl #5521 (abandoned, 2019): skipping the SerializingExecutor wrap for
  already-sequential executors showed 5–10% in TransportBenchmark; complementary to
  the proposals below.

Proposals (increasing ambition)

1. Merge MethodLookup + HandleServerCall into a single submission when no
   ServerCallExecutorSupplier is set. The two are queued back-to-back
   unconditionally; the split exists only so MethodLookup can call
   SerializingExecutor.setExecutor() before HandleServerCall is picked up. Without a
   supplier no switch can occur, so this is a pure removal of one dispatch per RPC with
   no ordering-contract change. We have a working prototype (core, inprocess, and netty
   test suites pass unmodified). Initial TransportBenchmark.unaryCall1024Latency
   numbers (NETTY, 32 threads, single fork — treat as directional): p50 −4.2%,
   mean −3.3% with directExecutor(); p50 −2.9%, mean −2.0% with the default executor.

2. Server-side WriteQueue.drainNow() at end of read batch (netty transport), to
   bound the deferred-flush amplification for direct/inline executors. Client precedent
   exists; prototype passes the netty suite. Evidence: in JMH (NETTY, 32 threads,
   directExecutor) the effect concentrates in the tail — p99 −3.0%, p99.9 −11.8%, median
   unchanged — and the on-cluster replication above shows −20–64% p99 in the saturation
   regime with goodput unchanged.

3. Coalesce MessagesAvailable + HalfClosed for the unary END_STREAM case — either
   a combined listener delivery from the transport or an enqueue-without-schedule + flush
   API on SerializingExecutor. This touches the internal transport SPI and all
   transports, so we'd want maintainer buy-in on direction before writing code. The
   streaming measurements above isolate the per-message submission cost this would
   address. Happy to sketch alternatives if there's interest.

We can rerun any benchmark shape on our cluster on request, and are prepared to submit
PRs for (1) and (2) if the direction sounds right.

[kannanjgithub]

On (2), we're not all that interested in optimizing further direct executors unless it is truly trivial; very few people should use direct executors.