Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
/*
* Copyright 2025 The gRPC Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package io.grpc.xds.internal.extauthz;

import com.google.common.collect.ImmutableList;
import io.envoyproxy.envoy.service.auth.v3.CheckResponse;
import io.grpc.CallOptions;
import io.grpc.Channel;
import io.grpc.ClientCall;
import io.grpc.Context;
import io.grpc.Metadata;
import io.grpc.MethodDescriptor;
import io.grpc.Status;
import io.grpc.internal.DelayedClientCall;
import io.grpc.stub.StreamObserver;
import io.grpc.xds.internal.grpcservice.HeaderValue;
import io.grpc.xds.internal.headermutations.HeaderMutations;
import io.grpc.xds.internal.headermutations.HeaderMutator;
import io.grpc.xds.internal.headermutations.HeaderValueOption;
import java.util.concurrent.Executor;

/**
* A {@link StreamObserver} that handles the response from the external authorization service,
* transitioning the delayed gRPC call based on the authorization decision.
*/
final class AuthzCallbackObserver<ReqT, RespT> implements StreamObserver<CheckResponse> {

private static final Metadata.Key<String> X_ENVOY_AUTH_FAILURE_MODE_ALLOWED =
Metadata.Key.of("x-envoy-auth-failure-mode-allowed", Metadata.ASCII_STRING_MARSHALLER);

private final DelayedClientCall<ReqT, RespT> delayedCall;
private final Channel next;
private final MethodDescriptor<ReqT, RespT> method;
private final CallOptions callOptions;
private final Executor callExecutor;
private final CheckResponseHandler responseHandler;
private final HeaderMutator headerMutator;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to float the HeaderMutator all the way from ExtAuthzFilter. It could just be created here. In fact we should make it static seeing that it holds no state (can be done separately).

private final ExtAuthzConfig config;
private final Context.CancellableContext authzContext;

AuthzCallbackObserver(
DelayedClientCall<ReqT, RespT> delayedCall,
Channel next,
MethodDescriptor<ReqT, RespT> method,
CallOptions callOptions,
Executor callExecutor,
CheckResponseHandler responseHandler,
HeaderMutator headerMutator,
ExtAuthzConfig config,
Context.CancellableContext authzContext) {
this.delayedCall = delayedCall;
this.next = next;
this.method = method;
this.callOptions = callOptions;
this.callExecutor = callExecutor;
this.responseHandler = responseHandler;
this.headerMutator = headerMutator;
this.config = config;
this.authzContext = authzContext;
}

@Override
public void onNext(CheckResponse value) {
// Note on exception safety: handleResponse() internally catches

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uncaught exception from onNext causes the stub to cancel the stream and route directly into onError(), which closes authzContext. Its closure does not rely on GC of the stream observer, and it cannot, because it has a direct reference to it from ExtAuthzClientCall.

In a unary rpc, onNext() delivers an inbound response payload; it does not signify that the RPC is complete. A unary RPC completes only when onClose / onCompleted() / onError() is invoked.

The comment needs to be corrected on both the above fronts.

Is relying on this mechanism "Valid Code"?

Mechanically, yes, gRPC will catch the exception and route to onError(). If an unhandled RuntimeException escapes onNext(), ClientCallImpl cancels the stream and delivers onError(), which drains delayedCall and closes authzContext.

However, intentionally relying on this as a design pattern is fragile and carries real risks:

A. Security Risk: Accidentally Failing Open on a DENY

This is the biggest risk:

// in onNext():
AuthzResponse authzResponse = responseHandler.handleResponse(value);
if (authzResponse.decision() == AuthzResponse.Decision.ALLOW) { ... }
else {
  // Suppose decision is DENY, but building FailingCallWithTrailerMutations throws an exception
  ...
}

If the external authorization server returned an explicit DENY response, but an unexpected RuntimeException is thrown while preparing the rejection:

  1. The exception escapes onNext().
  2. The stub cancels the stream and invokes onError(t).
  3. In onError():
    if (config.failureModeAllow()) {
      // FAILS OPEN: creates next.newCall() and forwards request to backend!
    }

If failure_mode_allow: true is configured (which is only intended for network/server outages reaching the authz service), a request that the authz server explicitly denied would be forwarded to the backend!

B. The "Throwing to Trigger Another Callback on Yourself" Anti-Pattern

Relying on throwing an unhandled exception out of onNext() so that the underlying framework cancels the transport stream and calls back into your own onError() is an indirect, surprising control flow:

  • It generates unnecessary transport RST_STREAM frames and Failed to read message error logs from ClientCallImpl exception handling.
  • It makes debugging difficult because the root cause in onNext gets swallowed into a transport Status.CANCELLED exception passed to onError.

C. Partial Failure State

If an exception occurs after delayedCall.setCall(call) was executed (for example, if callExecutor.execute(drain) throws a RejectedExecutionException):

  • When onError() subsequently runs, its call to delayedCall.setCall(...) returns null because realCall != null was already set. The failure call cannot be attached, leaving the call in an inconsistent state.

// HeaderMutationDisallowedException and converts it to a DENY response.
// Remaining calls (newCall, setCallAndDrain) use battle-tested
// infrastructure with protobuf-guaranteed non-null defaults. An
// uncaught RuntimeException here would propagate to the gRPC stub
// framework, which handles observer stream cleanup. The authzContext
// would remain uncancelled, but will be GC'd when this observer is
// collected — acceptable since the authz RPC would have already
// completed (we're inside onNext).
AuthzResponse authzResponse = responseHandler.handleResponse(value);
if (authzResponse.decision() == AuthzResponse.Decision.ALLOW) {
ClientCall<ReqT, RespT> delegate = next.newCall(method, callOptions);
ClientCall<ReqT, RespT> finalCall;
if (isHeaderMutationsEmpty(authzResponse.requestHeaderMutations())
&& isHeaderMutationsEmpty(authzResponse.responseHeaderMutations())) {
finalCall = delegate;
} else {
finalCall = new MutatingClientCall<>(
delegate,
authzResponse.requestHeaderMutations(),
authzResponse.responseHeaderMutations(),
headerMutator);
}
setCallAndDrain(finalCall);
} else {
// Defensive programming: status is always present on DENY branches since every
// deny() factory method requires a Status parameter. This orElseThrow satisfies
// static analysis and documents the invariant.
Status status = authzResponse.status().orElseThrow(
() -> new IllegalArgumentException("DENY response missing status"));
ClientCall<ReqT, RespT> failingCall;
if (isHeaderMutationsEmpty(authzResponse.responseHeaderMutations())) {
failingCall = new FailingClientCall<>(status);
} else {
failingCall = new FailingCallWithTrailerMutations<>(
status,
authzResponse.responseHeaderMutations(),
headerMutator);
}
setCallAndDrain(failingCall);
}
}

@Override
public void onError(Throwable t) {
if (config.failureModeAllow()) {
ClientCall<ReqT, RespT> delegate = next.newCall(method, callOptions);
ClientCall<ReqT, RespT> finalCall;
if (config.failureModeAllowHeaderAdd()) {
HeaderMutations requestMutations = HeaderMutations.create(
ImmutableList.of(HeaderValueOption.create(
HeaderValue.create(X_ENVOY_AUTH_FAILURE_MODE_ALLOWED.name(), "true"),
HeaderValueOption.HeaderAppendAction.APPEND_IF_EXISTS_OR_ADD)),
ImmutableList.of());
// TODO(sauravz): Consider using a static EMPTY_MUTATIONS constant
// to avoid repeated allocation of empty HeaderMutations.
HeaderMutations responseMutations =
HeaderMutations.create(ImmutableList.of(), ImmutableList.of());
finalCall = new MutatingClientCall<>(
delegate, requestMutations, responseMutations, headerMutator);
} else {
finalCall = delegate;
}
setCallAndDrain(finalCall);
authzContext.close();
} else {
Status statusToReturn = config.statusOnError().withCause(t);
setCallAndDrain(new FailingClientCall<>(statusToReturn));
authzContext.close();
}
}

@Override
public void onCompleted() {
// Defensive: if the authz server completed without sending a CheckResponse
// (buggy server), resolve the DelayedClientCall with a failure and close
// the authzContext. If onNext() already called setCall(), this is a no-op
// since DelayedClientCall.setCall() ignores subsequent calls.
Status status = config.statusOnError()
.withDescription("Authz server completed without sending CheckResponse");
setCallAndDrain(new FailingClientCall<>(status));
authzContext.close();
}

private void setCallAndDrain(ClientCall<ReqT, RespT> call) {
Runnable drain = delayedCall.setCall(call);
if (drain != null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this check (and the corresponding unit test allow_whenDelayedCallNotStarted_setCallReturnsNull) considering that the only code using this class is ExtAuthzClientCall that always starts the DelayedClientCall before invoking the rpc on the ext_auth stub?

callExecutor.execute(drain);
}
}

private static boolean isHeaderMutationsEmpty(HeaderMutations mutations) {
return mutations.headers().isEmpty() && mutations.headersToRemove().isEmpty();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
/*
* Copyright 2025 The gRPC Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package io.grpc.xds.internal.extauthz;

import com.google.common.annotations.VisibleForTesting;
import com.google.protobuf.util.Timestamps;
import io.envoyproxy.envoy.service.auth.v3.AuthorizationGrpc;
import io.envoyproxy.envoy.service.auth.v3.CheckRequest;
import io.grpc.CallOptions;
import io.grpc.Channel;
import io.grpc.ClientCall;
import io.grpc.Context;
import io.grpc.Deadline;
import io.grpc.ForwardingClientCall;
import io.grpc.Metadata;
import io.grpc.MethodDescriptor;
import io.grpc.internal.DelayedClientCall;
import io.grpc.xds.internal.headermutations.HeaderMutator;
import java.util.concurrent.Executor;
import java.util.concurrent.ScheduledExecutorService;
import javax.annotation.Nullable;

/**
* A {@link ForwardingClientCall} that delegates to a lightweight {@link DelayedClientCall}
* to buffer RPC requests safely while waiting for the asynchronous authorization decision.
*/
public final class ExtAuthzClientCall<ReqT, RespT> extends ForwardingClientCall<ReqT, RespT> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtAuthzClientInterceptor is actaully not added by this PR. Edit the PR title.


private final Channel next;
private final MethodDescriptor<ReqT, RespT> method;
private final CallOptions callOptions;
private final AuthorizationGrpc.AuthorizationStub authzStub;
private final CheckRequestBuilder checkRequestBuilder;
private final CheckResponseHandler responseHandler;
private final HeaderMutator headerMutator;
private final ExtAuthzConfig config;
private final Executor callExecutor;
private final Context.CancellableContext authzContext;

private final DelayedAuthzCall<ReqT, RespT> delegate;

public ExtAuthzClientCall(
Executor executor,
ScheduledExecutorService scheduler,
CallOptions callOptions,
Channel next,
MethodDescriptor<ReqT, RespT> method,
AuthorizationGrpc.AuthorizationStub authzStub,
CheckRequestBuilder checkRequestBuilder,
CheckResponseHandler responseHandler,
HeaderMutator headerMutator,
ExtAuthzConfig config) {
this.callOptions = callOptions;
this.next = next;
this.method = method;
this.authzStub = authzStub;
this.checkRequestBuilder = checkRequestBuilder;
this.responseHandler = responseHandler;
this.headerMutator = headerMutator;
this.config = config;
this.callExecutor = executor;
this.authzContext = Context.current().withCancellation();
this.delegate = new DelayedAuthzCall<>(executor, scheduler, callOptions.getDeadline());
}

@Override
protected ClientCall<ReqT, RespT> delegate() {
return delegate;
}

@Override
public void start(Listener<RespT> responseListener, Metadata headers) {
// 1. Construct the check request FIRST before handoff (respects metadata thread safety)
CheckRequest request = checkRequestBuilder.buildRequest(method, headers,
Timestamps.fromMillis(System.currentTimeMillis()));

// 2. Start the delayed call (buffers headers and outbound payloads)
super.start(responseListener, headers);

// 3. Trigger the async authorization check under the cancellable context
authzContext.run(() -> {
authzStub.check(request, new AuthzCallbackObserver<>(
delegate, next, method, callOptions, callExecutor, responseHandler, headerMutator,
config, authzContext));
});
}

@Override
public void cancel(
@Nullable String message, @Nullable Throwable cause) {
authzContext.cancel(cause);
super.cancel(message, cause);
}

/**
* Returns the cancellable context used for the authorization check.
* Visible for testing.
*/
@VisibleForTesting
Context.CancellableContext getAuthzContextForTest() {
return authzContext;
}

/**
* A lightweight package-private DelayedClientCall subclass to expose
* the protected constructor.
*/
private static final class DelayedAuthzCall<ReqT, RespT> extends DelayedClientCall<ReqT, RespT> {
DelayedAuthzCall(Executor executor, ScheduledExecutorService scheduler, Deadline deadline) {
super("ExtAuthzClientCall", executor, scheduler, deadline);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
/*
* Copyright 2025 The gRPC Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package io.grpc.xds.internal.extauthz;

import io.grpc.ClientCall;
import io.grpc.Metadata;
import io.grpc.Status;
import io.grpc.xds.internal.headermutations.HeaderMutations;
import io.grpc.xds.internal.headermutations.HeaderMutator;

/**
* A simple failing client call that lazily applies response mutations to trailers during start.
*/
final class FailingCallWithTrailerMutations<ReqT, RespT> extends ClientCall<ReqT, RespT> {
private final Status status;
private final HeaderMutations responseHeaderMutations;
private final HeaderMutator headerMutator;

FailingCallWithTrailerMutations(
Status status,
HeaderMutations responseHeaderMutations,
HeaderMutator headerMutator) {
this.status = status;
this.responseHeaderMutations = responseHeaderMutations;
this.headerMutator = headerMutator;
}

@Override
public void start(Listener<RespT> responseListener, Metadata headers) {
// Lazily allocate and apply response mutations to trailers copy on start!
Metadata trailers = new Metadata();
headerMutator.applyMutations(responseHeaderMutations, trailers);
responseListener.onClose(status, trailers);
}

@Override
public void request(int numMessages) {}

@Override
public void cancel(String message, Throwable cause) {}

@Override
public void halfClose() {}

@Override
public void sendMessage(ReqT message) {}
}
Loading
Loading