InvokeModelWithBidirectionalStreamAsync() can block the calling thread forever when the peer never sends any response - no timeout at any layer

Summary

BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync() is documented and named as an asynchronous method, but part of the initial event-stream write work runs synchronously on the calling thread, inside the call itself, with no timeout anywhere in the chain down to raw HTTP/2 flow control. If the server accepts the connection/stream but never sends any response (and therefore never sends a WINDOW_UPDATE frame either), the calling thread can hang forever inside the "async" call, not merely leave a callback unfired.

Environment

aws-sdk-cpp version: 1.11.890 (built with the CRT HTTP client, AWS_SDK_USE_CRT_HTTP / push-based WriteData path)
Service: Bedrock Runtime, InvokeModelWithBidirectionalStreamAsync (used for Nova Sonic real-time voice streaming)
Platform: Windows, MSVC, x86/Win32
Underlying CRT: aws-crt-cpp (commit pinned by aws-sdk-cpp 1.11.890) / aws-c-http (commit pinned by that aws-crt-cpp)

Repro

Stand up a local HTTP/2 (TLS, self-signed cert is fine) server that:
- Accepts the TCP/TLS/HTTP2 connection and the incoming stream normally.
- Never sends any WINDOW_UPDATE frame beyond the connection/stream defaults, and never sends any response (no headers, no data).
Call BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync() against it (dummy credentials are fine - the server never needs to validate auth), with a streamReadyHandler that sends a normal small sequence of bidirectional-streaming events (e.g. a Nova Sonic sessionStart/promptStart/contentStart/textInput/contentEnd/promptEnd/sessionEnd sequence) and then calls Close() on the input stream.
Place a diagnostic statement immediately after the call to InvokeModelWithBidirectionalStreamAsync() in your own code.

Expected: the call returns promptly (it's documented/named as async), and only the response-received/outcome callback is left waiting - ideally itself bounded by some timeout, or at least clearly the caller's own responsibility to bound since it's callback-based.

Actual: the diagnostic statement never executes. The calling thread is parked indefinitely (confirmed via process inspection: near-zero CPU, threads alive, no progress even after 9+ minutes in one run) inside the call itself.

Root cause (traced through the actual 1.11.890 source)

generated/src/aws-cpp-sdk-bedrock-runtime/source/BedrockRuntimeClient.cpp (~L401-427), the CRT/push-based branch of InvokeModelWithBidirectionalStreamAsync():
```
auto asyncTask = smithy::client::CreateSmithyBidirectionalWriteDataTask<InvokeModelWithBidirectionalStreamOutcome>(
    this, requestCopy, handler, handlerContext, eventEncoderStream, writeDataStreamBuf, std::move(endpointCallback),
    std::move(authCallback));
auto sem = asyncTask.GetSemaphore();
m_clientConfiguration.executor->Submit(std::move(asyncTask));
sem->WaitOne();
streamReadyHandler(*eventEncoderStream);
```
sem->WaitOne() is unconditional (no timeout). Once it returns (i.e. once the background task's HttpWriteDataStreamBuf::Initialize() succeeds), streamReadyHandler - the caller-supplied write callback - is invoked directly on the calling thread, as the last statement of this method. There is nothing else in the function body after this call; the method returns only once streamReadyHandler returns.
Inside streamReadyHandler, calling Close() on the event-stream input (Model::InvokeModelWithBidirectionalStreamInput → inherited Aws::Utils::Event::EventEncoderStream::Close()) is a pure passthrough down to HttpWriteDataStreamBuf::Close() → SendBuffer(endStream=true). The same SendBuffer() is used for every write (overflow(), xsputn(), sync(), and Close()).

src/aws-cpp-sdk-core/source/utils/stream/HttpWriteDataStreamBuf.cpp (~L148-181), SendBuffer():

m_stream->WriteData(data, [this](int errorCode) -> void {
    std::unique_lock<std::mutex> const lock{m_writeMutex};
    m_writeInProgress = false;
    m_writeError = (errorCode != AWS_ERROR_SUCCESS);
    m_writeComplete.notify_one();
}, endStream);

std::unique_lock<std::mutex> lock{m_writeMutex};
m_writeComplete.wait(lock, [this]() -> bool { return !m_writeInProgress; });

This wait has no timeout parameter and no timed overload anywhere in the class.

The completion callback only fires once aws-c-http's aws_h2_stream_encode_data_frame() (source/h2_stream.c, ~L829-846) actually encodes the write into an outgoing DATA frame:
```
if (stream->thread_data.window_size_peer <= AWS_H2_MIN_WINDOW_SIZE) {
    /* The stream is stalled now */
    *data_encode_status = AWS_H2_DATA_ENCODE_ONGOING_WINDOW_STALLED;
    return AWS_OP_SUCCESS;
}
```
If the stream is window-stalled, this returns having done nothing - no error, no retry scheduling, no deadline. The queued write just sits there.
window_size_peer is this side's outbound HTTP/2 flow-control credit as granted by the peer - seeded from the RFC 7540 §6.5.2 default (65535 bytes) at connection/stream setup (source/h2_connection.c ~L387-388, source/h2_stream.c ~L741-745), and only ever replenished by a WINDOW_UPDATE frame from the peer. A peer that never sends anything never sends one either. Once cumulative outstanding bytes across the whole stream exceed that window, every subsequent write - including, in this repro, the one inside Close() - stalls forever.
The one native mechanism that exists specifically for "peer never responds" is response_first_byte_timeout_ms (aws-c-http's aws_http_connection_manager_options / aws_http_make_request_options), but per its own doc comment it is explicitly HTTP/1.1-only:
```
/**
 * ...
 * TODO: Only supported in HTTP/1.1 now, support it in HTTP/2
 */
uint64_t response_first_byte_timeout_ms;
```
confirmed by grepping the whole aws-c-http tree: it's referenced only in h1_connection.c/h1_stream.c, never in h2_connection.c/h2_stream.c. It is also not exposed anywhere through aws-crt-cpp's or aws-sdk-cpp's C++ wrapper types, even for HTTP/1.1.
Separately, ClientConfiguration::requestTimeoutMs exists and is checked in the SDK's synchronous MakeRequest() response wait (src/aws-cpp-sdk-core/source/http/crt/CRTHttpClient.cpp ~L537-559), but is never referenced anywhere in the connection-acquisition (AcquireConnection, ~L721-745) or bidirectional-streaming write path - so it provides no protection here even when configured.

Impact

For a caller of InvokeModelWithBidirectionalStreamAsync(), there is currently no supported way, at any layer, to bound how long the call itself (not just the eventual response callback) can block against a peer that accepts the stream but never communicates further. This is worse than "the outcome callback never fires" - the calling application's own thread is the one that hangs, which for an application built expecting an async, callback-driven API (e.g. a call-handling engine dispatching this from a request-processing thread) can freeze application logic entirely, not just leave a background task dangling.

Suggested areas to address (not prescribing the fix, just where the gaps are)

Expose response_first_byte_timeout_ms (or an HTTP/2-appropriate equivalent) through aws-crt-cpp and aws-sdk-cpp's ClientConfiguration, and implement it for HTTP/2 in aws-c-http (currently HTTP/1.1-only by an existing TODO).
At minimum, give HttpWriteDataStreamBuf::SendBuffer()'s write-completion wait a timeout path, so a stalled write surfaces as an error to the caller instead of blocking forever.
Consider whether streamReadyHandler needs to run synchronously on the calling thread inside InvokeModelWithBidirectionalStreamAsync() at all - if the intent is a genuinely async method, dispatching this initial write burst through the executor (like the rest of the task) rather than back on the caller's thread would preserve the documented async contract even without a new timeout feature.

Related, but distinct issues (searched, not duplicates)

Possible regression introduced in 1.11.801 (2nd attempt) #3917 / Possible regression introduced in 1.11.801 #3911 - Aws::ShutdownAPI() hang after a stream has already completed (self-reference cycle pinning a connection) - already fixed in PR fix/bidirectional stream shutdown #3919 (1.11.890). This report is a different bug: the hang happens during the initial call, before any response, triggered by HTTP/2 flow control against a non-responsive peer, not by a resource cycle at shutdown.
Bedrock Runtime Nova Sonic bidirectional streaming on EC2 instance - no output event stream #3649, Stream error is not returned #3650 - connectivity/error-visibility issues, not hangs of this kind.

[Claude Code]

(Edited 2026-09-14 to fix the code sample below - see the correction comment further down the thread. The original version of this comment had the "never returns" marker on the wrong line.)

Follow-up with a concrete, minimal repro (stub server + C++ client), so this doesn't require guessing at API shapes. Re-verified against a fresh standalone build/run - InvokeModelWithBidirectionalStreamAsync() itself returns normally; the hang is in destructing/releasing the client afterward (see below and the later correction comment for the exact mechanism).

Stub server (Node.js `http2`, generic - accepts a stream, then stays silent forever)

const http2 = require('http2');
const fs = require('fs');

const server = http2.createSecureServer({
  key: fs.readFileSync('key.pem'),
  cert: fs.readFileSync('cert.pem'),
});

server.on('stream', (stream, headers) => {
  console.log(`stream received: ${headers[':method']} ${headers[':path']} - staying silent`);
  // Deliberately no respond()/write()/end() - connection and stream stay open forever.
});

server.listen(8443, () => console.log('listening on https://localhost:8443'));

(pair with a self-signed cert.pem/key.pem, CN=localhost)

Minimal C++ client

// Minimal repro: destructing/releasing a BedrockRuntimeClient can block the calling thread
// forever when a bidirectional stream is still incomplete (peer accepted the stream but never
// responded, so it never sent a WINDOW_UPDATE either). InvokeModelWithBidirectionalStreamAsync()
// itself returns normally - the hang is one step later, at client.reset() below.
//
// Requires the stub server above listening on localhost:8443. Dummy credentials are fine -
// the stub never checks auth.
//
// Define AWS_NO_STATIC_IMPL / USE_IMPORT_EXPORT / USE_WINDOWS_DLL_SEMANTICS only if your
// aws-sdk-cpp/aws-crt-cpp build is shared (DLL); omit for a static-lib build.

#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentials.h>
#include <aws/core/utils/json/JsonSerializer.h>
#include <aws/bedrock-runtime/BedrockRuntimeClient.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamRequest.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamHandler.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamInput.h>
#include <aws/bedrock-runtime/model/BidirectionalInputPayloadPart.h>
#include <iostream>

static void SendEvent(Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput,
                       const Aws::String& json)
{
    std::cout << ">> " << json << std::endl;
    Aws::BedrockRuntime::Model::BidirectionalInputPayloadPart part;
    part.SetBytes(Aws::Utils::CryptoBuffer(reinterpret_cast<const unsigned char*>(json.c_str()), json.size()));
    streamInput.WriteBidirectionalInputPayloadPart(part);
}

int main()
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    {
        Aws::Auth::AWSCredentials credentials("dummyAccessKeyId", "dummySecretAccessKey");

        Aws::Client::ClientConfigurationInitValues initValues{ /*shouldDisableIMDS:*/ true };
        Aws::Client::ClientConfiguration clientConfiguration(initValues);
        clientConfiguration.region = "us-east-1";
        clientConfiguration.endpointOverride = "localhost:8443";
        clientConfiguration.verifySSL = false;      // self-signed stub cert
        clientConfiguration.connectTimeoutMs = 5000;
        clientConfiguration.requestTimeoutMs = 0;

        auto client = Aws::MakeShared<Aws::BedrockRuntime::BedrockRuntimeClient>(
            "Repro", credentials, clientConfiguration);

        Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamHandler streamHandler;
        streamHandler.SetInitialResponseCallbackEx(
            [](const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInitialResponse& response,
               Aws::Utils::Event::InitialResponseType) {
                std::cout << "<< InitialResponse: " << response.Jsonize().View().WriteCompact() << std::endl;
            });
        streamHandler.SetOnErrorCallback(
            [](const Aws::Client::AWSError<Aws::BedrockRuntime::BedrockRuntimeErrors>& error) {
                std::cout << "<< StreamError: " << error.GetMessage() << std::endl;
            });

        Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest streamingRequest;
        streamingRequest.SetEventStreamHandler(streamHandler);
        streamingRequest.SetModelId("amazon.nova-2-sonic-v1:0");

        client->InvokeModelWithBidirectionalStreamAsync(
            streamingRequest,
            [](Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput) {
                Aws::Utils::Json::JsonValue inferenceConfiguration;
                inferenceConfiguration.WithInteger("maxTokens", 1024).WithDouble("topP", 0.9).WithDouble("temperature", 0.7);
                Aws::Utils::Json::JsonValue sessionStart;
                sessionStart.WithObject("inferenceConfiguration", inferenceConfiguration);
                Aws::Utils::Json::JsonValue event;
                event.WithObject("sessionStart", sessionStart);
                Aws::Utils::Json::JsonValue wrapper;
                wrapper.WithObject("event", event);
                SendEvent(streamInput, wrapper.View().WriteCompact());

                // ... send promptStart / contentStart / textInput / contentEnd / promptEnd /
                //     sessionEnd the same way (omitted here for brevity - shape doesn't matter,
                //     the bug doesn't depend on how much of a real conversation happens) ...

                streamInput.Close();
                std::cout << "Stream closed." << std::endl;
            },
            [](const Aws::BedrockRuntime::BedrockRuntimeClient*,
               const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest&,
               const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamOutcome& outcome,
               const std::shared_ptr<const Aws::Client::AsyncCallerContext>&) {
                std::cout << "Outcome: " << (outcome.IsSuccess() ? "SUCCESS" : outcome.GetError().GetMessage()) << std::endl;
            },
            nullptr);

        // <<< THIS LINE RUNS FINE - the async call returns normally >>>
        std::cout << "InvokeModelWithBidirectionalStreamAsync() returned." << std::endl;

        // <<< THIS LINE NEVER RETURNS when the bug reproduces >>>
        // Destroying the client tears down its DefaultExecutor, which waits (unbounded, no
        // timeout) for the write task submitted above to finish - and that task is stuck
        // forever waiting for HTTP/2 flow control the stub never grants.
        client.reset();
        std::cout << "client destroyed." << std::endl;
    }
    Aws::ShutdownAPI(options);
    return 0;
}

[Claude Code]

Correction to the original report and the follow-up repro comment above.

I re-tested the exact minimal repro I posted, but as a clean standalone main() program (not wrapped in a test framework), with a diagnostic print staged after every step. Result: the call to InvokeModelWithBidirectionalStreamAsync() returns normally. The "streamReadyHandler runs synchronously and the method never returns until it's done" description in the original report is technically accurate about how streamReadyHandler is invoked, but wrong about the practical consequence - the method itself does return.

The real, observable hang is one step later: destructing/releasing the client (client.reset(), or the shared_ptr going out of scope, or process shutdown) while the bidirectional stream is still incomplete. I isolated this precisely - a print immediately before client.reset() fires, a print immediately after it never does. Traced the exact mechanism in the 1.11.890 source:

~BedrockRuntimeClient() (generated/src/aws-cpp-sdk-bedrock-runtime/source/BedrockRuntimeClient.cpp:179) calls ShutdownSdkClient(this, -1).
ClientWithAsyncTemplateMethods<BedrockRuntimeClient>::ShutdownSdkClient (src/aws-cpp-sdk-core/include/aws/core/client/AWSClientAsyncCRTP.h:97-131) does a bounded wait on m_operationsProcessed==0, then at line 128 calls pClient->m_clientConfiguration.executor.reset();.
That bounded wait is a no-op here: the push-based bidirectional path (BedrockRuntimeClient.cpp:401-427, the same one from the original report) never increments m_operationsProcessed - unlike the sibling pull-based path in AWSClientBidirectionalStreaming.h:112. So step 2's timeout provides no protection for this code path even when requestTimeoutMs is set.
m_clientConfiguration.executor.reset() drops the last reference to the same DefaultExecutor instance that the original write task was submitted to (AwsSmithyClient.h:323 binds m_clientConfiguration by reference to the shared config object - confirmed identical instance, not a lookalike).
DefaultExecutor::~DefaultExecutor() (src/aws-cpp-sdk-core/source/utils/threading/DefaultExecutor.cpp:174-177) calls WaitUntilStopped().
WaitUntilStopped() (DefaultExecutor.cpp:132-157, line 149): m_cv.wait(lock, [this](){ return m_tasks.empty(); }); - unbounded, no timeout. A task removes itself from m_tasks only after it returns (via Detach(), lines 120-130).
The submitted task never returns - it's SmithyBidirectionalStreamingWriteDataTask::operator(), stuck inside HttpWriteDataStreamBuf::WaitForStreamComplete() (HttpWriteDataStreamBuf.cpp:176), which is the same unbounded HTTP/2 send-window stall already described in the original report. So m_tasks never empties, and the CV wait in step 6 blocks forever - taking down whatever thread calls client.reset()/destructs the client with it.

Net effect on the report: the underlying root cause (HTTP/2 flow-control window never replenished by a non-responding peer, no timeout anywhere in that wait) is unchanged and still the trigger. What's corrected is where it surfaces to a caller: not inside InvokeModelWithBidirectionalStreamAsync() itself, but inside destructing/releasing the client afterward (which is extremely common - RAII, shared_ptr scope exit, graceful shutdown). Practically this is at least as bad: application shutdown/cleanup code, not just the original call site, can hang forever.

One clarification I haven't fully verified: whether AwsSmithyClientBase::DisableRequestProcessing() (called from step 2, AWSClientAsyncCRTP.h:111) does anything blocking inside the CRT HTTP client - it looks flag-only from its call site but I didn't trace into that implementation. Doesn't change the conclusion above since step 6 is independently sufficient and unbounded.

Apologies for the churn - wanted to correct this precisely rather than leave the wrong call site on record.

@oleg-boulanov

@oleg-boulanov thanks for the precise trace, especially the correction pinning it to teardown. Status:

First-byte timeout (HTTP/2): implemented - the option was HTTP/1.1-only, now works for HTTP/2. PRs are open (not yet merged): adding ResponseFirstByteTimeout config awslabs/aws-c-http#586 and adding ResponseFirstByteTimeout config awslabs/aws-crt-cpp#908.
In the repro the stream now fails fast with AWS_ERROR_HTTP_RESPONSE_FIRST_BYTE_TIMEOUT instead of stalling.
Operation tracking: you were right that the push-based path never registers with m_operationsProcessed like the pull-based path does. I've wired it up (same RAIICounter), and client destruction now completes cleanly where it used to hang.
Not fully resolved yet: with both changes the failure moves into Aws::ShutdownAPI()/CRT cleanup and is nondeterministic (hang or crash). It looks like the async push path leaves CRT stream/connection references alive past SDK teardown, racing global CRT shutdown, so the counter fix is necessary but not sufficient. Still investigating that piece before I put up the aws-sdk-cpp PR.

Hey. I've opened up a PR #3929 which will re-introduce a configurable timeout which is going to apply for a time to first byte, and each individual write. Tested with your repro on that branch and looks like the behavior is correct. Although, setting a timeout of 0 (opting out), you can expect an indefinite hang with your repro since it's an explicit opt out of timeouts. Will close this out once that PR is merged

Thank you!

Fix is merged and shipped. Let us know if you run into any other issues.

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

#3929
This branch has not been deployed

Can you please provide the version number this fix is deployed in?

1.11.899

Navigation Menu

Repository navigation

InvokeModelWithBidirectionalStreamAsync() can block the calling thread forever when the peer never sends any response - no timeout at any layer #3923

Summary

Environment

Repro

Root cause (traced through the actual 1.11.890 source)

Impact

Suggested areas to address (not prescribing the fix, just where the gaps are)

Related, but distinct issues (searched, not duplicates)

Stub server (Node.js `http2`, generic - accepts a stream, then stays silent forever)

Minimal C++ client

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

InvokeModelWithBidirectionalStreamAsync() can block the calling thread forever when the peer never sends any response - no timeout at any layer #3923

Description

Summary

Environment

Repro

Root cause (traced through the actual 1.11.890 source)

Impact

Suggested areas to address (not prescribing the fix, just where the gaps are)

Related, but distinct issues (searched, not duplicates)

Activity

oleg-boulanov commented on Sep 14, 2026

Stub server (Node.js http2, generic - accepts a stream, then stays silent forever)

Minimal C++ client

oleg-boulanov commented on Sep 14, 2026

kai-ion commented on Sep 15, 2026

sbaluja commented on Sep 17, 2026

oleg-boulanov commented on Sep 17, 2026

sbaluja commented on Sep 21, 2026

github-actions commented on Sep 21, 2026

oleg-boulanov commented on Sep 22, 2026

oleg-boulanov commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Stub server (Node.js `http2`, generic - accepts a stream, then stays silent forever)