Repository navigation
Possible regression introduced in 1.11.801 (2nd attempt) #3917
Description
Describe the bug
In #3809, description states that "This is fully backwards compatible. No code changes required."
This may be not entirely correct. For example, after using Bedrock's InvokeModelWithBidirectionalStreamAsync(), Aws::ShutdownAPI(options) lock indefinitely.
Regression confirmed by building and using both 1.11.800 and 1.11.801 version with the same test code.
#include "pch.h"
// EZA-28656: standalone repro of Aws::ShutdownAPI() hanging forever after a Bedrock Runtime
// InvokeModelWithBidirectionalStream call, for reporting to aws/aws-sdk-cpp.
//
// This file is intentionally independent of every Eliza:: type - no SonicEngine, no AwsSdk
// wrapper, no InputEvent helpers. It uses only the AWS SDK for C++ (aws-cpp-sdk-core,
// aws-cpp-sdk-bedrock-runtime) plus the C++ standard library, so it can be lifted verbatim into
// a plain main() or a fresh aws-sdk-cpp sample project.
//
// Root cause (confirmed by live WinDbg inspection of the hung process, see EZA-28656): the
// generated Aws::BedrockRuntime::BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync
// installs a signing callback on the event-encoder stream that captures the stream's own
// shared_ptr by value - a self-reference cycle. Since aws-sdk-cpp PR #3809 ("use write data API
// in bidirectional streaming", first shipped in v1.11.801), that stream's backing buffer
// (HttpWriteDataStreamBuf) holds a live HTTP/2 Connection/ClientStream for as long as the cycle
// keeps it alive - so the cycle now pins a real connection forever instead of an inert buffer.
// Aws::ShutdownAPI() -> Aws::CleanupCrt() then blocks forever waiting for the CRT's native
// event-loop-group to report fully stopped, because that pinned connection never releases it.
//
// This repro deliberately does NOT work around the bug (unlike this repo's own SonicEngine,
// which calls streamInput.SetSigningCallback(nullptr) as a mitigation) - the whole point is to
// show the raw upstream behavior.
//
// Aws::InitAPI()/Aws::ShutdownAPI() must run on the SAME OS thread - Aws.h says so explicitly:
// "Please call this from the same thread from which InitAPI() has been called (use a dedicated
// thread if necessary)." So both calls run on one dedicated thread here (started before, and
// joined after, the Bedrock call), and that thread is given a bounded 5 seconds to finish before
// this test reports the outcome and returns - so this test is safe to run on its own, with no
// external timeout/process-kill needed, even though the bug it demonstrates is a genuine
// unbounded native hang.
//
// How to run (isolated, since it exercises real AWS SDK global init/shutdown state):
// Eliza.IVR.Sonic.ClientLib.Tests.exe --gtest_filter=Eza28656ShutdownHangRepro.ShutdownHangsAfterBidirectionalStream
// If it reproduces, this test FAILS with "Aws::ShutdownAPI() did not complete within 5000ms" -
// that failure IS the demonstration of the bug. Its dedicated thread is left running in the
// background (there's no way to safely abandon a thread that's stuck inside a native SDK call).
//
// Requires: valid AWS credentials in the default profile (~/.aws/credentials) with
// bedrock:InvokeModelWithBidirectionalStream permission for the model below, in the given region.
// This NuGet package builds aws-sdk-cpp/aws-crt-cpp as DLLs, not static libs - consumers must
// define these before including any AWS header (normally via that package's own
// eliza/aws-sdk-cpp-config.h, deliberately NOT included here to keep this file free of anything
// under an eliza/ path). Without AWS_NO_STATIC_IMPL, aws-c-common's "static inline" helper
// functions get compiled directly into this translation unit instead of imported from the DLL -
// and one of those (aws/common/math.h) picks its x64-only MSVC intrinics header
// (math.msvc_x64.inl, using _umul128/_addcarry_u64/_BitScanReverse64) even for this x86 build, an
// unrelated packaging bug that only surfaces when the code is compiled inline like this.
#define AWS_NO_STATIC_IMPL
#define USE_IMPORT_EXPORT
#define USE_WINDOWS_DLL_SEMANTICS
#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/core/utils/json/JsonSerializer.h>
#include <aws/bedrock-runtime/BedrockRuntimeClient.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamRequest.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamHandler.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamInput.h>
#include <aws/bedrock-runtime/model/BidirectionalInputPayloadPart.h>
#include
#include
#include <condition_variable>
#include
#include
#include
#include
#include
namespace {
// A random UUID isn't needed here - Nova Sonic just needs promptName/contentName to be unique
// within this one session. (Aws::Utils::UUID is deliberately avoided: it pulls in an aws-c-common
// header, math.msvc_x64.inl, that turned out to be broken for this project's x86 build - a
// separate, unrelated packaging issue not worth fighting for a minimal repro.)
Aws::String MakeId(const char* prefix)
{
static std::atomic counter{ 0 };
return Aws::String(prefix) + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count())
+ "-" + std::to_string(++counter);
}
// Owns the one dedicated thread that Aws::InitAPI()/Aws::ShutdownAPI() both run on.
struct ApiThread {
std::thread thread;
std::promise shutdownRequested;
};
std::unique_ptr StartApiThread(Aws::SDKOptions options)
{
auto state = std::make_unique();
std::promise initDone;
std::future initDoneFuture = initDone.get_future();
std::shared_future shutdownRequestedFuture = state->shutdownRequested.get_future().share();
state->thread = std::thread(options, initDone = std::move(initDone), shutdownRequestedFuture mutable {
Aws::InitAPI(options);
initDone.set_value();
shutdownRequestedFuture.wait();
Aws::ShutdownAPI(options);
});
initDoneFuture.wait();
return state;
}
// Signals the dedicated thread to proceed to Aws::ShutdownAPI(), then waits up to timeout for
// it to finish. The actual join happens on a THIRD thread - joining a std::thread has no
// thread-affinity requirement, unlike the AWS calls it's waiting on - so that this function itself
// can safely time out without violating the same-thread rule it exists to respect. Returns true if
// ShutdownAPI() completed in time; false if it's still stuck (its thread is then abandoned,
// running in the background - there's no safe way to cancel a thread blocked inside a native SDK
// call).
bool StopApiThreadWithTimeout(std::unique_ptr state, std::chrono::milliseconds timeout)
{
state->shutdownRequested.set_value();
std::thread apiThread = std::move(state->thread);
auto done = std::make_shared<std::promise>();
std::future doneFuture = done->get_future();
std::thread(apiThread = std::move(apiThread), done mutable {
apiThread.join();
done->set_value();
}).detach();
return doneFuture.wait_for(timeout) == std::future_status::ready;
}
// Builds one Nova Sonic input event as compact JSON: {"event":{: }}.
Aws::String WrapEvent(const Aws::String& eventName, Aws::Utils::Json::JsonValue&& body)
{
Aws::Utils::Json::JsonValue event;
event.WithObject(eventName, std::move(body));
Aws::Utils::Json::JsonValue wrapper;
wrapper.WithObject("event", std::move(event));
return wrapper.View().WriteCompact();
}
void SendEvent(
Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput,
const Aws::String& json)
{
std::cout << ">> " << json << std::endl;
Aws::BedrockRuntime::Model::BidirectionalInputPayloadPart part;
part.SetBytes(Aws::Utils::CryptoBuffer(reinterpret_cast<const unsigned char*>(json.c_str()), json.size()));
streamInput.WriteBidirectionalInputPayloadPart(part);
}
} // namespace
TEST(Eza28656ShutdownHangRepro, ShutdownHangsAfterBidirectionalStream)
{
Aws::SDKOptions options;
auto apiThread = StartApiThread(options);
{
Aws::Client::ClientConfigurationInitValues initValues{ /*shouldDisableIMDS:*/ true };
Aws::Client::ClientConfiguration clientConfiguration(initValues);
clientConfiguration.region = "us-east-1";
clientConfiguration.requestTimeoutMs = 30000;
Aws::Auth::ProfileConfigFileAWSCredentialsProvider credentialsProvider("default");
Aws::Auth::AWSCredentials credentials = credentialsProvider.GetAWSCredentials();
ASSERT_FALSE(credentials.IsEmpty()) << "No AWS credentials found in the default profile";
auto pClient = Aws::MakeShared<Aws::BedrockRuntime::BedrockRuntimeClient>(
"Eza28656Repro", credentials, clientConfiguration);
const Aws::String promptName = MakeId("prompt-");
const Aws::String contentName = MakeId("content-");
std::mutex doneMutex;
std::condition_variable doneCv;
bool outcomeReceived = false;
Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamHandler streamHandler;
streamHandler.SetInitialResponseCallbackEx(
[](const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInitialResponse& response,
Aws::Utils::Event::InitialResponseType) {
std::cout << "<< InitialResponse: " << response.Jsonize().View().WriteCompact() << std::endl;
});
streamHandler.SetOnErrorCallback(
[](const Aws::Client::AWSError<Aws::BedrockRuntime::BedrockRuntimeErrors>& error) {
std::cout << "<< StreamError: " << error.GetMessage() << std::endl;
});
streamHandler.SetBidirectionalOutputPayloadPartCallback(
[](const Aws::BedrockRuntime::Model::BidirectionalOutputPayloadPart& part) {
const auto& bytes = part.GetBytes();
std::cout << "<< " << Aws::String(reinterpret_cast<const char*>(bytes.GetUnderlyingData()), bytes.GetLength()) << std::endl;
});
Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest streamingRequest;
streamingRequest.SetEventStreamHandler(streamHandler);
streamingRequest.SetModelId("amazon.nova-2-sonic-v1:0");
pClient->InvokeModelWithBidirectionalStreamAsync(
streamingRequest,
[promptName, contentName](Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput) {
// Minimal session: start, one system-prompt turn, then close - no audio content
// needed. The bug is structural (a shared_ptr cycle created as soon as the stream
// exists), not dependent on how much of a real conversation happens.
Aws::Utils::Json::JsonValue inferenceConfiguration;
inferenceConfiguration.WithInteger("maxTokens", 1024).WithDouble("topP", 0.9).WithDouble("temperature", 0.7);
Aws::Utils::Json::JsonValue sessionStart;
sessionStart.WithObject("inferenceConfiguration", inferenceConfiguration);
SendEvent(streamInput, WrapEvent("sessionStart", std::move(sessionStart)));
Aws::Utils::Json::JsonValue textOutputConfiguration;
textOutputConfiguration.WithString("mediaType", "text/plain");
Aws::Utils::Json::JsonValue promptStart;
promptStart.WithString("promptName", promptName).WithObject("textOutputConfiguration", textOutputConfiguration);
SendEvent(streamInput, WrapEvent("promptStart", std::move(promptStart)));
Aws::Utils::Json::JsonValue textInputConfiguration;
textInputConfiguration.WithString("mediaType", "text/plain");
Aws::Utils::Json::JsonValue contentStart;
contentStart.WithString("promptName", promptName)
.WithString("contentName", contentName)
.WithString("type", "TEXT")
.WithString("role", "SYSTEM")
.WithObject("textInputConfiguration", textInputConfiguration);
SendEvent(streamInput, WrapEvent("contentStart", std::move(contentStart)));
Aws::Utils::Json::JsonValue textInput;
textInput.WithString("promptName", promptName).WithString("contentName", contentName).WithString("content", "Keep responses short.");
SendEvent(streamInput, WrapEvent("textInput", std::move(textInput)));
Aws::Utils::Json::JsonValue contentEnd;
contentEnd.WithString("promptName", promptName).WithString("contentName", contentName);
SendEvent(streamInput, WrapEvent("contentEnd", std::move(contentEnd)));
Aws::Utils::Json::JsonValue promptEnd;
promptEnd.WithString("promptName", promptName);
SendEvent(streamInput, WrapEvent("promptEnd", std::move(promptEnd)));
SendEvent(streamInput, WrapEvent("sessionEnd", Aws::Utils::Json::JsonValue()));
streamInput.Close();
// Deliberately NOT calling streamInput.SetSigningCallback(nullptr) here - that is
// this repo's own workaround (see SonicEngine::OnStreamReady), and the whole point
// of this file is to reproduce the raw, un-worked-around upstream bug.
std::cout << "Stream closed." << std::endl;
},
[&doneMutex, &doneCv, &outcomeReceived](
const Aws::BedrockRuntime::BedrockRuntimeClient*,
const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest&,
const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamOutcome& outcome,
const std::shared_ptr<const Aws::Client::AsyncCallerContext>&) {
std::cout << "Final outcome: " << (outcome.IsSuccess() ? Aws::String("SUCCESS") : outcome.GetError().GetMessage()) << std::endl;
{
std::lock_guard<std::mutex> lock(doneMutex);
outcomeReceived = true;
}
doneCv.notify_all();
},
nullptr);
{
std::unique_lock<std::mutex> lock(doneMutex);
ASSERT_TRUE(doneCv.wait_for(lock, std::chrono::seconds(30), [&] { return outcomeReceived; }))
<< "InvokeModelWithBidirectionalStreamAsync did not complete within 30s - this is a "
<< "different failure than EZA-28656 (which happens AFTER a successful call)";
}
std::cout << "pClient.use_count() before reset: " << pClient.use_count() << std::endl;
pClient.reset();
} // options/credentials/etc. go out of scope here; only apiThread itself outlives this block
// This is the call under test. If EZA-28656 reproduces, Aws::CleanupCrt() blocks forever
// inside Aws::ShutdownAPI(), waiting for the CRT event-loop group to report fully stopped -
// but this wait is bounded, so this test reports that outcome instead of hanging the process.
std::cout << "Requesting Aws::ShutdownAPI()..." << std::endl;
auto t0 = std::chrono::steady_clock::now();
bool completed = StopApiThreadWithTimeout(std::move(apiThread), std::chrono::seconds(5));
auto elapsedMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - t0).count();
if (completed) {
std::cout << "Aws::ShutdownAPI() returned after " << elapsedMs << "ms." << std::endl;
}
else {
std::cout << "Aws::ShutdownAPI() did NOT complete within 5000ms - its dedicated thread is "
<< "left running in the background." << std::endl;
}
EXPECT_TRUE(completed) << "Aws::ShutdownAPI() did not complete within 5000ms - EZA-28656 reproduced";
}
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
Aws::ShutdownAPI() exits cleanly after limited amount of time
Current Behavior
Aws::ShutdownAPI() hangs
Reproduction Steps
Build AWS SDK:
"C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" ...... -G "Visual Studio 18 2026" -A Win32 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX="........\install" -DBUILD_ONLY="bedrock-runtime" -DTARGET_ARCH=Windows -DBUILD_SHARED_LIBS=ON -DUSE_OPENSSL=OFF -DUSE_CRT_HTTP_CLIENT=ON
Build and run attached unit test using 1.11.800 and 1.11.801 ASW SDK versions
Possible Solution
No response
Additional Information/Context
No response
AWS CPP SDK version used
= 1.11.801
Compiler and Version used
Visual Studio 2026
Operating System and version
Windows 11
Activity
sbiscigl commentedon Sep 3, 2026
from the last ticket:
did you try picking up a version post that patch and seeing if that resolves your issue? that specifically dealt with internal issue titled
which sounds very similar to your case, sepcifically in you RCA
so i would suggest seeing if the issue is fixed on
mainbefore further debuggingoleg-boulanov commentedon Sep 3, 2026
oleg-boulanov commentedon Sep 3, 2026
sbiscigl commentedon Sep 3, 2026
small update, we were able to reproduce and are working on a fix we have that fix staged on a branch, will let you know when we get it out eddb695
sbiscigl commentedon Sep 8, 2026
just small update where we are at with the change, the changes in the PR do fix the hang issue. when testing out the fix we discovered a completely unrelated bug that we are also rolling into this fix that has to do with low speed limit of connections. where we will start honoring the configured low speed limit of requests as to destroy stalled connections. thanks for bearing with us throughout this and we should be able to finalize a change for fixing both issues soon.
related CRT PR: awslabs/aws-crt-cpp#905
oleg-boulanov commentedon Sep 8, 2026
sbiscigl commentedon Sep 10, 2026
the fix should be merged on main/be released as
1.11.890today. you should see a large CPU usage improvement too. give it a shot and let me know what you think.oleg-boulanov commentedon Sep 11, 2026
github-actions commentedon Sep 14, 2026
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.