From c029dcbc7c148a990ed058c6352f970ac2bb034b Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 13:23:41 +0500 Subject: [PATCH 01/13] Search diagnostics: fail fast outside the VPC, defuse the VPC-environment step Field feedback: an operator read Step 4's "create a VPC environment" as provisioning real infrastructure and skipped it, then ran the Step 5 script in a standard CloudShell, where it hung indefinitely with no diagnostic. Reassure on what Step 4 actually creates, make the script fail fast with a message pointing back to Step 4, add a CLI security-group lookup and a no-S3 copy-out fallback, and key Troubleshooting on the real error strings. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 32 ++++++++++++++++----- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 232de2f..9bce989 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -88,12 +88,19 @@ In a standard CloudShell you can download the files directly: **Actions → Down ## Step 4 — Open a shell inside the stack VPC -The domain only accepts connections from members of one security group in your Quilt deployment. Find it in the EC2 console by its description — *"For resources that need access to search cluster"* — or by name: depending on how your deployment was created it contains `search-accessor` or `SearchClusterAccessorSecurityGroup`. +The domain only accepts connections from members of one security group in your Quilt deployment. Find it in the EC2 console by its description — *"For resources that need access to search cluster"* — or by name: depending on how your deployment was created it contains `search-accessor` or `SearchClusterAccessorSecurityGroup`. Or from any shell (`` is the `vpc` value from Step 2): -In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Before creating, know: at most **two** VPC environments per user (delete one if you're at the limit), and network settings are fixed at creation — delete and recreate to change them. Fill in: +```bash +aws ec2 describe-security-groups --region \ + --filters Name=vpc-id,Values= \ + --query "SecurityGroups[?contains(Description, 'access to search cluster')].[GroupId,GroupName]" \ + --output text +``` + +In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Despite the "create", this is not new infrastructure — not a VPC endpoint, not an instance: a VPC environment is an ordinary CloudShell session whose network interface sits inside your VPC. It changes nothing in your Quilt deployment, costs nothing, and deleting it (see Cleanup) removes everything it made. Before creating, know: at most **two** VPC environments per user (delete one if you're at the limit), and network settings are fixed at creation — delete and recreate to change them. Fill in: - **VPC**: the `vpc` value from Step 2. -- **Subnet**: any subnet reaches the domain, but Step 6 needs an S3 route from it — an S3 gateway endpoint in the subnet's route table, NAT, or your usual egress path (e.g. Transit Gateway). In Quilt-created VPCs every subnet typically qualifies. +- **Subnet**: any subnet reaches the domain, but Step 6 copies the files out via S3, which needs an S3 route — an S3 gateway endpoint in the subnet's route table, NAT, or your usual egress path (e.g. Transit Gateway). This is about *picking* a subnet whose route already exists, not creating one: in Quilt-created VPCs every subnet typically qualifies, and if none does, Step 6 has a fallback that needs no S3 at all. - **Security group**: the one from above. Provisioning takes a minute or two; you're ready when the new environment opens with a shell prompt. @@ -106,6 +113,8 @@ Fill in the two placeholders — `` is the `endpoint` value from S ```bash python3 << 'EOF' +# Run this ONLY in the CloudShell VPC environment from Step 4 — +# from any other shell the domain is unreachable and every request fails. import boto3, urllib3 from botocore.auth import SigV4Auth from botocore.awsrequest import AWSRequest @@ -120,12 +129,19 @@ FILES = { "settings.json": "/_all/_settings?flat_settings=true", } creds = boto3.Session().get_credentials().get_frozen_credentials() -http = urllib3.PoolManager() +http = urllib3.PoolManager(timeout=urllib3.Timeout(connect=10, read=None), retries=False) for fname, path in FILES.items(): url = "https://" + HOST + path req = AWSRequest(method="GET", url=url) SigV4Auth(creds, "es", REGION).add_auth(req) - r = http.request("GET", url, headers=dict(req.headers)) + try: + r = http.request("GET", url, headers=dict(req.headers)) + except urllib3.exceptions.HTTPError as e: + raise SystemExit( + f"{type(e).__name__}: cannot reach {HOST} from this shell.\n" + "Not a script problem - run the script, unmodified, in the " + "CloudShell VPC environment from Step 4." + ) open(fname, "wb").write(r.data) print(fname, "HTTP", r.status) EOF @@ -155,6 +171,8 @@ aws s3 cp settings.json s3:///search-diagnostics/ Use any bucket you can write to, then download the files from the S3 console. +If no subnet with an S3 route was available in Step 4, skip S3 entirely — the files are small enough to move as text. In the VPC environment run `tar czf - cat_*.txt settings.json | base64`, copy the output from the terminal, and on your machine paste it into `base64 -d | tar xzf -` (finish with Ctrl-D). + ## Step 7 — Send the files Attach everything — the `cw_*.json` metric exports, the four cluster-state files, and the engine version from Step 2 — to your support thread or email them to support@quilt.bio. @@ -166,7 +184,7 @@ Delete the CloudShell VPC environment when you're done — it otherwise keeps ne ## Troubleshooting - **A `cw_*.json` file has an empty `Datapoints` array** (Step 3) — wrong `DOMAIN`, `ACCOUNT`, or region in the variables. -- **The script hangs, then times out** (Step 5) — the environment isn't in the stack VPC, or is missing the accessor security group from Step 4. +- **`cannot reach …` / `ConnectTimeoutError` / `Max retries exceeded` — or a silent hang** (Step 5) — the shell has no network path to the domain: you're not in the VPC environment from Step 4, or it's missing the accessor security group. This is a network-placement problem, not a script bug — modifying the script won't help; fix the environment and rerun the script unmodified. - **`HTTP 403`** (Step 5) — the credentials lack `es:ESHttpGet` on the domain (Step 1). - **`HTTP 401` with `"Your request … is not allowed"`** (Step 5) — the URL path isn't on AWS's supported-operations allowlist for managed domains; use the script exactly as given above. -- **`aws s3 cp` hangs or fails** (Step 6) — the subnet has no route to S3; recreate the environment in a subnet that has one (see the subnet guidance in Step 4). +- **`aws s3 cp` hangs or fails** (Step 6) — the subnet has no route to S3; recreate the environment in a subnet that has one (see the subnet guidance in Step 4), or use Step 6's no-S3 fallback. From f8bef28e12f1bea9a27d37ef1dbb1bdaabfb99a7 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 13:39:39 +0500 Subject: [PATCH 02/13] Subnet guidance instead of no-S3 fallback The fallback's premise (a VPC with no S3-routed subnet) can't occur in a working deployment - service subnets must reach S3. The realistic trap is the opposite: on newer network layouts the domain's own subnets are deliberately isolated, so the natural subnet choice breaks Step 6. Name the subnets that always work and how to verify a candidate. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 9bce989..299efa2 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -100,7 +100,7 @@ aws ec2 describe-security-groups --region \ In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Despite the "create", this is not new infrastructure — not a VPC endpoint, not an instance: a VPC environment is an ordinary CloudShell session whose network interface sits inside your VPC. It changes nothing in your Quilt deployment, costs nothing, and deleting it (see Cleanup) removes everything it made. Before creating, know: at most **two** VPC environments per user (delete one if you're at the limit), and network settings are fixed at creation — delete and recreate to change them. Fill in: - **VPC**: the `vpc` value from Step 2. -- **Subnet**: any subnet reaches the domain, but Step 6 copies the files out via S3, which needs an S3 route — an S3 gateway endpoint in the subnet's route table, NAT, or your usual egress path (e.g. Transit Gateway). This is about *picking* a subnet whose route already exists, not creating one: in Quilt-created VPCs every subnet typically qualifies, and if none does, Step 6 has a fallback that needs no S3 at all. +- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…` (older layouts: `-a`/`-b`); if you supplied your own VPC, use one of the stack's `Subnets` parameter values. To verify a candidate, check its route table (VPC console → Subnets → Route table tab) for an S3 gateway endpoint (`vpce-…`) or a default route to NAT/your usual egress path. This is about *picking* a subnet — never create routes or endpoints for this. - **Security group**: the one from above. Provisioning takes a minute or two; you're ready when the new environment opens with a shell prompt. @@ -171,8 +171,6 @@ aws s3 cp settings.json s3:///search-diagnostics/ Use any bucket you can write to, then download the files from the S3 console. -If no subnet with an S3 route was available in Step 4, skip S3 entirely — the files are small enough to move as text. In the VPC environment run `tar czf - cat_*.txt settings.json | base64`, copy the output from the terminal, and on your machine paste it into `base64 -d | tar xzf -` (finish with Ctrl-D). - ## Step 7 — Send the files Attach everything — the `cw_*.json` metric exports, the four cluster-state files, and the engine version from Step 2 — to your support thread or email them to support@quilt.bio. @@ -187,4 +185,4 @@ Delete the CloudShell VPC environment when you're done — it otherwise keeps ne - **`cannot reach …` / `ConnectTimeoutError` / `Max retries exceeded` — or a silent hang** (Step 5) — the shell has no network path to the domain: you're not in the VPC environment from Step 4, or it's missing the accessor security group. This is a network-placement problem, not a script bug — modifying the script won't help; fix the environment and rerun the script unmodified. - **`HTTP 403`** (Step 5) — the credentials lack `es:ESHttpGet` on the domain (Step 1). - **`HTTP 401` with `"Your request … is not allowed"`** (Step 5) — the URL path isn't on AWS's supported-operations allowlist for managed domains; use the script exactly as given above. -- **`aws s3 cp` hangs or fails** (Step 6) — the subnet has no route to S3; recreate the environment in a subnet that has one (see the subnet guidance in Step 4), or use Step 6's no-S3 fallback. +- **`aws s3 cp` hangs or fails** (Step 6) — the subnet has no route to S3 (the domain's own subnets often don't); delete the environment and recreate it in a subnet that has one — see the subnet guidance in Step 4. From 8829afa90d1936e0454ccfad6e53e1e7b80f6fa6 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 13:47:13 +0500 Subject: [PATCH 03/13] Drop the route-table verification from the subnet bullet The named service subnets are authoritative already, and asking the reader to hunt for an S3 gateway endpoint re-arms the very "I'd have to create a VPC endpoint" misreading Step 4 defuses. A wrong pick is cheap: Step 6 hangs, recreate with another subnet. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 299efa2..f921355 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -100,7 +100,7 @@ aws ec2 describe-security-groups --region \ In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Despite the "create", this is not new infrastructure — not a VPC endpoint, not an instance: a VPC environment is an ordinary CloudShell session whose network interface sits inside your VPC. It changes nothing in your Quilt deployment, costs nothing, and deleting it (see Cleanup) removes everything it made. Before creating, know: at most **two** VPC environments per user (delete one if you're at the limit), and network settings are fixed at creation — delete and recreate to change them. Fill in: - **VPC**: the `vpc` value from Step 2. -- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…` (older layouts: `-a`/`-b`); if you supplied your own VPC, use one of the stack's `Subnets` parameter values. To verify a candidate, check its route table (VPC console → Subnets → Route table tab) for an S3 gateway endpoint (`vpce-…`) or a default route to NAT/your usual egress path. This is about *picking* a subnet — never create routes or endpoints for this. +- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…` (older layouts: `-a`/`-b`); if you supplied your own VPC, use one of the stack's `Subnets` parameter values. Don't create routes or endpoints for this — the worst a wrong pick does is make Step 6 hang, and recreating the environment with another subnet fixes it. - **Security group**: the one from above. Provisioning takes a minute or two; you're ready when the new environment opens with a shell prompt. From 7d68b180d9fd97dfc04badc72281743d30c9cf29 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 13:52:42 +0500 Subject: [PATCH 04/13] Drop the create-nothing prohibition, keep the worst-case reassurance Nothing in the bullet suggests creating anything anymore, so the prohibition only re-plants the "endpoints" idea it guards against. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index f921355..02adf60 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -100,7 +100,7 @@ aws ec2 describe-security-groups --region \ In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Despite the "create", this is not new infrastructure — not a VPC endpoint, not an instance: a VPC environment is an ordinary CloudShell session whose network interface sits inside your VPC. It changes nothing in your Quilt deployment, costs nothing, and deleting it (see Cleanup) removes everything it made. Before creating, know: at most **two** VPC environments per user (delete one if you're at the limit), and network settings are fixed at creation — delete and recreate to change them. Fill in: - **VPC**: the `vpc` value from Step 2. -- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…` (older layouts: `-a`/`-b`); if you supplied your own VPC, use one of the stack's `Subnets` parameter values. Don't create routes or endpoints for this — the worst a wrong pick does is make Step 6 hang, and recreating the environment with another subnet fixes it. +- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…` (older layouts: `-a`/`-b`); if you supplied your own VPC, use one of the stack's `Subnets` parameter values. The worst a wrong pick does is make Step 6 hang; recreating the environment with another subnet fixes it. - **Security group**: the one from above. Provisioning takes a minute or two; you're ready when the new environment opens with a shell prompt. From b41303872d75624cd6bec856c5907a013eed5646 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 14:51:50 +0500 Subject: [PATCH 05/13] Drop the legacy-layout subnet-naming hint The -a/-b naming only helps on a Quilt-created network-1.0 VPC with an in-VPC domain: no such build exists (all current 1.0 variants leave the ES domain public), and that layout would not need the hint anyway - its two service subnets share the S3-gateway route table, so any pick works. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 02adf60..f92bd67 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -100,7 +100,7 @@ aws ec2 describe-security-groups --region \ In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Despite the "create", this is not new infrastructure — not a VPC endpoint, not an instance: a VPC environment is an ordinary CloudShell session whose network interface sits inside your VPC. It changes nothing in your Quilt deployment, costs nothing, and deleting it (see Cleanup) removes everything it made. Before creating, know: at most **two** VPC environments per user (delete one if you're at the limit), and network settings are fixed at creation — delete and recreate to change them. Fill in: - **VPC**: the `vpc` value from Step 2. -- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…` (older layouts: `-a`/`-b`); if you supplied your own VPC, use one of the stack's `Subnets` parameter values. The worst a wrong pick does is make Step 6 hang; recreating the environment with another subnet fixes it. +- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…`; if you supplied your own VPC, use one of the stack's `Subnets` parameter values. The worst a wrong pick does is make Step 6 hang; recreating the environment with another subnet fixes it. - **Security group**: the one from above. Provisioning takes a minute or two; you're ready when the new environment opens with a shell prompt. From 1ce5854d5fa811da718822541c01bde677bfeb71 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 15:11:17 +0500 Subject: [PATCH 06/13] Apply review-pass fixes From a full read-through review: the public-endpoint path is a labelled note and the script's precondition comment/error name it instead of contradicting it; add the 7.1+ write-rejection metrics (12->15 files) with an older-engine caveat; state the VPC-environment idle timeout; Cleanup covers a leftover ENI and the uploaded S3 copies; drop the unsourced search-accessor name and use "accessor security group" consistently; split Step 4's reassurance from its constraints; lead the subnet bullet with the instruction; quota is per IAM principal; soften the now-overbroad CloudWatch-route claim; correct the datapoint count. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 38 ++++++++++++--------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index f92bd67..4c13598 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -16,7 +16,7 @@ You'll work in two shells: a **standard CloudShell** for the performance metrics Performance history — one JSON file per CloudWatch metric (CPU, search/indexing latency and rates, queues and rejections, JVM pressure, storage): -- `cw_.json` (12 files) +- `cw_.json` (15 files) Cluster state — four files describing shard layout, index sizes, disk allocation, and index settings: @@ -35,7 +35,7 @@ The credentials you'll use — in CloudShell, the console role you're signed in - `es:ListDomainNames` and `es:DescribeDomain` — for Step 2 - `cloudwatch:GetMetricStatistics` — for Step 3 -- permission to create and use CloudShell VPC environments ([AWS's required IAM permissions](https://docs.aws.amazon.com/cloudshell/latest/userguide/sec-auth-with-identities.html)) — for Step 4 +- permission to create and use CloudShell VPC environments ([AWS's required IAM permissions](https://docs.aws.amazon.com/cloudshell/latest/userguide/sec-auth-with-identities.html)) — for Step 4 (this creates no infrastructure — see the note there) - `es:ESHttpGet` on the domain — the cluster-state requests (Step 5) are all read-only GETs - write access to some S3 bucket — for copying the results out (Step 6) @@ -58,11 +58,11 @@ aws opensearch describe-domain --domain-name --region \ Note all three — the endpoint and VPC are used below; include the engine version in what you send to support. If you run several Quilt stacks, the `vpc` value tells you which stack's VPC a domain belongs to. -(A null `endpoint` means the domain has a public endpoint instead — `DomainStatus.Endpoint`. Then skip Step 4, run the Step 5 script from any shell with that endpoint as `HOST`, and skip Step 6 — the files land wherever you ran the script.) +**If `endpoint` is null**, your domain has a public endpoint instead — read it from `DomainStatus.Endpoint`. Public-endpoint deployments skip Steps 4 and 6: run the Step 5 script from any shell, with the public endpoint as `HOST` — the files land wherever you run it. ## Step 3 — Export CloudWatch metrics -This step needs no VPC access — run it in a standard CloudShell (or any shell with the AWS CLI; Linux and macOS both work), **not** in the VPC environment you'll create in Step 4, which has no route to the CloudWatch API. Fill in the two variables at the top: +This step needs no VPC access — run it in a standard CloudShell (or any shell with the AWS CLI; Linux and macOS both work), **not** in the VPC environment you'll create in Step 4, which may have no route to the CloudWatch API. Fill in the two variables at the top: ```bash REGION= @@ -73,7 +73,8 @@ END=$(date +%s) for m in CPUUtilization SearchLatency IndexingLatency SearchRate IndexingRate \ JVMMemoryPressure ThreadpoolSearchQueue ThreadpoolSearchRejected \ - ThreadpoolWriteQueue ThreadpoolWriteRejected ClusterIndexWritesBlocked FreeStorageSpace; do + ThreadpoolWriteQueue ThreadpoolWriteRejected CoordinatingWriteRejected \ + PrimaryWriteRejected ReplicaWriteRejected ClusterIndexWritesBlocked FreeStorageSpace; do aws cloudwatch get-metric-statistics --region $REGION --namespace AWS/ES \ --metric-name $m --dimensions Name=DomainName,Value=$DOMAIN Name=ClientId,Value=$ACCOUNT \ --start-time $START --end-time $END --period 600 --statistics Average Maximum \ @@ -82,13 +83,13 @@ for m in CPUUtilization SearchLatency IndexingLatency SearchRate IndexingRate \ done ``` -This exports the last 7 days at 10-minute resolution (support may ask for a different window). Each file should contain a `Datapoints` array with several hundred entries — an empty array means a wrong `DOMAIN`, `ACCOUNT`, or region. +This exports the last 7 days at 10-minute resolution (support may ask for a different window). Each file should contain a `Datapoints` array with roughly a thousand entries — an empty array means a wrong `DOMAIN`, `ACCOUNT`, or region (exception: `CoordinatingWriteRejected`, `PrimaryWriteRejected`, and `ReplicaWriteRejected` only exist on engine versions 7.1+, so empty files for those are normal on older domains). In a standard CloudShell you can download the files directly: **Actions → Download file**. (Or `aws s3 cp` them to the same bucket you'll use in Step 6.) ## Step 4 — Open a shell inside the stack VPC -The domain only accepts connections from members of one security group in your Quilt deployment. Find it in the EC2 console by its description — *"For resources that need access to search cluster"* — or by name: depending on how your deployment was created it contains `search-accessor` or `SearchClusterAccessorSecurityGroup`. Or from any shell (`` is the `vpc` value from Step 2): +The domain only accepts connections from members of one security group in your Quilt deployment — the **accessor security group**. Find it in the EC2 console by its description — *"For resources that need access to search cluster"* — or by name (it contains `SearchClusterAccessorSecurityGroup`). Or from any shell (`` is the `vpc` value from Step 2): ```bash aws ec2 describe-security-groups --region \ @@ -97,11 +98,13 @@ aws ec2 describe-security-groups --region \ --output text ``` -In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Despite the "create", this is not new infrastructure — not a VPC endpoint, not an instance: a VPC environment is an ordinary CloudShell session whose network interface sits inside your VPC. It changes nothing in your Quilt deployment, costs nothing, and deleting it (see Cleanup) removes everything it made. Before creating, know: at most **two** VPC environments per user (delete one if you're at the limit), and network settings are fixed at creation — delete and recreate to change them. Fill in: +In **CloudShell** (in your stack's region), create a new **VPC environment** ([AWS walkthrough](https://docs.aws.amazon.com/cloudshell/latest/userguide/creating-vpc-environment.html)). Despite the "create", this is not new infrastructure — not a VPC endpoint, not an instance: a VPC environment is an ordinary CloudShell session whose network interface sits inside your VPC. It changes nothing in your Quilt deployment, costs nothing, and deleting it (see Cleanup) removes everything it made. + +Before creating, know two constraints: at most **two** VPC environments per IAM principal — shared with anyone else using the same role — so delete one if you're at the limit; and network settings are fixed at creation, so to change them you delete and recreate. Fill in: - **VPC**: the `vpc` value from Step 2. -- **Subnet**: any subnet in the VPC can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, so pick a subnet that can also reach S3. **Don't assume the domain's own subnets qualify** — in newer deployments they're deliberately isolated, with no route to S3. A subnet the Quilt services run on always works: in a Quilt-created VPC that's a subnet named `-private-…`; if you supplied your own VPC, use one of the stack's `Subnets` parameter values. The worst a wrong pick does is make Step 6 hang; recreating the environment with another subnet fixes it. -- **Security group**: the one from above. +- **Subnet**: pick a subnet the Quilt services run on — in a Quilt-created VPC, one named `-private-…`; if you supplied your own VPC, one of the stack's `Subnets` parameter values. Any subnet can reach the domain (access is gated by the security group), but Step 6 copies the files out via S3, and **the domain's own subnets often can't reach S3** — in newer deployments they're deliberately isolated. The worst a wrong pick does is make Step 6 hang; recreating the environment with another subnet fixes it. +- **Security group**: the accessor security group from above. Provisioning takes a minute or two; you're ready when the new environment opens with a shell prompt. @@ -113,8 +116,8 @@ Fill in the two placeholders — `` is the `endpoint` value from S ```bash python3 << 'EOF' -# Run this ONLY in the CloudShell VPC environment from Step 4 — -# from any other shell the domain is unreachable and every request fails. +# A VPC-internal domain is reachable ONLY from the CloudShell VPC environment +# from Step 4 — anywhere else, every request fails. (Public endpoint: any shell.) import boto3, urllib3 from botocore.auth import SigV4Auth from botocore.awsrequest import AWSRequest @@ -139,8 +142,9 @@ for fname, path in FILES.items(): except urllib3.exceptions.HTTPError as e: raise SystemExit( f"{type(e).__name__}: cannot reach {HOST} from this shell.\n" - "Not a script problem - run the script, unmodified, in the " - "CloudShell VPC environment from Step 4." + "Not a script problem. VPC-internal domain: run this, unmodified, " + "in the CloudShell VPC environment from Step 4. " + "Public endpoint: check the HOST value." ) open(fname, "wb").write(r.data) print(fname, "HTTP", r.status) @@ -160,7 +164,7 @@ Any other status means that request failed and its file contains the error messa ## Step 6 — Copy the cluster-state files out -CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends** — so move the files to S3 right away: +CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends** — idle sessions end after 20–30 minutes (10 in GovCloud) — so move the files to S3 right away: ```bash aws s3 cp cat_shards.txt s3:///search-diagnostics/ @@ -177,12 +181,12 @@ Attach everything — the `cw_*.json` metric exports, the four cluster-state fil ## Cleanup -Delete the CloudShell VPC environment when you're done — it otherwise keeps network interfaces in your VPC. +Delete the CloudShell VPC environment when you're done — it otherwise keeps network interfaces in your VPC. If the delete leaves a network interface behind (your role lacks `ec2:DeleteNetworkInterface`), remove it in the EC2 console. Once support confirms it has the files, you can also delete the `search-diagnostics/` copies from your bucket. ## Troubleshooting - **A `cw_*.json` file has an empty `Datapoints` array** (Step 3) — wrong `DOMAIN`, `ACCOUNT`, or region in the variables. -- **`cannot reach …` / `ConnectTimeoutError` / `Max retries exceeded` — or a silent hang** (Step 5) — the shell has no network path to the domain: you're not in the VPC environment from Step 4, or it's missing the accessor security group. This is a network-placement problem, not a script bug — modifying the script won't help; fix the environment and rerun the script unmodified. +- **`cannot reach …` (`ConnectTimeoutError`, `NewConnectionError`), `Max retries exceeded`, or a silent hang** (Step 5) — the shell has no network path to the domain: you're not in the VPC environment from Step 4, or it's missing the accessor security group. This is a network-placement problem, not a script bug — modifying the script won't help; fix the environment and rerun the script unmodified. - **`HTTP 403`** (Step 5) — the credentials lack `es:ESHttpGet` on the domain (Step 1). - **`HTTP 401` with `"Your request … is not allowed"`** (Step 5) — the URL path isn't on AWS's supported-operations allowlist for managed domains; use the script exactly as given above. - **`aws s3 cp` hangs or fails** (Step 6) — the subnet has no route to S3 (the domain's own subnets often don't); delete the environment and recreate it in a subnet that has one — see the subnet guidance in Step 4. From ac4822e07919c6c6dddcd9c96bdca7f02afd6e2e Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 15:16:01 +0500 Subject: [PATCH 07/13] Key the empty-Datapoints rule on every-file vs a-few Wrong DOMAIN/ACCOUNT/region empties every file; engine-version gaps empty only a few - the all-vs-some split makes the exception self-evident without naming metrics or versions. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 4c13598..5cb3de0 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -83,7 +83,7 @@ for m in CPUUtilization SearchLatency IndexingLatency SearchRate IndexingRate \ done ``` -This exports the last 7 days at 10-minute resolution (support may ask for a different window). Each file should contain a `Datapoints` array with roughly a thousand entries — an empty array means a wrong `DOMAIN`, `ACCOUNT`, or region (exception: `CoordinatingWriteRejected`, `PrimaryWriteRejected`, and `ReplicaWriteRejected` only exist on engine versions 7.1+, so empty files for those are normal on older domains). +This exports the last 7 days at 10-minute resolution (support may ask for a different window). Each file should contain a `Datapoints` array with roughly a thousand entries. Empty arrays in **every** file mean a wrong `DOMAIN`, `ACCOUNT`, or region; a few empty files are normal (not every metric exists on every engine version). In a standard CloudShell you can download the files directly: **Actions → Download file**. (Or `aws s3 cp` them to the same bucket you'll use in Step 6.) @@ -185,7 +185,7 @@ Delete the CloudShell VPC environment when you're done — it otherwise keeps ne ## Troubleshooting -- **A `cw_*.json` file has an empty `Datapoints` array** (Step 3) — wrong `DOMAIN`, `ACCOUNT`, or region in the variables. +- **Every `cw_*.json` file has an empty `Datapoints` array** (Step 3) — wrong `DOMAIN`, `ACCOUNT`, or region in the variables (a few empty files are normal). - **`cannot reach …` (`ConnectTimeoutError`, `NewConnectionError`), `Max retries exceeded`, or a silent hang** (Step 5) — the shell has no network path to the domain: you're not in the VPC environment from Step 4, or it's missing the accessor security group. This is a network-placement problem, not a script bug — modifying the script won't help; fix the environment and rerun the script unmodified. - **`HTTP 403`** (Step 5) — the credentials lack `es:ESHttpGet` on the domain (Step 1). - **`HTTP 401` with `"Your request … is not allowed"`** (Step 5) — the URL path isn't on AWS's supported-operations allowlist for managed domains; use the script exactly as given above. From 0837e5cecc213c1de2edf4987d198f308be4b9fa Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 15:18:09 +0500 Subject: [PATCH 08/13] Two micro-trims: drop the OS aside, untangle Step 6 punctuation Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 5cb3de0..1dd620a 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -62,7 +62,7 @@ Note all three — the endpoint and VPC are used below; include the engine versi ## Step 3 — Export CloudWatch metrics -This step needs no VPC access — run it in a standard CloudShell (or any shell with the AWS CLI; Linux and macOS both work), **not** in the VPC environment you'll create in Step 4, which may have no route to the CloudWatch API. Fill in the two variables at the top: +This step needs no VPC access — run it in a standard CloudShell (or any shell with the AWS CLI), **not** in the VPC environment you'll create in Step 4, which may have no route to the CloudWatch API. Fill in the two variables at the top: ```bash REGION= @@ -164,7 +164,7 @@ Any other status means that request failed and its file contains the error messa ## Step 6 — Copy the cluster-state files out -CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends** — idle sessions end after 20–30 minutes (10 in GovCloud) — so move the files to S3 right away: +CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends**, and idle sessions end after 20–30 minutes (10 in GovCloud) — so move the files to S3 right away: ```bash aws s3 cp cat_shards.txt s3:///search-diagnostics/ From 3ff6ba384adf11f6d46c94c7b1e41767325dca8a Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 15:20:37 +0500 Subject: [PATCH 09/13] Fold "What you'll collect" into the Summary The section was an inventory repeating the steps: the cluster-state filenames appear in the script, its expected output, and Step 6; the metric list is the loop itself. Only the two-shells roadmap and the disclosure note were load-bearing - both now close the Summary. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 23 +++------------------ 1 file changed, 3 insertions(+), 20 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 1dd620a..e935345 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -6,28 +6,11 @@ ## Summary -When you report search-related issues (slow or failing searches, incomplete results, high cluster CPU), Quilt support may ask for diagnostics from your deployment's Elasticsearch/OpenSearch domain: CloudWatch performance metrics, plus cluster-state files collected from inside the stack's VPC (the domain is VPC-internal in most deployments). This article shows how to do both with AWS CloudShell and a short read-only script — no software installation required. +When you report search-related issues (slow or failing searches, incomplete results, high cluster CPU), Quilt support may ask for diagnostics from your deployment's Elasticsearch/OpenSearch domain: CloudWatch performance metrics, plus cluster-state files collected from inside the stack's VPC (the domain is VPC-internal in most deployments). This article shows how to do both with AWS CloudShell and a short read-only script — no software installation required. You'll work in two shells: a **standard CloudShell** for the performance metrics (Step 3), and a **CloudShell VPC environment** for the cluster state (Steps 4–6). ---- - -## What you'll collect - -You'll work in two shells: a **standard CloudShell** for the performance metrics (Step 3), and a **CloudShell VPC environment** for the cluster state (Steps 4–6). - -Performance history — one JSON file per CloudWatch metric (CPU, search/indexing latency and rates, queues and rejections, JVM pressure, storage): - -- `cw_.json` (15 files) +The collected files disclose your registered bucket names (as index names), but no object contents, document data, or credentials. -Cluster state — four files describing shard layout, index sizes, disk allocation, and index settings: - -- `cat_shards.txt` -- `cat_indices.txt` -- `cat_allocation.txt` -- `settings.json` - -Plus one line of text: the domain's engine version (from Step 2). - -The files disclose your registered bucket names (as index names in the cluster-state files), but no object contents, document data, or credentials. +--- ## Step 1 — Check your credentials From 7936aeed90950b39be8e1027dd7004d991fcf7c2 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 15:35:04 +0500 Subject: [PATCH 10/13] Drop Step 3's premature VPC-environment warning It forward-references an environment that doesn't exist yet at that point, and Step 4's subnet guidance now steers to NAT'd subnets that usually do reach CloudWatch. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index e935345..6071443 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -45,7 +45,7 @@ Note all three — the endpoint and VPC are used below; include the engine versi ## Step 3 — Export CloudWatch metrics -This step needs no VPC access — run it in a standard CloudShell (or any shell with the AWS CLI), **not** in the VPC environment you'll create in Step 4, which may have no route to the CloudWatch API. Fill in the two variables at the top: +This step needs no VPC access — run it in a standard CloudShell (or any shell with the AWS CLI). Fill in the two variables at the top: ```bash REGION= From 7f60a0aab6f37b69c68a5cbe6d3413e040d6941f Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 17:02:20 +0500 Subject: [PATCH 11/13] Cut the allowlist aside; state the idle timeout as a floor The allowlist explanation lives in Troubleshooting's 401 entry, at the point of need. "As little as 10 minutes" is true in both partitions without the GovCloud parenthetical. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 6071443..0a192df 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -93,7 +93,7 @@ Provisioning takes a minute or two; you're ready when the new environment opens ## Step 5 — Collect the cluster state -Back in the **VPC environment** from Step 4: requests to the domain must be SigV4-signed, and the script below signs them with your session's credentials using CloudShell's preinstalled Python and boto3. The request paths are fixed — AWS-managed domains accept only an allowlisted subset of the domain's REST API. +Back in the **VPC environment** from Step 4: requests to the domain must be SigV4-signed, and the script below signs them with your session's credentials using CloudShell's preinstalled Python and boto3. Fill in the two placeholders — `` is the `endpoint` value from Step 2 (no `https://` prefix); `` as before — then paste the whole block: @@ -147,7 +147,7 @@ Any other status means that request failed and its file contains the error messa ## Step 6 — Copy the cluster-state files out -CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends**, and idle sessions end after 20–30 minutes (10 in GovCloud) — so move the files to S3 right away: +CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends**, and an idle session can end in as little as 10 minutes — so move the files to S3 right away: ```bash aws s3 cp cat_shards.txt s3:///search-diagnostics/ From 71beeb58fe1dc1c37847bb78069c7961ca9570f5 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 17:05:23 +0500 Subject: [PATCH 12/13] Fix Step 6 run-on: timeout becomes a parenthetical Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 0a192df..9404add 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -147,7 +147,7 @@ Any other status means that request failed and its file contains the error messa ## Step 6 — Copy the cluster-state files out -CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends**, and an idle session can end in as little as 10 minutes — so move the files to S3 right away: +CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends** (an idle session can end in as little as 10 minutes) — so move the files to S3 right away: ```bash aws s3 cp cat_shards.txt s3:///search-diagnostics/ From 8b1973afde67249a2967354aac6747432131e97d Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Mon, 3 Aug 2026 17:37:56 +0500 Subject: [PATCH 13/13] Address Copilot review Script precondition scoped to the VPC (not just the CloudShell environment); public-endpoint note names the Python+boto3 requirement. Co-Authored-By: Claude Fable 5 --- howto-collect-search-cluster-diagnostics.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/howto-collect-search-cluster-diagnostics.md b/howto-collect-search-cluster-diagnostics.md index 9404add..615e11e 100644 --- a/howto-collect-search-cluster-diagnostics.md +++ b/howto-collect-search-cluster-diagnostics.md @@ -41,7 +41,7 @@ aws opensearch describe-domain --domain-name --region \ Note all three — the endpoint and VPC are used below; include the engine version in what you send to support. If you run several Quilt stacks, the `vpc` value tells you which stack's VPC a domain belongs to. -**If `endpoint` is null**, your domain has a public endpoint instead — read it from `DomainStatus.Endpoint`. Public-endpoint deployments skip Steps 4 and 6: run the Step 5 script from any shell, with the public endpoint as `HOST` — the files land wherever you run it. +**If `endpoint` is null**, your domain has a public endpoint instead — read it from `DomainStatus.Endpoint`. Public-endpoint deployments skip Steps 4 and 6: run the Step 5 script from any shell with Python and boto3 (a standard CloudShell has both), with the public endpoint as `HOST` — the files land wherever you run it. ## Step 3 — Export CloudWatch metrics @@ -99,8 +99,9 @@ Fill in the two placeholders — `` is the `endpoint` value from S ```bash python3 << 'EOF' -# A VPC-internal domain is reachable ONLY from the CloudShell VPC environment -# from Step 4 — anywhere else, every request fails. (Public endpoint: any shell.) +# A VPC-internal domain is reachable only from inside the stack VPC — use the +# CloudShell VPC environment from Step 4; elsewhere every request fails. +# (Public endpoint: any shell.) import boto3, urllib3 from botocore.auth import SigV4Auth from botocore.awsrequest import AWSRequest