Search-diagnostics article: fail fast outside the VPC, defuse the VPC-environment step - #27
Merged
Conversation
…ment step Field feedback: an operator read Step 4's "create a VPC environment" as provisioning real infrastructure and skipped it, then ran the Step 5 script in a standard CloudShell, where it hung indefinitely with no diagnostic. Reassure on what Step 4 actually creates, make the script fail fast with a message pointing back to Step 4, add a CLI security-group lookup and a no-S3 copy-out fallback, and key Troubleshooting on the real error strings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| Use any bucket you can write to, then download the files from the S3 console. | ||
|
|
||
| If no subnet with an S3 route was available in Step 4, skip S3 entirely — the files are small enough to move as text. In the VPC environment run `tar czf - cat_*.txt settings.json | base64`, copy the output from the terminal, and on your machine paste it into `base64 -d | tar xzf -` (finish with Ctrl-D). |
Contributor
There was a problem hiding this comment.
The fallback prescribes a Unix-oriented base64 -d | tar xzf - pipeline and Ctrl-D without identifying the required local environment. On machines whose base64 utility uses different flags or lacks these commands, decoding fails or waits for input, preventing recovery of the diagnostics; please state the platform requirements or provide platform-specific alternatives.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
The fallback's premise (a VPC with no S3-routed subnet) can't occur in a working deployment - service subnets must reach S3. The realistic trap is the opposite: on newer network layouts the domain's own subnets are deliberately isolated, so the natural subnet choice breaks Step 6. Name the subnets that always work and how to verify a candidate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The named service subnets are authoritative already, and asking the reader to hunt for an S3 gateway endpoint re-arms the very "I'd have to create a VPC endpoint" misreading Step 4 defuses. A wrong pick is cheap: Step 6 hangs, recreate with another subnet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nothing in the bullet suggests creating anything anymore, so the prohibition only re-plants the "endpoints" idea it guards against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The <stack>-a/-b naming only helps on a Quilt-created network-1.0 VPC with an in-VPC domain: no such build exists (all current 1.0 variants leave the ES domain public), and that layout would not need the hint anyway - its two service subnets share the S3-gateway route table, so any pick works. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From a full read-through review: the public-endpoint path is a labelled note and the script's precondition comment/error name it instead of contradicting it; add the 7.1+ write-rejection metrics (12->15 files) with an older-engine caveat; state the VPC-environment idle timeout; Cleanup covers a leftover ENI and the uploaded S3 copies; drop the unsourced search-accessor name and use "accessor security group" consistently; split Step 4's reassurance from its constraints; lead the subnet bullet with the instruction; quota is per IAM principal; soften the now-overbroad CloudWatch-route claim; correct the datapoint count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wrong DOMAIN/ACCOUNT/region empties every file; engine-version gaps empty only a few - the all-vs-some split makes the exception self-evident without naming metrics or versions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The section was an inventory repeating the steps: the cluster-state filenames appear in the script, its expected output, and Step 6; the metric list is the loop itself. Only the two-shells roadmap and the disclosure note were load-bearing - both now close the Summary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
It forward-references an environment that doesn't exist yet at that point, and Step 4's subnet guidance now steers to NAT'd subnets that usually do reach CloudWatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The allowlist explanation lives in Troubleshooting's 401 entry, at the point of need. "As little as 10 minutes" is true in both partitions without the GovCloud parenthetical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the “collect search-cluster diagnostics” support article to prevent operators from running the cluster-state script outside the VPC, by clarifying CloudShell VPC environments and adding faster failure behavior and troubleshooting guidance.
Changes:
- Clarifies Step 4 (CloudShell VPC environment) to reduce confusion about “creating” infrastructure and improve guidance on picking the right subnet / security group.
- Updates the Step 5 script to fail fast when the endpoint is unreachable (connect timeout + clearer error) and adds copy-paste-surviving precondition guidance inside the script.
- Expands Step 3 CloudWatch metrics (adds write-rejection metrics) and refreshes troubleshooting/cleanup guidance.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+102
to
+103
| # A VPC-internal domain is reachable ONLY from the CloudShell VPC environment | ||
| # from Step 4 — anywhere else, every request fails. (Public endpoint: any shell.) |
| Note all three — the endpoint and VPC are used below; include the engine version in what you send to support. If you run several Quilt stacks, the `vpc` value tells you which stack's VPC a domain belongs to. | ||
|
|
||
| (A null `endpoint` means the domain has a public endpoint instead — `DomainStatus.Endpoint`. Then skip Step 4, run the Step 5 script from any shell with that endpoint as `HOST`, and skip Step 6 — the files land wherever you ran the script.) | ||
| **If `endpoint` is null**, your domain has a public endpoint instead — read it from `DomainStatus.Endpoint`. Public-endpoint deployments skip Steps 4 and 6: run the Step 5 script from any shell, with the public endpoint as `HOST` — the files land wherever you run it. |
| ## Step 6 — Copy the cluster-state files out | ||
|
|
||
| CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends** — so move the files to S3 right away: | ||
| CloudShell VPC environments can't use the console's upload/download menu, and their storage is **deleted when the session ends** (an idle session can end in as little as 10 minutes) — so move the files to S3 right away: |
Script precondition scoped to the VPC (not just the CloudShell environment); public-endpoint note names the Python+boto3 requirement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Field feedback from a support engagement: an operator following this article balked at Step 4 — read "create a VPC environment" as provisioning real infrastructure (a VPC endpoint) and skipped it — then ran the Step 5 script in a standard CloudShell, where it hung indefinitely with no diagnostic, prompting ad-hoc script modifications instead of a return to Step 4.
cannot reach,ConnectTimeoutError,Max retries exceeded) and say explicitly it's a network-placement problem — rerun the script unmodified.Coordinating/Primary/ReplicaWriteRejected, 12→15 files) with an older-engine caveat; stated the VPC-environment idle timeout (as little as 10 minutes — a floor that holds in GovCloud too); Cleanup now covers a leftover ENI (missingec2:DeleteNetworkInterface) and deleting the uploaded S3 copies; dropped the unsourcedsearch-accessorname; Step 4 flow split (reassurance vs constraints), subnet bullet leads with the instruction, quota corrected to per-IAM-principal.Wrong-shell path tested: against an unreachable host the script exits in ~10 s with the new message (exit 1). The happy path is unchanged apart from the timeout/except additions (
read=None, so slow responses from a degraded cluster are still never cut off).🤖 Generated with Claude Code