From f152fc6d9c92f18de568b2bbd76ab552741893d9 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 13:33:11 +0500 Subject: [PATCH 01/11] Add article: upgrading the search domain engine (ES 6.8 -> 7.10) Covers the drift case where the deployed template declares 7.10 but the live domain stayed on 6.8 after a stack-update-fired upgrade failed on a transient and CloudFormation recorded success. Direct CLI path: read-only until the single upgrade command, launch into the snapshot gap, escalation guidance for stalls. Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 104 ++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 howto-upgrade-search-domain-engine.md diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md new file mode 100644 index 0000000..37b3ce9 --- /dev/null +++ b/howto-upgrade-search-domain-engine.md @@ -0,0 +1,104 @@ +# How do I upgrade my Quilt deployment's search domain engine (Elasticsearch 6.8 → 7.10)? + +## Tags + +`aws`, `elasticsearch`, `opensearch`, `search`, `upgrade`, `cli`, `cloudformation` + +## Summary + +Some Quilt deployments run their search domain on Elasticsearch 6.8 even though the deployed CloudFormation template declares 7.10. This happens when the engine upgrade fired during a stack update fails on a transient condition — most commonly an automated snapshot running at that moment — and CloudFormation records the stack update as successful anyway. CloudFormation compares templates against templates, not against the live domain, so later deploys never retry the upgrade: the domain stays behind until someone upgrades it directly. + +This article is the direct path: an in-place engine upgrade via the AWS CLI. It is a short sequence in which **every command is read-only except one** — the upgrade trigger itself. Search keeps serving throughout (performance may dip while nodes are replaced; Kibana may be unavailable). Typical duration is minutes to hours depending on data size — plan for the possibility of most of a day on large domains. + +To check whether this applies to you: compare the live engine version with what your template declares. + +``` +aws es describe-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ + --query 'DomainStatus.ElasticsearchVersion' +``` + +(`aws es list-domain-names` shows your domain name. Quilt support can tell you what your release declares — for current releases it is 7.10.) + +## Before you start + +- **Set the variables** used by every command below: + + ``` + DOMAIN= + REGION= + ``` + +- **Permissions**: the credentials you use need `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeStatus`, `es:DescribeDomainAutoTunes`, and — for the one state-changing step — `es:UpgradeElasticsearchDomain`. + +- **Freeze the Quilt stack for the window**: no deploys, no CloudFormation changes, no admin "Re-index and repair" actions while the upgrade runs. The domain must be the only thing changing. + +- **Know your escalation path** (insurance only): check which AWS support plan is on the account (the Support Center page shows it) and who could open a technical case. In the rare case an upgrade stalls, the domain keeps serving search — it just refuses further configuration changes until AWS support unsticks it, and Business-or-above support is the channel for that. If the account is on Basic, note that Business support can be enabled in minutes and takes effect immediately, so the practical preparation is knowing who has the authority to approve it if ever needed. + +## Step 1 — Confirm the target version is a legal hop + +``` +aws es get-compatible-elasticsearch-versions --domain-name $DOMAIN --region $REGION +``` + +From 6.8, the list includes 7.10 — proceed with `TARGET=7.10`. + +If your domain is on a version **below** 6.8, note that 7.10 will not be in the list: Elasticsearch upgrades one step at a time (for example 6.7 → 6.8, then 6.8 → 7.10). Run this whole procedure once per hop. + +## Step 2 — Check nothing is scheduled to interfere + +``` +aws es describe-domain-auto-tunes --domain-name $DOMAIN --region $REGION +``` + +If no Auto-Tune action is scheduled for the coming day, proceed. If one is scheduled inside your window, move the window (or contact Quilt support about disabling Auto-Tune — the disable itself has options that matter). + +Also glance at the domain in the AWS console: cluster health should be green and the domain status Active before you continue. + +## Step 3 — Run the eligibility check + +This validates the upgrade without performing it — despite the command name, `--perform-check-only` changes nothing on the domain: + +``` +aws es upgrade-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ + --target-version $TARGET --perform-check-only +``` + +The check runs asynchronously. Fetch the verdict (repeat until the check completes — usually a few minutes): + +``` +aws es get-upgrade-status --domain-name $DOMAIN --region $REGION +``` + +- **`PRE_UPGRADE_CHECK: SUCCEEDED`** → go directly to Step 4, *now*. A passing check also means no automated snapshot is running at this moment — that is exactly the gap you want to launch into, because AWS takes hourly automated snapshots and an upgrade colliding with one fails. +- **Failed with "Prior snapshot operation has not yet completed"** → normal and harmless; an automated snapshot is running. Wait 10–15 minutes and repeat this step. (See [AWS's article on this error](https://repost.aws/knowledge-center/opensearch-prior-snapshot-error).) +- **Failed with anything else** → stop and send the output to Quilt support before proceeding. + +## Step 4 — Run the upgrade + +The one state-changing command: + +``` +aws es upgrade-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ + --target-version $TARGET +``` + +## Step 5 — Watch it + +Re-run the Step 3 status command occasionally, or watch the domain page in the console. The upgrade proceeds through `PRE_UPGRADE_CHECK` → `SNAPSHOT` → `UPGRADE` and takes anywhere from minutes to hours ([AWS's guidance on long-running upgrades](https://repost.aws/knowledge-center/opensearch-domain-upgrade)). + +If progress sits unchanged for several hours: **don't touch anything**. The domain keeps serving search while stuck; the recovery path is an AWS support case ([AWS's article on stuck upgrades](https://repost.aws/knowledge-center/opensearch-stuck-failed-upgrade)) — and contact Quilt support, we've been through this and will help draft the case. + +## Step 6 — Smoke test + +After the status shows the upgrade succeeded: + +1. Upload a small file to any bucket registered in your Quilt catalog. +2. Confirm it appears in catalog search within a couple of minutes. +3. Run a search you know should return results. + +If anything looks off, contact Quilt support — and tell us the upgrade completed either way, so we can verify from our side and plan any follow-up work with you. + +## Related + +- [Upgrading Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) — AWS's reference for the upgrade process +- [How do I collect search-cluster diagnostics for Quilt support?](howto-collect-search-cluster-diagnostics.md) From d43dbb00a70ddf706ff64e213d2f4703ec62c1c6 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 13:36:54 +0500 Subject: [PATCH 02/11] Lean pass on summary and escalation guidance Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index 37b3ce9..0e7a643 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -6,9 +6,9 @@ ## Summary -Some Quilt deployments run their search domain on Elasticsearch 6.8 even though the deployed CloudFormation template declares 7.10. This happens when the engine upgrade fired during a stack update fails on a transient condition — most commonly an automated snapshot running at that moment — and CloudFormation records the stack update as successful anyway. CloudFormation compares templates against templates, not against the live domain, so later deploys never retry the upgrade: the domain stays behind until someone upgrades it directly. +Some Quilt deployments run their search domain on Elasticsearch 6.8 even though the deployed CloudFormation template declares 7.10: an engine upgrade fired during a stack update can fail on a transient condition — most commonly an automated snapshot running at that moment — while CloudFormation records the stack update as successful. Because CloudFormation compares templates against templates, not against the live domain, later deploys never retry; the domain stays behind until someone upgrades it directly. -This article is the direct path: an in-place engine upgrade via the AWS CLI. It is a short sequence in which **every command is read-only except one** — the upgrade trigger itself. Search keeps serving throughout (performance may dip while nodes are replaced; Kibana may be unavailable). Typical duration is minutes to hours depending on data size — plan for the possibility of most of a day on large domains. +This article is the direct path: an in-place engine upgrade via the AWS CLI, in a short sequence where **every command is read-only except one** — the upgrade trigger itself. Search keeps serving throughout (performance may dip while nodes are replaced; Kibana may be unavailable). Expect minutes to hours depending on data size; plan for the possibility of most of a day on large domains. To check whether this applies to you: compare the live engine version with what your template declares. @@ -32,7 +32,7 @@ aws es describe-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ - **Freeze the Quilt stack for the window**: no deploys, no CloudFormation changes, no admin "Re-index and repair" actions while the upgrade runs. The domain must be the only thing changing. -- **Know your escalation path** (insurance only): check which AWS support plan is on the account (the Support Center page shows it) and who could open a technical case. In the rare case an upgrade stalls, the domain keeps serving search — it just refuses further configuration changes until AWS support unsticks it, and Business-or-above support is the channel for that. If the account is on Basic, note that Business support can be enabled in minutes and takes effect immediately, so the practical preparation is knowing who has the authority to approve it if ever needed. +- **Know your escalation path** (insurance only): which AWS support plan is on the account (the Support Center page shows it), and who could open a technical case. In the rare case an upgrade stalls, the domain keeps serving search — it just refuses further changes until AWS support unsticks it, and Business-or-above support is the channel for that. Basic is workable too: Business support enables in minutes with immediate effect, so the real preparation is knowing who can approve it. ## Step 1 — Confirm the target version is a legal hop From 3be1e071e4590b75e81f1ac5d4e4d7b58144fa95 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 13:52:33 +0500 Subject: [PATCH 03/11] Rework article per three-lens review - State irreversibility up front + rollback options (manual snapshot, Quilt S3 rebuild backstop); AWS test-domain suggestion - Verdicts and diagnosis via get-upgrade-history (get-upgrade-status carries no failure reasons, timestamps, or progress); baseline read first so stale entries can't masquerade as fresh ones - Handle SUCCEEDED_WITH_ISSUES, Step-5 CLI errors, early-step failures - Replace wrong-instrument Auto-Tune command with console Notifications glance; yellow-cluster rule; explicit done-criterion - Domain identified via the stack's Search resource, not guessed - Fix stale support-plan naming; self-service stall triage before case - Qualify search-availability claim (dedicated-masters conditional); cost note; scope note vs newer target versions; absolute Related link Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 105 +++++++++++++++----------- 1 file changed, 63 insertions(+), 42 deletions(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index 0e7a643..39a2833 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -4,101 +4,122 @@ `aws`, `elasticsearch`, `opensearch`, `search`, `upgrade`, `cli`, `cloudformation` +*Applies to Quilt releases whose template declares Elasticsearch 7.10 (all current releases).* + ## Summary -Some Quilt deployments run their search domain on Elasticsearch 6.8 even though the deployed CloudFormation template declares 7.10: an engine upgrade fired during a stack update can fail on a transient condition — most commonly an automated snapshot running at that moment — while CloudFormation records the stack update as successful. Because CloudFormation compares templates against templates, not against the live domain, later deploys never retry; the domain stays behind until someone upgrades it directly. +Some Quilt deployments run their search domain on Elasticsearch 6.8 even though the deployed CloudFormation template declares 7.10: an engine upgrade fired during a stack update can fail on a transient condition (such as an automated snapshot running at that moment) while CloudFormation records the stack update as successful. Because CloudFormation's normal update flow compares templates against templates, not against the live domain, later deploys never retry; the domain stays behind until someone upgrades it directly. -This article is the direct path: an in-place engine upgrade via the AWS CLI, in a short sequence where **every command is read-only except one** — the upgrade trigger itself. Search keeps serving throughout (performance may dip while nodes are replaced; Kibana may be unavailable). Expect minutes to hours depending on data size; plan for the possibility of most of a day on large domains. +This article is the direct path: an in-place engine upgrade via the AWS CLI. Only one command in the sequence changes the domain. Two things to know before starting: -To check whether this applies to you: compare the live engine version with what your template declares. +- **The upgrade is irreversible** — AWS states it "can't be paused or cancelled," and there is no downgrade. See the rollback options below before running Step 5. +- With Quilt's standard configuration (dedicated master nodes), search keeps serving through the upgrade, though performance may dip while nodes are replaced and Kibana may be unavailable. Masterless cost-sensitive configurations may additionally see a brief unresponsive period after the upgrade. AWS's guidance: the upgrade takes [15 minutes to several hours](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html); large domains can take longer. -``` -aws es describe-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ - --query 'DomainStatus.ElasticsearchVersion' -``` - -(`aws es list-domain-names` shows your domain name. Quilt support can tell you what your release declares — for current releases it is 7.10.) +The commands use the legacy `aws es` namespace, which matches these domains and takes plain version strings (`7.10`). The newer `aws opensearch` namespace works too, but expects `Elasticsearch_7.10`-style strings. ## Before you start -- **Set the variables** used by every command below: +- **Find your domain and set the variables** every command uses. The search domain belongs to your Quilt CloudFormation stack, logical resource `Search`: + + ``` + aws cloudformation describe-stack-resources --stack-name \ + --query "StackResources[?LogicalResourceId=='Search'].PhysicalResourceId" --output text + ``` ``` - DOMAIN= + DOMAIN= REGION= + TARGET=7.10 ``` -- **Permissions**: the credentials you use need `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeStatus`, `es:DescribeDomainAutoTunes`, and — for the one state-changing step — `es:UpgradeElasticsearchDomain`. + To confirm the drift, compare the live version with the target: + + ``` + aws es describe-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ + --query 'DomainStatus.ElasticsearchVersion' + ``` -- **Freeze the Quilt stack for the window**: no deploys, no CloudFormation changes, no admin "Re-index and repair" actions while the upgrade runs. The domain must be the only thing changing. +- **Permissions**: `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeHistory`, `es:GetUpgradeStatus`, and — for the upgrade itself — `es:UpgradeElasticsearchDomain` (grant the newer spellings too if writing a fresh policy: `es:DescribeDomain`, `es:GetCompatibleVersions`, `es:UpgradeDomain`), plus `cloudformation:DescribeStackResources` for the lookup above. -- **Know your escalation path** (insurance only): which AWS support plan is on the account (the Support Center page shows it), and who could open a technical case. In the rare case an upgrade stalls, the domain keeps serving search — it just refuses further changes until AWS support unsticks it, and Business-or-above support is the channel for that. Basic is workable too: Business support enables in minutes with immediate effect, so the real preparation is knowing who can approve it. +- **Rollback story, before the one-way door**: AWS's own procedure recommends [taking a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html) — it can be restored onto a *new* domain if you ever want the prior version back (automated snapshots can't do this). For a Quilt deployment there is a second, slower backstop: the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but no data is ever lost. Decide which insurance you want; for extra caution AWS also suggests restoring a snapshot onto a test domain first. -## Step 1 — Confirm the target version is a legal hop +- **Freeze the Quilt stack for the window**: no deploys, no CloudFormation changes, no admin "Re-index and repair" actions while the upgrade runs. Note that a stalled upgrade extends this freeze — the domain refuses configuration changes until the upgrade finishes. + +- **Know your escalation path** (insurance only): technical AWS support cases require a paid support plan — Basic can't open them; as of late 2025 the purchasable tier is Business Support+ (in AWS Organizations, support plans are often managed from the payer account — check with whoever owns that). If an upgrade stalls, self-service triage covers most cases (Step 6), with an AWS case as the final step. + +## Step 1 — Read the domain's upgrade history ``` -aws es get-compatible-elasticsearch-versions --domain-name $DOMAIN --region $REGION +aws es get-upgrade-history --domain-name $DOMAIN --region $REGION ``` -From 6.8, the list includes 7.10 — proceed with `TARGET=7.10`. +This is your baseline: it lists past upgrade attempts and eligibility checks with timestamps, step-by-step results, and — unlike `get-upgrade-status` — the actual failure reasons (`Issues`). On a drifted domain you'll typically see the original failed attempt here. Every later step reads this same command to tell fresh results from stale ones, so note what the latest entry is now. -If your domain is on a version **below** 6.8, note that 7.10 will not be in the list: Elasticsearch upgrades one step at a time (for example 6.7 → 6.8, then 6.8 → 7.10). Run this whole procedure once per hop. - -## Step 2 — Check nothing is scheduled to interfere +## Step 2 — Confirm the supported upgrade path ``` -aws es describe-domain-auto-tunes --domain-name $DOMAIN --region $REGION +aws es get-compatible-elasticsearch-versions --domain-name $DOMAIN --region $REGION ``` -If no Auto-Tune action is scheduled for the coming day, proceed. If one is scheduled inside your window, move the window (or contact Quilt support about disabling Auto-Tune — the disable itself has options that matter). +From 6.8 the list includes 7.10 (and newer options). **Pick exactly the version your template declares — 7.10 — even though AWS offers newer**: going past the template turns a harmless lag into a mismatch CloudFormation can never reconcile, since downgrades don't exist. -Also glance at the domain in the AWS console: cluster health should be green and the domain status Active before you continue. +If your domain is below 6.8: within a major version you can jump straight to its last minor (any 6.x → 6.8), but crossing to 7.x requires being on 6.8 first — run this procedure once per hop, and note that ES 6.0–6.7 domains accrue AWS extended-support charges and have an announced end-of-support date, so don't linger there. -## Step 3 — Run the eligibility check +## Step 3 — Quick health glance -This validates the upgrade without performing it — despite the command name, `--perform-check-only` changes nothing on the domain: +In the AWS console, the domain should show status **Active** (CLI: `DomainStatus.Processing` is `false`) and cluster health **green** — or your domain's normal color: single-data-node domains sit yellow permanently, and yellow doesn't block upgrades. **Red does** — stop and resolve that first ([AWS guidance](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html)). Also glance at the console **Notifications** page for any pending maintenance or Auto-Tune action scheduled into your window; if one is there, pick a different window. + +## Step 4 — Run the eligibility check + +This validates without upgrading — AWS's words: it "does not actually perform the upgrade": ``` aws es upgrade-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ --target-version $TARGET --perform-check-only ``` -The check runs asynchronously. Fetch the verdict (repeat until the check completes — usually a few minutes): +The check runs asynchronously, usually a few minutes. Fetch the verdict with Step 1's command — the new entry appears with a fresh timestamp and a name like "Pre-Upgrade Check for Elasticsearch 7.10": ``` -aws es get-upgrade-status --domain-name $DOMAIN --region $REGION +aws es get-upgrade-history --domain-name $DOMAIN --region $REGION ``` -- **`PRE_UPGRADE_CHECK: SUCCEEDED`** → go directly to Step 4, *now*. A passing check also means no automated snapshot is running at this moment — that is exactly the gap you want to launch into, because AWS takes hourly automated snapshots and an upgrade colliding with one fails. -- **Failed with "Prior snapshot operation has not yet completed"** → normal and harmless; an automated snapshot is running. Wait 10–15 minutes and repeat this step. (See [AWS's article on this error](https://repost.aws/knowledge-center/opensearch-prior-snapshot-error).) -- **Failed with anything else** → stop and send the output to Quilt support before proceeding. +- **`SUCCEEDED`** → proceed to Step 5 promptly: a passing check also means no automated snapshot is running right now (AWS takes them hourly, and they block upgrades), so you're in the gap. No need to rush beyond ordinary promptness — if the gap closes, the worst case is a harmless failed attempt and a retry. +- **Failed with a snapshot-related `Issue`** (e.g. "prior snapshot operation has not yet completed") → normal and harmless. Wait 10–15 minutes, re-run the check. +- **Failed with anything else** → the `Issues` text names the condition; most match AWS's [validation-failure table](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) with self-service fixes. Involve Quilt support if the fix touches Quilt-managed resources. +- **`SUCCEEDED_WITH_ISSUES`** → don't proceed; send the full history output to Quilt support first. -## Step 4 — Run the upgrade +## Step 5 — Run the upgrade -The one state-changing command: +The one command that changes the domain: ``` aws es upgrade-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ --target-version $TARGET ``` -## Step 5 — Watch it +It returns immediately with a JSON echo of the request. If it errors instead (for example, the domain slipped into a processing state), nothing has happened — resolve and return to Step 4. + +## Step 6 — Watch it + +Re-run the Step 1 history command occasionally (it shows the current attempt's steps and `ProgressPercent`), or watch the domain page in the console. The upgrade proceeds `PRE_UPGRADE_CHECK` → `SNAPSHOT` → `UPGRADE`. If the upgrade's own early steps fail (the snapshot gap can close), the domain is unharmed and still on 6.8 — return to Step 4. -Re-run the Step 3 status command occasionally, or watch the domain page in the console. The upgrade proceeds through `PRE_UPGRADE_CHECK` → `SNAPSHOT` → `UPGRADE` and takes anywhere from minutes to hours ([AWS's guidance on long-running upgrades](https://repost.aws/knowledge-center/opensearch-domain-upgrade)). +**Done** means: `aws es describe-elasticsearch-domain` shows `ElasticsearchVersion: "7.10"` and `Processing: false`. -If progress sits unchanged for several hours: **don't touch anything**. The domain keeps serving search while stuck; the recovery path is an AWS support case ([AWS's article on stuck upgrades](https://repost.aws/knowledge-center/opensearch-stuck-failed-upgrade)) — and contact Quilt support, we've been through this and will help draft the case. +If `ProgressPercent` sits unchanged for several hours: the domain typically keeps serving search meanwhile, and AWS's [stuck-upgrade guidance](https://repost.aws/knowledge-center/opensearch-stuck-failed-upgrade) prescribes self-service triage first — check `FreeStorageSpace`, cluster status, and JVM pressure in CloudWatch, free disk if it's low — and an AWS technical support case if none of that resolves it. Quilt support can help assemble the details AWS will ask for. -## Step 6 — Smoke test +One cost note: the upgrade is a blue/green deployment, so AWS bills both node fleets for the first hour — a one-time blip, not a rate change. -After the status shows the upgrade succeeded: +## Step 7 — Confirm and clean up -1. Upload a small file to any bucket registered in your Quilt catalog. -2. Confirm it appears in catalog search within a couple of minutes. -3. Run a search you know should return results. +1. Re-run the version check (see **Done** above) — this is the fact you came for. +2. Upload a small test file to a registered bucket you're comfortable writing to, and confirm it appears in catalog search. Right after an upgrade the indexing backlog may need time — give it up to ~15 minutes before reading anything into a delay. Delete the test file when done. (Read-only alternative: run a search whose results you know.) +3. Nothing else is needed: S3 event notifications queue and retry, so objects uploaded during the window get indexed without any re-index step, and there is nothing to re-enable. +4. Tell Quilt support the upgrade completed — your stack's standard telemetry reports daily index metrics, so we can confirm indexing health from those and plan any follow-up work with you. -If anything looks off, contact Quilt support — and tell us the upgrade completed either way, so we can verify from our side and plan any follow-up work with you. +If you run your own clients, dashboards, or saved Kibana objects against the domain (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) — Quilt's own software and indices are validated on 7.10. ## Related - [Upgrading Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) — AWS's reference for the upgrade process -- [How do I collect search-cluster diagnostics for Quilt support?](howto-collect-search-cluster-diagnostics.md) +- [How do I collect search-cluster diagnostics for Quilt support?](https://github.com/quiltdata/knowledge-base/blob/main/howto-collect-search-cluster-diagnostics.md) From 3a6d72e2bfa7f3794708e1ab98dbfcf5adf98c78 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 15:36:40 +0500 Subject: [PATCH 04/11] Scope to CFN deployments + old-release warning; honest snapshot limits; billing fix; breaking-changes check moved to preflight; drop telemetry mention Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index 39a2833..39de9a8 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -4,7 +4,7 @@ `aws`, `elasticsearch`, `opensearch`, `search`, `upgrade`, `cli`, `cloudformation` -*Applies to Quilt releases whose template declares Elasticsearch 7.10 (all current releases).* +*Applies to CloudFormation-based Quilt deployments running releases whose template declares Elasticsearch 7.10 (all current releases). If your release still declares 6.8, upgrade Quilt first — never move the domain ahead of its template. Terraform-based deployments manage the domain through Terraform — contact Quilt support instead of following this article.* ## Summary @@ -41,7 +41,9 @@ The commands use the legacy `aws es` namespace, which matches these domains and - **Permissions**: `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeHistory`, `es:GetUpgradeStatus`, and — for the upgrade itself — `es:UpgradeElasticsearchDomain` (grant the newer spellings too if writing a fresh policy: `es:DescribeDomain`, `es:GetCompatibleVersions`, `es:UpgradeDomain`), plus `cloudformation:DescribeStackResources` for the lookup above. -- **Rollback story, before the one-way door**: AWS's own procedure recommends [taking a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html) — it can be restored onto a *new* domain if you ever want the prior version back (automated snapshots can't do this). For a Quilt deployment there is a second, slower backstop: the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but no data is ever lost. Decide which insurance you want; for extra caution AWS also suggests restoring a snapshot onto a test domain first. +- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: it can only be restored onto a *new* domain your stack isn't wired to, and it goes stale as soon as indexing continues — useful for extra caution (or a test-domain dry run), not a practical undo. + +- **If you run your own clients, dashboards, or saved Kibana objects against the domain** (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) before deciding to upgrade — Quilt's own software and indices are validated on 7.10. - **Freeze the Quilt stack for the window**: no deploys, no CloudFormation changes, no admin "Re-index and repair" actions while the upgrade runs. Note that a stalled upgrade extends this freeze — the domain refuses configuration changes until the upgrade finishes. @@ -108,16 +110,14 @@ Re-run the Step 1 history command occasionally (it shows the current attempt's s If `ProgressPercent` sits unchanged for several hours: the domain typically keeps serving search meanwhile, and AWS's [stuck-upgrade guidance](https://repost.aws/knowledge-center/opensearch-stuck-failed-upgrade) prescribes self-service triage first — check `FreeStorageSpace`, cluster status, and JVM pressure in CloudWatch, free disk if it's low — and an AWS technical support case if none of that resolves it. Quilt support can help assemble the details AWS will ask for. -One cost note: the upgrade is a blue/green deployment, so AWS bills both node fleets for the first hour — a one-time blip, not a rate change. +One cost note: the upgrade is a blue/green deployment; [AWS charges for the largest cluster during the first hour only](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html), then for a single cluster even while the deployment continues — a one-time blip, not a rate change. ## Step 7 — Confirm and clean up 1. Re-run the version check (see **Done** above) — this is the fact you came for. 2. Upload a small test file to a registered bucket you're comfortable writing to, and confirm it appears in catalog search. Right after an upgrade the indexing backlog may need time — give it up to ~15 minutes before reading anything into a delay. Delete the test file when done. (Read-only alternative: run a search whose results you know.) 3. Nothing else is needed: S3 event notifications queue and retry, so objects uploaded during the window get indexed without any re-index step, and there is nothing to re-enable. -4. Tell Quilt support the upgrade completed — your stack's standard telemetry reports daily index metrics, so we can confirm indexing health from those and plan any follow-up work with you. - -If you run your own clients, dashboards, or saved Kibana objects against the domain (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) — Quilt's own software and indices are validated on 7.10. +4. Tell Quilt support the upgrade completed — a copy of the version-check output is perfect — so we can plan any follow-up work with you. ## Related From da4b0d493196ce8aa0becb90658db4c8496bab8e Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 15:39:19 +0500 Subject: [PATCH 05/11] Unpack the snapshot-restore limitation in plain terms Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index 39de9a8..0ce29f3 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -41,7 +41,7 @@ The commands use the legacy `aws es` namespace, which matches these domains and - **Permissions**: `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeHistory`, `es:GetUpgradeStatus`, and — for the upgrade itself — `es:UpgradeElasticsearchDomain` (grant the newer spellings too if writing a fresh policy: `es:DescribeDomain`, `es:GetCompatibleVersions`, `es:UpgradeDomain`), plus `cloudformation:DescribeStackResources` for the lookup above. -- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: it can only be restored onto a *new* domain your stack isn't wired to, and it goes stale as soon as indexing continues — useful for extra caution (or a test-domain dry run), not a practical undo. +- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: a snapshot can't downgrade the domain — restoring it means standing up a second, separate domain on the old version, and your Quilt deployment's services all point at the original, with no supported way to re-point them. It preserves a copy of the data, not your working deployment — useful for extra caution or a test-domain dry run, not a practical undo. - **If you run your own clients, dashboards, or saved Kibana objects against the domain** (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) before deciding to upgrade — Quilt's own software and indices are validated on 7.10. From 68d7e1d21c79dd6c6166b0b08bac5a39d6fb3b0c Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 15:42:40 +0500 Subject: [PATCH 06/11] Snapshot limits: restore the staleness clause Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index 0ce29f3..bfab172 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -41,7 +41,7 @@ The commands use the legacy `aws es` namespace, which matches these domains and - **Permissions**: `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeHistory`, `es:GetUpgradeStatus`, and — for the upgrade itself — `es:UpgradeElasticsearchDomain` (grant the newer spellings too if writing a fresh policy: `es:DescribeDomain`, `es:GetCompatibleVersions`, `es:UpgradeDomain`), plus `cloudformation:DescribeStackResources` for the lookup above. -- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: a snapshot can't downgrade the domain — restoring it means standing up a second, separate domain on the old version, and your Quilt deployment's services all point at the original, with no supported way to re-point them. It preserves a copy of the data, not your working deployment — useful for extra caution or a test-domain dry run, not a practical undo. +- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: a snapshot can't downgrade the domain — restoring it means standing up a second, separate domain on the old version, and your Quilt deployment's services all point at the original, with no supported way to re-point them. It preserves a copy of the data as of the moment it was taken — indexing continues past it — not your working deployment: useful for extra caution or a test-domain dry run, not a practical undo. - **If you run your own clients, dashboards, or saved Kibana objects against the domain** (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) before deciding to upgrade — Quilt's own software and indices are validated on 7.10. From 013647e5c5601657e300454ce2b48792cc203909 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 15:46:38 +0500 Subject: [PATCH 07/11] Snapshot restore: re-pointing is guided-migration surgery, not impossible Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index bfab172..e0acfa6 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -41,7 +41,7 @@ The commands use the legacy `aws es` namespace, which matches these domains and - **Permissions**: `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeHistory`, `es:GetUpgradeStatus`, and — for the upgrade itself — `es:UpgradeElasticsearchDomain` (grant the newer spellings too if writing a fresh policy: `es:DescribeDomain`, `es:GetCompatibleVersions`, `es:UpgradeDomain`), plus `cloudformation:DescribeStackResources` for the lookup above. -- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: a snapshot can't downgrade the domain — restoring it means standing up a second, separate domain on the old version, and your Quilt deployment's services all point at the original, with no supported way to re-point them. It preserves a copy of the data as of the moment it was taken — indexing continues past it — not your working deployment: useful for extra caution or a test-domain dry run, not a practical undo. +- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: a snapshot can't downgrade the domain — restoring it means standing up a second, separate domain on the old version, and re-pointing your Quilt deployment's services at it is not a self-service operation (it's the kind of template surgery Quilt engineers in guided migrations — a project, not an undo button). It preserves a copy of the data as of the moment it was taken — indexing continues past it — not your working deployment: useful for extra caution or a test-domain dry run, not a practical undo. - **If you run your own clients, dashboards, or saved Kibana objects against the domain** (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) before deciding to upgrade — Quilt's own software and indices are validated on 7.10. From 4dc6d19f2002aabae5dde182f286c811ffba92e9 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 15:50:32 +0500 Subject: [PATCH 08/11] Rollback paragraph: safety net first, one-way door second Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index e0acfa6..e483a35 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -41,7 +41,7 @@ The commands use the legacy `aws es` namespace, which matches these domains and - **Permissions**: `es:DescribeElasticsearchDomain`, `es:GetCompatibleElasticsearchVersions`, `es:GetUpgradeHistory`, `es:GetUpgradeStatus`, and — for the upgrade itself — `es:UpgradeElasticsearchDomain` (grant the newer spellings too if writing a fresh policy: `es:DescribeDomain`, `es:GetCompatibleVersions`, `es:UpgradeDomain`), plus `cloudformation:DescribeStackResources` for the lookup above. -- **Rollback story, before the one-way door**: for a Quilt deployment, the honest backstop is that the search index is derived data, rebuildable from S3 via a full re-index — days, not minutes, but nothing is ever lost. AWS's generic advice is [a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html); know its limits here: a snapshot can't downgrade the domain — restoring it means standing up a second, separate domain on the old version, and re-pointing your Quilt deployment's services at it is not a self-service operation (it's the kind of template surgery Quilt engineers in guided migrations — a project, not an undo button). It preserves a copy of the data as of the moment it was taken — indexing continues past it — not your working deployment: useful for extra caution or a test-domain dry run, not a practical undo. +- **Irreversible, but self-protecting**: there's no downgrade, yet the failure paths are covered. If the upgrade *fails*, [AWS restores the cluster automatically](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) from the snapshot it takes as part of the upgrade — you're back where you started, unharmed. And the data isn't at stake either way: the search index is derived from your S3 buckets and can always be rebuilt in full (a days-long re-index — the backstop of last resort, not a likely event). What doesn't exist is a way back after a *successful* upgrade — which you shouldn't need: 7.10 is the version your Quilt release was built for. AWS's generic advice to [take a manual snapshot first](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/starting-upgrades.html) is optional extra caution here — it preserves a data copy as of that moment (indexing continues past it), restorable only onto a separate domain, and re-pointing a Quilt deployment at a different domain is a guided-migration project, not an undo button. - **If you run your own clients, dashboards, or saved Kibana objects against the domain** (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) before deciding to upgrade — Quilt's own software and indices are validated on 7.10. From edae7a067a874b71f22ab3cc3b303bb943a0c40f Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 16:01:17 +0500 Subject: [PATCH 09/11] Retitle to the drift question; orientation line; define 'stalled' at first use Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index e483a35..67cd9a8 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -1,4 +1,4 @@ -# How do I upgrade my Quilt deployment's search domain engine (Elasticsearch 6.8 → 7.10)? +# Why is my Quilt deployment's search domain still on Elasticsearch 6.8, and how do I upgrade it? ## Tags @@ -8,7 +8,7 @@ ## Summary -Some Quilt deployments run their search domain on Elasticsearch 6.8 even though the deployed CloudFormation template declares 7.10: an engine upgrade fired during a stack update can fail on a transient condition (such as an automated snapshot running at that moment) while CloudFormation records the stack update as successful. Because CloudFormation's normal update flow compares templates against templates, not against the live domain, later deploys never retry; the domain stays behind until someone upgrades it directly. +Normally you never upgrade the search engine yourself — it rides Quilt's template upgrades. This article covers the exception. Some Quilt deployments run their search domain on Elasticsearch 6.8 even though the deployed CloudFormation template declares 7.10: an engine upgrade fired during a stack update can fail on a transient condition (such as an automated snapshot running at that moment) while CloudFormation records the stack update as successful. Because CloudFormation's normal update flow compares templates against templates, not against the live domain, later deploys never retry; the domain stays behind until someone upgrades it directly. This article is the direct path: an in-place engine upgrade via the AWS CLI. Only one command in the sequence changes the domain. Two things to know before starting: @@ -45,9 +45,9 @@ The commands use the legacy `aws es` namespace, which matches these domains and - **If you run your own clients, dashboards, or saved Kibana objects against the domain** (most deployments don't), review the [Elasticsearch 7 breaking changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) before deciding to upgrade — Quilt's own software and indices are validated on 7.10. -- **Freeze the Quilt stack for the window**: no deploys, no CloudFormation changes, no admin "Re-index and repair" actions while the upgrade runs. Note that a stalled upgrade extends this freeze — the domain refuses configuration changes until the upgrade finishes. +- **Freeze the Quilt stack for the window**: no deploys, no CloudFormation changes, no admin "Re-index and repair" actions while the upgrade runs. If the upgrade were ever to stall — stop making progress for hours; rare, and Step 6 covers the response — the freeze extends until it's resolved, because a domain mid-upgrade refuses further configuration changes. -- **Know your escalation path** (insurance only): technical AWS support cases require a paid support plan — Basic can't open them; as of late 2025 the purchasable tier is Business Support+ (in AWS Organizations, support plans are often managed from the payer account — check with whoever owns that). If an upgrade stalls, self-service triage covers most cases (Step 6), with an AWS case as the final step. +- **Know your escalation path** (insurance only): technical AWS support cases require a paid support plan — Basic can't open them; as of late 2025 the purchasable tier is Business Support+ (in AWS Organizations, support plans are often managed from the payer account — check with whoever owns that). For a stalled upgrade, self-service triage covers most cases (Step 6), with an AWS case as the final step. ## Step 1 — Read the domain's upgrade history From dafcf494fd2bc2c1f2bf7018eab3602cca75a37c Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 16:18:41 +0500 Subject: [PATCH 10/11] Fold rehearsal results in: observed ~50-min empty-domain duration, real verdict-entry JSON, mid-flight progress example Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index 67cd9a8..202e970 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -13,7 +13,7 @@ Normally you never upgrade the search engine yourself — it rides Quilt's templ This article is the direct path: an in-place engine upgrade via the AWS CLI. Only one command in the sequence changes the domain. Two things to know before starting: - **The upgrade is irreversible** — AWS states it "can't be paused or cancelled," and there is no downgrade. See the rollback options below before running Step 5. -- With Quilt's standard configuration (dedicated master nodes), search keeps serving through the upgrade, though performance may dip while nodes are replaced and Kibana may be unavailable. Masterless cost-sensitive configurations may additionally see a brief unresponsive period after the upgrade. AWS's guidance: the upgrade takes [15 minutes to several hours](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html); large domains can take longer. +- With Quilt's standard configuration (dedicated master nodes), search keeps serving through the upgrade, though performance may dip while nodes are replaced and Kibana may be unavailable. Masterless cost-sensitive configurations may additionally see a brief unresponsive period after the upgrade. AWS's guidance is [15 minutes to several hours](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html); in our test, a fresh *empty* domain took about 50 minutes end-to-end — data size scales that up, so plan for the possibility of most of a day on large domains. The commands use the legacy `aws es` namespace, which matches these domains and takes plain version strings (`7.10`). The newer `aws opensearch` namespace works too, but expects `Elasticsearch_7.10`-style strings. @@ -80,12 +80,21 @@ aws es upgrade-elasticsearch-domain --domain-name $DOMAIN --region $REGION \ --target-version $TARGET --perform-check-only ``` -The check runs asynchronously, usually a few minutes. Fetch the verdict with Step 1's command — the new entry appears with a fresh timestamp and a name like "Pre-Upgrade Check for Elasticsearch 7.10": +The check runs asynchronously, usually a minute or two. Fetch the verdict with Step 1's command — a new entry appears with a fresh timestamp: ``` aws es get-upgrade-history --domain-name $DOMAIN --region $REGION ``` +```json +{ + "UpgradeName": "Pre-Upgrade Check from 6.8 to 7.10", + "StartTimestamp": "2026-08-05T10:19:29+00:00", + "UpgradeStatus": "SUCCEEDED", + "StepsList": [ { "UpgradeStep": "PRE_UPGRADE_CHECK", "UpgradeStepStatus": "SUCCEEDED", ... } ] +} +``` + - **`SUCCEEDED`** → proceed to Step 5 promptly: a passing check also means no automated snapshot is running right now (AWS takes them hourly, and they block upgrades), so you're in the gap. No need to rush beyond ordinary promptness — if the gap closes, the worst case is a harmless failed attempt and a retry. - **Failed with a snapshot-related `Issue`** (e.g. "prior snapshot operation has not yet completed") → normal and harmless. Wait 10–15 minutes, re-run the check. - **Failed with anything else** → the `Issues` text names the condition; most match AWS's [validation-failure table](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) with self-service fixes. Involve Quilt support if the fix touches Quilt-managed resources. @@ -104,7 +113,7 @@ It returns immediately with a JSON echo of the request. If it errors instead (fo ## Step 6 — Watch it -Re-run the Step 1 history command occasionally (it shows the current attempt's steps and `ProgressPercent`), or watch the domain page in the console. The upgrade proceeds `PRE_UPGRADE_CHECK` → `SNAPSHOT` → `UPGRADE`. If the upgrade's own early steps fail (the snapshot gap can close), the domain is unharmed and still on 6.8 — return to Step 4. +Re-run the Step 1 history command occasionally (it shows the current attempt's steps and `ProgressPercent` — e.g. `"UpgradeStep": "UPGRADE", "UpgradeStepStatus": "IN_PROGRESS", "ProgressPercent": 55.0` mid-flight), or watch the domain page in the console. The upgrade proceeds `PRE_UPGRADE_CHECK` → `SNAPSHOT` → `UPGRADE`. If the upgrade's own early steps fail (the snapshot gap can close), the domain is unharmed and still on 6.8 — return to Step 4. **Done** means: `aws es describe-elasticsearch-domain` shows `ElasticsearchVersion: "7.10"` and `Processing: false`. From 15ad302bed9b738b9a0f8263c8c9a9f28d11ae14 Mon Sep 17 00:00:00 2001 From: Sergey Fedoseev Date: Wed, 5 Aug 2026 16:33:12 +0500 Subject: [PATCH 11/11] Quote placeholder values so copy-paste can't hit shell redirection (Greptile) Co-Authored-By: Claude Fable 5 --- howto-upgrade-search-domain-engine.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/howto-upgrade-search-domain-engine.md b/howto-upgrade-search-domain-engine.md index 202e970..e3787de 100644 --- a/howto-upgrade-search-domain-engine.md +++ b/howto-upgrade-search-domain-engine.md @@ -22,13 +22,13 @@ The commands use the legacy `aws es` namespace, which matches these domains and - **Find your domain and set the variables** every command uses. The search domain belongs to your Quilt CloudFormation stack, logical resource `Search`: ``` - aws cloudformation describe-stack-resources --stack-name \ + aws cloudformation describe-stack-resources --stack-name "your-quilt-stack-name" \ --query "StackResources[?LogicalResourceId=='Search'].PhysicalResourceId" --output text ``` ``` - DOMAIN= - REGION= + DOMAIN="the-domain-name-from-above" + REGION="your-stack-region" TARGET=7.10 ```