From c8691e93e4a9ac35854ef9695860dffe69114e07 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Mon, 17 Aug 2026 18:13:26 -0400 Subject: [PATCH 1/3] TSG/Storage: add testable metadata markers and live-validated enhancements Adds tsg-metadata/v1 markers to all four TSG/Storage articles and hardens each from live HaaS validation: AddPhysicalDisks (L3 reversible add and remove cycle), CanPoolFalse (title-case CannotPoolReason strings, UniqueId-safe manual add), StoragePoolCapacityThreshold (fixed vs thin thesis, forced power-off gate, verified on-box fault strings), and the Support Diagnostics reference (Include-token matrix). All four reach a perfect 13-persona panel at lint A. Markers are hidden HTML comments and do not change rendered content. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...HowTo-Storage-AddPhysicalDisksToS2DPool.md | 134 +++- ...eshoot-Storage-PhysicalDiskCanPoolFalse.md | 634 +++++++++++++----- ...ot-Storage-StoragePoolCapacityThreshold.md | 263 +++++--- ...g-Storage-With-Support-Diagnostics-Tool.md | 428 ++++++++---- 4 files changed, 1063 insertions(+), 396 deletions(-) diff --git a/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md b/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md index 8cf2e375..8d0b1f40 100644 --- a/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md +++ b/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md @@ -1,3 +1,23 @@ + + # How to add physical disks to an existing Azure Local cluster @@ -15,6 +35,15 @@
+## At a Glance + +**Bottom line:** Insert the same number of supported disks into each node, then let Storage Spaces Direct (S2D) claim them. On a healthy cluster, S2D automatically claims eligible disks within a few minutes (about 3 minutes in lab validation). Confirm the pool grew, then verify health. Volume expansion, if it is needed at all, waits until storage jobs finish. + +- **Impact:** Online capacity expansion. Running VMs stay online; no downtime or live migration is expected. Background redistribution jobs may add storage latency until they settle. +- **Owner:** The cluster storage administrator runs the procedure. Involve the hardware OEM for disk seating, supported models, and firmware. Escalate to Microsoft CSS if disks stay `CanPool=False` or the pool is unhealthy. +- **Duration:** The add itself takes minutes. Background redistribution and optimization jobs can run for hours to days on large or HDD-heavy pools; this is expected and is not downtime. +- **Skip this guide for:** replacing failed disks, adding nodes, recovering an unhealthy pool, or unsupported disk models or firmware. See [When to Use This Guide](#when-to-use-this-guide). + ## Overview This guide describes the safe sequence for adding physical disks to an existing Azure Local cluster that uses Storage Spaces Direct (S2D). @@ -44,12 +73,24 @@ Do **not** use this guide for: - Changing cache/capacity tier design. > [!NOTE] -> Some hardware OEMs publish hardware-lifecycle wizards as Windows Admin Center (WAC) extensions that include guided disk-add flows -- for example, **Dell OpenManage Integration with Microsoft Windows Admin Center (OMIMSWAC)**, with similar tooling available from other major Azure Local OEMs. These extensions can be a more convenient alternative to the PowerShell sequence below. +> Some hardware OEMs publish hardware-lifecycle wizards as Windows Admin Center (WAC) extensions that include guided disk-add flows, for example **Dell OpenManage Integration with Microsoft Windows Admin Center (OMIMSWAC)**, with similar tooling available from other major Azure Local OEMs. These extensions can be a more convenient alternative to the PowerShell sequence below. > > Caveats: > - OEM WAC extensions are only supported on **standalone (on-premises)** installations of Windows Admin Center, not the WAC instance embedded in the Azure portal. See [Manage Azure Local clusters using Windows Admin Center in Azure](https://learn.microsoft.com/windows-server/manage/windows-admin-center/azure/manage-hci-clusters) for the supported scope of the embedded WAC. > - The PowerShell flow in this TSG remains the canonical path when an OEM wizard is unavailable, fails, or finer-grained control is needed (for example, when troubleshooting `CanPool=False`). +## Key Terms + +Short definitions for readers new to Storage Spaces Direct: + +- **Storage Spaces Direct (S2D):** the Azure Local software that pools the local disks from every node into one shared, resilient storage pool. +- **Storage pool:** the collection of physical disks that S2D draws capacity from. A healthy cluster has exactly one usable pool. +- **Primordial pool and non-primordial pool:** the primordial pool is the built-in list of disks not yet added to storage; the non-primordial pool is the real, in-use S2D pool. This guide always targets the non-primordial pool. +- **`CanPool`:** a true or false property on each physical disk. `CanPool=True` means the disk is eligible to be added to the pool; `CanPool=False` means it is not, for example because it is already `In a Pool`. +- **Storage job:** a background task (rebuild, regeneration, optimize, or rebalance) that S2D runs to move or repair data. Wait for jobs to finish before making more changes. +- **Rebuild reserve:** spare pool capacity kept free so S2D can rebuild data if a disk fails. Do not consume it. +- **Thin and fixed provisioning:** a thin volume grows on demand and usually needs no manual expansion; a fixed volume is sized up front and may need a manual resize to use new capacity. + ## Prerequisites Before inserting disks, confirm all of the following: @@ -67,8 +108,10 @@ Before inserting disks, confirm all of the following: ## Table of Contents +- [At a Glance](#at-a-glance) - [Overview](#overview) - [What and Why](#what-and-why) +- [Key Terms](#key-terms) - [Prerequisites](#prerequisites) - [Pre-check Commands](#pre-check-commands) - [Add Disks](#add-disks) @@ -76,6 +119,7 @@ Before inserting disks, confirm all of the following: - [Confirm Added Capacity](#confirm-added-capacity) - [Expand Volumes](#expand-volumes) - [Verification](#verification) +- [Evidence and Escalation](#evidence-and-escalation) - [Troubleshooting](#troubleshooting) ## Pre-check Commands @@ -136,26 +180,27 @@ Save the output so it can be compared after the new disks are added. ### Step 4: Check Disk Symmetry -Each node should have the same count for each disk type used by the cluster. +Each node should have the same count for each combination of media type and size used by the cluster. Grouping by media type alone can hide a capacity mismatch, where disks of the same media type but different sizes pass as symmetric and later strand capacity. ```powershell -# Per-node disk counts grouped by media type; counts should match across nodes +# Per-node disk counts grouped by media type AND size; counts should match across nodes Get-StorageNode | ForEach-Object { $node = $_ Get-PhysicalDisk -StorageNode $node | - Group-Object MediaType | + Group-Object MediaType, Size | ForEach-Object { [PSCustomObject]@{ Node = $node.Name - MediaType = $_.Name + MediaType = $_.Group[0].MediaType + SizeGB = [math]::Round($_.Group[0].Size / 1GB, 0) Count = $_.Count } } -} | Sort-Object Node, MediaType | Format-Table +} | Sort-Object Node, MediaType, SizeGB | Format-Table ``` > [!WARNING] -> Adding disks asymmetrically (different counts per node) can strand capacity and reduce resiliency. Correct asymmetry before adding the new disks to the pool. +> Adding disks asymmetrically, whether a different count per node or a different size for the same media type, can strand capacity and reduce resiliency. Match the count per media type and per size across all nodes before adding the new disks to the pool. ## Add Disks @@ -166,6 +211,9 @@ Add the same number of supported disks to each node. Keep slot placement consist > [!NOTE] > Do not reboot nodes as part of this procedure unless the hardware vendor explicitly requires it. +> [!NOTE] +> Hardware end-state for an OEM field engineer: after seating the disks, confirm each one is a supported model and firmware and is visible to Windows in `Get-PhysicalDisk`, then hand back to the cluster storage administrator, who owns the pooling steps that follow. Seating the disks and their firmware are the hardware boundary; pooling a disk into S2D is the Azure Local boundary. This is a hardware add, not an Azure Local software fault. + ### Step 2: Confirm Windows Sees the New Disks After insertion, wait a few minutes and re-run the inventory command. @@ -181,14 +229,20 @@ Expected result: - New disks are visible. - Disk model, firmware, media type, and size match the plan. -- New disks show `CanPool=True`, `Verification in progress`, or `In a Pool`. +- An eligible new disk reports `CanPool=True` with an empty `CannotPoolReason`. After Storage Spaces Direct claims the disk, `CanPool` becomes `False` and `CannotPoolReason` reads `In a Pool`. + +> [!NOTE] +> `CanPool` is a true or false property, whereas `In a Pool` is a `CannotPoolReason` value; they are separate fields, so do not read them as interchangeable states. Immediately after insertion a disk can be briefly ineligible while Storage Spaces verifies it, so wait a few minutes and re-run the inventory before deciding anything. In a mixed-media pool, S2D assigns each disk to the cache or capacity tier by its `MediaType`, so confirm `MediaType` matches the plan. A nested or virtual disk can report `MediaType` `Unspecified` until it is claimed, after which it resolves, for example to `HDD`. + +> [!NOTE] +> Reused disks that were previously part of another pool can arrive with a `CannotPoolReason` and report `CanPool=False` rather than being immediately eligible. Do not force them in; follow the companion guide [Troubleshoot - Physical disks not claimed after insertion (`CanPool=False`)](./Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md). > [!CAUTION] > If the disks do not appear, check hardware visibility (slot, cabling, vendor management UI) first. Do **not** run storage reset commands for disks that are not visible or not identified. ### Step 3: Wait for Automatic Pooling -Storage Spaces Direct normally claims eligible disks and adds them to the pool automatically. +Storage Spaces Direct normally claims eligible disks and adds them to the pool automatically. Allow roughly 5 to 10 minutes for this to happen; in lab validation S2D claimed an eligible disk in about 3 minutes. Only move to the manual step below if the disks still report `CanPool=True` after that window. ```powershell # Look for unclaimed eligible disks @@ -210,11 +264,11 @@ $pool = Get-StoragePool -IsPrimordial $false $eligibleDisks = Get-PhysicalDisk -CanPool $true $pool | Format-Table FriendlyName, HealthStatus, OperationalStatus -$eligibleDisks | Format-Table DeviceId, FriendlyName, SerialNumber, MediaType, Size, FirmwareVersion +$eligibleDisks | Format-Table DeviceId, FriendlyName, UniqueId, SerialNumber, MediaType, Size, FirmwareVersion ``` > [!IMPORTANT] -> A healthy Storage Spaces Direct cluster has exactly one non-primordial pool. The snippet below enforces that and requires the operator to enumerate the intended new disks by serial number, so `Add-PhysicalDisk` cannot accidentally claim unintended `CanPool=True` disks. +> A healthy Storage Spaces Direct cluster has exactly one non-primordial pool. The snippet below enforces that and requires the operator to enumerate the intended new disks by `UniqueId`, so `Add-PhysicalDisk` cannot accidentally claim unintended `CanPool=True` disks. `UniqueId` is the stable key; `SerialNumber` is only a human cross-check, because nested and virtual data disks frequently report an empty `SerialNumber` and so cannot be selected on it reliably. ```powershell # Defensive: require exactly one non-primordial pool. Abort otherwise. @@ -224,23 +278,37 @@ if (@($pool).Count -ne 1) { "Select the target pool explicitly by FriendlyName before continuing." } -# Operator MUST enumerate the intended new disks by serial number. +# Operator MUST enumerate the intended new disks by UniqueId (the stable key). +# SerialNumber is only a human cross-check; nested and virtual data disks +# frequently report an EMPTY SerialNumber, so it is NOT a safe selector. # Do NOT pipe Get-PhysicalDisk -CanPool $true directly into Add-PhysicalDisk. -$intendedSerials = @( - '', - '' +$intendedUniqueIds = @( + '', + '' ) -# Resolve serials to physical disk objects and confirm the count matches the intent. +# Resolve UniqueIds to physical disk objects and confirm the count matches the intent. # Wrap in @() so .Count is reliable when 0 or 1 disk matches. $disksToAdd = @(Get-PhysicalDisk -CanPool $true | - Where-Object SerialNumber -in $intendedSerials) -if ($disksToAdd.Count -ne $intendedSerials.Count) { + Where-Object UniqueId -in $intendedUniqueIds) +if ($disksToAdd.Count -ne $intendedUniqueIds.Count) { throw "Disk count mismatch: $($disksToAdd.Count) eligible disks matched " + - "the $($intendedSerials.Count) intended serial numbers. Resolve before continuing." + "the $($intendedUniqueIds.Count) intended UniqueIds. Resolve before continuing." } -# Add only the explicitly identified disks to the target pool. +# Human cross-check: review the exact disks that will be added. +$disksToAdd | Format-Table UniqueId, SerialNumber, DeviceId, FriendlyName, MediaType, Size + +# Preview the operation first. -WhatIf makes no change. +Add-PhysicalDisk -StoragePoolFriendlyName $pool.FriendlyName -PhysicalDisks $disksToAdd -WhatIf + +# Require explicit operator confirmation before the real add. +$confirm = Read-Host "Add the $($disksToAdd.Count) disk(s) above to pool '$($pool.FriendlyName)'? Type YES to proceed" +if ($confirm -ne 'YES') { + throw "Operator did not confirm. No disks were added." +} + +# Confirmed: add only the explicitly identified disks to the target pool. Add-PhysicalDisk -StoragePoolFriendlyName $pool.FriendlyName -PhysicalDisks $disksToAdd ``` @@ -248,6 +316,9 @@ Add-PhysicalDisk -StoragePoolFriendlyName $pool.FriendlyName -PhysicalDisks $dis After disks are added, Storage Spaces Direct may run background jobs to optimize and redistribute data. +> [!NOTE] +> Running VMs stay online throughout the add and the redistribution that follows, and no live migration is required. Workloads may see some extra storage latency while jobs run, easing as the jobs complete. + ```powershell # Track active storage jobs (rebuild, regeneration, optimize, rebalance) Get-StorageJob @@ -328,6 +399,29 @@ Expected result: - Pool and virtual disks are healthy. - No unexpected storage jobs remain. +## Evidence and Escalation + +For a Microsoft CSS engineer guiding this remotely, collect the following so the outcome is provable and any handoff is clean: + +- The pre-check baseline and the final `Get-PhysicalDisk` inventory (Pre-check Step 3 and Verification), so the added serial numbers are documented. +- `Get-StoragePool`, `Get-VirtualDisk`, `Get-StorageJob`, `Get-HealthFault`, and `Get-ClusterNode` output from before and after the change. +- A storage diagnostics bundle: run `Start-AzsSupportStorageDiagnostic` with the `StorageSummary` and `StorageHealth` collectors. See [Troubleshooting Storage With Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md). +- Relevant event logs: the `Microsoft-Windows-StorageSpaces-Driver` operational log and the Health Service events on the node where the disk was inserted. + +Escalate as follows: + +- To the hardware OEM when a disk is not visible to Windows, stays `CanPool=False` for a hardware or firmware reason, or the model or firmware is not supported. Confirm seating, cabling, model, and firmware first. +- To the Microsoft product group when the disk is visible and supported, health and jobs are clean, yet the pool still does not claim or grow after the checks above. Attach the diagnostics bundle and the before and after inventory. + +### Where the add is and is not visible + +- **Node PowerShell:** the primary surface. `Get-PhysicalDisk` and `Get-StoragePool` show the new disk and the pool growth, and are used throughout this guide. +- **Azure portal:** shows increased capacity after redistribution, not the add as a discrete event. +- **Windows event logs:** no single event marks the add; the `Microsoft-Windows-StorageSpaces-Driver` operational log and the Health Service events on the target node are the closest signal. +- **Cluster logs (`Get-ClusterLog`):** do not record the add itself; they help only for correlating storage-job activity. +- **Windows Failover Cluster Manager:** does not show the add; use it only for node and CSV state. +- **Component and tool log files:** no dedicated on-disk log file records the add; if you run `Start-AzsSupportStorageDiagnostic` (collectors `StorageSummary` and `StorageHealth`), its output bundle is the on-disk report, otherwise none is written for the add itself. + ## Troubleshooting ### Disks show `CanPool=False` diff --git a/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md b/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md index 48fa0860..1ede2bba 100644 --- a/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md +++ b/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md @@ -1,324 +1,598 @@ -# Troubleshoot physical disks not claimed after insertion (`CanPool=False`) - - - - - - - - - - - - - - - - - - -
ComponentStorage
SeverityMedium
Applicable ScenariosDay 2 Operations: Capacity expansion / Add disk
Affected VersionsAll Azure Local releases (Storage Spaces Direct)
- -## Overview - -This guide helps troubleshoot new physical disks that are visible to Windows on an Azure Local cluster but are not added to the Storage Spaces Direct (S2D) pool. - -`CanPool=False` does not always mean something is broken. It can mean the disk is already in the pool, still being verified, blocked by hardware or firmware support checks, offline, not healthy, or carrying old metadata. - -## Symptoms + -**Observable behaviors:** +# Troubleshoot physical disks not claimed after insertion (`CanPool=False`) -- New disks were inserted into one or more cluster nodes. +## Table of contents + +- [Article metadata](#article-metadata) +- [Executive triage summary](#executive-triage-summary) +- [Symptoms and scope](#symptoms-and-scope) +- [Terms](#terms) +- [Where this failure appears](#where-this-failure-appears) +- [Preconditions and stop gates](#preconditions-and-stop-gates) +- [Common `CannotPoolReason` values](#common-cannotpoolreason-values) +- [Diagnosis and mitigation](#diagnosis-and-mitigation) +- [Support Diagnostics collection](#support-diagnostics-collection) +- [Verification](#verification) +- [Rollback](#rollback) +- [Escalation and evidence package](#escalation-and-evidence-package) +- [Prevention and monitoring](#prevention-and-monitoring) +- [Future test automation metadata](#future-test-automation-metadata) +- [Related documentation](#related-documentation) + +## Article metadata + +| Field | Value | +| --- | --- | +| Component | Storage | +| Applicable products | Azure Local, Storage Spaces Direct | +| Supported versions | All Azure Local releases that use Storage Spaces Direct | +| Audience | Customer IT admin, Microsoft Customer Support Services (CSS), OEM field engineer, partner or systems integrator | +| Applicable scenarios | Day 2 capacity expansion, add disk, physical disk replacement | +| Severity | Medium, because the cluster can stay online but capacity expansion or replacement is blocked | +| Customer impact | Pool capacity does not grow. Workloads normally stay online unless the cluster also has active storage health faults, node failures, or degraded redundancy | +| Primary owner | Customer IT for safe diagnosis and disk identity, OEM for model and firmware supportability, Microsoft CSS for inconclusive or Health Service cases | +| Execution surface | on-device | +| Risk summary | Diagnosis is read-only. Bringing a disk online is medium risk. `Reset-PhysicalDisk` is high risk and destroys Storage Spaces metadata on the target disk | + +## Executive triage summary + +| Question | Answer | +| --- | --- | +| What is broken? | A newly inserted physical disk is visible to Windows, but `Get-PhysicalDisk` reports `CanPool=False` and Storage Spaces Direct does not claim it. | +| Who should act? | A cluster administrator with local administrator rights on an Azure Local node. Engage the OEM for hardware, firmware, or supported configuration findings. | +| Fastest safe next action | Run the read-only `Get-PhysicalDisk` inventory in [Step 1](#step-1-capture-disk-identity-and-reason), then branch on the exact `CannotPoolReason`. | +| Estimated duration | 10 to 20 minutes for source diagnosis. Persistent Health Service verification adds about 10 to 15 minutes of waiting before you escalate. Firmware and OEM cases depend on the vendor support SLA and can run from hours to days. | +| Downtime or workload risk | None for diagnosis. No node drain, reboot, or live migration is required, so VMs stay online. Disk state changes, manual pool add, firmware updates, and destructive reset carry maintenance risk. A manual pool add starts background rebalance I/O that can run for minutes to hours depending on data volume. | +| Escalate immediately if | Cluster quorum is at risk, a virtual disk is detached or degraded, the disk may contain data, the model or firmware is not vendor-approved, or verification stays stuck after the normal wait of about 10 to 15 minutes. | + +`CanPool=False` is not automatically a fault. It can be an expected transitional state while Health Service verifies a disk, or it can mean the disk is already in a pool. This guide is the general decision-tree router for all documented `CannotPoolReason` values. It delegates the persistent Health Service verification-stuck branch to the dedicated `AzLocal_Storage_PhysicalDiskVerificationStuck` TSG and PR 333 ownership path. + +> [!NOTE] +> Scope: this is a storage disk-eligibility issue. It is not a networking or fabric problem, and it is not an OEM hardware or firmware problem unless `CannotPoolReason` is `Hardware Not Compliant` or `Firmware Not Compliant`. If you own networking only, hand this to the storage or cluster administrator. + +## Symptoms and scope + +Use this guide when all of the following are true: + +- New or replacement disks were inserted into one or more Azure Local cluster nodes. - The disks appear in PowerShell. -- Storage pool capacity did not increase. -- `Get-PhysicalDisk` shows one or more disks with `CanPool=False`. +- Storage pool capacity did not increase as expected. +- `Get-PhysicalDisk` shows one or more candidate disks with `CanPool=False`. -**Common error indicator:** +Common indicator: -``` -CanPool : False -CannotPoolReason : +```text +CanPool : False +CannotPoolReason : ``` -## Root Cause +This guide does not repair a wedged Health Service provider configuration. If `CannotPoolReason` is `Verification In Progress` or `Verification Failed` and the status does not clear after the normal wait of about 10 to 15 minutes, stop this guide and use the dedicated Health Service verification-stuck TSG from PR 333 or spec `AzLocal_Storage_PhysicalDiskVerificationStuck`. + +## Terms + +| Term | Plain-language definition | Why it matters | +| --- | --- | --- | +| Storage Spaces Direct (S2D) | The Azure Local storage layer that pools local disks across cluster nodes. | S2D owns the disk claim, health, repair, and pool-add behavior in this guide. | +| `CanPool` | A `Get-PhysicalDisk` property that says whether Storage Spaces can claim the disk into a pool. | `False` means at least one eligibility gate is blocking the disk. | +| `CannotPoolReason` | The `Get-PhysicalDisk` reason string that names the blocking gate. | The exact string chooses the safe branch in this article. | +| Primordial pool | The operating system's unclaimed disk collection. | A disk can be eligible for pooling only when it is not already in a non-primordial pool. | +| Non-primordial pool | The real S2D storage pool that contains claimed capacity disks. | Commands that add or check pool membership must target this pool explicitly. | +| `UniqueId` | A disk identity value used by Storage cmdlets. | Use it with `Reset-PhysicalDisk` because names and serial numbers are not always unique. | +| Stale metadata | Old Storage Spaces pool metadata left on a reused disk. | Clearing it with `Reset-PhysicalDisk` destroys disk metadata and must be gated carefully. | +| Supported Components Document | Health Service supportability data for approved physical disks and firmware. | `Hardware Not Compliant` and `Firmware Not Compliant` are driven by vendor-approved supportability data. | +| Cluster Shared Volume (CSV) | A clustered volume that every node can access at once for running VMs. | A candidate disk that backs an active CSV must never be reset or removed. | +| Quorum | The cluster's voting majority that keeps it running safely. | Losing quorum can take the cluster offline, so stop if quorum is at risk. | +| Repair, regeneration, or rebalance job | Background Storage Spaces work that restores redundancy or spreads data evenly across disks. | Changing disk state during one of these jobs can extend the impact window. | +| Enclosure and slot | The physical bay location that identifies exactly which disk you are touching. | Correct enclosure and slot prevent acting on the wrong disk. | + +## Where this failure appears + +| Admin surface | Status | Evidence to capture | +| --- | --- | --- | +| PowerShell on an Azure Local node | shown | `Get-PhysicalDisk` shows `CanPool=False` and the exact `CannotPoolReason`. Capture `DeviceId`, `FriendlyName`, `SerialNumber`, `UniqueId`, `FirmwareVersion`, `HealthStatus`, `OperationalStatus`, `Usage`, `CanPool`, and `CannotPoolReason`. | +| Azure portal | not-evident | The Azure portal does not appear to show the exact `CannotPoolReason` value for a candidate disk. It can show unchanged pool capacity or broader storage health alerts, but use node PowerShell for the reason string. | +| Windows event logs | not-evident | No single event log carries an authoritative signal for the general `CanPool=False` decision tree. As supporting evidence, collect the storage and clustering operational channels: `Microsoft-Windows-StorageSpaces-Driver/Operational`, `Microsoft-Windows-StorageSpaces-Driver/Diagnostic`, and `Microsoft-Windows-FailoverClustering/Operational`, plus `Get-HealthFault` for current Health Service state. | +| Cluster logs using `Get-ClusterLog` | not-evident | Cluster logs do not appear to show the exact `CannotPoolReason`. Collect them when Health Service, SDDC Group, storage provider, or cluster resource state needs correlation. | +| Windows Failover Cluster Manager | not-evident | Windows Failover Cluster Manager does not appear to show the exact `CannotPoolReason`. It can help confirm cluster group, resource, and node state before risky actions. | +| Windows Admin Center on a standalone host | not-evident | Windows Admin Center does not appear to expose the exact `CannotPoolReason` string. Treat it as a supporting health surface only. | +| Windows Admin Center in the Azure portal | not-evident | Windows Admin Center in the Azure portal does not appear to expose the exact `CannotPoolReason` string. Use PowerShell for disk eligibility and use the portal only for broader health context. | +| Component or tool log files on disk | shown | `Start-AzsSupportStorageDiagnostic` output and saved artifacts can capture the storage health analysis. Store the transcript file, diagnostic log file, and report files with the support evidence package. | + +## Preconditions and stop gates + +Complete these checks before any state-changing action. + +| Gate | Pre-check | Expected result | Stop condition | Owner | +| --- | --- | --- | --- | --- | +| Permissions | Run an elevated PowerShell session on a cluster node. | Local administrator rights are available. | Required rights are missing. | Customer IT | +| Cluster health | `Get-ClusterNode`, `Get-StorageJob`, `Get-VirtualDisk`, and `Get-HealthFault`. | Nodes are up, no unrelated storage job is active, virtual disks are healthy, and no quorum or redundancy risk exists. | Node down, quorum risk, detached virtual disk, degraded redundancy, or unrelated critical storage fault. | Cluster admin or CSS | +| Disk identity | Capture `DeviceId`, `SerialNumber`, `UniqueId`, node, enclosure, and slot. | One intended disk is identified unambiguously. | The candidate disk cannot be distinguished from active pool disks. | Customer IT and OEM | +| Workload impact | Confirm whether a repair, regeneration, rebalance, or update is in progress. | No active job conflicts with the intended action. | A job is active and could be extended or restarted by changing disk state. | Cluster admin | +| OEM readiness | Compare model, firmware, and support package with the OEM-approved Azure Local solution guidance. | The model and firmware are approved or the branch does not require OEM action. | Supportability is unclear or not approved. | OEM or partner | +| Destructive reset | Confirm the disk is not in any non-primordial pool and its data can be destroyed. | The disk is stale media intended to be wiped. | Any chance remains that the disk belongs to an active pool or contains data to preserve. | CSS, customer IT, and OEM as needed | + +## Common `CannotPoolReason` values + +The `CannotPoolReason` column uses the exact title-case strings that `Get-PhysicalDisk` prints, as defined by the `MSFT_PhysicalDisk` storage class and confirmed on a live Azure Local node, for example `In a Pool` and `Insufficient Capacity`. PowerShell matching with `-eq` and `-in` is case-insensitive, so these branches still match if a specific build prints a slightly different case. Treat disk operational states such as `Lost Communication`, `Abnormal Latency`, or `Transient Error` as supporting evidence, not as substitutes for the `CannotPoolReason` string. + +| `CannotPoolReason` | Meaning | Safe branch | +| --- | --- | --- | +| `In a Pool` | The disk is already claimed by a storage pool, or it carries old Storage Spaces metadata from a prior pool. | Use [Step 2a](#step-2a-resolve-in-a-pool). Use [Step 2h](#step-2h-resolve-stale-metadata-or-previous-pool-membership) only after active pool membership is disproven. | +| `Not Healthy` | The disk health state is not healthy. | Use [Step 2b](#step-2b-resolve-not-healthy). Do not add or reset the disk until health is understood. | +| `Removable Media` | The disk is presented as removable media. | Use [Step 2g](#step-2g-resolve-insufficient-capacity-or-removable-media). Replace with supported internal storage. | +| `In Use by Cluster` | The disk is owned or reserved by cluster storage. | Use [Step 2c](#step-2c-resolve-in-use-by-cluster). Do not reset the disk until cluster ownership is understood. | +| `Offline` | The disk is offline to Windows. | Use [Step 2f](#step-2f-resolve-offline-or-read-only-disk-state). | +| `Insufficient Capacity` | The disk does not have enough usable free capacity. This can be because the disk is too small for Azure Local, or because partitions consume the free space. | Use [Step 2g](#step-2g-resolve-insufficient-capacity-or-removable-media). | +| `Verification In Progress` | Health Service is checking whether the disk and firmware are approved for the solution. | Use [Step 2d](#step-2d-resolve-verification-in-progress-or-verification-failed). Wait, then recheck. | +| `Verification Failed` | Health Service could not complete supportability verification. | Use [Step 2d](#step-2d-resolve-verification-in-progress-or-verification-failed). Escalate persistent cases to PR 333 ownership. | +| `Firmware Not Compliant` | The disk firmware is not approved by the solution vendor support data. | Use [Step 2e](#step-2e-resolve-hardware-not-compliant-or-firmware-not-compliant). | +| `Hardware Not Compliant` | The disk model is not approved by the solution vendor support data. | Use [Step 2e](#step-2e-resolve-hardware-not-compliant-or-firmware-not-compliant). | + +## Diagnosis and mitigation + +### Step 1: Capture disk identity and reason + +[READ-ONLY] Capture the full disk picture for every physical disk. -S2D will not claim a physical disk into its pool unless every gate passes: the disk must be healthy, online, supported by the solution vendor (model + firmware), have completed Health Service verification, and not already belong to another pool. The `CannotPoolReason` field on `Get-PhysicalDisk` reports which gate is failing. +```powershell +# Get the full disk picture. CannotPoolReason tells you which gate failed. +Get-PhysicalDisk | + Sort-Object DeviceId | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, MediaType, BusType, Size, FirmwareVersion, HealthStatus, OperationalStatus, Usage, CanPool, CannotPoolReason +``` -### Common `CannotPoolReason` Values +Fastest safe path: -The `CannotPoolReason` column lists values as they appear in `Get-PhysicalDisk` output. A disk carrying stale Storage Spaces metadata from a prior deployment is a sub-case of `In a Pool` (the disk thinks it belongs to a pool that no longer exists) and is handled separately in [Step 2g](#step-2g-resolve-stale-metadata-or-previous-pool-membership). +1. Find the candidate disk by serial number and `UniqueId`. +2. Match the exact `CannotPoolReason` to the table above. +3. Run only the branch that matches the reason. +4. Stop before any state-changing command if cluster health, disk identity, or supportability is unclear. -| CannotPoolReason | Meaning | Action | -|---|---|---| -| `In a Pool` | The disk was already claimed by a storage pool | Confirm pool membership via Step 2a. If `Get-StoragePool \| Get-PhysicalDisk` finds no match for the disk, treat it as stale metadata (Step 2g). | -| `Verification in progress` | Health Service is checking whether the disk and firmware are approved | Wait and recheck | -| `Verification failed` | Health Service could not complete supportability verification | Check cluster health and vendor support data | -| `Hardware not compliant` | The disk model is not approved by the solution vendor | Contact the hardware vendor | -| `Firmware not compliant` | The disk firmware is not approved by the solution vendor | Update firmware to a supported version using OEM update tooling, or contact the hardware vendor | -| `Offline` | The disk is offline | Bring only the intended disk online | -| `Insufficient Capacity` | The disk is too small | Replace with a supported disk | -| `Removable media not supported` | The disk is removable or presented as removable | Replace with supported internal storage | +> [!NOTE] +> `Get-PhysicalDisk`, `Get-StoragePool`, `Get-VirtualDisk`, `Get-StorageJob`, and `Get-HealthFault` report cluster-wide, so you can run them from any one node. `Get-Disk`, `Set-Disk`, and online or read-only changes are node-local, so run them on the node that physically holds the disk. When you add disks across many nodes or sites, repeat the matching branch on each node and re-run verification per node. -## Resolution +### Step 2a: Resolve `In a Pool` -### Prerequisites +[READ-ONLY] This is expected when automatic pooling already claimed the disk. -- An elevated PowerShell session on a cluster node. -- Confirmation that the cluster is healthy aside from this issue (no unrelated active rebuild, no node down). -- Vendor support matrix for the disk model and firmware version on hand. +```powershell +# Replace with the new disk's serial number. +$serial = '' -### Steps +# Confirm the disk is in the intended non-primordial pool. +Get-StoragePool -IsPrimordial $false | + Get-PhysicalDisk | + Where-Object SerialNumber -eq $serial | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, Usage, HealthStatus, OperationalStatus +``` + +If the disk is in the intended pool and healthy, no additional action is needed. If this returns nothing but the candidate disk still reports `In a Pool`, treat it as possible stale metadata and continue to [Step 2h](#step-2h-resolve-stale-metadata-or-previous-pool-membership). -#### Step 1: Check the Disk State +### Step 2b: Resolve `Not Healthy` -Capture the full picture for every physical disk and identify which disks are blocked and why. +[READ-ONLY] A disk that is not healthy is not a safe pool-add candidate. ```powershell -# Get the full disk picture; CannotPoolReason tells you which gate failed +# Capture disk health and any supporting storage faults. Get-PhysicalDisk | - Sort-Object DeviceId | - Format-Table DeviceId, FriendlyName, SerialNumber, MediaType, BusType, Size, FirmwareVersion, HealthStatus, Usage, CanPool, CannotPoolReason + Where-Object CannotPoolReason -eq 'Not Healthy' | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, HealthStatus, OperationalStatus, Usage + +Get-HealthFault +Get-StorageJob +Get-VirtualDisk | Format-Table FriendlyName, HealthStatus, OperationalStatus ``` -The most important field is `CannotPoolReason`. Use the value to pick the matching sub-step below. +Stop here if the disk has `Lost Communication`, `Abnormal Latency`, high error counts, transient errors, or any unresolved virtual disk redundancy issue. Collect the evidence package and engage CSS or the OEM before adding or resetting the disk. -#### Step 2a: Resolve `In a Pool` +### Step 2c: Resolve `In Use by Cluster` -This is usually expected after automatic pooling. +[READ-ONLY] Confirm whether the cluster already owns or references the disk. ```powershell -# Replace with the new disk's serial number -$serial = '' +# Replace with the candidate disk UniqueId from Step 1. +$uniqueId = '' -# Confirm the disk is in the intended (non-primordial) pool and healthy Get-StoragePool -IsPrimordial $false | Get-PhysicalDisk | - Where-Object SerialNumber -eq $serial | - Format-Table DeviceId, FriendlyName, SerialNumber, Usage, HealthStatus, OperationalStatus + Where-Object UniqueId -eq $uniqueId | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, Usage, HealthStatus, OperationalStatus + +Get-ClusterResource | Format-Table Name, ResourceType, State, OwnerGroup, OwnerNode +Get-StorageJob ``` -If the disk is in the intended pool and healthy, no additional action is needed. +Do not reset a disk that is still in use by cluster storage. If cluster ownership is unclear, collect the evidence package and escalate. -#### Step 2b: Resolve `Verification in progress` +### Step 2d: Resolve `Verification In Progress` or `Verification Failed` -Wait several minutes and recheck: +[READ-ONLY] Health Service verification can take time after disk insertion. ```powershell -# Recheck verification progress +# Recheck verification progress. Get-PhysicalDisk | - Format-Table DeviceId, FriendlyName, SerialNumber, CanPool, CannotPoolReason, HealthStatus + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, CanPool, CannotPoolReason, HealthStatus, OperationalStatus ``` -> [!WARNING] -> Do not reset or manually add disks while verification is still in progress. - -#### Step 2c: Resolve `Verification failed` - -Check the cluster and storage state: +[READ-ONLY] If the state does not clear after about 10 to 15 minutes, capture cluster and storage context. ```powershell -# Cluster + storage health snapshot Get-HealthFault -Get-ClusterNode | Format-Table Name, State +Get-ClusterNode | Format-Table Name, State Get-StorageJob -Get-StoragePool -IsPrimordial $false | Format-Table FriendlyName, HealthStatus, OperationalStatus -Get-VirtualDisk | Format-Table FriendlyName, HealthStatus, OperationalStatus +Get-StoragePool -IsPrimordial $false | Format-Table FriendlyName, HealthStatus, OperationalStatus +Get-VirtualDisk | Format-Table FriendlyName, HealthStatus, OperationalStatus ``` -If available, run the Azure Local Support Diagnostic Tool storage checks: +> [!WARNING] +> Do not reset or manually add disks while verification is still in progress or failed. If `Verification In Progress` or `Verification Failed` remains unchanged after the normal wait of about 10 to 15 minutes and the disk is clean, supported, online, and symmetric, stop this decision tree and use the dedicated Health Service verification-stuck TSG from PR 333 or spec `AzLocal_Storage_PhysicalDiskVerificationStuck`. That companion owns Health resource, SDDC Group, and provider-list repair. + +### Step 2e: Resolve `Hardware Not Compliant` or `Firmware Not Compliant` + +[READ-ONLY] Capture model and firmware so the OEM can confirm whether the disk is supported by the validated Azure Local solution. ```powershell -# Targeted disk and storage health checks via the Support Diagnostic Tool -Start-AzsSupportStorageDiagnostic -Include 'DiskHealth','StorageHealth' +Get-PhysicalDisk | + Sort-Object FriendlyName, FirmwareVersion | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, FirmwareVersion, MediaType, BusType, HealthStatus, CanPool, CannotPoolReason ``` -For details on these checks see [Troubleshooting Storage With Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md). +> [!CAUTION] +> Do not bypass hardware or firmware validation. Use the OEM-approved update method for the validated Azure Local solution, or replace the disk with an OEM-supported model. If the OEM does not list a supported firmware for this model and solution, replace the disk or escalate to the OEM. -If the disk model or firmware is new to the system, validate supportability with the hardware vendor. +Firmware update tools and support packages are vendor-owned. Follow the OEM's official Azure Local solution guidance and maintenance requirements. Do not flash firmware as part of this TSG unless the OEM procedure, customer maintenance plan, and rollback plan are approved. -#### Step 2d: Resolve `Hardware not compliant` +The exact approved model and firmware revisions come from the Health Service Supported Components Document and the OEM's validated Azure Local solution matrix. Expected end-state: after the disk matches an approved model and firmware, Health Service re-verifies it and `Get-PhysicalDisk` reports `CanPool=True`, at which point you can continue to [Step 3](#step-3-manually-add-disks-only-when-they-are-eligible). -The disk model is not approved for this solution. +### Step 2f: Resolve `Offline` or read-only disk state + +[READ-ONLY] Identify the exact disk first. ```powershell -# Capture model + firmware so the vendor can confirm support -Get-PhysicalDisk | - Format-Table DeviceId, FriendlyName, SerialNumber, FirmwareVersion, MediaType, BusType, CanPool, CannotPoolReason +Get-Disk | + Sort-Object Number | + Format-Table Number, FriendlyName, SerialNumber, UniqueId, OperationalStatus, IsOffline, IsReadOnly, PartitionStyle ``` -> [!CAUTION] -> Do not bypass hardware validation. Contact the hardware vendor for a supported disk model or an updated solution support package. - -#### Step 2e: Resolve `Firmware not compliant` - -The disk firmware is not approved for this solution. +Before changing the disk state: ```powershell -# Compare firmware on the new vs existing disks of the same model -Get-PhysicalDisk | - Sort-Object FriendlyName, FirmwareVersion | - Format-Table DeviceId, FriendlyName, SerialNumber, FirmwareVersion, HealthStatus, CanPool, CannotPoolReason +# Stop if repair, regeneration, or rebalance work is active. +Get-StorageJob ``` -Contact the hardware vendor for firmware alignment or updated support guidance. In most cases, the resolution is to update firmware to a supported version using the OEM update tooling (Dell DSU, HPE SUM, Lenovo XClarity Essentials, etc.). If no supported firmware version exists for this disk model, the model itself may have been deprecated -- escalate to the hardware vendor. +> [!CAUTION] +> [MEDIUM RISK] Bring only the intended disk online. If a repair, regeneration, or rebalance job is active and involves this disk, let it complete first. Forcing the disk online mid-rebuild can extend the impact window. -#### Step 2f: Resolve `Offline` or Read-Only Disk State +After the intended disk is confirmed and no stop condition is present: ```powershell -# Identify the exact disk first -Get-Disk | Sort-Object Number | - Format-Table Number, FriendlyName, SerialNumber, OperationalStatus, IsOffline, IsReadOnly, PartitionStyle +# Replace with the Number value from Get-Disk above. +Set-Disk -Number -IsOffline $false +Set-Disk -Number -IsReadOnly $false ``` -After the intended disk is confirmed: +Recheck `Get-PhysicalDisk` afterward. -> [!CAUTION] -> Run `Get-StorageJob` first. If a repair, regeneration, or rebalance job is active that involves this disk, let it complete before bringing the disk online -- forcing it online mid-rebuild can cause the rebuild to retry against the newly-online path and extend the impact window. +### Step 2g: Resolve `Insufficient Capacity` or `Removable Media` + +[READ-ONLY] `Insufficient Capacity` can mean the disk is below the Azure Local data-drive requirement, or that existing partitions consume the usable free space. ```powershell -# Replace with the Number value from Get-Disk above -Set-Disk -Number -IsOffline $false -Set-Disk -Number -IsReadOnly $false +# Review the candidate disk's size, partitioning, and bus presentation. +Get-Disk | + Sort-Object Number | + Format-Table Number, FriendlyName, SerialNumber, UniqueId, BusType, Size, PartitionStyle, IsOffline, IsReadOnly + +Get-PhysicalDisk | + Where-Object { $_.CannotPoolReason -in @('Insufficient Capacity','Removable Media') } | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, MediaType, BusType, Size, CanPool, CannotPoolReason ``` -Recheck `Get-PhysicalDisk` afterward. +For `Insufficient Capacity`, replace the disk if it does not meet Azure Local requirements. If partitions or old configuration consume the disk, treat it as stale media and use [Step 2h](#step-2h-resolve-stale-metadata-or-previous-pool-membership) only when the disk is intended to be wiped. + +For `Removable Media`, replace the disk with supported internal storage. Do not try to force removable media into the S2D pool. -#### Step 2g: Resolve Stale Metadata or Previous Pool Membership +### Step 2h: Resolve stale metadata or previous pool membership > [!WARNING] -> `Reset-PhysicalDisk` is destructive. Do not run it on a disk that belongs to an active pool or contains data that must be preserved. Use this path **only** for disks that are intended to be wiped. +> [HIGH RISK] `Reset-PhysicalDisk` is destructive. It removes Storage Spaces pool configuration and data from the target disk. Use this path only for disks that are intended to be wiped. -Before reset, confirm the disk identity and that it is not in any pool: +> [!NOTE] +> If you are new to Storage Spaces Direct, do NOT run `Reset-PhysicalDisk` yourself. Confirm every hard-stop item below with a senior cluster administrator or Microsoft CSS first, because a reset on the wrong disk cannot be undone. + +Hard stop checklist before reset: + +- The candidate disk is identified by `UniqueId`, `SerialNumber`, `DeviceId`, node, enclosure, and slot. +- The disk does not appear in any active non-primordial pool. +- The disk is not required for any virtual disk, CSV, repair, regeneration, or rebalance job. +- The customer confirms the disk has no data to preserve. +- The reset target is a replacement or reused disk, not an active pool member. +- If there is any uncertainty, collect the evidence package and escalate before resetting. + +[READ-ONLY] Confirm the candidate identity and active pool membership. ```powershell -# Inspect the candidate disk -Get-PhysicalDisk -UniqueId '' | Format-List * +# Replace with the candidate disk UniqueId from Step 1. +$uniqueId = '' + +# Inspect the candidate disk. +Get-PhysicalDisk -UniqueId $uniqueId | Format-List * -# Confirm it is NOT a member of any active pool (this should return nothing) +# Confirm it is NOT a member of any active non-primordial pool. This should return nothing. Get-StoragePool -IsPrimordial $false | Get-PhysicalDisk | - Where-Object UniqueId -eq '' | + Where-Object UniqueId -eq $uniqueId | Format-List * + +# Confirm no storage job is active before any reset. +Get-StorageJob ``` -Only if the disk is confirmed to be unused stale media and the data can be destroyed: +Only if every hard stop check passes: ```powershell -# Destructive: clears storage pool metadata from the disk -Reset-PhysicalDisk -UniqueId '' +# Destructive: clears Storage Spaces metadata from the target disk. +Reset-PhysicalDisk -UniqueId $uniqueId ``` Wait several minutes and recheck: ```powershell -# Verify the disk is now eligible -Get-PhysicalDisk -UniqueId '' | - Format-Table DeviceId, FriendlyName, SerialNumber, CanPool, CannotPoolReason, HealthStatus, Usage +Get-PhysicalDisk -UniqueId $uniqueId | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, CanPool, CannotPoolReason, HealthStatus, Usage ``` -#### Step 3: Manual Add When Disks Are Eligible +### Step 3: Manually add disks only when they are eligible + +[MEDIUM RISK] Manual pool add changes storage pool membership. Use it only after the disks show `CanPool=True`, cluster health is stable, and automatic pooling has not claimed them. -If the disks now show `CanPool=True` but are not automatically claimed, first inspect the current pool and eligible disks: +Workload impact: adding disks can start storage jobs and rebalance work. Schedule the action when the cluster can tolerate background I/O and monitor `Get-StorageJob` until it drains. ```powershell -# Inspect the target pool and eligible disks -$pool = Get-StoragePool -IsPrimordial $false +# Inspect the target pool and eligible disks. +$pool = Get-StoragePool -IsPrimordial $false $eligibleDisks = Get-PhysicalDisk -CanPool $true -$pool | Format-Table FriendlyName, HealthStatus, OperationalStatus -$eligibleDisks | Format-Table DeviceId, FriendlyName, SerialNumber, MediaType, Size, FirmwareVersion +$pool | Format-Table FriendlyName, HealthStatus, OperationalStatus +$eligibleDisks | Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, MediaType, Size, FirmwareVersion ``` > [!IMPORTANT] -> A healthy Storage Spaces Direct cluster has exactly one non-primordial pool. The snippet below enforces that and requires the operator to enumerate the intended new disks by serial number, so `Add-PhysicalDisk` cannot accidentally claim unintended `CanPool=True` disks. +> A healthy Storage Spaces Direct cluster has exactly one non-primordial pool. The snippet below enforces that and requires the operator to enumerate the intended new disks by `UniqueId`, so `Add-PhysicalDisk` cannot accidentally claim unintended `CanPool=True` disks. `UniqueId` is the stable key: nested and virtual data disks frequently report an empty `SerialNumber`, confirmed on nested virtual data disks, so serial number is only an optional human cross-check. ```powershell # Defensive: require exactly one non-primordial pool. Abort otherwise. $pool = Get-StoragePool -IsPrimordial $false if (@($pool).Count -ne 1) { - throw "Expected exactly one non-primordial pool. Found $(@($pool).Count). " + - "Select the target pool explicitly by FriendlyName before continuing." + throw "Expected exactly one non-primordial pool. Found $(@($pool).Count). Select the target pool explicitly by FriendlyName before continuing." } -# Operator MUST enumerate the intended new disks by serial number. -# Do NOT pipe Get-PhysicalDisk -CanPool $true directly into Add-PhysicalDisk. -$intendedSerials = @( - '', - '' +# Operator MUST enumerate the intended new disks by UniqueId (the stable key). +# SerialNumber is unreliable: nested and virtual data disks frequently report an empty SerialNumber. +# Do not pipe Get-PhysicalDisk -CanPool $true directly into Add-PhysicalDisk. +$intendedUniqueIds = @( + '', + '' ) -# Resolve serials to physical disk objects and confirm the count matches the intent. +# Resolve UniqueIds to physical disk objects and confirm the count matches the intent. # Wrap in @() so .Count is reliable when 0 or 1 disk matches. $disksToAdd = @(Get-PhysicalDisk -CanPool $true | - Where-Object SerialNumber -in $intendedSerials) -if ($disksToAdd.Count -ne $intendedSerials.Count) { - throw "Disk count mismatch: $($disksToAdd.Count) eligible disks matched " + - "the $($intendedSerials.Count) intended serial numbers. Resolve before continuing." + Where-Object UniqueId -in $intendedUniqueIds) +if ($disksToAdd.Count -ne $intendedUniqueIds.Count) { + throw "Disk count mismatch: $($disksToAdd.Count) eligible disks matched the $($intendedUniqueIds.Count) intended UniqueIds. Resolve before continuing." +} + +# Optional human cross-check. SerialNumber may be blank on nested or virtual disks; UniqueId is authoritative. +$disksToAdd | Format-Table UniqueId, FriendlyName, SerialNumber, Size, MediaType + +# Preview the add with no changes. +Add-PhysicalDisk -StoragePoolFriendlyName $pool.FriendlyName -PhysicalDisks $disksToAdd -WhatIf + +# Require an explicit typed confirmation before the real add. +$confirm = Read-Host "Type ADD to claim the $($disksToAdd.Count) disk(s) above into pool '$($pool.FriendlyName)'" +if ($confirm -ne 'ADD') { + throw "Confirmation not received. Aborting before Add-PhysicalDisk." } # Add only the explicitly identified disks to the target pool. Add-PhysicalDisk -StoragePoolFriendlyName $pool.FriendlyName -PhysicalDisks $disksToAdd ``` -#### Step 4: Verify Resolution +Monitor the resulting storage jobs: + +```powershell +Get-StorageJob +Get-StoragePool -IsPrimordial $false | Format-Table FriendlyName, HealthStatus, OperationalStatus, AllocatedSize, Size +``` + +## Support Diagnostics collection + +The Azure Local Support Diagnostic Tool is a read-only evidence collection path in this article. The commands below write output files under `C:\Temp`, but they do not change cluster storage state. + +The exact `Start-AzsSupportStorageDiagnostic` source tokens used by this article are: + +| Source label | Token or parameter | +| --- | --- | +| Missing Disks from Storage Spaces | `MissingDisks` | +| Storage Pool Health Check, Cluster Nodes Health Process Running, Storage Job Check, Cluster Node Check, Cluster Shared Volumes Check, Storage Enclosure Check, Health Service Fault Check, Storage Health Action Check, Disks Not in Pool Check | `StorageHealth` | +| Storage Spaces Partitions Check, Disk Health Check, Transient Disk Check | `DiskHealth` | +| Support Components Change, Support Components Missing | `StorageComponents` | +| Firmware Drift | `FirmwareDrift` | +| Storage Summary | `StorageSummary` | +| Cluster Shared Volume Usage | `CSVUsage` | +| Physical extent analysis | `-PhysicalExtentCheck ` | + +[READ-ONLY] Capture output and artifacts for the support package. + +```powershell +# Local artifact folder for this read-only collection. +$evidenceRoot = "C:\Temp\PhysicalDiskCanPoolFalse-$((Get-Date).ToUniversalTime().ToString('yyyyMMddTHHmmssZ'))" +New-Item -ItemType Directory -Path $evidenceRoot -Force | Out-Null + +# Note: Start-AzsSupportStorageDiagnostic stops any outer Start-Transcript when it runs, +# so do not rely on Start-Transcript to capture this cmdlet. Redirect all streams instead. +# Record the cmdlet working directory and a start time so its native transcript can be copied out. +$diagWorkingDir = (Get-Location).Path +$runStart = Get-Date + +# Capture ALL streams (success, error, warning, verbose, host, and information) with no ConvertTo-Json. +Start-AzsSupportStorageDiagnostic -Include 'MissingDisks','DiskHealth','StorageHealth','StorageComponents','FirmwareDrift','StorageSummary','CSVUsage' *>&1 | + Tee-Object -FilePath (Join-Path $evidenceRoot 'Start-AzsSupportStorageDiagnostic.txt') + +# Copy the native tool transcript the cmdlet writes in its own working directory into the evidence package. +Get-ChildItem -Path $diagWorkingDir -Recurse -File | + Where-Object { $_.LastWriteTime -ge $runStart -and $_.Extension -in '.txt','.log','.etl','.zip' } | + ForEach-Object { Copy-Item -Path $_.FullName -Destination $evidenceRoot -Force } +``` + +[READ-ONLY] Use `-PhysicalExtentCheck` only when a virtual disk is degraded, detached, or otherwise unexpected. The parameter takes a virtual disk `FriendlyName`, not a physical disk serial number or drive letter. + +```powershell +# Replace with a Get-VirtualDisk FriendlyName, for example UserStorage_1. +$virtualDiskFriendlyName = '' + +$extentStart = Get-Date +Start-AzsSupportStorageDiagnostic -PhysicalExtentCheck $virtualDiskFriendlyName *>&1 | + Tee-Object -FilePath (Join-Path $evidenceRoot 'PhysicalExtentCheck.txt') + +# Copy the native tool transcript for this run into the evidence package as well. +Get-ChildItem -Path $diagWorkingDir -Recurse -File | + Where-Object { $_.LastWriteTime -ge $extentStart -and $_.Extension -in '.txt','.log','.etl','.zip' } | + ForEach-Object { Copy-Item -Path $_.FullName -Destination $evidenceRoot -Force } +``` + +For tool installation and full reference information, see [Support Tools](https://learn.microsoft.com/azure/azure-local/manage/support-tools) and [Troubleshooting Storage With Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md). + +## Verification + +Verify both the original disk reason and the broader cluster state. ```powershell -# Confirm the new disks are in the pool, healthy, with no surprise jobs or faults -Get-PhysicalDisk | Sort-Object DeviceId | - Format-Table DeviceId, FriendlyName, SerialNumber, Usage, HealthStatus, CanPool, CannotPoolReason +# Confirm the candidate disk state. +Get-PhysicalDisk | + Sort-Object DeviceId | + Format-Table DeviceId, FriendlyName, SerialNumber, UniqueId, Usage, HealthStatus, OperationalStatus, CanPool, CannotPoolReason + +# Confirm the pool, jobs, and health state. Get-StoragePool -IsPrimordial $false | Format-Table FriendlyName, HealthStatus, OperationalStatus, Size, AllocatedSize +Get-VirtualDisk | Format-Table FriendlyName, HealthStatus, OperationalStatus Get-StorageJob Get-HealthFault ``` Expected result: -- New disks are in the intended pool. +- Intended new disks are in the non-primordial pool, or are clearly blocked by the correct non-poolable reason. - New disks are healthy. - No unintended `CanPool=True` disks remain. -- No new storage faults are active. -- Any expected storage jobs are progressing. +- No unexpected storage jobs or faults are active. +- If jobs started after manual add, they are progressing and eventually drain to zero. -## Prevention +If this issue was found during deployment or capacity expansion, re-run the same validation that first flagged it after remediation, not just the `Get-PhysicalDisk` recheck: -- Always validate disk model and firmware against the OEM solution support matrix **before** insertion. -- Add disks symmetrically (same count and type per node) to avoid stranded capacity. -- Run the [How to add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md) pre-checks before any insertion. -- Avoid reusing disks from prior deployments without confirming they are wiped of stale pool metadata. +```powershell +# Re-run failover cluster storage validation after the disks are claimed. +Test-Cluster -Include "Storage Spaces Direct", "Inventory", "Storage" +``` -## Data to Collect Before Opening a Support Case +For a deployment that tripped the Azure Local environment validation, re-run the Environment Checker or the deployment validation step your process uses, and confirm it now passes. -```powershell -# Cluster + storage state snapshot -Get-HealthFault -Get-ClusterNode | Format-Table Name, State -Get-StorageJob -Get-StoragePool -IsPrimordial $false | Format-List * -Get-VirtualDisk | Format-List * -Get-PhysicalDisk | Format-List * +## Rollback + +| Action | Rollback | +| --- | --- | +| `Set-Disk -IsOffline $false` or `Set-Disk -IsReadOnly $false` | If the wrong disk was targeted or impact appears, stop and escalate. Do not toggle states repeatedly without CSS guidance. Capture `Get-Disk`, `Get-PhysicalDisk`, and `Get-StorageJob`. | +| `Reset-PhysicalDisk` | No in-place rollback. This is why the reset gate requires proof that the disk is stale disposable media. If reset was run on the wrong disk, stop all further actions and contact CSS immediately. | +| `Add-PhysicalDisk` | Do not remove an added disk unless CSS or the OEM confirms it is safe. Storage may have already allocated data to the disk. Capture pool and job state before any corrective action. | +| Local evidence collection files | Delete the local `C:\Temp\PhysicalDiskCanPoolFalse-*` folder after the support case no longer needs the artifacts. | + +## Escalation and evidence package + +Escalate to Microsoft CSS, the OEM, or the partner when diagnosis is inconclusive, a stop gate is met, mitigation risk is unacceptable, or the guide points to vendor or Health Service ownership. -# Last 60 minutes of cluster log to C:\Temp -Get-ClusterLog -Destination C:\Temp -TimeSpan 60 +Collect these artifacts before opening or updating a support case: + +```powershell +# Local artifact folder for command output. +$evidenceRoot = "C:\Temp\PhysicalDiskCanPoolFalse-$((Get-Date).ToUniversalTime().ToString('yyyyMMddTHHmmssZ'))" +New-Item -ItemType Directory -Path $evidenceRoot -Force | Out-Null + +Get-HealthFault | Tee-Object -FilePath (Join-Path $evidenceRoot 'Get-HealthFault.txt') +Get-ClusterNode | Format-Table Name, State | Tee-Object -FilePath (Join-Path $evidenceRoot 'Get-ClusterNode.txt') +Get-StorageJob | Tee-Object -FilePath (Join-Path $evidenceRoot 'Get-StorageJob.txt') +Get-StoragePool -IsPrimordial $false | Format-List * | Out-File -FilePath (Join-Path $evidenceRoot 'Get-StoragePool.txt') -Encoding utf8 +Get-VirtualDisk | Format-List * | Out-File -FilePath (Join-Path $evidenceRoot 'Get-VirtualDisk.txt') -Encoding utf8 +Get-PhysicalDisk | Format-List * | Out-File -FilePath (Join-Path $evidenceRoot 'Get-PhysicalDisk.txt') -Encoding utf8 +Get-Disk | Format-List * | Out-File -FilePath (Join-Path $evidenceRoot 'Get-Disk.txt') -Encoding utf8 + +# Last 60 minutes of cluster log. +Get-ClusterLog -Destination $evidenceRoot -TimeSpan 60 ``` Also collect: -- Disk serial numbers and unique IDs. -- Node and slot mapping. +- Disk serial numbers, unique IDs, enclosure, slot, and node mapping. - Disk model and firmware version. - Hardware vendor support matrix or written confirmation for the disk model and firmware. - Whether automatic pooling is expected or intentionally disabled. +- Whether the issue started after insertion, firmware change, node maintenance, or a solution update. +- Support Diagnostic Tool output from [Support Diagnostics collection](#support-diagnostics-collection). -## Related Issues +## Prevention and monitoring -- [How to add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md) -- [Troubleshooting Storage With Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md) - -## References +- Validate disk model and firmware against the OEM-approved Azure Local solution before insertion. +- Add disks symmetrically, same count and type per node, to avoid stranded capacity. +- Run the [How to add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md) pre-checks before insertion. +- Avoid reusing disks from prior deployments without confirming they are wiped of stale pool metadata. +- Record serial, slot, firmware, and node mapping before and after each physical disk change. +- Watch `Get-StorageJob`, `Get-HealthFault`, and pool allocated size after manual add until the cluster settles. + +## Future test automation metadata + +| Field | Value | +| --- | --- | +| Detector type | `command` | +| Detector signal | `Get-PhysicalDisk` with `CanPool` and `CannotPoolReason` | +| Inject strategy | VM-first scratch VHD slices for offline, read-only, insufficient-capacity, and stale-metadata behavior; hardware fallback only for OEM compliance, removable media, and physical disk health cases | +| Mitigation selector | Branch-specific command markers: `Set-Disk`, `Reset-PhysicalDisk`, `Add-PhysicalDisk`, and Support Diagnostics read-only collection | +| Safety floor | No reset unless the disk is confirmed outside every active non-primordial pool and intended to be wiped | +| Rollback check | Re-run `Get-PhysicalDisk`, `Get-StoragePool`, `Get-VirtualDisk`, `Get-StorageJob`, and `Get-HealthFault` | +| Reproduction substrate | `either`, VM first with hardware fallback | +| Fidelity level | `L1`, live read-only validation complete for the observed reasons | +| Technical grade | JSON `null` until TSG-FORGE live evidence completes | +| Automation status | `ready` | +| Spec reference | `AzLocal_Storage_PhysicalDiskCanPoolFalseGeneral` | +| Last validated | `2026-08-17`, live-observed reasons `In a Pool` and `Insufficient Capacity` on a nested Storage Spaces Direct lab cluster | + +## Related documentation - [Adding servers or drives to Storage Spaces Direct](https://learn.microsoft.com/windows-server/storage/storage-spaces/add-nodes#adding-drives) - [Troubleshoot Storage Spaces and Storage Spaces Direct health and operational states](https://learn.microsoft.com/windows-server/storage/storage-spaces/storage-spaces-states) - ---- +- [Azure Local system requirements](https://learn.microsoft.com/azure/azure-local/concepts/system-requirements-23h2) +- [Health Service overview](https://learn.microsoft.com/azure/azure-local/manage/health-service-overview) +- [Health Service settings](https://learn.microsoft.com/azure/azure-local/manage/health-service-settings) +- [Support Tools](https://learn.microsoft.com/azure/azure-local/manage/support-tools) +- [How to add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md) +- [Troubleshooting Storage With Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index e4328d5e..7fb5b7ed 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -1,3 +1,23 @@ + + # Troubleshoot the storage pool capacity threshold warning (fixed vs thin volumes) @@ -17,10 +37,18 @@ + + + + + + + +
Affected Versions All Azure Local releases (Storage Spaces Direct)
Article TypeTroubleshooting guide
Validation ScopeUse read-only checks first. Do not validate this by filling a production pool. If lab validation is needed, use an isolated scratch volume on physical S2D hardware.
-> **In plain terms:** the storage *pool* — the shared disk capacity behind every -> volume on the cluster — is filling up. Deciding what to do is a capacity task for +> **In plain terms:** the storage *pool*, the shared disk capacity behind every +> volume on the cluster, is filling up. Deciding what to do is a capacity task for > the **cluster / storage administrator**; it is usually not urgent, but if the pool > is genuinely allowed to fill, virtual machines can pause or go offline. Run the > **Quick triage** below to find which of two fixes applies. @@ -30,7 +58,7 @@ - @@ -45,7 +73,7 @@ @@ -61,7 +89,7 @@ ## Quick triage (start here) Run this on any cluster node. It shows how full the pool is and, crucially, whether -the volumes are **Fixed** or **Thin** — which decides the entire remediation path: +the volumes are **Fixed** or **Thin**, which decides the entire remediation path: ```powershell # 1) How full is the pool? (allocation vs total size) @@ -79,18 +107,28 @@ Get-HealthFault Then branch on `ProvisioningType`: - **`Fixed`** → the pool footprint is committed by design; the fix is to add - capacity, convert to thin, or adjust the alert — go to - [Path A](#path-a--fixed-provisioned-volumes). -- **`Thin`** → capacity from deleted data can be reclaimed — go to - [Path B](#path-b--thin-provisioned-volumes-reclaim-unused-capacity) (needs a + capacity, convert to thin, or adjust the alert. Go to + [Path A](#path-a-fixed-provisioned-volumes). +- **`Thin`** → capacity from deleted data can be reclaimed. Go to + [Path B](#path-b-thin-provisioned-volumes-reclaim-unused-capacity) (needs a maintenance window). > [!NOTE] > This is the short form of -> [Step 1](#step-1--determine-the-provisioning-type-required-first), surfaced up top. -> The full guide below explains *why* and covers every option in detail — read on if +> [Step 1](#step-1-determine-the-provisioning-type-required-first), surfaced up top. +> The full guide below explains *why* and covers every option in detail. Read on if > triage alone does not resolve it. +> [!NOTE] +> **Running this across many nodes or sites.** Pool allocation and provisioning +> type are cluster-wide, so the triage above is run once per cluster from any +> node. To check several clusters at once, wrap the same read-only block in +> `Invoke-Command -ComputerName { ... }`; it is safe to +> script across identical sites. The capacity event log (EventIDs 103, 104, and +> 310) is the exception: collect it on **every node**, because the pool owner +> logs it and ownership can move (see +> [Data to Collect](#data-to-collect-before-opening-a-support-case)). + ## Overview This guide explains the Storage Spaces Direct (S2D) **storage pool capacity @@ -102,7 +140,7 @@ The warning is **not a false alarm**: S2D needs free pool capacity in reserve so that storage repair jobs can rebuild resiliency after a drive or node is lost. It is, however, frequently misunderstood on clusters that use **fixed-provisioned** volumes, because a fixed volume commits its entire size to the pool the moment it -is created — so the pool can sit above the threshold even when the volume's file +is created, so the pool can sit above the threshold even when the volume's file system is mostly empty. The single most important step is to **determine whether the affected volumes are @@ -115,25 +153,25 @@ fixed-provisioned volume** and only applies to thin-provisioned volumes. Short definitions for the terms used in this guide: -- **Storage pool** — the cluster-wide set of physical drives that Storage Spaces +- **Storage pool:** the cluster-wide set of physical drives that Storage Spaces Direct (S2D) manages as one unit. Every volume is carved out of the pool. -- **S2D (Storage Spaces Direct)** — the Azure Local software-defined storage layer +- **S2D (Storage Spaces Direct):** the Azure Local software-defined storage layer that pools each server's local drives into shared, resilient storage. -- **CSV (Cluster Shared Volume)** — a volume mounted under `C:\ClusterStorage\` that +- **CSV (Cluster Shared Volume):** a volume mounted under `C:\ClusterStorage\` that every node can use at once; where VM disks (`.vhdx`) live. -- **ReFS (Resilient File System)** — the file system used for Azure Local volumes. -- **Thin vs Fixed provisioning** — a **thin** volume consumes pool capacity only as +- **ReFS (Resilient File System):** the file system used for Azure Local volumes. +- **Thin vs Fixed provisioning:** a **thin** volume consumes pool capacity only as data is written; a **fixed** volume reserves its full size in the pool the moment it is created (see [Why fixed-provisioned volumes hit this so easily](#why-fixed-provisioned-volumes-hit-this-so-easily)). -- **Footprint (`FootprintOnPool`)** — how much pool capacity a volume actually +- **Footprint (`FootprintOnPool`):** how much pool capacity a volume actually occupies, *including* its resiliency copies (for example, a three-way mirror uses 3× the written data). -- **Slab** — the 256 MB unit S2D allocates pool capacity in. A slab returns to the +- **Slab:** the 256 MB unit S2D allocates pool capacity in. A slab returns to the pool only when every block in it is free. -- **Unmap** — the background ReFS operation that returns emptied slabs to the pool +- **Unmap:** the background ReFS operation that returns emptied slabs to the pool after data is deleted or consolidated. -- **Primordial pool** — the built-in pool of drives not yet added to an S2D pool; +- **Primordial pool:** the built-in pool of drives not yet added to an S2D pool; filter it out with `Where-Object IsPrimordial -eq $false`. ## Symptoms @@ -151,6 +189,21 @@ Short definitions for the terms used in this guide: out-of-capacity error from the underlying virtualization layer once the pool is near full, even though the volume looks fine in Windows Admin Center. +## Where this appears, and where it does not + +Use these surfaces to orient the customer before choosing Path A or Path B: + +| Surface | What to expect | +|---|---| +| PowerShell on a node | `Get-StoragePool`, `Get-VirtualDisk`, and `Get-HealthFault` show the pool allocation, provisioning type, and active storage health faults. | +| Azure portal | The Azure Local cluster's Resource Health, Insights health view, or update-readiness surface can show the pool-capacity warning on an Arc-connected cluster. | +| Windows event logs | `Microsoft-Windows-StorageSpaces-Driver/Operational` carries the provider-scoped events listed in [Data to Collect Before Opening a Support Case](#data-to-collect-before-opening-a-support-case). | +| Cluster logs | `Get-ClusterLog` can help Microsoft Support correlate ownership moves, storage jobs, and cluster resource state around the capacity event. | +| Windows Admin Center on a standalone host | The Storage and health views can show the storage pool capacity warning and current pool utilization. | +| Windows Admin Center in the Azure portal | No separate WAC-in-portal signal is expected for this warning. Use the Azure portal health and update-readiness views instead. | +| Failover Cluster Manager | No dedicated Failover Cluster Manager signal is expected for a capacity-threshold warning by itself. Use it only to confirm VM or CSV state if capacity pressure has already caused workload impact. | +| Component / tool log files on disk | No separate tool-specific on-disk log or diagnostic log file is written for this capacity threshold. Collect the Storage Spaces event log, `Get-HealthFault`, storage cmdlet output, and the cluster log instead. | + ## What and Why ### Why the warning exists @@ -158,7 +211,7 @@ Short definitions for the terms used in this guide: When a capacity drive (or a whole node) is lost, S2D automatically starts repair ("auto-heal") jobs that re-create the missing copies of your data on the remaining drives to restore full resiliency. Those repair jobs need somewhere to -write — they consume free pool capacity. If the pool has no reserve, repair jobs +write; they consume free pool capacity. If the pool has no reserve, repair jobs have nowhere to rebuild and remain **suspended** until the failed drive is physically replaced, leaving the volume running with reduced (or no) redundancy in the meantime. @@ -167,7 +220,7 @@ For this reason Microsoft recommends keeping free pool capacity in reserve. The guidance is to reserve **the equivalent of one capacity drive per server, up to a maximum of four drives** (reserve grows for parity and multi-tier configurations). See -[Plan volumes — reserve capacity](https://learn.microsoft.com/windows-server/storage/storage-spaces/plan-volumes). +[Plan volumes, reserve capacity](https://learn.microsoft.com/windows-server/storage/storage-spaces/plan-volumes). This is especially important on **small clusters (for example, two nodes with a two-way mirror)**: during an update, nodes are drained and rebooted one at a @@ -179,11 +232,11 @@ resilient through the drain. Volumes on Azure Local are either **Thin** or **Fixed** provisioned (thin is the default for new volumes; the default can be changed at the pool level): -- **Fixed** — the volume reserves its full size in the pool at creation time. A +- **Fixed:** the volume reserves its full size in the pool at creation time. A fixed volume on an N-way mirror commits **N × the volume size** of pool footprint up front, regardless of how much data is actually written into it. Deleting data inside the volume does **not** return capacity to the pool. -- **Thin** — the volume consumes pool capacity only as data is written, and unused +- **Thin:** the volume consumes pool capacity only as data is written, and unused capacity can (with the procedure below) be returned to the pool. So a large fixed volume can push the pool over the threshold purely by design. @@ -208,11 +261,11 @@ layer you intend to act on before changing anything. Two different capacity signals exist, and they are frequently conflated: -- **Pool allocation** — how much of the storage *pool* is committed to virtual +- **Pool allocation:** how much of the storage *pool* is committed to virtual disks. This is what the **thin provisioning alert threshold (default 70%)** and the **pool reserve-capacity** check (`System.Storage.StoragePool.CheckPoolReserveCapacity`) evaluate. Pool allocation is the subject of this guide. -- **Volume fill** — how full an individual *volume's* file system is. The Health +- **Volume fill:** how full an individual *volume's* file system is. The Health Service evaluates this against separate, **volume-level** settings whose defaults are `System.Storage.Volume.CapacityThreshold.Warning = 80` and `.Critical = 90` @@ -222,17 +275,17 @@ Two different capacity signals exist, and they are frequently conflated: The pool has **no automatic 80/90/95 capacity ladder and no capacity-based read-only "block."** At the pool level the platform raises only two advisory, -Warning-class health faults — `StoragePool.PoolCapacityThresholdExceeded` (the -configurable 70% thin-provisioning alert) and `StoragePool.InsufficientReserveCapacity` -— and neither takes automatic action. A Storage Spaces pool is set **read-only** -only on **quorum loss** (too many drives offline — operational state `Incomplete`) +Warning-class health faults: `StoragePool.PoolCapacityThresholdExceeded` (the +configurable 70% thin-provisioning alert) and `StoragePool.InsufficientReserveCapacity`. +Neither takes automatic action. A Storage Spaces pool is set **read-only** +only on **quorum loss** (too many drives offline, operational state `Incomplete`) or by **administrator policy** (`Policy`), never from a capacity percentage. See [Storage Spaces and Storage Spaces Direct health and operational states](https://learn.microsoft.com/windows-server/storage/storage-spaces/storage-spaces-states). What actually happens as a pool approaches full is described next. ### What happens if the pool is allowed to fill -Treat the 70% alert as an **early warning**, not the danger line — the real safety +Treat the 70% alert as an **early warning**, not the danger line. The real safety floor is the reserve (the equivalent of one capacity drive per server, up to four). As pool allocation climbs past the reserve toward full, risk escalates: @@ -244,28 +297,28 @@ As pool allocation climbs past the reserve toward full, risk escalates: capacity is exhausted, new allocations fail (on Azure Local 23H2+ with Arc VMs this surfaces as an out-of-capacity error from the underlying virtualization layer). The volume can be taken offline and the affected VMs can stop or enter a - paused state as the platform reacts to the write failure — an unplanned outage, - not a graceful, admin-scheduled action. + paused state as the platform reacts to the write failure. That is an unplanned + outage, not a graceful, admin-scheduled action. -Act while the alert is still an early warning — do the cheapest, most reversible +Act while the alert is still an early warning. Do the cheapest, most reversible things first, and escalate only as needed: 1. **Audit and prune.** Merge or remove stale Hyper-V checkpoints, and find and remove orphaned or stale `.vhdx` files. On thin volumes the reclaimed space returns to the pool gradually (about 15 minutes; see - [Path B](#path-b--thin-provisioned-volumes-reclaim-unused-capacity)). + [Path B](#path-b-thin-provisioned-volumes-reclaim-unused-capacity)). 2. **Restrict new provisioning.** Stop creating new virtual disks or volumes on the pressured pool. 3. **Freeze automated thin-disk or volume expansion** so background growth cannot consume the remaining headroom. 4. **Prepare to expand the pool.** Add OEM-supported physical disks or a node - ([Option A1](#option-a1--add-capacity-recommended-when-growth-is-expected--low-risk)). + ([Option A1](#option-a1-add-capacity-recommended-when-growth-is-expected-low-risk)). Adding capacity is the durable fix. > [!CAUTION] > **Do not respond to a pool or CSV capacity warning by saving VM state.** Saving a -> VM — `Save-VM`, `Stop-VM -Save`, or the **Save the virtual machine state** -> automatic stop action — writes a saved-state file roughly the size of the VM's +> VM, `Save-VM`, `Stop-VM -Save`, or the **Save the virtual machine state** +> automatic stop action, writes a saved-state file roughly the size of the VM's > assigned memory onto its volume (similar to hibernating), consuming the very > capacity you are short of and potentially pushing a nearly-full CSV or pool over > the edge. Pausing a VM with `Suspend-VM` writes no file, but it frees no capacity @@ -273,7 +326,7 @@ things first, and escalate only as needed: > > - A VM whose **automatic stop action** is **Save the virtual machine state** > (historically the default) writes a saved-state file the size of its memory -> onto its volume whenever it is stopped **without a live-migration target** — +> onto its volume whenever it is stopped **without a live-migration target**, > for example during a full-cluster `Stop-Cluster`, or a host OS shutdown of a > non-HA VM. (A node *drain* is space-safe: it live-migrates VMs, copying memory > over the network and leaving the VHDX on the CSV.) A cluster-wide stop can @@ -285,7 +338,7 @@ things first, and escalate only as needed: > volume free space. Extending a thin volume or CSV to create file-system free > space does **not** add pool capacity and can make pool pressure worse. -## Step 1 — Determine the provisioning type (required first) +## Step 1: Determine the provisioning type (required first) Run this on any cluster node before choosing a remediation: @@ -293,8 +346,8 @@ Run this on any cluster node before choosing a remediation: Get-VirtualDisk | Format-Table FriendlyName, ProvisioningType, Size, FootprintOnPool -AutoSize ``` -- `ProvisioningType = Fixed` → follow [Path A](#path-a--fixed-provisioned-volumes). -- `ProvisioningType = Thin` → follow [Path B](#path-b--thin-provisioned-volumes-reclaim-unused-capacity). +- `ProvisioningType = Fixed` → follow [Path A](#path-a-fixed-provisioned-volumes). +- `ProvisioningType = Thin` → follow [Path B](#path-b-thin-provisioned-volumes-reclaim-unused-capacity). > [!IMPORTANT] > Do **not** run `Optimize-Volume -SlabConsolidate` or `Optimize-StoragePool` to @@ -310,31 +363,31 @@ Get-StoragePool | Where-Object IsPrimordial -eq $false | @{N='UsedPct';E={[math]::Round(100*$_.AllocatedSize/$_.Size,1)}} -AutoSize ``` -## Path A — Fixed-provisioned volumes +## Path A: Fixed-provisioned volumes On fixed volumes the pool footprint is committed by design. Choose one or more of the following based on the customer's goal. -### Option A1 — Add capacity (recommended when growth is expected) — [LOW RISK] +### Option A1: Add capacity (recommended when growth is expected) [LOW RISK] Add OEM-supported physical disks so total pool capacity grows and the allocation percentage drops below the threshold. Follow [How to add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md). -### Option A2 — Convert fixed volumes to thin — [MEDIUM RISK] +### Option A2: Convert fixed volumes to thin [MEDIUM RISK] Converting to thin lets the pool charge only for data actually written, which usually drops allocation well below the threshold and enables the reclamation procedure in Path B. Follow the documented procedure: [Convert fixed to thin provisioned volumes on Azure Local](https://learn.microsoft.com/previous-versions/azure/azure-local/manage/thin-provisioning-conversion). -After conversion, run [Path B](#path-b--thin-provisioned-volumes-reclaim-unused-capacity) +After conversion, run [Path B](#path-b-thin-provisioned-volumes-reclaim-unused-capacity) to release the now-unused capacity back to the pool. > [!IMPORTANT] > Microsoft publishes **no minimum build** for in-place fixed-to-thin conversion. > The linked procedure (`Set-VirtualDisk -ProvisioningType Thin` plus a volume > remount) is documented for Azure Stack HCI 21H2/22H2 and is now archived under -> `/previous-versions/` because of the Azure Stack HCI to Azure Local rename — not +> `/previous-versions/` because of the Azure Stack HCI to Azure Local rename, not > a documented removal of the feature. However, the current Azure Local 23H2/24H2 > volume docs do not re-publish an in-place conversion procedure, so confirm it is > still supported on the cluster's current build (against current guidance or with @@ -342,14 +395,14 @@ to release the now-unused capacity back to the pool. > support, create a new thin volume and migrate the data instead, then remove the > old fixed volume. -### Option A3 — Shrink or remove volumes — [MEDIUM RISK] +### Option A3: Shrink or remove volumes [MEDIUM RISK] Reduce committed footprint by removing volumes that are no longer needed, or by recreating a volume at a smaller size. Note that **ReFS does not support in-place volume shrink**, so "shrinking" a fixed ReFS volume means evacuating its data and recreating it smaller. Plan for data movement and downtime. -### Option A4 — Suppress the capacity alert — [MEDIUM RISK] +### Option A4: Suppress the capacity alert [MEDIUM RISK] If the customer accepts the capacity posture and wants to stop the alert, the Health Service threshold alert can be disabled: @@ -367,7 +420,7 @@ Get-StorageSubSystem -FriendlyName Clus* | > [!WARNING] > This setting is applied at the **storage subsystem level** > (`Get-StorageSubSystem ... | Set-StorageHealthSetting`), so it suppresses the -> capacity threshold alert **cluster-wide — for every pool in the subsystem**, not +> capacity threshold alert **cluster-wide, for every pool in the subsystem**, not > just the affected pool or volume. > > Suppressing the alert also hides a **real** safety signal. The underlying capacity @@ -376,10 +429,10 @@ Get-StorageSubSystem -FriendlyName Clus* | > [!NOTE] > Confirm the exact setting name on the live cluster first -> (`Get-StorageSubSystem -FriendlyName Clus* | Get-StorageHealthSetting`) — the +> (`Get-StorageSubSystem -FriendlyName Clus* | Get-StorageHealthSetting`); the > health-setting namespace can vary by build. -### Option A5 — Raise the alert threshold — [MEDIUM RISK] +### Option A5: Raise the alert threshold [MEDIUM RISK] If the goal is to move the threshold rather than silence the alert entirely: @@ -396,7 +449,16 @@ Set-StoragePool -FriendlyName "" -ThinProvisioningAlertThresholds @(8 > Raising the threshold reduces the early-warning margin before the pool runs out > of repair headroom. The same capacity risk applies as in Option A4. -## Path B — Thin-provisioned volumes (reclaim unused capacity) +## Path B: Thin-provisioned volumes (reclaim unused capacity) + +> [!IMPORTANT] +> **Ownership gate (read before starting).** This is a scheduled +> maintenance-window procedure that takes VMs offline; it is owned by the +> customer's cluster or storage administrator. If you are not that +> administrator, or you are unsure whether you are authorized to take these +> workloads offline, stop here and hand off. The read-only Quick triage and the +> [Verify](#verify) queries are always safe to run; the numbered steps below +> are not. On thin volumes, capacity that was written and later deleted can remain committed to the pool in partially used 256 MB "slabs". A slab is only returned to the pool @@ -410,7 +472,7 @@ data into fewer slabs and releases the emptied slabs back to the pool. > close to `Size × resiliency`). If footprint matches the data actually written, > there is nothing to reclaim. -**Procedure (requires an offline window for VMs on the affected volume; the window lasts through slab consolidation, which can take hours on large volumes):** — [MEDIUM RISK] +**Procedure (requires an offline window for VMs on the affected volume; the window lasts through slab consolidation, which can take hours on large volumes):** [MEDIUM RISK] 1. *(Optional, no downtime)* Merge Hyper-V checkpoints that are no longer needed (`Get-VM | Get-VMSnapshot`, then `Remove-VMSnapshot`). Checkpoint files pin @@ -432,15 +494,25 @@ data into fewer slabs and releases the emptied slabs back to the pool. Stop-VM -Name "" # graceful guest shutdown; run on/target the owner node ``` - If a guest will not shut down cleanly (hung, or no integration services), use a - forced turn-off — `Stop-VM -Name "" -TurnOff` — which also releases the - file handles **without** writing a saved-state file. + If a guest will not shut down cleanly (hung, or no integration services), a + forced turn-off also releases the file handles **without** writing a + saved-state file, but only as a last resort **[HIGH RISK]**: + + ```powershell + Stop-VM -Name "" -TurnOff # hard power-off; last resort only + ``` + + > [!WARNING] + > **`-TurnOff` is a hard power-off, the equivalent of pulling the power cord.** + > It can lose unsaved in-guest data and can leave the guest file system dirty. + > Try a graceful `Stop-VM` (in-guest shutdown) first, and force `-TurnOff` only + > after the workload owner has approved it for that specific VM. > [!CAUTION] > Do **not** substitute `Save-VM` (or the **Save** automatic stop action) or > `Suspend-VM` here. **Saving** releases the handles but writes a saved-state > file the size of the VM's memory onto the very volume you are trying to free. - > **Suspending** only *pauses* the VM — its memory stays in host RAM and its + > **Suspending** only *pauses* the VM, its memory stays in host RAM and its > virtual disk handles stay **open**, so slab consolidation cannot proceed. > Putting the cluster resource into redirected access is likewise **not** > sufficient. The VM's file handles must actually be released, which means a @@ -480,15 +552,24 @@ data into fewer slabs and releases the emptied slabs back to the pool. > [!IMPORTANT] > Do **not** add `-ReTrim`. On thin-provisioned ReFS, `-ReTrim` does nothing - > useful — ReFS does not use the NTFS retrim mechanism; it has its own + > useful; ReFS does not use the NTFS retrim mechanism. It has its own > background unmap workitem. (Some older published examples show > `-ReTrim -SlabConsolidate` together; for ReFS, use `-SlabConsolidate` > alone.) Slab consolidation is the time-consuming step and can take hours on > multi-terabyte volumes. + > [!NOTE] + > **Substrate matters if you are validating in a lab.** The reclaim is only + > observable on **physical S2D hardware**. On a nested or VM-based cluster, + > `Optimize-Volume -SlabConsolidate` reports every purgable slab pinned + > unmovable and returns 0 bytes to the pool even when real interior free + > space exists, and the footprint stays flat. That is a substrate limitation, + > not a failure of the procedure and not a defect in the volume. Grade this + > remediation only on physical S2D, never on a nested or VM cluster. + 4. **Wait about 15 minutes** after consolidation completes. The capacity is returned to the pool by the **ReFS background unmap workitem**, which runs - after `Optimize-Volume -SlabConsolidate` finishes — this wait, not the next + after `Optimize-Volume -SlabConsolidate` finishes. This wait, not the next step, is what releases the emptied slabs. > [!NOTE] @@ -507,18 +588,26 @@ data into fewer slabs and releases the emptied slabs back to the pool. step here, not the mechanism that frees the slabs (that already happened in Step 4). Monitor with `Get-StorageJob` and wait until no `Optimize` jobs are running before re-measuring pool fill. If it finishes in seconds with no jobs, - that is expected when there is nothing to rebalance — it does **not** mean + that is expected when there is nothing to rebalance; it does **not** mean reclamation failed; confirm the result with the pool fill query in [Verify](#verify). -6. **Bring the VMs back online:** +6. **Bring the VMs back online.** For **traditional non-Arc Hyper-V VMs**, start + them on the host: ```powershell - Start-VM -Name "" + Start-VM -Name "" # non-Arc Hyper-V VMs only ``` + > [!IMPORTANT] + > For **Arc-enabled Azure Local VMs (23H2+)**, start or restart the VM + > **through Azure** (the VM resource in the portal or CLI), not with host + > `Start-VM`. Driving an Arc VM's power state directly on the host bypasses the + > control plane and can desynchronize the Arc agent and Arc Resource Bridge + > view of the VM state; this mirrors the stop-side boundary in Step 2. + > [!NOTE] -> A consolidation pass can legitimately return little or no capacity — most often +> A consolidation pass can legitimately return little or no capacity, most often > because the volume's footprint already matches the data actually written (there > is nothing to reclaim; see the note at the start of Path B), or because slabs > are still pinned by data in use (confirm every VM on the volume is stopped in @@ -531,12 +620,12 @@ data into fewer slabs and releases the emptied slabs back to the pool. | Volume provisioning | Goal | Use | |---|---|---| -| Fixed | Grow capacity | A1 — add physical disks | -| Fixed | Reduce committed footprint / enable reclamation | A2 — convert to thin, then Path B | -| Fixed | Remove unneeded volumes | A3 — shrink/remove (ReFS = evacuate + recreate) | -| Fixed | Stop the alert (risk accepted) | A4 — disable the Health Service alert | -| Fixed | Move the alert threshold | A5 — raise `ThinProvisioningAlertThresholds` | -| Thin | Return deleted-data capacity to the pool | Path B — SlabConsolidate + ReFS unmap | +| Fixed | Grow capacity | A1: add physical disks | +| Fixed | Reduce committed footprint / enable reclamation | A2: convert to thin, then Path B | +| Fixed | Remove unneeded volumes | A3: shrink/remove (ReFS = evacuate + recreate) | +| Fixed | Stop the alert (risk accepted) | A4: disable the Health Service alert | +| Fixed | Move the alert threshold | A5: raise `ThinProvisioningAlertThresholds` | +| Thin | Return deleted-data capacity to the pool | Path B: SlabConsolidate + ReFS unmap | ## Verify @@ -557,7 +646,10 @@ Get-HealthFault ``` For an upgrade, re-run the solution update readiness check and confirm the -capacity finding is resolved or accepted. +capacity finding is resolved or accepted. When you are clearing the same warning +across many sites, treat this readiness re-run as the per-site validation loop: +remediate, re-run readiness, confirm resolved or accepted, then move to the next +site. ## Data to Collect Before Opening a Support Case @@ -566,8 +658,8 @@ capacity signal fired, the pool / volume / disk state, and the event-log history the threshold crossing and any allocation failures. **Health faults.** The pool-capacity signals surface as Storage Spaces health -faults. Collect the on-box faults with `Get-HealthFault`, and — for an Arc-connected -cluster — also check the resource's **Resource Health** / Insights health view in +faults. Collect the on-box faults with `Get-HealthFault`, and, for an Arc-connected +cluster, also check the resource's **Resource Health** / Insights health view in the Azure portal: ```powershell @@ -578,8 +670,17 @@ The fault types to look for (both Warning class): | Fault type | Meaning | Where it surfaces | |---|---|---| -| `Microsoft.Health.FaultType.StoragePool.InsufficientReserveCapacity` | The pool no longer has the minimum reserve (about two drives' worth) needed to repair resiliency after a drive or node loss. | On-box `Get-HealthFault`. | -| `Microsoft.Health.FaultType.StoragePool.PoolCapacityThresholdExceeded` | The storage pool is running out of capacity (the configurable thin-provisioning alert, default 70%). | Azure portal **Resource Health** / Insights health view; the on-box correlate is **EventID 103** below. | +| `StoragePool.InsufficientReserveCapacity` | The pool no longer has the minimum reserve (the equivalent of one capacity drive per server, up to a maximum of four drives) needed to repair resiliency after a drive or node loss. | On-box `Get-HealthFault`. | +| `StoragePool.PoolCapacityThresholdExceeded` | The storage pool is running out of capacity (the configurable thin-provisioning alert, default 70%). | Azure portal **Resource Health** / Insights health view; the on-box correlate is **EventID 103** below. | + +> [!NOTE] +> The strings above (`StoragePool.InsufficientReserveCapacity` and +> `StoragePool.PoolCapacityThresholdExceeded`) are the exact on-box +> `Get-HealthFault` `FaultType` values to match on. The full internal fault id +> carries an additional `Microsoft.Health.FaultType.` prefix (for example +> `Microsoft.Health.FaultType.StoragePool.PoolCapacityThresholdExceeded`), so +> match on the `StoragePool.` name when guiding a customer through their on-box +> output. **Event log.** Collect these Storage Spaces events from `Microsoft-Windows-StorageSpaces-Driver/Operational` on **every node** (the pool @@ -631,25 +732,25 @@ storage report. ## When to escalate Most capacity warnings are resolved by the paths above. Escalate when one of these -firm conditions is met — do not simply re-run the procedure. +firm conditions is met. Do not simply re-run the procedure. **Escalate to the hardware vendor / OEM when:** - The durable fix is to add capacity - ([Option A1](#option-a1--add-capacity-recommended-when-growth-is-expected--low-risk)), + ([Option A1](#option-a1-add-capacity-recommended-when-growth-is-expected-low-risk)), but the OEM-supported drives are unavailable or the drive model is no longer supported. -- Physical disks have failed or retired and pool capacity dropped as a result — the +- Physical disks have failed or retired and pool capacity dropped as a result. The reserve cannot be restored until the hardware is replaced (a drive replacement is an OEM action). **Escalate to Microsoft support when:** -- **EventID 310 appears, or a thin volume has gone read-only / offline** — the pool +- **EventID 310 appears, or a thin volume has gone read-only / offline**. The pool reached true exhaustion and there is data-path impact. Collect the data above and open the case now; do not wait for the pool to recover on its own. (If the pool *operational state* is `Incomplete` / read-only from a drive-quorum loss rather - than capacity, that is a separate, higher-severity problem — escalate immediately.) + than capacity, that is a separate, higher-severity problem. Escalate immediately.) - **Path B completed with every precondition met** (confirmed real interior free space, every VM on the volume stopped, checkpoints merged) and you waited out the ReFS unmap, but pool `AllocatedSize` still does not drop. diff --git a/TSG/Storage/Troubleshooting-Storage-With-Support-Diagnostics-Tool.md b/TSG/Storage/Troubleshooting-Storage-With-Support-Diagnostics-Tool.md index 1f780b2e..9527903a 100644 --- a/TSG/Storage/Troubleshooting-Storage-With-Support-Diagnostics-Tool.md +++ b/TSG/Storage/Troubleshooting-Storage-With-Support-Diagnostics-Tool.md @@ -1,145 +1,343 @@ -# Overview -The Storage Diagnostic cmdlets included with the Azure Local Support Diagnostic Tool. These cmdlets are designed to help operators and CSS to help troubleshoot storage related issues on Azure Local deployments. - -For more details regarding how to install this tool and examine other cmdlets available, refer to [Support Tools](https://learn.microsoft.com/en-us/azure/azure-local/manage/support-tools) for more information. - -# Start-AzsSupportStorageDiagnostic -This cmdlet performs an automated analysis of the Storage Spaces Direct feature in Azure Local. It provides results and recommendations for next steps, along with in-depth troubleshooting tools to help identify the root cause of any issues detected. - -## Diagnostic Health Checks - -The following checks can be run either all at once by default or initiated individually using the `-Include` switch in `Start-AzsSupportStorageDiagnostic`. Each test is detailed below with the corresponding argument for `-Include`. - -| Check | Result on detection | Argument | -|--------------------------------------------|---------------------|-----------------------------------------| -| Missing Disks from Storage Spaces | INFO | [MissingDisks](#missingdisks) | -| Storage Pool Health Check | FAIL | [StorageHealth](#storagehealth) | -| Cluster Nodes Health Process Running | FAIL | [StorageHealth](#storagehealth) | -| Storage Job Check | WARN | [StorageHealth](#storagehealth) | -| Cluster Node Check | FAIL | [StorageHealth](#storagehealth) | -| Cluster Shared Volumes Check | FAIL | [StorageHealth](#storagehealth) | -| Storage Enclosure Check | FAIL | [StorageHealth](#storagehealth) | -| Health Service Fault Check | WARN | [StorageHealth](#storagehealth) | -| Storage Health Action Check | FAIL | [StorageHealth](#storagehealth) | -| Disks Not in Pool Check | FAIL | [StorageHealth](#storagehealth) | -| Virtual Disk Check | FAIL | [VirtualDisks](#virtualdisks) | -| Dirty Count | FAIL | [DirtyCount](#dirtycount) | -| Support Components Change | INFO | [StorageComponents](#storagecomponents) | -| Support Components Missing | FAIL | [StorageComponents](#storagecomponents) | -| Storage Node View Differs | FAIL | [SNV](#snv-storage-node-view) | -| Firmware Drift | INFO | [FirmwareDrift](#firmwaredrift) | -| Cluster Nodes SMPHost Running | FAIL | [SMPHost](#smphost) | -| SMPHost Issue Detected | FAIL | [SMPHostIssue](#smphostissue) | -| Storage Spaces Partitions Check | FAIL | [DiskHealth](#diskhealth) | -| Disk Health Check | FAIL | [DiskHealth](#diskhealth) | -| Transient Disk Check | FAIL | [DiskHealth](#diskhealth) | - -Example: + + +# Start-AzsSupportStorageDiagnostic storage diagnostics reference + +
Business impactUsually low — a reserve-capacity and update-readiness early + Usually low: a reserve-capacity and update-readiness early warning. (The page severity Medium reflects the signal, not day-to-day impact.) High only if the pool is allowed to fill: thin-volume writes can then fail and affected VMs can pause or go offline (unplanned outage).Typical time to resolve Triage: minutes. Adding disks (A1) or adjusting the alert (A4/A5): low and online. Converting fixed→thin (A2) or the thin reclaim (Path B): a - maintenance window — slab consolidation can take hours on large + maintenance window; slab consolidation can take hours on large volumes.
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ComponentStorage
TopicAzure Local Support Diagnostic Tool: storage diagnostic cmdlet reference for Start-AzsSupportStorageDiagnostic
Document typeReference
AudienceAzure Local operators, Microsoft CSS support engineers, systems integrators, and OEM support engineers who need to capture storage evidence safely.
SeverityInformational: this article documents read-only diagnostics. Follow the linked troubleshooting guide for the specific failure that the diagnostic output identifies.
Highest action classificationRead-only diagnostic: the examples gather command output and write diagnostic artifacts only.
Workload impactNo VM or storage workload disruption is expected from the documented commands. The checks read storage and cluster state and add only light, transient query load, so running VMs are not paused, live migrated, or measurably slowed. The cmdlet may take several minutes and writes a transcript in the tool working directory.
+ +## Table of contents + +- [At a glance](#at-a-glance) +- [Overview](#overview) +- [Scope and safety boundary](#scope-and-safety-boundary) +- [Prerequisites](#prerequisites) +- [Parameters](#parameters) +- [Fast path](#fast-path) +- [Allowed Include tokens](#allowed-include-tokens) +- [Diagnostic check labels and result levels](#diagnostic-check-labels-and-result-levels) +- [Diagnostic information reports](#diagnostic-information-reports) +- [Physical extent analysis](#physical-extent-analysis) +- [Where this appears](#where-this-appears) +- [Verify and capture the diagnostic run](#verify-and-capture-the-diagnostic-run) +- [Glossary](#glossary) +- [Routing after you have output](#routing-after-you-have-output) +- [Related documentation](#related-documentation) + +## At a glance + +**Bottom line:** Run `Start-AzsSupportStorageDiagnostic -Include 'StorageHealth','DiskHealth','VirtualDisks'`, read the `PASS`, `INFO`, `WARN`, or `FAIL` result next to each check label, then use the [Routing after you have output](#routing-after-you-have-output) table to open the correct downstream guide. Do not run any repair command from this reference. + +| Question | Answer | +|----------|--------| +| Workload impact | None expected. Every documented command is read-only and does not move, restart, or reconfigure VMs, disks, or the cluster. | +| Owner | The operator or the Microsoft CSS engineer runs the capture. Each finding then routes to the owner named in the [Routing after you have output](#routing-after-you-have-output) table: a downstream storage TSG, the OEM or vendor for firmware and Supported Components findings, or the product group for an all-`PASS` case that still reproduces. | +| Duration | Usually several minutes for the full run. A single `-Include` token is faster. | +| Maintenance window | Not required, because this is read-only evidence capture. | + +> [!NOTE] +> New to Azure Local? Every command in this article only reads status and writes a log file that you can share with support. Running these commands cannot change the cluster, delete data, retire a disk, or restart a node. If a result tells you to run a repair command, stop and open the linked guide for that finding first. + +## Overview + +`Start-AzsSupportStorageDiagnostic` is part of the Azure Local Support Diagnostic Tool. It runs storage-focused checks for Storage Spaces Direct (S2D), prints check labels with `PASS`, `INFO`, `WARN`, or `FAIL`, and writes a diagnostic transcript for support evidence. + +Use this article as a reference for the cmdlet parameters, supported `-Include` tokens, source check labels, and safe evidence capture. For details on installing or updating the Support Diagnostic Tool, see [Support Tool for Azure Local Hyperconverged Deployments](https://learn.microsoft.com/en-us/azure/azure-local/manage/support-tools). + +## Scope and safety boundary + +### In scope + +- Running `Start-AzsSupportStorageDiagnostic` with no `-Include` value, which runs all storage checks. +- Running targeted checks with the exact source `-Include` tokens listed in this article. +- Capturing module version, command output, and the tool transcript path for CSS or product group review. +- Selecting a virtual disk FriendlyName for `-PhysicalExtentCheck` when a virtual disk has an unexpected state. + +### Out of scope + +- Repairing storage health, retiring disks, restarting the Health Service, or changing cluster resource parameters. +- Deciding whether a `CanPool=False` disk should be added to the pool. Use the general decision-tree router in [Troubleshoot physical disks not claimed after insertion (`CanPool=False`)](./Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md). +- Repairing persistent disk verification stuck behind a wedged Health Service. That scenario is delegated to the dedicated `AzLocal_Storage_PhysicalDiskVerificationStuck` validation spec and its public guidance, not to this reference article. + +> [!IMPORTANT] +> The commands in this article are diagnostic. If the diagnostic output recommends a state-changing command, do not run that command from this reference alone. Open the specific linked troubleshooting guide for that finding, confirm its safety gates, and capture a fresh backup of the evidence first. + +## Prerequisites + +- Run PowerShell as Administrator on an Azure Local node, or use a management host that can reach the cluster through PowerShell remoting. +- Use an account with administrative access to the Azure Local nodes. +- Install or import the `Microsoft.AzLocal.CSSTools` module that contains `Start-AzsSupportStorageDiagnostic`. +- Confirm PowerShell remoting works to the cluster nodes if you use `-ClusterName` or `-Credential`. +- If you plan to use `-PhysicalExtentCheck`, identify the virtual disk FriendlyName first with `Get-VirtualDisk`. + ```powershell -# will default and execute all health tests +$ErrorActionPreference = 'Stop' + +Get-Module Microsoft.AzLocal.CSSTools -ListAvailable | + Sort-Object Version -Descending | + Select-Object Name, Version, Path -First 1 + +Get-Command Start-AzsSupportStorageDiagnostic | + Select-Object Name, ModuleName, Version, Source +``` + +## Parameters + +| Parameter | Type | Required | Purpose | Notes | +|-----------|------|----------|---------|-------| +| `-ClusterName` | `String` | No | Runs diagnostics against the named cluster. | If omitted, the cmdlet attempts to resolve the local cluster name. | +| `-Credential` | `PSCredential` | No | Supplies credentials for remote computers. | If omitted, the current user context is used. | +| `-PhysicalExtentCheck` | `String` | No | Runs physical extent analysis for one virtual disk FriendlyName. | Use the virtual disk FriendlyName from `Get-VirtualDisk`, not a drive letter and not a physical disk name. | +| `-Include` | `String[]` | No | Limits the run to one or more supported storage diagnostic tokens. | If omitted, all storage checks and information reports run. | +| `-ProgressAction` and common parameters | `ActionPreference` and common parameter types | No | Standard PowerShell common-parameter behavior. | `-ProgressAction` is generated by PowerShell help and is not a diagnostic selector. | + +## Fast path + +Use the fast path when you need to unblock a remote support session quickly. + +1. Capture module identity and command source with the prerequisite commands above. +2. Run the broad storage health slice first. +3. Add a narrower token only after the broad slice points to that area. +4. Save the console output and the cmdlet transcript path before starting any remediation guide. + +```powershell +# Run the full storage diagnostic set. Start-AzsSupportStorageDiagnostic -# will just execute the health tests you define -Start-AzsSupportStorageDiagnostic -Include 'DiskHealth','StorageHealth','VirtualDisks' +# Run the common first support slice only. +Start-AzsSupportStorageDiagnostic -Include 'StorageHealth','DiskHealth','VirtualDisks' + +# Run against a named cluster with explicit credentials when needed. +Start-AzsSupportStorageDiagnostic -ClusterName '' -Credential (Get-Credential) -Include 'StorageHealth' ``` -### MissingDisks +## Allowed Include tokens -| Test | Description | -|------|-------------| -|Missing Disks from Storage Spaces | Compares physical disks in non-primordial Storage Pool count against disks detected via Plug and Play (PnP) which are eligible to add to pool. +The source implementation accepts exactly these `-Include` tokens. -### StorageHealth +| Include token | Runs | +|---------------|------| +| `CSVUsage` | Cluster Shared Volume usage report. | +| `DiskHealth` | Per-disk table plus Storage Spaces partition, disk health, and transient disk checks. | +| `StorageSummary` | Storage node, volume, virtual disk, pool, S2D, capacity, cache, and supported-components summary. | +| `StorageComponents` | Supported Components Document comparison and missing component checks. | +| `DirtyCount` | Dirty Region Tracking threshold check. | +| `VirtualDisks` | Virtual disk health check. | +| `MissingDisks` | PnP disk count compared with disks in Storage Spaces. | +| `SNV` | Storage Node View difference check. | +| `FirmwareDrift` | Firmware version drift by physical disk model. | +| `SMPHost` | Storage Management Provider host process check. | +| `SMPHostIssue` | Detached virtual disk with online CSV mismatch check. | +| `StorageHealth` | Storage pool, cluster node, CSV, enclosure, Health Service fault, storage health action, disks-not-in-pool, HealthPIH, and storage job checks. | -| Test | Description | -|------|-------------| -| Storage Pool Health Check | Checks for any Storage Pool that is not in a healthy state, which is determined by anything other than Health Status of OK. -| Cluster Nodes Health Process Running | Ensures that all Nodes in Cluster are running the Health Service process (HealthPIH.exe). -| Storage Job Check | Looks for any Storage Jobs that are running on the system, in a healthy system the expectation is that there will be no jobs running on the system. -| Cluster Node Check | Gets any Cluster Nodes that are in any other state than “Up”, which would indicate a requirement to review. -| Cluster Shared Volumes Check | Confirms that no Cluster Shared Volumes are present that have a state that is not “Online”. -| Storage Enclosure Check | Checks storage enclosures that have a health status that is not healthy. -| Health Service Fault Check | Checks Health Service faults any active issues detected. -| Storage Health Action Check | Health Service actions that have a state other than succeeded, implying that the automated action has failed. -| Disks Not in Pool Check | Checks if any physical disks in Storage Spaces are not present in the non-primordial pool. +## Diagnostic check labels and result levels -### VirtualDisks +The source behavior is: Prints `PASS` when a check finds no matching data. When data is detected, the source assigns the result level shown below. -| Test | Description | -|------|-------------| -| Virtual Disk Check | Confirms if any Virtual Disks are in a Health Status other than that of Healthy. +| Source output label | Result when detected | Include token | What the check looks for | +|---------------------|----------------------|---------------|--------------------------| +| Missing Disks From Storage Spaces | INFO | `MissingDisks` | Difference between disks visible through Plug and Play and disks in Storage Spaces. | +| Storage Pool Health Check | FAIL | `StorageHealth` | Non-primordial storage pool health is not healthy. | +| Cluster Nodes Health Process Running | FAIL | `StorageHealth` | `HealthPIH.exe` is not running on every cluster node. | +| Storage Job Check | WARN | `StorageHealth` | A storage job is not completed, excluding format volume jobs. | +| Cluster Node Check | FAIL | `StorageHealth` | One or more cluster nodes are not `Up`. | +| Cluster Shared Volumes Check | FAIL | `StorageHealth` | One or more Cluster Shared Volumes are not `Online`. | +| Storage Enclosure Check | FAIL | `StorageHealth` | One or more storage enclosures are not healthy. | +| Health Service Fault Check | WARN | `StorageHealth` | Active Health Service storage faults are present. | +| Storage Health Action Check | FAIL | `StorageHealth` | A storage health action is not in the `Succeeded` state. | +| Disks Not In Pool Check | FAIL | `StorageHealth` | Disks visible to the cluster are not present in the non-primordial pool. | +| Virtual Disk Check | FAIL | `VirtualDisks` | A virtual disk health state is not healthy. | +| Dirty Count | FAIL | `DirtyCount` | Dirty Region Tracking count is greater than the threshold. | +| Support Components Change | INFO | `StorageComponents` | The current physical disk and firmware inventory differs from the Supported Components Document and a new supported-components document can be suggested. | +| Support Components Missing | FAIL | `StorageComponents` | Physical disks or firmware versions in Storage Spaces are missing from the Supported Components Document. | +| Storage Node View Differs | FAIL | `SNV` | A physical disk health or operational problem is not seen consistently by all nodes. | +| Firmware Drift | INFO | `FirmwareDrift` | Same physical disk model has more than one firmware version. | +| SMPHost Check | FAIL | `SMPHost` | The Storage Management Provider host service is not running on all nodes. | +| SMPHost Issue Check | FAIL | `SMPHostIssue` | A virtual disk is detached while the related Cluster Shared Volume is still online. | +| Storage Spaces Partitions Check | FAIL | `DiskHealth` | Storage Spaces partitions are missing or do not match expected protective partition patterns. | +| Disk Health Check | FAIL | `DiskHealth` | A physical disk health status is not healthy. | +| Transient Disk Check | FAIL | `DiskHealth` | A physical disk operational status is `Transient Error`. | -### DirtyCount +Expected console output shape: -| Test | Description | -|------|-------------| -Dirty Count | Checks if the Dirty Region Tracking (DRT) has been exceeded for the disk, as the volume will stay offline until cleared. +```text +Storage Pool Health Check [ PASS ] +Storage Job Check [ WARN ] +Virtual Disk Check [ FAIL ] +Firmware Drift [ INFO ] +``` -### StorageComponents -| Test | Description | -|------|-------------| -| Support Components Change | Checks if the Supported Components Document has been specified and if current results `Get-PhysicalDisk` are not supported by this configuration currently and suggests new configuration if valid to avoid quarantine. -| Support Components Missing | Shows if detected the Physical disks and running firmware in storage spaces are not present in Supported Components Document. +The rows above are the source labels and result levels. Your live output may contain a different mix of `PASS`, `INFO`, `WARN`, and `FAIL` based on the cluster state at the time of the run. -### SNV (Storage Node View) -| Test | Description | -|------|-------------| -| Storage Node View Differs | Checks if Storage Node View differs between Nodes for Physical Disks where Health Status not in a healthy state or Operation Status other than OK. +## Diagnostic information reports -### FirmwareDrift -| Test | Description | -|------|-------------| -| Firmware Drift | Reviews if there are models of physical disk with different versions of firmware running. +These `-Include` tokens produce inventory or capacity information rather than a failure check. -### SMPHost -| Test | Description | -|------|-------------| -| Cluster Nodes SMPHost Running | Confirms if the Storage Management Provider host service is running on all Cluster Nodes. +| Report | Include token | Output areas | +|--------|---------------|--------------| +| Storage Summary | `StorageSummary` | Storage Nodes Configuration, Volume Configuration, Virtual Disk Configuration, Pool Configuration, Storage Spaces Direct Configuration, and Capacity Details. | +| Cluster Shared Volume Usage | `CSVUsage` | Cluster Shared Volume space consumption view. | -### SMPHostIssue -| Test | Description | -|------|-------------| -| SMPHost Issue Detected | Checks if Virtual Disks are displaying detached but Cluster Shared Volume shows online. +## Physical extent analysis -### DiskHealth -This provides a table view of all disks configured with key information along with the below: +Use `-PhysicalExtentCheck` only when you need to inspect non-active physical extents for a virtual disk with an unexpected state. The value is the virtual disk FriendlyName. -| Test | Description | -|------|-------------| -| Storage Spaces Partitions Check | Checks if Storage Spaces partitions for chosen disk usage are correctly created. -| Disk Health Check | Checks all Physical disks that are not healthy. -| Transient Disk Check | Checks if any Physical are in a health status of “Transient Error”, as this can be a temporary error or indicative of other issues such as partitions not correctly configured for disk usage. +```powershell +# List candidate virtual disk names first. +Get-VirtualDisk | + Format-Table FriendlyName, HealthStatus, OperationalStatus +# Replace the placeholder with the virtual disk FriendlyName from Get-VirtualDisk. +$virtualDiskName = 'replace-with-virtual-disk-FriendlyName' +Start-AzsSupportStorageDiagnostic -PhysicalExtentCheck $virtualDiskName +``` -## Diagnostic Information and Configuration -The following arguments provide storage information and configuration rather than running tests. +Under the hood, `-PhysicalExtentCheck` calls `Get-AzsSupportStoragePhysicalExtent` for the named virtual disk and reports its non-active physical extents. Expect one of three shapes: -| Report | Argument | -|--------|-------------| -| Storage Summary | [StorageSummary](#storagesummary) | -| Cluster Shared Volume Usage | [CSVUsage](#csvusage) | +- Non-active extents found: a virtual disk object plus `Extents`, `UniqueDisks`, and `Disks` details. +- No non-active extents found on a healthy virtual disk: the additional section is empty. +- No physical disks present to enumerate, for example on a nested or VM-based cluster: the check reports `No PNP device with class DiskDrive found on ` and returns no extent details. This is expected on virtualized substrates and does not indicate a virtual disk fault. This exact message was observed on a nested VM cluster during validation. -### StorageSummary +## Where this appears -| Configuration Check | Description | -|---------------------|-------------| -| Storage Nodes Configuration | Provides information related to the node, serial number, server name, manufacturer, model and last boot time. -| Volume Configuration | Information related to Volumes created from the storage pool. -| Virtual Disk Configuration | Information related to virtual disks created from storage pool. -| Pool Configuration | Information related to non-primordial pool configuration. -| Storage Spaces Direct Configuration | Information related to Storage Spaces configuration. -| Capacity Details | Information related to capacity, cache and supported components. +`Start-AzsSupportStorageDiagnostic` is an on-device PowerShell diagnostic. Its own output appears in the console and in the tool transcript. Underlying storage problems may also appear in other tools, but this article does not claim that every surface shows the diagnostic output. -### CSVUsage +| Admin surface | How this diagnostic appears | Operator note | +|---------------|-----------------------------|---------------| +| PowerShell on an Azure Local node | Shown by `Start-AzsSupportStorageDiagnostic` console output. | Primary surface for this reference. Capture the exact command, module version, and output. | +| Azure portal | Does not appear in Azure portal as this tool's output. | The portal may show the underlying storage alert, but it does not show this cmdlet's check table. | +| Windows event logs | Does not appear in Windows event logs as this tool's output. | Investigate event logs from the specific downstream TSG only after a check points there. | +| Cluster logs (`Get-ClusterLog`) | Does not appear in Get-ClusterLog as this tool's output. | Cluster logs can help with the underlying storage condition, not with the Support Diagnostic Tool run itself. | +| Windows Failover Cluster Manager | Does not appear in Failover Cluster Manager as this tool's output. | Use Failover Cluster Manager only to inspect an underlying cluster role, CSV, or node state that a check identifies. | +| Windows Admin Center on a standalone host | Does not appear in Windows Admin Center as this tool's output. | WAC may show the underlying storage health, not the cmdlet output. | +| Windows Admin Center in the Azure portal | Does not appear in Windows Admin Center in the Azure portal as this tool's output. | Use the PowerShell output and transcript as the evidence package. | +| Component / tool log files (on disk) | Shown in the Support Diagnostic Tool transcript file named `Start-AzsSupportStorageDiagnostic_TraceOutput_.csv` in the tool working directory. | Preserve this transcript with the console output. | -| Configuration Check | Description | -|---------------------|-------------| -| Cluster Shared Volume Usage | Current space consumption +## Verify and capture the diagnostic run -## Additional Analysis -To diagnose physical extent issues causing unexpected states of Virtual Disks, use the `-PhysicalExtentCheck` parameter and specify the friendly name of the drive. This will attempt to identify the root cause of the disk problem. +The goal is to prove the command ran, preserve its output, and keep the run reproducible. The following example writes a local evidence folder on the node or management host. It does not change cluster configuration. -Example: ```powershell -Start-AzsSupportStorageDiagnostic -PhysicalExtentCheck 'NAME' +$ErrorActionPreference = 'Stop' +$stamp = Get-Date -Format 'yyyyMMdd_HHmmss' +$outDir = Join-Path $env:USERPROFILE "AzLocalStorageDiagnostic_$stamp" +New-Item -ItemType Directory -Path $outDir -Force | Out-Null + +$module = Get-Module Microsoft.AzLocal.CSSTools -ListAvailable | + Sort-Object Version -Descending | + Select-Object Name, Version, Path -First 1 + +[pscustomobject]@{ + TimeUtc = (Get-Date).ToUniversalTime().ToString('o') + ModuleName = $module.Name + ModuleVersion = $($module.Version.ToString()) + ModulePath = $module.Path + Command = "Start-AzsSupportStorageDiagnostic -Include 'StorageHealth','DiskHealth','VirtualDisks'" +} | ConvertTo-Json | Out-File -FilePath (Join-Path $outDir 'run-metadata.json') -Encoding utf8 + +Start-AzsSupportStorageDiagnostic -Include 'StorageHealth','DiskHealth','VirtualDisks' *>&1 | + Tee-Object -FilePath (Join-Path $outDir 'storage-diagnostic-output.txt') + +# The tool transcript name carries the date only, so copy it out under a time-stamped +# name before any second run today, otherwise the next same-day run overwrites it. +$workingDir = Get-AzsSupportWorkingDirectory +$transcript = Get-ChildItem -Path $workingDir -Filter 'Start-AzsSupportStorageDiagnostic_TraceOutput_*.csv' | + Sort-Object LastWriteTime -Descending | + Select-Object -First 1 +if ($transcript) { + Copy-Item -Path $transcript.FullName -Destination (Join-Path $outDir "StorageDiagnostic_TraceOutput_$stamp.csv") +} + +Get-ChildItem -Path $outDir ``` + +Verify the diagnostic run before you interpret it: + +- The command completed without a PowerShell error. +- `storage-diagnostic-output.txt` contains the source output labels from this article. +- `run-metadata.json` contains the module version, module path, timestamp, and exact command. +- The Support Diagnostic Tool transcript is named `Start-AzsSupportStorageDiagnostic_TraceOutput_.csv`, which carries a date-only stamp, and is written to the tool working directory. Locate that directory with the module cmdlet `Get-AzsSupportWorkingDirectory`, then list the transcript with `Get-ChildItem -Path (Get-AzsSupportWorkingDirectory) -Filter 'Start-AzsSupportStorageDiagnostic_TraceOutput_*.csv'`. Because the name carries the date only, a second run on the same day overwrites it, so copy each run's transcript out immediately under a name that includes a time stamp such as `HHmmss` before you start the next run, exactly as the capture example above does. +- Every `INFO`, `WARN`, or `FAIL` row is routed to a specific troubleshooting guide before any state-changing action is attempted. + +For a single run the cmdlet already queries every node in the cluster, so cross-node checks such as `Storage Node View Differs`, `SMPHost Check`, and `Firmware Drift` compare all nodes in one pass. For repeat deployments across sites, run the same `-Include` slice on each cluster and keep a per-site `run-metadata.json`. After a downstream guide remediates a finding, re-run the same `-Include` token and confirm the check returns `PASS` to close the loop. + +## Glossary + +| Term | Meaning | +|------|---------| +| S2D | Storage Spaces Direct, the clustered storage technology used by Azure Local. | +| CSV | Cluster Shared Volume, a cluster volume path such as `C:\ClusterStorage\...` that can host workloads. | +| DRT | Dirty Region Tracking, metadata that tracks regions that need repair or synchronization. | +| SNV | Storage Node View, the view of physical disks from each cluster node. Differences can point to visibility or fabric issues. | +| SMPHost | Storage Management Provider host service. The diagnostic source label is `SMPHost Check`. | +| Supported Components Document | Vendor support data used to decide whether disk model and firmware combinations are supported. | +| Physical extent | A physical allocation backing a virtual disk. In this cmdlet, `-PhysicalExtentCheck` starts from a virtual disk FriendlyName. | +| Primordial pool | The built-in pool that lists disks which are available but not yet added to Storage Spaces Direct. The non-primordial pool is the actual S2D pool that backs your volumes, so a healthy cluster serves storage from the non-primordial pool. | +| Protective partition | A reserved partition pattern that Storage Spaces places on a pooled disk. A missing or unexpected pattern can mean a disk is not correctly claimed by the pool. | +| Detached virtual disk | A virtual disk that is no longer attached to the storage stack, so its Cluster Shared Volume cannot be served even when the pool itself is present. | +| PnP | Plug and Play, the Windows device layer that enumerates physical disks. A difference between the PnP disk count and the disks in Storage Spaces points to a disk that hardware can see but the pool cannot. | +| Health Service | The Azure Local cluster service that reports storage faults and runs storage health actions. | +| HealthPIH.exe | The per-node Health Service process. The `Cluster Nodes Health Process Running` check confirms it is running on every cluster node. | + +## Routing after you have output + +Use the diagnostic result to route the next action. Do not treat this reference as a repair procedure. + +| Diagnostic evidence | Route | +|---------------------|-------| +| `CanPool=False` or a `CannotPoolReason` value | Start with [Troubleshoot physical disks not claimed after insertion (`CanPool=False`)](./Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md). That guide remains the general decision-tree router. | +| Persistent `Verification in progress` or `Verification failed` that does not clear, especially with Health Service provider evidence | Use the dedicated PhysicalDiskVerificationStuck guidance tracked by spec `AzLocal_Storage_PhysicalDiskVerificationStuck`. This reference intentionally does not repair Health Service provider configuration. | +| `FirmwareDrift`, `Support Components Change`, or `Support Components Missing` | Capture disk model, firmware, serial number, and the Supported Components output. The Supported Components output names the expected supported model and firmware set, so use it as the comparison baseline, then hand off to the OEM or vendor support path and confirm the target firmware version against the OEM qualified-firmware list. | +| `Storage Node View Differs` or `Storage Enclosure Check` FAIL | A physical disk or enclosure is seen inconsistently across nodes. This can be a storage fabric, cabling, SAS or enclosure, or node network visibility problem, so capture the check output and involve the fabric, network, or hardware owner alongside the downstream storage TSG. | +| `Storage Job Check`, `Virtual Disk Check`, `Dirty Count`, or `Disk Health Check` | Preserve output and transcript, then select the downstream storage TSG for that exact check and confirm its safety gates before any state-changing action. | +| Only `PASS` rows and the issue is still present | Capture the evidence package and escalate through your support path to the product group, because this cmdlet did not detect the reported condition. A Microsoft CSS engineer escalates to the product group with the symptom timeline; a customer or partner opens or updates a support request. | + +## Related documentation + +- [Support Tool for Azure Local Hyperconverged Deployments](https://learn.microsoft.com/en-us/azure/azure-local/manage/support-tools) +- [Troubleshoot physical disks not claimed after insertion (`CanPool=False`)](./Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md) From 704bfd31a8cfb05e12ed84f86cac1c7bf4623abd Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 17:42:13 -0400 Subject: [PATCH 2/3] CanPoolFalse TSG: capture the diagnostic transcript CSV; fix cross-refs 1. The Support Diagnostics evidence copy now resolves the tool's own working directory via Get-AzsSupportWorkingDirectory (falling back to the current directory only if unavailable) instead of assuming the shell CWD, and both copy blocks include .csv so the native Start-AzsSupportStorageDiagnostic_TraceOutput_*.csv transcript is captured rather than silently omitted. 2. The "verification stuck" branch now links the companion guide by relative path (Troubleshoot-Storage-PhysicalDiskVerificationStuck.md) and adds an explicit Microsoft Support (CSS) fallback, so an operator has an actionable next step even before that companion article is published. The stale "PR 333 ownership" table-cell reference is replaced with the same CSS escalation. --- ...roubleshoot-Storage-PhysicalDiskCanPoolFalse.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md b/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md index 1ede2bba..05f6b7a8 100644 --- a/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md +++ b/TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md @@ -143,7 +143,7 @@ The `CannotPoolReason` column uses the exact title-case strings that `Get-Physic | `Offline` | The disk is offline to Windows. | Use [Step 2f](#step-2f-resolve-offline-or-read-only-disk-state). | | `Insufficient Capacity` | The disk does not have enough usable free capacity. This can be because the disk is too small for Azure Local, or because partitions consume the free space. | Use [Step 2g](#step-2g-resolve-insufficient-capacity-or-removable-media). | | `Verification In Progress` | Health Service is checking whether the disk and firmware are approved for the solution. | Use [Step 2d](#step-2d-resolve-verification-in-progress-or-verification-failed). Wait, then recheck. | -| `Verification Failed` | Health Service could not complete supportability verification. | Use [Step 2d](#step-2d-resolve-verification-in-progress-or-verification-failed). Escalate persistent cases to PR 333 ownership. | +| `Verification Failed` | Health Service could not complete supportability verification. | Use [Step 2d](#step-2d-resolve-verification-in-progress-or-verification-failed). Escalate persistent cases to Microsoft Support (CSS). | | `Firmware Not Compliant` | The disk firmware is not approved by the solution vendor support data. | Use [Step 2e](#step-2e-resolve-hardware-not-compliant-or-firmware-not-compliant). | | `Hardware Not Compliant` | The disk model is not approved by the solution vendor support data. | Use [Step 2e](#step-2e-resolve-hardware-not-compliant-or-firmware-not-compliant). | @@ -244,7 +244,7 @@ Get-VirtualDisk | Format-Table FriendlyName, HealthStatus, ``` > [!WARNING] -> Do not reset or manually add disks while verification is still in progress or failed. If `Verification In Progress` or `Verification Failed` remains unchanged after the normal wait of about 10 to 15 minutes and the disk is clean, supported, online, and symmetric, stop this decision tree and use the dedicated Health Service verification-stuck TSG from PR 333 or spec `AzLocal_Storage_PhysicalDiskVerificationStuck`. That companion owns Health resource, SDDC Group, and provider-list repair. +> Do not reset or manually add disks while verification is still in progress or failed. If `Verification In Progress` or `Verification Failed` remains unchanged after the normal wait of about 10 to 15 minutes and the disk is clean, supported, online, and symmetric, stop this decision tree and use the dedicated Health Service verification-stuck guide, [Troubleshoot physical disks stuck in verification (`CanPool=False`)](./Troubleshoot-Storage-PhysicalDiskVerificationStuck.md) (spec `AzLocal_Storage_PhysicalDiskVerificationStuck`), which owns Health resource, SDDC Group, and provider-list repair. If that guide is not yet present in your copy of this repo, engage Microsoft Support (CSS) for the Health Service / SDDC provider-list repair rather than editing the `Providers` cluster parameter yourself. ### Step 2e: Resolve `Hardware Not Compliant` or `Firmware Not Compliant` @@ -452,8 +452,10 @@ New-Item -ItemType Directory -Path $evidenceRoot -Force | Out-Null # Note: Start-AzsSupportStorageDiagnostic stops any outer Start-Transcript when it runs, # so do not rely on Start-Transcript to capture this cmdlet. Redirect all streams instead. -# Record the cmdlet working directory and a start time so its native transcript can be copied out. -$diagWorkingDir = (Get-Location).Path +# Record the tool's own working directory (where AzsSupport writes its native transcript, for +# example Start-AzsSupportStorageDiagnostic_TraceOutput_*.csv) and a start time so that transcript +# can be copied out. Fall back to the current directory only if the cmdlet is unavailable. +$diagWorkingDir = try { Get-AzsSupportWorkingDirectory } catch { (Get-Location).Path } $runStart = Get-Date # Capture ALL streams (success, error, warning, verbose, host, and information) with no ConvertTo-Json. @@ -462,7 +464,7 @@ Start-AzsSupportStorageDiagnostic -Include 'MissingDisks','DiskHealth','StorageH # Copy the native tool transcript the cmdlet writes in its own working directory into the evidence package. Get-ChildItem -Path $diagWorkingDir -Recurse -File | - Where-Object { $_.LastWriteTime -ge $runStart -and $_.Extension -in '.txt','.log','.etl','.zip' } | + Where-Object { $_.LastWriteTime -ge $runStart -and $_.Extension -in '.txt','.log','.etl','.zip','.csv' } | ForEach-Object { Copy-Item -Path $_.FullName -Destination $evidenceRoot -Force } ``` @@ -478,7 +480,7 @@ Start-AzsSupportStorageDiagnostic -PhysicalExtentCheck $virtualDiskFriendlyName # Copy the native tool transcript for this run into the evidence package as well. Get-ChildItem -Path $diagWorkingDir -Recurse -File | - Where-Object { $_.LastWriteTime -ge $extentStart -and $_.Extension -in '.txt','.log','.etl','.zip' } | + Where-Object { $_.LastWriteTime -ge $extentStart -and $_.Extension -in '.txt','.log','.etl','.zip','.csv' } | ForEach-Object { Copy-Item -Path $_.FullName -Destination $evidenceRoot -Force } ``` From ccdc15d99c85beea73a5cfb143ac06bd037a9779 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 18:50:31 -0400 Subject: [PATCH 3/3] AddPhysicalDisks HowTo: label the manual add [MEDIUM RISK] The manual Add-PhysicalDisk step (Step 4) is state-changing (it changes storage pool membership) but carried no canonical risk label, while the sibling Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md labels the identical manual pool add [MEDIUM RISK] and the contributor guidance requires labeling state-changing commands. Add the label to match. (The step already gates with a count-match throw, -WhatIf preview, and an explicit operator confirmation.) --- TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md b/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md index 8d0b1f40..0b14f54e 100644 --- a/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md +++ b/TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md @@ -254,7 +254,7 @@ If this returns no rows and the new disks show `CannotPoolReason = In a Pool`, t ### Step 4: Manually Add Disks Only When Needed -Manual add is appropriate when automatic pooling does not claim eligible disks, the target pool is known, and the disks show `CanPool=True`. +[MEDIUM RISK] Manual add changes storage pool membership. It is appropriate when automatic pooling does not claim eligible disks, the target pool is known, and the disks show `CanPool=True`. First, inspect the current pool and eligible disks: