Add @MinIndexUsageDays to IndexOptimize for safe skipping of unused indexes - #1
Add @MinIndexUsageDays to IndexOptimize for safe skipping of unused indexes#1forward-thinkers-lab wants to merge 301 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an opt-in @MinIndexUsageDays parameter to dbo.IndexOptimize to safely skip maintenance on unused non-clustered indexes by gating decisions on an estimated “age” of sys.dm_db_index_usage_stats.
Changes:
- Introduces
@MinIndexUsageDaysparameter, logs it in the run header, and validates it is non-negative. - Computes a per-database DMV “reset estimate” and trust flag, and (when trusted) deselects non-clustered indexes with zero reads in
sys.dm_db_index_usage_stats. - Adds diagnostic output about the computed reset estimate / trust status.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| -- 3) earliest activity in the DMV -- a lower bound; usage must be at least this old | ||
| IF @MinIndexUsageDays IS NOT NULL | ||
| AND @CurrentDatabaseState = 'ONLINE' | ||
| AND NOT (@CurrentAvailabilityGroup IS NOT NULL AND (@CurrentAvailabilityGroupRole <> 'PRIMARY' OR @CurrentAvailabilityGroupRole IS NULL)) |
There was a problem hiding this comment.
Thanks — agreed this is cleaner and removes the ordering dependency entirely, which is nice. One blocker before I can accept it though:
sys.fn_hadr_is_primary_replica was introduced in SQL Server 2012 along with Availability Groups. It does not exist on SQL Server 2008 / 2008 R2. The PR description commits to "Compatible with SQL Server 2008 through SQL Server 2025", and the surrounding code in IndexOptimize consistently guards HADR-specific calls with IF @Version >= 11 AND SERVERPROPERTY('IsHadrEnabled') = 1 for exactly this reason. Replacing the variable-based check with a direct call to the function would cause the procedure to fail on those older versions.
Two ways forward, happy with whichever the team prefers:
Keep the variable-based check as it stands now (already moved to after the AG/mirroring role discovery in the previous commit, so the ordering bug is fixed). Stays compatible with 2008+.
Adopt the sys.fn_hadr_is_primary_replica approach and drop SQL Server 2008 / 2008 R2 from the supported matrix. That's a bigger conversation than this PR, since both versions are out of extended support but the project still claims to support them.
My preference is option 1 for this PR and opening a separate issue to discuss dropping 2008/2008 R2 across the whole project. Let me know what you'd like.
| ---------------------------------------------------------------------------------------------------- | ||
| --// Log completing information //-- | ||
| ---------------------------------------------------------------------------------------------------- | ||
| ---------------------------------------------C:\Users\yportest\source\repos\sql-server-maintenance-solution\IndexOptimize.sql------------------------------------------------------- |
There was a problem hiding this comment.
Good catch...fixed
|
Hey @yllsuarez — could you take a look at this PR when you have a moment? PR description has the full context. Thanks! |
|
Will review it |
|
Hey @NickMa87 — could you take a look at this PR when you have a moment? PR description has the full context. Thanks! |
|
Very nice work! I will review in depth when have time! |
|
Looks good, clean implementation and the opt-in design with a NULL default is the right call. One small thing I noticed: the "reset internal trust state" block (the three SET lines for @CurrentStatsResetEstimate, @CurrentHoursSinceReset, @CurrentStatsTrusted) appears twice back-to-back around the trust check at I believe around line ~1470. Looks like a leftover from moving the block in the second commit. Easy cleanup to have noted. |
@paureis Thanks, will review your recommendation. |
Add support for updating statistics with PERSIST_SAMPLE_PERCENT
Fix issues with max_duration and low priority locks
Change default for @StatisticsPersistSample from 'N' to NULL
Improve logic for version checks
Add support for data compression for index rebuilds
Fix issue with backup on databases in single_user mode
Improve input parameter validation for @NumberOfFiles, @Directory, @MirrorDirectory, @url and @MirrorURL in DatabaseBackup
Allow the options EXPIREDATE and RETAINDAYS when backing up to URL
Add Is sysadmin logging
Add check for databases with double quotes when backing up using Data Domain Boost
Improve input parameter checks for file name tokens
Improve input parameter validation in DatabaseBackup
Update sql-server-backup.md
…n_updates Updates to backup documentation
…n_updates Update backup documentation
Remove support for ClusterName token in the non-AvailabilityGroup parameters
Fix a minor potential issue with the ServerName token on Managed Instance
Update error message in input parameter validation
Fix incorrect error message in input parameter validation in DatabaseBackup
TL;DR — Why you should care
This change lets
dbo.IndexOptimizestop wasting maintenance time on indexes nobody is using, while making sure it never makes that call on bad data.Concrete benefits:
How safe is the "trust" check?
Very. The script never asks SQL Server "is this DMV trustworthy?" — because SQL Server can't answer that. Instead, it derives a conservative lower bound on the DMV's age from three independent signals (instance uptime, database create date, earliest observed activity). It uses the most recent of the three, so any one of them being short is enough to mark the DMV as untrusted. The trust threshold itself is not user-configurable — it is locked to the same value (
@MinIndexUsageDays) the user already chose, so the trust check can never be weaker than the usage rule it protects. When in doubt, the script does nothing new and falls back to today's behavior. There is no scenario in which this change causes an index to be processed more aggressively than before, and the only way it processes an index less aggressively is when the DMV has demonstrably had enough time to prove that index is unused.What this change does
Adds a new optional parameter to
dbo.IndexOptimize:@MinIndexUsageDaysintNULLWhen set, the procedure will skip non-clustered indexes whose
user_seeks + user_scans + user_lookupsinsys.dm_db_index_usage_statsare zero — but only if the DMV itself has been collecting data for at least that many days. If it hasn't, the unused-index filter is bypassed for that database and normal fragmentation-based maintenance proceeds as usual.Why this is needed
sys.dm_db_index_usage_statsis the only built-in source of "is this index actually being used?" information, but it is volatile. Its counters are reset by:database_id)Because of this, reading the DMV without knowing how long it has been accumulating data can produce false negatives — an index that looks unused may simply not have had time to be queried since the last reset. Acting on that data risks skipping (or, in other tools, dropping) indexes that are in fact heavily used.
This change makes the decision safer by deriving an estimated DMV age per database and gating the unused-index logic on it.
What "trust" means in this change
Throughout the new code, an index usage decision is called trusted when we can reasonably believe
sys.dm_db_index_usage_statshas been collecting data long enough on the current instance to give a meaningful answer.Because SQL Server does not expose a "last reset" timestamp for the DMV, the script estimates the reset time per database as the most recent of three signals:
sys.dm_os_sys_info.sqlserver_start_time.sys.databases.create_date.last_user_seek,last_user_scan,last_user_lookup, andlast_user_updateacross all rows for the database.Taking the MAX of those three gives a conservative estimate of the earliest moment from which the DMV could be trusted. The DMV is then considered:
DATEDIFF(HOUR, estimate, SYSDATETIME()) >= @MinIndexUsageDays * 24How each reset scenario is accounted for
The "Why this is needed" section lists several events that wipe the DMV. Here is how each one is detected by the three signals above:
sqlserver_start_time. Any DMV age computed against it is automatically short until enough time passes.sqlserver_start_timereflects the new start, so the DMV is correctly treated as freshly reset.sqlserver_start_timeon the new primary is older than the failover, so signal 1 alone is not enough — but signal 3 (MIN(last_user_*)) reflects activity only since this replica became primary, which raises the estimate to the time of the role change. The script also explicitly skips the trust check on AG secondaries to avoid drawing conclusions from a DMV that does not represent user activity.database_idand a refreshedcreate_date. The DMV cannot contain rows for the database from before that moment.create_date, raising the estimate to the restore time.zero seeks/scans/lookups) will correctly show no usage history for that specific index. This is acceptable: an index that was just created will not be deselected as unused unless the DB-wide DMV is also old enough to trust, which by definition means enough time has passed.MIN(last_user_*)for the database to the time of the clear, pulling the estimate forward.The trust threshold is derived internally from
@MinIndexUsageDays. There is intentionally no separate parameter to override or weaken it — exposing one would allow the trust check to be set looser than the usage threshold itself, defeating the purpose.Behavior summary
@MinIndexUsageDays IS NULL(default)rdsadminon Amazon RDS, or is notONLINEBackward compatibility
NULL, which preserves existing behavior bit-for-bit.Testing performed
Tests were executed on a dedicated test SQL Server instance against a non-production database. Each scenario was verified by inspecting the procedure's output messages and the contents of
dbo.CommandLog.Regression — default parameters
Called
EXEC dbo.IndexOptimize @Databases = 'USER_DATABASES'with no new parameter set.Expected: identical behavior to the prior version. Result: same set of indexes processed, same commands logged, no new messages.
Trust granted — DMV old enough
On a database whose instance had been running for several days, called
EXEC dbo.IndexOptimize @Databases = 'TestDB', @MinIndexUsageDays = 1.Expected: "Trusted: Yes" line in output; non-clustered indexes with zero reads in
sys.dm_db_index_usage_statsdeselected.Result: matched expectation; the previously-processed unused indexes were skipped on this run.
Trust denied — DMV too young
Restarted the SQL Server service, waited under one hour, then called
EXEC dbo.IndexOptimize @Databases = 'TestDB', @MinIndexUsageDays = 30.Expected: "Trusted: No" line with an explanation; unused-index filter bypassed; fragmentation-based maintenance still runs.
Result: matched expectation; output explained the DMV age and the required age; fragmentation rebuilds/reorganizes proceeded normally.
Negative input validation
Called the procedure with
@MinIndexUsageDays = -1.Expected: validation error from the existing
@Errorsmechanism; procedure does not proceed.Result: error raised as expected.
Clustered indexes are never skipped
Confirmed by inspecting the
@tmpIndexesStatisticsupdate: the filter restricts itself toIndexID > 1. Verified on a test table whose clustered index had no recentuser_seeks/scans/lookups— it was still selected for processing.Multi-version smoke test
Re-deployed the altered procedure on SQL Server 2016, 2019, and 2022 instances and re-ran tests 1–3 on each. Behavior was consistent across all three versions.