MDEV-32286 ANALYZE displays a huge number of InnoDB secondary index pages_accessed - #5362
MDEV-32286 ANALYZE displays a huge number of InnoDB secondary index pages_accessed#5362iMineLink wants to merge 1 commit into
Conversation
|
|
There was a problem hiding this comment.
Pull request overview
This PR addresses MDEV-32286 by reducing inflated pages_accessed reporting during non-covering secondary-index scans in InnoDB, introducing a fast-path that reuses the previously accessed clustered-index leaf page when access patterns show correlation.
Changes:
- Add a clustered-leaf “hint” fast-path (
btr_cur_t::try_leaf_hint()) to avoid repeated root-to-leaf descents on correlated clustered lookups. - Track and reset per-statement hint state in
row_prebuilt_t(remembered leaf page + consecutive miss streak with auto-disable). - Add an MTR test suite case for non-covering secondary index scans and update expected outputs affected by the new access pattern.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| storage/innobase/row/row0sel.cc | Attempts clustered-leaf hint before full clustered-index descent; remembers last clustered leaf page id. |
| storage/innobase/include/row0mysql.h | Adds per-statement state to row_prebuilt_t for clustered leaf hinting and miss streak tracking. |
| storage/innobase/include/btr0cur.h | Declares btr_cur_t::try_leaf_hint() API for direct hinted leaf probing. |
| storage/innobase/handler/ha_innodb.cc | Resets hint state in ha_innobase::reset() to scope hinting to a single statement. |
| storage/innobase/btr/btr0cur.cc | Implements try_leaf_hint() using non-blocking, no-I/O page acquisition and page-local search. |
| mysql-test/suite/innodb/t/non_covering_sec_idx_scan.test | Adds regression/repro test for pages_accessed behavior across correlated and decorrelated patterns. |
| mysql-test/suite/innodb/r/non_covering_sec_idx_scan.result | Captures expected output for the new test case. |
| mysql-test/main/rowid_filter_innodb.result | Updates expected pages_accessed values affected by the optimization. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
4f30143 to
c1f5659
Compare
|
I reordered and adjusted the datatypes of the new On a side note, it could be possible to reorder the fields in the struct to optimize its layout. |
Thirunarayanan
left a comment
Thirunarayanan
left a comment
There was a problem hiding this comment.
Current limitation of this patch is that single page clustered index leaf hint.
Correlated scans: Works well(i.e. Target leaf advances monotonically, so the
single slot walks the tree)
Interleaved scans: consecutive rows bounce between a small set of hot leaves
Uncorrelated scans: Each secondary rows scan a random leaf. Basically
every attempt misses.
Can we cache the leaf's key range, not just its page number? Basically,
store {min_rec_key, max_rec_key, page_no} from the remembered leaf.
Before fetching the hint page, we could test min_rec_key <= clust_ref <= max_rec_key
with in-memory key comparison. How about having std::set<{min_rec_key, max_rec_key}, page_no>? Just an idea.
|
I implemented the key-caching variant similarly to what suggested by @Thirunarayanan, keeping an MRU cache with size 4 and a dedicated re-arm logic. This should prevent buffer pool access on miss (unless the cache is stale). |
| goto search_loop; | ||
| } | ||
|
|
||
| bool btr_cur_t::try_leaf_hint(const dtuple_t *tuple, page_id_t hint_page_id, |
There was a problem hiding this comment.
try_leaf_hint() holds the s-latch on leaf alone. Any caller of this function doesn't rely on tree latch being held? IIUC, descend logic do take tree latch.
There was a problem hiding this comment.
Yes, btr_cur_t::serach_leaf() takes the tree latch to descent (with no BTR_ALREADY_S_LATCHED, the case here): when returning though only the S-latch on the leaf is held, same condition that btr_cur_t::try_leaf_hint() leaves to the caller.
| && row_sel_clust_leaf_hint_armed(prebuilt); | ||
|
|
||
| if (hints_armed | ||
| && row_sel_clust_leaf_hint_search(prebuilt, clust_index, mtr)) { |
There was a problem hiding this comment.
cursor->tree_height could be stale if we jump into this leaf_hint_search. Will it have any consequence ?
There was a problem hiding this comment.
I think not: the critical use readers do is for sizing fsp_reserve_free_extents(), and they refresh their tree height by re-descending the tree with BTR_MODIFY_TREE latch mode.
| is enabled, the hints stay out of its way entirely, neither used nor | ||
| collected. */ | ||
| #ifdef BTR_CUR_HASH_ADAPT | ||
| const bool use_hints = !btr_search_enabled; |
There was a problem hiding this comment.
what if enable AHI in the middle of the query?
There was a problem hiding this comment.
The variable is re-read at every clustered leaf lookup. Therefore, next lookup will disable hints (they will age out at the next ha_innobase::reset() at the end of the statement) and start possibly using AHI if pattern permits. If AHI is disabled in middle of query, the hints will start get used (again) likely needing refresh before being effective. The test is coarse: it can be I think that AHI is enabled, but not usable on the index for some reasons (one could be MDEV-37070 in newer versions), still we disable hints; I guess it's an acceptable tradeoff, since both are heuristics and do not affect correctness.
Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree from the root for every row of a non-covering secondary-index scan, although consecutive rows often land on the same clustered leaf page. ANALYZE FORMAT=JSON charges each descent its full height, and those descents are nearly the whole cost: the secondary index is charged its own descent and one page for each further leaf, and nothing per row, because the position that its cursor holds between two rows is restored optimistically, which latches the leaf again without counting an access. So pages_accessed is the row count times the height of the clustered index, plus that handful. 1000 rows over a 2-level clustered index cost 2003, of which 3 are the secondary index, and 750 rows over a 3-level one cost 2277, where a full table scan of the same data costs 23 and 110. Remember, in the new row_prebuilt_t::clust_leaf_hint, the CLUST_LEAF_HINT_SLOTS (4) clustered leaves that the lookups of this statement reached, most recently used first. Each slot names one leaf: its page number, copies of its first and last user record truncated to the key fields, which bound the key range the leaf held when it was remembered, and the rec_get_offsets() of both. The copies are needed because the page is unlatched between two lookups, and the offsets spare a lookup the parsing of them. Several slots serve the scans that alternate between a few leaves, which one slot cannot serve at all, and a descent refreshes the slot of a leaf that is remembered already rather than spend a second one on the same page. A slot owns its key buffers and grows them only when a longer key arrives, so a row allocates nothing. The used-slot count and the miss counter are reset per statement in ha_innobase::reset(), matching autoinc_last_value. A lookup first compares its key against the remembered ranges, so an uncorrelated scan settles its misses in memory, with no buffer pool access and no pages_accessed. Only a covering range is probed, through the new btr_cur_t::try_leaf_hint(), which acquires the page with buf_page_try_get(): a hint is never derived from a latched parent page, so by the time it is tried it may precede the caller's already-latched secondary-index leaf in the latching order, where a blocking wait can deadlock. A stale range costs a wasted probe or a needless descent, never a wrong result, because the checks that try_leaf_hint() makes on the latched page remain the sole authority, and the ranges therefore need no invalidation protocol. After CLUST_LEAF_HINT_MAX_MISSES (8) consecutive unanswered lookups, a scan gives the slots up: row_sel_clust_leaf_hint_armed() stops both the test of the slots and the copies that refresh them, which are the larger half of their cost. One lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count again, so a scan whose order becomes correlated only later recovers, and the trial that this begins refreshes the slots as it goes. The run is short because a hit saves little where the pages above the leaf are resident: one buffer pool access and one page-local search for each level. Measured against the same tree built without the hints, at 16k with a resident working set and no adaptive hash index, a wholly correlated scan runs 30% faster over half the page accesses, a scan that answers three lookups in five runs level with it over 30% fewer, one whose locality appears only half way through runs 16% faster over a quarter fewer, and a scan that answers nothing stays within the noise. A run of 8 is what keeps that last one there. Where the adaptive hash index is enabled, the hints are neither used nor collected: its guess solves the same problem better, landing on the record with no page-local search and no page access to charge. It is off by default, so the hints are active in a default configuration. innodb.non_covering_sec_idx_scan measures pages_accessed over key orders that differ in how closely the secondary order tracks the clustered one, and seven further tables check query results over the record formats and key shapes that a clustered-index lookup has to read, down to the metadata pseudo-record of instant ALTER TABLE and to leaves that split and merge while a locking read walks them. non_covering_sec_idx_scan_debug runs the same body with the hints turned off, through a debug switch that returns before a lookup tests or refreshes the slots, so a diff of the two .result files is what the hints save: 2003 to 1025 (2-level clustered index), 2277 to 993 (3-level), 20010 to 15770 (decorrelated), 4003 to 2009 (two interleaved key ranges), 20010 to 19996 (shuffled) and 12008 to 9281 (locality in the second half alone). The last of those pins the retry: without it the count is 11926. main.rowid_filter_innodb: 84 to 72.
|
Superseded by #5625. |
Added two commits, one for reproducing the issue cited in the ticket, and the second one with an optimization that adds a shortcut to directly fetch the clustered index leaf page with an hint (without full traversal) in case there's correlation during a non-covering secondary index scan.The optimization self-disengages after 2 consecutive failures.That's the price these kind of queries have to pay in terms ofpages_accessedeven when they don't benefit from this (maybe due to fully decorrelated secondary-to-clustered index access pattern).Row_sel_get_clust_rec_for_mysql::operator() descends the clustered B-tree
from the root for every row of a non-covering secondary-index scan, although
consecutive rows often land on the same clustered leaf page. ANALYZE
FORMAT=JSON charges each descent its full height, and those descents are
nearly the whole cost: the secondary index is charged its own descent and
one page for each further leaf, and nothing per row, because the position
that its cursor holds between two rows is restored optimistically, which
latches the leaf again without counting an access. So pages_accessed is the
row count times the height of the clustered index, plus that handful. 1000
rows over a 2-level clustered index cost 2003, of which 3 are the secondary
index, and 750 rows over a 3-level one cost 2277, where a full table scan of
the same data costs 23 and 110.
Remember, in the new row_prebuilt_t::clust_leaf_hint, the
CLUST_LEAF_HINT_SLOTS (4) clustered leaves that the lookups of this
statement reached, most recently used first. Each slot names one leaf: its
page number, copies of its first and last user record truncated to the key
fields, which bound the key range the leaf held when it was remembered, and
the rec_get_offsets() of both. The copies are needed because the page is
unlatched between two lookups, and the offsets spare a lookup the parsing of
them. Several slots serve the scans that alternate between a few leaves,
which one slot cannot serve at all, and a descent refreshes the slot of a
leaf that is remembered already rather than spend a second one on the same
page. A slot owns its key buffers and grows them only when a longer key
arrives, so a row allocates nothing. The used-slot count and the miss
counter are reset per statement in ha_innobase::reset(), matching
autoinc_last_value.
A lookup first compares its key against the remembered ranges, so an
uncorrelated scan settles its misses in memory, with no buffer pool access
and no pages_accessed. Only a covering range is probed, through the new
btr_cur_t::try_leaf_hint(), which acquires the page with buf_page_try_get():
a hint is never derived from a latched parent page, so by the time it is
tried it may precede the caller's already-latched secondary-index leaf in
the latching order, where a blocking wait can deadlock. A stale range costs
a wasted probe or a needless descent, never a wrong result, because the
checks that try_leaf_hint() makes on the latched page remain the sole
authority, and the ranges therefore need no invalidation protocol.
After CLUST_LEAF_HINT_MAX_MISSES (8) consecutive unanswered lookups, a scan
gives the slots up: row_sel_clust_leaf_hint_armed() stops both the test of
the slots and the copies that refresh them, which are the larger half of
their cost. One lookup in CLUST_LEAF_HINT_RETRY (1024) starts the count
again, so a scan whose order becomes correlated only later recovers, and the
trial that this begins refreshes the slots as it goes.
The run is short because a hit saves little where the pages above the leaf
are resident: one buffer pool access and one page-local search for each
level. Measured against the same tree built without the hints, at 16k with a
resident working set and no adaptive hash index, a wholly correlated scan
runs 30% faster over half the page accesses, a scan that answers three
lookups in five runs level with it over 30% fewer, one whose locality
appears only half way through runs 16% faster over a quarter fewer, and a
scan that answers nothing stays within the noise. A run of 8 is what keeps
that last one there.
Where the adaptive hash index is enabled, the hints are neither used nor
collected: its guess solves the same problem better, landing on the record
with no page-local search and no page access to charge. It is off by
default, so the hints are active in a default configuration.
innodb.non_covering_sec_idx_scan measures pages_accessed over key orders
that differ in how closely the secondary order tracks the clustered one, and
seven further tables check query results over the record formats and key
shapes that a clustered-index lookup has to read, down to the metadata
pseudo-record of instant ALTER TABLE and to leaves that split and merge
while a locking read walks them. non_covering_sec_idx_scan_debug runs the
same body with the hints turned off, through a debug switch that returns
before a lookup tests or refreshes the slots, so a diff of the two .result
files is what the hints save: 2003 to 1025 (2-level clustered index), 2277
to 993 (3-level), 20010 to 15770 (decorrelated), 4003 to 2009 (two
interleaved key ranges), 20010 to 19996 (shuffled) and 12008 to 9281
(locality in the second half alone). The last of those pins the retry:
without it the count is 11926.
main.rowid_filter_innodb: 84 to 72.