diff --git a/.gitignore b/.gitignore index bbf3d54d032cc..5c58575e5a971 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,4 @@ gen /node_modules/ tmp/ +.qoder/settings.local.json diff --git a/TOC-ai.md b/TOC-ai.md index 6437c43935a39..7868ad1662b2a 100644 --- a/TOC-ai.md +++ b/TOC-ai.md @@ -82,3 +82,8 @@ - [Vector Search Index](/ai/reference/vector-search-index.md) - [Vector Search Performance Tuning](/ai/reference/vector-search-improve-performance.md) - [Vector Search Limitations](/ai/reference/vector-search-limitations.md) +- Full-Text Search + - [Full-Text Search Index](/ai/reference/full-text-search-index.md) + - [Full-Text Search Functions](/ai/reference/full-text-search-functions.md) + - [Full-Text Search Observability](/ai/reference/full-text-search-observability.md) + - [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md) diff --git a/ai/guides/vector-search-full-text-search-sql.md b/ai/guides/vector-search-full-text-search-sql.md index 22f89e1eb8dbf..d7679c69d3a2f 100644 --- a/ai/guides/vector-search-full-text-search-sql.md +++ b/ai/guides/vector-search-full-text-search-sql.md @@ -16,6 +16,12 @@ The full-text search feature in TiDB provides the following capabilities: - **Order by relevance**: the search result can be ordered by relevance using the widely adopted [BM25 ranking](https://en.wikipedia.org/wiki/Okapi_BM25) algorithm. +- **Multi-column search**: you can define multiple scored columns in one full-text index and search across them in a single query. BM25 scores are fused at the index level. + +- **Filter pushdown**: you can add filter columns (such as tenant IDs, status, and file paths) to a full-text index. Filter conditions on these columns are evaluated during the index scan, without accessing the table rows. + +- **Substring matching**: with the NGRAM parser, queries can match prefixes and substrings, such as matching `panic_handler` when searching for `panic`. + - **Fully compatible with SQL**: all SQL features, such as pre-filtering, post-filtering, grouping, and joining, can be used with full-text search. > **Tip:** @@ -65,6 +71,8 @@ CREATE TABLE stock_items( -- You might insert some data here. -- The full-text index can be created even if data is already in the table. +-- ADD_COLUMNAR_REPLICA_ON_DEMAND is optional. If you omit it, +-- make sure that a TiFlash replica is already created for the table. ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL ADD_COLUMNAR_REPLICA_ON_DEMAND; ``` @@ -74,19 +82,20 @@ The following parsers are accepted in the `WITH PARSER ` clause: - `MULTILINGUAL`: supports multiple languages, including English, Chinese, Japanese, and Korean. +- `NGRAM`: builds character-level n-grams so that queries can match prefixes and substrings. See [The NGRAM parser](/ai/reference/full-text-search-index.md#the-ngram-parser) for parameters. + ### Manage full-text indexes -When creating a full-text index, specifying an index name is optional. If you do not specify one, TiDB uses the name of the first indexed column as the index name by default. +When creating a full-text index, specifying an index name is optional. If you do not specify one, TiDB uses the name of the first indexed column as the index name by default. ```sql --- Without specifying an index name, TiDB uses the first indexed column name ("title") as the index name +-- Without specifying an index name, TiDB uses the first indexed column name ("title") as the index name ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL; -- Specifying an index name ALTER TABLE stock_items ADD FULLTEXT INDEX ft_title (title) WITH PARSER MULTILINGUAL; ``` -**View existing index names:** ```sql -- The Key_name column shows the index name @@ -98,7 +107,7 @@ FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = 'your_database' AND TABLE_NAME = 'stock_items'; ``` -**Drop a full-text index:** +**Drop a full-text index:** ```sql -- Use SHOW INDEX to confirm the index name first @@ -107,7 +116,7 @@ ALTER TABLE stock_items DROP INDEX title; #### Specify an index name -In both `CREATE TABLE` and `ALTER TABLE` statements, you can specify an index name after `FULLTEXT INDEX` or `FULLTEXT KEY`: +In both `CREATE TABLE` and `ALTER TABLE` statements, you can specify an index name after `FULLTEXT INDEX` or `FULLTEXT KEY`: ```sql -- Specifying a name in CREATE TABLE @@ -124,6 +133,60 @@ ALTER TABLE users ADD FULLTEXT INDEX ft_name (name) WITH PARSER STANDARD; CREATE FULLTEXT INDEX ft_name ON users (name) WITH PARSER STANDARD; ``` +### Create a multi-column full-text index + +A full-text index can contain multiple scored columns. Searching across them in one query fuses the BM25 scores at the index level, which replaces scanning one single-column index per column and merging results with `UNION ALL`. + +```sql +ALTER TABLE articles ADD FULLTEXT INDEX ft_article (title, body) WITH PARSER MULTILINGUAL; + +SELECT * FROM articles + WHERE fts_match_word('database', title, body) + ORDER BY fts_match_word('database', title, body) DESC LIMIT 10; +``` + +Columns in one call are combined with OR semantics: a document matches if any of the columns matches the query. For AND semantics and more details, see [Multi-column search](/ai/reference/full-text-search-functions.md#multi-column-search). + +### Filter pushdown + +Besides scored columns, a full-text index can contain filter columns. Filter conditions on these columns are evaluated during the index scan, so you get correct scoped Top-K results instead of filtering after a global Top-K. Filter columns are defined with the column-property syntax: + +```sql +ALTER TABLE files ADD FULLTEXT INDEX idx_fts ( + content_text WITH (multilingual), + path WITH (exact, path_hierarchy), + ext WITH (exact) +); + +SELECT * FROM files + WHERE fts_match_word('database', content_text) + AND path LIKE '/src/%' + AND ext IN ('go', 'rs') + ORDER BY fts_match_word('database', content_text) DESC LIMIT 10; +``` + +- The `exact` attribute supports `=` and `IN` matching during the index scan. It is suitable for tenant IDs, status, tags, and other low-cardinality columns. +- The `path_hierarchy` attribute supports hierarchical prefix matching such as `path LIKE '/src/%'`, where the prefix aligns with the `/` delimiter boundary. + +For the full attribute reference, syntax rules, and pushdown limitations, see [Full-Text Search Index](/ai/reference/full-text-search-index.md) and [Filter pushdown limitations](/ai/reference/full-text-search-limitations.md#filter-pushdown-limitations). + +### Substring matching with the NGRAM parser + +The `STANDARD` and `MULTILINGUAL` parsers match complete tokens only. For partial-recall scenarios such as code search, create a full-text index with the `NGRAM` parser to match prefixes and substrings: + +```sql +ALTER TABLE code_files ADD FULLTEXT INDEX idx_fts_ngram (content_text) + WITH PARSER NGRAM(min_gram=3, max_gram=3); + +-- Matches HandleRequest, RequestHandler, and handle_error +SELECT /*+ USE_INDEX(code_files, idx_fts_ngram) */ * +FROM code_files + WHERE fts_match_word('handle', content_text) + ORDER BY fts_match_word('handle', content_text) DESC LIMIT 10; +``` + +A table can have multiple full-text indexes, and the same column can participate in several of them with different parsers. Use the `USE_INDEX` hint to select an index at query time, or let the optimizer choose automatically. For parameters and query semantics, see [The NGRAM parser](/ai/reference/full-text-search-index.md#the-ngram-parser) and [Choose a full-text index at query time](/ai/reference/full-text-search-functions.md#choose-a-full-text-index-at-query-time). + ### Insert text data Inserting data into a table with a full-text index is identical to inserting data into any other tables. @@ -204,9 +267,9 @@ SELECT COUNT(*) FROM stock_items #### Multi-word search: tokenization and query semantics -When you use `fts_match_word()`, the query string is tokenized according to the parser's rules, and each token is matched independently. +When you use `fts_match_word()`, the query string is tokenized according to the parser's rules, and each token is matched independently. -The STANDARD parser tokenizes strings into words using spaces and punctuation as delimiters. The MULTILINGUAL parser tokenizes strings according to language-specific segmentation rules. +The STANDARD parser tokenizes strings into words using spaces and punctuation as delimiters. The MULTILINGUAL parser tokenizes strings according to language-specific segmentation rules. ```sql -- This query is tokenized into two tokens: "Alice" and "Smith" @@ -221,15 +284,15 @@ SELECT * FROM users WHERE fts_match_word('Alice Smith', name); SELECT * FROM users WHERE fts_match_word('Alice Smith', name); ``` -A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Because `X` is a very short query term, it can match many irrelevant documents. Avoid using very short query terms or single letters. +A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Because `X` is a very short query term, it can match many irrelevant documents. Avoid using very short query terms or single letters. -> **Note:** -> -> TiDB full-text search does not support exact phrase matching, where all query tokens must appear consecutively and in the specified order. +> **Note:** +> +> TiDB full-text search does not support exact phrase matching, where all query tokens must appear consecutively and in the specified order. -#### Prefix search +#### Prefix and substring search -**Not supported.** +To match prefixes or substrings, use a full-text index with the `NGRAM` parser. See [Substring matching with the NGRAM parser](#substring-matching-with-the-ngram-parser). To match path prefixes such as `/src/`, use a filter column with the `path_hierarchy` attribute. See [Filter pushdown](#filter-pushdown). #### Effect of repeated terms on relevance scores @@ -244,7 +307,7 @@ In this example, a document matching `Alice` receives twice the weight contribut #### Relevance scoring algorithm -TiDB full-text search uses the **BM25Tantivy** algorithm to calculate relevance scores. This algorithm is a variant of the classic BM25 (Okapi BM25) algorithm that uses Count-Min Sketch to approximate document frequency (DF) for improved performance. +TiDB full-text search uses the **BM25Tantivy** algorithm to calculate relevance scores. This algorithm is a variant of the classic BM25 (Okapi BM25) algorithm that uses Count-Min Sketch to approximate document frequency (DF) for improved performance. **BM25 formula (standard form):** @@ -316,6 +379,10 @@ WHERE t.author_id IN ## See also - [Hybrid Search](/ai/guides/vector-search-hybrid-search.md) +- [Full-Text Search Index](/ai/reference/full-text-search-index.md) +- [Full-Text Search Functions](/ai/reference/full-text-search-functions.md) +- [Full-Text Search Observability](/ai/reference/full-text-search-observability.md) +- [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md) ## Feedback & help diff --git a/ai/reference/full-text-search-functions.md b/ai/reference/full-text-search-functions.md new file mode 100644 index 0000000000000..86b5d715de518 --- /dev/null +++ b/ai/reference/full-text-search-functions.md @@ -0,0 +1,215 @@ +--- +title: Full-Text Search Functions +summary: Learn the full-text search functions in TiDB, including query syntax, multi-column search, parser selection, and index selection hints. +aliases: ['/tidb/stable/full-text-search-functions/','/tidbcloud/full-text-search-functions/'] +--- + +# Full-Text Search Functions + +This document describes the `FTS_MATCH_WORD()` function for full-text search in TiDB, including multi-column search, query semantics with different parsers, parser selection, and index selection hints. + +For how to create the full-text indexes that these functions use, see [Full-Text Search Index](/ai/reference/full-text-search-index.md). + +## FTS_MATCH_WORD() + +`FTS_MATCH_WORD()` performs a keyword search against one or more scored columns of a full-text index and returns a BM25 relevance score. + +``` +FTS_MATCH_WORD('query' [WITH PARSER parser_name], col [, col...]) +``` + +| Parameter | Description | +| :-- | :-- | +| `query` | The search text. It is tokenized according to the parser rules of the matched index. | +| `WITH PARSER parser_name` | Optional. Explicitly selects which parser to use when the scored column has multiple parsers. Only a parser name is accepted, such as `multilingual` or `ngram`. Parser parameters are not allowed. See [Select a parser explicitly](#select-a-parser-explicitly). | +| `col [, col...]` | One or more scored columns. The columns can be a subset of the scored columns of a full-text index. | + +Filter conditions on filter columns are written outside the function, as independent `WHERE` conditions: + +```sql +SELECT * FROM t +WHERE FTS_MATCH_WORD('database', content_text) + AND path LIKE '/src/%' + AND name = 'main.go' +ORDER BY FTS_MATCH_WORD('database', content_text) DESC +LIMIT 10; +``` + +If a column in `col [, col...]` does not exist in the table, TiDB reports `ERROR 1054 (42S22): Unknown column 'xxx' in 'fts_match_word'`. + +## Multi-column search + +You can search multiple scored columns in a single call. Columns are combined with **OR** semantics: `FTS_MATCH_WORD('query', col1, col2)` matches documents where `col1` matches the query or `col2` matches the query, and BM25 scores are fused at the index level. + +```sql +SELECT * FROM t +WHERE FTS_MATCH_WORD('database', content_text, description) +ORDER BY FTS_MATCH_WORD('database', content_text, description) DESC +LIMIT 10; +``` + +To require all columns to match (**AND** semantics), combine multiple `FTS_MATCH_WORD()` calls with `AND` in the `WHERE` clause: + +```sql +SELECT * FROM t +WHERE FTS_MATCH_WORD('database', content_text) + AND FTS_MATCH_WORD('database', description); +``` + +In a multi-column query, each column is tokenized by the parser defined for that column in the index, and the BM25 scores are fused at the index level. A query can also use a subset of the scored columns of an index. For example, if an index covers `content_text` and `description`, `FTS_MATCH_WORD('query', content_text)` still uses that index. + +## Multi-word query semantics + +When the query string contains multiple words, it is tokenized according to the parser rules, and each token is matched independently. `FTS_MATCH_WORD()` uses **OR** semantics across tokens: a document matches if it contains any of the tokens, and matching more tokens increases the relevance score. + +```sql +-- This query is tokenized into two tokens: "Alice" and "Smith". +-- It returns all rows where name contains "Alice" or "Smith" or both. +SELECT * FROM users WHERE FTS_MATCH_WORD('Alice Smith', name); +``` + +To require multiple words to match (**AND** semantics), combine multiple `FTS_MATCH_WORD()` calls with `AND`. The optimizer merges the conditions into a single index scan: + +```sql +SELECT * FROM t +WHERE FTS_MATCH_WORD('database', content_text) + AND FTS_MATCH_WORD('vector', content_text); +``` + +Similarly, multiple calls combined with `OR` are merged into one scan with OR semantics: + +```sql +SELECT * FROM t +WHERE FTS_MATCH_WORD('database', content_text) + OR FTS_MATCH_WORD('vector', content_text); +``` + +## Query semantics with the NGRAM parser + +When the matched index uses the `NGRAM` parser, `FTS_MATCH_WORD()` performs substring matching. The query string is split into n-grams using the same `min_gram` and `max_gram` settings as the index, and the n-grams are combined with **AND** semantics by default. + +For example, with `min_gram=max_gram=3`, searching `apple` is equivalent to searching `app` AND `ppl` AND `ple`: all n-grams must match in the same document. + +```sql +SELECT /*+ USE_INDEX(t, idx_fts_ngram) */ * +FROM t +WHERE FTS_MATCH_WORD('handle', content_text); +-- Matches HandleRequest, RequestHandler, and handle_error +``` + +To use OR semantics between n-grams, combine multiple `FTS_MATCH_WORD()` calls with `OR`: + +```sql +SELECT * FROM t +WHERE FTS_MATCH_WORD('app', content_text) + OR FTS_MATCH_WORD('ppl', content_text); +``` + +## Relevance scoring + +`FTS_MATCH_WORD()` returns a BM25 relevance score. The score is a non-negative floating-point number. A higher value indicates higher relevance. Scores are not directly comparable across different datasets. + +If the query string contains repeated terms, the term frequency of that term is counted multiple times in scoring. For example, in `FTS_MATCH_WORD('Alice alice bob', name)`, the term `Alice` contributes twice the weight of `bob`. This is expected BM25 behavior. + +For the full scoring formula and parameters, see [Relevance scoring algorithm](/ai/guides/vector-search-full-text-search-sql.md#relevance-scoring-algorithm). + +## Select a parser explicitly + +When a scored column has multiple parsers (for example, `content_text WITH (multilingual, ngram(...))`), use the `WITH PARSER` clause inside the function to select one explicitly. Only a parser name is accepted. Parser parameters such as `WITH PARSER ngram(min_gram=5)` are not allowed and return an error. + +```sql +-- Use the ngram parser explicitly +SELECT * FROM t +WHERE FTS_MATCH_WORD('handle' WITH PARSER ngram, content_text); + +-- Without WITH PARSER, the first compatible parser in the index definition is used +SELECT * FROM t +WHERE FTS_MATCH_WORD('database', content_text); +``` + +Parser selection rules: + +| Scenario | Behavior | +| :-- | :-- | +| `WITH PARSER parser_name` is specified, and the column has an index with that parser | The specified parser is used. | +| `WITH PARSER parser_name` is specified, but the column has no index with that parser | An error is returned: `ERROR: Parser 'xxx' not found for column 'yyy' in fts_match_word`. TiDB does not silently fall back to another parser. | +| `WITH PARSER` is omitted, and the column has one compatible parser | That parser is used. | +| `WITH PARSER` is omitted, and the column has multiple compatible parsers | The first compatible parser in the index definition is used. If `USE_INDEX` is also specified, the parser of the specified index is used. See [Choose a full-text index at query time](#choose-a-full-text-index-at-query-time). | + +> **Note:** +> +> Parser selection is part of query semantics, not performance tuning. Different parsers return different results. For this reason, you select a parser with `WITH PARSER` inside the function, not with an optimizer hint. If the specified parser does not exist, TiDB reports an error instead of falling back silently, so that results always match the parser you requested. + +## Choose a full-text index at query time + +When a table has multiple full-text indexes, you can control which index is used: + +### USE_INDEX: specify an index explicitly + +```sql +-- Use the NGRAM index (substring matching) +SELECT /*+ USE_INDEX(t, idx_fts_ngram) */ * +FROM t WHERE FTS_MATCH_WORD('handle', content_text); + +-- Use the MULTILINGUAL index (complete token matching) +SELECT /*+ USE_INDEX(t, idx_fts_ml) */ * +FROM t WHERE FTS_MATCH_WORD('database', content_text); +``` + +`USE_INDEX` semantics: + +| Scenario | Behavior | +| :-- | :-- | +| Intended use case | Select among multiple indexes that use the same parser. This is a pure execution-path choice. | +| Conflicts with `WITH PARSER` in the function (the specified index does not contain that parser) | `USE_INDEX` is ignored. The parser in `WITH PARSER` takes precedence, and TiDB selects another index that contains that parser. | +| `WITH PARSER` is not specified | TiDB follows `USE_INDEX` and uses the specified index and its parser. | +| The specified index does not exist | The hint is silently ignored, and the optimizer selects an index automatically. | + +### IGNORE_INDEX: exclude an index + +```sql +SELECT /*+ IGNORE_INDEX(t, idx_fts_ngram) */ * +FROM t WHERE FTS_MATCH_WORD('handle', content_text); +``` + +### Automatic index selection + +Without hints, the optimizer selects a full-text index by the following priorities: + +| Priority | Rule | +| :-- | :-- | +| 1 | Column coverage: prefer the index whose scored columns overlap most with the columns in the query. | +| 2 | Filter column coverage: prefer the index that matches more filter conditions in the `WHERE` clause. | +| 3 | Index size: prefer the smaller index to reduce scan cost. | + +The optimizer also compares the full-text index scan against a full table scan or a regular index scan, and chooses the path with the lowest estimated cost. See [Full-Text Search Observability](/ai/reference/full-text-search-observability.md) for how to check the selected plan. + +## Supported query patterns + +- Single-table queries with filter conditions, aggregation, ordering, and pagination: + + ```sql + SELECT category, COUNT(*) FROM t + WHERE FTS_MATCH_WORD('query', content_text) GROUP BY category; + + SELECT * FROM t WHERE FTS_MATCH_WORD('query', content_text) + ORDER BY FTS_MATCH_WORD('query', content_text) DESC LIMIT 10; + ``` + +- `INNER JOIN`, where the full-text search runs on the driving table and the join runs on the result set: + + ```sql + SELECT a.*, b.* FROM articles a + INNER JOIN authors b ON a.author_id = b.id + WHERE FTS_MATCH_WORD('database', a.content_text); + ``` + +- `UNION`, `UNION ALL`, `EXCEPT`, and `INTERSECT`. Each branch matches its full-text index independently, and the branch results are combined by the set operator. The outer `ORDER BY` and `LIMIT` of the compound statement run after the set operation and are not pushed into the full-text scans. + +For restrictions such as `ORDER BY` limitations and join support, see [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md). + +## See also + +- [Full-Text Search with SQL](/ai/guides/vector-search-full-text-search-sql.md) +- [Full-Text Search Index](/ai/reference/full-text-search-index.md) +- [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md) diff --git a/ai/reference/full-text-search-index.md b/ai/reference/full-text-search-index.md new file mode 100644 index 0000000000000..4d8d1e84b94fc --- /dev/null +++ b/ai/reference/full-text-search-index.md @@ -0,0 +1,263 @@ +--- +title: Full-Text Search Index +summary: Learn how to create and manage full-text indexes in TiDB, including syntax modes, column attributes, parsers, and DDL restrictions. +aliases: ['/tidb/stable/full-text-search-index/','/tidbcloud/full-text-search-index/'] +--- + +# Full-Text Search Index + +This document describes how to create and manage full-text indexes in TiDB, including the two index definition syntaxes, column attributes, parsers, multiple indexes on one table, and DDL restrictions on indexed columns and tables. + +To run full-text queries against a full-text index, see [Full-Text Search Functions](/ai/reference/full-text-search-functions.md). + +## Restrictions + +Full-text search is still in the early stages, and we are continuously rolling it out to more customers. Currently, full-text search is only available on {{{ .starter }}} in the following regions: + +- AWS: `Oregon (us-west-2)`, `N. Virginia (us-east-1)`, `Tokyo (ap-northeast-1)`, `Frankfurt (eu-central-1)`, and `Singapore (ap-southeast-1)` + +## Create a full-text index + +You can create a full-text index on a new table with `CREATE TABLE`, or add one to an existing table with `ALTER TABLE`. TiDB supports two definition syntaxes: + +- [Syntax sugar mode](#syntax-sugar-mode): `(col) WITH PARSER parser_name`. A simple form for single-purpose indexes. +- [Column-property mode](#column-property-mode): `col WITH (attribute1, attribute2(param=value))`. Configures tokenizers and filter attributes per column. + +The two modes are mutually exclusive within one index definition. Mixing them in the same `FULLTEXT INDEX` definition returns an error. + +### Syntax sugar mode + +```sql +ALTER TABLE t ADD FULLTEXT INDEX idx_fts (content_text) WITH PARSER MULTILINGUAL; +``` + +| Clause | Description | +| :-- | :-- | +| `(content_text)` | The scored column. It participates in the BM25 inverted index and relevance scoring. | +| `WITH PARSER MULTILINGUAL` | The tokenizer (parser) used to tokenize text for indexing and querying. | + +Accepted parsers in the `WITH PARSER ` clause: + +- `STANDARD`: fast, works for English content, splitting words by spaces and punctuation. All text is lowercased for indexing and search (case-insensitive matching). +- `MULTILINGUAL`: supports multiple languages, including English, Chinese, Japanese, and Korean. Case-insensitive for both indexing and querying. +- `NGRAM`: a character-level n-gram tokenizer that supports substring matching. See [The NGRAM parser](#the-ngram-parser) for parameters. + +```sql +ALTER TABLE t ADD FULLTEXT INDEX idx_fts_ngram (content_text) + WITH PARSER NGRAM(min_gram=3, max_gram=3); +``` + +Syntax sugar mode is equivalent to column-property mode with a single parser attribute: `(col) WITH PARSER MULTILINGUAL` equals `col WITH (multilingual)`. Syntax sugar mode does not support filter columns. To define filter columns, use column-property mode. + +> **Note:** +> +> In syntax sugar mode, specifying an index name is optional for the `MULTILINGUAL` parser. If you use the `NGRAM` parser or column-property mode, you must specify an index name explicitly. See [Index naming](#index-naming). + +> **Note:** +> +> The `ADD_COLUMNAR_REPLICA_ON_DEMAND` clause is optional. When specified, TiDB creates a TiFlash replica for the table on demand. If you omit it, make sure that a TiFlash replica is already created for the table before you use full-text search. + +### Column-property mode + +In column-property mode, each column declares one or more attributes in a `WITH (...)` clause: + +```sql +ALTER TABLE t ADD FULLTEXT INDEX idx_fts ( + content_text WITH (multilingual, ngram(min_gram=3, max_gram=3)), + path WITH (exact, path_hierarchy), + name WITH (exact), + ext WITH (exact) +); +``` + +Attributes fall into two categories: + +- **Parser attributes**: `multilingual` and `ngram`. Columns with parser attributes are scored columns that participate in BM25 scoring. +- **Filter attributes**: `exact` and `path_hierarchy`. Columns with filter attributes are filter columns. Filter conditions on these columns are evaluated during the full-text index scan, without accessing the table rows. See [Filter attributes](#filter-attributes). + +Rules for column attributes: + +- A column can have multiple attributes, separated by commas inside `WITH (...)`. For example, `content_text WITH (multilingual, ngram(...))` builds two tokenizer structures for the same column, and `path WITH (exact, path_hierarchy)` makes the column support both equality filters and path prefix filters. +- The same type of parser can appear only once per column. You cannot define two parsers of the same type with different parameters on one column. +- Parser attributes and filter attributes cannot coexist on the same column. A column is either a scored column or a filter column, not both. +- Attribute parameters have default values. You can omit any parameter to use its default. See [Column attribute reference](#column-attribute-reference). +- Filter attribute columns inherit the collation of the corresponding TiDB column. Case sensitivity of filter matching is determined by the column collation. For example, with `utf8mb4_bin` filters are case-sensitive; with `utf8mb4_general_ci` they are case-insensitive. + +#### Column attribute reference + +| Attribute | Parameters | Default values | Description | +| :-- | :-- | :-- | :-- | +| `multilingual` | None | - | Language-aware tokenizer that matches complete tokens. Case-insensitive. | +| `ngram` | `min_gram`, `max_gram`, `granularity`, `lower_case` | `min_gram=3`, `max_gram=3`, `granularity='word'`, `lower_case=true` | Character-level n-gram tokenizer that supports prefix, infix, and suffix substring matching. See [The NGRAM parser](#the-ngram-parser). | +| `exact` | None | - | Exact-value matching using inverted posting lists. Supports `=` and `IN`. Used for tenant IDs, status, tags, and other low-cardinality filters. | +| `path_hierarchy` | `delimiter` | `delimiter='/'` | Hierarchical prefix matching for path-like values. Supports `col LIKE '/src/%'` and `col UNDER '/src/'`. The prefix must align with a delimiter boundary. See [path_hierarchy prefix alignment](#path_hierarchy-prefix-alignment). | + +To customize the delimiter of `path_hierarchy`: + +```sql +ALTER TABLE t ADD FULLTEXT INDEX idx_fts ( + content_text WITH (multilingual), + path WITH (exact, path_hierarchy(delimiter='$')) +); +``` + +## The NGRAM parser + +The `NGRAM` parser builds character-level n-grams so that queries can match substrings. For example, searching `handle` matches documents containing `HandleRequest`, `RequestHandler`, or `handle_error`, which complete-token parsers such as `MULTILINGUAL` cannot match. + +| Parameter | Type | Valid values | Default | Description | +| :-- | :-- | :-- | :-- | :-- | +| `min_gram` | INTEGER | [2, `max_gram`] | `3` | The minimum n-gram length. | +| `max_gram` | INTEGER | [`min_gram`, 5] | `3` | The maximum n-gram length. | +| `granularity` | STRING | `'word'` or `'char'` | `'word'` | Controls how n-grams are generated. See below. | +| `lower_case` | BOOLEAN | `true` or `false` | `true` | Whether text is lowercased at indexing and query time. `true` means case-insensitive matching; `false` preserves the original case. | + +The `granularity` parameter controls n-gram generation: + +| granularity | Behavior | Output for `hello world` (`min_gram=max_gram=3`) | +| :-- | :-- | :-- | +| `word` (default) | Tokenizes the text into words first, then applies a character sliding window inside each word. Avoids meaningless n-grams crossing word boundaries. | `hel`, `ell`, `llo`, `wor`, `orl`, `rld` | +| `char` | Applies a character sliding window over the entire text, ignoring word or space boundaries. Suitable for languages without space-separated words, or for matching across symbols. | `hel`, `ell`, `llo`, `lo_`, `o_w`, `_wo`, `wor`, `orl`, `rld` | + +Examples: + +```sql +-- Defaults: 3-gram, word granularity, case-insensitive +WITH PARSER NGRAM(min_gram=3, max_gram=3) + +-- Character-level sliding window (includes spaces and symbols) +WITH PARSER NGRAM(min_gram=3, max_gram=3, granularity='char') + +-- Case-sensitive matching +WITH PARSER NGRAM(min_gram=3, max_gram=3, lower_case=false) + +-- All parameters can be omitted to use defaults +WITH PARSER NGRAM +``` + +### MULTILINGUAL versus NGRAM + +| Aspect | MULTILINGUAL | NGRAM | +| :-- | :-- | :-- | +| Tokenization | Language-aware segmentation | Character-level sliding window | +| Matching scope | Complete tokens | Prefix / infix / suffix substrings | +| Recall | Precise | Loose | +| Index size | Moderate | Larger | +| Suitable scenarios | Complete keyword search | Partial-recall and code snippet search | + +You can build both a `MULTILINGUAL` index and an `NGRAM` index on the same column and route queries to either one at query time. See [Multiple indexes on one table](#multiple-indexes-on-one-table). + +## Multi-column full-text indexes + +A full-text index can contain multiple scored columns. BM25 scores are fused at the index level in a single scan, which replaces the pattern of scanning one index per column and merging results with `UNION ALL` in the application. + +```sql +-- Syntax sugar mode: all columns share the same parser +ALTER TABLE t ADD FULLTEXT INDEX idx_fts_multi + (content_text, description) WITH PARSER MULTILINGUAL; + +-- Column-property mode: scored columns and filter columns in one index +ALTER TABLE t ADD FULLTEXT INDEX idx_fts_multi ( + content_text WITH (multilingual), + description WITH (multilingual), + path WITH (exact, path_hierarchy), + name WITH (exact) +); +``` + +To search across multiple scored columns, list them in the `FTS_MATCH_WORD()` call. For details, see [Full-Text Search Functions](/ai/reference/full-text-search-functions.md). + +## Multiple indexes on one table + +A table can have multiple full-text indexes. The same column can participate in multiple full-text indexes, each with a different parser. + +```sql +ALTER TABLE t ADD FULLTEXT INDEX idx_fts_ml + (content_text) WITH PARSER MULTILINGUAL; + +ALTER TABLE t ADD FULLTEXT INDEX idx_fts_ng + (content_text) WITH PARSER NGRAM(min_gram=3, max_gram=3); +``` + +At query time, you can select the index with the `USE_INDEX` or `IGNORE_INDEX` optimizer hint, or let the optimizer choose automatically. See [Choose a full-text index at query time](/ai/reference/full-text-search-functions.md#choose-a-full-text-index-at-query-time). + +### Index naming + +If you do not specify an index name, TiDB generates one automatically: + +1. By default, TiDB uses the name of the first indexed column as the index name. +2. If that name already exists, TiDB tries the `_2`, `_3`, and subsequent suffixes until the name is unique. +3. If the first indexed column name is the reserved word `PRIMARY`, TiDB starts from `primary_2`. + +The generated name can be referenced in the `USE_INDEX` hint. + +## View and drop full-text indexes + +`SHOW CREATE TABLE` outputs the full index definition, including the parser and per-column attributes: + +```sql +SHOW CREATE TABLE t; +``` + +``` +CREATE TABLE `t` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `content_text` text DEFAULT NULL, + `description` text DEFAULT NULL, + `path` varchar(512) DEFAULT NULL, + `name` varchar(256) DEFAULT NULL, + PRIMARY KEY (`id`), + FULLTEXT KEY `idx_fts_multi` (`content_text` WITH (multilingual), `description` WITH (multilingual), `path` WITH (exact, path_hierarchy), `name` WITH (exact)) +); +``` + +In column-property mode, every column's parser and filter attributes are shown in the `WITH (...)` clause after the column name. + +`SHOW INDEX` reports `FULLTEXT` in the `Index_type` column, compatible with MySQL: + +```sql +SHOW INDEX FROM t WHERE Key_name = 'idx_fts'; +``` + +You can also query `INFORMATION_SCHEMA`: + +- `INFORMATION_SCHEMA.STATISTICS`: `INDEX_TYPE` returns `FULLTEXT`. +- `INFORMATION_SCHEMA.TIDB_INDEXES`: `index_type` returns `FULLTEXT`. +- `INFORMATION_SCHEMA.TIDB_INDEX_USAGE`: reports access statistics for full-text indexes. + +To drop a full-text index: + +```sql +ALTER TABLE t DROP INDEX idx_fts; +``` + +## DDL restrictions + +Columns that participate in a full-text index and tables that contain a full-text index are subject to the following DDL restrictions. + +### Restrictions on indexed columns + +| DDL operation | Allowed | Notes | +| :-- | :-- | :-- | +| `DROP COLUMN` | No | Drop the full-text index first, then drop the column. | +| `RENAME COLUMN` | No | Drop the full-text index first, rename the column, and then re-create the index. | +| `MODIFY COLUMN` (narrowing) | No | For example, `INT` to `SMALLINT`, or `VARCHAR(40)` to `VARCHAR(20)`. | +| `MODIFY COLUMN` (widening) | Yes | For example, `INT` to `BIGINT`, or `VARCHAR(20)` to `VARCHAR(40)`. | +| `MODIFY COLUMN` (incompatible type) | No | For example, `TEXT` to `INT`. | + +### Restrictions on tables with full-text indexes + +| DDL operation | Allowed | Notes | +| :-- | :-- | :-- | +| `TRUNCATE TABLE` | No | Drop the full-text index first, then truncate the table. | +| `DROP TABLE` | No | Drop the full-text index first, then drop the table. | +| `RENAME TABLE` | No | Drop the full-text index first, rename the table, and then re-create the index. | + +For the full list of functional limitations, see [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md). + +## See also + +- [Full-Text Search with SQL](/ai/guides/vector-search-full-text-search-sql.md) +- [Full-Text Search Functions](/ai/reference/full-text-search-functions.md) +- [Full-Text Search Observability](/ai/reference/full-text-search-observability.md) +- [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md) diff --git a/ai/reference/full-text-search-limitations.md b/ai/reference/full-text-search-limitations.md new file mode 100644 index 0000000000000..fdb95c4cadb41 --- /dev/null +++ b/ai/reference/full-text-search-limitations.md @@ -0,0 +1,54 @@ +--- +title: Full-Text Search Limitations +summary: Learn the limitations of full-text search in TiDB, including index definition, filter pushdown, query syntax, and DDL restrictions. +aliases: ['/tidb/stable/full-text-search-limitations/','/tidbcloud/full-text-search-limitations/'] +--- + +# Full-Text Search Limitations + +This document describes the known limitations of full-text search in TiDB. + +> **Note:** +> +> Full-text search is still in the early stages, and we are continuously rolling it out to more customers. Currently, full-text search is only available on {{{ .starter }}} in selected regions. See [Restrictions](/ai/reference/full-text-search-index.md#restrictions) for the region list. + +## Index limitations + +- In one full-text index definition, the syntax sugar mode `(col) WITH PARSER parser_name` and the column-property mode `col WITH (...)` are mutually exclusive. Mixing both modes in the same index definition returns an error. +- On a single column, the same type of parser can appear only once. You cannot define multiple parsers of the same type with different parameters on one column. +- Parser attributes (`multilingual`, `ngram`) and filter attributes (`exact`, `path_hierarchy`) cannot coexist on the same column. A column is either a scored column or a filter column. +- Syntax sugar mode does not support filter columns. To define filter columns, use column-property mode. + +## Filter pushdown limitations + +- The `exact` attribute supports equality matching (`=` and `IN`) in the index scan. Range predicates (`<`, `>`, `BETWEEN`, and similar) on `exact` columns are not pushed down into the index scan. They are evaluated as residual predicates after row lookup, so query results remain correct but the filtering happens outside the index. +- `LIKE` conditions on `exact` columns are not pushed down into the index scan and are evaluated as residual predicates. `LIKE` prefix matching is pushed down only on columns with the `path_hierarchy` attribute. +- For `path_hierarchy` columns, a pushed-down prefix must align with a delimiter boundary. For example, with the default delimiter `/`, `path LIKE '/src/%'` is pushed down, but `path LIKE '/src/par%'` is not, because the prefix ends inside a directory name. Such conditions are evaluated as residual predicates after row lookup. +- Filter conditions that cannot match any filter column of the selected index are evaluated as residual predicates after row lookup. + +## Query limitations + +- `ORDER BY` with a match function is only supported when the `WHERE` clause contains a single match function call. When the `WHERE` clause contains multiple match functions (for example, multi-word AND or OR combinations), ordering by a match function in the same query is not supported. + + ```sql + -- Not supported: multiple match functions in WHERE and ORDER BY on a match function + SELECT * FROM t + WHERE FTS_MATCH_WORD('database', col) AND FTS_MATCH_WORD('vector', col) + ORDER BY FTS_MATCH_WORD('database', col) DESC; + ``` + +- `FTS_MATCH_WORD()` cannot appear in `GROUP BY` or `HAVING` clauses. +- Exact phrase matching, where all query tokens must appear consecutively and in the specified order, is not supported yet. +- Only `INNER JOIN` is supported with full-text search. Outer joins (`LEFT`, `RIGHT`, and `FULL`) are not supported yet. +- In compound statements (`UNION`, `UNION ALL`, `EXCEPT`, and `INTERSECT`), each branch matches its full-text index independently. Branches do not share a single index scan, and the outer `ORDER BY` and `LIMIT` are not pushed into the full-text scans. + +## DDL limitations + +See [DDL restrictions](/ai/reference/full-text-search-index.md#ddl-restrictions) in [Full-Text Search Index](/ai/reference/full-text-search-index.md). + +## Feedback + +We value your feedback and are always here to help: + +- Ask the community on [Discord](https://discord.gg/DQZ2dy3cuc?utm_source=doc) or [Slack](https://slack.tidb.io/invite?team=tidb-community&channel=everyone&ref=pingcap-docs). +- [Submit a support ticket for TiDB Cloud](https://tidb.support.pingcap.com/servicedesk/customer/portals)