Skip to content

Starter full-text search updates v830 #23615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zhaoshangzi wants to merge 1 commit into
base: release-8.5
Choose a base branch
from
Open

Starter full-text search updates v830 #23615

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ gen
/node_modules/

tmp/
.qoder/settings.local.json
5 changes: 5 additions & 0 deletions TOC-ai.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,8 @@
- [Vector Search Index](/ai/reference/vector-search-index.md)
- [Vector Search Performance Tuning](/ai/reference/vector-search-improve-performance.md)
- [Vector Search Limitations](/ai/reference/vector-search-limitations.md)
- Full-Text Search
- [Full-Text Search Index](/ai/reference/full-text-search-index.md)
- [Full-Text Search Functions](/ai/reference/full-text-search-functions.md)
- [Full-Text Search Observability](/ai/reference/full-text-search-observability.md)
- [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md)
95 changes: 81 additions & 14 deletions ai/guides/vector-search-full-text-search-sql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@

- **Order by relevance**: the search result can be ordered by relevance using the widely adopted [BM25 ranking](https://en.wikipedia.org/wiki/Okapi_BM25) algorithm.

- **Multi-column search**: you can define multiple scored columns in one full-text index and search across them in a single query. BM25 scores are fused at the index level.

- **Filter pushdown**: you can add filter columns (such as tenant IDs, status, and file paths) to a full-text index. Filter conditions on these columns are evaluated during the index scan, without accessing the table rows.

- **Substring matching**: with the NGRAM parser, queries can match prefixes and substrings, such as matching `panic_handler` when searching for `panic`.

- **Fully compatible with SQL**: all SQL features, such as pre-filtering, post-filtering, grouping, and joining, can be used with full-text search.

> **Tip:**
Expand Down Expand Up @@ -65,6 +71,8 @@
-- You might insert some data here.
-- The full-text index can be created even if data is already in the table.

-- ADD_COLUMNAR_REPLICA_ON_DEMAND is optional. If you omit it,
-- make sure that a TiFlash replica is already created for the table.
Comment on lines +74 to +75

@coderabbitai coderabbitai Bot Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

printf '%s\n' '--- applicable repository guidance ---'
find /tmp/coderabbit-repo-knowledge/pingcap-docs-9c73f70c -type f -name '*.md' -maxdepth 3 -print
printf '%s\n' '--- target excerpt ---'
sed -n '55,100p' ai/guides/vector-search-full-text-search-sql.md
printf '%s\n' '--- related clause references ---'
rg -n -C 3 'ADD_COLUMNAR_REPLICA_ON_DEMAND|FULLTEXT INDEX|TiFlash replica' ai resources docs 2>/dev/null | head -160

Repository: pingcap/docs

Length of output: 14857

🌐 Web query:

TiDB 8.5 ADD_COLUMNAR_REPLICA_ON_DEMAND FULLTEXT INDEX parse error issue 20964

💡 Result:

The issue referenced as 20964 in the context of TiDB's full-text search and the ADD_COLUMNAR_REPLICA_ON_DEMAND clause is a documentation/user experience issue identified in the pingcap/docs repository [1]. It specifically reports that executing an ALTER TABLE statement containing both a full-text index creation and the ADD_COLUMNAR_REPLICA_ON_DEMAND clause (e.g., ALTER TABLE t_text ADD FULLTEXT INDEX(ct) WITH PARSER MULTILINGUAL ADD_COLUMNAR_REPLICA_ON_DEMAND;) results in a SQL parsing error [1]. This error occurs because the TiDB SQL parser has historically not supported this specific combined syntax in certain environments, such as the TiDB Cloud SQL Editor [1]. As a workaround, users are advised to execute the operations in two separate steps [1]: 1. Set the TiFlash replica first: ALTER TABLE t_text SET TIFLASH REPLICA 2; [1] 2. Add the full-text index: ALTER TABLE t_text ADD FULLTEXT INDEX(ct) WITH PARSER MULTILINGUAL; [1] While some documentation examples may still display the combined syntax [2][3][4], it is known to fail in specific interfaces [1]. Development work has been ongoing to improve the parser to support such clauses [5]. It is important to note that the issue number 20964 in the pingcap/tidb repository refers to an unrelated historical pull request from 2020 regarding documentation links [6] and is not the same issue as the documentation issue 20964 in the pingcap/docs repository [1].

Citations:

🏁 Script executed:

printf '%s\n' '--- repository-wide review conventions ---'
cat /tmp/coderabbit-repo-knowledge/pingcap-docs-9c73f70c/conventions/repo-wide.md
printf '%s\n' '--- issue and parser-change metadata ---'
curl -L --fail --silent --show-error https://api.github.com/repos/pingcap/docs/issues/20964 | jq '{title,state,body,html_url,created_at,updated_at}'
curl -L --fail --silent --show-error https://api.github.com/repos/pingcap/tidb/pulls/61159 | jq '{title,state,merged_at,body,html_url,base:.base.ref,head:.head.ref}'

Repository: pingcap/docs

Length of output: 6850

🌐 Web query:

site:docs.pingcap.com/tidb/stable ADD_COLUMNAR_REPLICA_ON_DEMAND FULLTEXT TiDB 8.5

💡 Result:

In TiDB, the ADD_COLUMNAR_REPLICA_ON_DEMAND clause is used in conjunction with adding a FULLTEXT index to ensure the necessary columnar (TiFlash) infrastructure is available to support full-text search capabilities [1]. Key points regarding this feature: 1. Purpose: Full-text search in TiDB relies on columnar storage (TiFlash) [1]. Including ADD_COLUMNAR_REPLICA_ON_DEMAND during the creation of a FULLTEXT index ensures that the required TiFlash replica is provisioned for the table [1]. 2. Availability: While the syntax for FULLTEXT search is parsed by most TiDB environments for compatibility, actual support for full-text indexes is currently limited to specific environments, such as TiDB Cloud Starter and Essential instances in select AWS regions [2][3][4][5]. 3. Syntax Usage: The clause is typically appended to the ALTER TABLE ADD FULLTEXT INDEX statement [1]: ALTER TABLE table_name ADD FULLTEXT INDEX (column_name) WITH PARSER parser_name ADD_COLUMNAR_REPLICA_ON_DEMAND; The supported parsers for the WITH PARSER clause include STANDARD (for English, space/punctuation-based) and MULTILINGUAL (for broader language support, including Chinese, Japanese, and Korean) [1].

Citations:

🏁 Script executed:

set -e
printf '%s\n' '--- parser-support change ---'
curl -L --fail --silent --show-error https://api.github.com/repos/pingcap/tidb/pulls/61159/commits | jq '.[-1] | {sha,commit:{message,author}}'
curl -L --fail --silent --show-error https://api.github.com/repos/pingcap/tidb/pulls/61159 | jq '{merge_commit_sha,merged_at,base:.base.ref}'
printf '%s\n' '--- TiDB 8.5 refs ---'
git ls-remote --tags https://github.com/pingcap/tidb.git 'refs/tags/v8.5*' | tail -20
git ls-remote https://github.com/pingcap/tidb.git refs/heads/release-8.5
printf '%s\n' '--- parser change references on GitHub ---'
curl -L --fail --silent --show-error 'https://api.github.com/repos/pingcap/tidb/commits?path=parser/parser.y&sha=release-8.5&per_page=5' | jq '.[] | {sha,commit:{message,author}}'

Repository: pingcap/docs

Length of output: 9054

🏁 Script executed:

set -e
for ref in v8.5.0 v8.5.1 v8.5.6 v8.5.7 release-8.5; do
  printf '%s: ' "$ref"
  url="https://raw.githubusercontent.com/pingcap/tidb/$ref/parser/parser.y"
  if curl -L --fail --silent --show-error "$url" | rg -n -m 2 'ADD_COLUMNAR_REPLICA_ON_DEMAND'; then
    :
  else
    printf '%s\n' 'token not found or parser/parser.y unavailable'
  fi
done
printf '%s\n' '--- merge commit details ---'
curl -L --fail --silent --show-error https://api.github.com/repos/pingcap/tidb/commits/6148d608bf3b40fdfb455363aa0a32aa3e2ca962 | jq '{sha,commit:{message,author,date:.commit.author.date},parents:[.parents[].sha]}'

Repository: pingcap/docs

Length of output: 2030

🏁 Script executed:

set -e
printf '%s\n' '--- files changed by parser PR ---'
curl -L --fail --silent --show-error 'https://api.github.com/repos/pingcap/tidb/pulls/61159/files?per_page=100' |
  jq -r '.[].filename'
printf '%s\n' '--- matching parser files in v8.5.0 ---'
curl -L --fail --silent --show-error 'https://api.github.com/repos/pingcap/tidb/git/trees/v8.5.0?recursive=1' |
  jq -r '.tree[] | select(.path|test("parser|Parser|grammar";"i")) | .path' |
  head -80

Repository: pingcap/docs

Length of output: 2697

🏁 Script executed:

set -e
for ref in v8.5.0 v8.5.1 v8.5.6 v8.5.7 release-8.5; do
  printf '%s\n' "--- $ref ---"
  for path in pkg/parser/parser.y pkg/parser/keywords.go pkg/parser/ast/ddl.go; do
    printf '%s: ' "$path"
    body=$(curl -L --fail --silent --show-error "https://raw.githubusercontent.com/pingcap/tidb/$ref/$path")
    if printf '%s' "$body" | rg -n -m 3 'ADD_COLUMNAR_REPLICA_ON_DEMAND'; then
      :
    else
      printf '%s\n' 'token not found'
    fi
  done
done

Repository: pingcap/docs

Length of output: 810

Remove the unsupported clause from the setup example.

TiDB 8.5 does not recognize ADD_COLUMNAR_REPLICA_ON_DEMAND. Create the TiFlash replica before adding the full-text index.

Suggested change
-- ADD_COLUMNAR_REPLICA_ON_DEMAND is optional. If you omit it,
-- make sure that a TiFlash replica is already created for the table.
-- Ensure that a TiFlash replica is already created for the table.
ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL;

Source: MCP tools

ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL ADD_COLUMNAR_REPLICA_ON_DEMAND;
```

Expand All @@ -74,19 +82,20 @@

- `MULTILINGUAL`: supports multiple languages, including English, Chinese, Japanese, and Korean.

- `NGRAM`: builds character-level n-grams so that queries can match prefixes and substrings. See [The NGRAM parser](/ai/reference/full-text-search-index.md#the-ngram-parser) for parameters.

### Manage full-text indexes

When creating a full-text index, specifying an index name is optional. If you do not specify one, TiDB uses the name of the first indexed column as the index name by default.
When creating a full-text index, specifying an index name is optional. If you do not specify one, TiDB uses the name of the first indexed column as the index name by default.

```sql
-- Without specifying an index name, TiDB uses the first indexed column name ("title") as the index name
-- Without specifying an index name, TiDB uses the first indexed column name ("title") as the index name
ALTER TABLE stock_items ADD FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL;

-- Specifying an index name
ALTER TABLE stock_items ADD FULLTEXT INDEX ft_title (title) WITH PARSER MULTILINGUAL;
```

**View existing index names:**

```sql
-- The Key_name column shows the index name
Expand All @@ -98,7 +107,7 @@
WHERE TABLE_SCHEMA = 'your_database' AND TABLE_NAME = 'stock_items';
```

**Drop a full-text index:**
**Drop a full-text index:**

```sql
-- Use SHOW INDEX to confirm the index name first
Expand All @@ -107,7 +116,7 @@

#### Specify an index name

In both `CREATE TABLE` and `ALTER TABLE` statements, you can specify an index name after `FULLTEXT INDEX` or `FULLTEXT KEY`:
In both `CREATE TABLE` and `ALTER TABLE` statements, you can specify an index name after `FULLTEXT INDEX` or `FULLTEXT KEY`:

```sql
-- Specifying a name in CREATE TABLE
Expand All @@ -124,6 +133,60 @@
CREATE FULLTEXT INDEX ft_name ON users (name) WITH PARSER STANDARD;
```

### Create a multi-column full-text index

A full-text index can contain multiple scored columns. Searching across them in one query fuses the BM25 scores at the index level, which replaces scanning one single-column index per column and merging results with `UNION ALL`.

```sql
ALTER TABLE articles ADD FULLTEXT INDEX ft_article (title, body) WITH PARSER MULTILINGUAL;

SELECT * FROM articles
WHERE fts_match_word('database', title, body)
ORDER BY fts_match_word('database', title, body) DESC LIMIT 10;
```

Columns in one call are combined with OR semantics: a document matches if any of the columns matches the query. For AND semantics and more details, see [Multi-column search](/ai/reference/full-text-search-functions.md#multi-column-search).

### Filter pushdown

Besides scored columns, a full-text index can contain filter columns. Filter conditions on these columns are evaluated during the index scan, so you get correct scoped Top-K results instead of filtering after a global Top-K. Filter columns are defined with the column-property syntax:

```sql
ALTER TABLE files ADD FULLTEXT INDEX idx_fts (
content_text WITH (multilingual),
path WITH (exact, path_hierarchy),
ext WITH (exact)
);

SELECT * FROM files
WHERE fts_match_word('database', content_text)
AND path LIKE '/src/%'
AND ext IN ('go', 'rs')
ORDER BY fts_match_word('database', content_text) DESC LIMIT 10;
```

- The `exact` attribute supports `=` and `IN` matching during the index scan. It is suitable for tenant IDs, status, tags, and other low-cardinality columns.
- The `path_hierarchy` attribute supports hierarchical prefix matching such as `path LIKE '/src/%'`, where the prefix aligns with the `/` delimiter boundary.

For the full attribute reference, syntax rules, and pushdown limitations, see [Full-Text Search Index](/ai/reference/full-text-search-index.md) and [Filter pushdown limitations](/ai/reference/full-text-search-limitations.md#filter-pushdown-limitations).

### Substring matching with the NGRAM parser

The `STANDARD` and `MULTILINGUAL` parsers match complete tokens only. For partial-recall scenarios such as code search, create a full-text index with the `NGRAM` parser to match prefixes and substrings:

```sql
ALTER TABLE code_files ADD FULLTEXT INDEX idx_fts_ngram (content_text)
WITH PARSER NGRAM(min_gram=3, max_gram=3);

-- Matches HandleRequest, RequestHandler, and handle_error
SELECT /*+ USE_INDEX(code_files, idx_fts_ngram) */ *
FROM code_files
WHERE fts_match_word('handle', content_text)
ORDER BY fts_match_word('handle', content_text) DESC LIMIT 10;
```

A table can have multiple full-text indexes, and the same column can participate in several of them with different parsers. Use the `USE_INDEX` hint to select an index at query time, or let the optimizer choose automatically. For parameters and query semantics, see [The NGRAM parser](/ai/reference/full-text-search-index.md#the-ngram-parser) and [Choose a full-text index at query time](/ai/reference/full-text-search-functions.md#choose-a-full-text-index-at-query-time).

### Insert text data

Inserting data into a table with a full-text index is identical to inserting data into any other tables.
Expand Down Expand Up @@ -204,9 +267,9 @@

#### Multi-word search: tokenization and query semantics

When you use `fts_match_word()`, the query string is tokenized according to the parser's rules, and each token is matched independently.
When you use `fts_match_word()`, the query string is tokenized according to the parser's rules, and each token is matched independently.

The STANDARD parser tokenizes strings into words using spaces and punctuation as delimiters. The MULTILINGUAL parser tokenizes strings according to language-specific segmentation rules.
The STANDARD parser tokenizes strings into words using spaces and punctuation as delimiters. The MULTILINGUAL parser tokenizes strings according to language-specific segmentation rules.

```sql
-- This query is tokenized into two tokens: "Alice" and "Smith"
Expand All @@ -221,15 +284,15 @@
SELECT * FROM users WHERE fts_match_word('Alice Smith', name);
```

A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Because `X` is a very short query term, it can match many irrelevant documents. Avoid using very short query terms or single letters.
A common misconception is that `fts_match_word('Alice X', name)` treats `"Alice X"` as a single entity for exact matching. In reality, it is tokenized into `Alice` and `X`, using OR semantics. Because `X` is a very short query term, it can match many irrelevant documents. Avoid using very short query terms or single letters.

Check warning on line 287 in ai/guides/vector-search-full-text-search-sql.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[PingCAP.Ambiguous] Consider using a clearer word than 'many' because it may cause confusion.

Raw Output:
{"message":"[PingCAP.Ambiguous] Consider using a clearer word than 'many' because it may cause confusion.","location":{"path":"ai/guides/vector-search-full-text-search-sql.md","range":{"start":{"line":287,"column":247},"end":{"line":287,"column":251}}},"severity":"INFO","code":{"value":"PingCAP.Ambiguous"}}

> **Note:**
>
> TiDB full-text search does not support exact phrase matching, where all query tokens must appear consecutively and in the specified order.
> **Note:**
>
> TiDB full-text search does not support exact phrase matching, where all query tokens must appear consecutively and in the specified order.

#### Prefix search
#### Prefix and substring search

**Not supported.**
To match prefixes or substrings, use a full-text index with the `NGRAM` parser. See [Substring matching with the NGRAM parser](#substring-matching-with-the-ngram-parser). To match path prefixes such as `/src/`, use a filter column with the `path_hierarchy` attribute. See [Filter pushdown](#filter-pushdown).

#### Effect of repeated terms on relevance scores

Expand All @@ -244,7 +307,7 @@

#### Relevance scoring algorithm

TiDB full-text search uses the **BM25Tantivy** algorithm to calculate relevance scores. This algorithm is a variant of the classic BM25 (Okapi BM25) algorithm that uses Count-Min Sketch to approximate document frequency (DF) for improved performance.
TiDB full-text search uses the **BM25Tantivy** algorithm to calculate relevance scores. This algorithm is a variant of the classic BM25 (Okapi BM25) algorithm that uses Count-Min Sketch to approximate document frequency (DF) for improved performance.

**BM25 formula (standard form):**

Expand Down Expand Up @@ -316,6 +379,10 @@
## See also

- [Hybrid Search](/ai/guides/vector-search-hybrid-search.md)
- [Full-Text Search Index](/ai/reference/full-text-search-index.md)
- [Full-Text Search Functions](/ai/reference/full-text-search-functions.md)
- [Full-Text Search Observability](/ai/reference/full-text-search-observability.md)
- [Full-Text Search Limitations](/ai/reference/full-text-search-limitations.md)

## Feedback & help

Expand Down
Loading
Loading