From fd27c238d85766cd2c8c31234de1b78b4f7e905f Mon Sep 17 00:00:00 2001 From: Leon van Zantvoort <303196+zantvoort@users.noreply.github.com> Date: Fri, 23 Oct 2026 02:32:04 +0100 Subject: [PATCH] Let suggest ignore the words a search box names A word the substring index serves can be as useless as a short one when it occurs in a large share of the rows: "the" is three characters, so the trigram index on PostgreSQL serves it, but it matches a third of the titles in a movie catalog. suggest takes a set of words to ignore right after the input, in Java and Kotlin. An ignored word is treated as a short word is: it takes no part in the predicate, compared ignoring case, and an input of ignored words alone matches no rows. search takes no set. Fixes #741 --- CHANGELOG.md | 2 +- docs/search.md | 43 +++++++++++++++++++ .../impl/PredicateBuilderFactory.java | 39 +++++++++++++++-- .../st/orm/core/SearchIntegrationTest.java | 26 ++++++++++- .../java/st/orm/template/WhereBuilder.java | 38 +++++++++++++--- .../orm/template/impl/QueryBuilderImpl.java | 7 ++- .../st/orm/template/QueryBuilderTest.java | 16 +++++++ .../kotlin/st/orm/template/QueryBuilder.kt | 23 +++++++++- .../template/impl/PredicateBuilderFactory.kt | 8 ++-- .../st/orm/template/QueryBuilderTest.kt | 16 +++++++ website/static/skills/storm-query-java.md | 2 +- website/static/skills/storm-query-kotlin.md | 2 +- 12 files changed, 203 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d1ffacba..6a47c7f92 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ for the CLI, to [npm](https://www.npmjs.com/package/@storm-orm/cli) - `delete()` and `update()` refuse to run without a condition unless `unsafe()` is called, take explicit joins on every database, and refuse a joined inheritance hierarchy (#712). `SqlDialect.writeJoinSyntax()` replaces `supportsDeleteAlias()`. - Added `semiJoin` and `antiJoin`, which filter rows by a related table without adding its rows, so no `distinct` is needed (#690). `semiJoin().on()` keeps the owners with a pet matching the conditions that name `Pet`, `antiJoin` the owners without one. They filter `select()`, `delete()` and `update()`, and nest to filter through two to-many relations: `semiJoin().on().semiJoin().on()` (#717). A search over a row and its related rows reads each table through its own indexes: 1.3 s to a few milliseconds over 980,000 credits on PostgreSQL. - A typed join takes a condition: `on { Pet_.name eq "Leo" }` in Kotlin, `on(Owner.class, it -> ...)` in Java (#690). On a left join it keeps the rows without a match, which the same condition in `where` drops. -- Added `suggest`, the `search` for input typed as you go: it ignores words shorter than the database's substring index serves and matches nothing when no word is left, so it never reads every row. `search` tests such short words only on the rows its longer words found: "de niro" over 980,000 credits went from 2.3 s to 19 ms on PostgreSQL. The length comes from `SqlDialect.minimumSearchTermLength()`, 3 on PostgreSQL and 0 elsewhere, and `storm.search.min_term_length` overrides it. +- Added `suggest`, the `search` for input typed as you go: it ignores words shorter than the database's substring index serves and matches nothing when no word is left, so it never reads every row. `search` tests such short words only on the rows its longer words found: "de niro" over 980,000 credits went from 2.3 s to 19 ms on PostgreSQL. The length comes from `SqlDialect.minimumSearchTermLength()`, 3 on PostgreSQL and 0 elsewhere, and `storm.search.min_term_length` overrides it. `suggest` also takes the words a search box ignores, such as `the` in a catalog of titles, which the index serves but which narrow nothing: `suggest(input, setOf("the", "a", "of"), Movie_.title)` treats them as short words, compared ignoring case (#741). - Added `orderFirst`, which sorts the rows matching a predicate first: `orderFirst(Movie_.title eqIgnoreCase input)` in Kotlin, `orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input))` in Java, rendered as `CASE WHEN ... THEN 0 ELSE 1 END` on every database. A keyset `scroll` follows it, and `Scrollable.sortBy` and `sortByDescending` take a field of any table in the query. - Added `SqlFunction`, a one-argument SQL function applied to a column wherever a query takes one, so a query can match the expression an index is built on (#694). `foldAccents(Movie_.title)` in Kotlin, `foldAccents.apply(Movie_.title)` in Java; a comparison applies it to the column and the value alike, so an accent-insensitive search on PostgreSQL uses an index on `LOWER(fold_accents(title))`. - The Kotlin API takes a type argument where the call needs no `_` placeholders, and a class literal where it would (#719): `subquery()`, `orm.selectFrom()` and `orm.selectFrom()`, next to `orm.entity(User::class)` and `select(Owner::class)`. Ktor gains `entity(User::class)` and `projection(OwnerView::class)` on `Application`, `ApplicationCall` and routes. diff --git a/docs/search.md b/docs/search.md index 42bf03697..a3400201c 100644 --- a/docs/search.md +++ b/docs/search.md @@ -100,6 +100,49 @@ List suggestions = orm.entity(Movie.class) Every statement `suggest` renders is therefore served by the substring index, however the user types; the trade-off is that a short word does not narrow the suggestions. The search page the box leads to keeps `search`, which matches every word. Which words are short is the dialect's to say: three characters on PostgreSQL, whose trigram index needs three, and none on the other databases, where `suggest` matches as `search` does, except that an empty input matches no rows. The `storm.search.min_term_length` property sets another length, for an index of a different kind. +### Ignoring Words + +A word the index serves can be as useless as a short one when it occurs in a large share of the rows: `the` is three characters, so the trigram index serves it, but it matches a third of the titles in a movie catalog. Typed alone it returns a page of noise, and next to another word it narrows nothing. Which words those are depends on the column's content, `the` and `of` for titles, `de` and `van` for names, so the search box names them, as it names the columns. `suggest` takes the words to ignore after the input: + + + + +```kotlin +val titleNoise = setOf("the", "a", "of") + +val suggestions = orm.entity(Movie::class) + .select() + .where(suggest(input, titleNoise, Movie_.title, Movie_.director.name)) + .limit(6) + .resultList +``` + + + + +```java +private static final Set TITLE_NOISE = Set.of("the", "a", "of"); + +List suggestions = orm.entity(Movie.class) + .select() + .where(it -> it.suggest(input, TITLE_NOISE, Movie_.title, Movie_.director.name)) + .limit(6) + .getResultList(); +``` + + + + +An ignored word is treated as a short word is: it takes no part in the predicate, compared ignoring case as the words themselves match, and an input without a remaining word matches no rows. A quoted phrase is one word, ignored when it equals an ignored word, not when it contains one. + +| Input | Words `suggest` matches | +|-------|----------| +| `the god` | `god`: finds "The Godfather" and "City of God" | +| `The` | none: no rows | +| `"the good"` | `the good`: the phrase is one word | + +The set is a constant next to the box that uses it, typed by hand for most boxes. Where the catalog is large enough to measure, a query over the column counts the words and the set holds those that occur in more rows than the box shows. `search` takes no such set: on a results page a frequent word next to a longer one costs nothing and still narrows correctly, and a page of every title with `the` in it is a broad but legitimate query. + ## Searching Related Rows Sometimes the user remembers something about a related row rather than the row itself: the front desk knows the dog is called Leo, not the owner's last name. To search the rows related to each row, semi-join the related table and name its column next to the row's own. Every owner comes back once, however many pets match: diff --git a/storm-core/src/main/java/st/orm/core/template/impl/PredicateBuilderFactory.java b/storm-core/src/main/java/st/orm/core/template/impl/PredicateBuilderFactory.java index 7e1a9fd5f..2d62ff450 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/PredicateBuilderFactory.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/PredicateBuilderFactory.java @@ -15,12 +15,15 @@ */ package st.orm.core.template.impl; +import static java.util.Locale.ROOT; import static java.util.Objects.requireNonNull; +import static java.util.stream.Collectors.toSet; import static st.orm.Operator.CONTAINS_IGNORE_CASE; import static st.orm.core.template.TemplateString.wrap; import java.util.ArrayList; import java.util.List; +import java.util.Set; import st.orm.Data; import st.orm.Metamodel; import st.orm.Operator; @@ -115,7 +118,7 @@ static PredicateBuilder createWithId( */ static PredicateBuilder search(String input, List> paths) { - return words(input, paths, false); + return words(input, Set.of(), paths, false); } /** @@ -136,14 +139,38 @@ static PredicateBuilder search(String input, */ static PredicateBuilder suggest(String input, List> paths) { - return words(input, paths, true); + return suggest(input, Set.of(), paths); } /** - * Creates the predicate of {@link #search(String, List)} or {@link #suggest(String, List)}. Each word is kept - * apart until the query is built, since which words a substring index serves depends on the dialect. + * Creates the predicate of {@link #suggest(String, List)}, ignoring the given terms as well: a term that equals + * one of them, ignoring case, takes no part in the predicate, as a term the index does not serve takes none, so an + * input of ignored terms alone matches no rows. A search box names the words that occur in a large share of its + * rows, such as "the" in a catalog of titles, which the index serves but which narrow nothing. + * + * @param input the search input, as a search box receives it. + * @param ignore the terms to ignore, compared ignoring case. + * @param paths the text columns to search, at least one. + * @param the type of the table being queried. + * @param the type of the result. + * @param the type of the primary key. + * @return the predicate. + * @since 1.16 + */ + static PredicateBuilder suggest(String input, + Set ignore, + List> paths) { + requireNonNull(ignore, "ignore"); + return words(input, ignore, paths, true); + } + + /** + * Creates the predicate of {@link #search(String, List)} or {@link #suggest(String, Set, List)}. Each word is kept + * apart until the query is built, since which words a substring index serves depends on the dialect; an ignored + * word is left out here, since no dialect serves it. */ private static PredicateBuilder words(String input, + Set ignore, List> paths, boolean suggest) { requireNonNull(input, "input"); @@ -151,6 +178,10 @@ private static PredicateBuilder words(String i throw new IllegalArgumentException("%s needs at least one path to search.".formatted(suggest ? "suggest" : "search")); } List terms = SearchTerms.parse(input); + if (!ignore.isEmpty()) { + Set ignored = ignore.stream().map(word -> word.toLowerCase(ROOT)).collect(toSet()); + terms = terms.stream().filter(term -> !ignored.contains(term.toLowerCase(ROOT))).toList(); + } var words = new ArrayList(terms.size()); for (String term : terms) { var matches = new ArrayList(paths.size()); diff --git a/storm-core/src/test/java/st/orm/core/SearchIntegrationTest.java b/storm-core/src/test/java/st/orm/core/SearchIntegrationTest.java index 15308d61c..a9e6d8efa 100644 --- a/storm-core/src/test/java/st/orm/core/SearchIntegrationTest.java +++ b/storm-core/src/test/java/st/orm/core/SearchIntegrationTest.java @@ -22,6 +22,7 @@ import java.util.ArrayList; import java.util.List; import java.util.Map; +import java.util.Set; import javax.sql.DataSource; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; @@ -63,12 +64,17 @@ private List search(String input, Metamodel... paths) { @SafeVarargs private List suggest(String input, int minimumLength, Metamodel... paths) { + return suggest(input, Set.of(), minimumLength, paths); + } + + @SafeVarargs + private List suggest(String input, Set ignore, int minimumLength, Metamodel... paths) { var orm = ORMTemplate.of(dataSource, StormConfig.of(Map.of(StormConfig.SEARCH_MIN_TERM_LENGTH, String.valueOf(minimumLength)))); statements = new ArrayList<>(); List names = new ArrayList<>(); observe(sql -> statements.add(sql.statement()), () -> orm.entity(Owner.class).select() - .where(it -> PredicateBuilderFactory.suggest(input, List.of(paths))) + .where(it -> PredicateBuilderFactory.suggest(input, ignore, List.of(paths))) .getResultList() .forEach(owner -> names.add(owner.firstName()))); return names.stream().sorted().toList(); @@ -124,6 +130,24 @@ public void suggestMatchesTheWordsTheIndexServes() { assertEquals(List.of(), suggest("", 0, Owner_.lastName)); } + @Test + public void suggestIgnoresTheWordsTheSearchBoxNames() { + var ignore = Set.of("Davis", "the"); + // An ignored word takes no part, whatever its case; alone, it matches no rows. + assertEquals(List.of("Betty", "Harold"), suggest("davis", 3, Owner_.lastName)); + assertEquals(List.of(), suggest("davis", ignore, 3, Owner_.lastName)); + assertEquals("1 <> 1", whereClause()); + assertEquals(List.of(), suggest("DAVIS the", ignore, 3, Owner_.lastName)); + // The other words match as before, each over every path. + assertEquals(List.of("Betty"), suggest("the davis betty", ignore, 3, Owner_.lastName, Owner_.firstName)); + assertEquals(2, whereClause().split("LIKE").length - 1); + // A quoted phrase is one word, ignored when it equals an ignored word. + assertEquals(List.of(), suggest("\"Davis\"", ignore, 3, Owner_.lastName)); + // Ignored on a dialect that treats every word alike as well. + assertEquals(List.of("Betty"), suggest("the betty", ignore, 0, Owner_.firstName)); + assertEquals(List.of(), suggest("the", ignore, 0, Owner_.firstName)); + } + @Test public void inputWithoutTermsMatchesEveryRow() { assertEquals(10, search("", Owner_.lastName).size()); diff --git a/storm-java21/src/main/java/st/orm/template/WhereBuilder.java b/storm-java21/src/main/java/st/orm/template/WhereBuilder.java index ab5b01347..b421d3346 100644 --- a/storm-java21/src/main/java/st/orm/template/WhereBuilder.java +++ b/storm-java21/src/main/java/st/orm/template/WhereBuilder.java @@ -21,6 +21,7 @@ import java.util.ArrayList; import java.util.List; +import java.util.Set; import st.orm.Data; import st.orm.Navigable; import st.orm.Operator; @@ -286,10 +287,33 @@ public final PredicateBuilder search(String input, public final PredicateBuilder suggest(String input, Navigable path, Navigable... more) { + return suggest(input, Set.of(), path, more); + } + + /** + * Matches the words of a search box that queries as the user types against one or more text columns, as + * {@link #suggest(String, Navigable, Navigable[])} does, ignoring the given words as well. A word that equals one + * of them, ignoring case, takes no part in the predicate, as a word the index does not serve takes none, so an + * input of ignored words alone matches no rows. A search box names the words that occur in a large share of its + * rows, such as "the" in a catalog of titles, which the index serves but which narrow nothing: + * {@code suggest("the god", Set.of("the", "a", "of"), Movie_.title)} matches on "god" alone. + * + * @param input the search input, as a search box receives it. + * @param ignore the words to ignore, compared ignoring case. + * @param path the first text column to search. + * @param more further text columns to search. + * @return the predicate builder. + * @since 1.16 + */ + @SafeVarargs + public final PredicateBuilder suggest(String input, + Set ignore, + Navigable path, + Navigable... more) { var paths = new ArrayList>(more.length + 1); paths.add(path); paths.addAll(List.of(more)); - return suggestImpl(input, paths); + return suggestImpl(input, ignore, paths); } /** @@ -327,12 +351,16 @@ protected abstract PredicateBuilder whereImpl(Navigable searchImpl(String input, List> paths); /** - * Matches the words of a search input that the database's substring index serves against the given text columns. + * Matches the words of a search input that the database's substring index serves, and that the search box does + * not ignore, against the given text columns. * - * @param input the search input. - * @param paths the text columns to search, at least one. + * @param input the search input. + * @param ignore the words to ignore, compared ignoring case. + * @param paths the text columns to search, at least one. * @return the predicate builder. * @since 1.16 */ - protected abstract PredicateBuilder suggestImpl(String input, List> paths); + protected abstract PredicateBuilder suggestImpl(String input, + Set ignore, + List> paths); } diff --git a/storm-java21/src/main/java/st/orm/template/impl/QueryBuilderImpl.java b/storm-java21/src/main/java/st/orm/template/impl/QueryBuilderImpl.java index faa0a1437..3bd042f3e 100644 --- a/storm-java21/src/main/java/st/orm/template/impl/QueryBuilderImpl.java +++ b/storm-java21/src/main/java/st/orm/template/impl/QueryBuilderImpl.java @@ -24,6 +24,7 @@ import java.util.List; import java.util.Optional; +import java.util.Set; import java.util.function.Function; import java.util.stream.Stream; import org.jspecify.annotations.Nullable; @@ -634,8 +635,10 @@ protected PredicateBuilder searchImpl(String input, List suggestImpl(String input, List> paths) { - return new PredicateBuilderImpl<>(PredicateBuilderFactory.suggest(input, paths.stream().map(Navigable::asMetamodel).toList())); + protected PredicateBuilder suggestImpl(String input, + Set ignore, + List> paths) { + return new PredicateBuilderImpl<>(PredicateBuilderFactory.suggest(input, ignore, paths.stream().map(Navigable::asMetamodel).toList())); } } diff --git a/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java b/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java index c1643a5be..20a41c6b9 100644 --- a/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java +++ b/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java @@ -25,6 +25,8 @@ import java.util.List; import java.util.Optional; +import java.util.Set; +import java.util.function.Function; import java.util.stream.Stream; import javax.sql.DataSource; import org.junit.jupiter.api.Test; @@ -776,6 +778,20 @@ private static List semiJoinSuggestFirstNames(ORMTemplate template, Stri .getResultList().stream().map(Owner::firstName).sorted().toList(); } + @Test + public void testSuggestIgnoresTheWordsTheSearchBoxNames() { + // data.sql: owner 1 (Betty Davis) has pet Leo; owner 4 (Harold Davis) has pet Iggy. + var ignore = Set.of("davis", "the"); + Function> firstNames = input -> orm.entity(Owner.class).select() + .semiJoin(Pet.class).on(Owner.class) + .where(it -> it.suggest(input, ignore, Owner_.lastName, Pet_.name)) + .getResultList().stream().map(Owner::firstName).sorted().toList(); + assertEquals(List.of("Betty"), firstNames.apply("The Davis leo")); + assertEquals(List.of(), firstNames.apply("the davis")); + assertEquals(List.of(), firstNames.apply("")); + assertEquals(List.of("Betty", "Harold"), semiJoinSuggestFirstNames(orm, "davis")); + } + private List semiJoinSearchFirstNames(String input) { return orm.entity(Owner.class).select() .semiJoin(Pet.class).on(Owner.class) diff --git a/storm-kotlin/src/main/kotlin/st/orm/template/QueryBuilder.kt b/storm-kotlin/src/main/kotlin/st/orm/template/QueryBuilder.kt index 964053845..8d2a523a1 100644 --- a/storm-kotlin/src/main/kotlin/st/orm/template/QueryBuilder.kt +++ b/storm-kotlin/src/main/kotlin/st/orm/template/QueryBuilder.kt @@ -1776,7 +1776,28 @@ public fun suggest( input: String, path: Navigable, vararg more: Navigable, -): PredicateBuilder = createSuggest(input, listOf(path.asMetamodel()) + more.map { it.asMetamodel() }) +): PredicateBuilder = suggest(input, emptySet(), path, *more) + +/** + * Creates a predicate that matches the words of a search box that queries as the user types against one or more text + * columns, as [suggest] does, ignoring the given words as well. A word that equals one of them, ignoring case, takes + * no part in the predicate, as a word the index does not serve takes none, so an input of ignored words alone matches + * no rows. A search box names the words that occur in a large share of its rows, such as "the" in a catalog of + * titles, which the index serves but which narrow nothing: `suggest("the god", setOf("the", "a", "of"), Movie_.title)` + * matches on "god" alone. + * + * @param input the search input, as a search box receives it. + * @param ignore the words to ignore, compared ignoring case. + * @param path the first text column to search. + * @param more further text columns to search. + * @since 1.16 + */ +public fun suggest( + input: String, + ignore: Set, + path: Navigable, + vararg more: Navigable, +): PredicateBuilder = createSuggest(input, ignore, listOf(path.asMetamodel()) + more.map { it.asMetamodel() }) /** * Applies a SQL function to a column, for a query that names the expression an index is built on. The call is taken diff --git a/storm-kotlin/src/main/kotlin/st/orm/template/impl/PredicateBuilderFactory.kt b/storm-kotlin/src/main/kotlin/st/orm/template/impl/PredicateBuilderFactory.kt index ce750d9fd..adbf148dd 100644 --- a/storm-kotlin/src/main/kotlin/st/orm/template/impl/PredicateBuilderFactory.kt +++ b/storm-kotlin/src/main/kotlin/st/orm/template/impl/PredicateBuilderFactory.kt @@ -70,17 +70,19 @@ internal fun createSearch( ): PredicateBuilder = PredicateBuilderImpl(PredicateBuilderFactory.search(input, paths)) /** - * Creates a predicate that matches the terms of a search input the dialect's substring index serves in at least one - * of the paths, ignoring case, and no rows when the input holds none. + * Creates a predicate that matches the terms of a search input the dialect's substring index serves, and the search + * box does not ignore, in at least one of the paths, ignoring case, and no rows when the input holds none. * * @param input the search input. + * @param ignore the terms to ignore, compared ignoring case. * @param paths the text columns to search, at least one. * @return a new instance of [PredicateBuilder] */ internal fun createSuggest( input: String, + ignore: Set, paths: List>, -): PredicateBuilder = PredicateBuilderImpl(PredicateBuilderFactory.suggest(input, paths)) +): PredicateBuilder = PredicateBuilderImpl(PredicateBuilderFactory.suggest(input, ignore, paths)) /** * Combines two predicates using an AND condition, rooting the result at the operands' least common root. diff --git a/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt b/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt index d5e114432..0cc5f2d98 100644 --- a/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt +++ b/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt @@ -2260,6 +2260,22 @@ internal open class QueryBuilderTest( firstNames(orm, "le", suggest = true) shouldBe listOf("Betty", "Jean") } + @Test + fun `suggest should ignore the words the search box names`() { + // data.sql: owner 1 (Betty Davis) has pet Leo; owner 4 (Harold Davis) has pet Iggy. + val ownerRepo = orm.entity(Owner::class) + val lastName = metamodel(ownerRepo.model, "last_name") + val petName = metamodel(orm.entity(Pet::class).model, "name") + val ignore = setOf("davis", "the") + fun firstNames(input: String): List = ownerRepo.select() + .semiJoin().on() + .where(suggest(input, ignore, lastName, petName)) + .resultList.map { it.firstName }.sorted() + firstNames("The Davis leo") shouldBe listOf("Betty") + firstNames("the davis") shouldBe emptyList() + firstNames("") shouldBe emptyList() + } + @Test fun `orderFirst should put the matching rows first in the order of the calls`() { val ownerRepo = orm.entity(Owner::class) diff --git a/website/static/skills/storm-query-java.md b/website/static/skills/storm-query-java.md index 78b021468..a5e3f829e 100644 --- a/website/static/skills/storm-query-java.md +++ b/website/static/skills/storm-query-java.md @@ -135,7 +135,7 @@ Operators: EQUALS, NOT_EQUALS, LESS_THAN, LESS_THAN_OR_EQUAL, GREATER_THAN, GREA **A search box goes through `search`, never through hand-built `CONTAINS_IGNORE_CASE` chains.** `.where(it -> it.search(input, Movie_.title, Movie_.director.name))` takes the input as the box receives it: whitespace separates the words, a quoted phrase is one word, every word must occur in at least one of the columns (ignoring case, literally), and an empty input matches every row. It returns a predicate, so it combines with `.and()` and `.or()`. Do not split the input, reduce over the words or special-case the empty box in application code. To search the rows related to each row, such as an owner's pets, semi-join the related table and name its column: `.semiJoin(Pet.class).on(Owner.class).where(it -> it.search(input, Owner_.lastName, Pet_.name))`. Each word may then match a different related row, and every owner comes back once. -**A search box that queries as the user types goes through `suggest`, not `search`.** An auto-complete, a typeahead or any box that fetches on every keystroke uses `.where(it -> it.suggest(input, Movie_.title, Movie_.director.name))`: the same input and paths, matched the same way, on the words the database's substring index serves alone. A word shorter than the index serves (three characters on PostgreSQL, whose trigram index needs three) is ignored, and an input without such a word, an empty one included, matches no rows, so every statement it renders is served by the index however the user types. The search page the box leads to keeps `search`, which matches every word. Do not filter short words or an empty input out in application code; `suggest` does it per dialect. +**A search box that queries as the user types goes through `suggest`, not `search`.** An auto-complete, a typeahead or any box that fetches on every keystroke uses `.where(it -> it.suggest(input, Movie_.title, Movie_.director.name))`: the same input and paths, matched the same way, on the words the database's substring index serves alone. A word shorter than the index serves (three characters on PostgreSQL, whose trigram index needs three) is ignored, and an input without such a word, an empty one included, matches no rows, so every statement it renders is served by the index however the user types. The search page the box leads to keeps `search`, which matches every word. Do not filter short words or an empty input out in application code; `suggest` does it per dialect. A word the index serves but which occurs in a large share of the rows, such as `the` in a catalog of titles, is the box's to name: `it.suggest(input, Set.of("the", "a", "of"), Movie_.title)` ignores those words as it ignores short ones, compared ignoring case, so `the god` matches on `god` alone and `the` alone matches no rows. Keep the set a constant next to the box; never strip the words from the input in application code, and never pass the set to `search`, which takes none. **Rank search results with `orderFirst`, never with a `CASE` template.** `search` and `suggest` decide which rows match, not which come first. `orderFirst(predicate)` puts the rows matching the predicate first, appending a sort key as `orderBy` does, so the calls apply in order: `.orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input)).orderFirst(it -> it.where(Movie_.title, STARTS_WITH_IGNORE_CASE, input)).orderByDescending(Movie_.votes)` lists an exact title, then titles starting with the input, then the most popular. The operators match the input literally and ignoring case, as in `where`; combine columns with `.or(...)`. Rank only by columns of the queried or joined tables: a semi-joined table's columns are refused, as it adds no columns. A keyset `scroll` follows `orderFirst` (its keys lead the `Scrollable`'s sort fields and key, and the cursor carries the ranks), and a `Scrollable` may sort by a field of a joined table, such as the votes of a joined rating; `orderFirst` cannot order a `distinct` query. No index serves the sort, which is fine after a search's few matches. diff --git a/website/static/skills/storm-query-kotlin.md b/website/static/skills/storm-query-kotlin.md index 09916590a..67cd695f5 100644 --- a/website/static/skills/storm-query-kotlin.md +++ b/website/static/skills/storm-query-kotlin.md @@ -100,7 +100,7 @@ User_.email.isNotNull() // IS_NOT_NULL **A search box goes through `search`, never through hand-built `containsIgnoreCase` chains.** `.where(search(input, Movie_.title, Movie_.director.name))` takes the input as the box receives it: whitespace separates the words, a quoted phrase is one word, every word must occur in at least one of the columns (ignoring case, literally), and an empty input matches every row. It returns a predicate, so it combines with `and` and `or`. Do not split the input, `reduce` over the words or special-case the empty box in application code. To search the rows related to each row, such as an owner's pets, semi-join the related table and name its column: `.semiJoin().on().where(search(input, Owner_.lastName, Pet_.name))`. Each word may then match a different related row, and every owner comes back once. -**A search box that queries as the user types goes through `suggest`, not `search`.** An auto-complete, a typeahead or any box that fetches on every keystroke uses `.where(suggest(input, Movie_.title, Movie_.director.name))`: the same input and paths, matched the same way, on the words the database's substring index serves alone. A word shorter than the index serves (three characters on PostgreSQL, whose trigram index needs three) is ignored, and an input without such a word, an empty one included, matches no rows, so every statement it renders is served by the index however the user types. The search page the box leads to keeps `search`, which matches every word. Do not filter short words or an empty input out in application code; `suggest` does it per dialect. +**A search box that queries as the user types goes through `suggest`, not `search`.** An auto-complete, a typeahead or any box that fetches on every keystroke uses `.where(suggest(input, Movie_.title, Movie_.director.name))`: the same input and paths, matched the same way, on the words the database's substring index serves alone. A word shorter than the index serves (three characters on PostgreSQL, whose trigram index needs three) is ignored, and an input without such a word, an empty one included, matches no rows, so every statement it renders is served by the index however the user types. The search page the box leads to keeps `search`, which matches every word. Do not filter short words or an empty input out in application code; `suggest` does it per dialect. A word the index serves but which occurs in a large share of the rows, such as `the` in a catalog of titles, is the box's to name: `suggest(input, setOf("the", "a", "of"), Movie_.title)` ignores those words as it ignores short ones, compared ignoring case, so `the god` matches on `god` alone and `the` alone matches no rows. Keep the set a constant next to the box; never strip the words from the input in application code, and never pass the set to `search`, which takes none. **Rank search results with `orderFirst`, never with a `CASE` template.** `search` and `suggest` decide which rows match, not which come first. `orderFirst(predicate)` puts the rows matching the predicate first, appending a sort key as `orderBy` does, so the calls apply in order: `.orderFirst(Movie_.title eqIgnoreCase input).orderFirst(Movie_.title startsWithIgnoreCase input).orderByDescending(Movie_.votes)` lists an exact title, then titles starting with the input, then the most popular. The operators match the input literally and ignoring case, as in `where`; combine columns with `or`. Rank only by columns of the queried or joined tables: a semi-joined table's columns are refused, as it adds no columns. A keyset `scroll` follows `orderFirst` (its keys lead the `Scrollable`'s sort fields and key, and the cursor carries the ranks), and a `Scrollable` may sort by a field of a joined table, such as the votes of a joined rating; `orderFirst` cannot order a `distinct` query. No index serves the sort, which is fine after a search's few matches.