Let suggest ignore the words a search box names

CHANGELOG.md

@@ -24,7 +24,7 @@ for the CLI, to [npm](https://www.npmjs.com/package/@storm-orm/cli)
 - `delete()` and `update()` refuse to run without a condition unless `unsafe()` is called, take explicit joins on every database, and refuse a joined inheritance hierarchy (#712). `SqlDialect.writeJoinSyntax()` replaces `supportsDeleteAlias()`.
 - Added `semiJoin` and `antiJoin`, which filter rows by a related table without adding its rows, so no `distinct` is needed (#690). `semiJoin<Pet>().on<Owner>()` keeps the owners with a pet matching the conditions that name `Pet`, `antiJoin` the owners without one. They filter `select()`, `delete()` and `update()`, and nest to filter through two to-many relations: `semiJoin<Pet>().on<Owner>().semiJoin<Visit>().on<Pet>()` (#717). A search over a row and its related rows reads each table through its own indexes: 1.3 s to a few milliseconds over 980,000 credits on PostgreSQL.
 - A typed join takes a condition: `on<Owner> { Pet_.name eq "Leo" }` in Kotlin, `on(Owner.class, it -> ...)` in Java (#690). On a left join it keeps the rows without a match, which the same condition in `where` drops.
-- Added `suggest`, the `search` for input typed as you go: it ignores words shorter than the database's substring index serves and matches nothing when no word is left, so it never reads every row. `search` tests such short words only on the rows its longer words found: "de niro" over 980,000 credits went from 2.3 s to 19 ms on PostgreSQL. The length comes from `SqlDialect.minimumSearchTermLength()`, 3 on PostgreSQL and 0 elsewhere, and `storm.search.min_term_length` overrides it.
+- Added `suggest`, the `search` for input typed as you go: it ignores words shorter than the database's substring index serves and matches nothing when no word is left, so it never reads every row. `search` tests such short words only on the rows its longer words found: "de niro" over 980,000 credits went from 2.3 s to 19 ms on PostgreSQL. The length comes from `SqlDialect.minimumSearchTermLength()`, 3 on PostgreSQL and 0 elsewhere, and `storm.search.min_term_length` overrides it. `suggest` also takes the words a search box ignores, such as `the` in a catalog of titles, which the index serves but which narrow nothing: `suggest(input, setOf("the", "a", "of"), Movie_.title)` treats them as short words, compared ignoring case (#741).
 - Added `orderFirst`, which sorts the rows matching a predicate first: `orderFirst(Movie_.title eqIgnoreCase input)` in Kotlin, `orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input))` in Java, rendered as `CASE WHEN ... THEN 0 ELSE 1 END` on every database. A keyset `scroll` follows it, and `Scrollable.sortBy` and `sortByDescending` take a field of any table in the query.
 - Added `SqlFunction`, a one-argument SQL function applied to a column wherever a query takes one, so a query can match the expression an index is built on (#694). `foldAccents(Movie_.title)` in Kotlin, `foldAccents.apply(Movie_.title)` in Java; a comparison applies it to the column and the value alike, so an accent-insensitive search on PostgreSQL uses an index on `LOWER(fold_accents(title))`.
 - The Kotlin API takes a type argument where the call needs no `_` placeholders, and a class literal where it would (#719): `subquery<Visit>()`, `orm.selectFrom<Pet>()` and `orm.selectFrom<Pet, PetSummary>()`, next to `orm.entity(User::class)` and `select(Owner::class)`. Ktor gains `entity(User::class)` and `projection(OwnerView::class)` on `Application`, `ApplicationCall` and routes.

docs/search.md

@@ -100,6 +100,49 @@ List<Movie> suggestions = orm.entity(Movie.class)
 
 Every statement `suggest` renders is therefore served by the substring index, however the user types; the trade-off is that a short word does not narrow the suggestions. The search page the box leads to keeps `search`, which matches every word. Which words are short is the dialect's to say: three characters on PostgreSQL, whose trigram index needs three, and none on the other databases, where `suggest` matches as `search` does, except that an empty input matches no rows. The `storm.search.min_term_length` property sets another length, for an index of a different kind.
 
+### Ignoring Words
+
+A word the index serves can be as useless as a short one when it occurs in a large share of the rows: `the` is three characters, so the trigram index serves it, but it matches a third of the titles in a movie catalog. Typed alone it returns a page of noise, and next to another word it narrows nothing. Which words those are depends on the column's content, `the` and `of` for titles, `de` and `van` for names, so the search box names them, as it names the columns. `suggest` takes the words to ignore after the input:
+
+<Tabs groupId="language">
+<TabItem value="kotlin" label="Kotlin" default>
+
+```kotlin
+val titleNoise = setOf("the", "a", "of")
+
+val suggestions = orm.entity(Movie::class)
+    .select()
+    .where(suggest(input, titleNoise, Movie_.title, Movie_.director.name))
+    .limit(6)
+    .resultList
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+private static final Set<String> TITLE_NOISE = Set.of("the", "a", "of");
+
+List<Movie> suggestions = orm.entity(Movie.class)
+    .select()
+    .where(it -> it.suggest(input, TITLE_NOISE, Movie_.title, Movie_.director.name))
+    .limit(6)
+    .getResultList();
+```
+
+</TabItem>
+</Tabs>
+
+An ignored word is treated as a short word is: it takes no part in the predicate, compared ignoring case as the words themselves match, and an input without a remaining word matches no rows. A quoted phrase is one word, ignored when it equals an ignored word, not when it contains one.
+
+| Input | Words `suggest` matches |
+|-------|----------|
+| `the god` | `god`: finds "The Godfather" and "City of God" |
+| `The` | none: no rows |
+| `"the good"` | `the good`: the phrase is one word |
+
+The set is a constant next to the box that uses it, typed by hand for most boxes. Where the catalog is large enough to measure, a query over the column counts the words and the set holds those that occur in more rows than the box shows. `search` takes no such set: on a results page a frequent word next to a longer one costs nothing and still narrows correctly, and a page of every title with `the` in it is a broad but legitimate query.
+
 ## Searching Related Rows
 
 Sometimes the user remembers something about a related row rather than the row itself: the front desk knows the dog is called Leo, not the owner's last name. To search the rows related to each row, semi-join the related table and name its column next to the row's own. Every owner comes back once, however many pets match:

storm-core/src/main/java/st/orm/core/template/impl/PredicateBuilderFactory.java

@@ -15,12 +15,15 @@
  */
 package st.orm.core.template.impl;
 
+import static java.util.Locale.ROOT;
 import static java.util.Objects.requireNonNull;
+import static java.util.stream.Collectors.toSet;
 import static st.orm.Operator.CONTAINS_IGNORE_CASE;
 import static st.orm.core.template.TemplateString.wrap;
 
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
 import st.orm.Data;
 import st.orm.Metamodel;
 import st.orm.Operator;
@@ -115,7 +118,7 @@ static <T extends Data, R, ID, V> PredicateBuilder<T, R, ID> createWithId(
      */
     static <T extends Data, R, ID> PredicateBuilder<T, R, ID> search(String input,
                                                                      List<? extends Metamodel<?, String>> paths) {
-        return words(input, paths, false);
+        return words(input, Set.of(), paths, false);
     }
 
     /**
@@ -136,21 +139,49 @@ static <T extends Data, R, ID> PredicateBuilder<T, R, ID> search(String input,
      */
     static <T extends Data, R, ID> PredicateBuilder<T, R, ID> suggest(String input,
                                                                       List<? extends Metamodel<?, String>> paths) {
-        return words(input, paths, true);
+        return suggest(input, Set.of(), paths);
     }
 
     /**
-     * Creates the predicate of {@link #search(String, List)} or {@link #suggest(String, List)}. Each word is kept
-     * apart until the query is built, since which words a substring index serves depends on the dialect.
+     * Creates the predicate of {@link #suggest(String, List)}, ignoring the given terms as well: a term that equals
+     * one of them, ignoring case, takes no part in the predicate, as a term the index does not serve takes none, so an
+     * input of ignored terms alone matches no rows. A search box names the words that occur in a large share of its
+     * rows, such as "the" in a catalog of titles, which the index serves but which narrow nothing.
+     *
+     * @param input the search input, as a search box receives it.
+     * @param ignore the terms to ignore, compared ignoring case.
+     * @param paths the text columns to search, at least one.
+     * @param <T> the type of the table being queried.
+     * @param <R> the type of the result.
+     * @param <ID> the type of the primary key.
+     * @return the predicate.
+     * @since 1.16
+     */
+    static <T extends Data, R, ID> PredicateBuilder<T, R, ID> suggest(String input,
+                                                                      Set<String> ignore,
+                                                                      List<? extends Metamodel<?, String>> paths) {
+        requireNonNull(ignore, "ignore");
+        return words(input, ignore, paths, true);
+    }
+
+    /**
+     * Creates the predicate of {@link #search(String, List)} or {@link #suggest(String, Set, List)}. Each word is kept
+     * apart until the query is built, since which words a substring index serves depends on the dialect; an ignored
+     * word is left out here, since no dialect serves it.
      */
     private static <T extends Data, R, ID> PredicateBuilder<T, R, ID> words(String input,
+                                                                            Set<String> ignore,
                                                                             List<? extends Metamodel<?, String>> paths,
                                                                             boolean suggest) {
         requireNonNull(input, "input");
         if (paths.isEmpty()) {
             throw new IllegalArgumentException("%s needs at least one path to search.".formatted(suggest ? "suggest" : "search"));
         }
         List<String> terms = SearchTerms.parse(input);
+        if (!ignore.isEmpty()) {
+            Set<String> ignored = ignore.stream().map(word -> word.toLowerCase(ROOT)).collect(toSet());
+            terms = terms.stream().filter(term -> !ignored.contains(term.toLowerCase(ROOT))).toList();
+        }
         var words = new ArrayList<PredicateNode>(terms.size());
         for (String term : terms) {
             var matches = new ArrayList<PredicateNode>(paths.size());

storm-core/src/test/java/st/orm/core/SearchIntegrationTest.java

@@ -22,6 +22,7 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
 import javax.sql.DataSource;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
@@ -63,12 +64,17 @@ private List<String> search(String input, Metamodel<Owner, String>... paths) {
 
     @SafeVarargs
     private List<String> suggest(String input, int minimumLength, Metamodel<Owner, String>... paths) {
+        return suggest(input, Set.of(), minimumLength, paths);
+    }
+
+    @SafeVarargs
+    private List<String> suggest(String input, Set<String> ignore, int minimumLength, Metamodel<Owner, String>... paths) {
         var orm = ORMTemplate.of(dataSource,
                 StormConfig.of(Map.of(StormConfig.SEARCH_MIN_TERM_LENGTH, String.valueOf(minimumLength))));
         statements = new ArrayList<>();
         List<String> names = new ArrayList<>();
         observe(sql -> statements.add(sql.statement()), () -> orm.entity(Owner.class).select()
-                .where(it -> PredicateBuilderFactory.<Owner, Owner, Integer>suggest(input, List.of(paths)))
+                .where(it -> PredicateBuilderFactory.<Owner, Owner, Integer>suggest(input, ignore, List.of(paths)))
                 .getResultList()
                 .forEach(owner -> names.add(owner.firstName())));
         return names.stream().sorted().toList();
@@ -124,6 +130,24 @@ public void suggestMatchesTheWordsTheIndexServes() {
         assertEquals(List.of(), suggest("", 0, Owner_.lastName));
     }
 
+    @Test
+    public void suggestIgnoresTheWordsTheSearchBoxNames() {
+        var ignore = Set.of("Davis", "the");
+        // An ignored word takes no part, whatever its case; alone, it matches no rows.
+        assertEquals(List.of("Betty", "Harold"), suggest("davis", 3, Owner_.lastName));
+        assertEquals(List.of(), suggest("davis", ignore, 3, Owner_.lastName));
+        assertEquals("1 <> 1", whereClause());
+        assertEquals(List.of(), suggest("DAVIS the", ignore, 3, Owner_.lastName));
+        // The other words match as before, each over every path.
+        assertEquals(List.of("Betty"), suggest("the davis betty", ignore, 3, Owner_.lastName, Owner_.firstName));
+        assertEquals(2, whereClause().split("LIKE").length - 1);
+        // A quoted phrase is one word, ignored when it equals an ignored word.
+        assertEquals(List.of(), suggest("\"Davis\"", ignore, 3, Owner_.lastName));
+        // Ignored on a dialect that treats every word alike as well.
+        assertEquals(List.of("Betty"), suggest("the betty", ignore, 0, Owner_.firstName));
+        assertEquals(List.of(), suggest("the", ignore, 0, Owner_.firstName));
+    }
+
     @Test
     public void inputWithoutTermsMatchesEveryRow() {
         assertEquals(10, search("", Owner_.lastName).size());

storm-java21/src/main/java/st/orm/template/WhereBuilder.java

@@ -21,6 +21,7 @@
 
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
 import st.orm.Data;
 import st.orm.Navigable;
 import st.orm.Operator;
@@ -286,10 +287,33 @@ public final PredicateBuilder<T, R, ID> search(String input,
     public final PredicateBuilder<T, R, ID> suggest(String input,
                                                     Navigable<? extends T, String> path,
                                                     Navigable<? extends T, String>... more) {
+        return suggest(input, Set.of(), path, more);
+    }
+
+    /**
+     * Matches the words of a search box that queries as the user types against one or more text columns, as
+     * {@link #suggest(String, Navigable, Navigable[])} does, ignoring the given words as well. A word that equals one
+     * of them, ignoring case, takes no part in the predicate, as a word the index does not serve takes none, so an
+     * input of ignored words alone matches no rows. A search box names the words that occur in a large share of its
+     * rows, such as "the" in a catalog of titles, which the index serves but which narrow nothing:
+     * {@code suggest("the god", Set.of("the", "a", "of"), Movie_.title)} matches on "god" alone.
+     *
+     * @param input  the search input, as a search box receives it.
+     * @param ignore the words to ignore, compared ignoring case.
+     * @param path   the first text column to search.
+     * @param more   further text columns to search.
+     * @return the predicate builder.
+     * @since 1.16
+     */
+    @SafeVarargs
+    public final PredicateBuilder<T, R, ID> suggest(String input,
+                                                    Set<String> ignore,
+                                                    Navigable<? extends T, String> path,
+                                                    Navigable<? extends T, String>... more) {
         var paths = new ArrayList<Navigable<? extends T, String>>(more.length + 1);
         paths.add(path);
         paths.addAll(List.of(more));
-        return suggestImpl(input, paths);
+        return suggestImpl(input, ignore, paths);
     }
 
     /**
@@ -327,12 +351,16 @@ protected abstract <V> PredicateBuilder<T, R, ID> whereImpl(Navigable<? extends
     protected abstract PredicateBuilder<T, R, ID> searchImpl(String input, List<Navigable<? extends T, String>> paths);
 
     /**
-     * Matches the words of a search input that the database's substring index serves against the given text columns.
+     * Matches the words of a search input that the database's substring index serves, and that the search box does
+     * not ignore, against the given text columns.
      *
-     * @param input the search input.
-     * @param paths the text columns to search, at least one.
+     * @param input  the search input.
+     * @param ignore the words to ignore, compared ignoring case.
+     * @param paths  the text columns to search, at least one.
      * @return the predicate builder.
      * @since 1.16
      */
-    protected abstract PredicateBuilder<T, R, ID> suggestImpl(String input, List<Navigable<? extends T, String>> paths);
+    protected abstract PredicateBuilder<T, R, ID> suggestImpl(String input,
+                                                              Set<String> ignore,
+                                                              List<Navigable<? extends T, String>> paths);
 }

storm-java21/src/main/java/st/orm/template/impl/QueryBuilderImpl.java

@@ -24,6 +24,7 @@
 
 import java.util.List;
 import java.util.Optional;
+import java.util.Set;
 import java.util.function.Function;
 import java.util.stream.Stream;
 import org.jspecify.annotations.Nullable;
@@ -634,8 +635,10 @@ protected PredicateBuilder<TX, RX, IDX> searchImpl(String input, List<Navigable<
         }
 
         @Override
-        protected PredicateBuilder<TX, RX, IDX> suggestImpl(String input, List<Navigable<? extends TX, String>> paths) {
-            return new PredicateBuilderImpl<>(PredicateBuilderFactory.suggest(input, paths.stream().map(Navigable::asMetamodel).toList()));
+        protected PredicateBuilder<TX, RX, IDX> suggestImpl(String input,
+                                                            Set<String> ignore,
+                                                            List<Navigable<? extends TX, String>> paths) {
+            return new PredicateBuilderImpl<>(PredicateBuilderFactory.suggest(input, ignore, paths.stream().map(Navigable::asMetamodel).toList()));
         }
     }
 

storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java

@@ -25,6 +25,8 @@
 
 import java.util.List;
 import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
 import java.util.stream.Stream;
 import javax.sql.DataSource;
 import org.junit.jupiter.api.Test;
@@ -776,6 +778,20 @@ private static List<String> semiJoinSuggestFirstNames(ORMTemplate template, Stri
                 .getResultList().stream().map(Owner::firstName).sorted().toList();
     }
 
+    @Test
+    public void testSuggestIgnoresTheWordsTheSearchBoxNames() {
+        // data.sql: owner 1 (Betty Davis) has pet Leo; owner 4 (Harold Davis) has pet Iggy.
+        var ignore = Set.of("davis", "the");
+        Function<String, List<String>> firstNames = input -> orm.entity(Owner.class).select()
+                .semiJoin(Pet.class).on(Owner.class)
+                .where(it -> it.suggest(input, ignore, Owner_.lastName, Pet_.name))
+                .getResultList().stream().map(Owner::firstName).sorted().toList();
+        assertEquals(List.of("Betty"), firstNames.apply("The Davis leo"));
+        assertEquals(List.of(), firstNames.apply("the davis"));
+        assertEquals(List.of(), firstNames.apply(""));
+        assertEquals(List.of("Betty", "Harold"), semiJoinSuggestFirstNames(orm, "davis"));
+    }
+
     private List<String> semiJoinSearchFirstNames(String input) {
         return orm.entity(Owner.class).select()
                 .semiJoin(Pet.class).on(Owner.class)