diff --git a/CHANGELOG.md b/CHANGELOG.md index 47a61c5c4..a7bd801ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ for the CLI, to [npm](https://www.npmjs.com/package/@storm-orm/cli) - Added `semiJoin` and `antiJoin`, which filter rows by a related table without adding its rows, so no `distinct` is needed (#690). `semiJoin().on()` keeps the owners with a pet matching the conditions that name `Pet`, `antiJoin` the owners without one. They filter `select()`, `delete()` and `update()`, and nest to filter through two to-many relations: `semiJoin().on().semiJoin().on()` (#717). A search over a row and its related rows reads each table through its own indexes: 1.3 s to a few milliseconds over 980,000 credits on PostgreSQL. - A typed join takes a condition: `on { Pet_.name eq "Leo" }` in Kotlin, `on(Owner.class, it -> ...)` in Java (#690). On a left join it keeps the rows without a match, which the same condition in `where` drops. - Added query columns, the columns a query names but the entity does not carry (#743): `@QueryColumn(field = "titleFolded", type = String::class)` on an entity or projection generates `Movie_.titleFolded` as a metamodel field with no record component, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well. A generated column that stores a folded title for a search index is the typical case. Schema validation checks the column exists with a compatible type and, on an entity, reports a `NOT NULL` query column the database does not fill as unmapped (`UNMAPPED_COLUMN`), since every insert leaves it out. Schema validation reads whether a column is generated, and no longer reports a generated `NOT NULL` column no field maps as unmapped. `Model.queryColumns()` lists them. +- A query column names the function that computed it, `@QueryColumn(field = "titleFolded", type = String::class, function = "fold_accents")` (#745): a value the column is compared with passes through the function on the value side, `LOWER(m.title_folded) LIKE LOWER(fold_accents(?))`, so the column is read as stored and the database folds the input in the same statement. It applies to the comparison operators, `IN`, `BETWEEN`, the text operators, `search`, `suggest` and `orderFirst`; a keyset position holds the stored value and is compared as it is. `Column.function()` reports it. - Added `suggest`, the `search` for input typed as you go: it ignores words shorter than the database's substring index serves and matches nothing when no word is left, so it never reads every row. `search` tests such short words only on the rows its longer words found: "de niro" over 980,000 credits went from 2.3 s to 19 ms on PostgreSQL. The length comes from `SqlDialect.minimumSearchTermLength()`, 3 on PostgreSQL and 0 elsewhere, and `storm.search.min_term_length` overrides it. `suggest` also takes the words a search box ignores, such as `the` in a catalog of titles, which the index serves but which narrow nothing: `suggest(input, setOf("the", "a", "of"), Movie_.title)` treats them as short words, compared ignoring case (#741). - Added `orderFirst`, which sorts the rows matching a predicate first: `orderFirst(Movie_.title eqIgnoreCase input)` in Kotlin, `orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input))` in Java, rendered as `CASE WHEN ... THEN 0 ELSE 1 END` on every database. A keyset `scroll` follows it, and `Scrollable.sortBy` and `sortByDescending` take a field of any table in the query. - Added `SqlFunction`, a one-argument SQL function applied to a column wherever a query takes one, so a query can match the expression an index is built on (#694). `foldAccents(Movie_.title)` in Kotlin, `foldAccents.apply(Movie_.title)` in Java; a comparison applies it to the column and the value alike, so an accent-insensitive search on PostgreSQL uses an index on `LOWER(fold_accents(title))`. diff --git a/docs/entities.md b/docs/entities.md index d75d50381..4cf1e6dd2 100644 --- a/docs/entities.md +++ b/docs/entities.md @@ -699,7 +699,7 @@ A query column is a column a query names but the entity does not carry. `@QueryC ```kotlin -@QueryColumn(field = "titleFolded", type = String::class) +@QueryColumn(field = "titleFolded", type = String::class, function = "fold_accents") data class Movie( @PK val id: Int = 0, val title: String, @@ -721,7 +721,7 @@ orm.entity(Movie::class) ```java -@QueryColumn(field = "titleFolded", type = String.class) +@QueryColumn(field = "titleFolded", type = String.class, function = "fold_accents") record Movie(@PK Integer id, String title, Integer year) implements Entity {} ``` @@ -748,6 +748,7 @@ CREATE INDEX idx_movie_title_folded ON movie USING gin (LOWER(title_folded) gin_ - A query column is never selected, inserted or updated. A record holds no value for it: `Metamodel.getValue` throws, and `update().set` refuses it. - A column is carried by a field or named by a query column, not both. A query column whose column a field maps is refused when the model is built, as is one named after a record component. - It is reached through paths like any other field: a query column declared on `Movie` is `Credit_.movie.titleFolded` from a query on credits. Projections of the same table declare their own. +- A query column that stores a function's result names it: `function = "fold_accents"`. A value the column is compared with passes through the function, on the value side only, `title_folded = fold_accents(?)`, so the column is read as stored and the database computes the value in the same statement. It applies to the comparison operators, `IN`, `BETWEEN`, the text operators, `search`, `suggest` and `orderFirst`; a pattern passes through it too, so the function must leave the wildcards and the escape character as they are. A keyset position holds the stored value and is compared as it is. - `where`, `orderBy`, `orderFirst`, `search` and `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it. A keyset sort refuses a nullable query column, as it refuses a nullable field; `nullable = true` declares one. - Schema validation checks that the column exists with a compatible type. On an entity every insert leaves it out, so the database must fill it: a `NOT NULL` query column that is not generated, not an identity column and has no default is reported as an [unmapped column](validation.md) is. diff --git a/docs/search.md b/docs/search.md index 95b3eac3d..779c761f9 100644 --- a/docs/search.md +++ b/docs/search.md @@ -318,21 +318,19 @@ ALTER TABLE movie ADD COLUMN title_folded VARCHAR GENERATED ALWAYS AS (fold_acce CREATE INDEX idx_movie_title_folded ON movie USING gin (LOWER(title_folded) gin_trgm_ops); ``` -The search then compares `LOWER(m.title_folded) LIKE LOWER(?)`, the expression the index holds, and the recheck reads the stored column. The input must be folded by the same function, in the database, before the search; applying `foldAccents` to the query column instead would fold both sides again and bring the recheck back. +The query column names the function that computed it. A value compared with the column passes through that function, on the value side only, so the search compares `LOWER(m.title_folded) LIKE LOWER(fold_accents(?))`: the column as stored, which is the expression the index holds, and the input folded by the database in the same statement. The recheck reads the stored column, and a caller passes the input as it is. Applying `foldAccents` to the query column instead would fold both sides again and bring the recheck back. ```kotlin -@QueryColumn(field = "titleFolded", type = String::class) +@QueryColumn(field = "titleFolded", type = String::class, function = "fold_accents") data class Movie(...) : Entity -val folded = orm.query { "SELECT fold_accents($input)" }.singleResult() - orm.entity(Movie::class) .select() - .where(search(folded, Movie_.titleFolded)) - .orderFirst(Movie_.titleFolded eqIgnoreCase folded) + .where(search(input, Movie_.titleFolded)) + .orderFirst(Movie_.titleFolded eqIgnoreCase input) .resultList ``` @@ -340,15 +338,13 @@ orm.entity(Movie::class) ```java -@QueryColumn(field = "titleFolded", type = String.class) +@QueryColumn(field = "titleFolded", type = String.class, function = "fold_accents") record Movie(...) implements Entity {} -String folded = orm.query(RAW."SELECT fold_accents(\{input})").getSingleResult(String.class); - orm.entity(Movie.class) .select() - .where(it -> it.search(folded, Movie_.titleFolded)) - .orderFirst(it -> it.where(Movie_.titleFolded, EQUALS_IGNORE_CASE, folded)) + .where(it -> it.search(input, Movie_.titleFolded)) + .orderFirst(it -> it.where(Movie_.titleFolded, EQUALS_IGNORE_CASE, input)) .getResultList(); ``` diff --git a/storm-core/src/main/java/st/orm/core/template/Column.java b/storm-core/src/main/java/st/orm/core/template/Column.java index 728ba03f4..ae0c4cad1 100644 --- a/storm-core/src/main/java/st/orm/core/template/Column.java +++ b/storm-core/src/main/java/st/orm/core/template/Column.java @@ -150,6 +150,16 @@ public interface Column { */ boolean ref(); + /** + * Returns the SQL function whose result a query column stores, which a value compared with the column passes + * through on the value side: {@code title_folded = fold_accents(?)}. Empty for a column compared with a value as + * it is, which every column a field maps is. + * + * @return the function's name, or an empty string. + * @since 1.16 + */ + String function(); + /** * Returns the metamodel for this element. * diff --git a/storm-core/src/main/java/st/orm/core/template/impl/CacheableProcessor.java b/storm-core/src/main/java/st/orm/core/template/impl/CacheableProcessor.java index f63633e49..44473afb4 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/CacheableProcessor.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/CacheableProcessor.java @@ -74,16 +74,17 @@ private Object key( ) throws SqlTemplateException { return switch(cacheable.expression()) { case TemplateExpression it -> keyGenerator.apply(it.template()); - case ObjectExpression(var metamodel, var operator, var object, var indexable) -> { + case ObjectExpression(var metamodel, var operator, var object, var indexable, var fromRow) -> { var objectShape = getObjectShape(object, ignoreArity); if (objectShape == null) { yield null; } - // Whether the comparison asks for an index is part of what it compiles to. + // Whether the comparison asks for an index, and whether its value came from a row, are part of what it + // compiles to. if (metamodel == null) { - yield List.of(operator, objectShape, indexable); + yield List.of(operator, objectShape, indexable, fromRow); } - yield List.of(metamodel, operator, objectShape, indexable); + yield List.of(metamodel, operator, objectShape, indexable, fromRow); } }; } diff --git a/storm-core/src/main/java/st/orm/core/template/impl/ColumnImpl.java b/storm-core/src/main/java/st/orm/core/template/impl/ColumnImpl.java index c409bce87..525d57749 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/ColumnImpl.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/ColumnImpl.java @@ -39,6 +39,8 @@ * @param updatable whether the column is updatable. * @param version whether the column is a version column. * @param ref whether the column is a lazily fetched record. + * @param function the SQL function whose result a query column stores, which a value compared with the column + * passes through; empty for a column compared with a value as it is. * @param metamodel the metamodel for the column. * @param secondaryMetamodel the secondary metamodel for the column: the referenced primary key metamodel for * foreign keys, or the component's own metamodel for compound key components. @@ -59,6 +61,7 @@ public record ColumnImpl( boolean updatable, boolean version, boolean ref, + String function, Metamodel metamodel, Metamodel secondaryMetamodel ) implements Column { diff --git a/storm-core/src/main/java/st/orm/core/template/impl/Elements.java b/storm-core/src/main/java/st/orm/core/template/impl/Elements.java index af5be1040..c9d196640 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/Elements.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/Elements.java @@ -195,12 +195,15 @@ public sealed interface Expression {} * When the metamodel is {@code null}, the object is matched against the table's primary key. * *

{@code indexable} is {@code false} for a comparison that does not select rows, such as the condition an - * {@code orderFirst} ranks the rows by: an index on its column spares no read, so the comparison asks for none.

+ * {@code orderFirst} ranks the rows by: an index on its column spares no read, so the comparison asks for none. + * {@code fromRow} is {@code true} for a value read from a row of the table, as a keyset position holds one: it is + * in the column's own form already and is compared as it is, without the function a query column names for the + * values a caller supplies.

*/ public record ObjectExpression(@Nullable Metamodel metamodel, Operator operator, Object object, - boolean indexable) implements Expression { + boolean indexable, boolean fromRow) implements Expression { public ObjectExpression(@Nullable Metamodel metamodel, Operator operator, Object object) { - this(metamodel, operator, object, true); + this(metamodel, operator, object, true, false); } public ObjectExpression(Operator operator, Object object) { this(null, operator, object); @@ -213,7 +216,7 @@ public ObjectExpression(Object object) { * Returns this comparison as one that does not select rows, and so asks no index for its column. */ public ObjectExpression ranking() { - return indexable ? new ObjectExpression(metamodel, operator, object, false) : this; + return indexable ? new ObjectExpression(metamodel, operator, object, false, fromRow) : this; } public ObjectExpression { requireNonNull(object, "object"); diff --git a/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java b/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java index 69d928806..6ac3b047e 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java @@ -57,6 +57,7 @@ import st.orm.Persist; import st.orm.QueryColumn; import st.orm.Ref; +import st.orm.SqlFunction; import st.orm.SqlTemplateException; import st.orm.Version; import st.orm.core.spi.Name; @@ -175,7 +176,7 @@ private static List adjustColumnsForJoinedSubtype(List columns, ci.columnName(), ci.index(), ci.type(), ci.persistedType(), true, GenerationStrategy.NONE, "", ci.foreignKey(), ci.foreignKeyGeneration(), ci.keyIndex(), ci.nullable(), - ci.insertable(), ci.updatable(), ci.version(), ci.ref(), + ci.insertable(), ci.updatable(), ci.version(), ci.ref(), ci.function(), ci.metamodel(), ci.secondaryMetamodel() )); } else { @@ -192,7 +193,7 @@ private static List adjustColumnsForJoinedSubtype(List columns, ci.columnName(), ci.index(), ci.type(), ci.persistedType(), false, ci.generation(), ci.sequence(), ci.foreignKey(), ci.foreignKeyGeneration(), ci.keyIndex(), ci.nullable(), - false, false, ci.version(), ci.ref(), + false, false, ci.version(), ci.ref(), ci.function(), ci.metamodel(), ci.secondaryMetamodel() )); } else { @@ -258,6 +259,7 @@ private static Model createSealedModel(ModelBuilder discriminator, // updatable only when discriminator exists (supports type changes) false, // not version false, // not ref + "", (Metamodel) rootMetamodel, null )); @@ -350,6 +352,7 @@ private static Model createSealedModel(ModelBuilder !extensionColumn && !isPk, // updatable field.isAnnotationPresent(Version.class), false, // not ref + "", columnMetamodel, sealedFieldMetamodel )); @@ -568,6 +571,13 @@ private static void createQueryColumns(BuildContext ctx, ColumnName columnName = queryColumn.column().isEmpty() ? new ColumnName(ctx.builder().columnNameResolver().resolveColumnName(field)) : new ColumnName(queryColumn.column()); + if (!queryColumn.function().isEmpty()) { + try { + SqlFunction.of(queryColumn.function()); + } catch (IllegalArgumentException e) { + throw new SqlTemplateException("Query column '%s' of %s names function '%s', which is not a SQL identifier.".formatted(fieldName, type.getSimpleName(), queryColumn.function()), e); + } + } String mappedBy = fieldColumns.get(columnName.name().toLowerCase(ROOT)); if (mappedBy != null) { throw new SqlTemplateException("Query column '%s' of %s names column '%s', which field '%s' maps; a query column is a column the record does not carry.".formatted(fieldName, type.getSimpleName(), columnName.name(), mappedBy)); @@ -591,6 +601,7 @@ private static void createQueryColumns(BuildContext ctx, false, false, false, + queryColumn.function(), metamodel, null )); @@ -659,6 +670,7 @@ private static void emitColumns(BuildContext ctx, spec.updatable(), spec.version(), spec.ref(), + "", metamodel, secondaryMetamodel )); diff --git a/storm-core/src/main/java/st/orm/core/template/impl/QueryBuilderImpl.java b/storm-core/src/main/java/st/orm/core/template/impl/QueryBuilderImpl.java index 6903b422d..d674cf764 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/QueryBuilderImpl.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/QueryBuilderImpl.java @@ -246,7 +246,9 @@ private static PredicateBuilder compare(WhereB return wb.where(TemplateString.combine(ranks.get(index), TemplateString.of(" " + symbol + " "), wrap(value))); } int field = index - ranks.size(); - return wb.where((Metamodel) orders.get(field).field(), operator, new Object[] {values.get(field)}); + // The value was read from a row, so it is in the column's form already and is compared as it is, without the + // function a query column names for the values a caller supplies. + return new PredicateBuilderImpl<>(wrap(new ObjectExpression(orders.get(field).field(), operator, new Object[] {values.get(field)}, true, true))); } /** diff --git a/storm-core/src/main/java/st/orm/core/template/impl/QueryModelImpl.java b/storm-core/src/main/java/st/orm/core/template/impl/QueryModelImpl.java index 3ccc6ee57..6330b0e06 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/QueryModelImpl.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/QueryModelImpl.java @@ -252,7 +252,7 @@ public String compileExpression(Expression expression, TemplateCompiler compiler try { return switch (expression) { case TemplateExpression it -> compileTemplateExpression(it.template(), compiler); - case ObjectExpression it -> compileObjectExpression(getMetamodel(it), it.operator(), it.object(), it.indexable(), compiler); + case ObjectExpression it -> compileObjectExpression(getMetamodel(it), it.operator(), it.object(), it.indexable(), it.fromRow(), compiler); }; } catch (SqlTemplateException e) { throw new UncheckedSqlTemplateException(e); @@ -273,7 +273,7 @@ public void bindExpression(Expression expression, TemplateBinder binder) { try { switch (expression) { case TemplateExpression it -> bindTemplateExpression(it.template(), binder); - case ObjectExpression it -> bindObjectExpression(getMetamodel(it), it.operator(), it.object(), binder); + case ObjectExpression it -> bindObjectExpression(getMetamodel(it), it.operator(), it.object(), it.fromRow(), binder); } } catch (SqlTemplateException e) { throw new UncheckedSqlTemplateException(e); @@ -378,6 +378,7 @@ private String compileObjectExpression(Metamodel metamodel, Operator operator, Object object, boolean indexable, + boolean fromRow, TemplateCompiler compiler) throws SqlTemplateException { // A SqlFunction call compares the column it is applied to, with the call around the column and around each // value, so the database computes both sides alike. @@ -385,6 +386,9 @@ private String compileObjectExpression(Metamodel metamodel, metamodel = FunctionCalls.column(path); //noinspection DuplicatedCode Model model = getModel(metamodel); + // A query column that names a function takes each value a caller supplies through it, on the value side + // only; a value read from a row is in the column's form already. + String valueFunction = fromRow ? "" : valueFunction(model, metamodel); // A predicate column is looked up in the referenced table's model, which drops the path the reference was // navigated from. Keep it so the column resolves against the joined occurrence rather than the root one. String crossingPath = crossesReference(metamodel) ? metamodel.path() : null; @@ -459,7 +463,7 @@ private String compileObjectExpression(Metamodel metamodel, multiValues = InListShape.padRows(inListMode, multiValues, multiValues.getFirst().size(), compiler.dialect()); } else { - var shape = InListShape.shape(inListMode, values, path != metamodel, compiler.dialect()); + var shape = InListShape.shape(inListMode, values, path != metamodel || !valueFunction.isEmpty(), compiler.dialect()); if (shape.array() != null) { return compiler.dialect().arrayMembership(column, compiler.mapParameter(shape.array()), operator == Operator.NOT_IN); @@ -469,7 +473,7 @@ private String compileObjectExpression(Metamodel metamodel, } List placeholders = new ArrayList<>(values.size()); for (var value : values) { - placeholders.add(FunctionCalls.render(path, compiler.mapParameter(value))); + placeholders.add(FunctionCalls.render(path, applyFunction(valueFunction, compiler.mapParameter(value)))); } return ColumnComparison.render(operator, column, placeholders, multiValues, compiler::mapParameter, compiler.dialect()); } catch (IllegalArgumentException e) { @@ -531,13 +535,31 @@ SequencedMap toMap() { } } + /** + * Returns the function a value compared with the path passes through: the function a query column names, when + * the path resolves to that one column, or an empty string. + */ + private static String valueFunction(Model model, Metamodel metamodel) throws SqlTemplateException { + var columns = model.getColumns(metamodel); + return columns.size() == 1 ? columns.getFirst().function() : ""; + } + + /** + * Wraps the rendered value in the function, or returns it as it is when there is none. + */ + private static String applyFunction(String function, String sql) { + return function.isEmpty() ? sql : function + "(" + sql + ")"; + } + private void bindObjectExpression(Metamodel metamodel, Operator operator, Object object, + boolean fromRow, TemplateBinder binder) throws SqlTemplateException { Metamodel path = metamodel; metamodel = FunctionCalls.column(path); var model = getModel(metamodel); + boolean valueFunction = !fromRow && !valueFunction(model, metamodel).isEmpty(); List> multiValues = null; List singleValues = new ArrayList<>(); var values = new ObjectExpressionValues(); @@ -585,7 +607,7 @@ private void bindObjectExpression(Metamodel metamodel, return; } if (shaped) { - var shape = InListShape.shape(inListMode, singleValues, path != metamodel, template.dialect()); + var shape = InListShape.shape(inListMode, singleValues, path != metamodel || valueFunction, template.dialect()); if (shape.array() != null) { binder.bindParameter(shape.array()); return; diff --git a/storm-core/src/test/java/st/orm/core/QueryColumnIntegrationTest.java b/storm-core/src/test/java/st/orm/core/QueryColumnIntegrationTest.java index c0d0c147e..f004d69fc 100644 --- a/storm-core/src/test/java/st/orm/core/QueryColumnIntegrationTest.java +++ b/storm-core/src/test/java/st/orm/core/QueryColumnIntegrationTest.java @@ -20,6 +20,7 @@ import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import static st.orm.Operator.EQUALS; +import static st.orm.Operator.IN; import static st.orm.core.template.SqlInterceptor.observe; import static st.orm.core.template.TemplateString.raw; @@ -38,6 +39,7 @@ import st.orm.PersistenceException; import st.orm.QueryColumn; import st.orm.Scrollable; +import st.orm.SqlFunction; import st.orm.core.model.Owner; import st.orm.core.model.Owner_; import st.orm.core.model.Pet; @@ -156,7 +158,7 @@ public void insertLeavesTheQueryColumnOut() { @Test public void modelListsTheQueryColumnApart() throws Exception { var model = orm().entity(Owner.class).model(); - assertEquals(1, model.queryColumns().size()); + assertEquals(List.of("last_name_folded", "last_name_upper"), model.queryColumns().stream().map(st.orm.core.template.Column::name).toList()); var column = model.queryColumns().getFirst(); assertEquals("last_name_folded", column.name()); assertEquals(0, column.index()); @@ -204,4 +206,87 @@ public void queryColumnNamingAMappedColumnIsRefused() { } assertTrue(messages.stream().anyMatch(message -> message.contains("which field 'name' maps")), messages.toString()); } + + // A query column that names the function which computed it: owner.last_name_upper stores upper(last_name). + + @Test + public void queryColumnFunctionFoldsTheValueInTheStatement() { + List statements = new ArrayList<>(); + List owners = observe(sql -> statements.add(sql.statement().replaceAll("\\s+", " ")), + () -> orm().entity(Owner.class).select().where(Owner_.lastNameUpper, EQUALS, "davis").getResultList()); + assertEquals(List.of("Betty", "Harold"), firstNames(owners)); + // The column is read as stored; the value passes through the function. + assertTrue(statements.getFirst().contains("last_name_upper = upper(?)"), statements.getFirst()); + } + + @Test + public void queryColumnFunctionAppliesToSearchAndToInLists() { + List statements = new ArrayList<>(); + List searched = observe(sql -> statements.add(sql.statement().replaceAll("\\s+", " ")), + () -> orm().entity(Owner.class).select().where(it -> PredicateBuilderFactory.search("dav", List.of(Owner_.lastNameUpper))).getResultList()); + assertEquals(List.of("Betty", "Harold"), firstNames(searched)); + assertTrue(statements.getFirst().contains("LIKE LOWER(upper(?)) ESCAPE '!'"), statements.getFirst()); + List listed = observe(sql -> statements.add(sql.statement().replaceAll("\\s+", " ")), + () -> orm().entity(Owner.class).select().where(Owner_.lastNameUpper, IN, List.of("davis", "franklin")).getResultList()); + assertEquals(List.of("Betty", "George", "Harold"), firstNames(listed)); + assertTrue(statements.getLast().contains("IN (upper(?), upper(?))"), statements.getLast()); + } + + @Test + public void queryColumnFunctionAppliesToOrderFirst() { + List statements = new ArrayList<>(); + List owners = observe(sql -> statements.add(sql.statement().replaceAll("\\s+", " ")), + () -> orm().entity(Owner.class).select() + .orderFirst(it -> it.where(Owner_.lastNameUpper, EQUALS, "davis")) + .orderBy(Owner_.id) + .getResultList()); + assertEquals(List.of("Davis", "Davis"), owners.subList(0, 2).stream().map(Owner::lastName).toList()); + assertTrue(statements.getFirst().contains("upper(?)"), statements.getFirst()); + } + + @Test + public void queryColumnFunctionWrapsInsideASqlFunctionCall() { + SqlFunction lower = SqlFunction.of("lower"); + List statements = new ArrayList<>(); + List owners = observe(sql -> statements.add(sql.statement().replaceAll("\\s+", " ")), + () -> orm().entity(Owner.class).select().where(lower.apply(Owner_.lastNameUpper), EQUALS, "Davis").getResultList()); + assertEquals(List.of("Betty", "Harold"), firstNames(owners)); + // The column's own function is applied to the value first, the call around both sides after. + assertTrue(statements.getFirst().contains("lower(o.last_name_upper) = lower(upper(?))"), statements.getFirst()); + } + + @Test + public void keysetPositionIsComparedAsStored() { + List statements = new ArrayList<>(); + List ordered = orm().entity(Owner.class).select().orderBy(Owner_.lastName).orderBy(Owner_.id).getResultList(); + var first = orm().entity(Owner.class).select().scroll(Scrollable.of(Owner_.id, 3).sortBy(Owner_.lastNameUpper)); + var second = observe(sql -> statements.add(sql.statement().replaceAll("\\s+", " ")), + () -> orm().entity(Owner.class).select().scroll(first.next())); + assertEquals(ordered.subList(0, 3), first.content()); + assertEquals(ordered.subList(3, 6), second.content()); + // The position holds the value the previous window read from the column, so no function folds it again. + assertTrue(statements.getFirst().contains("last_name_upper > ?"), statements.getFirst()); + assertFalse(statements.getFirst().contains("upper(?)"), statements.getFirst()); + } + + @Test + public void modelReportsTheQueryColumnFunction() throws Exception { + var model = orm().entity(Owner.class).model(); + assertEquals("upper", model.getColumns(Owner_.lastNameUpper).getFirst().function()); + assertEquals("", model.getColumns(Owner_.lastNameFolded).getFirst().function()); + assertEquals("", model.getColumns(Owner_.lastName).getFirst().function()); + } + + @QueryColumn(field = "nameFolded", type = String.class, function = "not a function") + public record MisnamedFunctionQueryColumn(@PK Integer id, String name) implements Entity {} + + @Test + public void queryColumnFunctionMustBeAnIdentifier() { + var exception = assertThrows(PersistenceException.class, () -> orm().entity(MisnamedFunctionQueryColumn.class).model()); + var messages = new ArrayList(); + for (Throwable cause = exception; cause != null; cause = cause.getCause()) { + messages.add(String.valueOf(cause.getMessage())); + } + assertTrue(messages.stream().anyMatch(message -> message.contains("not a SQL identifier")), messages.toString()); + } } diff --git a/storm-core/src/test/java/st/orm/core/model/Owner.java b/storm-core/src/test/java/st/orm/core/model/Owner.java index 0377b8662..3b421c4cd 100644 --- a/storm-core/src/test/java/st/orm/core/model/Owner.java +++ b/storm-core/src/test/java/st/orm/core/model/Owner.java @@ -28,6 +28,7 @@ */ @Builder(toBuilder = true) @QueryColumn(field = "lastNameFolded", type = String.class) +@QueryColumn(field = "lastNameUpper", type = String.class, function = "upper") public record Owner( @PK Integer id, String firstName, diff --git a/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java b/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java index ce0bf1ff7..43c37ebe6 100644 --- a/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java +++ b/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java @@ -359,4 +359,12 @@ void testQueryColumnBehindAnUnknownPathIsRefused() { () -> MetamodelFactory.of(EntityWithQueryColumn.class, "unknown.nameFolded")); assertTrue(exception.getMessage().contains("unknown.nameFolded"), exception.getMessage()); } + + @Test + void testInlineRecordWithoutAUniqueConstraintIsNotNullable() { + Metamodel metamodel = MetamodelFactory.of(EntityWithInline.class, "address"); + + // No unique constraint restricts the inline record's NULL values, so nothing makes it nullable. + assertFalse(MetamodelFactory.isNullable(metamodel)); + } } diff --git a/storm-core/src/test/resources/data.sql b/storm-core/src/test/resources/data.sql index f88ead5e0..af5216a72 100644 --- a/storm-core/src/test/resources/data.sql +++ b/storm-core/src/test/resources/data.sql @@ -23,7 +23,7 @@ drop table if exists country CASCADE; drop table if exists country_city CASCADE; create table city (id integer auto_increment, name varchar(255) not null, primary key (id)); -create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255) not null, city_id integer, telephone varchar(255), primary key (id), version integer default 0 not null, last_name_folded varchar(255) generated always as (lower(last_name)) not null); +create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255) not null, city_id integer, telephone varchar(255), primary key (id), version integer default 0 not null, last_name_folded varchar(255) generated always as (lower(last_name)) not null, last_name_upper varchar(255) generated always as (upper(last_name)) not null); create table pet (id integer auto_increment, name varchar(255) not null, birth_date date not null, owner_id integer, type_id integer not null, primary key (id)); create table pet_type (id integer, name varchar(255) not null, primary key (id)); create table specialty (id integer auto_increment, name varchar(255) not null, primary key (id)); diff --git a/storm-foundation/src/main/java/st/orm/QueryColumn.java b/storm-foundation/src/main/java/st/orm/QueryColumn.java index d96bc6360..ec6a6d760 100644 --- a/storm-foundation/src/main/java/st/orm/QueryColumn.java +++ b/storm-foundation/src/main/java/st/orm/QueryColumn.java @@ -30,7 +30,7 @@ * value to give it. * * 
{@code
- * @QueryColumn(field = "titleFolded", type = String.class)
+ * @QueryColumn(field = "titleFolded", type = String.class, function = "fold_accents")
  * record Movie(@PK Integer id, String title, Integer year) implements Entity {}
  *
  * orm.entity(Movie.class).select()
@@ -38,6 +38,9 @@
  *     .getResultList();
  * }
* + *

The search compares {@code LOWER(m.title_folded) LIKE LOWER(fold_accents(?))}: the column as stored, the input + * folded by the database through the {@link #function()} the column names.

+ * *

The generated {@code Movie_.titleFolded} is a {@code Metamodel} and goes wherever a column does: * {@code where}, {@code orderBy}, {@code orderFirst}, {@code search} and {@code suggest}, a {@link SqlFunction}, a * keyset sort and a template placeholder. It is reached through paths like any other field, so a query column of a @@ -80,4 +83,17 @@ * keyset sort field, as a nullable field is. */ boolean nullable() default false; + + /** + * The SQL function whose result the column stores, such as {@code fold_accents} for a generated column holding + * {@code fold_accents(title)}. A value the column is compared with passes through it, on the value side only: + * {@code title_folded = fold_accents(?)}. The column is read as stored, which is what its index holds, and the + * database computes the value in the same statement, so a caller passes the input as it is. It applies wherever + * the column is compared with a value: the comparison operators, {@code IN}, {@code BETWEEN}, the text operators, + * {@code search}, {@code suggest} and {@code orderFirst}; a pattern passes through it too, so the function must + * leave the wildcards and the escape character as they are. A position of a keyset scroll holds the stored value + * and is compared as it is. A SQL identifier, optionally qualified by a schema, as {@link SqlFunction} takes. + * Empty for a column compared with a value as it is. + */ + String function() default ""; } diff --git a/storm-java21/src/main/java/st/orm/template/Column.java b/storm-java21/src/main/java/st/orm/template/Column.java index 5bbc5364e..a8d25c474 100644 --- a/storm-java21/src/main/java/st/orm/template/Column.java +++ b/storm-java21/src/main/java/st/orm/template/Column.java @@ -117,6 +117,16 @@ public interface Column { */ boolean ref(); + /** + * Returns the SQL function whose result a query column stores, which a value compared with the column passes + * through on the value side: {@code title_folded = fold_accents(?)}. Empty for a column compared with a value as + * it is, which every column a field maps is. + * + * @return the function's name, or an empty string. + * @since 1.16 + */ + String function(); + /** * Gets the metamodel of the column. * diff --git a/storm-java21/src/main/java/st/orm/template/impl/ColumnImpl.java b/storm-java21/src/main/java/st/orm/template/impl/ColumnImpl.java index ba2c0cfab..bd4f151b8 100644 --- a/storm-java21/src/main/java/st/orm/template/impl/ColumnImpl.java +++ b/storm-java21/src/main/java/st/orm/template/impl/ColumnImpl.java @@ -52,6 +52,7 @@ public record ColumnImpl( boolean updatable, boolean version, boolean ref, + String function, Metamodel metamodel ) implements Column { @@ -70,6 +71,7 @@ public ColumnImpl(st.orm.core.template.Column column) { column.updatable(), column.version(), column.ref(), + column.function(), column.metamodel() ); } diff --git a/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java b/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java index 3c1a6741b..3922962a6 100644 --- a/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java +++ b/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java @@ -1346,5 +1346,9 @@ public void testQueryColumnIsNamedByAQueryAndCarriedByNoRow() { assertEquals(List.of("last_name_folded"), model.queryColumns().stream().map(Column::name).toList()); assertTrue(model.columns().stream().noneMatch(column -> column.name().equals("last_name_folded"))); assertThrows(UnsupportedOperationException.class, () -> Owner_.lastNameFolded.getValue(owners.getFirst())); + // The query column names lower as the function that computed it, so the value side is lowered by the database. + assertEquals("lower", model.queryColumns().getFirst().function()); + List upperCased = orm.entity(Owner.class).select().where(Owner_.lastNameFolded, EQUALS, "DAVIS").getResultList(); + assertEquals(List.of("Betty", "Harold"), upperCased.stream().map(Owner::firstName).sorted().toList()); } } diff --git a/storm-java21/src/test/java/st/orm/template/model/Owner.java b/storm-java21/src/test/java/st/orm/template/model/Owner.java index 7b96f1964..b42bfe300 100644 --- a/storm-java21/src/test/java/st/orm/template/model/Owner.java +++ b/storm-java21/src/test/java/st/orm/template/model/Owner.java @@ -6,7 +6,7 @@ import st.orm.QueryColumn; import st.orm.Version; -@QueryColumn(field = "lastNameFolded", type = String.class) +@QueryColumn(field = "lastNameFolded", type = String.class, function = "lower") public record Owner( @PK Integer id, String firstName, diff --git a/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt b/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt index ec54291d9..946499f94 100644 --- a/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt +++ b/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt @@ -117,6 +117,15 @@ public interface Column { */ public val ref: Boolean + /** + * The SQL function whose result a query column stores, which a value compared with the column passes through on + * the value side: `title_folded = fold_accents(?)`. Null for a column compared with a value as it is, which every + * column a field maps is. + * + * @since 1.16 + */ + public val function: String? + /** * Gets the metamodel of the column. * diff --git a/storm-kotlin/src/main/kotlin/st/orm/template/impl/ColumnImpl.kt b/storm-kotlin/src/main/kotlin/st/orm/template/impl/ColumnImpl.kt index 363f40aa4..4800b98c8 100644 --- a/storm-kotlin/src/main/kotlin/st/orm/template/impl/ColumnImpl.kt +++ b/storm-kotlin/src/main/kotlin/st/orm/template/impl/ColumnImpl.kt @@ -36,5 +36,6 @@ internal class ColumnImpl( override val updatable: Boolean get() = core.updatable() override val version: Boolean get() = core.version() override val ref: Boolean get() = core.ref() + override val function: String? get() = core.function().ifEmpty { null } override val metamodel: Metamodel get() = core.metamodel() } diff --git a/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt b/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt index 1236a13f0..7d1c1a2eb 100644 --- a/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt +++ b/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt @@ -2293,6 +2293,10 @@ internal open class QueryBuilderTest( model.queryColumns.map { it.name } shouldBe listOf("last_name_folded") model.columns.none { it.name == "last_name_folded" } shouldBe true shouldThrow { lastNameFolded.getValue(owners.first()) } + // The query column names lower as the function that computed it, so the value side is lowered by the database. + model.queryColumns.first().function shouldBe "lower" + orm.entity(Owner::class).select().where(lastNameFolded eq "DAVIS").resultList + .map { it.firstName }.sorted() shouldBe listOf("Betty", "Harold") } @Test diff --git a/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt b/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt index 2d4a68319..1b9d8506a 100644 --- a/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt +++ b/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt @@ -24,7 +24,7 @@ import st.orm.Version * Simple domain object representing an owner. * */ -@QueryColumn(field = "lastNameFolded", type = String::class) +@QueryColumn(field = "lastNameFolded", type = String::class, function = "lower") internal data class Owner( @PK val id: Int = 0, override val firstName: String, diff --git a/website/static/skills/storm-entity-from-schema.md b/website/static/skills/storm-entity-from-schema.md index f4df3a448..52c1bf796 100644 --- a/website/static/skills/storm-entity-from-schema.md +++ b/website/static/skills/storm-entity-from-schema.md @@ -61,7 +61,7 @@ Generation/update rules: - **Composite unique constraints** (only when needed): composite unique constraints do NOT need to be modeled unless the user explicitly needs a composite `Metamodel.Key` (for keyset pagination or type-safe lookups). When needed, use an inline record + `@UK @Persist(insertable = false, updatable = false)` — ask the user if they need this. - `describe_table` reports unique constraints (top-level `uniqueConstraints` array with constraint name and column list) and FK cascade rules (onDelete/onUpdate). Both are exposed for context only. **Storm does not model cascade behavior or enforce uniqueness** — these are database-level concerns. Do not generate annotations or code for cascade rules. - @Version for version columns (confirm with user). -- Generated or computed columns that no caller reads, such as a folded or normalized copy of another column that a search index is built on: declare them as query columns on the type, `@QueryColumn(field = "titleFolded", type = String::class)` (Java: `String.class`), not as fields, so they are never selected or written. Map one as a `@Persist(insertable = false, updatable = false)` field only when the application reads its value (confirm with user). +- Generated or computed columns that no caller reads, such as a folded or normalized copy of another column that a search index is built on: declare them as query columns on the type, `@QueryColumn(field = "titleFolded", type = String::class, function = "fold_accents")` (Java: `String.class`), not as fields, so they are never selected or written; `function` names the function in the column's generation expression, so a compared value passes through it, and is left out for a column compared as it is. Map one as a `@Persist(insertable = false, updatable = false)` field only when the application reads its value (confirm with user). - When updating, preserve existing field order and custom annotations. Only add/modify what changed. - Use `@DbIgnore` on fields or types that should be excluded from schema validation. Note: ad-hoc query result types (aggregation DTOs) should not need this — write them as plain data classes/records without the `Data` marker instead of declaring `Data` and then suppressing validation with `@DbIgnore`. - Use `@PK(constraint = false)` if the table intentionally has no PK constraint in the database. diff --git a/website/static/skills/storm-entity-java.md b/website/static/skills/storm-entity-java.md index 6d59fea57..50fc0009a 100644 --- a/website/static/skills/storm-entity-java.md +++ b/website/static/skills/storm-entity-java.md @@ -146,7 +146,7 @@ Generation rules: ``` Use `insertable = false` alone for columns set by the database only on INSERT, or `updatable = false` alone for columns written once and never modified. -11c. Query columns: a column a query names but no row needs to carry, such as a generated column storing a folded title for a search index, a `tsvector`, a computed sort key or a large column a filter names, is declared on the type, not as a field: `@QueryColumn(field = "titleFolded", type = String.class)`, repeated per column; `column = "..."` names the column where the resolver does not, and `nullable = true` declares a nullable one. It generates `Movie_.titleFolded` with no record component, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well (`Credit_.movie.titleFolded`). Map such a column as a `@Persist(insertable = false, updatable = false)` field only when a caller reads its value. On an entity the database must fill a query column, since every insert leaves it out: generated, identity, with a default, or nullable. Schema validation reports one the database does not fill as an unmapped column. +11c. Query columns: a column a query names but no row needs to carry, such as a generated column storing a folded title for a search index, a `tsvector`, a computed sort key or a large column a filter names, is declared on the type, not as a field: `@QueryColumn(field = "titleFolded", type = String.class)`, repeated per column; `column = "..."` names the column where the resolver does not, `nullable = true` declares a nullable one, and `function = "fold_accents"` names the function the column stores the result of: a value compared with the column then passes through it on the value side (`title_folded = fold_accents(?)`), in `search`, `suggest`, `orderFirst`, `IN` and the operators alike, so the caller passes the input as it is and the column is read as stored. It generates `Movie_.titleFolded` with no record component, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well (`Credit_.movie.titleFolded`). Map such a column as a `@Persist(insertable = false, updatable = false)` field only when a caller reads its value. On an entity the database must fill a query column, since every insert leaves it out: generated, identity, with a default, or nullable. Schema validation reports one the database does not fill as an unmapped column. 11c. Update tuning: `@DynamicUpdate` sets how the entity's UPDATE statements are generated. The default mode (`ENTITY`) skips the UPDATE entirely when nothing changed and writes the full row when anything did. `@DynamicUpdate(FIELD)` writes only the changed columns, at the cost of one SQL shape per distinct change pattern (which splits batches); `@DynamicUpdate(OFF)` always writes all columns without comparison. `dirtyCheck = VALUE` switches change detection from reference identity to `equals()`. Semantics, trade-offs, and the foreign-key comparison rule: /storm-repository-java (Dirty Checking and Update Suppression). diff --git a/website/static/skills/storm-entity-kotlin.md b/website/static/skills/storm-entity-kotlin.md index b2cc9320e..58c3ab6da 100644 --- a/website/static/skills/storm-entity-kotlin.md +++ b/website/static/skills/storm-entity-kotlin.md @@ -166,7 +166,7 @@ Generation rules: ``` Use `insertable = false` alone for columns set by the database only on INSERT, or `updatable = false` alone for columns that are written once and never modified. -12c. Query columns: a column a query names but no row needs to carry, such as a generated column storing a folded title for a search index, a `tsvector`, a computed sort key or a large column a filter names, is declared on the type, not as a property: `@QueryColumn(field = "titleFolded", type = String::class)`, repeated per column; `column = "..."` names the column where the resolver does not, and `nullable = true` declares a nullable one. It generates `Movie_.titleFolded` with no constructor parameter, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well (`Credit_.movie.titleFolded`). Map such a column as a `@Persist(insertable = false, updatable = false)` property only when a caller reads its value. On an entity the database must fill a query column, since every insert leaves it out: generated, identity, with a default, or nullable. Schema validation reports one the database does not fill as an unmapped column. +12c. Query columns: a column a query names but no row needs to carry, such as a generated column storing a folded title for a search index, a `tsvector`, a computed sort key or a large column a filter names, is declared on the type, not as a property: `@QueryColumn(field = "titleFolded", type = String::class)`, repeated per column; `column = "..."` names the column where the resolver does not, `nullable = true` declares a nullable one, and `function = "fold_accents"` names the function the column stores the result of: a value compared with the column then passes through it on the value side (`title_folded = fold_accents(?)`), in `search`, `suggest`, `orderFirst`, `IN` and the operators alike, so the caller passes the input as it is and the column is read as stored. It generates `Movie_.titleFolded` with no constructor parameter, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well (`Credit_.movie.titleFolded`). Map such a column as a `@Persist(insertable = false, updatable = false)` property only when a caller reads its value. On an entity the database must fill a query column, since every insert leaves it out: generated, identity, with a default, or nullable. Schema validation reports one the database does not fill as an unmapped column. 12c. Update tuning: `@DynamicUpdate` sets how the entity's UPDATE statements are generated. The default mode (`ENTITY`) skips the UPDATE entirely when nothing changed and writes the full row when anything did. `@DynamicUpdate(FIELD)` writes only the changed columns, at the cost of one SQL shape per distinct change pattern (which splits batches); `@DynamicUpdate(OFF)` always writes all columns without comparison. `dirtyCheck = VALUE` switches change detection from reference identity to `equals()`. Semantics, trade-offs, and the foreign-key comparison rule: /storm-repository-kotlin (Dirty Checking and Update Suppression). diff --git a/website/static/skills/storm-query-java.md b/website/static/skills/storm-query-java.md index caf178474..02599fb94 100644 --- a/website/static/skills/storm-query-java.md +++ b/website/static/skills/storm-query-java.md @@ -139,7 +139,7 @@ Operators: EQUALS, NOT_EQUALS, LESS_THAN, LESS_THAN_OR_EQUAL, GREATER_THAN, GREA **Rank search results with `orderFirst`, never with a `CASE` template.** `search` and `suggest` decide which rows match, not which come first. `orderFirst(predicate)` puts the rows matching the predicate first, appending a sort key as `orderBy` does, so the calls apply in order: `.orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input)).orderFirst(it -> it.where(Movie_.title, STARTS_WITH_IGNORE_CASE, input)).orderByDescending(Movie_.votes)` lists an exact title, then titles starting with the input, then the most popular. The operators match the input literally and ignoring case, as in `where`; combine columns with `.or(...)`. Rank only by columns of the queried or joined tables: a semi-joined table's columns are refused, as it adds no columns. A keyset `scroll` follows `orderFirst` (its keys lead the `Scrollable`'s sort fields and key, and the cursor carries the ranks), and a `Scrollable` may sort by a field of a joined table, such as the votes of a joined rating; `orderFirst` cannot order a `distinct` query. No index serves the sort, which is fine after a search's few matches. -**A search needs an index built for it, created in the migration with the query.** `search` and the `IGNORE_CASE` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `SqlFunction foldAccents = SqlFunction.of("fold_accents")` once and searches `it.search(input, foldAccents.apply(Movie_.title))`, ranking with `orderFirst(it -> it.where(foldAccents.apply(Movie_.title), EQUALS_IGNORE_CASE, input))`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Where the recheck of the accent function on every candidate row dominates, which shows on a column where a common word matches many thousands of rows, store the folded text in a generated column (`title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL`), index `LOWER(title_folded)`, and declare it as a query column on the entity, `@QueryColumn(field = "titleFolded", type = String.class)`, so no row carries it (see the entity skill). Fold the input once in the database, `SELECT fold_accents(?)`, and search the column with the result, `it.search(folded, Movie_.titleFolded)`, so the column and the input are folded by the same function. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. +**A search needs an index built for it, created in the migration with the query.** `search` and the `IGNORE_CASE` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `SqlFunction foldAccents = SqlFunction.of("fold_accents")` once and searches `it.search(input, foldAccents.apply(Movie_.title))`, ranking with `orderFirst(it -> it.where(foldAccents.apply(Movie_.title), EQUALS_IGNORE_CASE, input))`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Where the recheck of the accent function on every candidate row dominates, which shows on a column where a common word matches many thousands of rows, store the folded text in a generated column (`title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL`), index `LOWER(title_folded)`, and declare it as a query column on the entity that names the function, `@QueryColumn(field = "titleFolded", type = String.class, function = "fold_accents")`, so no row carries it (see the entity skill) and a value compared with it passes through the function on the value side: `it.search(input, Movie_.titleFolded)` renders `LOWER(m.title_folded) LIKE LOWER(fold_accents(?))`, the column as stored and the input folded by the database in the same statement. Never fold the input in application code or in a query of its own. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. ### Naming a foreign key column diff --git a/website/static/skills/storm-query-kotlin.md b/website/static/skills/storm-query-kotlin.md index 16b545d9e..6911b5a43 100644 --- a/website/static/skills/storm-query-kotlin.md +++ b/website/static/skills/storm-query-kotlin.md @@ -104,7 +104,7 @@ User_.email.isNotNull() // IS_NOT_NULL **Rank search results with `orderFirst`, never with a `CASE` template.** `search` and `suggest` decide which rows match, not which come first. `orderFirst(predicate)` puts the rows matching the predicate first, appending a sort key as `orderBy` does, so the calls apply in order: `.orderFirst(Movie_.title eqIgnoreCase input).orderFirst(Movie_.title startsWithIgnoreCase input).orderByDescending(Movie_.votes)` lists an exact title, then titles starting with the input, then the most popular. The operators match the input literally and ignoring case, as in `where`; combine columns with `or`. Rank only by columns of the queried or joined tables: a semi-joined table's columns are refused, as it adds no columns. A keyset `scroll` follows `orderFirst` (its keys lead the `Scrollable`'s sort fields and key, and the cursor carries the ranks), and a `Scrollable` may sort by a field of a joined table, such as the votes of a joined rating; `orderFirst` cannot order a `distinct` query. No index serves the sort, which is fine after a search's few matches. -**A search needs an index built for it, created in the migration with the query.** `search` and the `IgnoreCase` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `val foldAccents = SqlFunction.of("fold_accents")` once, with an extension property that reads as a field of the column, `val Navigable.folded: Metamodel get() = foldAccents(this)`, and searches `search(input, Movie_.title.folded)`, ranking with `orderFirst(Movie_.title.folded eqIgnoreCase input)`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Where the recheck of the accent function on every candidate row dominates, which shows on a column where a common word matches many thousands of rows, store the folded text in a generated column (`title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL`), index `LOWER(title_folded)`, and declare it as a query column on the entity, `@QueryColumn(field = "titleFolded", type = String::class)`, so no row carries it (see the entity skill). Fold the input once in the database, `SELECT fold_accents(?)`, and search the column with the result, `search(folded, Movie_.titleFolded)`, so the column and the input are folded by the same function. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. +**A search needs an index built for it, created in the migration with the query.** `search` and the `IgnoreCase` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `val foldAccents = SqlFunction.of("fold_accents")` once, with an extension property that reads as a field of the column, `val Navigable.folded: Metamodel get() = foldAccents(this)`, and searches `search(input, Movie_.title.folded)`, ranking with `orderFirst(Movie_.title.folded eqIgnoreCase input)`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Where the recheck of the accent function on every candidate row dominates, which shows on a column where a common word matches many thousands of rows, store the folded text in a generated column (`title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL`), index `LOWER(title_folded)`, and declare it as a query column on the entity that names the function, `@QueryColumn(field = "titleFolded", type = String::class, function = "fold_accents")`, so no row carries it (see the entity skill) and a value compared with it passes through the function on the value side: `search(input, Movie_.titleFolded)` renders `LOWER(m.title_folded) LIKE LOWER(fold_accents(?))`, the column as stored and the input folded by the database in the same statement. Never fold the input in application code or in a query of its own. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. Combine with `and`/`or`: ```kotlin