diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dd7b2f214..cdb66bad0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,46 +79,45 @@ jobs: flags: unittests fail_ci_if_error: true - # Known-vulnerability gate over the dependency graph Storm ships (compile and runtime scope): OWASP - # Dependency-Check against the NVD, failing on a High or Critical advisory. Test and provided scope are the - # application's own dependencies and stay out of it. The NVD API key is a repository secret; Dependabot runs and - # forks receive no secrets, so the scan is skipped there and reported as such. + # Known-vulnerability gate over the dependency graph Storm ships (compile and runtime scope): the CycloneDX + # bill of materials of the reactor, scanned by Trivy, failing on a High or Critical advisory. Test and provided + # scope are the application's own dependencies and stay out of it. Trivy carries its own advisory database and + # needs no key, so the gate runs on every PR, Dependabot's and forks' included. dependency-check: runs-on: ubuntu-latest - timeout-minutes: 40 - env: - NVD_API_KEY: ${{ secrets.NVD_API_KEY }} + timeout-minutes: 20 steps: - name: Checkout repository uses: actions/checkout@v7 - name: Set up JDK 21 - if: env.NVD_API_KEY != '' uses: actions/setup-java@v6 with: distribution: 'temurin' java-version: '21' - # The Maven cache holds the NVD data the plugin keeps under ~/.m2 as well, so a run only pulls the - # advisories published since the last one. cache: 'maven' - - name: Scan the shipped dependency graph - if: env.NVD_API_KEY != '' - run: mvn -B -DskipTests -Dmaven.javadoc.skip=true -Dspotless.check.skip=true install org.owasp:dependency-check-maven:aggregate + - name: Write the bill of materials of the shipped dependency graph + run: mvn -B -DskipTests -Dmaven.javadoc.skip=true -Dspotless.check.skip=true package org.cyclonedx:cyclonedx-maven-plugin:makeAggregateBom - - name: Upload the Dependency-Check report - if: env.NVD_API_KEY != '' && always() + - name: Scan the bill of materials for known vulnerabilities + uses: aquasecurity/trivy-action@v0.36.0 + with: + scan-type: sbom + scan-ref: target/bom.json + scanners: vuln + severity: HIGH,CRITICAL + exit-code: '1' + + - name: Upload the bill of materials + if: always() uses: actions/upload-artifact@v7 with: - name: dependency-check-report - path: target/dependency-check-report.* + name: dependency-bom + path: target/bom.json if-no-files-found: ignore retention-days: 30 - - name: Report a skipped scan - if: env.NVD_API_KEY == '' - run: echo "::notice::Dependency-Check skipped - no NVD_API_KEY available to this run." - # The CLI ships to users via npx but is plain JavaScript with no compile step, # so at minimum its syntax must be checked before merge. cli-check: diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a47c7f92..47a61c5c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,7 @@ for the CLI, to [npm](https://www.npmjs.com/package/@storm-orm/cli) - `delete()` and `update()` refuse to run without a condition unless `unsafe()` is called, take explicit joins on every database, and refuse a joined inheritance hierarchy (#712). `SqlDialect.writeJoinSyntax()` replaces `supportsDeleteAlias()`. - Added `semiJoin` and `antiJoin`, which filter rows by a related table without adding its rows, so no `distinct` is needed (#690). `semiJoin().on()` keeps the owners with a pet matching the conditions that name `Pet`, `antiJoin` the owners without one. They filter `select()`, `delete()` and `update()`, and nest to filter through two to-many relations: `semiJoin().on().semiJoin().on()` (#717). A search over a row and its related rows reads each table through its own indexes: 1.3 s to a few milliseconds over 980,000 credits on PostgreSQL. - A typed join takes a condition: `on { Pet_.name eq "Leo" }` in Kotlin, `on(Owner.class, it -> ...)` in Java (#690). On a left join it keeps the rows without a match, which the same condition in `where` drops. +- Added query columns, the columns a query names but the entity does not carry (#743): `@QueryColumn(field = "titleFolded", type = String::class)` on an entity or projection generates `Movie_.titleFolded` as a metamodel field with no record component, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well. A generated column that stores a folded title for a search index is the typical case. Schema validation checks the column exists with a compatible type and, on an entity, reports a `NOT NULL` query column the database does not fill as unmapped (`UNMAPPED_COLUMN`), since every insert leaves it out. Schema validation reads whether a column is generated, and no longer reports a generated `NOT NULL` column no field maps as unmapped. `Model.queryColumns()` lists them. - Added `suggest`, the `search` for input typed as you go: it ignores words shorter than the database's substring index serves and matches nothing when no word is left, so it never reads every row. `search` tests such short words only on the rows its longer words found: "de niro" over 980,000 credits went from 2.3 s to 19 ms on PostgreSQL. The length comes from `SqlDialect.minimumSearchTermLength()`, 3 on PostgreSQL and 0 elsewhere, and `storm.search.min_term_length` overrides it. `suggest` also takes the words a search box ignores, such as `the` in a catalog of titles, which the index serves but which narrow nothing: `suggest(input, setOf("the", "a", "of"), Movie_.title)` treats them as short words, compared ignoring case (#741). - Added `orderFirst`, which sorts the rows matching a predicate first: `orderFirst(Movie_.title eqIgnoreCase input)` in Kotlin, `orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input))` in Java, rendered as `CASE WHEN ... THEN 0 ELSE 1 END` on every database. A keyset `scroll` follows it, and `Scrollable.sortBy` and `sortByDescending` take a field of any table in the query. - Added `SqlFunction`, a one-argument SQL function applied to a column wherever a query takes one, so a query can match the expression an index is built on (#694). `foldAccents(Movie_.title)` in Kotlin, `foldAccents.apply(Movie_.title)` in Java; a comparison applies it to the column and the value alike, so an accent-insensitive search on PostgreSQL uses an index on `LOWER(fold_accents(title))`. diff --git a/docs/entities.md b/docs/entities.md index 8a3fb0b59..d75d50381 100644 --- a/docs/entities.md +++ b/docs/entities.md @@ -689,6 +689,72 @@ record Pet(@PK Integer id, --- +## Query Columns + +A query can only name a column that a field maps, and every field is selected, inserted and updated. Some columns exist only to be queried: a generated column that stores a derived form of another column, which a search matches and an index is built on, but which no caller reads and the database computes. Mapped as a field it sits in every SELECT, needs `@Persist(insertable = false, updatable = false)`, and makes every caller that constructs the entity supply a value the database overwrites. The same holds for a `tsvector` column for full-text search, a normalized or hashed copy of a column, a computed sort key, and a large column a query filters on but should not load. + +A query column is a column a query names but the entity does not carry. `@QueryColumn` on the entity or projection declares it with the name of its metamodel field and the type of its values. The processors generate the field as they generate one for a record component, and the column follows from the field name through the column name resolver; a `column` attribute names it where the resolver does not. + + + + +```kotlin +@QueryColumn(field = "titleFolded", type = String::class) +data class Movie( + @PK val id: Int = 0, + val title: String, + val year: Int +) : Entity +``` + +The annotation repeats for each query column. `Movie_.titleFolded` is a `Metamodel` and goes wherever a column does: + +```kotlin +orm.entity(Movie::class) + .select() + .where(search(input, Movie_.titleFolded)) + .orderBy(Movie_.titleFolded) + .resultList +``` + + + + +```java +@QueryColumn(field = "titleFolded", type = String.class) +record Movie(@PK Integer id, String title, Integer year) implements Entity {} +``` + +The annotation repeats for each query column. `Movie_.titleFolded` is a `Metamodel` and goes wherever a column does: + +```java +orm.entity(Movie.class) + .select() + .where(it -> it.search(input, Movie_.titleFolded)) + .orderBy(Movie_.titleFolded) + .getResultList(); +``` + + + + +The migration creates the column, here generated from the title by the accent-folding function of [Storing the Folded Text](search.md#storing-the-folded-text), and the index on it: + +```sql +ALTER TABLE movie ADD COLUMN title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL; +CREATE INDEX idx_movie_title_folded ON movie USING gin (LOWER(title_folded) gin_trgm_ops); +``` + +- A query column is never selected, inserted or updated. A record holds no value for it: `Metamodel.getValue` throws, and `update().set` refuses it. +- A column is carried by a field or named by a query column, not both. A query column whose column a field maps is refused when the model is built, as is one named after a record component. +- It is reached through paths like any other field: a query column declared on `Movie` is `Credit_.movie.titleFolded` from a query on credits. Projections of the same table declare their own. +- `where`, `orderBy`, `orderFirst`, `search` and `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it. A keyset sort refuses a nullable query column, as it refuses a nullable field; `nullable = true` declares one. +- Schema validation checks that the column exists with a compatible type. On an entity every insert leaves it out, so the database must fill it: a `NOT NULL` query column that is not generated, not an identity column and has no default is reported as an [unmapped column](validation.md) is. + +Where a caller reads the value, map the column as a field with `@Persist(insertable = false, updatable = false)` instead: a query column is for a column no row needs to carry. + +--- + ## Modifying Entities Since Storm entities are immutable, updating a field means creating a new instance with the changed value. Kotlin data classes have a built-in `copy()` method for this. Java records do not provide an equivalent, but Lombok's `@Builder(toBuilder = true)` annotation generates a builder that copies all fields from an existing instance: diff --git a/docs/metamodel.md b/docs/metamodel.md index 4562b1348..8c4bf9b40 100644 --- a/docs/metamodel.md +++ b/docs/metamodel.md @@ -374,7 +374,7 @@ A direct (non-`Ref`) self-referential foreign key is rejected at generation time ## Generated Code -Understanding the generated code helps when debugging or reading compiler errors. The metamodel mirrors your entity structure, creating a static field for each entity field. Each field carries generic type parameters that encode both the root entity type and the field's value type, which is how the compiler enforces type safety in queries. +Understanding the generated code helps when debugging or reading compiler errors. The metamodel mirrors your entity structure, creating a static field for each entity field, and one for each [query column](entities.md#query-columns) the entity declares. Each field carries generic type parameters that encode both the root entity type and the field's value type, which is how the compiler enforces type safety in queries. ``` Entity Metamodel diff --git a/docs/search.md b/docs/search.md index a3400201c..95b3eac3d 100644 --- a/docs/search.md +++ b/docs/search.md @@ -308,3 +308,51 @@ The search compares `LOWER(fold_accents(m.title)) LIKE LOWER(fold_accents(?))`, - Name the function exactly as the index does. A different function, or the same one applied in another order, is another expression, and the plan falls back to reading every row. - The database folds the input too, with its own rules. PostgreSQL's `unaccent` maps `ø` to `o` and `æ` to `ae`, which removing combining marks in the application would leave alone; folding both sides in the database keeps the column and the input in step. - The function receives the search pattern, wildcards included, and an accent function leaves them alone, as a function compared through a text operator must. + +### Storing the Folded Text + +The expression index is enough for most catalogs. The index finds the candidate rows, and the database then rechecks each candidate by computing `fold_accents` from the title; on a column where a common word matches many thousands of rows per keystroke, that recheck is where the time goes. Where a search stays slow for that reason, and only then, store the folded title instead of computing it per search: a generated column holds `fold_accents(title)`, the index is built on that column, and the entity declares it as a [query column](entities.md#query-columns), a column the search names and no row carries. + +```sql +ALTER TABLE movie ADD COLUMN title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL; +CREATE INDEX idx_movie_title_folded ON movie USING gin (LOWER(title_folded) gin_trgm_ops); +``` + +The search then compares `LOWER(m.title_folded) LIKE LOWER(?)`, the expression the index holds, and the recheck reads the stored column. The input must be folded by the same function, in the database, before the search; applying `foldAccents` to the query column instead would fold both sides again and bring the recheck back. + + + + +```kotlin +@QueryColumn(field = "titleFolded", type = String::class) +data class Movie(...) : Entity + +val folded = orm.query { "SELECT fold_accents($input)" }.singleResult() + +orm.entity(Movie::class) + .select() + .where(search(folded, Movie_.titleFolded)) + .orderFirst(Movie_.titleFolded eqIgnoreCase folded) + .resultList +``` + + + + +```java +@QueryColumn(field = "titleFolded", type = String.class) +record Movie(...) implements Entity {} + +String folded = orm.query(RAW."SELECT fold_accents(\{input})").getSingleResult(String.class); + +orm.entity(Movie.class) + .select() + .where(it -> it.search(folded, Movie_.titleFolded)) + .orderFirst(it -> it.where(Movie_.titleFolded, EQUALS_IGNORE_CASE, folded)) + .getResultList(); +``` + + + + +The entity keeps one field for the title, which the query selects with its accents for display; the folded column stays out of every SELECT and INSERT, and the writes pay for the folding once per row. diff --git a/docs/validation.md b/docs/validation.md index ae3be65a5..ce0100db0 100644 --- a/docs/validation.md +++ b/docs/validation.md @@ -80,6 +80,7 @@ Schema validation compares your entity and projection definitions against the ac | Entity declares `@PK` but the database has no primary key constraint | `PRIMARY_KEY_MISSING` | Warning | | `@UK` field has a matching unique constraint in the database | `UNIQUE_KEY_MISSING` | Warning | | `@FK` field has a matching foreign key constraint in the database | `FOREIGN_KEY_MISSING` | Warning | +| A `NOT NULL` column without a default that no field maps, and that the database does not generate; a [query column](entities.md#query-columns) of an entity counts, since every insert leaves it out | `UNMAPPED_COLUMN` | Warning | **Errors** indicate definitive mismatches that will cause runtime failures, such as missing tables or columns. diff --git a/pom.xml b/pom.xml index e03062217..d4e043c81 100644 --- a/pom.xml +++ b/pom.xml @@ -265,26 +265,18 @@ - - org.owasp - dependency-check-maven - 13.0.0 + + org.cyclonedx + cyclonedx-maven-plugin + 2.9.1 - - 7 - true - true - NVD_API_KEY - - HTML - JSON - - - false + false + false + json diff --git a/storm-core/src/main/java/st/orm/core/template/Column.java b/storm-core/src/main/java/st/orm/core/template/Column.java index 0b7a5baff..728ba03f4 100644 --- a/storm-core/src/main/java/st/orm/core/template/Column.java +++ b/storm-core/src/main/java/st/orm/core/template/Column.java @@ -24,7 +24,8 @@ */ public interface Column { /** - * Gets the 1-based index of the column. + * Gets the 1-based index of the column among the model's columns, or {@code 0} for a query column, which has no + * position among them. * * @return the column index. */ diff --git a/storm-core/src/main/java/st/orm/core/template/Model.java b/storm-core/src/main/java/st/orm/core/template/Model.java index d1d15a5f0..1b9265a59 100644 --- a/storm-core/src/main/java/st/orm/core/template/Model.java +++ b/storm-core/src/main/java/st/orm/core/template/Model.java @@ -110,6 +110,20 @@ public interface Model { */ List declaredColumns(); + /** + * Returns the query columns declared on this model: the columns a query names but the record does not carry, + * declared with {@link st.orm.QueryColumn}. They are not among {@link #columns()} or {@link #declaredColumns()}, + * so a select, insert or update never names them; {@link #getColumns(Metamodel)} resolves them, for a query + * column of this type and for one of a type reached through an expanded relation. + * + *

A query column has no record field: its {@link Column#index()} is {@code 0}, and it is neither insertable + * nor updatable.

+ * + * @return the query columns declared on this model, in declaration order. + * @since 1.16 + */ + List queryColumns(); + /** * Returns the record type metadata. * diff --git a/storm-core/src/main/java/st/orm/core/template/impl/DatabaseSchema.java b/storm-core/src/main/java/st/orm/core/template/impl/DatabaseSchema.java index fa590b2b7..407b91549 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/DatabaseSchema.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/DatabaseSchema.java @@ -19,6 +19,7 @@ import java.sql.DatabaseMetaData; import java.sql.PreparedStatement; import java.sql.ResultSet; +import java.sql.ResultSetMetaData; import java.sql.SQLException; import java.util.ArrayList; import java.util.Collection; @@ -48,6 +49,18 @@ */ public final class DatabaseSchema { + /** + * Returns whether the result set has a column with the given label. + */ + private static boolean hasColumn(ResultSetMetaData resultSetMetaData, String label) throws SQLException { + for (int i = 1; i <= resultSetMetaData.getColumnCount(); i++) { + if (label.equalsIgnoreCase(resultSetMetaData.getColumnLabel(i))) { + return true; + } + } + return false; + } + private static final Logger LOGGER = LoggerFactory.getLogger("st.orm.validation"); /** @@ -61,6 +74,8 @@ public final class DatabaseSchema { * @param nullable whether the column allows NULL values. * @param autoIncrement whether the column is auto-incremented. * @param hasDefault whether the column has a default value defined. + * @param generated whether the database computes the column from an expression, as the driver reports it; + * a driver that does not report it reads as {@code false}. */ public record DbColumn( String tableName, @@ -70,7 +85,8 @@ public record DbColumn( int columnSize, boolean nullable, boolean autoIncrement, - boolean hasDefault + boolean hasDefault, + boolean generated ) {} /** @@ -329,6 +345,8 @@ public static DatabaseSchema read( } // Discover columns for all tables. try (ResultSet columns = metadata.getColumns(catalog, schemaPattern, "%", "%")) { + // IS_GENERATEDCOLUMN is the last column JDBC specifies, and not every driver returns it. + boolean reportsGenerated = hasColumn(columns.getMetaData(), "IS_GENERATEDCOLUMN"); while (columns.next()) { String tableName = columns.getString("TABLE_NAME"); // Only include columns for tables we discovered. @@ -347,9 +365,10 @@ public static DatabaseSchema read( boolean nullable = !"NO".equalsIgnoreCase(nullableStr); String autoIncrementStr = columns.getString("IS_AUTOINCREMENT"); boolean autoIncrement = "YES".equalsIgnoreCase(autoIncrementStr); + boolean generated = reportsGenerated && "YES".equalsIgnoreCase(columns.getString("IS_GENERATEDCOLUMN")); columnsByTable.computeIfAbsent(tableName, k -> new ArrayList<>()) - .add(new DbColumn(tableName, columnName, dataType, typeName, columnSize, nullable, autoIncrement, hasDefault)); + .add(new DbColumn(tableName, columnName, dataType, typeName, columnSize, nullable, autoIncrement, hasDefault, generated)); } } // A database that names its schemas as catalogs has no schema pattern; its catalog is the schema the diff --git a/storm-core/src/main/java/st/orm/core/template/impl/MetamodelFactory.java b/storm-core/src/main/java/st/orm/core/template/impl/MetamodelFactory.java index cd91872d5..d6289328c 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/MetamodelFactory.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/MetamodelFactory.java @@ -18,6 +18,7 @@ import static java.lang.invoke.MethodType.methodType; import static st.orm.core.spi.Providers.getORMConverter; import static st.orm.core.template.impl.RecordReflection.findPkField; +import static st.orm.core.template.impl.RecordReflection.findQueryColumn; import static st.orm.core.template.impl.RecordReflection.getRecordField; import static st.orm.core.template.impl.RecordReflection.getRecordFields; import static st.orm.core.template.impl.RecordReflection.getRefDataType; @@ -41,6 +42,7 @@ import st.orm.Navigable; import st.orm.PK; import st.orm.PersistenceException; +import st.orm.QueryColumn; import st.orm.Ref; import st.orm.SqlTemplateException; import st.orm.UK; @@ -208,31 +210,38 @@ public static boolean isNullable(Metamodel metamodel) { // The root metamodel designates the row itself rather than a column. return false; } + Class fieldResolutionClass = fieldResolutionClass(metamodel.root()); + RecordField field; try { - RecordField field = getRecordField(fieldResolutionClass(metamodel.root()), path); - boolean nullsDistinct = getNullsDistinct(field); - boolean inline = !Ref.class.isAssignableFrom(field.type()) - && getORMConverter(field).isEmpty() - && isRecord(field.type()) - && !Data.class.isAssignableFrom(field.type()); - if (inline) { - // Mirrors SimpleKeyMetamodel: an inline record is nullable when its unique constraint treats NULLs - // as distinct and any of its constituent fields is nullable. - boolean fieldIsUnique = field.isAnnotationPresent(UK.class) || field.isAnnotationPresent(PK.class); - if (!fieldIsUnique || !nullsDistinct) { - return false; - } - for (var leaf : metamodel.flatten()) { - if (leaf instanceof Metamodel.Key leafKey && leafKey.isNullable()) { - return true; - } - } - return false; - } - return field.nullable() && nullsDistinct; + field = getRecordField(fieldResolutionClass, path); } catch (SqlTemplateException e) { + // No record field at the path: a query column has none, and its declaration states its nullability. + QueryColumn queryColumn = findQueryColumnAtPath(fieldResolutionClass, path); + if (queryColumn != null) { + return queryColumn.nullable(); + } throw new PersistenceException("Failed to resolve nullability for metamodel field at path '%s' on type %s.".formatted(path, metamodel.root().getName()), e); } + boolean nullsDistinct = getNullsDistinct(field); + boolean inline = !Ref.class.isAssignableFrom(field.type()) + && getORMConverter(field).isEmpty() + && isRecord(field.type()) + && !Data.class.isAssignableFrom(field.type()); + if (inline) { + // Mirrors SimpleKeyMetamodel: an inline record is nullable when its unique constraint treats NULLs + // as distinct and any of its constituent fields is nullable. + boolean fieldIsUnique = field.isAnnotationPresent(UK.class) || field.isAnnotationPresent(PK.class); + if (!fieldIsUnique || !nullsDistinct) { + return false; + } + for (var leaf : metamodel.flatten()) { + if (leaf instanceof Metamodel.Key leafKey && leafKey.isNullable()) { + return true; + } + } + return false; + } + return field.nullable() && nullsDistinct; } /** @@ -390,6 +399,10 @@ private static Metamodel getModel(Class rootTable, return (Metamodel) root(rootTable); } Class fieldResolutionClass = fieldResolutionClass(rootTable); + QueryColumn queryColumn = findQueryColumnAtPath(fieldResolutionClass, path); + if (queryColumn != null) { + return queryColumnModel(rootTable, fieldResolutionClass, path, queryColumn); + } Class fieldType; String effectivePath; StringBuilder effectiveField; @@ -532,6 +545,66 @@ && isPrimaryKeyName(getRefDataType(referenceField), effectiveField.toString())) ); } + /** + * Returns the query column a path ends in, or {@code null} when its last segment is not one. The table the path + * reaches declares it, as it declares its components; an inline record declares none. + */ + @SuppressWarnings("unchecked") + private static @Nullable QueryColumn findQueryColumnAtPath(Class fieldResolutionClass, String path) { + String parentPath = stripLast(path); + String fieldName = parentPath.isEmpty() ? path : path.substring(parentPath.length() + 1); + Class tableType; + if (parentPath.isEmpty()) { + tableType = fieldResolutionClass; + } else { + try { + RecordField parent = getRecordField(fieldResolutionClass, parentPath); + if (Ref.class.isAssignableFrom(parent.type())) { + tableType = getRefDataType(parent); + } else if (Data.class.isAssignableFrom(parent.type())) { + tableType = (Class) parent.type(); + } else { + return null; + } + } catch (SqlTemplateException e) { + return null; + } + } + return findQueryColumn(tableType, fieldName).orElse(null); + } + + /** + * Creates the metamodel of a query column reached by path, with the table it belongs to resolved as a + * component's table is. + */ + @SuppressWarnings({"unchecked", "rawtypes"}) + private static Metamodel queryColumnModel(Class rootTable, + Class fieldResolutionClass, + String path, + QueryColumn queryColumn) { + String tablePathToParent = stripLast(path); + Metamodel rootModel = root(rootTable); + Metamodel tableModel; + if (tablePathToParent.isEmpty()) { + tableModel = rootModel; + } else { + String tablePath = getTablePath(fieldResolutionClass, tablePathToParent); + String tableField = tablePath.isEmpty() ? tablePathToParent : tablePathToParent.substring(tablePath.length() + 1); + Class tableType = resolveDataTypeAtPath(fieldResolutionClass, tablePathToParent); + tableModel = new SimpleMetamodel<>( + rootTable, + tablePath, + (Class) tableType, + tableField, + false, + false, + rootModel, + buildGetterHandle(rootTable, tablePathToParent) + ); + } + return new QueryColumnMetamodel<>(rootTable, tablePathToParent, (Class) queryColumn.type(), queryColumn.field(), tableModel); + } + @Nullable @SuppressWarnings("unchecked") private static Metamodel lookupGeneratedMetamodel( @@ -872,6 +945,57 @@ public boolean isNullable() { } } + /** + * The metamodel of a query column resolved by name: a column the query names and the record does not carry, so + * a record holds no value for it. + */ + private static final class QueryColumnMetamodel + extends AbstractMetamodel { + + private final Class root; + private final Metamodel table; + + QueryColumnMetamodel(Class root, + String path, + Class fieldType, + String field, + Metamodel table) { + super(fieldType, path, field, false, null, true); + this.root = root; + this.table = table; + } + + @Override + public Class root() { + return root; + } + + @Override + public Metamodel table() { + return table; + } + + @Override + public Object getValue(T record) { + throw noValue(); + } + + @Override + public boolean isIdentical(T a, T b) { + throw noValue(); + } + + @Override + public boolean isSame(T a, T b) { + throw noValue(); + } + + private UnsupportedOperationException noValue() { + return new UnsupportedOperationException("%s.%s is a query column; a record holds no value for it." + .formatted(table.fieldType().getSimpleName(), field())); + } + } + private static final class SimpleMetamodel extends AbstractMetamodel { diff --git a/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java b/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java index ba0b4fc2c..69d928806 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/ModelFactory.java @@ -16,6 +16,9 @@ package st.orm.core.template.impl; import static java.util.Collections.nCopies; +import static java.util.Locale.ROOT; +import static java.util.stream.Collectors.toMap; +import static java.util.stream.Collectors.toSet; import static st.orm.core.spi.Providers.getORMConverter; import static st.orm.core.template.impl.RecordReflection.detectSealedPattern; import static st.orm.core.template.impl.RecordReflection.findJoinedSealedParent; @@ -27,6 +30,8 @@ import static st.orm.core.template.impl.RecordReflection.getFkLeaves; import static st.orm.core.template.impl.RecordReflection.getForeignKeys; import static st.orm.core.template.impl.RecordReflection.getGenerationStrategy; +import static st.orm.core.template.impl.RecordReflection.getQueryColumnField; +import static st.orm.core.template.impl.RecordReflection.getQueryColumns; import static st.orm.core.template.impl.RecordReflection.getRecordType; import static st.orm.core.template.impl.RecordReflection.getRefDataType; import static st.orm.core.template.impl.RecordReflection.getSequence; @@ -37,6 +42,7 @@ import static st.orm.core.template.impl.RecordValidation.validateDataType; import java.util.ArrayList; +import java.util.HashSet; import java.util.List; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; @@ -49,6 +55,7 @@ import st.orm.Metamodel; import st.orm.PK; import st.orm.Persist; +import st.orm.QueryColumn; import st.orm.Ref; import st.orm.SqlTemplateException; import st.orm.Version; @@ -125,11 +132,13 @@ private static Model createModel(ModelBuilder builde RecordType recordType = getRecordType(type); List columns = new ArrayList<>(); List fields = new ArrayList<>(); + List queryColumns = new ArrayList<>(); try { - BuildContext ctx = new BuildContext(builder, columns, fields, Metamodel.root(type), new AtomicInteger(1), fetchPlan); + BuildContext ctx = new BuildContext(builder, columns, fields, queryColumns, Metamodel.root(type), new AtomicInteger(1), fetchPlan); for (var field : recordType.fields()) { createColumns(ctx, ctx.rootMetamodel(), field, false, null, KeyScope.none(), PkContext.none(), null, false); } + createQueryColumns(ctx, ctx.rootMetamodel(), type, false); var tableName = getTableName(type, builder.tableNameResolver(), builder.schemaResolver()); // For permitted subclasses of a JOINED sealed entity, adjust columns so that // base non-PK fields are not insertable/updatable (they belong to the base table), @@ -141,7 +150,7 @@ private static Model createModel(ModelBuilder builde } // The first model of an entity is the moment it is first used, where no startup check saw it. DesignCheck.check(type); - return new ModelImpl<>(recordType, tableName, fields, finalColumns); + return new ModelImpl<>(recordType, tableName, fields, finalColumns, queryColumns); } catch (UncheckedSqlTemplateException e) { throw e.getCause(); } @@ -349,10 +358,15 @@ private static Model createSealedModel(ModelBuilder var tableName = getTableName(sealedType, builder.tableNameResolver(), builder.schemaResolver()); // Override the type to the sealed interface so that Model.type() returns the sealed interface // (e.g., Vehicle.class) rather than the first permitted subclass (e.g., Car.class). - return new ModelImpl<>(firstRecordType, sealedType, tableName, fields, columns); + return new ModelImpl<>(firstRecordType, sealedType, tableName, fields, columns, List.of()); } - record BuildContext(ModelBuilder builder, List columns, List fields, Metamodel rootMetamodel, AtomicInteger index, FetchPlan fetchPlan) { + /** + * The state of a model under construction. {@code columns} and {@code fields} are aligned, one record field per + * column; {@code queryColumns} holds the columns the query side knows and the record does not carry, which have + * no field. + */ + record BuildContext(ModelBuilder builder, List columns, List fields, List queryColumns, Metamodel rootMetamodel, AtomicInteger index, FetchPlan fetchPlan) { } record ColumnSpec(boolean primaryKey, @@ -510,6 +524,80 @@ private static void expandForeignRelation(BuildContext ctx, for (var child : targetRecordType.fields()) { createColumns(ctx, foreignMetamodel, child, nullableDueToJoin, null, KeyScope.none(), PkContext.none(), null, true); } + createQueryColumns(ctx, foreignMetamodel, targetType, nullableDueToJoin); + } + + /** + * Adds the query columns the type declares, under the parent's path, so that a path through a relation reaches + * them as it reaches the components. A query column is never selected, inserted or updated, so it takes no + * position among the model's columns and no record field. + */ + private static void createQueryColumns(BuildContext ctx, + Metamodel parentMetamodel, + Class type, + boolean parentNullable) { + List queryColumns = getQueryColumns(type); + if (queryColumns.isEmpty()) { + return; + } + try { + var componentNames = getRecordType(type).fields().stream().map(RecordField::name).collect(toSet()); + // The columns the fields of this table map, as emitted: a query column cannot name one of them. + var tablePath = parentMetamodel.fieldPath(); + var fieldColumns = ctx.columns().stream() + .filter(column -> column.metamodel().table().fieldPath().equals(tablePath)) + .collect(toMap(column -> column.name().toLowerCase(ROOT), column -> column.metamodel().field(), (first, second) -> first)); + var declared = new HashSet(); + var declaredColumns = new HashSet(); + for (QueryColumn queryColumn : queryColumns) { + String fieldName = queryColumn.field(); + if (fieldName.isEmpty()) { + throw new SqlTemplateException("A query column of %s names no field.".formatted(type.getSimpleName())); + } + if (componentNames.contains(fieldName)) { + throw new SqlTemplateException("Query column '%s' of %s has the name of a record component; a query column is a column the record does not carry.".formatted(fieldName, type.getSimpleName())); + } + if (!declared.add(fieldName)) { + throw new SqlTemplateException("Query column '%s' of %s is declared twice.".formatted(fieldName, type.getSimpleName())); + } + RecordField field = getQueryColumnField(type, queryColumn); + @SuppressWarnings("unchecked") + Metamodel metamodel = (Metamodel) Metamodel.of( + (Class) parentMetamodel.root(), + parentMetamodel.fieldPath().isEmpty() ? fieldName : parentMetamodel.fieldPath() + "." + fieldName); + ColumnName columnName = queryColumn.column().isEmpty() + ? new ColumnName(ctx.builder().columnNameResolver().resolveColumnName(field)) + : new ColumnName(queryColumn.column()); + String mappedBy = fieldColumns.get(columnName.name().toLowerCase(ROOT)); + if (mappedBy != null) { + throw new SqlTemplateException("Query column '%s' of %s names column '%s', which field '%s' maps; a query column is a column the record does not carry.".formatted(fieldName, type.getSimpleName(), columnName.name(), mappedBy)); + } + if (!declaredColumns.add(columnName.name().toLowerCase(ROOT))) { + throw new SqlTemplateException("Query column '%s' of %s names column '%s', which another query column of %s names as well.".formatted(fieldName, type.getSimpleName(), columnName.name(), type.getSimpleName())); + } + ctx.queryColumns().add(new ColumnImpl( + columnName, + 0, + queryColumn.type(), + queryColumn.type(), + false, + GenerationStrategy.NONE, + "", + false, + GenerationStrategy.NONE, + -1, + parentNullable || queryColumn.nullable(), + false, + false, + false, + false, + metamodel, + null + )); + } + } catch (SqlTemplateException e) { + throw new UncheckedSqlTemplateException(e); + } } private static ColumnSpec buildSpec(RecordField field, diff --git a/storm-core/src/main/java/st/orm/core/template/impl/ModelImpl.java b/storm-core/src/main/java/st/orm/core/template/impl/ModelImpl.java index 2c4ea2038..892473348 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/ModelImpl.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/ModelImpl.java @@ -113,6 +113,7 @@ protected Map computeValue(Class type) { private final List columns; private final RecordField primaryKeyField; private final List declaredColumns; + private final List queryColumns; private final Map, List>> mappedMetamodels; private final Metamodel primaryKeyMetamodel; private final Map, List> columnMap; @@ -162,8 +163,9 @@ protected Map computeValue(Class type) { public ModelImpl(RecordType recordType, TableName tableName, List fields, - List columns) throws SqlTemplateException { - this(recordType, null, tableName, fields, columns); + List columns, + List queryColumns) throws SqlTemplateException { + this(recordType, null, tableName, fields, columns, queryColumns); } /** @@ -178,19 +180,24 @@ public ModelImpl(RecordType recordType, * @param tableName the table name. * @param fields the record fields aligned with columns. * @param columns the column metadata. + * @param queryColumns the columns a query names but the record does not carry, those of the type and those + * reached through its expanded relations; they are resolved by metamodel and take no position + * among {@code columns}. */ public ModelImpl(RecordType recordType, @Nullable Class typeOverride, TableName tableName, List fields, - List columns) throws SqlTemplateException { + List columns, + List queryColumns) throws SqlTemplateException { assert fields.size() == columns.size() : "Columns and fields must have the same size"; this.recordType = requireNonNull(recordType, "recordType"); this.typeOverride = typeOverride; this.tableName = requireNonNull(tableName, "tableName"); this.fields = copyOf(fields); this.columns = copyOf(columns); - this.columnMap = initColumnMap(this.columns); + this.queryColumns = initDeclaredColumns(queryColumns); + this.columnMap = initColumnMap(this.columns, queryColumns); this.discriminatorColumnIndex = initDiscriminatorColumnIndex(typeOverride); this.polymorphicFkDiscriminatorColumns = initPolymorphicFkDiscriminatorColumns(this.fields, this.columns); this.converters = initConverters(this.fields, this.columns); @@ -202,7 +209,7 @@ public ModelImpl(RecordType recordType, this.mappedMetamodels = initMappedMetamodels(this.fields, this.columns); } - private static Map, List> initColumnMap(List columns) { + private static Map, List> initColumnMap(List columns, List queryColumns) { var map = new HashMap, List>(); for (var column : columns) { map.computeIfAbsent(column.metamodel().canonical(), ignore -> new ArrayList<>()).add(column); @@ -210,6 +217,9 @@ public ModelImpl(RecordType recordType, map.computeIfAbsent(column.secondaryMetamodel().canonical(), ignore -> new ArrayList<>()).add(column); } } + for (var column : queryColumns) { + map.computeIfAbsent(column.metamodel().canonical(), ignore -> new ArrayList<>()).add(column); + } map.replaceAll((k, v) -> List.copyOf(v)); return Map.copyOf(map); } @@ -850,4 +860,9 @@ public List columns() { public List declaredColumns() { return declaredColumns; } + + @Override + public List queryColumns() { + return queryColumns; + } } diff --git a/storm-core/src/main/java/st/orm/core/template/impl/RecordReflection.java b/storm-core/src/main/java/st/orm/core/template/impl/RecordReflection.java index dbb245835..c28af84ee 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/RecordReflection.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/RecordReflection.java @@ -51,6 +51,7 @@ import st.orm.PersistenceException; import st.orm.Polymorphic; import st.orm.ProjectionQuery; +import st.orm.QueryColumn; import st.orm.Ref; import st.orm.SqlTemplateException; import st.orm.Version; @@ -826,6 +827,59 @@ private static Stream> columnNames(DbColumn[] dbColumns) { private static final List> COLUMN_ANNOTATIONS = List.of(PK.class, FK.class, DbColumn.class); + /** + * Returns the query columns a type declares, in declaration order: the columns a query names but the type does + * not carry. A repeated {@code @QueryColumn} and the {@code @QueryColumns} container read alike. + * + * @param type the entity or projection type. + * @return the query columns, empty for a type that declares none. + */ + static List getQueryColumns(Class type) { + return QUERY_COLUMN_CACHE.get(type); + } + + /** + * Query columns per declaring type, in declaration order. Reading a repeatable annotation assembles the list on + * every call, and a query's nullability check reads it per sort field; {@link ClassValue} ties each entry to the + * lifetime of the declaring class, so the cache never pins the class or its class loader. + */ + private static final ClassValue> QUERY_COLUMN_CACHE = new ClassValue<>() { + @Override + protected List computeValue(Class type) { + return List.of(type.getAnnotationsByType(QueryColumn.class)); + } + }; + + /** + * Returns the query column of the type with the given field name, if the type declares one. + * + * @param type the entity or projection type. + * @param field the metamodel field name. + * @return the query column, or empty when the type declares none by that name. + */ + static Optional findQueryColumn(Class type, String field) { + for (QueryColumn queryColumn : getQueryColumns(type)) { + if (queryColumn.field().equals(field)) { + return Optional.of(queryColumn); + } + } + return Optional.empty(); + } + + /** + * Returns a record field standing in for a query column, which has no component. It carries the field name, the + * declared type and nullability, and the declaration as its only annotation, which is what a column name resolver + * reads. It has no accessor, since a record holds no value for the column. + * + * @param declaringType the type that declares the query column. + * @param queryColumn the declaration. + * @return the record field. + */ + static RecordField getQueryColumnField(Class declaringType, QueryColumn queryColumn) { + return new RecordField(declaringType, queryColumn.field(), queryColumn.type(), queryColumn.type(), + queryColumn.nullable(), false, null, List.of(queryColumn)); + } + /** * Returns the column name for the specified record field taking the column name resolver into account, * if present. diff --git a/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidationError.java b/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidationError.java index afbeaed44..075302d57 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidationError.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidationError.java @@ -56,7 +56,10 @@ public enum ErrorKind { FOREIGN_KEY_MISMATCH, /** A {@code @FK} field does not have a matching foreign key constraint in the database. */ FOREIGN_KEY_MISSING(true), - /** A database column is NOT NULL without a default value, but is not mapped in the entity. @since 1.10 */ + /** + * A database column is NOT NULL without a default value, but is not mapped in the entity: no field maps it, + * or a query column names it, and the database does not generate it. @since 1.10 + */ UNMAPPED_COLUMN(true); private final boolean warning; diff --git a/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidator.java b/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidator.java index 4515e8acc..39c76c35b 100644 --- a/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidator.java +++ b/storm-core/src/main/java/st/orm/core/template/impl/SchemaValidator.java @@ -529,42 +529,16 @@ private void validateType( // Track all column names (including @DbIgnore) for the unmapped column check (step 9). Set mappedColumns = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); for (Column column : model.declaredColumns()) { - String columnName = column.name(); - mappedColumns.add(columnName); + mappedColumns.add(column.name()); if (isIgnored(column, ignoredComponents)) { continue; } - Optional dbColumn = schema.getColumn(tableName, columnName); - - if (dbColumn.isEmpty()) { - errors.add(new SchemaValidationError(type, ErrorKind.COLUMN_NOT_FOUND, - "Column '%s' not found in table '%s'.".formatted(columnName, qualifiedTableName))); - continue; - } - DbColumn dbCol = dbColumn.get(); - // Type compatibility check against the persisted type, so foreign key columns are validated - // against the referenced key's terminal type rather than being skipped as unknown entity types. - Compatibility compatibility = typeCompatibility.check(column.persistedType(), dbCol.dataType(), dbCol.typeName()); - if (compatibility == Compatibility.NARROWING) { - errors.add(new SchemaValidationError(type, ErrorKind.TYPE_NARROWING, - "Column '%s' in table '%s': Java type '%s' mapped to SQL type '%s' (%d) may involve precision or range loss." - .formatted(columnName, qualifiedTableName, column.persistedType().getSimpleName(), - dbCol.typeName(), dbCol.dataType()))); - } else if (compatibility == Compatibility.INCOMPATIBLE) { - errors.add(new SchemaValidationError(type, ErrorKind.TYPE_INCOMPATIBLE, - "Column '%s' in table '%s': Java type '%s' is not compatible with SQL type '%s' (%d)." - .formatted(columnName, qualifiedTableName, column.persistedType().getSimpleName(), - dbCol.typeName(), dbCol.dataType()))); - } - // Nullability check: entity field is non-nullable but database column allows NULL. - // We only flag the case where the entity says non-null but the DB says nullable, - // since the reverse (entity nullable, DB not null) is safe. A view's columns read as nullable - // whatever its query returns, so they carry no answer to check against. - if (!column.nullable() && dbCol.nullable() && !schema.isView(tableName)) { - errors.add(new SchemaValidationError(type, ErrorKind.NULLABILITY_MISMATCH, - "Column '%s' in table '%s': entity field is non-nullable but database column allows NULL." - .formatted(columnName, qualifiedTableName))); - } + validateColumn(type, column, schema, tableName, qualifiedTableName, errors); + } + // A query column is checked as a declared column is. It stays out of the mapped set: every insert leaves it + // out, so the unmapped column check (step 9) covers it as it covers a column no field maps. + for (Column column : model.queryColumns()) { + validateColumn(type, column, schema, tableName, qualifiedTableName, errors); } // 5. Primary key match. Skipped when this table's primary key could not be read: an empty result would // otherwise read as "the table has no primary key". @@ -830,8 +804,8 @@ private void validateUnmappedColumns( if (mappedColumns.contains(dbColumn.columnName())) { continue; // Column is mapped in the entity. } - // Skip columns that are safe to omit: nullable, has a default, or auto-incremented. - if (dbColumn.nullable() || dbColumn.hasDefault() || dbColumn.autoIncrement()) { + // Skip columns that are safe to omit: nullable, has a default, auto-incremented, or generated. + if (dbColumn.nullable() || dbColumn.hasDefault() || dbColumn.autoIncrement() || dbColumn.generated()) { continue; } errors.add(new SchemaValidationError(type, ErrorKind.UNMAPPED_COLUMN, @@ -840,6 +814,52 @@ private void validateUnmappedColumns( } } + /** + * Checks that a column exists in the table with a compatible type and nullability, and returns the database + * column, or {@code null} when the table has none by that name. + */ + private @Nullable DbColumn validateColumn( + Class type, + Column column, + DatabaseSchema schema, + String tableName, + String qualifiedTableName, + List errors + ) { + String columnName = column.name(); + Optional dbColumn = schema.getColumn(tableName, columnName); + if (dbColumn.isEmpty()) { + errors.add(new SchemaValidationError(type, ErrorKind.COLUMN_NOT_FOUND, + "Column '%s' not found in table '%s'.".formatted(columnName, qualifiedTableName))); + return null; + } + DbColumn dbCol = dbColumn.get(); + // Type compatibility check against the persisted type, so foreign key columns are validated + // against the referenced key's terminal type rather than being skipped as unknown entity types. + Compatibility compatibility = typeCompatibility.check(column.persistedType(), dbCol.dataType(), dbCol.typeName()); + if (compatibility == Compatibility.NARROWING) { + errors.add(new SchemaValidationError(type, ErrorKind.TYPE_NARROWING, + "Column '%s' in table '%s': Java type '%s' mapped to SQL type '%s' (%d) may involve precision or range loss." + .formatted(columnName, qualifiedTableName, column.persistedType().getSimpleName(), + dbCol.typeName(), dbCol.dataType()))); + } else if (compatibility == Compatibility.INCOMPATIBLE) { + errors.add(new SchemaValidationError(type, ErrorKind.TYPE_INCOMPATIBLE, + "Column '%s' in table '%s': Java type '%s' is not compatible with SQL type '%s' (%d)." + .formatted(columnName, qualifiedTableName, column.persistedType().getSimpleName(), + dbCol.typeName(), dbCol.dataType()))); + } + // Nullability check: entity field is non-nullable but database column allows NULL. + // We only flag the case where the entity says non-null but the DB says nullable, + // since the reverse (entity nullable, DB not null) is safe. A view's columns read as nullable + // whatever its query returns, so they carry no answer to check against. + if (!column.nullable() && dbCol.nullable() && !schema.isView(tableName)) { + errors.add(new SchemaValidationError(type, ErrorKind.NULLABILITY_MISMATCH, + "Column '%s' in table '%s': entity field is non-nullable but database column allows NULL." + .formatted(columnName, qualifiedTableName))); + } + return dbCol; + } + /** * Returns the names of fields annotated with {@link DbIgnore}. */ diff --git a/storm-core/src/test/java/st/orm/core/QueryColumnIntegrationTest.java b/storm-core/src/test/java/st/orm/core/QueryColumnIntegrationTest.java new file mode 100644 index 000000000..c0d0c147e --- /dev/null +++ b/storm-core/src/test/java/st/orm/core/QueryColumnIntegrationTest.java @@ -0,0 +1,207 @@ +/* + * Copyright 2024 - 2026 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package st.orm.core; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static st.orm.Operator.EQUALS; +import static st.orm.core.template.SqlInterceptor.observe; +import static st.orm.core.template.TemplateString.raw; + +import java.util.ArrayList; +import java.util.List; +import javax.sql.DataSource; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.autoconfigure.jdbc.JdbcTest; +import org.springframework.test.context.ContextConfiguration; +import org.springframework.test.context.junit.jupiter.SpringExtension; +import st.orm.Entity; +import st.orm.Metamodel; +import st.orm.PK; +import st.orm.PersistenceException; +import st.orm.QueryColumn; +import st.orm.Scrollable; +import st.orm.core.model.Owner; +import st.orm.core.model.Owner_; +import st.orm.core.model.Pet; +import st.orm.core.model.Pet_; +import st.orm.core.template.ORMTemplate; +import st.orm.core.template.impl.PredicateBuilderFactory; + +/** + * A query column is a column a query names but the record does not carry: {@code owner.last_name_folded} is generated + * by the database from the last name, and {@code Owner} declares it as {@code Owner_.lastNameFolded} with no + * component. + */ +@ExtendWith(SpringExtension.class) +@ContextConfiguration(classes = IntegrationConfig.class) +@JdbcTest +public class QueryColumnIntegrationTest { + + @Autowired + private DataSource dataSource; + + private final List statements = new ArrayList<>(); + + private ORMTemplate orm() { + return ORMTemplate.of(dataSource); + } + + private T observed(java.util.function.Supplier action) { + statements.clear(); + return observe(sql -> statements.add(sql.statement().replaceAll("\\s+", " ")), action); + } + + private List firstNames(List owners) { + return owners.stream().map(Owner::firstName).sorted().toList(); + } + + @Test + public void queryColumnIsNeverSelected() { + List owners = observed(() -> orm().entity(Owner.class).select().getResultList()); + assertEquals(10, owners.size()); + assertFalse(statements.getFirst().contains("last_name_folded"), statements.getFirst()); + } + + @Test + public void whereMatchesTheQueryColumn() { + List owners = observed(() -> orm().entity(Owner.class).select() + .where(Owner_.lastNameFolded, EQUALS, "davis") + .getResultList()); + assertEquals(List.of("Betty", "Harold"), firstNames(owners)); + assertTrue(statements.getFirst().contains("last_name_folded = ?"), statements.getFirst()); + } + + @Test + public void searchMatchesTheQueryColumn() { + List owners = orm().entity(Owner.class).select() + .where(it -> PredicateBuilderFactory.search("dav", List.of(Owner_.lastNameFolded))) + .getResultList(); + assertEquals(List.of("Betty", "Harold"), firstNames(owners)); + } + + @Test + public void pathReachesTheQueryColumnOfARelatedTable() { + List byFolded = observed(() -> orm().entity(Pet.class).select() + .where(Pet_.owner.lastNameFolded, EQUALS, "davis") + .getResultList()); + List byName = orm().entity(Pet.class).select() + .where(Pet_.owner.lastName, EQUALS, "Davis") + .getResultList(); + assertEquals(List.of("Iggy", "Leo"), byFolded.stream().map(Pet::name).sorted().toList()); + assertEquals(byName.stream().map(Pet::id).toList(), byFolded.stream().map(Pet::id).toList()); + assertFalse(statements.getFirst().contains("last_name_folded,"), statements.getFirst()); + } + + @Test + public void orderByAndKeysetSortByTheQueryColumn() { + List ordered = orm().entity(Owner.class).select() + .orderBy(Owner_.lastNameFolded) + .getResultList(); + assertEquals("Black", ordered.getFirst().lastName()); + var window = orm().entity(Owner.class).select() + .scroll(Scrollable.of(Owner_.id, 3).sortBy(Owner_.lastNameFolded)); + assertEquals(List.of("Black", "Coleman", "Davis"), window.content().stream().map(Owner::lastName).toList()); + } + + @Test + public void templatePlaceholderNamesTheQueryColumn() { + List folded = orm().query(raw("SELECT \0 FROM \0 WHERE \0 = \0", Owner_.lastNameFolded, Owner.class, Owner_.lastName, "Davis")) + .getResultList(String.class); + assertEquals(List.of("davis", "davis"), folded); + } + + @Test + public void recordHoldsNoValueForTheQueryColumn() { + Owner owner = orm().entity(Owner.class).select().where(Owner_.id, EQUALS, 1).getSingleResult(); + var exception = assertThrows(UnsupportedOperationException.class, () -> Owner_.lastNameFolded.getValue(owner)); + assertTrue(exception.getMessage().contains("query column"), exception.getMessage()); + } + + @Test + public void updateCannotSetTheQueryColumn() { + var exception = assertThrows(PersistenceException.class, + () -> orm().entity(Owner.class).update().set(Owner_.lastNameFolded, "x")); + assertTrue(exception.getMessage().contains("last_name_folded"), exception.getMessage()); + } + + @Test + public void insertLeavesTheQueryColumnOut() { + Owner inserted = observed(() -> orm().entity(Owner.class).insertAndFetch( + Owner.builder().firstName("Ada").lastName("Lovelace").address( + orm().entity(Owner.class).select().where(Owner_.id, EQUALS, 1).getSingleResult().address()).version(0).build())); + assertFalse(statements.getFirst().contains("last_name_folded"), statements.getFirst()); + assertEquals(List.of("Ada"), firstNames(orm().entity(Owner.class).select() + .where(Owner_.lastNameFolded, EQUALS, "lovelace").getResultList())); + assertEquals("Lovelace", inserted.lastName()); + } + + @Test + public void modelListsTheQueryColumnApart() throws Exception { + var model = orm().entity(Owner.class).model(); + assertEquals(1, model.queryColumns().size()); + var column = model.queryColumns().getFirst(); + assertEquals("last_name_folded", column.name()); + assertEquals(0, column.index()); + assertEquals(String.class, column.type()); + assertFalse(column.insertable()); + assertFalse(column.updatable()); + assertFalse(column.nullable()); + assertTrue(model.columns().stream().noneMatch(it -> it.name().equals("last_name_folded"))); + assertEquals(List.of(column), model.getColumns(Owner_.lastNameFolded)); + } + + @Test + public void queryColumnResolvesByName() { + assertEquals(Owner_.lastNameFolded, Metamodel.of(Owner.class, "lastNameFolded")); + Metamodel throughOwner = Metamodel.of(Pet.class, "owner.lastNameFolded"); + assertEquals("owner", throughOwner.path()); + assertEquals("lastNameFolded", throughOwner.field()); + assertEquals(String.class, throughOwner.fieldType()); + assertTrue(throughOwner.isColumn()); + assertEquals(Pet_.owner.lastNameFolded, throughOwner); + } + + @QueryColumn(field = "name", type = String.class) + public record ComponentNamedQueryColumn(@PK Integer id, String name) implements Entity {} + + @Test + public void queryColumnNamedAsAComponentIsRefused() { + var exception = assertThrows(PersistenceException.class, () -> orm().entity(ComponentNamedQueryColumn.class).model()); + var messages = new ArrayList(); + for (Throwable cause = exception; cause != null; cause = cause.getCause()) { + messages.add(String.valueOf(cause.getMessage())); + } + assertTrue(messages.stream().anyMatch(message -> message.contains("record component")), messages.toString()); + } + + @QueryColumn(field = "nameCopy", type = String.class, column = "name") + public record ColumnNamedQueryColumn(@PK Integer id, String name) implements Entity {} + + @Test + public void queryColumnNamingAMappedColumnIsRefused() { + var exception = assertThrows(PersistenceException.class, () -> orm().entity(ColumnNamedQueryColumn.class).model()); + var messages = new ArrayList(); + for (Throwable cause = exception; cause != null; cause = cause.getCause()) { + messages.add(String.valueOf(cause.getMessage())); + } + assertTrue(messages.stream().anyMatch(message -> message.contains("which field 'name' maps")), messages.toString()); + } +} diff --git a/storm-core/src/test/java/st/orm/core/model/Owner.java b/storm-core/src/test/java/st/orm/core/model/Owner.java index de36a2b58..0377b8662 100644 --- a/storm-core/src/test/java/st/orm/core/model/Owner.java +++ b/storm-core/src/test/java/st/orm/core/model/Owner.java @@ -19,6 +19,7 @@ import org.jspecify.annotations.Nullable; import st.orm.Entity; import st.orm.PK; +import st.orm.QueryColumn; import st.orm.Version; /** @@ -26,6 +27,7 @@ * */ @Builder(toBuilder = true) +@QueryColumn(field = "lastNameFolded", type = String.class) public record Owner( @PK Integer id, String firstName, diff --git a/storm-core/src/test/java/st/orm/core/template/SchemaValidatorTest.java b/storm-core/src/test/java/st/orm/core/template/SchemaValidatorTest.java index c8e4e7e44..665f082a9 100644 --- a/storm-core/src/test/java/st/orm/core/template/SchemaValidatorTest.java +++ b/storm-core/src/test/java/st/orm/core/template/SchemaValidatorTest.java @@ -43,7 +43,9 @@ import st.orm.GenerationStrategy; import st.orm.PK; import st.orm.Persist; +import st.orm.Projection; import st.orm.ProjectionQuery; +import st.orm.QueryColumn; import st.orm.Ref; import st.orm.UK; import st.orm.core.QuietValidationLog; @@ -240,6 +242,23 @@ public record ForeignKeyIgnoredEntity( @DbIgnore @Nullable @FK Ref foreignKeyTarget ) implements Entity {} + @QueryColumn(field = "nameFolded", type = String.class) + public record QueryColumnEntity(@PK Integer id, String name) implements Entity {} + + @QueryColumn(field = "nameFolded", type = String.class, nullable = true) + public record NullableQueryColumnEntity(@PK Integer id, String name) implements Entity {} + + @QueryColumn(field = "nameFolded", type = Integer.class) + public record MistypedQueryColumnEntity(@PK Integer id, String name) implements Entity {} + + @QueryColumn(field = "nameFolded", type = String.class, column = "folded") + public record NamedQueryColumnEntity(@PK Integer id, String name) implements Entity {} + + @QueryColumn(field = "nameFolded", type = String.class) + public record QueryColumnProjection(@PK Integer id, String name) implements Projection {} + + public record UnmappedGeneratedEntity(@PK Integer id, String name) implements Entity {} + @BeforeEach void setUp() { // Each test gets a fresh in-memory database to avoid table name collisions. @@ -790,6 +809,102 @@ void testUniqueKeyMissingSuppressedByConstraintFalse() throws SQLException { "Expected no UNIQUE_KEY_MISSING with @UK(constraint = false), but got: " + errors); } + // Query columns + + @Test + void testQueryColumnOfAGeneratedColumnPasses() throws SQLException { + execute("CREATE TABLE query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255) GENERATED ALWAYS AS (LOWER(name)) NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(QueryColumnEntity.class)); + + assertTrue(errors.isEmpty(), "Expected no errors but got: " + errors); + } + + @Test + void testQueryColumnWithADefaultPasses() throws SQLException { + execute("CREATE TABLE query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255) DEFAULT '' NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(QueryColumnEntity.class)); + + assertTrue(errors.isEmpty(), "Expected no errors but got: " + errors); + } + + @Test + void testQueryColumnNotFound() throws SQLException { + execute("CREATE TABLE query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(QueryColumnEntity.class)); + + assertEquals(1, errors.size(), errors.toString()); + assertEquals(ErrorKind.COLUMN_NOT_FOUND, errors.getFirst().kind()); + assertTrue(errors.getFirst().message().contains("name_folded"), errors.getFirst().message()); + } + + @Test + void testQueryColumnTheDatabaseDoesNotFillIsUnmapped() throws SQLException { + execute("CREATE TABLE query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255) NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(QueryColumnEntity.class)); + + // Every insert leaves the query column out, so the column is unmapped for the insert, as a column no field + // maps is, and reported the same way. + assertEquals(1, errors.size(), errors.toString()); + assertEquals(ErrorKind.UNMAPPED_COLUMN, errors.getFirst().kind()); + assertTrue(errors.getFirst().kind().warning()); + assertTrue(errors.getFirst().message().toLowerCase().contains("name_folded"), errors.getFirst().message()); + } + + @Test + void testQueryColumnOfAProjectionIsNotReportedAsUnmapped() throws SQLException { + execute("CREATE TABLE query_column_projection (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255) NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(QueryColumnProjection.class)); + + assertTrue(errors.isEmpty(), "Expected no errors but got: " + errors); + } + + @Test + void testQueryColumnNullability() throws SQLException { + execute("CREATE TABLE nullable_query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255), PRIMARY KEY (id))"); + execute("CREATE TABLE query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255), PRIMARY KEY (id))"); + + List nullableErrors = SchemaValidator.of(dataSource).validate(List.of(NullableQueryColumnEntity.class)); + List errors = SchemaValidator.of(dataSource).validate(List.of(QueryColumnEntity.class)); + + assertTrue(nullableErrors.isEmpty(), "Expected no errors but got: " + nullableErrors); + // A nullable column is filled with NULL; the declaration's nullability is checked as a field's is. + assertEquals(1, errors.size(), errors.toString()); + assertEquals(ErrorKind.NULLABILITY_MISMATCH, errors.getFirst().kind()); + } + + @Test + void testQueryColumnTypeIncompatible() throws SQLException { + execute("CREATE TABLE mistyped_query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255) GENERATED ALWAYS AS (LOWER(name)) NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(MistypedQueryColumnEntity.class)); + + assertEquals(1, errors.size(), errors.toString()); + assertEquals(ErrorKind.TYPE_INCOMPATIBLE, errors.getFirst().kind()); + } + + @Test + void testQueryColumnNamesItsColumn() throws SQLException { + execute("CREATE TABLE named_query_column_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, folded VARCHAR(255) GENERATED ALWAYS AS (LOWER(name)) NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(NamedQueryColumnEntity.class)); + + assertTrue(errors.isEmpty(), "Expected no errors but got: " + errors); + } + + @Test + void testUnmappedGeneratedColumnIsNotReported() throws SQLException { + execute("CREATE TABLE unmapped_generated_entity (id INTEGER AUTO_INCREMENT, name VARCHAR(255) NOT NULL, name_folded VARCHAR(255) GENERATED ALWAYS AS (LOWER(name)) NOT NULL, PRIMARY KEY (id))"); + + List errors = SchemaValidator.of(dataSource).validate(List.of(UnmappedGeneratedEntity.class)); + + assertTrue(errors.isEmpty(), "Expected no errors but got: " + errors); + } + // Helpers private void execute(String sql) throws SQLException { diff --git a/storm-core/src/test/java/st/orm/core/template/impl/DatabaseSchemaTest.java b/storm-core/src/test/java/st/orm/core/template/impl/DatabaseSchemaTest.java index d7e18fb3d..0aca9c13f 100644 --- a/storm-core/src/test/java/st/orm/core/template/impl/DatabaseSchemaTest.java +++ b/storm-core/src/test/java/st/orm/core/template/impl/DatabaseSchemaTest.java @@ -8,6 +8,8 @@ import java.sql.Connection; import java.sql.DatabaseMetaData; import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; import java.sql.SQLException; import java.util.List; import java.util.concurrent.atomic.AtomicInteger; @@ -178,6 +180,69 @@ void testGetColumn() throws SQLException { } } + @Test + void testGeneratedColumnIsReported() throws SQLException { + execute("CREATE TABLE generated_test (id INTEGER AUTO_INCREMENT, name VARCHAR(100) NOT NULL, name_folded VARCHAR(100) GENERATED ALWAYS AS (LOWER(name)) NOT NULL, PRIMARY KEY (id))"); + try (Connection connection = getConnection()) { + DatabaseSchema schema = DatabaseSchema.read(connection); + + DbColumn foldedColumn = schema.getColumn("generated_test", "name_folded").orElseThrow(); + assertTrue(foldedColumn.generated()); + assertFalse(foldedColumn.nullable()); + + DbColumn nameColumn = schema.getColumn("generated_test", "name").orElseThrow(); + assertFalse(nameColumn.generated()); + } + } + + /** + * Wraps the connection so the column metadata carries no {@code IS_GENERATEDCOLUMN} column, the way a driver + * that predates it reports columns. + */ + private Connection withoutGeneratedColumnMetadata() throws SQLException { + Connection delegate = getConnection(); + DatabaseMetaData realMetaData = delegate.getMetaData(); + DatabaseMetaData metaData = (DatabaseMetaData) Proxy.newProxyInstance( + DatabaseMetaData.class.getClassLoader(), new Class[]{DatabaseMetaData.class}, + (proxy, method, args) -> { + Object result = invoke(method, realMetaData, args); + if (!method.getName().equals("getColumns")) { + return result; + } + ResultSet columns = (ResultSet) result; + ResultSetMetaData realColumnsMetaData = columns.getMetaData(); + ResultSetMetaData columnsMetaData = (ResultSetMetaData) Proxy.newProxyInstance( + ResultSetMetaData.class.getClassLoader(), new Class[]{ResultSetMetaData.class}, + (p, m, a) -> { + Object value = invoke(m, realColumnsMetaData, a); + return m.getName().equals("getColumnLabel") && "IS_GENERATEDCOLUMN".equalsIgnoreCase(String.valueOf(value)) + ? "UNREPORTED" + : value; + }); + return Proxy.newProxyInstance( + ResultSet.class.getClassLoader(), new Class[]{ResultSet.class}, + (p, m, a) -> m.getName().equals("getMetaData") ? columnsMetaData : invoke(m, columns, a)); + }); + return (Connection) Proxy.newProxyInstance( + Connection.class.getClassLoader(), new Class[]{Connection.class}, + (proxy, method, args) -> method.getName().equals("getMetaData") + ? metaData + : invoke(method, delegate, args)); + } + + @Test + void testAGeneratedColumnIsNotReportedWithoutTheMetadataColumn() throws SQLException { + execute("CREATE TABLE generated_test (id INTEGER AUTO_INCREMENT, name VARCHAR(100) NOT NULL, name_folded VARCHAR(100) GENERATED ALWAYS AS (LOWER(name)) NOT NULL, PRIMARY KEY (id))"); + try (Connection connection = withoutGeneratedColumnMetadata()) { + DatabaseSchema schema = DatabaseSchema.read(connection); + + // The driver says nothing about generation, so the column reads as an ordinary one. + DbColumn foldedColumn = schema.getColumn("generated_test", "name_folded").orElseThrow(); + assertFalse(foldedColumn.generated()); + assertFalse(foldedColumn.nullable()); + } + } + @Test void testGetColumnProperties() throws SQLException { execute("CREATE TABLE prop_test (id INTEGER AUTO_INCREMENT, name VARCHAR(100) NOT NULL, description VARCHAR(500), PRIMARY KEY (id))"); diff --git a/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java b/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java index 51bf002e2..ce0bf1ff7 100644 --- a/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java +++ b/storm-core/src/test/java/st/orm/core/template/impl/MetamodelFactoryTest.java @@ -4,6 +4,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import java.util.List; @@ -13,6 +14,8 @@ import st.orm.Inline; import st.orm.Metamodel; import st.orm.PK; +import st.orm.PersistenceException; +import st.orm.QueryColumn; import st.orm.Ref; /** @@ -30,6 +33,12 @@ public record SimpleEntity( public record InlineAddress(String street, String zipCode) {} + @QueryColumn(field = "nameFolded", type = String.class) + public record EntityWithQueryColumn( + @PK Integer id, + String name + ) implements Entity {} + public record EntityWithInline( @PK Integer id, @Inline InlineAddress address @@ -316,4 +325,38 @@ void testOfIsSameForFkFieldDifferentPk() { EntityWithFk entityB = new EntityWithFk(2, new ReferencedEntity(20, "a")); assertFalse(model.isSame(entityA, entityB)); } + + // Query columns + + @Test + void testQueryColumnResolvesWithoutAGeneratedMetamodel() { + Metamodel metamodel = MetamodelFactory.of(EntityWithQueryColumn.class, "nameFolded"); + + assertEquals(EntityWithQueryColumn.class, metamodel.root()); + assertEquals(EntityWithQueryColumn.class, metamodel.table().fieldType()); + assertEquals("", metamodel.path()); + assertEquals("nameFolded", metamodel.field()); + assertEquals(String.class, metamodel.fieldType()); + assertTrue(metamodel.isColumn()); + assertFalse(MetamodelFactory.isNullable(metamodel)); + } + + @Test + void testQueryColumnHoldsNoValue() { + Metamodel metamodel = MetamodelFactory.of(EntityWithQueryColumn.class, "nameFolded"); + var a = new EntityWithQueryColumn(1, "a"); + var b = new EntityWithQueryColumn(2, "b"); + + var exception = assertThrows(UnsupportedOperationException.class, () -> metamodel.getValue(a)); + assertTrue(exception.getMessage().contains("query column"), exception.getMessage()); + assertThrows(UnsupportedOperationException.class, () -> metamodel.isSame(a, b)); + assertThrows(UnsupportedOperationException.class, () -> metamodel.isIdentical(a, b)); + } + + @Test + void testQueryColumnBehindAnUnknownPathIsRefused() { + var exception = assertThrows(PersistenceException.class, + () -> MetamodelFactory.of(EntityWithQueryColumn.class, "unknown.nameFolded")); + assertTrue(exception.getMessage().contains("unknown.nameFolded"), exception.getMessage()); + } } diff --git a/storm-core/src/test/resources/data.sql b/storm-core/src/test/resources/data.sql index f95a066dc..f88ead5e0 100644 --- a/storm-core/src/test/resources/data.sql +++ b/storm-core/src/test/resources/data.sql @@ -23,7 +23,7 @@ drop table if exists country CASCADE; drop table if exists country_city CASCADE; create table city (id integer auto_increment, name varchar(255) not null, primary key (id)); -create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255) not null, city_id integer, telephone varchar(255), primary key (id), version integer default 0 not null); +create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255) not null, city_id integer, telephone varchar(255), primary key (id), version integer default 0 not null, last_name_folded varchar(255) generated always as (lower(last_name)) not null); create table pet (id integer auto_increment, name varchar(255) not null, birth_date date not null, owner_id integer, type_id integer not null, primary key (id)); create table pet_type (id integer, name varchar(255) not null, primary key (id)); create table specialty (id integer auto_increment, name varchar(255) not null, primary key (id)); diff --git a/storm-dialect-tck/src/main/java/st/orm/tck/AbstractSchemaValidatorConformanceTest.java b/storm-dialect-tck/src/main/java/st/orm/tck/AbstractSchemaValidatorConformanceTest.java index d6d37325b..b89c0a366 100644 --- a/storm-dialect-tck/src/main/java/st/orm/tck/AbstractSchemaValidatorConformanceTest.java +++ b/storm-dialect-tck/src/main/java/st/orm/tck/AbstractSchemaValidatorConformanceTest.java @@ -25,6 +25,7 @@ import st.orm.DbTable; import st.orm.Entity; import st.orm.PK; +import st.orm.QueryColumn; import st.orm.StormConfig; import st.orm.UK; import st.orm.core.spi.Providers; @@ -77,6 +78,23 @@ protected boolean supportsSequences() { return true; } + /** + * The statement that creates {@code generated_probe} with an integer primary key {@code id}, a {@code NOT NULL} + * text column {@code name}, and a {@code NOT NULL} text column {@code name_folded} the database generates from + * {@code name}; {@code null} for a database without generated columns. + */ + protected @Nullable String generatedColumnProbeDdl() { + return null; + } + + /** {@code generated_probe} with its generated column declared as a query column. */ + @QueryColumn(field = "nameFolded", type = String.class) + public record GeneratedProbe(@PK Integer id, String name) implements Entity {} + + /** {@code generated_probe} with its generated column left unmapped. */ + @DbTable("generated_probe") + public record GeneratedProbeUnmapped(@PK Integer id, String name) implements Entity {} + public record Vet( @PK Integer id, @Nullable String firstName, @@ -268,6 +286,30 @@ public void testIndexesAreDiscovered() throws SQLException { } } + @Test + public void testAGeneratedColumnIsFilledByTheDatabase() throws SQLException { + var ddl = generatedColumnProbeDdl(); + assumeTrue(ddl != null); + try (var connection = dataSource.getConnection(); var statement = connection.createStatement()) { + statement.execute(ddl); + try { + var dialect = Providers.getSqlDialect(connection, StormConfig.defaults()); + var schema = DatabaseSchema.read(connection, connection.getCatalog(), connection.getSchema(), + DatabaseSchema.Discovery.of(dialect)); + var column = schema.getColumn("generated_probe", "name_folded").orElseThrow(); + // The driver reports the column as generated or with its expression as default; either way the + // database fills it, so neither a query column nor an unmapped column over it fails an insert. + assertTrue(column.generated() || column.hasDefault(), column::toString); + var errors = SchemaValidator.of(dataSource).validate(List.of(GeneratedProbe.class, GeneratedProbeUnmapped.class)); + assertFalse(errors.stream().anyMatch(error -> error.kind() == ErrorKind.UNMAPPED_COLUMN + || error.kind() == ErrorKind.COLUMN_NOT_FOUND || error.kind() == ErrorKind.TYPE_INCOMPATIBLE), + errors::toString); + } finally { + statement.execute("DROP TABLE generated_probe"); + } + } + } + @Test public void testAnExpressionIndexHasNoColumn() throws SQLException { var ddl = expressionIndexDdl(); diff --git a/storm-foundation/src/main/java/st/orm/QueryColumn.java b/storm-foundation/src/main/java/st/orm/QueryColumn.java new file mode 100644 index 000000000..d96bc6360 --- /dev/null +++ b/storm-foundation/src/main/java/st/orm/QueryColumn.java @@ -0,0 +1,83 @@ +/* + * Copyright 2024 - 2026 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package st.orm; + +import static java.lang.annotation.ElementType.TYPE; +import static java.lang.annotation.RetentionPolicy.RUNTIME; + +import java.lang.annotation.Repeatable; +import java.lang.annotation.Retention; +import java.lang.annotation.Target; + +/** + * Declares a column a query names but the entity or projection does not carry: a query column. It gets a metamodel + * field, generated as a record component's is, but no component, so it is never selected, inserted or updated. A + * generated column that stores a derived form of another column, which a search matches and an index is built on, + * is the typical case: the database computes it, no caller reads it, and a record built by the application has no + * value to give it. + * + * 
{@code
+ * @QueryColumn(field = "titleFolded", type = String.class)
+ * record Movie(@PK Integer id, String title, Integer year) implements Entity {}
+ *
+ * orm.entity(Movie.class).select()
+ *     .where(it -> it.search(input, Movie_.titleFolded))
+ *     .getResultList();
+ * }
+ * + *

The generated {@code Movie_.titleFolded} is a {@code Metamodel} and goes wherever a column does: + * {@code where}, {@code orderBy}, {@code orderFirst}, {@code search} and {@code suggest}, a {@link SqlFunction}, a + * keyset sort and a template placeholder. It is reached through paths like any other field, so a query column of a + * referenced table is {@code Credit_.movie.titleFolded} from a query on credits. A record holds no value for it: + * {@link Metamodel#getValue(Data)} throws, and an {@code update} cannot set it.

+ * + *

The column follows from the field name through the column name resolver, as a component's does; + * {@link #column()} names it where the resolver does not. A column is carried by a field or named by a query column, + * not both: a declaration whose column a field maps is refused when the model is built, as is one named after a + * record component. Schema validation checks that the column exists with a + * compatible type. On an entity every insert leaves it out, so the database must fill it: a NOT NULL query column + * that is not generated, not an identity column and has no default is reported as an unmapped column is.

+ * + * @since 1.16 + */ +@Target(TYPE) +@Retention(RUNTIME) +@Repeatable(QueryColumns.class) +public @interface QueryColumn { + + /** + * The name of the metamodel field, such as {@code titleFolded} for {@code Movie_.titleFolded}. + */ + String field(); + + /** + * The type of the column's values, as a record component's type: {@code String.class}, {@code Integer.class}, + * {@code LocalDate.class}, an enum type. + */ + Class type(); + + /** + * The database column name. By default the column name resolver derives it from the field name, as it does for a + * record component: {@code titleFolded} is {@code title_folded} under the default resolver. + */ + String column() default ""; + + /** + * Whether the column allows null. A nullable query column compares as a nullable field does and is refused as a + * keyset sort field, as a nullable field is. + */ + boolean nullable() default false; +} diff --git a/storm-foundation/src/main/java/st/orm/QueryColumns.java b/storm-foundation/src/main/java/st/orm/QueryColumns.java new file mode 100644 index 000000000..5bcf517bf --- /dev/null +++ b/storm-foundation/src/main/java/st/orm/QueryColumns.java @@ -0,0 +1,38 @@ +/* + * Copyright 2024 - 2026 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package st.orm; + +import static java.lang.annotation.ElementType.TYPE; +import static java.lang.annotation.RetentionPolicy.RUNTIME; + +import java.lang.annotation.Retention; +import java.lang.annotation.Target; + +/** + * Declares several {@link QueryColumn query columns} in one annotation. {@code @QueryColumn} is repeatable, so the + * container is needed only where repeating is not possible. + * + * @since 1.16 + */ +@Target(TYPE) +@Retention(RUNTIME) +public @interface QueryColumns { + + /** + * The query columns. + */ + QueryColumn[] value(); +} diff --git a/storm-h2/src/test/java/st/orm/spi/h2/H2SchemaValidatorConformanceTest.java b/storm-h2/src/test/java/st/orm/spi/h2/H2SchemaValidatorConformanceTest.java index 833bfdbe7..14a0cb178 100644 --- a/storm-h2/src/test/java/st/orm/spi/h2/H2SchemaValidatorConformanceTest.java +++ b/storm-h2/src/test/java/st/orm/spi/h2/H2SchemaValidatorConformanceTest.java @@ -9,4 +9,9 @@ public class H2SchemaValidatorConformanceTest extends AbstractSchemaValidatorCon protected boolean indexesForeignKeys() { return true; } + + @Override + protected String generatedColumnProbeDdl() { + return "CREATE TABLE generated_probe (id INTEGER NOT NULL PRIMARY KEY, name VARCHAR(50) NOT NULL, name_folded VARCHAR(50) GENERATED ALWAYS AS (LOWER(name)) NOT NULL)"; + } } diff --git a/storm-java21/src/main/java/st/orm/template/Column.java b/storm-java21/src/main/java/st/orm/template/Column.java index 3bc714ca7..5bbc5364e 100644 --- a/storm-java21/src/main/java/st/orm/template/Column.java +++ b/storm-java21/src/main/java/st/orm/template/Column.java @@ -33,7 +33,8 @@ */ public interface Column { /** - * Gets the 1-based index of the column. + * Gets the 1-based index of the column among the model's columns, or {@code 0} for a query column, which has no + * position among them. * * @return the column index. */ diff --git a/storm-java21/src/main/java/st/orm/template/Model.java b/storm-java21/src/main/java/st/orm/template/Model.java index 37750b381..3244a3097 100644 --- a/storm-java21/src/main/java/st/orm/template/Model.java +++ b/storm-java21/src/main/java/st/orm/template/Model.java @@ -100,6 +100,17 @@ public interface Model { */ List declaredColumns(); + /** + * Returns the query columns declared on this model: the columns a query names but the record does not carry, + * declared with {@link st.orm.QueryColumn}. They are not among {@link #columns()} or {@link #declaredColumns()}, + * so a select, insert or update never names them. A query column has no record field: its + * {@link Column#index()} is {@code 0}, and it is neither insertable nor updatable. + * + * @return the query columns declared on this model, in declaration order. + * @since 1.16 + */ + List queryColumns(); + /** *

This method is used to check if the primary key of the entity is a default value. This is useful when * determining if the entity is new or has been persisted before.

diff --git a/storm-java21/src/main/java/st/orm/template/impl/ModelImpl.java b/storm-java21/src/main/java/st/orm/template/impl/ModelImpl.java index 04e5b2188..5b3e51d08 100644 --- a/storm-java21/src/main/java/st/orm/template/impl/ModelImpl.java +++ b/storm-java21/src/main/java/st/orm/template/impl/ModelImpl.java @@ -31,6 +31,7 @@ public class ModelImpl implements Model { private final st.orm.core.template.impl.ModelImpl core; private final List columns; private final List declaredColumns; + private final List queryColumns; public ModelImpl(st.orm.core.template.impl.ModelImpl core) { this.core = core; @@ -42,7 +43,10 @@ public ModelImpl(st.orm.core.template.impl.ModelImpl core) { .map(ColumnImpl::new) .map(Column.class::cast) .toList(); - + this.queryColumns = core.queryColumns().stream() + .map(ColumnImpl::new) + .map(Column.class::cast) + .toList(); } /** @@ -133,6 +137,11 @@ public List declaredColumns() { return declaredColumns; } + @Override + public List queryColumns() { + return queryColumns; + } + /** * Iterates over the values of the given columns for the supplied record. * diff --git a/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java b/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java index 20a41c6b9..3c1a6741b 100644 --- a/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java +++ b/storm-java21/src/test/java/st/orm/template/QueryBuilderTest.java @@ -1332,4 +1332,19 @@ public void testGetOptionalResultThrowsNonUnique() { assertThrows(NonUniqueResultException.class, () -> orm.entity(City.class).select().getOptionalResult()); } + + @Test + public void testQueryColumnIsNamedByAQueryAndCarriedByNoRow() { + // data.sql: owner.last_name_folded is generated from the last name; Owner declares it as a query column. + List owners = orm.entity(Owner.class).select().where(Owner_.lastNameFolded, EQUALS, "davis").getResultList(); + assertEquals(List.of("Betty", "Harold"), owners.stream().map(Owner::firstName).sorted().toList()); + assertEquals("owner", Pet_.owner.lastNameFolded.path()); + assertEquals("lastNameFolded", Pet_.owner.lastNameFolded.field()); + List pets = orm.entity(Pet.class).select().where(Pet_.owner.lastNameFolded, EQUALS, "davis").getResultList(); + assertEquals(List.of("Iggy", "Leo"), pets.stream().map(Pet::name).sorted().toList()); + var model = orm.entity(Owner.class).model(); + assertEquals(List.of("last_name_folded"), model.queryColumns().stream().map(Column::name).toList()); + assertTrue(model.columns().stream().noneMatch(column -> column.name().equals("last_name_folded"))); + assertThrows(UnsupportedOperationException.class, () -> Owner_.lastNameFolded.getValue(owners.getFirst())); + } } diff --git a/storm-java21/src/test/java/st/orm/template/model/Owner.java b/storm-java21/src/test/java/st/orm/template/model/Owner.java index 8016cf64e..7b96f1964 100644 --- a/storm-java21/src/test/java/st/orm/template/model/Owner.java +++ b/storm-java21/src/test/java/st/orm/template/model/Owner.java @@ -3,8 +3,10 @@ import org.jspecify.annotations.Nullable; import st.orm.Entity; import st.orm.PK; +import st.orm.QueryColumn; import st.orm.Version; +@QueryColumn(field = "lastNameFolded", type = String.class) public record Owner( @PK Integer id, String firstName, diff --git a/storm-java21/src/test/resources/data.sql b/storm-java21/src/test/resources/data.sql index 377667b95..87d446b6b 100644 --- a/storm-java21/src/test/resources/data.sql +++ b/storm-java21/src/test/resources/data.sql @@ -8,7 +8,7 @@ drop table if exists vet_specialty CASCADE; drop table if exists visit CASCADE; create table city (id integer auto_increment, name varchar(255) not null, primary key (id)); -create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255) not null, city_id integer not null, telephone varchar(255), primary key (id), version integer default 0 not null); +create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255) not null, city_id integer not null, telephone varchar(255), primary key (id), version integer default 0 not null, last_name_folded varchar(255) generated always as (lower(last_name)) not null); create table pet (id integer auto_increment, name varchar(255) not null, birth_date date not null, owner_id integer, type_id integer not null, primary key (id)); create table pet_type (id integer, name varchar(255) not null, primary key (id)); create table specialty (id integer auto_increment, name varchar(255), primary key (id)); diff --git a/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt b/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt index a7ea5b6e4..ec54291d9 100644 --- a/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt +++ b/storm-kotlin/src/main/kotlin/st/orm/template/Column.kt @@ -34,7 +34,8 @@ import kotlin.reflect.KClass */ public interface Column { /** - * Gets the 1-based index of the column. + * Gets the 1-based index of the column among the model's columns, or `0` for a query column, which has no + * position among them. * * @return the column index. */ diff --git a/storm-kotlin/src/main/kotlin/st/orm/template/Model.kt b/storm-kotlin/src/main/kotlin/st/orm/template/Model.kt index b09adedea..01c4db3d0 100644 --- a/storm-kotlin/src/main/kotlin/st/orm/template/Model.kt +++ b/storm-kotlin/src/main/kotlin/st/orm/template/Model.kt @@ -87,6 +87,17 @@ public interface Model { */ public val declaredColumns: List + /** + * Returns the query columns declared on this model: the columns a query names but the record does not carry, + * declared with [st.orm.QueryColumn]. They are not among [columns] or [declaredColumns], so a select, insert or + * update never names them. A query column has no record field: its [Column.index] is `0`, and it is neither + * insertable nor updatable. + * + * @return the query columns declared on this model, in declaration order. + * @since 1.16 + */ + public val queryColumns: List + /** * * This method is used to check if the primary key of the entity is a default value. This is useful when diff --git a/storm-kotlin/src/main/kotlin/st/orm/template/impl/ModelImpl.kt b/storm-kotlin/src/main/kotlin/st/orm/template/impl/ModelImpl.kt index a0d299abf..6ae79a4a4 100644 --- a/storm-kotlin/src/main/kotlin/st/orm/template/impl/ModelImpl.kt +++ b/storm-kotlin/src/main/kotlin/st/orm/template/impl/ModelImpl.kt @@ -24,6 +24,7 @@ internal class ModelImpl( internal val core: st.orm.core.template.Model, override val columns: List = core.columns().map(::ColumnImpl), override val declaredColumns: List = core.declaredColumns().map(::ColumnImpl), + override val queryColumns: List = core.queryColumns().map(::ColumnImpl), ) : Model { override val schema: String get() = core.schema() diff --git a/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt b/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt index 0cc5f2d98..1236a13f0 100644 --- a/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt +++ b/storm-kotlin/src/test/kotlin/st/orm/template/QueryBuilderTest.kt @@ -4,6 +4,7 @@ import ch.qos.logback.classic.Level import ch.qos.logback.classic.LoggerContext import ch.qos.logback.classic.spi.ILoggingEvent import ch.qos.logback.core.read.ListAppender +import io.kotest.assertions.throwables.shouldThrow import io.kotest.matchers.collections.shouldHaveSize import io.kotest.matchers.shouldBe import io.kotest.matchers.shouldNotBe @@ -2276,6 +2277,24 @@ internal open class QueryBuilderTest( firstNames("") shouldBe emptyList() } + @Test + fun `query column is named by a query and carried by no row`() { + // data.sql: owner.last_name_folded is generated from the last name; Owner declares it as a query column. + val lastNameFolded: Metamodel = Metamodel.of(Owner::class.java, "lastNameFolded") + val owners = orm.entity(Owner::class).select().where(lastNameFolded eq "davis").resultList + owners.map { it.firstName }.sorted() shouldBe listOf("Betty", "Harold") + val ownerLastNameFolded: Metamodel = Metamodel.of(Pet::class.java, "owner.lastNameFolded") + ownerLastNameFolded.path() shouldBe "owner" + ownerLastNameFolded.field() shouldBe "lastNameFolded" + orm.entity(Pet::class).select() + .where(ownerLastNameFolded eq "davis") + .resultList.map { it.name }.sorted() shouldBe listOf("Iggy", "Leo") + val model = orm.entity(Owner::class).model + model.queryColumns.map { it.name } shouldBe listOf("last_name_folded") + model.columns.none { it.name == "last_name_folded" } shouldBe true + shouldThrow { lastNameFolded.getValue(owners.first()) } + } + @Test fun `orderFirst should put the matching rows first in the order of the calls`() { val ownerRepo = orm.entity(Owner::class) diff --git a/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt b/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt index 43e4dd860..2d4a68319 100644 --- a/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt +++ b/storm-kotlin/src/test/kotlin/st/orm/template/model/Owner.kt @@ -17,12 +17,14 @@ package st.orm.template.model import st.orm.Entity import st.orm.PK +import st.orm.QueryColumn import st.orm.Version /** * Simple domain object representing an owner. * */ +@QueryColumn(field = "lastNameFolded", type = String::class) internal data class Owner( @PK val id: Int = 0, override val firstName: String, diff --git a/storm-kotlin/src/test/resources/data.sql b/storm-kotlin/src/test/resources/data.sql index 388e922cb..6666aaad6 100644 --- a/storm-kotlin/src/test/resources/data.sql +++ b/storm-kotlin/src/test/resources/data.sql @@ -10,7 +10,7 @@ drop table if exists vet_specialty CASCADE; drop table if exists visit CASCADE; create table city (id integer auto_increment, name varchar(255) not null, primary key (id)); -create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255), city_id integer, telephone varchar(255), primary key (id), version integer default 0 not null); +create table owner (id integer auto_increment, first_name varchar(255) not null, last_name varchar(255) not null, address varchar(255), city_id integer, telephone varchar(255), primary key (id), version integer default 0 not null, last_name_folded varchar(255) generated always as (lower(last_name)) not null); create table pet (id integer auto_increment, name varchar(255) not null, birth_date date not null, owner_id integer, type_id integer not null, primary key (id)); create table pet_type (id integer, name varchar(255) not null, primary key (id)); create table specialty (id integer auto_increment, name varchar(255), primary key (id)); diff --git a/storm-ktor-test/pom.xml b/storm-ktor-test/pom.xml index ae9ebffaa..bd180d6d5 100644 --- a/storm-ktor-test/pom.xml +++ b/storm-ktor-test/pom.xml @@ -249,6 +249,20 @@ ktor-server-test-host-jvm ${ktor.version} + + + org.apache.httpcomponents.core5 + httpcore5 + 5.4.4 + + + org.apache.httpcomponents.core5 + httpcore5-h2 + 5.4.4 + com.h2database h2 diff --git a/storm-mariadb/src/test/java/st/orm/spi/mariadb/MariaDBSchemaValidatorConformanceTest.java b/storm-mariadb/src/test/java/st/orm/spi/mariadb/MariaDBSchemaValidatorConformanceTest.java index b88d1afa4..49a621fd2 100644 --- a/storm-mariadb/src/test/java/st/orm/spi/mariadb/MariaDBSchemaValidatorConformanceTest.java +++ b/storm-mariadb/src/test/java/st/orm/spi/mariadb/MariaDBSchemaValidatorConformanceTest.java @@ -22,4 +22,9 @@ public static synchronized DataSource dataSource() { protected boolean indexesForeignKeys() { return true; } + + @Override + protected String generatedColumnProbeDdl() { + return "CREATE TABLE generated_probe (id INTEGER NOT NULL PRIMARY KEY, name VARCHAR(50) NOT NULL, name_folded VARCHAR(50) AS (LOWER(name)) STORED)"; + } } diff --git a/storm-metamodel-ksp/src/main/kotlin/st/orm/metamodel/MetamodelProcessor.kt b/storm-metamodel-ksp/src/main/kotlin/st/orm/metamodel/MetamodelProcessor.kt index d6c577b1c..7c0a20c0b 100644 --- a/storm-metamodel-ksp/src/main/kotlin/st/orm/metamodel/MetamodelProcessor.kt +++ b/storm-metamodel-ksp/src/main/kotlin/st/orm/metamodel/MetamodelProcessor.kt @@ -95,6 +95,21 @@ class MetamodelProcessor( */ private val generatedInstantiators = mutableListOf>() + /** + * The valid query columns per data class, keyed by qualified name. Reading a type's query columns reports its + * invalid declarations, so the cache makes each diagnostic print once. + */ + private val queryColumns = mutableMapOf>() + + /** + * A column a data class declares through `@QueryColumn`: it gets a metamodel field but has no component. + * + * @property field the metamodel field name. + * @property type the column's value type, as the annotation names it. + * @property nullable whether the column allows null. + */ + private data class QueryColumnDeclaration(val field: String, val type: KSType, val nullable: Boolean) + companion object { /** * The `@Suppress` key that silences the warning for a nullable unique key, on the property or an enclosing @@ -146,6 +161,8 @@ class MetamodelProcessor( private const val FOREIGN_KEY = "st.orm.FK" private const val PRIMARY_KEY = "st.orm.PK" private const val UNIQUE_KEY = "st.orm.UK" + private const val QUERY_COLUMN = "st.orm.QueryColumn" + private const val QUERY_COLUMNS = "st.orm.QueryColumns" private const val K_BOOLEAN = "kotlin.Boolean" private const val K_BYTE = "kotlin.Byte" @@ -296,6 +313,102 @@ class MetamodelProcessor( private fun isSealedInterface(classDeclaration: KSClassDeclaration): Boolean = classDeclaration.classKind == ClassKind.INTERFACE && classDeclaration.modifiers.contains(Modifier.SEALED) + /** + * Returns the valid query columns the data class declares, in declaration order. Query columns apply to data + * classes that implement `Data` or carry `@GenerateMetamodel`; on any other type, a sealed interface included, the + * annotation is ignored. The first read of a type reports its invalid declarations (an empty field, a field that + * repeats a component or another query column) and leaves them out. + */ + private fun queryColumns(classDeclaration: KSClassDeclaration): List { + if (isSealedInterface(classDeclaration) || !classDeclaration.isDataClass()) return emptyList() + val qualifiedName = classDeclaration.qualifiedName?.asString() ?: return emptyList() + return queryColumns.getOrPut(qualifiedName) { readQueryColumns(classDeclaration) } + } + + private fun readQueryColumns(classDeclaration: KSClassDeclaration): List { + if (!classDeclaration.implementsInterface(DATA) && !classDeclaration.hasAnnotation(GENERATE_METAMODEL)) { + return emptyList() + } + val declarations = queryColumnAnnotations(classDeclaration) + if (declarations.isEmpty()) return emptyList() + val className = classDeclaration.simpleName.asString() + val componentNames = getModelProperties(classDeclaration).map { it.simpleName.asString() }.toSet() + val declaredFields = mutableSetOf() + val result = mutableListOf() + for (declaration in declarations) { + val fieldName = annotationArgument(declaration, "field") as? String ?: "" + if (fieldName.isEmpty()) { + logger.error( + "Query column on $className declares an empty field. " + + "Name the metamodel field the query column gets.", + classDeclaration, + ) + continue + } + if (fieldName in componentNames) { + logger.error( + "Query column field '$fieldName' on $className has the name of a component. " + + "A query column declares a column the data class does not carry; the component already has " + + "a metamodel field.", + classDeclaration, + ) + continue + } + if (!declaredFields.add(fieldName)) { + logger.error("Query column field '$fieldName' is declared more than once on $className.", classDeclaration) + continue + } + // The compiler reports an unresolvable class literal itself. + val type = annotationArgument(declaration, "type") as? KSType ?: continue + if (type.isError) continue + val nullable = annotationArgument(declaration, "nullable") as? Boolean ?: false + result.add(QueryColumnDeclaration(fieldName, type, nullable)) + } + return result.toList() + } + + /** + * Returns the `@QueryColumn` annotations on the declaration in declaration order: direct annotations (a repeated + * annotation may be presented as several) and the ones a `@QueryColumns` container holds. + */ + private fun queryColumnAnnotations(classDeclaration: KSClassDeclaration): List = classDeclaration.annotations.flatMap { annotation -> + when (annotation.annotationType.resolve().declaration.qualifiedName?.asString()) { + QUERY_COLUMN -> sequenceOf(annotation) + QUERY_COLUMNS -> when (val value = annotationArgument(annotation, "value")) { + is List<*> -> value.asSequence().filterIsInstance() + is Array<*> -> value.asSequence().filterIsInstance() + else -> emptySequence() + } + else -> emptySequence() + } + }.toList() + + private fun annotationArgument(annotation: KSAnnotation, name: String): Any? = annotation.arguments.firstOrNull { it.name?.asString() == name }?.value + ?: annotation.defaultArguments.firstOrNull { it.name?.asString() == name }?.value + + /** + * Renders the `E` type of a query column's metamodel as a component's addressed type renders, with star + * projections for a generic type. + */ + private fun queryColumnTypeName(queryColumn: QueryColumnDeclaration, packageName: String): String { + val baseName = kotlinTypeNameOf(queryColumn.type, packageName) + val typeParameterCount = (queryColumn.type.declaration as? KSClassDeclaration)?.typeParameters?.size ?: 0 + return if (typeParameterCount == 0) baseName else baseName + List(typeParameterCount) { "*" }.joinToString(", ", "<", ">") + } + + /** + * Renders the `V` type of a query column's metamodel: the `E` type, nullable when the column allows null or the + * metamodel is a nullable-chain variant. + */ + private fun queryColumnValueTypeName( + queryColumn: QueryColumnDeclaration, + packageName: String, + forceNullableChain: Boolean, + ): String { + val typeName = queryColumnTypeName(queryColumn, packageName) + return if (queryColumn.nullable || forceNullableChain) ensureNullable(typeName) else typeName + } + /** * Returns the properties to include in the metamodel for the given class declaration. * For sealed interfaces, only abstract declared properties are included: a defaulted property has no column. @@ -878,6 +991,16 @@ class MetamodelProcessor( ) } } + queryColumns(classDeclaration).forEach { queryColumn -> + val fieldRef = escaped(queryColumn.field) + val kotlinTypeName = queryColumnTypeName(queryColumn, packageName) + val valueKotlinTypeName = queryColumnValueTypeName(queryColumn, packageName, forceNullableChain = false) + builder.append(" /** Represents the ${queryColumn.field} query column of $className. */\n") + builder.append( + " val $fieldRef: AbstractMetamodel<$className, $kotlinTypeName, $valueKotlinTypeName> = " + + "$modelRef.$fieldRef\n", + ) + } if (builder.isNotEmpty()) builder.setLength(builder.length - 1) return builder.toString() @@ -914,6 +1037,11 @@ class MetamodelProcessor( builder.append(" val $fieldRef: $baseClass\n") } } + queryColumns(classDeclaration).forEach { queryColumn -> + val kotlinTypeName = queryColumnTypeName(queryColumn, packageName) + val valueKotlinTypeName = queryColumnValueTypeName(queryColumn, packageName, forceNullableChain) + builder.append(" val ${escaped(queryColumn.field)}: AbstractMetamodel\n") + } return builder.toString() } @@ -1080,6 +1208,24 @@ class MetamodelProcessor( ) } } + // A data class instance holds no value for a query column, so the value methods throw. + val className = classDeclaration.simpleName.asString() + queryColumns(classDeclaration).forEach { queryColumn -> + val fieldName = queryColumn.field + val kotlinTypeName = queryColumnTypeName(queryColumn, packageName) + val valueKotlinTypeName = queryColumnValueTypeName(queryColumn, packageName, forceNullableChain) + val javaTypeName = javaTypeNameOf(queryColumn.type, packageName) + val failure = "throw UnsupportedOperationException(" + + "\"$className.$fieldName is a query column; a record holds no value for it.\")" + builder.append( + " this.${escaped(fieldName)} = object : AbstractMetamodel(" + + "$javaTypeName, subPath, fieldBase + \"$fieldName\", false, this) {\n" + + " override fun getValue(record: T): $valueKotlinTypeName = $failure\n\n" + + " override fun isIdentical(a: T, b: T): Boolean = $failure\n\n" + + " override fun isSame(a: T, b: T): Boolean = $failure\n" + + " }\n", + ) + } if (builder.isNotEmpty()) builder.setLength(builder.length - 1) return builder.toString() @@ -1171,6 +1317,11 @@ class MetamodelProcessor( builder.append(" val $fieldRef: st.orm.AbstractNavigableMetamodel\n") } } + queryColumns(classDeclaration).forEach { queryColumn -> + val kotlinTypeName = queryColumnTypeName(queryColumn, packageName) + builder.append(" /** Represents navigation to the ${queryColumn.field} query column of $className. */\n") + builder.append(" val ${escaped(queryColumn.field)}: st.orm.AbstractNavigableMetamodel\n") + } return builder.toString() } @@ -1199,6 +1350,15 @@ class MetamodelProcessor( ) } } + queryColumns(classDeclaration).forEach { queryColumn -> + val fieldName = queryColumn.field + val kotlinTypeName = queryColumnTypeName(queryColumn, packageName) + val javaTypeName = javaTypeNameOf(queryColumn.type, packageName) + builder.append( + " this.${escaped(fieldName)} = object : st.orm.AbstractNavigableMetamodel(" + + "$javaTypeName, subPath, fieldBase + \"$fieldName\", false, this) {}\n", + ) + } if (builder.isNotEmpty()) builder.setLength(builder.length - 1) return builder.toString() } diff --git a/storm-metamodel-ksp/src/test/kotlin/st/orm/metamodel/MetamodelProcessorTest.kt b/storm-metamodel-ksp/src/test/kotlin/st/orm/metamodel/MetamodelProcessorTest.kt index 4642cf1e4..58c57f79a 100644 --- a/storm-metamodel-ksp/src/test/kotlin/st/orm/metamodel/MetamodelProcessorTest.kt +++ b/storm-metamodel-ksp/src/test/kotlin/st/orm/metamodel/MetamodelProcessorTest.kt @@ -303,6 +303,197 @@ class MetamodelProcessorTest { } } + @Test + fun `repeated query columns generate throwing fields`() { + val compilation = compile( + """ + package com.example + + import st.orm.Entity + import st.orm.FK + import st.orm.PK + import st.orm.QueryColumn + import st.orm.Ref + + @QueryColumn(field = "titleFolded", type = String::class) + @QueryColumn(field = "releaseYear", type = Int::class) + data class Movie(@PK val id: Int = 0, val title: String) : Entity + + data class Credit(@PK val id: Int = 0, @FK val movie: Ref) : Entity + """.trimIndent(), + ) + val movieInterface = compilation.generatedSource("Movie_.kt") + assertTrue( + " /** Represents the titleFolded query column of Movie. */\n" + + " val titleFolded: AbstractMetamodel = MovieMetamodel.instance().titleFolded" + in movieInterface, + ) { "expected the query column in the metamodel interface:\n$movieInterface" } + assertTrue("val releaseYear: AbstractMetamodel" in movieInterface) { + "expected the second query column in the metamodel interface:\n$movieInterface" + } + assertTrue(movieInterface.indexOf("val title:") < movieInterface.indexOf("val titleFolded:")) { + "query columns follow the components:\n$movieInterface" + } + assertTrue(movieInterface.indexOf("val titleFolded:") < movieInterface.indexOf("val releaseYear:")) { + "query columns keep their declaration order:\n$movieInterface" + } + val movieMetamodel = compilation.generatedSource("MovieMetamodel.kt") + assertTrue(" val titleFolded: AbstractMetamodel\n" in movieMetamodel) { + "expected the query column field in the metamodel class:\n$movieMetamodel" + } + val failure = "throw UnsupportedOperationException(\"Movie.titleFolded is a query column; a record holds no value for it.\")" + assertTrue( + " this.titleFolded = object : AbstractMetamodel(" + + "String::class.java, subPath, fieldBase + \"titleFolded\", false, this) {\n" + + " override fun getValue(record: T): String = $failure\n\n" + + " override fun isIdentical(a: T, b: T): Boolean = $failure\n\n" + + " override fun isSame(a: T, b: T): Boolean = $failure\n" + + " }" + in movieMetamodel, + ) { "a record holds no value for a query column, so its value methods throw:\n$movieMetamodel" } + assertTrue("object : AbstractMetamodel(Int::class.javaObjectType, subPath, fieldBase + \"releaseYear\"" in movieMetamodel) { + "a primitive query column type renders as its object type:\n$movieMetamodel" + } + val movieNullableMetamodel = compilation.generatedSource("MovieNullableMetamodel.kt") + assertTrue("val titleFolded: AbstractMetamodel" in movieNullableMetamodel) { + "the nullable-chain variant reads every value as nullable, as for a component:\n$movieNullableMetamodel" + } + val movieRefMetamodel = compilation.generatedSource("MovieRefMetamodel.kt") + assertTrue( + "this.titleFolded = object : st.orm.AbstractNavigableMetamodel(" + + "String::class.java, subPath, fieldBase + \"titleFolded\", false, this) {}" + in movieRefMetamodel, + ) { "a query column is reachable beyond a reference:\n$movieRefMetamodel" } + } + + @Test + fun `query columns in the container generate fields`() { + val compilation = compile( + """ + package com.example + + import st.orm.Entity + import st.orm.PK + import st.orm.QueryColumn + import st.orm.QueryColumns + + @QueryColumns( + QueryColumn(field = "titleFolded", type = String::class), + QueryColumn(field = "releaseYear", type = Int::class), + ) + data class Movie(@PK val id: Int = 0, val title: String) : Entity + """.trimIndent(), + ) + val movieInterface = compilation.generatedSource("Movie_.kt") + assertTrue("val titleFolded: AbstractMetamodel" in movieInterface) { + "expected the first contained query column in the interface:\n$movieInterface" + } + assertTrue("val releaseYear: AbstractMetamodel" in movieInterface) { + "expected the second contained query column in the interface:\n$movieInterface" + } + val movieMetamodel = compilation.generatedSource("MovieMetamodel.kt") + assertTrue("UnsupportedOperationException(\"Movie.releaseYear is a query column; a record holds no value for it.\")" in movieMetamodel) { + "expected throwing value methods for the contained query columns:\n$movieMetamodel" + } + } + + @Test + fun `a nullable query column renders a nullable value type`() { + val compilation = compile( + """ + package com.example + + import st.orm.Entity + import st.orm.PK + import st.orm.QueryColumn + + @QueryColumn(field = "titleFolded", type = String::class, nullable = true) + data class Movie(@PK val id: Int = 0, val title: String) : Entity + """.trimIndent(), + ) + val movieInterface = compilation.generatedSource("Movie_.kt") + assertTrue("val titleFolded: AbstractMetamodel" in movieInterface) { + "a nullable query column carries a nullable value type:\n$movieInterface" + } + val movieMetamodel = compilation.generatedSource("MovieMetamodel.kt") + assertTrue("override fun getValue(record: T): String? = throw" in movieMetamodel) { + "a nullable query column carries a nullable value type:\n$movieMetamodel" + } + } + + @Test + fun `a keyword-named query column is escaped`() { + val compilation = compile( + """ + package com.example + + import st.orm.Entity + import st.orm.PK + import st.orm.QueryColumn + + @QueryColumn(field = "object", type = String::class) + data class Movie(@PK val id: Int = 0, val title: String) : Entity + """.trimIndent(), + ) + val movieMetamodel = compilation.generatedSource("MovieMetamodel.kt") + assertTrue("this.`object` = object : AbstractMetamodel" in movieMetamodel) { + "expected the keyword-named query column backticked:\n$movieMetamodel" + } + assertTrue("fieldBase + \"object\"" in movieMetamodel) { "the field string literal keeps the raw name:\n$movieMetamodel" } + } + + @Test + fun `a duplicate query column field fails compilation`() { + val messages = compileExpectingError( + """ + package com.example + + import st.orm.Entity + import st.orm.PK + import st.orm.QueryColumn + + @QueryColumn(field = "titleFolded", type = String::class) + @QueryColumn(field = "titleFolded", type = String::class) + data class Movie(@PK val id: Int = 0, val title: String) : Entity + """.trimIndent(), + ) + assertTrue("Query column field 'titleFolded' is declared more than once on Movie." in messages) { messages } + } + + @Test + fun `a query column field named like a component fails compilation`() { + val messages = compileExpectingError( + """ + package com.example + + import st.orm.Entity + import st.orm.PK + import st.orm.QueryColumn + + @QueryColumn(field = "title", type = String::class) + data class Movie(@PK val id: Int = 0, val title: String) : Entity + """.trimIndent(), + ) + assertTrue("Query column field 'title' on Movie has the name of a component." in messages) { messages } + } + + @Test + fun `an empty query column field fails compilation`() { + val messages = compileExpectingError( + """ + package com.example + + import st.orm.Entity + import st.orm.PK + import st.orm.QueryColumn + + @QueryColumn(field = "", type = String::class) + data class Movie(@PK val id: Int = 0, val title: String) : Entity + """.trimIndent(), + ) + assertTrue("Query column on Movie declares an empty field." in messages) { messages } + } + @Test fun `ignores plain data class without annotation`() { val compilation = compile( diff --git a/storm-metamodel-processor/src/main/java/st/orm/metamodel/MetamodelProcessor.java b/storm-metamodel-processor/src/main/java/st/orm/metamodel/MetamodelProcessor.java index d89d6034c..212081810 100644 --- a/storm-metamodel-processor/src/main/java/st/orm/metamodel/MetamodelProcessor.java +++ b/storm-metamodel-processor/src/main/java/st/orm/metamodel/MetamodelProcessor.java @@ -29,6 +29,8 @@ import java.io.StringWriter; import java.io.UncheckedIOException; import java.io.Writer; +import java.util.ArrayList; +import java.util.HashMap; import java.util.HashSet; import java.util.LinkedHashSet; import java.util.List; @@ -80,6 +82,8 @@ public final class MetamodelProcessor extends AbstractProcessor { private static final String FOREIGN_KEY = "st.orm.FK"; private static final String PRIMARY_KEY = "st.orm.PK"; private static final String UNIQUE_KEY = "st.orm.UK"; + private static final String QUERY_COLUMN = "st.orm.QueryColumn"; + private static final String QUERY_COLUMNS = "st.orm.QueryColumns"; private static final Pattern REF_PATTERN = Pattern.compile("^st\\.orm\\.Ref<([^>]+)>$"); @@ -133,6 +137,21 @@ public final class MetamodelProcessor extends AbstractProcessor { */ private final Set generatedInstantiators; + /** + * The valid query columns per record type, keyed by qualified name. Reading a type's query columns reports its + * invalid declarations, so the cache makes each diagnostic print once. + */ + private final Map> queryColumns; + + /** + * A column a record declares through {@code @QueryColumn}: it gets a metamodel field but has no record component. + * + * @param field the metamodel field name. + * @param type the column's value type, as the annotation names it. + * @param nullable whether the column allows null. + */ + private record QueryColumnDeclaration(String field, TypeMirror type, boolean nullable) {} + private Elements elementUtils; private Types typeUtils; private boolean jspecifyAvailable; @@ -147,6 +166,7 @@ public MetamodelProcessor() { this.generatedNavigableMetamodels = new HashSet<>(); this.navPath = new HashSet<>(); this.generatedInstantiators = new LinkedHashSet<>(); + this.queryColumns = new HashMap<>(); } @Override @@ -322,18 +342,232 @@ private boolean implementsData(Element recordElement) { return implementsInterface(recordElement.asType(), DATA, typeUtils); } + private static boolean hasGenerateMetamodel(Element element) { + return element.getAnnotationMirrors().stream() + .anyMatch(annotationMirror -> GENERATE_METAMODEL.equals(annotationMirror.getAnnotationType().toString())); + } + + /** + * Returns the valid query columns the record declares, in declaration order. Query columns apply to records that + * implement {@code Data} or carry {@code @GenerateMetamodel}; on any other record the annotation is ignored. The + * first read of a type reports its invalid declarations (an empty field, a field that is not a Java identifier, a + * field that repeats a record component or another query column) and leaves them out. + */ + private List queryColumns(Element recordElement) { + if (!(recordElement instanceof TypeElement typeElement) || !isRecord(recordElement)) { + return List.of(); + } + String qualifiedName = typeElement.getQualifiedName().toString(); + List cached = queryColumns.get(qualifiedName); + if (cached == null) { + cached = readQueryColumns(typeElement); + queryColumns.put(qualifiedName, cached); + } + return cached; + } + + private List readQueryColumns(TypeElement recordElement) { + if (!implementsData(recordElement) && !hasGenerateMetamodel(recordElement)) { + return List.of(); + } + List declarations = queryColumnMirrors(recordElement); + if (declarations.isEmpty()) { + return List.of(); + } + String recordName = recordElement.getSimpleName().toString(); + Set componentNames = new HashSet<>(); + for (Element enclosed : recordElement.getEnclosedElements()) { + if (getRecordComponentType(enclosed).isEmpty()) continue; + String fieldName = enclosed.getSimpleName().toString(); + if (getTypeElement(recordElement, fieldName) != null) { + componentNames.add(fieldName); + } + } + Set declaredFields = new HashSet<>(); + List result = new ArrayList<>(); + for (AnnotationMirror declaration : declarations) { + Object field = annotationAttribute(declaration, "field"); + Object type = annotationAttribute(declaration, "type"); + Object nullable = annotationAttribute(declaration, "nullable"); + String fieldName = field instanceof String name ? name : ""; + if (fieldName.isEmpty()) { + processingEnv.getMessager().printMessage(ERROR, + "Query column on " + recordName + " declares an empty field. " + + "Name the metamodel field the query column gets.", + recordElement, declaration); + continue; + } + if (!SourceVersion.isIdentifier(fieldName) || SourceVersion.isKeyword(fieldName)) { + processingEnv.getMessager().printMessage(ERROR, + "Query column field '" + fieldName + "' on " + recordName + " is not a valid Java identifier.", + recordElement, declaration); + continue; + } + if (componentNames.contains(fieldName)) { + processingEnv.getMessager().printMessage(ERROR, + "Query column field '" + fieldName + "' on " + recordName + " has the name of a record " + + "component. A query column declares a column the record does not carry; the component " + + "already has a metamodel field.", + recordElement, declaration); + continue; + } + if (!declaredFields.add(fieldName)) { + processingEnv.getMessager().printMessage(ERROR, + "Query column field '" + fieldName + "' is declared more than once on " + recordName + ".", + recordElement, declaration); + continue; + } + if (!(type instanceof TypeMirror typeMirror) || typeMirror.getKind() == TypeKind.ERROR) { + // The compiler reports an unresolvable class literal itself. + continue; + } + if (typeMirror.getKind() == TypeKind.VOID) { + processingEnv.getMessager().printMessage(ERROR, + "Query column field '" + fieldName + "' on " + recordName + " declares type void.", + recordElement, declaration); + continue; + } + result.add(new QueryColumnDeclaration(fieldName, typeMirror, Boolean.TRUE.equals(nullable))); + } + return List.copyOf(result); + } + + /** + * Returns the {@code @QueryColumn} mirrors on the element in declaration order: a direct annotation, or the + * {@code @QueryColumns} container javac synthesizes for a repeated one (or that the source names explicitly). + */ + private static List queryColumnMirrors(Element element) { + List mirrors = new ArrayList<>(); + for (AnnotationMirror annotationMirror : element.getAnnotationMirrors()) { + String annotationName = annotationMirror.getAnnotationType().toString(); + if (QUERY_COLUMN.equals(annotationName)) { + mirrors.add(annotationMirror); + } else if (QUERY_COLUMNS.equals(annotationName)) { + for (var entry : annotationMirror.getElementValues().entrySet()) { + if (!"value".equals(entry.getKey().getSimpleName().toString())) continue; + if (entry.getValue().getValue() instanceof List values) { + for (Object value : values) { + if (value instanceof AnnotationValue annotationValue + && annotationValue.getValue() instanceof AnnotationMirror nested) { + mirrors.add(nested); + } + } + } + } + } + } + return mirrors; + } + + @Nullable + private Object annotationAttribute(AnnotationMirror annotationMirror, String name) { + for (var entry : elementUtils.getElementValuesWithDefaults(annotationMirror).entrySet()) { + if (name.equals(entry.getKey().getSimpleName().toString())) { + return entry.getValue().getValue(); + } + } + return null; + } + + /** + * Renders the {@code E} type of a query column's metamodel: the boxed type, qualified unless it is declared + * directly in the record's package, as a record component's type renders. + */ + private static String queryColumnTypeName(QueryColumnDeclaration queryColumn, String packageName) { + return getTypeName(queryColumn.type(), packageName); + } + + /** + * Renders the {@code V} type of a query column's metamodel: the {@code E} type, marked nullable when the column + * allows null. + */ + /** + * The value type of a query column field, which is its type: a component's value type carries no nullability + * annotation either. + */ + private String queryColumnValueTypeName(QueryColumnDeclaration queryColumn, String packageName) { + return queryColumnTypeName(queryColumn, packageName); + } + + private static String queryColumnDoc(String recordName, String fieldName) { + return "the {@code " + fieldName + "} query column of {@link " + recordName + "}"; + } + + private void appendQueryColumnInterfaceFields(StringBuilder builder, Element recordElement, String packageName, + String recordName, String modelRef) { + for (QueryColumnDeclaration queryColumn : queryColumns(recordElement)) { + String fieldName = queryColumn.field(); + builder.append(" /** Represents ").append(queryColumnDoc(recordName, fieldName)).append(". */\n"); + builder.append(" AbstractMetamodel<").append(recordName).append(", ") + .append(queryColumnTypeName(queryColumn, packageName)).append(", ") + .append(queryColumnValueTypeName(queryColumn, packageName)).append("> ") + .append(fieldName).append(" = ").append(modelRef).append(".").append(fieldName).append(";\n"); + } + } + + private void appendQueryColumnClassFields(StringBuilder builder, Element recordElement, String packageName, + String recordName) { + for (QueryColumnDeclaration queryColumn : queryColumns(recordElement)) { + String fieldName = queryColumn.field(); + builder.append(" /** Represents ").append(queryColumnDoc(recordName, fieldName)).append(". */\n"); + builder.append(" public final AbstractMetamodel ") + .append(fieldName).append(";\n"); + } + } + + /** + * Appends the initializers of the query column fields. A record holds no value for a query column, so the value + * methods throw. + */ + private void appendQueryColumnInitializers(StringBuilder builder, Element recordElement, String packageName, + String recordName) { + for (QueryColumnDeclaration queryColumn : queryColumns(recordElement)) { + String fieldName = queryColumn.field(); + String typeName = queryColumnTypeName(queryColumn, packageName); + String valueTypeName = queryColumnValueTypeName(queryColumn, packageName); + String failure = "throw new UnsupportedOperationException(\"" + recordName + "." + fieldName + + " is a query column; a record holds no value for it.\");"; + builder.append(" this.").append(fieldName).append(" = new AbstractMetamodel(") + .append(typeName).append(".class, subPath, fieldBase + \"").append(fieldName) + .append("\", false, this) {\n") + .append(" @Override public ").append(nullableReturnType(typeName)) + .append(" getValue(T record) { ").append(failure).append(" }\n") + .append(" @Override public boolean isIdentical(T a, T b) { ").append(failure).append(" }\n") + .append(" @Override public boolean isSame(T a, T b) { ").append(failure).append(" }\n") + .append(" };\n"); + } + } + + private void appendQueryColumnNavFields(StringBuilder builder, Element recordElement, String packageName, + String recordName) { + for (QueryColumnDeclaration queryColumn : queryColumns(recordElement)) { + String fieldName = queryColumn.field(); + builder.append(" /** Represents navigation to ").append(queryColumnDoc(recordName, fieldName)) + .append(". */\n"); + builder.append(" public final st.orm.AbstractNavigableMetamodel ").append(fieldName).append(";\n"); + } + } + + private void appendQueryColumnNavInitializers(StringBuilder builder, Element recordElement, String packageName) { + for (QueryColumnDeclaration queryColumn : queryColumns(recordElement)) { + String fieldName = queryColumn.field(); + String typeName = queryColumnTypeName(queryColumn, packageName); + builder.append(" this.").append(fieldName).append(" = new st.orm.AbstractNavigableMetamodel(").append(typeName).append(".class, subPath, fieldBase + \"") + .append(fieldName).append("\", false, this) {};\n"); + } + } + @Override public boolean process(Set annotations, RoundEnvironment roundEnv) { for (Element element : roundEnv.getRootElements()) { try { if (isRecord(element)) { - boolean hasGenerateMetamodel = element.getAnnotationMirrors().stream() - .anyMatch(annotationMirror -> GENERATE_METAMODEL - .equals(annotationMirror.getAnnotationType().toString())); - - boolean isData = implementsData(element); - if (hasGenerateMetamodel || isData) { + if (hasGenerateMetamodel(element) || implementsData(element)) { generateMetamodelArtifacts(element); } } else if (element.getKind() == ElementKind.INTERFACE @@ -1074,6 +1308,7 @@ private String buildInterfaceFields(Element recordElement, String packageName) { .append(fieldName).append(";\n"); } } + appendQueryColumnInterfaceFields(builder, recordElement, packageName, recordName, modelRef); if (!builder.isEmpty()) { builder.setLength(builder.length() - 1); } @@ -1247,6 +1482,7 @@ private String buildClassFields(Element recordElement, .append(";\n"); } } + appendQueryColumnClassFields(builder, recordElement, packageName, recordName); return builder.toString(); } @@ -1392,6 +1628,7 @@ private String initClassFields(Element recordElement, .append(" };\n"); } } + appendQueryColumnInitializers(builder, recordElement, packageName, recordName); if (!builder.isEmpty()) { builder.setLength(builder.length() - 1); } @@ -1569,6 +1806,7 @@ boolean record = isRecord(fieldType) && !isRefType(fieldType); builder.append(" public final st.orm.AbstractNavigableMetamodel ").append(fieldName).append(";\n"); } } + appendQueryColumnNavFields(builder, recordElement, packageName, recordName); return builder.toString(); } @@ -1594,6 +1832,7 @@ boolean record = isRecord(fieldType) && !isRefType(fieldType); .append(fieldName).append("\", false, this) {};\n"); } } + appendQueryColumnNavInitializers(builder, recordElement, packageName); if (!builder.isEmpty()) builder.setLength(builder.length() - 1); return builder.toString(); } diff --git a/storm-metamodel-processor/src/test/java/st/orm/metamodel/MetamodelProcessorTest.java b/storm-metamodel-processor/src/test/java/st/orm/metamodel/MetamodelProcessorTest.java index cb4036add..99d5f8a62 100644 --- a/storm-metamodel-processor/src/test/java/st/orm/metamodel/MetamodelProcessorTest.java +++ b/storm-metamodel-processor/src/test/java/st/orm/metamodel/MetamodelProcessorTest.java @@ -381,6 +381,162 @@ record City(@PK Integer id, @FK Ref country) implements Entity "navigation beyond the Ref boundary reaches the deeper graph"); } + @Test + void generatesThrowingFieldsForRepeatedQueryColumns() throws Exception { + Compilation compilation = compile("Movie.java", """ + import st.orm.Entity; + import st.orm.FK; + import st.orm.PK; + import st.orm.QueryColumn; + import st.orm.Ref; + + @QueryColumn(field = "titleFolded", type = String.class) + @QueryColumn(field = "releaseYear", type = int.class) + public record Movie(@PK Integer id, String title) implements Entity {} + + record Credit(@PK Integer id, @FK Ref movie) implements Entity {} + """); + assertTrue(compilation.success(), compilation.errors()); + String movieInterface = compilation.generatedSource("Movie_.java"); + assertTrue(movieInterface.contains(""" + /** Represents the {@code titleFolded} query column of {@link Movie}. */ + AbstractMetamodel titleFolded = \ + MovieMetamodel.instance().titleFolded;"""), + "expected the query column in the metamodel interface:\n" + movieInterface); + assertTrue(movieInterface.contains("AbstractMetamodel releaseYear"), + "a primitive query column type renders boxed:\n" + movieInterface); + assertTrue(movieInterface.indexOf(" title ") < movieInterface.indexOf("titleFolded") + && movieInterface.indexOf("titleFolded") < movieInterface.indexOf("releaseYear"), + "query columns follow the components in declaration order:\n" + movieInterface); + String movieMetamodel = compilation.generatedSource("MovieMetamodel.java"); + assertTrue(movieMetamodel.contains( + "public final AbstractMetamodel titleFolded;"), + "expected the query column field in the metamodel class:\n" + movieMetamodel); + assertTrue(movieMetamodel.contains(""" + this.titleFolded = new AbstractMetamodel(\ + java.lang.String.class, subPath, fieldBase + "titleFolded", false, this) { + @Override public java.lang.@org.jspecify.annotations.Nullable String getValue(T record) { \ + throw new UnsupportedOperationException("Movie.titleFolded is a query column; a record holds no value for it."); } + @Override public boolean isIdentical(T a, T b) { \ + throw new UnsupportedOperationException("Movie.titleFolded is a query column; a record holds no value for it."); } + @Override public boolean isSame(T a, T b) { \ + throw new UnsupportedOperationException("Movie.titleFolded is a query column; a record holds no value for it."); } + };"""), + "a record holds no value for a query column, so its value methods throw:\n" + movieMetamodel); + String movieNullableMetamodel = compilation.generatedSource("MovieNullableMetamodel.java"); + assertTrue(movieNullableMetamodel.contains( + "public final AbstractMetamodel releaseYear;"), + "the nullable-chain variant carries the query columns too:\n" + movieNullableMetamodel); + String movieRefMetamodel = compilation.generatedSource("MovieRefMetamodel.java"); + assertTrue(movieRefMetamodel.contains(""" + this.titleFolded = new st.orm.AbstractNavigableMetamodel(\ + java.lang.String.class, subPath, fieldBase + "titleFolded", false, this) {};"""), + "a query column is reachable beyond a reference:\n" + movieRefMetamodel); + assertTrue(Files.exists(compilation.classes().resolve("Movie_.class")) + && Files.exists(compilation.classes().resolve("MovieNullableMetamodel.class")) + && Files.exists(compilation.classes().resolve("MovieRefMetamodel.class")), + "expected the generated metamodels to compile"); + } + + @Test + void generatesQueryColumnsFromTheContainer() throws Exception { + Compilation compilation = compile("Movie.java", """ + import st.orm.Entity; + import st.orm.PK; + import st.orm.QueryColumn; + import st.orm.QueryColumns; + + @QueryColumns({ + @QueryColumn(field = "titleFolded", type = String.class), + @QueryColumn(field = "releaseYear", type = int.class) + }) + public record Movie(@PK Integer id, String title) implements Entity {} + """); + assertTrue(compilation.success(), compilation.errors()); + String movieInterface = compilation.generatedSource("Movie_.java"); + assertTrue(movieInterface.contains("AbstractMetamodel titleFolded"), + "expected the first contained query column in the interface:\n" + movieInterface); + assertTrue(movieInterface.contains("AbstractMetamodel releaseYear"), + "expected the second contained query column in the interface:\n" + movieInterface); + String movieMetamodel = compilation.generatedSource("MovieMetamodel.java"); + assertTrue(movieMetamodel.contains( + "throw new UnsupportedOperationException(\"Movie.releaseYear is a query column; a record holds no value for it.\");"), + "expected throwing value methods for the contained query columns:\n" + movieMetamodel); + assertTrue(Files.exists(compilation.classes().resolve("MovieMetamodel.class")), + "expected the generated metamodel to compile"); + } + + @Test + void rendersANullableQueryColumnAsANullableComponent() throws Exception { + Compilation compilation = compile("Movie.java", """ + import org.jspecify.annotations.Nullable; + import st.orm.Entity; + import st.orm.PK; + import st.orm.QueryColumn; + + @QueryColumn(field = "titleFolded", type = String.class, nullable = true) + public record Movie(@PK Integer id, @Nullable String title) implements Entity {} + """); + assertTrue(compilation.success(), compilation.errors()); + String movieInterface = compilation.generatedSource("Movie_.java"); + // A component's value type carries no nullability annotation; a query column's follows it. + assertTrue(movieInterface.contains("AbstractMetamodel title "), + "a nullable component carries a plain value type:\n" + movieInterface); + assertTrue(movieInterface.contains("AbstractMetamodel titleFolded"), + "a nullable query column carries the value type a nullable component carries:\n" + movieInterface); + String movieMetamodel = compilation.generatedSource("MovieMetamodel.java"); + assertTrue(movieMetamodel.contains("public final AbstractMetamodel titleFolded;"), + "a nullable query column carries the value type a nullable component carries:\n" + movieMetamodel); + assertTrue(Files.exists(compilation.classes().resolve("Movie_.class")), + "expected the generated metamodels to compile"); + } + + @Test + void rejectsADuplicateQueryColumnField() throws Exception { + Compilation compilation = compile("Movie.java", """ + import st.orm.Entity; + import st.orm.PK; + import st.orm.QueryColumn; + + @QueryColumn(field = "titleFolded", type = String.class) + @QueryColumn(field = "titleFolded", type = String.class) + public record Movie(@PK Integer id, String title) implements Entity {} + """); + assertFalse(compilation.success(), "a query column field declared twice must fail compilation"); + assertTrue(compilation.errors().contains("Query column field 'titleFolded' is declared more than once on Movie."), + compilation.errors()); + } + + @Test + void rejectsAQueryColumnFieldNamedLikeAComponent() throws Exception { + Compilation compilation = compile("Movie.java", """ + import st.orm.Entity; + import st.orm.PK; + import st.orm.QueryColumn; + + @QueryColumn(field = "title", type = String.class) + public record Movie(@PK Integer id, String title) implements Entity {} + """); + assertFalse(compilation.success(), "a query column named like a component must fail compilation"); + assertTrue(compilation.errors().contains("Query column field 'title' on Movie has the name of a record component."), + compilation.errors()); + } + + @Test + void rejectsAnEmptyQueryColumnField() throws Exception { + Compilation compilation = compile("Movie.java", """ + import st.orm.Entity; + import st.orm.PK; + import st.orm.QueryColumn; + + @QueryColumn(field = "", type = String.class) + public record Movie(@PK Integer id, String title) implements Entity {} + """); + assertFalse(compilation.success(), "a query column with an empty field must fail compilation"); + assertTrue(compilation.errors().contains("Query column on Movie declares an empty field."), + compilation.errors()); + } + @Test void ignoresPlainRecordWithoutAnnotation() throws Exception { Compilation compilation = compile("CityStats.java", """ diff --git a/storm-mssqlserver/src/test/java/st/orm/spi/mssqlserver/MSSQLServerSchemaValidatorConformanceTest.java b/storm-mssqlserver/src/test/java/st/orm/spi/mssqlserver/MSSQLServerSchemaValidatorConformanceTest.java index db7fa3183..ced9417ec 100644 --- a/storm-mssqlserver/src/test/java/st/orm/spi/mssqlserver/MSSQLServerSchemaValidatorConformanceTest.java +++ b/storm-mssqlserver/src/test/java/st/orm/spi/mssqlserver/MSSQLServerSchemaValidatorConformanceTest.java @@ -18,4 +18,9 @@ public static synchronized DataSource dataSource() { } return ContainerDataSource.of(container.getJdbcUrl(), container.getUsername(), container.getPassword()); } + + @Override + protected String generatedColumnProbeDdl() { + return "CREATE TABLE generated_probe (id INTEGER NOT NULL PRIMARY KEY, name VARCHAR(50) NOT NULL, name_folded AS (LOWER(name)) PERSISTED NOT NULL)"; + } } diff --git a/storm-mysql/src/test/java/st/orm/spi/mysql/MySQLSchemaValidatorConformanceTest.java b/storm-mysql/src/test/java/st/orm/spi/mysql/MySQLSchemaValidatorConformanceTest.java index 8409c9685..47937e1cc 100644 --- a/storm-mysql/src/test/java/st/orm/spi/mysql/MySQLSchemaValidatorConformanceTest.java +++ b/storm-mysql/src/test/java/st/orm/spi/mysql/MySQLSchemaValidatorConformanceTest.java @@ -32,4 +32,9 @@ protected String expressionIndexDdl() { protected boolean indexesForeignKeys() { return true; } + + @Override + protected String generatedColumnProbeDdl() { + return "CREATE TABLE generated_probe (id INTEGER NOT NULL PRIMARY KEY, name VARCHAR(50) NOT NULL, name_folded VARCHAR(50) AS (LOWER(name)) STORED NOT NULL)"; + } } diff --git a/storm-oracle/src/test/java/st/orm/spi/oracle/OracleSchemaValidatorConformanceTest.java b/storm-oracle/src/test/java/st/orm/spi/oracle/OracleSchemaValidatorConformanceTest.java index 990318e8c..306f46cf9 100644 --- a/storm-oracle/src/test/java/st/orm/spi/oracle/OracleSchemaValidatorConformanceTest.java +++ b/storm-oracle/src/test/java/st/orm/spi/oracle/OracleSchemaValidatorConformanceTest.java @@ -35,4 +35,9 @@ static synchronized void stopContainer() { protected String expressionIndexDdl() { return "CREATE UNIQUE INDEX index_probe_lower ON index_probe (LOWER(a))"; } + + @Override + protected String generatedColumnProbeDdl() { + return "CREATE TABLE generated_probe (id NUMBER(10) NOT NULL PRIMARY KEY, name VARCHAR2(50) NOT NULL, name_folded VARCHAR2(50) GENERATED ALWAYS AS (LOWER(name)) VIRTUAL NOT NULL)"; + } } diff --git a/storm-postgresql/src/test/java/st/orm/spi/postgresql/PostgreSQLSchemaValidatorConformanceTest.java b/storm-postgresql/src/test/java/st/orm/spi/postgresql/PostgreSQLSchemaValidatorConformanceTest.java index d737ddfa5..04d732938 100644 --- a/storm-postgresql/src/test/java/st/orm/spi/postgresql/PostgreSQLSchemaValidatorConformanceTest.java +++ b/storm-postgresql/src/test/java/st/orm/spi/postgresql/PostgreSQLSchemaValidatorConformanceTest.java @@ -55,4 +55,9 @@ public void testAPartialIndexIsMarkedAndAViewIndexIsLeftOut() throws SQLExceptio } } } + + @Override + protected String generatedColumnProbeDdl() { + return "CREATE TABLE generated_probe (id INTEGER NOT NULL PRIMARY KEY, name VARCHAR(50) NOT NULL, name_folded VARCHAR(50) GENERATED ALWAYS AS (LOWER(name)) STORED NOT NULL)"; + } } diff --git a/storm-sqlite/src/test/java/st/orm/spi/sqlite/SQLiteSchemaValidatorConformanceTest.java b/storm-sqlite/src/test/java/st/orm/spi/sqlite/SQLiteSchemaValidatorConformanceTest.java index bc7ece31a..cae7d93c4 100644 --- a/storm-sqlite/src/test/java/st/orm/spi/sqlite/SQLiteSchemaValidatorConformanceTest.java +++ b/storm-sqlite/src/test/java/st/orm/spi/sqlite/SQLiteSchemaValidatorConformanceTest.java @@ -15,4 +15,9 @@ protected boolean supportsSequences() { protected String expressionIndexDdl() { return "CREATE UNIQUE INDEX index_probe_lower ON index_probe (lower(a))"; } + + @Override + protected String generatedColumnProbeDdl() { + return "CREATE TABLE generated_probe (id INTEGER NOT NULL PRIMARY KEY, name TEXT NOT NULL, name_folded TEXT GENERATED ALWAYS AS (LOWER(name)) STORED NOT NULL)"; + } } diff --git a/website/static/skills/storm-entity-from-schema.md b/website/static/skills/storm-entity-from-schema.md index c66945975..f4df3a448 100644 --- a/website/static/skills/storm-entity-from-schema.md +++ b/website/static/skills/storm-entity-from-schema.md @@ -61,6 +61,7 @@ Generation/update rules: - **Composite unique constraints** (only when needed): composite unique constraints do NOT need to be modeled unless the user explicitly needs a composite `Metamodel.Key` (for keyset pagination or type-safe lookups). When needed, use an inline record + `@UK @Persist(insertable = false, updatable = false)` — ask the user if they need this. - `describe_table` reports unique constraints (top-level `uniqueConstraints` array with constraint name and column list) and FK cascade rules (onDelete/onUpdate). Both are exposed for context only. **Storm does not model cascade behavior or enforce uniqueness** — these are database-level concerns. Do not generate annotations or code for cascade rules. - @Version for version columns (confirm with user). +- Generated or computed columns that no caller reads, such as a folded or normalized copy of another column that a search index is built on: declare them as query columns on the type, `@QueryColumn(field = "titleFolded", type = String::class)` (Java: `String.class`), not as fields, so they are never selected or written. Map one as a `@Persist(insertable = false, updatable = false)` field only when the application reads its value (confirm with user). - When updating, preserve existing field order and custom annotations. Only add/modify what changed. - Use `@DbIgnore` on fields or types that should be excluded from schema validation. Note: ad-hoc query result types (aggregation DTOs) should not need this — write them as plain data classes/records without the `Data` marker instead of declaring `Data` and then suppressing validation with `@DbIgnore`. - Use `@PK(constraint = false)` if the table intentionally has no PK constraint in the database. diff --git a/website/static/skills/storm-entity-java.md b/website/static/skills/storm-entity-java.md index 9369369fd..6d59fea57 100644 --- a/website/static/skills/storm-entity-java.md +++ b/website/static/skills/storm-entity-java.md @@ -146,6 +146,8 @@ Generation rules: ``` Use `insertable = false` alone for columns set by the database only on INSERT, or `updatable = false` alone for columns written once and never modified. +11c. Query columns: a column a query names but no row needs to carry, such as a generated column storing a folded title for a search index, a `tsvector`, a computed sort key or a large column a filter names, is declared on the type, not as a field: `@QueryColumn(field = "titleFolded", type = String.class)`, repeated per column; `column = "..."` names the column where the resolver does not, and `nullable = true` declares a nullable one. It generates `Movie_.titleFolded` with no record component, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well (`Credit_.movie.titleFolded`). Map such a column as a `@Persist(insertable = false, updatable = false)` field only when a caller reads its value. On an entity the database must fill a query column, since every insert leaves it out: generated, identity, with a default, or nullable. Schema validation reports one the database does not fill as an unmapped column. + 11c. Update tuning: `@DynamicUpdate` sets how the entity's UPDATE statements are generated. The default mode (`ENTITY`) skips the UPDATE entirely when nothing changed and writes the full row when anything did. `@DynamicUpdate(FIELD)` writes only the changed columns, at the cost of one SQL shape per distinct change pattern (which splits batches); `@DynamicUpdate(OFF)` always writes all columns without comparison. `dirtyCheck = VALUE` switches change detection from reference identity to `equals()`. Semantics, trade-offs, and the foreign-key comparison rule: /storm-repository-java (Dirty Checking and Update Suppression). 12. Java records are immutable. Consider Lombok `@Builder(toBuilder = true)` for copy-with-modification. diff --git a/website/static/skills/storm-entity-kotlin.md b/website/static/skills/storm-entity-kotlin.md index f14ca329f..b2cc9320e 100644 --- a/website/static/skills/storm-entity-kotlin.md +++ b/website/static/skills/storm-entity-kotlin.md @@ -166,6 +166,8 @@ Generation rules: ``` Use `insertable = false` alone for columns set by the database only on INSERT, or `updatable = false` alone for columns that are written once and never modified. +12c. Query columns: a column a query names but no row needs to carry, such as a generated column storing a folded title for a search index, a `tsvector`, a computed sort key or a large column a filter names, is declared on the type, not as a property: `@QueryColumn(field = "titleFolded", type = String::class)`, repeated per column; `column = "..."` names the column where the resolver does not, and `nullable = true` declares a nullable one. It generates `Movie_.titleFolded` with no constructor parameter, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well (`Credit_.movie.titleFolded`). Map such a column as a `@Persist(insertable = false, updatable = false)` property only when a caller reads its value. On an entity the database must fill a query column, since every insert leaves it out: generated, identity, with a default, or nullable. Schema validation reports one the database does not fill as an unmapped column. + 12c. Update tuning: `@DynamicUpdate` sets how the entity's UPDATE statements are generated. The default mode (`ENTITY`) skips the UPDATE entirely when nothing changed and writes the full row when anything did. `@DynamicUpdate(FIELD)` writes only the changed columns, at the cost of one SQL shape per distinct change pattern (which splits batches); `@DynamicUpdate(OFF)` always writes all columns without comparison. `dirtyCheck = VALUE` switches change detection from reference identity to `equals()`. Semantics, trade-offs, and the foreign-key comparison rule: /storm-repository-kotlin (Dirty Checking and Update Suppression). 13. Use descriptive variable names, never abbreviated. diff --git a/website/static/skills/storm-query-java.md b/website/static/skills/storm-query-java.md index a5e3f829e..caf178474 100644 --- a/website/static/skills/storm-query-java.md +++ b/website/static/skills/storm-query-java.md @@ -139,7 +139,7 @@ Operators: EQUALS, NOT_EQUALS, LESS_THAN, LESS_THAN_OR_EQUAL, GREATER_THAN, GREA **Rank search results with `orderFirst`, never with a `CASE` template.** `search` and `suggest` decide which rows match, not which come first. `orderFirst(predicate)` puts the rows matching the predicate first, appending a sort key as `orderBy` does, so the calls apply in order: `.orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input)).orderFirst(it -> it.where(Movie_.title, STARTS_WITH_IGNORE_CASE, input)).orderByDescending(Movie_.votes)` lists an exact title, then titles starting with the input, then the most popular. The operators match the input literally and ignoring case, as in `where`; combine columns with `.or(...)`. Rank only by columns of the queried or joined tables: a semi-joined table's columns are refused, as it adds no columns. A keyset `scroll` follows `orderFirst` (its keys lead the `Scrollable`'s sort fields and key, and the cursor carries the ranks), and a `Scrollable` may sort by a field of a joined table, such as the votes of a joined rating; `orderFirst` cannot order a `distinct` query. No index serves the sort, which is fine after a search's few matches. -**A search needs an index built for it, created in the migration with the query.** `search` and the `IGNORE_CASE` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `SqlFunction foldAccents = SqlFunction.of("fold_accents")` once and searches `it.search(input, foldAccents.apply(Movie_.title))`, ranking with `orderFirst(it -> it.where(foldAccents.apply(Movie_.title), EQUALS_IGNORE_CASE, input))`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add a generated column or an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. +**A search needs an index built for it, created in the migration with the query.** `search` and the `IGNORE_CASE` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `SqlFunction foldAccents = SqlFunction.of("fold_accents")` once and searches `it.search(input, foldAccents.apply(Movie_.title))`, ranking with `orderFirst(it -> it.where(foldAccents.apply(Movie_.title), EQUALS_IGNORE_CASE, input))`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Where the recheck of the accent function on every candidate row dominates, which shows on a column where a common word matches many thousands of rows, store the folded text in a generated column (`title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL`), index `LOWER(title_folded)`, and declare it as a query column on the entity, `@QueryColumn(field = "titleFolded", type = String.class)`, so no row carries it (see the entity skill). Fold the input once in the database, `SELECT fold_accents(?)`, and search the column with the result, `it.search(folded, Movie_.titleFolded)`, so the column and the input are folded by the same function. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. ### Naming a foreign key column diff --git a/website/static/skills/storm-query-kotlin.md b/website/static/skills/storm-query-kotlin.md index 67cd695f5..16b545d9e 100644 --- a/website/static/skills/storm-query-kotlin.md +++ b/website/static/skills/storm-query-kotlin.md @@ -104,7 +104,7 @@ User_.email.isNotNull() // IS_NOT_NULL **Rank search results with `orderFirst`, never with a `CASE` template.** `search` and `suggest` decide which rows match, not which come first. `orderFirst(predicate)` puts the rows matching the predicate first, appending a sort key as `orderBy` does, so the calls apply in order: `.orderFirst(Movie_.title eqIgnoreCase input).orderFirst(Movie_.title startsWithIgnoreCase input).orderByDescending(Movie_.votes)` lists an exact title, then titles starting with the input, then the most popular. The operators match the input literally and ignoring case, as in `where`; combine columns with `or`. Rank only by columns of the queried or joined tables: a semi-joined table's columns are refused, as it adds no columns. A keyset `scroll` follows `orderFirst` (its keys lead the `Scrollable`'s sort fields and key, and the cursor carries the ranks), and a `Scrollable` may sort by a field of a joined table, such as the votes of a joined rating; `orderFirst` cannot order a `distinct` query. No index serves the sort, which is fine after a search's few matches. -**A search needs an index built for it, created in the migration with the query.** `search` and the `IgnoreCase` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `val foldAccents = SqlFunction.of("fold_accents")` once, with an extension property that reads as a field of the column, `val Navigable.folded: Metamodel get() = foldAccents(this)`, and searches `search(input, Movie_.title.folded)`, ranking with `orderFirst(Movie_.title.folded eqIgnoreCase input)`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add a generated column or an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. +**A search needs an index built for it, created in the migration with the query.** `search` and the `IgnoreCase` text operators compare `LOWER(column) LIKE LOWER('%word%')`: a lowered expression with a leading wildcard, which no B-tree index serves. Without an index for it, each search reads and lowers every row of every table it names, which goes unnoticed on test data and takes seconds on a production-sized table. Where a searched table can grow beyond a few thousand rows, add the target database's substring index (a trigram or n-gram index) to the migration, over exactly `LOWER(column)`, one per searched column, including the columns of a semi-joined table or of a table reached through a path. An index on the plain column does not match the lowered expression. A word shorter than the index's gram (usually three characters) cannot use it: `search` tests it only on the rows its longer words found, and a search of short words alone still reads every row; the slow log (`st.orm.sql.slow`) adds advice to such a statement's line. Confirm in the database's query plan that each table is read through its index, for the statement as the application sends it: with the word as a bind parameter, after the driver has prepared it, not with the word written into the SQL. A prepared statement may be planned once for every value; that plan cannot see the word, and a search ordered by an indexed column with a small limit, such as a list of suggestions, then walks that column's index testing every row. Where that happens, configure the driver or database to plan each execution for its values. After creating the index on a table that holds data, refresh the table's statistics; where a rare word is still estimated to match many rows, raise the statistics sample of the indexed expression. Never speed a search up by changing its predicate: a case-sensitive operator reads every row as well and changes what matches, and an `ILIKE` or a `LOWER(...)` template reads every row just the same. Where the database has no substring index, say so, and narrow the rows with a selective predicate instead. Whether accents count is the schema's decision, never the application's: the search follows the column's collation, which ignores accents on MySQL and MariaDB by default and compares them elsewhere. Where a search must ignore them on such a database, have the database remove them through a function an index is built on, and name that function in the query with `SqlFunction`: the migration wraps the database's accent function in one marked immutable where it is not (on PostgreSQL, `CREATE FUNCTION fold_accents(text) RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT RETURN public.unaccent('public.unaccent'::regdictionary, $1)`) and indexes `LOWER(fold_accents(column))`; the repository declares `val foldAccents = SqlFunction.of("fold_accents")` once, with an extension property that reads as a field of the column, `val Navigable.folded: Metamodel get() = foldAccents(this)`, and searches `search(input, Movie_.title.folded)`, ranking with `orderFirst(Movie_.title.folded eqIgnoreCase input)`. A comparison applies the function to the column and to the input, so the database folds both and the application folds neither: never remove accents from the input in application code, never add an entity field holding the folded text, and never wrap the column in an accent function inside a template, which the search operators cannot use. Where the recheck of the accent function on every candidate row dominates, which shows on a column where a common word matches many thousands of rows, store the folded text in a generated column (`title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL`), index `LOWER(title_folded)`, and declare it as a query column on the entity, `@QueryColumn(field = "titleFolded", type = String::class)`, so no row carries it (see the entity skill). Fold the input once in the database, `SELECT fold_accents(?)`, and search the column with the result, `search(folded, Movie_.titleFolded)`, so the column and the input are folded by the same function. Name exactly the function the index is built on; another expression is not served by it. A `Scrollable` refuses a function call as key or sort field. Combine with `and`/`or`: ```kotlin diff --git a/website/static/skills/storm-validate.md b/website/static/skills/storm-validate.md index d6cf1b568..36350e247 100644 --- a/website/static/skills/storm-validate.md +++ b/website/static/skills/storm-validate.md @@ -13,7 +13,7 @@ Compare Storm entities against the live database schema. - Entities without tables - Column type mismatches (incompatible types are errors; narrowing like Long mapped to INTEGER is a warning) - Missing columns (entity field has no matching DB column) - - Extra columns that are NOT NULL without a default value (these would cause INSERT failures) + - Extra columns that are NOT NULL without a default value and not generated (these would cause INSERT failures). A column declared with `@QueryColumn` on an entity counts, since every insert leaves it out: it must be generated, an identity column, have a default or be nullable - FK/`@FK` mismatches (FK referencing wrong table is an error; missing FK constraint is a warning) - Nullability differences (entity non-null but DB nullable is a warning) - Missing unique constraints for @UK fields (use `uniqueConstraints` array from `describe_table`)