Declare query columns an entity does not carry

.github/workflows/ci.yml

@@ -79,46 +79,45 @@ jobs:
           flags: unittests
           fail_ci_if_error: true
 
-  # Known-vulnerability gate over the dependency graph Storm ships (compile and runtime scope): OWASP
-  # Dependency-Check against the NVD, failing on a High or Critical advisory. Test and provided scope are the
-  # application's own dependencies and stay out of it. The NVD API key is a repository secret; Dependabot runs and
-  # forks receive no secrets, so the scan is skipped there and reported as such.
+  # Known-vulnerability gate over the dependency graph Storm ships (compile and runtime scope): the CycloneDX
+  # bill of materials of the reactor, scanned by Trivy, failing on a High or Critical advisory. Test and provided
+  # scope are the application's own dependencies and stay out of it. Trivy carries its own advisory database and
+  # needs no key, so the gate runs on every PR, Dependabot's and forks' included.
   dependency-check:
     runs-on: ubuntu-latest
-    timeout-minutes: 40
-    env:
-      NVD_API_KEY: ${{ secrets.NVD_API_KEY }}
+    timeout-minutes: 20
     steps:
       - name: Checkout repository
         uses: actions/checkout@v7
 
       - name: Set up JDK 21
-        if: env.NVD_API_KEY != ''
         uses: actions/setup-java@v6
         with:
           distribution: 'temurin'
           java-version: '21'
-          # The Maven cache holds the NVD data the plugin keeps under ~/.m2 as well, so a run only pulls the
-          # advisories published since the last one.
           cache: 'maven'
 
-      - name: Scan the shipped dependency graph
-        if: env.NVD_API_KEY != ''
-        run: mvn -B -DskipTests -Dmaven.javadoc.skip=true -Dspotless.check.skip=true install org.owasp:dependency-check-maven:aggregate
+      - name: Write the bill of materials of the shipped dependency graph
+        run: mvn -B -DskipTests -Dmaven.javadoc.skip=true -Dspotless.check.skip=true package org.cyclonedx:cyclonedx-maven-plugin:makeAggregateBom
 
-      - name: Upload the Dependency-Check report
-        if: env.NVD_API_KEY != '' && always()
+      - name: Scan the bill of materials for known vulnerabilities
+        uses: aquasecurity/trivy-action@v0.36.0
+        with:
+          scan-type: sbom
+          scan-ref: target/bom.json
+          scanners: vuln
+          severity: HIGH,CRITICAL
+          exit-code: '1'
+
+      - name: Upload the bill of materials
+        if: always()
         uses: actions/upload-artifact@v7
         with:
-          name: dependency-check-report
-          path: target/dependency-check-report.*
+          name: dependency-bom
+          path: target/bom.json
           if-no-files-found: ignore
           retention-days: 30
 
-      - name: Report a skipped scan
-        if: env.NVD_API_KEY == ''
-        run: echo "::notice::Dependency-Check skipped - no NVD_API_KEY available to this run."
-
   # The CLI ships to users via npx but is plain JavaScript with no compile step,
   # so at minimum its syntax must be checked before merge.
   cli-check:

CHANGELOG.md

@@ -24,6 +24,7 @@ for the CLI, to [npm](https://www.npmjs.com/package/@storm-orm/cli)
 - `delete()` and `update()` refuse to run without a condition unless `unsafe()` is called, take explicit joins on every database, and refuse a joined inheritance hierarchy (#712). `SqlDialect.writeJoinSyntax()` replaces `supportsDeleteAlias()`.
 - Added `semiJoin` and `antiJoin`, which filter rows by a related table without adding its rows, so no `distinct` is needed (#690). `semiJoin<Pet>().on<Owner>()` keeps the owners with a pet matching the conditions that name `Pet`, `antiJoin` the owners without one. They filter `select()`, `delete()` and `update()`, and nest to filter through two to-many relations: `semiJoin<Pet>().on<Owner>().semiJoin<Visit>().on<Pet>()` (#717). A search over a row and its related rows reads each table through its own indexes: 1.3 s to a few milliseconds over 980,000 credits on PostgreSQL.
 - A typed join takes a condition: `on<Owner> { Pet_.name eq "Leo" }` in Kotlin, `on(Owner.class, it -> ...)` in Java (#690). On a left join it keeps the rows without a match, which the same condition in `where` drops.
+- Added query columns, the columns a query names but the entity does not carry (#743): `@QueryColumn(field = "titleFolded", type = String::class)` on an entity or projection generates `Movie_.titleFolded` as a metamodel field with no record component, so the column is never selected, inserted or updated, while `where`, `orderBy`, `orderFirst`, `search`, `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it, through paths as well. A generated column that stores a folded title for a search index is the typical case. Schema validation checks the column exists with a compatible type and, on an entity, reports a `NOT NULL` query column the database does not fill as unmapped (`UNMAPPED_COLUMN`), since every insert leaves it out. Schema validation reads whether a column is generated, and no longer reports a generated `NOT NULL` column no field maps as unmapped. `Model.queryColumns()` lists them.
 - Added `suggest`, the `search` for input typed as you go: it ignores words shorter than the database's substring index serves and matches nothing when no word is left, so it never reads every row. `search` tests such short words only on the rows its longer words found: "de niro" over 980,000 credits went from 2.3 s to 19 ms on PostgreSQL. The length comes from `SqlDialect.minimumSearchTermLength()`, 3 on PostgreSQL and 0 elsewhere, and `storm.search.min_term_length` overrides it. `suggest` also takes the words a search box ignores, such as `the` in a catalog of titles, which the index serves but which narrow nothing: `suggest(input, setOf("the", "a", "of"), Movie_.title)` treats them as short words, compared ignoring case (#741).
 - Added `orderFirst`, which sorts the rows matching a predicate first: `orderFirst(Movie_.title eqIgnoreCase input)` in Kotlin, `orderFirst(it -> it.where(Movie_.title, EQUALS_IGNORE_CASE, input))` in Java, rendered as `CASE WHEN ... THEN 0 ELSE 1 END` on every database. A keyset `scroll` follows it, and `Scrollable.sortBy` and `sortByDescending` take a field of any table in the query.
 - Added `SqlFunction`, a one-argument SQL function applied to a column wherever a query takes one, so a query can match the expression an index is built on (#694). `foldAccents(Movie_.title)` in Kotlin, `foldAccents.apply(Movie_.title)` in Java; a comparison applies it to the column and the value alike, so an accent-insensitive search on PostgreSQL uses an index on `LOWER(fold_accents(title))`.

docs/entities.md

@@ -689,6 +689,72 @@ record Pet(@PK Integer id,
 
 ---
 
+## Query Columns
+
+A query can only name a column that a field maps, and every field is selected, inserted and updated. Some columns exist only to be queried: a generated column that stores a derived form of another column, which a search matches and an index is built on, but which no caller reads and the database computes. Mapped as a field it sits in every SELECT, needs `@Persist(insertable = false, updatable = false)`, and makes every caller that constructs the entity supply a value the database overwrites. The same holds for a `tsvector` column for full-text search, a normalized or hashed copy of a column, a computed sort key, and a large column a query filters on but should not load.
+
+A query column is a column a query names but the entity does not carry. `@QueryColumn` on the entity or projection declares it with the name of its metamodel field and the type of its values. The processors generate the field as they generate one for a record component, and the column follows from the field name through the column name resolver; a `column` attribute names it where the resolver does not.
+
+<Tabs groupId="language">
+<TabItem value="kotlin" label="Kotlin" default>
+
+```kotlin
+@QueryColumn(field = "titleFolded", type = String::class)
+data class Movie(
+    @PK val id: Int = 0,
+    val title: String,
+    val year: Int
+) : Entity<Int>
+```
+
+The annotation repeats for each query column. `Movie_.titleFolded` is a `Metamodel<Movie, String>` and goes wherever a column does:
+
+```kotlin
+orm.entity(Movie::class)
+    .select()
+    .where(search(input, Movie_.titleFolded))
+    .orderBy(Movie_.titleFolded)
+    .resultList
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+@QueryColumn(field = "titleFolded", type = String.class)
+record Movie(@PK Integer id, String title, Integer year) implements Entity<Integer> {}
+```
+
+The annotation repeats for each query column. `Movie_.titleFolded` is a `Metamodel<Movie, String>` and goes wherever a column does:
+
+```java
+orm.entity(Movie.class)
+    .select()
+    .where(it -> it.search(input, Movie_.titleFolded))
+    .orderBy(Movie_.titleFolded)
+    .getResultList();
+```
+
+</TabItem>
+</Tabs>
+
+The migration creates the column, here generated from the title by the accent-folding function of [Storing the Folded Text](search.md#storing-the-folded-text), and the index on it:
+
+```sql
+ALTER TABLE movie ADD COLUMN title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL;
+CREATE INDEX idx_movie_title_folded ON movie USING gin (LOWER(title_folded) gin_trgm_ops);
+```
+
+- A query column is never selected, inserted or updated. A record holds no value for it: `Metamodel.getValue` throws, and `update().set` refuses it.
+- A column is carried by a field or named by a query column, not both. A query column whose column a field maps is refused when the model is built, as is one named after a record component.
+- It is reached through paths like any other field: a query column declared on `Movie` is `Credit_.movie.titleFolded` from a query on credits. Projections of the same table declare their own.
+- `where`, `orderBy`, `orderFirst`, `search` and `suggest`, a `SqlFunction`, a keyset sort and a template placeholder take it. A keyset sort refuses a nullable query column, as it refuses a nullable field; `nullable = true` declares one.
+- Schema validation checks that the column exists with a compatible type. On an entity every insert leaves it out, so the database must fill it: a `NOT NULL` query column that is not generated, not an identity column and has no default is reported as an [unmapped column](validation.md) is.
+
+Where a caller reads the value, map the column as a field with `@Persist(insertable = false, updatable = false)` instead: a query column is for a column no row needs to carry.
+
+---
+
 ## Modifying Entities
 
 Since Storm entities are immutable, updating a field means creating a new instance with the changed value. Kotlin data classes have a built-in `copy()` method for this. Java records do not provide an equivalent, but Lombok's `@Builder(toBuilder = true)` annotation generates a builder that copies all fields from an existing instance:

docs/metamodel.md

@@ -374,7 +374,7 @@ A direct (non-`Ref`) self-referential foreign key is rejected at generation time
 
 ## Generated Code
 
-Understanding the generated code helps when debugging or reading compiler errors. The metamodel mirrors your entity structure, creating a static field for each entity field. Each field carries generic type parameters that encode both the root entity type and the field's value type, which is how the compiler enforces type safety in queries.
+Understanding the generated code helps when debugging or reading compiler errors. The metamodel mirrors your entity structure, creating a static field for each entity field, and one for each [query column](entities.md#query-columns) the entity declares. Each field carries generic type parameters that encode both the root entity type and the field's value type, which is how the compiler enforces type safety in queries.
 
 ```
 Entity                              Metamodel

docs/search.md

@@ -308,3 +308,51 @@ The search compares `LOWER(fold_accents(m.title)) LIKE LOWER(fold_accents(?))`,
 - Name the function exactly as the index does. A different function, or the same one applied in another order, is another expression, and the plan falls back to reading every row.
 - The database folds the input too, with its own rules. PostgreSQL's `unaccent` maps `ø` to `o` and `æ` to `ae`, which removing combining marks in the application would leave alone; folding both sides in the database keeps the column and the input in step.
 - The function receives the search pattern, wildcards included, and an accent function leaves them alone, as a function compared through a text operator must.
+
+### Storing the Folded Text
+
+The expression index is enough for most catalogs. The index finds the candidate rows, and the database then rechecks each candidate by computing `fold_accents` from the title; on a column where a common word matches many thousands of rows per keystroke, that recheck is where the time goes. Where a search stays slow for that reason, and only then, store the folded title instead of computing it per search: a generated column holds `fold_accents(title)`, the index is built on that column, and the entity declares it as a [query column](entities.md#query-columns), a column the search names and no row carries.
+
+```sql
+ALTER TABLE movie ADD COLUMN title_folded VARCHAR GENERATED ALWAYS AS (fold_accents(title)) STORED NOT NULL;
+CREATE INDEX idx_movie_title_folded ON movie USING gin (LOWER(title_folded) gin_trgm_ops);
+```
+
+The search then compares `LOWER(m.title_folded) LIKE LOWER(?)`, the expression the index holds, and the recheck reads the stored column. The input must be folded by the same function, in the database, before the search; applying `foldAccents` to the query column instead would fold both sides again and bring the recheck back.
+
+<Tabs groupId="language">
+<TabItem value="kotlin" label="Kotlin" default>
+
+```kotlin
+@QueryColumn(field = "titleFolded", type = String::class)
+data class Movie(...) : Entity<Int>
+
+val folded = orm.query { "SELECT fold_accents($input)" }.singleResult<String>()
+
+orm.entity(Movie::class)
+    .select()
+    .where(search(folded, Movie_.titleFolded))
+    .orderFirst(Movie_.titleFolded eqIgnoreCase folded)
+    .resultList
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+@QueryColumn(field = "titleFolded", type = String.class)
+record Movie(...) implements Entity<Integer> {}
+
+String folded = orm.query(RAW."SELECT fold_accents(\{input})").getSingleResult(String.class);
+
+orm.entity(Movie.class)
+    .select()
+    .where(it -> it.search(folded, Movie_.titleFolded))
+    .orderFirst(it -> it.where(Movie_.titleFolded, EQUALS_IGNORE_CASE, folded))
+    .getResultList();
+```
+
+</TabItem>
+</Tabs>
+
+The entity keeps one field for the title, which the query selects with its accents for display; the folded column stays out of every SELECT and INSERT, and the writes pay for the folding once per row.

docs/validation.md

@@ -80,6 +80,7 @@ Schema validation compares your entity and projection definitions against the ac
 | Entity declares `@PK` but the database has no primary key constraint | `PRIMARY_KEY_MISSING` | Warning |
 | `@UK` field has a matching unique constraint in the database | `UNIQUE_KEY_MISSING` | Warning |
 | `@FK` field has a matching foreign key constraint in the database | `FOREIGN_KEY_MISSING` | Warning |
+| A `NOT NULL` column without a default that no field maps, and that the database does not generate; a [query column](entities.md#query-columns) of an entity counts, since every insert leaves it out | `UNMAPPED_COLUMN` | Warning |
 
 **Errors** indicate definitive mismatches that will cause runtime failures, such as missing tables or columns.
 

pom.xml

@@ -265,26 +265,18 @@
                     </executions>
                 </plugin>
                 <plugin>
-                    <!-- The known-vulnerability gate CI runs over the dependency graph Storm ships: compile and
-                         runtime scope. Test and provided scope (Spring Boot, Jackson, the drivers) are the
-                         application's own dependencies and stay out of the gate. Invoked explicitly, never bound
-                         to a phase: mvn org.owasp:dependency-check-maven:aggregate, with the NVD API key in
-                         NVD_API_KEY. -->
-                    <groupId>org.owasp</groupId>
-                    <artifactId>dependency-check-maven</artifactId>
-                    <version>13.0.0</version>
+                    <!-- The software bill of materials of the dependency graph Storm ships, which the
+                         known-vulnerability gate in CI scans: compile and runtime scope. Test and provided scope
+                         (Spring Boot, Jackson, the drivers) are the application's own dependencies and stay out
+                         of it. Invoked explicitly, never bound to a phase:
+                         mvn package org.cyclonedx:cyclonedx-maven-plugin:makeAggregateBom. -->
+                    <groupId>org.cyclonedx</groupId>
+                    <artifactId>cyclonedx-maven-plugin</artifactId>
+                    <version>2.9.1</version>
                     <configuration>
-                        <!-- CVSS 7 is the floor of High. -->
-                        <failBuildOnCVSS>7</failBuildOnCVSS>
-                        <skipTestScope>true</skipTestScope>
-                        <skipProvidedScope>true</skipProvidedScope>
-                        <nvdApiKeyEnvironmentVariable>NVD_API_KEY</nvdApiKeyEnvironmentVariable>
-                        <formats>
-                            <format>HTML</format>
-                            <format>JSON</format>
-                        </formats>
-                        <!-- The .NET analyzer needs the dotnet tool and only warns without it. -->
-                        <assemblyAnalyzerEnabled>false</assemblyAnalyzerEnabled>
+                        <includeTestScope>false</includeTestScope>
+                        <includeProvidedScope>false</includeProvidedScope>
+                        <outputFormat>json</outputFormat>
                     </configuration>
                 </plugin>
             </plugins>

storm-core/src/main/java/st/orm/core/template/Column.java

@@ -24,7 +24,8 @@
  */
 public interface Column {
     /**
-     * Gets the 1-based index of the column.
+     * Gets the 1-based index of the column among the model's columns, or {@code 0} for a query column, which has no
+     * position among them.
      *
      * @return the column index.
      */

storm-core/src/main/java/st/orm/core/template/Model.java

@@ -110,6 +110,20 @@ public interface Model<E extends Data, ID> {
      */
     List<Column> declaredColumns();
 
+    /**
+     * Returns the query columns declared on this model: the columns a query names but the record does not carry,
+     * declared with {@link st.orm.QueryColumn}. They are not among {@link #columns()} or {@link #declaredColumns()},
+     * so a select, insert or update never names them; {@link #getColumns(Metamodel)} resolves them, for a query
+     * column of this type and for one of a type reached through an expanded relation.
+     *
+     * <p>A query column has no record field: its {@link Column#index()} is {@code 0}, and it is neither insertable
+     * nor updatable.</p>
+     *
+     * @return the query columns declared on this model, in declaration order.
+     * @since 1.16
+     */
+    List<Column> queryColumns();
+
     /**
      * Returns the record type metadata.
      *