diff --git a/.changeset/eql-v4-ordered-ope-types.md b/.changeset/eql-v4-ordered-ope-types.md new file mode 100644 index 000000000..27a8fb5a7 --- /dev/null +++ b/.changeset/eql-v4-ordered-ope-types.md @@ -0,0 +1,5 @@ +--- +'@cipherstash/eql': minor +--- + +**Stack Encrypt produces the OPE-ordered EQL v4 types `IntegerOrd`, `SmallintOrd`, `BigintOrd`, `RealOrd`, `DoubleOrd`, `TextOrd` and their `OrdOpe` twins** (`eql-bindings`, `stack-encrypt` feature). Their `op` term orders by ADR-0002's order key, so `<`, `<=`, `>`, `>=` and `ORDER BY eql_v4.ord_term(col)` compare integers by value, floats as PostgreSQL's `float8` does and text by its fold (case and accents folded). `=` on text reads `TextEq`'s equality term. `TextSearch` stays unproducible, with a reason: its query carries a match term that short text cannot make. diff --git a/languages/golang/CHANGELOG.md b/languages/golang/CHANGELOG.md index c0c98d873..6a5a03e8e 100644 --- a/languages/golang/CHANGELOG.md +++ b/languages/golang/CHANGELOG.md @@ -88,6 +88,12 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`. type, range checked as for `Integer`) tagged `encrypt_into=` one of them seals at 16 bits; `SmallintEq` gets `Eq`, with the same equality term as an integer of the same value under the same column. +- **`IntegerOrd`, `SmallintOrd`, `BigintOrd`, `RealOrd`, `DoubleOrd`, + `TextOrd` and their `OrdOpe` twins**: a field tagged `encrypt_into=` one of them gets `Lt`, + `Lte`, `Gt` and `Gte`, and a text one `Eq` too. They store an OPE term + that orders by ADR-0002's order key: floats as PostgreSQL's `float8` + sorts them, text by its fold. No `Between` yet: it needs EQL v4's range + query domains. - **Ordering methods on text.** A string field with `index=ore` or `index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its folded form (ADR-0002: case and accents folded, under Unicode 16), so diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 40dd3753a..3462fa1c2 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -158,15 +158,16 @@ The engine produces these EQL types today, each from one Go field type: | `encrypt_into` | Go field | |---|---| -| `TextEq`, `Text`, `TextMatch` | `string` | -| `Smallint`, `SmallintEq` | `int16`, or any Go integer type | -| `Integer`, `IntegerEq` | `int32`, or any Go integer type | -| `Bigint`, `BigintEq` | `int64`, or any Go integer type | +| `TextEq`, `Text`, `TextMatch`, `TextOrd`, `TextOrdOpe` | `string` | +| `Smallint`, `SmallintEq`, `SmallintOrd`, `SmallintOrdOpe` | `int16`, or any Go integer type | +| `Integer`, `IntegerEq`, `IntegerOrd`, `IntegerOrdOpe` | `int32`, or any Go integer type | +| `Bigint`, `BigintEq`, `BigintOrd`, `BigintOrdOpe` | `int64`, or any Go integer type | | `Boolean` | `bool` | -| `Real`, `RealEq` | `float32` | -| `Double`, `DoubleEq` | `float64` | +| `Real`, `RealEq`, `RealOrd`, `RealOrdOpe` | `float32` | +| `Double`, `DoubleEq`, `DoubleOrd`, `DoubleOrdOpe` | `float64` | -The `Eq` types answer `Eq`; `Text`, `Smallint`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double` are stored and read only, with no query method. +The `Eq` types answer `Eq`; the `Ord` and `OrdOpe` types answer `Lt`, `Lte`, `Gt` and `Gte`, and on text `Eq` too; `Text`, `Smallint`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double` are stored and read only, with no query method. +An ordered type has no `Between` method yet, because a `BETWEEN` operand needs EQL v4's range query domains. Any built-in Go integer field takes an integer type (ADR-0002) and seals at the type's width: an `int16` field always fits an `Integer`, and an `int64` field is range checked value by value, `Encrypt` refusing a value that does not fit with `encrypt.ErrEncoding`. `encrypt.Int128` and `encrypt.Uint128` are not converted. A float's width must match: a `float64` field is not a `Real`. A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too. diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index 603dd025a..e681bfd04 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -475,18 +475,24 @@ The EQL types the engine produces, and the Go field types each takes: |---|---|---| | `TextEq` | `string` | `=`, `<>` | | `TextMatch` | `string` | `@@` (substring match, 3-grams, case-insensitive) | +| `TextOrd`, `TextOrdOpe` | `string` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | | `Text` | `string` | none: stored and read only | | `Smallint` | `int16`, or any Go integer type (see below) | none | | `SmallintEq` | the same | `=`, `<>` | +| `SmallintOrd`, `SmallintOrdOpe` | the same | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | | `Integer` | `int32`, or any Go integer type (see below) | none | | `IntegerEq` | the same | `=`, `<>` | +| `IntegerOrd`, `IntegerOrdOpe` | the same | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | | `Bigint` | `int64`, or any Go integer type | none | | `BigintEq` | the same | `=`, `<>` | +| `BigintOrd`, `BigintOrdOpe` | the same | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | | `Boolean` | `bool` | none | | `Real` | `float32` | none | | `RealEq` | `float32` | `=`, `<>` | +| `RealOrd`, `RealOrdOpe` | `float32` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | | `Double` | `float64` | none | | `DoubleEq` | `float64` | `=`, `<>` | +| `DoubleOrd`, `DoubleOrdOpe` | `float64` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | Any built-in Go integer field takes an integer EQL type (ADR-0002): the field is sealed at the type's width, and opens back into the field's type. A type @@ -502,8 +508,25 @@ A `TextMatch` value must be at least three characters long: shorter text has no 3-grams to index and is refused. The number equality types compare as PostgreSQL compares the plain types: an integer equals itself whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN`; the stored value still -decrypts to exactly what was written. The rest of -the catalog (ordering, dates, decimals, JSON) is listed in +decrypts to exactly what was written. + +The `Ord` and `OrdOpe` types store an OPE term, which EQL compares as bytes, +and order by ADR-0002's order key: integers by value, floats as PostgreSQL's +`float8` sorts them (`-0.0` ties `0.0`, every NaN is one value after +`+Infinity`), text by its fold (case and accents folded, under Unicode 16). +A field gets `Lt`, `Lte`, `Gt` and `Gte`, and a text field `Eq` too; sort +with `ORDER BY eql_v4.ord_term(col)`. `ord_term` returns a `bytea` domain, so +a plain functional btree index, `CREATE INDEX ON t (eql_v4.ord_term(col))`, +serves `<`, `>` and `ORDER BY` with no operator class. That is why these +types work on an install without superuser rights, where the ORE operator +class is skipped. On a number column SQL's `=` compares +the order term; stashgen writes no `Eq` method for one yet, since the engine +answers `Eq` through an equality term only. A text order type also stores +`TextEq`'s equality term, and `=` reads that, so `Café` and `cafe` are +neither `<` nor `>` each other and not `=` either. Text holding a code point +Unicode 16 does not assign has no order term and is refused. There is no +`Between` method yet: a `BETWEEN` needs EQL v4's range query domains. The rest of +the catalog (ORE ordering, dates, decimals, JSON) is listed in `eql.Types` with the reason it is not produced yet, and `stashgen` refuses it with that reason. The guest builds the value: the generated code imports `encrypt/eql`, which embeds the build of the engine that holds the diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index d463252d3..05c485dbc 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -28,7 +28,9 @@ // also takes any other built-in Go integer type, sealed at the type's width: // one every value of which fits it (an int16 into Integer) always seals, and // a wider one (an int64 into Integer) is range checked, a value that does not -// fit refused with [ErrEncoding]. The *Eq types answer Eq; Smallint, +// fit refused with [ErrEncoding]. The *Eq types answer Eq; the *Ord and +// *OrdOpe types (IntegerOrd, SmallintOrd, BigintOrd, RealOrd, DoubleOrd, +// TextOrd and their OrdOpe twins) answer Lt, Lte, Gt and Gte, and Eq too on text; Smallint, // Integer, Bigint, Boolean, Real, Double and Text are stored and read only. // // 3. Run the generator. It writes user_stash.go beside the struct: diff --git a/languages/golang/encrypt/eql/eql_gen.go b/languages/golang/encrypt/eql/eql_gen.go index 6942ed5f2..d1da7b798 100644 --- a/languages/golang/encrypt/eql/eql_gen.go +++ b/languages/golang/encrypt/eql/eql_gen.go @@ -12,18 +12,18 @@ var Types = []Type{ {Name: "Integer", GoName: "Integer", Family: "integer", Suffix: "", Plaintext: "int32", SQLDomain: "public.eql_v4_integer", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "IntegerEq", GoName: "IntegerEq", Family: "integer", Suffix: "Eq", Plaintext: "int32", SQLDomain: "public.eql_v4_integer_eq", Indexes: []string{"eq"}, Query: "IntegerEqQuery", QuerySQLDomain: "eql_v4.query_integer_eq", Producible: true, Reason: ""}, {Name: "IntegerOrdOre", GoName: "IntegerOrdOre", Family: "integer", Suffix: "OrdOre", Plaintext: "int32", SQLDomain: "public.eql_v4_integer_ord_ore", Indexes: []string{"ore"}, Query: "IntegerOrdOreQuery", QuerySQLDomain: "eql_v4.query_integer_ord_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, - {Name: "IntegerOrd", GoName: "IntegerOrd", Family: "integer", Suffix: "Ord", Plaintext: "int32", SQLDomain: "public.eql_v4_integer_ord", Indexes: []string{"ope"}, Query: "IntegerOrdQuery", QuerySQLDomain: "eql_v4.query_integer_ord", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, - {Name: "IntegerOrdOpe", GoName: "IntegerOrdOpe", Family: "integer", Suffix: "OrdOpe", Plaintext: "int32", SQLDomain: "public.eql_v4_integer_ord_ope", Indexes: []string{"ope"}, Query: "IntegerOrdOpeQuery", QuerySQLDomain: "eql_v4.query_integer_ord_ope", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, + {Name: "IntegerOrd", GoName: "IntegerOrd", Family: "integer", Suffix: "Ord", Plaintext: "int32", SQLDomain: "public.eql_v4_integer_ord", Indexes: []string{"ope"}, Query: "IntegerOrdQuery", QuerySQLDomain: "eql_v4.query_integer_ord", Producible: true, Reason: ""}, + {Name: "IntegerOrdOpe", GoName: "IntegerOrdOpe", Family: "integer", Suffix: "OrdOpe", Plaintext: "int32", SQLDomain: "public.eql_v4_integer_ord_ope", Indexes: []string{"ope"}, Query: "IntegerOrdOpeQuery", QuerySQLDomain: "eql_v4.query_integer_ord_ope", Producible: true, Reason: ""}, {Name: "Smallint", GoName: "Smallint", Family: "smallint", Suffix: "", Plaintext: "int16", SQLDomain: "public.eql_v4_smallint", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "SmallintEq", GoName: "SmallintEq", Family: "smallint", Suffix: "Eq", Plaintext: "int16", SQLDomain: "public.eql_v4_smallint_eq", Indexes: []string{"eq"}, Query: "SmallintEqQuery", QuerySQLDomain: "eql_v4.query_smallint_eq", Producible: true, Reason: ""}, {Name: "SmallintOrdOre", GoName: "SmallintOrdOre", Family: "smallint", Suffix: "OrdOre", Plaintext: "int16", SQLDomain: "public.eql_v4_smallint_ord_ore", Indexes: []string{"ore"}, Query: "SmallintOrdOreQuery", QuerySQLDomain: "eql_v4.query_smallint_ord_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, - {Name: "SmallintOrd", GoName: "SmallintOrd", Family: "smallint", Suffix: "Ord", Plaintext: "int16", SQLDomain: "public.eql_v4_smallint_ord", Indexes: []string{"ope"}, Query: "SmallintOrdQuery", QuerySQLDomain: "eql_v4.query_smallint_ord", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, - {Name: "SmallintOrdOpe", GoName: "SmallintOrdOpe", Family: "smallint", Suffix: "OrdOpe", Plaintext: "int16", SQLDomain: "public.eql_v4_smallint_ord_ope", Indexes: []string{"ope"}, Query: "SmallintOrdOpeQuery", QuerySQLDomain: "eql_v4.query_smallint_ord_ope", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, + {Name: "SmallintOrd", GoName: "SmallintOrd", Family: "smallint", Suffix: "Ord", Plaintext: "int16", SQLDomain: "public.eql_v4_smallint_ord", Indexes: []string{"ope"}, Query: "SmallintOrdQuery", QuerySQLDomain: "eql_v4.query_smallint_ord", Producible: true, Reason: ""}, + {Name: "SmallintOrdOpe", GoName: "SmallintOrdOpe", Family: "smallint", Suffix: "OrdOpe", Plaintext: "int16", SQLDomain: "public.eql_v4_smallint_ord_ope", Indexes: []string{"ope"}, Query: "SmallintOrdOpeQuery", QuerySQLDomain: "eql_v4.query_smallint_ord_ope", Producible: true, Reason: ""}, {Name: "Bigint", GoName: "Bigint", Family: "bigint", Suffix: "", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "BigintEq", GoName: "BigintEq", Family: "bigint", Suffix: "Eq", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_eq", Indexes: []string{"eq"}, Query: "BigintEqQuery", QuerySQLDomain: "eql_v4.query_bigint_eq", Producible: true, Reason: ""}, {Name: "BigintOrdOre", GoName: "BigintOrdOre", Family: "bigint", Suffix: "OrdOre", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord_ore", Indexes: []string{"ore"}, Query: "BigintOrdOreQuery", QuerySQLDomain: "eql_v4.query_bigint_ord_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, - {Name: "BigintOrd", GoName: "BigintOrd", Family: "bigint", Suffix: "Ord", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord", Indexes: []string{"ope"}, Query: "BigintOrdQuery", QuerySQLDomain: "eql_v4.query_bigint_ord", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, - {Name: "BigintOrdOpe", GoName: "BigintOrdOpe", Family: "bigint", Suffix: "OrdOpe", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord_ope", Indexes: []string{"ope"}, Query: "BigintOrdOpeQuery", QuerySQLDomain: "eql_v4.query_bigint_ord_ope", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, + {Name: "BigintOrd", GoName: "BigintOrd", Family: "bigint", Suffix: "Ord", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord", Indexes: []string{"ope"}, Query: "BigintOrdQuery", QuerySQLDomain: "eql_v4.query_bigint_ord", Producible: true, Reason: ""}, + {Name: "BigintOrdOpe", GoName: "BigintOrdOpe", Family: "bigint", Suffix: "OrdOpe", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord_ope", Indexes: []string{"ope"}, Query: "BigintOrdOpeQuery", QuerySQLDomain: "eql_v4.query_bigint_ord_ope", Producible: true, Reason: ""}, {Name: "Date", GoName: "Date", Family: "date", Suffix: "", Plaintext: "", SQLDomain: "public.eql_v4_date", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, {Name: "DateEq", GoName: "DateEq", Family: "date", Suffix: "Eq", Plaintext: "", SQLDomain: "public.eql_v4_date_eq", Indexes: []string{"eq"}, Query: "DateEqQuery", QuerySQLDomain: "eql_v4.query_date_eq", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, {Name: "DateOrdOre", GoName: "DateOrdOre", Family: "date", Suffix: "OrdOre", Plaintext: "", SQLDomain: "public.eql_v4_date_ord_ore", Indexes: []string{"ore"}, Query: "DateOrdOreQuery", QuerySQLDomain: "eql_v4.query_date_ord_ore", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, @@ -43,27 +43,27 @@ var Types = []Type{ {Name: "TextEq", GoName: "TextEq", Family: "text", Suffix: "Eq", Plaintext: "string", SQLDomain: "public.eql_v4_text_eq", Indexes: []string{"eq"}, Query: "TextEqQuery", QuerySQLDomain: "eql_v4.query_text_eq", Producible: true, Reason: ""}, {Name: "TextMatch", GoName: "TextMatch", Family: "text", Suffix: "Match", Plaintext: "string", SQLDomain: "public.eql_v4_text_match", Indexes: []string{"match"}, Query: "TextMatchQuery", QuerySQLDomain: "eql_v4.query_text_match", Producible: true, Reason: ""}, {Name: "TextOrdOre", GoName: "TextOrdOre", Family: "text", Suffix: "OrdOre", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord_ore", Indexes: []string{"eq", "ore"}, Query: "TextOrdOreQuery", QuerySQLDomain: "eql_v4.query_text_ord_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, - {Name: "TextOrd", GoName: "TextOrd", Family: "text", Suffix: "Ord", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord", Indexes: []string{"eq", "ope"}, Query: "TextOrdQuery", QuerySQLDomain: "eql_v4.query_text_ord", Producible: false, Reason: "no EQL type is built from the engine's text order terms yet (#1131)"}, - {Name: "TextOrdOpe", GoName: "TextOrdOpe", Family: "text", Suffix: "OrdOpe", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord_ope", Indexes: []string{"eq", "ope"}, Query: "TextOrdOpeQuery", QuerySQLDomain: "eql_v4.query_text_ord_ope", Producible: false, Reason: "no EQL type is built from the engine's text order terms yet (#1131)"}, + {Name: "TextOrd", GoName: "TextOrd", Family: "text", Suffix: "Ord", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord", Indexes: []string{"eq", "ope"}, Query: "TextOrdQuery", QuerySQLDomain: "eql_v4.query_text_ord", Producible: true, Reason: ""}, + {Name: "TextOrdOpe", GoName: "TextOrdOpe", Family: "text", Suffix: "OrdOpe", Plaintext: "string", SQLDomain: "public.eql_v4_text_ord_ope", Indexes: []string{"eq", "ope"}, Query: "TextOrdOpeQuery", QuerySQLDomain: "eql_v4.query_text_ord_ope", Producible: true, Reason: ""}, {Name: "TextSearchOre", GoName: "TextSearchOre", Family: "text", Suffix: "SearchOre", Plaintext: "string", SQLDomain: "public.eql_v4_text_search_ore", Indexes: []string{"eq", "ore", "match"}, Query: "TextSearchOreQuery", QuerySQLDomain: "eql_v4.query_text_search_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, - {Name: "TextSearch", GoName: "TextSearch", Family: "text", Suffix: "Search", Plaintext: "string", SQLDomain: "public.eql_v4_text_search", Indexes: []string{"eq", "ope", "match"}, Query: "TextSearchQuery", QuerySQLDomain: "eql_v4.query_text_search", Producible: false, Reason: "no EQL type is built from the engine's text order terms yet (#1131)"}, + {Name: "TextSearch", GoName: "TextSearch", Family: "text", Suffix: "Search", Plaintext: "string", SQLDomain: "public.eql_v4_text_search", Indexes: []string{"eq", "ope", "match"}, Query: "TextSearchQuery", QuerySQLDomain: "eql_v4.query_text_search", Producible: false, Reason: "every query of this type carries a match term, which text shorter than one 3-gram cannot make, so `=` or `<` on short text would be refused; how a query that is not a match leaves it out is not decided (#1131)"}, {Name: "Boolean", GoName: "Boolean", Family: "boolean", Suffix: "", Plaintext: "bool", SQLDomain: "public.eql_v4_boolean", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "Real", GoName: "Real", Family: "real", Suffix: "", Plaintext: "float32", SQLDomain: "public.eql_v4_real", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "RealEq", GoName: "RealEq", Family: "real", Suffix: "Eq", Plaintext: "float32", SQLDomain: "public.eql_v4_real_eq", Indexes: []string{"eq"}, Query: "RealEqQuery", QuerySQLDomain: "eql_v4.query_real_eq", Producible: true, Reason: ""}, {Name: "RealOrdOre", GoName: "RealOrdOre", Family: "real", Suffix: "OrdOre", Plaintext: "float32", SQLDomain: "public.eql_v4_real_ord_ore", Indexes: []string{"ore"}, Query: "RealOrdOreQuery", QuerySQLDomain: "eql_v4.query_real_ord_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, - {Name: "RealOrd", GoName: "RealOrd", Family: "real", Suffix: "Ord", Plaintext: "float32", SQLDomain: "public.eql_v4_real_ord", Indexes: []string{"ope"}, Query: "RealOrdQuery", QuerySQLDomain: "eql_v4.query_real_ord", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, - {Name: "RealOrdOpe", GoName: "RealOrdOpe", Family: "real", Suffix: "OrdOpe", Plaintext: "float32", SQLDomain: "public.eql_v4_real_ord_ope", Indexes: []string{"ope"}, Query: "RealOrdOpeQuery", QuerySQLDomain: "eql_v4.query_real_ord_ope", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, + {Name: "RealOrd", GoName: "RealOrd", Family: "real", Suffix: "Ord", Plaintext: "float32", SQLDomain: "public.eql_v4_real_ord", Indexes: []string{"ope"}, Query: "RealOrdQuery", QuerySQLDomain: "eql_v4.query_real_ord", Producible: true, Reason: ""}, + {Name: "RealOrdOpe", GoName: "RealOrdOpe", Family: "real", Suffix: "OrdOpe", Plaintext: "float32", SQLDomain: "public.eql_v4_real_ord_ope", Indexes: []string{"ope"}, Query: "RealOrdOpeQuery", QuerySQLDomain: "eql_v4.query_real_ord_ope", Producible: true, Reason: ""}, {Name: "Double", GoName: "Double", Family: "double", Suffix: "", Plaintext: "float64", SQLDomain: "public.eql_v4_double", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "DoubleEq", GoName: "DoubleEq", Family: "double", Suffix: "Eq", Plaintext: "float64", SQLDomain: "public.eql_v4_double_eq", Indexes: []string{"eq"}, Query: "DoubleEqQuery", QuerySQLDomain: "eql_v4.query_double_eq", Producible: true, Reason: ""}, {Name: "DoubleOrdOre", GoName: "DoubleOrdOre", Family: "double", Suffix: "OrdOre", Plaintext: "float64", SQLDomain: "public.eql_v4_double_ord_ore", Indexes: []string{"ore"}, Query: "DoubleOrdOreQuery", QuerySQLDomain: "eql_v4.query_double_ord_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, - {Name: "DoubleOrd", GoName: "DoubleOrd", Family: "double", Suffix: "Ord", Plaintext: "float64", SQLDomain: "public.eql_v4_double_ord", Indexes: []string{"ope"}, Query: "DoubleOrdQuery", QuerySQLDomain: "eql_v4.query_double_ord", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, - {Name: "DoubleOrdOpe", GoName: "DoubleOrdOpe", Family: "double", Suffix: "OrdOpe", Plaintext: "float64", SQLDomain: "public.eql_v4_double_ord_ope", Indexes: []string{"ope"}, Query: "DoubleOrdOpeQuery", QuerySQLDomain: "eql_v4.query_double_ord_ope", Producible: false, Reason: "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)"}, + {Name: "DoubleOrd", GoName: "DoubleOrd", Family: "double", Suffix: "Ord", Plaintext: "float64", SQLDomain: "public.eql_v4_double_ord", Indexes: []string{"ope"}, Query: "DoubleOrdQuery", QuerySQLDomain: "eql_v4.query_double_ord", Producible: true, Reason: ""}, + {Name: "DoubleOrdOpe", GoName: "DoubleOrdOpe", Family: "double", Suffix: "OrdOpe", Plaintext: "float64", SQLDomain: "public.eql_v4_double_ord_ope", Indexes: []string{"ope"}, Query: "DoubleOrdOpeQuery", QuerySQLDomain: "eql_v4.query_double_ord_ope", Producible: true, Reason: ""}, {Name: "SteVecDocument", GoName: "SteVecDocument", Family: "json", Suffix: "Search", Plaintext: "", SQLDomain: "public.eql_v4_json_search", Indexes: []string{"json"}, Query: "SteVecQuery", QuerySQLDomain: "eql_v4.query_json", Producible: false, Reason: "the JSON index is a new operation in the engine (#1060)"}, {Name: "Json", GoName: "JSON", Family: "json", Suffix: "", Plaintext: "", SQLDomain: "public.eql_v4_json", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: false, Reason: "a JSON value seals as a tree of leaves and EQL's `c` holds one; how a document becomes one sealed leaf is not decided"}, } // Integer is the EQL type stored in public.eql_v4_integer: the integer family with no index. -// The engine produces it from a int32. +// The engine produces it from an int32. type Integer []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -87,7 +87,7 @@ func (v *Integer) UnmarshalJSON(b []byte) error { } // IntegerEq is the EQL type stored in public.eql_v4_integer_eq: the integer family with the eq index. -// The engine produces it from a int32. +// The engine produces it from an int32. type IntegerEq []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -181,7 +181,7 @@ func (v *IntegerOrdOreQuery) UnmarshalJSON(b []byte) error { } // IntegerOrd is the EQL type stored in public.eql_v4_integer_ord: the integer family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from an int32. type IntegerOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -228,7 +228,7 @@ func (v *IntegerOrdQuery) UnmarshalJSON(b []byte) error { } // IntegerOrdOpe is the EQL type stored in public.eql_v4_integer_ord_ope: the integer family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from an int32. type IntegerOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -275,7 +275,7 @@ func (v *IntegerOrdOpeQuery) UnmarshalJSON(b []byte) error { } // Smallint is the EQL type stored in public.eql_v4_smallint: the smallint family with no index. -// The engine produces it from a int16. +// The engine produces it from an int16. type Smallint []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -299,7 +299,7 @@ func (v *Smallint) UnmarshalJSON(b []byte) error { } // SmallintEq is the EQL type stored in public.eql_v4_smallint_eq: the smallint family with the eq index. -// The engine produces it from a int16. +// The engine produces it from an int16. type SmallintEq []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -393,7 +393,7 @@ func (v *SmallintOrdOreQuery) UnmarshalJSON(b []byte) error { } // SmallintOrd is the EQL type stored in public.eql_v4_smallint_ord: the smallint family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from an int16. type SmallintOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -440,7 +440,7 @@ func (v *SmallintOrdQuery) UnmarshalJSON(b []byte) error { } // SmallintOrdOpe is the EQL type stored in public.eql_v4_smallint_ord_ope: the smallint family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from an int16. type SmallintOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -487,7 +487,7 @@ func (v *SmallintOrdOpeQuery) UnmarshalJSON(b []byte) error { } // Bigint is the EQL type stored in public.eql_v4_bigint: the bigint family with no index. -// The engine produces it from a int64. +// The engine produces it from an int64. type Bigint []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -511,7 +511,7 @@ func (v *Bigint) UnmarshalJSON(b []byte) error { } // BigintEq is the EQL type stored in public.eql_v4_bigint_eq: the bigint family with the eq index. -// The engine produces it from a int64. +// The engine produces it from an int64. type BigintEq []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -605,7 +605,7 @@ func (v *BigintOrdOreQuery) UnmarshalJSON(b []byte) error { } // BigintOrd is the EQL type stored in public.eql_v4_bigint_ord: the bigint family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from an int64. type BigintOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -652,7 +652,7 @@ func (v *BigintOrdQuery) UnmarshalJSON(b []byte) error { } // BigintOrdOpe is the EQL type stored in public.eql_v4_bigint_ord_ope: the bigint family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from an int64. type BigintOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1500,7 +1500,7 @@ func (v *TextOrdOreQuery) UnmarshalJSON(b []byte) error { } // TextOrd is the EQL type stored in public.eql_v4_text_ord: the text family with the eq and ope indexes. -// The engine cannot produce it yet: no EQL type is built from the engine's text order terms yet (#1131). +// The engine produces it from a string. type TextOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1547,7 +1547,7 @@ func (v *TextOrdQuery) UnmarshalJSON(b []byte) error { } // TextOrdOpe is the EQL type stored in public.eql_v4_text_ord_ope: the text family with the eq and ope indexes. -// The engine cannot produce it yet: no EQL type is built from the engine's text order terms yet (#1131). +// The engine produces it from a string. type TextOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1641,7 +1641,7 @@ func (v *TextSearchOreQuery) UnmarshalJSON(b []byte) error { } // TextSearch is the EQL type stored in public.eql_v4_text_search: the text family with the eq and ope and match indexes. -// The engine cannot produce it yet: no EQL type is built from the engine's text order terms yet (#1131). +// The engine cannot produce it yet: every query of this type carries a match term, which text shorter than one 3-gram cannot make, so `=` or `<` on short text would be refused; how a query that is not a match leaves it out is not decided (#1131). type TextSearch []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1830,7 +1830,7 @@ func (v *RealOrdOreQuery) UnmarshalJSON(b []byte) error { } // RealOrd is the EQL type stored in public.eql_v4_real_ord: the real family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from a float32. type RealOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1877,7 +1877,7 @@ func (v *RealOrdQuery) UnmarshalJSON(b []byte) error { } // RealOrdOpe is the EQL type stored in public.eql_v4_real_ord_ope: the real family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from a float32. type RealOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -2042,7 +2042,7 @@ func (v *DoubleOrdOreQuery) UnmarshalJSON(b []byte) error { } // DoubleOrd is the EQL type stored in public.eql_v4_double_ord: the double family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from a float64. type DoubleOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -2089,7 +2089,7 @@ func (v *DoubleOrdQuery) UnmarshalJSON(b []byte) error { } // DoubleOrdOpe is the EQL type stored in public.eql_v4_double_ord_ope: the double family with the ope index. -// The engine cannot produce it yet: the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236). +// The engine produces it from a float64. type DoubleOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. diff --git a/languages/golang/encrypt/eql/eql_test.go b/languages/golang/encrypt/eql/eql_test.go index c7ee88ef0..78596a96d 100644 --- a/languages/golang/encrypt/eql/eql_test.go +++ b/languages/golang/encrypt/eql/eql_test.go @@ -19,7 +19,15 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) { t.Errorf("%s: an unproducible type has a reason", typ.Name) } } - want := []string{"Integer", "IntegerEq", "Smallint", "SmallintEq", "Bigint", "BigintEq", "Text", "TextEq", "TextMatch", "Boolean", "Real", "RealEq", "Double", "DoubleEq"} + want := []string{ + "Integer", "IntegerEq", "IntegerOrd", "IntegerOrdOpe", + "Smallint", "SmallintEq", "SmallintOrd", "SmallintOrdOpe", + "Bigint", "BigintEq", "BigintOrd", "BigintOrdOpe", + "Text", "TextEq", "TextMatch", "TextOrd", "TextOrdOpe", + "Boolean", + "Real", "RealEq", "RealOrd", "RealOrdOpe", + "Double", "DoubleEq", "DoubleOrd", "DoubleOrdOpe", + } if len(producible) != len(want) { t.Fatalf("producible = %v, want %v", producible, want) } @@ -28,7 +36,7 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) { t.Fatalf("producible = %v, want %v", producible, want) } } - for name, plaintext := range map[string]string{"Integer": "int32", "IntegerEq": "int32", "Smallint": "int16", "SmallintEq": "int16", "Bigint": "int64", "BigintEq": "int64", "Boolean": "bool", "Real": "float32", "RealEq": "float32", "Double": "float64", "DoubleEq": "float64"} { + for name, plaintext := range map[string]string{"Integer": "int32", "IntegerEq": "int32", "IntegerOrd": "int32", "Smallint": "int16", "SmallintEq": "int16", "SmallintOrdOpe": "int16", "Bigint": "int64", "BigintEq": "int64", "BigintOrdOpe": "int64", "Boolean": "bool", "Real": "float32", "RealEq": "float32", "RealOrd": "float32", "Double": "float64", "DoubleEq": "float64", "DoubleOrdOpe": "float64", "TextOrd": "string"} { if typ, ok := eql.Lookup(name); !ok || typ.Plaintext != plaintext { t.Errorf("%s: Plaintext = %q, want %q", name, typ.Plaintext, plaintext) } diff --git a/languages/golang/encrypt/eql_v4_order_test.go b/languages/golang/encrypt/eql_v4_order_test.go new file mode 100644 index 000000000..2a4619997 --- /dev/null +++ b/languages/golang/encrypt/eql_v4_order_test.go @@ -0,0 +1,386 @@ +package encrypt_test + +import ( + "bytes" + "context" + "encoding/json" + "math" + "os" + "sort" + "strings" + "testing" + + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" +) + +// The OPE-ordered EQL v4 types (IntegerOrd, BigintOrd, RealOrd, DoubleOrd, +// TextOrd and their OrdOpe twins), sealed through generated code over the +// deterministic guest build with the EQL types. PostgreSQL sees these +// values through the Rust suite: TestV4OrderedRowsMatchTheCommittedFixture +// commits them to v4_ordered_rows.json, and +// packages/eql/tests/encryption/tests/v4_order.rs inserts that file into +// eql_v4 columns and compares it. + +// rankings are ordered by every field's order key: each row's value is +// below the next row's in every column, except the pairs rankingTies +// records. +var rankings = []testusers.Ranking{ + {ID: 1, Level: -5, Floor: -5, Points: math.MinInt64, Total: -1 << 40, Speed: float32(math.Inf(-1)), Pace: -2.5, Score: -1.5, Mean: math.Inf(-1), Title: "alice", Handle: "al"}, + {ID: 2, Level: 0, Floor: 0, Points: 0, Total: 0, Speed: 0, Pace: 0, Score: 0, Mean: 0, Title: "Cafe", Handle: "bob"}, + {ID: 3, Level: 0, Floor: 0, Points: 0, Total: 0, Speed: float32(math.Copysign(0, -1)), Pace: float32(math.Copysign(0, -1)), Score: math.Copysign(0, -1), Mean: math.Copysign(0, -1), Title: "café", Handle: "BOB"}, + {ID: 4, Level: 7, Floor: 70000, Points: 5_000_000_001, Total: 1 << 40, Speed: 2.5, Pace: 1.5, Score: 9.75, Mean: 1e300, Title: "Mango", Handle: "carol"}, + {ID: 5, Level: math.MaxInt32, Floor: math.MaxInt32, Points: math.MaxInt64, Total: math.MaxInt64, Speed: float32(math.Inf(1)), Pace: float32(math.Inf(1)), Score: math.Inf(1), Mean: math.Inf(1), Title: "Zoe", Handle: "zed"}, +} + +// rankingTies are the adjacent pairs of rankings, by ID, that tie in every +// order key: equal integers, -0.0 against 0.0, and text that differs only +// in case and accents. They are written down rather than derived because +// the text ties need Unicode case folding, which the engine does and this +// module has no dependency for. +var rankingTies = map[[2]int64]bool{{2, 3}: true} + +// storedRanking is each sealed field of an encrypted ranking by column. +func storedRanking(e testusers.EncryptedRanking) map[string][]byte { + return map[string][]byte{ + "level": e.Level, + "floor": e.Floor, + "points": e.Points, + "total": e.Total, + "speed": e.Speed, + "pace": e.Pace, + "score": e.Score, + "mean": e.Mean, + "title": e.Title, + "handle": e.Handle, + } +} + +// orderTerm is a stored value's or query's `op`, as its hex. Lower-case +// hex sorts as the bytes it encodes, which is how EQL compares `op`. +func orderTerm(t *testing.T, value []byte) string { + t.Helper() + var v struct { + OP string `json:"op"` + } + if err := json.Unmarshal(value, &v); err != nil || v.OP == "" { + t.Fatalf("no op in %s: %v", value, err) + } + return v.OP +} + +func TestV4OrderedFieldsSealToTheirEQLTypesAndOpenBack(t *testing.T) { + c := deterministicEQLClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + + encrypted, err := testusers.EncryptRanking(ctx, cipher, rankings) + if err != nil { + t.Fatal(err) + } + for i, e := range encrypted { + for column, value := range storedRanking(e) { + want := "c,i,op,v" + if column == "title" || column == "handle" { + want = "c,hm,i,op,v" + } + if got := rankingKeys(t, column, value); got != want { + t.Errorf("row %d %s: keys %s, want %s", i, column, got, want) + } + } + } + + // The stored order terms sort as the rows do: each row below the next, + // except the pairs rankingTies records. A recorded tie that is not an + // adjacent pair of rankings would be checked as nothing, so each one + // must be found, its numbers must be equal in Go too, and there must be + // at least one. + if len(rankingTies) == 0 { + t.Fatal("rankingTies records no tie: the equal-order-key case is untested") + } + found := map[[2]int64]bool{} + for i := 0; i+1 < len(rankings); i++ { + a, b := rankings[i], rankings[i+1] + pair := [2]int64{a.ID, b.ID} + if !rankingTies[pair] { + continue + } + found[pair] = true + if a.Level != b.Level || a.Floor != b.Floor || a.Points != b.Points || a.Total != b.Total || + a.Speed != b.Speed || a.Pace != b.Pace || a.Score != b.Score || a.Mean != b.Mean { + t.Errorf("rows %d and %d are recorded as a tie but their numbers differ", a.ID, b.ID) + } + } + for pair := range rankingTies { + if !found[pair] { + t.Errorf("rankingTies records rows %d and %d, which are not adjacent in rankings", pair[0], pair[1]) + } + } + for column := range storedRanking(encrypted[0]) { + for i := 0; i+1 < len(encrypted); i++ { + a := orderTerm(t, storedRanking(encrypted[i])[column]) + b := orderTerm(t, storedRanking(encrypted[i+1])[column]) + tie := rankingTies[[2]int64{rankings[i].ID, rankings[i+1].ID}] + switch { + case tie && a != b: + t.Errorf("%s: rows %d and %d tie in the order key but store different terms", column, rankings[i].ID, rankings[i+1].ID) + case !tie && a >= b: + t.Errorf("%s: row %d's term does not sort below row %d's", column, rankings[i].ID, rankings[i+1].ID) + } + } + } + + decrypted, err := testusers.DecryptRanking(ctx, cipher, encrypted) + if err != nil { + t.Fatal(err) + } + for i := range rankings { + got, want := decrypted[i], rankings[i] + // The ciphertext keeps -0.0's sign; == would not tell. + if math.Signbit(float64(got.Speed)) != math.Signbit(float64(want.Speed)) || math.Signbit(got.Mean) != math.Signbit(want.Mean) { + t.Errorf("row %d: a zero lost its sign", want.ID) + } + if got != want { + t.Errorf("row %d: %+v, want %+v", i, got, want) + } + } +} + +// rankingKeys is the sorted top-level keys of an EQL value stored under +// rankings, checking its envelope as keysOf does for profiles. +func rankingKeys(t *testing.T, column string, value []byte) string { + t.Helper() + var raw map[string]json.RawMessage + if err := json.Unmarshal(value, &raw); err != nil { + t.Fatalf("the EQL value is not JSON: %v: %s", err, value) + } + var i struct{ T, C string } + if err := json.Unmarshal(raw["i"], &i); err != nil || i.T != "rankings" || i.C != column { + t.Errorf("i = %s, want {\"t\":\"rankings\",\"c\":%q}", raw["i"], column) + } + if string(raw["v"]) != "4" { + t.Errorf("v = %s, want 4", raw["v"]) + } + keys := make([]string, 0, len(raw)) + for k := range raw { + keys = append(keys, k) + } + sort.Strings(keys) + return strings.Join(keys, ",") +} + +func TestV4OrderedQueriesMatchTheRustFixture(t *testing.T) { + c := deterministicEQLClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + + raw, err := os.ReadFile(fixturePath("v4_ordered_queries.json")) + if err != nil { + t.Fatal(err) + } + type entry[T any] struct { + Plaintext T + Query string + } + var fixture struct { + Table string + Level entry[int32] + Floor entry[int32] + Points entry[int64] + Total entry[int64] + Speed entry[float32] + Pace entry[float32] + Score entry[float64] + Mean entry[float64] + Title entry[string] + Handle entry[string] + } + if err := json.Unmarshal(raw, &fixture); err != nil { + t.Fatal(err) + } + if fixture.Table != "rankings" { + t.Fatalf("the fixture's table changed: %q", fixture.Table) + } + f := testusers.RankingFields + // Byte for byte what eql-bindings' own dispatch derived under the same + // index key. Every ordering method of a field returns one operand: the + // operator is SQL's. + cases := []struct { + field string + lt func() ([]byte, error) + gte func() ([]byte, error) + want string + }{ + {"Level", func() ([]byte, error) { return f.Level.Lt(ctx, cipher, fixture.Level.Plaintext) }, func() ([]byte, error) { return f.Level.Gte(ctx, cipher, fixture.Level.Plaintext) }, fixture.Level.Query}, + {"Floor", func() ([]byte, error) { return f.Floor.Lt(ctx, cipher, fixture.Floor.Plaintext) }, func() ([]byte, error) { return f.Floor.Gte(ctx, cipher, fixture.Floor.Plaintext) }, fixture.Floor.Query}, + {"Points", func() ([]byte, error) { return f.Points.Lt(ctx, cipher, fixture.Points.Plaintext) }, func() ([]byte, error) { return f.Points.Gte(ctx, cipher, fixture.Points.Plaintext) }, fixture.Points.Query}, + {"Total", func() ([]byte, error) { return f.Total.Lt(ctx, cipher, fixture.Total.Plaintext) }, func() ([]byte, error) { return f.Total.Gte(ctx, cipher, fixture.Total.Plaintext) }, fixture.Total.Query}, + {"Speed", func() ([]byte, error) { return f.Speed.Lt(ctx, cipher, fixture.Speed.Plaintext) }, func() ([]byte, error) { return f.Speed.Gte(ctx, cipher, fixture.Speed.Plaintext) }, fixture.Speed.Query}, + {"Pace", func() ([]byte, error) { return f.Pace.Lt(ctx, cipher, fixture.Pace.Plaintext) }, func() ([]byte, error) { return f.Pace.Gte(ctx, cipher, fixture.Pace.Plaintext) }, fixture.Pace.Query}, + {"Score", func() ([]byte, error) { return f.Score.Lt(ctx, cipher, fixture.Score.Plaintext) }, func() ([]byte, error) { return f.Score.Gte(ctx, cipher, fixture.Score.Plaintext) }, fixture.Score.Query}, + {"Mean", func() ([]byte, error) { return f.Mean.Lt(ctx, cipher, fixture.Mean.Plaintext) }, func() ([]byte, error) { return f.Mean.Gte(ctx, cipher, fixture.Mean.Plaintext) }, fixture.Mean.Query}, + {"Title", func() ([]byte, error) { return f.Title.Lt(ctx, cipher, fixture.Title.Plaintext) }, func() ([]byte, error) { return f.Title.Eq(ctx, cipher, fixture.Title.Plaintext) }, fixture.Title.Query}, + {"Handle", func() ([]byte, error) { return f.Handle.Lt(ctx, cipher, fixture.Handle.Plaintext) }, func() ([]byte, error) { return f.Handle.Eq(ctx, cipher, fixture.Handle.Plaintext) }, fixture.Handle.Query}, + } + for _, tc := range cases { + for _, method := range []func() ([]byte, error){tc.lt, tc.gte} { + got, err := method() + if err != nil { + t.Fatal(err) + } + if tc.want == "" || string(got) != tc.want { + t.Errorf("%s = %s\nwant %s", tc.field, got, tc.want) + } + } + } + + // Text with a code point Unicode 16 does not assign has no fold, so no + // order term. + if q, err := f.Title.Lt(ctx, cipher, "a͸b"); err == nil { + t.Errorf("Title.Lt over unassigned text = %s, want a refusal", q) + } +} + +// orderedRowsFixture is v4_ordered_rows.json: the rankings above as the +// generated code seals them under the deterministic guest and testSeed. +type orderedRowsFixture struct { + Comment string `json:"_comment"` + Seed string `json:"seed"` + Table string `json:"table"` + Rows []orderedRow `json:"rows"` +} + +type orderedRow struct { + ID int64 `json:"id"` + Plaintext orderedPlain `json:"plaintext"` + Stored map[string]json.RawMessage `json:"stored"` +} + +// orderedPlain is a ranking's plaintext as the fixture records it. JSON has +// no infinity, so the fixture's rows hold finite values only; the infinite +// and -0.0 cases are covered by TestV4OrderedFieldsSealToTheirEQLTypesAndOpenBack +// and the Rust suite. +type orderedPlain struct { + Level int32 `json:"level"` + Floor int32 `json:"floor"` + Points int64 `json:"points"` + Total int64 `json:"total"` + Speed float32 `json:"speed"` + Pace float32 `json:"pace"` + Score float64 `json:"score"` + Mean float64 `json:"mean"` + Title string `json:"title"` + Handle string `json:"handle"` +} + +func orderedPlainOf(r testusers.Ranking) orderedPlain { + return orderedPlain{r.Level, r.Floor, r.Points, r.Total, r.Speed, r.Pace, r.Score, r.Mean, r.Title, r.Handle} +} + +// fixtureRankings are the rows v4_ordered_rows.json holds: finite values +// on both sides of the fixture's query plaintexts in every column. +var fixtureRankings = []testusers.Ranking{ + {ID: 1, Level: -20, Floor: -30, Points: -5, Total: -9, Speed: -1.25, Pace: -3.5, Score: -2.5, Mean: -0.25, Title: "apple", Handle: "Alice"}, + {ID: 2, Level: 10, Floor: -3, Points: 5_000_000_000, Total: -1, Speed: 2.5, Pace: -0.5, Score: 9.75, Mean: 0, Title: "mango", Handle: "Carol"}, + {ID: 3, Level: 99, Floor: 4, Points: 6_000_000_000, Total: 12, Speed: 8, Pace: 0.75, Score: 100, Mean: 3.5, Title: "Zucchini", Handle: "dave"}, +} + +// TestV4OrderedRowsMatchTheCommittedFixture checks the committed +// v4_ordered_rows.json against what the generated code seals today, as +// TestV4RowsMatchTheCommittedFixture does for v4_rows.json: everything but +// each `c` must match, and the committed `c` values must open. +// EQL_UPDATE_FIXTURES rewrites the file when something other than the +// ciphertexts moved. +func TestV4OrderedRowsMatchTheCommittedFixture(t *testing.T) { + c := deterministicEQLClient(t) + ctx := context.Background() + encrypted, err := testusers.EncryptRanking(ctx, c.DefaultKeyset(), fixtureRankings) + if err != nil { + t.Fatal(err) + } + fixture := orderedRowsFixture{ + Comment: "Ranking rows sealed by the Go SDK's generated code over the deterministic EQL guest with the seed below; the Rust suite opens and compares them. Regenerate with EQL_UPDATE_FIXTURES=1 go test ./encrypt -run TestV4OrderedRows, do not edit.", + Seed: string(testSeed[:]), + Table: "rankings", + } + for i, e := range encrypted { + row := orderedRow{ID: fixtureRankings[i].ID, Plaintext: orderedPlainOf(fixtureRankings[i]), Stored: map[string]json.RawMessage{}} + for column, value := range storedRanking(e) { + row.Stored[column] = json.RawMessage(value) + } + fixture.Rows = append(fixture.Rows, row) + } + rendered, err := json.MarshalIndent(fixture, "", " ") + if err != nil { + t.Fatal(err) + } + rendered = append(rendered, '\n') + path := fixturePath("v4_ordered_rows.json") + var committed orderedRowsFixture + committedBytes, err := os.ReadFile(path) + if err == nil { + err = json.Unmarshal(committedBytes, &committed) + } + same := err == nil && bytes.Equal(orderedWithoutCiphertexts(t, committed), orderedWithoutCiphertexts(t, fixture)) + if os.Getenv("EQL_UPDATE_FIXTURES") != "" && !same { + if err := os.WriteFile(path, rendered, 0o644); err != nil { + t.Fatal(err) + } + t.Logf("%s: rewritten", path) + committed, same, err = fixture, true, nil + } + if err != nil { + t.Fatalf("%v: run with EQL_UPDATE_FIXTURES=1 to write it", err) + } + if !same { + t.Errorf("%s differs from what the generated code seals today beyond the ciphertexts; regenerate it with EQL_UPDATE_FIXTURES=1", path) + } + + // The committed ciphertexts still open, to the committed plaintexts. + sealed := make([]testusers.EncryptedRanking, len(committed.Rows)) + for i, row := range committed.Rows { + s := row.Stored + sealed[i] = testusers.EncryptedRanking{ + ID: row.ID, Level: []byte(s["level"]), Floor: []byte(s["floor"]), Points: []byte(s["points"]), Total: []byte(s["total"]), + Speed: []byte(s["speed"]), Pace: []byte(s["pace"]), Score: []byte(s["score"]), Mean: []byte(s["mean"]), + Title: []byte(s["title"]), Handle: []byte(s["handle"]), + } + } + opened, err := testusers.DecryptRanking(ctx, c.DefaultKeyset(), sealed) + if err != nil { + t.Fatal(err) + } + for i, row := range committed.Rows { + if got := orderedPlainOf(opened[i]); got != row.Plaintext { + t.Errorf("row %d opens to %+v, want %+v", row.ID, got, row.Plaintext) + } + } +} + +// orderedWithoutCiphertexts renders a fixture with every stored value's `c` +// removed: the part of the file that is the same on every run. +func orderedWithoutCiphertexts(t *testing.T, f orderedRowsFixture) []byte { + t.Helper() + out := f + out.Rows = nil + for _, row := range f.Rows { + stripped := orderedRow{ID: row.ID, Plaintext: row.Plaintext, Stored: map[string]json.RawMessage{}} + for column, value := range row.Stored { + var fields map[string]json.RawMessage + if err := json.Unmarshal(value, &fields); err != nil { + t.Fatal(err) + } + delete(fields, "c") + b, err := json.Marshal(fields) + if err != nil { + t.Fatal(err) + } + stripped.Stored[column] = b + } + out.Rows = append(out.Rows, stripped) + } + b, err := json.Marshal(out) + if err != nil { + t.Fatal(err) + } + return b +} diff --git a/languages/golang/encrypt/guest/src/targets.rs b/languages/golang/encrypt/guest/src/targets.rs index 1f4443eff..416d2e70d 100644 --- a/languages/golang/encrypt/guest/src/targets.rs +++ b/languages/golang/encrypt/guest/src/targets.rs @@ -191,18 +191,30 @@ mod tests { [ "Integer", "IntegerEq", + "IntegerOrd", + "IntegerOrdOpe", "Smallint", "SmallintEq", + "SmallintOrd", + "SmallintOrdOpe", "Bigint", "BigintEq", + "BigintOrd", + "BigintOrdOpe", "Text", "TextEq", "TextMatch", + "TextOrd", + "TextOrdOpe", "Boolean", "Real", "RealEq", + "RealOrd", + "RealOrdOpe", "Double", - "DoubleEq" + "DoubleEq", + "DoubleOrd", + "DoubleOrdOpe" ], "the engine's producible types, in catalog order" ); diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index 866c304e4..acbc63b45 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -1365,18 +1365,30 @@ fn targets_lists_what_the_build_holds() { [ "Integer", "IntegerEq", + "IntegerOrd", + "IntegerOrdOpe", "Smallint", "SmallintEq", + "SmallintOrd", + "SmallintOrdOpe", "Bigint", "BigintEq", + "BigintOrd", + "BigintOrdOpe", "Text", "TextEq", "TextMatch", + "TextOrd", + "TextOrdOpe", "Boolean", "Real", "RealEq", + "RealOrd", + "RealOrdOpe", "Double", - "DoubleEq" + "DoubleEq", + "DoubleOrd", + "DoubleOrdOpe" ], "the engine's producible types, in catalog order" ); @@ -1727,8 +1739,9 @@ fn target_limit_sizes_a_fetch_from_the_terms_bytes() { } /// The `eql` build's catalog lists, per type, the operators the engine -/// answers: `TextEq` answers `eq`, exactly, and a type the engine does not -/// produce answers nothing. +/// answers: `TextEq` answers `eq`, exactly, an OPE-ordered type the four +/// orderings of one value through `ope` (and `eq` first when it stores an +/// equality term), and a type the engine does not produce answers nothing. #[test] fn the_catalog_lists_each_types_operators() { if !stack_encrypt_guest::targets::HOLDS_EQL { @@ -1749,15 +1762,37 @@ fn the_catalog_lists_each_types_operators() { panic!("{name}: operators is a list"); }; let producible = matches!(entry(fields, "producible"), Tree::Bool(true)); + let listed: Vec> = operators + .iter() + .map(|op| { + let Tree::Object(op) = op else { + panic!("an object"); + }; + op.iter().map(|(k, v)| (k.as_str(), text(v))).collect() + }) + .collect(); let only = |want: [(&str, &str); 4]| { - assert_eq!(operators.len(), 1, "{name} answers one operator"); - let Tree::Object(op) = &operators[0] else { - panic!("an object"); - }; - let op: Vec<(&str, &str)> = op.iter().map(|(k, v)| (k.as_str(), text(v))).collect(); - assert_eq!(op, want, "{name}"); + assert_eq!(listed, [want.to_vec()], "{name}"); }; + let exact = |operator: &'static str, index: &'static str| { + vec![ + ("operator", operator), + ("fidelity", "exact"), + ("arity", "one"), + ("index", index), + ] + }; + let orderings = || ["lt", "lte", "gt", "gte"].map(|op| exact(op, "ope")); match name.as_str() { + "IntegerOrd" | "IntegerOrdOpe" | "SmallintOrd" | "SmallintOrdOpe" | "BigintOrd" + | "BigintOrdOpe" | "RealOrd" | "RealOrdOpe" | "DoubleOrd" | "DoubleOrdOpe" => { + assert_eq!(listed, orderings().to_vec(), "{name}"); + } + "TextOrd" | "TextOrdOpe" => { + let mut want = vec![exact("eq", "eq")]; + want.extend(orderings()); + assert_eq!(listed, want, "{name}"); + } "TextEq" | "IntegerEq" | "SmallintEq" | "BigintEq" | "RealEq" | "DoubleEq" => only([ ("operator", "eq"), ("fidelity", "exact"), diff --git a/languages/golang/encrypt/internal/testusers/ranking.go b/languages/golang/encrypt/internal/testusers/ranking.go new file mode 100644 index 000000000..b7aff488a --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/ranking.go @@ -0,0 +1,24 @@ +package testusers + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Ranking -name Ranking + +// Ranking stores one field in each OPE-ordered EQL v4 type the engine +// produces, the shape the ordered cross-language fixtures under +// packages/eql/tests/encryption/fixtures (v4_ordered_queries.json, +// v4_ordered_rows.json) are derived under (table rankings). The `Ord` and +// `OrdOpe` types of a family store the same terms; each gets a column so +// both plans run. +type Ranking struct { + _ struct{} `stash:"context=rankings"` + ID int64 `stash:"id,passthrough"` + Level int32 `stash:"level,encrypt_into=IntegerOrd"` + Floor int32 `stash:"floor,encrypt_into=IntegerOrdOpe"` + Points int64 `stash:"points,encrypt_into=BigintOrd"` + Total int64 `stash:"total,encrypt_into=BigintOrdOpe"` + Speed float32 `stash:"speed,encrypt_into=RealOrd"` + Pace float32 `stash:"pace,encrypt_into=RealOrdOpe"` + Score float64 `stash:"score,encrypt_into=DoubleOrd"` + Mean float64 `stash:"mean,encrypt_into=DoubleOrdOpe"` + Title string `stash:"title,encrypt_into=TextOrd"` + Handle string `stash:"handle,encrypt_into=TextOrdOpe"` +} diff --git a/languages/golang/encrypt/internal/testusers/ranking_stash.go b/languages/golang/encrypt/internal/testusers/ranking_stash.go new file mode 100644 index 000000000..96c368b90 --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/ranking_stash.go @@ -0,0 +1,500 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Ranking prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedRanking struct { + ID int64 + Level eql.IntegerOrd + Floor eql.IntegerOrdOpe + Points eql.BigintOrd + Total eql.BigintOrdOpe + Speed eql.RealOrd + Pace eql.RealOrdOpe + Score eql.DoubleOrd + Mean eql.DoubleOrdOpe + Title eql.TextOrd + Handle eql.TextOrdOpe +} + +func (e EncryptedRanking) String() string { + return gensupport.Redacted("EncryptedRanking", map[string]any{"ID": e.ID}, "Level", "Floor", "Points", "Total", "Speed", "Pace", "Score", "Mean", "Title", "Handle") +} + +func (e EncryptedRanking) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Level", "Floor", "Points", "Total", "Speed", "Pace", "Score", "Mean", "Title", "Handle") +} + +// Stops compiling when Ranking gains, loses, reorders or retypes a field. +var _ = rankingShape(Ranking{}) + +type rankingShape struct { + _ struct{} + ID int64 + Level int32 + Floor int32 + Points int64 + Total int64 + Speed float32 + Pace float32 + Score float64 + Mean float64 + Title string + Handle string +} + +var rankingDeclaration = gensupport.Declare("rankings"). + Passthrough("id"). + EncryptInto("level", gensupport.Int32, "IntegerOrd"). + EncryptInto("floor", gensupport.Int32, "IntegerOrdOpe"). + EncryptInto("points", gensupport.Int64, "BigintOrd"). + EncryptInto("total", gensupport.Int64, "BigintOrdOpe"). + EncryptInto("speed", gensupport.Float32, "RealOrd"). + EncryptInto("pace", gensupport.Float32, "RealOrdOpe"). + EncryptInto("score", gensupport.Float64, "DoubleOrd"). + EncryptInto("mean", gensupport.Float64, "DoubleOrdOpe"). + EncryptInto("title", gensupport.String, "TextOrd"). + EncryptInto("handle", gensupport.String, "TextOrdOpe") + +var rankingCodec = gensupport.New(gensupport.Generated[Ranking, EncryptedRanking]{ + TypeName: "Ranking", + Declaration: rankingDeclaration, + PrintsPlaintext: true, + Source: func(v Ranking) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "level": v.Level, + "floor": v.Floor, + "points": v.Points, + "total": v.Total, + "speed": v.Speed, + "pace": v.Pace, + "score": v.Score, + "mean": v.Mean, + "title": v.Title, + "handle": v.Handle, + } + }, + Seal: func(rec gensupport.Record) (EncryptedRanking, error) { + var e EncryptedRanking + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedRanking{}, err + } + e.Level = eql.IntegerOrd(rec["level"].EQL) + e.Floor = eql.IntegerOrdOpe(rec["floor"].EQL) + e.Points = eql.BigintOrd(rec["points"].EQL) + e.Total = eql.BigintOrdOpe(rec["total"].EQL) + e.Speed = eql.RealOrd(rec["speed"].EQL) + e.Pace = eql.RealOrdOpe(rec["pace"].EQL) + e.Score = eql.DoubleOrd(rec["score"].EQL) + e.Mean = eql.DoubleOrdOpe(rec["mean"].EQL) + e.Title = eql.TextOrd(rec["title"].EQL) + e.Handle = eql.TextOrdOpe(rec["handle"].EQL) + return e, nil + }, + Open: func(e EncryptedRanking) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "level": {EQL: e.Level}, + "floor": {EQL: e.Floor}, + "points": {EQL: e.Points}, + "total": {EQL: e.Total}, + "speed": {EQL: e.Speed}, + "pace": {EQL: e.Pace}, + "score": {EQL: e.Score}, + "mean": {EQL: e.Mean}, + "title": {EQL: e.Title}, + "handle": {EQL: e.Handle}, + } + }, + Value: func(e EncryptedRanking, vals gensupport.Values) (Ranking, error) { + var v Ranking + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return Ranking{}, err + } + if v.Level, err = gensupport.Get[int32](vals, "level"); err != nil { + return Ranking{}, err + } + if v.Floor, err = gensupport.Get[int32](vals, "floor"); err != nil { + return Ranking{}, err + } + if v.Points, err = gensupport.Get[int64](vals, "points"); err != nil { + return Ranking{}, err + } + if v.Total, err = gensupport.Get[int64](vals, "total"); err != nil { + return Ranking{}, err + } + if v.Speed, err = gensupport.Get[float32](vals, "speed"); err != nil { + return Ranking{}, err + } + if v.Pace, err = gensupport.Get[float32](vals, "pace"); err != nil { + return Ranking{}, err + } + if v.Score, err = gensupport.Get[float64](vals, "score"); err != nil { + return Ranking{}, err + } + if v.Mean, err = gensupport.Get[float64](vals, "mean"); err != nil { + return Ranking{}, err + } + if v.Title, err = gensupport.Get[string](vals, "title"); err != nil { + return Ranking{}, err + } + if v.Handle, err = gensupport.Get[string](vals, "handle"); err != nil { + return Ranking{}, err + } + return v, nil + }, +}) + +// EncryptRanking seals each Ranking, with one ZeroKMS request for each 500 +// sealed values, plus one the first time a keyset is used. The result has one +// element for each input, in the same order. +func EncryptRanking(ctx context.Context, cipher *encrypt.Cipher, values []Ranking) ([]EncryptedRanking, error) { + return rankingCodec.Encrypt(ctx, cipher, values) +} + +// DecryptRanking opens each EncryptedRanking, with one ZeroKMS request for each +// 500 sealed values, plus one the first time a keyset is used. +func DecryptRanking(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedRanking) ([]Ranking, error) { + return rankingCodec.Decrypt(ctx, d, encrypted) +} + +var RankingFields = struct { + Level RankingLevelField + Floor RankingFloorField + Points RankingPointsField + Total RankingTotalField + Speed RankingSpeedField + Pace RankingPaceField + Score RankingScoreField + Mean RankingMeanField + Title RankingTitleField + Handle RankingHandleField +}{ + Level: RankingLevelField{gensupport.NewField[int32](rankingDeclaration, "level")}, + Floor: RankingFloorField{gensupport.NewField[int32](rankingDeclaration, "floor")}, + Points: RankingPointsField{gensupport.NewField[int64](rankingDeclaration, "points")}, + Total: RankingTotalField{gensupport.NewField[int64](rankingDeclaration, "total")}, + Speed: RankingSpeedField{gensupport.NewField[float32](rankingDeclaration, "speed")}, + Pace: RankingPaceField{gensupport.NewField[float32](rankingDeclaration, "pace")}, + Score: RankingScoreField{gensupport.NewField[float64](rankingDeclaration, "score")}, + Mean: RankingMeanField{gensupport.NewField[float64](rankingDeclaration, "mean")}, + Title: RankingTitleField{gensupport.NewField[string](rankingDeclaration, "title")}, + Handle: RankingHandleField{gensupport.NewField[string](rankingDeclaration, "handle")}, +} + +type RankingLevelField struct { + field gensupport.Field[int32] +} + +func (f RankingLevelField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrd, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.IntegerOrd(out.EQL), err +} + +func (f RankingLevelField) Lt(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.IntegerOrdQuery(term), err +} + +func (f RankingLevelField) Lte(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.IntegerOrdQuery(term), err +} + +func (f RankingLevelField) Gt(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.IntegerOrdQuery(term), err +} + +func (f RankingLevelField) Gte(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.IntegerOrdQuery(term), err +} + +type RankingFloorField struct { + field gensupport.Field[int32] +} + +func (f RankingFloorField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdOpe, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.IntegerOrdOpe(out.EQL), err +} + +func (f RankingFloorField) Lt(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.IntegerOrdOpeQuery(term), err +} + +func (f RankingFloorField) Lte(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.IntegerOrdOpeQuery(term), err +} + +func (f RankingFloorField) Gt(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.IntegerOrdOpeQuery(term), err +} + +func (f RankingFloorField) Gte(ctx context.Context, c *encrypt.Cipher, v int32) (eql.IntegerOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.IntegerOrdOpeQuery(term), err +} + +type RankingPointsField struct { + field gensupport.Field[int64] +} + +func (f RankingPointsField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrd, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.BigintOrd(out.EQL), err +} + +func (f RankingPointsField) Lt(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.BigintOrdQuery(term), err +} + +func (f RankingPointsField) Lte(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.BigintOrdQuery(term), err +} + +func (f RankingPointsField) Gt(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.BigintOrdQuery(term), err +} + +func (f RankingPointsField) Gte(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.BigintOrdQuery(term), err +} + +type RankingTotalField struct { + field gensupport.Field[int64] +} + +func (f RankingTotalField) Encrypt(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdOpe, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.BigintOrdOpe(out.EQL), err +} + +func (f RankingTotalField) Lt(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.BigintOrdOpeQuery(term), err +} + +func (f RankingTotalField) Lte(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.BigintOrdOpeQuery(term), err +} + +func (f RankingTotalField) Gt(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.BigintOrdOpeQuery(term), err +} + +func (f RankingTotalField) Gte(ctx context.Context, c *encrypt.Cipher, v int64) (eql.BigintOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.BigintOrdOpeQuery(term), err +} + +type RankingSpeedField struct { + field gensupport.Field[float32] +} + +func (f RankingSpeedField) Encrypt(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrd, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.RealOrd(out.EQL), err +} + +func (f RankingSpeedField) Lt(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.RealOrdQuery(term), err +} + +func (f RankingSpeedField) Lte(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.RealOrdQuery(term), err +} + +func (f RankingSpeedField) Gt(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.RealOrdQuery(term), err +} + +func (f RankingSpeedField) Gte(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.RealOrdQuery(term), err +} + +type RankingPaceField struct { + field gensupport.Field[float32] +} + +func (f RankingPaceField) Encrypt(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdOpe, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.RealOrdOpe(out.EQL), err +} + +func (f RankingPaceField) Lt(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.RealOrdOpeQuery(term), err +} + +func (f RankingPaceField) Lte(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.RealOrdOpeQuery(term), err +} + +func (f RankingPaceField) Gt(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.RealOrdOpeQuery(term), err +} + +func (f RankingPaceField) Gte(ctx context.Context, c *encrypt.Cipher, v float32) (eql.RealOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.RealOrdOpeQuery(term), err +} + +type RankingScoreField struct { + field gensupport.Field[float64] +} + +func (f RankingScoreField) Encrypt(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrd, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.DoubleOrd(out.EQL), err +} + +func (f RankingScoreField) Lt(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.DoubleOrdQuery(term), err +} + +func (f RankingScoreField) Lte(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.DoubleOrdQuery(term), err +} + +func (f RankingScoreField) Gt(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.DoubleOrdQuery(term), err +} + +func (f RankingScoreField) Gte(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.DoubleOrdQuery(term), err +} + +type RankingMeanField struct { + field gensupport.Field[float64] +} + +func (f RankingMeanField) Encrypt(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdOpe, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.DoubleOrdOpe(out.EQL), err +} + +func (f RankingMeanField) Lt(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.DoubleOrdOpeQuery(term), err +} + +func (f RankingMeanField) Lte(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.DoubleOrdOpeQuery(term), err +} + +func (f RankingMeanField) Gt(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.DoubleOrdOpeQuery(term), err +} + +func (f RankingMeanField) Gte(ctx context.Context, c *encrypt.Cipher, v float64) (eql.DoubleOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.DoubleOrdOpeQuery(term), err +} + +type RankingTitleField struct { + field gensupport.Field[string] +} + +func (f RankingTitleField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrd, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextOrd(out.EQL), err +} + +func (f RankingTitleField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextOrdQuery(term), err +} + +func (f RankingTitleField) Lt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.TextOrdQuery(term), err +} + +func (f RankingTitleField) Lte(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.TextOrdQuery(term), err +} + +func (f RankingTitleField) Gt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.TextOrdQuery(term), err +} + +func (f RankingTitleField) Gte(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.TextOrdQuery(term), err +} + +type RankingHandleField struct { + field gensupport.Field[string] +} + +func (f RankingHandleField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdOpe, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextOrdOpe(out.EQL), err +} + +func (f RankingHandleField) Eq(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TextOrdOpeQuery(term), err +} + +func (f RankingHandleField) Lt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lt", v) + return eql.TextOrdOpeQuery(term), err +} + +func (f RankingHandleField) Lte(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "lte", v) + return eql.TextOrdOpeQuery(term), err +} + +func (f RankingHandleField) Gt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gt", v) + return eql.TextOrdOpeQuery(term), err +} + +func (f RankingHandleField) Gte(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextOrdOpeQuery, error) { + term, err := f.field.Term(ctx, c, "gte", v) + return eql.TextOrdOpeQuery(term), err +} diff --git a/languages/golang/encrypt/query_test.go b/languages/golang/encrypt/query_test.go index aa9a92ec1..6a5200345 100644 --- a/languages/golang/encrypt/query_test.go +++ b/languages/golang/encrypt/query_test.go @@ -382,7 +382,10 @@ func TestVerifyAndTargetLimitNeedNoCipher(t *testing.T) { } // The catalog reports, per EQL type, the operators the engine answers: -// TextEq answers eq, exactly, through its equality index. +// TextEq answers eq, exactly, through its equality index, and an OPE-ordered +// type the four orderings of one value through its ope index (Between takes +// a pair, which a type's one query value does not carry), and eq too when it +// also stores an equality term. func TestTheCatalogListsEachTypesOperators(t *testing.T) { wasm, err := encrypt.EmbeddedGuest() if errors.Is(err, encrypt.ErrGuestNotBuilt) { @@ -411,6 +414,17 @@ func TestTheCatalogListsEachTypesOperators(t *testing.T) { if len(target.Operators) != 1 || target.Operators[0] != want[0] { t.Fatalf("%s answers %+v, want %+v", target.Name, target.Operators, want) } + case slices.Contains([]string{"IntegerOrd", "IntegerOrdOpe", "SmallintOrd", "SmallintOrdOpe", "BigintOrd", "BigintOrdOpe", "RealOrd", "RealOrdOpe", "DoubleOrd", "DoubleOrdOpe", "TextOrd", "TextOrdOpe"}, target.Name): + var want []record.Operator + if strings.HasPrefix(target.Name, "Text") { + want = append(want, record.Operator{Name: "eq", Fidelity: record.Exact, Index: record.Equality}) + } + for _, name := range []string{"lt", "lte", "gt", "gte"} { + want = append(want, record.Operator{Name: name, Fidelity: record.Exact, Index: record.Ope}) + } + if !slices.Equal(target.Operators, want) { + t.Fatalf("%s answers %+v, want %+v", target.Name, target.Operators, want) + } case target.Name == "TextMatch": want := record.Operator{Name: "matches", Fidelity: record.Probabilistic, Index: record.Match} if len(target.Operators) != 1 || target.Operators[0] != want { diff --git a/languages/golang/stashgen/refusal_test.go b/languages/golang/stashgen/refusal_test.go index e1e8b77a6..d91f5de3e 100644 --- a/languages/golang/stashgen/refusal_test.go +++ b/languages/golang/stashgen/refusal_test.go @@ -93,7 +93,7 @@ func TestRefusals(t *testing.T) { {"an opaque struct with an interface", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tAny any"), stashgen.Request{Type: "User"}, "Any", "is not a type JSON carries"}, {"an opaque struct with a struct JSON would truncate", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tTx hidden") + "\ntype hidden struct {\n\tA int\n\tb int\n}\n", stashgen.Request{Type: "User"}, "Tx", "JSON would drop"}, {"an opaque struct with a map keyed by a struct", user("\t_ struct{} `stash:\"context=users,opaque\"`\n\tM map[hidden]int") + "\ntype hidden struct{ A int }\n", stashgen.Request{Type: "User"}, "M", "map key that is not a string or an integer"}, - {"an EQL type the engine cannot produce yet", user(ctx + "\tAge int32 `stash:\"age,encrypt_into=IntegerOrd\"`"), stashgen.Request{Type: "User"}, "Age", "has no EQL type IntegerOrd"}, + {"an EQL type the engine cannot produce yet", user(ctx + "\tAge int32 `stash:\"age,encrypt_into=IntegerOrdOre\"`"), stashgen.Request{Type: "User"}, "Age", "has no EQL type IntegerOrdOre"}, {"an index on a composite", user(ctx + "\tAttrs map[string]string `stash:\"attrs,encrypt,index=equality\"`"), stashgen.Request{Type: "User"}, "Attrs", "seals only as part of an opaque struct"}, {"the json index, not in the engine yet", user(ctx + "\tAttrs string `stash:\"attrs,index=json\"`"), stashgen.Request{Type: "User"}, "Attrs", "cannot derive the json index yet"}, {"an index option a query term cannot use", user(ctx + "\tEmail string `stash:\"email,encrypt,index=match(k=3)\"`"), stashgen.Request{Type: "User"}, "Email", "a query term uses only the default index options"}, diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index f8f0de918..864b46d7e 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -30,6 +30,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 The `date`, `timestamp` and `numeric` families say why they are still unproducible: the engine has their kinds, and their Rust plaintext types are not chosen yet. +- **The OPE-ordered types are producible** (`stack-encrypt` feature): + `IntegerOrd`, `IntegerOrdOpe`, `SmallintOrd`, `SmallintOrdOpe`, + `BigintOrd`, `BigintOrdOpe`, `RealOrd`, `RealOrdOpe`, `DoubleOrd`, + `DoubleOrdOpe`, `TextOrd` and `TextOrdOpe`, from `i32`, `i16`, `i64`, + `f32`, `f64` and `String`. Their `op` term is Stack + Encrypt's OPE term over ADR-0002's order key, hex encoded, which EQL + compares as bytes: floats order as PostgreSQL's `float8` does, and text by + its fold (case and accents folded), so text order types keep `TextEq`'s + equality term for `=`. `OpeCllw` implements `EncryptFrom` for any + orderable plaintext. Their query twins carry one bound, so the engine + answers `<`, `<=`, `>` and `>=` (and `=` on text) through them, not + `BETWEEN`. - **`Target::match_options`** (`targets` feature): the options a type's match index is built under, `EqlMatch`'s, for a type that carries one (`TextMatch`), and `None` otherwise. Stack Encrypt answers `matches` diff --git a/packages/eql/crates/eql-bindings/README.md b/packages/eql/crates/eql-bindings/README.md index 1c7f16db9..c519e91e5 100644 --- a/packages/eql/crates/eql-bindings/README.md +++ b/packages/eql/crates/eql-bindings/README.md @@ -111,19 +111,31 @@ plaintext: | v4 type | Plaintext | | --- | --- | -| `Text`, `TextMatch` | `String` | -| `Smallint`, `SmallintEq` | `i16` | -| `Integer`, `IntegerEq` | `i32` | -| `Bigint`, `BigintEq` | `i64` | +| `Text`, `TextMatch`, `TextOrd`, `TextOrdOpe` | `String` | +| `Smallint`, `SmallintEq`, `SmallintOrd`, `SmallintOrdOpe` | `i16` | +| `Integer`, `IntegerEq`, `IntegerOrd`, `IntegerOrdOpe` | `i32` | +| `Bigint`, `BigintEq`, `BigintOrd`, `BigintOrdOpe` | `i64` | | `Boolean` | `bool` | -| `Real`, `RealEq` | `f32` | -| `Double`, `DoubleEq` | `f64` | +| `Real`, `RealEq`, `RealOrd`, `RealOrdOpe` | `f32` | +| `Double`, `DoubleEq`, `DoubleOrd`, `DoubleOrdOpe` | `f64` | Each seals the same leaf the Go SDK seals for the same value, so values written here and by the Go SDK open each other. On the equality types, `=` behaves as Postgres's own does: an integer has one term whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN`. The ciphertext keeps the exact value. +The `Ord` and `OrdOpe` types store an OPE term (`op`), which EQL compares as +bytes, so `<`, `<=`, `>`, `>=` and `ORDER BY eql_v4.ord_term(col)` follow +ADR-0002's order key: integers by value, floats as PostgreSQL's `float8` +sorts them (`-0.0` ties `0.0`, every NaN is one value after `+Infinity`, and +an `f32` orders as the `f64` it widens to), and text by its fold (case and +accents folded, under Unicode 16). On a number column `=` compares the order +term too. A text order type also stores `TextEq`'s equality term, and `=` +reads that one, so `Café` and `cafe` are neither `<` nor `>` each other, and +not `=` either. Text with a code point Unicode 16 does not assign has no +order term and is refused, stored or queried. A family's `Ord` and `OrdOpe` +types store the same term under one column. The query value carries one +bound: `BETWEEN` needs EQL v4's range query domains and is not produced yet. `TextMatch` builds its filter under a configuration pinned in this crate (`EqlMatch`: 3-grams of the lower-cased text, `k = 6`, a 2048-bit filter) for stored values and queries alike. The configuration is part of the stored @@ -133,11 +145,11 @@ where that matters. Write the stored column on the left, `col @@ query`: the other order finds nothing until #1171 is fixed. Text too short to make a 3-gram (empty strings, two-letter codes) cannot be stored in a `TextMatch` column yet, and a `TextMatchQuery` for it is refused too: one filter type -builds both, and an empty filter as a query would match every row. The -ordering types wait for the engine's order keys (#1236), text's order types -(#1131) and block ORE (#1132), `Date`, `Timestamp` and `Numeric` for a -choice of their Rust plaintext types (#1130), and `Json` for a decision on -sealing a document as one leaf. +builds both, and an empty filter as a query would match every row. The ORE +types (`*OrdOre`, `TextSearchOre`) wait for block ORE (#1132), `TextSearch` +for how a query that is not a match leaves out its match term (#1131), +`Date`, `Timestamp` and `Numeric` for a choice of their Rust plaintext types +(#1130), and `Json` for a decision on sealing a document as one leaf. The generated API docs include a complete, executable example in the `eql_bindings::encryption` module, from cipher setup through decryption. Its diff --git a/packages/eql/crates/eql-bindings/src/encryption.rs b/packages/eql/crates/eql-bindings/src/encryption.rs index f95172c1f..ee0c14727 100644 --- a/packages/eql/crates/eql-bindings/src/encryption.rs +++ b/packages/eql/crates/eql-bindings/src/encryption.rs @@ -7,6 +7,11 @@ //! precomposed `é` and with `e` and a combining accent is one value, `Café` //! another. The stored identifier supplies context for ciphertext and terms. //! +//! Order (the `op` term of the `Ord` and `OrdOpe` types) compares ADR-0002's +//! order key, which folds text's case and accents: `Café` and `cafe` are +//! neither `<` nor `>` each other, while `=` on a `TextOrd` column, which +//! reads the equality term, still tells them apart. +//! //! This is a new producer profile, independent of existing cipherstash-client //! ciphertext and terms, so it writes the `eql_v4` domains ([`crate::v4`]), //! never `eql_v3` (ADR-0002). @@ -93,7 +98,9 @@ pub mod targets; use base64::{engine::general_purpose::STANDARD, Engine as _}; #[allow(deprecated)] use stack_encrypt::sem::MatchTerm; -use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, TermError, Tokenizer}; +use stack_encrypt::sem::{ + EqualityTerm, MatchConfig, MatchOptions, OpeTerm, Orderable, TermError, Tokenizer, +}; use stack_encrypt::target::transcode::{Reader, Transcode, Visitor}; use stack_encrypt::target::{self, AeadContext, CallerContext}; use stack_encrypt::{ @@ -103,7 +110,7 @@ use stack_encrypt::{ }; use crate::{ - v3::terms::{BloomFilter, Ciphertext, Hmac256}, + v3::terms::{BloomFilter, Ciphertext, Hmac256, OpeCllw}, Identifier, }; @@ -360,6 +367,51 @@ impl DecryptField for BloomFilter { } } +/// Builds EQL's hex OPE term from the native order-preserving term. +/// +/// The bytes are Stack Encrypt's CLLW OPE ciphertext of the value's +/// ADR-0002 order key, which `vitaminc-ore` encodes: an integer at its own +/// width, a float as the `f64` it widens to (`-0.0` ties `0.0`, every NaN is +/// one value after `+Infinity`), text in Unicode 16 NFC with case and accents +/// folded. EQL compares `op` as `bytea`, which is the lexicographic byte +/// order the scheme preserves, so `<` on the stored terms is `<` on the +/// order keys. Matching cipherstash-client's bytes is not a goal: +/// cipherstash-client never writes an `eql_v4` column. +/// +/// Text with no fold (a code point Unicode 16 does not assign) has no order +/// term and is refused, stored or queried. +#[doc(hidden)] +pub struct OpeVisitor; +impl Visitor for OpeVisitor { + type Value = OpeCllw; + fn ope(self, term: OpeTerm) -> Result { + Ok(OpeCllw(hex::encode(term.as_bytes()))) + } +} +impl Transcode for OpeCllw { + type Visitor = OpeVisitor; + fn visitor() -> Self::Visitor { + OpeVisitor + } +} +impl EncryptFrom for OpeCllw { + type Context = CallerContext; + fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context> + where + S: 's, + { + target::ope().transcode() + } +} +impl Decryptable for OpeCllw { + const DECRYPTABLE: bool = false; +} +impl DecryptField for OpeCllw { + fn decryption_field(self, _: Ctx) -> Option> { + None + } +} + #[cfg(test)] mod tests { use stack_encrypt::{nonempty, Descriptor, Label}; diff --git a/packages/eql/crates/eql-bindings/src/v4/bigint.rs b/packages/eql/crates/eql-bindings/src/v4/bigint.rs index 9d1009024..9fc19460d 100644 --- a/packages/eql/crates/eql-bindings/src/v4/bigint.rs +++ b/packages/eql/crates/eql-bindings/src/v4/bigint.rs @@ -155,10 +155,18 @@ impl DomainType for BigintOrdOre { doc, doc = " See the [EQL bigint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`i64`](i64). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = i64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct BigintOrd { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -197,10 +205,18 @@ impl DomainType for BigintOrd { doc, doc = " See the [EQL bigint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`i64`](i64). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = i64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct BigintOrdOpe { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -329,10 +345,18 @@ impl DomainType for BigintOrdOreQuery { doc, doc = " See the [EQL bigint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`i64`](i64). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = i64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct BigintOrdQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } @@ -370,10 +394,18 @@ impl DomainType for BigintOrdQuery { doc, doc = " See the [EQL bigint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`i64`](i64). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = i64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct BigintOrdOpeQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } diff --git a/packages/eql/crates/eql-bindings/src/v4/double.rs b/packages/eql/crates/eql-bindings/src/v4/double.rs index f20074728..b45aa291c 100644 --- a/packages/eql/crates/eql-bindings/src/v4/double.rs +++ b/packages/eql/crates/eql-bindings/src/v4/double.rs @@ -155,10 +155,18 @@ impl DomainType for DoubleOrdOre { doc, doc = " See the [EQL double reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`f64`](f64). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = f64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct DoubleOrd { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -197,10 +205,18 @@ impl DomainType for DoubleOrd { doc, doc = " See the [EQL double reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`f64`](f64). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = f64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct DoubleOrdOpe { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -329,10 +345,18 @@ impl DomainType for DoubleOrdOreQuery { doc, doc = " See the [EQL double reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`f64`](f64). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = f64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct DoubleOrdQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } @@ -370,10 +394,18 @@ impl DomainType for DoubleOrdQuery { doc, doc = " See the [EQL double reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`f64`](f64). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = f64)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct DoubleOrdOpeQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } diff --git a/packages/eql/crates/eql-bindings/src/v4/integer.rs b/packages/eql/crates/eql-bindings/src/v4/integer.rs index c6b0037dc..72fed0f39 100644 --- a/packages/eql/crates/eql-bindings/src/v4/integer.rs +++ b/packages/eql/crates/eql-bindings/src/v4/integer.rs @@ -155,10 +155,18 @@ impl DomainType for IntegerOrdOre { doc, doc = " See the [EQL integer reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`i32`](i32). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = i32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct IntegerOrd { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -197,10 +205,18 @@ impl DomainType for IntegerOrd { doc, doc = " See the [EQL integer reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`i32`](i32). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = i32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct IntegerOrdOpe { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -329,10 +345,18 @@ impl DomainType for IntegerOrdOreQuery { doc, doc = " See the [EQL integer reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`i32`](i32). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = i32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct IntegerOrdQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } @@ -370,10 +394,18 @@ impl DomainType for IntegerOrdQuery { doc, doc = " See the [EQL integer reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`i32`](i32). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = i32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct IntegerOrdOpeQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } diff --git a/packages/eql/crates/eql-bindings/src/v4/real.rs b/packages/eql/crates/eql-bindings/src/v4/real.rs index e6d316bc6..171906eee 100644 --- a/packages/eql/crates/eql-bindings/src/v4/real.rs +++ b/packages/eql/crates/eql-bindings/src/v4/real.rs @@ -155,10 +155,18 @@ impl DomainType for RealOrdOre { doc, doc = " See the [EQL real reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`f32`](f32). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = f32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct RealOrd { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -197,10 +205,18 @@ impl DomainType for RealOrd { doc, doc = " See the [EQL real reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`f32`](f32). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = f32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct RealOrdOpe { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -329,10 +345,18 @@ impl DomainType for RealOrdOreQuery { doc, doc = " See the [EQL real reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`f32`](f32). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = f32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct RealOrdQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } @@ -370,10 +394,18 @@ impl DomainType for RealOrdQuery { doc, doc = " See the [EQL real reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`f32`](f32). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = f32)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct RealOrdOpeQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } diff --git a/packages/eql/crates/eql-bindings/src/v4/smallint.rs b/packages/eql/crates/eql-bindings/src/v4/smallint.rs index 424182bdf..68aeee4f5 100644 --- a/packages/eql/crates/eql-bindings/src/v4/smallint.rs +++ b/packages/eql/crates/eql-bindings/src/v4/smallint.rs @@ -155,10 +155,18 @@ impl DomainType for SmallintOrdOre { doc, doc = " See the [EQL smallint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`i16`](i16). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = i16)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct SmallintOrd { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -197,10 +205,18 @@ impl DomainType for SmallintOrd { doc, doc = " See the [EQL smallint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`i16`](i16). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = i16)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct SmallintOrdOpe { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub op: OpeCllw, @@ -329,10 +345,18 @@ impl DomainType for SmallintOrdOreQuery { doc, doc = " See the [EQL smallint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`i16`](i16). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = i16)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct SmallintOrdQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } @@ -370,10 +394,18 @@ impl DomainType for SmallintOrdQuery { doc, doc = " See the [EQL smallint reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`i16`](i16). Implements `EncryptFrom` to build order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = i16)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct SmallintOrdOpeQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub op: OpeCllw, } diff --git a/packages/eql/crates/eql-bindings/src/v4/targets.rs b/packages/eql/crates/eql-bindings/src/v4/targets.rs index 846b7bf36..8f352b7a7 100644 --- a/packages/eql/crates/eql-bindings/src/v4/targets.rs +++ b/packages/eql/crates/eql-bindings/src/v4/targets.rs @@ -58,10 +58,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("IntegerOrdQuery"), query_sql_domain: Some("eql_v4.query_integer_ord"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "IntegerOrdOpe", @@ -72,10 +70,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("IntegerOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_integer_ord_ope"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "Smallint", @@ -124,10 +120,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("SmallintOrdQuery"), query_sql_domain: Some("eql_v4.query_smallint_ord"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "SmallintOrdOpe", @@ -138,10 +132,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("SmallintOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_smallint_ord_ope"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "Bigint", @@ -190,10 +182,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("BigintOrdQuery"), query_sql_domain: Some("eql_v4.query_bigint_ord"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "BigintOrdOpe", @@ -204,10 +194,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("BigintOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_bigint_ord_ope"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "Date", @@ -478,10 +466,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["eq", "ope"], query: Some("TextOrdQuery"), query_sql_domain: Some("eql_v4.query_text_ord"), - producible: false, - reason: Some( - "no EQL type is built from the engine's text order terms yet (#1131)", - ), + producible: true, + reason: None, }, Target { name: "TextOrdOpe", @@ -492,10 +478,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["eq", "ope"], query: Some("TextOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_text_ord_ope"), - producible: false, - reason: Some( - "no EQL type is built from the engine's text order terms yet (#1131)", - ), + producible: true, + reason: None, }, Target { name: "TextSearchOre", @@ -522,7 +506,7 @@ pub const TARGETS: &[Target] = &[ query_sql_domain: Some("eql_v4.query_text_search"), producible: false, reason: Some( - "no EQL type is built from the engine's text order terms yet (#1131)", + "every query of this type carries a match term, which text shorter than one 3-gram cannot make, so `=` or `<` on short text would be refused; how a query that is not a match leaves it out is not decided (#1131)", ), }, Target { @@ -584,10 +568,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("RealOrdQuery"), query_sql_domain: Some("eql_v4.query_real_ord"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "RealOrdOpe", @@ -598,10 +580,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("RealOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_real_ord_ope"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "Double", @@ -650,10 +630,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("DoubleOrdQuery"), query_sql_domain: Some("eql_v4.query_double_ord"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "DoubleOrdOpe", @@ -664,10 +642,8 @@ pub const TARGETS: &[Target] = &[ indexes: &["ope"], query: Some("DoubleOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_double_ord_ope"), - producible: false, - reason: Some( - "the engine does not yet derive this family's order terms over ADR-0002's order keys (#1236)", - ), + producible: true, + reason: None, }, Target { name: "SteVecDocument", @@ -712,6 +688,18 @@ pub(crate) fn encrypt_named<'a, K: 'static>( "IntegerEq" => { run_target::("IntegerEq", keyset, column, plaintext) } + "IntegerOrd" => run_target::( + "IntegerOrd", + keyset, + column, + plaintext, + ), + "IntegerOrdOpe" => run_target::( + "IntegerOrdOpe", + keyset, + column, + plaintext, + ), "Smallint" => { run_target::("Smallint", keyset, column, plaintext) } @@ -721,12 +709,33 @@ pub(crate) fn encrypt_named<'a, K: 'static>( column, plaintext, ), + "SmallintOrd" => run_target::( + "SmallintOrd", + keyset, + column, + plaintext, + ), + "SmallintOrdOpe" => run_target::( + "SmallintOrdOpe", + keyset, + column, + plaintext, + ), "Bigint" => { run_target::("Bigint", keyset, column, plaintext) } "BigintEq" => { run_target::("BigintEq", keyset, column, plaintext) } + "BigintOrd" => { + run_target::("BigintOrd", keyset, column, plaintext) + } + "BigintOrdOpe" => run_target::( + "BigintOrdOpe", + keyset, + column, + plaintext, + ), "Text" => run_target::("Text", keyset, column, plaintext), "TextEq" => { run_target::("TextEq", keyset, column, plaintext) @@ -734,17 +743,41 @@ pub(crate) fn encrypt_named<'a, K: 'static>( "TextMatch" => { run_target::("TextMatch", keyset, column, plaintext) } + "TextOrd" => { + run_target::("TextOrd", keyset, column, plaintext) + } + "TextOrdOpe" => run_target::( + "TextOrdOpe", + keyset, + column, + plaintext, + ), "Boolean" => { run_target::("Boolean", keyset, column, plaintext) } "Real" => run_target::("Real", keyset, column, plaintext), "RealEq" => run_target::("RealEq", keyset, column, plaintext), + "RealOrd" => { + run_target::("RealOrd", keyset, column, plaintext) + } + "RealOrdOpe" => { + run_target::("RealOrdOpe", keyset, column, plaintext) + } "Double" => { run_target::("Double", keyset, column, plaintext) } "DoubleEq" => { run_target::("DoubleEq", keyset, column, plaintext) } + "DoubleOrd" => { + run_target::("DoubleOrd", keyset, column, plaintext) + } + "DoubleOrdOpe" => run_target::( + "DoubleOrdOpe", + keyset, + column, + plaintext, + ), _ => Err(refuse(name)), } } @@ -763,30 +796,79 @@ pub(crate) fn decrypt_named<'a, K: 'static>( "IntegerEq" => { open_target::("IntegerEq", opener, column, stored) } + "IntegerOrd" => { + open_target::("IntegerOrd", opener, column, stored) + } + "IntegerOrdOpe" => open_target::( + "IntegerOrdOpe", + opener, + column, + stored, + ), "Smallint" => { open_target::("Smallint", opener, column, stored) } "SmallintEq" => { open_target::("SmallintEq", opener, column, stored) } + "SmallintOrd" => open_target::( + "SmallintOrd", + opener, + column, + stored, + ), + "SmallintOrdOpe" => open_target::( + "SmallintOrdOpe", + opener, + column, + stored, + ), "Bigint" => open_target::("Bigint", opener, column, stored), "BigintEq" => { open_target::("BigintEq", opener, column, stored) } + "BigintOrd" => { + open_target::("BigintOrd", opener, column, stored) + } + "BigintOrdOpe" => open_target::( + "BigintOrdOpe", + opener, + column, + stored, + ), "Text" => open_target::("Text", opener, column, stored), "TextEq" => open_target::("TextEq", opener, column, stored), "TextMatch" => { open_target::("TextMatch", opener, column, stored) } + "TextOrd" => { + open_target::("TextOrd", opener, column, stored) + } + "TextOrdOpe" => { + open_target::("TextOrdOpe", opener, column, stored) + } "Boolean" => { open_target::("Boolean", opener, column, stored) } "Real" => open_target::("Real", opener, column, stored), "RealEq" => open_target::("RealEq", opener, column, stored), + "RealOrd" => open_target::("RealOrd", opener, column, stored), + "RealOrdOpe" => { + open_target::("RealOrdOpe", opener, column, stored) + } "Double" => open_target::("Double", opener, column, stored), "DoubleEq" => { open_target::("DoubleEq", opener, column, stored) } + "DoubleOrd" => { + open_target::("DoubleOrd", opener, column, stored) + } + "DoubleOrdOpe" => open_target::( + "DoubleOrdOpe", + opener, + column, + stored, + ), _ => Err(refuse(name)), } } @@ -806,15 +888,51 @@ pub(crate) fn query_named<'a, K: 'static>( column, plaintext, ), + "IntegerOrd" => run_target::( + "IntegerOrd", + keyset, + column, + plaintext, + ), + "IntegerOrdOpe" => run_target::( + "IntegerOrdOpe", + keyset, + column, + plaintext, + ), "SmallintEq" => run_target::( "SmallintEq", keyset, column, plaintext, ), + "SmallintOrd" => run_target::( + "SmallintOrd", + keyset, + column, + plaintext, + ), + "SmallintOrdOpe" => run_target::( + "SmallintOrdOpe", + keyset, + column, + plaintext, + ), "BigintEq" => run_target::( "BigintEq", keyset, column, plaintext, ), + "BigintOrd" => run_target::( + "BigintOrd", + keyset, + column, + plaintext, + ), + "BigintOrdOpe" => run_target::( + "BigintOrdOpe", + keyset, + column, + plaintext, + ), "TextEq" => { run_target::("TextEq", keyset, column, plaintext) } @@ -824,12 +942,42 @@ pub(crate) fn query_named<'a, K: 'static>( column, plaintext, ), + "TextOrd" => { + run_target::("TextOrd", keyset, column, plaintext) + } + "TextOrdOpe" => run_target::( + "TextOrdOpe", + keyset, + column, + plaintext, + ), "RealEq" => { run_target::("RealEq", keyset, column, plaintext) } + "RealOrd" => { + run_target::("RealOrd", keyset, column, plaintext) + } + "RealOrdOpe" => run_target::( + "RealOrdOpe", + keyset, + column, + plaintext, + ), "DoubleEq" => run_target::( "DoubleEq", keyset, column, plaintext, ), + "DoubleOrd" => run_target::( + "DoubleOrd", + keyset, + column, + plaintext, + ), + "DoubleOrdOpe" => run_target::( + "DoubleOrdOpe", + keyset, + column, + plaintext, + ), _ => Err(refuse_query(name)), } } diff --git a/packages/eql/crates/eql-bindings/src/v4/text.rs b/packages/eql/crates/eql-bindings/src/v4/text.rs index 2e549eca0..e454fa534 100644 --- a/packages/eql/crates/eql-bindings/src/v4/text.rs +++ b/packages/eql/crates/eql-bindings/src/v4/text.rs @@ -206,10 +206,18 @@ impl DomainType for TextOrdOre { doc, doc = " See the [EQL text reference](https://cipherstash.com/docs/reference/eql/text) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`String`](String). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = String)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct TextOrd { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub hm: Hmac256, @@ -249,10 +257,18 @@ impl DomainType for TextOrd { doc, doc = " See the [EQL text reference](https://cipherstash.com/docs/reference/eql/text) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`String`](String). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = String)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct TextOrdOpe { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub hm: Hmac256, @@ -520,10 +536,18 @@ impl DomainType for TextOrdOreQuery { doc, doc = " See the [EQL text reference](https://cipherstash.com/docs/reference/eql/text) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`String`](String). Implements `EncryptFrom` to build equality and order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = String)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct TextOrdQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub hm: Hmac256, pub op: OpeCllw, @@ -562,10 +586,18 @@ impl DomainType for TextOrdQuery { doc, doc = " See the [EQL text reference](https://cipherstash.com/docs/reference/eql/text) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`String`](String). Implements `EncryptFrom` to build equality and order query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = String)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct TextOrdOpeQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub hm: Hmac256, pub op: OpeCllw, diff --git a/packages/eql/crates/eql-codegen/src/bindings.rs b/packages/eql/crates/eql-codegen/src/bindings.rs index 2259d99ab..a385acedd 100644 --- a/packages/eql/crates/eql-codegen/src/bindings.rs +++ b/packages/eql/crates/eql-codegen/src/bindings.rs @@ -335,24 +335,37 @@ fn render_query_struct(eql: EqlVersion, family: &DomainFamily, domain: &Domain) /// The `(family, domain)` pairs whose generated v4 structs carry the opt-in /// `stack-encrypt` derives: the storage and equality domains of every family -/// whose plaintext eql-bindings names (`plaintext_of`), and text match. Every +/// whose plaintext eql-bindings names (`plaintext_of`), the OPE-ordered +/// domains (`ord`, `ord_ope`) of the families listed, and text match. Every /// other stored domain has a reason in [`encryption_gap`], and a test holds the two together so a new domain /// cannot arrive without either a derive or a reason. pub const ENCRYPTION_DOMAINS: &[(&str, &str)] = &[ ("integer", ""), ("integer", "eq"), + ("integer", "ord"), + ("integer", "ord_ope"), ("smallint", ""), ("smallint", "eq"), + ("smallint", "ord"), + ("smallint", "ord_ope"), ("bigint", ""), ("bigint", "eq"), + ("bigint", "ord"), + ("bigint", "ord_ope"), ("text", ""), ("text", "eq"), ("text", "match"), + ("text", "ord"), + ("text", "ord_ope"), ("boolean", ""), ("real", ""), ("real", "eq"), + ("real", "ord"), + ("real", "ord_ope"), ("double", ""), ("double", "eq"), + ("double", "ord"), + ("double", "ord_ope"), ]; /// Why no `eql_v3` struct derives Stack Encrypt's traits. @@ -403,14 +416,15 @@ pub fn encryption_gap( not decided (#1130)" } else if domain.terms.contains(&Term::Ore) { "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)" - } else if domain.terms.contains(&Term::Ope) && family.name == "text" { - "no EQL type is built from the engine's text order terms yet (#1131)" - } else if domain.terms.contains(&Term::Ope) { - "the engine does not yet derive this family's order terms over ADR-0002's order keys \ - (#1236)" + } else if domain.terms.contains(&Term::Bloom) && domain.terms.contains(&Term::Ope) { + "every query of this type carries a match term, which text shorter than one 3-gram \ + cannot make, so `=` or `<` on short text would be refused; how a query that is not a \ + match leaves it out is not decided (#1131)" } else { - // Reached by an equality-only domain of a family whose plaintext is - // specified but which is not listed: nothing in the catalog today. + // Reached by any unlisted domain of a family whose plaintext is + // specified and whose terms are neither ORE nor match-and-order (an + // equality or OPE-ordered domain of a family that gains a plaintext, + // say): nothing in the catalog today. "this domain derives the way its family's other domains do; it is not listed in \ ENCRYPTION_DOMAINS yet" }) @@ -436,10 +450,15 @@ fn encryption_attrs( let shown = rust.rsplit("::").next().unwrap_or(rust); let plaintext_doc = |named: &str| { if query { - let terms = if domain.terms.contains(&Term::Bloom) { - "match" - } else { - "equality" + let terms = match ( + domain.terms.contains(&Term::Bloom), + domain.terms.contains(&Term::Hm), + domain.terms.contains(&Term::Ope), + ) { + (true, _, _) => "match", + (false, true, true) => "equality and order", + (false, false, true) => "order", + _ => "equality", }; format!( " Plaintext input: {named}. Implements `EncryptFrom<{shown}>` to build {terms} \ @@ -1725,14 +1744,15 @@ mod tests { "no Rust plaintext type for this family" } else if domain.terms.contains(&Term::Ore) { "block ORE terms" - } else if domain.terms.contains(&Term::Ope) { - if family.name == "text" { - "text order terms" - } else { - "order keys (#1236)" - } + } else if domain.terms.contains(&Term::Bloom) + && domain.terms.contains(&Term::Ope) + { + "carries a match term" } else { - panic!("{full}: every specified, unordered domain is derived: {reason}") + panic!( + "{full}: every specified domain without ORE or a match-and-order \ + query is derived: {reason}" + ) }; assert!(reason.contains(expected), "{full}: {reason}"); } diff --git a/packages/eql/crates/eql-codegen/src/go_eql.rs b/packages/eql/crates/eql-codegen/src/go_eql.rs index e644e0a73..11650f3db 100644 --- a/packages/eql/crates/eql-codegen/src/go_eql.rs +++ b/packages/eql/crates/eql-codegen/src/go_eql.rs @@ -127,7 +127,15 @@ pub fn render_go_eql() -> String { ) }; let status = match (row.reason, row.plaintext) { - (None, Some(kind)) => format!("The engine produces it from a {kind}."), + (None, Some(kind)) => { + // The Go kinds are read as words: "an int32", "a float64". + let article = if kind.starts_with(['a', 'e', 'i', 'o', 'u']) { + "an" + } else { + "a" + }; + format!("The engine produces it from {article} {kind}.") + } (None, None) => "The engine produces it.".to_owned(), (Some(reason), _) => format!("The engine cannot produce it yet: {reason}."), }; @@ -213,6 +221,14 @@ mod tests { } } + #[test] + fn the_plaintext_kind_takes_the_article_it_is_read_with() { + let out = render_go_eql(); + assert!(out.contains("The engine produces it from an int32.")); + assert!(out.contains("The engine produces it from a float64.")); + assert!(!out.contains("from a int"), "a int32 or a int64"); + } + #[test] fn rendering_is_deterministic() { assert_eq!(render_go_eql(), render_go_eql()); diff --git a/packages/eql/crates/eql-codegen/src/targets.rs b/packages/eql/crates/eql-codegen/src/targets.rs index d63ece55b..73b6b3f4a 100644 --- a/packages/eql/crates/eql-codegen/src/targets.rs +++ b/packages/eql/crates/eql-codegen/src/targets.rs @@ -19,11 +19,10 @@ //! [`encryption_gap`](crate::bindings::encryption_gap) on every domain. //! //! The reasons a type is not producible, each naming the issue that tracks -//! it: no Rust plaintext type is chosen for the family yet (#1130); the -//! engine's order terms are not yet derived over ADR-0002's order keys -//! (#1236) or, for text, no EQL type is built from them (#1131); EQL's ORE -//! term is block ORE where the engine derives CLLW ORE (#1132); and the JSON -//! index is a new engine operation (#1060). +//! it: no Rust plaintext type is chosen for the family yet (#1130); EQL's +//! ORE term is block ORE where the engine derives CLLW ORE (#1132); a +//! `TextSearch` query carries a match term that short text cannot make +//! (#1131); and the JSON index is a new engine operation (#1060). use proc_macro2::TokenStream; use quote::{format_ident, quote}; @@ -391,15 +390,13 @@ mod tests { target_gap(family, family.domain_by_name(name).expect("domain")) }; let text = eql_domains::TEXT; - for producible in ["", "eq", "match"] { + for producible in ["", "eq", "match", "ord", "ord_ope"] { assert!(gap(&text, producible).is_none(), "text {producible:?}"); } - for ordered in ["ord", "ord_ope", "search"] { - assert!( - gap(&text, ordered).unwrap().contains("text order terms"), - "{ordered}" - ); - } + assert!( + gap(&text, "search").unwrap().contains("match term"), + "TextSearch waits on how a query leaves out its match term" + ); for block_ore in ["ord_ore", "search_ore"] { assert!( gap(&text, block_ore).unwrap().contains("block ORE"), @@ -416,11 +413,12 @@ mod tests { for family_name in ["integer", "smallint", "bigint", "real", "double"] { let family = family(family_name); assert!(gap(family, "").is_none(), "{family_name}"); - assert!(gap(family, "eq").is_none(), "{family_name}Eq is producible"); - assert!( - gap(family, "ord").unwrap().contains("#1236"), - "{family_name}" - ); + for producible in ["eq", "ord", "ord_ope"] { + assert!( + gap(family, producible).is_none(), + "{family_name} {producible}" + ); + } assert!( gap(family, "ord_ore").unwrap().contains("block ORE"), "{family_name}" diff --git a/packages/eql/mise.toml b/packages/eql/mise.toml index 98d1d10d4..4af6bdf8e 100644 --- a/packages/eql/mise.toml +++ b/packages/eql/mise.toml @@ -340,7 +340,7 @@ psql "$EQL_TEST_DATABASE_URL" -v ON_ERROR_STOP=1 -f release/cipherstash-encrypt- # `--ignored` selects by attribute, and cargo exits 0 on an empty selection, so # a test that loses its `#[ignore]` or is renamed would leave this task passing # on nothing. List what the filter selects before running it. -for suite in text_eq v4_types; do +for suite in text_eq v4_types v4_order; do listing=$(cargo test --locked -p eql-encryption-tests --test "$suite" -- --ignored --list | sed -n 's/: test$//p') [ -n "$listing" ] || { echo "error: '--ignored' selects no test in $suite; the PostgreSQL test has gone missing." >&2; exit 1; } cargo test --locked -p eql-encryption-tests --test "$suite" -- --ignored diff --git a/packages/eql/tests/encryption/fixtures/README.md b/packages/eql/tests/encryption/fixtures/README.md index 4fb784ea0..4a7969a45 100644 --- a/packages/eql/tests/encryption/fixtures/README.md +++ b/packages/eql/tests/encryption/fixtures/README.md @@ -4,10 +4,19 @@ | --- | --- | --- | | `v4_rows.json` | the Go SDK's `TestV4RowsMatchTheCommittedFixture` | yes, with `EQL_UPDATE_FIXTURES=1` | | `v4_queries.json` | the by-name dispatch in `tests/v4_types.rs` | yes, with `EQL_UPDATE_FIXTURES=1` | +| `v4_ordered_queries.json` | the by-name dispatch in `tests/v4_order.rs` | yes, with `EQL_UPDATE_FIXTURES=1` | +| `v4_ordered_rows.json` | the Go SDK's `TestV4OrderedRowsMatchTheCommittedFixture` | yes, with `EQL_UPDATE_FIXTURES=1` | | `text_eq_query.json`, `ste_vec_selectors.json` | `tests/text_eq.rs`, `tests/ste_vec_selectors.rs` | yes, see each test | | `v4_smallint_rows.json` | the Go SDK's `TestV4SmallintRowsMatchTheCommittedFixture` | yes, with `EQL_UPDATE_FIXTURES=1` | | `stored_v4_rows_v1.json` | copied from `v4_rows.json` at the next build | **never: frozen** | +`v4_ordered_queries.json` and `v4_ordered_rows.json` are not frozen: their +`op` bytes are the order terms Stack Encrypt writes today, so a change that moves +those bytes regenerates both together (Go writes the rows, Rust writes the +queries, each with `EQL_UPDATE_FIXTURES=1`). No frozen +`stored_v4_ordered_rows_*` file exists yet; one is cut once the order-term +bytes are final. + ## `stored_v4_rows_v1.json` (FROZEN) `Profile` rows the Go SDK's generated code stored with the `next` build, one diff --git a/packages/eql/tests/encryption/fixtures/v4_ordered_queries.json b/packages/eql/tests/encryption/fixtures/v4_ordered_queries.json new file mode 100644 index 000000000..a494c0791 --- /dev/null +++ b/packages/eql/tests/encryption/fixtures/v4_ordered_queries.json @@ -0,0 +1,44 @@ +{ + "_comment": "The v4 query operands eql-bindings derives for one plaintext of each OPE-ordered type under FakeDataKeySource's index key (keyset nil), read by the Go SDK's hermetic test; regenerate with EQL_UPDATE_FIXTURES=1, do not edit.", + "floor": { + "plaintext": -3, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"floor\"},\"op\":\"0068773fbf20d83c18c9e2c158d14a9601d12eab933e59d10e163110e4f030377b\"}" + }, + "handle": { + "plaintext": "carol", + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"handle\"},\"hm\":\"30fca14a65e5695fc96bd9fc7e3747afe79c7ff2993d532e3ad97f8e8b95ffac\",\"op\":\"00e169c8297d7be0a24e597d20e140b6a098c488dda2a5fab8b1a7071ff703df7f0880519d1ea23890b5c6a01066f62633\"}" + }, + "level": { + "plaintext": 10, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"level\"},\"op\":\"0159d19c111791f639ba0638692ca6f6c53ecac713e356a7e8bf95fc2d839691f4\"}" + }, + "mean": { + "plaintext": 0.0, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"mean\"},\"op\":\"017bc9729378e320ff0c1504dc995074a7db7b05d24bed8a923f6944899e4517d213bba21ee639bb01063902e92cad61b3f3b0755b0dacc66fa5188b434c455360\"}" + }, + "pace": { + "plaintext": -0.5, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"pace\"},\"op\":\"00028af974e429d69504e291c6615487c98d3abbaf6a36a012b22bd60ddb569ba83f09243bb5f22d9df419926a7b9e31c7c4411c822471fd427c7a74ac57df0482\"}" + }, + "points": { + "plaintext": 5000000000, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"points\"},\"op\":\"0124da6c76e6a9d3572f27a08b17b843e6581a23213f34fea500706db6d0edf539fee6624d544ab352bbfcd0987056c61e7e2b808fcb36eef07e673ec3639e4235\"}" + }, + "score": { + "plaintext": 9.75, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"score\"},\"op\":\"00cf2820721cd8c187f434f8986fd01c7eb126c713b0619086b0ca6d0b2120f8d322bbc60d5e40ac640ec49f271c0894a6965ab22b7da190f2e2af8f157a7b503f\"}" + }, + "speed": { + "plaintext": 2.5, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"speed\"},\"op\":\"00bbc2ac417a8cb6a4b5b69c9f81223f9016f7ffc9e250417dd5afa5cfd4844dac6f9f72bf972a4aba57ae0f7e64bc7ce315095aab620f1f0f7a567d525837af8c\"}" + }, + "table": "rankings", + "title": { + "plaintext": "Mango", + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"title\"},\"hm\":\"5d14d6d4a7e93c15cf6b7a0e92dc102adbfcbfb7c0492ad09d6b1938bafb748b\",\"op\":\"00344fe120b7abb89fdad4d2ace56e65d225dc12e1f08841a86ad18728afb74d36c3bdd298ed9a2e94e98a2e9328a2fa76\"}" + }, + "total": { + "plaintext": -1, + "query": "{\"v\":4,\"i\":{\"t\":\"rankings\",\"c\":\"total\"},\"op\":\"007152e605e2cc13823bd1379c7733939403474f5dfe1a65fa73eb63ea134f4a0505668fc80076fb24aaefd4763b1d1a2be9bf875178cf8f4abdbdce276f071ed9\"}" + } +} diff --git a/packages/eql/tests/encryption/fixtures/v4_ordered_rows.json b/packages/eql/tests/encryption/fixtures/v4_ordered_rows.json new file mode 100644 index 000000000..4fa6224dc --- /dev/null +++ b/packages/eql/tests/encryption/fixtures/v4_ordered_rows.json @@ -0,0 +1,334 @@ +{ + "_comment": "Ranking rows sealed by the Go SDK's generated code over the deterministic EQL guest with the seed below; the Rust suite opens and compares them. Regenerate with EQL_UPDATE_FIXTURES=1 go test ./encrypt -run TestV4OrderedRows, do not edit.", + "seed": "encrypt round-trip tests seed v1", + "table": "rankings", + "rows": [ + { + "id": 1, + "plaintext": { + "level": -20, + "floor": -30, + "points": -5, + "total": -9, + "speed": -1.25, + "pace": -3.5, + "score": -2.5, + "mean": -0.25, + "title": "apple", + "handle": "Alice" + }, + "stored": { + "floor": { + "v": 4, + "i": { + "t": "rankings", + "c": "floor" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABn4yMF67G89OFjQZZuk7z/IAAPy1eGXRyPxy1IfAwzQCPlOMdditoc/8fNRNI2Iw3VQwJX+YR4eii1fcxB+KJtUD2cDQHwylkneQqa8OgbYJYiWTc=", + "op": "0068773fbf20d83c18c9e2c158d14a9601d12eab933e59d10e16311064f92eac47" + }, + "handle": { + "v": 4, + "i": { + "t": "rankings", + "c": "handle" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACao1GJ/o4L0iCBtDrzF6P4IADDycFng7kuIpwyCSaqaE7ZJDE3q+nAEKaMK9GelLdntgK3RJEIWuJIwJoaqe10CfLP0ObM8l+bgj/harYScfipBL4g", + "hm": "fd9e4ef66848b4dada73e4262e85c9994af16aff271638ef9ec695d460e55076", + "op": "00e169c8297d7be0a24e597d20e14036df866a797b2cf54c44b7ae8b5540f7055f407ed6f91137efd272331b242fbe1ba1" + }, + "level": { + "v": 4, + "i": { + "t": "rankings", + "c": "level" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADOELB7oFk4gGnte9gWUzX+IAA/eO5LaKSJjCCWDqc30OLhtVNEMuxqU8YpeFdvKCq/zwJuoJAKC6CVvRXxddKK93rVWrvRV5ZLBIdjTRO5wEkx4/M=", + "op": "00da028bda7156e277d149180f09a615bb7f3f45efabe00137a04f5514176bc162" + }, + "mean": { + "v": 4, + "i": { + "t": "rankings", + "c": "mean" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAXtGGpXlAfrs+IrloNwZO2IACelhdo3FYPTm2Ee6gu3+WpSHxU3IZtSHO2tiiQdtYltALT0WMwiFSu9v/C372nHnQ9Y2F5lQ+SJ9a6UGl7q1Wnzg6OavYb", + "op": "00fbce95d005970edd747e9dbff02f9824d8fedff874593ae7ddceabb8034978ead8b5f3a99c55793ed7912cb85f2cb5b600cc608ead9edfbf551e4d1544fff0d7" + }, + "pace": { + "v": 4, + "i": { + "t": "rankings", + "c": "pace" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADbT5WtnwkRYMqrlrr5MYHyIAB3uJpHJLZ4qKadvcMBxL3zAMxDkh86LddqQyvJSQAvugKLD9zlce2vNCOnYmInd4TrsCm94ahuAenaAUtKbFc5Fhk=", + "op": "00020a960d8eb4ca100a36cdccaca4b2685bcd7b343e092f8133a320b1d0cb6ec85408fb3485228c62b3512f97dc9b86ff4021b728ce7aec0b99f5bbe72a3fd870" + }, + "points": { + "v": 4, + "i": { + "t": "rankings", + "c": "points" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADE2rUct3jjTKCWivCCCvn+IACDxItrcNhNJjpoOqJKOryPLgDOg+vvfrH4031ThcjDcgL2v4cWtT1QKjil3dECXavlyHzUuyoinNxTfIYEdhSNTtbKxK1z", + "op": "00a53e38612eddc330547218dda2f5ee376e718061a8ea3af479e8473fadca71aef7476bd681e502cb239d515f23f7ef925e386904bfa7dcea145d06c9bcec551b" + }, + "score": { + "v": 4, + "i": { + "t": "rankings", + "c": "score" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAdcnRvoEEyjJO4SJ0QoQxyIAAWeon6kK0bnzHkJPNFV77QcE9ws0SBOC8udyFu1wQVVQJ2t+3/Rkek59/J7fRQYHVLNvDLxukNBbokL5IOQCiE/3SVDWb0", + "op": "004e3516c0d33a94c4ba52f438645a422d570ec9e8ff91214f13848b1eb09310c1a23351ac149e998374c4baea8eaf9247fcf7465dd3a946be01332c42889aeb49" + }, + "speed": { + "v": 4, + "i": { + "t": "rankings", + "c": "speed" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACQhhB1srtFMhmPGNU3YdsWIADsfyGa4Tr54eU6CvZPZuGD26cBMDuzAO68y/GPMEqBTALILPPKyY05f6gsuB1SdbmqiH4qDmgFT5U58X3BsRGNNfM=", + "op": "003c173a74a6502886339a78c9c32c6aae7ebde1d06257855d71e33b84651ef15d77ddbbc7980a6fa7634fb604fd5b2f52c248e3531b61f181a2a9662ae051aaff" + }, + "title": { + "v": 4, + "i": { + "t": "rankings", + "c": "title" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABDsM58wLdL/OmzeE1bqk6fIAAZc0G+IFRon1pJAfT2YQm7ENl+LSkdnO6ABKfec8JJOwIZax/COEcTjpdSfUvH35SFQKaypGbuq36W7Qkriuu7eEkm", + "hm": "3bb29d6ed8db30040350b25bdb47263cb156666c5214d96f991850a4672eefc0", + "op": "00344fe120b7abb89fdad4d2ac644367e74b2ca4e6a9b6d4115e85b5966136c35ca2f2645bf578d8d1fe730b1465312f27" + }, + "total": { + "v": 4, + "i": { + "t": "rankings", + "c": "total" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAATWgrbBtfjqeFUZIJyCg/TIADRzWl3oad1fwHQ28RfJWb4tEsJ+2+pskxX7xjJHYOtxwKaanS7nY7gY/0rP/z1BlJQivmSTGhqkM8GGU3HYF54RbD163HI", + "op": "007152e605e2cc13823bd1379c7733939403474f5dfe1a65fa73eb63ea134f4a0505668fc80076fb24aaefd4763b1d1a2be9bf875178cf8f4abdbdce26ef1e3d98" + } + } + }, + { + "id": 2, + "plaintext": { + "level": 10, + "floor": -3, + "points": 5000000000, + "total": -1, + "speed": 2.5, + "pace": -0.5, + "score": 9.75, + "mean": 0, + "title": "mango", + "handle": "Carol" + }, + "stored": { + "floor": { + "v": 4, + "i": { + "t": "rankings", + "c": "floor" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAD8EXObyO8iZAsGLtyoF8OzIAAUnZe1c/yU3tUu7btj/FmcuWh0G3VlpdjE4Y1DYkuXYwKjI6FNx98wap6YRPYq0Pr6xumDkjkem3g1aD9Bd15gGJE=", + "op": "0068773fbf20d83c18c9e2c158d14a9601d12eab933e59d10e163110e4f030377b" + }, + "handle": { + "v": 4, + "i": { + "t": "rankings", + "c": "handle" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAJSAHc0qCJ5PzM6mX2ijrQIAAOAb/DTu0ioK8DD+OJLmo37ro6p5goGfRbZvv+4GZmmwLyq1pNP+xziI6NvqEt7inQUrLLPhDVG8j3/FIywLlE/SJx", + "hm": "28fe3661678ecdba7c98355d901c9a8a591c1175c1d7b403bd3d1ec949c8b2e0", + "op": "00e169c8297d7be0a24e597d20e140b6a098c488dda2a5fab8b1a7071ff703df7f0880519d1ea23890b5c6a01066f62633" + }, + "level": { + "v": 4, + "i": { + "t": "rankings", + "c": "level" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAhOR6KLBknqtSBVRnlASoSIAAkxpt5V1/27p77EpAoxlp3ceWsCtSnFCzVx1mJZ9TRkwK8xMXduV+d2TMvfn7+P2QF8EnlLYAiRPNnN29ANmoVSts=", + "op": "0159d19c111791f639ba0638692ca6f6c53ecac713e356a7e8bf95fc2d839691f4" + }, + "mean": { + "v": 4, + "i": { + "t": "rankings", + "c": "mean" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAipqSO612avBu8ILwcZ4zfIAClyVXmOu93Sq1j8QZEePtr9Yn/DaGkc/jIzWIJfzRtqwJ1jc06IDRZmX8wTBKbh4NQUgAPFduL3qFTTmzXxfzmMverhaSn", + "op": "017bc9729378e320ff0c1504dc995074a7db7b05d24bed8a923f6944899e4517d213bba21ee639bb01063902e92cad61b3f3b0755b0dacc66fa5188b434c455360" + }, + "pace": { + "v": 4, + "i": { + "t": "rankings", + "c": "pace" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACSUbSNa21Cz2EdccB0dU93IAB5ZsgjlyQaH70IyAEmbajCC+g5pPkdckAlAyLzt31TpwJZCCAoMewAOd8y+7jLdEfzU5mQ3zdV4dpjHzvMNoIl6q0=", + "op": "00028af974e429d69504e291c6615487c98d3abbaf6a36a012b22bd60ddb569ba83f09243bb5f22d9df419926a7b9e31c7c4411c822471fd427c7a74ac57df0482" + }, + "points": { + "v": 4, + "i": { + "t": "rankings", + "c": "points" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAD7BDP9k+0Hn5JvKavkeIEeIACwwz3ffvBIoUS+WhQi678dNUBCkgT8GduW9s20XVgb7AKy1BxwBvFhYdWu8TjON31J/XFZE6F1lgnSVZLat6Po+utVYHS2", + "op": "0124da6c76e6a9d3572f27a08b17b843e6581a23213f34fea500706db6d0edf539fee6624d544ab352bbfcd0987056c61e7e2b808fcb36eef07e673ec3639e4235" + }, + "score": { + "v": 4, + "i": { + "t": "rankings", + "c": "score" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAC8mEb7pUk1HxVAoAej3DB3IABYatmLY126M9pYkSV/txgFWt8Dn0gqikv85rJxYY3DsAK7P9SKYcuFN3NNXWdqAQ301cj85RYYFUnxeu9PUP6Hv3Rc0TAf", + "op": "00cf2820721cd8c187f434f8986fd01c7eb126c713b0619086b0ca6d0b2120f8d322bbc60d5e40ac640ec49f271c0894a6965ab22b7da190f2e2af8f157a7b503f" + }, + "speed": { + "v": 4, + "i": { + "t": "rankings", + "c": "speed" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAu+yQLbuCeiI1Kc93l5gM9IABDnI3BuN2mIRVGQNCz+OT9g1NBNpy73iq+RyQMrDq2SgLdiotIJVKFxL7zjyPD19PobupEvKh4GhfipnC5MhqxmUc=", + "op": "00bbc2ac417a8cb6a4b5b69c9f81223f9016f7ffc9e250417dd5afa5cfd4844dac6f9f72bf972a4aba57ae0f7e64bc7ce315095aab620f1f0f7a567d525837af8c" + }, + "title": { + "v": 4, + "i": { + "t": "rankings", + "c": "title" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAB8xVf2YzyG2GIJsjlxprEhIAARLofbMnEJkC3zRNC1Liyue93ywxEK707OQBNyU3GRRAIR6N/Yy+L9m18gSMba81M6XZu9nnwSMGCjCFmC6FtEdAC0", + "hm": "96c9b4169dddc5b00a92aca4f2141eaa81586bb3c0459c9621c6152a938bccc2", + "op": "00344fe120b7abb89fdad4d2ace56e65d225dc12e1f08841a86ad18728afb74d36c3bdd298ed9a2e94e98a2e9328a2fa76" + }, + "total": { + "v": 4, + "i": { + "t": "rankings", + "c": "total" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABsGQzdwhlf/5JQSTVnel1lIADu5SaW0n7eTUy2k1WrssfM1qoPuD3RSy12zQc69AvRNwKYqTPivNDeEnZygzTBpFhBGNtG92Y0Kltt9qEE0OEUYseWMu0H", + "op": "007152e605e2cc13823bd1379c7733939403474f5dfe1a65fa73eb63ea134f4a0505668fc80076fb24aaefd4763b1d1a2be9bf875178cf8f4abdbdce276f071ed9" + } + } + }, + { + "id": 3, + "plaintext": { + "level": 99, + "floor": 4, + "points": 6000000000, + "total": 12, + "speed": 8, + "pace": 0.75, + "score": 100, + "mean": 3.5, + "title": "Zucchini", + "handle": "dave" + }, + "stored": { + "floor": { + "v": 4, + "i": { + "t": "rankings", + "c": "floor" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAahRXPLhcOhZalhxaRKtwfIACZJ24pqBpQ/xkQSjuIcUV0/DY8mrP3JugbZIjLooy/agJXmfRZQv8gd+32H9A96zYw31oM8RyIVpR/HrQbVP/a6vM=", + "op": "00e7a865a4a6a28052352a8eb6c1265f682958013daa3f37fce5f07022a097aa85" + }, + "handle": { + "v": 4, + "i": { + "t": "rankings", + "c": "handle" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACAaXkgSLAngfKw540Hh/cyIABUzHBgSmtLU4FEA1PiYLaeQb+T1nu8+sQyX6e1dXFFnAJ8t2RiTgBT5L5jai/lA/1J1ipI8gx72xi4G78HNRc8LzU=", + "hm": "ec1d4753450fceec15afd623ce671fff76c236cbbbc81af81cef9e694a493ac8", + "op": "00e169c8297d7be0a24e597d20e1c094556c47b21bced759dd8e83e5a075f1b02cf50ad54f524ee158" + }, + "level": { + "v": 4, + "i": { + "t": "rankings", + "c": "level" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAxIHgvAEtydMinzMsMC9maIABcXRm4Ec9fNHs/VMPHO2YcFzhLBtvGDaO5/vL94Up4DwKIvL/fnqO4f4Bu9W6FNibpKRmt9aQmi/P+igsXeqf1LXc=", + "op": "0159d19c111791f639ba0638692ca6f6c53ecac713e356a7e8c016099dca23df54" + }, + "mean": { + "v": 4, + "i": { + "t": "rankings", + "c": "mean" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAA3MuHYMxBxol/l3CWUUdMwIACBix0i7ecA6Ldg1y72uB17y+CSAZFHm7KMBeJIDB9SSQJr4639om/GxjhRNe96mH/k6IVqzDKdy1vQcaiHL6M/rhySzCh1", + "op": "017c49c16621a2476ed2e40c07fd2d0b35d8f722f0b0e7bca5a4681d69a349806e3554d39f0d13392b43cfe477aed64f8d13ba4fb8e786dbb8c60b454a8a50197c" + }, + "pace": { + "v": 4, + "i": { + "t": "rankings", + "c": "pace" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADsocba1ACUuZab/S3cR0ewIACbWZ1UAMwA6rnu+PuFd4RCwsXLXgkmZwavUB2HQ5lIMAKNgBydDXntlpuWxH4iE3ALX8ukrBS2pqIyrliA+pyNHzI=", + "op": "00822a022b1129809c146cada9cd22fa5208b482279f0e58376d67e7619bf3c39ab962520422fd56d4c43422be205365bfe363bf466ba709ced360af209fdf2c8c" + }, + "points": { + "v": 4, + "i": { + "t": "rankings", + "c": "points" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACyINaK+oMYBWGqilpz+Y8WIAAAAwXxsCfja/ehP5So7kKPSpDckgOjdWAFkQ0JFNatGgIjwqEjrceceNJS9EXjqbF+syMBx0UeigKdwBZvpKg1VJUetU8L", + "op": "0124da6c76e6a9d3572f27a08b17b843e6581a23213f34fea500706db6d0edf539ff65903adb0c33d49877a8393f339203152f20ddbf199339e4e940bf5754b804" + }, + "score": { + "v": 4, + "i": { + "t": "rankings", + "c": "score" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADLJrDpNuSIT1MzDrC3NqkdIADeidZb+K23yM3QNYs+J9lmB9ImCm/0mFW58UkeELUnfgLrLahFD5GGGpELE5j98cbiAvLw/UzqRHXRZdjAzrhSGvAhJpcP", + "op": "00cf2820721cd8c187f4b49ef818bca2626025cd2725b90a8daab1fff0d28e46031a6cf188bff7fd4095aac4fd74613c395a909cb2f6ee863ff59d364234862a3a" + }, + "speed": { + "v": 4, + "i": { + "t": "rankings", + "c": "speed" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACgW67nY8FTYuEOauFB2JqpIAB5N3v5rvteb1kWiSLlblOJ9Bvwa4mzUHOFsJbqvqlsrQL8ByKpYorieYyq9NnIkuVxcznI5zABTlD5OzN1KGkjl+M=", + "op": "00bbc2ac417a8cb6a4b5b71c87a011af4b942767664acfa469bbd491bf66466451a9ee3aac2b56a2f81fcdb0a2b6c84725f60ad5bb727b2c14eda1eb054fb92592" + }, + "title": { + "v": 4, + "i": { + "t": "rankings", + "c": "title" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACoLw9QzYPeYb2lWmo5koQrIAAWqW7Qj+fimzpnYilKdFOpSi5wedGlAhzFUlLDVfQXrgI200vJrAyvuyAWRolG7M4tS7aNYBt3umCJUE7DlkMSjIO3fe+U", + "hm": "0784eaafee80c50e6a9856e256e99dd536f50c9668cba8bfef433b7bf75e22d0", + "op": "00344fe120b7abb89fdad4d32c82054f71be09c4ac588f34f122f9c53b1fdf516a7cd4f4017f13ffd88a1757c19da753ff8eb398b6832378da3944d7e286e6cb345a82887f75e76591" + }, + "total": { + "v": 4, + "i": { + "t": "rankings", + "c": "total" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADqzuNSUCG8WYJqEd0qG67nIAAN5uG4KA4yAHLXuPwfwCvXit4ti1MjxU/VymYIyK+Q9AKCtwW2fHKhvHt8T3IIrxXZAUpi7B02kqV6TuyK+8SRr5pGIgLH", + "op": "00f029fe2d68c79a0ca72ec21e5352a77673be1b39fae030898356ddae1d2cf9f0d771411be2b36c41f5215db20a3f965b37a8a6f7ff1e80c3be894eeb226759a2" + } + } + } + ] +} diff --git a/packages/eql/tests/encryption/tests/targets.rs b/packages/eql/tests/encryption/tests/targets.rs index 1d4cc4f48..cf623efe7 100644 --- a/packages/eql/tests/encryption/tests/targets.rs +++ b/packages/eql/tests/encryption/tests/targets.rs @@ -560,11 +560,11 @@ mod the_table { } } - /// The types the engine produces in EQL v4 today: the storage and - /// equality domains of every family whose plaintext eql-bindings names, - /// and text match. + /// The types the engine produces in EQL v4 today: the storage, equality + /// and OPE-ordered domains of the families eql-bindings names a plaintext + /// for, and text match. #[test] - fn the_producible_types_are_the_unordered_ones_with_a_rust_plaintext() { + fn the_producible_types_are_the_ones_with_no_ore_and_a_rust_plaintext() { let producible: Vec<&str> = targets::targets() .iter() .filter(|t| t.producible) @@ -575,18 +575,30 @@ mod the_table { [ "Integer", "IntegerEq", + "IntegerOrd", + "IntegerOrdOpe", "Smallint", "SmallintEq", + "SmallintOrd", + "SmallintOrdOpe", "Bigint", "BigintEq", + "BigintOrd", + "BigintOrdOpe", "Text", "TextEq", "TextMatch", + "TextOrd", + "TextOrdOpe", "Boolean", "Real", "RealEq", + "RealOrd", + "RealOrdOpe", "Double", - "DoubleEq" + "DoubleEq", + "DoubleOrd", + "DoubleOrdOpe" ], ); for row in targets::targets().iter().filter(|t| !t.producible) { @@ -596,8 +608,7 @@ mod the_table { ("json", "") => "tree of leaves", ("date" | "timestamp" | "numeric", _) => "no Rust plaintext type for this family", (_, "OrdOre" | "SearchOre") => "derives CLLW ORE (#1132)", - ("text", "Ord" | "OrdOpe" | "Search") => "text order terms yet (#1131)", - (_, "Ord" | "OrdOpe") => "order keys (#1236)", + ("text", "Search") => "carries a match term", (family, suffix) => panic!("{}: unexpected {family} {suffix}", row.name), }; assert!(reason.contains(expected), "{}: {reason}", row.name); diff --git a/packages/eql/tests/encryption/tests/v4_order.rs b/packages/eql/tests/encryption/tests/v4_order.rs new file mode 100644 index 000000000..f9f76da38 --- /dev/null +++ b/packages/eql/tests/encryption/tests/v4_order.rs @@ -0,0 +1,886 @@ +//! The OPE-ordered EQL v4 types Stack Encrypt produces: `IntegerOrd`, +//! `BigintOrd`, `RealOrd`, `DoubleOrd`, `TextOrd` and their `OrdOpe` twins. +//! +//! EQL compares the `op` term as `bytea` after decoding its hex, which is +//! lexicographic byte order. So the stored terms must sort as ADR-0002's +//! order keys sort: integers by value, floats as PostgreSQL's `float8` sorts +//! them (`-0.0` ties `0.0`, every NaN is one value after `+Infinity`), and +//! text by its fold (case and accents folded, Unicode 16). These tests hold +//! that through both entry points, pin the query twins to the stored terms, +//! and, against an EQL v4 database, check that `<`, `<=`, `>`, `>=`, `=` and +//! `ORDER BY` find exactly the rows the plaintexts say. + +mod common; + +use eql_bindings::encryption::targets::{self as targets}; +use eql_bindings::v4::{bigint, double, integer, real, text}; +use eql_bindings::Identifier; +use stack_encrypt::dynamic::ValueFrame; +use stack_encrypt::{Encrypt, Label, NonEmpty}; + +fn frame(value: T) -> ValueFrame { + ValueFrame::of(value).unwrap() +} + +fn column() -> NonEmpty { + Identifier::for_column("rankings", "level").unwrap() +} + +fn label() -> Label { + Label::new(["rankings", "level"]).unwrap() +} + +/// Every type this file covers, with the plaintext kind it takes. +const ORDERED: &[&str] = &[ + "IntegerOrd", + "IntegerOrdOpe", + "SmallintOrd", + "SmallintOrdOpe", + "BigintOrd", + "BigintOrdOpe", + "RealOrd", + "RealOrdOpe", + "DoubleOrd", + "DoubleOrdOpe", + "TextOrd", + "TextOrdOpe", +]; + +/// The stored JSON the named type writes for `value`, through the by-name +/// dispatch a Go plan reaches. +async fn stored( + keyset: &stack_encrypt::KeysetCipher<'_, common::Observed>, + name: &str, + value: ValueFrame, +) -> serde_json::Value { + let bytes = targets::encrypt(name, keyset, &label(), value) + .unwrap() + .await + .unwrap(); + serde_json::from_slice(&bytes).unwrap() +} + +/// The query operand the named type's twin derives for `value`. +async fn query( + keyset: &stack_encrypt::KeysetCipher<'_, common::Observed>, + name: &str, + value: ValueFrame, +) -> serde_json::Value { + let bytes = targets::query(name, keyset, &label(), value) + .unwrap() + .await + .unwrap(); + serde_json::from_slice(&bytes).unwrap() +} + +/// The `op` term of a stored value or query, as its hex. Lower-case hex +/// sorts as the bytes it encodes (each byte is two digits, and `0`–`9` sort +/// before `a`–`f`), so comparing the strings is comparing the `bytea` EQL +/// compares. +fn op(value: &serde_json::Value) -> &str { + value["op"].as_str().expect("an ordered value carries op") +} + +/// Every producible ordered type is in [`ORDERED`] and the other way +/// round: a type that becomes producible is covered here or fails. +#[test] +fn the_ordered_producible_types_are_these() { + let mut ordered: Vec<&str> = targets::targets() + .iter() + .filter(|t| t.producible && t.indexes.contains(&"ope")) + .map(|t| t.name) + .collect(); + ordered.sort_unstable(); + let mut expected = ORDERED.to_vec(); + expected.sort_unstable(); + assert_eq!(ordered, expected); + // No producible type carries an ORE term: EQL's is block ORE, and the + // engine's is CLLW ORE. + assert!(targets::targets() + .iter() + .filter(|t| t.producible) + .all(|t| !t.indexes.contains(&"ore"))); +} + +/// The stored keys of each type: the ciphertext and the OPE term, and the +/// equality term too for text, whose order term folds case and so cannot +/// answer `=`. +#[tokio::test] +async fn each_ordered_type_stores_its_catalog_terms() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for name in ORDERED { + let value = if name.starts_with("Text") { + frame("alice") + } else if name.starts_with("Integer") { + frame::(7) + } else if name.starts_with("Smallint") { + frame::(7) + } else if name.starts_with("Bigint") { + frame::(7) + } else if name.starts_with("Real") { + frame::(7.0) + } else { + frame::(7.0) + }; + let json = stored(&keyset, name, value.clone()).await; + let mut keys: Vec<&str> = json + .as_object() + .unwrap() + .keys() + .map(String::as_str) + .collect(); + keys.sort_unstable(); + let expected: &[&str] = if name.starts_with("Text") { + &["c", "hm", "i", "op", "v"] + } else { + &["c", "i", "op", "v"] + }; + assert_eq!(keys, expected, "{name}"); + assert_eq!(json["v"], 4, "{name}"); + + let q = query(&keyset, name, value.clone()).await; + assert!( + q.get("c").is_none(), + "{name}: a query carries no ciphertext" + ); + assert_eq!(op(&q), op(&json), "{name}: the query's op is the stored op"); + assert_eq!(q["hm"], json["hm"], "{name}: and so is its hm, if any"); + + let opened = targets::decrypt(name, &cipher, &label(), json.to_string().as_bytes()) + .unwrap() + .await + .unwrap(); + assert_eq!(opened.risky_ref(), value.risky_ref(), "{name}: opens back"); + } +} + +/// Assert that the terms `values` store under `name` sort as `expected` +/// says: each entry's group is its rank, so equal groups must store one +/// term and a lower group a lower term. +async fn sorts_as( + keyset: &stack_encrypt::KeysetCipher<'_, common::Observed>, + name: &str, + values: Vec<(usize, ValueFrame, &str)>, +) { + let mut terms = Vec::new(); + for (group, value, shown) in values { + let json = stored(keyset, name, value).await; + terms.push((group, op(&json).to_owned(), shown)); + } + for a in &terms { + for b in &terms { + assert_eq!( + a.1.cmp(&b.1), + a.0.cmp(&b.0), + "{name}: {} vs {}: the terms must order as the order keys do", + a.2, + b.2 + ); + } + } +} + +#[tokio::test] +async fn integer_terms_order_by_value() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + let ints = [i32::MIN, -70_000, -1, 0, 1, 255, 256, 70_000, i32::MAX]; + for name in ["IntegerOrd", "IntegerOrdOpe"] { + let values = ints + .iter() + .enumerate() + .map(|(rank, v)| (rank, frame(*v), "an i32")) + .collect(); + sorts_as(&keyset, name, values).await; + } + let smalls = [i16::MIN, -300, -1, 0, 1, 255, 256, i16::MAX]; + for name in ["SmallintOrd", "SmallintOrdOpe"] { + let values = smalls + .iter() + .enumerate() + .map(|(rank, v)| (rank, frame(*v), "an i16")) + .collect(); + sorts_as(&keyset, name, values).await; + } + let bigs = [i64::MIN, -(1 << 40), -1, 0, 1, 1 << 40, i64::MAX]; + for name in ["BigintOrd", "BigintOrdOpe"] { + let values = bigs + .iter() + .enumerate() + .map(|(rank, v)| (rank, frame(*v), "an i64")) + .collect(); + sorts_as(&keyset, name, values).await; + } +} + +#[tokio::test] +async fn float_terms_order_as_postgres_float8_does() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for name in ["DoubleOrd", "DoubleOrdOpe"] { + let values = vec![ + (0, frame(f64::NEG_INFINITY), "-inf"), + (1, frame(f64::MIN), "MIN"), + (2, frame(-1.5f64), "-1.5"), + (3, frame(-0.0f64), "-0.0"), + (3, frame(0.0f64), "0.0"), + (4, frame(f64::MIN_POSITIVE), "MIN_POSITIVE"), + (5, frame(1.5f64), "1.5"), + (6, frame(f64::MAX), "MAX"), + (7, frame(f64::INFINITY), "+inf"), + (8, frame(f64::NAN), "NaN"), + ( + 8, + frame(f64::from_bits(0xfff8_0000_0000_beef)), + "another NaN", + ), + ]; + sorts_as(&keyset, name, values).await; + } + for name in ["RealOrd", "RealOrdOpe"] { + let values = vec![ + (0, frame(f32::NEG_INFINITY), "-inf"), + (1, frame(f32::MIN), "MIN"), + (2, frame(-1.5f32), "-1.5"), + (3, frame(-0.0f32), "-0.0"), + (3, frame(0.0f32), "0.0"), + (4, frame(1.5f32), "1.5"), + (5, frame(f32::MAX), "MAX"), + (6, frame(f32::INFINITY), "+inf"), + (7, frame(f32::NAN), "NaN"), + (7, frame(f32::from_bits(0xffc0_1234)), "another NaN"), + ]; + sorts_as(&keyset, name, values).await; + } +} + +/// A `float32` orders as the `float64` it widens to, so a `RealOrd` and a +/// `DoubleOrd` column under one name store one term for one value. +#[tokio::test] +async fn a_real_orders_as_the_double_it_widens_to() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for v in [-1.5f32, 0.0, 1.5, f32::INFINITY, f32::NAN] { + let real = stored(&keyset, "RealOrd", frame(v)).await; + let double = stored(&keyset, "DoubleOrd", frame(f64::from(v))).await; + assert_eq!(op(&real), op(&double), "{v}"); + } +} + +/// Text orders by its fold: case and accents tie, `é` sorts with `e`, and +/// `alice` comes before `Zoe`. A prefix sorts first. +#[tokio::test] +async fn text_terms_order_by_the_fold() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for name in ["TextOrd", "TextOrdOpe"] { + let values = vec![ + (0, frame(""), "the empty string"), + (1, frame("al"), "al"), + (2, frame("alice"), "alice"), + (2, frame("ALICE"), "ALICE"), + (3, frame("bob"), "bob"), + (4, frame("cafe"), "cafe"), + (4, frame("Café"), "Café"), + (4, frame("CAFE\u{301}"), "CAFE with a combining accent"), + (5, frame("cafes"), "cafes"), + (6, frame("Zoe"), "Zoe"), + ]; + sorts_as(&keyset, name, values).await; + } +} + +/// `=` on a `TextOrd` column reads the equality term, which folds no case: +/// `Café` and `cafe` tie in order and are still two values to `=`. The +/// equality term is `TextEq`'s for the same column. +#[tokio::test] +async fn text_order_ties_are_not_equal() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + let a = stored(&keyset, "TextOrd", frame("Café")).await; + let b = stored(&keyset, "TextOrd", frame("cafe")).await; + assert_eq!(op(&a), op(&b)); + assert_ne!(a["hm"], b["hm"]); + let eq = stored(&keyset, "TextEq", frame("Café")).await; + assert_eq!(a["hm"], eq["hm"], "TextOrd's hm is TextEq's"); +} + +/// A family's `Ord` and `OrdOpe` types are the same index under two names: +/// they store one term for one value under one column. +#[tokio::test] +async fn ord_and_ord_ope_store_one_term() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + let pairs = [ + ("IntegerOrd", "IntegerOrdOpe", frame::(-9)), + ("SmallintOrd", "SmallintOrdOpe", frame::(-9)), + ("BigintOrd", "BigintOrdOpe", frame::(1 << 50)), + ("RealOrd", "RealOrdOpe", frame::(2.5)), + ("DoubleOrd", "DoubleOrdOpe", frame::(-2.5)), + ("TextOrd", "TextOrdOpe", frame("Zoë")), + ]; + for (ord, ope, value) in pairs { + let a = stored(&keyset, ord, value.clone()).await; + let b = stored(&keyset, ope, value).await; + assert_eq!(op(&a), op(&b), "{ord} and {ope}"); + } +} + +/// Text with a code point Unicode 16 does not assign has no fold, so no +/// order term: refused stored and queried, before any term reaches a row. +#[tokio::test] +async fn text_with_no_fold_is_refused() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + let unassigned = "a\u{0378}b"; + for name in ["TextOrd", "TextOrdOpe"] { + let stored = match targets::encrypt(name, &keyset, &label(), frame(unassigned)) { + Ok(pending) => pending.await, + Err(error) => panic!("{name}: refused before the plan ran: {error}"), + }; + assert!(stored.is_err(), "{name}: stored text with no fold"); + let queried = match targets::query(name, &keyset, &label(), frame(unassigned)) { + Ok(pending) => pending.await, + Err(error) => panic!("{name}: refused before the plan ran: {error}"), + }; + assert!(queried.is_err(), "{name}: queried text with no fold"); + } + let typed: Result = keyset.encrypt_as(&unassigned.to_owned(), column()).await; + assert!(typed.is_err(), "the typed entry point refuses it too"); +} + +/// The typed entry point writes what the by-name dispatch writes: the same +/// terms, and a ciphertext that opens to the exact value. +#[tokio::test] +async fn the_typed_entry_point_stores_the_dispatchs_terms() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + + let typed: integer::IntegerOrd = keyset.encrypt_as(&-9i32, column()).await.unwrap(); + let named = stored(&keyset, "IntegerOrd", frame(-9i32)).await; + assert_eq!(serde_json::to_value(&typed.op).unwrap(), named["op"]); + let opened: i32 = cipher.decrypt_as(typed, column().into()).await.unwrap(); + assert_eq!(opened, -9); + + let typed: bigint::BigintOrdOpeQuery = keyset.encrypt_as(&7i64, column()).await.unwrap(); + let named = query(&keyset, "BigintOrdOpe", frame(7i64)).await; + assert_eq!(serde_json::to_value(&typed.op).unwrap(), named["op"]); + + let typed: real::RealOrd = keyset.encrypt_as(&-0.0f32, column()).await.unwrap(); + let named = stored(&keyset, "RealOrd", frame(0.0f32)).await; + assert_eq!( + serde_json::to_value(&typed.op).unwrap(), + named["op"], + "-0.0 ties 0.0 in the term" + ); + let opened: f32 = cipher.decrypt_as(typed, column().into()).await.unwrap(); + assert_eq!( + opened.to_bits(), + (-0.0f32).to_bits(), + "and keeps its sign in c" + ); + + let typed: double::DoubleOrdOpe = keyset.encrypt_as(&f64::NAN, column()).await.unwrap(); + let named = stored(&keyset, "DoubleOrdOpe", frame(f64::NAN)).await; + assert_eq!(serde_json::to_value(&typed.op).unwrap(), named["op"]); + + let typed: text::TextOrdQuery = keyset + .encrypt_as(&"Café".to_owned(), column()) + .await + .unwrap(); + let named = query(&keyset, "TextOrd", frame("Café")).await; + assert_eq!(serde_json::to_value(&typed).unwrap(), named); +} + +/// `fixtures/v4_ordered_queries.json`: a query operand for one value of each +/// ordered type, under `FakeDataKeySource`'s index key (keyset nil), read by +/// the Go SDK's hermetic test, which derives the same operands through +/// generated `Lt` methods and asserts the bytes are equal. Regenerate with +/// `EQL_UPDATE_FIXTURES=1`; the bytes must not change otherwise. +#[tokio::test] +async fn the_ordered_query_fixture_is_what_the_dispatch_derives() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + let query = |name: &'static str, column: &'static str, value: ValueFrame| { + let keyset = &keyset; + async move { + let label = Label::new(["rankings", column]).unwrap(); + let bytes = targets::query(name, keyset, &label, value) + .unwrap() + .await + .unwrap(); + String::from_utf8(bytes).unwrap() + } + }; + let fixture = serde_json::json!({ + "_comment": "The v4 query operands eql-bindings derives for one plaintext of each OPE-ordered type under FakeDataKeySource's index key (keyset nil), read by the Go SDK's hermetic test; regenerate with EQL_UPDATE_FIXTURES=1, do not edit.", + "table": "rankings", + "level": {"plaintext": 10, "query": query("IntegerOrd", "level", frame::(10)).await}, + "floor": {"plaintext": -3, "query": query("IntegerOrdOpe", "floor", frame::(-3)).await}, + "points": {"plaintext": 5_000_000_000_i64, "query": query("BigintOrd", "points", frame::(5_000_000_000)).await}, + "total": {"plaintext": -1, "query": query("BigintOrdOpe", "total", frame::(-1)).await}, + "speed": {"plaintext": 2.5, "query": query("RealOrd", "speed", frame::(2.5)).await}, + "pace": {"plaintext": -0.5, "query": query("RealOrdOpe", "pace", frame::(-0.5)).await}, + "score": {"plaintext": 9.75, "query": query("DoubleOrd", "score", frame::(9.75)).await}, + "mean": {"plaintext": 0.0, "query": query("DoubleOrdOpe", "mean", frame::(0.0)).await}, + "title": {"plaintext": "Mango", "query": query("TextOrd", "title", frame("Mango")).await}, + "handle": {"plaintext": "carol", "query": query("TextOrdOpe", "handle", frame("carol")).await}, + }); + let path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("fixtures/v4_ordered_queries.json"); + let rendered = serde_json::to_string_pretty(&fixture).unwrap() + "\n"; + if std::env::var_os("EQL_UPDATE_FIXTURES").is_some() { + std::fs::write(&path, &rendered).unwrap(); + } + let committed = std::fs::read_to_string(&path) + .expect("fixtures/v4_ordered_queries.json is committed; EQL_UPDATE_FIXTURES=1 writes it"); + assert_eq!( + committed, rendered, + "the committed fixture is what the dispatch derives today" + ); +} + +/// `fixtures/v4_ordered_rows.json`: `Ranking` rows the Go SDK's generated +/// code sealed over the deterministic EQL guest +/// (`TestV4OrderedRowsMatchTheCommittedFixture` in languages/golang/encrypt +/// writes it). +mod go_rows { + use super::*; + use serde::de::DeserializeOwned; + use std::collections::BTreeMap; + + #[derive(serde::Deserialize)] + pub struct Fixture { + pub seed: String, + pub table: String, + pub rows: Vec, + } + #[derive(serde::Deserialize)] + pub struct Row { + pub id: i32, + pub plaintext: Plain, + pub stored: BTreeMap, + } + #[derive(serde::Deserialize, Clone)] + pub struct Plain { + pub level: i32, + pub floor: i32, + pub points: i64, + pub total: i64, + pub speed: f32, + pub pace: f32, + pub score: f64, + pub mean: f64, + pub title: String, + pub handle: String, + } + + pub fn fixture() -> Fixture { + let path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("fixtures/v4_ordered_rows.json"); + let raw = std::fs::read_to_string(path).expect( + "fixtures/v4_ordered_rows.json is written by the Go SDK's \ + TestV4OrderedRowsMatchTheCommittedFixture", + ); + serde_json::from_str(&raw).unwrap() + } + + pub async fn cipher(seed: &str) -> stack_encrypt::StackCipher { + let seed: [u8; 32] = seed.as_bytes().try_into().expect("a 32-byte seed"); + stack_encrypt::StackCipher::builder() + .kms(stack_kms::DeterministicSource::new(seed)) + .init() + .await + .unwrap() + } + + pub fn column(name: &str) -> NonEmpty { + Identifier::for_column("rankings", name).unwrap() + } + + pub fn parsed(row: &Row, column: &str) -> T { + serde_json::from_value(row.stored[column].clone()) + .unwrap_or_else(|e| panic!("row {} {column}: {e}", row.id)) + } + + /// Open every field of `row` and check it against what Go sealed. + pub async fn open_row( + cipher: &stack_encrypt::StackCipher, + row: &Row, + ) { + let p = &row.plaintext; + macro_rules! open { + ($ty:ty, $plain:ty, $column:literal) => {{ + let opened: $plain = cipher + .decrypt_as(parsed::<$ty>(row, $column), column($column).into()) + .await + .unwrap(); + opened + }}; + } + assert_eq!(open!(integer::IntegerOrd, i32, "level"), p.level); + assert_eq!(open!(integer::IntegerOrdOpe, i32, "floor"), p.floor); + assert_eq!(open!(bigint::BigintOrd, i64, "points"), p.points); + assert_eq!(open!(bigint::BigintOrdOpe, i64, "total"), p.total); + assert_eq!( + open!(real::RealOrd, f32, "speed").to_bits(), + p.speed.to_bits() + ); + assert_eq!( + open!(real::RealOrdOpe, f32, "pace").to_bits(), + p.pace.to_bits() + ); + assert_eq!( + open!(double::DoubleOrd, f64, "score").to_bits(), + p.score.to_bits() + ); + assert_eq!( + open!(double::DoubleOrdOpe, f64, "mean").to_bits(), + p.mean.to_bits() + ); + assert_eq!(open!(text::TextOrd, String, "title"), p.title); + assert_eq!(open!(text::TextOrdOpe, String, "handle"), p.handle); + } +} + +/// What the Go SDK sealed parses as each ordered v4 type and opens in Rust. +#[tokio::test] +async fn ordered_rows_the_go_sdk_sealed_open_in_rust() { + let fixture = go_rows::fixture(); + assert_eq!(fixture.table, "rankings"); + assert!(fixture.rows.len() > 1); + let cipher = go_rows::cipher(&fixture.seed).await; + for row in &fixture.rows { + go_rows::open_row(&cipher, row).await; + } +} + +/// The ordered columns and their SQL types, for the PostgreSQL tests: +/// (column, stored domain, query domain). +const COLUMNS: &[(&str, &str, &str)] = &[ + ("level", "integer_ord", "query_integer_ord"), + ("floor", "integer_ord_ope", "query_integer_ord_ope"), + ("points", "bigint_ord", "query_bigint_ord"), + ("total", "bigint_ord_ope", "query_bigint_ord_ope"), + ("speed", "real_ord", "query_real_ord"), + ("pace", "real_ord_ope", "query_real_ord_ope"), + ("score", "double_ord", "query_double_ord"), + ("mean", "double_ord_ope", "query_double_ord_ope"), + ("title", "text_ord", "query_text_ord"), + ("handle", "text_ord_ope", "query_text_ord_ope"), +]; + +fn connect() -> postgres::Client { + postgres::Client::connect( + &std::env::var("EQL_TEST_DATABASE_URL").expect("EQL_TEST_DATABASE_URL"), + postgres::NoTls, + ) + .unwrap() +} + +fn create_table(db: &mut postgres::Client, table: &str) { + let columns: Vec = COLUMNS + .iter() + .map(|(column, domain, _)| format!("{column} public.eql_v4_{domain}")) + .collect(); + db.batch_execute(&format!( + "CREATE TEMP TABLE {table} (id int PRIMARY KEY, {});", + columns.join(", ") + )) + .unwrap(); +} + +fn insert(db: &mut postgres::Client, table: &str, id: i32, stored: &[String]) { + let placeholders: Vec = COLUMNS + .iter() + .enumerate() + .map(|(n, (_, domain, _))| format!("${}::text::jsonb::public.eql_v4_{domain}", n + 2)) + .collect(); + let mut params: Vec<&(dyn postgres::types::ToSql + Sync)> = vec![&id]; + params.extend( + stored + .iter() + .map(|s| s as &(dyn postgres::types::ToSql + Sync)), + ); + db.execute( + &format!( + "INSERT INTO {table} VALUES ($1, {})", + placeholders.join(", ") + ), + ¶ms, + ) + .unwrap(); +} + +fn ids(db: &mut postgres::Client, sql: &str, query: &str) -> Vec { + db.query(sql, &[&query]) + .unwrap_or_else(|e| panic!("{sql}: {e}")) + .iter() + .map(|r| r.get(0)) + .collect() +} + +/// Needs an EQL v4 installation. Rows written through the by-name dispatch: +/// each ordered column answers `<`, `<=`, `>`, `>=` and `=` with exactly the +/// rows the plaintexts say, and `ORDER BY eql_v4.ord_term(column)` sorts +/// them as the order keys do. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_compares_and_orders_ope_rows() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (cipher, _) = runtime.block_on(common::cipher()); + let keyset = cipher.default_keyset(); + let mut db = connect(); + create_table(&mut db, "v4_ordered"); + + // One value per column per row. Row 3 ties row 2 in every column's order + // key: equal integers, `-0.0` against `0.0`, and text that differs only + // in case and accents. + struct Plain(i32, i64, f32, f64, &'static str); + let rows = [ + (1, Plain(-5, i64::MIN, f32::NEG_INFINITY, -1.5, "alice")), + (2, Plain(0, 0, 0.0, 0.0, "Cafe")), + (3, Plain(0, 0, -0.0, -0.0, "café")), + (4, Plain(7, 1 << 40, 2.5, 1e300, "carol")), + (5, Plain(i32::MAX, i64::MAX, f32::NAN, f64::NAN, "Zoe")), + ]; + let frames = |p: &Plain| -> Vec<(&'static str, ValueFrame)> { + vec![ + ("IntegerOrd", frame(p.0)), + ("IntegerOrdOpe", frame(p.0)), + ("BigintOrd", frame(p.1)), + ("BigintOrdOpe", frame(p.1)), + ("RealOrd", frame(p.2)), + ("RealOrdOpe", frame(p.2)), + ("DoubleOrd", frame(p.3)), + ("DoubleOrdOpe", frame(p.3)), + ("TextOrd", frame(p.4)), + ("TextOrdOpe", frame(p.4)), + ] + }; + let under = |column: &str| Label::new(["rankings", column]).unwrap(); + for (id, plain) in &rows { + let stored: Vec = runtime.block_on(async { + let mut stored = Vec::new(); + for ((column, _, _), (name, value)) in COLUMNS.iter().zip(frames(plain)) { + let bytes = targets::encrypt(name, &keyset, &under(column), value) + .unwrap() + .await + .unwrap(); + stored.push(String::from_utf8(bytes).unwrap()); + } + stored + }); + insert(&mut db, "v4_ordered", *id, &stored); + } + + // The query operand for the middle row's values (row 2), per column. + let probes: Vec = runtime.block_on(async { + let mut probes = Vec::new(); + for ((column, _, _), (name, value)) in COLUMNS.iter().zip(frames(&rows[1].1)) { + let bytes = targets::query(name, &keyset, &under(column), value) + .unwrap() + .await + .unwrap(); + probes.push(String::from_utf8(bytes).unwrap()); + } + probes + }); + + for ((column, _, query_domain), probe) in COLUMNS.iter().zip(&probes) { + let cmp = |op: &str| { + format!( + "SELECT id FROM v4_ordered WHERE {column} {op} $1::text::jsonb::eql_v4.{query_domain} ORDER BY id" + ) + }; + let text = column == &"title" || column == &"handle"; + assert_eq!(ids(&mut db, &cmp("<"), probe), [1], "{column} <"); + assert_eq!(ids(&mut db, &cmp("<="), probe), [1, 2, 3], "{column} <="); + assert_eq!(ids(&mut db, &cmp(">"), probe), [4, 5], "{column} >"); + assert_eq!(ids(&mut db, &cmp(">="), probe), [2, 3, 4, 5], "{column} >="); + // `=` reads the order term on a number column, where -0.0 ties 0.0 + // as in float8, and the case-sensitive equality term on text. + let equal: &[i32] = if text { &[2] } else { &[2, 3] }; + assert_eq!(ids(&mut db, &cmp("="), probe), equal, "{column} ="); + + // The query operand may stand on the left too. + let flipped = format!( + "SELECT id FROM v4_ordered WHERE $1::text::jsonb::eql_v4.{query_domain} > {column} ORDER BY id" + ); + assert_eq!( + ids(&mut db, &flipped, probe), + [1], + "{column}: query > column" + ); + + // Ascending by the order term: row 1 first, the tied rows 2 and 3 + // together (broken by id), then 4 and 5. + let ordered: Vec = db + .query( + &format!("SELECT id FROM v4_ordered ORDER BY eql_v4.ord_term({column}), id"), + &[], + ) + .unwrap() + .iter() + .map(|r| r.get(0)) + .collect(); + assert_eq!(ordered, [1, 2, 3, 4, 5], "{column}: ORDER BY ord_term"); + } + + // MIN and MAX over the order term name the extreme rows' values. + let back: String = db + .query_one("SELECT eql_v4.max(level)::text FROM v4_ordered", &[]) + .map(|r| r.get(0)) + .unwrap_or_else(|e| panic!("max(level): {e}")); + let opened: i32 = runtime + .block_on(async { + cipher + .decrypt_as( + serde_json::from_str::(&back).unwrap(), + column().into(), + ) + .await + }) + .unwrap(); + assert_eq!(opened, i32::MAX); +} + +/// The Go-sealed rows in PostgreSQL: each value passes its domain check, +/// `<`, `<=`, `>` and `>=` with the fixture's query operands (which the Go +/// SDK's test derives byte for byte) find the rows whose plaintext is on +/// that side of the query's, and `ORDER BY eql_v4.ord_term(column)` sorts +/// Go's rows as their plaintexts do. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_compares_rows_the_go_sdk_sealed() { + let fixture = go_rows::fixture(); + let mut db = connect(); + create_table(&mut db, "go_rankings"); + for row in &fixture.rows { + let stored: Vec = COLUMNS + .iter() + .map(|(column, _, _)| row.stored[*column].to_string()) + .collect(); + insert(&mut db, "go_rankings", row.id, &stored); + } + let queries: serde_json::Value = serde_json::from_str( + &std::fs::read_to_string( + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("fixtures/v4_ordered_queries.json"), + ) + .unwrap(), + ) + .unwrap(); + // A column's order key, as a value that sorts as the order term does. + #[derive(PartialEq, PartialOrd, Debug)] + enum Key { + Int(i64), + Float(f64), + Text(String), + } + let fold = |s: &str| { + // The fixture's text is ASCII, so lower-casing is the fold here. + assert!(s.is_ascii(), "{s}: the expectation folds ASCII only"); + Key::Text(s.to_ascii_lowercase()) + }; + let key = |column: &str, p: &go_rows::Plain| -> Key { + match column { + "level" => Key::Int(p.level.into()), + "floor" => Key::Int(p.floor.into()), + "points" => Key::Int(p.points), + "total" => Key::Int(p.total), + "speed" => Key::Float(p.speed.into()), + "pace" => Key::Float(p.pace.into()), + "score" => Key::Float(p.score), + "mean" => Key::Float(p.mean), + "title" => fold(&p.title), + "handle" => fold(&p.handle), + other => panic!("no order key for {other}"), + } + }; + let query_key = |column: &str, plain: &serde_json::Value| -> Key { + match column { + "level" | "floor" | "points" | "total" => Key::Int(plain.as_i64().unwrap()), + "speed" | "pace" | "score" | "mean" => Key::Float(plain.as_f64().unwrap()), + "title" | "handle" => fold(plain.as_str().unwrap()), + other => panic!("no order key for {other}"), + } + }; + for (column, _, query_domain) in COLUMNS { + let probe = query_key(column, &queries[*column]["plaintext"]); + // Sorted, as the SQL below sorts its ids: the assertion is about + // which rows, not the order the fixture lists them in. + let side = |keep: &dyn Fn(&Key) -> bool| -> Vec { + let mut ids: Vec = fixture + .rows + .iter() + .filter(|r| keep(&key(column, &r.plaintext))) + .map(|r| r.id) + .collect(); + ids.sort_unstable(); + ids + }; + let below = side(&|k| *k < probe); + let above = side(&|k| *k > probe); + assert!( + !below.is_empty() && !above.is_empty(), + "{column}: the fixture has rows on both sides of the query" + ); + let not_below = side(&|k| *k >= probe); + let not_above = side(&|k| *k <= probe); + + let query = queries[*column]["query"].as_str().unwrap(); + let cmp = |op: &str| { + format!( + "SELECT id FROM go_rankings WHERE {column} {op} $1::text::jsonb::eql_v4.{query_domain} ORDER BY id" + ) + }; + assert_eq!( + ids(&mut db, &cmp("<"), query), + below, + "< over Go-sealed {column}" + ); + assert_eq!( + ids(&mut db, &cmp("<="), query), + not_above, + "<= over Go-sealed {column}" + ); + assert_eq!( + ids(&mut db, &cmp(">"), query), + above, + "> over Go-sealed {column}" + ); + assert_eq!( + ids(&mut db, &cmp(">="), query), + not_below, + ">= over Go-sealed {column}" + ); + + // Go's terms sort among themselves as the plaintexts do, ties + // broken by id. + let mut by_plaintext: Vec<(Key, i32)> = fixture + .rows + .iter() + .map(|r| (key(column, &r.plaintext), r.id)) + .collect(); + by_plaintext.sort_by(|a, b| a.partial_cmp(b).expect("the fixture's keys are finite")); + let expected: Vec = by_plaintext.into_iter().map(|(_, id)| id).collect(); + let ordered: Vec = db + .query( + &format!("SELECT id FROM go_rankings ORDER BY eql_v4.ord_term({column}), id"), + &[], + ) + .unwrap() + .iter() + .map(|r| r.get(0)) + .collect(); + assert_eq!( + ordered, expected, + "ORDER BY ord_term over Go-sealed {column}" + ); + } +} diff --git a/packages/eql/tests/encryption/tests/v4_types.rs b/packages/eql/tests/encryption/tests/v4_types.rs index 0e4be75a5..ecd8c4cb6 100644 --- a/packages/eql/tests/encryption/tests/v4_types.rs +++ b/packages/eql/tests/encryption/tests/v4_types.rs @@ -450,6 +450,13 @@ async fn every_producible_type_dispatches_by_name() { "{name}: the query probes the stored term" ); } + if target.indexes.contains(&"ope") { + assert!(json["op"].is_string(), "{name} stores an op"); + assert_eq!( + query["op"], json["op"], + "{name}: the query compares the stored order term" + ); + } } } }