diff --git a/.changeset/eql-v4-date-timestamp-numeric.md b/.changeset/eql-v4-date-timestamp-numeric.md new file mode 100644 index 000000000..96ffccffb --- /dev/null +++ b/.changeset/eql-v4-date-timestamp-numeric.md @@ -0,0 +1,5 @@ +--- +'@cipherstash/eql': minor +--- + +**Stack Encrypt produces the EQL v4 date, timestamp and numeric types `Date`, `DateEq`, `Timestamp`, `TimestampEq`, `Numeric` and `NumericEq`** (`eql-bindings`, `stack-encrypt` feature). Their Rust plaintexts are the crate's own `v4::plaintext::Date`, `Timestamp` and `Decimal`, which hold the integers the engine stores (days since 1970-01-01; Unix seconds and nanoseconds; a mantissa and a scale), are range checked when built and are wiped when dropped. The `chrono` and `rust_decimal` features convert to and from those crates' types. `=` on `NumericEq` normalises the scale (`1.5 = 1.50`) and on `TimestampEq` compares to the microsecond, truncating (never rounding) the nanoseconds below it; each value decrypts as it was stored. The ordered types of these families stay unproducible until the engine derives their order terms. The Go SDK's `encrypt/eql` table marks the six producible, from `encrypt.Date`, `time.Time` and `encrypt.Decimal`. diff --git a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md index 01926da4f..8230b207b 100644 --- a/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md +++ b/docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md @@ -1,7 +1,7 @@ --- status: accepted date: 2026-10-08 -revised: 2026-10-12, amended 2026-10-18 +revised: 2026-10-12, amended 2026-10-18 and 2026-10-19 relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 --- @@ -30,6 +30,10 @@ relates-to: ADR-0001; stack-encrypt ADR-0003, ADR-0007 > design (see "Exhaustive handling of kinds"). > > The bullets below that this changes say so where they stand. +> +> **Amended 2026-10-19.** A timestamp's microsecond resolution truncates +> rather than rounding half to even, as vitaminc does for both the equality +> and the order term. The reason, and the rule, are under "Lossy transforms". Every value Stack Encrypt handles is encoded three times: as a **ciphertext** (reversible and self-describing), as an **equality term** (a keyed hash that @@ -194,13 +198,22 @@ domain: timestamp resolution, text folding and prefix truncation. Each transform and its version is named in the domain label, for example `…/timestamp/day/v1`. The ones defined now: -- **Timestamp resolution**: exact, microsecond or day. Microsecond rounds to - the nearest microsecond, ties to even, on the signed microsecond count +- **Timestamp resolution**: exact, microsecond or day. ~~Microsecond rounds + to the nearest microsecond, ties to even, on the signed microsecond count since the epoch. That is what Postgres does when it parses a timestamp with more digits than it keeps (`.1234565` becomes `.123456` and `.1234575` - becomes `.123458`, before and after 1970; checked on Postgres 17.5). A - value that already has microsecond precision is unchanged. Day takes the - UTC calendar day. + becomes `.123458`, before and after 1970; checked on Postgres 17.5).~~ + *Amended 2026-10-19:* microsecond truncates: the nanoseconds below the + microsecond are dropped, never rounded, which is floor on the signed count + since the epoch (`.1234567` becomes `.123456`, and before 1970 an instant + moves earlier). vitaminc applies this rule to the equality term and to the + order term alike (vitaminc-ore's timestamp domain is + `vitaminc/ore/value/timestamp-micros/v1`, and both terms normalise through + `vitaminc_normalize::timestamp`), so `=` and the order agree on which + instants are one. Matching PostgreSQL's parse-time rounding is not a goal: + timestamps are stored in many places besides PostgreSQL, and a term must + mean the same thing whichever store holds it. A value that already has + microsecond precision is unchanged. Day takes the UTC calendar day. - **Text fold, v1**: NFD, then full Unicode case folding (`CaseFolding.txt` statuses C and F, so `ß` becomes `ss`), then NFD again, then removal of every accent, then NFC. An accent is a mark (General_Category `Mn`, `Mc` diff --git a/languages/golang/CHANGELOG.md b/languages/golang/CHANGELOG.md index 6a5a03e8e..738869d02 100644 --- a/languages/golang/CHANGELOG.md +++ b/languages/golang/CHANGELOG.md @@ -52,6 +52,15 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`. ### Added +- **The date, timestamp and numeric EQL v4 types are producible** (#1130): + `Date` and `DateEq` from an `encrypt.Date`, `Timestamp` and `TimestampEq` + from a `time.Time`, and `Numeric` and `NumericEq` from an + `encrypt.Decimal`, with `encrypt_into=`. The `Eq` types answer `Eq`: + `NumericEq` normalises the scale (`1.5 = 1.50`) and `TimestampEq` compares + to the microsecond (truncated, never rounded), while each value opens as it was sealed. A + `time.Time` seals its instant and opens in UTC. Their `Ord`, `OrdOpe` and + `OrdOre` types stay unproducible until the engine derives order terms + for these kinds (#1236). - **Every integer width, 128-bit integers, dates, timestamps and decimals as sealed fields** (#1144). Each integer type seals at its own width (`int8` as an int8, `uint16` as a uint16), so `record.Kind` and diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 3462fa1c2..89e095cae 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -165,8 +165,12 @@ The engine produces these EQL types today, each from one Go field type: | `Boolean` | `bool` | | `Real`, `RealEq`, `RealOrd`, `RealOrdOpe` | `float32` | | `Double`, `DoubleEq`, `DoubleOrd`, `DoubleOrdOpe` | `float64` | +| `Date`, `DateEq` | `encrypt.Date` | +| `Timestamp`, `TimestampEq` | `time.Time` | +| `Numeric`, `NumericEq` | `encrypt.Decimal` | -The `Eq` types answer `Eq`; the `Ord` and `OrdOpe` types answer `Lt`, `Lte`, `Gt` and `Gte`, and on text `Eq` too; `Text`, `Smallint`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double` are stored and read only, with no query method. +The `Eq` types answer `Eq`; the `Ord` and `OrdOpe` types answer `Lt`, `Lte`, `Gt` and `Gte`, and on text `Eq` too; `Text`, `Smallint`, `Integer`, `Bigint`, `Boolean`, `Real`, `Double`, `Date`, `Timestamp` and `Numeric` are stored and read only, with no query method. +The ordered date, timestamp and numeric types are refused: the engine derives no order term for these kinds yet. An ordered type has no `Between` method yet, because a `BETWEEN` operand needs EQL v4's range query domains. Any built-in Go integer field takes an integer type (ADR-0002) and seals at the type's width: an `int16` field always fits an `Integer`, and an `int64` field is range checked value by value, `Encrypt` refusing a value that does not fit with `encrypt.ErrEncoding`. `encrypt.Int128` and `encrypt.Uint128` are not converted. A float's width must match: a `float64` field is not a `Real`. diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index e681bfd04..23c458b5f 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -260,7 +260,7 @@ index: with 256 values, an order term would rank every row among 256, which a few known rows resolve to the values, as a `bool`'s would say which rows are true. Equality follows vitaminc's normalisation: an integer equals itself at every width, decimals compare at any scale -(`1.50 = 1.5`), and a timestamp compares to the microsecond. An order index on +(`1.50 = 1.5`), and a timestamp compares to the microsecond, truncated, never rounded. An order index on a decimal, date or timestamp is refused for now: the engine derives no order term for those kinds yet. @@ -493,6 +493,12 @@ The EQL types the engine produces, and the Go field types each takes: | `Double` | `float64` | none | | `DoubleEq` | `float64` | `=`, `<>` | | `DoubleOrd`, `DoubleOrdOpe` | `float64` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | +| `Date` | `encrypt.Date` | none | +| `DateEq` | `encrypt.Date` | `=`, `<>` | +| `Timestamp` | `time.Time` | none | +| `TimestampEq` | `time.Time` | `=`, `<>` (to the microsecond, truncated) | +| `Numeric` | `encrypt.Decimal` | none | +| `NumericEq` | `encrypt.Decimal` | `=`, `<>` (`1.5 = 1.50`) | Any built-in Go integer field takes an integer EQL type (ADR-0002): the field is sealed at the type's width, and opens back into the field's type. A type @@ -508,7 +514,17 @@ A `TextMatch` value must be at least three characters long: shorter text has no 3-grams to index and is refused. The number equality types compare as PostgreSQL compares the plain types: an integer equals itself whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN`; the stored value still -decrypts to exactly what was written. +decrypts to exactly what was written. The same holds for the date, +timestamp and numeric types: `NumericEq` normalises the scale, so `1.5` and +`1.50` are equal, and `TimestampEq` compares to the microsecond, truncating +(never rounding) the nanoseconds below it, so two instants in one +microsecond are equal; each still opens as it was written, a +decimal with its scale and a `time.Time` with its nanoseconds, in UTC. A +`time.Time` is sealed as its instant, whatever its location. A zero +`encrypt.Date`, a date or instant outside years -262143 to 262142 and a +decimal with more than 28 digits after the point are refused with +`ErrEncoding`. The ordered date, timestamp and numeric types are not +produced yet: the engine derives no order term for these kinds. The `Ord` and `OrdOpe` types store an OPE term, which EQL compares as bytes, and order by ADR-0002's order key: integers by value, floats as PostgreSQL's @@ -526,7 +542,7 @@ answers `Eq` through an equality term only. A text order type also stores neither `<` nor `>` each other and not `=` either. Text holding a code point Unicode 16 does not assign has no order term and is refused. There is no `Between` method yet: a `BETWEEN` needs EQL v4's range query domains. The rest of -the catalog (ORE ordering, dates, decimals, JSON) is listed in +the catalog (ORE ordering, ordered dates, timestamps and decimals, JSON) is listed in `eql.Types` with the reason it is not produced yet, and `stashgen` refuses it with that reason. The guest builds the value: the generated code imports `encrypt/eql`, which embeds the build of the engine that holds the diff --git a/languages/golang/encrypt/doc.go b/languages/golang/encrypt/doc.go index 05c485dbc..cb7e45937 100644 --- a/languages/golang/encrypt/doc.go +++ b/languages/golang/encrypt/doc.go @@ -23,15 +23,19 @@ // encrypt_into names the EQL type the field is stored as. The Go field type // must match it: TextEq, TextMatch and Text take a string, Smallint and // SmallintEq an int16, Integer and IntegerEq an int32, Bigint and BigintEq -// an int64, Boolean a bool, Real and RealEq a float32, and Double and -// DoubleEq a float64. An integer EQL type +// an int64, Boolean a bool, Real and RealEq a float32, Double and DoubleEq a +// float64, Date and DateEq a [Date], Timestamp and TimestampEq a time.Time, +// and Numeric and NumericEq a [Decimal]. An integer EQL type // also takes any other built-in Go integer type, sealed at the type's width: // one every value of which fits it (an int16 into Integer) always seals, and // a wider one (an int64 into Integer) is range checked, a value that does not // fit refused with [ErrEncoding]. The *Eq types answer Eq; the *Ord and // *OrdOpe types (IntegerOrd, SmallintOrd, BigintOrd, RealOrd, DoubleOrd, -// TextOrd and their OrdOpe twins) answer Lt, Lte, Gt and Gte, and Eq too on text; Smallint, -// Integer, Bigint, Boolean, Real, Double and Text are stored and read only. +// TextOrd and their OrdOpe twins) answer Lt, Lte, Gt and Gte, and Eq too on +// text; Smallint, Integer, Bigint, Boolean, Real, Double, Text, Date, +// Timestamp and Numeric are stored and read only. NumericEq compares +// decimals with their scale normalised (1.5 = 1.50), and TimestampEq +// compares instants to the microsecond, truncated, never rounded. // // 3. Run the generator. It writes user_stash.go beside the struct: // diff --git a/languages/golang/encrypt/eql/eql_gen.go b/languages/golang/encrypt/eql/eql_gen.go index d1da7b798..f3e19ac2d 100644 --- a/languages/golang/encrypt/eql/eql_gen.go +++ b/languages/golang/encrypt/eql/eql_gen.go @@ -24,21 +24,21 @@ var Types = []Type{ {Name: "BigintOrdOre", GoName: "BigintOrdOre", Family: "bigint", Suffix: "OrdOre", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord_ore", Indexes: []string{"ore"}, Query: "BigintOrdOreQuery", QuerySQLDomain: "eql_v4.query_bigint_ord_ore", Producible: false, Reason: "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)"}, {Name: "BigintOrd", GoName: "BigintOrd", Family: "bigint", Suffix: "Ord", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord", Indexes: []string{"ope"}, Query: "BigintOrdQuery", QuerySQLDomain: "eql_v4.query_bigint_ord", Producible: true, Reason: ""}, {Name: "BigintOrdOpe", GoName: "BigintOrdOpe", Family: "bigint", Suffix: "OrdOpe", Plaintext: "int64", SQLDomain: "public.eql_v4_bigint_ord_ope", Indexes: []string{"ope"}, Query: "BigintOrdOpeQuery", QuerySQLDomain: "eql_v4.query_bigint_ord_ope", Producible: true, Reason: ""}, - {Name: "Date", GoName: "Date", Family: "date", Suffix: "", Plaintext: "", SQLDomain: "public.eql_v4_date", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "DateEq", GoName: "DateEq", Family: "date", Suffix: "Eq", Plaintext: "", SQLDomain: "public.eql_v4_date_eq", Indexes: []string{"eq"}, Query: "DateEqQuery", QuerySQLDomain: "eql_v4.query_date_eq", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "DateOrdOre", GoName: "DateOrdOre", Family: "date", Suffix: "OrdOre", Plaintext: "", SQLDomain: "public.eql_v4_date_ord_ore", Indexes: []string{"ore"}, Query: "DateOrdOreQuery", QuerySQLDomain: "eql_v4.query_date_ord_ore", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "DateOrd", GoName: "DateOrd", Family: "date", Suffix: "Ord", Plaintext: "", SQLDomain: "public.eql_v4_date_ord", Indexes: []string{"ope"}, Query: "DateOrdQuery", QuerySQLDomain: "eql_v4.query_date_ord", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "DateOrdOpe", GoName: "DateOrdOpe", Family: "date", Suffix: "OrdOpe", Plaintext: "", SQLDomain: "public.eql_v4_date_ord_ope", Indexes: []string{"ope"}, Query: "DateOrdOpeQuery", QuerySQLDomain: "eql_v4.query_date_ord_ope", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "Timestamp", GoName: "Timestamp", Family: "timestamp", Suffix: "", Plaintext: "", SQLDomain: "public.eql_v4_timestamp", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "TimestampEq", GoName: "TimestampEq", Family: "timestamp", Suffix: "Eq", Plaintext: "", SQLDomain: "public.eql_v4_timestamp_eq", Indexes: []string{"eq"}, Query: "TimestampEqQuery", QuerySQLDomain: "eql_v4.query_timestamp_eq", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "TimestampOrdOre", GoName: "TimestampOrdOre", Family: "timestamp", Suffix: "OrdOre", Plaintext: "", SQLDomain: "public.eql_v4_timestamp_ord_ore", Indexes: []string{"ore"}, Query: "TimestampOrdOreQuery", QuerySQLDomain: "eql_v4.query_timestamp_ord_ore", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "TimestampOrd", GoName: "TimestampOrd", Family: "timestamp", Suffix: "Ord", Plaintext: "", SQLDomain: "public.eql_v4_timestamp_ord", Indexes: []string{"ope"}, Query: "TimestampOrdQuery", QuerySQLDomain: "eql_v4.query_timestamp_ord", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "TimestampOrdOpe", GoName: "TimestampOrdOpe", Family: "timestamp", Suffix: "OrdOpe", Plaintext: "", SQLDomain: "public.eql_v4_timestamp_ord_ope", Indexes: []string{"ope"}, Query: "TimestampOrdOpeQuery", QuerySQLDomain: "eql_v4.query_timestamp_ord_ope", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "Numeric", GoName: "Numeric", Family: "numeric", Suffix: "", Plaintext: "", SQLDomain: "public.eql_v4_numeric", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "NumericEq", GoName: "NumericEq", Family: "numeric", Suffix: "Eq", Plaintext: "", SQLDomain: "public.eql_v4_numeric_eq", Indexes: []string{"eq"}, Query: "NumericEqQuery", QuerySQLDomain: "eql_v4.query_numeric_eq", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "NumericOrdOre", GoName: "NumericOrdOre", Family: "numeric", Suffix: "OrdOre", Plaintext: "", SQLDomain: "public.eql_v4_numeric_ord_ore", Indexes: []string{"ore"}, Query: "NumericOrdOreQuery", QuerySQLDomain: "eql_v4.query_numeric_ord_ore", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "NumericOrd", GoName: "NumericOrd", Family: "numeric", Suffix: "Ord", Plaintext: "", SQLDomain: "public.eql_v4_numeric_ord", Indexes: []string{"ope"}, Query: "NumericOrdQuery", QuerySQLDomain: "eql_v4.query_numeric_ord", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, - {Name: "NumericOrdOpe", GoName: "NumericOrdOpe", Family: "numeric", Suffix: "OrdOpe", Plaintext: "", SQLDomain: "public.eql_v4_numeric_ord_ope", Indexes: []string{"ope"}, Query: "NumericOrdOpeQuery", QuerySQLDomain: "eql_v4.query_numeric_ord_ope", Producible: false, Reason: "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)"}, + {Name: "Date", GoName: "Date", Family: "date", Suffix: "", Plaintext: "date", SQLDomain: "public.eql_v4_date", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, + {Name: "DateEq", GoName: "DateEq", Family: "date", Suffix: "Eq", Plaintext: "date", SQLDomain: "public.eql_v4_date_eq", Indexes: []string{"eq"}, Query: "DateEqQuery", QuerySQLDomain: "eql_v4.query_date_eq", Producible: true, Reason: ""}, + {Name: "DateOrdOre", GoName: "DateOrdOre", Family: "date", Suffix: "OrdOre", Plaintext: "date", SQLDomain: "public.eql_v4_date_ord_ore", Indexes: []string{"ore"}, Query: "DateOrdOreQuery", QuerySQLDomain: "eql_v4.query_date_ord_ore", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "DateOrd", GoName: "DateOrd", Family: "date", Suffix: "Ord", Plaintext: "date", SQLDomain: "public.eql_v4_date_ord", Indexes: []string{"ope"}, Query: "DateOrdQuery", QuerySQLDomain: "eql_v4.query_date_ord", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "DateOrdOpe", GoName: "DateOrdOpe", Family: "date", Suffix: "OrdOpe", Plaintext: "date", SQLDomain: "public.eql_v4_date_ord_ope", Indexes: []string{"ope"}, Query: "DateOrdOpeQuery", QuerySQLDomain: "eql_v4.query_date_ord_ope", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "Timestamp", GoName: "Timestamp", Family: "timestamp", Suffix: "", Plaintext: "timestamp", SQLDomain: "public.eql_v4_timestamp", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, + {Name: "TimestampEq", GoName: "TimestampEq", Family: "timestamp", Suffix: "Eq", Plaintext: "timestamp", SQLDomain: "public.eql_v4_timestamp_eq", Indexes: []string{"eq"}, Query: "TimestampEqQuery", QuerySQLDomain: "eql_v4.query_timestamp_eq", Producible: true, Reason: ""}, + {Name: "TimestampOrdOre", GoName: "TimestampOrdOre", Family: "timestamp", Suffix: "OrdOre", Plaintext: "timestamp", SQLDomain: "public.eql_v4_timestamp_ord_ore", Indexes: []string{"ore"}, Query: "TimestampOrdOreQuery", QuerySQLDomain: "eql_v4.query_timestamp_ord_ore", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "TimestampOrd", GoName: "TimestampOrd", Family: "timestamp", Suffix: "Ord", Plaintext: "timestamp", SQLDomain: "public.eql_v4_timestamp_ord", Indexes: []string{"ope"}, Query: "TimestampOrdQuery", QuerySQLDomain: "eql_v4.query_timestamp_ord", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "TimestampOrdOpe", GoName: "TimestampOrdOpe", Family: "timestamp", Suffix: "OrdOpe", Plaintext: "timestamp", SQLDomain: "public.eql_v4_timestamp_ord_ope", Indexes: []string{"ope"}, Query: "TimestampOrdOpeQuery", QuerySQLDomain: "eql_v4.query_timestamp_ord_ope", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "Numeric", GoName: "Numeric", Family: "numeric", Suffix: "", Plaintext: "decimal", SQLDomain: "public.eql_v4_numeric", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, + {Name: "NumericEq", GoName: "NumericEq", Family: "numeric", Suffix: "Eq", Plaintext: "decimal", SQLDomain: "public.eql_v4_numeric_eq", Indexes: []string{"eq"}, Query: "NumericEqQuery", QuerySQLDomain: "eql_v4.query_numeric_eq", Producible: true, Reason: ""}, + {Name: "NumericOrdOre", GoName: "NumericOrdOre", Family: "numeric", Suffix: "OrdOre", Plaintext: "decimal", SQLDomain: "public.eql_v4_numeric_ord_ore", Indexes: []string{"ore"}, Query: "NumericOrdOreQuery", QuerySQLDomain: "eql_v4.query_numeric_ord_ore", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "NumericOrd", GoName: "NumericOrd", Family: "numeric", Suffix: "Ord", Plaintext: "decimal", SQLDomain: "public.eql_v4_numeric_ord", Indexes: []string{"ope"}, Query: "NumericOrdQuery", QuerySQLDomain: "eql_v4.query_numeric_ord", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, + {Name: "NumericOrdOpe", GoName: "NumericOrdOpe", Family: "numeric", Suffix: "OrdOpe", Plaintext: "decimal", SQLDomain: "public.eql_v4_numeric_ord_ope", Indexes: []string{"ope"}, Query: "NumericOrdOpeQuery", QuerySQLDomain: "eql_v4.query_numeric_ord_ope", Producible: false, Reason: "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)"}, {Name: "Text", GoName: "Text", Family: "text", Suffix: "", Plaintext: "string", SQLDomain: "public.eql_v4_text", Indexes: nil, Query: "", QuerySQLDomain: "", Producible: true, Reason: ""}, {Name: "TextEq", GoName: "TextEq", Family: "text", Suffix: "Eq", Plaintext: "string", SQLDomain: "public.eql_v4_text_eq", Indexes: []string{"eq"}, Query: "TextEqQuery", QuerySQLDomain: "eql_v4.query_text_eq", Producible: true, Reason: ""}, {Name: "TextMatch", GoName: "TextMatch", Family: "text", Suffix: "Match", Plaintext: "string", SQLDomain: "public.eql_v4_text_match", Indexes: []string{"match"}, Query: "TextMatchQuery", QuerySQLDomain: "eql_v4.query_text_match", Producible: true, Reason: ""}, @@ -699,7 +699,7 @@ func (v *BigintOrdOpeQuery) UnmarshalJSON(b []byte) error { } // Date is the EQL type stored in public.eql_v4_date: the date family with no index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine produces it from a date. type Date []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -723,7 +723,7 @@ func (v *Date) UnmarshalJSON(b []byte) error { } // DateEq is the EQL type stored in public.eql_v4_date_eq: the date family with the eq index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine produces it from a date. type DateEq []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -770,7 +770,7 @@ func (v *DateEqQuery) UnmarshalJSON(b []byte) error { } // DateOrdOre is the EQL type stored in public.eql_v4_date_ord_ore: the date family with the ore index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type DateOrdOre []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -817,7 +817,7 @@ func (v *DateOrdOreQuery) UnmarshalJSON(b []byte) error { } // DateOrd is the EQL type stored in public.eql_v4_date_ord: the date family with the ope index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type DateOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -864,7 +864,7 @@ func (v *DateOrdQuery) UnmarshalJSON(b []byte) error { } // DateOrdOpe is the EQL type stored in public.eql_v4_date_ord_ope: the date family with the ope index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type DateOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -911,7 +911,7 @@ func (v *DateOrdOpeQuery) UnmarshalJSON(b []byte) error { } // Timestamp is the EQL type stored in public.eql_v4_timestamp: the timestamp family with no index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine produces it from a timestamp. type Timestamp []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -935,7 +935,7 @@ func (v *Timestamp) UnmarshalJSON(b []byte) error { } // TimestampEq is the EQL type stored in public.eql_v4_timestamp_eq: the timestamp family with the eq index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine produces it from a timestamp. type TimestampEq []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -982,7 +982,7 @@ func (v *TimestampEqQuery) UnmarshalJSON(b []byte) error { } // TimestampOrdOre is the EQL type stored in public.eql_v4_timestamp_ord_ore: the timestamp family with the ore index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type TimestampOrdOre []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1029,7 +1029,7 @@ func (v *TimestampOrdOreQuery) UnmarshalJSON(b []byte) error { } // TimestampOrd is the EQL type stored in public.eql_v4_timestamp_ord: the timestamp family with the ope index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type TimestampOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1076,7 +1076,7 @@ func (v *TimestampOrdQuery) UnmarshalJSON(b []byte) error { } // TimestampOrdOpe is the EQL type stored in public.eql_v4_timestamp_ord_ope: the timestamp family with the ope index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type TimestampOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1123,7 +1123,7 @@ func (v *TimestampOrdOpeQuery) UnmarshalJSON(b []byte) error { } // Numeric is the EQL type stored in public.eql_v4_numeric: the numeric family with no index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine produces it from a decimal. type Numeric []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1147,7 +1147,7 @@ func (v *Numeric) UnmarshalJSON(b []byte) error { } // NumericEq is the EQL type stored in public.eql_v4_numeric_eq: the numeric family with the eq index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine produces it from a decimal. type NumericEq []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1194,7 +1194,7 @@ func (v *NumericEqQuery) UnmarshalJSON(b []byte) error { } // NumericOrdOre is the EQL type stored in public.eql_v4_numeric_ord_ore: the numeric family with the ore index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type NumericOrdOre []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1241,7 +1241,7 @@ func (v *NumericOrdOreQuery) UnmarshalJSON(b []byte) error { } // NumericOrd is the EQL type stored in public.eql_v4_numeric_ord: the numeric family with the ope index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type NumericOrd []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. @@ -1288,7 +1288,7 @@ func (v *NumericOrdQuery) UnmarshalJSON(b []byte) error { } // NumericOrdOpe is the EQL type stored in public.eql_v4_numeric_ord_ope: the numeric family with the ope index. -// The engine cannot produce it yet: eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130). +// The engine cannot produce it yet: the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236). type NumericOrdOpe []byte // Value is the EQL value as the driver takes it: the JSON text, or NULL when empty. diff --git a/languages/golang/encrypt/eql/eql_test.go b/languages/golang/encrypt/eql/eql_test.go index 78596a96d..3758e261f 100644 --- a/languages/golang/encrypt/eql/eql_test.go +++ b/languages/golang/encrypt/eql/eql_test.go @@ -23,6 +23,7 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) { "Integer", "IntegerEq", "IntegerOrd", "IntegerOrdOpe", "Smallint", "SmallintEq", "SmallintOrd", "SmallintOrdOpe", "Bigint", "BigintEq", "BigintOrd", "BigintOrdOpe", + "Date", "DateEq", "Timestamp", "TimestampEq", "Numeric", "NumericEq", "Text", "TextEq", "TextMatch", "TextOrd", "TextOrdOpe", "Boolean", "Real", "RealEq", "RealOrd", "RealOrdOpe", @@ -36,7 +37,7 @@ func TestTypesTableNamesTheProducibleTypes(t *testing.T) { t.Fatalf("producible = %v, want %v", producible, want) } } - for name, plaintext := range map[string]string{"Integer": "int32", "IntegerEq": "int32", "IntegerOrd": "int32", "Smallint": "int16", "SmallintEq": "int16", "SmallintOrdOpe": "int16", "Bigint": "int64", "BigintEq": "int64", "BigintOrdOpe": "int64", "Boolean": "bool", "Real": "float32", "RealEq": "float32", "RealOrd": "float32", "Double": "float64", "DoubleEq": "float64", "DoubleOrdOpe": "float64", "TextOrd": "string"} { + for name, plaintext := range map[string]string{"Integer": "int32", "IntegerEq": "int32", "IntegerOrd": "int32", "Smallint": "int16", "SmallintEq": "int16", "SmallintOrdOpe": "int16", "Bigint": "int64", "BigintEq": "int64", "BigintOrdOpe": "int64", "Boolean": "bool", "Real": "float32", "RealEq": "float32", "RealOrd": "float32", "Double": "float64", "DoubleEq": "float64", "DoubleOrdOpe": "float64", "TextOrd": "string", "Date": "date", "DateEq": "date", "Timestamp": "timestamp", "TimestampEq": "timestamp", "Numeric": "decimal", "NumericEq": "decimal"} { if typ, ok := eql.Lookup(name); !ok || typ.Plaintext != plaintext { t.Errorf("%s: Plaintext = %q, want %q", name, typ.Plaintext, plaintext) } diff --git a/languages/golang/encrypt/eql_v4_ledger_test.go b/languages/golang/encrypt/eql_v4_ledger_test.go new file mode 100644 index 000000000..8e63c31a2 --- /dev/null +++ b/languages/golang/encrypt/eql_v4_ledger_test.go @@ -0,0 +1,358 @@ +package encrypt_test + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "os" + "testing" + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/internal/testusers" +) + +func mustDecimal(t *testing.T, s string) encrypt.Decimal { + t.Helper() + d, err := encrypt.ParseDecimal(s) + if err != nil { + t.Fatal(err) + } + return d +} + +func mustInstant(t *testing.T, s string) time.Time { + t.Helper() + v, err := time.Parse(time.RFC3339Nano, s) + if err != nil { + t.Fatal(err) + } + return v +} + +func storedLedger(e testusers.EncryptedLedger) map[string][]byte { + return map[string][]byte{"opened": e.Opened, "due": e.Due, "created": e.Created, "posted": e.Posted, "total": e.Total, "amount": e.Amount} +} + +// sameLedger compares two ledgers as values: instants with Equal, since an +// opened time.Time is in UTC and the original may not be. +func sameLedger(a, b testusers.Ledger) bool { + return a.ID == b.ID && a.Opened == b.Opened && a.Due == b.Due && + a.Created.Equal(b.Created) && a.Posted.Equal(b.Posted) && + a.Total == b.Total && a.Amount == b.Amount +} + +// The date, timestamp and numeric EQL v4 types, sealed through generated +// code over the deterministic guest build with the EQL types: an +// encrypt.Date, a time.Time and an encrypt.Decimal reach each family's +// storage and equality types, open back, and their Eq queries are the ones +// eql-bindings' own dispatch derives (v4_ledger_queries.json). +func TestV4DateTimestampNumericFieldsSealAndOpenBack(t *testing.T) { + c := deterministicEQLClient(t) + ctx := context.Background() + cipher := c.DefaultKeyset() + + east := time.FixedZone("UTC+10", 10*60*60) + ledgers := []testusers.Ledger{ + { + ID: 1, Opened: encrypt.Date{Year: 2024, Month: time.February, Day: 29}, Due: encrypt.Date{Year: 1, Month: time.January, Day: 1}, + Created: mustInstant(t, "1969-12-31T23:59:59.999999999Z"), Posted: time.Date(2024, time.March, 1, 9, 0, 0, 1, east), + Total: mustDecimal(t, "-79228162514264337593543950335"), Amount: mustDecimal(t, "1.50"), + }, + { + ID: 2, Opened: encrypt.Date{Year: -262143, Month: time.January, Day: 1}, Due: encrypt.Date{Year: 262142, Month: time.December, Day: 31}, + Created: mustInstant(t, "2024-02-29T12:34:56.123456789Z"), Posted: time.Unix(0, 0), + Total: mustDecimal(t, "0.0000000000000000000000000001"), Amount: mustDecimal(t, "-0.00"), + }, + } + encrypted, err := testusers.EncryptLedger(ctx, cipher, ledgers) + if err != nil { + t.Fatal(err) + } + for i, e := range encrypted { + for column, value := range storedLedger(e) { + want := "c,hm,i,v" + if column == "opened" || column == "created" || column == "total" { + want = "c,i,v" + } + if got := keysOf(t, column, value); got != want { + t.Errorf("row %d %s: keys %s, want %s", i, column, got, want) + } + } + } + decrypted, err := testusers.DecryptLedger(ctx, cipher, encrypted) + if err != nil { + t.Fatal(err) + } + for i := range ledgers { + if !sameLedger(decrypted[i], ledgers[i]) { + t.Errorf("row %d: %+v, want %+v", i, decrypted[i], ledgers[i]) + } + // An instant opens in UTC, with its nanoseconds. + if decrypted[i].Posted.Location() != time.UTC || decrypted[i].Posted.Nanosecond() != ledgers[i].Posted.Nanosecond() { + t.Errorf("row %d: posted opens as %v", i, decrypted[i].Posted) + } + } + // A decimal keeps its scale; a negative zero opens as zero. + if got := decrypted[0].Amount.String(); got != "1.50" { + t.Errorf("amount opens as %s, want 1.50", got) + } + if got := decrypted[1].Amount.String(); got != "0.00" { + t.Errorf("amount opens as %s, want 0.00", got) + } + + // Equality follows the baseline: a decimal's scale is normalised, and + // an instant keeps the microsecond. + amount := func(s string) []byte { + q, err := testusers.LedgerFields.Amount.Eq(ctx, cipher, mustDecimal(t, s)) + if err != nil { + t.Fatal(err) + } + return q + } + posted := func(v time.Time) []byte { + q, err := testusers.LedgerFields.Posted.Eq(ctx, cipher, v) + if err != nil { + t.Fatal(err) + } + return q + } + if !bytes.Equal(amount("1.5"), amount("1.50")) || bytes.Equal(amount("1.5"), amount("1.51")) { + t.Error("Amount.Eq does not normalise the scale") + } + early := posted(mustInstant(t, "2024-02-29T12:00:00.123456001Z")) + if !bytes.Equal(early, posted(mustInstant(t, "2024-02-29T12:00:00.123456999Z"))) || bytes.Equal(early, posted(mustInstant(t, "2024-02-29T12:00:00.123457Z"))) { + t.Error("Posted.Eq does not compare to the microsecond") + } + // The instant is compared, not the wall clock: the same instant in two + // locations is one term. + if !bytes.Equal(early, posted(time.Date(2024, time.February, 29, 22, 0, 0, 123456001, east))) { + t.Error("Posted.Eq differs for the same instant in another location") + } + + // What the engine cannot store is refused on every path, naming the + // field: a zero Date, a date past the range, an instant past it, and + // a decimal scale past 28. + refusals := map[string]error{} + _, refusals["EncryptLedger zero date"] = testusers.EncryptLedger(ctx, cipher, []testusers.Ledger{{ID: 3, Opened: ledgers[0].Opened, Created: ledgers[0].Created, Posted: ledgers[0].Posted}}) + _, refusals["Due.Encrypt zero"] = testusers.LedgerFields.Due.Encrypt(ctx, cipher, encrypt.Date{}) + _, refusals["Due.Eq past the range"] = testusers.LedgerFields.Due.Eq(ctx, cipher, encrypt.Date{Year: 262143, Month: time.January, Day: 1}) + _, refusals["Posted.Eq past the range"] = testusers.LedgerFields.Posted.Eq(ctx, cipher, time.Date(262143, time.January, 1, 0, 0, 0, 0, time.UTC)) + _, refusals["Amount.Eq scale 29"] = testusers.LedgerFields.Amount.Eq(ctx, cipher, encrypt.Decimal{Scale: 29}) + for path, err := range refusals { + if !errors.Is(err, encrypt.ErrEncoding) { + t.Errorf("%s: %v, want ErrEncoding", path, err) + } + } + + raw, err := os.ReadFile(fixturePath("v4_ledger_queries.json")) + if err != nil { + t.Fatal(err) + } + var fixture struct { + Due, Posted, Amount struct { + Plaintext string + Query string + } + } + if err := json.Unmarshal(raw, &fixture); err != nil { + t.Fatal(err) + } + queries := map[string]func(string) ([]byte, error){ + "due": func(s string) ([]byte, error) { + v, err := time.Parse(time.DateOnly, s) + if err != nil { + return nil, err + } + return testusers.LedgerFields.Due.Eq(ctx, cipher, encrypt.DateOf(v)) + }, + "posted": func(s string) ([]byte, error) { + return testusers.LedgerFields.Posted.Eq(ctx, cipher, mustInstant(t, s)) + }, + "amount": func(s string) ([]byte, error) { + return testusers.LedgerFields.Amount.Eq(ctx, cipher, mustDecimal(t, s)) + }, + } + wants := map[string]struct { + Plaintext string + Query string + }{"due": fixture.Due, "posted": fixture.Posted, "amount": fixture.Amount} + for column, derive := range queries { + want := wants[column] + got, err := derive(want.Plaintext) + if err != nil { + t.Fatal(err) + } + if want.Query == "" || string(got) != want.Query { + t.Errorf("%s.Eq = %s\nwant %s", column, got, want.Query) + } + } +} + +// ledgerRows are the rows v4_ledger_rows.json holds: the range's edges, a +// day, instant and number each of the fixture's queries finds, a value in +// the same microsecond and an amount at another scale, which the queries +// find too, and rows they do not. +func ledgerRows(t *testing.T) []testusers.Ledger { + return []testusers.Ledger{ + { + ID: 1, Opened: encrypt.Date{Year: -262143, Month: time.January, Day: 1}, Due: encrypt.Date{Year: 2024, Month: time.February, Day: 29}, + Created: mustInstant(t, "1969-12-31T23:59:59.999999999Z"), Posted: mustInstant(t, "2024-02-29T12:34:56.123456Z"), + Total: mustDecimal(t, "-79228162514264337593543950335"), Amount: mustDecimal(t, "1.50"), + }, + { + ID: 2, Opened: encrypt.Date{Year: 262142, Month: time.December, Day: 31}, Due: encrypt.Date{Year: 2024, Month: time.March, Day: 1}, + Created: mustInstant(t, "2024-02-29T12:34:56.123456789Z"), Posted: mustInstant(t, "2024-02-29T12:34:56.123456999Z"), + Total: mustDecimal(t, "0.0000000000000000000000000001"), Amount: mustDecimal(t, "1.5"), + }, + { + ID: 3, Opened: encrypt.Date{Year: 1970, Month: time.January, Day: 1}, Due: encrypt.Date{Year: 2024, Month: time.February, Day: 29}, + Created: mustInstant(t, "1970-01-01T00:00:00Z"), Posted: mustInstant(t, "2024-02-29T12:34:56.123457Z"), + Total: mustDecimal(t, "1234.5600"), Amount: mustDecimal(t, "1.51"), + }, + } +} + +type ledgerRowsFixture struct { + Comment string `json:"_comment"` + Seed string `json:"seed"` + Table string `json:"table"` + Rows []ledgerRow `json:"rows"` +} + +type ledgerRow struct { + ID int64 `json:"id"` + Plaintext ledgerPlain `json:"plaintext"` + Stored map[string]json.RawMessage `json:"stored"` +} + +// ledgerPlain is a row's plaintext as text: dates in ISO 8601, instants in +// RFC 3339 with nanoseconds, decimals as fixed-point text with their scale. +type ledgerPlain struct { + Opened string `json:"opened"` + Due string `json:"due"` + Created string `json:"created"` + Posted string `json:"posted"` + Total string `json:"total"` + Amount string `json:"amount"` +} + +// isoDate is a Date as ISO 8601, with the signed year chrono reads for +// years outside 0000 to 9999. +func isoDate(d encrypt.Date) string { + switch { + case d.Year < 0: + return fmt.Sprintf("-%04d-%02d-%02d", -d.Year, d.Month, d.Day) + case d.Year > 9999: + return fmt.Sprintf("+%04d-%02d-%02d", d.Year, d.Month, d.Day) + } + return fmt.Sprintf("%04d-%02d-%02d", d.Year, d.Month, d.Day) +} + +func plainLedger(l testusers.Ledger) ledgerPlain { + return ledgerPlain{ + Opened: isoDate(l.Opened), Due: isoDate(l.Due), + Created: l.Created.UTC().Format(time.RFC3339Nano), Posted: l.Posted.UTC().Format(time.RFC3339Nano), + Total: l.Total.String(), Amount: l.Amount.String(), + } +} + +// TestV4LedgerRowsMatchTheCommittedFixture checks the committed +// v4_ledger_rows.json against what the generated code seals today, as +// TestV4RowsMatchTheCommittedFixture does for v4_rows.json: everything but +// each `c` must match, and the committed `c` values must open. The Rust +// suite opens the same rows and stores them in PostgreSQL. EQL_UPDATE_FIXTURES +// rewrites the file when something other than the ciphertexts moved. +func TestV4LedgerRowsMatchTheCommittedFixture(t *testing.T) { + c := deterministicEQLClient(t) + ctx := context.Background() + rows := ledgerRows(t) + encrypted, err := testusers.EncryptLedger(ctx, c.DefaultKeyset(), rows) + if err != nil { + t.Fatal(err) + } + fixture := ledgerRowsFixture{ + Comment: "Ledger rows sealed by the Go SDK's generated code over the deterministic EQL guest with the seed below; the Rust suite opens and queries them. Regenerate with EQL_UPDATE_FIXTURES=1 go test ./encrypt -run TestV4LedgerRows, do not edit.", + Seed: string(testSeed[:]), + Table: "profiles", + } + for i, e := range encrypted { + row := ledgerRow{ID: rows[i].ID, Plaintext: plainLedger(rows[i]), Stored: map[string]json.RawMessage{}} + for column, value := range storedLedger(e) { + row.Stored[column] = json.RawMessage(value) + } + fixture.Rows = append(fixture.Rows, row) + } + rendered, err := json.MarshalIndent(fixture, "", " ") + if err != nil { + t.Fatal(err) + } + rendered = append(rendered, '\n') + path := fixturePath("v4_ledger_rows.json") + var committed ledgerRowsFixture + committedBytes, err := os.ReadFile(path) + if err == nil { + err = json.Unmarshal(committedBytes, &committed) + } + same := err == nil && bytes.Equal(ledgerWithoutCiphertexts(t, committed), ledgerWithoutCiphertexts(t, fixture)) + if os.Getenv("EQL_UPDATE_FIXTURES") != "" && !same { + if err := os.WriteFile(path, rendered, 0o644); err != nil { + t.Fatal(err) + } + t.Logf("%s: rewritten", path) + committed, same, err = fixture, true, nil + } + if err != nil { + t.Fatalf("%v: run with EQL_UPDATE_FIXTURES=1 to write it", err) + } + if !same { + t.Errorf("%s differs from what the generated code seals today beyond the ciphertexts; regenerate it with EQL_UPDATE_FIXTURES=1", path) + } + + sealed := make([]testusers.EncryptedLedger, len(committed.Rows)) + for i, row := range committed.Rows { + s := row.Stored + sealed[i] = testusers.EncryptedLedger{ + ID: row.ID, Opened: []byte(s["opened"]), Due: []byte(s["due"]), Created: []byte(s["created"]), + Posted: []byte(s["posted"]), Total: []byte(s["total"]), Amount: []byte(s["amount"]), + } + } + opened, err := testusers.DecryptLedger(ctx, c.DefaultKeyset(), sealed) + if err != nil { + t.Fatal(err) + } + for i, row := range committed.Rows { + if got := plainLedger(opened[i]); got != row.Plaintext { + t.Errorf("row %d opens to %+v, want %+v", row.ID, got, row.Plaintext) + } + } +} + +func ledgerWithoutCiphertexts(t *testing.T, f ledgerRowsFixture) []byte { + t.Helper() + out := f + out.Rows = nil + for _, row := range f.Rows { + stripped := ledgerRow{ID: row.ID, Plaintext: row.Plaintext, Stored: map[string]json.RawMessage{}} + for column, value := range row.Stored { + var fields map[string]json.RawMessage + if err := json.Unmarshal(value, &fields); err != nil { + t.Fatal(err) + } + delete(fields, "c") + b, err := json.Marshal(fields) + if err != nil { + t.Fatal(err) + } + stripped.Stored[column] = b + } + out.Rows = append(out.Rows, stripped) + } + b, err := json.Marshal(out) + if err != nil { + t.Fatal(err) + } + return b +} diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index 649c9a375..a5bb1941d 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -711,6 +711,7 @@ dependencies = [ "stack-encrypt", "thiserror 2.0.20", "ts-rs", + "vitaminc-aead 0.6.1", "zeroize", ] diff --git a/languages/golang/encrypt/guest/src/targets.rs b/languages/golang/encrypt/guest/src/targets.rs index 416d2e70d..b863f0669 100644 --- a/languages/golang/encrypt/guest/src/targets.rs +++ b/languages/golang/encrypt/guest/src/targets.rs @@ -201,6 +201,12 @@ mod tests { "BigintEq", "BigintOrd", "BigintOrdOpe", + "Date", + "DateEq", + "Timestamp", + "TimestampEq", + "Numeric", + "NumericEq", "Text", "TextEq", "TextMatch", diff --git a/languages/golang/encrypt/guest/tests/native_ops.rs b/languages/golang/encrypt/guest/tests/native_ops.rs index acbc63b45..4da201ab8 100644 --- a/languages/golang/encrypt/guest/tests/native_ops.rs +++ b/languages/golang/encrypt/guest/tests/native_ops.rs @@ -1375,6 +1375,12 @@ fn targets_lists_what_the_build_holds() { "BigintEq", "BigintOrd", "BigintOrdOpe", + "Date", + "DateEq", + "Timestamp", + "TimestampEq", + "Numeric", + "NumericEq", "Text", "TextEq", "TextMatch", @@ -1793,7 +1799,8 @@ fn the_catalog_lists_each_types_operators() { want.extend(orderings()); assert_eq!(listed, want, "{name}"); } - "TextEq" | "IntegerEq" | "SmallintEq" | "BigintEq" | "RealEq" | "DoubleEq" => only([ + "TextEq" | "IntegerEq" | "SmallintEq" | "BigintEq" | "RealEq" | "DoubleEq" + | "DateEq" | "TimestampEq" | "NumericEq" => only([ ("operator", "eq"), ("fidelity", "exact"), ("arity", "one"), diff --git a/languages/golang/encrypt/internal/testusers/ledger.go b/languages/golang/encrypt/internal/testusers/ledger.go new file mode 100644 index 000000000..d1a657e1f --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/ledger.go @@ -0,0 +1,27 @@ +package testusers + +import ( + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt" +) + +//go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Ledger -name Ledger + +// Ledger stores the date, timestamp and numeric EQL v4 types: each family's +// storage-only type and its equality type, from the Go types ADR-0002 maps +// them to (encrypt.Date, time.Time, encrypt.Decimal). Its columns are under +// Profile's table, with names of their own, so its queries sit beside +// v4_queries.json's in v4_ledger_queries.json. It is a model of its own +// because Profile's rows are frozen in stored_v4_rows_v1.json: a new Profile +// field would change what those rows store. +type Ledger struct { + _ struct{} `stash:"context=profiles"` + ID int64 `stash:"id,passthrough"` + Opened encrypt.Date `stash:"opened,encrypt_into=Date"` + Due encrypt.Date `stash:"due,encrypt_into=DateEq"` + Created time.Time `stash:"created,encrypt_into=Timestamp"` + Posted time.Time `stash:"posted,encrypt_into=TimestampEq"` + Total encrypt.Decimal `stash:"total,encrypt_into=Numeric"` + Amount encrypt.Decimal `stash:"amount,encrypt_into=NumericEq"` +} diff --git a/languages/golang/encrypt/internal/testusers/ledger_stash.go b/languages/golang/encrypt/internal/testusers/ledger_stash.go new file mode 100644 index 000000000..eb1db959d --- /dev/null +++ b/languages/golang/encrypt/internal/testusers/ledger_stash.go @@ -0,0 +1,226 @@ +// Code generated by stashgen. DO NOT EDIT. + +package testusers + +import ( + "context" + "log/slog" + "time" + + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/eql" + "github.com/cipherstash/stack/languages/golang/encrypt/gensupport" +) + +// Stops compiling when the library does not accept this version of generated file. +const _ = gensupport.GeneratedVersion1 + +// Ledger prints its sealed fields in the clear: it has no String or LogValue +// method. Write them, or run stashgen with -redact. + +type EncryptedLedger struct { + ID int64 + Opened eql.Date + Due eql.DateEq + Created eql.Timestamp + Posted eql.TimestampEq + Total eql.Numeric + Amount eql.NumericEq +} + +func (e EncryptedLedger) String() string { + return gensupport.Redacted("EncryptedLedger", map[string]any{"ID": e.ID}, "Opened", "Due", "Created", "Posted", "Total", "Amount") +} + +func (e EncryptedLedger) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Opened", "Due", "Created", "Posted", "Total", "Amount") +} + +// Stops compiling when Ledger gains, loses, reorders or retypes a field. +var _ = ledgerShape(Ledger{}) + +type ledgerShape struct { + _ struct{} + ID int64 + Opened encrypt.Date + Due encrypt.Date + Created time.Time + Posted time.Time + Total encrypt.Decimal + Amount encrypt.Decimal +} + +var ledgerDeclaration = gensupport.Declare("profiles"). + Passthrough("id"). + EncryptInto("opened", gensupport.Date, "Date"). + EncryptInto("due", gensupport.Date, "DateEq"). + EncryptInto("created", gensupport.Timestamp, "Timestamp"). + EncryptInto("posted", gensupport.Timestamp, "TimestampEq"). + EncryptInto("total", gensupport.Decimal, "Numeric"). + EncryptInto("amount", gensupport.Decimal, "NumericEq") + +var ledgerCodec = gensupport.New(gensupport.Generated[Ledger, EncryptedLedger]{ + TypeName: "Ledger", + Declaration: ledgerDeclaration, + PrintsPlaintext: true, + Source: func(v Ledger) gensupport.Values { + return gensupport.Values{ + "id": v.ID, + "opened": v.Opened, + "due": v.Due, + "created": v.Created, + "posted": v.Posted, + "total": v.Total, + "amount": v.Amount, + } + }, + Seal: func(rec gensupport.Record) (EncryptedLedger, error) { + var e EncryptedLedger + var err error + if e.ID, err = gensupport.Passthrough[int64](rec, "id"); err != nil { + return EncryptedLedger{}, err + } + e.Opened = eql.Date(rec["opened"].EQL) + e.Due = eql.DateEq(rec["due"].EQL) + e.Created = eql.Timestamp(rec["created"].EQL) + e.Posted = eql.TimestampEq(rec["posted"].EQL) + e.Total = eql.Numeric(rec["total"].EQL) + e.Amount = eql.NumericEq(rec["amount"].EQL) + return e, nil + }, + Open: func(e EncryptedLedger) gensupport.Record { + return gensupport.Record{ + "id": {Value: e.ID}, + "opened": {EQL: e.Opened}, + "due": {EQL: e.Due}, + "created": {EQL: e.Created}, + "posted": {EQL: e.Posted}, + "total": {EQL: e.Total}, + "amount": {EQL: e.Amount}, + } + }, + Value: func(e EncryptedLedger, vals gensupport.Values) (Ledger, error) { + var v Ledger + var err error + if v.ID, err = gensupport.Get[int64](vals, "id"); err != nil { + return Ledger{}, err + } + if v.Opened, err = gensupport.Get[encrypt.Date](vals, "opened"); err != nil { + return Ledger{}, err + } + if v.Due, err = gensupport.Get[encrypt.Date](vals, "due"); err != nil { + return Ledger{}, err + } + if v.Created, err = gensupport.Get[time.Time](vals, "created"); err != nil { + return Ledger{}, err + } + if v.Posted, err = gensupport.Get[time.Time](vals, "posted"); err != nil { + return Ledger{}, err + } + if v.Total, err = gensupport.Get[encrypt.Decimal](vals, "total"); err != nil { + return Ledger{}, err + } + if v.Amount, err = gensupport.Get[encrypt.Decimal](vals, "amount"); err != nil { + return Ledger{}, err + } + return v, nil + }, +}) + +// EncryptLedger seals each Ledger, with one ZeroKMS request for each 500 sealed +// values, plus one the first time a keyset is used. The result has one element +// for each input, in the same order. +func EncryptLedger(ctx context.Context, cipher *encrypt.Cipher, values []Ledger) ([]EncryptedLedger, error) { + return ledgerCodec.Encrypt(ctx, cipher, values) +} + +// DecryptLedger opens each EncryptedLedger, with one ZeroKMS request for each +// 500 sealed values, plus one the first time a keyset is used. +func DecryptLedger(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedLedger) ([]Ledger, error) { + return ledgerCodec.Decrypt(ctx, d, encrypted) +} + +var LedgerFields = struct { + Opened LedgerOpenedField + Due LedgerDueField + Created LedgerCreatedField + Posted LedgerPostedField + Total LedgerTotalField + Amount LedgerAmountField +}{ + Opened: LedgerOpenedField{gensupport.NewField[encrypt.Date](ledgerDeclaration, "opened")}, + Due: LedgerDueField{gensupport.NewField[encrypt.Date](ledgerDeclaration, "due")}, + Created: LedgerCreatedField{gensupport.NewField[time.Time](ledgerDeclaration, "created")}, + Posted: LedgerPostedField{gensupport.NewField[time.Time](ledgerDeclaration, "posted")}, + Total: LedgerTotalField{gensupport.NewField[encrypt.Decimal](ledgerDeclaration, "total")}, + Amount: LedgerAmountField{gensupport.NewField[encrypt.Decimal](ledgerDeclaration, "amount")}, +} + +type LedgerOpenedField struct { + field gensupport.Field[encrypt.Date] +} + +func (f LedgerOpenedField) Encrypt(ctx context.Context, c *encrypt.Cipher, v encrypt.Date) (eql.Date, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.Date(out.EQL), err +} + +type LedgerDueField struct { + field gensupport.Field[encrypt.Date] +} + +func (f LedgerDueField) Encrypt(ctx context.Context, c *encrypt.Cipher, v encrypt.Date) (eql.DateEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.DateEq(out.EQL), err +} + +func (f LedgerDueField) Eq(ctx context.Context, c *encrypt.Cipher, v encrypt.Date) (eql.DateEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.DateEqQuery(term), err +} + +type LedgerCreatedField struct { + field gensupport.Field[time.Time] +} + +func (f LedgerCreatedField) Encrypt(ctx context.Context, c *encrypt.Cipher, v time.Time) (eql.Timestamp, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.Timestamp(out.EQL), err +} + +type LedgerPostedField struct { + field gensupport.Field[time.Time] +} + +func (f LedgerPostedField) Encrypt(ctx context.Context, c *encrypt.Cipher, v time.Time) (eql.TimestampEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TimestampEq(out.EQL), err +} + +func (f LedgerPostedField) Eq(ctx context.Context, c *encrypt.Cipher, v time.Time) (eql.TimestampEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.TimestampEqQuery(term), err +} + +type LedgerTotalField struct { + field gensupport.Field[encrypt.Decimal] +} + +func (f LedgerTotalField) Encrypt(ctx context.Context, c *encrypt.Cipher, v encrypt.Decimal) (eql.Numeric, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.Numeric(out.EQL), err +} + +type LedgerAmountField struct { + field gensupport.Field[encrypt.Decimal] +} + +func (f LedgerAmountField) Encrypt(ctx context.Context, c *encrypt.Cipher, v encrypt.Decimal) (eql.NumericEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.NumericEq(out.EQL), err +} + +func (f LedgerAmountField) Eq(ctx context.Context, c *encrypt.Cipher, v encrypt.Decimal) (eql.NumericEqQuery, error) { + term, err := f.field.Term(ctx, c, "eq", v) + return eql.NumericEqQuery(term), err +} diff --git a/languages/golang/encrypt/query_test.go b/languages/golang/encrypt/query_test.go index 6a5200345..7b948450a 100644 --- a/languages/golang/encrypt/query_test.go +++ b/languages/golang/encrypt/query_test.go @@ -409,7 +409,7 @@ func TestTheCatalogListsEachTypesOperators(t *testing.T) { } for _, target := range targets { switch { - case slices.Contains([]string{"TextEq", "IntegerEq", "SmallintEq", "BigintEq", "RealEq", "DoubleEq"}, target.Name): + case slices.Contains([]string{"TextEq", "IntegerEq", "SmallintEq", "BigintEq", "RealEq", "DoubleEq", "DateEq", "TimestampEq", "NumericEq"}, target.Name): want := []record.Operator{{Name: "eq", Fidelity: record.Exact, Index: record.Equality}} if len(target.Operators) != 1 || target.Operators[0] != want[0] { t.Fatalf("%s answers %+v, want %+v", target.Name, target.Operators, want) diff --git a/packages/eql/Cargo.lock b/packages/eql/Cargo.lock index cc35cb4e8..e85b3af88 100644 --- a/packages/eql/Cargo.lock +++ b/packages/eql/Cargo.lock @@ -1199,14 +1199,17 @@ name = "eql-bindings" version = "3.0.6" dependencies = [ "base64", + "chrono", "eql-domains", "hex", + "rust_decimal", "schemars", "serde", "serde_json", "stack-encrypt", "thiserror 2.0.20", "ts-rs", + "vitaminc-aead 0.6.1", "zeroize", ] @@ -1237,10 +1240,12 @@ name = "eql-encryption-tests" version = "0.1.0" dependencies = [ "base64", + "chrono", "cipherstash-client", "eql-bindings", "eql-domains", "postgres", + "rust_decimal", "serde", "serde_json", "stack-encrypt", @@ -5095,7 +5100,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "71b8c5c4ced92545dce5fd7cf04581d08210def174cc81475cf227fa2506c5b7" dependencies = [ "bytes", + "chrono", "futures-util", + "rust_decimal", "serde", "thiserror 2.0.20", "vitaminc-aead-derive 0.6.1", diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index 864b46d7e..d2a2b7c15 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -30,6 +30,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 The `date`, `timestamp` and `numeric` families say why they are still unproducible: the engine has their kinds, and their Rust plaintext types are not chosen yet. +- **`Date`, `DateEq`, `Timestamp`, `TimestampEq`, `Numeric` and `NumericEq` + are producible** (`stack-encrypt` feature), from the new + `v4::plaintext::Date` (an `i32` of days since 1970-01-01), + `v4::plaintext::Timestamp` (`i64` Unix seconds and `u32` nanoseconds, + UTC) and `v4::plaintext::Decimal` (an `i128` mantissa and a `u8` scale). + Each is built range checked against vitaminc's ranges, refused with a + `v4::plaintext::RangeError` that does not show the value, wiped on drop + (`Zeroize`, `ZeroizeOnDrop`), and not `Copy`: chrono's and + rust_decimal's types cannot be wiped, so the by-name dispatch could not + hold them. The `chrono` and `rust_decimal` features add conversions to + and from `NaiveDate`, `DateTime` and rust_decimal's `Decimal`; a + leap second is refused. `NumericEq`'s `=` normalises the scale and + `TimestampEq`'s compares to the microsecond, truncating (never + rounding) the nanoseconds below it. The ordered types of these + families stay unproducible, with a reason: the engine derives no order + term for these kinds yet (#1236). - **The OPE-ordered types are producible** (`stack-encrypt` feature): `IntegerOrd`, `IntegerOrdOpe`, `SmallintOrd`, `SmallintOrdOpe`, `BigintOrd`, `BigintOrdOpe`, `RealOrd`, `RealOrdOpe`, `DoubleOrd`, diff --git a/packages/eql/crates/eql-bindings/Cargo.toml b/packages/eql/crates/eql-bindings/Cargo.toml index eae736220..0e4be7dab 100644 --- a/packages/eql/crates/eql-bindings/Cargo.toml +++ b/packages/eql/crates/eql-bindings/Cargo.toml @@ -46,12 +46,22 @@ schemars = "1" # No default features and no `dynamic`: the typed EQL v4 types need only # Stack Encrypt's typed API. The `targets` feature turns `dynamic` on. stack-encrypt = { path = "../../../stack-encrypt", version = "0.3.0", optional = true, default-features = false } -# `targets` only. The plaintext a target reads out of its `Protected` -# runtime value is held in `Zeroizing` until it is sealed, so the copy is -# wiped like the original. +# The plaintext a target reads out of its `Protected` runtime value is held +# in `Zeroizing` until it is sealed, so the copy is wiped like the original; +# and the date, timestamp and decimal plaintexts (`v4::plaintext`) wipe +# themselves on drop. zeroize = { version = "1", optional = true } -# `targets` only: `TargetError`. +# `TargetError` and `v4::plaintext::RangeError`. thiserror = { version = "2", optional = true } +# `stack-encrypt` only: the sink and scalar vocabulary the date, timestamp +# and decimal plaintexts (`v4::plaintext`) describe themselves with, and +# vitaminc's range checks for them. The version Stack Encrypt uses, so the +# traits are the ones it calls. +vitaminc-aead = { version = "0.6.0", optional = true } +# The `chrono` and `rust_decimal` features: conversions between those +# crates' types and `v4::plaintext`'s. +chrono = { version = "0.4.44", optional = true, default-features = false } +rust_decimal = { version = "1", optional = true, default-features = false } base64 = { version = "0.22", optional = true } hex = { version = "0.4", optional = true } @@ -64,7 +74,15 @@ stack-encrypt = [ "dep:stack-encrypt", "dep:base64", "dep:hex", + "dep:vitaminc-aead", + "dep:zeroize", + "dep:thiserror", ] +# Conversions between chrono's `NaiveDate` / `DateTime` and +# `v4::plaintext::Date` / `Timestamp`. +chrono = ["stack-encrypt", "dep:chrono", "vitaminc-aead/chrono"] +# Conversions between rust_decimal's `Decimal` and `v4::plaintext::Decimal`. +rust_decimal = ["stack-encrypt", "dep:rust_decimal", "vitaminc-aead/rust_decimal"] # The by-name dispatch a language binding needs (`encryption::targets`, # `v4::targets`): an EQL type named at run time, a `ValueFrame` plaintext, # and the target table a guest serializes. Turns on stack-encrypt's @@ -73,12 +91,10 @@ stack-encrypt = [ targets = [ "stack-encrypt", "stack-encrypt/dynamic", - "dep:zeroize", - "dep:thiserror", ] [package.metadata.docs.rs] -features = ["targets"] +features = ["targets", "chrono", "rust_decimal"] [dev-dependencies] # Parity oracle: tests/catalog_parity.rs asserts the v3 domain inventory diff --git a/packages/eql/crates/eql-bindings/README.md b/packages/eql/crates/eql-bindings/README.md index c519e91e5..d8e278d9d 100644 --- a/packages/eql/crates/eql-bindings/README.md +++ b/packages/eql/crates/eql-bindings/README.md @@ -107,7 +107,8 @@ The optional `stack-encrypt` feature enables `EncryptFrom` for implements these traits. The derives are emitted by `eql-codegen`. The other v4 types Stack Encrypt produces take a Rust primitive as their -plaintext: +plaintext, or, for a date, a timestamp or a decimal, one of +`eql_bindings::v4::plaintext`'s types: | v4 type | Plaintext | | --- | --- | @@ -118,6 +119,22 @@ plaintext: | `Boolean` | `bool` | | `Real`, `RealEq`, `RealOrd`, `RealOrdOpe` | `f32` | | `Double`, `DoubleEq`, `DoubleOrd`, `DoubleOrdOpe` | `f64` | +| `Date`, `DateEq` | `v4::plaintext::Date`: an `i32` of days since 1970-01-01 | +| `Timestamp`, `TimestampEq` | `v4::plaintext::Timestamp`: `i64` Unix seconds and `u32` nanoseconds, UTC, no leap seconds | +| `Numeric`, `NumericEq` | `v4::plaintext::Decimal`: an `i128` mantissa (below 2^96) and a `u8` scale (at most 28) | + +A date, a timestamp and a decimal have no Rust standard type, and chrono's +and rust_decimal's types cannot be wiped, so these three are this crate's +own. Each is built range checked (vitaminc's ranges, which are chrono's and +rust_decimal's), refusing a value outside them with an error that does not +show it, and wipes its integers when dropped. It is not `Copy`. The `chrono` +and `rust_decimal` features convert to and from `NaiveDate`, +`DateTime` and rust_decimal's `Decimal`; the value converted to is the +caller's, and is not wiped. Only a leap second, which a `DateTime` can +hold and a `Timestamp` cannot, is refused in a conversion. On `NumericEq`, +`1.5 = 1.50` (the scale is normalised); on `TimestampEq`, an instant is truncated to the microsecond (the nanoseconds below it are dropped, not rounded), so two instants in the same microsecond are equal. Each opens as +it was stored. The ordered date, timestamp and numeric types are refused +until the engine derives their order terms. Each seals the same leaf the Go SDK seals for the same value, so values written here and by the Go SDK open each other. On the equality diff --git a/packages/eql/crates/eql-bindings/src/encryption/targets.rs b/packages/eql/crates/eql-bindings/src/encryption/targets.rs index 0e5ac90d0..5eb2f91a8 100644 --- a/packages/eql/crates/eql-bindings/src/encryption/targets.rs +++ b/packages/eql/crates/eql-bindings/src/encryption/targets.rs @@ -414,6 +414,9 @@ mod sealed { impl Sealed for bool {} impl Sealed for f32 {} impl Sealed for f64 {} + impl Sealed for crate::v4::plaintext::Date {} + impl Sealed for crate::v4::plaintext::Timestamp {} + impl Sealed for crate::v4::plaintext::Decimal {} } /// A Rust plaintext an EQL type is produced from, read out of and written @@ -495,6 +498,20 @@ scalar_plaintext!( f64 => Float64, ); +/// The date, timestamp and decimal plaintexts read exactly their own kind, +/// and are range checked as they are read: a frame vitaminc's value model +/// admits is always in range, so the check refuses nothing a binding can +/// send. Each wipes itself on drop as well as inside `Zeroizing`. +mod logical { + use super::*; + use crate::v4::plaintext::{Date, Decimal, Timestamp}; + scalar_plaintext!( + Date => Date, + Timestamp => Timestamp, + Decimal => Decimal, + ); +} + /// Run a target's own plan over one runtime value and resolve to the EQL /// value's JSON bytes. The generated dispatch calls this with the type and /// its plaintext; it is the one place the typed `encrypt_as` is reached from diff --git a/packages/eql/crates/eql-bindings/src/v4/date.rs b/packages/eql/crates/eql-bindings/src/v4/date.rs index f704939bb..8a161b347 100644 --- a/packages/eql/crates/eql-bindings/src/v4/date.rs +++ b/packages/eql/crates/eql-bindings/src/v4/date.rs @@ -14,10 +14,18 @@ use ts_rs::TS; doc, doc = " See the [EQL date reference](https://cipherstash.com/docs/reference/eql/dates-and-times) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`Date`](crate::v4::plaintext::Date). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = crate::v4::plaintext::Date)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct Date { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, } @@ -55,10 +63,18 @@ impl DomainType for Date { doc, doc = " See the [EQL date reference](https://cipherstash.com/docs/reference/eql/dates-and-times) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`Date`](crate::v4::plaintext::Date). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = crate::v4::plaintext::Date)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct DateEq { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub hm: Hmac256, @@ -223,10 +239,18 @@ impl DomainType for DateOrdOpe { doc, doc = " See the [EQL date reference](https://cipherstash.com/docs/reference/eql/dates-and-times) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`Date`](crate::v4::plaintext::Date). Implements `EncryptFrom` to build equality query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = crate::v4::plaintext::Date)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct DateEqQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub hm: Hmac256, } diff --git a/packages/eql/crates/eql-bindings/src/v4/mod.rs b/packages/eql/crates/eql-bindings/src/v4/mod.rs index b795fb9fa..09b9eed5c 100644 --- a/packages/eql/crates/eql-bindings/src/v4/mod.rs +++ b/packages/eql/crates/eql-bindings/src/v4/mod.rs @@ -30,6 +30,10 @@ pub mod inventory; /// (`public.eql_v4_json`, the `Json` struct). pub mod json; pub mod numeric; +// The Rust plaintexts of the `date`, `timestamp` and `numeric` families. A +// plain comment: an outer doc comment here would resolve the module's own +// links from this scope. +pub mod plaintext; pub mod real; pub mod smallint; /// Generated: the table of EQL types a Stack Encrypt data plan may name as diff --git a/packages/eql/crates/eql-bindings/src/v4/numeric.rs b/packages/eql/crates/eql-bindings/src/v4/numeric.rs index ff46246d4..61d02e4b1 100644 --- a/packages/eql/crates/eql-bindings/src/v4/numeric.rs +++ b/packages/eql/crates/eql-bindings/src/v4/numeric.rs @@ -14,10 +14,18 @@ use ts_rs::TS; doc, doc = " See the [EQL numeric reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`Decimal`](crate::v4::plaintext::Decimal). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = crate::v4::plaintext::Decimal)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct Numeric { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, } @@ -55,10 +63,18 @@ impl DomainType for Numeric { doc, doc = " See the [EQL numeric reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`Decimal`](crate::v4::plaintext::Decimal). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = crate::v4::plaintext::Decimal)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct NumericEq { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub hm: Hmac256, @@ -223,10 +239,18 @@ impl DomainType for NumericOrdOpe { doc, doc = " See the [EQL numeric reference](https://cipherstash.com/docs/reference/eql/numbers) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`Decimal`](crate::v4::plaintext::Decimal). Implements `EncryptFrom` to build equality query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = crate::v4::plaintext::Decimal)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct NumericEqQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub hm: Hmac256, } diff --git a/packages/eql/crates/eql-bindings/src/v4/plaintext.rs b/packages/eql/crates/eql-bindings/src/v4/plaintext.rs new file mode 100644 index 000000000..de602810c --- /dev/null +++ b/packages/eql/crates/eql-bindings/src/v4/plaintext.rs @@ -0,0 +1,792 @@ +//! The plaintexts of the `date`, `timestamp` and `numeric` families: a +//! calendar date, a UTC instant and a decimal number, each held as the +//! integers Stack Encrypt's value model stores it as (ADR-0002). +//! +//! The other families are produced from Rust's own types (`i32`, `f64`, +//! `String`). These three have no Rust standard type, and the types the +//! ecosystem offers (chrono's `NaiveDate` and `DateTime`, +//! rust_decimal's `Decimal`) keep their fields private, so their bytes +//! cannot be wiped. A plaintext the by-name dispatch copies out of the +//! engine's `Protected` buffer must be wiped when it is dropped, so the +//! families are produced from the types here instead. +//! +//! vitaminc-aead has a wipeable `Timestamp` of its own, and it is not +//! reused: Stack Encrypt hands each term of a type its own copy of the +//! plaintext, so its ciphertext and equality targets need a `Clone` +//! plaintext, and vitaminc's type is deliberately not `Clone`. It is also +//! built only from a `DateTime`, which would make chrono a dependency +//! of every build. The types here are `Clone`, and a clone is wiped too. +//! +//! | Type | Holds | Range | +//! |---|---|---| +//! | [`Date`] | an `i32` count of days since 1970-01-01, in the proleptic Gregorian calendar | years -262143 to 262142 | +//! | [`Timestamp`] | an `i64` count of seconds since the Unix epoch and a `u32` of nanoseconds below one second, UTC, with no leap seconds | years -262143 to 262142 | +//! | [`Decimal`] | an `i128` mantissa and a `u8` scale: the value is `mantissa × 10^-scale` | scale at most 28, mantissa below 2^96 in magnitude | +//! +//! The ranges are vitaminc's, which are chrono's and rust_decimal's: a +//! value outside them has no encoding in the engine and is refused when the +//! type is built, so a value of one of these types always encrypts. A +//! refusal ([`RangeError`]) names what was out of range, never the value. +//! +//! # What wiping covers +//! +//! Each type implements [`Zeroize`] and wipes its integers when it is +//! dropped ([`ZeroizeOnDrop`]), as the engine wipes the copy of an `i64` it +//! opens. That covers every owned copy this crate and Stack Encrypt make. +//! The integer a value describes itself to vitaminc's sink with is a `Copy` +//! stack temporary, as it is for the engine's own primitive, chrono and +//! rust_decimal sources, and is not wiped. Nor is a value the caller builds +//! the type from, or converts it to: a +//! `NaiveDate`, `DateTime` or rust_decimal `Decimal` (with the +//! `chrono` and `rust_decimal` features), or the integers read through an +//! accessor, are ordinary values the caller looks after. +//! +//! None of the types is `Copy`, so a copy is never made without a trace. +//! Each is `Clone`, because Stack Encrypt hands each term of a type its own +//! copy of the plaintext; a clone is wiped when it is dropped too. `Debug` +//! is redacted, and there is no `PartialEq`, `Ord` or `Hash`, which would +//! compare plaintext in variable time: compare the accessors' integers. +//! +//! # Equality +//! +//! The ciphertext keeps the value exactly as given, and an equality term is +//! derived from vitaminc's equality encoding of it: +//! +//! - a decimal's scale is normalised, so `1.5` and `1.50` have one term and +//! `=` matches them, while each opens as it was stored; +//! - a timestamp is truncated to the microsecond: the nanoseconds below +//! it are dropped, never rounded, so two instants in the same microsecond +//! match under `=`, while each opens with its own nanoseconds. Truncation +//! is floor on the count since the epoch, so it moves an instant before +//! 1970 earlier too. It is the rule the order term applies, so `=` and the +//! order agree, and it does not depend on any one store's rounding; +//! - a date is its day. + +use std::fmt; + +use stack_encrypt::Encrypt; +use vitaminc_aead::plaintext::depth; +use vitaminc_aead::plaintext::kinds::{timestamp_nanos, MAX_DECIMAL_SCALE}; +use vitaminc_aead::{FromScalar, Int, Logical, Physical, Sink, TimeUnit}; +use zeroize::{Zeroize, ZeroizeOnDrop}; + +const NANOS_PER_SECOND: u32 = 1_000_000_000; + +/// Why a value could not be built as one of this module's plaintexts. +/// +/// The message names the range, never the value: the value is plaintext, +/// and an error is what a program logs. +#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)] +#[non_exhaustive] +pub enum RangeError { + /// A day count outside chrono's date range, years -262143 to 262142. + #[error("the date is outside the representable range, years -262143 to 262142")] + Date, + /// An instant outside chrono's range, years -262143 to 262142. + #[error("the timestamp is outside the representable range, years -262143 to 262142")] + Timestamp, + /// Nanoseconds of one second or more: not a nanosecond field, and + /// chrono's representation of a leap second, which has no count of + /// seconds since the epoch. + #[error( + "the timestamp's nanoseconds must be below one second (chrono holds a leap second \ + this way, and it has no encoding)" + )] + LeapSecond, + /// A decimal scale above 28. + #[error("the decimal's scale is above 28")] + DecimalScale, + /// A decimal mantissa of 2^96 or more in magnitude. + #[error("the decimal's mantissa does not fit 96 bits")] + DecimalMantissa, +} + +/// A calendar date with no time of day and no time zone: a count of days +/// since 1970-01-01 in the proleptic Gregorian calendar. The plaintext of +/// the `date` family's EQL v4 types (`Date`, `DateEq`). +/// +/// 1970-01-01 is day 0 and 1969-12-31 is day -1: the day count vitaminc's +/// value model stores a date as, and the one a `NaiveDate` converts to with +/// the `chrono` feature. See the [module documentation](self) for what +/// wiping covers. +/// +/// ``` +/// use eql_bindings::v4::plaintext::Date; +/// +/// let day = Date::from_days_since_epoch(19_782)?; // 2024-02-29 +/// assert_eq!(day.days_since_epoch(), 19_782); +/// # Ok::<(), eql_bindings::v4::plaintext::RangeError>(()) +/// ``` +#[derive(Clone)] +pub struct Date { + days: i32, +} + +impl Date { + /// The date `days` after 1970-01-01, or before it for a negative count. + /// + /// # Errors + /// + /// [`RangeError::Date`] outside years -262143 to 262142. + pub fn from_days_since_epoch(days: i32) -> Result { + Logical::Date + .admits(&Physical::Int(Int::I32(days))) + .then_some(Self { days }) + .ok_or(RangeError::Date) + } + + /// The count of days since 1970-01-01. The integer is the caller's to + /// look after: it is a copy, and is not wiped. + pub fn days_since_epoch(&self) -> i32 { + self.days + } +} + +/// A UTC instant with nanosecond precision and no leap seconds: whole +/// seconds since the Unix epoch, and nanoseconds past them. The plaintext +/// of the `timestamp` family's EQL v4 types (`Timestamp`, `TimestampEq`). +/// +/// The ciphertext keeps the nanoseconds; an equality term keeps the +/// microseconds, truncated rather than rounded (see the [module documentation](self#equality), which also +/// says what wiping covers). +/// +/// ``` +/// use eql_bindings::v4::plaintext::{RangeError, Timestamp}; +/// +/// let instant = Timestamp::new(1_700_000_000, 500)?; +/// assert_eq!((instant.seconds(), instant.nanoseconds()), (1_700_000_000, 500)); +/// assert_eq!(Timestamp::new(0, 1_000_000_000).unwrap_err(), RangeError::LeapSecond); +/// # Ok::<(), RangeError>(()) +/// ``` +#[derive(Clone)] +pub struct Timestamp { + seconds: i64, + nanos: u32, +} + +impl Timestamp { + /// The instant `seconds` after the Unix epoch, plus `nanoseconds`. + /// A negative `seconds` is before 1970; the nanoseconds still count + /// forward from it, so -0.5 seconds is `new(-1, 500_000_000)`. + /// + /// # Errors + /// + /// [`RangeError::LeapSecond`] for nanoseconds of one second or more, + /// and [`RangeError::Timestamp`] outside years -262143 to 262142. + pub fn new(seconds: i64, nanoseconds: u32) -> Result { + if nanoseconds >= NANOS_PER_SECOND { + return Err(RangeError::LeapSecond); + } + let instant = Self { + seconds, + nanos: nanoseconds, + }; + timestamp_nanos(TimeUnit::Nanos, Int::I128(instant.count())) + .map(|_| instant) + .ok_or(RangeError::Timestamp) + } + + /// Whole seconds since the Unix epoch, negative before 1970. A copy, + /// not wiped. + pub fn seconds(&self) -> i64 { + self.seconds + } + + /// Nanoseconds past [`seconds`](Self::seconds), below one second. A + /// copy, not wiped. + pub fn nanoseconds(&self) -> u32 { + self.nanos + } + + /// Nanoseconds since the epoch: the count vitaminc stores a timestamp + /// as, and the one a `DateTime` encrypts to, so the two seal the + /// same leaf. Cannot overflow: an `i64` of seconds times 10^9 is far + /// inside an `i128`. + fn count(&self) -> i128 { + i128::from(self.seconds) * i128::from(NANOS_PER_SECOND) + i128::from(self.nanos) + } + + /// The instant for a count of nanoseconds already checked to be in + /// range. + fn from_nanos(nanos: i128) -> Option { + let per_second = i128::from(NANOS_PER_SECOND); + Some(Self { + seconds: i64::try_from(nanos.div_euclid(per_second)).ok()?, + nanos: u32::try_from(nanos.rem_euclid(per_second)).ok()?, + }) + } +} + +/// A finite decimal number with its scale: `mantissa × 10^-scale`. The +/// plaintext of the `numeric` family's EQL v4 types (`Numeric`, +/// `NumericEq`). +/// +/// The scale is kept: `1.50` (mantissa 150, scale 2) opens as `1.50`, not +/// `1.5`. An equality term normalises it, so `1.5` and `1.50` match under +/// `=`. There is no NaN and no infinity, and a negative zero is zero. See +/// the [module documentation](self) for what wiping covers. +/// +/// PostgreSQL's `numeric` holds values this type cannot (more than 28 +/// digits after the point, or a mantissa past 96 bits); such a value is +/// refused, never rounded. +/// +/// ``` +/// use eql_bindings::v4::plaintext::{Decimal, RangeError}; +/// +/// let price = Decimal::new(-123_450, 2)?; // -1234.50 +/// assert_eq!((price.mantissa(), price.scale()), (-123_450, 2)); +/// assert_eq!(Decimal::new(1, 29).unwrap_err(), RangeError::DecimalScale); +/// # Ok::<(), RangeError>(()) +/// ``` +#[derive(Clone)] +pub struct Decimal { + mantissa: i128, + scale: u8, +} + +impl Decimal { + /// `mantissa × 10^-scale`. + /// + /// # Errors + /// + /// [`RangeError::DecimalScale`] for a scale above 28, and + /// [`RangeError::DecimalMantissa`] for a mantissa of 2^96 or more in + /// magnitude. + pub fn new(mantissa: i128, scale: u8) -> Result { + if scale > MAX_DECIMAL_SCALE { + return Err(RangeError::DecimalScale); + } + Logical::Decimal { scale } + .admits(&Physical::Int(Int::I128(mantissa))) + .then_some(Self { mantissa, scale }) + .ok_or(RangeError::DecimalMantissa) + } + + /// The mantissa: the value is `mantissa × 10^-scale`. A copy, not + /// wiped. + pub fn mantissa(&self) -> i128 { + self.mantissa + } + + /// The number of digits after the point, at most 28. A copy, not + /// wiped. + pub fn scale(&self) -> u8 { + self.scale + } +} + +/// `Zeroize`, `ZeroizeOnDrop` and a redacted `Debug` for each type: wiping +/// sets every integer to zero. +macro_rules! wiped { + ($($ty:ident { $($field:ident),+ }),+ $(,)?) => {$( + impl Zeroize for $ty { + fn zeroize(&mut self) { + $(self.$field.zeroize();)+ + } + } + + impl Drop for $ty { + fn drop(&mut self) { + self.zeroize(); + } + } + + impl ZeroizeOnDrop for $ty {} + + impl fmt::Debug for $ty { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(concat!(stringify!($ty), "()")) + } + } + )+}; +} +wiped!( + Date { days }, + Timestamp { seconds, nanos }, + Decimal { mantissa, scale }, +); + +/// Each type describes itself to vitaminc as the logical type over an int +/// it is, the same scalar the chrono and rust_decimal types describe, so a +/// value converted from one of those seals the same leaf. The owned form +/// visits through the borrowed one and is then dropped, and wiped. +macro_rules! source { + ($($ty:ident => |$value:ident| $scalar:expr),+ $(,)?) => {$( + impl Encrypt for &$ty { + const DEPTH: Option = depth::SCALAR; + fn visit(self, sink: S) -> Result { + let $value = self; + let (logical, int) = $scalar; + sink.scalar(Some(logical), Physical::Int(int)) + } + } + + impl Encrypt for $ty { + const DEPTH: Option = depth::SCALAR; + fn visit(self, sink: S) -> Result { + (&self).visit(sink) + } + } + )+}; +} +source!( + Date => |date| (Logical::Date, Int::I32(date.days)), + Timestamp => |instant| (Logical::Timestamp(TimeUnit::Nanos), Int::I128(instant.count())), + Decimal => |decimal| (Logical::Decimal { scale: decimal.scale }, Int::I128(decimal.mantissa)), +); + +/// A date reads exactly a date scalar, at any integer width, in range. +impl FromScalar for Date { + const EXPECTED: &'static str = "date"; + fn from_scalar(ty: Option, value: Physical<'_>) -> Option { + match (ty, value) { + (Some(Logical::Date), Physical::Int(days)) => { + Date::from_days_since_epoch(i32::try_from(days.to_i128()?).ok()?).ok() + } + _ => None, + } + } +} + +/// A timestamp reads a timestamp scalar in any unit (a JavaScript `Date` +/// is in milliseconds, a Rust or Go instant in nanoseconds), in range. +impl FromScalar for Timestamp { + const EXPECTED: &'static str = "timestamp"; + fn from_scalar(ty: Option, value: Physical<'_>) -> Option { + match (ty, value) { + (Some(Logical::Timestamp(unit)), Physical::Int(count)) => { + Timestamp::from_nanos(timestamp_nanos(unit, count)?) + } + _ => None, + } + } +} + +/// A decimal reads a decimal scalar, keeping its scale, in range. +impl FromScalar for Decimal { + const EXPECTED: &'static str = "decimal"; + fn from_scalar(ty: Option, value: Physical<'_>) -> Option { + match (ty, value) { + (Some(Logical::Decimal { scale }), Physical::Int(mantissa)) => { + Decimal::new(mantissa.to_i128()?, scale).ok() + } + _ => None, + } + } +} + +/// Conversions to and from chrono's `NaiveDate` and `DateTime`. The +/// ranges are the same (vitaminc pins them to chrono's), so only a leap +/// second, which a `DateTime` can hold and a [`Timestamp`] cannot, is +/// refused. The chrono value is outside this crate's care: it is not wiped. +#[cfg(feature = "chrono")] +mod chrono_conversions { + use super::{Date, RangeError, Timestamp}; + use chrono::{DateTime, NaiveDate, Utc}; + use vitaminc_aead::plaintext::kinds::chrono as conv; + use vitaminc_aead::{Int, TimeUnit}; + + impl From for Date { + /// Every `NaiveDate` is in range: chrono's range is the one a + /// `Date` checks. + fn from(date: NaiveDate) -> Self { + Date { + days: conv::date_days(date), + } + } + } + + impl From<&Date> for NaiveDate { + fn from(date: &Date) -> Self { + // A `Date` is only built in chrono's range, so the conversion + // always succeeds. If the two ranges ever drifted apart, this + // fails loudly rather than returning a wrong date. + conv::date(Int::I32(date.days)).expect("a Date is built in chrono's NaiveDate range") + } + } + + impl TryFrom> for Timestamp { + type Error = RangeError; + + /// # Errors + /// + /// [`RangeError::LeapSecond`] for a leap second, which chrono holds + /// as nanoseconds of one second or more. + fn try_from(instant: DateTime) -> Result { + Timestamp::new(instant.timestamp(), instant.timestamp_subsec_nanos()) + } + } + + impl From<&Timestamp> for DateTime { + fn from(instant: &Timestamp) -> Self { + // A `Timestamp` is only built in chrono's range. If the two + // ranges ever drifted apart, this fails loudly rather than + // returning a wrong instant. + conv::timestamp(TimeUnit::Nanos, Int::I128(instant.count())) + .expect("a Timestamp is built in chrono's DateTime range") + } + } +} + +/// Conversions to and from rust_decimal's `Decimal`, whose range (scale at +/// most 28, a 96-bit mantissa) is the one a [`Decimal`] checks, so neither +/// direction can fail. A rust_decimal negative zero converts to zero at its +/// scale. The rust_decimal value is outside this crate's care: it is not +/// wiped. +#[cfg(feature = "rust_decimal")] +mod rust_decimal_conversions { + use super::Decimal; + use vitaminc_aead::plaintext::kinds::decimal as conv; + use vitaminc_aead::Int; + + impl From for Decimal { + fn from(decimal: rust_decimal::Decimal) -> Self { + let (scale, mantissa) = conv::parts(decimal); + Decimal { mantissa, scale } + } + } + + impl From<&Decimal> for rust_decimal::Decimal { + fn from(decimal: &Decimal) -> Self { + // A `Decimal` is only built in rust_decimal's range. If the two + // ranges ever drifted apart, this fails loudly rather than + // returning a wrong number. + conv::decimal(decimal.scale, Int::I128(decimal.mantissa)) + .expect("a Decimal is built in rust_decimal's range") + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use stack_encrypt::Decrypt; + use vitaminc_aead::plaintext::kinds::{ + MAX_DATE_DAYS, MAX_TIMESTAMP_NANOS, MIN_DATE_DAYS, MIN_TIMESTAMP_NANOS, + }; + + /// The scalar a value describes itself as. + struct Scalar; + impl Sink for Scalar { + type Ok = (Option, Option); + type Error = vitaminc_aead::DecodeError; + type Seq = vitaminc_aead::typed::Never; + type Map = vitaminc_aead::typed::Never; + fn scalar(self, ty: Option, v: Physical<'_>) -> Result { + Ok(( + ty, + match v { + Physical::Int(i) => i.to_i128(), + _ => None, + }, + )) + } + fn seq(self, _: Option) -> Result { + unreachable!("a plaintext here is a scalar") + } + fn map(self, _: Option) -> Result { + unreachable!("a plaintext here is a scalar") + } + fn passthrough(self, _: P) -> Result { + unreachable!("a plaintext here is a scalar") + } + } + + fn scalar(value: impl Encrypt) -> (Option, Option) { + value.visit(Scalar).unwrap() + } + + /// What the type's own decrypt sink builds from one scalar: the path + /// an opened leaf takes. + fn read(ty: Logical, int: Int) -> Option { + T::sink().scalar(Some(ty), Physical::Int(int)).ok() + } + + #[test] + fn a_date_is_its_day_count_and_refuses_one_outside_the_range() { + for days in [0, -1, 19_782, MIN_DATE_DAYS, MAX_DATE_DAYS] { + let date = Date::from_days_since_epoch(days).unwrap(); + assert_eq!(date.days_since_epoch(), days); + assert_eq!(scalar(&date), (Some(Logical::Date), Some(days.into()))); + let back: Date = read(Logical::Date, Int::I32(days)).unwrap(); + assert_eq!(back.days_since_epoch(), days); + } + for days in [MIN_DATE_DAYS - 1, MAX_DATE_DAYS + 1, i32::MIN, i32::MAX] { + assert_eq!( + Date::from_days_since_epoch(days).unwrap_err(), + RangeError::Date + ); + assert!(read::(Logical::Date, Int::I32(days)).is_none()); + } + // Another width reads too; another logical type does not. + let wide: Date = read(Logical::Date, Int::I64(-3)).unwrap(); + assert_eq!(wide.days_since_epoch(), -3); + assert!(read::(Logical::Date, Int::I64(i64::MAX)).is_none()); + assert!(read::(Logical::Timestamp(TimeUnit::Nanos), Int::I32(0)).is_none()); + assert!(::from_scalar(None, Physical::Int(Int::I32(0))).is_none()); + } + + #[test] + fn a_timestamp_is_its_nanosecond_count_and_refuses_a_leap_second() { + for (seconds, nanos) in [ + (0, 0), + (1_700_000_000, 123_456_789), + (-1, 999_999_999), + (-1, 500_000_000), + ] { + let instant = Timestamp::new(seconds, nanos).unwrap(); + assert_eq!((instant.seconds(), instant.nanoseconds()), (seconds, nanos)); + let count = i128::from(seconds) * 1_000_000_000 + i128::from(nanos); + assert_eq!( + scalar(&instant), + (Some(Logical::Timestamp(TimeUnit::Nanos)), Some(count)) + ); + let back: Timestamp = + read(Logical::Timestamp(TimeUnit::Nanos), Int::I128(count)).unwrap(); + assert_eq!((back.seconds(), back.nanoseconds()), (seconds, nanos)); + } + // Any unit reads: a millisecond count is a JavaScript `Date`'s. + let millis: Timestamp = + read(Logical::Timestamp(TimeUnit::Millis), Int::I64(-1_500)).unwrap(); + assert_eq!((millis.seconds(), millis.nanoseconds()), (-2, 500_000_000)); + + assert_eq!( + Timestamp::new(0, 1_000_000_000).unwrap_err(), + RangeError::LeapSecond + ); + assert_eq!( + Timestamp::new(0, u32::MAX).unwrap_err(), + RangeError::LeapSecond + ); + // The range's two edges, and one nanosecond past each. + let edge = |nanos: i128| { + let n = i128::from(NANOS_PER_SECOND); + Timestamp::new( + i64::try_from(nanos.div_euclid(n)).unwrap(), + u32::try_from(nanos.rem_euclid(n)).unwrap(), + ) + }; + assert!(edge(MIN_TIMESTAMP_NANOS).is_ok()); + assert!(edge(MAX_TIMESTAMP_NANOS).is_ok()); + assert_eq!( + edge(MIN_TIMESTAMP_NANOS - 1).unwrap_err(), + RangeError::Timestamp + ); + assert_eq!( + edge(MAX_TIMESTAMP_NANOS + 1).unwrap_err(), + RangeError::Timestamp + ); + for seconds in [i64::MIN, i64::MAX] { + assert_eq!( + Timestamp::new(seconds, 0).unwrap_err(), + RangeError::Timestamp + ); + } + assert!(read::( + Logical::Timestamp(TimeUnit::Nanos), + Int::I128(MAX_TIMESTAMP_NANOS + 1) + ) + .is_none()); + assert!(read::(Logical::Date, Int::I128(0)).is_none()); + } + + #[test] + fn a_decimal_keeps_its_scale_and_refuses_what_rust_decimal_cannot_hold() { + let limit = 1i128 << 96; + for (mantissa, scale) in [(150, 2), (-1, 0), (0, 28), (limit - 1, 28), (1 - limit, 0)] { + let decimal = Decimal::new(mantissa, scale).unwrap(); + assert_eq!((decimal.mantissa(), decimal.scale()), (mantissa, scale)); + assert_eq!( + scalar(&decimal), + (Some(Logical::Decimal { scale }), Some(mantissa)) + ); + let back: Decimal = read(Logical::Decimal { scale }, Int::I128(mantissa)).unwrap(); + assert_eq!((back.mantissa(), back.scale()), (mantissa, scale)); + } + assert_eq!(Decimal::new(1, 29).unwrap_err(), RangeError::DecimalScale); + assert_eq!( + Decimal::new(1, u8::MAX).unwrap_err(), + RangeError::DecimalScale + ); + for mantissa in [limit, -limit, i128::MAX, i128::MIN] { + assert_eq!( + Decimal::new(mantissa, 0).unwrap_err(), + RangeError::DecimalMantissa + ); + assert!(read::(Logical::Decimal { scale: 0 }, Int::I128(mantissa)).is_none()); + } + assert!(read::(Logical::Date, Int::I128(1)).is_none()); + } + + #[test] + fn an_owned_value_seals_the_scalar_its_borrow_does() { + let date = Date::from_days_since_epoch(5).unwrap(); + assert_eq!(scalar(date.clone()), scalar(&date)); + let instant = Timestamp::new(5, 6).unwrap(); + assert_eq!(scalar(instant.clone()), scalar(&instant)); + let decimal = Decimal::new(5, 6).unwrap(); + assert_eq!(scalar(decimal.clone()), scalar(&decimal)); + } + + #[test] + fn zeroize_clears_every_field() { + let mut date = Date::from_days_since_epoch(-42).unwrap(); + date.zeroize(); + assert_eq!(date.days_since_epoch(), 0); + let mut instant = Timestamp::new(-7, 999).unwrap(); + instant.zeroize(); + assert_eq!((instant.seconds(), instant.nanoseconds()), (0, 0)); + let mut decimal = Decimal::new(-123, 4).unwrap(); + decimal.zeroize(); + assert_eq!((decimal.mantissa(), decimal.scale()), (0, 0)); + } + + /// Dropping wipes the value where it lies. Read through a + /// `ManuallyDrop` so the memory is still the test's after the drop + /// glue has run: `drop_in_place` runs `Drop` without freeing. + #[test] + fn dropping_wipes_the_value_in_place() { + use std::mem::ManuallyDrop; + let mut date = ManuallyDrop::new(Date::from_days_since_epoch(-42).unwrap()); + let mut instant = ManuallyDrop::new(Timestamp::new(-7, 999).unwrap()); + let mut decimal = ManuallyDrop::new(Decimal::new(-123, 4).unwrap()); + // SAFETY: each value is dropped once and only its integers, which + // need no drop of their own, are read afterwards. + unsafe { + ManuallyDrop::drop(&mut date); + ManuallyDrop::drop(&mut instant); + ManuallyDrop::drop(&mut decimal); + } + assert_eq!(date.days, 0); + assert_eq!((instant.seconds, instant.nanos), (0, 0)); + assert_eq!((decimal.mantissa, decimal.scale), (0, 0)); + } + + #[test] + fn debug_and_errors_never_show_the_value() { + let date = Date::from_days_since_epoch(19_782).unwrap(); + let instant = Timestamp::new(1_700_000_000, 0).unwrap(); + let decimal = Decimal::new(123_450, 2).unwrap(); + for shown in [ + format!("{date:?}"), + format!("{instant:?}"), + format!("{decimal:?}"), + ] { + assert!(shown.ends_with("()"), "{shown}"); + assert!(!shown.chars().any(|c| c.is_ascii_digit()), "{shown}"); + } + for error in [ + Date::from_days_since_epoch(i32::MAX).unwrap_err(), + Timestamp::new(i64::MAX, 0).unwrap_err(), + Timestamp::new(0, 1_234_567_890).unwrap_err(), + Decimal::new(i128::MAX, 0).unwrap_err(), + Decimal::new(1, 200).unwrap_err(), + ] { + let message = error.to_string(); + for value in ["2147483647", "9223372036854775807", "1234567890", "200"] { + assert!(!message.contains(value), "{message}"); + } + } + } + + #[cfg(feature = "chrono")] + #[test] + fn chrono_values_convert_both_ways() { + use chrono::{DateTime, NaiveDate, Utc}; + + for date in [ + NaiveDate::from_ymd_opt(2024, 2, 29).unwrap(), + NaiveDate::from_ymd_opt(1969, 12, 31).unwrap(), + NaiveDate::from_ymd_opt(1, 1, 1).unwrap(), + NaiveDate::MIN, + NaiveDate::MAX, + ] { + let ours = Date::from(date); + assert_eq!(NaiveDate::from(&ours), date); + // The same scalar chrono's own source describes. + assert_eq!(scalar(&ours), scalar(date)); + } + assert_eq!( + Date::from(NaiveDate::from_ymd_opt(1970, 1, 1).unwrap()).days_since_epoch(), + 0 + ); + // The ends of the range a `Date` checks are chrono's ends: if the + // two drift apart, the `expect` in the conversion fires here. + for (days, date) in [ + (MIN_DATE_DAYS, NaiveDate::MIN), + (MAX_DATE_DAYS, NaiveDate::MAX), + ] { + let ours = Date::from_days_since_epoch(days).unwrap(); + assert_eq!(NaiveDate::from(&ours), date); + } + + for instant in [ + DateTime::from_timestamp(1_700_000_000, 123_456_789).unwrap(), + DateTime::from_timestamp(-1, 1).unwrap(), + chrono::NaiveDateTime::MIN.and_utc(), + chrono::NaiveDateTime::MAX.and_utc(), + ] { + let ours = Timestamp::try_from(instant).unwrap(); + assert_eq!(DateTime::::from(&ours), instant); + assert_eq!(scalar(&ours), scalar(instant)); + } + // The same for a `Timestamp`'s range. + for (nanos, instant) in [ + (MIN_TIMESTAMP_NANOS, chrono::NaiveDateTime::MIN.and_utc()), + (MAX_TIMESTAMP_NANOS, chrono::NaiveDateTime::MAX.and_utc()), + ] { + let ours = + read::(Logical::Timestamp(TimeUnit::Nanos), Int::I128(nanos)).unwrap(); + assert_eq!(DateTime::::from(&ours), instant); + } + let leap = NaiveDate::from_ymd_opt(2016, 12, 31) + .and_then(|d| d.and_hms_nano_opt(23, 59, 59, 1_500_000_000)) + .unwrap() + .and_utc(); + assert_eq!( + Timestamp::try_from(leap).unwrap_err(), + RangeError::LeapSecond + ); + } + + #[cfg(feature = "rust_decimal")] + #[test] + fn rust_decimal_values_convert_both_ways() { + use std::str::FromStr; + + for text in [ + "1.50", + "-1234.5678", + "0", + "79228162514264337593543950335", + "-0.0000000000000000000000000001", + ] { + let theirs = rust_decimal::Decimal::from_str(text).unwrap(); + let ours = Decimal::from(theirs); + let back = rust_decimal::Decimal::from(&ours); + assert_eq!(back.to_string(), theirs.to_string(), "{text}"); + assert_eq!(scalar(&ours), scalar(theirs), "{text}"); + } + // The ends of rust_decimal's range, at the smallest and the largest + // scale: if the ranges drift apart, the `expect` in the conversion + // fires here. + let limit = 79_228_162_514_264_337_593_543_950_335_i128; + for theirs in [rust_decimal::Decimal::MIN, rust_decimal::Decimal::MAX] { + let ours = Decimal::from(theirs); + assert_eq!(rust_decimal::Decimal::from(&ours), theirs); + } + for mantissa in [limit, -limit] { + let ours = Decimal::new(mantissa, MAX_DECIMAL_SCALE).unwrap(); + let back = rust_decimal::Decimal::from(&ours); + assert_eq!((back.mantissa(), back.scale()), (mantissa, 28)); + } + let ours = Decimal::from(rust_decimal::Decimal::from_str("1.50").unwrap()); + assert_eq!((ours.mantissa(), ours.scale()), (150, 2)); + // A negative zero is zero, at its scale. + let mut negative_zero = rust_decimal::Decimal::new(0, 2); + negative_zero.set_sign_negative(true); + let ours = Decimal::from(negative_zero); + assert_eq!((ours.mantissa(), ours.scale()), (0, 2)); + assert_eq!(rust_decimal::Decimal::from(&ours).to_string(), "0.00"); + } +} diff --git a/packages/eql/crates/eql-bindings/src/v4/targets.rs b/packages/eql/crates/eql-bindings/src/v4/targets.rs index 8f352b7a7..67d8e09e0 100644 --- a/packages/eql/crates/eql-bindings/src/v4/targets.rs +++ b/packages/eql/crates/eql-bindings/src/v4/targets.rs @@ -3,6 +3,9 @@ use crate::encryption::targets::{ open_target, refuse, refuse_query, run_target, Opener, Target, TargetError, }; +use crate::v4::plaintext::Date; +use crate::v4::plaintext::Decimal; +use crate::v4::plaintext::Timestamp; use crate::Identifier; use stack_encrypt::dynamic::ValueFrame; use stack_encrypt::{KeysetCipher, NonEmpty, Pending}; @@ -201,210 +204,198 @@ pub const TARGETS: &[Target] = &[ name: "Date", family: "date", suffix: "", - plaintext: None, + plaintext: Some("date"), sql_domain: "public.eql_v4_date", indexes: &[], query: None, query_sql_domain: None, - producible: false, - reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", - ), + producible: true, + reason: None, }, Target { name: "DateEq", family: "date", suffix: "Eq", - plaintext: None, + plaintext: Some("date"), sql_domain: "public.eql_v4_date_eq", indexes: &["eq"], query: Some("DateEqQuery"), query_sql_domain: Some("eql_v4.query_date_eq"), - producible: false, - reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", - ), + producible: true, + reason: None, }, Target { name: "DateOrdOre", family: "date", suffix: "OrdOre", - plaintext: None, + plaintext: Some("date"), sql_domain: "public.eql_v4_date_ord_ore", indexes: &["ore"], query: Some("DateOrdOreQuery"), query_sql_domain: Some("eql_v4.query_date_ord_ore"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "DateOrd", family: "date", suffix: "Ord", - plaintext: None, + plaintext: Some("date"), sql_domain: "public.eql_v4_date_ord", indexes: &["ope"], query: Some("DateOrdQuery"), query_sql_domain: Some("eql_v4.query_date_ord"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "DateOrdOpe", family: "date", suffix: "OrdOpe", - plaintext: None, + plaintext: Some("date"), sql_domain: "public.eql_v4_date_ord_ope", indexes: &["ope"], query: Some("DateOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_date_ord_ope"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "Timestamp", family: "timestamp", suffix: "", - plaintext: None, + plaintext: Some("timestamp"), sql_domain: "public.eql_v4_timestamp", indexes: &[], query: None, query_sql_domain: None, - producible: false, - reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", - ), + producible: true, + reason: None, }, Target { name: "TimestampEq", family: "timestamp", suffix: "Eq", - plaintext: None, + plaintext: Some("timestamp"), sql_domain: "public.eql_v4_timestamp_eq", indexes: &["eq"], query: Some("TimestampEqQuery"), query_sql_domain: Some("eql_v4.query_timestamp_eq"), - producible: false, - reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", - ), + producible: true, + reason: None, }, Target { name: "TimestampOrdOre", family: "timestamp", suffix: "OrdOre", - plaintext: None, + plaintext: Some("timestamp"), sql_domain: "public.eql_v4_timestamp_ord_ore", indexes: &["ore"], query: Some("TimestampOrdOreQuery"), query_sql_domain: Some("eql_v4.query_timestamp_ord_ore"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "TimestampOrd", family: "timestamp", suffix: "Ord", - plaintext: None, + plaintext: Some("timestamp"), sql_domain: "public.eql_v4_timestamp_ord", indexes: &["ope"], query: Some("TimestampOrdQuery"), query_sql_domain: Some("eql_v4.query_timestamp_ord"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "TimestampOrdOpe", family: "timestamp", suffix: "OrdOpe", - plaintext: None, + plaintext: Some("timestamp"), sql_domain: "public.eql_v4_timestamp_ord_ope", indexes: &["ope"], query: Some("TimestampOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_timestamp_ord_ope"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "Numeric", family: "numeric", suffix: "", - plaintext: None, + plaintext: Some("decimal"), sql_domain: "public.eql_v4_numeric", indexes: &[], query: None, query_sql_domain: None, - producible: false, - reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", - ), + producible: true, + reason: None, }, Target { name: "NumericEq", family: "numeric", suffix: "Eq", - plaintext: None, + plaintext: Some("decimal"), sql_domain: "public.eql_v4_numeric_eq", indexes: &["eq"], query: Some("NumericEqQuery"), query_sql_domain: Some("eql_v4.query_numeric_eq"), - producible: false, - reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", - ), + producible: true, + reason: None, }, Target { name: "NumericOrdOre", family: "numeric", suffix: "OrdOre", - plaintext: None, + plaintext: Some("decimal"), sql_domain: "public.eql_v4_numeric_ord_ore", indexes: &["ore"], query: Some("NumericOrdOreQuery"), query_sql_domain: Some("eql_v4.query_numeric_ord_ore"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "NumericOrd", family: "numeric", suffix: "Ord", - plaintext: None, + plaintext: Some("decimal"), sql_domain: "public.eql_v4_numeric_ord", indexes: &["ope"], query: Some("NumericOrdQuery"), query_sql_domain: Some("eql_v4.query_numeric_ord"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { name: "NumericOrdOpe", family: "numeric", suffix: "OrdOpe", - plaintext: None, + plaintext: Some("decimal"), sql_domain: "public.eql_v4_numeric_ord_ope", indexes: &["ope"], query: Some("NumericOrdOpeQuery"), query_sql_domain: Some("eql_v4.query_numeric_ord_ope"), producible: false, reason: Some( - "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK carry the kind, and which Rust type a date, timestamp or decimal is produced from is not decided (#1130)", + "the engine derives no order term for a date, timestamp or decimal yet: how they are ordered is not decided (#1236)", ), }, Target { @@ -736,6 +727,29 @@ pub(crate) fn encrypt_named<'a, K: 'static>( column, plaintext, ), + "Date" => run_target::("Date", keyset, column, plaintext), + "DateEq" => run_target::("DateEq", keyset, column, plaintext), + "Timestamp" => run_target::( + "Timestamp", + keyset, + column, + plaintext, + ), + "TimestampEq" => run_target::( + "TimestampEq", + keyset, + column, + plaintext, + ), + "Numeric" => { + run_target::("Numeric", keyset, column, plaintext) + } + "NumericEq" => run_target::( + "NumericEq", + keyset, + column, + plaintext, + ), "Text" => run_target::("Text", keyset, column, plaintext), "TextEq" => { run_target::("TextEq", keyset, column, plaintext) @@ -836,6 +850,29 @@ pub(crate) fn decrypt_named<'a, K: 'static>( column, stored, ), + "Date" => open_target::("Date", opener, column, stored), + "DateEq" => open_target::("DateEq", opener, column, stored), + "Timestamp" => open_target::( + "Timestamp", + opener, + column, + stored, + ), + "TimestampEq" => open_target::( + "TimestampEq", + opener, + column, + stored, + ), + "Numeric" => { + open_target::("Numeric", opener, column, stored) + } + "NumericEq" => open_target::( + "NumericEq", + opener, + column, + stored, + ), "Text" => open_target::("Text", opener, column, stored), "TextEq" => open_target::("TextEq", opener, column, stored), "TextMatch" => { @@ -933,6 +970,21 @@ pub(crate) fn query_named<'a, K: 'static>( column, plaintext, ), + "DateEq" => { + run_target::("DateEq", keyset, column, plaintext) + } + "TimestampEq" => run_target::( + "TimestampEq", + keyset, + column, + plaintext, + ), + "NumericEq" => run_target::( + "NumericEq", + keyset, + column, + plaintext, + ), "TextEq" => { run_target::("TextEq", keyset, column, plaintext) } diff --git a/packages/eql/crates/eql-bindings/src/v4/timestamp.rs b/packages/eql/crates/eql-bindings/src/v4/timestamp.rs index ef9545462..5099b567d 100644 --- a/packages/eql/crates/eql-bindings/src/v4/timestamp.rs +++ b/packages/eql/crates/eql-bindings/src/v4/timestamp.rs @@ -14,10 +14,18 @@ use ts_rs::TS; doc, doc = " See the [EQL timestamp reference](https://cipherstash.com/docs/reference/eql/dates-and-times) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`Timestamp`](crate::v4::plaintext::Timestamp). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = crate::v4::plaintext::Timestamp)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct Timestamp { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, } @@ -55,10 +63,18 @@ impl DomainType for Timestamp { doc, doc = " See the [EQL timestamp reference](https://cipherstash.com/docs/reference/eql/dates-and-times) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input and decrypted output: [`Timestamp`](crate::v4::plaintext::Timestamp). Implements `EncryptFrom` and `DecryptInto`. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom, stack_encrypt::DecryptInto)] +#[stash(plaintext = crate::v4::plaintext::Timestamp)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct TimestampEq { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub c: Ciphertext, pub hm: Hmac256, @@ -223,10 +239,18 @@ impl DomainType for TimestampOrdOpe { doc, doc = " See the [EQL timestamp reference](https://cipherstash.com/docs/reference/eql/dates-and-times) for SQL domain variants, operators, and query examples." )] +/// +/// Plaintext input: [`Timestamp`](crate::v4::plaintext::Timestamp). Implements `EncryptFrom` to build equality query terms. Query operands do not decrypt. +/// Encryption context: `NonEmpty`, constructed with `Identifier::for_column(table, column)` and stored in `i`. +/// See the [complete encryption example](crate::encryption#example). +#[derive(stack_encrypt::EncryptFrom)] +#[stash(plaintext = crate::v4::plaintext::Timestamp)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct TimestampEqQuery { + #[stash(default)] pub v: SchemaVersion, + #[stash(context_field)] pub i: Identifier, pub hm: Hmac256, } diff --git a/packages/eql/crates/eql-codegen/src/bindings.rs b/packages/eql/crates/eql-codegen/src/bindings.rs index a385acedd..198093c7a 100644 --- a/packages/eql/crates/eql-codegen/src/bindings.rs +++ b/packages/eql/crates/eql-codegen/src/bindings.rs @@ -336,8 +336,9 @@ fn render_query_struct(eql: EqlVersion, family: &DomainFamily, domain: &Domain) /// The `(family, domain)` pairs whose generated v4 structs carry the opt-in /// `stack-encrypt` derives: the storage and equality domains of every family /// whose plaintext eql-bindings names (`plaintext_of`), the OPE-ordered -/// domains (`ord`, `ord_ope`) of the families listed, and text match. Every -/// other stored domain has a reason in [`encryption_gap`], and a test holds the two together so a new domain +/// domains (`ord`, `ord_ope`) of those whose order terms the engine derives, +/// and text match. Every other stored domain has a reason in +/// [`encryption_gap`], and a test holds the two together so a new domain /// cannot arrive without either a derive or a reason. pub const ENCRYPTION_DOMAINS: &[(&str, &str)] = &[ ("integer", ""), @@ -352,6 +353,12 @@ pub const ENCRYPTION_DOMAINS: &[(&str, &str)] = &[ ("bigint", "eq"), ("bigint", "ord"), ("bigint", "ord_ope"), + ("date", ""), + ("date", "eq"), + ("timestamp", ""), + ("timestamp", "eq"), + ("numeric", ""), + ("numeric", "eq"), ("text", ""), ("text", "eq"), ("text", "match"), @@ -368,6 +375,11 @@ pub const ENCRYPTION_DOMAINS: &[(&str, &str)] = &[ ("double", "ord_ope"), ]; +/// The families whose plaintext is a vitaminc logical type (a date, a +/// timestamp, a decimal) rather than a physical one. Their equality terms +/// are derived; their order terms are not (`encryption_gap`). +const ORDER_PENDING_FAMILIES: &[&str] = &["date", "timestamp", "numeric"]; + /// Why no `eql_v3` struct derives Stack Encrypt's traits. pub const V3_ENCRYPTION_GAP: &str = "Stack Encrypt produces EQL v4 only: its output is not byte-for-byte compatible with \ @@ -376,7 +388,9 @@ pub const V3_ENCRYPTION_GAP: &str = /// The plaintext a family's v4 domains are produced from: the vitaminc /// `ValueKind` name a plan field's `"type"` key spells, and the Rust type the /// derive and the by-name dispatch name. `None` for a family eql-bindings -/// names no Rust plaintext type for yet. +/// names no Rust plaintext type for yet. A date, a timestamp and a decimal +/// are eql-bindings' own types (`v4::plaintext`), which can be wiped where +/// chrono's and rust_decimal's cannot. pub fn plaintext_of(family: &str) -> Option<(&'static str, &'static str)> { Some(match family { "text" => ("string", "String"), @@ -386,6 +400,9 @@ pub fn plaintext_of(family: &str) -> Option<(&'static str, &'static str)> { "boolean" => ("bool", "bool"), "real" => ("float32", "f32"), "double" => ("float64", "f64"), + "date" => ("date", "crate::v4::plaintext::Date"), + "timestamp" => ("timestamp", "crate::v4::plaintext::Timestamp"), + "numeric" => ("decimal", "crate::v4::plaintext::Decimal"), _ => return None, }) } @@ -412,8 +429,15 @@ pub fn encryption_gap( becomes one sealed leaf is not decided" } else if plaintext_of(family.name).is_none() { "eql-bindings has no Rust plaintext type for this family yet: the engine and the Go SDK \ - carry the kind, and which Rust type a date, timestamp or decimal is produced from is \ - not decided (#1130)" + carry the kind" + } else if ORDER_PENDING_FAMILIES.contains(&family.name) + && (domain.terms.contains(&Term::Ope) || domain.terms.contains(&Term::Ore)) + { + // Before the ORE branch: the engine derives no order term of either + // scheme for these kinds, so block ORE is not the first thing a + // `*OrdOre` type of these families waits for. + "the engine derives no order term for a date, timestamp or decimal yet: how they are \ + ordered is not decided (#1236)" } else if domain.terms.contains(&Term::Ore) { "EQL stores block ORE terms and the engine derives CLLW ORE (#1132)" } else if domain.terms.contains(&Term::Bloom) && domain.terms.contains(&Term::Ope) { @@ -1742,6 +1766,8 @@ mod tests { "tree of leaves" } else if plaintext_of(family.name).is_none() { "no Rust plaintext type for this family" + } else if ORDER_PENDING_FAMILIES.contains(&family.name) { + "no order term for a date, timestamp or decimal" } else if domain.terms.contains(&Term::Ore) { "block ORE terms" } else if domain.terms.contains(&Term::Bloom) diff --git a/packages/eql/crates/eql-codegen/src/targets.rs b/packages/eql/crates/eql-codegen/src/targets.rs index 73b6b3f4a..c8fe66c67 100644 --- a/packages/eql/crates/eql-codegen/src/targets.rs +++ b/packages/eql/crates/eql-codegen/src/targets.rs @@ -18,9 +18,10 @@ //! the derive, and [`target_gap`] is tested to agree with //! [`encryption_gap`](crate::bindings::encryption_gap) on every domain. //! -//! The reasons a type is not producible, each naming the issue that tracks -//! it: no Rust plaintext type is chosen for the family yet (#1130); EQL's -//! ORE term is block ORE where the engine derives CLLW ORE (#1132); a +//! The reasons a type is not producible: eql-bindings names no Rust +//! plaintext type for the family; the engine derives no order term for a +//! date, timestamp or decimal; EQL's ORE term is block ORE where the engine +//! derives CLLW ORE (#1132); a //! `TextSearch` query carries a match term that short text cannot make //! (#1131); and the JSON index is a new engine operation (#1060). @@ -424,13 +425,19 @@ mod tests { "{family_name}" ); } + // A date, a timestamp and a decimal are stored and compared for + // equality; every ordered domain of theirs waits for an order term, + // block ORE included, which waits for that first. for family_name in ["date", "timestamp", "numeric"] { - for domain in ["", "eq", "ord"] { + let family = family(family_name); + assert!(gap(family, "").is_none(), "{family_name}"); + assert!(gap(family, "eq").is_none(), "{family_name}Eq is producible"); + for ordered in ["ord", "ord_ope", "ord_ore"] { assert!( - gap(family(family_name), domain) + gap(family, ordered) .unwrap() - .contains("no Rust plaintext type for this family"), - "{family_name} {domain:?}" + .contains("no order term for a date, timestamp or decimal"), + "{family_name} {ordered}" ); } } diff --git a/packages/eql/mise.toml b/packages/eql/mise.toml index 4af6bdf8e..89e4a7557 100644 --- a/packages/eql/mise.toml +++ b/packages/eql/mise.toml @@ -340,7 +340,7 @@ psql "$EQL_TEST_DATABASE_URL" -v ON_ERROR_STOP=1 -f release/cipherstash-encrypt- # `--ignored` selects by attribute, and cargo exits 0 on an empty selection, so # a test that loses its `#[ignore]` or is renamed would leave this task passing # on nothing. List what the filter selects before running it. -for suite in text_eq v4_types v4_order; do +for suite in text_eq v4_types v4_order v4_date_time_numeric; do listing=$(cargo test --locked -p eql-encryption-tests --test "$suite" -- --ignored --list | sed -n 's/: test$//p') [ -n "$listing" ] || { echo "error: '--ignored' selects no test in $suite; the PostgreSQL test has gone missing." >&2; exit 1; } cargo test --locked -p eql-encryption-tests --test "$suite" -- --ignored diff --git a/packages/eql/tests/encryption/Cargo.toml b/packages/eql/tests/encryption/Cargo.toml index 443a1fd4c..d945e5fcf 100644 --- a/packages/eql/tests/encryption/Cargo.toml +++ b/packages/eql/tests/encryption/Cargo.toml @@ -13,15 +13,21 @@ harness = false # Keep crypto/Postgres test dependencies out of eql-bindings' default test build. [dev-dependencies] -eql-bindings = { path = "../../crates/eql-bindings", features = ["targets"] } +# `chrono` and `rust_decimal`: the date, timestamp and decimal plaintexts +# convert from those crates' values, which build the test values here. +eql-bindings = { path = "../../crates/eql-bindings", features = ["targets", "chrono", "rust_decimal"] } # The parity oracle for the generated target table (tests/targets.rs): the # table must name every stored catalog domain, in catalog order. eql-domains = { path = "../../crates/eql-domains" } # `dynamic`: `ValueFrame`, the runtime plaintext the target dispatch takes, # as eql-bindings names it. stack-encrypt = { path = "../../../stack-encrypt", default-features = false, features = ["dynamic"] } -# `plaintext::Sink`, for a plaintext type with its own `Encrypt` impl. -vitaminc-aead = "0.6.0" +# `plaintext::Sink`, for a plaintext type with its own `Encrypt` impl; and +# chrono's and rust_decimal's own sources, which seal the leaf a date, +# timestamp or decimal from any other producer seals. +vitaminc-aead = { version = "0.6.0", features = ["chrono", "rust_decimal"] } +chrono = { version = "0.4.44", default-features = false, features = ["std"] } +rust_decimal = { version = "1", default-features = false, features = ["std"] } stack-kms = { path = "../../../stack-kms", default-features = false, features = ["test-support"] } serde = "1" serde_json = "1" diff --git a/packages/eql/tests/encryption/fixtures/README.md b/packages/eql/tests/encryption/fixtures/README.md index 4a7969a45..7e1f183b6 100644 --- a/packages/eql/tests/encryption/fixtures/README.md +++ b/packages/eql/tests/encryption/fixtures/README.md @@ -6,6 +6,8 @@ | `v4_queries.json` | the by-name dispatch in `tests/v4_types.rs` | yes, with `EQL_UPDATE_FIXTURES=1` | | `v4_ordered_queries.json` | the by-name dispatch in `tests/v4_order.rs` | yes, with `EQL_UPDATE_FIXTURES=1` | | `v4_ordered_rows.json` | the Go SDK's `TestV4OrderedRowsMatchTheCommittedFixture` | yes, with `EQL_UPDATE_FIXTURES=1` | +| `v4_ledger_queries.json` | the by-name dispatch in `tests/v4_date_time_numeric.rs` | yes, with `EQL_UPDATE_FIXTURES=1` | +| `v4_ledger_rows.json` | the Go SDK's `TestV4LedgerRowsMatchTheCommittedFixture` | yes, with `EQL_UPDATE_FIXTURES=1` | | `text_eq_query.json`, `ste_vec_selectors.json` | `tests/text_eq.rs`, `tests/ste_vec_selectors.rs` | yes, see each test | | `v4_smallint_rows.json` | the Go SDK's `TestV4SmallintRowsMatchTheCommittedFixture` | yes, with `EQL_UPDATE_FIXTURES=1` | | `stored_v4_rows_v1.json` | copied from `v4_rows.json` at the next build | **never: frozen** | @@ -21,8 +23,10 @@ bytes are final. `Profile` rows the Go SDK's generated code stored with the `next` build, one value of every EQL v4 type that build produced. Types made producible since -are sealed into their own regenerable row fixtures (see the table above); a -later frozen file covers them once it is cut. `v4_rows.json` proves +are sealed into their own regenerable row fixtures (see the table above), +the date, timestamp and numeric types by their own Go model (`Ledger`) into +`v4_ledger_rows.json`; a later frozen file covers them once it is cut. +`v4_rows.json` proves the Go SDK and Rust agree *today* and is regenerated when the encoding moves. This file is the opposite: it is never regenerated or edited, so a change that breaks rows already stored fails instead of being regenerated away. diff --git a/packages/eql/tests/encryption/fixtures/v4_ledger_queries.json b/packages/eql/tests/encryption/fixtures/v4_ledger_queries.json new file mode 100644 index 000000000..cf59509ae --- /dev/null +++ b/packages/eql/tests/encryption/fixtures/v4_ledger_queries.json @@ -0,0 +1,16 @@ +{ + "_comment": "The DateEq, TimestampEq and NumericEq query operands eql-bindings derives for each plaintext under FakeDataKeySource's index key (keyset nil), read by the Go SDK's hermetic test; regenerate with EQL_UPDATE_FIXTURES=1, do not edit.", + "amount": { + "plaintext": "1.50", + "query": "{\"v\":4,\"i\":{\"t\":\"profiles\",\"c\":\"amount\"},\"hm\":\"ff787d4a33e41d24d30fb95098c57bdffeef072fb9e0907a2dd7fa621a5eda03\"}" + }, + "due": { + "plaintext": "2024-02-29", + "query": "{\"v\":4,\"i\":{\"t\":\"profiles\",\"c\":\"due\"},\"hm\":\"decc03613957c47004860bc9bc70770aed4c6ba3f0da2aaea3da9fe7367af6a9\"}" + }, + "posted": { + "plaintext": "2024-02-29T12:34:56.123456Z", + "query": "{\"v\":4,\"i\":{\"t\":\"profiles\",\"c\":\"posted\"},\"hm\":\"b3a2b5c8d5ea562f472a55eb57f3b49aec3d6cc3731dad16c4f8999394a95c55\"}" + }, + "table": "profiles" +} diff --git a/packages/eql/tests/encryption/fixtures/v4_ledger_rows.json b/packages/eql/tests/encryption/fixtures/v4_ledger_rows.json new file mode 100644 index 000000000..63777e6f4 --- /dev/null +++ b/packages/eql/tests/encryption/fixtures/v4_ledger_rows.json @@ -0,0 +1,199 @@ +{ + "_comment": "Ledger rows sealed by the Go SDK's generated code over the deterministic EQL guest with the seed below; the Rust suite opens and queries them. Regenerate with EQL_UPDATE_FIXTURES=1 go test ./encrypt -run TestV4LedgerRows, do not edit.", + "seed": "encrypt round-trip tests seed v1", + "table": "profiles", + "rows": [ + { + "id": 1, + "plaintext": { + "opened": "-262143-01-01", + "due": "2024-02-29", + "created": "1969-12-31T23:59:59.999999999Z", + "posted": "2024-02-29T12:34:56.123456Z", + "total": "-79228162514264337593543950335", + "amount": "1.50" + }, + "stored": { + "amount": { + "v": 4, + "i": { + "t": "profiles", + "c": "amount" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABkTJd2+gSGUCXq5Bog0ODUIACQ+t9l6f3shvibRWnk1Ihw7dEY6c0vRmYYIIFt9OMzRAIL8M8TSHbtBLpRD8evgfwfWeQncgvb4wFTmJ2Dl2n60SGuhBcL2PnYmDbY/GM=", + "hm": "ff787d4a33e41d24d30fb95098c57bdffeef072fb9e0907a2dd7fa621a5eda03" + }, + "created": { + "v": 4, + "i": { + "t": "profiles", + "c": "created" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABGuTbf5RPT5TP4jkMDscaIIADvvVQ1X3uY1KcA+8e4xNhSomXJsaXQbJQFq2COXq6V/gKCU8qm+3AzsbJ2iZdccv/Iaxb+WgqDN6aoF4MRVZU74p5qKfXg12dXAA==" + }, + "due": { + "v": 4, + "i": { + "t": "profiles", + "c": "due" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABxVAjbrawodyK39W6/wh82IABYPU+0mX8CHFvfIfoChRygxSp/EY9wA5Tt3u09Pc3bgwIDcMmFSrX/0VrXRIQ4VO1CORi2fLaZvUHGHKyQ51NO9OU=", + "hm": "decc03613957c47004860bc9bc70770aed4c6ba3f0da2aaea3da9fe7367af6a9" + }, + "opened": { + "v": 4, + "i": { + "t": "profiles", + "c": "opened" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABac+LBy0VG2VPg3ZA53wR5IADij0/N3siwDhSUWwCHNy3M63fuNYPaRFubg/d7h/snCAL03BHwsPgXawwBQRFdQWpQKS5jg02BxED9R4DALnq5jis=" + }, + "posted": { + "v": 4, + "i": { + "t": "profiles", + "c": "posted" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAr9Mw7BH8KdqEzSM13/Zi7IADyLjNo7TciDvsKQU+xOu/8uQlNYMYFE1U+bVt29bUpOQIHRhk22OLKCoM9GSa2moqwpXD2u/VhhJfzC9Yt+Nj6OJMZhFCoU4Fuhg==", + "hm": "b3a2b5c8d5ea562f472a55eb57f3b49aec3d6cc3731dad16c4f8999394a95c55" + }, + "total": { + "v": 4, + "i": { + "t": "profiles", + "c": "total" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAATyRC14+m8N7oH7Js7ZVqRIAAMq9Za2+D51qL5ba1r6/Rtqzc3kwtt8DctI1tVVhdvgQKko9YUgjcDJdcUOhpOnbxcyAs53dUr8W3bkDuQJ2+tOMjWzVbZi3+3UY9THL8=" + } + } + }, + { + "id": 2, + "plaintext": { + "opened": "+262142-12-31", + "due": "2024-03-01", + "created": "2024-02-29T12:34:56.123456789Z", + "posted": "2024-02-29T12:34:56.123456999Z", + "total": "0.0000000000000000000000000001", + "amount": "1.5" + }, + "stored": { + "amount": { + "v": 4, + "i": { + "t": "profiles", + "c": "amount" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAKybDqXBsszdixWIGR8SZBIAD/drxmP5dzmNDKUjYtiDgZvFcHhOZOJzrc6eex0q0MCgJ/PdJDq9vGicUmekPKAKWfOpPYO9BZKtoa1mnfIssxqlPAj08l1SM/ZUkyzdg=", + "hm": "ff787d4a33e41d24d30fb95098c57bdffeef072fb9e0907a2dd7fa621a5eda03" + }, + "created": { + "v": 4, + "i": { + "t": "profiles", + "c": "created" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACseL3lfg99+b/uhGmba7LRIADKXzDfsoVOBKIKwpw6XN/IWltOxnK8Q1qzZ0YkAqK0WwI+rlmCIdTAihn74kvROTOrdSmPQRCT1v1M110EAPHAat3QVBGT+2H9AA==" + }, + "due": { + "v": 4, + "i": { + "t": "profiles", + "c": "due" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAXndapLjoyx2wsK2yToOIfIACwJln1DyojCIuwH8XHvJodMunCGS0dfEYintuFQrNXLALNoZ3nax4abrlBe0je4lvugniEDqDjJ/5+nrsWCL19o84=", + "hm": "086ccfab604ccf449a731ad7e01161bfd80c12f25ba5832e92b709f42b2b7aa4" + }, + "opened": { + "v": 4, + "i": { + "t": "profiles", + "c": "opened" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADhmok01VlvzDZ54Ve4NYs+IAA9SQIB8WxjSZDilDh2n9Ym6YccQQWBJ7ef2clbtu4SgQLl1OmZPlTj2ZHK0QyC215t27avkXEaIiiXstXCtLjsXhQ=" + }, + "posted": { + "v": 4, + "i": { + "t": "profiles", + "c": "posted" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABaXm2d096n8ZEEJiaB3NxKIACL8OhWrR2p4fZZazuJKKgRsv+lJPaamyXwxGVq9vT2EAK7AmoJ7W7R19iCPgf9Vgr+k6s4qT1n8cIt4USX0NfAI+pE7FlsCyeD7A==", + "hm": "b3a2b5c8d5ea562f472a55eb57f3b49aec3d6cc3731dad16c4f8999394a95c55" + }, + "total": { + "v": 4, + "i": { + "t": "profiles", + "c": "total" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABLXWj8ZGmyU14mfULzczbUIADg6xfObeAiOuQQcg49dZ3rMkOc2WqCL0NPepGJDhhslgLyLRD8HyuEqddrbsIzlXzyWBYWvk46vlaHtQK02b+Task10kxCuTxm7gJtg0s=" + } + } + }, + { + "id": 3, + "plaintext": { + "opened": "1970-01-01", + "due": "2024-02-29", + "created": "1970-01-01T00:00:00Z", + "posted": "2024-02-29T12:34:56.123457Z", + "total": "1234.5600", + "amount": "1.51" + }, + "stored": { + "amount": { + "v": 4, + "i": { + "t": "profiles", + "c": "amount" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABnPkyUpuwK99phMHDnRbhCIABCbSCYPvkH92/UTxfFMiL/7X01xvH3mgh3pxeV+v59vwJJQAnSyyjouUFH1r1Y5SrLasHKlU9oED93T3O5LqvtTWwiArdsKBHHpx71T4Y=", + "hm": "7fbed23fa93b6d89ecf0883cddd191d4260d1cc8af68ec9c60d997365dbf4d5f" + }, + "created": { + "v": 4, + "i": { + "t": "profiles", + "c": "created" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAADb10mF0XME/G6O/c19GNdtIAC6NkocrbY4y7PuHSJX/nEKsSq6uC8EKGhUL69Xkgz4MQJh2UUJ/xMpXG6j+ij6GOMZn9R/bakY9mtlssoB5upT/pIJA+hV4N478g==" + }, + "due": { + "v": 4, + "i": { + "t": "profiles", + "c": "due" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAMtFomHN3dgKH8FZNzXqxbIAAX/CwMjLj4HTN/Mo0hRbmUt+V5v4bI366uKHRN55IcpQKhu13tqJuXm3Uw6yrB8GapAq7TEKzLqcOkNT2vSyNPLvk=", + "hm": "decc03613957c47004860bc9bc70770aed4c6ba3f0da2aaea3da9fe7367af6a9" + }, + "opened": { + "v": 4, + "i": { + "t": "profiles", + "c": "opened" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAAAUfff4RXAt9HIL0ezwFl5oIABloL38OjJ3H7ZOBpgYefpdza2nKMMccXMcD8sSAdaR9AIG69Y8fG9aO6AFuMG4AkH92lz4Plj5TuGR3crB+83JuHc=" + }, + "posted": { + "v": 4, + "i": { + "t": "profiles", + "c": "posted" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAABTyd02/xTIqENG5WzqeH+kIACTeY3JPRFisVuLrMf6rcKnordc0e9lrrATavl1B/cL4wJ2f82ziXPF0/JepP3C5XPnAxP/UVQemoyzemJeRGdkqegUvASIHcRrgQ==", + "hm": "b75c44fffb471158dd836f858f88d04658cd8985d0f52ccf582af259eeda030f" + }, + "total": { + "v": 4, + "i": { + "t": "profiles", + "c": "total" + }, + "c": "stack-encrypt:1:AQAAAAAAAAAAAAAAAAAAAACT3tVNTBXZ5pFHbNGibsVoIADBAZSy1MjXUm3HtQF3xbv35hpGRWHsK4FDx5KsARB6cgLDY8S5OvL6ob8+2pUnCBw7Eq1x5kSeBw1lpoRqjUmzugxrQ9NTSGwi1ObgAuk=" + } + } + } + ] +} diff --git a/packages/eql/tests/encryption/tests/targets.rs b/packages/eql/tests/encryption/tests/targets.rs index cf623efe7..ad3020dba 100644 --- a/packages/eql/tests/encryption/tests/targets.rs +++ b/packages/eql/tests/encryption/tests/targets.rs @@ -486,6 +486,9 @@ mod the_table { "boolean" => Some(ValueKind::Bool), "real" => Some(ValueKind::Float32), "double" => Some(ValueKind::Float64), + "date" => Some(ValueKind::Date), + "timestamp" => Some(ValueKind::Timestamp), + "numeric" => Some(ValueKind::Decimal), // The rest have no Rust plaintext type in eql-bindings yet. _ => None, }; @@ -560,8 +563,9 @@ mod the_table { } } - /// The types the engine produces in EQL v4 today: the storage, equality - /// and OPE-ordered domains of the families eql-bindings names a plaintext + /// The types the engine produces in EQL v4 today: the storage and + /// equality domains of every family whose plaintext eql-bindings names, + /// the OPE-ordered domains of those the engine derives an order term /// for, and text match. #[test] fn the_producible_types_are_the_ones_with_no_ore_and_a_rust_plaintext() { @@ -585,6 +589,12 @@ mod the_table { "BigintEq", "BigintOrd", "BigintOrdOpe", + "Date", + "DateEq", + "Timestamp", + "TimestampEq", + "Numeric", + "NumericEq", "Text", "TextEq", "TextMatch", @@ -606,7 +616,9 @@ mod the_table { let expected = match (row.family, row.suffix) { ("json", "Search") => "JSON index", ("json", "") => "tree of leaves", - ("date" | "timestamp" | "numeric", _) => "no Rust plaintext type for this family", + ("date" | "timestamp" | "numeric", "Ord" | "OrdOpe" | "OrdOre") => { + "no order term for a date, timestamp or decimal yet" + } (_, "OrdOre" | "SearchOre") => "derives CLLW ORE (#1132)", ("text", "Search") => "carries a match term", (family, suffix) => panic!("{}: unexpected {family} {suffix}", row.name), @@ -635,7 +647,7 @@ mod the_table { "reason": null }) ); - let row = targets::target("Date").unwrap(); + let row = targets::target("Json").unwrap(); let json = serde_json::to_value(row).unwrap(); assert_eq!( json["plaintext"], diff --git a/packages/eql/tests/encryption/tests/v4_date_time_numeric.rs b/packages/eql/tests/encryption/tests/v4_date_time_numeric.rs new file mode 100644 index 000000000..8026446eb --- /dev/null +++ b/packages/eql/tests/encryption/tests/v4_date_time_numeric.rs @@ -0,0 +1,909 @@ +//! The EQL v4 date, timestamp and numeric types Stack Encrypt produces: +//! each family's storage-only type (`Date`, `Timestamp`, `Numeric`) and its +//! equality type (`DateEq`, `TimestampEq`, `NumericEq`), from eql-bindings' +//! own plaintexts (`v4::plaintext`). Their ordered types wait for the +//! engine's order terms and are refused. +//! +//! Each type must round-trip through both entry points, seal the leaf a +//! chrono or rust_decimal value of the same instant, day or number seals +//! (what any other producer writes for that kind), open a leaf such a value +//! sealed, and refuse a value of another kind. The Go SDK's rows +//! (`fixtures/v4_ledger_rows.json`) open here and are queried in +//! PostgreSQL with the queries this file derives +//! (`fixtures/v4_ledger_queries.json`). + +mod common; + +use std::str::FromStr; + +use chrono::{DateTime, NaiveDate, Utc}; +use eql_bindings::encryption::targets::{self as targets, TargetError}; +use eql_bindings::encryption::NativeCiphertextVisitor as NativeLeaf; +use eql_bindings::v3::terms::Ciphertext; +use eql_bindings::v4::plaintext::{Date, Decimal, Timestamp}; +use eql_bindings::v4::{date, numeric, timestamp}; +use eql_bindings::Identifier; +use serde::de::DeserializeOwned; +use stack_encrypt::dynamic::{ValueFrame, ValueKind}; +use stack_encrypt::target::transcode::Reader; +use stack_encrypt::{Encrypt, Label, NonEmpty}; + +fn frame(value: T) -> ValueFrame { + ValueFrame::of(value).unwrap() +} + +fn same(a: &ValueFrame, b: &ValueFrame) -> bool { + a.risky_ref() == b.risky_ref() +} + +fn column() -> NonEmpty { + Identifier::for_column("profiles", "entry").unwrap() +} + +fn label() -> Label { + Label::new(["profiles", "entry"]).unwrap() +} + +fn refused(result: Result) -> TargetError { + match result { + Err(error) => error, + Ok(_) => panic!("the dispatch accepted what it should have refused"), + } +} + +fn day(text: &str) -> NaiveDate { + NaiveDate::from_str(text).unwrap() +} + +fn instant(text: &str) -> DateTime { + DateTime::parse_from_rfc3339(text).unwrap().to_utc() +} + +fn number(text: &str) -> rust_decimal::Decimal { + rust_decimal::Decimal::from_str(text).unwrap() +} + +/// What a stored `c` opens to as a frame, without any EQL type: the leaf +/// as another producer would read it. +async fn as_frame( + cipher: &stack_encrypt::StackCipher, + c: Ciphertext, +) -> ValueFrame { + let native = c.read(NativeLeaf).unwrap(); + cipher.decrypt(native, column()).await.unwrap() +} + +async fn by_name( + keyset: &stack_encrypt::KeysetCipher<'_, common::Observed>, + name: &str, + value: ValueFrame, +) -> T { + let bytes = targets::encrypt(name, keyset, &label(), value) + .unwrap() + .await + .unwrap(); + serde_json::from_slice(&bytes).unwrap() +} + +/// The `hm` of `name` over `value`, through the by-name query dispatch. +async fn hm( + keyset: &stack_encrypt::KeysetCipher<'_, common::Observed>, + name: &str, + value: ValueFrame, +) -> String { + let bytes = targets::query(name, keyset, &label(), value) + .unwrap() + .await + .unwrap(); + let json: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + json["hm"].as_str().unwrap().to_owned() +} + +#[tokio::test] +async fn dates_round_trip_and_seal_the_leaf_a_naive_date_seals() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for text in ["2024-02-29", "1969-12-31", "0001-01-01", "1970-01-01"] { + let theirs = day(text); + let value = Date::from(theirs); + let stored: date::Date = keyset.encrypt_as(&value, column()).await.unwrap(); + let opened: Date = cipher + .decrypt_as(stored.clone(), column().into()) + .await + .unwrap(); + assert_eq!(NaiveDate::from(&opened), theirs, "{text}"); + assert!( + same(&as_frame(&cipher, stored.c.clone()).await, &frame(theirs)), + "{text}: the leaf a NaiveDate seals" + ); + + let searchable: date::DateEq = keyset.encrypt_as(&value, column()).await.unwrap(); + let named: date::DateEq = by_name(&keyset, "DateEq", frame(theirs)).await; + assert_eq!(searchable.hm, named.hm, "{text}: one term on both paths"); + let opened = targets::decrypt( + "DateEq", + &cipher, + &label(), + &serde_json::to_vec(&named).unwrap(), + ) + .unwrap() + .await + .unwrap(); + assert!(same(&opened, &frame(theirs)), "{text}: by name"); + } + for edge in [NaiveDate::MIN, NaiveDate::MAX] { + let stored: date::DateEq = keyset + .encrypt_as(&Date::from(edge), column()) + .await + .unwrap(); + let opened: Date = cipher.decrypt_as(stored, column().into()).await.unwrap(); + assert_eq!(NaiveDate::from(&opened), edge); + } +} + +#[tokio::test] +async fn timestamps_round_trip_to_the_nanosecond_and_seal_the_leaf_a_date_time_seals() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for text in [ + "2024-02-29T12:34:56.123456789Z", + "1969-12-31T23:59:59.999999999Z", + "1970-01-01T00:00:00Z", + ] { + let theirs = instant(text); + let value = Timestamp::try_from(theirs).unwrap(); + let stored: timestamp::Timestamp = keyset.encrypt_as(&value, column()).await.unwrap(); + let opened: Timestamp = cipher + .decrypt_as(stored.clone(), column().into()) + .await + .unwrap(); + assert_eq!(DateTime::::from(&opened), theirs, "{text}"); + assert!( + same(&as_frame(&cipher, stored.c.clone()).await, &frame(theirs)), + "{text}: the leaf a DateTime seals" + ); + + let searchable: timestamp::TimestampEq = keyset.encrypt_as(&value, column()).await.unwrap(); + let named: timestamp::TimestampEq = by_name(&keyset, "TimestampEq", frame(theirs)).await; + assert_eq!(searchable.hm, named.hm, "{text}: one term on both paths"); + let opened: Timestamp = cipher + .decrypt_as(searchable, column().into()) + .await + .unwrap(); + assert_eq!( + DateTime::::from(&opened), + theirs, + "{text}: nanoseconds kept" + ); + } + for edge in [ + chrono::NaiveDateTime::MIN.and_utc(), + chrono::NaiveDateTime::MAX.and_utc(), + ] { + let stored: timestamp::TimestampEq = keyset + .encrypt_as(&Timestamp::try_from(edge).unwrap(), column()) + .await + .unwrap(); + let opened: Timestamp = cipher.decrypt_as(stored, column().into()).await.unwrap(); + assert_eq!(DateTime::::from(&opened), edge); + } +} + +#[tokio::test] +async fn decimals_round_trip_with_their_scale_and_seal_the_leaf_rust_decimal_seals() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for text in [ + "1.50", + "-1234.5678", + "0.00", + "79228162514264337593543950335", + "-0.0000000000000000000000000001", + ] { + let theirs = number(text); + let value = Decimal::from(theirs); + let stored: numeric::Numeric = keyset.encrypt_as(&value, column()).await.unwrap(); + let opened: Decimal = cipher + .decrypt_as(stored.clone(), column().into()) + .await + .unwrap(); + assert_eq!( + rust_decimal::Decimal::from(&opened).to_string(), + text, + "the scale is kept" + ); + assert!( + same(&as_frame(&cipher, stored.c.clone()).await, &frame(theirs)), + "{text}: the leaf a rust_decimal Decimal seals" + ); + + let searchable: numeric::NumericEq = keyset.encrypt_as(&value, column()).await.unwrap(); + let named: numeric::NumericEq = by_name(&keyset, "NumericEq", frame(theirs)).await; + assert_eq!(searchable.hm, named.hm, "{text}: one term on both paths"); + let opened = targets::decrypt( + "NumericEq", + &cipher, + &label(), + &serde_json::to_vec(&named).unwrap(), + ) + .unwrap() + .await + .unwrap(); + assert!(same(&opened, &frame(theirs)), "{text}: by name"); + } +} + +/// ADR-0002's baseline and vitaminc's equality encodings: a decimal's scale +/// is normalised, a timestamp is truncated to the microsecond, and distinct +/// values have distinct terms. +#[tokio::test] +async fn equality_terms_follow_the_baseline() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + let n = |text: &str| frame(number(text)); + assert_eq!( + hm(&keyset, "NumericEq", n("1.5")).await, + hm(&keyset, "NumericEq", n("1.50")).await, + "1.5 = 1.50" + ); + assert_eq!( + hm(&keyset, "NumericEq", n("0")).await, + hm(&keyset, "NumericEq", n("-0.000")).await, + "every zero is one value" + ); + assert_ne!( + hm(&keyset, "NumericEq", n("1.5")).await, + hm(&keyset, "NumericEq", n("1.51")).await + ); + assert_ne!( + hm(&keyset, "NumericEq", n("1.5")).await, + hm(&keyset, "NumericEq", n("-1.5")).await + ); + + let t = |text: &str| frame(instant(text)); + assert_eq!( + hm(&keyset, "TimestampEq", t("2024-02-29T12:00:00.123456001Z")).await, + hm(&keyset, "TimestampEq", t("2024-02-29T12:00:00.123456999Z")).await, + "one microsecond, one term" + ); + // Truncated, never rounded: the nanoseconds below the microsecond are + // dropped, as the order term drops them. Before 1970 that is floor on + // the count since the epoch. + assert_eq!( + hm(&keyset, "TimestampEq", t("2024-02-29T12:00:00.1234567Z")).await, + hm(&keyset, "TimestampEq", t("2024-02-29T12:00:00.123456Z")).await, + "truncated, not rounded" + ); + assert_eq!( + hm(&keyset, "TimestampEq", t("1969-12-31T23:59:59.9999995Z")).await, + hm(&keyset, "TimestampEq", t("1969-12-31T23:59:59.999999Z")).await, + "floor before the epoch" + ); + assert_ne!( + hm(&keyset, "TimestampEq", t("2024-02-29T12:00:00.123456Z")).await, + hm(&keyset, "TimestampEq", t("2024-02-29T12:00:00.123457Z")).await + ); + + let d = |text: &str| frame(day(text)); + assert_ne!( + hm(&keyset, "DateEq", d("2024-02-29")).await, + hm(&keyset, "DateEq", d("2024-03-01")).await + ); + // A date, an instant and a number are separate domains: the same + // integer underneath is no match. + assert_ne!( + hm( + &keyset, + "DateEq", + frame(Date::from_days_since_epoch(0).unwrap()) + ) + .await, + hm(&keyset, "TimestampEq", frame(Timestamp::new(0, 0).unwrap())).await + ); + assert_ne!( + hm( + &keyset, + "DateEq", + frame(Date::from_days_since_epoch(0).unwrap()) + ) + .await, + hm(&keyset, "NumericEq", frame(Decimal::new(0, 0).unwrap())).await + ); +} + +#[tokio::test] +async fn a_value_of_another_kind_is_refused_not_converted() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for (name, value, expected) in [ + ( + "Date", + frame(instant("2024-02-29T00:00:00Z")), + ValueKind::Date, + ), + ("DateEq", frame::(19_782), ValueKind::Date), + ("Timestamp", frame(day("2024-02-29")), ValueKind::Timestamp), + ("TimestampEq", frame::(0), ValueKind::Timestamp), + ("Numeric", frame::(1.5), ValueKind::Decimal), + ("NumericEq", frame("1.50"), ValueKind::Decimal), + ] { + let error = refused(targets::encrypt(name, &keyset, &label(), value)); + assert!( + matches!(error, TargetError::Plaintext { target, expected: e, .. } if target == name && e == expected), + "{name}: {error}" + ); + } + + // A stored date moved into a timestamp is not read as one. + let stored: date::Date = keyset + .encrypt_as(&Date::from(day("2024-02-29")), column()) + .await + .unwrap(); + let moved = timestamp::Timestamp { + v: stored.v, + i: stored.i, + c: stored.c, + }; + let opened: Result = cipher.decrypt_as(moved, column().into()).await; + assert!(opened.is_err(), "a date is not a timestamp"); +} + +/// The ordered types wait for the engine's order terms over these kinds, +/// and say so; block ORE waits for that first. +#[tokio::test] +async fn the_ordered_types_are_refused_with_their_reason() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for family in ["Date", "Timestamp", "Numeric"] { + for suffix in ["Ord", "OrdOpe", "OrdOre"] { + let name = format!("{family}{suffix}"); + let value = frame(Decimal::new(1, 0).unwrap()); + match refused(targets::encrypt(&name, &keyset, &label(), value)) { + TargetError::Unproducible { reason, .. } => assert!( + reason.contains("no order term for a date, timestamp or decimal"), + "{name}: {reason}" + ), + other => panic!("{name}: {other}"), + } + } + } +} + +/// `fixtures/v4_ledger_queries.json`: the `DateEq`, `TimestampEq` and +/// `NumericEq` query operands the dispatch derives under +/// `FakeDataKeySource`'s index key, which the Go SDK's deterministic guest +/// shares. Its test derives the same queries through generated code and +/// requires the same bytes. Regenerate with `EQL_UPDATE_FIXTURES=1`. +#[tokio::test] +async fn the_ledger_query_fixture_is_what_the_dispatch_derives() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + let query = |name: &'static str, column: &'static str, value: ValueFrame| { + let keyset = &keyset; + async move { + let label = Label::new(["profiles", column]).unwrap(); + let bytes = targets::query(name, keyset, &label, value) + .unwrap() + .await + .unwrap(); + String::from_utf8(bytes).unwrap() + } + }; + let fixture = serde_json::json!({ + "_comment": "The DateEq, TimestampEq and NumericEq query operands eql-bindings derives for each plaintext under FakeDataKeySource's index key (keyset nil), read by the Go SDK's hermetic test; regenerate with EQL_UPDATE_FIXTURES=1, do not edit.", + "table": "profiles", + "due": { + "plaintext": "2024-02-29", + "query": query("DateEq", "due", frame(day("2024-02-29"))).await, + }, + "posted": { + "plaintext": "2024-02-29T12:34:56.123456Z", + "query": query("TimestampEq", "posted", frame(instant("2024-02-29T12:34:56.123456Z"))).await, + }, + "amount": { + "plaintext": "1.50", + "query": query("NumericEq", "amount", frame(number("1.50"))).await, + }, + }); + let path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("fixtures/v4_ledger_queries.json"); + let rendered = serde_json::to_string_pretty(&fixture).unwrap() + "\n"; + if std::env::var_os("EQL_UPDATE_FIXTURES").is_some() { + std::fs::write(&path, &rendered).unwrap(); + } + let committed = std::fs::read_to_string(&path) + .expect("fixtures/v4_ledger_queries.json is committed; EQL_UPDATE_FIXTURES=1 writes it"); + assert_eq!( + committed, rendered, + "the committed fixture is what the dispatch derives today" + ); +} + +fn database() -> postgres::Client { + postgres::Client::connect( + &std::env::var("EQL_TEST_DATABASE_URL").expect("EQL_TEST_DATABASE_URL"), + postgres::NoTls, + ) + .unwrap() +} + +fn json(value: &T) -> String { + serde_json::to_string(value).unwrap() +} + +/// Needs an EQL v4 installation. Rust-sealed values of each type pass their +/// domain check and read back, `=` on each equality column finds the rows +/// its query names (1.5 and 1.50 are one number; one microsecond is one +/// instant), and the storage-only columns refuse `=`. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_stores_and_finds_date_timestamp_and_numeric_rows() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (cipher, _) = runtime.block_on(common::cipher()); + let keyset = cipher.default_keyset(); + let mut db = database(); + db.batch_execute( + "CREATE TEMP TABLE v4_ledger ( + id int PRIMARY KEY, + opened public.eql_v4_date, + due public.eql_v4_date_eq, + created public.eql_v4_timestamp, + posted public.eql_v4_timestamp_eq, + total public.eql_v4_numeric, + amount public.eql_v4_numeric_eq + );", + ) + .unwrap(); + let rows = [ + (1, "2024-02-29", "2024-02-29T12:00:00.123456001Z", "1.5"), + ( + 2, + "0001-01-01", + "1969-12-31T23:59:59.999999999Z", + "-79228162514264337593543950335", + ), + (3, "2024-02-29", "2024-02-29T12:00:00.123456999Z", "1.50"), + (4, "2024-03-01", "2024-02-29T12:00:00.123457Z", "1.51"), + ]; + for (id, d, t, n) in rows { + let (opened, due, created, posted, total, amount) = runtime.block_on(async { + let d = Date::from(day(d)); + let t = Timestamp::try_from(instant(t)).unwrap(); + let n = Decimal::from(number(n)); + ( + json( + &keyset + .encrypt_as::<_, date::Date>(&d, column()) + .await + .unwrap(), + ), + json( + &keyset + .encrypt_as::<_, date::DateEq>(&d, column()) + .await + .unwrap(), + ), + json( + &keyset + .encrypt_as::<_, timestamp::Timestamp>(&t, column()) + .await + .unwrap(), + ), + json( + &keyset + .encrypt_as::<_, timestamp::TimestampEq>(&t, column()) + .await + .unwrap(), + ), + json( + &keyset + .encrypt_as::<_, numeric::Numeric>(&n, column()) + .await + .unwrap(), + ), + json( + &keyset + .encrypt_as::<_, numeric::NumericEq>(&n, column()) + .await + .unwrap(), + ), + ) + }); + db.execute( + "INSERT INTO v4_ledger VALUES ($1, + $2::text::jsonb::public.eql_v4_date, + $3::text::jsonb::public.eql_v4_date_eq, + $4::text::jsonb::public.eql_v4_timestamp, + $5::text::jsonb::public.eql_v4_timestamp_eq, + $6::text::jsonb::public.eql_v4_numeric, + $7::text::jsonb::public.eql_v4_numeric_eq)", + &[&id, &opened, &due, &created, &posted, &total, &amount], + ) + .unwrap(); + } + let mut ids = |column_name: &str, query_domain: &str, query: String| -> Vec { + db.query( + &format!( + "SELECT id FROM v4_ledger WHERE {column_name} = $1::text::jsonb::eql_v4.{query_domain} ORDER BY id" + ), + &[&query], + ) + .unwrap() + .iter() + .map(|r| r.get(0)) + .collect() + }; + let date_query = |text: &str| { + let value = Date::from(day(text)); + runtime.block_on(async { + json( + &keyset + .encrypt_as::<_, date::DateEqQuery>(&value, column()) + .await + .unwrap(), + ) + }) + }; + let timestamp_query = |text: &str| { + let value = Timestamp::try_from(instant(text)).unwrap(); + runtime.block_on(async { + json( + &keyset + .encrypt_as::<_, timestamp::TimestampEqQuery>(&value, column()) + .await + .unwrap(), + ) + }) + }; + let numeric_query = |text: &str| { + let value = Decimal::from(number(text)); + runtime.block_on(async { + json( + &keyset + .encrypt_as::<_, numeric::NumericEqQuery>(&value, column()) + .await + .unwrap(), + ) + }) + }; + assert_eq!( + ids("due", "query_date_eq", date_query("2024-02-29")), + [1, 3] + ); + assert_eq!(ids("due", "query_date_eq", date_query("0001-01-01")), [2]); + assert_eq!( + ids("due", "query_date_eq", date_query("2025-01-01")), + Vec::::new() + ); + assert_eq!( + ids( + "posted", + "query_timestamp_eq", + timestamp_query("2024-02-29T12:00:00.123456Z") + ), + [1, 3], + "one microsecond is one instant" + ); + assert_eq!( + ids( + "posted", + "query_timestamp_eq", + timestamp_query("2024-02-29T12:00:00.123457Z") + ), + [4] + ); + assert_eq!( + ids("amount", "query_numeric_eq", numeric_query("1.500")), + [1, 3], + "1.5 = 1.50 = 1.500" + ); + assert_eq!( + ids( + "amount", + "query_numeric_eq", + numeric_query("-79228162514264337593543950335") + ), + [2] + ); + + // A stored value reads back and opens with its scale and nanoseconds. + let row = db + .query_one( + "SELECT opened::text, created::text, total::text FROM v4_ledger WHERE id = 3", + &[], + ) + .unwrap(); + let (opened, created, total): (String, String, String) = (row.get(0), row.get(1), row.get(2)); + runtime.block_on(async { + let opened: Date = cipher + .decrypt_as( + serde_json::from_str::(&opened).unwrap(), + column().into(), + ) + .await + .unwrap(); + assert_eq!(NaiveDate::from(&opened), day("2024-02-29")); + let created: Timestamp = cipher + .decrypt_as( + serde_json::from_str::(&created).unwrap(), + column().into(), + ) + .await + .unwrap(); + assert_eq!(created.nanoseconds(), 123_456_999); + let total: Decimal = cipher + .decrypt_as( + serde_json::from_str::(&total).unwrap(), + column().into(), + ) + .await + .unwrap(); + assert_eq!(rust_decimal::Decimal::from(&total).to_string(), "1.50"); + }); + + // The storage-only columns have no equality operator. + for (column_name, domain) in [ + ("opened", "eql_v4_date"), + ("created", "eql_v4_timestamp"), + ("total", "eql_v4_numeric"), + ] { + let err = db + .query( + &format!("SELECT id FROM v4_ledger WHERE {column_name} = {column_name}"), + &[], + ) + .unwrap_err(); + let message = err.as_db_error().map(|e| e.message()).unwrap_or_default(); + assert!( + message.contains(&format!("is not supported for public.{domain}")), + "{column_name}: {message}" + ); + } +} + +/// `fixtures/v4_ledger_rows.json`: `Ledger` rows the Go SDK's generated code +/// sealed over the deterministic EQL guest +/// (`TestV4LedgerRowsMatchTheCommittedFixture` writes it). +mod go_ledger { + use super::*; + use std::collections::BTreeMap; + + #[derive(serde::Deserialize)] + pub struct Fixture { + pub seed: String, + pub table: String, + pub rows: Vec, + } + #[derive(serde::Deserialize)] + pub struct Row { + pub id: i32, + pub plaintext: Plain, + pub stored: BTreeMap, + } + /// Dates in ISO 8601, instants in RFC 3339 with nanoseconds, decimals + /// as fixed-point text with their scale. + #[derive(serde::Deserialize)] + pub struct Plain { + pub opened: String, + pub due: String, + pub created: String, + pub posted: String, + pub total: String, + pub amount: String, + } + + pub fn fixture() -> Fixture { + let path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("fixtures/v4_ledger_rows.json"); + let raw = std::fs::read_to_string(path).expect( + "fixtures/v4_ledger_rows.json is written by the Go SDK's \ + TestV4LedgerRowsMatchTheCommittedFixture", + ); + serde_json::from_str(&raw).unwrap() + } + + pub async fn cipher(seed: &str) -> stack_encrypt::StackCipher { + let seed: [u8; 32] = seed.as_bytes().try_into().expect("a 32-byte seed"); + stack_encrypt::StackCipher::builder() + .kms(stack_kms::DeterministicSource::new(seed)) + .init() + .await + .unwrap() + } + + fn parsed(row: &Row, column: &str) -> T { + serde_json::from_value(row.stored[column].clone()) + .unwrap_or_else(|e| panic!("row {} {column}: {e}", row.id)) + } + + /// Open every field of `row` as its v4 type, into eql-bindings' + /// plaintexts, and check it against what Go sealed. + pub async fn open_row( + cipher: &stack_encrypt::StackCipher, + row: &Row, + ) { + let column = |name: &str| Identifier::for_column("profiles", name).unwrap(); + let p = &row.plaintext; + let id = row.id; + let opened: Date = cipher + .decrypt_as(parsed::(row, "opened"), column("opened").into()) + .await + .unwrap(); + assert_eq!(NaiveDate::from(&opened), day(&p.opened), "row {id} opened"); + let due: Date = cipher + .decrypt_as(parsed::(row, "due"), column("due").into()) + .await + .unwrap(); + assert_eq!(NaiveDate::from(&due), day(&p.due), "row {id} due"); + let created: Timestamp = cipher + .decrypt_as( + parsed::(row, "created"), + column("created").into(), + ) + .await + .unwrap(); + assert_eq!( + DateTime::::from(&created), + instant(&p.created), + "row {id} created" + ); + let posted: Timestamp = cipher + .decrypt_as( + parsed::(row, "posted"), + column("posted").into(), + ) + .await + .unwrap(); + assert_eq!( + DateTime::::from(&posted), + instant(&p.posted), + "row {id} posted" + ); + let total: Decimal = cipher + .decrypt_as( + parsed::(row, "total"), + column("total").into(), + ) + .await + .unwrap(); + assert_eq!( + rust_decimal::Decimal::from(&total).to_string(), + p.total, + "row {id} total keeps its scale" + ); + let amount: Decimal = cipher + .decrypt_as( + parsed::(row, "amount"), + column("amount").into(), + ) + .await + .unwrap(); + assert_eq!( + rust_decimal::Decimal::from(&amount).to_string(), + p.amount, + "row {id} amount keeps its scale" + ); + } +} + +/// What the Go SDK sealed as the date, timestamp and numeric types parses as +/// each type and opens in Rust to the same day, instant and number. +#[tokio::test] +async fn ledger_rows_the_go_sdk_sealed_open_in_rust() { + let fixture = go_ledger::fixture(); + assert_eq!(fixture.table, "profiles"); + assert!(fixture.rows.len() > 1); + let cipher = go_ledger::cipher(&fixture.seed).await; + for row in &fixture.rows { + go_ledger::open_row(&cipher, row).await; + } +} + +/// The Go-sealed ledger rows in PostgreSQL: each value passes its domain +/// check, and `=` with this file's queries (which the Go SDK's test derives +/// byte for byte) finds the rows holding that day, microsecond and number. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_stores_and_queries_ledger_rows_the_go_sdk_sealed() { + let fixture = go_ledger::fixture(); + let mut db = database(); + db.batch_execute( + "CREATE TEMP TABLE go_ledgers ( + id int PRIMARY KEY, + opened public.eql_v4_date, + due public.eql_v4_date_eq, + created public.eql_v4_timestamp, + posted public.eql_v4_timestamp_eq, + total public.eql_v4_numeric, + amount public.eql_v4_numeric_eq + );", + ) + .unwrap(); + for row in &fixture.rows { + let v = |column: &str| row.stored[column].to_string(); + db.execute( + "INSERT INTO go_ledgers VALUES ($1, + $2::text::jsonb::public.eql_v4_date, + $3::text::jsonb::public.eql_v4_date_eq, + $4::text::jsonb::public.eql_v4_timestamp, + $5::text::jsonb::public.eql_v4_timestamp_eq, + $6::text::jsonb::public.eql_v4_numeric, + $7::text::jsonb::public.eql_v4_numeric_eq)", + &[ + &row.id, + &v("opened"), + &v("due"), + &v("created"), + &v("posted"), + &v("total"), + &v("amount"), + ], + ) + .unwrap(); + } + let queries: serde_json::Value = serde_json::from_str( + &std::fs::read_to_string( + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("fixtures/v4_ledger_queries.json"), + ) + .unwrap(), + ) + .unwrap(); + let plaintext = |key: &str| queries[key]["plaintext"].as_str().unwrap().to_owned(); + let due = day(&plaintext("due")); + let posted = instant(&plaintext("posted")).timestamp_micros(); + let amount = number(&plaintext("amount")).normalize(); + for (column_name, query_domain, expected) in [ + ( + "due", + "query_date_eq", + fixture + .rows + .iter() + .filter(|r| day(&r.plaintext.due) == due) + .map(|r| r.id) + .collect::>(), + ), + ( + "posted", + "query_timestamp_eq", + fixture + .rows + .iter() + .filter(|r| instant(&r.plaintext.posted).timestamp_micros() == posted) + .map(|r| r.id) + .collect(), + ), + ( + "amount", + "query_numeric_eq", + fixture + .rows + .iter() + .filter(|r| number(&r.plaintext.amount).normalize() == amount) + .map(|r| r.id) + .collect(), + ), + ] { + assert!( + !expected.is_empty() && expected.len() < fixture.rows.len(), + "{column_name}: the fixture splits the rows" + ); + let key = column_name; + let ids: Vec = db + .query( + &format!( + "SELECT id FROM go_ledgers WHERE {column_name} = $1::text::jsonb::eql_v4.{query_domain} ORDER BY id" + ), + &[&queries[key]["query"].as_str().unwrap()], + ) + .unwrap() + .iter() + .map(|r| r.get(0)) + .collect(); + assert_eq!(ids, expected, "= over Go-sealed {column_name}"); + } +} diff --git a/packages/eql/tests/encryption/tests/v4_types.rs b/packages/eql/tests/encryption/tests/v4_types.rs index ecd8c4cb6..84f0c32a6 100644 --- a/packages/eql/tests/encryption/tests/v4_types.rs +++ b/packages/eql/tests/encryption/tests/v4_types.rs @@ -14,7 +14,7 @@ mod common; use eql_bindings::encryption::targets::{self as targets, TargetError}; use eql_bindings::encryption::{CiphertextVisitor, NativeCiphertextVisitor as NativeLeaf}; use eql_bindings::v3::terms::Ciphertext; -use eql_bindings::v4::{bigint, boolean, double, integer, real, smallint, text}; +use eql_bindings::v4::{bigint, boolean, double, integer, plaintext, real, smallint, text}; use eql_bindings::Identifier; use serde::{de::DeserializeOwned, Serialize}; use stack_encrypt::dynamic::{ValueFrame, ValueKind}; @@ -123,6 +123,9 @@ fn sample(kind: ValueKind) -> ValueFrame { ValueKind::Float32 => frame::(-0.0), ValueKind::Float64 => frame::(f64::NAN), ValueKind::String => frame("a private note"), + ValueKind::Date => frame(plaintext::Date::from_days_since_epoch(-719_162).unwrap()), + ValueKind::Timestamp => frame(plaintext::Timestamp::new(-1, 999_999_999).unwrap()), + ValueKind::Decimal => frame(plaintext::Decimal::new(-123_450, 2).unwrap()), other => panic!("no sample for {other:?}: add one with the type that takes it"), } }