diff --git a/docs/plans/2026-10-24-stack-encrypt-api-plan.md b/docs/plans/2026-10-24-stack-encrypt-api-plan.md new file mode 100644 index 000000000..00969faf7 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-plan.md @@ -0,0 +1,1537 @@ +# Stack Encrypt API implementation plan + +Date: 2026-10-24. Status: proposed implementation direction, developed from +the [review log](2026-10-24-stack-encrypt-api-review.md), +[public examples](2026-10-24-stack-encrypt-public-examples.md), and +[context examples](2026-10-24-stack-encrypt-context-examples.md). + +This document turns the review into implementation choices and completion +criteria. User requirements are distinguished from recommendations that +have not yet been accepted. Examples below are proposed APIs, not working +code for the current release. + +**Implementation reconciliation, 2026-10-27:** reviewed merged `next` at +`ffeed7adb3f7d8739cda53261046147852bffc82`. Go JSON catalog, configured +targets, queries, request-bound reads, typed documents and executable docs +have landed (#1372, #1373, #1378, #1402). Opaque Go storage adoption has +landed (#1369/#1370). The active customer work is now contract conformance, +remaining read/configuration gaps and release readiness, not implementing +those foundations again. See the refreshed +[Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and [JSON inventory](2026-10-25-stack-encrypt-json-operations.md). + +#1379 added cascade authentication and randomized storage/projection order. +Wildcard `->` now selects an unspecified element; old first-in-document-order +recipes are invalid. Freeze the post-cascade format and compare query +semantics rather than randomized byte ordering. #1397 implements short +whole-word terms in the shared Bloom filter; configurable native Go match +options and broader tokenizer/filter work remain. This is implementation +status, not acceptance of a stronger privacy claim for those terms. + +Go's Vitamin C storage modules are temporarily pinned to `06568ba93bd9`. +The final-release guard requires replacing pseudo-versions with released +module tags; prereleases may carry them. Rust remains on crates.io 0.8.0. +Keep crate moves and the final Rust facade behind customer delivery. + +**Latest delivery decisions, 2026-10-25:** encoded ciphertext field targets +are accepted. The comprehensive JSON first-release contract and Rust/dynamic +examples are in [JSON operations](2026-10-25-stack-encrypt-json-operations.md), +including extraction and walks. ORE does not provide equality initially; +retain a future explicit capability for equality-preserving encodings. +Prioritize stable blob storage bytes via +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); +zeroization/timing implementation improvements may follow without changing +stored format. The naming/package review is now in progress. These decisions supersede +earlier sequencing recommendations below where they conflict. + +**Follow-up decisions:** custom Rust query stages may be explicitly +Rust-only, with `NotPortable` on dynamic export until registered; built-in +capabilities retain Rust/Go parity. Every validation independent of keys +runs before key resolution/loading. The user is still reviewing the JSON +operation contract; its detailed API proposals are not collectively +approved. New keyset-routing, lock-context and provider-capability questions +are assessed in [provider capabilities](2026-10-25-stack-encrypt-provider-capabilities.md). + +**Packaging review:** the user favors retiring Vitamin C `AsyncCipher` and +moving application encryption behavior into Stack. The +[packaging and naming recommendation](2026-10-25-stack-encrypt-packaging.md) +defines crate ownership, an acyclic dependency graph, the proposed shared +format crate, role-based public names and coordinated migration work. It is +a proposal for review, not an accepted rename or an implementation. + +**Compatibility scope (user clarification, 2026-10-25):** this API has only +existed as an early prototype. Do not preserve its APIs, stored ciphertexts, +or encodings for backward compatibility. No legacy decoders, compatibility +aliases, deprecation period, or data migration is required. Choose the +coherent new design and update its Rust, Go, guest, and EQL consumers +together. Engine output must still agree with the SQL that consumes it; +that is current interoperability, not a legacy-format requirement. +The merged Go implementation nevertheless retains a legacy JSON opaque +reader; record it as existing behavior, not a new requirement to add +prototype migration machinery. This exemption ends for customer profiles when released: subsequent +Rust API/package refactors must preserve the Go interface and stored +data/search compatibility. + +## Requirements established in the review + +- Ordinary encryption is keyset-directed, with `encrypt` and + `encrypt_with_context`. Keep the separate target-directed `EncryptInto` + conversion interface; remove redundant public `encrypt_as`. +- Single-value encryption produces opaque ciphertext; `OpaqueBlob` + explicitly encodes a composite value before sealing it as one value. + Direct `encrypt` does not automatically lift over collections. Use + `EncryptInto` for collection batching, including `Vec` into + `Vec` without caller context. This division is agreed as of + 2026-10-25. The current byte-vector recommendation follows that same + collection rule and uses `OpaqueBlob` for a whole vector; there is no + prototype-compatibility reason to add a raw-byte exception. +- Rust traits and derives are the primary static interface. Plans support + those declarations and dynamic/FFI lowering. +- Design static and dynamic support against the same engine/definition + contract. Deliver the customer Go interface first; polished public Rust + traits/derives may follow. Engine-level Rust interoperability tests remain + required, so this does not introduce a second Go encryption implementation. +- Provide decrypt entry points with and without explicit context. The + no-context variant must not invent a default required application value. +- `OpaqueEncode` / `OpaqueDecode` name the single-ciphertext representation + capabilities. Both `encrypt` and `encrypt_into`, including their + context-taking variants, consume input and take responsibility for + cleaning up owned plaintext and plaintext artifacts. Ownership alone is + not a zeroization implementation; require `Zeroize` on plaintext types + and enforce cleanup through guarded storage and lifecycle checks. Deliberate passthrough + fields transfer to the output rather than being erased. +- Keep EQL's engine/SQL contract coherent and retain shared batching/execution + behavior; prototype representations need not be preserved. +- Investigate and replace the EQL JSON adapter's serialization round trips + with direct typed conversion, maintaining engine/SQL agreement. + +## Recommended decision: general configured targets + +Use a general target-plus-typed-options contract. Do not add a public +`json_tree(...)` derive construct. Rust constructs concrete options directly; +the dynamic interface deserializes its options into that same type. JSON +supplies the options type, validation, and implementation behind the general +mechanism. Sharing a contract does not require sharing input syntax or +making static Rust configuration pass through deserialization. + +Three approaches were considered: + +| Approach | Benefit | Limitation | Recommendation | +| --- | --- | --- | --- | +| Typed Rust options factory | Compiler checks the options type; normal Rust APIs and tooling | The function itself is not portable, but its concrete output has a data form | Preferred static surface | +| Named policy type holding options | Reusable and discoverable | Adds a provider trait and marker type where a function can suffice | Optional later convenience | +| Target plus data options | Expressible in Go declarations and dynamic data | Requires deserialization and semantic validation | Dynamic surface, producing the same typed options | + +This supersedes the initial recommendation to put option field literals in +Rust attributes. Rejecting a Rust factory because its code cannot cross FFI +was unnecessarily restrictive: the common boundary is its typed result, +not the code used to construct it. No arbitrary encryption callback is +introduced; the factory only supplies configuration. + +The target owns its source capabilities, canonical output, options/defaults, +validation, encryption description, index/query capabilities, and opening +behavior. Context remains a separate input. Configuration does not supply +arbitrary execution callbacks or let a caller redefine EQL's wire fields. + +Implement configuration as a target capability (working name: +`ConfigurableTarget`), lowering into the existing operation descriptions. +Its associated options type has a data representation. Rust factories +return that type directly; registered dynamic targets deserialize into it. +Validate and normalize once when a reusable declaration is built. Both +paths use the same target-owned semantic validator, before key I/O. Use +validated deserialization, for example a wire helper plus `TryFrom`, rather +than a derived deserializer that bypasses checked builders. Keep any raw +options fields private if the public type promises validated invariants. + +Compile-time guarantees cover factory resolution and signatures, the +target/options association, and typed builder arguments. They do not imply +that a string is a valid JSON path or that the factory evaluates at compile +time. JSON path construction is currently fallible. Begin with a fallible +factory contract, `fn() -> Result`, with errors converted +to the engine's configuration error path. Propagate errors without panics +or `unwrap`; a later literal-path macro can add stronger static validation. + +Resolve the factory once per reusable configured declaration and reuse its +normalized options for writes and queries. A factory can read mutable +external state, so its type alone cannot prove a fixed policy; document +that fixed-policy providers must be stable. Do not independently reevaluate +the provider when generating a query. Intentionally variable configuration +belongs in an explicitly constructed configured-target instance. + +Unlike today's `Layout`, the general capability must not require one index: +targets may have none or multiple. Adapt the existing JSON layout internally +while migrating; do not expose two competing configured-target concepts. +The target catalog resolves registered targets for Go. Custom Rust targets +remain possible without registration, but cannot be dynamically addressed +until their adapter and descriptor are included in the guest build. + +## Candidate static syntax + +`target(options = payroll_options)` names a Rust factory for the inner +target of a transparent newtype. The inner Rust type selects the output +representation and associated options type. The spelling is proposed; +typed factories for Rust and typed deserialization for dynamic callers are +the revised recommendation. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; +use stack_encrypt::json::JsonOptions; +use stack_encrypt::sem::TermError; + +fn payroll_options() -> Result { + JsonOptions::new().ordering(["$.salary"]) +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = serde_json::Value, + transparent, + target(options = payroll_options) +)] +struct EncryptedPayroll(EqlEncryptedJsonTree); +``` + +The factory body uses today's `JsonOptions` builder; the derive support is +proposed. The derive calls the factory and type-checks its result against +the inner target's associated options type. It knows neither JSON path +semantics nor JSON-specific option names, and does not serialize or +deserialize the factory's result. The configured target constructs its +output, which the derive wraps. Serde transparency preserves that output's +wire envelope; SQL driver traits are a separate integration requirement. +The same factory attribute should work on a field of a derived output +record, with existing context selection rules. + +```rust +let encrypted: EncryptedPayroll = payroll + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +let restored = serde_json::Value::decrypt_from_with_context( + encrypted, &keyset, column, +).await?; +``` + +Fixed policy is a recipe for writes and queries. Transparent deserialization +does not attest that an existing row was created with that policy. In +particular, the full ordered-path policy is not stored in the JSON header. + +## Candidate dynamic representation and Go requirements + +Implemented by #1372: string targets and configured object targets share +the native options validator. Use the landed `JsonSearch` identifier: + +```json +{ + "payroll": { + "context": ["employees", "payroll"], + "target": { + "name": "JsonSearch", + "options": { "ordered": ["$.salary"] } + } + } +} +``` + +Keep `"target": "TextEq"` as shorthand for that target's default options. +The object form and JSON target options are now implemented. +Reject unknown options, incompatible options, unsupported target names, and +invalid paths during declaration validation. Do not silently fall back to +defaults. Specify one consistent JSON wire format and context encoding +across the Rust, dynamic, Go, and database paths. + +Go's generator and policy API must represent these same options, emit the +same declaration, and obtain the EQL value from the guest. Go must not +assemble the EQL envelope or reproduce JSON indexing. Discovery must expose +the options contract and capabilities; extend the existing target catalog +and resolver rather than installing a parallel JSON registry. +The dynamic registry invokes the concrete target options deserializer; it +does not look up or invoke the Rust application's factory by name. A Go +caller supplies equivalent options data. If a Rust policy must be exported, +serialize its resolved options, not its function identifier. This contract +does not require all Rust-only targets to depend on Serde; data conversion +is required for targets registered with the dynamic interface. + +The catalog/guest, JSON value kind, strict JSON text parser and generated +Go target options are now implemented. #1373 adds operation input shapes and +operand encodings, and #1378 adds typed Go document adapters. Remaining +work includes the formal versioned specification, generalized definition +lifecycle/caching, direct typed EQL conversion and context-role separation. +The current data-plan grammar is still a field map; the illustrative +versioned schema below is not already accepted by the runtime. + +Writes and queries must reuse the normalized configured target in Rust and +Go. An ordering query on an undeclared path must fail before key I/O. +Containment, path equality, ordering, and extraction need capability metadata +that can express their different operand/result shapes, not a single +assumed scalar query twin. Decryption must use stored cryptographic metadata +and context, not require the caller to remember the write's ordered paths. + +## Accepted decrypt surface and return ownership + +**Decision, 2026-10-25:** retain ordinary keyset-directed decryption for one +opaque ciphertext and destination-directed `DecryptFrom` for declared +targets/layouts. Both paths have context-free and context-taking variants: + +```rust +let value: i32 = keyset.decrypt(encrypted).await?; +let value: i32 = keyset.decrypt_with_context(encrypted, context).await?; + +let plaintext = User::decrypt_from(encrypted, &keyset).await?; +let plaintext = User::decrypt_from_with_context(encrypted, &keyset, context) + .await?; +``` + +These are independent call examples, not sequential reuse of a consumed +ciphertext. Direct decrypt mirrors direct encrypt; `DecryptFrom` mirrors +target-directed `EncryptInto`. The existing input capabilities still govern +whether omission of context is available. Do not add a competing public +ciphertext-receiver `decrypt_into` method. + +Successful decryption returns an ordinary owned `T: Zeroize`. Stack guards +recovered bytes and intermediate plaintext while it owns them; successful +return transfers responsibility for the value to the caller. `Zeroize` does +not automatically wipe that returned value on drop. Support explicitly +requested drop-guarded results without requiring a wrapper around every +decrypted application struct; the precise opt-in adapter is implementation +design within this decision. Guarding the completed value does not resolve +partial-deserialization cleanup, which remains required work. + +This reads as constructing `User` from ciphertext. Calling +`encrypted.decrypt_from(...)` would conflict with the ordinary From/Into +receiver convention; retain the agreed destination receiver. +The destination-directed pair already exists on the blanket `DecryptFrom` +extension trait. Today's ciphertext method is +`encrypted.decrypt_into(cipher, context)`, also on `DecryptFrom`, and has +no separate context-free overload. The current declaration trait +`DecryptInto

` is another layer; do not rename it accidentally while +settling method ergonomics. Consolidate declaration and convenience-trait +naming with review issue 5. The public examples use the destination-directed +pair; remove the competing ciphertext method when implementing this surface. + +## Recommended public traits and context input capabilities + +Status: direction accepted by the user on 2026-10-24: separate conversion +methods from `EncryptTarget`/`DecryptSource` declarations, and use explicit +caller-context capabilities instead of `Default`. Implementation, complete +signatures, and call-site updates remain pending. This section resolves the +direction of work areas A and B below; earlier examples retain historical +derive names until their migration. + +Keep two user-facing conversion traits, implemented by blanket library +implementations: + +| Trait | Public methods | Receiver | +| --- | --- | --- | +| `EncryptInto` | `encrypt_into`, `encrypt_into_with_context` | Plaintext value; output inferred or selected | +| `DecryptFrom` | `decrypt_from`, `decrypt_from_with_context` | Plaintext destination type; ciphertext supplied as an argument | + +These methods run declarations through the common engine and return the +existing batched awaitable. Target implementers do not override execution. +Ordinary `keyset.encrypt` and `encrypt_with_context` remain inherent methods +and do not gain a plaintext-side `encrypt` synonym. Their accepted decrypt +counterparts are `keyset.decrypt` and `decrypt_with_context`. + +Give the declaration traits role names, rather than From/Into names that +promise conversion methods they do not define: + +| Proposed declaration | Implemented on | Describes | +| --- | --- | --- | +| `target::EncryptTarget

` | Encrypted output or term target | How to produce this target from `P` | +| `target::DecryptSource

` | Recoverable encrypted output | How this stored value yields `P` | + +The corresponding derives should use the same names. This replaces today's +declaration traits `EncryptFrom

` and `DecryptInto

`; it does not remove +the retained `EncryptInto` extension trait. A term-only target implements +only `EncryptTarget`: no decrypt method that always fails. Plaintext visitor +traits remain a separate Stack-owned representation layer; moving/renaming +them must retain a single canonical trait implementation and a consistent +wire contract across the new consumers; prototype compatibility is not needed. +Configured-target options are another associated capability, not context. + +Declaration methods continue returning core-owned operation descriptions. +Encryption declarations do not receive the plaintext and cipher together; +decrypt declarations inspect stored output and select its opening, without +performing key I/O. Plans, derives, and dynamic resolution reuse those +descriptions. Keep the declaration module public for custom targets and +keep normal method imports/derives available from the Stack entry point. + +Use `ContextInput` as the associated-type name on each declaration trait. +It describes only what the caller must or may supply. It is not the complete +cryptographic context, which can also include a prefix and fields selected +from plaintext or stored output. Encrypt and decrypt declare this type +independently. + +Recommend library-owned input wrappers with explicit capabilities: + +| Proposed input | Omitted argument | Supplied argument | Meaning | +| --- | --- | --- | --- | +| `NoCallerContext` | Allowed | Not accepted | Declaration derives all context and exposes no caller input | +| `RequiredContext` | Unavailable | Required `C` | Supplies context or the missing component selected by the declaration | +| `OptionalContext` | Allowed | Accepted `C` | For ordinary ciphertext: omission means empty context; supplied context is used directly | +| `ContextExtension` | Allowed | Accepted `C` | Omission preserves the declared base; supply extends it | +| `ExpectedContext` | Allowed | Accepted `C` | Omission reconstructs stored context; supply additionally checks it | + +`OptionalContext` does not append a synthetic empty prefix, and omission +of an extension does not append `()`. `ExpectedContext` never extends AAD. +The table specifies behavior; names may be refined without changing it. +Nonempty requirements belong to the accepted argument type or validated +resolved context, not to whether an input is optional. Search targets still +require a nonempty effective context. Ordinary scalar ciphertext should +accept the same empty context through both keyset and target-directed APIs. + +No-context methods require an explicit, sealed `AllowsNoContext` capability; +`_with_context` methods require a corresponding `AcceptsContext` capability +with its accepted argument type. Neither is inferred from `C: Default`. +`RequiredContext` consequently remains required even though `u64` +defaults to zero. The helpers accept ordinary argument values (or ergonomic +checked conversions); users do not manually wrap `RequiredContext` or +`ExpectedContext` at every call. Conversion errors propagate before key I/O. + +The core-owned wrappers carry presence and intent; target declarations +perform source selection and context composition. For composite required +input, `C` may be a tuple or named struct; it does not have to be the full +logical context. Preserve existing grouping rules and context encodings. + +Examples of independent encrypt/decrypt requirements: + +| Declaration | Encrypt input | Decrypt input | +| --- | --- | --- | +| Caller row ID stored in output | `RequiredContext` | `ExpectedContext` | +| Source row ID also stored in output | `NoCallerContext` | `ExpectedContext` | +| Source row ID not recoverable before decrypt | `NoCallerContext` | `RequiredContext` | +| Static field base with optional unstored extension | `ContextExtension` | `ContextExtension` | + +In the last row, decrypting an extended ciphertext requires supplying the +same extension; availability of the omitted method is not a promise that +it opens every ciphertext of that Rust type. A derive can expose an optional +extension instead of `NoCallerContext` where its declaration explicitly +allows that behavior. Expected-context checks do not authenticate stored +data themselves: successful cryptographic opening remains necessary. + +The dynamic plan must encode the same requirements, source selectors, and +storage rules. Rust enforces omitted/supplied method availability with trait +bounds; the engine enforces dynamic input presence and validity before key +I/O. Omitted input, explicit zero, and an empty supplied value are distinct. +Go generation can expose convenient signatures using that metadata, but +the guest remains authoritative for runtime requests. + +Implementation proof points: compile-fail required-input calls even for +`C: Default`, reject context for a no-input-only target, preserve optional +extension versus expectation semantics, and execute all nine directional +scenarios through Rust and dynamic plans. Preserve owned-source support +and batching; this trait separation must not add a blanket `Clone` bound. +The final signatures must show where borrowed and owned sources are +supported rather than concealing the existing difference behind the rename. + +## Go JSON implementation and PR #1353 + +The [Go JSON plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +has been updated against merged `next`. Tracking issue #1347 and steps +#1348–#1352 are closed. #1372, #1373 and #1378 implement the catalog, +configured JSON targets, dynamic execution, operation-specific operands, +request-bound opening and generated typed Go frontend; #1402 supplies +current docs and the executable database example. + +The shipped target identifier is `JsonSearch`; Go uses `eql.JSONSearch`, +`JSONSearchQuery`, `JSONSearchLeafQuery` and `JSONEntry`. Do not impose the +proposed `EncryptedJson` glossary as a pre-release rewrite. Preserve the +role distinctions and freeze actual customer names before later Rust renames. +Prepared Go reads retain their cipher and use `Open(ctx, rows)`. The engine +rederives trusted selectors/read keys from the original request. No new +persistent guest-handle protocol is needed to implement that verification. + +Residual requirements must be explicit: SQL children/counts exist but a +parent-bound plaintext child opener does not; selector opening accepts at +most one entry per document group. Use subtree reads for all-match plaintext +retrieval. General scope/row-binding inputs remain unsupported for EQL +targets. The existing column/keyset profile can ship without claiming them. + +The JSON contract now uses randomized projection order and cascade-bound +ciphertexts. Semantic parity and cross-opening are required; raw term byte +equality is meaningful only with controlled test randomness. Go typed JSON +adapters follow `encoding/json`, while raw JSON goes through the strict +engine parser. Do not claim arbitrary Go encoders preserve every input byte. + +## Prior query design: baseline and reconciliation + +**Decision update, 2026-10-25:** ORE does not answer equality in the initial +API, for any plaintext type. Equality uses its declared equality-capable +index. Keep capability selection parameterized by the index, plaintext +encoding/domain and options, so a future explicit exact/injective integer +encoding can opt into equality without changing query syntax or wire +operator identity. Never infer `Eq` merely from ordering support: text and +other lossy encodings cannot establish plaintext equality. A future opt-in +must retain overlap rejection (one selected provider per operator) and +must not silently invalidate existing `(Equality, Ore)` definitions. This +policy does not disable JSON scalar equality, which uses value selectors +rather than ORE/OPE terms. + +Read in full on 2026-10-25: the user-supplied **Stack Encrypt query design**, +dated 2026-10-21, at +`/Users/dan/Downloads/Stack Encrypt query design.md` (654 lines). The source +document was not modified. It is explicitly a discussion draft, based on +an earlier `next` snapshot; its historical code/release claims were not +revalidated as part of this reading. Use it as the query-design baseline, +not as a settled specification or a reason to redesign queries from scratch. + +### Existing proposal to carry forward + +- The caller selects an operator and a typed field/target handle. The field + supplies its own configured index; remove `DefaultIndex` and caller `.via`. +- Each field has at most one index answering each operator. Refuse overlaps + at declaration validation instead of choosing an index by priority. +- Declare operators, index capabilities, and admitted source types once; + derive or verify typed lookup, dynamic lookup, and discovery from that + declaration. Avoid another capability table in EQL. +- Share query derivation and refusals through the same semantic functions. + In particular, tokenize match input once and retain the tokens needed for + probabilistic verification in the query object. +- Retain index/source types in typed field handles. Unsupported operators + are compile errors on the typed path and validated errors dynamically. +- Model analyzer case behavior once in its stages, rather than restating + it in both the match configuration and the operator. + +The draft already proposes the query call +`cipher.query(op::Matches, text).on(&field).await?`, typed +`FieldRef` handles, and a loaded configuration/analyzer owned by the dynamic +plan (`Loaded` / `Arc`). Those are substantive designs to reuse; +they do not yet settle how our output-derived typed factory options are +initialized and shared with convenient `encrypt_into` calls. + +### Reconciliation with this review + +1. **Generate handles from the same output-target declaration.** The query + appendix proposes a plaintext-side `EncryptionPlan` derive and generated + encrypted counterparts. Our agreed starting point is output-side + `EncryptTarget`/`DecryptSource` derives. Recommend emitting typed handles + from that declaration, without separately restating indexes on plaintext. + A source-side convenience derive can later generate the same target. + Hand-written plans remain lower-level composition, not mandatory usage. +2. **Resolve typed factory ownership.** A generic `Match` can fix a policy + identity in its type, but an arbitrary factory can still compute different + values on different calls. Carry a resolved configuration instance through + writes and field handles; settle the owner/initialization API. The draft's + dynamic plan ownership is useful precedent, not proof that all static + factory results are compile-time constants. +3. **Separate portable options from execution types.** JSON's Rust factory + and dynamic deserializer can construct the same `JsonOptions`. The draft's + `Match` and `Match` intentionally have different execution + types to accommodate generic/custom analyzer stages. Preserve shared + validated options and semantic implementations for the portable subset; + do not force all typed analyzers through a serializable stage enum. +4. **Accepted portability scope.** Ship built-in advertised + capabilities in Rust and Go together. Downstream custom Rust stages can + remain explicitly typed-only with `NotPortable` until registered. This + qualifies the draft's blanket "lands in Rust first" rule; it must not + become permission to defer the Go JSON capability under this plan. +5. **Include JSON capabilities before fixing lookup representations.** The + draft's 16-shape dynamic enum covers equality, match, and scalar ordering, + not the new JSON target. Extend operator identity, operand/result shapes, + capability validation, and overlap rules for containment, `AtPath`, and + extraction. Its interim validated index list using one capability table + is a viable starting point; the specific closed enum is not a decision. +6. **Use the agreed context/scope model.** Replace label-only assumptions + with separate ciphertext binding and search scope. Row binding does not + flow into search terms; `PrimaryKey` is context-eligible and `ScopeKey` + is also scope-eligible. Apply the same declared scope to index generation + and queries. Preserve the requirement to return or supply row context + when decrypting selected results. +7. **Accepted validation timing.** All checks independent of keys precede + key resolution/loading, including query-input checks. The draft's possible + named-keyset resolution before those checks is superseded. Reuse the + same validator for Go's `check_query` and execution rather than copying + it or tokenization. Provider facts and cryptographic checks that require + keys remain later stages. Explicit earlier client initialization/keyset + calls are not retroactively prevented by this operation-validation rule. +8. **Apply the user's prototype scope.** Legacy match-option decoding, + deprecation/release-preservation requirements, and frozen Go wire formats + in the older document are not constraints here. Keep new Rust/Go/EQL + consumers aligned. The appendix's `cipher.encrypt(plan, value)` and + plaintext derive are illustrative older proposals, not replacements for + our agreed entry points. + +ORE/Eq is now decided as described above. Retain integration work on +typed field-handle construction/spelling, dynamic +`Index` versus a shared semantic-function boundary, feasibility +of compile-time match-option checks, Go's `check_query`, and ADR updates. +Type-safe stage ordering does not by itself validate runtime numeric values +or string paths; stronger const-check claims require implementation proof. +Use these as specific integration decisions rather than an unbounded query +API task. Coordinate their Go surface with PR #1353. + +## Recommended owner: a reusable typed target definition + +Status: recommendation for review. Use an immutable **target definition** +to own resolved options, prepared indexes/analyzers, context-selection rules, +and operation metadata. The output derive generates this definition's typed +field accessors from the same declaration that generates encryption. It +contains no plaintext, row values, credentials, or selected keyset. A field +handle borrows or shares that definition and retains its plaintext/index +types; it is not an independent name plus a newly constructed default index. + +**Selector naming, 2026-10-25:** the user finds encryption's `using` and +query's `on` confusing and requests considering one name. Recommend +`.using(...)` for both: each selects the definition or field whose declared +policy governs the operation. This spelling is proposed, not yet accepted. +Encryption still accepts the requested target's associated definition type; +query still requires `QueryTarget` on a selected field or +single-value definition. Shared naming does not weaken either bound or +make an entire multi-field record a query target. Older `on` examples in +the supplied query draft describe its earlier spelling. + +Proposed usage, assuming the previously described `EncryptedUser` target: + +```rust +let users = EncryptedUser::definition()?; + +let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; + +let query = cipher + .using(&users.email()) + .query(op::Eq, "alice@example.com") + .keyset(&keyset) + .await?; +``` + +`definition()` obtains the shared resolved definition of that source/target +mapping. Plain `encrypt_into` obtains the same definition internally. It +does not independently invoke the options factory. `.email()` is generated +from the output field declaration, not a hand-maintained second schema. +The method name `definition` is proposed; it names a resolved declaration, +not a builder that users must reconstruct before every operation. + +### Call order and strict typestate + +User requirement, 2026-10-25: use strict typestate for the fluent API. The +user prefers the reading `cipher.using(&field).query(op, input)` and is open +to supporting both call orders. Recommend definition-first as the canonical +documented query form, while supporting operator-first through the same +request states: + +```rust +let a = cipher.using(&users.email()).query(op::Eq, email).await?; +let b = cipher.query(op::Eq, email).using(&users.email()).await?; +``` + +Both are proposed. The current implementation is query-first, followed by +`.on(...)`; this recommendation changes that interface. + +| Starting state | Method | Resulting state | +| --- | --- | --- | +| Cipher | `using(&definition)` | Bound definition, no query selected | +| Bound definition | `query(op, input)` | Query with definition/operator/input known | +| Cipher | `query(op, input)` | Query awaiting a definition | +| Query awaiting a definition | `using(&definition)` | The same query state as the definition-first path | + +The transition that brings the definition and operator/input together has +the `QueryTarget` bound. A definition-first bound handle need not +pretend to know the future operator. Once combined, the index and context +requirements are determined. Only the executable state implements +`IntoFuture`; a bound definition alone, a query missing its definition, or +a query missing required context cannot be awaited. Required context is +completed by a typed context-input transition; optional/no-input cases +start with a valid omitted state. Context completion must not be represented +solely as an unchecked `Option` plus a runtime missing-input error on the +static path. + +Do not expose another `using` transition after a query's definition has +been selected, or another query-selection transition on the request. +Supporting these two orders does not promise every permutation of keyset, +context, and operation selection. Specify those transitions separately, +starting with context supplied after its required type is known. Share +the final query request, validation, and execution implementation rather +than duplicate them per call order. Input-dependent errors such as an +invalid path still need runtime validation; typestate checks completeness +and capability, not arbitrary value semantics. + +The same discipline applies to encryption requests: a fixed-policy +`encrypt_into` has an implicit definition and can be awaited without +`using`; selecting an explicit compatible definition is a one-time +transition before execution. This does not add a new ordinary `encrypt` +receiver or require fixed-policy callers to bind definitions manually. + +Completion tests must include both query orders with identical semantics, +and compile-fail cases for unsupported operators/inputs, premature await, +missing required context, and repeated definition selection. Definition +borrows/shared handles must remain valid for the lifetime of the request. + +### Fixed declarations + +Use lazy, fallible initialization for the fixed definition, retaining each +successful result for reuse. Allow explicit startup initialization via +`EncryptedUser::definition()?`, while the shorthand write initializes it +on demand and returns any error through its awaitable. Factory errors must +not panic or perform key I/O; do not permanently cache initialization +failures. Coordinate concurrent first use so successful initialization is +performed once, and account for reentrant/recursive initialization in the +implementation rather than allowing a deadlock. + +The identity is the complete concrete declaration (including source/target +mapping and generic policy parameters), not merely the ciphertext's output +type. Different instantiations must not share a cache slot by accident. +This is an implementation requirement, not a claim that a static inside a +generic Rust function is automatically per-monomorphization. + +Fixed factories must describe stable application policy; they are not the +interface for reloading environment settings or tenant-specific options. +The cache makes writes and queries in one process share the resolved policy; +it does not prove another process or an older stored row uses that policy. +Per-call context, source values, and keyset choice remain separate inputs. + +### Runtime declarations + +Load/create a distinct definition instance for runtime options. A configured +instance is not stored in the fixed-declaration cache. Use the same type of +definition and field handles, so queries and writes share that instance. + +Proposed Rust spelling for a configurable library target: + +```rust +let payroll = EqlEncryptedJsonTree::configure(options)?; + +let encrypted: EqlEncryptedJsonTree = value + .encrypt_into_with_context(&keyset, column) + .using(&payroll) + .await?; +``` + +Here `using` takes a resolved typed target definition, not a hand-built +plan. This requires a lazy conversion request: select the definition before +building the operation or invoking a default factory. Its await lowers to +the existing `Pending`/batch executor; the shared engine remains unchanged. +The definition's output/source types constrain the conversion. Queries use +the same definition (or its field handles), supplying the required context +and keyset for that call. + +A fixed-policy wrapper such as `EncryptedPayroll` does not expose runtime +override of its declared options. Use the configurable underlying target +for the runtime case, avoiding silent overrides of a fixed policy. Exact +configuration APIs for multi-field runtime definitions are still to be +specified; they must select typed options without introducing another +execution mechanism. + +Go's loaded dynamic plan owns its resolved definitions and generated field +accessors reach those definitions. It has no Rust factory to invoke. The +guest validates and builds the configuration once per loaded declaration, +then writes/queries use the loaded instance. A different runtime policy is +a different instance; it does not mutate one used by in-flight operations. + +### Implementation proof and boundary + +Prove the cache/handle design early with two fixed mappings, multiple generic +instantiations, concurrent first use, initialization failure/retry, and +separate runtime instances. Prove a query and its write use the same options +without calling the factory twice. Preserve source lifetimes, owned-source +support, batch lowering, and no-HTTP/WASM builds. Queries on a field without +the operator must remain compile errors, with dynamic equivalents checked +at declaration/query validation. + +Prefer this generated-definition approach over requiring callers to author +a second plan or putting configuration state in each ciphertext. Plans are +still the lower-level implementation of the definition. Portable option +serialization and index capability discovery remain part of its dynamic +contract; the definition object does not itself need to serialize arbitrary +Rust closures or custom analyzer code. + +## Single-ciphertext representation and plaintext ownership + +Accepted decisions, 2026-10-25: use `OpaqueEncode` / `OpaqueDecode` for +single-ciphertext representation, replacing the working names `EncodeValue` +/ `DecodeValue`. Both direct `encrypt` and target-directed `encrypt_into` +consume their values and take responsibility for owned plaintext cleanup, +including their context-taking variants. These are representation +capabilities; the engine still performs cryptography. `EncryptTarget` / +`DecryptSource` describe how encrypted outputs and layouts compose those +operations. The lifecycle and cleanup scope below are part of the accepted +design. The plaintext cleanup bound is `Zeroize`; implementation remains +to be completed. + +### Owned input is the direct API contract + +`keyset.encrypt(value)` and its context-taking counterpart take an owned +single-value input. No implicit clone and no `Clone` bound. For example, +passing a `String` transfers it; a borrowed `&str` does not transfer the +storage it refers to and must not receive the same cleanup promise. Do not +add a silent borrowed-input fallback to the consuming operation. + +**Decision, 2026-10-25:** require `Zeroize` for plaintext types and use a +library-owned guard. Use the existing ecosystem trait rather than defining +a new cleanup trait or alternative protected-ownership adapter for now. +`Zeroize` supplies the operation; the library must arrange to invoke it on +every supported exit path. A bound alone does not arrange cleanup on drop. + +Acquire protected ownership synchronously when creating the encryption +request, before returning its future/builder. An `async fn` body is not run +until polling, so placing the first guard inside that body would leave an +unpolled request with an ordinarily dropped source. A sync constructor +returning the typed future can establish the guard first. Every subsequent +builder transition moves that guarded ownership rather than making a new +unprotected copy. + +The representation trait may encode through a temporary internal borrow of +the guarded value into a library-owned sensitive sink. Public consumption +does not require giving an encoder unrestricted ownership of the source or +an ordinary growable `Vec` for scratch work. Wipe and release the source +as soon as encoding no longer needs it; keep encoded plaintext guarded until +sealing completes. Preserve the single-leaf representation boundary. + +### Cleanup scope and implementation obligations + +- Cover owned source storage, encoded buffers, codec scratch, and any + library-created plaintext copies. Use established non-elidable wiping + primitives/protected storage rather than `clear()` or ordinary drop alone. +- Cover success, encoding/configuration/KMS/AEAD errors, partially prepared + batches, request cancellation, and an unpolled request being dropped. + Use drop guards for ordinary unwinding too; process abort or a leaked + future is not a destructor guarantee. +- Avoid reallocations that abandon uncleared buffers; growing sensitive + sinks must have a cleanup strategy for old allocations. Prevent plaintext + from escaping in diagnostics, Debug, caches, and instrumentation. +- On decrypt, protect recovered bytes and partial decode state until cleanup + or successful ownership transfer to the caller. Successful returned + plaintext is intentionally retained; its subsequent lifetime is the + caller's responsibility unless the return type itself is protected. +- A Rust move does not prove all physical copies were erased. In particular, + `Copy` inputs leave caller-owned originals usable, and compiler temporaries + may exist. The enforceable API promise concerns supported owned storage + and library-managed artifacts, not caller-retained copies or erasure of + every register/stack remnant. Verify stronger claims with compiler evidence. + +The [zeroize documentation](https://docs.rs/zeroize/latest/zeroize/) +documents drop-based guards, non-elidable wiping, and limits involving +moves, stack copies, and reallocations. The source currently uses +`Protected>` at the AEAD boundary, but that does not establish +end-to-end cleanup of every caller value and intermediate. This discussion +is a design requirement, not a completed source/assembly zeroization audit. + +### Consequences for blobs, layouts, and Go + +`OpaqueBlob` cannot promise source cleanup from `T: Serialize` alone. +Require `T: Serialize + Zeroize` for the proposed Serde-backed blob input; +owned blob decoding requires `T: DeserializeOwned + Zeroize`. Ordinary +application structs may derive `Zeroize`. A +generic Serde implementation can allocate or retain hidden copies, so strong +claims also require a supported codec/implementation contract; wrapping its +final byte buffer does not make arbitrary user serialization code compliant. + +In particular, `T: Zeroize` only supplies cleanup once a `T` exists. If +derived `Deserialize` allocates a `String` field and fails on a later field, +the partially built fields ordinarily drop without invoking `T::zeroize`. +Even adding `ZeroizeOnDrop` to the enclosing struct cannot fix a struct +that was never constructed. Keep this as a concrete decoder design and +acceptance requirement; do not claim the new bound solves partial decoding. + +### Blob codec assessment: Vitamin C frames + +**Priority decision, 2026-10-25:** first define a stable, versioned binary +storage profile. The codec was originally ephemeral transport; stored blobs +need a stronger compatibility commitment. Zeroization and constant-time +implementation improvements can follow, provided they preserve stored +format semantics. Do not defer decisions that affect stored bytes (option +presence, type mapping, versions) or claim timing/cleanup properties not +implemented. The `Zeroize` input bound and ownership direction remain. +Track this action in +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519), created +for this review. It explicitly distinguishes storage stability from the +transport-only rationale in Vitamin C #332. + +**Implemented upstream and in Go, 2026-10-27:** Vitamin C's dedicated +storage profile and Go reader/writer address the format work from #519 +(#520/#526, carried onto upstream main via #544). Stack #1369/#1370 adopt +the Go codec: opaque generated structs call `vcffi.MarshalStorage` and +`UnmarshalStorage`. The profile is versioned (`VCS` plus version), supports +explicit optional values and typed map keys, and is separate from transport. +The transport mapping's `Some(())` ambiguity and text-only map-key rules +must not be reported as defects of this new storage profile. + +Go generated shapes use `vc` tags and `GeneratedVersion2`. A reader rejects +unsupported future blob versions distinctly and retains the old JSON +fallback. Go storage mapping has documented qualifications: nil containers +become empty, supported nonfinite floats round-trip, and unsupported opaque +types are refused by generation. Go pointers and Rust `Option` do not always +write identical bytes; cross-reading and the normative corpus matter. + +Rust workspaces still use crates.io Vitamin C 0.8.0; the final proposed Rust +`OpaqueBlob` facade has not thereby been implemented. Adopt the dedicated +storage entry points when that facade lands rather than repurposing the +transport Serde codec. Go's temporary pseudo-version dependency must move +to released module tags for a final Go release. A new serializer evaluation +is not on the customer critical path. + +Stable bytes do not establish constant-time behavior or complete cleanup. +Retain the remaining guarded-output, temporary-buffer, partial decode and +caller-owned-value work, without changing stored format. JSON tree retains +its own node/envelope encoding; its post-cascade compatibility profile must +be frozen separately from opaque blobs. + +The same consuming contract applies to target-directed `encrypt_into`. +For a structural target, declared passthrough fields intentionally +move into the output; wipe encrypted source fields and temporary copies, +not plaintext fields the caller explicitly chose to retain. Fan-out to +ciphertext and index operations should read a guarded source where possible +instead of requiring a plaintext clone per operation. + +Specify cleanup for dynamic host/guest transport and JSON parsing buffers +as part of the Go implementation. The Rust guest cannot erase Go caller +objects it never owned; host-managed copies need host cleanup and ordinary +Go values do not acquire Rust move semantics. Document that boundary rather +than claiming guest cleanup erases every copy in the calling process. + +Acceptance work: source-cleanup/drop observation on successful and failing +requests, cancellation/unpolled-request tests, nested source/blob cases, +and owned non-Clone inputs. Inspect optimized code for wipe/copy behavior +where necessary; ordinary tests alone cannot prove absence of every residual +copy. Do not infer a current vulnerability or a completed guarantee from +this design discussion. + +## Configured-target implementation sequence and completion criteria + +Customer commitments supersede the earlier Rust/Go front-end lockstep sequence. +The foundational Go JSON slice and opaque storage adoption are now merged. + +1. **Specify the implemented contract.** Capture target options, operator + input/result encodings, grouped reads, errors and the column/keyset + profile in the formal interoperability spec and conformance fixtures. + Keep proposed schema envelopes distinct from the current field-map grammar. +2. **Close actual remaining customer gaps.** Parent-bound child opening, + complex ordered-path declarations if needed, configurable Go match + options (#1112) and the tokenizer/filter pipeline. #1397 already supplies + short whole-word matching through EQL; confirm its accepted leakage and + freeze its index semantics rather than planning it as unimplemented. +3. **Validate and freeze the release candidate.** Run the existing engine, + Go/guest and PostgreSQL suites against the selected bundle. Preserve + numeric/null behavior, cascade tamper checks, wildcard/fallback semantics, + known omission limitations and post-cascade stored vectors. Tests already + landed are not proof this review reran a release build. +4. **Release the dependencies and customer surface.** Replace Vitamin C's + temporary Go pseudo-version pins for a final release; the existing guard + allows prereleases. Opaque storage is already integrated; keep it separate + from the JSON-tree storage profile. Build production guests and matching SQL. +5. **Complete Rust ergonomics and packaging.** Implement public traits, + derives, definition lifecycle and crate moves behind the frozen Go/data + contract. Direct EQL conversion is a format-preserving optimization. + +Use one execution engine throughout. The final Rust facade and package +layout are not gates for the existing Go JSON profile. Do not reimplement +configured JSON options or request-bound reads that have already landed. + +### Required deliverable: dynamic interoperability specification + +**User requirement:** develop a formal specification of the dynamic +interoperability layer during implementation, not as documentation added +after the Rust and Go APIs have diverged. The current examples remain +proposals until reconciled into that specification; adding this deliverable +does not claim the specification has already been written. + +**Recommended format:** a versioned specification package containing +[JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) schemas, +normative semantic and wire-format rules, and executable conformance +fixtures. JSON Schema describes declaration/request/response structure; +it is not the entire behavioral or binary transport specification. Keep +application semantic-type declarations distinct from JSON Schema's own +`type`, `$defs`, and `$ref` vocabulary. + +Required coverage: + +- Definition grammar: built-in and named types, optional `types` section, + inline role wrappers, context selectors, scope eligibility, passthrough + and context storage, targets/options, query capabilities, defaults, and + unknown-field/extension handling. Named ordinary types need no role; + naming a type does not grant scope eligibility. +- Semantic validation: resolve named types and field references, reject + invalid/cyclic references where unsupported, enforce primary-key versus + scope-key capabilities recursively, check source/output compatibility, + and derive write/query scope agreement. Do not imply generic JSON Schema + validators perform the engine's reference resolution or capability checks. +- Operation contracts: input presence and context/scope roles, target and + definition identity, results, stable error codes and locations without + plaintext values, validation stages, and which failures precede key I/O. +- Wire mapping: JSON and frame representations, integer widths/ranges and + lossless numeric transport, bytes/text encodings, missing versus null, + context ordering/grouping, descriptor/AAD/index-scope lowering, and + normative references to the ciphertext/SQL envelope and tag extraction. + Specify the host/guest ABI or reference its versioned specification; + JSON authoring syntax must not silently redefine the binary transport. +- Versioning: explicit protocol/schema identifiers, supported-version and + capability handling, and rejection of unsupported features. Define future + evolution without adding compatibility obligations for the prototype. + +Conformance fixtures include accepted and rejected definitions/requests, +expected error codes/paths, normalized definitions, and deterministic +context/encoding vectors. Exercise the same fixtures through Rust and the +Go guest path, plus SQL extraction where relevant. Randomized ciphertext +comparisons use cross-decryption and query behavior, not byte equality. +Schema validation is one layer of the suite; semantic rejection tests are +required even for structurally valid JSON. + +Keep a single maintained contract: prefer generating structural schemas and +capability metadata from the shared declaration models/catalog where that +faithfully represents the specified grammar, with committed artifacts and +CI drift checks. Review generated schemas against normative rules rather +than treating Serde derives as a complete specification. Do not introduce +OpenAPI or a new transport solely to document this non-HTTP interface. + +## Coverage of the complete review + +The configured-target work above addresses issue 13 in depth. The full API +redesign also requires the following work. Each row identifies a concrete +deliverable; recommendations do not silently settle the open decisions. + +| Work area | Review issues | Required deliverable | +| --- | --- | --- | +| Public methods, traits, and ownership | 1, 3, 5, 9, 10, 14 | One signature/import/receiver map, operation-trait design, and dependency migration | +| Context contract | 1, 3, 4, 11, 12, 14 | Scalar and record context rules, explicit optionality, and matching dynamic representation | +| Ciphertext persistence | 2, 8 | Opaque scalar output, canonical bytes, Serde/Base64 behavior, and a defined boundary with structural trees | +| Whole-value wrapper | 8 | Codec, authenticated format version, decode contract, and bounded timing claims | +| Mixed records and collections | 7, 8, 11 | Passthrough/renaming/context-field derive support and retained collection batching | +| Configured targets and queries | 13 | Typed Rust factories, dynamic options, configuration lifetime, JSON conversion and Go coverage | +| Names, integration, and release | 5, 6, 9, 10, 13, 14 | Final glossary, coordinated consumer updates, format fixtures, and executable public examples | +| Parallel Go JSON work | 15 | Reconcile PR #1353 and tasks #1348–#1352 with this contract; reuse compatible implementation work | +| Prior query design | 16 | Reuse the supplied field-owned query model; reconcile handles, configuration ownership, JSON capabilities, and portability | +| Consistent definition selector | 17 | Choose one method name for encrypt and query; recommend `using` with operation-specific bounds | +| Single-ciphertext encoding and cleanup | 18 | `OpaqueEncode`/`OpaqueDecode`, consuming encrypt and encrypt_into, guarded source/buffer lifetimes, and explicit cleanup scope | + +### A. Consolidate the public Rust method and trait structure + +This is the Rust-facade workstream. Under the latest customer ordering, do +only the engine/contract changes needed by Go before its release; defer +standalone Rust ergonomics and trait relocation. The sections below organize +the full review scope, not the customer critical path. + +Keep the agreed ordinary keyset receiver and `EncryptInto` conversion forms. +Use the accepted destination-directed decrypt pair. Write +out all signatures, imports, bounds, and return types before renaming traits. +Retain ordinary keyset `decrypt` / `decrypt_with_context` for opaque +ciphertexts, and remove redundant prototype conveniences without +compatibility shims. + +Separate plaintext representation, target operation declarations, and +execution conveniences explicitly. Resolve why `EncryptFrom` declares +`encryption()` while `encrypt_from` lives on `EncryptInto`, and why +`DecryptFrom` supplies methods while `DecryptInto` describes opening. A +rename without an ownership/signature map leaves the discovery problem. + +Recommendation: Stack owns the public value-encryption model; Vitamin C +retains general primitives. Specify the dependency boundary before moving +code, retaining one canonical trait identity and the no-HTTP/WASM builds. +Reassess the original Vitamin C cipher-trait mismatch in that architecture: +do not add a public trait implementation merely for symmetry if it cannot +carry structured KMS context or support its advertised operations. + +Completion: representative scalar, blob, typed record, searchable record, +and JSON calls compile with the documented imports, including generic +callers. The method inventory has no unexplained duplicate execution path. +Compile-fail tests cover genuinely unsupported trait capabilities. + +### B. Turn the context scenarios into a normative contract + +The nine scenarios in the context document are the acceptance matrix, not +optional examples. Cover scalar static context plus caller extension, +source-field selection, caller input, output storage, and decrypt recovery +or expected-context checks. Preserve the accepted `field = "id"` shorthand +as selection on both inputs, without implicit copying. + +Fix composition order and grouping once: literal/field base, composite row +context, and any caller extension. Specify the exact descriptor and AAD +encoding for each case. An expected context checks stored context; it is +not another extension. Never need to open ciphertext to discover the +context required to open that ciphertext. + +Recommendation: represent no input, required input, optional extension, +and optional expectation as explicit capabilities. Do not infer optionality +from the application's `Default` implementation. Settle the wrapper or +trait mechanism alongside the public signatures. Resolve the existing +empty-context mismatch between ordinary scalar encryption and ciphertext +targets without weakening nonempty-context requirements for search targets. + +Give each source/storage/composition rule a dynamic representation and Go +authoring path. Current EQL identifier constraints must be reconciled with +row/caller context explicitly, not bypassed or silently dropped. + +**Next discussion — recommendation, not yet a decision:** distinguish +ciphertext binding from search scope. A row ID may bind a ciphertext to its +record without making every index term row-specific. The target definition +should explicitly describe both scopes; query construction uses the same +declared search scope as index generation. For example, a searchable email +may bind its ciphertext to `(users.email, tenant_id, user_id)` while its +equality index is scoped to `(users.email, tenant_id)`, allowing a query +before the user ID is known. Do not silently remove components from one +generic context argument. Index-key derivation/KMS descriptors must respect +this separation too; changing only leaf AAD is not a complete design. +Finalize the conceptual split before choosing attribute names or exact +tuple encodings, and represent it in the dynamic definition as well. + +**User direction:** allow row-bound ciphertext and document that reads must +provide the row ID to decrypt. Search scope should cover multiple rows, +such as a tenant; automatically including a row ID defeats ordinary search. + +**Misuse-resistant API recommendation (not yet accepted):** + +- Give existing record `context(field = "id")` and its directional variants + ciphertext-binding semantics. Do not implicitly feed those selected fields + or caller extensions into index keys or terms. This is a proposed + clarification for searchable targets; older examples did not distinguish + the roles, and must be updated together before implementation. +- Declare search scope separately on the shared target definition. A named + application scope input (for example `TenantScope`) can come from operation + configuration/caller input. The capability proposal below also permits + record-field selectors when their types explicitly qualify as scopes. + The same typed scope is required for both + writing index terms and constructing queries. A tenant-bound cipher may + supply it only through an explicit declared mapping; never guess it from + unrelated metadata. +- A binding input and a search-scope input are distinct capabilities even + when their underlying values have the same Rust type. Do not provide + blanket implicit conversion between the roles. If one operation needs both + caller inputs, use named members in its typed input rather than one + unlabelled value routed to both operations. +- Missing required scope prevents awaiting a Rust request; dynamic/Go + validation rejects it before key I/O. A default column-wide scope is not + substituted when a tenant scope is missing. Scope-free searches use an + explicitly declared column-wide policy, with column identity still bound. +- The effective ciphertext binding should include the declared search scope + as well as record binding, so changing a supplied tenant also prevents + opening the ciphertext. Index derivation sees column identity plus search + scope only. Query requests never require record binding input. +- Encrypted output must expose whatever the decrypt selector needs or + require it from the caller. Record-projection examples must select the ID; + there is no decrypt fallback which drops row binding. Selecting a field + as context does not implicitly store that field. + +Working declaration spelling: +`#[stash(context(field = "id"), search_scope(caller = TenantScope))]`. +The spelling is illustrative; it preserves the agreed directional-context +shorthand and gives search an explicitly separate role. Dynamic definitions +must preserve this separation rather than providing a shared context bag. + +This prevents accidental propagation of row IDs into search terms; it +cannot prove the application chose a scope containing multiple rows. A user +can deliberately put a row ID inside `TenantScope`. Do not claim otherwise, +or attempt to infer semantics from field names such as `id`. Use explicit +types/declarations and documentation for that remaining application choice. + +#### Scope capability and role wrappers — assessment + +**Proposal under discussion:** use both ideas together. Scope is the subset +of context permitted for search terms; semantic wrappers provide positive +trait implementations identifying that subset. Working capability names: +`CiphertextContext` and `SearchScope: CiphertextContext`. These names describe +eligibility, not source selection, optionality, or encoding. Integrate with +the existing context representation rather than introducing a competing +serializer. The previously agreed `ContextInput` capabilities still describe +whether the caller must supply a value. + +For example, `PrimaryKey` implements ciphertext-context eligibility but +not search-scope eligibility. `ScopeKey` implements both, subject to +the inner value's supported context encoding. Generic parameters are Rust +types (`PrimaryKey`); `PrimaryKey(42)` is a value constructor. Prefer +`ScopeKey` as the name for the second wrapper; the user accepted this +name in preference to `Collection`, which can read as a container. +These wrappers need the appropriate `Zeroize` support when they are owned +plaintext fields, consistent with the existing ownership decision. + +Crucially, neither wrapper automatically adds the field to context or scope, +stores it in encrypted output, or changes the plaintext's encryption layout. +The target declaration still selects sources. An illustrative declaration +is `context(field = "id"), search_scope(field = "tenant_id")`. With +`id: PrimaryKey` and `tenant_id: ScopeKey`, the selection is +valid; selecting `id` as search scope is a compile-time error. This makes +typed field selection a reasonable alternative to the earlier recommendation +to accept only caller-supplied search scope. + +Misuse-resistance requirements: + +- No blanket `SearchScope` implementation for every context, serializable + type, raw number, or raw string. Scope use is an explicit classification. +- Composite scopes qualify only when every component is a scope. A tuple + containing a primary key cannot become a scope merely by tuple composition. + Context composites may contain either category. +- Do not forward scope eligibility through `Deref`, `AsRef`, or implicit + conversions that erase a primary-key wrapper. Typed target/query handles + retain the definition's concrete scope type, rather than accepting any + type implementing `SearchScope` interchangeably. Distinct application + newtypes can distinguish tenant and project scopes with equal wire types. +- Permit explicit application opt-in through a documented trait/derive + contract. This is a semantic capability, not an unsafe-memory contract. + Sealing every implementation cannot prove that a value groups multiple + records: a caller can still explicitly rewrap a row identifier as a scope. + Guarantee rejection of accidental role mixing, not business semantics. + +Dynamic definitions must retain field roles and context-versus-scope +selection. Validate those roles before execution: an explicitly declared +primary-key field is not scope-eligible. Plain JSON numbers carry no such +role by themselves; do not infer it from the field name or wire encoding. +Generated Go types can represent the distinction, while the engine validates +dynamic declarations. Rust and dynamic lowering must produce the same +canonical context/scope values; never use Rust type names as wire domains. +Scope classification is not evidence of authorization to access a tenant. + +Acceptance work: valid typed field/caller scopes, rejected primary-key and +mixed-tuple scopes, no automatic selection of marked fields, exact scope +type agreement between write/query, and equivalent dynamic validation. + +#### Dynamic scope roles and validation + +**Recommended design:** declare semantic types and source selectors in the +definition, validate once on load, and supply values separately per operation. +Do not let an individual request assert `scope_key` on an arbitrary value +to override the loaded definition. A definition author can explicitly +classify values, just as a Rust author can construct wrappers; this is +misuse resistance, not authorization or proof of multi-row cardinality. + +**Accepted clarification:** `types` is optional; it holds reusable named +types. Fields may use built-ins or inline role wrappers. Ordinary built-in +or named values need no `kind`. Supported primitives remain eligible for +ciphertext context, but search eligibility requires explicit `scope_key` +classification (or another explicitly supported scope-capable type). + +Illustrative declaration fragment (proposed schema, not today's wire API): + +```json +{ + "types": { + "UserId": { "kind": "primary_key", "value": "u64" }, + "TenantId": { "kind": "scope_key", "value": "u64" } + }, + "fields": { + "id": { "type": "UserId", "passthrough": true }, + "tenant_id": { "type": "TenantId", "passthrough": true } + }, + "context": { "field": "id" }, + "search_scope": { "field": "tenant_id" } +} +``` + +This is the context-related portion of a record target definition; encrypted +email targets and their search options are declared separately as usual. +The shared `field` shorthand selects plaintext fields on encrypt and stored +fields on decrypt; it does not imply a field exists in a query request. +For a query on the derived email handle, the definition exposes a required +`TenantId` scope argument. No `UserId` argument is required or accepted as +an alternative scope. A named/composite scope uses the same ordered schema +on the write and query paths, regardless of their different source selectors. + +Definition loading resolves field references and source/output types, +computes the context/scope capabilities, verifies all composite components, +and rejects a primary-key search selector before key I/O. It also verifies +decrypt selectors are available from stored fields or declared caller input. +Compile the validated definition into an immutable handle shared by encrypt, +decrypt and query operations. No second query-specific scope policy. + +At execution validate presence, nominal type when supplied by a typed handle, +physical representation, range, and composite shape against the selected +definition. Raw wire values obtain their expected semantic type from the +definition; they cannot prove their application provenance. Unknown role +overrides or mismatched nominal value handles are rejected. No fallback to +unscoped querying when a required scope is absent. When JSON is the authoring +or request format, preserve full-width integers through lossless parsing; +the frame transport already distinguishes supported integer widths. + +Recommended Go generation uses distinct named ID types and query methods +with the required scope argument (`TenantID` rather than `uint64` or an +untyped context map). Generated wrappers remain conveniences over the same +validated runtime definition; ungenerated FFI callers receive runtime errors +for invalid declarations/requests. Do not assert Go shares Rust's ownership +or trait-bound guarantees. + +Lower the chosen scope and binding through the existing canonical context +machinery, retaining scalar widths, ordering and grouping. Current dynamic +`context.rs` proves parity between runtime `ContextPiece` trees and Rust +context derivations, but has no primary-key/scope-role distinction. Add role +validation before lowering erases those distinctions, and keep the separate +resolved scope and binding available to their respective operations. +Do not persist role labels in every row just to recover them at runtime; +definition identity and deployment remain explicit concerns of loading the +same policy. Add Rust/Go/dynamic fixtures for valid scopes, primary-key +refusals, nested scopes, missing inputs and identical derived terms/AAD. + +Completion: all nine cases execute through Rust and dynamic declarations; +required input is enforced, altered/wrong context is refused, and existing +context encodings agree across all new implementations. + +### C. Finish the ciphertext and whole-value storage contracts + +Accepted direction: a single-value plaintext encoding produces one opaque +`Ciphertext`. A normal application struct uses `OpaqueBlob` to encode +and seal it as one value, or an explicit target for field-level encryption; +Serde alone does not imply automatic blob serialization. A custom type can +explicitly implement the single-value encoding capability. The wrapper +name and detailed persistence contract remain proposals. +Recommend canonical byte parsing/encoding, human-readable Serde as Base64 +of those bytes, binary Serde as +bytes, and safe Debug behavior. Parsing an envelope does not authenticate +it; authentication occurs during decrypt. + +The existing encrypted-output transcoder can implement the adapter: +`Node::Single(SealedValue)` is consumed by `Visitor::sealed` and moved into +the opaque wrapper. Keep the sealed value internally rather than forcing +a byte serialization at construction. There is no new encryption or KMS +request. On decrypt the wrapper delegates to the same leaf-opening path, +using a single-node wrapper if the internal API requires it. + +Do not convert maps, sequences, passthrough nodes, or sealed empty-container +markers into this type merely because they have a byte representation. +Serializing a whole encrypted tree retains its multiple ciphertexts and +authentication boundaries; it does not produce single-value encryption. +The leaf visitor's unsupported-shape errors are a defensive check, not the +desired public discovery mechanism. Constrain ordinary encryption to +single-value encodable inputs so unsupported structural inputs are rejected +before encryption, statically in Rust where possible and during dynamic +validation. Settle the exact plaintext capability with the representation +traits; do not discover an unsupported shape only after minting keys. + +Keep generic structural encryption and searchable JSON explicit. Specify +whether structural-tree persistence is supported and under what stable +format. Any protocol changes must be coordinated with their consumers. + +#### Typed encodings and extractable ZeroKMS tags + +**Requirements added, 2026-10-25:** ciphertext text conversion returns the +existing concrete encoding wrappers, not `String`. The user also wants to +explore selecting an encoded ciphertext as a field's output type. The +ZeroKMS key tag must be easily extractable from stored ciphertext, including +in PL/pgSQL, to support a separately designed refinable-tag mechanism. +The storage/authentication contract is not yet locked in. + +Current `eql-bindings::encoding::{Base64Text, Base85Text}` validates canonical +padded standard Base64 and the crate's RFC 1924 Base85 encoding respectively. +Their private strings, encoding constructors and validating deserializers +make decoding infallible. They guarantee the text encoding, not that the +bytes form a ciphertext. Reuse these types and add ordinary fallible text +parsing conveniences rather than requiring Serde to construct them from text. + +Recommended ownership: a small Vitamin C encoding component, re-exported by +Stack Encrypt and EQL. These byte/text utilities do not depend on encryption, +SQL or EQL layouts. Keep TypeScript/schema integration optional in the owning +crate, accounting for Rust's orphan rules when moving the existing trait +implementations. Do not introduce a Stack Encrypt dependency on EQL merely +to reuse its text wrappers. Final component placement remains a proposal. + +Proposed ciphertext methods return `Base64Text` / `Base85Text` from +`to_base64` / `to_base85`; corresponding constructors accept the typed text +and still return a ciphertext parse error when its decoded envelope is +invalid. Encoding validation, envelope parsing and authentication are three +different guarantees. Use one canonical byte envelope under both encodings. + +**Accepted, 2026-10-25:** support explicit encoded ciphertext field targets. +Use a ciphertext-specific storage wrapper +(working names `Base64Ciphertext` / `Base85Ciphertext`) over making every +`Base64Text` an encryption target. The wrapper validates both text and +envelope, serializes as the text string, and composes with `EncryptTarget` +and `DecryptSource`: seal once, then encode; parse before opening. These +names remain provisional until the final naming pass; the first-class +encoded target capability is settled. A structural output +can mix `Ciphertext` and encoded ciphertext fields without additional KMS +calls. Do not silently change the prescribed representation of EQL targets. + +The current `SealedValue` v1 envelope already uses a simple prefix: +`version(1) | keyset_id(16) | iv(16) | tag_len(2, LE) | tag | local_ciphertext`. +Its ZeroKMS tag begins at byte offset 35, with length at offsets 33–34. +This is evidence that extraction can be simple, not a commitment to keep +those offsets. Keep the chosen version's layout documented and parseable +without the plaintext codec, key retrieval, or decryption. Prefer a checked +borrowed header/envelope view and an accessor explicitly named `key_tag` +over decomposing the whole ciphertext or conflating it with the AEAD tag. +Extraction does not authenticate the returned tag. + +Ship a matching SQL/PL/pgSQL extraction helper and Rust/Go/SQL fixtures with +the storage format. Check versions, lengths and truncation before slicing. +For text storage prefer Base64 when SQL must inspect the header: +[PostgreSQL](https://www.postgresql.org/docs/current/functions-binarystring.html) +has built-in Base64 decoding and byte slicing, but not Base85 decoding. +If SQL extraction from a Base85 envelope is supported, supply an explicit +decoder; do not leave it as an assumed capability. PostgreSQL Base64 output +also inserts line breaks, unlike the canonical Rust wrapper: normalize SQL +output or validate it against the same canonical rules in cross-language +tests. JSON-tree storage needs its own header adapter: its shared key header +is distinct from each node's ciphertext and parent tag. + +The proposed refinement uses the ZeroKMS tag as an HMAC key and replaces it +with a derived tag. **User clarification:** at decryption, ZeroKMS uses the +refined tag for policy checks and logging, and returns the original tag. +The decrypt path uses that returned original tag to reconstruct leaf AAD. +This resolves the conceptual conflict with binding the original tag into +AEAD: the stored/request tag and the authentication tag input need not be +identical after refinement. Keep those roles explicit in the KMS response +and decrypt implementation. Tag extraction returns the tag actually stored +in the envelope, which may be refined. Protocol details remain deferred; +this is the intended behavior, not a claim that the current client/engine +already implements it. Add refinement/decrypt integration coverage when +that work lands. + +**Decision, 2026-10-25:** direct `encrypt` handles one single-value input; +`EncryptInto` handles collection composition and batching. This supersedes +the earlier open question about automatic collection lifting. The agreed +public examples are: + +```rust +let scalar: Ciphertext = keyset.encrypt(1_i32).await?; + +let batch: Vec = vec![1_i32, 3_i32] + .encrypt_into(&keyset) + .await?; + +let blob: Ciphertext = keyset + .encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])) + .await?; +``` + +The batch produces two independent ciphertexts through the shared batch +executor; the blob encodes and seals the complete collection as one value. +Do not add implicit collection handling to direct `encrypt`, or a separate +batch entry point merely to provide functionality already covered by +`EncryptInto`. Context-taking variants follow the same division. + +Acceptance checks: reject direct `encrypt(Vec)` before execution; +support context-free `Vec` to `Vec`; preserve target-directed +collection opening and batching; and recover the blob as a whole collection. +Existing collection composition is retained, while opaque `Ciphertext` and +its no-context target support are new work. + +The current prototype treats `Vec` as one byte value, but that is not a +compatibility constraint. The latest recommendation is uniform collection +behavior: `Vec` to `Vec` encrypts individual bytes, while +`OpaqueBlob>` encodes the complete vector as one value. Do not add +`Vec` to `Ciphertext` conversion or a bytes wrapper solely to retain +prototype behavior. A distinct raw-byte capability would need its own use +case. The blob codec need not reproduce the prototype's raw-byte encoding. + +For `OpaqueBlob`, select the library-owned codec, authenticated codec/type +discriminator and version, schema-evolution policy, error behavior, and +supported source/decoded types. Authenticate before deserializing plaintext. +Do not silently reinterpret an ordinary byte ciphertext as a different +encoding. Preserve an explicit wrapper rather than implicit Serde fallback. + +The constant-time question needs a concrete conclusion: assess `serdect`, +the codec, both sides of Serde (value traits and serializer/deserializer), +length/shape disclosure, and permitted custom implementations. If arbitrary +Serde types remain supported, do not promise end-to-end constant-time +serialization. Select a restricted profile separately if that property is +required; it cannot be inferred from the wrapper or its encryption primitive. + +Completion: scalar and blob examples persist and reload through a fresh +cipher, reject invalid encodings, and verify fixtures for the selected +format. The supported timing contract and codec version are documented. + +### D. Complete record derives and preserve collection composition + +Implement explicit passthrough and source-field renaming, independently of +context selection and storage. Generate decrypt reconstruction for the +same record. Support nested targets with and without configuration, and +report incompatible context or source types at their declarations. + +Keep the collection behavior already present: `Vec` to `Vec` composes +the target operations into one logical batch, rather than awaiting every +row separately. Preserve document/leaf key granularity; one logical batch +does not promise one physical ZeroKMS request beyond backend limits. + +Completion: the public mixed-record and searchable-record examples run, +including batch encrypt/decrypt, nested targets, renamed passthrough fields, +and stored context. Dynamic/Go declarations produce equivalent semantics. +A plaintext-side derive generating output types is deferred convenience, +not a prerequisite for these workflows. + +### E. Finish the configured policy's lifetime and query surface + +Use the prior query-design baseline and reconciliation above. The query +model is not an empty design space: field-owned index selection, typed +handles, and shared derivation already have a detailed proposal. + +The typed-factory direction is established. The target-definition section +above now proposes its owner: a shared immutable fixed definition, or an +explicit runtime instance, with generated typed field handles. Review that +proposal and prove its initialization/lifetime behavior before treating the +once-only reuse claim as implemented. Multi-field runtime configuration and +the exact generated signatures still require specification. + +Write complete static and Go examples for equality, JSON containment, +ordered paths, and extraction, all using the write policy. Specify query +results and unsupported-operation errors. Keep those examples free of +mandatory hand-written plans. Preserve direct typed EQL conversion and +catalog/guest support as deliverables of the configured-target sequence. + +### F. Make naming and coordinated integration part of completion + +Use the [packaging glossary](2026-10-25-stack-encrypt-packaging.md) for the +proposed role-based JSON names, retained `OpaqueBlob`/`Ciphertext` spellings, +shared format crate and primitive/application split. Retain accepted +declaration/convenience trait names. Recommend renaming the plan builder's +output-selection method to `target`; distinguish it from executing +`EncryptInto`. New spellings and crate moves still require review. + +Remove replaced APIs directly and update EQL, Go, and guest consumers +together. There is no requirement to read prototype ciphertext or preserve +old imports. Specify the selected encoding and cryptographic semantics, +and establish fixtures for that contract going forward. + +Completion: all public examples are executable tests, static/dynamic +cross-opening and querying pass against the selected storage format, batching +checks pass, and the guest builds remain supported. No new API is described +as implemented until its Rust and Go acceptance criteria are met. +These are full redesign completion criteria. The earlier Go release uses +the facade-independent Go, engine and storage gates listed above. + +## Recommended order for the remaining design decisions + +### Decision checkpoint, 2026-10-25 + +Settled direction: keyset-directed single-value encryption; target-directed +collection batching; explicit `OpaqueBlob`; conversion methods separate +from `EncryptTarget`/`DecryptSource` declarations; explicit context-input +capabilities instead of `Default`; typed Rust options factories and dynamic +deserialization into the same options; static and dynamic support together; +shared typed definitions and field handles; a consistent `using` selector +with strict typestate and definition-first query spelling; `OpaqueEncode` +/ `OpaqueDecode`; consuming encrypt and encrypt_into with guarded cleanup; +ordinary keyset decrypt alongside the destination-directed decrypt pair, +both with context variants and ordinary owned `T: Zeroize` results; +separate ciphertext context and search scope with `PrimaryKey`/`ScopeKey` +eligibility; definition-time role validation in the dynamic interface, +optional named-type declarations and explicit scope eligibility; and a +versioned interoperability specification with shared conformance coverage; +and no backward compatibility with the prototype. Do not reopen these +choices merely because older examples or work-area descriptions retain +provisional wording. + +The remaining product/API decisions are narrower. Distinguish them from +specification and implementation obligations: + +| Remaining question | Decision required | +| --- | --- | +| JSON contract completion | Main Go surface is implemented. Close parent-bound child opening and formal conformance gaps; use current cascade/random-order semantics. Do not reopen operator design or rename shipped Go APIs for cosmetic consistency. | +| Key selection and optional provider policy | Review plaintext-derived keyset selection, client-scoped routed execution, and ZeroKMS lock context as optional provider capabilities. Assess the neutral data/index-key boundary and the intentional non-use of Vitamin C AsyncCipher in the provider-capabilities companion. | +| Customer storage profiles | Go opaque storage is integrated. Replace temporary Go dependency pins for final release and freeze post-cascade JSON/search profiles. Rust blob conveniences and crate relocation may follow. | +| Configurable match and short input | #1397 implements deterministic short-word terms in the same Bloom filter. Broader configurable Go tokenizers/filters remain, including #1112. Reconcile the earlier privacy concern explicitly; merging support is not proof of unchanged leakage. | +| Packaging and names — review in progress | Review the packaging companion's shared format crate, Stack-owned value execution, primitive-only Vitamin C boundary, AsyncCipher retirement, ZeroKMS client rename and role-based JSON names. Encoded ciphertext field targets themselves are accepted. The accepted trait names, `ScopeKey`, and receiver choices are not open again. | + +Next complete the supported-contract specification and remaining customer +configuration/read gaps, then validate the candidate and resolve final-release +pins. The original JSON vertical slice is already implemented. Packaging +remains deferred; keep initial ORE equality policy unchanged. + +Specification/implementation work still includes exact context/scope grouping +and encoding, dynamic grammar and version identifiers, error codes, binary +envelope fields, Rust/Go/SQL fixtures, and the refined-versus-original tag +flow. Record candidate choices in the formal specification for review; do +not treat every byte offset or adapter signature as a new architectural +decision. The initial persistence recommendation remains ordinary Serde +composition for explicit layouts, without a new arbitrary-tree wire format. + +Cleanup implementation still needs guarded encoder sinks, partial decode +handling, cancellation/drop checks, and an explicit drop-guarded result +adapter. Under the user's latest sequencing decision, implementation +improvements may follow the stable format; preserve the stored contract and +document actual guarantees instead of claiming the eventual goals are +already satisfied. No completed constant-time/zeroization audit is implied. + +The shared-definition caching, initialization error/retry behavior, generic +identity, and request lifetimes still need concrete signatures and proof. +That is implementation design within the recorded ownership model, not a +reason to reopen the question of whether writes and queries share a policy. + +Passthrough implementation, transcoder adapters, collection composition, +trait relocation, documentation updates, and acceptance tests remain work +items. They are not reasons to characterize the agreed API direction as +undecided. No backward-compatibility phase is needed for the prototype; +the customer release establishes a new Go and persisted-data commitment. diff --git a/docs/plans/2026-10-24-stack-encrypt-api-review.md b/docs/plans/2026-10-24-stack-encrypt-api-review.md new file mode 100644 index 000000000..4fa7e186d --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-api-review.md @@ -0,0 +1,1176 @@ +# Stack Encrypt API review + +Started: 2026-10-24. Scope: the current Rust `stack-encrypt` API, starting +with encrypting one `i32` into opaque ciphertext and storing the result. + +Companion: [complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md), +an API design draft covering encryption, persistence, decryption, and queries. +The [directional context examples](2026-10-24-stack-encrypt-context-examples.md) +explore context sources and storage using Serde-inspired attribute syntax. +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records +requirements, recommended decisions, sequencing, and completion criteria. + +This is a running issue log. Add a numbered section as each issue is found; +keep observations, proposed changes, and unresolved decisions distinct. +All issues below remain unimplemented. Individual decisions are recorded +within each issue; proposed APIs are illustrative, not implemented. + +**User clarification, 2026-10-25:** this API has never shipped beyond an +early prototype. Earlier suggestions to preserve old encodings or public +imports are superseded: backward compatibility and prototype data migration +are not requirements. New Rust, Go, guest, and EQL implementations must +agree with each other, but need not retain the prototype's representations. + +## 1. Cipher trait mismatch and explicit context ergonomics + +**Status:** Open. + +### Current behavior + +`KeysetCipher::encrypt` is an inherent method, not a trait method: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let ciphertext = keyset.encrypt(42_i32, "users/age").await?; +``` + +Vitamin C (`vitaminc-aead` 0.8.0) provides both `Cipher` and `AsyncCipher`. +Stack implements the synchronous `Cipher` trait on internal adapters after +fetching keys: `MintedKeys` seals leaves and `BoundKeys` opens them. Neither +public cipher implements `AsyncCipher`, so callers cannot use +`KeysetCipher` directly with generic Vitamin C code requiring that trait. + +Each internal adapter implements only half of `Cipher`: `MintedKeys::open` +and `BoundKeys::seal` always return errors. + +### Why the mismatch exists + +`AsyncCipher` already supports batching through `seal_many` and `open_many`; +batching alone does not explain the separate API. Its methods receive each +leaf's derived AAD, while ZeroKMS needs the original structured context's +descriptor. That original context is not carried by the derived leaf AAD. +Stack's pending-operation layer also combines key requests across documents, +fields, and rows, beyond the trait's per-document batch interface. + +These constraints need to survive any trait integration. The current +separation is deliberate, but leaves an interoperability gap and internal +implementations with unsupported operations. + +### Requested API direction + +Requiring an empty tuple for the common case of no application AAD is +unnecessary ceremony. Use distinct methods, following the convention used +elsewhere: + +```rust +keyset.encrypt(42_i32).await?; +keyset.encrypt_with_context(42_i32, "users/age").await?; +``` + +The no-context method should select the same empty context currently +represented by `()`. This distinction should exist even if `KeysetCipher` +continues to use inherent methods rather than implementing a Vitamin C trait. + +### Open design questions + +- Can a context-bound adapter implement `AsyncCipher`, or should Vitamin C's + interface preserve the original context explicitly? +- How does that integration preserve batching across multiple documents? +- Should sealing and opening be separate trait capabilities, avoiding methods + that are present but always fail? +- Should decryption expose the corresponding `decrypt` / + `decrypt_with_context` distinction? + +**Source:** [cipher implementation and rationale](../../packages/stack-encrypt/src/cipher.rs) +(`KeysetCipher::encrypt`, `MintedKeys`, `BoundKeys`, and the module's batching +discussion). Vitamin C trait definitions are in `vitaminc-aead` 0.8.0, +`src/seal.rs`. + +## 2. The Node / SealedValue boundary complicates opaque ciphertext storage + +**Status:** Open. + +### Current behavior + +`KeysetCipher::encrypt` returns `StackCipherText`, an alias for Vitamin C's +`Node`. Encrypting an `i32` produces `Single(SealedValue)`, but +the return type also admits sequences, maps, empty containers, and passthrough +nodes. It does not retain the plaintext type as a type parameter. + +| Capability | `StackCipherText` tree | `SealedValue` leaf | +| --- | --- | --- | +| Debug printing (`{:?}`) | Yes | Yes | +| Serde serialization/deserialization | No | Yes | +| Canonical byte encoding | None | `to_bytes()` / `from_bytes()` | +| Direct Base64 conversion | None | None; encode the canonical bytes separately | + +The leaf's canonical bytes include its keyset id, key retrieval metadata, +and local ciphertext. That versioned byte format is the storage commitment; +the leaf's Serde representation is not the same compatibility commitment. + +### Caller impact + +Even the scalar storage path requires exposing the container representation: + +```rust +let ciphertext = keyset.encrypt(42_i32, ()).await?; +let StackCipherText::Single(leaf) = ciphertext else { + unreachable!("encrypting an i32 produces a single leaf"); +}; +let bytes = leaf.to_bytes(); + +// After loading the bytes: +let leaf = SealedValue::from_bytes(&bytes)?; +let ciphertext = StackCipherText::Single(leaf); +let value: i32 = cipher.decrypt(ciphertext, ()).await?; +``` + +The caller must handle a shape mismatch that cannot occur for this input, +extract the leaf, and reconstruct the node to decrypt. There is no direct +`ciphertext.to_bytes()` or `serde_json::to_string(&ciphertext)` path. Base64 +adds a separate encoding step. Collections additionally require the caller +to choose a representation preserving the tree structure and node variants. + +Stored bytes are not sufficient by themselves to decrypt: the caller still +needs authorized ZeroKMS access and the same application context. The storage +API should make that requirement clear without exposing tree internals for +the simple scalar case. + +### Proposed direction and open decisions + +**Subsequent decision:** ordinary encryption should produce one opaque +ciphertext; choose structural encryption through an explicit target, or +encode a composite plaintext first with `OpaqueBlob`. The +[implementation plan](2026-10-24-stack-encrypt-api-plan.md) recommends +adapting a native single sealed leaf through the existing transcoder, +without serializing an entire encrypted tree or re-encrypting it. + +**Collection decision, 2026-10-25:** direct `encrypt` handles a single-value +input and does not automatically lift over collections. Retain collection +composition through `EncryptInto`, including +`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. +Use `OpaqueBlob>` to encode and seal a complete collection as one +ciphertext. This settles the previously reopened collection question. +The spelling for byte buffers (`Vec` versus an explicit bytes wrapper) +remains a separate detail to resolve. + +Provide a straightforward opaque ciphertext storage round trip for scalar +values, with direct byte conversion and reconstruction. Decide whether this +means a dedicated scalar result type, a wrapper, or conversion helpers on +the existing API. + +Also decide whether Base64 and Serde should be supported conveniences, and +whether whole-tree serialization belongs in Stack or remains caller-owned. +If it remains caller-owned, make that distinction explicit and keep it out +of the simple scalar path. The prototype's canonical leaf encoding is a +useful starting point, not a backward-compatibility obligation. + +**Follow-up, 2026-10-25:** text conversions must use the existing +`Base64Text` / `Base85Text` wrappers from EQL bindings. Explore moving these +general encoding types below Stack/EQL and supporting ciphertext-specific +encoded field targets. The persisted ZeroKMS key tag must be extractable +simply in Rust and PL/pgSQL. A refinable-tag protocol is being designed +separately. The user clarified that ZeroKMS checks policy/logs against the +refined tag and returns the original tag for reconstruction of leaf AAD; +record these distinct roles in the implementation. The +[persistence requirements](2026-10-24-stack-encrypt-api-plan.md#typed-encodings-and-extractable-zerokms-tags) +record the proposals and the Base85/SQL decoding constraint. + +**Sources:** [ciphertext alias and leaf encoding](../../packages/stack-encrypt/src/cipher.rs) +(`StackCipherText`, `SealedValue`) and +[storage documentation and public re-exports](../../packages/stack-encrypt/src/lib.rs). +Vitamin C's container definition is in `vitaminc-aead` 0.8.0, +`src/container.rs`. + +## 3. Plaintext-side encryption has different syntax and context requirements + +**Status:** Open. + +### Current behavior + +The current Vitamin C `Encrypt` trait describes a plaintext through `visit`; +it does not provide an `encrypt` method. Importing Stack's re-export of that +trait does not enable `10_i32.encrypt(&keyset)`. Vitamin C 0.8.0 documents +this trait as replacing the older visitor-based trait that drove a cipher +directly. Its `ContextTag` wrapper has encryption methods, but those require +Vitamin C cipher traits and do not make the syntax available on bare integers +or accept Stack's public ciphers directly. + +Stack supplies its own blanket `EncryptInto` extension trait: + +```rust +use stack_encrypt::{nonempty, EncryptInto, StackCipherText}; + +let ciphertext: StackCipherText = 10_i32 + .encrypt_into_with_context(&keyset, nonempty!("users/age")) + .await?; +``` + +The output type selects the `EncryptFrom` implementation. For +`StackCipherText`, that implementation requires an `Encrypt + Clone` source +and an `AeadContext`, which must be nonempty. + +Although `EncryptInto` also has `encrypt_into(&keyset)`, that method requires +the target's context to implement `Default`. `AeadContext` does not, so this +no-context form cannot produce `StackCipherText`. It is available for other +targets with default contexts, such as declarations carrying their own +contexts. + +The scalar derive's implicit context choice is also a discoverability concern: +`#[stash(plaintext = i32)]` generates `type Context = CallerContext`, even +for a record containing only a `StackCipherText` field (whose own context is +`AeadContext`). The generated operation converts between those types. The +reviewer found that hidden choice surprising; whether it needs an API change +or clearer diagnostics/documentation remains undecided. The `Default` bound +belongs to `EncryptInto::encrypt_into`, not to the `EncryptFrom` trait itself. + +### Caller impact and open decisions + +Plaintext-side encryption exists, but is not a symmetric spelling of +`keyset.encrypt(value, context)`: it uses different method names, selects an +output through a target trait, adds a `Clone` bound for this output, and +refuses the empty context accepted by the inherent method. + +Resolve the differing empty-context requirements alongside issue 1; +renaming alone would not make the two paths equivalent. This inconsistency +is separate from the receiver choice below. + +### Receiver choice: plaintext or keyset? + +**Decision (2026-10-24):** use only the keyset receiver for the ordinary +`encrypt` method: + +```rust +tenant_keyset.encrypt(value).await?; +``` + +Do not add a plaintext-side `value.encrypt(&keyset)` counterpart. +**`EncryptInto` remains.** This decision concerns only the `encrypt` method, +not removal of the target-directed encryption API. Implementation is pending; +the current inherent method still requires its context argument (issue 1). + +The two receiver forms express the same inputs and operation. Neither inherently provides +better type safety, ownership behavior, performance, or batching; those +depend on the signatures and implementation, not which argument is `self`. + +| Consideration | Plaintext receiver | Keyset receiver | +| --- | --- | --- | +| Mental model | Transform this value using a capability | Ask this encryption capability to process a value | +| Discoverability | Requires importing an extension trait; methods appear on plaintext types | Encryption methods are found on the encryption handle | +| Fluent composition | Reads naturally after an expression producing plaintext | Reads naturally when selecting a keyset before an operation | +| Operational scope | Keyset and its authority appear as an argument | Keyset and its authority lead the call | +| API surface | Adds methods to primitives and application types | Keeps execution methods on library-owned handles | +| Generic integration | Can work through a cipher trait | Can work through a cipher trait | + +**Rationale:** Stack performs an asynchronous operation through a configured KMS +capability; putting that capability first makes its scope visible and gives +callers one place to discover operations. Plaintext traits can describe how +values are represented without also owning the execution API. + +Keeping both equivalent spellings adds documentation, completion, and naming +choices without adding capability. Familiarity with an earlier API is a +migration consideration, but does not by itself justify two permanent entry +points. + +The target-directed `EncryptInto` surface can choose outputs other than +ordinary ciphertext and is retained explicitly. Its context inconsistency +remains open independently of this receiver decision. No decision about +decryption or batch method receivers is made here. + +**Sources:** [plaintext-side methods and ciphertext target implementation](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto`, `EncryptFrom for StackCipherText`) and +[context types](../../packages/stack-encrypt/src/target/context.rs) +(`AeadContext`). Vitamin C's source trait is in `vitaminc-aead` 0.8.0, +`src/plaintext/mod.rs`. + +## 4. The scalar derive cannot declare a fixed literal context + +**Status:** Direction agreed; implementation pending. + +The desired combination is a derived output for a bare `i32`, a fixed +context declared on that output, and `10_i32.encrypt_into(&keyset)`. +The current macro rejects `#[stash(plaintext = i32, context = "example")]`: +the literal `context` attribute is supported only with `struct = ..`. + +A supported derive-only workaround wraps the integer in a plaintext struct: + +```rust +struct IntegerInput { + value: i32, +} + +#[derive(stack_encrypt::EncryptFrom)] +#[stash(struct = IntegerInput, context = "example")] +struct EncryptedInteger { + value: stack_encrypt::StackCipherText, +} + +// With EncryptInto imported: +let encrypted: EncryptedInteger = IntegerInput { value: 10 } + .encrypt_into(&keyset) + .await?; +``` + +This form declares the field's context as the pair `("example", "value")`, +rendered as the ZeroKMS descriptor `example/value`, and uses defaultable +`DeclaredContext` for optional caller extensions. It is not a direct +conversion from a bare integer. A manual `EncryptFrom` implementation +can supply its own context; the ordinary literal attribute cannot express +that on a scalar derive today. + +### Agreed direction + +Allow a scalar `plaintext = T` derive to declare a fixed literal context: + +```rust +#[derive(EncryptFrom)] +#[stash(plaintext = i32, context = "example/integer")] +struct EncryptedInteger { + ciphertext: StackCipherText, +} + +let encrypted: EncryptedInteger = 10_i32.encrypt_into(&keyset).await?; +``` + +With a fixed context, the generated operation supplies that context and +exposes a defaultable caller-context type, enabling `encrypt_into`. Without +a fixed context, the ordinary scalar derive continues to use `CallerContext` +and requires `encrypt_into_with_context`. Preserve explicit `context_type` +and `context_field` configurations; their interaction with a fixed context +needs validation rules rather than a silent override. + +This addresses the scalar gap at the derive layer without requiring +`CallerContext` itself to implement `Default` or changing the `EncryptFrom` +trait. A bare scalar has no input field from which the derive can extract +context; runtime context can still be supplied by the caller. + +Additional caller context is accepted. The existing mechanism to reuse is +`DeclaredContext::under(own)`, exposed on operation descriptions through +`Encryption::under(own)`. `DeclaredContext` implements `Default`; its empty +default preserves the declared context unchanged. A caller extension is +combined as `(own, caller)`, never substituted for the declared context. + +Recommended combination semantics follow this existing mechanism: + +| Caller input | Effective structured context, with static context `C` | +| --- | --- | +| No extension | `C` | +| One extension `P` | `(C, P)` | +| Successive extensions `P`, then `Q` via `DeclaredContext::with` | `((C, P), Q)` | +| One extension containing the pair `(P, Q)` | `(C, (P, Q))` | + +This is structured composition, not string concatenation. Order and grouping +matter, and decrypt must reproduce the same structure. `NonEmpty::with` +builds the corresponding pairs directly. The existing test +`under_gives_a_subtree_its_own_context_which_the_callers_extends` in +[target.rs](../../packages/stack-encrypt/tests/target.rs) verifies that an +operation's declared context plus a caller extension agrees with a directly +composed context. + +The exact representation of a static literal (notably text versus a parsed +path label) still needs to be specified when implementing the derive, so +the descriptor and AEAD context remain consistent across encryption and +decryption. This decision is separate from retaining the `EncryptInto` trait. + +**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`a_container_context_requires_a_struct`) and +[generated implementations](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`struct_derive`, `value_derive`). + +## 5. EncryptFrom does not define encrypt_from + +**Status:** Open. + +`EncryptFrom` defines the associated type `Context`, the operation-description +constructor `encryption()`, and `indexes()`. It does not define a function +named `encrypt_from`. + +A function with that name does exist, but on the blanket extension trait +`EncryptInto`: `Target::encrypt_from(&value, &keyset, context)` delegates to +`keyset.encrypt_as`. Calling that function requires `EncryptInto` in scope +and a target implementing `EncryptFrom`. + +This differs from the standard Rust naming relationship: `From` defines +`from`, while `Into` defines `into`. The names suggest that same +relationship here, but `EncryptFrom` declares an operation rather than +executing a conversion, and the target-side execution function belongs to +the source-side extension trait. That makes method discovery and the +required trait import surprising. + +Review the trait and method names, or where the convenience function lives, +while preserving the distinction between declaring an operation and +executing it. No replacement design has been selected. + +**Source:** [target operation traits](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptFrom`, `EncryptInto`, including `EncryptInto::encrypt_from`). + +## 6. Plan builder encrypt_into reads like the execution method + +**Status:** Open; revisit during the plan API discussion. + +`Plan::value::().encrypt_into::()` is an inherent builder method that +selects an output target requiring `T: EncryptFrom`. It does not implement +`EncryptInto` or execute encryption. Sharing the name with +`value.encrypt_into(&keyset).await?` caused confusion during this review. + +Consider renaming the builder method to `target`: + +```rust +Plan::value::() + .target::() + .build() +``` + +This names the selection being made and distinguishes it from executing a +conversion. The rename is a proposal for later discussion, not a decision +or an implemented API change. `EncryptInto::encrypt_into` remains unchanged +by this proposal. + +**Source:** [value plan builder](../../packages/stack-encrypt/src/plan/value.rs) +(`ValueStart::encrypt_into`). + +## 7. Mixed encrypted and plaintext fields lack a direct output-derive spelling + +**Status:** Passthrough attribute proposal accepted in principle; implementation pending. + +For a `User` with encrypted `name` and `email` and plaintext `id` and +`created_at`, the fields-plan API directly supports `.encrypt(pick(..))` +and `.passthrough(pick(..))`. It batches the encrypted fields and returns +`FieldValues`; callers can extract those values into a typed storage struct. + +The current `EncryptFrom` derive has no passthrough/copy attribute for source +fields. `#[stash(default)]` initializes an output from a default or expression; +it does not copy a field from the input. Merely keeping the original field +type does not select passthrough either: derived fields require target traits. + +The proposed `#[stash(passthrough)]` attribute on output fields was accepted +as sensible: it copies the matching source field unchanged, alongside fields +whose types select encryption. Plaintext passthrough fields remain unencrypted and +unauthenticated; making a field part of the encryption context is a separate +choice, not implied by copying it. + +This gap is specific to Stack's output-side `EncryptFrom` derive. Vitamin C +already has plaintext-side `#[derive(Encrypt, Decrypt)]` with +`#[aead(passthrough)]` on source fields. Stack's `keyset.encrypt(&user, aad)` +can consume that derived source, yielding a generic `StackCipherText` map +with sealed and passthrough nodes. It does not generate a named +`EncryptedUser` type or expose the output as typed Rust fields. Its map-entry +AAD derivation also differs from the per-field contexts of a Stack fields +plan; the two routes should not be presented as interchangeable storage +formats. There is currently no source-side derive that generates the typed +encrypted counterpart illustrated in this review. + +**Sources:** [plan passthrough](../../packages/stack-encrypt/src/plan/build.rs) +(`FieldsBuilder::passthrough`) and +[derive field attributes](../../packages/stack-encrypt-derive/src/attrs.rs) +(`FieldAttrs`). + +## 8. Clarify whole-value encryption, layouts, and searchable outputs + +**Status:** Open architectural discussion; recommendations below are not decisions. + +Three user workflows need clear support: + +1. Serialize a complete value into bytes and encrypt those bytes as one + opaque ciphertext. +2. Produce a chosen layout, such as `EncryptedUser`, with independently + encrypted fields and explicit plaintext fields. +3. Produce a layout with additional encrypted forms or search terms, such + as EQL targets. + +The existing Vitamin C plaintext derive produces a structural ciphertext +tree, not a single encrypted serialization. It is a generic output layout +for workflow 2, not workflow 1. The distinction affects key count, +independent field access, visible structure, and authenticated boundaries. +Encoding the completed ciphertext tree into one byte string would not +change those properties into whole-value encryption. + +### Recommended model + +Distinguish the encryption granularity (one encoded value or selected +fields) from the output representation and its requested operations. +Workflow 3 should extend workflow 2 through field targets that request +ciphertext, terms, or both, rather than introducing a separate execution +system. The generic tree can remain a layout option without being the +mandatory scalar storage representation. + +- **Whole-value path:** the follow-up review favors an explicit named + wrapper, provisionally `OpaqueBlob`, which owns serialization and + deserialization through one library-selected codec. It produces one opaque + persistable ciphertext; decrypting into `OpaqueBlob` selects the matching + decode path. Today callers can serialize into `Vec`, which Vitamin C + treats as a single byte leaf, but still encounter the node/leaf storage gap. +- **Layout path:** select a target with `EncryptFrom`/`EncryptInto`, or a + plan, with an explicit passthrough choice. A source-side derive that + generates a target would be authoring convenience over the same system, + not another set of encryption semantics. +- **Searchable layout path:** compose core encryption/index operations + through targets. EQL adapters own their storage representation; Stack + owns execution. Write and query operations must share the relevant + context and normalization rules. + +Keep source representation, requested operations, output encoding, and +execution distinct internally, while giving each workflow a short public +path. An application should not need to learn visitor machinery or plans +to encrypt and persist a scalar. Arbitrary serialization must not become an +implicit replacement for the typed normalization used by indexes. + +Before selecting more trait names, write end-to-end examples for all three +workflows: encrypt, persist, load, decrypt, plus query for searchable targets. +Use those examples to evaluate the public surface and error messages. + +### Follow-up: opinionated whole-value wrapper + +The first public example now uses +`keyset.encrypt_with_context(OpaqueBlob::new(user), context)` rather than +manual JSON encoding. The wrapper name is provisional; the direction is to +make blob serialization an explicit library-owned behavior. The stored result +remains opaque ciphertext, and decryption returns an `OpaqueBlob` that +can be unwrapped. The codec and its authenticated versioning must be specified; +the wrapper does not silently change how unwrapped structs are encrypted. + +Constant-time serialization/deserialization is a desired property to +investigate, not a property established by choosing a binary codec or using +constant-time AEAD. The first revised example provisionally uses Serde traits, +whose arbitrary user implementations cannot carry that guarantee. Define +which lengths and shapes may be public and whether a stricter supported-type +profile is needed before promising secret-independent execution. The examples +document this unresolved boundary separately from the wrapper ergonomics. + +Investigate `serdect` as part of that work. It is already a Stack Encrypt +dependency; check its encoding/decoding guarantees and their scope, and +evaluate applicability to ciphertext serialization and the proposed blob +codec. No whole-object constant-time guarantee or codec choice is inferred +from its use. + +## 9. Ownership of the value-encryption API across Stack and Vitamin C + +**Status:** Open architectural discussion; moving the traits is proposed, +not authorized implementation work. + +The review has repeatedly crossed from Stack's target declarations into +Vitamin C's plaintext visitor traits, cipher traits, contexts, derives, and +ciphertext tree. The user proposes moving the encryption traits into Stack +Encrypt and retaining general primitives such as protected values and +randomness in Vitamin C. + +**Recommendation:** Stack should own the public value-encryption model, +including the plaintext traits/derives, target declarations, contexts, +ciphertext persistence, and execution semantics. Vitamin C can retain +reusable cryptographic and memory-protection primitives. Moving files alone +does not resolve ambiguous names (`Encrypt` currently describes a source), +duplicate entry points, or the missing whole-value/storage path. + +One Stack-owned core layer may serve Rust and language guests without +pulling in KMS transport or HTTP. Applications should still have one +coherent `stack_encrypt` entry point; an internal dependency split need not +become another public API users must understand. + +### Migration constraints + +- Today `stack-encrypt` depends on both `vitaminc-aead` and + `vitaminc-encrypt`, while `vitaminc-encrypt` also depends on + `vitaminc-aead`. Moving the latter's traits into the top-level Stack crate + and making the existing backend depend on Stack would introduce a cycle. + Move the value-level backend adapters too, or separate a lower-level + core/primitive boundary. +- Context composition, type tags, ciphertext framing, index normalization, + and binding transport frames must agree across the new consumers. They + may change without preserving prototype data or providing migration. +- Retain one canonical trait/type identity. Update consumers directly; + compatibility re-exports are unnecessary. Copied independent trait + definitions would require duplicate implementations and adapters. +- Account for guest/FFI consumers and EQL target implementations, not just + Rust application imports. The no-HTTP/WASM path must remain usable. + +**Sources:** [Stack dependencies](../../packages/stack-encrypt/Cargo.toml), +[guest ABI dependencies](../../packages/stack-guest-abi/Cargo.toml), and +[EQL encryption adapters](../../packages/eql/crates/eql-bindings/src/encryption.rs). +The installed `vitaminc-encrypt` 0.8.0 manifest confirms its dependency on +`vitaminc-aead`. + +## 10. Public encrypt_as duplicates encrypt_into + +**Status:** Decision agreed; implementation pending. + +`KeysetCipher::encrypt_as` is currently an inherent method requiring the +output to implement `EncryptFrom`. It executes that target's operation +declaration. `EncryptInto::encrypt_into` and +`EncryptInto::encrypt_into_with_context` delegate to it, supplying either +the default context or the caller's context. + +**Decision:** remove `encrypt_as` from the public API. Keep `encrypt_into` +and `encrypt_into_with_context` as the public target-directed entry points: + +```rust +let encrypted: EncryptedInteger = value.encrypt_into(&keyset).await?; +let encrypted: EncryptedInteger = value + .encrypt_into_with_context(&keyset, context) + .await?; +``` + +The operation-execution machinery may remain internal; removing the public +method does not remove target declarations, batching, or execution. Public +examples should use the retained entry points when illustrating the proposed +API. Earlier descriptions of `encrypt_as` in this log describe current code. + +This is distinct from issue 3's decision about ordinary encryption: +`keyset.encrypt(value)` remains the sole ordinary `encrypt` spelling, while +`value.encrypt_into(&keyset)` selects a particular output type. The +`encrypt_from` naming/location question in issue 5 remains open. + +**Source:** [target operations](../../packages/stack-encrypt/src/target/operations.rs) +(`KeysetCipher::encrypt_as`, `EncryptInto`). + +## 11. Context-field storage does not yet extract context from an input struct + +**Status:** Open. + +`#[stash(context_field)]` exists on derived output types. On encryption it +stores the supplied context in that output field; on decryption it uses the +stored context, optionally checking an independently supplied expected value. +It does not automatically select a field from a plaintext input struct. + +The macro currently treats this field as the complete context and rejects +it alongside a literal `context`. Since `struct = User` requires a literal +context, that form cannot currently mark its `id` with `context_field`. +The public examples originally described passing the id explicitly without +explaining this distinction. + +### Clarified model: context source and context storage are independent + +Two similar scenarios must be distinguished: + +1. The caller supplies context as a function argument; the encrypted output + stores it for later use. This is the existing scalar-output + `context_field` behavior. +2. The encryption declaration obtains context from one or more fields of + the plaintext. Whether those values are also stored is a separate choice. + +The second scenario can include storage too: read `User.id`, use it to bind +the encrypted fields, and carry it into `EncryptedUser.id`. The caller need +not pass the same value twice. Several fields, such as a tenant id and user +id, can supply a structured composite context. + +| Context source | Stored in the encrypted output? | Recovery requirement | +| --- | --- | --- | +| Caller argument | Yes | Read the stored context; optionally check an independently supplied expected value | +| Caller argument | No | Supply the same context again | +| Plaintext field(s) | Yes | Reconstruct context from the stored field(s); optionally check expected values | +| Plaintext field(s) | No | Supply equivalent context independently before decrypting | + +Store a source-derived context independently of the ciphertext that needs +it, so decrypt does not depend on first opening that ciphertext to discover +its own context. Binding ciphertext to a stored context authenticates that +relationship; it does not by itself establish that the entire record is at +the destination the application expected. + +The derive should express source selection and output storage as separate +concepts, even if a shorthand covers the common read-and-store case. Avoid +assuming that one `context_field` attribute should silently perform both. +Syntax remains open. Define field ordering/grouping for composite context, +literal prefixes, optional caller extensions, `from` mappings, and expected +context checks. These capabilities are proposed, not implemented. + +The recommended common-case shorthand is `context(field = "id")`, expanding +exactly to `encrypt(field = "id"), decrypt(field = "id")`. It selects the +same Rust field name on the two different inputs; it does not implicitly +copy or store that field. Keep the directional form for different names or +sources. Initially reject mixing shorthand and directional selectors to +avoid ambiguous precedence. The context examples now use this shorthand +for the plaintext-id-to-stored-id case. + +**Sources:** [macro validation](../../packages/stack-encrypt-derive/src/shape.rs) +(`Record::parse`, `context_type`) and +[context-field output assembly](../../packages/stack-encrypt-derive/src/encrypt.rs) +(`assignments`, `value_derive`). + +## 12. Default is not the same as optional caller context + +**Status:** Open; exposed by the directional-context examples. + +`EncryptInto::encrypt_into` and `DecryptFrom::decrypt_from` make the +no-context form available whenever the relevant associated `Context` type +implements `Default`. That works for deliberately defaultable types such as +`DeclaredContext` and `ExpectedContext`, and currently excludes +`CallerContext` and `AeadContext`. + +However, a required caller value can have a Rust default: if a declaration +exposes `Context = u64`, the no-context form supplies zero. The directional +proposal `encrypt(caller = u64)` must not accidentally mean "optional, with +zero as the default". + +Separate the application's context value type from the trait's caller-input +type or capability. Options include a non-defaultable `RequiredContext` +wrapper that accepts ergonomic conversions from `T`, or a dedicated marker +controlling context-free entry points. No mechanism is decided yet. + +This matters equally to required context on decrypt. Source-derived context +can require no encrypt argument, while decrypt may require an external +value or accept an optional expectation of stored context. Do not assume the +two associated input types must match, or that either always equals the +complete logical context used by encryption. + +See the [context scenario examples](2026-10-24-stack-encrypt-context-examples.md) +for the proposed selectors and input requirements. + +**Source:** [convenience methods](../../packages/stack-encrypt/src/target/operations.rs) +(`EncryptInto::encrypt_into`, `DecryptFrom::decrypt_from`). + +## 13. JSON layouts and encrypted-output adaptation + +**Status:** Open; initial assessment of the JSON tree implementation on +`next` at `b1049824b`. No replacement API selected. + +**Agreed follow-up:** investigate replacing the EQL JSON adapter's +serialization round trips with direct typed conversions. This does not yet +select an extended visitor protocol or authorize a particular replacement +API. + +`Layout` describes a target selected by a value carrying runtime options. +It associates a `Target: EncryptFrom` and an `Index`, and supplies an +`encryption(&self)` description plus the configured index for queries. +`EncryptFrom::encryption()` instead selects a description by output type. +For JSON, both routes can produce `JsonDocument`: selecting that type uses +default options, while `Json::new(options)` can specify ordered paths and +whether the root is sealed. A plan retains those options for its queries. + +This is broader than choosing a Rust record's field layout. JSON options +affect cryptographic structure, query capability, and information exposed +to the server. The same output type can represent different configurations; +the type alone does not identify them. The JSON tree belongs to the +searchable-layout workflow, rather than being an opaque serialized blob. + +The existing `target::transcode` protocol serves a different stage: a +`Reader` hands completed ciphertext, structural markers, metadata, and terms +to a destination `Visitor`. `Encryption::transcode()` uses it to adapt +encrypted output without an intermediate serialization tree. It does not +choose JSON indexing options or generate missing search terms. + +The new EQL adapter currently takes another route. `JsonTree` implements +`Layout`, delegates encryption to the +engine's `Json` layout, and wraps its completed `JsonDocument` with EQL +version and column metadata. Its envelope conversion uses a +`serde_json::Value` intermediate; a private helper also named `transcode` +converts query/extraction types through `to_value`/`from_value`. Neither is +the existing `target::transcode` protocol. That protocol currently has no +JSON-document reader or visitor event. The conversion processes encrypted +output and query metadata; it is not a plaintext serialization bypass. + +On decrypt, the EQL target validates its stored column against any supplied +expectation, reconstructs the engine document through the wire form, and +delegates opening to the engine. Its layout object is not required to +perform that conversion. + +### Questions to resolve + +- Keep encryption configuration distinct from encrypted-output adaptation. + Decide whether `Layout` is the clearest public name for a configured + target, especially alongside the earlier use of "layout" for output + records. +- Decide how runtime target options fit the retained `EncryptInto` API; + currently type-directed conversion selects the target's defaults, while + plans accept a configured layout via `encrypt_with_layout`. +- Evaluate a direct typed JSON-document adapter versus extending the + encrypted-output visitor. Avoid forcing the JSON protocol into generic + map/sequence events that lose its header, shared document key, projection, + and extraction semantics. A shared architectural boundary does not + necessarily require one universal visitor. +- Consider whether a general configured-target abstraction should require + an index: today's `Layout` requires one even though encryption-only + targets can also need options. +- Preserve the link between write configuration and query configuration; + a plan's reuse of the configured index is useful independently of naming. + +**Sources:** [layout and operation descriptions](../../packages/stack-encrypt/src/target/operations.rs) +(`Layout`, `json_document`, `EncryptFrom for JsonDocument`), +[encrypted-output protocol](../../packages/stack-encrypt/src/target/transcode.rs), +[JSON options and document](../../packages/stack-encrypt/src/json/mod.rs), +and [EQL JSON adapter](../../packages/eql/crates/eql-bindings/src/v4/json_tree.rs) +(`JsonTree`, `JsonTreeDocument::envelope`, `transcode`, `DecryptInto`). + +### Static Rust surface versus runtime configuration + +The intended public Rust experience is traits and derives. Treat plans as +the lower-level composition interface supporting those APIs and the dynamic +FFI interface, rather than requiring Rust callers to write a plan for JSON. +The derive already emits plans; its implementation can retain that shared +execution path without making the plan builder the primary user example. + +JSON already has a type-directed path. Given an initialized `cipher` and +its `keyset`, this uses existing methods and default JSON options: + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use eql_bindings::Identifier; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, EncryptInto}; + +let profile = json!({"name": "Alice", "age": 34}); +let column = Identifier::for_column("users", "profile")?; + +let encrypted: JsonTreeDocument = profile + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +let restored: Value = encrypted.decrypt_into(&keyset, column).await?; +assert_eq!(restored, profile); +``` + +This produces searchable JSON with a sealed root and no ordered paths. +The explicit decrypt context checks the expected column against the stored +identifier. The snippet is source-checked, not compiled here. + +The earlier `SearchableEmail` example fixes its recipe in the target type: +ciphertext plus an equality term. `EncryptFrom::encryption()` takes no +configuration value. JSON's default target also fixes a recipe in its type. +What `Layout::encryption(&self)` adds is access to instance configuration: +one `JsonTreeDocument` output type can be produced with `$.age` ordered, +with `$.salary` ordered, or with different root settings, selected at +runtime. Its configured index also supplies the matching query behavior. + +Search metadata itself does not require `Layout`. A custom target could +fix ordered JSON paths in a manual `EncryptFrom` implementation using the +existing JSON operation. A derive could eventually generate such a fixed +policy. Runtime-configured targets need an additional configuration input; +today `encrypt_into_with_context` accepts context but no layout/options, +and custom JSON configurations use the lower-level plan or `keyset.run` +path. Keep those options separate from cryptographic context. + +The cryptographic distinction is separate from the API distinction: JSON +uses a specialized document operation with a shared document data key and +node/extraction semantics. It cannot be implemented merely by adding a +generic equality-term field to the ordinary ciphertext tree. That explains +the JSON operation, but does not by itself require a public `Layout` trait. + +### Naming obscures roles + +| Current type | Role | +| --- | --- | +| `serde_json::Value` | Plaintext input and recovered plaintext | +| `stack_encrypt::json::JsonOptions` | Runtime configuration | +| `stack_encrypt::target::Json` | Configured index; also implements `Layout` | +| `stack_encrypt::json::JsonDocument` | Encrypted engine document, including search metadata | +| `eql_bindings::v4::json_tree::JsonTree` | Configured EQL target/layout | +| `eql_bindings::v4::json_tree::JsonTreeDocument` | Encrypted EQL storage envelope | + +Review names that state these roles, using `Encrypted` for ciphertext-bearing +documents, `Index` for index descriptions, and `Target` or `Config` for +configuration, with module paths distinguishing EQL from engine types. +No replacement names are selected. `Json`'s dual index/layout role should +be considered before simply renaming it. + +### Candidate: a fixed-policy wrapper around the EQL output + +**Proposal, not implemented or agreed syntax.** Attach the fixed policy to +an output newtype, while the inner library type owns the database format. +The alias below only clarifies the existing type's role; it is not a +library rename. + +The later [implementation plan](2026-10-24-stack-encrypt-api-plan.md) +recommends a general typed-options factory, `target(options = factory)`, +instead of the JSON-specific attribute below. Dynamic callers deserialize +options into the same concrete type; Rust and Go support ship together. +This sketch is retained to explain the earlier alternative. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument as EqlEncryptedJsonTree; +use eql_bindings::Identifier; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use stack_encrypt::{DecryptFrom, DecryptInto, EncryptFrom, EncryptInto}; + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[serde(transparent)] +#[stash( + plaintext = Value, + transparent, + json_tree(ordering_paths = ["$.salary"]) +)] +struct EncryptedPayroll(EqlEncryptedJsonTree); + +// Given a keyset: +let payroll = json!({"name": "Alice", "salary": 120_000}); +let column = Identifier::for_column("employees", "payroll")?; + +let encrypted: EncryptedPayroll = payroll + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + +// The wire representation is the inner EQL envelope, with no extra field. +let wire = serde_json::to_vec(&encrypted)?; +let loaded: EncryptedPayroll = serde_json::from_slice(&wire)?; +let restored: Value = loaded.decrypt_into(&keyset, column).await?; +assert_eq!(restored, payroll); +``` + +The proposed `stash(transparent, json_tree(...))` selects the configured +JSON operation for the inner target and wraps its completed output. It must +not invoke the ordinary scalar/record derivation on the wrapper, nor invoke +the inner target's default encryption and then try to add ordering terms. +It inherits the EQL target's caller-context and decrypt behavior. Serde's +existing `transparent` attribute preserves the EQL envelope; it does not +provide database-driver trait implementations, which remain a separate +integration concern. The root stays sealed under this candidate's defaults. + +The generated declaration must also expose the configured index metadata +for future typed queries, so queries can reuse the policy instead of +repeating paths. The query-facing syntax remains undesigned. The core derive +must delegate configured output construction to a capability of the inner +target, rather than depending directly on the EQL crate. Whether to express +that capability through `Layout` or another trait remains open. + +### Alternative: configuration on a plaintext wrapper + +The user's proposed `JsonPlaintext::new(value).with_ordering_path(path)` can +still yield the exact EQL wire format. Configuration source and output +representation are independent: an explicit `EqlEncryptedJsonTree` target +can accept that input and execute the same configured JSON operation. The +wrapper must be treated as an encryption request, not serialized as part of +the plaintext. The current API does not provide this wrapper. + +This is attractive when paths come from runtime configuration, but repeats +fixed policy at call sites and needs a way to reuse that policy for queries. +Prefer the output newtype for the fixed-policy example. If the runtime +wrapper is added, initially do not implement conversion from it into +fixed-policy targets: avoid silently overriding or merging conflicting +policies. Neither syntax permits callers to redefine EQL's required fields +or its cryptographic wire protocol. + +A fixed Rust type describes the policy for new writes; transparent Serde +deserialization alone does not prove a stored document was produced with +that policy. Ordered paths are not all recorded in the document header. +Policy changes and heterogeneous existing rows need explicit compatibility +and migration semantics before claiming stronger read-time guarantees. + +## 14. Decrypt convenience methods need explicit context variants + +**Required direction:** provide a no-context method and a separate +`_with_context` method, mirroring the encrypt-side distinction. The user +prefers the name `decrypt_from`. + +Today `ciphertext.decrypt_into(cipher, context)` exists on the blanket +`DecryptFrom` extension trait. `Plaintext::decrypt_from(ciphertext, cipher)` +and `Plaintext::decrypt_from_with_context(ciphertext, cipher, context)` also +exist on that trait. `DecryptInto

` separately declares the opening +operation. These method/trait names are easy to confuse. + +**Recommendation:** use the destination-directed pair in the preferred +public examples. `User::decrypt_from(encrypted, &keyset)` follows Rust's +From convention; `encrypted.decrypt_from(...)` would reverse it. The +receiver and migration of the competing method remain recommendations to +confirm, not settled user decisions. Do not change the declaration trait +merely because its convenience spelling changes. Address trait discoverability +with issue 5 and optional-context capability with issue 12. + +## 15. Coordinate the Go JSON work with the shared API redesign + +The user identified [PR #1353](https://github.com/cipherstash/stack/pull/1353) +as parallel agent work. The PR is currently a Go JSON design document with +implementation tasks #1348 through #1352, tracked by #1347. Its dynamic +JSON/options path, public names, context behavior, and staged extraction +support must be reconciled with the configured-target and trait/context +contracts developed here. Preserve compatible catalog, transport, engine, +and testing work rather than designing or implementing it twice. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) records +the concrete reconciliation points after reading the PR write-up. This is +a local coordination item; no message or review was posted to that agent +or PR. + +## 16. Integrate the prior query design rather than reopening it + +The user supplied **Stack Encrypt query design** (2026-10-21), read in full +on 2026-10-25 from `/Users/dan/Downloads/Stack Encrypt query design.md`. +It proposes field-owned index selection, one answering index per operator, +typed field handles, a single capability declaration, and shared query +derivation/refusals. It is explicitly a discussion draft with open questions. + +The [implementation plan](2026-10-24-stack-encrypt-api-plan.md) now uses that +draft as the baseline and records concrete reconciliation points: our +output-side derives and configured options factories, JSON operators and +extraction, context inputs, static/dynamic portability, and validation +timing. Older compatibility requirements are superseded by the user's +prototype clarification. The source document was left unchanged. + +## 17. Use consistent definition-selection names for encrypt and query + +The user finds `.using(...)` for encryption and `.on(...)` for queries +confusing and asks to consider one name for both. Recommend `.using(...)`: +the selected definition or field supplies the operation's policy. Final +spelling is not yet accepted. Retain the operation-specific bounds: +encryption selects the output target's associated definition; queries +require a field or single-value target implementing `QueryTarget`. +This is a naming change, not a merger of those capabilities. + +The user also prefers `cipher.using(&field).query(op, input)` and explicitly +requires strict typestate. Supporting both orders is acceptable to them. +The plan recommends definition-first as canonical, with operator-first +converging to the same typed request. Missing definition/required context +must prevent awaiting on the static path; repeated selection must not +silently overwrite a request's definition. The current `.query(...).on(...)` +implementation is not claimed to already provide this API. + +## 18. Single-ciphertext representation needs consuming, cleanup-aware inputs + +**Decision, 2026-10-25:** use `OpaqueEncode` / `OpaqueDecode` for single-value +encoding/decoding, distinguishing one opaque ciphertext from target layouts. +Both direct `encrypt` and target-directed `encrypt_into` consume input and +are responsible for cleanup of owned plaintext artifacts, including the +context-taking variants. Explicit passthrough fields move to the output. +The plan records source cleanup capabilities, +immediate guarding before returning an unpolled request, cleanup of encoded +buffers/scratch on all ordinary exit paths, and boundaries for caller copies, +custom serialization, passthrough output, and Go host/guest ownership. +`Serialize` alone cannot establish cleanup of an `OpaqueBlob` source. +Consumption and zeroization are distinct; no completed audit is claimed. + +**Follow-up decision:** plaintext types must implement the existing +`Zeroize` trait; do not introduce our own cleanup trait for now. For the +proposed Serde blob, this means `Serialize + Zeroize` on input and +`DeserializeOwned + Zeroize` on owned decode. Library guards must actually +invoke cleanup; partially constructed Serde values need separate treatment. + +**Decrypt decision:** retain `keyset.decrypt` / `decrypt_with_context` for +ordinary opaque ciphertext and `T::decrypt_from` / +`decrypt_from_with_context` for declared targets. Return an ordinary owned +`T: Zeroize` on success, transferring cleanup responsibility to the caller. +Stack guards plaintext intermediates while it owns them. Explicit +drop-guarded results should be supported without becoming the default; +`Zeroize` alone does not imply cleanup on drop. This does not resolve the +separate partial-deserialization cleanup requirement. + +**Codec recommendation:** reuse Vitamin C frame machinery with a blob +mapping. The 0.8.0 transport Serde implementation already exists and guards +serialization buffers, and its frame writer wipes allocations on growth. +Do not reuse that mapping unchanged: option presence can be lost +(`Some(())` and `None` both encode as null), maps require text keys, and +successful `to_vec` returns an unguarded vector. See the +[codec assessment](2026-10-24-stack-encrypt-api-plan.md#blob-codec-assessment-vitamin-c-frames) +for the required work and Borsh fallback. Neither codec choice nor the +`Zeroize` bound establishes constant-time behavior or partial-decode cleanup. + +**Priority update:** first specify stable, versioned storage bytes; the +existing frame codec was ephemeral transport. Further zeroization and timing +implementation improvements may follow without changing stored format. +The upstream action is +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519). + +## 19. First-release completeness and remaining review decisions + +**Accepted, 2026-10-25:** encoded ciphertext wrappers are first-class field +targets; final wrapper names/package placement come last. ORE does not +answer equality initially, even for integers. Preserve a future explicit +capability for equality-preserving encodings; never infer exact equality +from ordering support for text or other lossy encodings. Keep operator +provider overlap validation and avoid silently changing existing definitions. + +The JSON first version must be comprehensive across Rust and dynamic/Go, +including extraction, opening entries, children/counts and subtree walks. +Claude's earlier Go plan deferred walks; the new scope supersedes that +deferral. The [operation contract and examples](2026-10-25-stack-encrypt-json-operations.md) +lists required operations, current evidence, proposed Rust/dynamic surfaces, +SQL use, missing-path/null semantics and implementation acceptance gates. +Names and packaging are explicitly the last design activity. + +**Further accepted decisions:** custom Rust stages can remain explicitly +Rust-only with `NotPortable` for unsupported dynamic export; built-ins retain +Rust/Go parity. All key-independent checks precede key resolution/loading. +The detailed JSON operation contract is still under user review. + +## 20. Plaintext-derived keysets and optional provider policies + +The user requested routing a plaintext record to a keyset by ID/name from +one of its fields; assessment of Vitamin C's unused `AsyncCipher`; and +ZeroKMS lock context including principal JWT claim selectors. Keyset and +lock-context capabilities must remain optional for future AWS KMS/Vault +backends. See the [provider-capabilities assessment](2026-10-25-stack-encrypt-provider-capabilities.md) +for current source evidence, proposed static/dynamic declarations and +acceptance work. + +Current facts: Stack resolves keysets by ID/name but generation batches use +one preselected keyset. Stack deliberately orchestrates provider requests +outside Vitamin C's leaf walk and already implements synchronous `Cipher`. +Its provider payload layer supports lock context but Stack Encrypt sends +empty lock context. The existing data/index key abstractions still embed +ZeroKMS protocol types; future providers need a neutral base plus optional +capabilities, not no-op implementations of mandatory ZeroKMS features. + +Recommendations, not yet accepted: derive key routing as an explicit target +selector and resolve it in the client executor; partition batches by resolved +key/provider/principal requirements; carry lock context as a separate typed +provider policy on both generation and retrieval; reject unsupported required +features before I/O. Keep key routing, ciphertext context, search scope and +principal policy separate. Revisit cipher adapter cleanup during packaging +without discarding cross-record batching or native/WASM support. + +## 21. Packaging, naming and retirement of AsyncCipher + +The user favors removing Vitamin C `AsyncCipher` and moving application +encryption behavior into Stack Encrypt. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md) proposes +Stack ownership of value traversal, declarations, target/index adapters and +execution; a small shared `stack-encrypt-format` crate for codecs and typed +encodings; a ZeroKMS-specific client crate; and reusable primitives remaining +in Vitamin C. It includes an acyclic dependency map and public name glossary. + +Stack does not use `AsyncCipher`, but Vitamin C's async helpers and tests do. +Retire that execution path with its consumers and move relevant coverage; +do not treat deletion of the trait alone as complete. Normalization/PRF/ORE +adapters also depend on the existing value model and need separation from +their primitives. Moving the frame implementation refines the earlier codec +ownership recommendation; coordinate stable storage work under Vitamin C +#519. Names and package moves remain proposals, not implementation changes. + +## 22. Customer-driven sequencing: Go JSON and match search first + +The customer needs Go JSON support and robust Bloom-filter handling with +tokenizers/token filters, including two-character input. Standalone Rust +release quality is secondary. Claude's JSON-tree PR stack headed by #1363 has now merged. Revise the +older Go plan in #1353 against it and the agreed API direction. Vitamin C +#520 and #526 implement the storage profiles; colleague review and release +are pending. Stack still needs to adopt that release. + +Freeze the public Go contract and stored binary/search semantics before the +customer release. Internal plans may change behind that interface. Adopt and +stabilize the Vitamin C codec in its current repository now; defer physical +crate moves, AsyncCipher retirement and final Rust derives. The earlier +prototype compatibility exemption does not permit breaking customer data +after this release. + +The supplied `Short-ngrams.md` explores short whole-word filters and variant +queries, which must be distinguished from arbitrary two-character substring +search. The current tokenizer permits length 2, but the current one-match +output and one/range-query-term shapes need expansion for named filters and +Boolean alternatives. Resolve the scheme's leakage, SQL representation and +performance before release; do not silently defer the customer requirement. + +### Merged implementation reconciliation, 2026-10-27 + +Reviewed `next` at `ffeed7adb3f7d8739cda53261046147852bffc82` against the +previous `ebdb5de0a` baseline. #1372/#1373/#1378/#1402 deliver the Go JSON +catalog, configured targets, operators, request-bound opening, typed document +adapters and executable documentation. #1347–#1352 are closed. #1369/#1370 +integrate Go opaque storage; final Go releases still require replacing the +temporary Vitamin C pseudo-version pins. Rust dependencies remain on 0.8.0. + +#1379 adds cascade authentication and randomized stored/projection order. +Wildcard `->` returns an unspecified match, not the first document element. +Post-cascade data is the format to freeze; unchanged header version 1 does +not imply compatibility with the old prototype AAD. Read verification still +does not authenticate result completeness, absence or database row identity. + +#1397 implements deterministic, domain-separated short-word terms inside +the existing Bloom filter, enabled by EQL's pinned match profile. This avoids +a separate short-word index but is not the random-variant mechanism explored +earlier, and it does not by itself resolve the user's frequency/leakage +concern. Native Go index options still refuse configuration (#1112). The +uncommitted ngram research and Go delivery document are unchanged. + +The refreshed API plan and JSON operation inventory mark landed work as +implemented. Remaining items include the formal interoperability spec and +conformance corpus, parent-bound child opening, configurable Go match +pipelines, direct typed EQL conversion and later context-role separation. +The customer release does not wait for final Rust traits or crate moves. +This review inspected source and existing tests; runtime/database suites +were not rerun and no production code was changed. diff --git a/docs/plans/2026-10-24-stack-encrypt-context-examples.md b/docs/plans/2026-10-24-stack-encrypt-context-examples.md new file mode 100644 index 000000000..c927b7d67 --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-context-examples.md @@ -0,0 +1,723 @@ +# Context sources and storage: directional API examples + +Date: 2026-10-24. Design exploration accompanying the +[API review log](2026-10-24-stack-encrypt-api-review.md) and +[complete public workflow examples](2026-10-24-stack-encrypt-public-examples.md). + +**Everything introduced here is proposed syntax, not an implemented API.** +The purpose is to compare context behavior across the meaningful source and +storage combinations before selecting attributes or changing traits. The +`Ciphertext` and `passthrough` proposals are inherited from the workflow draft. + +## Recommendation: directional sources, separate storage + +Borrow Serde's nested directional attribute style: it supports separate +serialization and deserialization names within one `rename` attribute. +See [Serde's field attributes](https://serde.rs/field-attrs.html). + +For encryption, use direction names to say **where to obtain context**: + +```rust +#[stash(context( + prefix = "users", + encrypt(field = "id"), + decrypt(field = "id") +))] +``` + +The two `field` selectors refer to different objects: the plaintext input +when encrypting, and the encrypted input when decrypting. They must yield the +same typed context value. Unlike a Serde rename, these are not independent +cryptographic names: changing the context value or its shape between the two +directions prevents decryption. + +For the common case where both directions use the same Rust field name, +the recommended shorthand is: + +```rust +#[stash(context(prefix = "users", field = "id"))] +``` + +This expands exactly to `encrypt(field = "id"), decrypt(field = "id")`. +It still reads two different objects: the plaintext input on encrypt and the +encrypted input on decrypt. It does not copy or store the field implicitly; +the output layout must make that value available, normally with passthrough. +Use the explicit directional form when the names or sources differ. + +Use a separate field annotation to **store the computed context** when it is +not already carried by ordinary passthrough fields: + +```rust +#[stash(from_context)] +scope: u64, +``` + +`from_context` fills an output metadata field from the computed logical +context. It does not choose the context source. A passthrough source field +can already carry the context value, in which case no `from_context` field +is needed. No new source-side encryption methods or public `encrypt_as` +entry point are introduced. + +This is preferable to bare `context(in)` / `context(out)`: either term could +mean an operation direction, an input/output object, or storing a value. +`encrypt`, `decrypt`, and `from_context` give those roles distinct names. + +## Proposed vocabulary and semantics + +| Syntax inside `context(...)` | Meaning | +| --- | --- | +| `prefix = "users"` | Static base for this layout's field contexts | +| `field = "id"` | Shorthand for both `encrypt(field = "id")` and `decrypt(field = "id")` | +| `encrypt(caller = u64)` | Obtain the logical context from a required `u64` argument | +| `encrypt(field = "id")` | Obtain it from the plaintext input's `id` | +| `decrypt(caller = u64)` | Obtain it from a required `u64` argument on decrypt | +| `decrypt(field = "scope")` | Obtain it from the encrypted input's cleartext `scope` | +| `encrypt(tuple(field = "tenant_id", field = "id"))` | Build one ordered, typed pair from plaintext fields | +| `decrypt(tuple(field = "id", caller = u64))` | Build one pair from a stored field and the decrypt argument | + +`tuple(...)` is a declarative selector, not a Rust expression or a function +that gets access to keys. It can nest. `field` names resolve at compile time; +the first design supports direct fields, not arbitrary paths or callbacks. +There is at most one caller argument selector per direction; use a tuple or +typed context object as that argument when it carries several values. + +Use either the shared `field` shorthand or both explicit directional +selectors. For the initial API, reject mixing the shorthand with directional +selectors instead of inventing precedence rules. That keeps the expansion +predictable; a record with asymmetric names uses the explicit form. +`context = "users"` +remains the existing simple spelling for a static prefix with optional caller +extensions; it is used in scenario 0. + +### Composition rule + +Call the logical row context `R`. For a field named `name`, its static base is +`B = ("users", "name")` and its effective context is `(B, R)`. With only a +static prefix and no extension it is just `B`, not `(B, ())`. + +The selector's result is **one structured value**. A pair `(tenant_id, id)` +produces `(("users", "name"), (tenant_id, id))`, not +`((("users", "name"), tenant_id), id)`. `from_context` stores `R`, not the +field-dependent effective context. Static prefixes and field identities are +reconstructed from the declaration. + +Names in this proposal do not change existing byte encodings. Mapping new +selectors onto the existing context machinery must preserve types, order, +grouping, and descriptor rendering. A new explicit selector is a new +declaration; it must not silently reinterpret existing stored data. + +### Storage and expected context + +When all of `R` can be reconstructed from stored fields, the default decrypt +call uses those fields. The caller may additionally supply an expected `R` +through the existing `ExpectedContext` pattern: check equality first, then +use the stored value. That expected value is **not another extension**. + +When a decrypt selector needs a caller argument, that argument is required +to construct `R`; no no-context convenience is available. It is not an +expected-value check on context already fully available from the record. + +If a future caller needs both a required input and a separate expectation +on the assembled `R`, use a distinct checked-context input or a named API. +Do not overload one unlabelled argument to mean both. That combined check +API remains outside this proposal; scenario 7 makes the need visible. + +## Scenario matrix + +| Scenario | Encrypt obtains context from | Context available in stored output | Decrypt obtains context from | +| --- | --- | --- | --- | +| 0 | Static prefix; no row context | Declaration | Declaration | +| 1 | Caller argument | Not stored | Caller argument | +| 2 | Caller argument | Dedicated metadata field | Metadata field | +| 3 | Plaintext `id` | No cleartext copy; `id` itself is encrypted | Caller argument | +| 4 | Plaintext `id` | Passthrough `id` | Stored `id` | +| 5 | Plaintext `(tenant_id, id)` | Two passthrough fields | Same pair of stored fields | +| 6 | Plaintext `id` plus caller scope | Dedicated pair-valued metadata field | Stored pair | +| 7 | Plaintext `id` plus caller scope | Only `id` is stored in cleartext | Stored `id` plus caller scope | +| 8 | Plaintext `(tenant_id, id)` | Neither component is stored in cleartext | Caller supplies the pair | + +An encrypted copy of a context field does not make that context available +before decrypt. Scenarios 3 and 8 retain encrypted copies to recover the +complete plaintext object; callers still need the context independently. + +## Shared code for the examples + +Each scenario defines an output type and an async function. Combine this +prelude, the scenario blocks, and the final `main` into one candidate program. +They use real file serialization through the same helper, so no example +relies on a context value being secretly carried inside an in-memory handle. +All use proposed APIs and therefore cannot yet be typechecked against Stack. + +Dependencies match the workflow draft: Stack Encrypt, Serde with derives, +serde_json, tempfile, and Tokio. ZeroKMS credentials are required once these +APIs exist. `scope = 7` in caller examples is external application context, +not a keyset selector or an authorization credential. + +```rust +use serde::{de::DeserializeOwned, Deserialize, Serialize}; +use stack_encrypt::{ + Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, NonEmpty, + StackCipher, +}; +use std::error::Error; + +type Result = std::result::Result>; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +fn sample_user() -> User { + User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + } +} + +// The candidate Ciphertext Serde contract encodes its canonical envelope. +fn store_and_reload(value: &T) -> Result { + let directory = tempfile::tempdir()?; + let path = directory.path().join("record.json"); + std::fs::write(&path, serde_json::to_vec(value)?)?; + Ok(serde_json::from_slice(&std::fs::read(path)?)?) +} +``` + +## 0. Static prefix only + +This is the baseline. There is no dynamic row context to store or supply. +It retains the existing static-context form and proposed passthrough support. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct StaticUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_0() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StaticUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +For this shorthand, caller extensions remain available as previously agreed: +encrypt and decrypt must receive the same extension. The remaining examples +use explicit directional selectors, so their argument requirements follow +those selectors; there is no additional implicit extension layered on top. + +## 1. Caller context, not stored + +`scope` is absent from the output. The caller must provide it again. A +plaintext `id` in this layout is unrelated to the chosen context source. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(caller = u64), + decrypt(caller = u64) +))] +struct ExternalScopeUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_1() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let scope = 7_u64; + + let stored: ExternalScopeUser = user + .encrypt_into_with_context(&keyset, scope) + .await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Logical context: `7_u64`. Effective name context: `(("users", "name"), 7_u64)`. + +## 2. Caller context, stored in an output metadata field + +The source is still the function argument. `from_context` stores its value; +the decrypt selector explicitly chooses that stored field. `scope` is output +metadata, not a field to add to the recovered `User`. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(caller = u64), + decrypt(field = "scope") +))] +struct StoredScopeUser { + #[stash(from_context)] + scope: u64, + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_2() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StoredScopeUser = user + .encrypt_into_with_context(&keyset, 7_u64) + .await?; + assert_eq!(stored.scope, 7); + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Logical context remains `7_u64`. It is stored once, not appended a second time +on decrypt. This generalizes the existing caller-supplied-and-stored case to +the field-by-field layout. + +## 3. Plaintext field as context, no cleartext context stored + +The encrypt selector reads `User.id`; no argument is needed. The output's +`id` is encrypted, so it cannot supply the context required to open itself. +The reader obtains the id from its external record lookup. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(field = "id"), + decrypt(caller = u64) +))] +struct HiddenIdUser { + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_3() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let lookup_id = user.id; + + let stored: HiddenIdUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, lookup_id).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Both directions construct `R = 42_u64`. Supplying a different id fails to +open the encrypted fields. The id is not dropped or defaulted when rebuilding +`User`; its encrypted copy is opened using the externally supplied context. + +## 4. Plaintext field as context, stored as the corresponding output field + +This is the natural ID-bound form of workflow example 2. The context selector +and the passthrough annotation have separate jobs: select `id`, and copy it. +It uses the common-field shorthand, equivalent to the explicit pair of +directional selectors introduced above. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context(prefix = "users", field = "id"))] +struct IdBoundUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_4() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: IdBoundUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + + // Alternatively, require the stored context to match the lookup's id. + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context( + loaded, + &cipher, + NonEmpty::from(user.id), + ) + .await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +The second decrypt call uses the `ExpectedContext` conversion. It checks +the stored id instead of constructing `(stored_id, expected_id)`. Changing +`id` alone breaks decryption; moving an entire consistent record is detected +only when the reader supplies the expected id (or another placement check). +`created_at` remains ordinary unauthenticated passthrough metadata. + +## 5. Several plaintext fields as context, stored in separate fields + +Add a tenant id to the input. The context is the ordered pair `(tenant_id, id)`. +The output renames `id` to `user_id` to demonstrate why direction-specific +field selectors are useful. No duplicate context metadata field is needed. + +```rust +#[derive(Debug, PartialEq)] +struct TenantUser { + tenant_id: u64, + id: u64, + name: String, + email: String, + created_at: i64, +} + +fn sample_tenant_user() -> TenantUser { + TenantUser { + tenant_id: 7, + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + } +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = TenantUser)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "tenant_id", field = "id")), + decrypt(tuple(field = "tenant_id", field = "user_id")) +))] +struct TenantBoundUser { + #[stash(passthrough)] + tenant_id: u64, + #[stash(passthrough, from = "id")] + user_id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_5() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_tenant_user(); + + let stored: TenantBoundUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = TenantUser::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +Both directions construct `(7_u64, 42_u64)`. Rust field order does not choose +the context order: the selector does. A Serde wire rename also does not change +the selector; it names Rust fields, not serialized property names. + +## 6. Plaintext field plus caller argument, store the combined context + +Here `R = (User.id, caller_scope)`. `binding` stores that complete pair. The +original id is also retained as ciphertext so `User` can be reconstructed +without adding a new rule that maps pieces of metadata into plaintext fields. +That deliberately redundant representation keeps this example focused on +context composition rather than projection conveniences. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "id", caller = u64)), + decrypt(field = "binding") +))] +struct StoredCombinedUser { + #[stash(from_context)] + binding: (u64, u64), + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_6() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + + let stored: StoredCombinedUser = user + .encrypt_into_with_context(&keyset, 7_u64) + .await?; + assert_eq!(stored.binding, (42, 7)); + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from(loaded, &cipher).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +The encryption function accepts only the caller-supplied component (`7`), not +the pair the derive can construct itself. A checked decrypt would expect the +whole reconstructed pair `(42, 7)`, not just that caller component. + +## 7. Plaintext field plus caller argument, store only the field component + +Store `id` as a normal field, but keep the external scope out of the output. +Decryption combines the stored id with the scope supplied by the reader. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "id", caller = u64)), + decrypt(tuple(field = "id", caller = u64)) +))] +struct PartiallyStoredUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_7() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_user(); + let scope = 7_u64; + + let stored: PartiallyStoredUser = user + .encrypt_into_with_context(&keyset, scope) + .await?; + let loaded = store_and_reload(&stored)?; + let restored = User::decrypt_from_with_context(loaded, &cipher, scope).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +This also constructs `(42_u64, 7_u64)` in both directions. It shows why +"stored context" cannot always be represented as a single yes/no flag: +some components may be stored and others supplied externally. A separate +expected-id check can be performed against `loaded.id` before opening when +the application also needs to assert the record's destination. + +## 8. Several plaintext fields as context, keep neither component in cleartext + +The encrypt side obtains `(tenant_id, id)` from the source. The reader supplies +the complete pair from outside the encrypted record; both source fields are +recovered from ciphertext after that context has been established. + +```rust +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = TenantUser)] +#[stash(context( + prefix = "users", + encrypt(tuple(field = "tenant_id", field = "id")), + decrypt(caller = (u64, u64)) +))] +struct HiddenTenantUser { + tenant_id: Ciphertext, + id: Ciphertext, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +async fn example_8() -> Result<()> { + let cipher = StackCipher::new().await?; + let keyset = cipher.default_keyset(); + let user = sample_tenant_user(); + let lookup_context = (user.tenant_id, user.id); + + let stored: HiddenTenantUser = user.encrypt_into(&keyset).await?; + let loaded = store_and_reload(&stored)?; + let restored = + TenantUser::decrypt_from_with_context(loaded, &cipher, lookup_context).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +This is a new typed selector contract. It does not mean flattening the pair +into text, or treating separately chained extensions as interchangeable with +one tuple-valued argument. + +## Candidate program entry point + +```rust +#[tokio::main] +async fn main() -> Result<()> { + example_0().await?; + example_1().await?; + example_2().await?; + example_3().await?; + example_4().await?; + example_5().await?; + example_6().await?; + example_7().await?; + example_8().await?; + Ok(()) +} +``` + +## Consequences for the traits and derive + +The logical context `R` is not always the value a caller passes. For example: + +| Scenario | Logical `R` | Caller input on encrypt | Caller input on decrypt | +| --- | --- | --- | --- | +| 1 | `u64` | Required `u64` | Required `u64` | +| 2 | `u64` | Required `u64` | Optional expectation of `u64` | +| 3 | `u64` | None | Required `u64` | +| 4 | `u64` | None | Optional expectation of `u64` | +| 6 | `(u64, u64)` | Required `u64` component | Optional expectation of the whole pair | +| 7 | `(u64, u64)` | Required `u64` component | Required `u64` component | +| 8 | `(u64, u64)` | None | Required pair | + +Today `EncryptFrom::Context` and `DecryptInto

::Context` already belong +to separate traits and can differ. They describe the input the caller needs +to supply, not necessarily the final context used by every operation. That +distinction should be explicit in documentation; `ContextInput` might be a +clearer name, but a rename is not decided here. + +No-caller-input selectors can expose a defaultable input, such as `()`, to +support `encrypt_into`. Stored-context decrypt uses the existing +`ExpectedContext` idea. Required caller selectors must remain required +even if their Rust type implements `Default`: a required `u64` must not +silently become zero merely because `u64: Default`. + +That exposes a real constraint in today's convenience methods, whose only +test is `Context: Default`. An implementation of this proposal would need +a non-defaultable `RequiredContext` input (with an ergonomic conversion +from `T`), or a separate trait capability controlling context-free calls. +The examples show bare caller values at call sites, but do not assume that +the associated input type can be bare `u64`. Choose that mechanism before +claiming the directional API enforces required inputs. + +The current public `Encryption` constructors must also gain or expose the +core-owned source-selection machinery needed by the derive. Existing plan +context fields demonstrate source-dependent context, but are label-based, +carry the field, and use different composition rules. They are not already +an implementation of this proposal's general typed, optionally stored `R`. + +## Rules the implementation should enforce + +- Selectors for both directions must produce the same context type and + ordered structure. Different Rust field names are fine; different values + or grouping are not a supported way to encrypt and decrypt one record. +- Resolve field names and types at compile time. A decrypt selector cannot + read an encrypted field to obtain the context needed to decrypt it. +- A `from_context` field is cleartext output metadata with the logical + context's type. It cannot also be encrypted, defaulted, or independently + copied from an unrelated source. Ordinary passthrough fields do not + become context merely because their names are `id` or `tenant_id`. +- Extra metadata is not invented as a plaintext field during recovery. + Every original plaintext field must have a defined recovery path; no + silent omission or defaulting to compensate for a context-only source. +- Check supplied expectations before key retrieval where possible. A + context value read from storage remains untrusted until encrypted fields + authenticate under it. An expectation check alone is not authentication. +- If a selected source value is duplicated in stored context metadata and + in a separately recovered plaintext field, validate their agreement + before returning the reconstructed object. Do not return a different id + from the one the encrypted fields were bound to. +- Stored context should be required in the serialized schema; a missing + field must not silently fall back to a default context. Empty-context + policy must be explicit and retain the current nonempty target rule. +- For indexed targets, the selected context also scopes their terms unless + a future API explicitly separates index context from ciphertext context. + A row-specific id therefore changes search semantics; these examples + concern ciphertext layouts and do not silently promise cross-row queries. + +Required tests include storage round trips for every scenario, missing +required arguments, wrong expected values, altered stored context, tuple +reordering/regrouping, renamed source/output fields, and attempted circular +decrypt dependencies. Same-typed field swaps may evade a type check, so +round-trip and mismatch tests remain necessary even with typed selectors. + +## Alternatives worth comparing + +1. **Field-level `context(encrypt, decrypt)` flags.** Concise for one id, + but references the output declaration while the encrypt source lives + elsewhere, leaves multi-field order implicit, and cannot readily describe + caller-plus-field composition. Prefer explicit container selectors first. +2. **One `context = expression` callback.** Flexible, but hides dependencies, + makes reverse reconstruction and cycle checks harder, and encourages + arbitrary application code in the declaration. Keep simple declarative + selectors as the initial surface. +3. **A typed context object with a dedicated derive.** Could reduce repetition + for large composite contexts and carry a stronger encoding contract. + Consider it after the scalar/tuple cases establish the desired semantics. + +The recommendation is the directional selector plus independent storage +model, not a commitment to every spelling shown here. In particular, +`from_context`, `prefix`, the tuple notation, and how required inputs are +represented remain review choices. + +## Evidence and validation + +Current behavior was checked in +[context types](../../packages/stack-encrypt/src/target/context.rs), +[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), +[derive validation](../../packages/stack-encrypt-derive/src/shape.rs), and +[plan context fields](../../packages/stack-encrypt/src/plan/build.rs). +The directional nesting is inspired by Serde, not provided by Serde. + +These are candidate API examples. Syntax/link checks do not establish macro +support, type correctness, or encryption behavior. No live encryption or +database tests are claimed, and no library implementation is changed. + +Validation completed: `rustfmt` parsed the combined nine-scenario candidate +program and the introductory fragments (wrapped as Rust items), using edition +2024. Local links, Markdown fences, and trailing whitespace were checked in +this document and both companion documents. No proposed macro was expanded +or typechecked. diff --git a/docs/plans/2026-10-24-stack-encrypt-public-examples.md b/docs/plans/2026-10-24-stack-encrypt-public-examples.md new file mode 100644 index 000000000..ee700cabc --- /dev/null +++ b/docs/plans/2026-10-24-stack-encrypt-public-examples.md @@ -0,0 +1,609 @@ +# Stack Encrypt: complete public API examples for review + +Date: 2026-10-24. Companion to the +[API review log](2026-10-24-stack-encrypt-api-review.md). +The separate [context examples](2026-10-24-stack-encrypt-context-examples.md) +explore directional source selection, storing context, and decrypt inputs +across nine scenarios. Their attributes are proposals, not implemented APIs. +The [JSON operation contract](2026-10-25-stack-encrypt-json-operations.md) +covers the comprehensive first-release JSON workflows with Rust, dynamic +and SQL examples, including extraction and walks. + +The user clarified on 2026-10-25 that this API is an early prototype: +backward compatibility with its old APIs and encodings is not required. + +This document tests the proposed public API against three complete workflows: +encrypt, persist, load, and decrypt a user; the searchable workflow also +constructs and applies a query. It is an API design draft, not documentation +for a released version. The examples deliberately leave plans and operation +descriptions out of the application code. + +Each main example is a separate program, with all imports and types shown. +They use the same user fields and synthetic data. `created_at` is an `i64` +Unix timestamp in seconds so that date-library configuration does not distract +from the encryption API. The storage files live in a temporary directory and +are removed when the program exits. Files stand in for a database boundary; +the examples really encode, write, read, and decode their stored forms. + +## What is agreed, and what these examples propose + +**Ownership decision (2026-10-25):** `encrypt` and `encrypt_into` consume +their inputs, including context-taking variants. Their implementations must +guard owned plaintext and temporary buffers for cleanup on success, error, +and request drop; passthrough fields intentionally transfer to the output. +The single-ciphertext capabilities are named `OpaqueEncode` / `OpaqueDecode`. +Plaintext types must implement `Zeroize`; no new cleanup trait is planned. +The candidate programs below predate that bound: update their source derives +and any reuse of consumed input when making them executable. +`Serialize` alone does not establish the cleanup contract for `OpaqueBlob`. + +**Decrypt decision (2026-10-25):** retain direct `keyset.decrypt` and +`decrypt_with_context` for opaque ciphertext, alongside destination-directed +`T::decrypt_from` and `decrypt_from_with_context` for declared targets. +Successful calls return ordinary owned `T: Zeroize`; the caller then owns +cleanup responsibility. Stack guards intermediate plaintext. Support +explicit drop-guarded results without imposing a wrapper by default; +`Zeroize` alone does not wipe returned values on drop. These are accepted +API contracts, not a claim that the prototype already implements them. + +**Collection decision (2026-10-25):** direct `keyset.encrypt(value)` handles +one single-value input and produces `Ciphertext`; it does not automatically +encrypt collections element by element. Collection batching uses +`EncryptInto`, for example +`let encrypted: Vec = vec![1_i32, 3_i32].encrypt_into(&keyset).await?`. +`keyset.encrypt(OpaqueBlob::new(vec![1_i32, 3_i32])).await?` instead produces +one ciphertext for the encoded collection. The latest recommendation +applies the same rule to byte vectors, with no wrapper or special case +needed to preserve prototype behavior. These calls require the proposed +opaque ciphertext and no-context target support. + +| Surface | Status in this review | +| --- | --- | +| `keyset.encrypt(value)` and `encrypt_with_context(value, context)` | Agreed direction; current code still has the two-argument `encrypt` method | +| `value.encrypt_into(&keyset)` and `encrypt_into_with_context` | Retained existing entry points | +| No public `encrypt_as` in these examples | Agreed direction; current implementation still exposes it | +| `#[stash(passthrough)]` on output fields | Accepted in principle; needs implementation on both encrypt and decrypt derives | +| A fixed context on a scalar output derive | Agreed direction; not required by these particular examples | +| `OpaqueBlob` | Agreed wrapper direction; provisional name. Requests whole-value serialization and one encrypted leaf, using a library-owned codec | +| `Ciphertext` | Proposed name for an opaque single-value ciphertext; **not** today's `StackCipherText` or `CipherText` enum | +| `Ciphertext::to_bytes` / `from_bytes` | Proposed direct access to the existing canonical sealed-leaf encoding, without node extraction | +| `Ciphertext: Serialize + Deserialize` | Proposed storage contract: Base64 of the canonical bytes for human-readable serializers, bytes for binary serializers | +| Encoded ciphertext field targets | Accepted: dedicated ciphertext wrappers for Base64/Base85 storage compose with encrypt/decrypt targets; final names remain for the naming pass | +| `Ciphertext` as an encryption/decryption target | Proposed single-value target, including scalars and `String`; use `OpaqueBlob>` for a whole byte vector under the latest recommendation | +| Decrypt into `OpaqueBlob`, then `into_inner()` | Proposed symmetric decoding path; returns the original application value after authentication and decoding | +| Direct `decrypt` / `decrypt_with_context` | Accepted ordinary decrypt pair, symmetric with direct encryption | +| `User::decrypt_from` / `decrypt_from_with_context` | Accepted destination-directed pair on `DecryptFrom` | +| `SearchableEmail` in example 3 | Application-defined target composed from ciphertext and an existing `EqualityTerm` | + +The first three Rust blocks use proposed `OpaqueBlob`, `Ciphertext`, and/or +passthrough APIs. They are complete candidate programs, not programs claimed +to compile against today's crate. The names and contracts above remain +subject to review where marked proposed or provisional. + +The proposed opaque ciphertext is intentionally untyped: the decrypt call's +expected type selects decoding. A typed wrapper remains an open alternative. +Its byte decoder parses a stored envelope; authentication happens on decrypt. +Context and leaf encodings must agree across the new Rust and dynamic +implementations; preserving the prototype's encoding is not required. + +The trait name `DecryptFrom` is somewhat surprising: it supplies the +convenience functions, while `#[derive(DecryptInto)]` describes how an +encrypted output recovers plaintext. These examples keep that existing +distinction visible rather than silently redesigning it. + +## Setup shared by the programs + +The proposed programs use `stack-encrypt` with its default `http` feature, +`serde` with `derive`, `serde_json`, `tempfile`, and `tokio` with `macros` and +`rt-multi-thread`. No published version currently supplies all proposed APIs, +so this draft does not give a misleading installable Cargo manifest. + +`StackCipher::new()` uses the configured ZeroKMS credentials: a CLI login or +`CS_CLIENT_ID`, `CS_CLIENT_KEY`, `CS_CLIENT_ACCESS_KEY`, and `CS_WORKSPACE_CRN`. +The second client in each program represents a later reader using the same +authorized configuration. The default keyset is selected for brevity; an +already-selected tenant keyset can be used in its place. + +Decryption uses the client because the sealed value carries its keyset id. +It still requires access to the corresponding keys and the correct context. +The examples do not depend on in-memory fake keys surviving a reload. + +## 1. Encrypt the whole user through an explicit blob wrapper + +**Intent:** store one encrypted document. All four fields, including the ID +and creation time, are inside one encrypted serialization. There are no +independently readable fields or search terms in the stored ciphertext. + +`OpaqueBlob::new(user)` explicitly requests that treatment. Stack owns the +serialization and matching deserialization, rather than requiring every +application to select and invoke a codec. The proposed wrapper owns the input; +`new` only wraps it, and serialization errors surface from the encryption +operation. The context identifies the application's document schema and binds +this document to an externally known record id. + +For this draft the wrapper accepts `Serialize + Zeroize` on write and +`DeserializeOwned + Zeroize` on read. This does +not claim that arbitrary Serde implementations can satisfy a constant-time +contract. The initial wrapper has one library-selected, versioned binary +codec, not a caller-selected codec parameter. Selecting and specifying that +codec remains implementation/design work; no particular binary format is +claimed to be implemented or constant-time here. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{nonempty, Ciphertext, OpaqueBlob, StackCipher}; +use std::error::Error; + +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let expected_id = 42_u64; // Supplied by the application's record lookup. + let user = User { + id: expected_id, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + let expected = user.clone(); // Retained only for the round-trip assertion. + + // The wrapper requests one blob and lets Stack handle serialization. + let sealed: Ciphertext = keyset + .encrypt_with_context( + OpaqueBlob::new(user), + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + + // Persist the complete ciphertext envelope as binary bytes. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.ciphertext"); + std::fs::write(&path, sealed.to_bytes())?; + + // Later: reconstruct the ciphertext from storage using a new client. + let reader = StackCipher::new().await?; + let stored_bytes = std::fs::read(&path)?; + let loaded = Ciphertext::from_bytes(&stored_bytes)?; + + // Stack authenticates/decrypts first, then decodes the blob into User. + let opened: OpaqueBlob = reader + .decrypt_with_context( + loaded, + nonempty!("user-document-v1").with(expected_id), + ) + .await?; + let restored = opened.into_inner(); + assert_eq!(restored, expected); + Ok(()) +} +``` + +**What the API must guarantee:** the serialized bytes are one encrypted leaf +under one data key. The envelope contains the retrieval metadata, so the +caller never needs to save an IV or key tag separately. A different context +must fail to open it. Stack owns the matching encode/decode pair and its +compatibility contract. A blob-format discriminator and version must be +authenticated, for example inside the encrypted payload, so decoding does +not rely on guessing a codec or on unauthenticated format metadata. This +can reuse the canonical outer ciphertext envelope; the blob payload is a +new, explicit format rather than silently reinterpreting previously encrypted +raw bytes. Application schema evolution remains explicit: a codec version +does not automatically migrate changes to `User`. + +**Without application context:** use `keyset.encrypt(OpaqueBlob::new(user))` +form. A matching no-context `reader.decrypt(loaded)` is accepted for symmetry; +it is not the signature of today's two-argument decrypt method. This document +uses explicit context in the full example so the saved format is unambiguous. + +**Meaning of the wrapper:** `OpaqueBlob` is the plaintext-side request +and the decoded value, not the stored ciphertext type. `Ciphertext` remains +the persisted envelope. `into_inner()` returns an ordinary `User`. The +wrapper does not by itself promise to zeroize arbitrary input/returned types; +the codec should keep temporary plaintext buffers in wipeable storage. +`#[derive(Encrypt)]` on `User` currently means a structural ciphertext tree +and is not a substitute for wrapping the value as one blob. + +### Timing contract to investigate + +The desired property is serialization/deserialization whose timing does not +depend on secret contents, under an explicit policy for lengths and shape. +Constant time does not mean identical elapsed time for differently sized +messages; what may vary must be declared public. The general requirement is +secret-independent control flow and memory access, including outside the +cryptographic primitive. See [BearSSL's constant-time guidance](https://www.bearssl.org/constanttime.html). + +A library-owned binary codec gives us control over that implementation, but +does not establish the property by itself. Strings, collection lengths, +enum variants, validation errors, allocations, and user-defined Serde code +can all introduce value-dependent work. Supporting arbitrary `Serialize` / +`Deserialize` implementations cannot guarantee constant-time behavior for +the whole operation. + +Before claiming such a guarantee, define supported types, permitted public +shape/length information, and restrictions on custom implementations. A +stricter profile may need bounded shapes, fixed-width encodings, padding, +and controlled encoders/decoders, with implementation-level verification. +Padding ciphertext to a fixed size alone does not make the preceding +serialization constant-time. Authentication must complete before decoding +the plaintext blob. This is a design requirement to investigate, not a +guarantee made by these example programs or a constant-time audit result. + +**Investigation candidate: `serdect`.** It is already a dependency of +`stack-encrypt`. Review its supported encodings and precise constant-time +guarantees, including decoding/error behavior and length assumptions. Assess +where it can help with ciphertext/byte serialization and whether any of its +techniques apply to the `OpaqueBlob` codec. Do not assume guarantees for byte +encoding extend to serializing arbitrary objects through Serde. This is a +follow-up investigation, not a completed assessment or codec selection. + +## 2. Encrypt selected fields into a chosen output layout + +**Intent:** store an ordinary record with plaintext `id` and `created_at`, +and independently encrypted `name` and `email`. The output type states the +layout. The two encrypted fields need no search indexes. + +The proposed ciphertext Serde implementation makes persistence compositional: +ordinary Serde derives on the output encode plaintext fields normally and +ciphertext fields as Base64 envelopes. There is no manual `Node` traversal. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::{ + Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct EncryptedUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: Ciphertext, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let user = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + + // The output type selects the conversion and declares its own contexts. + let encrypted: EncryptedUser = user.encrypt_into(&keyset).await?; + assert_eq!(encrypted.id, user.id); + assert_eq!(encrypted.created_at, user.created_at); + + // Persist the typed output, including both plaintext and ciphertext fields. + let directory = tempfile::tempdir()?; + let path = directory.path().join("user.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: deserialize the same output layout and recover the plaintext type. + let reader = StackCipher::new().await?; + let loaded: EncryptedUser = serde_json::from_slice(&std::fs::read(&path)?)?; + let restored: User = User::decrypt_from(loaded, &reader).await?; + assert_eq!(restored, user); + Ok(()) +} +``` + +**Context and execution:** `name` uses the structured context +`("users", "name")`, and `email` uses `("users", "email")`. The keyset is +shared, but each encrypted field receives its own data key. One conversion +batches those key requests. The decrypt derive reconstructs `User`, opening +the ciphertext fields and copying the passthrough fields back. + +**Plaintext semantics:** copying `id` does not automatically bind the sealed +fields to it. Both plaintext fields remain unencrypted and unauthenticated. +That is the requested passthrough behavior, not a promise of whole-record +integrity. The example's name/email contexts distinguish columns, not rows. + +If the application wants row binding, it can use the existing +`user.encrypt_into_with_context(&keyset, expected_id)` entry point and the +matching `User::decrypt_from_with_context(loaded, &reader, expected_id)`. +The effective contexts then become `(("users", "name"), expected_id)` and +`(("users", "email"), expected_id)`. The reader supplies the expected id +from its record lookup. That is an explicit variation, not an automatic +effect of the passthrough attribute. + +**Existing `context_field` support:** the derive does have +`#[stash(context_field)]`, but its current behavior is to store the +caller-supplied context in an output field and use it again when decrypting. +It does not read an id from the plaintext `User`. It also supplies the complete +context: the macro rejects combining it with a literal `context = "users"`. +Because the current `struct = User` form requires that literal, the attribute +cannot currently be used on example 2's field-by-field declaration. + +**Separate source from storage:** obtaining context from plaintext fields +and carrying context into the encrypted output are independent choices. +An extended derive could read `User.id`, combine it with the declared field +contexts, and also copy it into `EncryptedUser.id`. That would enable row +binding without passing the id twice. The same model should accommodate +several source fields, such as `(tenant_id, id)`, with explicit ordering and +grouping. + +Alternatively, context can be passed as an argument and stored, passed and +not stored, or derived from plaintext fields without being stored. In either +non-stored case, the reader needs equivalent context from elsewhere before +decrypting. Source selection and storage should be separately expressible; +whether one annotation can conveniently request both remains a syntax choice. +Today's `context_field` only covers the caller-supplied-and-stored case. + +For the source-derived-and-stored case, a reader could use the stored id, +with an optional independently supplied expected id to reject a whole record +moved to another destination. Changing the stored id alone must fail +decryption. The values needed to reconstruct context must be available +without first opening the ciphertext that requires them. + +**Review point:** the declaration is concise, but requires an explicit +`EncryptedUser` type. A future source-side derive could generate that type; +it should generate this same layout and context behavior. Also decide whether +`User::decrypt_from(...)` is the desired public decrypt spelling before +adding a competing no-context `loaded.decrypt_into(...)` convenience. + +## 3. Use the same layout model with a searchable email target + +**Intent:** keep the same plaintext fields, encrypt the name normally, and +store the email's ciphertext plus an equality term. Search terms come from +the original typed plaintext through their declared operation, not from the +JSON encoding used to persist the output. + +`SearchableEmail` is an application-defined scalar target. It uses the caller +context supplied by the outer record: `("users", "email")`. It does not +declare another fixed context or append another `email` component. The +`ciphertext` and `equality` member names are output layout, not additional +cryptographic context segments. + +```rust +use serde::{Deserialize, Serialize}; +use stack_encrypt::sem::EqualityTerm; +use stack_encrypt::{ + nonempty, Ciphertext, DecryptFrom, DecryptInto, EncryptFrom, EncryptInto, + StackCipher, +}; +use std::error::Error; + +#[derive(Debug, PartialEq)] +struct User { + id: u64, + name: String, + email: String, + created_at: i64, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(plaintext = String)] +struct SearchableEmail { + ciphertext: Ciphertext, + equality: EqualityTerm, +} + +#[derive(EncryptFrom, DecryptInto, Serialize, Deserialize)] +#[stash(struct = User, context = "users")] +struct SearchableUser { + #[stash(passthrough)] + id: u64, + name: Ciphertext, + email: SearchableEmail, + #[stash(passthrough)] + created_at: i64, +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let alice = User { + id: 42, + name: "Alice".to_owned(), + email: "alice@example.com".to_owned(), + created_at: 1_700_000_000, + }; + let bob = User { + id: 43, + name: "Bob".to_owned(), + email: "bob@example.com".to_owned(), + created_at: 1_700_000_100, + }; + + // One collection conversion batches key generation across both users. + let users = vec![alice, bob]; + let encrypted: Vec = users.encrypt_into(&keyset).await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("users.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + // Later: a query uses the same keyset, field context, and equality operation. + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let wanted_email = "alice@example.com".to_owned(); + let query: EqualityTerm = wanted_email + .encrypt_into_with_context( + &query_keyset, + nonempty!("users").with("email"), + ) + .await?; + + // A file-backed comparison demonstrates matching without a SQL adapter. + let loaded: Vec = + serde_json::from_slice(&std::fs::read(&path)?)?; + let matches: Vec = loaded + .into_iter() + .filter(|row| row.email.equality == query) + .collect(); + assert_eq!(matches.len(), 1); + + // The query term cannot decrypt. Open the matched ciphertext rows in a batch. + let restored: Vec = Vec::::decrypt_from(matches, &reader).await?; + assert_eq!(restored.as_slice(), &users[..1]); + Ok(()) +} +``` + +The filter is an in-memory scan over persisted encrypted records, not a +claim of database index use. It demonstrates the entire query contract: +derive the term, compare it with the stored term, and open the matched +ciphertext. An equality match is not authentication of the stored record; +the decrypt operation authenticates its encrypted fields. Passthrough +semantics are the same as in example 2. + +Collection conversion is already supported by Stack's blanket +`EncryptFrom> for Vec` implementation when `T: EncryptFrom` +and its context supports cloning. It collects every row's pending operations +before dispatch: these two users require four data keys (name and email for +each), issued through one batched `generate_keys` call. Equality terms use +the current local index-key backend. This count concerns encryption work, +not client initialization or authentication. + +One logical `generate_keys` call is not an unlimited single HTTP request: +the KMS client chunks batches according to its configured key-count limit. +This four-key example fits in one normal request; larger collections may +span several requests without changing the public call. + +The matching collection decrypt implementation batches retrieval across +rows, with one `retrieve_keys` call per keyset represented in the ciphertext. +An empty collection requires no data-key call. Collection order is preserved +by the conversion; no new whole-collection authentication is implied. + +This target-directed vector is a vector of independent output records. It +is distinct from encrypting a plaintext vector into a single +`StackCipherText` sequence tree, and from wrapping the vector in `OpaqueBlob` +to encode/encrypt the whole collection as one value. Batching is a property +of execution, not a change to the chosen layout. + +This example uses the common field context for all rows so one query term +can search them. Adding a row id to that context would also scope the term +to that row. Separating ciphertext row binding from column-wide search +context is a further design question, not something these examples silently +promise. Likewise, applications querying across keysets need the appropriate +term per keyset; no cross-keyset comparison is implied. + +### EQL targets: a concrete adapter boundary + +The example above is a generic searchable layout, **not an EQL payload**. +For EQL, use the adapter's types, metadata, serialization, and query target. +For example, today's `eql_bindings::v4::text::TextEq` already represents an +email ciphertext plus equality term; `TextEqQuery` represents the query. + +The following separate field-level program uses existing APIs and avoids +the proposed `Ciphertext` and passthrough attributes. It needs `eql-bindings` +with its `stack-encrypt` feature, `stack-encrypt`, `serde_json`, `tempfile`, +and `tokio`. It is marked `no_run` because it needs live credentials; it has +been checked against source signatures here, not compiled or executed. + +```rust +use eql_bindings::{ + v4::text::{TextEq, TextEqQuery}, + Identifier, +}; +use stack_encrypt::{DecryptFrom, EncryptInto, StackCipher}; +use std::error::Error; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let writer = StackCipher::new().await?; + let keyset = writer.default_keyset(); + let column = Identifier::for_column("users", "email")?; + let email = "alice@example.com".to_owned(); + + let encrypted: TextEq = email + .encrypt_into_with_context(&keyset, column.clone()) + .await?; + + let directory = tempfile::tempdir()?; + let path = directory.path().join("email.eql.json"); + std::fs::write(&path, serde_json::to_vec(&encrypted)?)?; + + let reader = StackCipher::new().await?; + let query_keyset = reader.default_keyset(); + let query: TextEqQuery = email + .encrypt_into_with_context(&query_keyset, column.clone()) + .await?; + let loaded: TextEq = serde_json::from_slice(&std::fs::read(&path)?)?; + assert_eq!(loaded.hm, query.hm); + + // Supply the expected column, so stored identifier metadata is checked too. + let restored: String = loaded.decrypt_into(&reader, column).await?; + assert_eq!(restored, email); + Ok(()) +} +``` + +`TextEq` includes its EQL version, identifier, ciphertext, and equality term. +Its context is `NonEmpty`. The typed identifier comes from the +application's expected table/column on both writes and reads; decryption +checks it against the stored identifier before opening the ciphertext. +The `v4` namespace is intentional: this Rust producer profile targets EQL +v4, not the TypeScript EQL v3 integrations. + +There is a gap before we can simply replace `SearchableEmail` with `TextEq` +in the outer derive. That derive currently passes `CallerContext`; EQL's +target needs `NonEmpty` and carries its identifier in the output. +The context adaptation must be designed explicitly, including whether caller +extensions are representable. The EQL example demonstrates what that adapter +must preserve without inventing an implicit conversion or a nonexistent SQL +client API. PostgreSQL DDL, parameter binding, and index engagement are a +separate adapter example to add once that boundary is settled. + +## What to review across the three workflows + +| Question | Concrete choice made by these examples | +| --- | --- | +| What is one encrypted unit? | One `OpaqueBlob` encoding in example 1; one selected field in examples 2 and 3 | +| How is the layout selected? | Ordinary encrypt returns an opaque value; `EncryptInto` selects a declared output | +| How is persistence exposed? | Canonical bytes for one ciphertext; Serde composed over a typed layout | +| How is plaintext recovered? | Decrypt/decode into `OpaqueBlob` and unwrap, or use the target's `DecryptInto` declaration | +| How do indexes fit? | Additional field targets under the same layout/execution model | +| Who chooses context? | The application for a blob; the layout plus optional caller extensions for fields; an explicit identifier for EQL | +| Do examples need plans or visitors? | No; those remain implementation and advanced composition mechanisms | +| Is an encrypted output type generated automatically? | No; the layout examples deliberately define it | + +The principal unresolved choices exposed by the examples are the opaque +ciphertext name and storage contract, the blob +codec and timing contract, EQL context adaptation, and optional source-side output +generation. None requires reintroducing public `encrypt_as`. + +## Source checks and validation limits + +The draft was checked against the existing +[target traits and convenience methods](../../packages/stack-encrypt/src/target/operations.rs), +[context composition](../../packages/stack-encrypt/src/target/context.rs), +[derive implementation](../../packages/stack-encrypt-derive/src/encrypt.rs), +[decrypt derive](../../packages/stack-encrypt-derive/src/decrypt.rs), +[canonical leaf encoding](../../packages/stack-encrypt/src/cipher.rs), and +[EQL targets](../../packages/eql/crates/eql-bindings/src/v4/text.rs). + +All four programs were parsed successfully by `rustfmt` using Rust edition +2024. All local links resolve; code fences and whitespace were checked. +These checks establish syntax and document integrity, not type correctness. + +The candidate APIs are not implemented by this document. The three proposed +programs cannot yet be typechecked against the repository. No live encryption, +database execution, or cryptographic compatibility tests are claimed. Once the surface is agreed +and implemented, promote these programs to executable examples and verify +storage round trips, matching queries, and context mismatches. diff --git a/docs/plans/2026-10-25-stack-encrypt-json-operations.md b/docs/plans/2026-10-25-stack-encrypt-json-operations.md new file mode 100644 index 000000000..dc6d2e7e0 --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-json-operations.md @@ -0,0 +1,279 @@ +# JSON tree: operation inventory and release contract + +Updated: 2026-10-27 against merged `next` +`ffeed7adb3f7d8739cda53261046147852bffc82`. Companion to the +[API plan](2026-10-24-stack-encrypt-api-plan.md). + +The Go JSON feature is implemented through #1372/#1373/#1378/#1402. This +inventory distinguishes current support from remaining requirements. The +public Rust derive/facade example below remains proposed, not executable +code for the current engine. The +[refreshed Go plan](https://github.com/cipherstash/stack/blob/docs/go-json-tree-plan/docs/plans/2026-10-24-go-json-tree.md) +and its engine/Go findings document the exact source evidence. + +## Current names and policy + +| Role | Current public Go / dynamic / SQL name | +| --- | --- | +| Configured target | `JsonSearch` | +| Encrypted document | `eql.JSONSearch`, `public.eql_v4_json_search` | +| Containment / EqAt operand | `eql.JSONSearchQuery`, `eql_v4.query_json` | +| Ordering operand | `eql.JSONSearchLeafQuery`, `eql_v4.query_json_leaf` | +| Encrypted entry | `eql.JSONEntry`, `public.eql_v4_json_entry` | +| Prepared selector / subtree | `encrypt.JSONSelection`, `encrypt.JSONRead` | +| SQL capability values | `JSONSelector`, `JSONToken`, `JSONTokens` | +| Singular plaintext result | `JSONValue{Found, JSON}` | + +Handwritten Rust EQL adapters retain their `JsonTree*` names; the internal +engine `JsonDocument` is not the Go ciphertext type. `eql.JSON` is the +storage-only type. Do not use the old `query_json_tree` / `eql_v4_json_tree` +domains in current recipes. + +A field's configured target options are resolved into native `JsonOptions` +and reused for writes, queries and reads. The shipped profile uses column +identity and keyset selection. Separate row binding and search scope remain +accepted future API work, not supported EQL context inputs. Reject those +unsupported declarations; do not append row IDs to the existing shared +context. Queries do not gain row isolation merely because their returned +ciphertexts authenticate. + +## Required inventory and current support + +| ID | Operation | Current behavior / remaining work | +| --- | --- | --- | +| J01 | Encrypt object/array/scalar/null | Implemented; raw JSON supports every root, typed adapters support declared structs/maps/slices | +| J02 | Decrypt complete document | Implemented; raw JSON text or declared Go type | +| J03 | Batch write/read | Implemented; shared executor and provider chunking | +| J04 | Contains | Implemented `Contains`, `contains`, `@>` with `query_json`; generated needle guards apply | +| J05 | Contained by | Implemented `ContainedBy`, `contained_by`, `<@` | +| J06 | Scalar equality at path | Implemented `EqAt`, `eq_at`; value selectors, not ordering equality | +| J07 | Four ordering predicates | Implemented runtime paths and `At.` accessors; ordered number/string paths only | +| J08 | Inclusive range | Implemented `Between`, `between_at`, named lower/upper operands | +| J09 | Sort | Selector plus explicit SQL; wildcard sort needs defined reduction, not an assumed first array element | +| J10 | Path exists | Selector plus `json_tree_path_exists` | +| J11 | Extract one match | Selector plus `->` / path-query-first; wildcard choice is unspecified | +| J12 | Extract as encrypted text | `->>`; still encrypted entry JSON, not plaintext | +| J13 | Enumerate all matches | SQL path query exists; bounded wildcard subtree is the supported grouped plaintext route | +| J14 | Open extracted scalars | `JSONSelection.Open` handles one optional entry per document group. Multi-entry selector groups are refused; do not label them supported | +| J15 | Prepare positional/wildcard read | Implemented `Subtree`, `subtree`; token list plus retained trusted request | +| J16 | Fetch/open subtree | `json_tree_subtree(document, tokens, budget)`, `JSONRead.Open` / `OpenOne` | +| J17 | Immediate children | `Children` prepares a concrete token; SQL returns child entries. Parent-bound child-scalar opening remains unimplemented | +| J18 | Count children | `ChildCount` token and SQL nullable count; not authenticated completeness | +| J19 | Budget/fallback | Implemented, with original requested selection applied after root fallback | +| J20 | Stored-value comparisons | Existing document containment and entry ordering, within compatible column/keyset/profile | + +The comprehensive release requirement remains. Do not equate closed tracking +issues with implementation of a parent-bound child opener, or feed child +rows into an unrelated selector/subtree request. Complete that opener for +the J17 plaintext-child requirement; until then, fetch and open the parent's +subtree to inspect its children, documenting the different transfer/leakage +cost. SQL all-match enumeration is distinct from the supported bounded +subtree plaintext interface. + +## Semantics that changed or are now implemented + +- #1379 randomizes stored entries, object ordinals and containment projection + order. `->` at a wildcard selects an unspecified element. Use all matches + with an explicit `EXISTS`, `MIN` or `MAX` rule for ordering/sorting. +- Cascade authentication commits containers to their child ciphertext tags. + Nonempty standalone container entries fail `JsonNotALeaf`, also used for + alteration; empty container entries authenticate but fail `OpenedType`. + All ordinary containers need a subtree read. A requested sealed root can + still open as a complete document. +- The dynamic engine now rejects positional selector/ordering/EqAt paths. + Concrete `[n]` positions belong to subtree reads; wildcard selectors address + a path class. Independent EqAt predicates need not match one array element; + one containment needle preserves their correlation. +- The strict Rust parser now rejects duplicate keys and out-of-range integer + literals before key I/O. Typed Go encoding occurs first and follows + `encoding/json`, including invalid-UTF8 replacement. Raw JSON is checked + directly. Neither path promises byte-for-byte JSON text preservation. +- SQL NULL, missing selection, JSON null and empty containers are distinct. + `JSONValue.Found` carries selection presence. `OpenOne` rejects wildcards; + plural subtree reads return requested values in document order. +- Prepared reads retain the cipher, field definition, path and wildcard + bound. `Open(ctx, rows)` asks Rust to rederive and verify the request. + No serialized guest-local key handle is required. Rows cannot supply their + own expected path. Keep one group per database document. +- Cascade strengthens verification beneath a returned root. It does not + authenticate the whole result set, absence, freshness or the intended + database row. Whole results/probes can still be withheld. +- Wildcard bound defaults to 64 with boundary probes and an 8,192-token cap. + Walk budget independently defaults to 64; zero disables root fallback. + An open-root document cannot use sealed-root fallback. The budget is not + a universal response-size cap. +- Header version remains 1 despite the changed cascade/AAD semantics. Freeze + the post-cascade profile explicitly; the unchanged header is not evidence + of compatibility with older prototype ciphertexts. Randomized query + projection order requires canonical semantic comparisons in parity tests. + +## Current SQL recipes + +The examples assume `profile public.eql_v4_json_search`. Parameters are the +SDK's generated query/selector/token values, never plaintext paths. Add the +application's authorization and tenant predicates as usual. + +```sql +-- Contains / contained by / EqAt operands. +SELECT id, profile FROM users WHERE profile @> $1::eql_v4.query_json; +SELECT id, profile FROM users WHERE profile <@ $1::eql_v4.query_json; + +-- A single-valued ordered path: selector, lower bound, upper bound. +SELECT id, profile FROM users +WHERE (profile -> $1::text) >= $2::eql_v4.query_json_leaf + AND (profile -> $1::text) <= $3::eql_v4.query_json_leaf +ORDER BY eql_v4.ord_term(profile -> $1::text) ASC NULLS LAST; + +-- Wildcard ordering: explicitly existential, not an unspecified -> match. +SELECT u.id FROM users AS u +WHERE EXISTS ( + SELECT 1 + FROM eql_v4.json_tree_path_query(u.profile::jsonb, $1::text) AS x(entry) + WHERE x.entry > $2::eql_v4.query_json_leaf +); + +SELECT id FROM users +WHERE eql_v4.json_tree_path_exists(profile::jsonb, $1::text); +SELECT id, profile -> $1::text AS entry FROM users; +SELECT id, profile ->> $1::text AS encrypted_entry_text FROM users; + +-- Token list, then walk budget. Group entries by id before opening. +SELECT u.id, w.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_subtree( + u.profile, $1::text[], $2::integer +) AS w(entry); + +-- Children/count take one concrete node token, with no gates parameter. +SELECT u.id, c.entry FROM users AS u +CROSS JOIN LATERAL eql_v4.json_tree_children(u.profile, $1::text) AS c(entry); +SELECT id, eql_v4.json_tree_child_count(profile, $1::text) FROM users; +``` + +Child count is zero for scalars/empty containers and NULL for a missing node +or the sealed root. Do not convert NULL to zero. The former token/gates and +`json_tree_subtree_at` recipes are removed. Current full Go programs and +parameter binding examples live in +[the JSON example](../../languages/golang/encrypt/example/json/main.go). + +## Current dynamic grammar and results + +This is implemented data-plan syntax, without an illustrative versioned +outer envelope: + +```json +{ + "profile": { + "type": "json", + "context": ["users", "profile"], + "target": { + "name": "JsonSearch", + "options": {"sealed_root": true, "ordered": ["$.age"]} + } + } +} +``` + +Operations select the field through the ABI. Their named inputs include +`{"json"}`, `{"path","json"}`, `{"path","low","high"}`, `{"path"}` +and `{"path","wildcard_bound"}`. JSON values are UTF-8 JSON text. Query +results identify `encoding` (`eql`, `text`, `text_array`, or scalar `bytes`), +SQL domain and operands. Capabilities identify input shapes. Opening uses +`json_open` / `se_json_open` with the original `selector` or `subtree` +request and grouped encrypted rows. The general versioned definition/schema +package remains required work; it is not the current runtime grammar. + +## Rust write/read example + +Proposed public code. `target(column = ...)` is illustrative declaration +syntax for an explicit EQL column identity; the exact attribute spelling +belongs to the forthcoming specification/naming pass. The options factory +is typed Rust, not JSON hidden inside an attribute. + +```rust +use eql_bindings::v4::json_tree::JsonTreeDocument; +use serde::{Deserialize, Serialize}; +use stack_encrypt::{DecryptFrom, DecryptSource, EncryptInto, EncryptTarget, StackCipher}; +use stack_encrypt::json::{JsonOptions, JsonPlaintext}; +use stack_encrypt::sem::TermError; +use std::error::Error; +use zeroize::Zeroize; + +fn profile_options() -> Result { + JsonOptions::new().ordering(["$.age", "$.visits[*].date"]) +} + +#[derive(Serialize, Deserialize, EncryptTarget, DecryptSource)] +#[serde(transparent)] +#[stash( + plaintext = JsonPlaintext, + transparent, + target(column = ("users", "profile"), options = profile_options) +)] +struct EncryptedProfile(JsonTreeDocument); + +#[tokio::main] +async fn main() -> Result<(), Box> { + let client = StackCipher::new().await?; + let keyset = client.default_keyset(); + let _definition = EncryptedProfile::definition()?; + + let profile = JsonPlaintext::parse( + r#"{"age":42,"name":"Alice","visits":[{"date":"2026-10-01"}]}"#, + )?; + let encrypted: EncryptedProfile = profile.encrypt_into(&keyset).await?; + + // These bytes can instead be bound to the EQL PostgreSQL column. + let stored = serde_json::to_vec(&encrypted)?; + let loaded: EncryptedProfile = serde_json::from_slice(&stored)?; + let mut opened = JsonPlaintext::decrypt_from(loaded, &keyset).await?; + // The caller now owns the returned plaintext and its cleanup. + opened.zeroize(); + + let profiles = vec![ + JsonPlaintext::parse(r#"{"age":42}"#)?, + JsonPlaintext::parse(r#"{"age":27}"#)?, + ]; + let encrypted: Vec = profiles.encrypt_into(&keyset).await?; + let mut opened = Vec::::decrypt_from(encrypted, &keyset).await?; + assert_eq!(opened.len(), 2); + opened.zeroize(); + Ok(()) +} +``` + +This example explicitly declares column-wide scope and no row binding by +its fixed standalone definition; the normative schema must express that +choice rather than silently replacing missing required tenant scope. +For an output-derived `EncryptedUser` with `id: PrimaryKey` and +`tenant_id: ScopeKey`, select those fields through the context/scope +attributes described in the API plan. Obtain its JSON field handle from +`EncryptedUser::definition()?.profile()` and require `TenantId` on query +requests. Store/return both ID fields for decrypt. If caller input is used, +use the agreed `_with_context` variants with a typed input separating +binding and scope. No row ID is ever an index/query input implicitly. + +## Remaining work and release gates + +1. Capture the implemented wire grammar, errors, presence states and context + profile in the promised formal specification and conformance corpus. + General Rust definitions/caching and additional context roles can follow. +2. Close the explicit J14/J17 scope gaps. The supported all-match plaintext + route is subtree reads; a parent-bound child opener is separate engine + work. Do not add generic raw-entry decryption as a shortcut. +3. Replace the internal EQL serialize-to-Value/deserialize bridge with typed + conversion while preserving the frozen profile. This is an optimization, + not a missing Go JSON execution path. +4. Run the existing Rust/guest/Go/PostgreSQL suites on the release candidate, + mapping tests to J01–J20. Include typed/raw inputs, numeric/null cases, + both root policies, wildcard bounds, fallback, cascade tampering and known + omission limitations. Inspect real SQL parameter casts and index use. +5. Freeze post-cascade storage/search vectors and the matching EQL bundle. + Compare canonical projection semantics and cross-opening; ordinary + randomized bytes need not match. Replace temporary Go Vitamin C pins + before a final release. Opaque storage adoption is already done and is + a separate format from JSON-tree storage. + +This refresh is source and documentation review, not a fresh runtime test +run or a complete cryptographic audit. Final Rust example compilation, +package extraction and stronger zeroization/constant-time work remain later +API-plan activities; they must preserve the released Go/data contract. diff --git a/docs/plans/2026-10-25-stack-encrypt-packaging.md b/docs/plans/2026-10-25-stack-encrypt-packaging.md new file mode 100644 index 000000000..09c7bd248 --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-packaging.md @@ -0,0 +1,268 @@ +# Stack Encrypt packaging and public names + +Date: 2026-10-25. Recommendation for review, accompanying the +[implementation plan](2026-10-24-stack-encrypt-api-plan.md). +These are proposed boundaries and names, not implemented APIs or accepted +renames. Existing companion examples retain their working names until this +proposal is accepted. The detailed JSON operation contract remains under +separate user review. + +**Implementation update, 2026-10-27:** Go JSON and opaque storage have +landed on `next` through #1369/#1370/#1372/#1373/#1378/#1402; #1397 adds +short whole-word match terms. The customer path is remaining configuration +and conformance work plus release readiness, including replacing temporary +Vitamin C Go module pins. Rust stays on crates.io Vitamin C 0.8.0. Neither +crate extraction nor final Rust facade completion is a customer release gate. + +Source baseline: the current documentation branch and inspected local Vitamin +C 0.8.0 sources. Stack's +[cipher implementation notes](../../packages/stack-encrypt/src/cipher.rs) +explain its existing batching boundary. The +[EQL manifest](../../packages/eql/crates/eql-bindings/Cargo.toml) establishes +the optional downstream engine dependency, and +[encoding wrappers](../../packages/eql/crates/eql-bindings/src/encoding.rs) +are the concrete shared-type extraction candidates. This is a source-level +design assessment, not an audit of every external Vitamin C consumer. + +## Ownership rule + +Stack Encrypt owns the application value model and its execution: turning +owned Rust/dynamic values into ciphertext, structured targets and search +operands, and reconstructing plaintext. Vitamin C owns reusable cryptographic +and protected-memory primitives. A normal Stack Encrypt user should not need +to import Vitamin C encryption traits to implement an application type. + +Moving code should remove duplicate abstractions, not preserve both public +surfaces under new names. The user's prototype compatibility exemption +applies to this API; review other Vitamin C consumers before deleting their +dependencies upstream. + +## Proposed packages + +| Package | Responsibility | +| --- | --- | +| `stack-encrypt` | Public Rust API, value traversal/reconstruction, target declarations and definitions, context/scope roles, normalization and searchable-target adapters, plans, batching, JSON-tree execution, and dynamic definition validation/execution | +| `stack-encrypt-derive` | Derives generating the same declarations and definitions available by hand; re-export derives from `stack-encrypt` | +| `stack-encrypt-format` — new | Pure data representations/codecs shared by the engine and bindings: frame primitives and required scalar representations, stable blob profile, ciphertext envelope parsing/encoding, and validated Base64/Base85 text; no key requests or execution runtime | +| `stack-zerokms` — rename current `stack-kms` | ZeroKMS protocol client, authentication integration, native keysets, lock-context request types, and responses; it is not the neutral multi-provider abstraction | +| `eql-bindings` | EQL versioned storage/query types, SQL-compatible JSON serialization, catalog and Stack target adapters; stays downstream of the engine | +| Vitamin C crates | Protected memory, randomness, byte-level AEAD/key operations, HMAC/PRF and ORE algorithm primitives, and reusable canonical context encoding | + +Keep `stack-auth` and `stack-profile` separate: they have consumers outside +encryption. Keep `stack-guest-abi` for the host/guest calling convention. +Neither the ABI nor the Go host should own a second target-policy model. +Put the formal dynamic contract, schemas and conformance fixtures beside +Stack Encrypt; its implementation uses the same definition types and +validation as static Rust. + +Use modules for the remaining boundaries initially. In particular, do not +create separate execution, JSON-policy, query, context or provider-traits +crates merely to mirror each concept. + +The format crate has a concrete consumer justification: EQL's wire types +currently work without its optional `stack-encrypt` integration. They need +shared encoding types without an encryption client/runtime dependency. Make +Serde, schema and TypeScript support feature-scoped where practical; define +implementations alongside the owned types so downstream orphan rules do not +force duplicate wrappers. Re-export the ordinary format types through Stack +Encrypt, so users normally need only its dependency. + +The format crate is a single implementation with distinct profiles. Blob +storage, host/guest transport, and ciphertext envelopes have different +contracts and versioning requirements; sharing code must not silently make +the ephemeral transport Serde mapping the persisted blob format. This does +not introduce persistence for arbitrary internal `Node` trees: structured +outputs still use their explicit target layout and ordinary Serde composition. + +## Dependency direction + +```mermaid +flowchart TD + App[Application / Rust or guest adapter] --> EQL[eql-bindings] + App --> Engine[stack-encrypt] + EQL -->|optional target integration| Engine + EQL --> Format[stack-encrypt-format] + Engine --> Format + Engine --> Derive[stack-encrypt-derive] + Engine -->|optional ZeroKMS adapter| KMS[stack-zerokms] + Engine --> Vitamin[Vitamin C primitives] +``` + +The derive emits Stack paths but does not depend on the engine as a normal +runtime dependency. No Vitamin C primitive depends back on Stack Encrypt. +`stack-zerokms` does not depend on the engine. Provider traits belong to +`stack_encrypt::provider`; a Stack-owned adapter implements them using the +ZeroKMS client. This avoids an engine/client dependency cycle. + +Make ZeroKMS an optional provider feature independently of the HTTP feature, +so a different provider does not pull its protocol into the core contract. +Preserve injected providers and guest/native builds. Data-key support is the +base capability; index-key support, named-keyset resolution and lock context +remain explicit optional capabilities. Unsupported required capabilities +fail before key I/O. Future provider adapters can become separate downstream +crates when needed; do not design fictitious AWS/Vault keysets now. + +Do not re-export EQL from Stack Encrypt: EQL's target integration already +depends on Stack Encrypt. Applications explicitly import their EQL types. + +## What moves out of Vitamin C + +Move application-facing plaintext/ciphertext visitors, structural nodes, +value conversion traits and derives, tree walking, collection composition, +and application encryption helpers into Stack's value/target/plan layers. +Keep advanced extension points public only where target implementers need +them; they are not the normal root API. Internal nodes do not become a +second public opaque ciphertext type. + +Move the generic frame machinery and codecs needed by Stack into the format +crate. Stack owns the stored blob profile and its compatibility promise. +This refines the earlier recommendation to reuse Vitamin C's frame codec: +reuse its implementation, with a lossless storage mapping, while changing +its long-term ownership. Coordinate this handoff under +[Vitamin C #519](https://github.com/cipherstash/vitaminc/issues/519); retain +the issue's format specification and fixture work. No upstream issue or +repository has been changed by this proposal. + +Also move normalization and PRF/ORE adapters that understand Stack's logical +values, traversal, targets or query semantics. Keep independent cryptographic +algorithms below them. The current `vitaminc-normalize`, `vitaminc-prf` and +`vitaminc-ore` manifests depend on `vitaminc-aead`; normalization even uses +its physical scalar types. Moving only the encryption traits would therefore +leave a dependency back to the old value model. Separate primitive inputs +from application adapters as part of the move; do not add a reverse +Vitamin C dependency on Stack to make it compile. + +Retain canonical context byte construction in `vitaminc-context` where it +serves independent algorithms. Stack owns the semantic roles and selection +rules (`PrimaryKey`, `ScopeKey`, ciphertext context, query scope and caller +input capabilities), reusing that canonical encoding instead of inventing +an incompatible second one. Provider policy remains a separate role. + +## Retire AsyncCipher + +Recommend removing Vitamin C's `AsyncCipher` and its associated application +execution helpers as the Stack executor takes ownership. Do not introduce a +replacement public catch-all async cipher trait in Stack. + +Stack already plans and groups remote key operations, then performs local +synchronous cryptography. Its current `MintedKeys`/`BoundKeys` implementations +use synchronous leaf cipher operations; `AsyncCipher` models a different +place to put asynchronous work. The meaningful async boundary is the provider +and the public pending operation, with batching controlled by Stack. + +It is unused by Stack, but not unused inside Vitamin C: `aead/src/seal.rs`, +`context.rs`, `typed.rs`, `walk.rs` and `normalize/src/terms.rs` expose or use +async helpers, and their tests implement it. Migrate the relevant batching, +ordering, failure and reconstruction coverage with the code, then delete +the retired trait/helper surface. Audit external consumers before the +upstream release. Separate internal leaf seal/open capabilities where useful +instead of retaining implementations that reject half of a combined trait. + +## Recommended public glossary + +| Name/path | Meaning | +| --- | --- | +| `stack_encrypt::Cipher` | Provider-backed execution handle; replaces `StackCipher` | +| `KeysetCipher` | Execution handle bound to a resolved keyset, available for providers supporting keysets | +| `Ciphertext` | One opaque encrypted value with a specified storage envelope; replaces the public use of `StackCipherText = Node` | +| `OpaqueBlob` | Explicit owned whole-value serialization adapter | +| `EncryptInto`, `DecryptFrom` | Target-directed encryption and destination-directed decryption convenience traits, with context variants | +| `OpaqueEncode`, `OpaqueDecode` | Single-ciphertext representation traits | +| `target::{EncryptTarget, DecryptSource}` | Declaration traits; matching derives are re-exported at the crate root | +| `target::TargetDefinition` | Validated immutable definition shared by static and dynamic paths | +| `context::{PrimaryKey, ScopeKey}` | Eligibility wrappers; neither automatically selects a context or keyset | +| `encoding::{Base64Text, Base85Text}` | Validated encoding of arbitrary bytes; owned in the format crate and re-exported here | +| `encoding::{Base64Ciphertext, Base85Ciphertext}` | Validated encoded opaque-ciphertext representations and supported field targets; encoding validity alone is insufficient | +| `plan::Plan` | Explicit low-level composition interface; output selection is `.target::()` | +| `provider` | Neutral provider contracts and capability bounds | +| `provider::zerokms` | ZeroKMS adapter and policy extension surface, including `LockContext` | +| `dynamic` | Versioned definition loading, capability discovery and execution through the common engine | + +Keep the already-agreed `.using(...)`, `definition()`, `encrypt`, +`encrypt_with_context`, `encrypt_into`, `encrypt_into_with_context`, +`decrypt_from` and `decrypt_from_with_context` spellings. Keep direct +`cipher.decrypt`/`decrypt_with_context`. Remove plaintext `.encrypt`, +redundant `encrypt_as`, and public `decrypt_into` as already agreed. + +Use `TargetDefinition` for the configured public contract and `Layout` only +for the advanced internal composition concept where that distinction is +needed. A layout does not name a plaintext document or a stored ciphertext. +Name low-level operations `seal`/`open` where they act on bytes; reserve the +application encrypt/decrypt vocabulary for the public value operations. + +Keep ciphertext representations in the format crate independent of traits; +Stack implements its own target traits for those types. Plain encoded text +is not itself an encryption target. Construction must validate both layers +for the ciphertext wrappers; avoid an unchecked public string constructor. + +## JSON names by role + +Keep the existing EQL package and version modules. The merged public Go +surface is `eql.JSONSearch`, `JSONSearchQuery`, `JSONSearchLeafQuery` and +`JSONEntry`, configured through `JsonSearch`; SQL uses `eql_v4_json_search`, +`query_json`, `query_json_leaf` and `eql_v4_json_entry`. Prepared Go objects +are `encrypt.JSONSelection` and `JSONRead`. + +The table below remains a proposed Rust facade glossary. Its `JsonTree*` +names describe handwritten Rust adapters, not the registered Go target. +Do not rename the working Go/SQL surface to this glossary as a customer +release prerequisite. Later Rust renames must preserve frozen dynamic IDs. + +| Working/current name | Recommended name | Role / home | +| --- | --- | --- | +| `JsonPlaintext` | `JsonPlaintext` | Owned plaintext adapter, `stack_encrypt::json` | +| `JsonOptions` | `JsonTreeOptions` | Validated tree-target policy, `stack_encrypt::json` | +| `JsonSelection` | `JsonSelection` | Proposed plaintext selection result, `stack_encrypt::json` | +| `JsonTreeDocument` | `EncryptedJson` | Stored EQL encrypted document, `eql_bindings::v4` | +| `JsonTreeEntry` | `EncryptedJsonEntry` | Extracted encrypted entry, `eql_bindings::v4` | +| `JsonTreeSubtree` | `EncryptedJsonSubtree` | Encrypted subtree result, `eql_bindings::v4` | +| `JsonTreeQuery` | `JsonContainmentQuery` | Containment/path-value query operand, `eql_bindings::v4` | +| `JsonTreeLeafQuery` | `JsonOrderingQuery` | Ordering query operand, `eql_bindings::v4` | +| `JsonTreeSelector` | `JsonSelector` | Encoded path selector, `eql_bindings::v4` | +| `JsonSubtreeKey` | `JsonWalkRequest` | Client-held walk token and request binding; engine type with a typed EQL adapter | + +`JsonContainmentQuery` can still be the physical operand for a path equality +operation where the agreed SQL recipe uses containment; operation and wire +operand need not have the same name. `JsonWalkRequest` describes the actual +role better than "key": it carries sensitive capability/request state, +not the data-encryption key. Exact operation behavior remains governed by +the separate JSON contract review. + +Rename the internal target index `Json` to `JsonTreeIndex`. Keep the internal +engine `JsonDocument` out of the ordinary facade, with a role-qualified name +such as `EncryptedJsonTree` if adapter authors require public access. +`EncryptedJson` always denotes the EQL-compatible stored form, not any +arbitrary JSON serialization of encrypted nodes. An application's +`EncryptedUser` remains its own declared record layout. + +Dynamic target/operation identifiers are explicitly versioned contract +identifiers, not Rust type names serialized into JSON. Select their exact +spellings in the interoperability specification and use the same glossary +in Go. Rust renames alone must not accidentally redefine stored SQL formats +or dynamic discriminants. + +## Implementation order and acceptance + +1. Specify and validate the now-implemented Go JSON and opaque storage + contracts. Freeze post-cascade JSON/search semantics and exact fixtures. + Replace temporary Vitamin C Go module pins for a final release. +2. Complete outstanding Go match configuration and JSON read/conformance + gaps. Short whole-word matching is implemented in EQL; its leakage and + profile compatibility still require explicit release acceptance. +3. After delivery, establish the shared format crate and move the value model, + traversal and dependent normalization/index adapters into Stack. Preserve + frozen encodings and an acyclic graph; move tests with their code. +4. Complete neutral provider extraction and protocol-package renaming, then + retire Vitamin C's async application execution surface. Required Go-facing + policy and routing contracts must already have been handled before release. +5. Complete public Rust trait/derive names, imports and executable examples. + These changes must not rename the shipped Go API, reinterpret stored + profiles or require customer re-encryption/reindexing. + +Acceptance includes an acyclic Cargo graph; an EQL wire-only build without +the engine; engine builds without ZeroKMS and without HTTP; native/guest +builds; compiled public examples; the moved behavior tests; and shared +Rust/Go/SQL format/operation fixtures. Full timing and memory-cleanup +improvements remain follow-up work under the accepted format-first priority; +record actual guarantees honestly. No implementation changes, tests of the +proposed API, code moves, or upstream removals are claimed by this document. diff --git a/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md new file mode 100644 index 000000000..0ce88185e --- /dev/null +++ b/docs/plans/2026-10-25-stack-encrypt-provider-capabilities.md @@ -0,0 +1,351 @@ +# Key selection, lock context and provider capabilities + +Date: 2026-10-25. Assessment accompanying the +[API implementation plan](2026-10-24-stack-encrypt-api-plan.md). + +Status: recommendations for review, not accepted public signatures or an +implementation. The user requested plaintext-derived ZeroKMS keyset +selection, review of Vitamin C `AsyncCipher`, and optional lock-context +support without requiring future AWS KMS/Vault backends to emulate it. +The JSON operation contract remains under user review independently. + +## Distinct roles + +| Role | Selects/binds | Example | +| --- | --- | --- | +| Key selection | Provider key domain used to encrypt and derive index terms | ZeroKMS keyset ID or name selected from `tenant_id` | +| Ciphertext context | Application binding verified when opening ciphertext | Column + tenant scope + row ID | +| Search scope | Domain in which index/query terms compare | Column + tenant ID | +| Provider policy | Conditions the provider applies to key generation/retrieval | ZeroKMS lock context selecting authenticated JWT `sub` | + +These are separate logical inputs, even where the protocol binds several +into its tag/AAD. A `ScopeKey` does not select a ZeroKMS keyset by itself; +a keyset selector does not insert its source field into AAD; lock context +is not an alias for either ciphertext context or search scope. Sharing a +source value requires explicit declaration, not inference from a name. + +## Current implementation evidence + +**Update, 2026-10-27:** #1379 makes open-root JSON verification load the +index keyset as well as the data key. `Pending` settlement and generic +execution entry points now require `DataKeySource + IndexKeySource`. +The future neutral provider split below must preserve that verification +requirement for JSON while allowing data-key-only capabilities where an +operation genuinely needs no index key. This change does not implement +plaintext-derived keyset routing or ZeroKMS lock-context plumbing. + + +- [StackCipher::keyset](../../packages/stack-encrypt/src/cipher.rs) accepts + `IdentifiedBy` by UUID or name, resolves the keyset and its index key, + and returns a bound `KeysetCipher`. The client has a bounded keyset cache + and name-resolution freshness policy. `.default_keyset()` is already + bound at initialization. Current initialization requires both + `DataKeySource` and `IndexKeySource`. +- A sealed leaf carries the resolved keyset UUID. Client-scoped decrypt + groups retrieval by the UUIDs in the leaves; a bound keyset can constrain + which keyset is permitted. Names are lookup conveniences, not persistent + key identities. See [ADR 0002](../../packages/stack-encrypt/docs/adr/0002-keyset-id-in-the-leaf-and-names-as-bounded-lookups.md). +- [Pending dispatch](../../packages/stack-encrypt/src/target/pending.rs) + currently takes one selected keyset for generates and groups retrieves by + their stored keyset IDs. It constructs generate/retrieve payloads without + lock context and passes no unverified request context. +- [Key-source traits](../../packages/stack-kms/src/key_source.rs) and + [payloads](../../packages/stack-kms/src/payload.rs) already model ZeroKMS + lock-context lists, optional decryption policies and unverified context. + They are ZeroKMS-specific abstractions used for provider injection/testing, + not yet a neutral contract for arbitrary KMS backends. Returned key types + also reflect the existing ZeroKMS/recipher integration. +- [TypeScript identity API](../../languages/typescript/packages/stack/src/identity/index.ts) + accepts `.withLockContext({ identityClaim: ["sub"] })`. Its documented + behavior is server resolution of claim values from the authenticating + principal. Encrypt/decrypt operations pass that selector to protect-ffi; + audit metadata travels separately as `unverifiedContext`. +- The pinned `zerokms-protocol` 0.12.31 `Context` enum includes `Tag`, + `Value` and `IdentityClaim`. Its `RetrievedKey` currently exposes key + material only; `stack-kms::DataKey` contains IV and key, not the original + authentication tag returned by the proposed refinement flow. Supporting + that flow remains a protocol/client response integration requirement, + consistent with the user's earlier explanation of intended behavior. + +## 1. Plaintext-derived keyset selection + +Add a key-selection declaration to the existing target definition, parallel +to context/scope source selectors. A useful minimal portable form is an +explicit ID or name selector over a plaintext field: + +```rust +// Proposed attribute fragment. tenant_id contains the actual keyset name. +#[stash(keyset(name(field = "tenant_id")))] +``` + +```rust +// Alternative: the selected field contains a keyset UUID, not an arbitrary +// application UUID assumed to be a keyset. +#[stash(keyset(id(field = "keyset_id")))] +``` + +Neither selector creates a keyset. Missing, unknown, unauthorized, disabled +or incorrectly typed keysets are errors; never fall back to the default. +A business tenant ID need not be a keyset name/UUID. Where they differ, +configure an explicit tenant-to-key mapping/resolver. Start with direct +selectors and a shared registered resolver contract; do not encode arbitrary +Rust callbacks in dynamic definitions. Pure mappings can return a selector; +network lookup belongs to the execution/provider layer, not the derive. + +The existing manually selected form already fits the agreed API: + +```rust +let tenant = cipher.keyset(IdentifiedBy::Name(user.tenant_id.clone().into())).await?; +let encrypted: EncryptedUser = user.encrypt_into(&tenant).await?; +``` + +The proposed derived route removes that repetition: + +```rust +// EncryptedUser's definition declares how to select the keyset. +let encrypted: EncryptedUser = user.encrypt_into(&cipher).await?; +let encrypted: Vec = users.encrypt_into(&cipher).await?; +``` + +`&cipher` is the client/provider executor which can resolve keysets; it is +not an unscoped request that silently encrypts under the default. Extend the +conversion executor capability to accept either a resolved keyset or a +client plus a declared selection recipe. The target's representation and +encryption operation description remain unchanged. Direct +`tenant_keyset.encrypt(value)` retains its accepted meaning. + +Execution order: + +1. Acquire guarded source ownership at request construction as already + required. Validate target/options, input shape, selectors, role/capability + requirements and every key-independent value check before provider I/O. +2. Evaluate the key-selection recipe from the guarded plaintext. Resolve + selectors through the configured provider; pin each resolved identity + for the duration of the request, including index derivation. +3. Partition a batch by resolved key identity, provider/authentication + session and compatible request-level options. Preserve original result + ordering; many tenants do not imply one remote call for the whole batch. +4. Use the selected keyset's index material for its searchable fields and + mint data keys in the selected keyset. Persist the resolved identity, + not a mutable name, in the ciphertext's provider metadata. + +Define conflict behavior: a preselected keyset cannot silently be overridden +by a target's route. If both are provided, validate that the resolved route +matches the bound keyset, or reject the combination explicitly. No implicit +precedence or fallback. Errors requiring name resolution can only occur +after that resolution; the accepted preflight rule covers key-independent +checks, not provider facts unavailable locally. + +For queries there is no plaintext user to inspect. The same target policy +must expose a caller-supplied tenant routing input or use an explicitly +selected keyset. A typed input may deliberately supply both key routing and +search scope; declaration determines both uses. Do not derive a keyset from +the email search value. Searching across multiple keysets is explicit +fan-out with a term per keyset, not one universally comparable query term. + +Decrypt uses the stable key identity stored with the ciphertext, as today. +Do not re-resolve a historical keyset name and substitute its current target. +An expected tenant/keyset restriction can constrain retrieval, with any +historical key mapping supplied explicitly. Merely trusting a tenant field +returned alongside a ciphertext does not authenticate application tenancy; +row binding or trusted expected context supplies that separate check. +Likewise, choosing a keyset from caller-controlled plaintext is routing, +not permission to use every tenant keyset accessible to a service account. + +## 2. Vitamin C AsyncCipher + +**Packaging follow-up:** the user favors removal. The +[packaging recommendation](2026-10-25-stack-encrypt-packaging.md#retire-asynccipher) +now recommends retiring this trait and its application helpers as execution +moves into Stack, rather than adding the optional adapter considered below. +Vitamin C still has helper/test consumers to migrate; no removal has been +implemented. + +The absence of `AsyncCipher` is intentional in the current design, not an +unexplained omission. [cipher.rs](../../packages/stack-encrypt/src/cipher.rs) +documents the reason, and implements the synchronous Vitamin C `Cipher` +through `MintedKeys` (seal) and `BoundKeys` (open). + +Vitamin C 0.8.0 `AsyncCipher` takes leaf plaintext and derived AAD, with +`seal_many`/`open_many` operating on the leaves of one document. Stack's +executor retains the original provider descriptor and combines requests +across fields/documents/records before calling the leaf cipher. Leaf AAD +alone is not a reversible representation of those execution inputs. +Future routed keys and lock policies add more execution metadata rather +than removing the mismatch. The trait also requires `Sync` and `Send` +futures; Stack's provider traits use `MaybeSend` for native/WASM support. + +Recommendation: retain Stack's batch/provider orchestration and Vitamin C's +leaf cryptography. Do not implement `AsyncCipher` on `StackCipher` merely to +claim trait symmetry, encode provider policy secretly into AAD, or lose +cross-record batching. An explicitly configured adapter could implement the +trait for a restricted use case with provider descriptor/key route/policy +already supplied, but it is not required for the public redesign and must +advertise its limitations. A general refactor would need a richer request +contract and evidence that batching/metadata/WASM behavior remain intact. + +There is a smaller cleanup opportunity: each synchronous adapter currently +implements an unused half of `Cipher` by returning an error. Consider +separate seal/open capabilities or an internal adapter improvement during +the final dependency/packaging pass. Do not conflate that with requiring the +public Stack client to implement the async trait. + +## 3. Lock context as an optional provider policy + +Use a dedicated, typed ZeroKMS policy input, not the ordinary +`encrypt_with_context` argument and not a `SearchScope`. Proposed spelling: + +```rust +use stack_encrypt::providers::zerokms::{LockContext, LockContextExt}; + +let lock = LockContext::identity_claim("sub"); +let encrypted = user.encrypt_into(&tenant_keyset) + .with_lock_context(&lock) + .await?; +let user = User::decrypt_from(encrypted, &tenant_keyset) + .with_lock_context(&lock) + .await?; +``` + +This is illustrative module/method syntax pending the naming pass. The +operation builder retains owned plaintext guards while policy is attached. +The extension is present for a provider with the required capability; a +runtime-erased provider checks the equivalent capability before key I/O. +Ordinary encryption without this optional policy works on providers that +do not support it. A request/definition requiring it must fail on those +providers, never discard it or substitute literal AAD. + +The input selects a claim **name**. ZeroKMS verifies the authenticating +principal and resolves its claim value; do not decode an unverified JWT in +the application and turn its `sub` into a caller assertion. Authentication +belongs to the configured provider/session. Use explicit, immutable +request/session binding when identities differ concurrently; avoid mutating +a global client's identity between operations. + +Support explicit literal tag/value items through the same provider policy, +while distinguishing those caller assertions from principal-derived claims. +Missing claims or provider policy failures are errors with no retry without +the lock. Forward policy on generate and retrieve. A required policy can +live in the target definition so callers cannot accidentally omit it; an +operation may supply an undeclared optional policy, but cannot replace or +weaken a definition-required one. Define composition/conflict rules rather +than last-call-wins behavior. Policy selectors may need to be available on +reads; define their storage or required caller input without serializing +JWTs, credentials or claim values resolved by the provider. + +Keep audit metadata (`UnverifiedContext`) separate: it is not trusted +identity, ciphertext context or a policy assertion. Preserve distinctions +between lock context and the protocol's separate decryption-policy feature; +do not silently reinterpret one as the other. + +Index/query derivation may use an already-loaded deterministic index key +without generating/retrieving a data key. Therefore lock context on those +data-key endpoints does not by itself restrict who can derive search terms +or view query results. If search authorization is required, define that +capability separately; do not infer that the TypeScript query method's +acceptance of `withLockContext` proves the new index path is protected. +Do not implicitly make search scope principal-specific to emulate a lock. + +Batching must preserve per-item lock inputs and separate incompatible +principal/provider sessions. If a provider supports per-payload policy, +retain it in each payload; if an option is request-wide, partition by it. +Any future data-key cache must respect policy/principal checking semantics; +a cache hit must not silently replace a required provider authorization or +audit event. This is a design requirement, not a claim of an existing cache +vulnerability. Do not broaden the current keyset/index-key cache into an +unqualified data-key authorization cache. + +## Provider-neutral base and progressive capabilities + +The public value/target model needs extensions, not replacement. The larger +refactor is the provider boundary, which currently imports ZeroKMS payloads, +UUID keyset IDs, policies, errors and returned key types directly. + +Recommended capability split (working names): + +| Capability | Required for | +| --- | --- | +| Data-key generation/recovery and protected key material | Ordinary opaque and structural encryption/decryption | +| Index-key support | Searchable targets and query derivation | +| Keyset resolution by ID/name | ZeroKMS-style keyset selectors; not every provider | +| Lock-context support, including claim selectors | ZeroKMS policy extension; not every provider | + +Define a resolved provider key reference and versioned opaque recovery +metadata. Do not force every provider to supply a ZeroKMS UUID, IV/tag or +recipher response. A caller selects/configures a provider; ciphertext +metadata identifies a registered provider/key format, never arbitrary +network endpoints or credentials to execute. Keep ZeroKMS's key tag simple +to extract within its documented envelope profile for Rust/SQL consumers. +Other providers' metadata need not invent such a tag. Stable blob payload +encoding is independent of these provider-specific envelope profiles. + +Future AWS KMS/Vault integrations implement the minimal capabilities they +offer. They need not implement lock context, named keysets or searchable +index-key management merely to encrypt an opaque value. The application +can configure its own tenant-to-provider-key mapping without representing +it as a native ZeroKMS keyset. A searchable target requires an explicitly +supported index-key strategy; the engine must not assume every data-key +backend supplies deterministic search keys automatically. + +Refined-tag support also needs distinct stored/request tag and returned +original authentication-tag input. Preserve the user's intended ZeroKMS +flow when designing neutral responses; do not require that every provider +implements refinable tags or that a generic KMS returns that concept. + +## Dynamic interoperability + +Provider capabilities join target capabilities in discovery and definition +validation. Example provider-specific extension (proposed logical schema): + +```json +{ + "provider": "zerokms", + "key_selection": { + "keyset": { "by": "name", "encrypt": { "field": "tenant_id" } } + }, + "provider_policy": { + "zerokms": { + "lock_context": [{ "identity_claim": "sub" }] + } + } +} +``` + +The field's declared type must admit the chosen selector; missing fields +are definition/input errors before provider resolution. Query routing +declares an equivalent caller input because it has no source record. +Providers and extensions are registered, versioned capabilities. Unsupported +required key selection/policy is an error during loading or binding to an +execution provider, before key I/O. Do not accept arbitrary extension maps +and ignore unfamiliar entries. Auth credentials stay in provider setup, not +in this declaration or persisted policy metadata. + +Go generation uses named selectors/policies and the same validated dynamic +definition; provider-specific methods/capabilities are explicit. The +interoperability specification must include mixed-keyset batches, policy +inputs, capability refusals, key identity persistence and the distinctions +between authenticated claims and caller-supplied audit metadata. + +## Implementation and acceptance work + +1. Specify neutral key references/recovery metadata and optional capability + traits without altering the accepted encrypt/decrypt conversion roles. +2. Add definition selectors and client-scoped routed execution. Cover ID + versus name, explicit mappings, conflict errors and no default fallback. +3. Extend request planning/grouping to carry resolved route and provider + policy. Preserve per-record ordering and guard lifetimes across async + resolution/cancellation. +4. Wire ZeroKMS lock context through both generate/retrieve, including + literal and principal-claim forms, and test required-policy enforcement. +5. Verify mixed-tenant batches, matching write/query keys, historical ID + decrypt, wrong expected keysets, missing claims, unsupported-provider + failures before I/O, and concurrent principal isolation. A minimal fake + provider without keysets/locks/index keys must still encrypt opaque values. +6. Resolve refined-tag response support with the ZeroKMS client/protocol. + Add integration fixtures without assuming the pinned API already returns + the original tag. +7. Defer the `AsyncCipher` adapter/trait cleanup decision to the packaging + pass; preserve the documented batching and WASM constraints meanwhile. + +No implementation changes, protocol guarantees for future providers, or +completed security audit are claimed by this assessment.