Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing to CipherStash Stack

Thank you for your interest in contributing to the CipherStash Stack for TypeScript! This document walks you through the repository's structure, how to build and run the project locally, and how to make contributions effectively.
Thank you for your interest in contributing to CipherStash Stack. This document walks you through the repository's structure, how to build and run the project locally, and how to make contributions effectively.

## I want to report a bug, or make a feature request

Expand All @@ -20,6 +20,7 @@ This is a [Turborepo](https://turbo.build/) monorepo managed with [pnpm](https:/
│ │ ├── cli/ <-- The `stash` CLI
│ │ └── ... <-- stack-drizzle, stack-supabase, stack-prisma, nextjs, migrate, wizard, ...
│ └── examples/ <-- Runnable example apps
├── languages/golang/ <-- Go SDK, stashgen and credential packages
├── packages/eql/ <-- EQL (@cipherstash/eql and its Rust crates)
├── e2e/ <-- Cross-package end-to-end tests
├── skills/ <-- Agent skills
Expand All @@ -29,6 +30,8 @@ This is a [Turborepo](https://turbo.build/) monorepo managed with [pnpm](https:/

See [AGENTS.md](./AGENTS.md) for a detailed layout, key APIs, environment variables, and gotchas — it's written for coding agents but is the most complete developer reference in the repo.

For the Go module, start with [languages/golang/README.md](languages/golang/README.md).

### `languages/typescript/packages/stack`

**@cipherstash/stack** is the main package published to npm. It contains the encryption client and all integrations (Drizzle, Supabase, DynamoDB, secrets, identity). This is likely where you'll spend most of your time.
Expand All @@ -44,6 +47,7 @@ The `languages/typescript/examples/` directory contains applications demonstrati
- **Node.js** >= 22
- **pnpm** (the version pinned in `package.json`'s `packageManager` field)
- CipherStash credentials if you want to run integration tests or examples (see [AGENTS.md](./AGENTS.md) for the required environment variables)
- For Go work: mise and the Go and Rust toolchains pinned in `mise.toml`

### 1. Clone the Repo

Expand Down Expand Up @@ -76,6 +80,27 @@ pnpm run dev

Then navigate to one of the examples in `languages/typescript/examples/` and follow its README.

### Go module

The Go module is in `languages/golang/`, outside the pnpm workspace. Its
`encrypt` and `auth` packages embed WASI guests built from Rust. In a source
checkout, build those guests before running the Go suite:

```bash
mise run wasm:guest:build
mise run wasm:guest:build:eql
mise run wasm:guest:build:deterministic
mise run wasm:guest:build:eql:deterministic
mise run wasm:auth-guest:build
mise run go:test
```

For EQL v4 PostgreSQL coverage, set `EQL_TEST_DATABASE_URL` and run
`mise run go:test:postgres`. Go examples that connect to ZeroKMS need
CipherStash credentials. The [Go module guide](languages/golang/README.md)
links to the SDK, generator, credentials, examples and release notes. Commit
generated `*_stash.go` files when changing Go model tags; CI checks for drift.

## Making Changes

1. **Create a new branch** from `main`:
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ npx stash init

> The Stack also ships a `stash` CLI for auth, schema, and database setup. See the [SDK reference][reference].

Building in Go? Start with the [Go SDK guide](languages/golang/README.md) for
typed field encryption, generated methods and EQL v4 PostgreSQL queries.

## What's in the Stack

### 🔐 Searchable encryption
Expand Down
14 changes: 8 additions & 6 deletions languages/golang/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
`eql.JSONSearchLeafQuery` and its entry `eql.JSONEntry`,
generated from the EQL catalog. The names are the ones EQL v3 gives its
searchable JSON document; `eql.JSON` stays the storage-only type. The
engine stores one from the module's own plans; `stashgen` does not
generate a field of it yet.
engine stores one from the module's own plans, and `stashgen` generates
fields tagged `encrypt_into=JsonSearch`.

- **Decimals, dates and timestamps take an order index (#1236).** An
`encrypt.Decimal`, `encrypt.Date` or `time.Time` field takes `ore` and
Expand Down Expand Up @@ -252,7 +252,8 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
implements `driver.Valuer`, `sql.Scanner` and JSON like the other query
types. They are not in
`eql.Types`, which lists stored types, and no generated method returns one
yet: the ordered EQL types are not producible.
yet: `stashgen` does not construct a range query operand, although the
ordered EQL types are producible and expose single-bound comparisons.
- **`encrypt.MatchQuery` and `encrypt.EQLMatchQuery`**: what a `Matches`
method returns. Each carries the term or EQL query value to bind,
`Verify` (check decrypted rows, since a match can return false positives)
Expand Down Expand Up @@ -290,12 +291,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
these types: it is not a tag word.
- **`TextSearch`**: a `string` field tagged `encrypt_into=TextSearch` gets
`Eq`, `Lt`, `Lte`, `Gt`, `Gte` and `Matches`, one column for all of them.
The stored value keeps all three terms. Text shorter than three
characters, an empty string included, is stored with an empty match
The stored value keeps all three terms. Text that cuts no match token,
such as an empty string or one letter, is stored with an empty match
filter, where `TextMatch` refuses it with `ErrTerm`: `Eq`, `Lt`, `Lte`,
`Gt` and `Gte` find the row, and no `Matches` query does. Each query
carries only the term its operator reads, so `Eq` and `Lt` take a
two-letter value and `Matches` refuses one (`ErrTerm`). Needs the EQL v4
one-letter value and `Matches` refuses one (`ErrTerm`). A two-letter
word has its own match token, as described above. Needs the EQL v4
SQL from this release, whose `eql_v4.query_text_search` accepts one term.
- **Ordering methods on text.** A string field with `index=ore` or
`index=ope` gets `Lt`, `Lte`, `Gt`, `Gte` and `Between`. Text orders by its
Expand Down
52 changes: 52 additions & 0 deletions languages/golang/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# CipherStash Stack for Go

This module provides searchable field encryption for Go structs. `stashgen`
turns `stash` tags into typed encryption, decryption and query methods. The
`encrypt` package runs the Rust encryption engine in an embedded WASI guest
with wazero; applications do not need cgo.

Start with the [SDK guide](encrypt/README.md) for installation, credentials,
generation and PostgreSQL storage. The [stashgen reference](cmd/stashgen/README.md)
explains tags, supported field types and generated methods. The
[typed JSON document example](encrypt/example/json/README.md) walks through
an EQL v4 table, queries and partial reads. The [credential guide](auth/README.md)
covers profile, access key and federation strategies. See the
[Go changelog](CHANGELOG.md) for release and compatibility notes.

## Using a released module

Use Go 1.26.10 or later (or 1.27.3 or later on the 1.27 line). Add the
generator to your application module, declare a struct with `stash`
tags, then generate and commit its `*_stash.go` file:

```sh
go get -tool github.com/cipherstash/stack/languages/golang/cmd/stashgen
go generate ./...
```

Use a tagged Go module version. Release tags include the embedded WASI guests;
a checkout or an untagged commit does not. The [SDK guide](encrypt/README.md#building)
explains how to build them locally. Searchable `encrypt_into` fields require
EQL v4 in PostgreSQL; the EQL v4 SQL bundle is attached to the matching Go
release. Go and TypeScript currently use different EQL versions, so follow
the [EQL column guide](encrypt/README.md#eql-columns) when choosing a schema.

## Working in this repository

From the repository root, install the toolchain pinned by `mise.toml` and
build the guests before running Go tests:

```sh
mise run wasm:guest:build
mise run wasm:guest:build:eql
mise run wasm:guest:build:deterministic
mise run wasm:guest:build:eql:deterministic
mise run wasm:auth-guest:build
mise run go:test
Comment on lines +42 to +45

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Build the production guests before invoking the Go test task

On a fresh checkout, this sequence does not produce the production encryption guests required by mise run go:test. In mise.toml, the two deterministic tasks copy their outputs only into encrypt/testdata/, whereas scripts/go-binding-test.sh explicitly requires encrypt/wasm/stack_encrypt_guest.wasm and encrypt/eql/wasm/stack_encrypt_guest_eql.wasm and exits before running tests if either is absent. Add wasm:guest:build and wasm:guest:build:eql alongside the deterministic builds, and update the identical sequence in CONTRIBUTING.md as well.

Useful? React with 👍 / 👎.

```

The `.wasm` files are build outputs and are not committed on development
branches. `mise run go:test:postgres` also runs the PostgreSQL suite against
the EQL v4 bundle built from this tree; it needs the database named by
`EQL_TEST_DATABASE_URL`. See `mise.toml` and [AGENTS.md](../../AGENTS.md)
for the build and release details.
8 changes: 6 additions & 2 deletions languages/golang/cmd/stashgen/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ These words are the same as the Rust API's words for the same behaviour, with tw
### Encrypted JSON documents

A field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
For a complete generated model and PostgreSQL query flow, see the [runnable typed-document example](../../encrypt/example/json/README.md).
Declare it as a struct, map or slice type, or as a `json.RawMessage` that holds the document's JSON text:

```go
Expand Down Expand Up @@ -289,12 +290,15 @@ The engine produces these EQL types today, each from one Go field type:
| `Real`, `RealEq`, `RealOrd`, `RealOrdOpe`, `RealOrdOre` | `float32` |
| `Double`, `DoubleEq`, `DoubleOrd`, `DoubleOrdOpe`, `DoubleOrdOre` | `float64` |
| `Date`, `DateEq` | `encrypt.Date` |
| `DateOrd`, `DateOrdOpe`, `DateOrdOre` | `encrypt.Date` |
| `Timestamp`, `TimestampEq` | `time.Time` |
| `TimestampOrd`, `TimestampOrdOpe`, `TimestampOrdOre` | `time.Time` |
| `Numeric`, `NumericEq` | `encrypt.Decimal` |
| `NumericOrd`, `NumericOrdOpe`, `NumericOrdOre` | `encrypt.Decimal` |
| `JsonSearch` | struct, map, slice, or `json.RawMessage`; see [Encrypted JSON documents](#encrypted-json-documents) |

The `Eq` types answer `Eq`; the `Ord`, `OrdOpe` and `OrdOre` types answer `Lt`, `Lte`, `Gt` and `Gte`, and on text `Eq` too; `TextMatch` answers `Matches`, and `TextSearch` and `TextSearchOre` all of `Eq`, `Lt`, `Lte`, `Gt`, `Gte` and `Matches`, each query carrying only the term its operator reads; `Text`, `Smallint`, `Integer`, `Bigint`, `Boolean`, `Real`, `Double`, `Date`, `Timestamp` and `Numeric` are stored and read only, with no query method.
The ordered date, timestamp and numeric types are refused: the engine derives no order term for these kinds yet.
An ordered type has no `Between` method yet, because a `BETWEEN` operand needs EQL v4's range query domains.
The ordered date, timestamp and numeric types answer the four comparisons, as do the other ordered EQL types. The generated EQL fields have no `Between` method yet: EQL v4 has range query domains, but `stashgen` does not construct their operands. A JSON document's declared ordered paths have their own `Between` method, which returns two leaf query values; see [Encrypted JSON documents](#encrypted-json-documents).

Any built-in Go integer field takes an integer type (ADR-0002) and seals at the type's width: an `int16` field always fits an `Integer`, and an `int64` field is range checked value by value, `Encrypt` refusing a value that does not fit with `encrypt.ErrEncoding`. `encrypt.Int128` and `encrypt.Uint128` are not converted. A float's width must match: a `float64` field is not a `Real`.
A `TextMatch` field indexes the 3-grams of its text and each two-letter word whole, so a two-letter surname such as "Ng" is stored and found by the query "ng"; a query's two-letter words must be whole words of the row.
Expand Down
28 changes: 18 additions & 10 deletions languages/golang/encrypt/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,8 +349,9 @@ operand of `eql_v4.between(column, $1::eql_v4.query_integer_range_ore)`. Keep
the cast to the exact range domain: an uncast `$1`, a `jsonb` parameter, or a
range of another term or family fails with SQLSTATE 42725 ("function ... is
not unique"), unlike the single-bound operators, which resolve an untyped
`$1`. No generated method returns one yet, because the ordered EQL types are
not producible.
`$1`. The ordered EQL types are producible, but `stashgen` does not yet
generate a `Between` method or construct these range query values. Their
generated fields expose the four single-bound comparisons.

`Eq`, the orderings and `Between` are exact: the server's answer is the
answer. `Matches` is probabilistic: the server answers with every row whose
Expand Down Expand Up @@ -512,10 +513,14 @@ The EQL types the engine produces, and the Go field types each takes:
| `DoubleOrd`, `DoubleOrdOpe`, `DoubleOrdOre` | `float64` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` |
| `Date` | `encrypt.Date` | none |
| `DateEq` | `encrypt.Date` | `=`, `<>` |
| `DateOrd`, `DateOrdOpe`, `DateOrdOre` | `encrypt.Date` | `<`, `<=`, `>`, `>=`, `ORDER BY` |
| `Timestamp` | `time.Time` | none |
| `TimestampEq` | `time.Time` | `=`, `<>` (to the microsecond, truncated) |
| `TimestampOrd`, `TimestampOrdOpe`, `TimestampOrdOre` | `time.Time` | `<`, `<=`, `>`, `>=`, `ORDER BY` |
| `Numeric` | `encrypt.Decimal` | none |
| `NumericEq` | `encrypt.Decimal` | `=`, `<>` (`1.5 = 1.50`) |
| `NumericOrd`, `NumericOrdOpe`, `NumericOrdOre` | `encrypt.Decimal` | `<`, `<=`, `>`, `>=`, `ORDER BY` |
| `JsonSearch` (`eql.JSONSearch`) | struct, map, slice, or `json.RawMessage` | containment, equality at a path, ordering at declared paths, selection, subtree reads; see [Encrypted JSON documents](#encrypted-json-documents) |

Any built-in Go integer field takes an integer EQL type (ADR-0002): the field
is sealed at the type's width, and opens back into the field's type. A type
Expand Down Expand Up @@ -552,8 +557,8 @@ decimal with its scale and a `time.Time` with its nanoseconds, in UTC. A
`time.Time` is sealed as its instant, whatever its location. A zero
`encrypt.Date`, a date or instant outside years -262143 to 262142 and a
decimal with more than 28 digits after the point are refused with
`ErrEncoding`. The ordered date, timestamp and numeric types are not
produced yet: the engine derives no order term for these kinds.
`ErrEncoding`. Ordered dates compare by day, timestamps by the truncated
microsecond, and decimals by value (`1.5` ties `1.50`).

The `Ord` and `OrdOpe` types store an OPE term, which EQL compares as bytes,
and order by ADR-0002's order key: integers by value, floats as PostgreSQL's
Expand All @@ -570,7 +575,8 @@ answers `Eq` through an equality term only. A text order type also stores
`TextEq`'s equality term, and `=` reads that, so `Café` and `cafe` are
neither `<` nor `>` each other and not `=` either. Text holding a code point
Unicode 16 does not assign has no order term and is refused. There is no
`Between` method yet: a `BETWEEN` needs EQL v4's range query domains.
generated `Between` method on these EQL fields yet, although EQL v4 has
range query domains and `eql_v4.between` for them.

The `OrdOre` types and `TextSearchOre` order by the same key, through a block
ORE term (ore-rs's 6-bit schemes) in place of the OPE term. Each term
Expand All @@ -582,9 +588,9 @@ the installer disables these domains, and the `Ord` types are the ones to
use. Text whose fold is longer than 1023 bytes, or ends in NUL, has no block
ORE term and is refused, stored or queried.

The rest of the catalog (ordered dates, timestamps and decimals, the v3 JSON types) is listed in
`eql.Types` with the reason it is not produced yet, and `stashgen` refuses it
with that reason. The guest builds the value: the generated code
The catalog in `eql.Types` also lists EQL types the engine does not produce,
with the reason for each; `stashgen` refuses those types with that reason.
The guest builds a producible value: the generated code
imports `encrypt/eql`, which embeds the build of the engine that holds the
EQL types and registers it on import, so the program runs that build and
stores what it returns. `Fields.Email.Eq` returns the `eql.TextEqQuery`
Expand All @@ -601,8 +607,10 @@ column.

`JsonSearch` stores a whole JSON document in one `public.eql_v4_json_search`
column, from a struct, map or slice field, or a `json.RawMessage`
(`stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"`; see the
stashgen README). A struct is marshalled with `encoding/json` when it is
(`stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"`). See the
[stashgen reference](../cmd/stashgen/README.md#encrypted-json-documents) and
the [runnable typed-document example](example/json/README.md). A struct is
marshalled with `encoding/json` when it is
sealed and read back into its type when it is opened. Its numbers come back
exactly; the stashgen README lists what `encoding/json` does not keep
(invalid UTF-8, a nil map or slice, a time's zone). Unlike the types above,
Expand Down
48 changes: 48 additions & 0 deletions languages/golang/encrypt/example/json/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Typed encrypted JSON document example

This program stores a Go `Profile` struct as one encrypted EQL v4 JSON column.
It finds a row by containment, compares an ordered path, decrypts a whole
profile, and opens only the visit times from another query.

## Run

You need Go 1.26.10 or later, PostgreSQL with the EQL v4 SQL bundle installed, a `DATABASE_URL`
pointing to it, and CipherStash credentials. `stash auth login` writes a
developer profile; the `CS_*` environment variables also work. The example
creates a temporary table, so it does not leave data behind.

From the repository root:

```sh
mise run wasm:guest:build:eql wasm:auth-guest:build
export DATABASE_URL='postgres://localhost:5432/your_database'
cd languages/golang
go run ./encrypt/example/json
```

For a fresh database, install the `eql-v4-install.sql` asset attached to the
Go module release before running the program. The guest builds are needed in
a repository checkout because its WASM files are not committed. A tagged Go
module version carries them.

## Change the model

The declaration is in `model.go`. `stashgen` writes `customer_stash.go` from
its `stash` tags. After changing the struct or tags, run this from
`languages/golang` and commit the changed generated file:

```sh
go generate ./encrypt/example/json
```

`Profile` uses `encoding/json` for the document shape. The `ordered=$.age`
tag adds an ordering term for that path. A containment needle made from a
struct includes every field without `omitempty`, including zero values;
`main.go` uses a map to name only the member being queried. The prepared
subtree read returns JSON text, which `encrypt.DecodeJSON[int64]` decodes.

The example's `SELECT` statements cast parameters to EQL's operand types.
The age comparison uses the selector derived from the same ordered path as
its query term. A JSON selector or read token is a capability over a column;
do not log it. Partial reads authenticate returned entries against the
prepared request, but do not prove that the database returned every entry.
Loading
Loading