-
Notifications
You must be signed in to change notification settings - Fork 0
feat: effect 4 rewrite #7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,206 +1,114 @@ | ||
| # JSON Evolutions | ||
|
|
||
| [](https://github.com/toolsplus/json-evolutions/actions/workflows/ci.yml) | ||
| [](https://www.npmjs.com/package/@toolsplus/json-evolutions) | ||
| `@toolsplus/json-evolutions` evolves stored JSON objects through explicitly versioned changesets while application code works with the latest Effect Schema representation. | ||
|
|
||
| JSON evolutions is a small library that aims to help with two things: | ||
| Version 2 is Effect 4-native and ESM-only. It deliberately replaces the v1 io-ts/fp-ts interface while preserving the stored `_version` protocol. | ||
|
|
||
| * keep track of changes to JSON data over time | ||
| * evolve JSON data in an older format to the latest version | ||
| ## Install | ||
|
|
||
| ## Motivation | ||
|
|
||
| Atlassian apps can use [entity properties](https://developer.atlassian.com/cloud/jira/platform/jira-entity-properties/) to store JSON data against various Atlassian product entities. This JSON data is stored in the Atlassian product and can be queried and updated via REST API. From the point of view of an app it is similar to a schemaless, distributed JSON storage. | ||
|
|
||
| Unfortunately, there is no easy way for apps to update data stored in entity property storage if the data schema evolves. For example if the app adds a new property to the data, existing data cannot easily be migrated. There is no easy way to backfill a default value into existing data like you can for example with database evolution tools like Flyway, Liquibase, or Play evolutions. | ||
|
|
||
| To help with this, JSON evolutions introduces a versioning mechanism, and a changelog that describes how to migrate JSON data from one version to the next. It is assumed that the JSON evolution consumer app always works with the latest version of data. Data read from an external store must always include a version number. JSON evolutions will then read that version number and apply all outstanding changesets in a changelog in sequence to migrate the data to the latest format. If the app writes to the external storage, JSON evolutions will inject the latest version number into the data. | ||
|
|
||
| Using this technique allows the data consumer to introduce schema changes without immediately updating already stored data records. Existing records will be migrated on the fly. | ||
|
|
||
| ## Usage | ||
| ```shell | ||
| $ npm add @toolsplus/json-evolutions | ||
| npm install @toolsplus/json-evolutions effect@4.0.0-beta.107 | ||
| ``` | ||
|
|
||
| ### Example | ||
|
|
||
| Let's assume our app stores a configuration object and wants to evolve old values on read while always writing the latest version. | ||
|
|
||
| #### Version 0 | ||
| The Effect peer is pinned exactly while Effect 4 Schema remains beta-sensitive. Node.js 24 or newer is required. | ||
|
|
||
| Let's start with the initial version of the stored data. At version `0`, the changelog is empty because there are no migrations to apply yet. | ||
| ## Complete example | ||
|
|
||
| We also define an [io-ts](https://github.com/gcanti/io-ts) codec using the `versioned` combinator. The `versioned` combinator injects the latest `_version` when encoding and expects your strict `io-ts` codec to strip `_version` again when decoding. Using `io-ts` is optional, but it is a convenient way to keep the version marker as a storage concern instead of leaking it into the rest of the app. | ||
| This example is compiled and executed against the packed npm artifact during the package smoke test. | ||
|
|
||
| ```typescript | ||
| import * as E from "fp-ts/Either"; | ||
| import * as t from "io-ts"; | ||
| ```typescript package-smoke | ||
| import {Effect, Result, Schema} from "effect"; | ||
| import { | ||
| createChangelog, | ||
| latestVersion, | ||
| VersionedJsonObject, | ||
| evolve, | ||
| evolveAndDecode, | ||
| immutabilityHelperChangeset, | ||
| jsonPatchChangeset, | ||
| versioned, | ||
| } from "@toolsplus/json-evolutions"; | ||
|
|
||
| export const changelog = E.getOrElseW((error) => { | ||
| throw new Error(error.message); | ||
| })(createChangelog()); | ||
|
|
||
| export interface Configuration { | ||
| defaultFields: string[]; | ||
| } | ||
|
|
||
| export const codec: t.Type<Configuration, VersionedJsonObject> = versioned( | ||
| t.strict({ | ||
| defaultFields: t.array(t.string), | ||
| }), | ||
| latestVersion(changelog), | ||
| ); | ||
| ``` | ||
|
|
||
| Configuration records can now be written using: | ||
|
|
||
| ```typescript | ||
| codec.encode({defaultFields: ["field1", "field2"]}); | ||
|
|
||
| // {_version: 0, defaultFields: ["field1", "field2"]} | ||
| ``` | ||
|
|
||
| Because the codec uses `versioned`, the latest `_version` is injected automatically when encoding. | ||
|
|
||
| To read a stored configuration value, first pass it through `evolve`. With an empty changelog there is nothing to migrate, so the value is returned unchanged. After that, decode it with the `io-ts` codec to validate the structure and drop `_version`. | ||
|
|
||
| ```typescript | ||
| import * as E from "fp-ts/Either"; | ||
| import {pipe} from "fp-ts/function"; | ||
| import {evolve} from "@toolsplus/json-evolutions"; | ||
|
|
||
| pipe( | ||
| {_version: 0, defaultFields: ["field1", "field2"]}, | ||
| evolve(changelog), | ||
| E.chain(codec.decode), | ||
| ); | ||
|
|
||
| // Right({defaultFields: ["field1", "field2"]}) | ||
| ``` | ||
|
|
||
| The important detail here is that `evolve` returns an `Either`, and `codec.decode` also returns an `Either`, so `E.chain(codec.decode)` keeps the two validation steps in the same error pipeline. | ||
|
|
||
| #### Version 1 | ||
| const addEnabled = jsonPatchChangeset({ | ||
| _version: 1, | ||
| patch: [{op: "add", path: "/enabled", value: true}], | ||
| }); | ||
| const addLabel = immutabilityHelperChangeset({ | ||
| _version: 2, | ||
| spec: {$merge: {label: "current"}}, | ||
| }); | ||
| const changelog = Result.getOrThrow(createChangelog(addLabel, addEnabled)); | ||
|
|
||
| Now let's evolve the schema by adding a new `isEnabled` field. To do that, define a validated changelog containing a version `1` changeset. Changelog versions must be sequential and start at `1`, and `createChangelog` checks that rule up front. | ||
| const Configuration = Schema.Struct({ | ||
| enabled: Schema.Boolean, | ||
| label: Schema.String, | ||
| }); | ||
| const StoredConfiguration = Configuration.pipe(versioned(changelog)); | ||
|
|
||
| ```typescript | ||
| import * as E from "fp-ts/Either"; | ||
| import * as t from "io-ts"; | ||
| import { | ||
| createChangelog, | ||
| jsonPatchChangeset, | ||
| latestVersion, | ||
| VersionedJsonObject, | ||
| versioned, | ||
| } from "@toolsplus/json-evolutions"; | ||
| const program = Effect.gen(function* () { | ||
| const stored = yield* Schema.encodeUnknownEffect(StoredConfiguration)({ | ||
| enabled: false, | ||
| label: "saved", | ||
| }); | ||
| const configuration = yield* evolveAndDecode(StoredConfiguration)({ | ||
| _version: 0, | ||
| }); | ||
| const evolved = yield* evolve(changelog)({ | ||
| _version: 1, | ||
| enabled: false, | ||
| }); | ||
| const initialized = yield* evolve(changelog, { | ||
| initializeFromUnversioned: (input) => | ||
| Effect.succeed({...input, _version: 0 as const}), | ||
| })({}); | ||
| return {stored, configuration, evolved, initialized}; | ||
| }); | ||
|
|
||
| export const changelog = E.getOrElseW((error) => { | ||
| throw new Error(error.message); | ||
| })( | ||
| createChangelog( | ||
| jsonPatchChangeset({ | ||
| _version: 1, | ||
| patch: [ | ||
| { | ||
| op: "add", | ||
| path: "/isEnabled", | ||
| value: true, | ||
| }, | ||
| ], | ||
| }), | ||
| ), | ||
| const fallback = {_version: 2, enabled: false, label: "fallback"} as const; | ||
| const recovered = evolve(changelog)({_version: "invalid"}).pipe( | ||
| Effect.catchTags({ | ||
| InvalidStoredValue: (error) => | ||
| Effect.logWarning(error.message).pipe(Effect.as(fallback)), | ||
| UnsupportedFutureVersion: (error) => | ||
| Effect.fail( | ||
| new Error(`Cannot read stored version ${error.version}`), | ||
| ), | ||
| }), | ||
| ); | ||
|
|
||
| export interface Configuration { | ||
| defaultFields: string[]; | ||
| isEnabled: boolean; | ||
| const result = await Effect.runPromise(program); | ||
| if (result.stored._version !== 2 || !result.configuration.enabled) { | ||
| throw new Error("JSON evolution example failed"); | ||
| } | ||
|
|
||
| export const codec: t.Type<Configuration, VersionedJsonObject> = versioned( | ||
| t.strict({ | ||
| defaultFields: t.array(t.string), | ||
| isEnabled: t.boolean, | ||
| }), | ||
| latestVersion(changelog), | ||
| ); | ||
| void recovered; | ||
| ``` | ||
|
|
||
| Writing values still happens through `versioned`, which now injects version `1`: | ||
| `StoredConfiguration.Type` is the application value `{readonly enabled: boolean; readonly label: string}`. Its encoded representation adds `_version: number`. Runtime decoding accepts only `_version: 2`, the exact latest version derived from the retained changelog. Field transformations and their service requirements are preserved. | ||
|
|
||
| ```typescript | ||
| codec.encode({defaultFields: ["field1", "field2"], isEnabled: false}); | ||
| `evolveAndDecode` evolves historical input, validates the exact current marker, decodes the business representation, and removes `_version`. Use `evolve` when business decoding is not wanted. An initializer runs only for a strict JSON root object with no own version marker; its service requirements propagate, typed failures are wrapped, and defects remain defects. | ||
|
|
||
| // {_version: 1, defaultFields: ["field1", "field2"], isEnabled: false} | ||
| ``` | ||
| ## Changeset adapters | ||
|
|
||
| Reading a previously stored version `0` value now goes through a strict migration boundary. `evolve` will validate the stored value, determine that version `1` is still pending, apply the configured changeset, and return the migrated shape. The codec decode step then validates the business shape and strips `_version`. | ||
| `jsonPatchChangeset` exposes only the six RFC 6902 operations: `add`, `remove`, `replace`, `move`, `copy`, and `test`. JSON Pointer syntax and document-dependent applicability are delegated to `fast-json-patch`. The root `_version` path and source are rejected by `createChangelog`; nested properties such as `/settings/_version` remain valid. | ||
|
|
||
| ```typescript | ||
| import * as E from "fp-ts/Either"; | ||
| import {pipe} from "fp-ts/function"; | ||
| import {evolve} from "@toolsplus/json-evolutions"; | ||
| Changelogs are trusted source declarations. An `immutabilityHelperChangeset` spec is deliberately opaque and receives only a shallow object-or-function guard during changelog construction. Helper functions, `$apply`, Map/Set commands, and registered custom commands are not claimed to be serializable. Every delegate result is still strictly validated before evolution continues. | ||
|
|
||
| pipe( | ||
| {_version: 0, defaultFields: ["field1", "field2"]}, | ||
| evolve(changelog), | ||
| E.chain(codec.decode), | ||
| ); | ||
| ## Tagged errors | ||
|
|
||
| // Right({defaultFields: ["field1", "field2"], isEnabled: true}) | ||
| ``` | ||
|
|
||
| ### Initializing older unversioned values | ||
|
|
||
| If your storage contains historic values from before `_version` existed, you can opt in to `initializeFromUnversioned`. The hook is only called when `_version` is missing and must return a valid version `0` stored value. | ||
|
|
||
| ```typescript | ||
| import * as E from "fp-ts/Either"; | ||
| import {evolve, InitializeFromUnversioned} from "@toolsplus/json-evolutions"; | ||
|
|
||
| const initializeFromUnversioned: InitializeFromUnversioned = (input) => { | ||
| if ( | ||
| typeof input !== "object" || | ||
| input === null || | ||
| Array.isArray(input) || | ||
| !("defaultFields" in input) | ||
| ) { | ||
| return E.left({ | ||
| errorCode: "INVALID_STORED_VALUE_ERROR", | ||
| message: "Cannot initialize value.", | ||
| }); | ||
| } | ||
|
|
||
| return E.right({ | ||
| ...(input as Record<string, unknown>), | ||
| _version: 0, | ||
| }); | ||
| }; | ||
|
|
||
| evolve(changelog, {initializeFromUnversioned})({ | ||
| defaultFields: ["field1", "field2"], | ||
| }); | ||
| ``` | ||
| Evolution uses these schema-backed, yieldable error classes: | ||
|
|
||
| In other words, `initializeFromUnversioned` is a one-time bridge from pre-versioned data into the normal versioned migration flow. Once the hook has returned a valid version `0` value, the regular changelog semantics apply. | ||
| - `InvalidStoredValue` | ||
| - `UnsupportedFutureVersion` | ||
| - `InitializeFromUnversionedFailed` | ||
| - `InitializeFromUnversionedReturnedInvalidValue` | ||
| - `InvalidChangelog` | ||
| - `JsonPatchEvolutionError` | ||
| - `ImmutabilityHelperEvolutionError` | ||
|
|
||
| ### Rules | ||
| Match them through `_tag`, as the complete example does with `Effect.catchTags`. Business-schema failures from `evolveAndDecode` remain `Schema.SchemaError`. Foreign and schema causes are retained through `Schema.Defect()`. | ||
|
|
||
| To make sure the concepts implemented in this library work as intended, follow these rules when you code your evolutions: | ||
| ## Stored-value guarantees | ||
|
|
||
| * Existing changesets **must** never be changed after they have been shipped to production. | ||
| * New changesets **must** always have a sequentially increasing version number. | ||
| * Call `createChangelog` or `validateChangelog` once at startup and reuse the validated result. | ||
| A stored value must be a genuine JSON root object with an own non-negative safe-integer `_version`. Nested values may contain only objects, arrays, strings, finite numbers, booleans, and `null`. | ||
|
|
||
| ### Assumptions | ||
| Functions, `undefined`, symbols, bigint, `Date`, `Map`, `Set`, non-finite numbers, cycles, and root arrays are rejected. Changesets never mutate the original input. The engine owns `_version`, stamps it after each successful changeset, and validates the newly stamped value before continuing. Business structs may not declare the reserved root marker. | ||
|
|
||
| * Stored values must be JSON objects. | ||
| * `initializeFromUnversioned` is disabled by default and should only be used for known pre-versioning records. | ||
| * The library validates stored values and changelogs eagerly and returns an error instead of silently accepting unsupported shapes. | ||
| See [the v1-to-v2 migration guide](docs/migration-v2.md) for source migration details. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.