Repository navigation
docs(openspec): propose data node and value operations #211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gohabereg
wants to merge
1
commit into
docs/openspec-fix-ot-client-convergence
Choose a base branch
from
docs/openspec-add-data-node-operations
base: docs/openspec-fix-ot-client-convergence
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| ## Context | ||
|
|
||
| The v3 List tool keeps its items in one flat array inside a single block: | ||
|
|
||
| ```ts | ||
| { style, start?, counterType?, items: [{ content: Text, depth: number, checked?: boolean }] } | ||
| ``` | ||
|
|
||
| Every List interaction becomes one of three model calls on that array: `createDataNode('items.N', …)`, `removeDataNode('items.N')`, or `updateValue('items.N.depth' | 'items.N.checked', …)`. The model already supports them. `BlockNode.createDataNode` splices when the parent is an array, `removeDataNode` splices it out, and `ValueNode.update` emits `ValueModifiedEvent` with `{ value, previous }`. What is missing is everything downstream of the event: attribution, operations, transforms and replay. | ||
|
|
||
| Two undo managers consume model events. `core`'s `UndoRedoManager` records raw event payloads and replays their inverse through `EditorJSModel.insertData` / `removeData` / `modifyData`. `collaboration-manager`'s records `Operation`s and cancels core's with `preventDefault()`. Both replay through the same three model methods, and so does `ot-server`'s `DocumentManager`. Fixing those methods once fixes undo in all three places. | ||
|
|
||
| ## Goals / Non-Goals | ||
|
|
||
| **Goals** | ||
| - Data-node insert/delete and value modify on any block reach peers, are applied by `ot-server`, and can be undone and redone locally. | ||
| - Concurrent structural edits in the same array converge, including text edits inside a shifted element. | ||
|
|
||
| **Non-Goals** | ||
| - Re-keying model listeners or `InputsRegistry` after an array splice (a separate change, see proposal). | ||
| - Grouping several operations into one undo step in collaboration mode. | ||
| - Fixing the client/server protocol (revision stamping, in-flight rebasing, explicit transform order). That is `fix-ot-client-convergence`, which this change depends on. | ||
|
|
||
| ## Decisions | ||
|
|
||
| ### D1. Initialization events are marked, not removed | ||
|
|
||
| `BlockNode#initialize` calls `createDataNode` once per top-level key. The resulting `DataNodeAddedEvent`s are the only signal `BlockToolAdapter` gets to dispatch `KeyAddedEvent`, which tools such as Paragraph use to create their inputs data-first. They must keep flowing. But they duplicate the `BlockAddedEvent` that precedes them. Turning them into operations would make every block insert also send N data-node inserts, and redo would throw `AlreadyExistingKeyError`. | ||
|
|
||
| `DataNodeAddedEvent` therefore gets a boolean marker, `detail.isInitialization`. `BlockNode` sets it for events dispatched from `#initialize`. `CollaborationManager` and core's `UndoRedoManager` skip marked events entirely: no operation, no undo entry and no transform of the undo stacks. `BlockToolAdapter` does not look at the marker. | ||
|
|
||
| The marker is lost today when the event bubbles, because `EditorDocument#listenAndBubbleBlockEvent` and `EditorJSModel#listenAndBubbleDocumentEvents` re-create events as `new ctor(index, data, userId)`. Both re-creation sites forward it. | ||
|
|
||
| *Alternatives*: a separate `DataNodeInitializedEvent` class would make every adapter-side consumer map two classes to `KeyAdded`. Keeping the no-`userId` quirk would leave it impossible to tell an initialization event from a genuine one, and D2 removes the quirk anyway. | ||
|
|
||
| ### D2. Explicit `createDataNode` dispatches synchronously | ||
|
|
||
| The microtask in `createDataNode` exists only to order initialization events after `BlockAddedEvent`. It is kept for `#initialize` only. A call on an already constructed block dispatches synchronously, so `EditorJSModel` reads the `userId` from the live `WithContext` frame, as it does for every other event. | ||
|
|
||
| ### D3. Operation shapes | ||
|
|
||
| | Model event | Operation | `index` | `data` | | ||
| |---|---|---|---| | ||
| | `DataNodeAddedEvent` (not initialization) | `Insert` | `DataIndex` | `{ payload: [subtree] }` | | ||
| | `DataNodeRemovedEvent` | `Delete` | `DataIndex` | `{ payload: [subtree] }` | | ||
| | `ValueModifiedEvent` | `Modify` | `DataIndex` | `{ payload: value, prevPayload: previous }` | | ||
|
|
||
| `subtree` is the event's `BlockNodeDataSerializedValue`. Wrapping it in a one-element array keeps `InsertOrDeleteOperationData.payload` `ArrayLike`, matching block operations (`[BlockNodeSerialized]`). It also means the text transformer's `payload.length` reads never see an arbitrary object. `ModifyOperationData.payload` / `prevPayload` widen from `Record<any, any>` to any JSON value (`depth: 2`, `checked: false`), the same widening #192 makes for plugin data. | ||
|
|
||
| The model's `insertData` / `removeData` take the raw subtree for a `DataIndex`. A single exported helper, `toModelData(operation)`, unwraps `[subtree]` for a `DataIndex` Insert/Delete and returns `payload` unchanged otherwise. `CollaborationManager.applyOperation` and `ot-server`'s `DocumentManager#applyOperationToModel` both call it, so there is one source of truth for the shape. | ||
|
|
||
| `BatchedOperation.canAdd` already returns `false` for non-text indexes, so each data operation is its own undo step. | ||
|
|
||
| ### D4. Model replay semantics for `DataIndex` | ||
|
|
||
| - `modifyData(DataIndex, { value })` calls `updateValue(blockIndex, dataKey, value)`. It creates the node if it is absent, as `updateValue` already does. | ||
| - `insertData(DataIndex, subtree)`: | ||
| - when the parent is an array, it splices, as `createDataNode` does now; | ||
| - when the key does not exist, it creates the node; | ||
| - when the key already exists outside an array, it removes the existing node and then creates the new one. This emits `DataNodeRemoved` and then `DataNodeAdded`. | ||
|
|
||
| The direct `createDataNode` API keeps throwing `AlreadyExistingKeyError`. Only the replay path replaces. | ||
| - `removeData(DataIndex)` is unchanged (`removeDataNode`). | ||
|
|
||
| Replacing on insert is what lets two concurrent creations of the same object key converge (D6). Without it, `ot-server` throws on the second one. | ||
|
|
||
| ### D5. Array positions in the transformer | ||
|
|
||
| The transformer has no model, so it reads array positions from the key: **a path segment that is a non-negative integer addresses an array element.** This matches `keypath.insert` / `remove` and `renumberKeys`, which already treat numeric segments as array indices. Object keys made only of digits are not supported for array semantics. A tool must not use them, and the spec says so. | ||
|
|
||
| For an operation `against` at `P.i` (prefix `P`, last segment the integer `i`) and an operation `op` whose key (`DataIndex.dataKey` or `TextIndex.dataKey`) is `P.j` or `P.j.rest`, in the same block: | ||
|
|
||
| | `against` | `op` relation | Result | | ||
| |---|---|---| | ||
| | Insert at `P.i` | `j > i` | `j → j+1` | | ||
| | Insert at `P.i` | `j == i`, `op` key is deeper (`P.i.rest`) | `j → j+1` (the element moved) | | ||
| | Insert at `P.i` | `j == i`, `op` is an Insert at exactly `P.i` | tie-break (below) | | ||
| | Insert at `P.i` | `j == i`, `op` is a Delete at exactly `P.i` | `j → j+1` | | ||
| | Delete at `P.i` | `j > i` | `j → j−1` | | ||
| | Delete at `P.i` | `j == i`, `op` is an Insert at exactly `P.i` | unchanged | | ||
| | Delete at `P.i` | `j == i`, anything else (same element or inside it) | `Neutral` | | ||
| | any | `j < i`, or `op` key outside `P` | unchanged | | ||
|
|
||
| For an operation `against` that deletes a key `K` whose last segment is **not** an integer (an object member), any `op` at `K` or under `K.` becomes `Neutral`, except an Insert at exactly `K`, which re-creates it and is unchanged. | ||
|
|
||
| Prefixes nest: `P` itself may contain array segments (`rows.2.cells.0`). Only the segment at depth `|P|` is compared and rewritten. A text operation keeps its `textRange` and only its `dataKey` is rewritten (a `TextIndex` with the new key). | ||
|
|
||
| **Tie-break for two Inserts at the same array position.** This uses the explicit order argument from `fix-ot-client-convergence`, `op.transform(against, againstIsLater)`. `op` shifts to `i+1` iff `against` comes earlier (`againstIsLater` is false). Revisions are not consulted: remote operations reach the undo stacks rebuilt from model events, without one. All three paths give the same order: | ||
|
|
||
| - *Server.* Incoming `A` is reduced through history op `B` (default order), so `A` shifts. `B` stays at `i` and `A` goes to `i+1`. | ||
| - *Author of `A`.* `OTClient` transforms incoming `B` against its unacknowledged `A` with `againstIsLater: true`, so `B` stays at `i`. The local `A` moves to `i+1` because the model splices, and the client rebases `A` against `B` with the default order. The result is the same order. | ||
| - *Undo stacks and the open batch.* An entry transformed against a remote insert at the same position (default order) shifts, which is correct: the remote element now sits in front of it. | ||
|
|
||
| ### D6. Same-key conflicts that do not shift | ||
|
|
||
| - **Modify vs. Modify, same block and key.** Last-writer-wins in server order: `op` becomes `Neutral` iff `against` comes later (`againstIsLater`). Otherwise `op` is kept and applied after `against`. This is the rule `fix-ot-client-convergence` sets for values. | ||
| - **Insert vs. Insert on the same object key** (last segment not an integer). The same rule as Modify. With replace-on-insert (D4), the later write in server order wins on the server and on every client, exactly as for values. | ||
| - **Modify vs. Insert / Delete** follows D5. A Modify under a deleted element or member becomes `Neutral`. A Modify inside a shifted element is re-keyed. | ||
|
|
||
| Every new rule is added to the existing `#transformAgainstDataOperation` branch, which `#applyTransformation` already dispatches to for any `DataIndex` `againstOp`. Text operations stop being returned unchanged there. Data operations transformed against *text* operations, and anything transformed against *block* operations, keep their current rules: `DataIndex` already has `withBlockIndex`. | ||
|
|
||
| ### D7. Undo/redo | ||
|
|
||
| `Operation.inverse()` already swaps Insert ↔ Delete with the same payload and swaps `payload` / `prevPayload` for Modify. With D3 and D4 in place, collaboration undo of a data operation works without new code. Core's `UndoRedoManager` replays `event.data` for Added/Removed (the raw subtree, which matches the model) and inverts Modified, so it works once D1 (skip the marker), D2 (attribution) and D4 (`modifyData`) are in place. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Two local inserts at |
||
|
|
||
| ## Risks / Trade-offs | ||
|
|
||
| - **Stale delete payloads.** A Delete carries a snapshot of the removed subtree. If a peer edits inside the element concurrently, the server still deletes the right element, but undoing the delete restores the snapshot without the peer's edit. Block deletes behave the same way today. This is accepted and documented. | ||
| - **Undoing a Modify overwrites a newer remote value.** Undo stack entries are transformed with the default order, so a remote Modify on the same key does not neutralize them. Undo then restores the local `prevPayload` over the peer's value. This is current behavior for data operations and is unchanged. | ||
| - **Dependency on `fix-ot-client-convergence`.** Until it lands, the server never transforms client operations and the client skips its in-flight operation, so concurrent array edits diverge exactly as concurrent text edits do today. The rules here are correct only on top of that fix. | ||
| - **The numeric-segment convention** rules out digit-only object keys for array semantics. It is cheap to state, and no current tool uses such keys. | ||
| - **Wire compatibility.** An older `ot-server` silently ignores `DataIndex` Insert/Delete (no model support) and drops `Modify`. Clients and server deploy together, which is already the rule for this monorepo. | ||
|
|
||
| ## Migration Plan | ||
|
|
||
| No data migration. Ship `model-types`, `model`, `collaboration-manager`, `core` and `ot-server` together. | ||
|
|
||
| ## Open Questions | ||
|
|
||
| None blocking. The array re-keying change must land before the List tool enables structural editing in collaboration mode. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| ## Why | ||
|
|
||
| Changes to a block's data tree other than text are local-only today. A data node created or removed (`DataNodeAddedEvent` / `DataNodeRemovedEvent`) or a value updated (`ValueModifiedEvent`) is never sent to collaborators and cannot be undone: | ||
|
|
||
| - `CollaborationManager#handleEvent` only maps text and block events to `Operation`s. The three data events fall through to `console.error('Unknown event type')`. | ||
| - `EditorDocument.modifyData` handles only `TextIndex` (`@todo implement other actions`). So a `Modify` on a `DataIndex` does nothing, whether it comes from undo, a remote peer or `ot-server`. This is true for the core `UndoRedoManager` as well as the collaboration one. | ||
| - A locally created data node reports no `userId`. `BlockNode.createDataNode` dispatches `DataNodeAddedEvent` in a microtask, `EditorDocument` drops `detail.userId` when it bubbles the event, and `EditorJSModel` re-reads an execution context that has already been popped. Even if the event were mapped, it would be treated as remote and never sent. | ||
| - `OperationsTransformer` handles `DataIndex` with last-writer-wins on the exact same key only. Nothing shifts positions in an array, so concurrent inserts and deletes in the same array (or text edits inside a shifted element, e.g. `items.3.content` after an insert at `items.1`) land on the wrong element. | ||
|
|
||
| The v3 List tool depends on all four. It stores items as one flat array, `items: [{ content, depth, checked? }]`. Tab/Shift+Tab is `updateValue('items.N.depth')`, Enter inserts an array element, a Backspace merge removes one, and a checkbox toggle is `updateValue('items.N.checked')`. Each of these must sync to peers and be undoable. | ||
|
|
||
| ## What Changes | ||
|
|
||
| - **Model events carry their author.** `createDataNode` called on an existing block dispatches `DataNodeAddedEvent` synchronously with the context `userId`. `DataNodeAddedEvent`s dispatched while a `BlockNode` is being constructed are marked as initialization events. They still reach adapters, which is how tools render their inputs data-first, but they never become operations or undo steps, because `BlockAddedEvent` already carries the whole block. | ||
| - **The model applies data operations by index.** `modifyData` applies a `DataIndex` change through `updateValue`. `insertData` on a `DataIndex` whose key already exists (outside an array) replaces the node instead of throwing, so concurrent creations of the same key converge on last-writer-wins. | ||
| - **New operation kinds in `collaboration-manager`.** `DataNodeAdded` becomes an `Insert` and `DataNodeRemoved` a `Delete` on a `DataIndex`, with a one-element `payload: [subtree]`. `ValueModified` becomes a `Modify` with `payload` / `prevPayload`. `ModifyOperationData.payload` widens to any JSON value, the same widening as #192. The client and `ot-server` share one helper that turns an operation into model-shaped data. | ||
| - **Transformer rules for array positions.** Inserts and deletes of array elements shift the positions of other operations in the same array: data operations and text operations whose `dataKey` passes through that array. Operations inside a deleted element or a deleted subtree become `Neutral`. Two inserts at the same position break the tie with the explicit transform order from `fix-ot-client-convergence`, so the server and every client order them the same way. Value vs. value on the same key keeps last-writer-wins under that same order. | ||
| - **Undo/redo of data changes works** in both the core and collaboration undo managers. No new mechanism is needed: the existing inversion works once the model applies the inverted change. | ||
|
|
||
| ## Capabilities | ||
|
|
||
| ### New Capabilities | ||
|
|
||
| None. | ||
|
|
||
| ### Modified Capabilities | ||
|
|
||
| - `collaboration-manager`: data-node and value events become operations; new transformation rules for array positions and subtree deletion; undo/redo of data operations. | ||
| - `model`: `modifyData` / `insertData` semantics for `DataIndex`; synchronous, attributed `DataNodeAddedEvent` outside construction. | ||
| - `model-types`: `DataNodeAddedEvent` gains the initialization marker, and bubbling keeps it. | ||
| - `core`: the local `UndoRedoManager` records data-node and value changes and skips initialization events. | ||
| - `ot-server`: `DocumentManager` applies data-node and value operations to its model. | ||
|
|
||
| ## Out of scope / dependencies | ||
|
|
||
| - **Array element re-keying in the model and the DOM (separate change, hard prerequisite for List).** `BlockNode` node listeners capture their key at creation (`#listenAndBubbleTextNodeEvent` / `#listenAndBubbleValueNodeEvent`), and `InputsRegistry` in `dom-adapters` is not re-keyed. After an insert at `items.1`, the node now at `items.2` keeps reporting `items.1`, so local edits there produce wrongly addressed operations. This change only makes *operations in flight* account for shifts. The tests here apply operations and never edit a node whose position has moved. | ||
| - **Undo grouping in collaboration mode.** Data operations are never batched in `BatchedOperation`, so List Enter (text delete + item insert) is two undo steps when collaboration is enabled. Core's grouping by browser task (#201 / #203) does not cover `CollaborationManager`. #201 already lists this as a follow-up. | ||
| - **Interaction with #201 / #203.** #201's design (R2) relies on construction-time `DataNodeAddedEvent`s having no `userId`. This change replaces that accident with an explicit marker. Whichever lands second updates the R2 test to assert on the marker. | ||
| - **Depends on `fix-ot-client-convergence`.** That change makes the server actually transform client operations, makes the client transform against its in-flight operation and rebase its queue, and adds `transform(against, againstIsLater)`. The tie-break and same-key rules here are written in terms of that order argument, and convergence for array edits holds only once it has landed. | ||
| - **Interaction with #192.** #192 widens `ModifyOperationData` in the same way. Whichever lands second drops its duplicate. | ||
|
|
||
| ## Impact | ||
|
|
||
| - **Code**: `packages/model-types/src/events/DataNodeAddedEvent.ts`; `packages/model/src/entities/BlockNode/index.ts`, `EditorDocument/index.ts`, `EditorJSModel.ts`; `packages/collaboration-manager/src/{CollaborationManager,Operation,OperationsTransformer}.ts`; `packages/core/src/components/UndoRedoManager.ts`; `packages/ot-server/src/DocumentManager.ts`. | ||
| - **Wire format**: additive. `Insert` / `Delete` / `Modify` on `k: "data"` indexes, which already serialize. Clients and `ot-server` must deploy together, because an older server drops these operations. | ||
| - **Docs**: `docs/collaboration.md` (operation kinds and the transform rules table), `docs/events.md` (`DataNodeAddedEvent` marker and synchronous dispatch). The "Step 1: transform each pending local op against the incoming remote op" in `docs/collaboration.md` does not match `OTClient`, which only transforms the incoming op against pending ones. This change does not alter that and notes it in design.md. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For concurrent inserts at
meta({url: 'a'}) andmeta.url('b'), processingmetafirst leaves the server atb, but its broadcast overwrites the second client's pendingbwitha. Define ancestor/descendant conflicts for replace-on-insert as well as exact-key ties.