-
Notifications
You must be signed in to change notification settings - Fork 14
Encrypted fields: isEncrypted support, explicit type tagging, and ADP findings #485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| --- | ||
| name: encrypted-fields-adp-probe | ||
| description: Issue #392 live characterization — an ADP-enabled iCloud test account exists (2026-09-14); run `mistdemo probe-encrypted` per account, not test-private | ||
| metadata: | ||
| type: project | ||
| --- | ||
|
|
||
| As of 2026-09-14 Leo has an Advanced Data Protection (ADP) iCloud test account for | ||
| issue #392 (encrypted fields over CloudKit Web Services). The live comparison | ||
| (standard protection vs ADP: scenarios B0/C4, B1a/B1b/B1c, B3) is run with | ||
| `swift run mistdemo probe-encrypted` (Examples/MistDemo), one run per account's | ||
| `CLOUDKIT_WEB_AUTH_TOKEN`, and the summary block pasted into issue #392. | ||
|
|
||
| **Why:** `test-private` aborts on the first failing phase and `EncryptedFieldsPhase` | ||
| deletes its zone, so it cannot persist a record under standard protection and read it | ||
| back under ADP, nor reach the asset-download step once the encrypted step fails. | ||
| `lookupRecords` also has no `zoneID` parameter, so the probe defaults to `_defaultZone`. | ||
|
|
||
| **Results so far (2026-09-14):** | ||
| - **B0/C4 standard account — PASS.** Encrypted STRING write needs an explicit `type` | ||
| (untagged → `BAD_REQUEST … ENCRYPTED_BYTES … defined to be: ENCRYPTED_STRING`); with | ||
| `type: STRING` CloudKit echoes `isEncrypted: true` + plaintext on modify, lookup and | ||
| query. Asset upload/download in the same record works. The asset came back with | ||
| `wrappingKey`/`referenceChecksum` set (contradicts the older "never appear live" note in | ||
| CLAUDE.md; not yet known whether that depends on the encrypted sibling field). | ||
| - **B1a ADP account — sign-in never yields a token.** CloudKit JS popup: with web access | ||
| off → "iCloud Data Web Access is Off"; with web access on + trusted device armed via | ||
| icloud.com → "Authentication Error — This action could not be completed." Standard | ||
| account signs in within a minute through the same flow. Chrome HAR (2026-09-14): Apple ID | ||
| sign-in + device approval succeed and 302 to `cdn.apple-cloudkit.com/ck-auth/` with an | ||
| `oauth_token`; the failing step is `setup.apple-cloudkit.com/setup/ws/1/oauth/validateToken` | ||
| → 200 `{"status":13}`, a status `ckauth.js` has no handler for (0 success, 7 TOS, 11 poll `requestPCS`, 12 web-access-off); telemetry `CKAUTHunknownError` / `CKJSUnexpectedAuthError`. | ||
| Chrome == Safari. iCloud.com control (same ADP account): `setup.icloud.com/setup/ws/1/` | ||
| `requestWebAccessState` → `enableDeviceConsentForPCS` → `requestPCS {appName:"notes3"}`, then | ||
| Notes' `/database/1/com.apple.notes/production/private/*` calls all 200 incl. encrypted fields — | ||
| first-party apps get per-app key release; third-party containers are refused before that loop. | ||
| Research write-up: `.claude/docs/research/adp-web-auth-signin.md` §1.1. | ||
| - `records/changes` is invalid in `_defaultZone` ("cannot get changes in default zone"); | ||
| `lookupRecords` has no `zoneID`, so the probe defaults to the default zone and skips | ||
| the changes step there (use `--zone-name` to swap which read is skipped). | ||
|
|
||
| **How to apply:** Prerequisite is `"secret" ENCRYPTED STRING` on `Note` deployed to the | ||
| `iCloud.com.brightdigit.MistDemo` development schema. Keep `CloudKitError` mapping as-is | ||
| (no ADP-specific case) until an ADP account actually reaches the API. | ||
|
|
||
| ## Session handoff — 2026-09-14 (stopped for a machine restart) | ||
|
|
||
| **Branch `392-encrypted-fields`, all work UNCOMMITTED in the worktree** (`git status` lists it). | ||
| Tracking issue for ADP: #486. Results comment posted on #392 (links #486). Research doc: | ||
| `.claude/docs/research/adp-web-auth-signin.md` (§1.1 HAR, §1.2 `{"status":13}` + `ckauth.js` table, | ||
| §1.3 icloud.com control). Both confirmed: nothing for MistKit to handle under ADP. | ||
|
|
||
| ### Done and verified | ||
| - `probe-encrypted` MistDemo command (5 new files under `Examples/MistDemo/Sources/MistDemoKit/`). | ||
| - MistKit fixes with tests: explicit `type` on encrypted fields (`Components.Schemas.RecordOperation`, | ||
| `FilterBuilder.cloudKitListType` now internal); `RecordInfo` tolerant decoder; logging middleware | ||
| 64 KiB replay via `Sources/MistKit/OpenAPI/ReplayingBodyIterator.swift` (actor). | ||
| - Live B0/C4 pass; C2 filter → `BAD_REQUEST "Field 'secret' has a value type of ENCRYPTED_STRING and | ||
| cannot be queried using filter type EQUALS"`; C2 sort → accepted, meaningless order. A1 client guard | ||
| fires; server-side writing a schema-`ENCRYPTED` field to the **public** DB without the flag → | ||
| per-record `BAD_REQUEST "encrypted fields are not supported in the public database"`. | ||
| - DocC: `WorkingWithRecords.md` "Encrypted fields" section + fixed stale `createRecord`/`updateRecord` | ||
| symbol links (now include `encryptedFields:`); `AuthenticationAndDatabases.md` ADP `> Important` note. | ||
| - MistDemo `create --encrypted-fields a,b` (key `record.encrypted.fields`… see `MistDemoKeys+Record`); | ||
| shared single-record table header now "Found 1 record(s)" and `create` prints its own ✅ line. | ||
| - Last green: MistKit 689 tests, MistDemo 1020 tests, `./Scripts/lint.sh` clean (before the last | ||
| MistDemo edits — re-run). | ||
|
|
||
| ### C3 answered (2026-09-14) | ||
| Writing `isEncrypted` to a field the schema declares **plain** → `BAD_REQUEST | ||
| "Attempt to save encrypted data in non encrypted field type"`. Note this differs from the | ||
| inverse direction (an `ENCRYPTED` field written **without** the flag), which reports | ||
| `invalid attempt to set value type ENCRYPTED_BYTES … defined to be: ENCRYPTED_STRING`. | ||
| The schema declaration governs; the flag must agree with it both ways. Control | ||
| (`secret:string:… --encrypted-fields secret`) succeeded and the read-back echoed | ||
| `"encryptedFields":["secret"]` with the value in plaintext. Posted to #392; the DocC | ||
| bullet "Existing fields cannot be converted" now carries both strings. | ||
|
|
||
| ### Pitfall that cost time — RESOLVED, it was a CLI bug (issue #487) | ||
| The earlier "No fields provided" failures were **not** zsh word-splitting (each flag was | ||
| quoted explicitly and it still failed). A comma-bearing `--field` value is mis-parsed by | ||
| `CommandLineArgumentsProvider`: `MistDemoKeys.Record.field` is an `OptionalConfigKey<String>` | ||
| and swift-configuration's `arraySeparator` defaults to `,`, so the value reads back as nil | ||
| and `CreateConfig.parseFieldsFromSources` throws `noFieldsProvided`. Proof: the **same** | ||
| string via `CLOUDKIT_FIELD=…` env works; `--field "title:string:C3 probe"` (no comma, spaces | ||
| fine) works; two separate `--field` flags also fail. Pre-existing, not a #392 regression. | ||
| **Workaround for live probes: one `--field` per value, no commas, or use the env var.** | ||
|
|
||
| ### Remaining | ||
| - ADP (#486) stays blocked — no supported way to get a `ckWebAuthToken` for a third-party | ||
| container while ADP is on. | ||
| - `Examples/MistDemo/.env` has **two** `CLOUDKIT_WEB_AUTH_TOKEN` lines (6 and 9) with | ||
| different values. Which wins is unverified; auth currently works, but edit the wrong one | ||
| and you will debug a "fresh" token that is not in effect. Worth de-duplicating. | ||
| - Leave `EncryptedFieldsPhase` pre-existing lint warnings (function length, type order) | ||
| unless cheap. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| --- | ||
| name: reference_android_emulator_flake | ||
| description: Android CI can fail after a fully successful build when the emulator never starts — adb "could not connect to TCP port 5554"; re-run, don't debug the code | ||
| metadata: | ||
| type: reference | ||
| --- | ||
|
|
||
| A `Build on Android (<swift>, <api>)` job can fail with **no compiler error at all**. The signature, | ||
| observed 2026-09-14 on `Build on Android (6.3, 33)` (run 34888402078): | ||
|
|
||
| ``` | ||
| [1195/1204] Compiling MistKitTests QueryFilterTests+ComplexFields.swift | ||
| [1200/1208] Emitting module MistKitPackageDiscoveredTests | ||
| [1207/1211] Emitting module MistKitPackageTests | ||
| error: could not connect to TCP port 5554: Connection refused | ||
| The process '.../platform-tools/adb' failed with exit code 1 | ||
| ##[error]The process '/usr/bin/sh' failed with exit code 1 | ||
| ``` | ||
|
|
||
| The Swift build **completed** — every target compiled and both test modules emitted. The failure is | ||
| `skiptools/swift-android-action` failing to reach the emulator over adb, i.e. the AVD never came up. | ||
|
|
||
| **Why this matters:** the failing step is `Run brightdigit/swift-build@v1`, so `gh pr checks` just | ||
| says the Android job failed, and it looks like a source break. It is not. Distinguish it from a real | ||
| failure by checking for `Emitting module MistKitPackageTests` in the log — if that line is present and | ||
| there is no `error:` from swift-frontend, the code built fine. | ||
|
|
||
| **How to apply:** re-run the job (`gh run rerun <run-id> --job <job-id>`) rather than changing source. | ||
| Corroborating check: the other Android matrix entries (other Swift versions / API levels) pass on the | ||
| same commit — a genuine source break fails all of them, a flake fails one. Distinct from | ||
| [[reference_wasm_ci_signatures]] (wasm OOM vs SDK-download curl exit 7) and from | ||
| [[reference_windows_62_mistkittests_emit_abort]] (Windows 6.2, which fails *before* emitting | ||
| MistKitTests and is a real size tip-over). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| name: reference_lint_needs_mise_trust | ||
| description: An untrusted mise.toml makes Scripts/lint.sh silently use the wrong SwiftLint and skip Periphery, producing phantom findings | ||
| metadata: | ||
| type: reference | ||
| --- | ||
|
|
||
| `Scripts/lint.sh` puts mise-managed tools on PATH via `eval "$(mise -C "$PACKAGE_DIR" env -s bash)"`. | ||
| If `mise.toml` is **untrusted** in this worktree, that `eval` silently produces nothing — mise exits | ||
| with `error parsing config file … are not trusted` on stderr and the script keeps going. The run then | ||
| uses whatever is on the bare PATH instead of the pinned versions. | ||
|
|
||
| Observed 2026-09-14 in the `392-encrypted-fields` worktree: | ||
|
|
||
| - SwiftLint fell back to bare **0.65.1** instead of the pinned **0.62.2**, inventing two | ||
| `superfluous_disable_command` errors against `force_unwrapping` disable blocks in | ||
| `CloudKitService.swift` and `Sharing/CreatedShare.swift`. Both files are untouched by the branch and | ||
| carry the same blocks on `main`; the pinned version does not flag them. Acting on those would have | ||
| deleted disable blocks that `.claude/agent-notes.md` explicitly endorses. | ||
| - **Periphery exited 127** (not installed on the bare PATH), so that leg never ran while still being | ||
| reported as `"skipped": false`. | ||
|
|
||
| **Why:** a lint report that looks complete but silently ran the wrong tools is worse than no report — | ||
| the phantom findings point at correct, deliberate code. | ||
|
|
||
| **How to apply:** run `mise trust` once per worktree (each `git trees add` worktree needs its own), | ||
| then confirm `mise exec -- swiftlint version` reports the pinned version and | ||
| `mise exec -- periphery version` resolves before believing a lint run. In the JSON report, treat | ||
| `periphery.exitCode == 127` or a surprise SwiftLint rule as "the tools were wrong", not "the code is | ||
| wrong". Verify with `swift-format`/`swiftlint`/`swift-build`/`periphery` all at `exitCode` 0. | ||
|
|
||
| Note `Scripts/lint.sh` also shells out to `swift build`; see [[reference_use_xcrun_for_swift]] — under | ||
| the swiftly-managed toolchain that leg crashes swift-frontend and reports `exitCode: 1`, which is | ||
| likewise a toolchain artifact rather than a code error. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| name: reference_use_xcrun_for_swift | ||
| description: On macOS, when swiftly lacks the pinned toolchain, run Swift via xcrun — the swiftly-managed swift crashes swift-frontend | ||
| metadata: | ||
| type: reference | ||
| --- | ||
|
|
||
| **On macOS only, and only when swiftly has no toolchain matching the pinned `.swift-version`**, | ||
| invoke Swift through `xcrun` — `xcrun swift build`, `xcrun swift test` — rather than the bare | ||
| `swift` on PATH. When swiftly *does* have the pinned toolchain, use it; this is a fallback, not a | ||
| blanket rule. | ||
|
|
||
| `xcrun` is an Xcode tool and **does not exist on Linux**. Ubuntu CI and any Linux agent invoke | ||
| `swift` directly — applying this rule there breaks the build. | ||
|
|
||
| Observed 2026-09-14 on the `392-encrypted-fields` worktree: the swiftly-managed default | ||
| (`~/Library/Developer/Toolchains/swift-6.3.2-RELEASE.xctoolchain`, matching the root `.swift-version` | ||
| of `6.3.2`) **crashed swift-frontend** with a stack dump while building the root package. `xcrun` | ||
| resolves to `/Applications/Xcode.app/…/XcodeDefault.xctoolchain` (Swift 6.4) and builds and tests | ||
| cleanly: 689 root tests, 1020 MistDemo tests. | ||
|
|
||
| `xcrun` also sidesteps a second problem: `Examples/MistDemo/.swift-version` pins `6.4`, and swiftly has | ||
| only `6.4.x-snapshot-*` installed (no release 6.4), so swiftly refuses to run there at all | ||
| ("uses toolchain version 6.4, but it doesn't match any of the installed toolchains"). | ||
|
|
||
| **Why:** a swift-frontend crash reads like a code error but is a toolchain artifact. It also corrupts | ||
| the `Scripts/lint.sh` report, whose `swift-build` leg exits 1 for the same reason — see | ||
| [[reference_lint_needs_mise_trust]]. | ||
|
|
||
| **How to apply:** prefix Swift commands with `xcrun`. When a script shells out to `swift` internally | ||
| (`Scripts/lint.sh` does), put the Xcode toolchain first on PATH for that invocation: | ||
| `export PATH="$(dirname "$(xcrun --find swift)"):$PATH"`. Beware that piping to `tail`/`grep` makes | ||
| `$?` the pipe's exit code, not the compiler's — capture `${PIPESTATUS[0]}` or echo an explicit | ||
| `EXIT=$?` marker before filtering, or a crashed build reads as success. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.