feat: make owner-only state persistence configurable and turn it off for enclaves

[koenvanderveen]

An enclave gets an ephemeral keypair on every boot, so restoring a previous boot's state does not fit its security model — and it wipes its state anyway under fresh_state. Every write the DO makes purely so it can rebuild itself later is therefore pure cost, spent on a rate-limited Drive API inside a 1-second poll loop. Until now there was no way to switch that off; the only control was the per-call sync(auto_checkpoint=...) kwarg, and the enclave calls sync() with no arguments from ~12 sites.

What "owner-only state" means

State read back by nobody but this datasite, existing solely to restore it:

1. The append-only event log (owner_write_events_message_to_syftbox)
2. The rolling state — uploaded on every event, since the threshold is 1
3. Incremental and full checkpoints
4. All the reads of the above, which happen only in pull_initial_state

Peer-facing state is explicitly not in scope: the outbox, collections and peer files are untouched.

Two things worth knowing, both verified: the owner log and the peer outbox are separate queues drained independently in process_syftbox_events_queue, so dropping the log cannot affect peer delivery. And the DO's own local changes never reach that log — _queue_events_for_outbox only fills the outbox — so they were durably recorded only via rolling state → checkpoints. Turning all four off therefore leaves genuinely zero restore state.

Changes

• persist_owner_state flag, default True, so notebook and Colab clients are unaffected. It sits on DatasiteOwnerSyncerConfig and SyftboxManagerConfig and is threaded through every factory, mirroring use_in_memory_cache.
• Gated at four points, one per item above. The event log is gated at its single enqueue site in queue_event_for_syftbox, which avoids an unbounded queue rather than draining-and-discarding.
• pull_initial_state skips the whole owner-state restore when off, rather than falling through to replaying the log. Collections are still restored either way.
• Config wins over the per-call arg. sync(auto_checkpoint=...) is unchanged but subordinate, so no existing sync() call site needed touching. The automatic path returns None; the manual checkpoint methods are simply not called rather than raising.
• Enclaves default to off via EnclaveSettings.persist_owner_state (SYFT_ENCLAVE_PERSIST_OWNER_STATE), flowing through for_enclave into the DO syncer config. Both enclave entrypoints pass and log it.
• pull_initial_state split up. It was ~100 lines and its restore half is exactly what needed skipping, so it is now _restore_owner_state plus one helper per step. One subtlety preserved: a full checkpoint carrying no last_event_timestamp must not trigger a download-all, hence _CheckpointRestore.found_checkpoint being tracked separately from the cursor.

Deploy

Because the default is False, every deploy path is already correct with no template change — just start, start-debug, terraform apply, local-run and the inference-* variants all get it for free. The only deploy edit is adding SYFT_ENCLAVE_PERSIST_OWNER_STATE to both Dockerfiles' allow_env_override labels (Confidential Space rejects un-allowlisted tee-env-* overrides) so it can be switched on later. The Justfile / terraform tee-env- keys are deliberately left alone.

Test plan

• just test-unit: 431 passed. The 2 failures in tests/unit/syft_bg/test_init.py are a pre-existing macOS /private/var symlink issue, confirmed on a stashed clean tree.
• just test-unit-rds (79), just test-unit-enclave (73), just test-unit-enclave-model-api (5): all pass.
• The 30 existing checkpoint/rolling-state tests pass untouched — the main signal that the default path is unchanged.
• New tests: no owner-only state of any kind reaches the backend when off (checkpoints, incrementals, rolling state, event log, local rolling_state.json) while the peer still receives every change; a cold start restores nothing and attempts zero event downloads; the owner state is all present by default; enclave settings default + env override; and a regression test on the **kw passthrough in syft-rds, which nothing type-checks and where a rename would silently hand an enclave back the state it must not keep.
• pre-commit run passes on all changed files.

Not run: the just local-run end-to-end check, which needs docker plus real Drive credentials and writes to a live datasite.