[Feature] Separate configuration, secrets, recovery state and disposable data with a tested backup/restore workflow

[nostalume]

Please confirm the following

• I have read and agree to AGPL-3.0 Section 15. The program is provided "as is" without any warranties; you bear all risks of using it.
• I have read and agree to AGPL-3.0 Section 16. The copyright holders and distributors are not liable for any damages resulting from the use or inability to use the program.
• I confirm my description is clear, polite, helps developers quickly locate the issue, and complies with community rules.
• I have read the OpenList documentation.
• I confirm there are no duplicate issues or discussions.
• I believe this issue must be handled by OpenList and not by a third party.
• I confirm this feature has not been implemented yet.
• I confirm this feature is reasonable and has general demand, not just my personal need.
• I have not read these checkboxes and therefore I just ticked them all, Please close this issue.

Feature Description

OpenList currently groups data by implementation location rather than by truth,
retention and sensitivity. config.json and environment variables jointly
determine process configuration; bootstrap rewrites the file before environment
overrides. The DB holds users, mounts, Driver Addition, settings, tasks and
other records; Driver initialization can refresh credentials into the storage
row. Settings flags distinguish public API visibility, not all secret handling.
Search state, multipart/temp bytes and task records have different recovery
needs despite sharing data-directory/storage mechanisms.

The frontend adds a separate client-device boundary: it stores an auth token
and a “remember password” value in localStorage. Its backup UI exports selected
settings/users/storages/metas/shares, optionally encrypted; no password yields
plaintext. That export is useful but does not by itself prove a full
disaster-recovery backup, and it may contain sensitive settings or Driver
Addition values.

Please define an operator workflow from first configuration through credential
rotation, consistent backup, isolated restore and verification, with explicit
classification of secret/public and authoritative/recoverable/rebuildable/
disposable state. Avoid a filesystem-only split that loses token refresh,
in-flight transfers or compatibility.

Suggested Solution

1. Inventory every config field, DB record, browser value and data-directory
   artifact by owner/writer, source of truth, precedence, API visibility,
   sensitivity, retention, backup, restore and rotation rule. Mark mixed records
   rather than forcing each into one category. Define behavior for unknown or
   obsolete settings during migration.
2. Define an explicit precedence/version contract for file, environment and
   admin-set values. Preserve Driver credential refresh and make effective
   sources inspectable without revealing values. Review sensitive logging,
   file creation/replacement permissions and redacted admin/export responses;
   do not assume one chmod rule works across platforms.
3. Separate durable authoritative records from reconstructible search/cache,
   recoverable task/upload state and disposable temp/logs by lifecycle. Define
   crash cleanup and pending-operation reconciliation before changing startup
   temp cleanup or retention.
4. Provide a consistent operator backup of the DB plus required config and
   secret sources; declare excluded projections and in-flight state. If a
   distributed pool is adopted, include committed manifests and stable account
   mapping. Restore into an isolated instance, rotate/resolve credentials,
   rebuild projections and verify list/read/write/authorization; pooled mode
   must also verify full and Range reads and shard health.
5. Review browser password/token persistence and the plaintext export path as
   a separate client-side secret workflow. Label configuration export honestly
   if it cannot restore all required server state.

Additional Information

Relevant source: internal/conf/config.go, internal/bootstrap/config.go,
internal/bootstrap/data/setting.go, internal/db/db.go, internal/op/storage.go,
model/storage.go, internal/bootstrap/task.go, internal/multipart,
internal/search, frontend src/utils/request.ts,
src/pages/login/index.tsx, src/pages/manage/backup-restore.tsx.
This proposal does not assert a verified exploit or a tested restore. It asks
for a contract and bounded validation before migration. Existing issues about
config behavior should be reviewed for overlap before posting: #1887 describes
a config rewrite bug and #1736 asks for temp cleanup, while this issue owns the
broader truth/secret/recovery classification.

AI Generated Content

• I used AI tools to generate this content
• I did not use AI tools to generate this content

AI model used

OpenAI Codex (exact model identifier not exposed to the agent)

[nostalume]

Architecture update (AI-assisted investigation; proposal, not implemented). This is based on backend origin/main@54ae9d7451707d76b4c92f0e946a407fb6bb4481 and frontend 031570fc7ae0ef18a4e36d68497ea7067b81fb16; it does not claim a tested backup/restore or a completed Driver audit.

More precise problem

The current boundary is not simply “file versus database.” The same records mix desired settings, runtime refresh state and secrets: bootstrap rewrites config.json before applying environment overrides (source); storage initialization can persist the whole Driver Addition, including Driver-mutated fields (source); the frontend currently edits the whole Addition through a generic form (source). A read-only source inventory found 87 Addition definitions, but that is not a field-level security or migration certification. Application release, Gorm table schema, per-Driver JSON shape, UI default and provider credential mode are different version axes. Existing version-keyed patches and Driver-local rewrites therefore cannot be treated as one atomic semantic migration.

Candidate architecture

1. Keep one backend-managed database as live authority for managed durable data, but classify each field and effect independently by lifecycle (declarative desired, durable domain, recoverable operational, derived, ephemeral) and sensitivity (public, restricted, secret, verifier/capability). A Go object may contain several classes. Frontend/browser preferences are a separate device owner; a public render snapshot is a derived, allowlisted backend projection, not a second configuration owner.
2. Give desired records stable logical IDs and typed, versioned semantics. Admission validates category/Driver-specific fields; a stored desired revision is distinct from effective activation/health. Driver manifests should reuse ordinary field/secret/reference primitives, while products own mode-specific validation, credential acquisition/refresh and legacy conversion. A single reflected Addition or field-name heuristic cannot preserve all products (e.g. one Google Drive field can mean an OAuth token or a host-local service-account path).
3. Provide categorized, text-editable export → validate → import into a fresh DB, with logical references rather than physical row IDs or mount paths. Configuration export contains secret references, never values, and is not advertised as full disaster-recovery backup. A writable encrypted-DB secret backend with an unlock key held outside the DB is a candidate, not yet selected; key custody, rotation and restore must be proved. Full-state recovery is a distinct operation covering durable identities/shares and recoverable work, with rebuildable indexes excluded.
4. Migrate one way, per product/observed legacy shape/mount instance. A read-only preflight reports automatic conversion, exact operator input needed, bounded legacy support, or blocked. Preserve pre-existing mounts at their paths while they await safe conversion, but create no new legacy mounts through UI, API, CLI or import once rollout starts. Per-mount cutover has one active writer and an idempotent receipt; failed Driver activation leaves typed desired state with an honest error/retry, not a raw-Addition fallback. A complete portable export is claimed only when no relevant legacy exclusions remain. Lossless old-binary downgrade after new-authority writes is not promised.

Dependency-ordered investigation/delivery stages

1. Close the pilot's evidence gates: one product's exact field/effect/legacy-shape audit, secret-key recovery, finite DB discovery, frontend/CLI writer inventory and preflight contract.
2. Prove one typed product end to end in a disposable/gated pilot: create, migrate, redact, refresh, fail/retry and partial export.
3. Expand Driver coverage and one-way rollout; independently classify process/site/identity/share/runtime categories and backend projections.
4. Prove logical config import plus distinct full-state backup/restore; validate split-frontend cache/session/custom-content trust separately before claiming static-host equivalence.

Open gates before a production migration: all product-specific conversions and writers, the writable secret/key choice, crash/restart behavior, missing or rotated credentials, stale frontend/CLI behavior, share-link capability policy, and the exact edit/retirement window for pre-existing legacy mounts. No production code or migration has been run as part of this investigation.

[nostalume]

S0 investigation update (AI-assisted; no production change or migration yet), scoped to the same backend/frontend revisions as the earlier comment.

• Historical S3 compatibility has a concrete counterexample: the 2022 change renamed the embedded root JSON key from root_folder to root_folder_path. In a removed, fake-key in-memory SQLite probe, passing the old key through today's op.CreateStorage/Driver init produced a saved row with root_folder_path:"": the old value was not preserved. This is not evidence that every real upgrade takes that path, but it rules out a blind old-JSON-to-current-struct converter. Preflight must inspect raw legacy rows before startup patches or Driver initialization can rewrite them. Old/new key conflicts, absent/default fields and unknown fields need explicit results.
• Policy decisions for the proposed one-way migration: pre-existing legacy mounts may keep running, but administrator setting/credential edits are blocked until conversion; new legacy creation and typed-to-legacy fallback remain forbidden. A separately typed, field-scoped enable/disable control may be allowed only after its writer/race behavior is tested. The backend gate must include stale API clients, frontend backup import, CLI and startup paths, not just the current form.
• DB selection: an explicit selector must win or fail visibly; without it, search only a documented finite, deduplicated SQLite path set. Multiple plausible DBs fail closed. Zero matches require explicit new-instance initialization, not silent creation. A legacy config.json can be a one-time read-only locator for nonstandard old DB settings, not a new live owner; custom/network DBs need an explicit selector afterward.
• If encrypted secret records in the one managed DB pass security/restore review, the selected pilot key-delivery default is an operator-provisioned, read-only secret file whose path is supplied through an environment variable. The path is not the key; OpenList does not generate/rewrite it or derive it from JwtSecret. Config export remains reference-only, and ciphertext restore requires separately protected key custody. This does not select the encrypted-DB provider for production.

Still open before a typed S3 pilot: historical converter fixtures, packaged DB-selector tests, key-file backup/wrong-key/rotation tests, exact ingress rejection and crash/restart/activation evidence. The full Driver inventory remains a later category gate.

[nostalume]

S0 database-admission follow-up (AI-assisted investigation; candidate design, not implemented). Backend evidence is from b25ba197881667e83a0dc10e31673534570a041f (the relevant paths match upstream 54ae9d7451707d76b4c92f0e946a407fb6bb4481).

What the bounded tests established. A temporary, read-only legacy-locator function matched the current effective database fields in five disposable Windows cases without modifying the config or creating a data directory; six separate current-behavior cases also passed. Normal mode used OPENLIST_DB_FILE, while --no-prefix used DB_FILE. Persisted force:true suppressed env parsing, but OPENLIST_FORCE=true supplied by env did not suppress DB overrides in that same parse. A relative explicit --config was based on the caller cwd; the relative DB file was not rebased to the config-file directory. With a nonempty PostgreSQL DSN, changing OPENLIST_DB_HOST changed the parsed field but not the DSN that current InitDB would use. No network DB was contacted. Both temporary test files were removed; no production code changed.

Refinement to the earlier selector sketch. Do not scan an unconditional union of cwd, executable, Docker and historical paths. Use an invocation-context default plus a read-only legacy locator, and treat other paths as hints. Conflict detection must compare the effective selected DB after old precedence and DB-type/DSN rules, not raw env-variable presence. A stale default config.json can cease being authoritative only after its configuration category has moved to the DB; an active old env/flag selector naming another DB remains a conflict. A converted custom/network DB needs a canonical selector rather than permanent dependence on the old file. A proposed DB-owned instance/version record must distinguish incomplete, transitional and admitted states, but it is not evidence that every mount or setting is converted.

A disposable decision model passed 22 named and 29 bounded invariant cases after exposing the custom-DB and active-selector corrections. This checks contract consistency, not current OpenList runtime behavior. Before implementation: prove version-tolerant read-only DB classification (including older schemas, WAL and foreign DBs), engine-specific single-writer cutover/revalidation, explicit-init crash recovery, and truthful start/restart child admission receipts.

[nostalume]

S0 compatibility evidence update (AI-assisted, no production code or live-instance migration). The current-backend check used b25ba197881667e83a0dc10e31673534570a041f; the release fixture was the official AList v3.20.0 Windows/amd64 binary, verified against its release MD5. All generated data was isolated and removed afterward.

• The old binary produced a real five-table SQLite DB with both admin and guest. It exited leaving a 4 KiB main DB plus committed WAL. Reading/copying the main file alone omitted x_users; a read-only VACUUM INTO snapshot retained both users. Therefore a legacy classifier/backup must account for WAL, not just the apparent .db file.
• On a complete isolated copy of that fixture, the current binary's first admin --data exited successfully and produced an integrity-clean eight-table DB, but the admin row still had its legacy password value and an empty pwd_hash. Guest was converted. A second run converted admin too. This is an observed one-pass migration postcondition failure for this one fixture, despite successful process exit; it is not a claim about all historical versions or a tested HTTP login response.
• A source-level explanation fits the observation: data initialization caches admin before patches; password conversion updates a fresh DB row without invalidating that cache; the later permission patch can save the stale cached admin row. The fixture began with admin permission zero, and the first upgraded row had the grant mask. This causal path is source-supported, not separately instrumented patch-by-patch.

Design consequence: admission/cutover must not publish a managed/ready receipt merely because startup and quick_check succeed. Each applicable legacy patch needs an idempotent, checked postcondition (including removal of applicable raw password fields), with an explicit incomplete/repairable result if it fails. A second boot cannot be the required completion step. This is a compatibility-migration gate, distinct from DB selection and later per-mount conversion. Other historical shapes, network DB ownership and explicit-init recovery remain open.

[nostalume]

Correction and additional S0 evidence (AI-assisted; isolated fixtures only, no production change). The previous comment's “second run converted admin” observation used a current-source binary with the default dev version, which reruns every upgrade patch on every startup. It does not describe release-version retry behavior.

I rebuilt the unchanged current source at b25ba197881667e83a0dc10e31673534570a041f with the project's release-style conf.Version=v4.2.6 ldflag and reran a checksum-verified AList v3.20.0 fixture on a complete copied DB/WAL/SHM set. First admin --data: successful exit, SQLite quick_check=ok, but admin still had an empty pwd_hash and a nonempty legacy password; config.json already recorded last_launched_version=v4.2.6. Second identical run: admin remained unconverted. The source explains the retry gate: InitConfig persists the new release version before patches, then InitUpgradePatch skips them when the persisted version equals the binary version. This is a stronger incomplete-migration risk than the previous dev-build result. Password login code compares against PwdHash, but I did not run an HTTP login attempt.

Separately, the official v4.1.2 release, checked against its release MD5, generated a real eight-table SQLite fixture (two hash-only users, zero mounts, 100 settings). An isolated first run with the same release-like current build retained the eight-table/two-user shape and added 16 settings with quick_check=ok. That bounds an empty-mount v4.1.2 upgrade only; it does not validate Alias/WebDAV mount patches or prove a table fingerprint authenticates a legacy instance.

Architecture correction: the old config's last_launched_version cannot serve as a migration-complete receipt. A DB-owned transition should keep the prior source version and an explicit incomplete state until required patches report success and their data postconditions pass. Moving the version write later alone is insufficient while patches log-and-return on errors and cached whole-row saves can undo earlier writes. This needs a targeted first-run compatibility regression test before any automatic one-way admission. All disposable fixtures and binaries were removed.

[nostalume]

S0 public-contract proposal for review (AI-assisted; not implemented or maintainer-approved). This narrows the earlier DB-admission design to a concrete selector and one-shot initialization boundary. Source basis: current config/env parsing, CLI flags, Docker entrypoint, and current setup handler.

Canonical DB locator: propose fixed, --no-prefix-independent OPENLIST_INSTANCE_DB_TYPE=sqlite3|mysql|postgres plus OPENLIST_INSTANCE_DB_TARGET (absolute host-native SQLite path, or complete engine-native network DSN). Optional OPENLIST_INSTANCE_DB_TABLE_PREFIX selects the existing Gorm table namespace; unset means x_, explicitly empty means no prefix. The prefix is physical instance binding, not portable site configuration. If any of these variables is supplied without a valid type/target pair, fail before fallback. DSN credentials and all three locator values are excluded from logical config export and redacted from diagnostics. Existing OPENLIST_DB_*/Docker DB_* remain one-time legacy locator inputs, not aliases; their effective target/namespace must agree with an explicit new selector. Old force:true cannot suppress the new selector. Without it, inspect only the context-derived default and effective old locator; never create a blank DB on ordinary startup.

Explicit creation: propose one-shot openlist init, not a persistent initialize-if-empty server flag. No canonical pair means only the resolved --data SQLite target; a pair means exactly that endpoint and namespace. Reject --config and --dev for fresh init, existing SQLite DB files, nonempty/foreign selected network table namespaces, and conflicting plausible old locators. Network init does not create the network database. Interrupted init remains distinguishable; rerunning it never silently starts a second instance. Successful DB provisioning is setup-required, not admin/site-ready; ordinary server/start/restart/admin never initialize on missing DB.

Not frozen for default-on rollout: the current public /api/public/init/setup writes admin then site title without a setup capability/atomic claim, while OPENLIST_ADMIN_PASSWORD can implicitly create admin. S3 must define an authorized first-admin/site claim and incomplete-state recovery before general routes are served. The image must expose the one-shot CLI invocation; its current entrypoint ignores custom commands except version. SQLite/network migration ownership, legacy patch postconditions and truthful parent/child readiness remain separate gates. The first bounded code slice should be read-only resolution/inspection, non-default until these integrated gates land. No new historical-release survey is required to choose this boundary.

[nostalume]

S0→S3/S4 first-run user-state refinement (AI-assisted architecture proposal; not implemented or maintainer-approved). This follows the S0 selector/init proposal above. Current-source basis: backend b25ba197881667e83a0dc10e31673534570a041f and frontend 031570fc7ae0ef18a4e36d68497ea7067b81fb16.

Why this needs a separate contract. Today public setup treats an admin row as “initialized” and writes admin then site title separately; startup can also create an admin from OPENLIST_ADMIN_PASSWORD. The current browser login sends a reusable client-side password hash and can retain the raw password locally; browser bearer tokens are kept in localStorage. These source observations motivate a new state boundary, not a claim of a demonstrated exploit.

Proposed new-instance contract.

1. openlist init provisions only a checked setup-required DB. No first web visitor, server startup, or missing-path fallback creates the admin. A local operator either issues a one-use, short-lived high-entropy setup code to an attached terminal for a trusted setup page, or uses a separate local headless claim action reading the password from an operator secret file. Both modes target the same selected DB and the same one-winner transaction.
2. That transaction verifies admission generation and claim authority; creates a stable logical admin principal and versioned slow password verifier; optionally updates the DB-owned public site title; consumes the code; and commits db-admitted. A failure changes none of those facts. Claim returns no login token/session. The existing customized /@init page must not be treated as trusted code-entry UI while arbitrary site scripts can reach it.
3. Distinguish DB admission from process activating/ready. Core login becomes ready only when the admitted DB, admin verifier, guest-policy projection, session repository and trusted admin transport work. Contradictory states (e.g. admitted with no valid admin) fail into operator recovery, never reopen unauthenticated setup. The frontend consumes explicit status and server /me; it does not infer readiness or manufacture guest identity after an auth error.
4. For the new trusted browser path, prefer an opaque random DB-backed session bound to immutable principal ID and monotonic auth generation, with only a hash of the session identifier stored server-side. Issue it in a host-only Secure, HttpOnly, SameSite=Lax, __Host- cookie; revoke on logout/security changes and enforce server-side expiry. Use Argon2id for newly claimed passwords. This removes a JWT signing-key dependency from core first login, not from existing JWT-dependent SSO/API features. Cookies require exact-origin/CSRF checks; an invalid credential must not fall back to guest.
5. For remote browser admin, propose HTTPS (localhost-only development exception). A separately deployed frontend can be composed with the API under one browser origin by a reverse proxy; true cross-origin admin hosting needs its own S4 review. Arbitrary public customization scripts cannot share the admin-cookie origin.

Still open before default-on migration: legacy DBs without an admin; old password-hash verification/upgrade and SSO/WebAuthn/API-client adapters; exact CLI/API and trusted-proxy/origin grammar; network-DB one-winner execution; session-query/KDF cost; and documented TLS transition for existing plain-HTTP LAN deployments. Legacy mounts and older-binary downgrade retain the one-way constraints discussed above. The next bounded code candidate remains the non-default, read-only S0 DB selector/inspector; this comment does not assert a completed implementation, security review, or end-to-end migration test.