Skip to content

RUST-REF-016 - Move blocking SQLite work behind bounded async persistence #155

Open

Description

@itlezy

Durable spec: https://github.com/emulebb/emulebb-tooling/blob/main/docs/products/emulebb-rust/active/items/RUST-REF-016.md

RUST-REF-016 - Move blocking SQLite work behind bounded async persistence

Summary

Remove synchronous SQLite waits and durable flushes from Tokio worker threads
and global async state locks. Introduce bounded persistence ownership that keeps
the network runtime responsive while preserving crash consistency and explicit
backpressure.

Current State

The metadata layer shares a synchronous rusqlite::Connection behind
std::sync::Mutex, uses synchronous=FULL, and allows a five-second busy
timeout. Core API methods and search paths call that storage directly from async
contexts, sometimes while holding the global core state or search-index lock.
Upload request handling also performs synchronous counter and metadata writes
before continuing peer work. A busy database or durable flush can therefore
block a Tokio worker and serialize unrelated network, REST, or scheduler work.

RUST-REF-008 already owns large-library ingestion batching. This item owns the
broader runtime boundary for interactive reads, compound writes, counters, and
hot peer paths.

Why This Matters

Blocking a cooperative executor on database mutex acquisition, query work, or
filesystem synchronization converts storage latency into protocol latency.
Adding more spawn_blocking calls ad hoc would hide ownership, permit
unbounded work, and make ordering harder to reason about.

Representative Sites

  • crates/emulebb-metadata/src/store.rs connection ownership and pragmas.
  • crates/emulebb-core/src/category_runtime.rs, server_api.rs,
    friend_api.rs, app_api.rs, and search_api.rs async callers.
  • crates/emulebb-metadata/src/transfer_store.rs transfer metadata and counter
    updates.
  • crates/emulebb-ed2k/src/ed2k_transfer/upload.rs upload accounting called
    from the peer payload path.

Intended Shape

  • Own SQLite writes in a dedicated bounded persistence executor or actor with
    typed commands, explicit queue capacity, and measured latency.
  • Use set-oriented transactions for compound state changes and bounded
    batching/coalescing for lossy counters where the durability contract permits.
  • Run blocking reads through a bounded read lane or dedicated read connections;
    never hold a Tokio mutex guard while waiting for SQLite.
  • Define per-command completion semantics so callers know whether they require
    durable commit, ordered enqueue, a snapshot read, or best-effort accounting.
  • Surface overload and database unavailability as typed errors or controlled
    backpressure rather than unbounded task creation.

Accepted Sharing-Path Follow-up (2026-11-06)

This owns approved follow-up 3 from the async-I/O review. It remains in the
post-beta backlog; the update does not promote it into a blocker for the
validated stock-import workflow. Main e3372aba and the import/root branch at
43855fba were reviewed before recording the following sites.

shared_file_api.rs still calls index.upsert_file synchronously after
offloaded ingest and calls index.replace_shared_directory_roots while holding
the async index guard. Ignore-rule list/add/remove/toggle paths also call the
synchronous metadata layer directly. The async guard does not make the SQLite
query, connection-mutex wait, busy wait, or durable commit asynchronous.

Route these live operations through the same bounded persistence owner as the
other runtime callers. Capture owned arguments, release global state/index
guards before queueing or awaiting storage, and publish the result according to
the command's required durable-commit semantics. Serialize logical root edits
or use configuration revisions so removing a guard does not introduce lost
updates. Do not replace a bounded owner with independent unbounded
spawn_blocking calls at each handler.

Keep SQLite synchronous inside that worker. The current Arc<Mutex<Connection>>
serializes users of each shared connection even with WAL: a five-second SQLite
busy timeout does not bound time waiting for the Rust connection mutex. Expose
both queue/connection wait and execution/commit latency. A read-lane or
connection-pool expansion needs measured justification and explicit snapshot
consistency; it is not a prerequisite to isolating blocking work.

Whole-file ingest, its local-content/AICH persistence, media extraction, root
membership updates, and paged catalog reads already use blocking boundaries in
important paths. Preserve and consolidate that good behavior. Exclusive offline
imports may keep bounded synchronous transactions because they serve no live
network/API workload.

RUST-REF-018 owns filesystem probes and root state lock scope. RUST-REF-008 owns
the compiled ignore-policy cache and eliminating per-file query/compile work;
its rule edits/invalidation use this persistence boundary. RUST-REF-019 owns
CPU-only bursts. These items must not introduce competing SQLite actors.

Additional representative sites:

  • crates/emulebb-core/src/shared_file_api.rs: final index upsert, root
    replacement, ignore-rule operations, and current cached progress accessor.
  • crates/emulebb-index/src/lib.rs: synchronous metadata adapter methods.
  • crates/emulebb-ed2k/src/ed2k_transfer/ingest.rs: already-offloaded durable
    local-content/AICH writes to preserve.

Additional acceptance criteria:

  • Live root edits, post-ingest index updates, and ignore-rule operations use
    the bounded storage boundary without global async guards across waits.
  • Injected SQLite mutex contention, busy waits, and slow FULL commits leave
    unrelated REST requests, peer timers, and cached progress responsive.
  • Queue saturation and shutdown preserve required commit guarantees and do
    not silently lose root edits or mark uncommitted content durable.

Scope Constraints

  • Preserve synchronous=FULL where the durability contract requires it; improve
    scheduling and transaction shape rather than weakening correctness.
  • Keep password/secret handling and profile isolation unchanged; encryption
    feature work is not part of this scheduling change.
  • Preserve stock eMule wire timing rules and packet behavior.
  • Keep large-library batching performance work in RUST-REF-008 and category
    deletion atomicity in RUST-BUG-131; share one persistence architecture.

Acceptance Criteria

  • No synchronous SQLite operation runs on a Tokio worker thread while
    holding global core, search, or peer-session async state.
  • Persistence queues and blocking read concurrency are explicitly bounded
    and expose queue depth, wait time, execution time, and failure counters.
  • Compound operations use one transaction and publish in-memory state only
    after the required commit succeeds.
  • Upload accounting cannot delay payload service on a database busy timeout;
    its durability and coalescing policy is explicit and tested.
  • Shutdown drains or rejects persistence commands according to documented
    command guarantees without silently losing required writes.
  • Database contention tests prove REST, search, and peer timers remain
    responsive under the configured busy timeout and durable flush latency.

Validation

  • Tokio paused-time and multi-thread contention tests with injected slow SQLite
    operations and queue saturation.
  • Crash/restart tests for commands that promise durable completion.
  • Upload/search/API latency regressions under forced database contention.
  • Current-schema rejection, metadata durability, REST, eD2K, Kad, and
    large-library validation lanes. Product code gains no migrations or reset path.

Audit Release Placement

This remains parallel follow-up work and does not independently block Beta 3.
Its existing Client, Indexer, or Arr phase ownership remains unchanged;
non-gating Beta 3 placement does not bypass the separate whole-phase gates.

Audit Reconciliation (2026-11-09)

The complete audit collection was reconciled against current source and
owning issues. The following are retained findings, corrected assumptions,
or required evidence for this owner; inspection does not replace runtime proof.

  • emulebb-rust-prompt_audit_findings_persistence.md section 2.4 (existing): Do not add speculative pools or independent actors. Existing REF016 owns queue/wait/commit semantics and read-lane justification.
  • emulebb-rust-prompt_audit_findings_product_direction_amule_future_parity.md section scale-proof (existing): The aMule scaling wave is inspiration, not evidence of the same defect in Rust. Existing storage-domain, scan-I/O, async SQLite/filesystem/CPU and upload-performance items own the remaining work.
  • emulebb-rust-prompt_audit_findings_product_direction_amule_future_parity.md section A3-kad-cache-load (existing): Kad cache hydration calls SQLite synchronously during core construction. Existing async SQLite persistence work is the nearest owner; the review did not measure startup blocking or establish a new release defect.

Scoped Follow-up Acceptance

  • Preserve FULL durability and measure connection/queue latency; keep one storage owner.
  • Use bounded representative benchmarks and latency evidence.
  • Reuse existing 100k fixture and cross-platform owners.
  • Required Beta 3 Windows/Linux evidence is selected under RUST-CI-009 - Certify a Beta 3 candidate without publishing #142; optional platforms remain parallel.
  • Measure representative cache startup and executor responsiveness.
  • Offload substantial work through the bounded persistence owner if required.
  • Preserve startup consistency and the existing cache model.

Retained Public Issue Context

The following pre-intake context is retained for provenance. The release
placement and scope stated in this item govern current execution.

Reviewable branch spec: https://github.com/emulebb/emulebb-tooling/blob/fix/stock-hash-share-tree/docs/products/emulebb-rust/active/items/RUST-REF-016.md

Activity

  1. added this to the post-beta-polish milestone on Oct 23, 2026
  2. added
    rustPull requests that update rust code
    on Oct 23, 2026
  3. added
    auditReviewed audit finding, evidence or disposition
    parallel-non-gatingParallel follow-up outside the Beta 3 release gate
    on Nov 9, 2026
  4. itlezy commented on Nov 10, 2026

    @itlezy
    ContributorAuthor

    Readiness review on Rust 15020b76: WAL/FULL and a five-second SQLite busy timeout already exist, and several transfer/catalog/ingest paths already offload SQL. The remaining gaps are shared admission and shutdown bounds, synchronous SQL under core/index guards, upload request accounting before payload service, and credit flush jobs that can accumulate under contention.

    Sequence this after #248, #299 and #247 are integrated:

    1. Prove one bounded executor per live profile with friend CRUD: typed admission/completion, bounded payloads, queue/connection/execute metrics, close/drain/join, and lease retention through shutdown. Start on the existing connection to preserve its connection-local verified-path state.
    2. Move interactive producers in bounded batches: settings/categories, search/servers, and sharing roots/ignore rules. Reserve one owner for shared core wiring. Order read-modify-write intent and publish committed state even if the original caller cancels.
    3. Move upload accounting off the payload path. Reuse the credit coalescer with bounded keys and one pending flush; retain durable ordering for secure identity binds and wipes. Measure Kad hydration and 100k-file behavior before adding more lanes.

    Proof must cover queue saturation, a held store mutex, SQLite busy contention, slow and failed commits, and shutdown. Unrelated cached routes and peer timers must continue; acknowledge durable writes only after commit and prove recovery by restart. No schema, TOML or migration change is needed for the foundation. Decide credit retention/ranking behavior before any outage fallback that changes accounting.

    This is design preparation; implementation and runtime qualification remain pending. The item stays parallel and non-gating.

  5. itlezy commented on Nov 12, 2026

    @itlezy
    ContributorAuthor

    Parent coordination: the approved next reliability slice is 155-A: one bounded persistence owner plus friend CRUD, after the change-aware CI batch #327. This accepts the implementation boundary, not a completed implementation or all of #155.

    Contract accepted after current-source review at Rust 080a5de and independent review:

    • One explicitly composed fixed worker and typed FIFO for migrated operations; bound admitted command count and owned payload bytes through completion. Reject saturation/closing explicitly; no unbounded spawn_blocking or producer tasks.
    • Preserve schema 68, WAL/FULL durability, current busy behavior, secrets and OS-backed profile lease. Offline synchronous constructors must remain usable without a Tokio runtime.
    • Normalize/capture friend input and release global CoreState before storage admission/wait. Serialize each friend mutation through durable commit AND cache publication; caller cancellation cannot release ordering or leave committed DB/cache divergent. Reply with success only after required publication.
    • Release the SQLite connection mutex before waiting for any cache/CoreState publication. Existing legacy synchronous DB-under-CoreState paths must not deadlock the new lane.
    • Shutdown rejects new work, drains accepted required writes and their finalizers, joins the worker before releasing the connection/profile lease, and reports a failed drain honestly.
    • Test slot/byte saturation, forced connection contention/slow commit with responsive unrelated work, failed commit, concurrent same-key mutations, cancellation before/after admission and drain/reopen outcomes. Measure queue/connection/execution/commit behavior without claiming performance from source inspection.

    Reserve metadata executor/exports and composition seam, core friend APIs and their domain-order boundary, daemon drain seam, and existing friends REST error mapping/tests. No schema edits in this slice. #233 owns later typed/generation-safe Kad snapshots and removal of met-file writes using this same owner. Parent serializes shared Core/schema/shutdown changes.

    Legacy callers remain during the first slice: do not claim global FIFO, complete off-executor persistence or closure of #155. Existing Project phase and release placement are not silently changed. Config/schema reviews #297/#298 retain their owners; Beta3, APK/signing and long soak remain held.

  6. itlezy commented on Nov 13, 2026

    @itlezy
    ContributorAuthor

    Additional #155-A integration constraints from a read-only review of Rust b73e16e7497766b6ad5f75a19b28b03aa18d9d79. The accepted bounded friend-CRUD contract is unchanged; this records concrete seams needed before implementation, not tested behavior.

    • Core's public constructors are synchronous and converge on the common constructor in crates/emulebb-core/src/lib.rs:575. Existing plain tests construct Core without Tokio. Compose one runtime-independent fixed worker there, after the shared state exists; Core clones share its handle. Constructors must not acquire a Tokio runtime or spawn per clone.
    • Retain the metadata store and narrow publication state in the worker, not the full Core or its owning client handle. Keep join ownership take-once and avoid a strong reference cycle. The cloned metadata store already retains the OS-backed profile lease through its shared file handle.
    • Daemon run_inner constructs Core before recovery, WebUI and listener operations that can return early errors. A drain added only to normal graceful teardown misses those exits. Provide one explicit post-construction success/error boundary that closes admission, drains accepted writes and publication, and joins before returning. Preserve both the primary failure and any drain failure.
    • The summary task retains a Core clone, so last-Arc destruction is not the shutdown contract. Existing network teardown has a cancellable timeout and returns only warnings; required persistence drain must sit outside that cancellation boundary and report failure honestly. The CLI already retains the profile lease across run and logging shutdown; persistence completion belongs before run returns.

    Reuse existing take-once owner/join and close-before-drain patterns where suitable, without importing lossy logging or forced-abort semantics into required durable writes. No implementation, schema change, runtime execution or new queue is introduced by this review.

  7. moved this from Todo to In Progress in eMuleBB Roadmapon Nov 15, 2026
  8. itlezy commented on Nov 15, 2026

    @itlezy
    ContributorAuthor

    A bounded failure-atomicity repair is now allocated on fix/peer-credit-settlement, based on current main 23e685b8. Source inspection confirmed settle_parked_peer_credit removes pending deltas before fallible SQL and loses them on error, while the batch flush already restores failed deltas.

    The selected repair restores by saturating merge under the existing flush gate and propagates the original error. Real-SQL failure, unchanged durable state, retained/concurrently added pending deltas, successful retry and exact-once reopen are the acceptance proof. Rust execution is pending the current full-validation owner; no runtime pass is claimed.

    This preserves schema 69, FULL durability, current storage ownership, secure-ident wipe ordering and the existing best-effort crash window. The broader bounded persistence actor, queue/backpressure and shutdown contracts remain unimplemented; this slice does not close #155 or #233. Project status tracks active execution of this slice, with phase and priority unchanged.

  9. itlezy commented on Nov 15, 2026

    @itlezy
    ContributorAuthor

    The scoped restore patch is pushed at Rust 4e6e0b86; source review accepted it for testing. Rust execution remains pending the shared validation slot. Allocation and remaining boundaries are merged in emulebb/emulebb-tooling#190.

    Independent review also confirmed a separate existing first-bind ordering gap: settle_parked_peer_credit releases the flush gate before pin_verified_secure_ident, while producers can still add parked deltas. A concurrent pre-bind delta can therefore be applied after the first-key wipe. Fixing that requires a coherent linearization of additions, drains and bind/wipe, plus deterministic interleaving and SQL-failure coverage. Extending only the flush gate does not cover additions. This remains outside the narrow restore patch and is not claimed fixed.

    The analogous absolute-seed helper currently has only test callers; no production race is established for that path. The broad persistence owner remains unimplemented. Current schema is 69; the older schema-68 references identify historical review baselines.

  10. itlezy commented on Nov 15, 2026

    @itlezy
    ContributorAuthor

    Read-only follow-up review identified a candidate for a separate first-bind repair: existing flush gate, one metadata transaction, then a brief same-peer ledger handoff while the connection is owned. Check the authoritative key before detaching; reject a mismatched key without mutations. Concurrent adds must remain free of SQL/flush-gate waits. Combine the detached totals with durable state and apply stock first-bind zero versus 1/1 wiping in that transaction. Roll back and merge detached credit before releasing connection ownership on failure.

    This is a proposal, not an allocated or implemented extension to the current restore slice. Before implementation, resolve same-key settlement/completion semantics, rollback-failure behavior, and verify every production identity reader uses the shared connection. Current read-through scoring reads SQL and parked totals separately; coherent first-bind snapshots also need one connection-ownership interval. No inspected production path currently nests ledger ownership around connection acquisition.

    Required proof includes controlled additions, competing flush/bind/readers, mismatched-key rejection, SQL and commit failures, exact restoration, retry/reopen and progress of additions during blocked SQL. A later-bucket sweep after SQL returns is unsafe. The existing batch-flush drain-before-connection snapshot gap remains separate; this proposal does not claim to fix all read-through races. No schema or generation registry is currently justified by this review.

  11. itlezy commented on Nov 15, 2026

    @itlezy
    ContributorAuthor

    Delivered failed parked-peer-credit settlement preservation in PR #335, merged at 0840e90: restore pending deltas after SQL failure, merge additions without overwrite, and retry exactly once. The broader persistence actor/friend CRUD and separate first-bind/add/flush boundary remain open. Durable evidence: https://github.com/emulebb/emulebb-tooling/blob/main/docs/products/emulebb-rust/active/items/RUST-REF-016.md

    The final pre-merge source c6b9cef passed the six-stage full local owner gate and all applicable hosted checks. Android local proof is compile/link; public APK emulator evidence is separately source-bound in the Android item. Fresh actual-main workflows are tracked separately. Tooling PR #191 merged the final records.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    auditReviewed audit finding, evidence or dispositionparallel-non-gatingParallel follow-up outside the Beta 3 release gaterustPull requests that update rust code

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions