Repository navigation
Description
Durable spec: https://github.com/emulebb/emulebb-tooling/blob/main/docs/products/emulebb-rust/active/items/RUST-REF-016.md
RUST-REF-016 - Move blocking SQLite work behind bounded async persistence
Summary
Remove synchronous SQLite waits and durable flushes from Tokio worker threads
and global async state locks. Introduce bounded persistence ownership that keeps
the network runtime responsive while preserving crash consistency and explicit
backpressure.
Current State
The metadata layer shares a synchronous rusqlite::Connection behind
std::sync::Mutex, uses synchronous=FULL, and allows a five-second busy
timeout. Core API methods and search paths call that storage directly from async
contexts, sometimes while holding the global core state or search-index lock.
Upload request handling also performs synchronous counter and metadata writes
before continuing peer work. A busy database or durable flush can therefore
block a Tokio worker and serialize unrelated network, REST, or scheduler work.
RUST-REF-008 already owns large-library ingestion batching. This item owns the
broader runtime boundary for interactive reads, compound writes, counters, and
hot peer paths.
Why This Matters
Blocking a cooperative executor on database mutex acquisition, query work, or
filesystem synchronization converts storage latency into protocol latency.
Adding more spawn_blocking calls ad hoc would hide ownership, permit
unbounded work, and make ordering harder to reason about.
Representative Sites
crates/emulebb-metadata/src/store.rsconnection ownership and pragmas.crates/emulebb-core/src/category_runtime.rs,server_api.rs,
friend_api.rs,app_api.rs, andsearch_api.rsasync callers.crates/emulebb-metadata/src/transfer_store.rstransfer metadata and counter
updates.crates/emulebb-ed2k/src/ed2k_transfer/upload.rsupload accounting called
from the peer payload path.
Intended Shape
- Own SQLite writes in a dedicated bounded persistence executor or actor with
typed commands, explicit queue capacity, and measured latency. - Use set-oriented transactions for compound state changes and bounded
batching/coalescing for lossy counters where the durability contract permits. - Run blocking reads through a bounded read lane or dedicated read connections;
never hold a Tokio mutex guard while waiting for SQLite. - Define per-command completion semantics so callers know whether they require
durable commit, ordered enqueue, a snapshot read, or best-effort accounting. - Surface overload and database unavailability as typed errors or controlled
backpressure rather than unbounded task creation.
Accepted Sharing-Path Follow-up (2026-11-06)
This owns approved follow-up 3 from the async-I/O review. It remains in the
post-beta backlog; the update does not promote it into a blocker for the
validated stock-import workflow. Main e3372aba and the import/root branch at
43855fba were reviewed before recording the following sites.
shared_file_api.rs still calls index.upsert_file synchronously after
offloaded ingest and calls index.replace_shared_directory_roots while holding
the async index guard. Ignore-rule list/add/remove/toggle paths also call the
synchronous metadata layer directly. The async guard does not make the SQLite
query, connection-mutex wait, busy wait, or durable commit asynchronous.
Route these live operations through the same bounded persistence owner as the
other runtime callers. Capture owned arguments, release global state/index
guards before queueing or awaiting storage, and publish the result according to
the command's required durable-commit semantics. Serialize logical root edits
or use configuration revisions so removing a guard does not introduce lost
updates. Do not replace a bounded owner with independent unbounded
spawn_blocking calls at each handler.
Keep SQLite synchronous inside that worker. The current Arc<Mutex<Connection>>
serializes users of each shared connection even with WAL: a five-second SQLite
busy timeout does not bound time waiting for the Rust connection mutex. Expose
both queue/connection wait and execution/commit latency. A read-lane or
connection-pool expansion needs measured justification and explicit snapshot
consistency; it is not a prerequisite to isolating blocking work.
Whole-file ingest, its local-content/AICH persistence, media extraction, root
membership updates, and paged catalog reads already use blocking boundaries in
important paths. Preserve and consolidate that good behavior. Exclusive offline
imports may keep bounded synchronous transactions because they serve no live
network/API workload.
RUST-REF-018 owns filesystem probes and root state lock scope. RUST-REF-008 owns
the compiled ignore-policy cache and eliminating per-file query/compile work;
its rule edits/invalidation use this persistence boundary. RUST-REF-019 owns
CPU-only bursts. These items must not introduce competing SQLite actors.
Additional representative sites:
crates/emulebb-core/src/shared_file_api.rs: final index upsert, root
replacement, ignore-rule operations, and current cached progress accessor.crates/emulebb-index/src/lib.rs: synchronous metadata adapter methods.crates/emulebb-ed2k/src/ed2k_transfer/ingest.rs: already-offloaded durable
local-content/AICH writes to preserve.
Additional acceptance criteria:
- Live root edits, post-ingest index updates, and ignore-rule operations use
the bounded storage boundary without global async guards across waits. - Injected SQLite mutex contention, busy waits, and slow FULL commits leave
unrelated REST requests, peer timers, and cached progress responsive. - Queue saturation and shutdown preserve required commit guarantees and do
not silently lose root edits or mark uncommitted content durable.
Scope Constraints
- Preserve
synchronous=FULLwhere the durability contract requires it; improve
scheduling and transaction shape rather than weakening correctness. - Keep password/secret handling and profile isolation unchanged; encryption
feature work is not part of this scheduling change. - Preserve stock eMule wire timing rules and packet behavior.
- Keep large-library batching performance work in
RUST-REF-008and category
deletion atomicity inRUST-BUG-131; share one persistence architecture.
Acceptance Criteria
- No synchronous SQLite operation runs on a Tokio worker thread while
holding global core, search, or peer-session async state. - Persistence queues and blocking read concurrency are explicitly bounded
and expose queue depth, wait time, execution time, and failure counters. - Compound operations use one transaction and publish in-memory state only
after the required commit succeeds. - Upload accounting cannot delay payload service on a database busy timeout;
its durability and coalescing policy is explicit and tested. - Shutdown drains or rejects persistence commands according to documented
command guarantees without silently losing required writes. - Database contention tests prove REST, search, and peer timers remain
responsive under the configured busy timeout and durable flush latency.
Validation
- Tokio paused-time and multi-thread contention tests with injected slow SQLite
operations and queue saturation. - Crash/restart tests for commands that promise durable completion.
- Upload/search/API latency regressions under forced database contention.
- Current-schema rejection, metadata durability, REST, eD2K, Kad, and
large-library validation lanes. Product code gains no migrations or reset path.
Audit Release Placement
This remains parallel follow-up work and does not independently block Beta 3.
Its existing Client, Indexer, or Arr phase ownership remains unchanged;
non-gating Beta 3 placement does not bypass the separate whole-phase gates.
Audit Reconciliation (2026-11-09)
The complete audit collection was reconciled against current source and
owning issues. The following are retained findings, corrected assumptions,
or required evidence for this owner; inspection does not replace runtime proof.
emulebb-rust-prompt_audit_findings_persistence.mdsection 2.4 (existing): Do not add speculative pools or independent actors. Existing REF016 owns queue/wait/commit semantics and read-lane justification.emulebb-rust-prompt_audit_findings_product_direction_amule_future_parity.mdsection scale-proof (existing): The aMule scaling wave is inspiration, not evidence of the same defect in Rust. Existing storage-domain, scan-I/O, async SQLite/filesystem/CPU and upload-performance items own the remaining work.emulebb-rust-prompt_audit_findings_product_direction_amule_future_parity.mdsection A3-kad-cache-load (existing): Kad cache hydration calls SQLite synchronously during core construction. Existing async SQLite persistence work is the nearest owner; the review did not measure startup blocking or establish a new release defect.
Scoped Follow-up Acceptance
- Preserve FULL durability and measure connection/queue latency; keep one storage owner.
- Use bounded representative benchmarks and latency evidence.
- Reuse existing 100k fixture and cross-platform owners.
- Required Beta 3 Windows/Linux evidence is selected under RUST-CI-009 - Certify a Beta 3 candidate without publishing #142; optional platforms remain parallel.
- Measure representative cache startup and executor responsiveness.
- Offload substantial work through the bounded persistence owner if required.
- Preserve startup consistency and the existing cache model.
Retained Public Issue Context
The following pre-intake context is retained for provenance. The release
placement and scope stated in this item govern current execution.
Reviewable branch spec: https://github.com/emulebb/emulebb-tooling/blob/fix/stock-hash-share-tree/docs/products/emulebb-rust/active/items/RUST-REF-016.md
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsIn Progress
Activity
itlezy commentedon Nov 10, 2026
Readiness review on Rust
15020b76: WAL/FULL and a five-second SQLite busy timeout already exist, and several transfer/catalog/ingest paths already offload SQL. The remaining gaps are shared admission and shutdown bounds, synchronous SQL under core/index guards, upload request accounting before payload service, and credit flush jobs that can accumulate under contention.Sequence this after #248, #299 and #247 are integrated:
Proof must cover queue saturation, a held store mutex, SQLite busy contention, slow and failed commits, and shutdown. Unrelated cached routes and peer timers must continue; acknowledge durable writes only after commit and prove recovery by restart. No schema, TOML or migration change is needed for the foundation. Decide credit retention/ranking behavior before any outage fallback that changes accounting.
This is design preparation; implementation and runtime qualification remain pending. The item stays parallel and non-gating.
itlezy commentedon Nov 12, 2026
Parent coordination: the approved next reliability slice is 155-A: one bounded persistence owner plus friend CRUD, after the change-aware CI batch #327. This accepts the implementation boundary, not a completed implementation or all of #155.
Contract accepted after current-source review at Rust 080a5de and independent review:
Reserve metadata executor/exports and composition seam, core friend APIs and their domain-order boundary, daemon drain seam, and existing friends REST error mapping/tests. No schema edits in this slice. #233 owns later typed/generation-safe Kad snapshots and removal of met-file writes using this same owner. Parent serializes shared Core/schema/shutdown changes.
Legacy callers remain during the first slice: do not claim global FIFO, complete off-executor persistence or closure of #155. Existing Project phase and release placement are not silently changed. Config/schema reviews #297/#298 retain their owners; Beta3, APK/signing and long soak remain held.
itlezy commentedon Nov 13, 2026
Additional #155-A integration constraints from a read-only review of Rust
b73e16e7497766b6ad5f75a19b28b03aa18d9d79. The accepted bounded friend-CRUD contract is unchanged; this records concrete seams needed before implementation, not tested behavior.crates/emulebb-core/src/lib.rs:575. Existing plain tests construct Core without Tokio. Compose one runtime-independent fixed worker there, after the shared state exists; Core clones share its handle. Constructors must not acquire a Tokio runtime or spawn per clone.run_innerconstructs Core before recovery, WebUI and listener operations that can return early errors. A drain added only to normal graceful teardown misses those exits. Provide one explicit post-construction success/error boundary that closes admission, drains accepted writes and publication, and joins before returning. Preserve both the primary failure and any drain failure.runand logging shutdown; persistence completion belongs beforerunreturns.Reuse existing take-once owner/join and close-before-drain patterns where suitable, without importing lossy logging or forced-abort semantics into required durable writes. No implementation, schema change, runtime execution or new queue is introduced by this review.
itlezy commentedon Nov 15, 2026
A bounded failure-atomicity repair is now allocated on
fix/peer-credit-settlement, based on current main23e685b8. Source inspection confirmedsettle_parked_peer_creditremoves pending deltas before fallible SQL and loses them on error, while the batch flush already restores failed deltas.The selected repair restores by saturating merge under the existing flush gate and propagates the original error. Real-SQL failure, unchanged durable state, retained/concurrently added pending deltas, successful retry and exact-once reopen are the acceptance proof. Rust execution is pending the current full-validation owner; no runtime pass is claimed.
This preserves schema 69, FULL durability, current storage ownership, secure-ident wipe ordering and the existing best-effort crash window. The broader bounded persistence actor, queue/backpressure and shutdown contracts remain unimplemented; this slice does not close #155 or #233. Project status tracks active execution of this slice, with phase and priority unchanged.
itlezy commentedon Nov 15, 2026
The scoped restore patch is pushed at Rust
4e6e0b86; source review accepted it for testing. Rust execution remains pending the shared validation slot. Allocation and remaining boundaries are merged in emulebb/emulebb-tooling#190.Independent review also confirmed a separate existing first-bind ordering gap:
settle_parked_peer_creditreleases the flush gate beforepin_verified_secure_ident, while producers can still add parked deltas. A concurrent pre-bind delta can therefore be applied after the first-key wipe. Fixing that requires a coherent linearization of additions, drains and bind/wipe, plus deterministic interleaving and SQL-failure coverage. Extending only the flush gate does not cover additions. This remains outside the narrow restore patch and is not claimed fixed.The analogous absolute-seed helper currently has only test callers; no production race is established for that path. The broad persistence owner remains unimplemented. Current schema is 69; the older schema-68 references identify historical review baselines.
itlezy commentedon Nov 15, 2026
Read-only follow-up review identified a candidate for a separate first-bind repair: existing flush gate, one metadata transaction, then a brief same-peer ledger handoff while the connection is owned. Check the authoritative key before detaching; reject a mismatched key without mutations. Concurrent adds must remain free of SQL/flush-gate waits. Combine the detached totals with durable state and apply stock first-bind zero versus 1/1 wiping in that transaction. Roll back and merge detached credit before releasing connection ownership on failure.
This is a proposal, not an allocated or implemented extension to the current restore slice. Before implementation, resolve same-key settlement/completion semantics, rollback-failure behavior, and verify every production identity reader uses the shared connection. Current read-through scoring reads SQL and parked totals separately; coherent first-bind snapshots also need one connection-ownership interval. No inspected production path currently nests ledger ownership around connection acquisition.
Required proof includes controlled additions, competing flush/bind/readers, mismatched-key rejection, SQL and commit failures, exact restoration, retry/reopen and progress of additions during blocked SQL. A later-bucket sweep after SQL returns is unsafe. The existing batch-flush drain-before-connection snapshot gap remains separate; this proposal does not claim to fix all read-through races. No schema or generation registry is currently justified by this review.
itlezy commentedon Nov 15, 2026
Delivered failed parked-peer-credit settlement preservation in PR #335, merged at 0840e90: restore pending deltas after SQL failure, merge additions without overwrite, and retry exactly once. The broader persistence actor/friend CRUD and separate first-bind/add/flush boundary remain open. Durable evidence: https://github.com/emulebb/emulebb-tooling/blob/main/docs/products/emulebb-rust/active/items/RUST-REF-016.md
The final pre-merge source c6b9cef passed the six-stage full local owner gate and all applicable hosted checks. Android local proof is compile/link; public APK emulator evidence is separately source-bound in the Android item. Fresh actual-main workflows are tracked separately. Tooling PR #191 merged the final records.