[Feature] Normalize cached link ownership and resource lifetime

[nostalume]

Please confirm the following

• I have read and agree to AGPL-3.0 Section 15. The program is provided "as is" without any warranties; you bear all risks of using it.
• I have read and agree to AGPL-3.0 Section 16. The copyright holders and distributors are not liable for any damages resulting from the use or inability to use the program.
• I confirm my description is clear, polite, helps developers quickly locate the issue, and complies with community rules.
• I have read the OpenList documentation.
• I confirm there are no duplicate issues or discussions.
• I believe this issue must be handled by OpenList and not by a third party.
• I confirm this feature has not been implemented yet.
• I confirm this feature is reasonable and has general demand, not just my personal need.
• I have not read these checkboxes and therefore I just ticked them all, Please close this issue.

Feature Description

OpenList's download path currently represents several different responsibilities with model.Link:

• a URL or RangeReader transport description;
• request headers, content length, and partitioning hints;
• cache expiration policy;
• shared SyncClosers reference state; and
• the close handle returned to one caller.

The same value is constructed by drivers, cached and reference-acquired by internal/op, normalized by internal/fs, converted to readers by internal/stream, and finally adopted by HTTP, WebDAV, S3, FTP, MCP, transfer tasks, and wrapper drivers. Ordinary links and archive extraction currently repeat the cache-expiration, reference-acquisition, and retry protocol. GetRangeReaderFromLink returns the reader capability separately from the link whose resources keep it valid, leaving adapters to reconstruct close ownership.

This has a large compatibility surface: model.Link appears in 129 Go files and 89 driver methods implement Link. It should therefore be treated as a multi-PR migration, not a single type replacement. Recent focused fixes illustrate the boundary risk: #3090 preserves cancellation failures in partitioned downloads, and #3095 makes S3 ranged response bodies own both the opened reader and link reference. #3082's SFTP throughput investigation is a related performance constraint that the migration must not regress.

The goal is one explicit owner for each acquired download resource, while preserving current routes, protocol responses, driver behavior, cache keys and expiration, cancellation, and redirect/proxy policy. This issue does not propose a user-facing API or configuration change.

Suggested Solution

Use an incremental roadmap behind the existing driver interface:

1. Characterize direct/partitioned URL and RangeReader paths, expiring and reference-required cache reuse, archive extraction, redirects, cancellation, and close order through observable tests.
2. Remove internal application packages' dependency on server/common for request-origin calculation. Admit request origin once at the delivery boundary and preserve generated URL behavior.
3. Introduce one internal acquired-link lifetime behind op.Link without changing all drivers at once. A migrated caller must have exactly one close obligation; do not retain parallel old/new ownership conventions.
4. Make ordinary link caching and archive extraction share the acquisition, expiration, reference, and retry policy, then delete the duplicate protocol.
5. Make range-reader adoption return a resource that owns both the opened body and acquired link in the required close order. Migrate HTTP, WebDAV, S3, FTP, MCP, transfer tasks, and wrapper drivers as separate reviewable slices.
6. After every consumer migrates, reassess and remove cache/lifetime fields from model.Link only where no compatibility consumer remains.

Each implementation PR should be independently revertible and must include focused behavior, failure/cancellation, close-count, and Linux race evidence. Driver-facing interfaces should remain compatible until the final migration gate. Performance validation must include the SFTP/high-latency case from #3082 where applicable.

We plan to submit future PRs in this dependency order rather than combine the migration into one large diff. The design should be reopened if a driver requires a genuinely different lifetime, if an adapter cannot adopt the common acquired resource without changing protocol behavior, or if the shared acquisition policy regresses cache freshness, cancellation, redirect policy, or throughput.

Relationship to current work

• fix(net): preserve cancellation errors during partitioned downloads #3090 is an independent cancellation-correctness fix. It defines failure and concurrency behavior that later lifecycle PRs must preserve; future lifecycle work should start after it lands or its disposition is otherwise settled, rather than stack on its open branch.
• fix(s3): close ranged response bodies #3095 is an independent S3 body/link ownership fix. It defines the close contract that later range-reader adoption must preserve; future lifecycle work should start after it lands or carry an explicitly equivalent contract.
• Draft chore(proxy): add download behavior tests #3089 is not intended to merge alone. Its durable proxy behavior tests are inputs to the existing local Stage 1 ranged-proxy reduction, which itself waits for fix(net): preserve cancellation errors during partitioned downloads #3090's disposition. Those tests may become roadmap characterization evidence without making the roadmap depend on a test-only PR.
• fix(aliyundrive): limit callback concurrency #3071 is independent callback-admission work. It intersects fix(s3): close ranged response bodies #3095 in S3 regression coverage but is not a roadmap predecessor.
• refactor(offline): centralize native tool setup #3096 is an independent offline-tool authority reduction and has no code or merge dependency on this roadmap.
• refactor(context): centralize request origin access #3100 implements roadmap item 2 by moving request-origin context decoding to internal/conf and removing the eight origin-only internal dependencies on server/common. It targets main directly and has no code or merge dependency on fix(net): preserve cancellation errors during partitioned downloads #3090, fix(s3): close ranged response bodies #3095, refactor(offline): centralize native tool setup #3096, or later lifecycle work.
• fix(op): enforce link cache lifecycle policy #3101 is the first bounded lifecycle-policy slice across roadmap items 3-4. It admits ordinary and archive links through one policy, rejects TTL plus owned-resource conflicts, and prevents wrapper clones from inheriting source TTL. It is based directly on main@4580c4db, is independent of fix(net): preserve cancellation errors during partitioned downloads #3090/fix(s3): close ranged response bodies #3095/refactor(context): centralize request origin access #3100, and deliberately does not introduce the still-unsettled acquired-resource carrier.

These are behavioral and verification relationships, not a stacked-PR chain. Roadmap implementation PRs should branch from refreshed main after the relevant predecessors are settled, remain independently revertible, and name any contract evidence they reuse.

Delivery policy: item 1's characterization evidence travels with the first PR that changes its owning boundary rather than becoming a standalone test PR. Item 2 may be developed on an independent, main-based fork branch, but should be submitted upstream only if its actual diff removes all eight origin-only internal → server/common dependencies without adding a package, facade, configuration field, or behavior change. Otherwise retain the incumbent or fold the boundary change into the first lifecycle PR that genuinely needs it. Dependent later work may proceed on local child branches or worktrees, but cross-fork branches are not proposed as a public stacked-review chain.

Additional Information

Current module evidence at main@4580c4db:

• internal/model: 1,032 production lines, imported by 371 Go files, no package tests;
• internal/op: 3,163 production lines, imported by 244 Go files;
• internal/fs: 1,180 production lines, imported by 33 Go files;
• model.Link: referenced by 129 Go files;
• op.Link: 14 call sites; fs.Link: 11 call sites; GetRangeReaderFromLink: 11 call sites;
• eleven internal files import packages under server/, including five internal/fs files.
• acquisition analysis identifies TTL-shared, reference-shared, and close-invalidated legacy modes. Follow-up tracing found the conflicting state is reachable when Link.Clone copies TTL while owning its source; fix(op): enforce link cache lifecycle policy #3101 makes internal/op the single admission owner, rejects TTL plus owned resources, and stops clones from propagating cache policy. The acquired-resource carrier remains a later decision requiring wrapper-transfer and close-order evidence.
• item 2 is published as refactor(context): centralize request origin access #3100: it removes all eight origin-only internal dependencies on server/common, retains a narrow compatibility bridge for existing server/driver consumers, and passes focused behavior and Linux race checks.

Repository issue searches for link lifecycle, RangeReader close, link cache reference closer, and server common dependency found no existing roadmap. #3082 is related to SFTP streaming performance but does not cover ownership migration.

AI Generated Content

• I used AI tools to generate this content
• I did not use AI tools to generate this content

AI model used

Codex (GPT-5)

[nostalume]

Development status — 2026-09-20

This is a progress report, not a claim that the roadmap has landed upstream.
PRs #3071, #3090, #3095, #3100, and #3101 are still open and unmerged.

Transfer-lifecycle roadmap

• The remaining roadmap items have been implemented and exercised on the private
  refactor/architecture-integration
  branch, based on the current fix(aliyundrive): limit callback concurrency #3071 head 6d4dd259.
• The transfer-lifecycle portion has one acquired-resource owner after admission,
  shared ordinary/archive acquisition policy, consumer-owned response bodies, and
  no parallel legacy lifetime convention. Its production delta is exactly neutral
  (+121/-121) rather than an additive facade.
• Focused behavior, cancellation/failure, close-count, race, compile, removed-symbol,
  and integration checks pass on that private line. Characterization evidence was
  kept with the owning changes; no tests-only roadmap PR is planned.
• This private result is review evidence only. It does not change the merge status
  or review ownership of the open PRs above, and it is not a request to merge the
  complete integration branch as one PR.

Boundary with the wider reduction work

Issue #3098 continues to own only transfer acquisition and resource lifetime.
The later namespace/cache/task/search/STRM convergence work is a separate
architecture domain. It consumes the settled lifetime invariants but does not
reopen them.

That namespace domain is implemented privately and is currently at its Phase G
publication-evidence gate. Local configured-storage calibration is green for:

• SQLite DB search and a real local STRM projection;
• single-upload mutation → reconciliation → projection completion;
• 100-object repeated snapshots, 32 concurrent parents, recursive traversal, and
  cached-list non-publication;
• bounded refusal, zero representative-run rejection, direct-sink comparison, and
  race verification.

The provisional queue/worker constants remain unchanged. Phase G is not yet
closed: a configured remote-storage workload and a real Meilisearch run remain.
Those are publication gates for the namespace domain, not prerequisites for the
transfer-lifecycle correctness claim in this issue.

The dependency consequences are:

1. transfer lifetime precedes data-plane scheduling work and the link-resource
   portion of cache work;
2. authorization is a separate topology predecessor for namespace convergence;
3. directory indexing and Driver registration remain independently gated and are
   not being folded into Phase G; and
4. recently found OSS-test, network-observation, and BufferStore race repairs are
   separate tail fixes. They improve verification trustworthiness but do not define
   or precede the architecture roadmap.

No new upstream PR or mainline change is proposed by this status update.

[nostalume]

Phase G calibration follow-up — 2026-09-20

The two configured-backend items described as open in the previous status comment
have now been exercised. This does not change issue #3098's ownership: the result
belongs to the separate namespace-convergence domain and is reported here only to
keep the cross-domain dependency status current.

• The real OpenList WebDav Driver was configured against a disposable HTTP/WebDAV
  endpoint using the same 100-object, upload, concurrent-parent, recursive, cache,
  and STRM fixture.
• A checksum-verified official Meilisearch v1.53.2 process was used with telemetry
  disabled and disposable storage.
• Across five combined WebDav/Meilisearch/STRM runs, all 87 accepted updates
  completed per sink with zero rejection. Maximum sampled retention was 33 keys
  and 453 object references; the existing capacity/worker constants were retained.
• Single-upload completion was 0.361-0.461 s. The 32-parent run completed in
  4.71-5.05 s, with Meilisearch p95 4.39-4.94 s. The combined path also passes the
  race detector and deadline-bounded shutdown.

Calibration exposed an independent, pre-existing sink cost: Meilisearch task
status polling explicitly used a one-second interval although the locked SDK's
default and the older in-source recommendation are 50 ms. A one-line, code-neutral
local branch (perf/meilisearch-task-polling-standalone@72d4cdb8) reproduces the
correction directly from current main; it has not been pushed or proposed as a
PR. In the combined workload it reduced single-upload search completion from about
1.02 s to 0.36-0.46 s and 32-parent search p95 from about 9.32 s to 4.39-4.94 s.
The remaining time is actual Meilisearch indexing work rather than mailbox
admission or the former polling sleep.

The defined Phase G configured-storage evidence gate is therefore complete, with
the qualification that a loopback WebDAV endpoint proves the real Driver and
protocol contracts but is not a WAN/provider SLA. The next namespace-domain step
is a publication-boundary decision for the complete migration, not another
architecture phase. No new upstream PR or mainline change is authorized or made by
this update, and the transfer-lifecycle status in #3098 is unchanged.

[nostalume]

Domain 3 closure follow-up — 2026-09-20

This remains a cross-domain status note only; issue #3098 continues to own transfer acquisition and resource lifetime.

The namespace-convergence publication boundary described in the prior comment is now closed locally. The architecture body remains separate from its verification tails, and no upstream PR, push, or mainline change is authorized by this note.

A compact opt-in repository benchmark now retains the repeatable configured mutation workload (internal/bootstrap/projection_benchmark_test.go, build tag benchmark) instead of the discarded 500-plus-line exploratory harness. Ordinary tests exclude it. Final-state-checked Local/SQLite/STRM and real OpenList WebDav/SQLite/STRM runs, plus a tagged race run, pass on the private integration line.

The benchmark exposed an independent pre-existing DB-search deletion defect: path.Split returned a trailing-slash parent while stored parent paths are normalized, so a renamed node could remain indexed. The one-line code-neutral correction and behavior test are isolated from current main at fix/search-node-delete-path-standalone@637a20b5; it is a correctness prerequisite for the benchmark's rename assertion, not part of namespace ownership or the transfer-lifecycle roadmap.

The resulting delivery decision is: keep that standalone correctness fix and the Meilisearch polling fix independently reviewable; carry the durable benchmark with the eventual complete namespace-convergence change rather than publish a tests-only PR. Domain 3 is therefore closed for local architecture work, while publication remains separately unauthorized.

[nostalume]

Cross-domain architecture anchor — Driver-system linearization (Domain 6)

This comment records the current Driver-system design boundary so later changes
can reference one stable, reviewable specification. It is a cross-domain anchor:
issue #3098 continues to own transfer acquisition and resource lifetime. No code
described here has landed upstream, and this comment does not request a new PR or
change the status of existing PRs.

Goal and invariant

The goal is not to replace small Driver methods with a larger framework. It is to
remove duplicated or negative capability declarations, hidden product
registration, repeated protocol implementations, and implicit resource
ownership, while keeping one forward path for each operation.

Each fact has one owner:

• product inclusion -> one explicit compiled catalog;
• behavior availability -> Go interface satisfaction;
• product policy -> driver.Config;
• provider variation -> family-local data or a narrow strategy;
• instance resources -> storage activation plus Init/Drop;
• Link/body ownership -> the already settled transfer-lifecycle boundary.

The normal operation path is:

resolve instance
  -> admit instance state
  -> admit policy
  -> select capability
  -> invoke once
  -> normalize compatibility result
  -> commit cache/projection effects
  -> return

A helper or abstraction is acceptable only when it deletes an old interpretation
point in the same change. A maximal base Driver, Driver-name switch, second
capability registry, second Link carrier, or compatibility facade beside the old
path is out of scope.

Ordered Driver work

6B — truthful capabilities

Static support is represented positively by interface satisfaction. Static
absence is represented by interface absence, not by a method whose entire body
returns errs.NotImplement or errs.NotSupport. NotImplement remains meaningful
only for a real capability that dynamically requests a documented fallback.

An AST inventory found 41 unconditional NotImplement methods in 16 product
packages whose interface-absence path returns the same sentinel and performs no
Driver-side effect. They occupy 154 physical production lines. This exact set is
the first implementation-ready boundary; it adds no helper, flag, interface, or
compatibility shim.

Two tempting extensions were rejected:

• Four three-method ArchiveReader stub groups remain because removing the
  bundled interface changes direct extract failure from NotImplement to
  DriverExtractNotSupported; splitting the interface would add more contract
  surface to remove only 48 lines.
• Twenty-seven unconditional NotSupport methods remain because interface
  absence currently returns NotImplement; canonicalizing that error is a
  behavior decision, not cleanup.

Capability and policy are deliberately distinct. In particular,
effective upload = Put capability && policy permits: NoUpload cannot replace
interface discovery because at least one Driver has a real Put implementation
while that policy is set.

6C — provider-family composition

Reuse is selected by behavior, not naming:

• use one parameterized implementation when only endpoints/defaults differ;
• use one family-local protocol mechanism plus thin product surfaces when
  request, error, retry, cancellation, and resource laws are identical;
• use a narrow strategy only for one varying step such as authentication or
  signing;
• use explicit delegation for alias/chunk/crypt/strm/virtual-style composite
  Drivers;
• otherwise keep providers separate.

The first bounded family is AList V3/OpenList. Their four production files total
1,309 physical lines; about 601 AList lines have unchanged OpenList counterparts.
The complete semantic differences are limited to product identity and Addition
metadata, scalar-versus-scalar/array role decoding, optional refresh propagation,
and archive-overwrite projection. Login/token, request/retry/error translation,
object conversion, Link behavior, mutation/upload, and the remaining archive flow
are identical.

The private prototype must produce one protocol mechanism and retain both product
names and persisted Addition schemas. It is accepted only if the complete family
deletes at least 350 net production lines with no second protocol implementation.
Partial common-layer extraction is not an integrable result.

6A — explicit product composition

The compiled Driver set still needs one source-level catalog, but distributed
package init mutation and blank-import activation are accidental complexity.
The existing private 6A prototype replaces them with local constructors, one
drivers.All() catalog, and one explicit installation before command execution.
Its implementation is now the dependent catalog layer after 6B/6C.

The order is therefore:

6B exact capability deletion
  -> 6C complete family prototype (accept or discard)
  -> replay 6A explicit catalog
  -> integrated compatibility and net-reduction audit

Resource and initialization boundary

model.Link remains the Driver/API compatibility carrier. The transfer roadmap
already linearizes its ownership:

Driver Link -> internal/op admission/cache -> borrower lease
  -> byte-source factory -> owned body -> terminal consumer -> close/release

Domain 6 must not add another Link wrapper or move cache/lifetime interpretation
back into provider families.

Storage activation is a separate unresolved lifecycle domain. Today
initStorage combines Addition decoding, reference binding, Driver.Init, map
publication, status persistence, and panic recovery, and failed instances may be
published beside ready instances. That is why 29 CheckStatus predicates remain.
A shared helper would retain every admission point and add a relay, so it was
rejected. A future activation design must first settle ready/failed publication,
reference dependency order and cycles, rollback after partial Init, atomic
replacement, and Init/Drop obligations.

Likewise, eight NoUpload gates intentionally run at different early resource
boundaries (multipart parsing, stream caching, task admission, and S3 redirect
selection). Moving all of them to op.Put would reject too late; an admission
token/facade would add more state than it removes. They remain explicit.

Review and delivery rules

• Every accepted slice must remove an authority, false capability, repeated
  mechanism, dependency edge, or ownership convention.
• Production LOC must be negative for 6B, the complete 6C family, 6A, and their
  integration; moving code or adding a parallel abstraction is a discard result.
• Driver names, Addition JSON/defaults, persisted storage records, errors,
  fallback, retry, cancellation, upload progress, and Link/resource ownership
  remain compatible unless separately approved.
• Tests protect shared behavior and the real dialect/failure edges; they do not
  preserve deleted helper or stub structure.
• Stage, branch, commit, and PR boundaries remain independent. The published Driver PRs are open for review, not approved for merge.

This anchor should be reopened only if maintained external source compatibility
requires the negative concrete methods, a provider family proves materially
different lifecycle/error laws, dynamic Driver loading replaces the static
binary model, or actual diffs fail the stated deletion and compatibility gates.

Published Driver PRs (2026-09-25)

• #3133: 6B/6C capability-family contains both truthful-capability deletion and shared AList V3/OpenList protocol; it is open for independent review. No duplicate 6B/6C PR is needed.
• #3134: 6A explicit catalog is open for parallel review, but must merge after refactor(drivers)!: remove false capabilities and share OpenList protocol #3133. Until then, use the catalog-only comparison.
• The cross-domain review map records the full merge order. Both still require human review; title-check success is not full CI.

[nostalume]

Driver-system roadmap status: 6C-1 accepted locally

This is a private implementation update only; it does not publish a branch or open/change an upstream PR.

The ordered Driver-system work now has two accepted local outcomes:

• 6B-1: removed 41 unconditional optional-capability stubs (205 production lines, no additions).
• 6C-1: replaced the duplicated AList V3/OpenList implementations with one internal protocol-family mechanism and thin product shells.

6C-1 preserves the product-specific boundaries recorded in the architecture anchor:

• distinct product names, concrete types, Addition schemas, and persisted JSON;
• AList V3 scalar-or-array role decoding vs. OpenList scalar role decoding;
• AList V3 refresh suppression vs. OpenList opt-in refresh forwarding;
• AList V3 omission vs. OpenList forwarding of archive overwrite;
• unchanged request/login/retry, link/header/cache, mutation/upload, and archive behavior.

Reduction result:

• production: 724 additions / 1,074 deletions = -350 lines;
• whole change including focused tests: 879 additions / 1,101 deletions = -222 lines.

Validation completed:

• focused family behavior and dialect tests;
• affected-package race tests;
• compile check for every Driver package;
• internal/search, internal/op, and internal/fs tests.

The next dependency-ordered task is to replay the frozen 6A explicit catalog prototype on top of 6B/6C, then audit the combined Driver-system reduction and compatibility gates. Link ownership and storage activation remain outside this change.

[nostalume]

Driver-system roadmap status: 6A/6B/6C lineage closed locally

This remains private implementation evidence only; no branch was pushed and no upstream PR was opened or changed.

The explicit catalog prototype was replayed after the accepted 6B and 6C changes and is now accepted locally:

• one explicit, atomically published Driver catalog;
• 93 product constructors, with the product-name set matching the frozen baseline;
• no remaining RegisterDriver calls;
• no blank imports used for Driver activation;
• duplicate names, nil constructors, and nil instances are rejected before publication;
• command startup installs the complete catalog once; focused tests install only their required constructors.

Reduction:

• 6A replay: 649 additions / 733 deletions = -84 lines;
• complete Domain 6 from the pre-Driver-system integration base: 1,528 additions / 2,039 deletions = -511 lines across 124 files.

Validation:

• all Driver packages compile;
• catalog parity and affected integration tests pass;
• internal/op, GitHub Driver, and S3 race tests pass;
• the full test run exposed and corrected one catalog-dependent GitHub test;
• remaining full-suite failures are bounded to existing environment/global-config/flaky cases (external aria2 service, OneDrive test configuration, task cancellation timing).

The Driver-system roadmap has therefore reached its planned closure. Link ownership and storage activation remain separate domains.

[nostalume]

Architecture PR review map (updated 2026-09-25)

This is a cross-domain delivery note for reviewers, not a change to this issue's scope: #3098 continues to own link acquisition and resource lifetime. The Driver-system design anchor remains here. The aim is to delete duplicated interpretation, false capabilities, and implicit lifecycle conventions, not retain a second framework beside the old paths.

All six PRs below are open against upstream main for parallel review. A code dependency restricts merge order, not when review can begin. Each dependent PR description links to its layer-only comparison because GitHub's default upstream-main diff temporarily includes the predecessor. After a predecessor merges, we will update the same successor PR branch, narrow its diff, and rerun the relevant checks rather than submit a replacement PR.

Architecture delivery	Review status and merge order	Layer claim
#3131, Domain 1	Open; independent in this queue	One acquired-link borrowing and terminal-ownership path. This is the delivery directly relevant to #3098.
#3132, Domain 2	Open; independent in this queue	One protocol-neutral path-authorization policy owner.
#3133, Driver family	Open; independent in this queue	Remove false optional capabilities and share the AList V3/OpenList protocol mechanism while retaining product dialects.
#3134, Driver catalog	Open for review; merge after #3133	Replace distributed registration and blank-import activation with an explicit 93-product catalog. Catalog-only comparison.
#3135, Domain 5	Open for review; merge after #3131	Remove native-range-to-synthetic-HTTP wrapping and keep bounded scheduling separate from URL status/retry interpretation. Scheduler-only comparison.
#3136, Domain 3	Open for review; merge after #3132	Converge namespace mutation and bounded search/STRM projections; delete the old hook/queue paths. Namespace-only comparison.

Domain 4 directory indexing remains a no-PR decision pending representative workload evidence. #3136's retained benchmark currently uses baseline Driver blank-import registration; if #3134 lands first, its fixture will be adapted in #3136 before merge. This is a cross-PR integration check, not a semantic dependency between Domain 3 and Driver design.

All six PRs still require human review; mergeability and passing PR-title checks are not approval or full CI. Their descriptions disclose source compatibility, the checks actually run, non-green/unrun full-suite boundaries, and substantial Codex assistance. Human contribution attestations remain with the contributor.