Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,11 @@ To maintain a clear separation between fast, offline unit tests and containerize
### 20. Explicit Top-Level Class Imports
- Always use top-level `import` statements at the top of Java files instead of writing fully qualified package class names inline in method signatures or method bodies (e.g. add `import me.desair.tus.server.util.Utils;` at the top of the file and call `Utils.interruptStream(...)` instead of writing `me.desair.tus.server.util.Utils.interruptStream(...)`).

### 21. Single Responsibility Principle & Constructor Simplicity
- Every class MUST have a single, well-defined main purpose (Single Responsibility Principle).
- Do NOT mix data serialization models (DTOs / JSON metadata objects) with active process or lifecycle management components (such as lock handles with scheduled thread executors or storage clients).
- Keep constructors focused and minimal (typically 1 or 2 constructors per class). Classes requiring data models MUST accept the dedicated data object (e.g., `LeaseData`) in their constructor rather than defining multiple telescoping or metadata-only constructor overloads.

## IETF Resumable Uploads for HTTP (RUFH) Spec Maintenance & Update Playbook

### 1. Spec Diff Review
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ All notable changes to this project will be documented in this file.

### New

- **NFS- & SMB-Safe Lease Locking (`LeaseFileLockingService`)**: Added distributed, container-safe filesystem locking using atomic directory creation (`mkdir`) and TTL-based JSON lease files with background heartbeat renewal. Operates reliably across multi-server replicas on NFS (v3/v4), AWS EFS, Azure Files, Windows SMB/CIFS, and local disks without requiring Redis, ZooKeeper, or OS-level `FileLock` daemons. Comprehensive guide and legacy opt-out instructions available in `docs/DISK_BASED_LOCKING.md`.
- **NFS- & SMB-Safe Lease Locking (`LeaseFileLockingService` & `LeaseFileMutex`)**: Added distributed, container-safe filesystem locking using atomic sibling mutex directories (`<UploadId>.mutex/`), in-place expired lock takeover, and TTL-based JSON lease files with background heartbeat renewal and ownership fencing. Operates reliably across multi-server replicas on NFS (v3/v4), AWS EFS, Azure Files, Windows SMB/CIFS, and local disks without requiring Redis, ZooKeeper, or OS-level `FileLock` daemons. Comprehensive guide and legacy opt-out instructions available in `docs/DISK_BASED_LOCKING.md`.
- **S3-Compatible Storage & Distributed Locking**: Added native S3 storage support via `S3StorageService` (MinIO SDK), distributed locking via `S3LockingService` (S3 conditional writes with TTL leases and interrupt signals for multi-replica container deployments), S3-native concatenation via `S3ConcatenationService`, and complete documentation in `docs/S3_STORAGE.md`.
- **Azure Blob Storage & Distributed Leases**: Added native Azure Blob Storage support via `AzureBlobStorageService` (Block Blob staging with streaming appends, sub-threshold buffering, truncation, and deduplication), distributed locking via `AzureBlobLockingService` (Azure Blob Leases with auto-renewal, JVM interruption, cross-replica `.stop` signals, and clean shutdown), zero-copy server-side concatenation via `AzureBlobConcatenationService` (`stageBlockFromUrl`), and comprehensive documentation in `docs/AZURE_BLOB_STORAGE.md`.
- **IETF Resumable Uploads for HTTP (RUFH) Protocol**: Implemented full support for the official IETF Resumable Uploads for HTTP specification (`draft-ietf-httpbis-resumable-upload-12`).
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ You can add the latest stable version of this library to your application using
<dependency>
<groupId>me.desair.tus</groupId>
<artifactId>tus-java-server</artifactId>
<version>2.0.0-SNAPSHOT</version>
<version>2.0.0</version>
</dependency>
```

Expand Down Expand Up @@ -117,7 +117,7 @@ Besides the [core protocol](https://tus.io/protocols/resumable-upload.html#core-
The first step is to create a `TusFileUploadService` object using its constructor. You can make this object available as a (Spring bean) singleton or create a new instance for each request. After creating the object, you can configure it using the following methods:

* `withUploadUri(String)`: Set the relative path (e.g. `/files/upload`) or absolute base URL (e.g. `https://upload.example.com/files/upload`) under which the main tus upload endpoint will be made available. When configured with an absolute URL, the `Location` header returned upon upload creation (201 Created, 200 OK, or 104 Interim Response) will contain the full URL. Optionally, this URI may contain regex parameters in order to support endpoints that contain URL parameters, for example `/users/[0-9]+/files/upload` or `https://upload.example.com/users/[0-9]+/files/upload`.
* `withSupportedProtocolVersions(ProtocolVersion)`: Configure supported protocol versions (`ProtocolVersion.AUTO` for automatic header-based detection, `ProtocolVersion.TUS_1_0_0` for Tus 1.0.0 only, or `ProtocolVersion.IETF` for IETF Resumable Uploads only).
* `withSupportedProtocolVersions(ProtocolVersion)`: Configure supported protocol versions (`ProtocolVersion.AUTO` for automatic header-based detection, `ProtocolVersion.TUS_1_0_0` for Tus 1.0.0 only, or `ProtocolVersion.RUFH` for IETF Resumable Uploads only).
* `withMaxUploadSize(Long)`: Specify the maximum number of bytes that can be uploaded per upload. If you don't call this method, the maximum number of bytes is `Long.MAX_VALUE`.
* `withStoragePath(String)`: If you're using the default file system-based storage service, you can use this method to specify the path where to store the uploaded bytes and upload information.
* `withMaxLockRetries(int)`: Specify the maximum number of retries the service will attempt to acquire an upload lock before failing with an `UploadAlreadyLockedException` during lock contention resolution (e.g. for `HEAD` or `DELETE` requests). Default is `40` retries (with a 200ms sleep between retries, resulting in an 8.0-second retry budget).
Expand Down Expand Up @@ -181,7 +181,7 @@ This server implementation has been tested with:

For detailed instructions on running our native conformity test suite and interpreting results, see the **[Conformity Testing Guide (docs/CONFORMITY_TESTING.md)](docs/CONFORMITY_TESTING.md)**.

This repository also contains comprehensive automated integration test suites (`ITTusFileUploadService`, `IetfProtocolCreationTest`, `IetfProtocolAppendTest`, `IetfProtocolHeadTest`, `IetfProtocolCancellationTest`) validating both protocol specifications.
This repository also contains comprehensive automated integration test suites (`ITTusFileUploadService`, `RufhProtocolCreationTest`, `RufhProtocolAppendTest`, `RufhProtocolHeadTest`, `RufhProtocolCancellationTest`) validating both protocol specifications.

## Versioning
This artifact follows `MAJOR.MINOR.PATCH` semantic versioning. Version `2.0.0` introduces major dual-protocol support for both Tus 1.0.0 and the IETF Resumable Uploads for HTTP specification (`draft-ietf-httpbis-resumable-upload`). Version `1.0.0-3.3` was the last Tus protocol-only version.
Expand Down
58 changes: 28 additions & 30 deletions docs/DISK_BASED_LOCKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,20 @@ The legacy `DiskLockingService` relies on OS kernel-level file locks (`java.nio.
- **Ungraceful Crashes**: Pod crashes (`kill -9`, OOM killer, node eviction) leave locks stuck in NFS server state for minutes or indefinitely.
- **Cross-Pod Coordination**: Kernel file locks are tracked in OS memory and do not coordinate cleanly across multi-replica container clusters.

### The Solution: Application-Level Lease Directories
`LeaseFileLockingService` replaces OS kernel locks with atomic directory creation (`mkdir`) and JSON lease files with background heartbeat renewal. This provides **zero external dependencies** (no Redis, ZooKeeper, or etcd cluster required) and works seamlessly on both local disks and distributed network shares.
### The Solution: Application-Level Lease Directories & Sibling Mutexes
`LeaseFileLockingService` replaces OS kernel locks with atomic sibling mutex directory creation (`mkdir`) and JSON lease files with background heartbeat renewal. This provides **zero external dependencies** (no Redis, ZooKeeper, or etcd cluster required) and works seamlessly on both local disks and distributed network shares.

---

## 2. Lock Directory Layout & Mechanics

Locks are structured as a dedicated directory containing a JSON lease metadata file:
Locks are structured as a dedicated directory containing a JSON lease metadata file, synchronized via a transient sibling mutex directory:

```
<storagePath>/locks/
├── <UploadId>.lock/ # Dedicated lock directory (Atomic existence primitive)
├── <UploadId>.lock/ # Dedicated lock directory (contains lease.json)
│ └── lease.json # JSON lease metadata (holderId, expiresAt, acquiredAt)
├── <UploadId>.mutex/ # Transient sibling atomic mutex directory (age <= 5s)
└── <UploadId>.stop # Empty signal file for lock contention interruption
```

Expand All @@ -45,40 +46,37 @@ Locks are structured as a dedicated directory containing a JSON lease metadata f
```

### Architectural Rationale:
1. **Universal Atomic Directory Staging & Renames**: Rather than creating an empty directory directly at `<UploadId>.lock` and subsequently writing metadata into it (which creates a window where concurrent nodes observe an empty, uninitialized directory), new locks are staged in a temporary sibling directory (`<UploadId>.lock.stage.<uuid>`) with `lease.json` pre-written, then moved atomically into place (`Files.move` with `StandardCopyOption.ATOMIC_MOVE`). Directory moves map directly to atomic server-side RPCs on both POSIX NFS (`rename(2)`) and Windows SMB (`SetFileInformationByHandle`), ensuring `<UploadId>.lock` is born on disk 100% complete and valid.
2. **Clean Encapsulation**: Placing `lease.json` inside `<UploadId>.lock/` prevents metadata clutter and guarantees that lease updates and watchdog renewals are scoped directly to the lock entity.
3. **Atomic Eviction & Move Isolation**: Stale lock cleanup isolates the target directory by atomically renaming it (`StandardCopyOption.ATOMIC_MOVE` to `.evicting.<uuid>`) before inspecting and deleting its contents. This isolates expired state and allows post-move rollback verification, preventing race collisions between multiple recovering nodes.
1. **Sibling Mutex Isolation (`<UploadId>.mutex/`)**: All state-modifying operations (acquisition, in-place takeover, release, and cleanup) acquire `<UploadId>.mutex/` via atomic `Files.createDirectory`. Because the mutex is a sibling of `<UploadId>.lock/`, the lock directory contains only `lease.json`, avoiding nested directory deletion races (`DirectoryNotEmptyException`) and Windows handle locking conflicts.
2. **In-Place Takeover (Zero Directory Moves)**: Rather than moving or displacing the lock directory during eviction (which creates a TOCTOU hole where the directory temporarily vanishes from disk), expired locks are updated in place directly inside `<UploadId>.lock/` under mutex protection.
3. **Fencing & Ownership Verification**: Both `lock.close()` and heartbeat renewals verify that `holderId` in `lease.json` matches the current holder before modifying or deleting files, ensuring a paused node never corrupts a successor's active lease.
4. **5-Second Crash Recovery**: Stale mutex directories left behind by crashed nodes are detected via `now - mtime >= 5000ms`, cleanly recovered, and retried.

---

## 3. Distributed Concurrency & Contention Resolution

### 1. Lock Acquisition Flow (Atomic Directory Staging)
To ensure that an observing process never encounters an empty or partially written lock directory, lock creation uses **Atomic Directory Staging**:
### 1. Lock Acquisition Flow (`tryAcquireLock`)
To acquire or take over a lock:
1. Extract `UploadId` from the request URI.
2. Verify that `<storagePath>/locks/<UploadId>.lock` does not already exist. If it exists:
- If `lease.json` is unexpired: Lock is actively held on another replica $\rightarrow$ throw `UploadAlreadyLockedException`.
- If `lease.json` is expired: Holder crashed $\rightarrow$ proceed to **Safe Atomic Eviction** and retry acquisition.
3. Create a unique temporary staging directory: `<storagePath>/locks/<UploadId>.lock.stage.<uuid>`.
4. Write the complete `lease.json` file inside the staging directory.
5. Execute `Files.move(stageDir, lockDirPath, StandardCopyOption.ATOMIC_MOVE)`.
- **Success**: The lock directory appears on disk atomically with a valid, fully populated `lease.json` already inside it. Start the background heartbeat daemon (renews every $\text{leaseDuration} / 3$) and return `LeaseFileUploadLock`.
- **Collision (Already Exists)**: `Files.move` fails because another node acquired the lock in the interim. Clean up `stageDir` and throw `UploadAlreadyLockedException`.

### 2. TOCTOU Mitigation in Expired Lock Eviction (Post-Move Verification & Rollback)
When multiple cluster nodes concurrently discover an expired lock left behind by a crashed pod, a **Time-of-Check to Time-of-Use (TOCTOU)** race condition can arise:
1. **Time of Check (TOC)**: Node A and Node B both inspect `<UploadId>.lock` and observe that its lease has expired.
2. **Node A Wins**: Node A renames the expired directory to `.evicting.<uuid-a>`, deletes it, and stages/moves a brand-new active lock.
3. **Time of Use (TOU) Hazard**: Node B (having verified expiration in Step 1) executes eviction on Node A's **active** directory. Without post-move verification, Node B destroys Node A's directory and acquires a second lock handle, causing dual ownership.

**The Solution: Post-Move Verification & Rollback**:
- When Node B isolates the directory via `Files.move(lockDirPath, evictPath, ATOMIC_MOVE)`, it immediately re-inspects `evictPath` post-move.
- If `evictPath` contains an active lease (created by Node A right before Node B's move), Node B recognizes that it lost the race.
- Node B immediately rolls back the move via `Files.move(evictPath, lockDirPath, ATOMIC_MOVE)` and aborts eviction.
- Exactly one node wins the eviction and acquisition, preserving single-owner lock exclusivity.
2. Acquire sibling mutex `<storagePath>/locks/<UploadId>.mutex/` via `Files.createDirectory`.
- **Collision (Already Exists)**: If `mtime` is $< 5$s old, another live node is modifying the lock $\rightarrow$ throw `UploadAlreadyLockedException`. If $\ge 5$s old, clean up stale mutex and retry.
3. Under mutex protection:
- If `<UploadId>.lock/lease.json` exists and is **unexpired**: lock is actively held on another replica $\rightarrow$ throw `UploadAlreadyLockedException`.
- If `<UploadId>.lock/` does not exist: create directory.
- If `lease.json` is missing or expired: write updated `lease.json` via a temporary file and atomically rename it into place (`Files.move` with `ATOMIC_MOVE, REPLACE_EXISTING`). This prevents concurrent read-only queries (like `isLocked()`) from ever observing an empty (0-byte) or partially written JSON file during disk flushes.
- Start background heartbeat daemon (renews every $\text{leaseDuration} / 3$) and return `LeaseFileUploadLock`.
4. Release `<UploadId>.mutex/`.

### 2. Lock Release Flow (`lock.close()`)
When an active upload completes or is aborted:
1. Stop background heartbeat daemon.
2. Acquire sibling mutex `<UploadId>.mutex/`.
3. Read `lease.json` and verify `holderId == this.holderId`.
4. If ownership matches: delete `lease.json` and delete `<UploadId>.lock/`.
5. Release `<UploadId>.mutex/`.

### 3. Heartbeat Lease Auto-Renewal
Active streaming uploads periodically renew their lease by updating `expiresAt` in `lease.json` every $\text{leaseDuration} / 3$ (default: every 10 seconds for a 30s lease). When the request completes, `lock.close()` stops the daemon and removes the lock directory.
Active streaming uploads periodically renew their lease by updating `expiresAt` in `lease.json` every $\text{leaseDuration} / 3$ (default: every 10 seconds for a 30s lease). The renewal verifies `holderId` ownership under the sibling mutex to ensure it aborts if the lease was taken over after a long pause.

### 4. Lock Contention & `.stop` Signal Files
When a client sends a `HEAD` or `DELETE` request to resume or cancel an upload while a stalled `PATCH` stream holds the lock:
Expand Down
14 changes: 7 additions & 7 deletions docs/LOCKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ All locking mechanisms in `tus-java-server` implement the `UploadLockingService`
### `UploadLockingService` Interface

```java
public interface UploadLockingService extends Closeable {
public interface UploadLockingService {

// Acquires an exclusive lock on an upload resource
UploadLock lockUploadByUri(String requestUri) throws TusException, IOException;
Expand All @@ -37,17 +37,17 @@ public interface UploadLockingService extends Closeable {
// Cleans up stale or expired locks
void cleanupStaleLocks() throws IOException;

// Injects the UploadIdFactory instance used to parse upload IDs from request URIs
void setIdFactory(UploadIdFactory idFactory);

// Registers the active request input stream so it can be interrupted cleanly
default void registerInputStream(String requestUri, InputStream inputStream) {}

// Requests that any active lock for the URI be released
default void requestLockRelease(String requestUri) {}

// Injects the UploadIdFactory instance used to parse upload IDs from request URIs
default void setIdFactory(UploadIdFactory idFactory) {}

// Injects the upload expiration period in milliseconds
default void setUploadExpirationPeriod(Long expirationPeriod) {}
// Closes resources, interrupts in-flight streams, and shuts down background daemon threads
default void close() throws IOException {}
}
```

Expand Down Expand Up @@ -84,7 +84,7 @@ public interface UploadLock extends Closeable {

| Storage Backend / Environment | Locking Service Class | Key Characteristics & Architecture | Documentation File |
|---|---|---|---|
| **Disk & Network Filesystems (Default)** | `LeaseFileLockingService` | Atomic directory staging & renames (`mkdir`/`rename`), TTL-based JSON lease files with heartbeat renewal, TOCTOU-safe eviction with rollback, and `.stop` signal files. Fully safe on NFSv3/v4, AWS EFS, SMB/CIFS, Kubernetes containers, and local disks. | [`docs/DISK_BASED_LOCKING.md`](file:///Users/tom/projects/tus-java-server/docs/DISK_BASED_LOCKING.md) |
| **Disk & Network Filesystems (Default)** | `LeaseFileLockingService` | Atomic sibling mutex directory (`<UploadId>.mutex/`), in-place expired lock takeover, TTL-based JSON lease files with heartbeat renewal, ownership fencing, and `.stop` signal files. Fully safe on NFSv3/v4, AWS EFS, SMB/CIFS, Kubernetes containers, and local disks. | [`docs/DISK_BASED_LOCKING.md`](file:///Users/tom/projects/tus-java-server/docs/DISK_BASED_LOCKING.md) |
| **Local File System (Legacy Opt-Out)** | `DiskLockingService` | OS kernel-level exclusive POSIX `FileLock` (`fcntl`) with JVM shutdown hooks and `.stop` signal files. Best for single-node deployments on local disk. | [`docs/DISK_BASED_LOCKING.md`](file:///Users/tom/projects/tus-java-server/docs/DISK_BASED_LOCKING.md) |
| **Amazon S3 / S3-Compatible** | `S3LockingService` | S3 object-backed TTL lease objects (`.lock`), conditional writes (`If-None-Match: *`), jittered read-after-write verification, heartbeat renewal, and cross-pod `.stop` signal object polling watchdog. | [`docs/S3_STORAGE.md`](file:///Users/tom/projects/tus-java-server/docs/S3_STORAGE.md) |
| **Azure Blob Storage** | `AzureBlobLockingService` | Native Azure Blob Storage exclusive 30-second leases (`BlobLeaseClient`), background daemon renewal, and `.stop` signal blob polling watchdog. | [`docs/AZURE_BLOB_STORAGE.md`](file:///Users/tom/projects/tus-java-server/docs/AZURE_BLOB_STORAGE.md) |
Expand Down
2 changes: 1 addition & 1 deletion docs/MIGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import me.desair.tus.server.TusFileUploadService;

TusFileUploadService tusService = new TusFileUploadService()
.withUploadUri("/files")
// Enabled by default: AUTO mode detects TUS_1_0_0 vs IETF per request
// Enabled by default: AUTO mode detects TUS_1_0_0 vs RUFH per request
.withSupportedProtocolVersions(ProtocolVersion.AUTO);
```

Expand Down
Loading
Loading