bug(podman): watch initialization hangs with more than 256 existing sandbox snapshots #4717

Summary

The Podman driver's start_watch() can deadlock during initial state synchronization when more than 256 existing workload containers produce valid sandbox snapshots.

The initial snapshot loop awaits sends into a bounded Tokio channel with capacity 256 before returning the receiver to its caller. Once 256 messages have been enqueued, send 257 waits for a receiver that the caller cannot yet access.

This is supported by source inspection and a minimal reproduction using real Tokio. An end-to-end OpenShell/Podman reproduction has not yet been run.

Source and failure sequence

Source: watcher.rs on main, inspected October 19, 2026. The same pattern is present in v0.1.4.

1. start_watch() creates mpsc::channel::<Result<WatchSandboxesEvent, ComputeDriverError>>(256).
2. It subscribes to Podman events and lists existing managed workload containers.
3. The initial inventory loop calls tx.send(...).await once per eligible sandbox snapshot.
4. The 257th send waits for channel capacity.
5. ReceiverStream::new(rx) is returned only after that loop completes, so the consumer cannot drain the channel.

The threshold concerns emitted initial snapshots, not the number of raw Podman containers or a 256-sandbox limit on an already-established watch.

Minimal Tokio reproduction

Executed locally with Rust 1.99.1, Tokio 1.53.3, and four Tokio worker threads. This isolates the channel ordering; it does not run OpenShell or Podman.

Cargo.toml:

[package]
name = "watch-channel-repro"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "=1.53.3", features = ["macros", "rt-multi-thread", "sync", "time"] }

The essential reproduction is:

use tokio::{sync::mpsc, time::{timeout, Duration}};

async fn start_watch(n: usize) -> mpsc::Receiver<usize> {
    let (tx, rx) = mpsc::channel(256);
    for sandbox in 0..n {
        tx.send(sandbox).await.unwrap();
    }
    rx
}

#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() {
    for n in [255, 256, 257, 1000] {
        let result = timeout(Duration::from_millis(300), start_watch(n)).await;
        println!("n={n}: returned_receiver={}", result.is_ok());
        assert_eq!(result.is_ok(), n <= 256);
    }
}

In the executed harness, an atomic counter after each successful send additionally confirmed that both failing cases completed exactly 256 sends and timed out waiting on send 257.

The concurrent-producer variant retained channel capacity 256, spawned the sending loop, returned the receiver immediately, and drained all messages in the caller.

Expected behavior and impact

Watch initialization should complete with an inventory larger than the output channel capacity.

A pool may grow beyond 256 eligible sandboxes while an established watch is consuming events normally, then encounter this problem on gateway/watch startup or reconnection when the existing inventory must be replayed. Existing workload processes need not stop, but the affected watch cannot finish initialization.

Suggested fix and regression coverage

Allow the initial snapshot producer and receiver consumer to run concurrently, while preserving subscription-before-list ordering and initial-snapshot-before-live-event delivery. For example, move snapshot production and subsequent live-event forwarding into a producer task, then return the receiver promptly. Preserve cancellation, receiver-drop, and error handling.

Increasing channel capacity only moves the threshold.

A regression test should initialize a watch against more than 256 valid existing sandbox records and verify, under a timeout, that the watch returns and the complete initial inventory can be consumed.