Docs/abot world multi session

docs/en/blog/abot_world_multi_session.md

@@ -0,0 +1,179 @@
+---
+title: "ABot Multi-Session Serving: State Isolation, Compatible Batching, and Multi-GPU Scheduling"
+description: Session-isolated ABot-World serving with compatible batching, capacity-aware admission, and multi-GPU workers.
+date: 2026-08-19
+status: validated
+validated_revision: 52491cb
+hardware: 4 x NVIDIA H100 80 GB HBM3
+tags:
+  - abot-world
+  - multi-session
+  - batched-serving
+  - process-nccl
+  - livekit
+---
+
+# ABot Multi-Session Serving: State Isolation, Compatible Batching, and Multi-GPU Scheduling
+
+An ABot-World interaction is not a request that can be released immediately. Once a user connects, the service keeps
+the prompt embedding, DiT KV cache, scheduler, and video temporal state until the interaction ends. A single-session
+path makes this state easy to manage, but concurrent users normally execute denoising independently: they share model
+weights, not the computation.
+
+The sessions also cannot always be concatenated safely. They may be at different chunk stages, use different control
+latent counts, or refer to different latent-frame positions. Multi-session serving therefore does more than concatenate
+requests: it checks state compatibility, shares one model execution, and returns state and output to the original
+session.
+
+## Design Goals
+
+The implementation follows five boundaries:
+
+1. Keep generation and decode state independent per session.
+2. Batch only sessions with compatible control frames, cache shapes, and RoPE positions.
+3. Make admission depend on actual GPU capacity, not only connection count.
+4. Separate LiveKit control from model workers and support multi-GPU migration.
+5. Preserve B1/eager fallback when batching, memory, or CUDA Graph constraints are not met.
+
+## Architecture Overview
+
+```text
+LiveKit / HTTP control plane
+  | session creation, admission, ownership, control messages
+  v
+ABotWorldLiveKitService
+  | deadline-aware scheduling, compatibility, backpressure
+  v
+Model workers (one per GPU)
+  | collate session state -> one DiT execution -> scatter state
+  v
+Per-session TAEW/VAE decode and output queues
+```
+
+The main ownership boundaries are:
+
+| Area | Owner | Responsibility |
+|---|---|---|
+| Session state | `ABotWorldInteractiveSession` | Prompt, KV cache, scheduler, RNG, and temporal decode state |
+| Single-GPU service | `ABotWorldLiveKitService` | Admission, scheduling, batching, pacing, and lifecycle |
+| Process workers | `process_worker_pool.py` | Model execution outside the control plane |
+| Multi-GPU workers | `nccl_process_worker_pool.py` | One model worker per GPU and fixed NCCL communicator |
+| Cluster routing | `turboserve.py` | Load-aware placement, ownership, and migration decisions |
+
+Model code owns model inputs, cache updates, and decode. Connection management, GPU selection, and cross-worker routing
+remain outside the model.
+
+## Session Isolation
+
+Each `ABotWorldInteractiveSession` owns its prompt embedding, first-frame latent, self/cross KV caches, scheduler and
+RNG state, VAE/TAEW temporal decode state, and chunk/frame counters. An idle session can move retained tensors to CPU
+and later restore them without changing its logical owner.
+
+The batch therefore shares only short-lived model computation. Mutable generation state is never shared: after one batch
+finishes, caches, decode state, and outputs return to their respective sessions.
+
+## Compatible Batching
+
+The scheduler orders ready sessions by their earliest playout deadline. Before forming a batch it validates the control
+latent count, first-versus-continuation stage, absolute-RoPE `next_latent_frame`, and latent/cache shape and dtype.
+
+Compatible sessions form B2 or B3 within a short formation window:
+
+```text
+ready sessions
+      | compatibility check
+      v
+collate prompt / noise / action / KV cache
+      | one denoise_interactive_blocks call
+      v
+scatter cache and latent outputs
+      | per-session decode and queue
+      v
+independent LiveKit delivery
+```
+
+When first chunks and continuations cannot be mixed safely, the scheduler reduces the batch or uses B1. This trades
+some throughput for deterministic cache pointers, position encoding, and frame ordering.
+
+## Multi-GPU Workers and Migration
+
+The parent process owns LiveKit, HTTP admission, and ownership; worker processes own model instances and GPU tensors.
+In `process-nccl` mode, one worker is started per GPU with a fixed NCCL communicator. Clients see one service endpoint
+and never select a GPU.
+
+Migration is a chunk-boundary transaction: quiesce new work, drain output and publisher credit, snapshot state, restore
+it on the target worker, and commit a new route with a monotonically increasing ownership epoch. If the commit fails,
+the source owner remains authoritative instead of leaving two workers with partial state.
+
+## Backpressure and Fallback
+
+Real-time output supports `latest`, which drops the oldest complete block when a queue is full, and `lossless`, which
+blocks only the affected session. CUDA Graph replays are used only for profiled-compatible shapes and state; other inputs
+use eager execution. Failed compatibility checks similarly fall back to a smaller batch or B1.
+
+## Performance Results
+
+### Target compute on one H100
+
+In the LF3 microbenchmark, aggregate throughput increased from 30.98 FPS at B1 to 37.01 FPS at B4, a 19.5% gain. B4
+met an 8 FPS, 1.5-second chunk deadline; larger batches did not. This measures model compute only and excludes
+network, LiveKit, and browser delivery.
+
+### Four-GPU trace
+
+The 30-minute four-GPU trace on 2026-08-19 produced the following results:
+
+| Metric | Result |
+|---|---:|
+| Admission | 295/295 immediate, 0 failed |
+| Real model calls | 5,475, all successful |
+| B1 / B2 / B3 calls | 4,533 / 907 / 35 |
+| B2 chunk-item share | 28.1% |
+| B3 chunk-item share | 1.6% |
+| Fewer calls than all-B1 execution | 977 (15.1%) |
+| Graph fallback | 0 |
+| Demand SLO | 81.8% |
+| Mean active-user FPS | 11.13 |
+| A2F p95 | 4.13 s |
+
+The dispatch histogram represents:
+
+```text
+4533 + 2 × 907 + 3 × 35 = 6452
+```
+
+All-B1 execution would require 6,452 model calls. The trace used 5,475, saving 977 calls. This difference reflects
+model-execution reuse from compatible batching rather than dropping sessions.
+
+## Correctness and Applicability
+
+Tests cover session isolation, 30-block generation, batch ordering, cache scatter, worker lifecycle, and migration
+transactions. The long trace also exercises admission, output queues, and multi-worker dispatch. Every model dispatch in
+the trace returned `ok`, and none of the four workers used Graph fallback.
+
+Mean active-user FPS was 11.13 and demand-SLO attainment was 81.8%. Batching gains depend on control-message arrival
+timing and tensor compatibility; these results primarily show the capacity and compute-reuse benefits of multi-user
+serving.
+
+## Reproduction Record
+
+The experiment record is under:
+
+```text
+results/experiments/abot_4gpu1237_lf3_12fps_publicdemo_b3_f36_graph_30min_20260819T060933Z/
+```
+
+The workload is normalized from a public TurboServe trace and covers admission, batch dispatch, state retention, output
+delivery, and multi-GPU worker behavior.
+
+## Limitations
+
+- Results cover one host with four H100 GPUs;
+- batch formation depends on session arrival timing and input compatibility;
+- migration cost depends on worker topology, memory capacity, and transfer configuration;
+- demand-SLO attainment in this trace is below 100%.
+
+## Related Work
+
+The implementation combines continuous batching, deadline-aware scheduling, and multi-GPU worker pools with
+ABot-World KV caches, TAEW/VAE temporal state, and LiveKit real-time delivery.

docs/en/blog/index.md

@@ -13,6 +13,7 @@ these articles explain the evidence, alternatives, implementation tradeoffs, and
 
 | Date | Article | Status | Validated platform |
 |---|---|---|---|
+| 2026-08-19 | [ABot Multi-Session Serving: State Isolation, Compatible Batching, and Multi-GPU Scheduling](abot_world_multi_session.md) | Validated | 4 x H100 80 GB |
 | 2026-08-19 | [FP8 Sol-Attn: Quantized Sparse Attention for Video DiTs on H100](fp8_sol_attention.md) | Validated | 1 x H100 80 GB |
 | 2026-08-20 | [Agent Dev First Through the MiniMax H3 Integration](agent_dev_first_minimax_h3.md) | Validated | 1, 2, and 4 x H100 80 GB |
 | 2026-08-06 | [CUDA IPC Ulysses: Overlapping Attention Communication on H100](cuda_ipc_ulysses.md) | Validated | 4 x H100 80 GB |

docs/zh/blog/abot_world_multi_session.md

@@ -0,0 +1,176 @@
+---
+title: "ABot 多会话服务：状态隔离、兼容批处理与多卡调度"
+description: 为 ABot-World 引入 session 状态隔离、兼容批处理、容量感知 admission 和多 GPU worker 调度。
+date: 2026-08-19
+status: validated
+validated_revision: 52491cb
+hardware: 4 x NVIDIA H100 80 GB HBM3
+tags:
+  - abot-world
+  - multi-session
+  - batched-serving
+  - process-nccl
+  - livekit
+---
+
+# ABot 多会话服务：状态隔离、兼容批处理与多卡调度
+
+ABot-World 的一次交互并不是一个可以立即释放的请求。用户建立连接后，服务需要持续保存 prompt
+embedding、DiT 的 KV cache、scheduler 以及视频解码的 temporal state，直到这次交互结束。单会话路径很容易
+实现这些状态，但当多个用户同时在线时，每个用户各自执行一次 denoising，GPU 只共享了模型权重，计算却没有
+得到复用。
+
+问题的另一面是，这些 session 并不总是可以直接拼成一个 batch：它们可能处在不同的 chunk 阶段，拥有不同的
+控制帧数量，或者对应不同的 latent frame 位置。multi-session serving 的核心不是“把请求拼起来”，而是先
+确认状态兼容，再共享一次模型执行，最后把状态和输出分别还给原来的 session。
+
+## 设计目标
+
+实现遵循以下边界：
+
+1. session 的生成状态和解码状态始终独立；
+2. 只有控制帧、cache shape 和 RoPE 位置兼容的 session 才进入同一个 batch；
+3. admission 受实际 GPU 容量约束，而不是只由连接数决定；
+4. LiveKit 控制面与模型 worker 解耦，支持多 GPU 和 chunk 边界迁移；
+5. batch 不兼容、显存不足或 CUDA Graph 不适用时，保留 B1/eager fallback。
+
+## 架构总览
+
+```text
+LiveKit / HTTP 控制面
+  | session 创建、admission、ownership、控制消息
+  v
+ABotWorldLiveKitService
+  | deadline-aware scheduler、batch compatibility、backpressure
+  v
+模型 worker（每张 GPU 一个）
+  | collate session state -> 一次 DiT 执行 -> scatter session state
+  v
+每个 session 的 TAEW/VAE decode 与输出队列
+```
+
+主要职责由以下组件承担：
+
+| 范围 | 负责模块 | 职责 |
+|---|---|---|
+| Session 状态 | `ABotWorldInteractiveSession` | prompt、KV cache、scheduler、随机数和 temporal decode state |
+| 单卡服务 | `ABotWorldLiveKitService` | admission、调度、批处理、输出节流和 session 生命周期 |
+| 进程 worker | `process_worker_pool.py` | 将模型执行移出控制面，管理 worker 生命周期 |
+| 多卡 worker | `nccl_process_worker_pool.py` | 每 GPU 一个模型 worker，固定 NCCL communicator |
+| 集群路由 | `turboserve.py` | 负载感知 placement、ownership 和迁移决策 |
+
+模型代码只负责模型输入、cache 更新和输出解码；连接管理、GPU 选择和跨 worker 路由不下沉到模型内部。
+
+## Session 状态隔离
+
+每个 `ABotWorldInteractiveSession` 持有自己的 prompt embedding、首帧 latent、self/cross KV cache、scheduler
+和 RNG state，同时保存 VAE/TAEW temporal decode state 以及 chunk/frame 计数。session 被挂起时，retained
+tensor 可以移到 CPU；恢复时再回到原有的逻辑 owner。
+
+因此，batch 只共享短暂的模型计算，不共享可变的生成状态：一次 batch 执行结束后，cache、解码状态和输出
+仍然回到各自的 session。
+
+## 兼容批处理
+
+调度器按照最早 playout deadline 选择 ready session。进入 batch 前会验证 control latent frame 数、首 chunk
+与 continuation 阶段、absolute RoPE 的 `next_latent_frame`，以及 latent/cache 的 shape 和 dtype。
+
+兼容的 session 在短 formation window 内组成 B2 或 B3：
+
+```text
+ready sessions
+      | compatibility check
+      v
+collate prompt / noise / action / KV cache
+      | one denoise_interactive_blocks call
+      v
+scatter cache and latent outputs
+      | per-session decode and queue
+      v
+independent LiveKit delivery
+```
+
+首 chunk 与 continuation 不能安全混合时，调度器会缩小 batch 或退回 B1。这样做牺牲一部分吞吐，换取
+cache 指针、位置编码和帧顺序的确定性。
+
+## 多 GPU 与 session 迁移
+
+父进程拥有 LiveKit、HTTP admission 和 ownership；模型 worker 拥有模型实例和 GPU tensor。`process-nccl`
+模式为每张 GPU 启动一个 worker，并在 worker 之间建立固定 NCCL communicator。客户端只看到一个服务端点，
+不需要选择 GPU。
+
+迁移是一个 chunk-boundary transaction：先 quiesce session，排空输出和 publisher credit，快照状态，在目标
+worker 恢复，再用递增的 ownership epoch 提交新路由。提交失败时保留 source owner，不会让一半状态同时被两个
+worker 接管。
+
+## 输出背压与 fallback
+
+实时视频输出有两种语义：`latest` 在队列满时丢弃最旧的完整 block，`lossless` 则只阻塞当前 session。
+CUDA Graph 只在输入 shape 和状态满足 profile 时 replay；其他情况使用 eager 路径。batch 兼容性检查失败时，
+同样回退到较小 batch 或 B1。
+
+## 性能结果
+
+### 单 H100 目标端计算
+
+LF3 microbenchmark 中，聚合吞吐从 B1 的 30.98 FPS 增加到 B4 的 37.01 FPS，提升约 19.5%。B4 满足 8 FPS、
+1.5 秒 chunk deadline；继续增加 batch 后无法满足该 deadline。这里测量的是模型计算，不包含网络、LiveKit
+和浏览器交付。
+
+### 四卡 trace
+
+2026-08-19 的四卡、30 分钟 trace 得到以下结果：
+
+| 指标 | 结果 |
+|---|---:|
+| Admission | 295/295 立即分配，0 failed |
+| 真实模型调用 | 5,475，全部成功 |
+| B1 / B2 / B3 调用次数 | 4,533 / 907 / 35 |
+| B2 chunk item 占比 | 28.1% |
+| B3 chunk item 占比 | 1.6% |
+| 相比全部 B1 少的模型调用 | 977 次（15.1%） |
+| Graph fallback | 0 |
+| Demand SLO | 81.8% |
+| 活跃用户平均 FPS | 11.13 |
+| A2F p95 | 4.13 s |
+
+调用直方图对应的 chunk item 数为：
+
+```text
+4533 + 2 × 907 + 3 × 35 = 6452
+```
+
+如果每个 item 都使用 B1，需要 6,452 次模型调用；实际执行 5,475 次，减少 977 次。这个差值直接反映了
+兼容批处理带来的模型执行复用，而不是通过丢弃 session 得到的数字。
+
+## 正确性与适用边界
+
+测试覆盖 session 状态隔离、30-block 生成、batch ordering、cache scatter、worker 生命周期和迁移事务。
+长时间 trace 还验证了 admission、输出队列和多 worker dispatch。实验中四个 worker 均未发生 Graph fallback，
+所有模型 dispatch 都返回 `ok`。
+
+平均活跃用户 FPS 为 11.13，demand SLO 达成率为 81.8%。batch 收益取决于 session 控制消息的到达时间和
+tensor 兼容性，这组结果主要体现了多用户场景下的容量和计算复用收益。
+
+## 复现记录
+
+实验记录位于：
+
+```text
+results/experiments/abot_4gpu1237_lf3_12fps_publicdemo_b3_f36_graph_30min_20260819T060933Z/
+```
+
+workload 使用公开 TurboServe trace 归一化得到，覆盖 admission、batch dispatch、状态保持、输出交付和多
+GPU worker 行为。
+
+## 限制
+
+- 当前结果覆盖单机四张 H100；
+- batch 形成依赖 session 到达时间和输入兼容性；
+- 迁移开销依赖 worker 拓扑、显存容量和传输配置；
+- 当前 trace 的 demand SLO 尚未达到 100%。
+
+## Related Work
+
+实现结合了 continuous batching、deadline-aware scheduling 和多 GPU worker pool，并将这些机制接入
+ABot-World 的 KV cache、TAEW/VAE temporal state 和 LiveKit 实时交付路径。

docs/zh/blog/index.md

@@ -12,6 +12,7 @@ description: 记录 TeleFuser 模型集成、工程架构、性能与运行时
 
 | 日期 | 文章 | 状态 | 验证平台 |
 |---|---|---|---|
+| 2026-08-19 | [ABot 多会话服务：状态隔离、兼容批处理与多卡调度](abot_world_multi_session.md) | 已验证 | 4 x H100 80 GB |
 | 2026-08-19 | [FP8 Sol-Attn：H100 视频 DiT 的量化稀疏注意力](fp8_sol_attention.md) | 已验证 | 1 x H100 80 GB |
 | 2026-08-20 | [从 MiniMax H3 适配看 Agent Dev First](agent_dev_first_minimax_h3.md) | 已验证 | 1、2、4 x H100 80 GB |
 | 2026-08-06 | [CUDA IPC Ulysses：在 H100 上重叠 Attention 通信](cuda_ipc_ulysses.md) | 已验证 | 4 x H100 80 GB |

mkdocs.yml

@@ -130,6 +130,7 @@ plugins:
             FP8 Sol-Attn: FP8 Sol-Attn
             Agent Dev First: Agent Dev First
             CUDA IPC Ulysses: CUDA IPC Ulysses
+            ABot Multi-Session: ABot 多会话服务
             Parallel Inference: 并行推理
             Communication Architecture: 通信架构
             Developer Guides: 开发者指南
@@ -187,6 +188,7 @@ nav:
       - FP8 Sol-Attn: blog/fp8_sol_attention.md
       - Agent Dev First: blog/agent_dev_first_minimax_h3.md
       - CUDA IPC Ulysses: blog/cuda_ipc_ulysses.md
+      - ABot Multi-Session: blog/abot_world_multi_session.md
   - Configuration:
       - configuration.md
       - Quantization: quantization.md