diff --git a/.ai_partners/features/workstreams/2026/06/moshi-three-modes/FEATURE.md b/.ai_partners/features/workstreams/2026/06/moshi-three-modes/FEATURE.md new file mode 100644 index 00000000..88477d97 --- /dev/null +++ b/.ai_partners/features/workstreams/2026/06/moshi-three-modes/FEATURE.md @@ -0,0 +1,394 @@ +--- +title: Moshi Three Modes — 魔师内容传递协作系统 +status: in-progress +priority: P1 +created: 2026-06-20 +updated: 2026-06-22T22:00 +depends: [] +milestone: +description: >- + 开发 Moshi 的三大场景模式（备课/人练/AI主讲），先跑通讲课流程，再推进解耦。 +--- + +# Moshi Three Modes + +## 当前优先级 + +**双阻塞**：bug #23（字幕链路不通）和 bug #24（Ghost 不调 advance_point 导致卡住）必须解决才能继续集成测试。 + +## 当前状态（2026-06-22 会话，第五轮 — 集成测试受阻） + +**v2 代码实施已完成**（8 个文件），但手工集成测试暴露了两个新问题，均阻塞 Phase 2b 闭环。 + +### 集成测试日志分析（moss.log, 2026-06-22 16:48-16:54） + +运行命令：`moss-run-ghost echo --mode default`（注意：是 `default` 不是 `show`） + +关键日志行： + +| 行号 | 时间 | 内容 | 含义 | +|------|------|------|------| +| 846 | 16:48:55 | `MOSS_MODE_NAME=default` | Moshi 进程 mode=default | +| 849 | 16:48:55 | `enable_topic=False, topic_path=moshi/subtitle` | ConfigStore 未命中覆盖（default 模式下无 subtitle 配置） | +| 852 | 16:48:55 | `字幕回调已注入 Speech（HTTP 旁路，同进程）` | 回退链三层全失败，走入 HTTP 旁路 | +| 876 | 16:48:58 | `lecture_start Signal 已发送, points=5` | 讲课启动正常 | +| 930-949 | 16:49:00-14 | TTS batch 创建→连接→接收→完成 | TTS 工作正常，~12s 音频 | +| 950 | 16:49:14 | `wait 28.38s for playing` | 音频播放队列预估 28s | +| 958 | 16:49:43 | `speaker running=False` | 播放完成 | +| 964-966 | 16:49:44 | `interpreter stopped`, `compiled=0 done=1` | **Ghost 只执行了 1 个命令（`__content__`），没有 `advance_point`** | +| 966-1045 | 16:49:44→16:54 | 只有 SSE keep-alive 重连 | **Ghost 完全静默，无新输出** | + +### 字幕回退链分析：三层全失败 + +``` +Layer 1: ConfigStore.get(SubtitleTopicConfig) → enable_topic=False ← 模式覆盖未生效 +Layer 2: Environment.discover().moss_mode_name → "default" + → import MOSS.modes.default.configs → 无 subtitle_topic_config ← 配置只在 show 模式 + → getattr 返回 None → isinstance 跳过 +Layer 3: HTTP 旁路 new_subtitle_callback() → 只在同进程有效 ← Ghost 进程 Speech 未收到回调 +``` + +**结论**：三层回退链无一有效。要打通字幕链路，至少要修复 Layer 1（ConfigStore 把 show mode 的覆盖应用到 Ghost 进程），或直接用 `--mode show` 并确保 ConfigStore 正确工作。 + +### v2 实施完成项（不变） + +| # | 步骤 | 文件 | 状态 | +|---|------|------|------| +| 1 | 注册 `TTSSpeechServiceProvider` | `manifests/providers.py` | ✅ | +| 2 | 定义 `SubtitleTopic` TopicModel | `topics/audio.py` | ✅ | +| 3 | 定义 `SubtitleTopicConfig` ConfigType | `core/speech/subtitle_config.py`（新） | ✅ | +| 4 | 注册 Topic + Config 到 manifests | `manifests/configs.py`, `manifests/topics.py` | ✅ | +| 5 | 扩展回调签名 | `stream_tts_speech.py` | ✅ | +| 6 | 改造 `SpeechServiceProvider.factory()` | `speech_service_provider.py` | ✅ | +| 7 | Mode config 中启用 | `modes/show/configs.py` | ✅ | +| 8 | Moshi 侧 Topic 消费协程 | `moss_in_reflex.py` | ✅ | +| 9 | HTTP 旁路 fallback | `moss_in_reflex.py` | ✅ （三层回退） | +| 10 | 集成测试 | 跨进程手工测试 | 🚫 受阻于 #23 #24 | +| 11 | 清理旧 HTTP 旁路 | — | ⏳ | + +**本轮会话新增踩坑**： +- bug #23：字幕回退链三层全失败（ConfigStore 模式覆盖不生效 + mode=default 无配置 + HTTP 同进程无效） +- bug #24：Ghost 不调 advance_point — 叙述完第一段后静默，无后续 Signal → 卡死 +- 附注：第一轮测试（16:46）还发生了 `attention aborted during execute` + `RuntimeError: MossRuntime is not running`（MossRuntime 提前退出），第二轮未复现 + +## Design Index + +- 对话环路架构（含踩坑）：`design/2026-06-20_chat_loop_architecture.md` +- **讲课模式完整方案**：`design/2026-06-21_teaching_mode.md` +- **bug #22 修复方案 v1**：`.moss_ws/apps/ui/moshi/tmp/subtitle.md` +- **bug #22 修复方案 v2（最终采纳）**：`.moss_ws/apps/ui/moshi/tmp/subtitle_v2.md` +- TTS 字幕方案调研：`.moss_ws/apps/ui/moshi/research-tts-subtitle-progress.md` +- 产品方案：`.moss_ws/apps/ui/moshi/魔师Moshi (3).md` +- App 代码入口：`.moss_ws/apps/ui/moshi/main.py` +- 核心模块：`.moss_ws/apps/ui/moshi/moss_in_reflex/` +- 独立 Reflex App：`.moss_ws/apps/ui/reflex/` +- Topic 桥接参考实现：`.moss_ws/apps/sensors/` + +## Key Decisions + +### 聊天：走 Signal（决策已定，基础设施完成） + +- 删除 ChatTopic/TopicWindow，用 InputSignal 唤醒 Ghost +- SSE 改用 asyncio.Event 驱动 +- context_messages 按模式做信息隔离 +- **新增**：teaching 模式下不注入 chat-instruction，杜绝 Ghost 自言自语 + +### Signal vs Topic（已验证，来自 sensors 参考实现） + +**Topic 不能通知 Ghost。** sensors 的标准模式是 Topic + Signal 同时发布： +- Topic：跨 app 数据共享（Zenoh pub/sub，Ghost 不感知） +- Signal：唤醒 Ghost（mindflow → Nucleus → Impulse → Attention） + +因此在讲课流程中，Moshi↔Reflex 数据桥接用 Topic（Phase 4），唤醒 Ghost 永远用 Signal。 + +### 讲课编排权归服务端（已实施） + +**不让 Ghost 做编排决策。** `/_internal/lecture/start` 在服务端原子完成： +load_course + 强制切 lesson 布局 + 渲染首章 + 初始化 LectureBrain， +然后发 Signal 通知 Ghost。Ghost 只需叙述 + 调 advance_point。 + +编排逻辑在服务端内部，Phase 4 解耦时只需换底层通信机制（Queue → Topic），编排本身不变。 + +### advance_point 后发 Signal 驱动继续（本轮新增） + +Ghost 输出 CTML 命令后倾向于停止生成（function call = 回合结束）。参考 `.moss_ws/apps/genkits/image` 的 `emit_generation_signal` 模式，`advance_point` 执行完毕后发 `InputSignal(Priority.NOTICE)` 唤醒 Ghost 继续叙述。Signal 携带当前 active 段落文本，Ghost 可直接继续。 + +`lecture_ended` 不发 Signal——讲课结束，Ghost 应停止。 + +### ChannelCtx 不在 HTTP handler 中可用 + +aiohttp 的 HTTP handler 运行在裸 asyncio 上下文中，没有 MOSS 的 ChannelCtx。 +在 `moss()` 初始化阶段捕获 `_course_storage`、`_registry` 等引用， +供内部 HTTP handler（`_internal_*`）使用。channel 命令仍然用 `ChannelCtx.container()`。 + +### 数据桥接：Moshi ↔ Reflex 通过 Topic（Phase 4 方案） + +- 参考 `.moss_ws/apps/sensors/` 的实现——audio_capture 和 listener 两个独立 App + 通过 Topic + Matrix 直接通信，不经过 Ghost +- Moshi 和 Reflex 拆分到两个独立 cell，用 Topic 桥接页面状态和章节加载 + +### 翻章同步：AudioRuntimeTopic 门控（Phase 2 设计） + +**问题**：Ghost 裸文本走 `__main__.__content__`（TTS），`advance_point` 走 `apps.ui_moshi` channel。跨 channel 并行执行导致 TTS 还在播上一章，翻章已触发。 + +**约束**：不能强行把 TTS 搬到 moshi channel（narrate 方案违反跨 channel 并行架构，且 `chunks__` 跨协程迭代触发 bug #19/#20）。 + +**方案**：复用 sensors/listener 的 `AudioRuntimeTopic` 门控模式。`MiniAudioStreamPlayer` 已发布 `AudioRuntimeTopic(device_name="speaker", running=True/False)`。`advance_point` 通过 `TopicWindow[AudioRuntimeTopic]` 订阅，翻章前等待 `running=False`。 + +**不改**：`__main__.__content__`、跨 channel 并行、Ghost 裸文本输出方式。 + +### 句级字幕：TTSSpeechStream subtitle_callback（Phase 2 设计） + +**问题**：需要 TTS 实际朗读的文本（同语音完全一致），逐句显示为字幕。 + +**约束**：裸文本→TTS 管道在核心 speech 模块内，moshi channel 不可见。 + +**方案（方案 C）**：在 `TTSSpeechStream`（`src/ghoshell_moss/core/speech/stream_tts_speech.py`）注入可选 `subtitle_callback(text, is_final)`。`_buffer()` 累积文本按标点（`。！？；\n`）切句入队，`_play_loop()` 消费 `tts_batch.items()` 时逐句回调。moshi 侧注入回调写 SSE → 字幕同步显示。 + +**为什么不选其他方案**： +- 方案 A（_feed_stream 层分句）：CTML parser chunk 不按句子对齐，且 feed 时机早于音频播放 +- 方案 B（SpeechTopic 整段字幕）：10-30 秒延迟，讲课不可用 +- 方案 D（时间估算）：文本→音频非线性，精度太低 + +### Context 调整：每轮一个 talking point + +**改动**：`context_messages.py` 讲课指令从 "调完立即继续叙述，不要停顿" 改为 "调完 advance_point 后停止输出，等待系统 Signal 后再继续下一段"。 + +**原因**：Ghost 每轮只讲一个 talking point。TTS 播完 → advance_point 门控放行 → Signal 唤醒 → 下一轮。一轮一点，时序自然对齐。 + +## Implementation Phases + +| Phase | 内容 | 状态 | +|-------|------|------| +| 1 | Signal 对话环路 + 讲课骨架 | ✅ 核心环路跑通 | +| 2a | TTS 翻章同步（AudioRuntimeTopic 门控） | ✅ 已实施（2026-06-22） | +| 2b | 句级字幕（SubtitleTopic 跨进程 Topic 总线） | ✅ 已实施，待集成测试 | +| 2c | 弹幕（飞书消息 → SSE） | 待开始（方案需调整） | +| 3 | 语音打断 + 总结 | 待开始 | +| 4 | Topic 桥接解耦 | 设计完成，待实现 | + +## Phase 2b 修订：SubtitleTopic 跨进程 Topic 总线（v2） + +> 方案分析：`.moss_ws/apps/ui/moshi/tmp/subtitle_v2.md` +> 核心变更：用 Zenoh Topic 总线替代 HTTP 旁路（`new_subtitle_callback()` → `/_internal/subtitle_in`） +> 过渡策略：旧 HTTP 旁路代码保留，通过 `SubtitleTopicConfig.enable_topic` 开关控制 + +### v2 全链路数据流 + +``` +Reflex 进程声明 SubtitleTopicConfig(enable_topic=True, topic_path="moshi/subtitle") + │ + ▼ ConfigStore (workspace/configs/) ← 文件系统，两端共享 + │ +Ghost 进程 SpeechServiceProvider.factory() 读配置 → 创建 _publish_subtitle 闭包 + │ + ▼ _subtitle_callback(text, is_final, batch_id) + │ └─► SubtitleTopic.to_topic() → TopicService.pub(topic) + │ └─► Zenoh session.put(key_expr, json) [线程池] +Zenoh 总线 ← MOSS/{session_scope}/topics/moshi/subtitle + │ + ▼ Reflex 进程 _consume_subtitle() 协程 poll_model() + │ ├─► 压入 _SUBTITLE_QUEUE (async Lock) + │ └─► _SUBTITLE_EVENT.set() + │ └─► _internal_subtitle_stream (SSE) → main.py proxy → Browser +``` + +### v2 实施步骤 + +按依赖关系排列，标注风险等级（🔴高 🟡中 🟢低）： + +| # | 步骤 | 文件 | 风险 | 说明 | +|---|------|------|------|------| +| 1 | 注册 `TTSSpeechServiceProvider` | `manifests/providers.py` | 🟢 | 1 行 import。当前未注册导致 IoC 无法发现 | +| 2 | 定义 `SubtitleTopic` TopicModel | `src/ghoshell_moss/topics/audio.py` | 🟢 | ~20 行，与 `AudioRuntimeTopic` 同文件。字段：text, is_final, batch_id。topic_type="core/speech/subtitle" | +| 3 | 定义 `SubtitleTopicConfig` ConfigType | `src/ghoshell_moss/core/speech/subtitle_config.py`（新） | 🟢 | ~25 行。字段：enable_topic: bool = False, topic_path: str = "moshi/subtitle" | +| 4 | 注册 Topic + Config 到 manifests | `manifests/topics.py`, `manifests/configs.py` | 🟢 | 各 1 行 import | +| 5 | 扩展回调签名 → `(text, is_final, batch_id)` | `stream_tts_speech.py` | 🟡 | `_subtitle_callback` 签名 + `TTSSpeechStream.__init__` 新增 `batch_id` 参数。上游调用方（`SpeechModule` / `__content__`）需追溯一轮 | +| 6 | 改造 `SpeechServiceProvider.factory()` | `speech_service_provider.py` | 🟡 | 读 `SubtitleTopicConfig` → 获取 `TopicService` → 创建 `_publish_subtitle` 闭包（用 `TopicService.pub()` 直接发布，不用 Publisher 抽象） | +| 7 | mode config 中启用 | `modes/show/configs.py`（或 default） | 🟢 | `SubtitleTopicConfig(enable_topic=True, topic_path="moshi/subtitle")` | +| 8 | Moshi 侧 Topic 消费协程 | `moss_in_reflex.py` | 🟡 | `_consume_subtitle()` 协程：`TopicWindow` → `subscriber.poll_model()` → `_SUBTITLE_QUEUE` → `_SUBTITLE_EVENT.set()`。避免 `on_change` 回调（线程池→事件循环桥接问题） | +| 9 | 保留旧 HTTP 旁路，config 开关控制 | `moss_in_reflex.py` | 🟢 | `enable_topic=True` → Topic 路径；`False`（默认）→ HTTP 旁路。过渡期两套共存 | +| 10 | 集成测试 | 跨进程手工测试 | 🟡 | Ghost 进程 ↔ Reflex 进程，确认字幕跨 Zenoh 到达 | +| 11 | 清理旧 HTTP 旁路 | `moss_in_reflex.py`, `main.py` | 🟢 | 删除 `new_subtitle_callback()`、`/_internal/subtitle_in` 端点、`force_fetch(Speech).set_subtitle_callback()`。Topic 路径稳定后执行 | + +### v2 关键设计决策 + +**放弃 Publisher 抽象，改用 `TopicService.pub()` 直接发布**（解决问题 2）： +- `TopicService.pub()` 无需预先建立连接，每次调用即时 pub +- 线程安全已验证（内部 `run_in_executor`） +- 避免了 `ZenohTopicPublisher` async context manager 生命周期绑定问题 +- 代价：失去 `declare_publisher` 预声明优化，但字幕消息频率低（每句一次），可忽略 + +**使用专用消费协程替代 `on_change` 回调**（解决问题 4）： +- `TopicWindow.on_change` 在线程池执行，不能操作 `asyncio.Lock`/`asyncio.Event` +- 消费协程 `_consume_subtitle()` 天然在事件循环上运行，无线程安全问题 +- 与 `AudioRuntimeTopic` 的轮询模式一致 + +**保留旧 HTTP 旁路**（解决问题 5）： +- `SubtitleTopicConfig.enable_topic` 默认 `False`，保持向后兼容 +- 确认 Topic 路径稳定后再清理旧代码 + +### v2 未解决的开放问题 + +1. **`batch_id` 上游来源**：需在调用 `new_tts_stream()` 时传入。确认是 `SpeechModule` 层还是 `__content__` 命令层生成 +2. **TopicWindow 底层 Subscriber 访问**：消费协程需要 `window._subscriber`（私有属性），或需在 `TopicWindow` ABC 上暴露 `poll()` 方法 +3. **Zenoh 断连恢复**：`ZenohTopicService` session 断连时的重连行为需确认 +4. **多 Ghost 实例**：同一 `session_scope` 下多 Ghost 同时说话时，字幕 topic 混在一起。`batch_id` 可用于前端过滤但需前端配合 + +### Phase 2a：TTS 翻章同步（AudioRuntimeTopic 门控）✅ + +| # | 内容 | 涉及文件 | +|---|------|---------| +| 2a.1 | `moss()` 初始化时创建 `TopicWindow[AudioRuntimeTopic]` | `moss_in_reflex.py` | +| 2a.2 | `advance_point` 翻章前等待 `device_name="speaker"` 的 `running=False`（150ms 轮询） | `moss_in_reflex.py` | +| 2a.3 | context 指令改为"调完停止，等 Signal 再继续" | `context_messages.py` | + +### Phase 2c：弹幕（后续） + +| # | 内容 | 涉及文件 | +|---|------|---------| +| 2c.1 | `check_messages` 实际对接飞书 channel | `moss_in_reflex.py` | +| 2c.2 | 飞书消息 → `_DANMAKU_QUEUE` → SSE 弹幕 | `moss_in_reflex.py` | +| 2c.3 | 弹幕频率控制（缓冲 50 条，堆积 20+ 丢弃非@旧消息） | `moss_in_reflex.py` | + +## Phase 1 实现进度 + +| # | 内容 | 状态 | +|---|------|------| +| 1.1 | chapter_data 增加 speaker_notes 字段 | ✅ | +| 1.2 | LectureBrain 状态机（lecture_brain.py） | ✅ | +| 1.3 | 新命令（advance_point、check_messages、resume_lecture） | ✅ | +| 1.4 | 课程列表 API（/_internal/courses → main.py proxy） | ✅ | +| 1.5 | 前端讲课页面（teachingL0.html） | ✅ | +| 1.6 | Ghost context 更新（speaker_notes + 约束 + Signal 驱动） | ✅ | +| 1.7 | 字幕通道 | 延至 Phase 2 | + +### 新增 API 路由 + +| 路由 | Method | 说明 | +|------|--------|------| +| `/api/courses` | POST | 课程列表（proxy → /_internal/courses） | +| `/api/lecture/start` | POST | 讲课启动（proxy → /_internal/lecture/start） | + +### 新增 Channel 命令 + +| 命令 | 说明 | +|------|------| +| `advance_point` | 段落推进，返回 point_advanced / chapter_advanced / lecture_ended。执行后发 Signal 唤醒 Ghost | +| `check_messages` | 检查飞书消息（Phase 1-2 桩） | +| `resume_lecture` | 暂停后恢复讲课 | + +### 新增文件 + +| 文件 | 说明 | +|------|------| +| `moss_in_reflex/lecture_brain.py` | 讲课状态机，不依赖 Reflex | + +## 踩坑记录 + +| # | 现象 | 根因 | 解决 | 日期 | +|---|------|------|------|------| +| 1 | main.py `Matrix already started` | Matrix cell 单实例锁 | HTTP 桥接 | 2026-06-20 | +| 2 | ChatTopic/_CHAT_WINDOW 被 hot-reload 重置 | 模块级变量 | 换 Signal（不依赖模块变量） | 2026-06-20 | +| 3 | Ghost 不主动读 context | Topic 是数据非事件 | Signal 唤醒 | 2026-06-20 | +| 4 | `aspect-video` 裁切聊天面板 | CSS 布局 | `h-full` | 2026-06-20 | +| 5 | logger 不可见 | `getLogger(__name__)` 无 handler | `getLogger("moss")` | 2026-06-20 | +| 6 | `to_signal()` keyword error | 参数是 `*messages` 位置参数 | `to_signal(Message(...))` | 2026-06-20 | +| 7 | `Text()` positional arg | 需 keyword | `Text(text=...)` | 2026-06-20 | +| 8 | 前端讲课页蒙层不消失 | Ghost 讲课时不走 chat_reply | 服务端 `/_internal/lecture/start` 原子完成 | 2026-06-21 | +| 9 | Ghost 乱切布局 | context 没说用哪个布局 | 服务端强制 LayoutEvent；context 约束 | 2026-06-21 | +| 10 | Ghost 重复调 load_course | context 未告知已加载 | context 约束 + 服务端 guard | 2026-06-21 | +| 11 | 5 个 advance_point 后返回 lecture_ended 而非翻章 | Ghost 在 CTML 命令后停止生成，无新 attention 周期推进翻章逻辑 | Signal 驱动：advance_point 后发 InputSignal 唤醒 Ghost | 2026-06-21 | +| 12 | `/_internal/lecture/start` 崩溃 | `_switch_to_chapter` 调 ChannelCtx 但 HTTP handler 无此上下文 | 初始化时捕获 `_registry` 引用 | 2026-06-21 | +| 13 | Ghost 输出 `<_>` 破坏 CTML 解析 | Ghost 用 `<_>` 做段落分隔，CTML 解析器把 `<` 当标签 | context 加 CTML 卫生约束：禁止自创标签 | 2026-06-21 | +| 14 | 讲课模式下 Ghost 自言自语调 chat_reply | chat-instruction 对所有模式注入 | teaching 模式不注入 chat-instruction | 2026-06-21 | +| 15 | Ghost 调 advance_point 后静默 | CTML function call 后模型停止生成，COMMAND-RESULT 不触发新 attention | advance_point 后发 InputSignal(Priority.NOTICE) 唤醒 | 2026-06-21 | +| 16 | lecture_ended 后 Ghost 继续调 check_messages + advance_point | context 无停止规则 + advance_point 在 ended 状态抛异常 | 幂等保护 + context 明确停止规则 | 2026-06-21 | +| 17 | Ghost 用 `` 调命令报 not found | context 写的是短名 `moshi`，运行时注册的是全名 `apps.ui_moshi` | 全部改为 `apps.ui_moshi` | 2026-06-21 | +| 18 | context 中 speaker_notes 和 lecture-state 展示矛盾 | teaching() 读持久化数据（全 pending），lecture-state 读内存（实时状态） | 移除 context-mode 的 talking_points 展示，只保留 lecture-state 的实时进度 | 2026-06-21 | +| 19 | narrate: `'NoneType' object has no attribute 'call_soon_threadsafe'` | `asyncio.create_task(_feed())` 把 `chunks__` 放到后台协程迭代，底层 janus 队列的 `call_soon_threadsafe` 找不到 event loop | 不要跨协程传递 `chunks__`，始终在主协程迭代。`SpeechStream.speak()` 官方 API 已处理此约束 | 2026-06-21 | +| 20 | narrate: `speech unavailable` | `ChannelCtx.container().force_fetch(Speech)` 拿到的 Speech 实例与 Shell `_speech_context_manager` 启动的不是同一个，`is_running()` 返回 False | 在 `moss()` 初始化时用 `matrix.container` 直接捕获（与 `_course_storage`/`_registry` 一致） | 2026-06-21 | +| 21 | narrate 方案整体否决 | 试图把 TTS 搬到 moshi channel 强制串行，违反 MOSS 跨 channel 并行核心设计。用户指正"不符合 moss 架构思想" | 不在 channel 层面强行串行。在共享状态上做协调——翻章用 AudioRuntimeTopic 门控，字幕用 SpeechStream 回调注入 | 2026-06-22 | +| 22 | subtitle_callback 注入到错误的 Speech 实例 | Reflex 进程和 Ghost 进程各有独立的 IoC Container，`matrix.container.force_fetch(Speech)` 在两个进程中返回不同的 `BaseTTSSpeech` 实例。`moss()` 中注入的 callback 设在 Reflex 进程的 Speech 上，Ghost 进程中 Shell 用的 Speech 未收到回调 → `TTSSpeechStream._subtitle_callback is None` → 字幕数据永远不到达 SSE | v2 方案：在 Ghost 进程 `SpeechServiceProvider.factory()` 中注入 `_publish_subtitle` 闭包，通过 Zenoh Topic 总线跨进程发布字幕。代码已实施，但集成测试中被 #23 阻塞 | 2026-06-22 | +| 23 | 字幕 ConfigStore 回退链三层全失败 | 三层回退无一打通：(1) ConfigStore 模式覆盖不生效，Ghost/Moshi 进程均读到 `enable_topic=False` 的默认值（根因可能是 `HostEnvConfigStoreProvider` 未正确扫描 mode config 模块，或 `search_config_infos_from_package` 未找到 instance 类型的 override）；(2) `Environment.discover().moss_mode_name` 返回 `"default"` 而非 `"show"`，`MOSS.modes.default.configs` 无 `subtitle_topic_config` 属性 → 回退导入静默跳过；(3) HTTP 旁路 `new_subtitle_callback()` 只在 Reflex 同进程有效，Ghost 进程 Speech 收不到。**当前日志确认**：`enable_topic=False` 在两个进程中都出现，走 HTTP 旁路，无字幕数据发布 | **方向 A**：确保 `--mode show` 运行，并修复 ConfigStore 模式覆盖链路（诊断 `HostEnvConfigStoreProvider.bootstrap()` → `search_config_infos_from_package`）。**方向 B（临时）**：在 `MOSS.modes.default.configs` 中重复定义 `subtitle_topic_config`，或在 `speech_service_provider.py` / `moss_in_reflex.py` 中硬编码 `enable_topic=True` | 2026-06-22 | +| 24 | Ghost 不调 `advance_point` → 叙述完第一段后静默 | 日志 `interpreter settled: compiled=0 done=1`：Ghost 只执行了 `__main__.__content__`（TTS 叙述），没有输出 `` CTML 命令。模型讲完第一个 talking point 即停止，不调 advance_point → 无 Signal → 永远不会醒来继续。context 指令（`context_messages.py:119-130`）已写明"每讲完一个要点后调 advance_point"，但 LLM 模型行为不可靠。第一轮测试（16:46）还发生了 `attention aborted during execute` + `RuntimeError: MossRuntime is not running`（MossRuntime 提前退出崩溃），第二轮未复现 | **方向 A**：增强 context 约束（如要求 Ghost 必须在同一轮输出的末尾包含 ``，否则视为违规）。**方向 B**：服务端增加超时兜底——TTS 播放完成后 N 秒若无 `advance_point` 调用，自动推进并发送 Signal。**方向 C**：检查 Ghost LLM 的原始输出，确认是模型没生成还是 CTML 解析器未识别 | 2026-06-22 | + +## 架构教训 + +### IoC Container 是进程内对象，不跨进程共享 + +`TTSSpeechServiceProvider(singleton=True)` 的 singleton 作用域是单个 `IoCContainer` 实例内部——不是跨 OS 进程的。Reflex 进程和 Ghost 进程各自调用 `Host.discover()`，各自创建独立的 Host/Matrix/IoCContainer，各自在首次 `force_fetch(Speech)` 时触发 Provider 创建自己的 `BaseTTSSpeech` 实例。两个 Python 对象在不同的内存空间，互不影响。 + +**可跨进程工作的机制**：Zenoh Topic（`AudioRuntimeTopic`）— pub/sub 走网络协议，天然跨进程。 +**不可跨进程工作的机制**：Python 函数指针（`subtitle_callback`）— 只在设置它的进程内存中有效。 + +**推论**：任何需要在 Ghost 进程中生效的运行时注入，必须在 Ghost 进程侧执行。Reflex 进程侧的 `matrix.container` 操作只影响 Reflex 进程自身。 + +Ghost 的职责是**叙述**，不是**编排**。布局选择、课程加载、章节导航——这些都是确定性操作，应放在服务端，不经过 Ghost 的理解层。把编排交给 Ghost 引入了三类问题： +1. 幻觉（选错布局、重复操作） +2. 时序依赖（Ghost 的 CTML 执行顺序不可靠） +3. 上下文污染（编排细节占用了叙述需要的 token） + +原则：**确定性操作走 API，创造性工作走 Ghost。** + +### ChannelCtx 的作用域 + +`ChannelCtx.container()` 只在 MOSS channel 命令和 context_messages 回调中可用。 +HTTP handler（aiohttp）运行在裸 asyncio 上下文，需要在 `moss()` 初始化阶段捕获所需 IoC 引用。 +这是一个通用模式——任何在 `moss()` 闭包内但不在 channel 命令链中的代码，都应使用捕获引用。 + +### CTML 命令 ≠ 回合延续 + +模型倾向于在 function call 后停止生成。CTML 命令返回的 `` 是 percept 不是 Signal——它能被 Ghost 看到，但不能触发新的 attention 周期。要让 Ghost 在命令后继续工作，必须发 Signal 来驱动下一轮 mindflow。 + +模式：**命令执行 → 状态变更 → 发 Signal(Priority.NOTICE) → Ghost 醒来 → 读 context → 继续**。 + +参考实现：`.moss_ws/apps/genkits/image/main.py` 的 `emit_generation_signal`。 + +### Channel 短名 ≠ 运行时全名 + +`PyChannel(name="moshi")` 注册的短名在 app 体系下会被自动加上 `apps._` 前缀。context 中引用的命令必须使用运行时全名，否则 Ghost 每次调命令都先吃一个 `INTERPRET_ERROR`。 + +## 下次会话起点 + +### 优先级 0：修复 Ghost 卡住（bug #24） + +Ghost 叙述完第一段后不调 `advance_point`，整个讲课流程死锁。 + +**诊断步骤**： +1. 查看 Ghost LLM 的原始输出 — 确认模型是否生成了 ``。如果是，说明 CTML 解析器未识别；如果否，是 prompt 问题 +2. 检查 `__content__` 命令执行后 Ghost 的 mindflow 状态 — `interpreter settled` 后应有新 attention 周期，但 `observe=False` 暗示没有 +3. 对比 chat 模式（聊天时 Ghost 能正常调 `chat_reply`）与 teaching 模式的行为差异 + +**可能修复**： +- 增强 context 约束：明确要求 Ghost "本轮输出末尾必须包含 ``" +- 服务端超时兜底：TTS `running=False` 后 5s 无 `advance_point` → 自动推进 + 发 Signal +- 如果是 CTML 解析器问题：检查 `` 语法是否被正确识别 + +### 优先级 1：打通字幕链路（bug #23） + +**短期可用路径**（绕过 ConfigStore bug）： +- 方案 A：在 `MOSS.modes.default.configs` 中加 `subtitle_topic_config = SubtitleTopicConfig(enable_topic=True)` — 1 行改动，让 default 模式也启用 Topic 字幕 +- 方案 B：在 `speech_service_provider.py` 和 `moss_in_reflex.py` 的回退代码中，不仅尝试 `mode_name` 的 config，也尝试 `"show"` 的 config — 比方案 A 更脏 + +**正确路径**（修复 ConfigStore 模式覆盖）： +1. 添加诊断日志到 `HostEnvConfigStoreProvider.bootstrap()` — 确认 `search_config_infos_from_package("MOSS.modes.show.configs")` 是否找到 instance 类型的 `subtitle_topic_config` +2. 若未找到 → 修复扫描逻辑；若找到但未应用 → 修复 `set_config` 调用链 +3. 验证：`moss --mode show manifests configs` 输出应包含 `subtitle_topic: SubtitleTopicConfig(enable_topic=True)` + +### 优先级 2：ConfigStore 根因修复 + +独立 feature：ConfigStore 模式覆盖（`is_override=True` 的 ConfigType instance）不生效。影响范围超出字幕——任何在 mode configs 中定义的 instance 覆盖都无效。需要端到端追踪：`PackageManifests` 扫描 → `ConfigInfo(is_override=True)` → `HostEnvConfigStoreProvider.bootstrap()` → `set_config()`。 + +### 测试命令 + +```bash +# 正确命令（用 show mode） +moss-run-ghost echo --mode show + +# 验证 ConfigStore +moss --mode show manifests configs | grep subtitle + +# 日志关键字 +grep -E "subtitle|advance_point|interpreter settled|回退导入" .moss_ws/runtime/logs/moss.log +``` diff --git a/.ai_partners/features/workstreams/2026/06/moshi-three-modes/design/2026-06-20_chat_loop_architecture.md b/.ai_partners/features/workstreams/2026/06/moshi-three-modes/design/2026-06-20_chat_loop_architecture.md new file mode 100644 index 00000000..457790c8 --- /dev/null +++ b/.ai_partners/features/workstreams/2026/06/moshi-three-modes/design/2026-06-20_chat_loop_architecture.md @@ -0,0 +1,152 @@ +# Moshi 对话环路 + 架构解耦设计 + +## 背景 + +Moshi 当前聊天是 echo 原型。需打通 用户↔Ghost 的对话环路，同时解决 Moshi 与 Reflex 的耦合问题。 + +## 架构演进 + +### v1：Topic 方案（已废弃） + +ChatTopic pub/sub，Ghost 被动读 TopicWindow。基础设施全通但 Ghost 不主动来取消息。 + +**废弃原因**：聊天是"事件"而非"数据"。Topic 是被动数据结构。 + +### v2：Moshi ↔ Reflex 分裂为两个 Matrix Cell（备选，暂不采用） + +两个独立 cell，Ghost 跨 channel 编排命令。 + +**搁置原因**：Ghost 需要两步操作（读快照→存档），增加幻觉风险。 + +### v3：单 Channel + 模块化（当前代码结构） + +一个 "moshi" Channel 承载所有命令，模块化拆分代码。Signal 驱动对话。 + +### v4：Topic 桥接解耦（当前设计方向，参考 sensors 模式） + +**参考实现**：`.moss_ws/apps/sensors/` + +audio_capture 和 listener 是两个独立 App，通过 Topic + Matrix 直接通信： + +``` +audio_capture 进程 listener 进程 + │ │ + ├─ MatrixAudioTransport ├─ MatrixAudioTransport + │ ├─ pub PCM 音频流 │ ├─ 消费 PCM 流 + │ └─ topic_window(AudioRuntimeTopic)│ ├─ pub_topic(SpeechTopic) + │ │ └─ add_signal(AudioSignal) + └────────────────┬───────────────────┘ + │ + Matrix/Zenoh Topic 总线 + （零 Ghost 参与） +``` + +**应用到 Moshi**： + +``` +Reflex App（渲染 cell） Moshi App（课程逻辑 cell） + │ │ + │ 页面变更 → pub PageSnapshotTopic │ + │ (debounce 500ms) ├─ topic_window(PageSnapshotTopic) + │ │ └─ save_chapter 时读最新快照 + │ │ + ├─ topic_window(ChapterLoadTopic) │ + │ └─ 收到 → 渲染到白板 │ + │ ├─ start_teaching → pub ChapterLoadTopic + └──────────────────┬───────────────────┘ + │ + Matrix/Zenoh Topic 总线 + （零 Ghost 参与） +``` + +**新增两个 Topic**： + +```python +class PageSnapshotTopic(TopicModel): + """Reflex 页面状态快照""" + layout: str + fields: dict # {title, sub_title, main_text, annotations, images_count} + +class ChapterLoadTopic(TopicModel): + """Moshi 请求 Reflex 加载章节""" + course_name: str + chapter_index: int + chapter_data: dict # {title, sub_title, main_text, annotations, images} +``` + +**save_chapter 改造**：不再读 `_SNAPSHOTS`，改为读 `PageSnapshotTopic` TopicWindow。 + +**switch_to_chapter 改造**：不再操作 QUEUE，改为 pub `ChapterLoadTopic`。 + +### 方案对比 + +| | v2（Ghost 编排） | v4（Topic 桥接） | +|---|---|---| +| Ghost 参与 | Ghost 跨 channel 编排 | **零 Ghost 参与** | +| 参考实现 | 无 | **sensors 已验证** | +| 改动量 | 大 | 中（两个 Topic + 消费端） | + +## 当前架构（v3，代码实现中） + +### 进程拓扑 + +``` +Reflex 进程 (:3000) — lifespan → moss() + ├─ Matrix.discover() + ├─ 注册 Channel "moshi"（渲染 + 课程 + 聊天命令） + ├─ context_messages（课程状态 + chat-instruction + 模式感知信息隔离） + └─ 内部 HTTP 桥接 (:9733) + ├─ POST /_internal/chat_in → InputSignal.add_signal() + └─ GET /_internal/chat_stream → SSE（asyncio.Event 驱动） + +main.py (:9731) — HTTP proxy，不碰 Matrix +Ghost 进程 — 发现 Channel "moshi" +``` + +### 对话环路（Signal 驱动） + +``` +用户输入 → main.py → :9733/_internal/chat_in + → InputSignal().to_signal(Message(text)) + → matrix.session.add_signal(signal) # Zenoh "MOSS/default/signals" + → Ghost InputSignalNucleus → Impulse → Attention + → 读 context_messages → 看到 chat-instruction + → 回复 + → _SSE_QUEUE.append + _SSE_EVENT.set() + → SSE 推送 → main.py pipe → 浏览器 +``` + +## 关键架构决策 + +| # | 决策 | 状态 | +|---|------|------| +| 1 | 聊天走 Signal 而非 Topic | ✅ 实现 | +| 2 | 删除 ChatTopic/TopicWindow | ✅ 实现 | +| 3 | context 按模式信息隔离（preparing 才发 Reflex 命令信息） | ✅ 实现 | +| 4 | 暂不拆分 cell，模块化组织代码 | ✅ 当前策略 | +| 5 | Moshi ↔ Reflex 通过 Topic 桥接（v4） | ⏳ 设计完成，等解耦阶段实现 | + +## 踩坑记录 + +| # | 现象 | 根因 | 解决 | +|---|------|------|------| +| 1 | main.py `Matrix already started` | Matrix cell 单实例锁 | HTTP 桥接 | +| 2 | ChatTopic/_CHAT_WINDOW 被 hot-reload 重置 | 模块级变量 | 换 Signal（不依赖模块变量） | +| 3 | Ghost 不主动读 context | Topic 是数据非事件 | Signal 唤醒 | +| 4 | `aspect-video` 裁切聊天面板 | CSS 布局 | `h-full` | +| 5 | logger 不可见 | `getLogger(__name__)` 无 handler | `getLogger("moss")` | +| 6 | `to_signal()` keyword error | 参数是 `*messages` 位置参数 | `to_signal(Message(...))` | +| 7 | `Text()` positional arg | 需 keyword `Text(text=...)` | `Text(text=...)` | + +## 实现进度 + +| Phase | 内容 | 状态 | +|-------|------|------| +| 1.1 | 删除 ChatTopic | ✅ | +| 1.2 | Signal 发消息 | ✅ | +| 1.3 | 删除 TopicService/TopicWindow | ✅ | +| 1.4 | SSE 改用 asyncio.Event | ✅ | +| 1.5 | context chat-instruction + 信息隔离 | ✅ | +| 1.6 | chat_reply prompt 调优 | ⏳ 待调 | +| 2 | 讲课流程梳理 + 跑通 | ⏳ 进行中 | +| 3 | Topic 桥接解耦 | ⏳ 设计完成 | diff --git a/.ai_partners/features/workstreams/2026/06/moshi-three-modes/design/2026-06-21_teaching_mode.md b/.ai_partners/features/workstreams/2026/06/moshi-three-modes/design/2026-06-21_teaching_mode.md new file mode 100644 index 00000000..92f5780f --- /dev/null +++ b/.ai_partners/features/workstreams/2026/06/moshi-three-modes/design/2026-06-21_teaching_mode.md @@ -0,0 +1,479 @@ +# 讲课模式完整技术方案 + +> **实施策略**：按 FEATURE.md 优先级——先单体跑通讲课流程（Phase 1-3），最后 Topic 桥接解耦（Phase 4）。每 Phase 产出可体验的端到端功能。 + +## 用户故事（完整流程） + +1. 用户在模式选择页点击 **"讲课"** +2. 弹出课程列表，用户点击选择一门课 +3. 页面切换到讲课模式：**全屏 Reflex iframe（白板） + 底部字幕区** +4. AI 自动开讲（TTS），字幕同步展示 +5. 讲课时飞书群消息以**弹幕形式从右飘过**，AI 在飞书群回复（不打断讲课） +6. 人类**语音提问**→ AI 暂停讲课 → 回答语音问题 → 继续讲课 +7. 全部讲完后，AI 总结所有问题（飞书群 + 语音打断） + +## 核心设计：双重视图 + +Reflex 白板是「观众视图」，Ghost 看到的是「演讲者视图」——两者主题相同，内容不需要逐字对应。 + +| | 观众视图（Reflex 白板） | 演讲者视图（Ghost context） | +|---|---|---| +| **内容** | 标题 + 要点 + 图片（投屏用） | 每段演讲要点、过渡词、关键数据、预期时长 | +| **给谁看** | 听众/学员 | 主讲人（AI 或人类） | +| **数据来源** | `slide_data` | `speaker_notes` | +| **生成时机** | 备课阶段 AI 生成 | 备课阶段 AI 生成（与 slide_data 同时） | +| **练习模式映射** | — | 演讲者手卡（📌要点 / 🔄过渡 / 📊关键数据） | + +### speaker_notes 的生成流程 + +在备课（preparing）阶段，Ghost 为每个章节生成内容时，同时产出两份数据： + +1. **slide_data**（观众视图）— 投屏用的标题、要点、图片 +2. **speaker_notes**（演讲者视图）— 讲课时 Ghost 自己的"手卡" + +Ghost 调 `save_chapter` 时，slide_data 和 speaker_notes 一起持久化到章节 JSON 中。讲课阶段 Ghost 从章节数据中读取 speaker_notes 作为叙述指南。 + +**章节数据结构**： + +```python +chapter_data: + slide_data: # 观众视图（已有，原 chapter_data 内容） + sub_title: str + main_text: str + annotations: list[str] + images: list[str] # locator 列表 + + speaker_notes: # 演讲者视图（备课阶段 AI 生成，与 slide_data 同时持久化） + talking_points: [ + {"id": "0", "text": "开场：欢迎各位", "status": "done"}, + {"id": "1", "text": "要点1：核心用户场景", "status": "done"}, + {"id": "2", "text": "要点2：小步实验验证", "status": "active"}, + {"id": "3", "text": "要点3：沉淀指标看板", "status": "pending"}, + ] + # status: done=已讲完, active=正在讲（唯一）, pending=待讲 + transitions: list[str] # 过渡词 + key_data: list[str] # 关键数据 + estimated_duration: int # 预计讲解时长（秒） +``` + +**段落驱动 + 间隙回复**：每个 talking_point 讲完后 Ghost 调 `advance_point` 标记完成。段落间隙调 `check_messages` 批量回复飞书消息。 + +**打断恢复**：context 中展示 `✓ done / → active / … pending` 的段落状态列表，Ghost 直接从 `→` 标记处继续。 + +**advance_point 兜底**：若 Ghost 未在合理时间内调 `advance_point`（当前 active 段落持续超过 estimated_duration × 1.5），LectureBrain 自动将当前段落标记为 done 并推进到下一段，同时向 Ghost context 注入提示"上一段已超时自动推进"。 + +## 架构概览 + +### Phase 1-3：单体架构（当前阶段） + +Moshi 和 Reflex 在同一进程内，通过内部 Queue + 命令直接通信。讲课状态机、TTS 控制、字幕生成均在 moshi Channel 内完成。 + +``` +┌─ 浏览器 ──────────────────────────────────────────────────────────────┐ +│ ModeSelect → CourseSelect → LecturePage(iframe + subtitle + danmaku) │ +└──────────────────────────────┬──────────────────────────────────────────┘ + │ HTTP + SSE (/api/chat, /api/chat/stream, + │ /api/courses, /api/mode) +┌─ main.py (:9731) ──────────────────────────────────────────────────────┐ +│ HTTP proxy：不碰 Matrix │ +│ 路由：/api/courses（课程列表）、/api/mode（发 Signal） │ +│ 新增 SSE 频道：/api/chat/stream/subtitle、/api/chat/stream/danmaku │ +└──────────────────────────────┬──────────────────────────────────────────┘ + │ HTTP POST :9733/_internal/* +┌─ Moshi + Reflex（同一进程）────────────────────────────────────────────┐ +│ Channel "moshi"：课程逻辑 + 讲课控制 + 聊天 + 渲染 │ +│ LectureBrain：讲课状态机（idle→loading→lecturing→paused→ended） │ +│ TTS 控制 → 字幕文本 → SSE 推送 Subtitle │ +│ 飞书群消息 → 弹幕 → SSE 推送 Danmaku │ +│ 语音打断 ← AudioSignal → pause → answer → resume │ +│ │ +│ 内部通信（Queue，不经过 Matrix）： │ +│ _switch_to_chapter → QUEUE.put(LayoutEvent) → Reflex 渲染 │ +│ Reflex 页面变更 → LayoutSnapshot → save_chapter 读取 │ +│ Subtitle/Danmaku → _SSE_QUEUE → main.py pipe → 浏览器 │ +└────────────────────────────────────────────────────────────────────────┘ +``` + +### Phase 4：Topic 桥接解耦（未来） + +``` +┌─ 浏览器 ──────────────────────────────────────────────────────────────┐ +│ ModeSelect → CourseSelect → LecturePage(iframe + subtitle + danmaku) │ +└──────────────────────────────┬──────────────────────────────────────────┘ + │ HTTP + SSE +┌─ main.py (:9731) ──────────────────────────────────────────────────────┐ +│ HTTP proxy：不碰 Matrix │ +└──────────────────────────────┬──────────────────────────────────────────┘ + │ HTTP POST :9733/_internal/* +┌─ Moshi Cell（课程逻辑）────────────────────────────────────────────────┐ +│ Channel "moshi"：课程逻辑 + 讲课控制 + 聊天 │ +│ LectureBrain：讲课状态机 │ +│ TTS 控制 → pub SubtitleTopic │ +│ 飞书群消息 → pub DanmakuTopic │ +│ │ │ +│ Topic Bridge（不经过 Ghost） │ │ +│ ┌────────────────────────────────────────────────────────┼──┐ │ +│ │ pub ChapterLoadTopic → Reflex 渲染章节 │ │ │ +│ │ sub PageSnapshotTopic ← Reflex 页面快照 │ │ │ +│ │ pub SubtitleTopic → Reflex 字幕 │ │ │ +│ │ pub DanmakuTopic → Reflex 弹幕 │ │ │ +│ │ pub LectureStateTopic → Reflex 状态同步 │ │ │ +│ └────────────────────────────────────────────────────────┼──┘ │ +└───────────────────────────────────────────────────────────┼────────────┘ + │ Matrix/Zenoh +┌─ Reflex Cell ──────────────────────────────────────────────────────────┐ +│ Channel "reflex"：纯白板渲染 │ +│ topic_window(ChapterLoadTopic) → 渲染章节 │ +│ topic_window(SubtitleTopic) → 字幕展示 │ +│ topic_window(DanmakuTopic) → 弹幕动画 │ +│ pub PageSnapshotTopic（debounce 500ms） │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +## Topic 定义（Phase 4 启用，Phase 1-3 用内部 Queue 等效实现） + +### 1. ChapterLoadTopic — Moshi → Reflex（章节渲染） + +```python +class ChapterLoadTopic(TopicModel): + course_name: str + chapter_index: int = 0 + total_chapters: int = 0 + chapter_data: dict # slide_data 部分（观众视图） + topic_type → "moshi/chapter_load" +``` + +Phase 1-3 等价实现：`_switch_to_chapter()` → `QUEUE.put(LoadChapterEvent)`。 + +### 2. PageSnapshotTopic — Reflex → Moshi（页面快照） + +```python +class PageSnapshotTopic(TopicModel): + layout: str + fields: dict # {title, sub_title, main_text, annotations_count, images_count} + topic_type → "moshi/page_snapshot" +``` + +Phase 1-3 等价实现：`LayoutSnapshot.refresh()` 读取当前 Reflex State。 + +### 3. SubtitleTopic — Moshi → Reflex（字幕同步） + +```python +class SubtitleTopic(TopicModel): + text: str # 当前字幕文本 + chapter_index: int = 0 + is_final: bool = False # True=本段讲完, False=流式增量 + topic_type → "moshi/subtitle" +``` + +Phase 1-3 等价实现：写入 `_SUBTITLE_QUEUE`，SSE 推送到浏览器。 + +### 4. DanmakuTopic — Moshi → Reflex（弹幕） + +```python +class DanmakuTopic(TopicModel): + sender: str # 发送者昵称 + text: str # 弹幕文本 + color: str = "white" # 颜色标签（white/green/yellow/red） + source: str = "feishu" # feishu / system + topic_type → "moshi/danmaku" +``` + +Phase 1-3 等价实现：写入 `_DANMAKU_QUEUE`，SSE 推送到浏览器。 + +**弹幕频率控制**：飞书群消息先入缓冲窗口（最近 50 条），按到达时间戳均匀发射。若缓冲堆积超过 20 条，丢弃最旧的非@消息，保留 @消息优先展示。 + +### 5. LectureStateTopic — Moshi → Reflex（讲课状态） + +```python +class LectureStateTopic(TopicModel): + state: Literal["loading", "lecturing", "paused", "ended"] + course_name: str = "" + chapter_index: int = 0 + total_chapters: int = 0 + topic_type → "moshi/lecture_state" +``` + +## 讲课状态机（LectureBrain） + +``` +idle → loading → lecturing ⇄ paused → ended + ↑ + 语音打断或人类控场 +``` + +| 状态 | 行为 | +|------|------| +| idle | 等待开始讲课指令 | +| loading | load_course + 渲染首章（slide_data 推送白板，speaker_notes 进 context） | +| lecturing | TTS 播放 + 字幕同步；Ghost 按 speaker_notes 逐要点讲，讲完后自动翻页 | +| paused | TTS 暂停；Ghost 回答问题；恢复时重新读 speaker_notes，自行判断从哪继续 | +| ended | TTS 停止，总结飞书+语音问题 | + +**核心数据**： + +```python +class LectureBrain: + status: str = "idle" + current_course: str = "" + current_chapter: int = 0 + points: list[dict] = [] # 当前章节 talking_points（从 speaker_notes 加载） + questions: list[dict] = [] # [{source:"voice"/"feishu", text:"...", sender:"..."}] + point_started_at: float = 0 # 当前 active 段落开始时间（用于超时检测） +``` + +**新命令**： + +```python +# advance_point — 标记当前 active→done，下一个 pending→active +# 全部 done 时返回 "chapter_done"（Ghost 据此翻页） +# 兜底：若超过 estimated_duration * 1.5 未调用，LectureBrain 自动推进 + + +# check_messages — 读 feishu_window，有消息就回复，返回消息数 +# Ghost 在段落间隙调用 + +``` + +## 前端页面流 + +``` +┌─ 模式选择页（/）───────────────────────────────────────┐ +│ ┌─ TopBar ───────────────────────────────────────┐ │ +│ │ [备课] [练习] [讲课] │ │ +│ └────────────────────────────────────────────────┘ │ +│ ┌─ 内容区 ───────────────────────────────────────┐ │ +│ │ 默认：显示 Logo/提示文字 │ │ +│ │ 点击"讲课"后→ 弹出课程选择列表 │ │ +│ └────────────────────────────────────────────────┘ │ +└───────────────────────────────────────────────────────┘ + │ 点击具体课程 + ▼ +┌─ 讲课页面（/lecture?course=xxx）───────────────────────┐ +│ ┌─ Reflex iframe（全屏白板）─────────────────────┐ │ +│ │ :3000，嵌入，占满整个页面 │ │ +│ │ 弹幕层：文字从右飘过，覆盖在白板上方 │ │ +│ └──────────────────────────────────────────────┘ │ +│ ┌─ 字幕区（底部，半透明背景）─────────────────────┐ │ +│ │ AI 主讲字幕，TTS 所说文字同步展示 │ │ +│ └──────────────────────────────────────────────-┘ │ +│ ┌─ 退出按钮（右下角，3s 无操作半透明）──────────────┐ │ +│ │ ✕ 仅此一个操作入口 │ │ +│ └────────────────────────────────────────────────┘ │ +└───────────────────────────────────────────────────────┘ +``` + +前端基于现有 `static/teachingL0.html` 改造——该文件已有弹幕 CSS 动画和演讲者手卡 UI 组件，可直接复用。 + +## TTS 音频通道 + +TTS 音频不经过浏览器——由服务端 Volcengine TTS 直接合成播放（通过 MOSS TTSManager + AudioPlayer provider）。浏览器端通过字幕同步跟进内容。 + +流程： +``` +Ghost 生成叙述文本 → + → TTSManager.synthesize(text) → AudioPlayer 播放音频 + → 每句播放时 pub SubtitleTopic（或写 SSE queue）→ 浏览器字幕更新 +``` + +如需在浏览器端播放音频（例如远程听课场景），可在 Phase 2 评估增加音频流推送，但当前以服务端播放 + 浏览器字幕同步为主方案。 + +## 交互数据流 + +### 流 A：开讲（Phase 1-3 走 Queue，Phase 4 走 Topic） + +``` +前端 POST /api/mode {mode:"lecture", course:"xxx"} + → main.py → :9733/_internal/course/start_teaching + → LectureBrain.status = "loading" + → CourseManager.load_course() + start_teaching() + → _switch_to_chapter(0) → QUEUE.put(LoadChapterEvent) → Reflex 渲染 + → speaker_notes 注入 Ghost context + → Ghost 开始叙述 → TTS 启动 +``` + +### 流 B：段落推进 + 翻页 + +``` +Ghost 即兴叙述当前 active 要点 → TTS 播放 → 字幕同步 +Ghost 认为本段讲够了： + → 调 + → 有飞书消息就回复，没有就跳过 + → 调 + → LectureBrain：active→done，下一段→active + → 如本章全部 done：自动翻页（Ghost 不参与翻页决策） + → 返回 "point_advanced" / "chapter_advanced" / "lecture_ended" + → Ghost 继续讲下一段（或结束） + +兜底：若 advance_point 超时未调用（estimated_duration * 1.5）， + LectureBrain 自动推进并在 context 注入提示。 +``` + +**翻页决策权在 LectureBrain，不在 Ghost**。`advance_point` 内部判断是否翻页： + +```python +# advance_point 内部逻辑 +if 下一段 is None（本章全部 done）： + if 还有下一章： + _switch_to_chapter(current + 1) # Phase 4: pub ChapterLoadTopic + LectureBrain.current_chapter += 1，加载新的 talking_points + return "chapter_advanced" + else： + LectureBrain.status = "ended" + return "lecture_ended" +else： + return "point_advanced" +``` + +### 流 C：字幕同步 + +``` +TTS 播放每个句子： + → 写 SSE queue：SubtitleTopic(text="...", is_final=False) # 流式增量 + → 写 SSE queue：SubtitleTopic(text="...", is_final=True) # 本句结束 + → main.py SSE pipe → 浏览器 → 更新字幕区文本 +``` + +### 流 D：飞书弹幕 + 段落间隙回复 + +``` +飞书群消息到达 → Feishu Channel → Moshi 消费 + → 写 SSE queue：DanmakuTopic（弹幕，即时显示，不经过 Ghost） + → 飞书消息存入 feishu_window + → 弹幕频率控制：缓冲最近 50 条，超 20 条堆积时丢弃最旧非@消息 + +Ghost 讲完一个要点（advance_point 后，翻页前）： + → 调 + → 读 feishu_window → 批量回复 → 继续讲 +``` + +### 流 E：语音打断 + 恢复 + +``` +用户语音输入 → ASR → AudioSignal(SPEECH_FINAL, priority=WARNING) + → 当前讲课 Attention 被 abort + → LectureBrain.status = "paused" + → LectureBrain.questions.append({source:"voice", text:"..."}) + → TTS 停止（当前 active 段落保持 active） + + → Ghost 新 Attention：回答问题 + → context 包含： + 段落进度： + ✓ 要点1：核心用户场景 + → 要点2：小步实验验证 ← 继续这个 + … 要点3：沉淀指标看板 + + 已播字幕（当前 active 段落的最后 5 句）： + 1. "我们来谈谈增长策略中最重要的实验方法" + 2. "核心在于用最小成本验证最大假设" + 3. "举个例子，某电商平台通过A/B测试..." + + → Ghost 回答问题 + + → Ghost 调 + → context 同样提供段落进度 + 已播字幕 + → "请从第 3 句之后继续讲，不要重复已讲内容" + → Ghost 从断点后继续即兴叙述 + → TTS 恢复 → 字幕同步 +``` + +**恢复信息全部由已有数据派生，不额外存状态**： +- 段落进度：读 `talking_points[*].status`（`done` / `active` / `pending`） +- 已播字幕：读 SubtitleTopic 窗口最近 5 条 + +### 流 F：讲课结束 + 总结 + +``` +Ghost 讲完最后一章 + → LectureBrain.status = "ended" + → 收集所有飞书问题 + 语音打断问题 + → 生成总结（通过 chat_reply 返回给人类） +``` + +## 实施步骤 + +### Phase 1：单体讲课骨架（产出：可从头讲到尾） + +| # | 内容 | 涉及文件 | +|---|------|---------| +| 1.1 | 扩展章节数据结构：chapter_data 增加 speaker_notes 字段 | `course_manager.py`, `course_storage.py` | +| 1.2 | 实现 LectureBrain 状态机（idle→loading→lecturing→paused→ended） | `moss_in_reflex.py`（新建 `lecture_brain.py`） | +| 1.3 | 新增命令：`start_teaching`、`advance_point`、`check_messages`、`resume_lecture` | `moss_in_reflex.py` | +| 1.4 | 课程列表 API 对接 `CourseStorage.list_infos()` | `main.py` | +| 1.5 | 前端讲课页面（基于 `teachingL0.html` 改造：iframe + 字幕区 + 弹幕层 + 退出按钮） | `static/teachingL0.html` 或新建 | +| 1.6 | Ghost context 更新：teaching 模式下注入 speaker_notes + 段落进度 | `context_messages.py` | +| 1.7 | 字幕推送通道：SSE `/api/chat/stream/subtitle` | `main.py`, `moss_in_reflex.py` | + +### Phase 2：TTS + 字幕 + 弹幕（产出：有声音有互动） + +| # | 内容 | +|---|------| +| 2.1 | 接入 MOSS TTS 系统（Volcengine TTSManager），新增 `speak` 命令 | +| 2.2 | TTS 每句字幕 → SSE subtitle 推送 → 浏览器字幕同步 | +| 2.3 | 弹幕推送通道：SSE `/api/chat/stream/danmaku` | +| 2.4 | 飞书消息 → 弹幕（频率控制：缓冲 50 条，堆积 20+ 丢弃非@旧消息） | +| 2.5 | 段落间隙飞书回复（check_messages 实际对接 feishu channel） | + +### Phase 3：语音打断 + 总结（产出：完整讲课体验） + +| # | 内容 | +|---|------| +| 3.1 | Audio Signal 打断流程：识别 SPEECH_FINAL → LectureBrain pause | +| 3.2 | 打断后恢复：context 注入段落进度 + 已播字幕窗口 | +| 3.3 | 讲课结束总结：收集 questions → Ghost 生成总结 → chat_reply 返回 | +| 3.4 | 断线重连恢复：重连时从 LectureBrain 当前状态 + 当前章节恢复 | + +### Phase 4：Topic 桥接解耦（产出：架构清洁） + +| # | 内容 | +|---|------| +| 4.1 | 定义 5 个 Topic Model（ChapterLoad / PageSnapshot / Subtitle / Danmaku / LectureState） | +| 4.2 | Reflex Cell 改造：去掉课程逻辑，只保留渲染 + Topic 消费 | +| 4.3 | Moshi Cell 改造：去掉 Reflex 渲染代码，改用 Topic 发布 | +| 4.4 | save_chapter 改用 PageSnapshotTopic TopicWindow | +| 4.5 | _switch_to_chapter 改用 ChapterLoadTopic.pub() | +| 4.6 | Subtitle/Danmaku 改走 Topic，Reflex cell 内 topic_window 消费后 SSE 推送 | + +## 不改动的部分 + +- main.py HTTP proxy 基础结构（Phase 1-3 沿用，Phase 4 可能微调） +- Signal 对话环路（Phase 1 已完成） +- CourseManager / CourseStorage +- context_messages 模式感知机制 +- Reflex 布局系统（Layout、EventModel、build()） + +## Phase 1 实现记录（2026-06-21） + +### 已实现 + +| 组件 | 文件 | 说明 | +|------|------|------| +| 章节数据结构扩展 | `course_manager.py` | `save_chapter()` 新增 `speaker_notes` 参数；`get_speaker_notes()` | +| LectureBrain 状态机 | `lecture_brain.py`（新文件） | idle→loading→lecturing⇄paused→ended，超时检测，问题收集 | +| 讲课命令 | `moss_in_reflex.py` | `advance_point`（段落推进+自动翻页）、`check_messages`（桩）、`resume_lecture` | +| 课程列表 API | `moss_in_reflex.py` + `main.py` | `/_internal/courses` → `POST /api/courses` proxy | +| 讲课启动 API | `moss_in_reflex.py` + `main.py` | `/_internal/lecture/start`（原子编排）→ `POST /api/lecture/start` proxy | +| 前端讲课页面 | `teachingL0.html` | Reflex iframe + 字幕区 + 弹幕层 + 课程选择弹窗 + SSE | +| Ghost context | `context_messages.py` | teaching 模式展示 speaker_notes + 约束（不切布局、不重复 load）；preparing 模式提示生成 speaker_notes | + +### 架构决策 + +1. **编排权归服务端**：`/_internal/lecture/start` 原子完成 load + 布局 + 渲染 + 状态机初始化 + Signal，不经过 Ghost 理解层。 +2. **ChannelCtx 作用域**：aiohttp HTTP handler 无 ChannelCtx。在 `moss()` 初始化阶段捕获 `_course_storage`、`_registry` 等引用供内部 handler 使用。 +3. **Signal 唤醒 Ghost**：编排完成后用 `InputSignal`（非 Topic）通知 Ghost。Topic 不能唤醒 Ghost——这是 sensors 参考实现验证过的模式。 +4. **advance_point 后发 Signal 驱动连续叙述**：模型倾向于在 function call 后停止生成，COMMAND-RESULT 是 percept 不是 Signal，不能触发新 attention 周期。参考 `.moss_ws/apps/genkits/image` 的 `emit_generation_signal` 模式，`advance_point` 执行完毕后发 `InputSignal(Priority.NOTICE)`，携带当前 active 段落文本。`lecture_ended` 不发 Signal。 +5. **context 命令引用用运行时全名**：`PyChannel(name="moshi")` 在 app 体系下注册为 `apps.ui_moshi`，context 中所有 CTML 命令引用必须用全名。 +6. **teaching 模式隔离聊天功能**：`chat-instruction` 仅在 idle/discussing/preparing 模式注入，teaching 模式不注入——防止 Ghost 在讲课中自言自语调 `chat_reply`。 + +### Phase 1 调试实录（2026-06-21 第二轮） + +在端到端测试中发现并修复了以下问题，Phase 1 核心环路（叙述→推进→翻章→结束）最终跑通： + +| # | 问题 | 修复 | +|---|------|------| +| 1 | channel 名不匹配（`moshi` vs `apps.ui_moshi`）→ 每次命令调用先报错 | context 全部改为 `apps.ui_moshi`（6 处） | +| 2 | context-mode 与 lecture-state 重复展示 talking_points | 移除 context-mode 的 points 列表，只保留 lecture-state 的实时进度 | +| 3 | chat-instruction 对所有模式生效 → Ghost 讲课中自言自语 | `if _COURSE_MGR.mode != "teaching"` 守卫 | +| 4 | Ghost 输出 `<_>` 做段落分隔 → CTML 解析器崩溃 | context 加 CTML 卫生约束 | +| 5 | Ghost 调 advance_point 后静默 → function call 是回合终点 | advance_point 末尾发 InputSignal(Priority.NOTICE) | +| 6 | lecture_ended 后 Ghost 继续调命令 → 报 UNKNOWN_ERROR | advance_point 对 ENDED 状态幂等保护；context 加停止规则 | diff --git a/.gitignore b/.gitignore index 838ce764..811930a1 100644 --- a/.gitignore +++ b/.gitignore @@ -152,3 +152,4 @@ cython_debug/ # g1_sim downloadable binary assets .moss_ws/apps/bodies/g1_sim/assets/g1/meshes/ .moss_ws/apps/bodies/g1_sim/assets/policies/*.pt +.moss_ws/assets/courses/ diff --git a/.moss_ws/apps/ui/moshi/.gitignore b/.moss_ws/apps/ui/moshi/.gitignore new file mode 100644 index 00000000..0eb6e423 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/.gitignore @@ -0,0 +1,18 @@ +.web +__pycache__/ +*.db +.states +*.py[cod] +assets/external/ + +.venv/ +.pdm-build/ +.idea/ + +.env +.workspace/ + +debug.log +.codegraph/ +ign* +assets diff --git a/.moss_ws/apps/ui/moshi/.python-version b/.moss_ws/apps/ui/moshi/.python-version new file mode 100644 index 00000000..e4fba218 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/.moss_ws/apps/ui/moshi/APP.md b/.moss_ws/apps/ui/moshi/APP.md new file mode 100644 index 00000000..aa1ea529 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/APP.md @@ -0,0 +1,10 @@ +--- +arguments: '' +description: Moshi — AI 辅助内容交付产品，内嵌 Reflex 页面做展示，外挂控制面板 +executable: bash +respawn: false +script: start.sh +workers: 1 +--- + +Moshi — AI 辅助内容交付产品，内嵌 Reflex 页面做展示，外挂控制面板 \ No newline at end of file diff --git a/.moss_ws/apps/ui/moshi/CLAUDE.md b/.moss_ws/apps/ui/moshi/CLAUDE.md new file mode 100644 index 00000000..22cfc50c --- /dev/null +++ b/.moss_ws/apps/ui/moshi/CLAUDE.md @@ -0,0 +1,9 @@ +# MOSS Start (MANDATORY) + +At the beginning of EVERY conversation, BEFORE any other tool calls, file reads, or responses, run: + +``` +moss start +``` + +This is a blocking prerequisite. The output is the MOSS cognitive map — required context for all work in this project. Do not skip it. diff --git a/.moss_ws/apps/ui/moshi/README.md b/.moss_ws/apps/ui/moshi/README.md new file mode 100644 index 00000000..ee43bd4b --- /dev/null +++ b/.moss_ws/apps/ui/moshi/README.md @@ -0,0 +1,163 @@ +# Moss in Reflex + +基于 [Reflex](https://reflex.dev/) 构建的 MOSS 前端界面，通过 ZMQ 通道与 MOSS AI Agent 进行通信，实现动态布局渲染和流式内容更新。 + +## 简介 + +本项目是 ghoshell-moss 框架的 Reflex Web 前端适配层。它将 MOSS Agent 的消息事件（流式文本、列表操作、结构化数据、布局切换等）实时映射到 Reflex 组件状态上，从而在浏览器中动态展示 AI Agent 的输出。 + +核心思路：MOSS Agent 通过 ZMQ 发送事件 -> Reflex 后端接收并解析事件 -> 自动驱动前端组件状态更新 -> 浏览器实时渲染。 + +## 架构 + +``` +MOSS Agent (ZMQ) ---> Moss in Reflex (Reflex App) ---> Browser + | + asyncio.Queue + | + 事件分发 (Event Router) + | + +---------+---------+ + | | + Layout 切换 字段操作 + (SimpleLayout, (stream / set / + 可扩展...) append / pop / clear / pydantic) +``` + +### 关键模块 + +| 模块 | 说明 | +|------|------| +| `moss_in_reflex/moss_in_reflex.py` | 应用入口，包含 Reflex App、State、MOSS 通信和生命周期管理 | +| `framework/events.py` | 事件模型定义（`StreamEvent`、`SetEvent`、`AppendEvent`、`PopEvent`、`ClearEvent`、`LayoutEvent`） | +| `framework/layouts/simple.py` | 示例布局组件，展示标题、副标题、标签、段落和 Markdown 内容 | +| `framework/layouts/helpers/mixin.py` | 布局组件的 `NameMixin` 抽象基类 | +| `framework/runtime/event_generator.py` | 根据 State 类的类型注解动态生成事件处理方法 | +| `examples/run_with_simple_agent.py` | 示例：启动一个 MOSS SimpleAgent 并通过 ZMQ 连接到本前端 | + +## 事件类型 + +| 事件 | 说明 | +|------|------| +| `LayoutEvent` | 切换当前布局 | +| `StreamEvent` | 流式追加文本到 `str` 类型字段 | +| `SetEvent` | 设置任意字段值（支持 JSON 设置 BaseModel 字段） | +| `AppendEvent` | 向 `list` 类型字段追加元素（支持 JSON 追加 dict/BaseModel 元素） | +| `PopEvent` | 弹出 `list` 类型字段的最后一个元素 | +| `ClearEvent` | 清空指定字段 | + +## 支持的字段类型 + +| 字段类型 | 自动生成的方法 | 说明 | +|---------|--------------|------| +| `str` | `stream_`, `clear_` | 流式文本追加、清空 | +| `list[str]` | `push_`, `pop_`, `clear_`, `stream_append_` | 列表操作 + 流式追加到最后一个元素 | +| `list[T]`（T 为 dict 或 BaseModel） | `push_`, `pop_`, `clear_` | 通过 JSON 操作列表 | +| `BaseModel` 子类 | `set_`, `clear_` | 通过 JSON 设置/重置结构化数据 | + +## 媒体与图表组件 + +布局中可使用以下 Reflex 内置组件渲染富媒体内容： + +- **图片**：`rx.image(src=url)` — 展示 URL 或 assets 路径的图片 +- **视频**：`rx.video(url=url)` — 嵌入视频播放器 +- **图表**：`rx.recharts.*` — 折线图、柱状图、饼图等，数据字段通常为 `list[dict]` + +## 快速开始 + +### 环境要求 + +- Python >= 3.12 +- uv 或 pdm 包管理器 + +### 安装 + +```bash +uv venv +source .venv/bin/activate +uv sync +``` + +### 运行前端 + +```bash +reflex run +``` + +启动后访问 `http://localhost:3000` 查看界面。 + +### 运行示例 Agent + +在另一个终端中运行： + +```bash +python examples/run_with_simple_agent.py +``` + +该示例会启动一个 MOSS SimpleAgent，通过 ZMQ（`tcp://127.0.0.1:9528`）与 Reflex 前端通信。 + +## 扩展布局 + +1. 在 `framework/layouts/` 下创建新的布局类，继承 `rx.ComponentState` 和 `NameMixin`： + +```python +import reflex as rx +from pydantic import BaseModel, Field +from framework.layouts.helpers.mixin import NameMixin + +# 可选：定义 Pydantic 模型用于结构化数据 +class CardData(BaseModel): + title: str = Field(default="") + image_url: str = Field(default="") + description: str = Field(default="") + +class MyLayout(rx.ComponentState, NameMixin): + """My custom layout description.""" + content: str = "" + images: list[str] = [] + card: CardData = CardData() + chart_data: list[dict] = [] + + @classmethod + def name(cls) -> str: + return "my_layout" + + @classmethod + def get_component(cls, **props) -> rx.Component: + return rx.vstack( + rx.text(cls.content), + rx.foreach(cls.images, lambda url: rx.image(src=url, width="200px")), + rx.heading(cls.card.title), + rx.text(cls.card.description), + rx.recharts.line_chart( + rx.recharts.line(data_key="value"), + rx.recharts.x_axis(data_key="name"), + data=cls.chart_data, + width="100%", height=300, + ), + **props, + ) +``` + +2. 在 `moss_in_reflex/layouts.toml` 的 `layouts` 数组中注册： + +```toml +layouts = [ + "framework.layouts.simple.SimpleLayout", + "framework.layouts.my_layout.MyLayout", +] +``` + +框架会自动为新布局的 State 字段生成对应的事件处理方法。 + +## 技术栈 + +- **Reflex** - Python 全栈 Web 框架 +- **ghoshell-moss** - MOSS AI Agent 框架 +- **ZMQ (ZeroMQ)** - 进程间通信 +- **Pydantic** - 数据模型验证 +- **Tailwind CSS v4** - 样式（通过 Reflex 插件集成） + +## License + +Apache License 2.0 diff --git a/.moss_ws/apps/ui/moshi/framework/__init__.py b/.moss_ws/apps/ui/moshi/framework/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/.moss_ws/apps/ui/moshi/framework/components/__init__.py b/.moss_ws/apps/ui/moshi/framework/components/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/.moss_ws/apps/ui/moshi/framework/events.py b/.moss_ws/apps/ui/moshi/framework/events.py new file mode 100644 index 00000000..487e6685 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/framework/events.py @@ -0,0 +1,76 @@ +import asyncio +from dataclasses import dataclass, field as dc_field +from typing import Any, ClassVar + +from ghoshell_common.helpers import uuid + + +@dataclass(kw_only=True) +class EventModel: + event_type: ClassVar[str] = "" + + event_id: str = dc_field(default_factory=uuid) + future: asyncio.Future | None = dc_field(default=None, repr=False) + + +@dataclass(kw_only=True) +class LayoutEvent(EventModel): + event_type: ClassVar[str] = "set_layout" + + layout: str = "" + + +@dataclass(kw_only=True) +class StreamEvent(EventModel): + event_type: ClassVar[str] = "stream" + + field: str = "" + chunk: str = "" + + +@dataclass(kw_only=True) +class SetEvent(EventModel): + event_type: ClassVar[str] = "set" + + field: str = "" + data: Any = None + + +@dataclass(kw_only=True) +class AppendEvent(EventModel): + event_type: ClassVar[str] = "append" + + field: str = "" + data: Any = None + + +@dataclass(kw_only=True) +class UpdateEvent(EventModel): + event_type: ClassVar[str] = "update" + + field: str = "" + index: int = 0 + data: Any = None + + +@dataclass(kw_only=True) +class PopEvent(EventModel): + event_type: ClassVar[str] = "pop" + + field: str = "" + + +@dataclass(kw_only=True) +class ClearEvent(EventModel): + event_type: ClassVar[str] = "clear" + + field: str = "" + + +@dataclass(kw_only=True) +class LoadChapterEvent(EventModel): + """原子替换当前章节全部字段。chapter_data 包含 sub_title/main_text/annotations/appreciation/images。""" + + event_type: ClassVar[str] = "load_chapter" + + chapter_data: dict = dc_field(default_factory=dict) diff --git a/.moss_ws/apps/ui/moshi/framework/helpers/__init__.py b/.moss_ws/apps/ui/moshi/framework/helpers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/.moss_ws/apps/ui/moshi/framework/helpers/layout_snapshot.py b/.moss_ws/apps/ui/moshi/framework/helpers/layout_snapshot.py new file mode 100644 index 00000000..e66c4dbb --- /dev/null +++ b/.moss_ws/apps/ui/moshi/framework/helpers/layout_snapshot.py @@ -0,0 +1,149 @@ +"""Layout state snapshot — 每个 layout 独立管理，支持持久化和版本控制。 + +通过 Reflex get_state API 一把读取 ComponentState 所有字段值， +写入内存快照并可选持久化到 JSON 文件。 +""" + +import json +import logging +import typing +from pathlib import Path + +import reflex as rx +from PIL import Image + +logger = logging.getLogger(__name__) + +SNAPSHOT_VERSION = 1 + + +class LayoutSnapshot: + """单个 layout 的状态快照管理器。""" + + def __init__(self, name: str, state_class: type[rx.State], storage_dir: Path) -> None: + self._name = name + self._state_class = state_class + self._storage_path = storage_dir / name / "snapshot.json" + self._data: dict[str, object] = {} + self._data_full: dict[str, object] = {} + self._init_defaults() + + # ---- public ---- + + async def refresh(self, root_state: rx.State) -> None: + """通过 root_state.get_state() 一把读取所有字段运行时值。""" + try: + layout_state = await root_state.get_state(self._state_class) + except Exception: + logger.warning("get_state failed for layout %r", self._name, exc_info=True) + return + + for name, type_hint in self._state_class.__annotations__.items(): + val = getattr(layout_state, name, None) + self._data_full[name] = self._serialize_full(val, type_hint) + self._data[name] = self._summarize(val, type_hint) + + def get(self) -> dict[str, object]: + return self._data + + def get_full(self) -> dict[str, object]: + """返回全量数据（不截断），供 save_chapter 等持久化场景使用。""" + return dict(self._data_full) + + def save(self) -> None: + self._storage_path.parent.mkdir(parents=True, exist_ok=True) + payload = {"version": SNAPSHOT_VERSION, "data": self._data_full} + self._storage_path.write_text( + json.dumps(payload, ensure_ascii=False, default=str) + ) + + # ---- internal ---- + + def _init_defaults(self) -> None: + for name, type_hint in self._state_class.__annotations__.items(): + self._data[name] = _default_for(type_hint) + self._data_full[name] = _default_for(type_hint) + + @staticmethod + def _serialize_full(val: object, type_hint: type) -> object: + """将运行时值序列化为全量数据（不截断），供持久化使用。 + + - str → 原样（不截断） + - list[str] → 全量列表 + - list[Image] → 仅记数量（locator 由 Ghost 在 save_chapter 时传入） + - Image.Image → 哨兵值 1（用于校验数量） + - None → 按类型给默认哨兵值 + """ + if val is None: + return _default_for(type_hint) + if isinstance(val, str): + return val + if isinstance(val, list): + args = typing.get_args(type_hint) + elem_type = args[0] if args else None + if elem_type is Image.Image: + return len(val) + return list(val) + if isinstance(val, Image.Image): + return 1 + return val + + @staticmethod + def _summarize(val: object, type_hint: type) -> object: + """将运行时值压缩为 token 友好的摘要。 + + 阶段规则： + - None → 按类型给默认哨兵值 + - str → 截断 200 字符 + - list[str] → 前 5 项，每项 ≤60 字符，总量 ≤300 字符 + - list[Image] → 前 5 项，每项报尺寸 + - 其他 list → 仅报长度 + - Image.Image → 报尺寸 + - 其他 → str() 后截断 200 字符 + """ + if val is None: + return _default_for(type_hint) + if isinstance(val, str): + return val[:200] + "..." if len(val) > 200 else val + if isinstance(val, list): + return _summarize_list(val, type_hint) + if isinstance(val, Image.Image): + return f"" + return str(val)[:200] + + +def _summarize_list(val: list, type_hint: type) -> object: + """根据元素类型选择列表摘要策略。""" + args = typing.get_args(type_hint) + elem_type = args[0] if args else None + if elem_type is str: + return _summarize_str_list(val) + if elem_type is Image.Image: + return f"" + return len(val) + + +def _summarize_str_list(val: list[str]) -> str: + """list[str]：展示前 5 项内容，每项 ≤60 字符，总量 ≤300 字符。""" + parts: list[str] = [] + for item in val[:5]: + s = str(item) + parts.append(s[:60] + "..." if len(s) > 60 else s) + result = ", ".join(parts) + if len(val) > 5: + result += f", ... (+{len(val) - 5} more)" + if len(result) > 300: + result = result[:300] + "..." + return result + + +def _default_for(type_hint: type) -> object: + """类型对应的默认哨兵值（refresh 前使用）。""" + origin = typing.get_origin(type_hint) + if origin is list: + return 0 + if type_hint is str: + return "" + if type_hint is Image.Image: + return "" + return "(unknown)" diff --git a/.moss_ws/apps/ui/moshi/framework/helpers/mixin.py b/.moss_ws/apps/ui/moshi/framework/helpers/mixin.py new file mode 100644 index 00000000..2f2b8f79 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/framework/helpers/mixin.py @@ -0,0 +1,8 @@ +import abc + + +class NameMixin(abc.ABC): + + @abc.abstractmethod + def name(self) -> str: + pass \ No newline at end of file diff --git a/.moss_ws/apps/ui/moshi/framework/layouts/__init__.py b/.moss_ws/apps/ui/moshi/framework/layouts/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/.moss_ws/apps/ui/moshi/framework/layouts/lesson.py b/.moss_ws/apps/ui/moshi/framework/layouts/lesson.py new file mode 100644 index 00000000..fe5e24fd --- /dev/null +++ b/.moss_ws/apps/ui/moshi/framework/layouts/lesson.py @@ -0,0 +1,83 @@ +import reflex as rx +from PIL import Image + +from framework.helpers.mixin import NameMixin + + +class LessonLayout(rx.ComponentState, NameMixin): + """语文备课布局：课程标题、作者信息、课文插图、正文、注释、赏析。""" + + title: str = "" + sub_title: str = "" + image: list[Image.Image] = [] + main_text: str = "" + annotations: list[str] = [] + appreciation: str = "" + + @classmethod + def name(cls) -> str: + return "lesson" + + @classmethod + def get_component(cls, **props) -> rx.Component: + return rx.vstack( + # ── 课程标题 ── + rx.skeleton( + rx.heading(cls.title, size="8"), + width="500px", + height="36px", + loading=cls.title == "", + ), + # ── 副标题（作者/朝代）── + rx.skeleton( + rx.text(cls.sub_title, color_scheme="gray", size="3"), + width="300px", + height="20px", + loading=cls.sub_title == "", + ), + rx.divider(), + # ── 课文插图 ── + rx.skeleton( + rx.hstack( + rx.foreach( + cls.image, + lambda img: rx.image(img, width="400px", height="auto", border_radius="8px"), + ), + ), + width="400px", + height="250px", + loading=cls.image.length() == 0, + ), + # ── 正文 ── + rx.skeleton( + rx.markdown(cls.main_text), + width="800px", + height="150px", + loading=cls.main_text == "", + ), + # ── 注释 ── + rx.skeleton( + rx.box( + rx.heading("注释", size="5"), + rx.foreach( + cls.annotations, + lambda text, idx: rx.text( + rx.text.span(f"[{idx + 1}] ", color_scheme="blue"), + text, + size="2", + ), + ), + ), + width="800px", + height="100px", + loading=cls.annotations.length() == 0, + ), + # ── 赏析 ── + rx.skeleton( + rx.markdown(cls.appreciation), + width="800px", + height="100px", + loading=cls.appreciation == "", + ), + **props, + ) diff --git a/.moss_ws/apps/ui/moshi/framework/layouts/media.py b/.moss_ws/apps/ui/moshi/framework/layouts/media.py new file mode 100644 index 00000000..048de176 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/framework/layouts/media.py @@ -0,0 +1,50 @@ +import reflex as rx +from PIL import Image + +from framework.helpers.mixin import NameMixin + + +class MediaLayout(rx.ComponentState, NameMixin): + """文字 + 媒体混排布局。 + + 文字部分（title / sub_title / tags / paragraph / markdown）复用 simple 骨架渲染模式， + 媒体部分（media: list[MediaItem]）按槽位展示，支持 pending / ready / failed 三种状态。 + """ + + title: str = "" + sub_title: str = "" + image: list[Image.Image] = [] + + @classmethod + def name(cls) -> str: + return "media" + + @classmethod + def get_component(cls, **props) -> rx.Component: + return rx.vstack( + # ── 文字区域（复用 simple 骨架模式）── + rx.skeleton( + rx.heading(cls.title, size="7"), + width="500px", + height="30px", + loading=cls.title == "", + ), + rx.skeleton( + rx.text(cls.sub_title, color_scheme="gray"), + width="400px", + height="25px", + loading=cls.sub_title == "", + ), + # ── 媒体区域 ── + rx.skeleton( + rx.hstack( + rx.foreach( + cls.image, lambda i: rx.image(i, width="400px", height="auto"), + ), + ), + width="200px", + height="200px", + loading=cls.image.length() == 0 + ), + **props, + ) diff --git a/.moss_ws/apps/ui/moshi/framework/layouts/simple.py b/.moss_ws/apps/ui/moshi/framework/layouts/simple.py new file mode 100644 index 00000000..90441714 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/framework/layouts/simple.py @@ -0,0 +1,59 @@ +import reflex as rx + +from framework.helpers.mixin import NameMixin + + +class SimpleLayout(rx.ComponentState, NameMixin): + """ + A Simple Layout + """ + title: str = "" + sub_title: str = "" + tags: list[str] = [] + paragraph: list[str] = [] + markdown: str = "" + + @classmethod + def name(cls) -> str: + return "simple" + + @classmethod + def get_component(cls, **props) -> rx.Component: + return rx.vstack( + rx.skeleton( + rx.heading(cls.title, size="7"), + width="500px", + height="30px", + loading=cls.title=="", + ), + rx.skeleton( + rx.text(cls.sub_title, color_scheme="gray"), + width="400px", + height="25px", + loading=cls.sub_title=="", + ), + rx.skeleton( + rx.hstack( + rx.foreach(cls.tags, rx.badge), + spacing="1", + ), + width="400px", + height="25px", + loading=cls.tags.length()==0, + ), + rx.skeleton( + rx.foreach( + cls.paragraph, rx.text + ), + width="800px", + height="100px", + loading=cls.paragraph.length()==0, + ), + rx.skeleton( + rx.markdown(cls.markdown), + width="800px", + height="100px", + loading=cls.markdown == "", + ), + **props, + ) diff --git a/.moss_ws/apps/ui/moshi/framework/runtime/__init__.py b/.moss_ws/apps/ui/moshi/framework/runtime/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/.moss_ws/apps/ui/moshi/framework/runtime/event_generator.py b/.moss_ws/apps/ui/moshi/framework/runtime/event_generator.py new file mode 100644 index 00000000..52aa84f2 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/framework/runtime/event_generator.py @@ -0,0 +1,364 @@ +import asyncio +import inspect +import json +import typing + +import pydantic +import reflex as rx +from PIL import Image +from ghoshell_moss import CommandError, CommandErrorCode, new_channel_state +from ghoshell_moss.contracts import ResourceRegistry +from ghoshell_moss.core import PyCommand, ChannelCtx + +from framework.events import StreamEvent, ClearEvent, AppendEvent, PopEvent, UpdateEvent, SetEvent, LayoutEvent, EventModel + +_CMD_TIMEOUT = 5.0 # 非流式命令等待 UI 处理的超时秒数 + + +async def _await_event(queue: asyncio.Queue, event: EventModel, timeout: float = _CMD_TIMEOUT) -> None: + """塞 future 进事件，入队后同步等待 UI 侧 resolve/reject。""" + fut = asyncio.get_event_loop().create_future() + event.future = fut + await queue.put(event) + try: + await asyncio.wait_for(fut, timeout=timeout) + except TimeoutError: + raise CommandError( + code=CommandErrorCode.FAILED, + message="UI 处理超时", + ) + except Exception as e: + raise CommandError( + code=CommandErrorCode.FAILED, + message=f"UI 处理失败: {e}", + ) from e + + +def build(state_name, state_class: type[rx.State], layout_state_class: type[rx.ComponentState], queue: asyncio.Queue): + commands = [] + annotations = state_class.__annotations__ + for name, type_ in annotations.items(): + ori_type = typing.get_origin(type_) + if str in [type_, ori_type]: + commands.extend( + generate_stream_command(name, state_class, queue) + ) + elif list in [type_, ori_type]: + elem_type = typing.get_args(type_)[0] + commands.extend( + generate_list_command(name, state_class, elem_type, queue) + ) + elif issubclass(type_, pydantic.BaseModel): + commands.extend( + generate_pydantic_model_command(name, state_class, type_, queue) + ) + elif Image.Image in [type_, ori_type]: + commands.extend( + generate_image_command(name, state_class, queue) + ) + + layout_doc = inspect.getdoc(layout_state_class) or "" + state = new_channel_state(name=state_name, description=layout_doc) + + async def state_startup(): + await queue.put(LayoutEvent(layout=state_name)) + + state.startup(state_startup) + for command in commands: + state.add_command(command) + return state + + +def generate_stream_command(name: str, state_class: type[rx.State], queue: asyncio.Queue): + # 1. 定义函数体逻辑 + async def stream(state: state_class, t: str): # type: ignore[valid-type] + # 动态给 state 的属性赋值 + setattr(state, name, getattr(state, name) + t) + + async def clear(state: state_class): # type: ignore[valid-type] + setattr(state, name, "") + + # 2. 动态修改函数名（符合 stream_{title} 格式） + stream.__name__ = stream.__qualname__ = f"stream_{name}" + clear.__name__ = clear.__qualname__ = f"clear_{name}" + + # 3. 用 rx.event 装饰并返回 + stream_decorated = rx.event(stream) + clear_decorated = rx.event(clear) + setattr(state_class, f"stream_{name}", stream_decorated) + setattr(state_class, f"clear_{name}", clear_decorated) + + async def stream_command(chunks__): + async for chunk in chunks__: + await queue.put(StreamEvent(field=name, chunk=chunk)) + + async def clear_command(): + await _await_event(queue, ClearEvent(field=name)) + + return [ + PyCommand( + func=stream_command, + name=f"stream_{name}", + doc=f"流式渲染{name}字段", + ), + PyCommand( + func=clear_command, + name=f"clear_{name}", + doc=f"清空{name}字段" + ) + ] + + +def generate_list_command(name: str, state_class: type[rx.State], field_type, queue: asyncio.Queue): + async def append(state: state_class, t: field_type): # type: ignore[valid-type] + if issubclass(field_type, pydantic.BaseModel): + t = field_type.model_validate_json(t) + if issubclass(field_type, dict): + t = json.loads(t) + getattr(state, name).append(t) + + async def pop(state: state_class): # type: ignore[valid-type] + ll = getattr(state, name) + if len(ll) == 0: + return + getattr(state, name).pop() + + async def clear(state: state_class): # type: ignore[valid-type] + getattr(state, name).clear() + + async def stream_append(state: state_class, t: str): # type: ignore[valid-type] + l = getattr(state, name) + l[-1] += t + + async def update(state: state_class, index: int, t: field_type): # type: ignore[valid-type] + if issubclass(field_type, pydantic.BaseModel): + t = field_type.model_validate_json(t) + if issubclass(field_type, dict): + t = json.loads(t) + l = getattr(state, name) + if index >= len(l): + l.append(t) + else: + l[index] = t + + # 2. 动态修改函数名（符合 stream_{title} 格式） + append.__name__ = append.__qualname__ = f"append_{name}" + pop.__name__ = pop.__qualname__ = f"pop_{name}" + clear.__name__ = clear.__qualname__ = f"clear_{name}" + stream_append.__name__ = stream_append.__qualname__ = f"stream_{name}" + update.__name__ = update.__qualname__ = f"update_{name}" + + append_decorated = rx.event(append) + pop_decorated = rx.event(pop) + clear_decorated = rx.event(clear) + stream_last_decorated = rx.event(stream_append) + update_decorated = rx.event(update) + + setattr(state_class, f"append_{name}", append_decorated) + setattr(state_class, f"pop_{name}", pop_decorated) + setattr(state_class, f"clear_{name}", clear_decorated) + + if field_type == str: + setattr(state_class, f"stream_{name}", stream_last_decorated) + + # update 仅对结构化元素（BaseModel / dict）有意义，str 列表无 id 概念 + if issubclass(field_type, (pydantic.BaseModel, dict)): + setattr(state_class, f"update_{name}", update_decorated) + + async def append_command(text__): + # 数据结构校验 + try: + json.loads(text__) + except json.decoder.JSONDecodeError as e: + raise CommandError( + code=CommandErrorCode.FAILED, + message=e.msg, + ) + await _await_event(queue, AppendEvent(field=name, data=text__)) + + async def pop_command(): + await _await_event(queue, PopEvent(field=name)) + + async def clear_command(): + await _await_event(queue, ClearEvent(field=name)) + + async def update_command(index: int, text__): + try: + json.loads(text__) + except json.decoder.JSONDecodeError as e: + raise CommandError( + code=CommandErrorCode.FAILED, + message=e.msg, + ) + await _await_event(queue, UpdateEvent(field=name, index=index, data=text__)) + + async def stream_command(chunks__): + # 先创建一个最新的元素 + await queue.put(AppendEvent(field=name, data="")) + + # 流式追加内容到最新的元素 + async for chunk in chunks__: + await queue.put(StreamEvent(field=name, chunk=chunk)) + + async def append_image_command(locator: str): + registry = ChannelCtx.container().force_fetch(ResourceRegistry) + item = await registry.get(locator) + if not item: + raise CommandError( + code=CommandErrorCode.FAILED, + message=f"{locator} not found", + ) + image = await item.get() + if not isinstance(image, Image.Image): + raise CommandError( + code=CommandErrorCode.FAILED, + message=f"{locator} is not an image", + ) + + await _await_event(queue, AppendEvent(field=name, data=image)) + + commands = [ + PyCommand( + func=pop_command, + name=f"pop_{name}", + doc=f"弹出{name}列表最后一个元素", + ), + PyCommand( + func=clear_command, + name=f"clear_{name}", + doc=f"清空{name}列表", + ), + ] + + if field_type == str: + commands.append(PyCommand( + func=stream_command, + name=f"stream_{name}", + doc=f"追加一个元素到{name}，并流式渲染", + )) + if issubclass(field_type, pydantic.BaseModel): + commands.extend([ + PyCommand( + func=update_command, + name=f"update_{name}", + doc=f"更新{name}列表中指定索引的元素，字段格式为{field_type.model_json_schema()}", + ), + PyCommand( + func=append_command, + name=f"append_{name}", + doc=f"追加元素到{name}列表，字段格式为{field_type.model_json_schema()}", + ), + ]) + if field_type == dict: + commands.extend([ + PyCommand( + func=update_command, + name=f"update_{name}", + doc=f"更新{name}列表中指定索引的元素", + ), + PyCommand( + func=append_command, + name=f"append_{name}", + doc=f"追加元素到{name}列表", + ), + ]) + if field_type == Image.Image: + commands.extend([ + PyCommand( + func=append_image_command, + name=f"append_{name}", + doc=f"追加渲染一张图片，使用locator格式的字符串" + ) + ]) + + return commands + + +def generate_pydantic_model_command(name: str, state_class: type[rx.State], field_type: type[pydantic.BaseModel], queue: asyncio.Queue): + # 1. 定义函数体逻辑 + async def set_(state: state_class, t: str): # type: ignore[valid-type] + # 动态给 state 的属性赋值 + model = field_type.model_validate_json(t) + setattr(state, name, model) + + async def clear(state: state_class): # type: ignore[valid-type] + setattr(state, name, field_type()) + + # 2. 动态修改函数名（符合 stream_{title} 格式） + set_.__name__ = set_.__qualname__ = f"set_{name}" + clear.__name__ = clear.__qualname__ = f"clear_{name}" + + # 3. 用 rx.event 装饰并返回 + set_decorated = rx.event(set_) + clear_decorated = rx.event(clear) + setattr(state_class, f"set_{name}", set_decorated) + setattr(state_class, f"clear_{name}", clear_decorated) + + async def set_command(t: str): + model = field_type.model_validate_json(t) + await _await_event(queue, SetEvent(field=name, data=model)) + + async def clear_command(): + await _await_event(queue, ClearEvent(field=name)) + + return [ + PyCommand( + func=set_command, + name=f"set_{name}", + doc=f"设置{name}字段，字段格式为{field_type.model_json_schema()}", + ), + PyCommand( + func=clear_command, + name=f"clear_{name}", + doc=f"清空{name}字段" + ) + ] + + +def generate_image_command(name: str, state_class: type[rx.State], queue: asyncio.Queue): + async def set_(state: state_class, t: Image.Image): # type: ignore[valid-type] + setattr(state, name, t) + + async def clear(state: state_class): # type: ignore[valid-type] + setattr(state, name, None) + + set_.__name__ = set_.__qualname__ = f"set_{name}" + clear.__name__ = clear.__qualname__ = f"clear_{name}" + + set_decorated = rx.event(set_) + clear_decorated = rx.event(clear) + setattr(state_class, f"set_{name}", set_decorated) + setattr(state_class, f"clear_{name}", clear_decorated) + + async def set_command(locator: str): + registry = ChannelCtx.container().force_fetch(ResourceRegistry) + item = await registry.get(locator) + if not item: + raise CommandError( + code=CommandErrorCode.FAILED, + message=f"{locator} not found", + ) + image = await item.get() + if not isinstance(image, Image.Image): + raise CommandError( + code=CommandErrorCode.FAILED, + message=f"{locator} is not an image", + ) + + await _await_event(queue, SetEvent(field=name, data=image)) + + async def clear_command(): + await _await_event(queue, ClearEvent(field=name)) + + return [ + PyCommand( + func=set_command, + name=f"set_{name}", + doc=f"渲染一张图片，使用locator格式的字符串", + ), + PyCommand( + func=clear_command, + name=f"clear_{name}", + doc=f"清空{name}字段" + ) + ] diff --git a/.moss_ws/apps/ui/moshi/main.py b/.moss_ws/apps/ui/moshi/main.py new file mode 100644 index 00000000..d046e639 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/main.py @@ -0,0 +1,245 @@ +"""Moshi — AI 辅助内容交付产品原型。 + +Web 服务层：静态文件 + REST API。 +不直接连接 Matrix，而是通过 moss() 的内部 HTTP 桥接 (:9733) 转发聊天消息。 +""" + +import json +import logging +from pathlib import Path + +import aiohttp +from aiohttp import web + +logger = logging.getLogger("moshi") + +HOST = "127.0.0.1" +PORT = 9731 +REFLEX_URL = "http://localhost:3000" +INTERNAL_BRIDGE = "http://127.0.0.1:9733" +STATIC_DIR = Path(__file__).resolve().parent / "static" + + +# ── API Handlers ────────────────────────────────────────────────────────── + +async def handle_chat(request: web.Request) -> web.Response: + """转发用户消息到 moss() 内部桥接 → 发 Signal 给 Ghost。""" + data = await request.json() + text = (data.get("text") or "").strip() + mode = data.get("mode", "idle") + if not text: + return web.json_response({"error": "empty text"}, status=400) + + logger.info("[proxy] 转发 chat_in → 内部桥接, text=%.60s", text) + try: + async with aiohttp.ClientSession() as session: + async with session.post( + f"{INTERNAL_BRIDGE}/_internal/chat_in", + json={"text": text, "mode": mode}, + timeout=aiohttp.ClientTimeout(total=5), + ) as resp: + if resp.status != 200: + body = await resp.text() + logger.warning("[proxy] chat_in 失败: %s", body) + return web.json_response({"error": body}, status=502) + logger.info("[proxy] chat_in 转发成功") + return web.json_response({"ok": True}) + except aiohttp.ClientError as e: + logger.error("[proxy] chat_in 连接失败: %s", e) + return web.json_response({"error": "内部桥接不可用"}, status=503) + + +async def handle_chat_stream(request: web.Request) -> web.StreamResponse: + """转发 moss() 内部桥接的 SSE 流到浏览器。""" + resp = web.StreamResponse( + status=200, + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + "Connection": "keep-alive", + }, + ) + await resp.prepare(request) + + logger.info("[proxy] SSE pipe 已连接 → 内部桥接") + try: + async with aiohttp.ClientSession() as session: + async with session.get( + f"{INTERNAL_BRIDGE}/_internal/chat_stream", + timeout=aiohttp.ClientTimeout(total=None, sock_read=None), + ) as upstream: + async for line in upstream.content: + await resp.write(line) + except aiohttp.ClientError as e: + logger.info("[proxy] SSE pipe 断开: %s", e) + except (ConnectionResetError, ConnectionAbortedError): + pass + + return resp + + +async def handle_subtitle_stream(request: web.Request) -> web.StreamResponse: + """转发 moss() 内部桥接的字幕 SSE 流到浏览器。""" + resp = web.StreamResponse( + status=200, + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + "Connection": "keep-alive", + }, + ) + await resp.prepare(request) + + logger.info("[proxy] 字幕 SSE pipe 已连接 → 内部桥接") + try: + async with aiohttp.ClientSession() as session: + async with session.get( + f"{INTERNAL_BRIDGE}/_internal/subtitle_stream", + timeout=aiohttp.ClientTimeout(total=None, sock_read=None), + ) as upstream: + async for line in upstream.content: + await resp.write(line) + except aiohttp.ClientError as e: + logger.info("[proxy] 字幕 SSE pipe 断开: %s", e) + except (ConnectionResetError, ConnectionAbortedError): + pass + + return resp + + +async def handle_danmaku_stream(request: web.Request) -> web.StreamResponse: + """转发 moss() 内部桥接的弹幕 SSE 流到浏览器。""" + resp = web.StreamResponse( + status=200, + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + "Connection": "keep-alive", + }, + ) + await resp.prepare(request) + + logger.info("[proxy] 弹幕 SSE pipe 已连接 → 内部桥接") + try: + async with aiohttp.ClientSession() as session: + async with session.get( + f"{INTERNAL_BRIDGE}/_internal/danmaku_stream", + timeout=aiohttp.ClientTimeout(total=None, sock_read=None), + ) as upstream: + async for line in upstream.content: + await resp.write(line) + except aiohttp.ClientError as e: + logger.info("[proxy] 弹幕 SSE pipe 断开: %s", e) + except (ConnectionResetError, ConnectionAbortedError): + pass + + return resp + + +async def handle_state(request: web.Request) -> web.Response: + return web.json_response({"mode": "idle", "status": "就绪"}) + + +async def handle_courses(request: web.Request) -> web.Response: + """从内部桥接获取课程列表（代理 moss() 的 /_internal/courses）。""" + try: + async with aiohttp.ClientSession() as session: + async with session.get( + f"{INTERNAL_BRIDGE}/_internal/courses", + timeout=aiohttp.ClientTimeout(total=5), + ) as resp: + if resp.status != 200: + body = await resp.text() + logger.warning("[proxy] courses 失败: %s", body) + return web.json_response([], status=502) + data = await resp.json() + return web.json_response(data) + except aiohttp.ClientError as e: + logger.error("[proxy] courses 连接失败: %s", e) + return web.json_response([], status=503) + + +async def handle_mode(request: web.Request) -> web.Response: + data = await request.json() + mode = data.get("mode", "idle") + return web.json_response({ + "mode": mode, + "status": f"切换到{mode}", + "msg": f"切换到{mode}模式", + }) + + +async def handle_course_select(request: web.Request) -> web.Response: + data = await request.json() + name = data.get("name", "") + return web.json_response({ + "mode": "idle", + "status": f"已加载: {name}", + "msg": f"已加载: {name}", + }) + + +async def handle_lecture_start(request: web.Request) -> web.Response: + """原子编排讲课启动：转发到 moss() 内部桥接。""" + data = await request.json() + course = (data.get("course") or "").strip() + if not course: + return web.json_response({"error": "course is required"}, status=400) + + logger.info("[proxy] 转发 lecture_start → 内部桥接, course=%s", course) + try: + async with aiohttp.ClientSession() as session: + async with session.post( + f"{INTERNAL_BRIDGE}/_internal/lecture/start", + json={"course": course}, + timeout=aiohttp.ClientTimeout(total=15), + ) as resp: + result = await resp.json() + if resp.status != 200: + logger.warning("[proxy] lecture_start 失败: %s", result) + return web.json_response(result, status=502) + logger.info("[proxy] lecture_start 成功: %s", result) + return web.json_response(result) + except aiohttp.ClientError as e: + logger.error("[proxy] lecture_start 连接失败: %s", e) + return web.json_response({"error": "内部桥接不可用"}, status=503) + + +async def handle_cmd(request: web.Request) -> web.Response: + data = await request.json() + cmd = data.get("cmd", "") + return web.json_response({"msg": f"命令 {cmd} 已发送"}) + + +# ── App ─────────────────────────────────────────────────────────────────── + +def create_app() -> web.Application: + app = web.Application() + app.router.add_static("/static/", path=str(STATIC_DIR), name="static") + app.router.add_get("/", lambda r: web.FileResponse(STATIC_DIR / "prepareL0.html")) + app.router.add_get("/lecture", lambda r: web.FileResponse(STATIC_DIR / "teachingL0.html")) + app.router.add_post("/api/state", handle_state) + app.router.add_post("/api/courses", handle_courses) + app.router.add_post("/api/mode", handle_mode) + app.router.add_post("/api/course/select", handle_course_select) + app.router.add_post("/api/cmd", handle_cmd) + app.router.add_post("/api/chat", handle_chat) + app.router.add_get("/api/chat/stream", handle_chat_stream) + app.router.add_get("/api/subtitle/stream", handle_subtitle_stream) + app.router.add_get("/api/danmaku/stream", handle_danmaku_stream) + app.router.add_post("/api/lecture/start", handle_lecture_start) + return app + + +def main() -> None: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(name)s] %(message)s", + ) + app = create_app() + logger.info(f"Moshi GUI → http://{HOST}:{PORT}") + web.run_app(app, host=HOST, port=PORT, print=logger.info) + + +if __name__ == "__main__": + main() diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/__init__.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/channel_commands.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/channel_commands.py new file mode 100644 index 00000000..a7d6a65d --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/channel_commands.py @@ -0,0 +1,323 @@ +"""Channel 命令定义——所有 MOSS channel 命令的独立实现。 + +每个命令是模块级 async 函数，接受 MossRuntime 获取运行时依赖。 +register_commands() 将它们注册到 PyChannel。 +""" + +import asyncio as _asyncio +import json +from dataclasses import dataclass + +from ghoshell_moss import CommandError, CommandErrorCode, Message, Text +from ghoshell_moss.core import ChannelCtx +from ghoshell_moss.core.blueprint.mindflow import InputSignal, Priority + +from framework.events import LoadChapterEvent +from framework.runtime.event_generator import _await_event + +from moss_in_reflex.course_manager import CourseManager +from moss_in_reflex.course_storage import CourseStorage +from moss_in_reflex.lecture_brain import ( + POINT_ADVANCED, CHAPTER_ADVANCED, LECTURE_ENDED, + LOADING, LECTURING, PAUSED, ENDED, +) + +from moss_in_reflex.state import ( + logger, + QUEUE, + _LAYOUT, _SNAPSHOTS, + _COURSE_MGR, _LECTURE_BRAIN, + _SSE_EVENT, _SSE_QUEUE, _SSE_LOCK, +) + + +# =========================== Runtime Context =========================== + +@dataclass +class MossRuntime: + """运行时依赖——在 Matrix 初始化后创建，传递给需要 IoC/Matrix 引用的命令。""" + matrix: object # Matrix instance + registry: object # ResourceRegistry + course_storage: object # CourseStorage + runtime_window: object # TopicWindow[AudioRuntimeTopic] + + +# =========================== Helpers =========================== + +async def _switch_to_chapter(n: int, runtime: MossRuntime) -> None: + """从 _COURSE_MGR 取章节数据，解析图片，通过 LoadChapterEvent 上屏。""" + chapter_data = _COURSE_MGR.get_chapter_data(n) + + # 解析图片 locator → PIL 对象 + images = [] + for loc in chapter_data.get("images", []): + item = await runtime.registry.get(loc) + if item: + images.append(await item.get()) + chapter_data["images"] = images + + await _await_event(QUEUE, LoadChapterEvent(chapter_data=chapter_data)) + + _COURSE_MGR.set_chapter_index(n) + _LECTURE_BRAIN.current_chapter = n + + +async def _wait_tts_done(runtime: MossRuntime): + """等待 TTS speaker 播放完成（150ms 轮询）。 + + 无 speaker topic 时最多等 ~450ms（3 次轮询）， + 之后视为无 TTS 运行，立即返回以免死等。 + """ + empty_count = 0 + while True: + for topic in reversed(runtime.runtime_window.values()): + if topic.device_name == "speaker": + if topic.running: + break # 仍在播放，继续等 + else: + return # 播放完成 + else: + empty_count += 1 + if empty_count > 3: + return + await _asyncio.sleep(0.15) + + +# =========================== 命令函数 =========================== + +async def cmd_start_prepare(course: str = "") -> str: + return _COURSE_MGR.start_prepare(course) + + +async def cmd_set_outline(course: str, title: str, chapters: str) -> str: + storage = ChannelCtx.container().force_fetch(CourseStorage) + return await _COURSE_MGR.set_outline(course, title, chapters, storage) + + +async def cmd_save_chapter(chapter: str, locators: str = "", speaker_notes: str = "") -> str: + snap = _SNAPSHOTS.get(_LAYOUT.name) + if snap is None: + raise RuntimeError("no active layout snapshot") + + full = snap.get_full() + locator_list = CourseManager.parse_locators(locators) + + sn: dict | None = None + if speaker_notes.strip(): + try: + sn = json.loads(speaker_notes) + except json.JSONDecodeError as e: + raise CommandError( + code=CommandErrorCode.FAILED, + message=f"speaker_notes JSON 解析失败: {e}", + ) + + image_count = full.get("image", 0) + if not isinstance(image_count, int): + image_count = 0 + if len(locator_list) != image_count: + from ghoshell_moss import CommandError, CommandErrorCode + raise CommandError( + code=CommandErrorCode.FAILED, + message=( + f"locator count mismatch: Ghost 传了 {len(locator_list)} 个，" + f"页面有 {image_count} 张图" + ), + ) + + storage = ChannelCtx.container().force_fetch(CourseStorage) + return await _COURSE_MGR.save_chapter(chapter, locator_list, full, storage, speaker_notes=sn) + + +async def cmd_load_course(course: str) -> str: + storage = ChannelCtx.container().force_fetch(CourseStorage) + return await _COURSE_MGR.load_course(course, storage) + + +async def cmd_start_teaching(course: str = "", start_chapter: int = 0, runtime: MossRuntime = None) -> str: + if not _COURSE_MGR.is_loaded: + if not course: + raise RuntimeError("no course loaded — pass course name or call load_course first") + storage = ChannelCtx.container().force_fetch(CourseStorage) + await _COURSE_MGR.load_course(course, storage) + + result = _COURSE_MGR.start_teaching(start_chapter) + await _switch_to_chapter(start_chapter, runtime) + + sn = _COURSE_MGR.get_speaker_notes(start_chapter) + _LECTURE_BRAIN.start_loading( + course=_COURSE_MGR.name, + chapter=start_chapter, + total_chapters=len(_COURSE_MGR.outline), + talking_points=sn.get("talking_points", []), + transitions=sn.get("transitions", []), + key_data=sn.get("key_data", []), + estimated_duration=sn.get("estimated_duration", 0), + ) + _LECTURE_BRAIN.start_lecturing() + logger.info("[start_teaching] LectureBrain → lecturing, chapter=%d, points=%d", + start_chapter, len(_LECTURE_BRAIN.points)) + + return result + + +async def cmd_switch_chapter(index: int = 0, runtime: MossRuntime = None) -> str: + if not _COURSE_MGR.is_loaded: + raise RuntimeError("no course loaded — call load_course first") + if index < 0 or index >= len(_COURSE_MGR.outline): + raise RuntimeError( + f"chapter index {index} out of range (0-{len(_COURSE_MGR.outline) - 1})" + ) + await _switch_to_chapter(index, runtime) + return f"chapter {index}: {_COURSE_MGR.outline[index]}" + + +async def cmd_advance_point(runtime: MossRuntime) -> str: + if _LECTURE_BRAIN.status == ENDED: + return LECTURE_ENDED + + result = _LECTURE_BRAIN.advance_point() + logger.info("[advance_point] → %s, progress=%s", result, _LECTURE_BRAIN.progress_summary) + + if result == CHAPTER_ADVANCED: + await _wait_tts_done(runtime) + logger.info("[advance_point] TTS 播放完成，开始翻章") + + next_ch = _COURSE_MGR.chapter_index + 1 + if next_ch < len(_COURSE_MGR.outline): + await _switch_to_chapter(next_ch, runtime) + sn = _COURSE_MGR.get_speaker_notes(next_ch) + _LECTURE_BRAIN.advance_chapter( + chapter=next_ch, + talking_points=sn.get("talking_points", []), + transitions=sn.get("transitions", []), + key_data=sn.get("key_data", []), + estimated_duration=sn.get("estimated_duration", 0), + ) + logger.info("[advance_point] 翻页 → chapter %d: %s", next_ch, _COURSE_MGR.outline[next_ch]) + result = f"chapter_advanced → 第 {next_ch + 1} 章: {_COURSE_MGR.outline[next_ch]}" + else: + _LECTURE_BRAIN.end() + logger.info("[advance_point] 全部讲完 → ended") + return LECTURE_ENDED + else: + await _wait_tts_done(runtime) + logger.info("[advance_point] TTS 播放完成，发送 Signal") + + active = _LECTURE_BRAIN.active_point + next_text = active.get("text", "") if active else "" + hint = f"advance_point → {result}。当前段落：{next_text}。请继续叙述。" + signal = InputSignal().to_signal( + priority=Priority.NOTICE, + description=f"讲课推进: {_COURSE_MGR.name}", + hint=hint, + ) + runtime.matrix.session.add_signal(signal) + logger.info("[advance_point] Signal 已发送, hint=%.80s", hint) + return result + + +async def cmd_check_messages() -> str: + return "0 messages (feishu not connected)" + + +async def cmd_resume_lecture() -> str: + _LECTURE_BRAIN.resume() + logger.info("[resume_lecture] → lecturing, chapter=%d", _LECTURE_BRAIN.current_chapter) + return f"resumed at chapter {_LECTURE_BRAIN.current_chapter}" + + +async def cmd_chat_reply(text: str) -> str: + msg = {"role": "ai", "text": text} + async with _SSE_LOCK: + _SSE_QUEUE.append(msg) + _SSE_EVENT.set() + logger.info("[chat_reply] SSE 推送 role=ai text=%.80s", text) + return "ok" + + +# =========================== 注册 =========================== + +def register_commands(chan, runtime: MossRuntime): + """将所有命令注册到 PyChannel。""" + + @chan.build.command( + name="start_prepare", + doc="进入讨论大纲阶段。带 course 参数=开始备课，不带参数=重置回空闲", + timeout=5.0, blocking=True, + ) + async def _start_prepare(course: str = "") -> str: + return await cmd_start_prepare(course) + + @chan.build.command( + name="set_outline", + doc="锁定课程大纲。course=课程名, title=课程标题, chapters=逗号分隔的章节标识列表", + timeout=5.0, blocking=True, + ) + async def _set_outline(course: str, title: str, chapters: str) -> str: + return await cmd_set_outline(course, title, chapters) + + @chan.build.command( + name="save_chapter", + doc="存档当前章节。chapter=章节标识, locators=逗号分隔的图片 locator, speaker_notes=演讲者笔记 JSON。", + timeout=10.0, blocking=True, + ) + async def _save_chapter(chapter: str, locators: str = "", speaker_notes: str = "") -> str: + return await cmd_save_chapter(chapter, locators, speaker_notes) + + @chan.build.command( + name="load_course", + doc="加载指定课程并继续备课。course=课程名", + timeout=15.0, blocking=True, + ) + async def _load_course(course: str) -> str: + return await cmd_load_course(course) + + @chan.build.command( + name="start_teaching", + doc="讲课模式，开讲课程。course=课程名（未加载时必传），start_chapter=起始章节序号（默认 0）", + timeout=15.0, blocking=True, + ) + async def _start_teaching(course: str = "", start_chapter: int = 0) -> str: + return await cmd_start_teaching(course, start_chapter, runtime) + + @chan.build.command( + name="switch_chapter", + doc="切换到指定章节序号（从 0 开始）", + timeout=10.0, blocking=True, + ) + async def _switch_chapter(index: int = 0) -> str: + return await cmd_switch_chapter(index, runtime) + + @chan.build.command( + name="advance_point", + doc="标记当前讲完的要点为 done，推进到下一个。Ghost 在每段讲完后调用。" + "等待 TTS 播放完成后发 Signal 唤醒 Ghost，确保音频不重叠。", + timeout=120.0, blocking=True, + ) + async def _advance_point() -> str: + return await cmd_advance_point(runtime) + + @chan.build.command( + name="check_messages", + doc="检查飞书群消息并回复。Phase 1-2 为桩，返回无消息。", + timeout=5.0, blocking=True, + ) + async def _check_messages() -> str: + return await cmd_check_messages() + + @chan.build.command( + name="resume_lecture", + doc="暂停后恢复讲课。Ghost 回答完打断问题后调用。", + timeout=5.0, blocking=True, + ) + async def _resume_lecture() -> str: + return await cmd_resume_lecture() + + @chan.build.command( + name="chat_reply", + doc="回复聊天消息。text=回复正文", + timeout=5.0, blocking=False, + ) + async def _chat_reply(text: str) -> str: + return await cmd_chat_reply(text) diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/config.yaml b/.moss_ws/apps/ui/moshi/moss_in_reflex/config.yaml new file mode 100644 index 00000000..035aab41 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/config.yaml @@ -0,0 +1,8 @@ +# Layout 配置文件 +# 在这里注册需要加载的 Layout，格式为 Python 模块路径 +# 添加新 Layout 后只需在此文件中添加一行即可启用 + +layouts: + - "framework.layouts.simple.SimpleLayout" + - "framework.layouts.media.MediaLayout" + - "framework.layouts.lesson.LessonLayout" \ No newline at end of file diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/context_messages.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/context_messages.py new file mode 100644 index 00000000..12c27645 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/context_messages.py @@ -0,0 +1,139 @@ +"""各阶段 context_messages 文案模板。 + +四个纯函数，不依赖任何模块状态变量——入参即所需信息。 +moss_in_reflex.py 的 context_messages() 负责判断阶段、传入参数、拼 Message 对象。 +""" + + +def idle() -> str: + return ( + "## 当前状态：空闲\n\n" + "等待人类指令。课程命令（start_prepare、set_outline、save_chapter、" + "load_course、start_teaching）仅在人类明确要求时调用。" + ) + + +def discussing(course_name: str) -> str: + return ( + f"## 当前状态：讨论大纲\n\n" + f"课程：{course_name}\n" + f"请与人类讨论章节结构和各章内容侧重。" + f"必须在人类确认大纲后，调用 set_outline 锁定，这也是此阶段你唯一可用的指令。\n\n" + f"约束：\n" + f" ❌ 不要操作 UI（不要 stream_title 等）\n" + f" ❌ 不要调用 save_chapter、switch_chapter\n" + f" ❌ 不要直接填内容——先定结构" + ) + + +def preparing(outline: list[str], saved_chapters: list[str]) -> str: + saved = set(saved_chapters) + total = len(outline) + current_idx = len(saved_chapters) + + lines = [ + f"## 当前状态：{'备课完成' if len(saved) >= total else '备课'}", + f"", + f"课程：{total} 章", + ] + + for i, ch in enumerate(outline): + if ch in saved: + lines.append(f" ✓ {ch}") + elif i == current_idx: + lines.append(f" → {ch}") + else: + lines.append(f" {ch}") + + lines.append("") + + if len(saved) >= total: + lines.append("全部完成 ✓ 可以开始讲课。") + else: + lines.append( + f"已存档 {len(saved)} 章，当前填充第 {current_idx + 1} 章。" + ) + + lines.append("") + lines.append("约束：") + lines.append(" ❌ 不要调用 switch_chapter（那是讲课命令）") + lines.append(" ❌ 不要自己决定大纲——需要调整时先和人类讨论") + lines.append(" ✓ 每章填完后等人类说\"好了，存档\"再调用 save_chapter") + lines.append("") + lines.append("### speaker_notes（演讲者笔记）") + lines.append("存档每章时，除页面内容外，还需生成演讲者笔记。") + lines.append("save_chapter 的 speaker_notes 参数为 JSON，结构：") + lines.append(' {"talking_points": [{"id":"0","text":"开场","status":"pending"},...],') + lines.append(' "transitions": ["接下来...","现在看..."],') + lines.append(' "key_data": ["关键数据1"],') + lines.append(' "estimated_duration": 120}') + lines.append("talking_points 按叙述顺序排列，status 统一为 pending。") + + return "\n".join(lines) + + +def teaching(course_name: str, outline: list[str], chapter_index: int, + speaker_notes: dict | None = None) -> str: + total = len(outline) + cur = outline[chapter_index] if 0 <= chapter_index < total else "（未知）" + prev_ch = outline[chapter_index - 1] if chapter_index > 0 else "（无）" + next_ch = outline[chapter_index + 1] if chapter_index + 1 < total else "（无）" + + lines = [ + f"## 当前状态：讲课", + f"", + f"课程：{course_name}", + f"进度：第 {chapter_index + 1}/{total} 章 — {cur}", + f"上一章：{prev_ch}", + f"下一章：{next_ch}", + f"", + ] + + # ── 演讲者手卡（静态参考，实时进度见 lecture-state）── + sn = speaker_notes or {} + talking_points = sn.get("talking_points", []) + if talking_points: + # 过渡词 + transitions = sn.get("transitions", []) + if transitions: + lines.append("### 过渡词") + for t in transitions: + lines.append(f" 🔄 {t}") + lines.append("") + + # 关键数据 + key_data = sn.get("key_data", []) + if key_data: + lines.append("### 关键数据") + for kd in key_data: + lines.append(f" 📊 {kd}") + lines.append("") + + # 预计时长 + est = sn.get("estimated_duration", 0) + if est: + lines.append(f"预计时长：约 {est} 秒") + lines.append("") + + lines.append( + f"请按演讲要点逐段讲解。每讲完一个要点后调 " + f"标记完成，调完后**停止输出**，等待系统 Signal 唤醒后再继续下一段。" + f"段落间隙调 检查飞书消息。" + ) + lines.append("") + lines.append("【CTML 规范】：") + lines.append(" ❌ 禁止输出 <_> 或任何自创 CTML 标签——会破坏命令解析器") + lines.append(" ❌ 禁止调用 chat_reply——讲课模式下没有聊天功能") + lines.append(" ❌ 禁止自言自语或模拟对话——你是主讲人，不是聊天对象") + lines.append(" ✓ 用纯文本叙述段落，只在推进时输出 ") + lines.append(" ✓ advance_point 必须是本轮最后一条命令——调完即停，等 Signal 再开新一轮") + lines.append(" ✓ 当 advance_point 返回 lecture_ended 时，讲课已结束。停止输出，不要再调任何命令。") + lines.append("") + lines.append("【强约束】：") + lines.append(" ❌ 不要修改/重新渲染页面内容——只在人类明确指出问题时才改") + lines.append(" ❌ 不要跳到备课行为（除非人类说\"改一下\"）") + lines.append(" ❌ 不要调 switch_state——布局已自动设为 lesson") + lines.append(" ❌ 不要调 load_course——课程已加载完毕") + lines.append(" ✓ 全部要点讲完后调 advance_point → 自动翻页或返回 lecture_ended，结束后不要再调命令") + + return "\n".join(lines) diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/course_manager.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/course_manager.py new file mode 100644 index 00000000..8b7815c4 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/course_manager.py @@ -0,0 +1,234 @@ +"""CourseManager — 课程状态机与持久化。 + +不依赖 Reflex，可独立单元测试。 +""" + +from framework.helpers.layout_snapshot import SNAPSHOT_VERSION +from moss_in_reflex.course_storage import CourseInfo, CourseItem, CourseStorage + + +class CourseManager: + """课程四阶段状态机 + 章节持久化。 + + 四阶段：idle → discussing → preparing → teaching + """ + + def __init__(self) -> None: + self._mode: str = "idle" # idle | discussing | preparing | teaching + self._name: str = "" # 课程标识（目录名），如 "水调歌头" + self._title: str = "" # 课程显示标题，如 "水调歌头·明月几时有" + self._outline: list[str] = [] # 章节名有序列表 + self._chapters: dict[str, dict] = {} # 内存章节数据 {章节名: data_dict} + self._chapter_index: int = 0 # 当前章节序号 + + # ── 只读属性 ────────────────────────────────────────────── + + @property + def mode(self) -> str: + return self._mode + + @property + def name(self) -> str: + return self._name + + @property + def title(self) -> str: + return self._title + + @property + def outline(self) -> tuple[str, ...]: + return tuple(self._outline) + + @property + def saved_chapters(self) -> tuple[str, ...]: + return tuple(self._chapters.keys()) + + @property + def chapter_index(self) -> int: + return self._chapter_index + + @property + def is_loaded(self) -> bool: + return bool(self._chapters) + + # ── 纯状态转换（同步，无 IO）────────────────────────────── + + def start_prepare(self, course: str = "") -> str: + """有参→设课名+切讨论大纲；无参→重置回空闲。""" + if not course: + self._name = "" + self._title = "" + self._outline = [] + self._chapters = {} + self._mode = "idle" + self._chapter_index = 0 + return "reset to idle" + else: + self._name = course + self._title = "" + self._outline = [] + self._chapters = {} + self._mode = "discussing" + self._chapter_index = 0 + return f"preparing {course}" + + def start_teaching(self, start_chapter: int = 0) -> str: + """切到讲课模式。调用方负责 load + _switch_to_chapter。""" + if not self._outline: + raise ValueError("no outline set — call set_outline first") + if start_chapter < 0 or start_chapter >= len(self._outline): + raise ValueError( + f"start_chapter {start_chapter} out of range (0-{len(self._outline) - 1})" + ) + self._mode = "teaching" + return f"teaching from chapter {start_chapter}: {self._outline[start_chapter]}" + + # ── 持久化操作（异步，需要 CourseStorage）───────────────── + + async def set_outline( + self, course: str, title: str, chapters: str, storage: CourseStorage, + ) -> str: + """锁定大纲，写 meta.json，切备课模式。""" + chapter_list = [c.strip() for c in chapters.split(",") if c.strip()] + if not chapter_list: + raise ValueError("chapters 不能为空") + + self._name = course + self._title = title + self._outline = chapter_list + self._chapters = {} + self._mode = "preparing" + self._chapter_index = 0 + + meta_data = { + "version": SNAPSHOT_VERSION, + "title": title, + "chapters": chapter_list, + "chapters_count": len(chapter_list), + } + info = CourseInfo( + host="workspace-assets", + path=f"{course}/meta", + title=title, + chapters_count=len(chapter_list), + ) + item = CourseItem(info, meta_data) + await storage.put(item) + + return f"meta://{course}" + + async def save_chapter( + self, + chapter: str, + locator_list: list[str], + snapshot: dict, + storage: CourseStorage, + speaker_notes: dict | None = None, + ) -> str: + """存档当前章节到 JSON 文件，更新内存进度。 + + Args: + chapter: 章节标识，必须在大纲中 + locator_list: Ghost 传入的图片 locator 列表 + snapshot: LayoutSnapshot.get_full() 返回的页面状态全量数据 + storage: 课程持久化存储 + speaker_notes: 演讲者笔记（备课阶段 AI 生成）。 + {talking_points: [{id, text, status}], transitions: [...], + key_data: [...], estimated_duration: int} + """ + if not self._name: + raise ValueError("no course set — call set_outline first") + if chapter not in self._outline: + raise ValueError( + f"章节 '{chapter}' 不在大纲中。大纲：{', '.join(self._outline)}" + ) + + chapter_data = { + "version": SNAPSHOT_VERSION, + "sub_title": snapshot.get("sub_title", ""), + "main_text": snapshot.get("main_text", ""), + "annotations": snapshot.get("annotations", []), + "appreciation": snapshot.get("appreciation", ""), + "images": locator_list, + "speaker_notes": speaker_notes or {}, + } + + info = CourseInfo(host="workspace-assets", path=f"{self._name}/{chapter}") + item = CourseItem(info, chapter_data) + locator = await storage.put(item) + + self._chapters[chapter] = chapter_data + + return locator + + async def load_course(self, course: str, storage: CourseStorage) -> str: + """从文件系统读 meta + 全部章节到内存，切备课模式。""" + meta_item = await storage.get(f"{course}/meta") + if meta_item is None: + raise ValueError(f"课程 {course} 不存在") + meta = await meta_item.get() + chapter_list = meta.get("chapters", []) + + chapters: dict[str, dict] = {} + for ch_name in chapter_list: + ch_item = await storage.get(f"{course}/{ch_name}") + if ch_item is not None: + chapters[ch_name] = await ch_item.get() + + self._name = course + self._title = meta.get("title", course) + self._outline = chapter_list + self._chapters = chapters + self._mode = "preparing" + self._chapter_index = len(chapters) + + loaded = len(chapters) + pending = len(chapter_list) - loaded + return f"loaded {loaded} chapters, {pending} pending" + + # ── 章节数据（供 _switch_to_chapter 构建 LoadChapterEvent）── + + def get_chapter_data(self, n: int) -> dict: + """返回指定章节的数据字典。 + + images 字段是 locator 字符串列表，由调用方解析为 PIL 对象。 + """ + ch = self._chapters[self._outline[n]] + return { + "title": ch.get("sub_title", self._title or self._name), + "sub_title": ch.get("sub_title", ""), + "main_text": ch.get("main_text", ""), + "annotations": ch.get("annotations", []), + "appreciation": ch.get("appreciation", ""), + "images": list(ch.get("images", [])), + } + + def get_speaker_notes(self, n: int) -> dict: + """返回指定章节的演讲者笔记。 + + Returns: + {talking_points: [...], transitions: [...], key_data: [...], + estimated_duration: int} + 若章节无 speaker_notes 则返回空结构。 + """ + ch = self._chapters.get(self._outline[n], {}) + sn = ch.get("speaker_notes", {}) or {} + return { + "talking_points": sn.get("talking_points", []), + "transitions": sn.get("transitions", []), + "key_data": sn.get("key_data", []), + "estimated_duration": sn.get("estimated_duration", 0), + } + + def set_chapter_index(self, n: int) -> None: + """翻页后更新当前章节序号。""" + self._chapter_index = n + + # ── 工具 ────────────────────────────────────────────────── + + @staticmethod + def parse_locators(locators: str) -> list[str]: + """逗号分隔的 locator 字符串 → 列表。""" + if not locators.strip(): + return [] + return [l.strip() for l in locators.split(",") if l.strip()] diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/course_storage.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/course_storage.py new file mode 100644 index 00000000..8ba1325a --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/course_storage.py @@ -0,0 +1,255 @@ +"""CourseStorage — JSON 文件系统的本地课程资源存储。 + +每个课程一个目录，meta.json + 各章 JSON 文件。 +注册进 ResourceRegistry 后，Ghost 通过 scheme://host/path 访问， +和 pil-image 用完全相同的 API。 +""" + +import json +from pathlib import Path +from typing import Sequence + +from ghoshell_moss.contracts.resource import ( + ResourceInfo, + ResourceItem, + ResourceStorage, + ResourceStorageMeta, +) +from ghoshell_container import IoCContainer, INSTANCE + + +# -- Meta & Item ------------------------------------------------------- + +class CourseInfo(ResourceInfo): + """课程资源元信息。""" + + host: str = "workspace-assets" + path: str = "" + description: str = "" + title: str = "" + chapters_count: int = 0 + + @classmethod + def scheme(cls) -> str: + return "course" + + @classmethod + def scheme_description(cls) -> str: + return "课程资源，包含备课章节的结构化数据" + + +class CourseItem(ResourceItem[CourseInfo, dict]): + """课程资源项。meta 立即可用，get() 直接返回内存 dict。""" + + def __init__(self, meta: CourseInfo, data: dict) -> None: + self._meta = meta + self._data = data + + @classmethod + def meta_type(cls) -> type[CourseInfo]: + return CourseInfo + + @property + def info(self) -> CourseInfo: + return self._meta + + async def get(self) -> dict: + return self._data + + +# -- Storage ----------------------------------------------------------- + +class CourseStorage(ResourceStorage[CourseInfo, dict]): + """JSON 文件系统的本地课程存储。 + + 目录结构: + {data_dir}/ + 水调歌头/ + meta.json # 课程元信息 + ch00_导语.json # 章节文件 + ch01_背景.json + ... + + query 支持: 无参数浏览全部课程，keyword 匹配课程 title。 + """ + + META_FILE = "meta.json" + + def __init__(self, data_dir: str | Path, host: str = "workspace-assets") -> None: + self._host = host + self._data_dir = Path(data_dir) + + # -- class-level -------------------------------------------------- + + @classmethod + def scheme(cls) -> str: + return CourseInfo.scheme() + + @classmethod + def scheme_description(cls) -> str: + return CourseInfo.scheme_description() + + # -- instance-level ------------------------------------------------ + + @property + def host(self) -> str: + return self._host + + # -- self-describing ----------------------------------------------- + + def usage(self) -> str: + return """\ +course: 本地课程资源存储 + +查询语法: keyword（匹配课程标题，大小写不敏感） + course list → 列出全部课程 + course list "水调歌头" → 搜索包含关键字的课程 + +返回的 ResourceMeta 字段: + host, path, description, title, chapters_count + +路径格式: {课程名}/{章节名} 或 {课程名}/meta + course://workspace-assets/水调歌头/meta → 课程元信息 + course://workspace-assets/水调歌头/ch01_背景 → 章节内容""" + + async def help(self, question: str | None = None) -> str: + if question is None: + return self.usage() + q = question.lower() + if "格式" in q or "format" in q or "字段" in q: + return ( + "章节 JSON 格式: {version, sub_title, main_text, annotations, " + "appreciation, images (locator 列表), " + "speaker_notes: {talking_points: [{id, text, status}], " + "transitions: [...], key_data: [...], estimated_duration: int}}" + ) + if "查询" in q or "query" in q: + return "query 支持 keyword 匹配课程 title，大小写不敏感。不传 query 列出全部课程。" + return f"[course help] {question}\n此问题无预设答案。用法概览:\n{self.usage()}" + + # -- CRUD (writing) ------------------------------------------------- + + async def put(self, item: ResourceItem[CourseInfo, dict]) -> str: + meta = item.info + data = await item.get() + + path = meta.path + if not path: + raise ValueError("CourseInfo.path is required for put") + + meta.host = self._host + + file_path = self._data_dir / f"{path}.json" + file_path.parent.mkdir(parents=True, exist_ok=True) + content = json.dumps(data, ensure_ascii=False, default=str) + file_path.write_text(content, encoding="utf-8") + + return meta.locator + + async def delete(self, path: str) -> bool: + file_path = self._data_dir / f"{path}.json" + if not file_path.exists(): + return False + file_path.unlink() + + # 如果所属目录下无文件，清理空目录 + parent = file_path.parent + if parent.is_dir() and not any(parent.iterdir()): + parent.rmdir() + return True + + # -- CRUD (reading) -------------------------------------------------- + + async def get(self, path: str) -> CourseItem | None: + file_path = self._data_dir / f"{path}.json" + if not file_path.exists(): + return None + + try: + data = json.loads(file_path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return None + + meta = CourseInfo( + host=self._host, + path=path, + title=data.get("title", ""), + chapters_count=data.get("chapters_count", 0), + ) + return CourseItem(meta, data) + + async def list_infos( + self, query: str | None = None, limit: int = 50 + ) -> Sequence[CourseInfo]: + infos: list[CourseInfo] = [] + + if not self._data_dir.exists(): + return infos + + for course_dir in sorted(self._data_dir.iterdir()): + if not course_dir.is_dir(): + continue + + meta_file = course_dir / self.META_FILE + if not meta_file.exists(): + continue + + try: + meta_data = json.loads(meta_file.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + continue + + title = meta_data.get("title", course_dir.name) + if query and query.lower() not in title.lower(): + continue + + infos.append(CourseInfo( + host=self._host, + path=f"{course_dir.name}/meta", + title=title, + description=meta_data.get("title", ""), + chapters_count=meta_data.get("chapters_count", 0), + )) + + if 0 <= limit <= len(infos): + break + + return infos + + +# -- ResourceStorageMeta (for manifest discovery) ----------------------- + +class CourseStorageMeta(ResourceStorageMeta): + """IoC 工厂：从 Workspace 拿 data_dir，注册 CourseStorage 到 ResourceRegistry。 + + Matrix 扫描 mode manifests 时通过 isinstance(obj, ResourceStorageMeta) + 发现此实例，自动调用 factory() 并注册到 ResourceRegistry(scheme="course")。 + """ + + def __init__( + self, + host: str = "workspace-assets", + assets_sub_path: str = "courses", + ): + self._host = host + self._assets_sub_path = assets_sub_path + + def factory(self, con: IoCContainer) -> INSTANCE: + from ghoshell_moss.contracts.workspace import Workspace + + workspace = con.force_fetch(Workspace) + data_dir = workspace.assets().abspath() / self._assets_sub_path + return CourseStorage(data_dir, host=self._host) + + # -- ResourceStorageMeta -------------------------------------------- + + @classmethod + def scheme(cls) -> str: + return CourseStorage.scheme() + + @property + def host(self) -> str: + return self._host + + def description(self) -> str: + return "Local course resource storage backed by JSON filesystem" diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/http_endpoints.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/http_endpoints.py new file mode 100644 index 00000000..a8189fcf --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/http_endpoints.py @@ -0,0 +1,268 @@ +"""内部 HTTP 桥接端点 — 供 main.py 发 Signal + 收 SSE。 + +create_internal_app() 创建 aiohttp Application 并注册所有路由。 +""" + +import asyncio +import json + +from aiohttp import web as aiohttp_web + +from ghoshell_moss import Message, Text +from ghoshell_moss.core.blueprint.mindflow import InputSignal + +from framework.events import LayoutEvent +from framework.runtime.event_generator import _await_event + +from moss_in_reflex.channel_commands import _switch_to_chapter +from moss_in_reflex.state import ( + logger, + QUEUE, + _COURSE_MGR, _LECTURE_BRAIN, + _SSE_EVENT, _SSE_QUEUE, _SSE_LOCK, + _SUBTITLE_EVENT, _SUBTITLE_QUEUE, _SUBTITLE_LOCK, + _DANMAKU_EVENT, _DANMAKU_QUEUE, _DANMAKU_LOCK, +) + + +# =========================== SSE 流端点 =========================== + +async def _chat_stream(request: aiohttp_web.Request): + """SSE：监听 asyncio.Event，推送 AI 回复。""" + resp = aiohttp_web.StreamResponse( + status=200, + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + "Connection": "keep-alive", + }, + ) + await resp.prepare(request) + logger.info("[chat_stream] SSE 客户端连接") + await resp.write(b": connected\n\n") + sent = 0 + try: + while True: + await asyncio.wait_for(_SSE_EVENT.wait(), timeout=30) + _SSE_EVENT.clear() + async with _SSE_LOCK: + while _SSE_QUEUE: + msg = _SSE_QUEUE.pop(0) + payload = json.dumps(msg, ensure_ascii=False) + await resp.write(f"data: {payload}\n\n".encode()) + sent += 1 + logger.info("[chat_stream] 推送给前端 role=%s text=%.60s", + msg.get("role"), msg.get("text")) + except asyncio.TimeoutError: + pass + except (ConnectionResetError, ConnectionAbortedError): + logger.info("[chat_stream] SSE 客户端断开, 共推送 %d 条", sent) + except Exception: + logger.exception("[chat_stream] 内部错误") + return resp + + +async def _subtitle_stream(request: aiohttp_web.Request): + """SSE: 推送字幕流。""" + resp = aiohttp_web.StreamResponse( + status=200, + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + "Connection": "keep-alive", + }, + ) + await resp.prepare(request) + logger.info("[subtitle_stream] SSE 客户端连接") + await resp.write(b": connected\n\n") + try: + while True: + await asyncio.wait_for(_SUBTITLE_EVENT.wait(), timeout=30) + _SUBTITLE_EVENT.clear() + async with _SUBTITLE_LOCK: + while _SUBTITLE_QUEUE: + msg = _SUBTITLE_QUEUE.pop(0) + payload = json.dumps(msg, ensure_ascii=False) + await resp.write(f"data: {payload}\n\n".encode()) + except asyncio.TimeoutError: + pass + except (ConnectionResetError, ConnectionAbortedError): + logger.info("[subtitle_stream] SSE 客户端断开") + except Exception: + logger.exception("[subtitle_stream] 内部错误") + return resp + + +async def _danmaku_stream(request: aiohttp_web.Request): + """SSE: 推送弹幕流。""" + resp = aiohttp_web.StreamResponse( + status=200, + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + "Connection": "keep-alive", + }, + ) + await resp.prepare(request) + logger.info("[danmaku_stream] SSE 客户端连接") + await resp.write(b": connected\n\n") + try: + while True: + await asyncio.wait_for(_DANMAKU_EVENT.wait(), timeout=30) + _DANMAKU_EVENT.clear() + async with _DANMAKU_LOCK: + while _DANMAKU_QUEUE: + msg = _DANMAKU_QUEUE.pop(0) + payload = json.dumps(msg, ensure_ascii=False) + await resp.write(f"data: {payload}\n\n".encode()) + except asyncio.TimeoutError: + pass + except (ConnectionResetError, ConnectionAbortedError): + logger.info("[danmaku_stream] SSE 客户端断开") + except Exception: + logger.exception("[danmaku_stream] 内部错误") + return resp + + +async def _subtitle_in(request: aiohttp_web.Request): + """接收来自 Speech 进程的字幕数据（HTTP 旁路），推入 SSE 队列。""" + try: + data = await request.json() + text = (data.get("text") or "").strip() + is_final = data.get("is_final", False) + if text or is_final: + async with _SUBTITLE_LOCK: + _SUBTITLE_QUEUE.append( + {"type": "full" if is_final else "chunk", "text": text} + ) + _SUBTITLE_EVENT.set() + except Exception: + pass # 字幕非关键，静默吞错 + return aiohttp_web.json_response({"ok": True}) + + +# =========================== Runtime-dependent 端点 =========================== + +def _create_chat_in(matrix): + """main.py 转发用户消息 → 发 Signal 唤醒 Ghost。""" + async def handler(request: aiohttp_web.Request): + data = await request.json() + text = (data.get("text") or "").strip() + if not text: + return aiohttp_web.json_response({"error": "empty text"}, status=400) + signal = InputSignal().to_signal( + Message.new(tag="chat-input").with_content(Text(text=text)), + description=f"聊天: {text[:50]}", + ) + matrix.session.add_signal(signal) + logger.info("[chat_in] Signal 已发送, text=%.60s", text) + return aiohttp_web.json_response({"ok": True}) + return handler + + +def _create_courses(course_storage): + """返回课程列表，供 main.py 代理。""" + async def handler(request: aiohttp_web.Request): + infos = await course_storage.list_infos() + courses = [ + { + "name": info.path.rsplit("/", 1)[0] if "/" in info.path else info.path, + "title": info.title, + "chapters_count": info.chapters_count, + "locator": info.locator, + } + for info in infos + ] + return aiohttp_web.json_response(courses) + return handler + + +def _create_lecture_start(matrix, course_storage, runtime): + """原子编排：加载+布局+渲染+状态机+唤醒Ghost。""" + async def handler(request: aiohttp_web.Request): + data = await request.json() + course = (data.get("course") or "").strip() + if not course: + return aiohttp_web.json_response({"error": "course is required"}, status=400) + + logger.info("[lecture_start] 开始编排课程 %s", course) + + # 1. 加载课程 + if not _COURSE_MGR.is_loaded or _COURSE_MGR.name != course: + await _COURSE_MGR.load_course(course, course_storage) + + # 2. 强制切到 lesson 布局 + await _await_event(QUEUE, LayoutEvent(layout="lesson")) + logger.info("[lecture_start] 布局 → lesson") + + # 3. 进入讲课模式 + 渲染首章 + start_chapter = data.get("start_chapter", 0) + _COURSE_MGR.start_teaching(start_chapter) + await _switch_to_chapter(start_chapter, runtime) + + # 4. 初始化讲课状态机 + sn = _COURSE_MGR.get_speaker_notes(start_chapter) + _LECTURE_BRAIN.start_loading( + course=_COURSE_MGR.name, + chapter=start_chapter, + total_chapters=len(_COURSE_MGR.outline), + talking_points=sn.get("talking_points", []), + transitions=sn.get("transitions", []), + key_data=sn.get("key_data", []), + estimated_duration=sn.get("estimated_duration", 0), + ) + _LECTURE_BRAIN.start_lecturing() + + # 5. 发 Signal 唤醒 Ghost + ready_msg = ( + f"课程《{_COURSE_MGR.name}》已就绪。" + f"当前第 1/{len(_COURSE_MGR.outline)} 章：{_COURSE_MGR.outline[0]}。" + f"布局已自动设为 lesson。请按 speaker_notes 开始叙述。" + ) + signal = InputSignal().to_signal( + Message.new(tag="lecture-ready").with_content(Text(text=ready_msg)), + description=f"讲课就绪: {_COURSE_MGR.name}", + ) + matrix.session.add_signal(signal) + + logger.info("[lecture_start] Signal 已发送, course=%s, chapter=%d, points=%d", + _COURSE_MGR.name, start_chapter, len(_LECTURE_BRAIN.points)) + + return aiohttp_web.json_response({ + "ok": True, + "course": _COURSE_MGR.name, + "title": _COURSE_MGR.title or _COURSE_MGR.name, + "chapter": start_chapter, + "total_chapters": len(_COURSE_MGR.outline), + "chapter_name": _COURSE_MGR.outline[start_chapter] if _COURSE_MGR.outline else "", + }) + return handler + + +# =========================== App 创建 =========================== + +def create_internal_app(matrix, course_storage, runtime) -> aiohttp_web.Application: + """创建内部 HTTP 桥接应用，注册所有路由。 + + 返回已注册好全部端点的 aiohttp_web.Application（尚未启动）， + 由调用方负责 AppRunner + TCPSite 生命周期。 + """ + app = aiohttp_web.Application() + + # Signal 输入 + app.router.add_post("/_internal/chat_in", _create_chat_in(matrix)) + + # SSE 流（纯模块级状态，无需闭包） + app.router.add_get("/_internal/chat_stream", _chat_stream) + app.router.add_get("/_internal/subtitle_stream", _subtitle_stream) + app.router.add_get("/_internal/danmaku_stream", _danmaku_stream) + + # HTTP 旁路字幕输入 + app.router.add_post("/_internal/subtitle_in", _subtitle_in) + + # 课程 & 讲课 + app.router.add_get("/_internal/courses", _create_courses(course_storage)) + app.router.add_post("/_internal/lecture/start", + _create_lecture_start(matrix, course_storage, runtime)) + + return app diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/lecture_brain.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/lecture_brain.py new file mode 100644 index 00000000..6de2b6e3 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/lecture_brain.py @@ -0,0 +1,272 @@ +"""LectureBrain — 讲课状态机。 + +管理讲课流程的实时状态：段落推进、暂停/恢复、问题收集。 +不依赖 Reflex，不持有 CourseManager 引用——翻页决策由命令侧根据返回值执行。 +""" + +import time +from dataclasses import dataclass, field + + +# ── 状态常量 ── + +IDLE = "idle" +LOADING = "loading" +LECTURING = "lecturing" +PAUSED = "paused" +ENDED = "ended" + +# ── advance_point 返回值 ── + +POINT_ADVANCED = "point_advanced" # 段内推进，继续讲 +CHAPTER_ADVANCED = "chapter_advanced" # 本章讲完，需翻页（命令侧处理） +LECTURE_ENDED = "lecture_ended" # 全部讲完 + + +@dataclass +class LectureBrain: + """讲课状态机。 + + idle → loading → lecturing ⇄ paused → ended + + 核心职责： + - 管理 talking_points 生命周期（pending → active → done） + - 处理暂停/恢复（语音打断 + Ghost 回答后继续） + - advance_point 自动判断翻页 + - 收集讲课过程中的问题 + """ + + status: str = IDLE + current_course: str = "" + current_chapter: int = 0 + total_chapters: int = 0 + points: list[dict] = field(default_factory=list) # talking_points（可变） + transitions: list[str] = field(default_factory=list) + key_data: list[str] = field(default_factory=list) + estimated_duration: int = 0 + questions: list[dict] = field(default_factory=list) # [{source, text, sender}] + point_started_at: float = 0.0 + + # ── 状态查询 ── + + @property + def is_active(self) -> bool: + return self.status in (LOADING, LECTURING, PAUSED) + + @property + def active_point(self) -> dict | None: + """当前正在讲的要点。""" + for p in self.points: + if p.get("status") == "active": + return p + return None + + @property + def progress_summary(self) -> str: + """段落进度摘要，供 context 展示。""" + done = sum(1 for p in self.points if p.get("status") == "done") + total = len(self.points) + return f"{done}/{total}" + + # ── 状态转换 ── + + def start_loading( + self, + course: str, + chapter: int, + total_chapters: int, + talking_points: list[dict], + transitions: list[str] | None = None, + key_data: list[str] | None = None, + estimated_duration: int = 0, + ) -> None: + """进入 loading 状态。由 start_teaching 命令调用。 + + talking_points 应为 [{id, text, status:"pending"}] 格式。 + 首个要点自动标为 active。 + """ + self.status = LOADING + self.current_course = course + self.current_chapter = chapter + self.total_chapters = total_chapters + + # 深拷贝 talking_points，统一 status + self.points = [ + {"id": p.get("id", str(i)), "text": p.get("text", ""), "status": "pending"} + for i, p in enumerate(talking_points) + ] + self.transitions = list(transitions or []) + self.key_data = list(key_data or []) + self.estimated_duration = estimated_duration + self.questions = [] + + # 首个要点激活 + if self.points: + self.points[0]["status"] = "active" + self.point_started_at = time.monotonic() + + def start_lecturing(self) -> None: + """loading → lecturing。页面渲染完成后调用。""" + if self.status != LOADING: + raise RuntimeError(f"只能在 loading 状态调 start_lecturing，当前 {self.status}") + self.status = LECTURING + + def pause(self, source: str = "voice", text: str = "", sender: str = "") -> None: + """lecturing → paused。语音打断或人工控场。 + + Args: + source: "voice" | "manual" + text: 问题文本 + sender: 提问者标识 + """ + if self.status not in (LECTURING, LOADING): + return # 不报错，幂等 + + self.status = PAUSED + if text.strip(): + self.questions.append({ + "source": source, + "text": text.strip(), + "sender": sender, + }) + # 当前 active 段落保持 active，point_started_at 不动 + + def resume(self) -> None: + """paused → lecturing。Ghost 回答完问题后恢复讲课。""" + if self.status != PAUSED: + raise RuntimeError(f"只能在 paused 状态调 resume，当前 {self.status}") + self.status = LECTURING + # point_started_at 重置，给恢复后的段落全新计时 + self.point_started_at = time.monotonic() + + def end(self) -> str: + """强制结束讲课 → ended。""" + self.status = ENDED + return LECTURE_ENDED + + # ── 段落推进 ── + + def advance_point(self) -> str: + """标记当前 active → done，激活下一个 pending。 + + Returns: + "point_advanced" — 段内推进，继续讲当前章 + "chapter_advanced" — 本章全部 done，需要翻页（命令侧负责） + "lecture_ended" — 全部章节讲完 + """ + if self.status not in (LECTURING, PAUSED): + raise RuntimeError(f"只能在 lecturing/paused 状态调 advance_point，当前 {self.status}") + + # 标记当前 active → done + for p in self.points: + if p.get("status") == "active": + p["status"] = "done" + break + + # 找下一个 pending + next_point = None + for p in self.points: + if p.get("status") == "pending": + next_point = p + break + + if next_point is not None: + next_point["status"] = "active" + self.point_started_at = time.monotonic() + return POINT_ADVANCED + + # 本章全部 done + return CHAPTER_ADVANCED + + # ── 翻页（命令侧调用） ── + + def advance_chapter(self, chapter: int, talking_points: list[dict], + transitions: list[str] | None = None, + key_data: list[str] | None = None, + estimated_duration: int = 0) -> str: + """加载新章节的 talking_points。命令侧判断还有章节时调用。 + + Returns: + "chapter_advanced" — 新章节已加载 + "lecture_ended" — 这是最后一章，讲课结束 + """ + self.current_chapter = chapter + + self.points = [ + {"id": p.get("id", str(i)), "text": p.get("text", ""), "status": "pending"} + for i, p in enumerate(talking_points) + ] + self.transitions = list(transitions or []) + self.key_data = list(key_data or []) + self.estimated_duration = estimated_duration + + if self.points: + self.points[0]["status"] = "active" + self.point_started_at = time.monotonic() + + # 保持 lecturing 状态（pause 后翻页的场景：仍在 pause 中） + if self.status == PAUSED: + # 翻页后仍暂停，等 Ghost 调 resume + pass + else: + self.status = LECTURING + + return CHAPTER_ADVANCED + + # ── 超时检测 ── + + def check_timeout(self) -> bool: + """检查当前 active 段落是否超时。 + + Returns True 如果超时（超过 estimated_duration * 1.5 且至少 30 秒）。 + 命令侧根据返回值决定是否自动推进。 + """ + if self.status != LECTURING: + return False + if self.estimated_duration <= 0: + return False + if self.point_started_at <= 0: + return False + + elapsed = time.monotonic() - self.point_started_at + timeout = max(self.estimated_duration * 1.5, 30) + return elapsed > timeout + + # ── 问题收集 ── + + def add_question(self, source: str, text: str, sender: str = "") -> None: + """添加问题记录（用于结束后总结）。""" + self.questions.append({ + "source": source, + "text": text, + "sender": sender, + }) + + def summary(self) -> str: + """生成讲课总结文本。""" + if not self.questions: + return "本次讲课无提问记录。" + + voice_qs = [q for q in self.questions if q.get("source") == "voice"] + feishu_qs = [q for q in self.questions if q.get("source") == "feishu"] + + lines = [f"## 讲课总结\n"] + lines.append(f"课程：{self.current_course}") + lines.append(f"共收到 {len(self.questions)} 个问题") + lines.append(f" - 语音打断提问：{len(voice_qs)} 个") + lines.append(f" - 飞书群提问：{len(feishu_qs)} 个") + lines.append("") + + if voice_qs: + lines.append("### 语音提问") + for q in voice_qs: + lines.append(f" - {q.get('sender', '匿名')}: {q.get('text', '')}") + lines.append("") + + if feishu_qs: + lines.append("### 飞书群提问") + for q in feishu_qs: + lines.append(f" - {q.get('sender', '匿名')}: {q.get('text', '')}") + lines.append("") + + return "\n".join(lines) diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/moss_in_reflex.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/moss_in_reflex.py new file mode 100644 index 00000000..08c25a93 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/moss_in_reflex.py @@ -0,0 +1,387 @@ +import asyncio +import inspect +from contextlib import asynccontextmanager + +from aiohttp import web as aiohttp_web +import reflex as rx +from ghoshell_moss import PyChannel, Message, Text, Matrix +from ghoshell_moss.contracts import ResourceRegistry +from ghoshell_moss.core import ChannelCtx + +from framework.events import EventModel, LayoutEvent, LoadChapterEvent, StreamEvent, SetEvent, AppendEvent, UpdateEvent, PopEvent, ClearEvent +from moss_in_reflex.course_storage import CourseStorage, CourseStorageMeta +from moss_in_reflex.context_messages import idle, discussing, preparing, teaching +from moss_in_reflex.lecture_brain import ( + POINT_ADVANCED, CHAPTER_ADVANCED, LECTURE_ENDED, + LOADING, LECTURING, PAUSED, ENDED, +) +from ghoshell_moss.topics.audio import AudioRuntimeTopic +from ghoshell_moss.contracts.resource import ResourceStorageFactoryBootstrapper +from ghoshell_moss.contracts.configs import ConfigStore + +# ── 子模块 ── +from moss_in_reflex.subtitle import setup_subtitle +from moss_in_reflex.channel_commands import MossRuntime, register_commands +from moss_in_reflex.http_endpoints import create_internal_app + +# ── 模块级全局状态（state.py）── +from moss_in_reflex.state import ( + logger, + CONFIG, LAYOUTS, LAYOUT_COMPONENTS, LAYOUT_CHANNEL_STATES, + QUEUE, + _LAYOUT, _SNAPSHOTS, + _COURSE_MGR, _LECTURE_BRAIN, + _SSE_EVENT, _SSE_QUEUE, _SSE_LOCK, + _SUBTITLE_EVENT, _SUBTITLE_QUEUE, _SUBTITLE_LOCK, + _DANMAKU_EVENT, _DANMAKU_QUEUE, _DANMAKU_LOCK, +) + +# =========================== Reflex =========================== +class State(rx.State): + """The app state.""" + + layout: str = "simple" + + @rx.event(background=True) + async def moss_listener(self): + # Yield immediately so Reflex can finish on_load and render the page + # yield + + # hot-reload 后模块全局变量被重置，从 Reflex State 权威恢复 + # self.layout 存储在 Reflex StateManager 中，不随模块 reload 丢失 + if self.layout != _LAYOUT.name: + _LAYOUT.name = self.layout + logger.info("Layout 从 Reflex State 恢复: %s", self.layout) + + while True: + # 先取事件；若 background task 被取消，get() 抛 CancelledError， + # 此时没有取到 item，直接退出，绝不能调用 task_done()（否则超调）。 + try: + event = await QUEUE.get() + except asyncio.CancelledError: + return + + # 取到 item 后，无论处理成功与否，finally 都恰好 task_done() 一次， + # 与 get() 一一对应。 + fut = event.future # 命令侧通过 _await_event 塞入，否则为 None + + try: + logger.info(f"moss_listener {event}") + + if isinstance(event, LayoutEvent): + _LAYOUT.name = event.layout + async with self: + self.layout = event.layout + + current = _LAYOUT.get_component() + if not current: + logger.warning("Current layout is None") + if fut and not fut.done(): + fut.set_exception(RuntimeError("no active layout")) + continue + + handler_missing = None + if isinstance(event, StreamEvent): + handler = f"stream_{event.field}" + if hasattr(current.State, handler): + yield getattr(current.State, handler)(event.chunk) + else: + handler_missing = handler + if isinstance(event, SetEvent): + handler = f"set_{event.field}" + if hasattr(current.State, handler): + yield getattr(current.State, handler)(event.data) + else: + handler_missing = handler + if isinstance(event, AppendEvent): + handler = f"append_{event.field}" + if hasattr(current.State, handler): + yield getattr(current.State, handler)(event.data) + else: + handler_missing = handler + if isinstance(event, UpdateEvent): + handler = f"update_{event.field}" + if hasattr(current.State, handler): + yield getattr(current.State, handler)(event.index, event.data) + else: + handler_missing = handler + if isinstance(event, PopEvent): + handler = f"pop_{event.field}" + if hasattr(current.State, handler): + yield getattr(current.State, handler)() + else: + handler_missing = handler + if isinstance(event, ClearEvent): + handler = f"clear_{event.field}" + if hasattr(current.State, handler): + yield getattr(current.State, handler)() + else: + handler_missing = handler + if isinstance(event, LoadChapterEvent): + ch = event.chapter_data + + # 清空全部字段 + for field in ("title", "sub_title", "image", "main_text", "annotations", "appreciation"): + handler = f"clear_{field}" + if hasattr(current.State, handler): + yield getattr(current.State, handler)() + + # 逐字段填充 + if ch.get("title"): + yield current.State.stream_title(ch["title"]) + if ch.get("sub_title"): + yield current.State.stream_sub_title(ch["sub_title"]) + if ch.get("main_text"): + yield current.State.stream_main_text(ch["main_text"]) + for ann in ch.get("annotations", []): + yield current.State.append_annotations(ann) + if ch.get("appreciation"): + yield current.State.stream_appreciation(ch["appreciation"]) + for img in ch.get("images", []): + yield current.State.append_image(img) + + handler_missing = None # 已处理，不需要单独的 callback + + if handler_missing: + logger.warning("Layout %r has no handler %r", _LAYOUT.name, handler_missing) + if fut and not fut.done(): + fut.set_exception( + RuntimeError(f"handler {handler_missing} not found on layout {_LAYOUT.name}") + ) + + except asyncio.CancelledError: + if fut and not fut.done(): + fut.cancel() + return + except Exception as ex: + logger.error(f"Exception: {ex}") + if fut and not fut.done(): + fut.set_exception(ex) + else: + if handler_missing is None: + async with self: + await _SNAPSHOTS[_LAYOUT.name].refresh(self) + if fut and not fut.done(): + fut.set_result(None) + finally: + QUEUE.task_done() + + +def index() -> rx.Component: + return rx.container( + rx.match( + State.layout, + *LAYOUT_COMPONENTS, + rx.text("default") + ), + ) +# =========================== Reflex =========================== + +# =========================== MOSS =========================== +async def context_messages(): + logger.info("[context_messages] >>> 被调用, mode=%s", _COURSE_MGR.mode) + messages = [] + + # ── 情境感知：调模板函数获取文案 ── + + if _COURSE_MGR.mode == "discussing": + messages.append( + Message.new(tag="context-mode").with_content( + Text(text=discussing(_COURSE_MGR.name or "未命名")) + ) + ) + elif _COURSE_MGR.mode == "preparing" and _COURSE_MGR.outline: + messages.append( + Message.new(tag="context-mode").with_content( + Text(text=preparing(list(_COURSE_MGR.outline), list(_COURSE_MGR.saved_chapters))) + ) + ) + elif _COURSE_MGR.mode == "teaching" and _COURSE_MGR.is_loaded: + sn = _COURSE_MGR.get_speaker_notes(_COURSE_MGR.chapter_index) + # 将 LectureBrain 实时状态合并到 talking_points，避免与持久化数据矛盾 + if _LECTURE_BRAIN.points: + lb_status = {p.get("id"): p.get("status") for p in _LECTURE_BRAIN.points} + merged_points = [] + for tp in sn.get("talking_points", []): + tp_copy = dict(tp) + tp_id = tp_copy.get("id", "") + if tp_id in lb_status: + tp_copy["status"] = lb_status[tp_id] + merged_points.append(tp_copy) + sn = {**sn, "talking_points": merged_points} + messages.append( + Message.new(tag="context-mode").with_content( + Text(text=teaching(_COURSE_MGR.name, list(_COURSE_MGR.outline), + _COURSE_MGR.chapter_index, sn)) + ) + ) + # 附加 LectureBrain 实时状态 + if _LECTURE_BRAIN.is_active: + lb_lines = [ + f"### 讲课实时状态: {_LECTURE_BRAIN.status}", + f"章节: {_LECTURE_BRAIN.current_chapter + 1}/{_LECTURE_BRAIN.total_chapters}", + f"段落进度: {_LECTURE_BRAIN.progress_summary}", + ] + if _LECTURE_BRAIN.points: + lb_lines.append("") + for tp in _LECTURE_BRAIN.points: + mark = {"done": "✓", "active": "→", "pending": "…"}.get( + tp.get("status", ""), "…" + ) + lb_lines.append(f" {mark} {tp.get('text', '')}") + lb_lines.append("") + lb_lines.append( + "讲完当前 → 要点后调 ，" + "调完后**停止输出**，等待系统 Signal 唤醒后再继续下一段。" + "段落间隙调 。" + ) + if _LECTURE_BRAIN.status == PAUSED: + lb_lines.append("⚠ 当前已暂停，回答完问题后调 恢复。") + messages.append( + Message.new(tag="lecture-state").with_content( + Text(text="\n".join(lb_lines)) + ) + ) + else: + messages.append( + Message.new(tag="context-mode").with_content(Text(text=idle())) + ) + + # ── 布局源码 + State 快照（仅备课/讨论模式，按模式信息隔离） ── + if _COURSE_MGR.mode in ("discussing", "preparing"): + for l in LAYOUTS: + if l.name() == _LAYOUT.name: + module = inspect.getmodule(l) + source_code = inspect.getsource(module) + messages.append( + Message.new(tag="layout-source-code").with_content( + Text(text=f"当前 layout: {l.name()}，reflex 源码如下\n"), + Text(text=source_code), + ) + ) + + snap = _SNAPSHOTS.get(_LAYOUT.name) + if snap is not None: + data = snap.get() + if data: + state_lines = [f"{k}: {v}" for k, v in data.items()] + messages.append( + Message.new(tag="current-state").with_content( + Text(text="\n".join(state_lines)) + ) + ) + + # ── 可用资源（图片 + 课程） ── + registry = ChannelCtx.container().force_fetch(ResourceRegistry) + resource_msg = Message.new(tag="resources") + + pil_infos = await registry.list_infos(scheme="pil-image") + for info in pil_infos: + resource_msg.with_content( + f"locator: {info.locator} description: {info.description}\n" + ) + + course_infos = await registry.list_infos(scheme="course") + for info in course_infos: + course_name = info.path.rsplit("/", 1)[0] if "/" in info.path else info.path + resource_msg.with_content( + f"locator: {info.locator} course: {course_name} title: {info.title} 章节数: {info.chapters_count}\n" + ) + + if pil_infos or course_infos: + messages.append(resource_msg) + + # ── 聊天指令（按模式隔离：teaching 模式不开放聊天功能）── + if _COURSE_MGR.mode != "teaching": + messages.append( + Message.new(tag="chat-instruction").with_content( + Text(text="\n".join([ + "## 聊天功能", + "你正在和用户进行实时文字对话。当收到用户消息（Signal name='input'）时，", + "请使用 命令回复用户。", + "你的回复应该是自然的中文对话，直接写回复内容即可，不需要任何标记或前缀。", + ])) + ) + ) + + return messages + + +async def moss(): + chan = PyChannel(name="moshi", description="魔师 Moshi — AI 内容传递协作助手。提供课程管理、聊天交互、白板渲染能力") + chan.build.context_messages(context_messages) + + first = True + for state in LAYOUT_CHANNEL_STATES: + if first: + chan.with_state(state, is_default=True) + first = False + continue + chan.with_state(state) + + matrix = Matrix.discover() + _internal_runner = None + async with matrix: + + meta = CourseStorageMeta() + ResourceStorageFactoryBootstrapper(meta).bootstrap(matrix.container) + matrix.container.set(CourseStorage, meta.factory(matrix.container)) + _course_storage = meta.factory(matrix.container) + _registry = matrix.container.force_fetch(ResourceRegistry) + + # ── TTS 门控：订阅 AudioRuntimeTopic 窗口 ── + _runtime_window = matrix.session.topics.create_window_for( + AudioRuntimeTopic, max_size=10 + ) + await _runtime_window.wait_started() + logger.info("[moss] AudioRuntimeTopic 窗口已就绪") + + # ── 字幕订阅：Topic 总线 或 HTTP 旁路 ── + await setup_subtitle(matrix, matrix.container.force_fetch(ConfigStore)) + + # ── 注册 Channel 命令 ── + runtime = MossRuntime( + matrix=matrix, + registry=_registry, + course_storage=_course_storage, + runtime_window=_runtime_window, + ) + register_commands(chan, runtime) + + # ── 内部 HTTP 桥接 (:9733) — 供 main.py 发 Signal + 收 SSE ── + internal_app = create_internal_app(matrix, _course_storage, runtime) + + _internal_runner = aiohttp_web.AppRunner(internal_app) + await _internal_runner.setup() + _internal_site = aiohttp_web.TCPSite(_internal_runner, "127.0.0.1", 9733) + await _internal_site.start() + logger.info("内部 HTTP 桥接 → http://127.0.0.1:9733") + + await matrix.provide_channel(chan) + + # moss() 退出时清理内部 HTTP 桥接 + if _internal_runner is not None: + try: + await _internal_runner.cleanup() + except Exception: + pass + + +# =========================== MOSS =========================== + +# =========================== Bootstrap =========================== +@asynccontextmanager +async def lifespan(): + task = asyncio.create_task(moss()) + yield # ← 在 yield 之前是 startup，之后是 shutdown + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + + +app = rx.App() +app.register_lifespan_task(lifespan) +app.add_page(index, on_load=State.moss_listener) +# =========================== Bootstrap =========================== diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/state.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/state.py new file mode 100644 index 00000000..6d5ee485 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/state.py @@ -0,0 +1,116 @@ +import asyncio +import importlib +import logging +from dataclasses import dataclass +from pathlib import Path + +from ghoshell_common.contracts import YamlConfig, WorkspaceConfigs, DefaultFileStorage +from pydantic import Field + +from framework.helpers.layout_snapshot import LayoutSnapshot +from framework.runtime.event_generator import build +from moss_in_reflex.course_manager import CourseManager +from moss_in_reflex.lecture_brain import LectureBrain + +# =========================== Logger =========================== +logger = logging.getLogger("moss") +logger.setLevel(logging.DEBUG) +if not logger.handlers: + _h = logging.StreamHandler() + _h.setFormatter(logging.Formatter( + "%(asctime)s [moss] %(levelname)s %(message)s", + datefmt="%H:%M:%S", + )) + logger.addHandler(_h) + logger.propagate = False + + +# =========================== Config & Layouts =========================== + +class Config(YamlConfig): + relative_path = "config.yaml" + layouts: list[str] = Field(default_factory=list, description="List of layout names") + + +CONFIG = WorkspaceConfigs( + DefaultFileStorage(dir_=str(Path(__file__).parent.absolute())) +).get_or_create(Config()) + + +def load_layouts() -> list: + """从 layouts.toml 动态加载 Layout 类。""" + layouts = [] + for module_path in CONFIG.layouts: + module_name, class_name = module_path.rsplit(".", 1) + module = importlib.import_module(module_name) + cls = getattr(module, class_name) + layouts.append(cls) + logger.info(f"Loaded layout: {module_path}") + return layouts + + +# 外部系统和Reflex系统通信媒介 +QUEUE: asyncio.Queue = asyncio.Queue() # type: ignore[valid-type] + +# 注册可用的LAYOUT（从 layouts.toml 动态加载） +LAYOUTS = load_layouts() + +# 动态生成LAYOUT运行时 +LAYOUT_COMPONENTS = [(l.name(), l.create()) for l in LAYOUTS] +LAYOUT_CHANNEL_STATES = [] + +for i, _l in enumerate(LAYOUTS): + _n, _lc = LAYOUT_COMPONENTS[i] + LAYOUT_CHANNEL_STATES.append(build(_n, _lc.State, _l, QUEUE)) + + +# =========================== Layout Mirror =========================== + +@dataclass +class _LayoutMirror: + """Layout 状态镜像 — Reflex State 的模块级只读缓存。 + + self.layout 是权威数据源（持久化，hot-reload 存活），_LAYOUT 是模块级镜像。 + moss_listener 同步写，context_messages() 只读。 + """ + name: str = "simple" + + def get_component(self): + for n, c in LAYOUT_COMPONENTS: + if n == self.name: + return c + return LAYOUT_COMPONENTS[0][1] + + +_LAYOUT = _LayoutMirror() + +# Layout 状态快照 — moss_listener 循环末尾一把读，context_messages() 消费 +_SNAPSHOT_DIR = Path(__file__).resolve().parent / "snapshots" +_SNAPSHOTS: dict[str, LayoutSnapshot] = { + name: LayoutSnapshot(name, component.State, _SNAPSHOT_DIR) + for name, component in LAYOUT_COMPONENTS +} + + +# =========================== 课程 & 讲课状态 =========================== + +_COURSE_MGR = CourseManager() +_LECTURE_BRAIN = LectureBrain() + + +# =========================== SSE 队列 =========================== + +# 聊天 SSE +_SSE_EVENT: asyncio.Event = asyncio.Event() +_SSE_QUEUE: list[dict] = [] +_SSE_LOCK: asyncio.Lock = asyncio.Lock() + +# 字幕 SSE +_SUBTITLE_EVENT: asyncio.Event = asyncio.Event() +_SUBTITLE_QUEUE: list[dict] = [] +_SUBTITLE_LOCK: asyncio.Lock = asyncio.Lock() + +# 弹幕 SSE +_DANMAKU_EVENT: asyncio.Event = asyncio.Event() +_DANMAKU_QUEUE: list[dict] = [] +_DANMAKU_LOCK: asyncio.Lock = asyncio.Lock() diff --git a/.moss_ws/apps/ui/moshi/moss_in_reflex/subtitle.py b/.moss_ws/apps/ui/moshi/moss_in_reflex/subtitle.py new file mode 100644 index 00000000..a66b9e58 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/moss_in_reflex/subtitle.py @@ -0,0 +1,133 @@ +"""字幕子系统：Topic 总线消费 + HTTP 旁路兼容。 + +提供: +- new_subtitle_callback(): HTTP 旁路回调（fire-and-forget POST → :9733/_internal/subtitle_in） +- setup_subtitle(): 三层回退 → TopicWindow 消费协程 或 HTTP 旁路 +""" + +import asyncio +import importlib + +import aiohttp + +from ghoshell_moss.core.speech.subtitle_config import SubtitleTopicConfig +from ghoshell_moss.topics.audio import SubtitleTopic +from ghoshell_moss.contracts.speech import Speech +from ghoshell_moss.core.blueprint.environment import Environment + +from moss_in_reflex.state import ( + logger, + _SUBTITLE_LOCK, _SUBTITLE_QUEUE, _SUBTITLE_EVENT, +) + + +def new_subtitle_callback(): + """创建句级字幕回调函数（fire-and-forget HTTP POST 到内部桥接）。 + + 可用于： + - 同进程：moss() 中注入 matrix.container 的 Speech + - 跨进程：Ghost 运行时设置到其 Speech 实例 + + 回调签名：(text: str, is_final: bool, batch_id: str = "") -> None + """ + def _post(text: str, is_final: bool, batch_id: str = "") -> None: + async def _do(): + try: + async with aiohttp.ClientSession() as session: + await session.post( + "http://127.0.0.1:9733/_internal/subtitle_in", + json={"text": text, "is_final": is_final}, + timeout=aiohttp.ClientTimeout(total=2), + ) + except Exception: + pass # 字幕丢失非关键故障 + + try: + asyncio.get_running_loop().create_task(_do()) + except RuntimeError: + pass + + return _post + + +async def setup_subtitle(matrix, config_store) -> bool: + """配置字幕消费链路。 + + 三层回退： + 1. ConfigStore.get(SubtitleTopicConfig) — 读 mode 覆盖配置 + 2. Mode config 模块回退导入 — 直读 MOSS.modes..configs + 3. HTTP 旁路 — 同进程 Speech 注入（仅同进程有效） + + Returns: + True if Topic 路径已启用（TopicWindow 创建 + 消费协程已启动）， + False if HTTP 旁路（同进程兼容路径）。 + """ + subtitle_config = None + + # Layer 1: ConfigStore + try: + subtitle_config = config_store.get(SubtitleTopicConfig) + logger.info("[subtitle] SubtitleTopicConfig from ConfigStore: enable_topic=%s, topic_path=%s", + subtitle_config.enable_topic, subtitle_config.topic_path) + except Exception as e: + logger.warning("[subtitle] ConfigStore.get(SubtitleTopicConfig) 异常: %s (%s)", e, type(e).__name__) + + # Layer 2: Mode config 模块回退 + if subtitle_config is None or not subtitle_config.enable_topic: + mode_name = Environment.discover().moss_mode_name + if mode_name: + try: + mode_configs = importlib.import_module(f"MOSS.modes.{mode_name}.configs") + _stc = getattr(mode_configs, "subtitle_topic_config", None) + if isinstance(_stc, SubtitleTopicConfig): + subtitle_config = _stc + logger.info("[subtitle] SubtitleTopicConfig 从 mode %s 回退导入: enable_topic=%s", + mode_name, _stc.enable_topic) + except ImportError: + pass + + # Layer 3: Topic 路径 或 HTTP 旁路 + if subtitle_config is not None and subtitle_config.enable_topic: + # ── Topic 路径（新）：创建 TopicWindow + 消费协程 ── + _subtitle_window = matrix.session.topics.create_window_for( + SubtitleTopic, max_size=100, + topic_name=subtitle_config.topic_path, + ) + await _subtitle_window.wait_started() + logger.info("[subtitle] SubtitleTopic 窗口已就绪（Topic 总线）") + + async def _consume_subtitle(): + """在 asyncio 事件循环上消费 SubtitleTopic 窗口，写入 SSE 队列。 + + 通过 len()/values() 轮询（~100ms），与 AudioRuntimeTopic 门控的 + 轮询模式一致。每次只处理新增项，无线程安全问题。 + """ + consumed = 0 + while True: + try: + current_len = len(_subtitle_window) + if current_len > consumed: + for topic in _subtitle_window.values()[consumed:]: + async with _SUBTITLE_LOCK: + _SUBTITLE_QUEUE.append({ + "type": "full" if topic.is_final else "chunk", + "text": topic.text, + }) + _SUBTITLE_EVENT.set() + consumed = current_len + except Exception: + pass + await asyncio.sleep(0.1) + + asyncio.create_task(_consume_subtitle()) + return True + else: + # ── HTTP 旁路（旧，兼容）：同进程 Speech 注入 ── + _subtitle_cb = new_subtitle_callback() + try: + _speech = matrix.container.force_fetch(Speech) + _speech.set_subtitle_callback(_subtitle_cb) + logger.info("[subtitle] 字幕回调已注入 Speech（HTTP 旁路，同进程）") + except Exception: + logger.info("[subtitle] Speech 不在当前容器（HTTP 旁路在跨进程场景失效）") + return False diff --git a/.moss_ws/apps/ui/moshi/pyproject.toml b/.moss_ws/apps/ui/moshi/pyproject.toml new file mode 100644 index 00000000..5ac109e6 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/pyproject.toml @@ -0,0 +1,30 @@ +[project] +name = "moss-in-reflex" +version = "0.1.0" +description = "Add your description here" +license = { text = "Apache License 2.0" } +readme = "README.md" +requires-python = ">=3.12" +dependencies = [ + "aiohttp>=3.9", + "ghoshell-moss", + "ghoshell-moss[audio,zmq,host,ghost]", + "reflex==0.9.1", +] + + +[tool.uv.sources] +ghoshell-moss = { path = "../../../..", editable = true } + +[tool.pdm.build] +includes = [] +[build-system] +requires = ["pdm-backend"] +build-backend = "pdm.backend" + +[dependency-groups] +dev = [ + "ruff>=0.15.0", + "pytest>=8.3.5", + "pytest-asyncio>=1.1.0", +] diff --git a/.moss_ws/apps/ui/moshi/rxconfig.py b/.moss_ws/apps/ui/moshi/rxconfig.py new file mode 100644 index 00000000..9fe3ae61 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/rxconfig.py @@ -0,0 +1,11 @@ +import reflex as rx + +config = rx.Config( + app_name="moss_in_reflex", + plugins=[ + rx.plugins.SitemapPlugin(), + rx.plugins.TailwindV4Plugin(), + ], + frontend_packages=[ + ] +) \ No newline at end of file diff --git a/.moss_ws/apps/ui/moshi/scripts/dev_restart.sh b/.moss_ws/apps/ui/moshi/scripts/dev_restart.sh new file mode 100755 index 00000000..d71f2c65 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/scripts/dev_restart.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# 杀掉所有 reflex 旧进程 +set -euo pipefail + +PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$PROJECT_ROOT" + +echo "==> 查找并杀掉旧进程..." + +KILL_PIDS=() + +# reflex 相关 +for pid in $(pgrep -f "reflex run" 2>/dev/null || true); do + KILL_PIDS+=("$pid") + echo " 找到 reflex run (PID $pid)" +done +for pid in $(pgrep -f "react-router dev" 2>/dev/null || true); do + KILL_PIDS+=("$pid") + echo " 找到 react-router dev (PID $pid)" +done + +# moshi main.py 相关 +for pid in $(pgrep -f "moshi.*main.py" 2>/dev/null || true); do + KILL_PIDS+=("$pid") + echo " 找到 moshi main.py (PID $pid)" +done + +# 占用 moshi 端口 9731 的进程 +PORT_PID=$(lsof -ti:9731 2>/dev/null || true) +if [ -n "$PORT_PID" ]; then + KILL_PIDS+=($PORT_PID) + echo " 找到占用 9731 端口的进程 (PID $PORT_PID)" +fi + +if [ ${#KILL_PIDS[@]} -gt 0 ]; then + UNIQUE_PIDS=($(printf '%s\n' "${KILL_PIDS[@]}" | sort -n | uniq)) + echo "==> 杀掉 ${#UNIQUE_PIDS[@]} 个进程..." + for pid in "${UNIQUE_PIDS[@]}"; do + kill "$pid" 2>/dev/null || true + done + sleep 1 + for pid in "${UNIQUE_PIDS[@]}"; do + kill -9 "$pid" 2>/dev/null || true + done + echo " 已全部杀掉。" +else + echo " 没有找到需要杀掉的进程。" +fi + +echo "==> 清理完毕。" diff --git a/.moss_ws/apps/ui/moshi/start.sh b/.moss_ws/apps/ui/moshi/start.sh new file mode 100644 index 00000000..8d98e295 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/start.sh @@ -0,0 +1,4 @@ +bash "$(dirname "$0")/scripts/dev_restart.sh" +uv run reflex run & +sleep 2 +exec uv run python main.py \ No newline at end of file diff --git a/.moss_ws/apps/ui/moshi/static/app.js b/.moss_ws/apps/ui/moshi/static/app.js new file mode 100644 index 00000000..71d71699 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/static/app.js @@ -0,0 +1,126 @@ +let currentMode = "idle"; + +// ── Chat ───────────────────────────────────────────────────────────────── + +function addMessage(role, text) { + const div = document.createElement("div"); + div.className = "msg " + role; + const now = new Date(); + const sender = role === "user" ? "你" : role === "ai" ? "AI" : ""; + div.innerHTML = + '' + sender + "" + + '' + now.toLocaleTimeString() + "
" + + escapeHtml(text); + const msgs = document.getElementById("messages"); + msgs.appendChild(div); + msgs.scrollTop = msgs.scrollHeight; +} + +function escapeHtml(s) { + const d = document.createElement("div"); + d.textContent = s; + return d.innerHTML; +} + +// ── API ────────────────────────────────────────────────────────────────── + +async function api(path, body) { + if (body === undefined) body = {}; + const r = await fetch("/api" + path, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }); + return r.json(); +} + +// ── Actions ────────────────────────────────────────────────────────────── + +async function setMode(mode) { + currentMode = mode; + const r = await api("/mode", { mode }); + updateToolbar(mode); + document.getElementById("status").textContent = r.status || ""; + if (r.msg) addMessage("system", r.msg); +} + +async function sendCmd(cmd) { + const r = await api("/cmd", { cmd }); + document.getElementById("status").textContent = r.status || ""; + if (r.info) document.getElementById("chapter-info").textContent = r.info; + if (r.msg) addMessage("system", r.msg); +} + +async function selectCourse(name) { + const r = await api("/course/select", { name }); + document.getElementById("status").textContent = r.status || ""; + if (r.msg) addMessage("system", r.msg); +} + +async function sendChat() { + const input = document.getElementById("chat-input"); + const text = input.value.trim(); + if (!text) return; + input.value = ""; + addMessage("user", text); + + const r = await api("/chat", { text, mode: currentMode }); + if (r.reply) { + addMessage("ai", r.reply); + } else if (r.error) { + addMessage("system", "错误: " + r.error); + } + if (r.status) document.getElementById("status").textContent = r.status; + if (r.info) document.getElementById("chapter-info").textContent = r.info; +} + +function updateToolbar(mode) { + document.querySelectorAll("#toolbar button[data-mode]").forEach(function (b) { + b.classList.toggle("active", b.dataset.mode === mode); + }); +} + +// ── Init ───────────────────────────────────────────────────────────────── + +document.addEventListener("DOMContentLoaded", async function () { + // Mode buttons + document.querySelectorAll("#toolbar button[data-mode]").forEach(function (b) { + b.addEventListener("click", function () { + setMode(b.dataset.mode); + }); + }); + + // Course select + document.getElementById("course-select").addEventListener("change", function () { + selectCourse(this.value); + }); + + // Control buttons + document.querySelectorAll("#progress button[data-cmd]").forEach(function (b) { + b.addEventListener("click", function () { + sendCmd(b.dataset.cmd); + }); + }); + + // Chat + document.getElementById("send-btn").addEventListener("click", sendChat); + document.getElementById("chat-input").addEventListener("keydown", function (e) { + if (e.key === "Enter") sendChat(); + }); + + // Load courses + const cs = await api("/courses"); + const sel = document.getElementById("course-select"); + (cs || []).forEach(function (c) { + const o = document.createElement("option"); + o.value = c.name; + o.text = c.name + " (" + c.chapters + ")"; + sel.appendChild(o); + }); + + // Init state + const s = await api("/state"); + currentMode = s.mode || "idle"; + updateToolbar(currentMode); + document.getElementById("status").textContent = s.status || "就绪"; +}); diff --git a/.moss_ws/apps/ui/moshi/static/frontPrompt.md b/.moss_ws/apps/ui/moshi/static/frontPrompt.md new file mode 100644 index 00000000..482eb4b0 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/static/frontPrompt.md @@ -0,0 +1,137 @@ +# AI 原型图生成提示词 + +> 每条提示词均为**独立完整版本**，可直接复制到 Midjourney / DALL-E / ComfyUI 运行。 +> 全局风格：深色主题（#1a1a2e）、中文无衬线字体、16:9 横屏、极简沉浸、产品截图风格。 +> 每条提供 English 和中文两个版本，任选其一使用。 + +--- + +## B1. 讲课 L0 / L1 — 全屏舞台 + 字幕 + 弹幕 + +> L1 悬浮控制点为隐形热区，静态截图不可见，与 L0 共用同一提示词。 + +**English：** + +``` +A clean, immersive fullscreen lecture UI for a live teaching app, product screenshot style. The entire screen is a whiteboard canvas filling 100% of the 16:9 viewport — absolutely no toolbar, no sidebar, no chat panel, no menus. At the bottom of the screen, a semi-transparent dark gradient overlay bar displays live speech-to-text subtitles in large white sans-serif Chinese characters (e.g. "今天我们讨论三个核心问题"). In the middle area, multiple lines of colorful danmaku bullet comments drift from right to left at varying speeds, showing real audience messages like "这个讲得好！" in green, "能解释下这个概念吗？" in yellow, "666" in white. In the bottom-right corner, a single small circular "✕" exit button — 48px diameter, dark semi-transparent background, the icon subtly visible, designed to fade to 30% opacity after inactivity. The overall feel: a stage, not a software tool — the presenter stands beside the screen, the content fills every pixel, the interface disappears. Dark theme background (#1a1a2e), clean Chinese typography (PingFang SC style). Minimal, immersive, professional. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个简洁、沉浸的全屏讲课模式 UI，产品截图风格。整个 16:9 屏幕被一个白板画布 100% 填满——完全没有工具栏、没有侧边栏、没有聊天面板、没有菜单。屏幕底部有一条半透明深色渐变遮罩条，上面显示大号白色无衬线中文字体的实时语音转文字字幕（如"今天我们讨论三个核心问题"）。屏幕中间区域有多行彩色弹幕以不同速度从右向左飘过，内容为真实观众消息：绿色"这个讲得好！"、黄色"能解释下这个概念吗？"、白色"666"。右下角有一个极小的圆形"✕"退出按钮——直径 48px，深色半透明背景，图标若隐若现，无操作 3 秒后淡化为 30% 不透明度。整体感觉：不是软件工具，是舞台——讲师站在屏幕旁，内容填满每一个像素，界面消失了。深色主题背景（#1a1a2e），干净的中文排版（PingFang SC 风格）。极简、沉浸、专业。--ar 16:9 --v 6 +``` + +--- + +## B2. 讲课 L2 — 全屏舞台 + 字幕 + 弹幕 + 讲师后台面板 + +**English：** + +``` +A clean, immersive fullscreen lecture UI for a live teaching app, product screenshot style. The entire screen is a whiteboard canvas filling 100% of the 16:9 viewport — no toolbar, no sidebar, no chat panel. At the bottom, a semi-transparent dark gradient overlay bar displays live speech-to-text subtitles in large white sans-serif Chinese characters. In the middle area, multiple lines of colorful danmaku bullet comments drift from right to left with audience messages like "这个讲得好！" and "能解释下这个概念吗？". In the bottom-right corner, a small circular "✕" exit button (48px, dark semi-transparent). NEW ELEMENT — in the bottom-left corner, a small compact dark semi-transparent card panel (about 280x200px) serving as a presenter backend monitor. The card has a title row "🔥 热点问题" at top with a small fire emoji. Below it, a list of 3-5 clustered audience questions, each showing the question text in white (e.g. "AI 能替代讲师吗？") and a heat number badge in orange on the right (e.g. "23", "18", "12", "9", "5"). At the bottom of the card, a green status indicator dot with text "🤖 自动回复已开启". The overall feel: same immersive stage as the base lecture mode, but with a subtle ops panel for the presenter to monitor audience sentiment. Dark theme (#1a1a2e), clean Chinese typography, 16:9. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个简洁、沉浸的全屏讲课模式 UI，产品截图风格。整个 16:9 屏幕被白板画布 100% 填满——没有工具栏、没有侧边栏、没有聊天面板。屏幕底部有一条半透明深色渐变遮罩条，显示大号白色无衬线中文字体的实时语音转文字字幕。中间区域有多行彩色弹幕以不同速度从右向左飘过，内容为观众消息。右下角有一个极小的圆形"✕"退出按钮（48px，深色半透明）。新增元素——屏幕左下角有一个紧凑的深色半透明卡片面板（约 280x200px），作为讲师后台监控面板。卡片顶部标题行"🔥 热点问题"带有小火苗图标。下方是 3-5 条聚类后的观众提问列表，每条左侧显示白色提问文字（如"AI 能替代讲师吗？"），右侧显示橙色热度数字徽章（如"23"、"18"、"12"、"9"、"5"）。卡片底部有一个绿色状态指示灯和文字"🤖 自动回复已开启"。整体感觉：与基础讲课模式相同的沉浸舞台感，但增加了一个低调的后台面板供讲师监控观众情绪。深色主题（#1a1a2e），干净的中文排版，16:9 横屏。--ar 16:9 --v 6 +``` + +--- + +## B3. 练习 L0 — 主屏 + 演讲者手卡 + 计时器 + +**English：** + +``` +A clean practice mode UI for a presentation rehearsal app, product screenshot style. The full 16:9 screen shows a whiteboard/slide canvas filling the entire viewport — this is what the audience would see. There is absolutely no toolbar, no chat panel, no sidebar, no menus. In the bottom-right corner, a small floating dark semi-transparent card (about 300x160px) labeled "📝 演讲者手卡" (Speaker Cue Card) — this is visible only to the presenter, not the audience. The card shows three lines of text in white Chinese characters: Line 1 "📌 要点: 回顾核心论点——杠杆率决定天花板", Line 2 "🔄 过渡: 接下来我们看第三个案例...", Line 3 "📊 关键数据: 同比增长 37%". Prominently displayed in the center-bottom area is a large digital timer "⏱ 03:24" in white monospace font on a dark semi-transparent pill-shaped background. No webcam, no camera feed, no picture-in-picture. The feel: a private rehearsal booth — just the content, the cue cards to prevent forgetting lines, and the timer. Dark theme (#1a1a2e), clean Chinese typography, 16:9. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个简洁的练习模式 UI，用于演讲排练应用，产品截图风格。整个 16:9 屏幕展示白板/幻灯片画布填满视口——这是听众看到的画面。完全没有工具栏、没有聊天面板、没有侧边栏、没有菜单。右下角有一个浮动的小型深色半透明卡片（约 300x160px），标注"📝 演讲者手卡"——仅讲师可见，听众看不到。卡片显示三行白色中文文字：第一行"📌 要点: 回顾核心论点——杠杆率决定天花板"，第二行"🔄 过渡: 接下来我们看第三个案例..."，第三行"📊 关键数据: 同比增长 37%"。屏幕中央偏下位置醒目地显示一个大号数字计时器"⏱ 03:24"，白色等宽字体，深色半透明胶囊形背景。没有摄像头画面、没有 PIP 画中画。整体感觉：一间私人排练室——只有内容、防忘词的手卡、以及计时器。深色主题（#1a1a2e），干净的中文排版，16:9 横屏。--ar 16:9 --v 6 +``` + +--- + +## B4. 练习 L1 — 主屏 + 手卡 + 计时器 + AI 弹幕 + 压力滑块 + +**English：** + +``` +A clean practice mode UI for a presentation rehearsal app, product screenshot style. The full 16:9 screen shows a whiteboard/slide canvas filling the viewport (audience view). In the bottom-right corner, a small floating dark semi-transparent speaker cue card labeled "📝 演讲者手卡" showing three lines: "📌 要点: 回顾核心论点", "🔄 过渡: 接下来我们看第三个案例...", "📊 关键数据: 同比增长 37%". In the center-bottom area, a large digital timer "⏱ 03:24" in white monospace font on a dark pill-shaped background. NEW ELEMENTS — ① A danmaku layer in the middle area: AI-generated simulated audience comments drift right-to-left at varying speeds. Examples: a red skeptical comment "这个逻辑有问题吧？", a green encouraging comment "讲得不错！", a yellow question "能举个实际例子吗？", a white neutral "666". ② At the very bottom of the screen, a pressure level slider control bar: a horizontal bar spanning about 60% of the screen width, left label "低压" in green, right label "高压" in red, the track showing a green-to-red gradient, with a circular thumb indicator positioned at the current level. Above the slider, a small label "压力等级 Lv.2". The speaker cue card and timer remain visible. No webcam, no PIP. Dark theme (#1a1a2e), clean Chinese typography, 16:9. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个简洁的练习模式 UI，用于演讲排练应用，产品截图风格。整个 16:9 屏幕展示白板/幻灯片画布填满视口（听众视角）。右下角有一个浮动的小型深色半透明演讲者手卡，标注"📝 演讲者手卡"，三行内容："📌 要点: 回顾核心论点"、"🔄 过渡: 接下来我们看第三个案例..."、"📊 关键数据: 同比增长 37%"。屏幕中央偏下位置有大号数字计时器"⏱ 03:24"，白色等宽字体，深色胶囊形背景。新增元素——① 屏幕中间弹幕层：AI 生成的模拟观众弹幕以不同速度从右向左飘过。示例：红色质疑"这个逻辑有问题吧？"，绿色鼓励"讲得不错！"，黄色提问"能举个实际例子吗？"，白色中性"666"。② 屏幕最底部增加压力等级滑块控制条：一条横跨屏幕约 60% 宽度的水平滑轨，左侧绿色标签"低压"，右侧红色标签"高压"，轨道呈绿到红渐变，圆形滑块指示器停在当前档位。滑块上方有小字标注"压力等级 Lv.2"。手卡和计时器保持可见。没有摄像头、没有 PIP。深色主题（#1a1a2e），干净的中文排版，16:9 横屏。--ar 16:9 --v 6 +``` + +--- + +## B5. 练习 L2 — 复盘评分卡（练习结束后的界面） + +**English：** + +``` +A post-practice review scorecard UI for a presentation rehearsal app, product screenshot style. This is a fullscreen overlay that appears after the practice session ends. The background is a dark semi-transparent overlay (#1a1a2e at 95% opacity) covering the entire 16:9 viewport. At the top center, a title "🎯 练习复盘" in large white Chinese characters. Below it, centered prominently, a four-dimension radar/spider chart with labeled axes: 清晰度 (Clarity), 节奏 (Pacing), 互动 (Engagement), 内容 (Content). Each axis has a score number (e.g. 85, 72, 90, 78) and the filled area forms an irregular polygon in a semi-transparent blue. Below the radar chart, a horizontal timeline bar spanning about 80% of screen width, labeled "⏱ 时间轴". The timeline has colored markers: green dots at smooth sections, red dots at danmaku spike moments (labeled "弹幕密集"), amber warning dots at weak segments (labeled "建议改进"). Each marker has a small timestamp below it (e.g. 01:23, 05:47). At the bottom of the screen, two action buttons side by side: a primary blue button "✏️ 去改课" (Go Edit Course) on the left with a bold style, and a secondary gray button "🔄 再练一次" (Practice Again) on the right with an outlined style. The overall feel: a coach's data-driven feedback report — friendly but professional. Dark theme, Chinese typography, 16:9. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个练习结束后的复盘评分卡 UI，用于演讲排练应用，产品截图风格。这是一个全屏覆盖层，在练习 session 结束后弹出。背景为深色半透明遮罩（#1a1a2e，95% 不透明度）覆盖整个 16:9 视口。顶部居中显示大号白色中文标题"🎯 练习复盘"。其下方居中位置是一个四维雷达图/蛛网图，坐标轴标注：清晰度、节奏、互动、内容，每个轴上有分数数字（如 85、72、90、78），填充区域形成不规则多边形，使用半透明蓝色。雷达图下方是一条横跨屏幕约 80% 宽度的水平时间轴，标注"⏱ 时间轴"。时间轴上有彩色标记点：绿色圆点标注流畅段落，红色圆点标注弹幕密集时刻（旁注"弹幕密集"），琥珀色警告点标注建议改进段落（旁注"建议改进"）。每个标记点下方有小字时间戳（如 01:23、05:47）。屏幕底部有两个并排按钮：左侧主按钮"✏️ 去改课"，蓝色填充，粗体样式；右侧次按钮"🔄 再练一次"，灰色描边样式。整体感觉：一份教练视角的数据驱动反馈报告——友好但专业。深色主题，中文排版，16:9 横屏。--ar 16:9 --v 6 +``` + +--- + +## B6. 备课 L0 — 主屏 + AI 对话框 + 保存/退出 + +**English：** + +``` +A clean content editing mode UI for a course authoring app, product screenshot style. The 16:9 layout has three zones. Zone 1 — a very slim top bar (about 40px height): on the left, three mode tab labels in white Chinese text "备课" (highlighted/active, with a subtle blue underline), "练习", "讲课"; on the right, two buttons — "💾 保存" (Save, subtle blue background) and "✕ 退出" (Exit, dark gray). Zone 2 — the central editing area fills most of the remaining space, showing a whiteboard canvas with sample slide content: a title in large bold Chinese characters "产品增长策略", a subtitle, a bullet list of 3 items, and a placeholder image area on the right side. Zone 3 — a collapsible AI chat panel at the bottom (about 180px height): the panel has a dark background slightly lighter than the main theme, showing a conversation bubble from AI "需要我帮你调整第三章的结构吗？" in white text on a blue-tinted bubble, and below it a text input field with placeholder text "输入你的问题..." and a "发送" (Send) button on the right. No left sidebar, no right panel, no discussion area, no toolbar menus. The feel: a minimal writing workshop — just the content, an AI assistant in the bottom panel, and save/exit controls at top. Dark theme (#1a1a2e), Chinese typography, 16:9. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个简洁的内容编辑模式 UI，用于课程创作应用，产品截图风格。16:9 布局分为三区。区域一——顶部极窄栏（约 40px 高）：左侧三个白色中文模式标签"备课"（高亮/激活态，带淡蓝色下划线）、"练习"、"讲课"；右侧两个按钮——"💾 保存"（淡蓝色背景）和"✕ 退出"（深灰色）。区域二——中央编辑区填满大部分剩余空间，展示白板画布上的示例幻灯片内容：大号粗体中文标题"产品增长策略"、一行副标题、三条要点列表、右侧一个图片占位区域。区域三——底部可折叠 AI 聊天面板（约 180px 高）：面板背景为比主背景稍亮的深色，显示一条 AI 对话气泡"需要我帮你调整第三章的结构吗？"，白色文字，蓝色调气泡背景；下方是文本输入框，占位文字"输入你的问题..."，右侧"发送"按钮。没有左侧边栏、没有右侧面板、没有讨论区、没有工具栏菜单。整体感觉：一个极简的写作工坊——只有内容、底部的 AI 助手、以及顶部的保存和退出。深色主题（#1a1a2e），中文排版，16:9 横屏。--ar 16:9 --v 6 +``` + +--- + +## B7. 备课 L1 — 主屏 + 左侧大纲 + AI 对话框 + 保存 + +**English：** + +``` +A clean content editing mode UI for a course authoring app, product screenshot style. The 16:9 layout has four zones. Zone 1 — slim top bar (40px): left side mode tabs "备课" (active, blue underline), "练习", "讲课"; right side "💾 保存" and "✕ 退出" buttons. Zone 2 — NEW left sidebar: a dark panel (about 220px width) showing a course outline. The sidebar header says "📑 课程大纲" in white. Below it, a vertical list of chapters: "第一章 引言" with a filled green progress dot (completed), "第二章 市场分析" with a filled green dot, "第三章 增长策略" highlighted with a blue background and a half-filled dot (in progress), "第四章 案例研究" with an empty gray dot (pending), "第五章 总结" with an empty gray dot. Each chapter row has a small drag handle icon on the right. The right edge of the sidebar has a subtle vertical divider line with a small collapse arrow "◀" button. Zone 3 — central editing area (narrowed to accommodate sidebar), showing a whiteboard canvas with slide content. Zone 4 — bottom AI chat panel (same as L0). Dark theme (#1a1a2e), Chinese typography, 16:9. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个简洁的内容编辑模式 UI，用于课程创作应用，产品截图风格。16:9 布局分为四区。区域一——顶部极窄栏（40px）：左侧模式标签"备课"（激活态，蓝色下划线）、"练习"、"讲课"；右侧"💾 保存"和"✕ 退出"按钮。区域二——新增左侧大纲侧边栏：深色面板（约 220px 宽），顶部白色标题"📑 课程大纲"。下方是纵向章节列表："第一章 引言"带绿色实心进度圆点（已完成），"第二章 市场分析"带绿色实心点，"第三章 增长策略"以蓝色背景高亮，带半填充圆点（进行中），"第四章 案例研究"带灰色空心圆点（待完成），"第五章 总结"带灰色空心点。每行右侧有一个小型拖拽手柄图标。侧边栏右边缘有一条细微的纵向分隔线，带有一个小折叠箭头"◀"按钮。区域三——中央编辑区（因侧边栏而收窄），展示白板画布上的幻灯片内容。区域四——底部 AI 聊天面板（与 L0 相同）。深色主题（#1a1a2e），中文排版，16:9 横屏。--ar 16:9 --v 6 +``` + +--- + +## B8. 备课 L2 — 主屏 + 大纲 + 素材导入 + AI 蒸馏 + 保存 + +**English：** + +``` +A clean content editing mode UI for a course authoring app, product screenshot style. The 16:9 layout has all L1 elements plus two new additions. Zone 1 — slim top bar with mode tabs and save/exit buttons. Zone 2 — left outline sidebar with chapter list and progress dots (same as L1). Zone 3 — NEW element: a file drop zone at the top of the editing area. A dashed-border rectangular area spanning the width of the editing panel, with a central cloud-upload icon, primary text "📁 拖拽文件到此处导入" in white, secondary text "支持 PDF、图片、网页链接" in smaller gray text below, and a "浏览文件" (Browse) button on the right side. Below the drop zone, the whiteboard canvas shows slide content. Zone 4 — ENHANCED bottom AI chat panel (taller than L1, about 260px). Inside the panel: a progress indicator showing "🔍 正在蒸馏素材..." with a thin blue progress bar at 60%, below it a row of three story template cards — "🏆 英雄之旅" (selected, blue border), "💡 颠覆认知" (default), "📖 案例驱动" (default) — each as a small rounded card with title and brief description. Below the templates, a horizontal difficulty slider labeled "知识点深浅: 浅 ← → 深" with a circular thumb at the middle position. Below that, the standard text input field with "发送" button. Dark theme (#1a1a2e), Chinese typography, 16:9. --ar 16:9 --v 6 +``` + +**中文版：** + +``` +一个简洁的内容编辑模式 UI，用于课程创作应用，产品截图风格。16:9 布局包含所有 L1 元素外加两个新增。区域一——顶部极窄栏，模式标签和保存/退出按钮。区域二——左侧大纲侧边栏，章节列表和进度圆点（与 L1 相同）。区域三——新增元素：编辑区顶部出现一个文件拖拽区。一个虚线边框矩形区域横跨编辑面板宽度，中央有一个云上传图标，主要文字"📁 拖拽文件到此处导入"为白色，下方较小的灰色辅助文字"支持 PDF、图片、网页链接"，右侧有一个"浏览文件"按钮。拖拽区下方是白板画布展示幻灯片内容。区域四——增强版底部 AI 聊天面板（比 L1 更高，约 260px）。面板内从上到下依次为：进度指示器显示"🔍 正在蒸馏素材..."，带一条细蓝色进度条（60%）；下方一排三个故事线模板卡片——"🏆 英雄之旅"（已选中，蓝色边框）、"💡 颠覆认知"（默认态）、"📖 案例驱动"（默认态）——每张为小型圆角卡片，包含标题和简短描述。模板下方是一条水平深浅滑块，标注"知识点深浅: 浅 ← → 深"，圆形滑块停在中位。滑块下方是标准文本输入框和"发送"按钮。深色主题（#1a1a2e），中文排版，16:9 横屏。--ar 16:9 --v 6 +``` + +--- diff --git a/.moss_ws/apps/ui/moshi/static/index.html b/.moss_ws/apps/ui/moshi/static/index.html new file mode 100644 index 00000000..6c919f1d --- /dev/null +++ b/.moss_ws/apps/ui/moshi/static/index.html @@ -0,0 +1,49 @@ + + + + + +Moshi + + + + +
+ + + + + + + 就绪 +
+ +
+
+ +
+ + - + + + + +
+
+ + +
+ + + + diff --git a/.moss_ws/apps/ui/moshi/static/prepareL0.html b/.moss_ws/apps/ui/moshi/static/prepareL0.html new file mode 100644 index 00000000..2d0fb24e --- /dev/null +++ b/.moss_ws/apps/ui/moshi/static/prepareL0.html @@ -0,0 +1,219 @@ + + + + + + 备课模式 - 课件制作 UI + + + + + +
+ +
+
+
+ 备课 +
+
+
练习
+
讲课
+
+ +
+ + +
+
+ + +
+ +
+ + +
+
+
+ + AI 助手 +
+ +
+ +
+ +
+ + +
+
+ +
+ + + + + \ No newline at end of file diff --git a/.moss_ws/apps/ui/moshi/static/style.css b/.moss_ws/apps/ui/moshi/static/style.css new file mode 100644 index 00000000..000709e9 --- /dev/null +++ b/.moss_ws/apps/ui/moshi/static/style.css @@ -0,0 +1,202 @@ +* { margin: 0; padding: 0; box-sizing: border-box; } + +body { + font-family: -apple-system, system-ui, sans-serif; + background: #0f0f14; + color: #e0e0e0; + height: 100vh; + display: flex; + flex-direction: column; +} + +/* ── Toolbar ─────────────────────────────────────────────── */ + +#toolbar { + display: flex; + gap: 6px; + padding: 8px 12px; + background: #1a1a24; + border-bottom: 1px solid #2a2a3a; + align-items: center; + flex-shrink: 0; +} + +#toolbar button { + padding: 5px 14px; + border: 1px solid #3a3a4a; + border-radius: 5px; + background: #252535; + color: #ccc; + cursor: pointer; + font-size: 13px; + transition: 0.15s; +} + +#toolbar button:hover { + background: #353550; +} + +#toolbar button.active { + background: #c44; + border-color: #c44; + color: #fff; +} + +#toolbar select { + padding: 5px 10px; + border-radius: 5px; + background: #252535; + color: #ccc; + border: 1px solid #3a3a4a; + font-size: 13px; +} + +.spacer { flex: 1; } + +#status { + font-size: 12px; + color: #888; +} + +/* ── Main layout ─────────────────────────────────────────── */ + +#main { + display: flex; + flex: 1; + min-height: 0; +} + +/* ── Stage ───────────────────────────────────────────────── */ + +#stage-wrapper { + flex: 1; + display: flex; + flex-direction: column; +} + +#stage { + flex: 1; + border: none; + background: #fff; +} + +#progress { + display: flex; + gap: 8px; + padding: 6px 12px; + background: #1a1a24; + align-items: center; + flex-shrink: 0; +} + +#progress button { + padding: 4px 12px; + border: 1px solid #3a3a4a; + border-radius: 4px; + background: #252535; + color: #ccc; + cursor: pointer; + font-size: 12px; +} + +#progress button.primary { + background: #c44; + border-color: #c44; +} + +#progress .info { + font-size: 12px; + color: #888; +} + +/* ── Sidebar / Chat ──────────────────────────────────────── */ + +#sidebar { + width: 360px; + display: flex; + flex-direction: column; + background: #14141e; + border-left: 1px solid #2a2a3a; + flex-shrink: 0; +} + +#messages { + flex: 1; + overflow-y: auto; + padding: 12px; + display: flex; + flex-direction: column; + gap: 8px; +} + +.msg { + padding: 8px 10px; + border-radius: 6px; + font-size: 13px; + line-height: 1.5; + max-width: 100%; + word-break: break-word; +} + +.msg.ai { + background: #1a2a3a; + align-self: flex-start; +} + +.msg.user { + background: #2a1a2a; + align-self: flex-end; +} + +.msg.system { + background: #1a1a1a; + color: #888; + align-self: center; + font-size: 11px; +} + +.msg .sender { + font-size: 10px; + color: #888; + margin-bottom: 2px; +} + +.msg .time { + font-size: 10px; + color: #555; + float: right; +} + +/* ── Input ───────────────────────────────────────────────── */ + +#input-area { + display: flex; + padding: 8px; + gap: 6px; + border-top: 1px solid #2a2a3a; +} + +#input-area input { + flex: 1; + padding: 8px 10px; + border-radius: 5px; + border: 1px solid #3a3a4a; + background: #1a1a24; + color: #eee; + font-size: 13px; + outline: none; +} + +#input-area input:focus { + border-color: #c44; +} + +#input-area button { + padding: 8px 16px; + border-radius: 5px; + border: none; + background: #c44; + color: #fff; + cursor: pointer; + font-size: 13px; +} diff --git a/.moss_ws/apps/ui/moshi/static/teachingL0.html b/.moss_ws/apps/ui/moshi/static/teachingL0.html new file mode 100644 index 00000000..852f454e --- /dev/null +++ b/.moss_ws/apps/ui/moshi/static/teachingL0.html @@ -0,0 +1,399 @@ + + + + + + 讲课模式 — Moshi + + + + + + +
+ + + + +
+
+ + + +
+ + +
+

+ +

+
+ + + + + + + + + + + diff --git a/.moss_ws/apps/ui/moshi/tests/__init__.py b/.moss_ws/apps/ui/moshi/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/.moss_ws/configs/subtitle_topic.yml b/.moss_ws/configs/subtitle_topic.yml new file mode 100644 index 00000000..d1c28948 --- /dev/null +++ b/.moss_ws/configs/subtitle_topic.yml @@ -0,0 +1,3 @@ +# dump from `ghoshell_moss.core.speech.subtitle_config:SubtitleTopicConfig` +enable_topic: false +topic_path: moshi/subtitle diff --git a/.moss_ws/ghosts/echo/mode-default/mcp_hub.yml b/.moss_ws/ghosts/echo/mode-default/mcp_hub.yml new file mode 100644 index 00000000..8334de11 --- /dev/null +++ b/.moss_ws/ghosts/echo/mode-default/mcp_hub.yml @@ -0,0 +1,2 @@ +# dump from `ghoshell_moss.channels.mcp_hub:MCPHubConfig` +servers: {} diff --git a/.moss_ws/src/MOSS/manifests/configs.py b/.moss_ws/src/MOSS/manifests/configs.py index f5785ca0..d4b27cc8 100644 --- a/.moss_ws/src/MOSS/manifests/configs.py +++ b/.moss_ws/src/MOSS/manifests/configs.py @@ -21,6 +21,7 @@ from ghoshell_moss.channels.mcp_hub import MCPHubConfig from ghoshell_moss.contracts.audio import AudioCaptureConfig from ghoshell_moss.contracts.llms import LLMConfig +from ghoshell_moss.core.speech.subtitle_config import SubtitleTopicConfig # 全局只做类型注册（import 即注册类引用 → is_override=False → 文件持久化）。 # 类实例覆盖（is_override=True，仅内存不写文件）在 mode configs.py 中声明。 diff --git a/.moss_ws/src/MOSS/manifests/topics.py b/.moss_ws/src/MOSS/manifests/topics.py index 19479590..88969b31 100644 --- a/.moss_ws/src/MOSS/manifests/topics.py +++ b/.moss_ws/src/MOSS/manifests/topics.py @@ -10,3 +10,4 @@ # # 发现路径：MOSS.manifests.topics # 深入：moss codex get-interface ghoshell_moss.core.concepts.topic:TopicModel +from ghoshell_moss.topics.audio import SubtitleTopic diff --git a/.moss_ws/src/MOSS/modes/show/configs.py b/.moss_ws/src/MOSS/modes/show/configs.py index 36e9450c..a09421bf 100644 --- a/.moss_ws/src/MOSS/modes/show/configs.py +++ b/.moss_ws/src/MOSS/modes/show/configs.py @@ -4,3 +4,8 @@ # tts_override = TTSManagerConfig(default_speaker="大壹") # is_override=True from MOSS.manifests.configs import * # noqa: F403 from ghoshell_moss.channels.web_bookmark import WebConfig + +# ── 字幕 Topic 总线：替代旧 HTTP 旁路 ── +# 启用后 Ghost 进程 SpeechServiceProvider 通过 Zenoh 发布 SubtitleTopic， +# Reflex 进程通过 TopicWindow 订阅后渲染为 SSE 字幕。 +subtitle_topic_config = SubtitleTopicConfig(enable_topic=True, topic_path="moshi/subtitle") diff --git a/src/ghoshell_moss/core/speech/__init__.py b/src/ghoshell_moss/core/speech/__init__.py index c805f13d..638dce99 100644 --- a/src/ghoshell_moss/core/speech/__init__.py +++ b/src/ghoshell_moss/core/speech/__init__.py @@ -5,3 +5,4 @@ from ghoshell_moss.core.speech.null import NullSpeech from ghoshell_moss.core.speech.speech_module import SpeechChannelModule, build_content_command from ghoshell_moss.core.speech.stream_tts_speech import BaseTTSSpeech, TTSSpeechStream +from ghoshell_moss.core.speech.subtitle_config import SubtitleTopicConfig diff --git a/src/ghoshell_moss/core/speech/stream_tts_speech.py b/src/ghoshell_moss/core/speech/stream_tts_speech.py index 1f1aefbe..dd8d9510 100644 --- a/src/ghoshell_moss/core/speech/stream_tts_speech.py +++ b/src/ghoshell_moss/core/speech/stream_tts_speech.py @@ -1,5 +1,6 @@ import asyncio import logging +import re from typing import Optional, Callable, Coroutine import numpy as np @@ -28,6 +29,7 @@ def __init__( player: StreamAudioPlayer, tts_batch: TTSBatch, logger: LoggerItf, + subtitle_callback: Callable[[str, bool], None] | None = None, ): batch_id = tts_batch.batch_id() super().__init__(id=batch_id) @@ -50,10 +52,35 @@ def __init__( self._has_audio_data = False self._log_prefix = "[TTSSpeechStream id=%s] " % batch_id + # ── 句级字幕 ── + self._batch_id = batch_id + self._subtitle_callback = subtitle_callback + self._subtitle_buffer = "" # 累积文本，按标点切句 + self._subtitle_queue: list[str] = [] # 待回调的句子 + def _buffer(self, text: str) -> None: self._text_buffer += text self._tts_batch.feed(text) + # 句级字幕：累积文本，按标点切句入队 + if self._subtitle_callback is not None: + self._subtitle_buffer += text + while True: + match = re.search(r'[。！？；\n]', self._subtitle_buffer) + if not match: + break + idx = match.end() + sentence = self._subtitle_buffer[:idx].strip() + self._subtitle_buffer = self._subtitle_buffer[idx:] + if sentence: + self._subtitle_queue.append(sentence) + + def _flush_subtitle(self) -> None: + """将 subtitle buffer 中剩余文本作为最后一句推入队列。""" + if self._subtitle_buffer.strip(): + self._subtitle_queue.append(self._subtitle_buffer.strip()) + self._subtitle_buffer = "" + def _commit(self) -> None: self._tts_batch.commit() @@ -94,7 +121,6 @@ async def _play_loop(self) -> None: await self.start_synthesis() self.logger.debug("%s start new audio playing", self._log_prefix) async for item in self._tts_batch.items(): - # 将 buffer 的内容 data = item["audio"] self._player.add( data, @@ -102,6 +128,19 @@ async def _play_loop(self) -> None: audio_type=self._audio_type, rate=self._sample_rate, ) + + # 句级字幕：每帧音频对应一句字幕 + if self._subtitle_callback is not None: + if self._subtitle_queue: + text = self._subtitle_queue.pop(0) + self._subtitle_callback(text, False, self._batch_id) + elif self._subtitle_buffer.strip(): + # 队列空但 buffer 有余留（标点未触发切句） + self._flush_subtitle() + if self._subtitle_queue: + text = self._subtitle_queue.pop(0) + self._subtitle_callback(text, False, self._batch_id) + await asyncio.sleep(0) self.logger.debug("%s add audio %d bytes", self._log_prefix, len(data)) await self._player.wait_play_done() @@ -110,6 +149,19 @@ async def _play_loop(self) -> None: except Exception as e: self.logger.exception("%s play failed: %s", self._log_prefix, e) finally: + # 句级字幕：flush 所有剩余 + 发送 final + if self._subtitle_callback is not None: + self._flush_subtitle() + while self._subtitle_queue: + text = self._subtitle_queue.pop(0) + try: + self._subtitle_callback(text, False, self._batch_id) + except Exception: + pass + try: + self._subtitle_callback("", True, self._batch_id) + except Exception: + pass self._play_done_event.set() # 冗余的 clear. await self._player.clear() @@ -149,6 +201,7 @@ def __init__( player: StreamAudioPlayer, tts: TTS, logger: Optional[LoggerItf] = None, + subtitle_callback: Callable[[str, bool], None] | None = None, ): self.logger = logger or logging.getLogger("moss") self._player = player @@ -161,6 +214,7 @@ def __init__( self._started = False self._closing = False self._closed_event = ThreadSafeEvent() + self._subtitle_callback = subtitle_callback def tts(self) -> TTS: return self._tts @@ -168,6 +222,13 @@ def tts(self) -> TTS: def player(self) -> StreamAudioPlayer: return self._player + def set_subtitle_callback(self, callback: Callable[[str, bool], None] | None) -> None: + """注入句级字幕回调，所有后续 new_stream 创建的流都会收到回调。 + + 可在运行时动态设置。讲课场景注入，非讲课场景设为 None。 + """ + self._subtitle_callback = callback + def new_stream(self, *, batch_id: Optional[str] = None) -> SpeechStream: batch_id = batch_id or unique_id() tts_batch = self._tts.new_batch(batch_id=batch_id) @@ -182,6 +243,7 @@ def new_tts_stream(self, batch: TTSBatch) -> SpeechStream: player=self._player, tts_batch=batch, logger=self.logger, + subtitle_callback=self._subtitle_callback, ) return stream diff --git a/src/ghoshell_moss/core/speech/subtitle_config.py b/src/ghoshell_moss/core/speech/subtitle_config.py new file mode 100644 index 00000000..0d5bcfb8 --- /dev/null +++ b/src/ghoshell_moss/core/speech/subtitle_config.py @@ -0,0 +1,35 @@ +"""字幕 Topic 发布配置。 + +Ghost 进程 SpeechServiceProvider 读取此配置，决定是否启用 Zenoh Topic +总线字幕发布（替代旧 HTTP 旁路）。 + +过渡策略：enable_topic 默认 False（保持向后兼容，走旧 HTTP 旁路）。 +设置 True 后走 Topic 总线路径。 +""" + +from pydantic import Field +from ghoshell_moss.contracts.configs import ConfigType + +__all__ = ["SubtitleTopicConfig"] + + +class SubtitleTopicConfig(ConfigType): + """字幕 Topic 发布配置。 + + Ghost 进程 SpeechServiceProvider.factory() 读取此配置： + - enable_topic=False（默认）：不注入字幕回调，字幕功能禁用 + - enable_topic=True：从容器获取 TopicService，创建 pub 闭包注入给 BaseTTSSpeech + """ + + enable_topic: bool = Field( + default=False, + description="是否启用 Topic 总线字幕发布。True 时 SpeechServiceProvider 创建 Topic 发布闭包", + ) + topic_path: str = Field( + default="moshi/subtitle", + description="字幕发布的 Topic 路径。消费端（Reflex）需订阅同一路径", + ) + + @classmethod + def conf_name(cls) -> str: + return "subtitle_topic" diff --git a/src/ghoshell_moss/host/providers/speech_service_provider.py b/src/ghoshell_moss/host/providers/speech_service_provider.py index 55c30780..56c1d10b 100644 --- a/src/ghoshell_moss/host/providers/speech_service_provider.py +++ b/src/ghoshell_moss/host/providers/speech_service_provider.py @@ -1,6 +1,9 @@ from ghoshell_moss.contracts.speech import Speech, TTS, StreamAudioPlayer from ghoshell_moss.contracts.logger import LoggerItf from ghoshell_moss.core.speech import BaseTTSSpeech +from ghoshell_moss.core.speech.subtitle_config import SubtitleTopicConfig +from ghoshell_moss.core.concepts.topic import TopicName +from ghoshell_moss.topics.audio import SubtitleTopic from ghoshell_container import IoCContainer, Provider, INSTANCE __all__ = ['TTSSpeechServiceProvider'] @@ -15,8 +18,59 @@ def factory(self, con: IoCContainer) -> INSTANCE: logger = con.force_fetch(LoggerItf) player = con.force_fetch(StreamAudioPlayer) tts = con.force_fetch(TTS) + + # ── 字幕 Topic 总线回调（可选，由 SubtitleTopicConfig 控制）── + subtitle_callback = None + try: + from ghoshell_moss.contracts.configs import ConfigStore + from ghoshell_moss.core.concepts.topic import TopicService + config_store = con.force_fetch(ConfigStore) + subtitle_config = config_store.get(SubtitleTopicConfig) + logger.info("[speech] SubtitleTopicConfig from ConfigStore: enable_topic=%s, topic_path=%s", + subtitle_config.enable_topic, subtitle_config.topic_path) + except Exception as e: + logger.warning("[speech] ConfigStore.get(SubtitleTopicConfig) 异常: %s (%s)", e, type(e).__name__) + subtitle_config = None + + # ConfigStore 未命中 mode 覆盖时，直接从 mode config 模块回退 + if subtitle_config is None or not subtitle_config.enable_topic: + import importlib + from ghoshell_moss.core.blueprint.environment import Environment + mode_name = Environment.discover().moss_mode_name + if mode_name: + try: + mode_configs = importlib.import_module(f"MOSS.modes.{mode_name}.configs") + _stc = getattr(mode_configs, "subtitle_topic_config", None) + if isinstance(_stc, SubtitleTopicConfig): + subtitle_config = _stc + logger.info("[speech] SubtitleTopicConfig 从 mode %s 回退导入: enable_topic=%s", + mode_name, _stc.enable_topic) + except ImportError: + pass + + if subtitle_config is not None and subtitle_config.enable_topic: + topic_service = con.force_fetch(TopicService) + topic_name = TopicName(subtitle_config.topic_path) + + def _publish_subtitle(text: str, is_final: bool, batch_id: str = "") -> None: + """句级字幕发布闭包：组装 SubtitleTopic → pub 到 Zenoh 总线。 + + 由 TTSSpeechStream._play_loop() 在音频播放线程中逐句调用。 + TopicService.pub() 内部通过 run_in_executor 安全投递到 Zenoh。 + """ + try: + topic_service.pub( + SubtitleTopic(text=text, is_final=is_final, batch_id=batch_id), + name=str(topic_name), + ) + except Exception: + pass # 字幕丢失非关键故障，与旧 HTTP 旁路行为一致 + + subtitle_callback = _publish_subtitle + return BaseTTSSpeech( logger=logger, player=player, tts=tts, + subtitle_callback=subtitle_callback, ) diff --git a/src/ghoshell_moss/topics/__init__.py b/src/ghoshell_moss/topics/__init__.py index 10c28cbb..a88a52dc 100644 --- a/src/ghoshell_moss/topics/__init__.py +++ b/src/ghoshell_moss/topics/__init__.py @@ -6,4 +6,4 @@ by the TopicService at runtime. """ from ghoshell_moss.core.concepts.topic import TopicModel, TopicService, Subscriber, Publisher -from .audio import AudioRuntimeTopic, SpeechTopic +from .audio import AudioRuntimeTopic, SpeechTopic, SubtitleTopic diff --git a/src/ghoshell_moss/topics/audio.py b/src/ghoshell_moss/topics/audio.py index 6c0799df..a3572ebd 100644 --- a/src/ghoshell_moss/topics/audio.py +++ b/src/ghoshell_moss/topics/audio.py @@ -11,6 +11,7 @@ __all__ = [ "AudioRuntimeTopic", "SpeechTopic", + "SubtitleTopic", ] @@ -74,3 +75,26 @@ def topic_type(cls) -> str: @classmethod def default_topic_name(cls) -> str: return "speech" + + +class SubtitleTopic(TopicModel): + """句级字幕 Topic — 跨进程投递实时字幕文本。 + + Published by the Ghost process Speech pipeline (TTSSpeechStream) + via TopicService.pub(), consumed by the Reflex process via + TopicWindow for SSE-based browser rendering. + + Replaces the old HTTP bypass (new_subtitle_callback() → :9733/_internal/subtitle_in). + """ + + text: str = Field(default="", description="字幕文本内容") + is_final: bool = Field(default=False, description="当前句子或音频流是否结束") + batch_id: str = Field(default="", description="语音流批次 ID，用于多轮对话下的字幕流对齐") + + @classmethod + def topic_type(cls) -> str: + return "core/speech/subtitle" + + @classmethod + def default_topic_name(cls) -> str: + return "moshi/subtitle"