The Autonomous Showrunner

An AI stream director for Twitch. It watches your chat and your gameplay on screen in real time and produces the show for you — switches OBS scenes, plays sounds, posts chat messages, shows on-screen overlays, and (optionally) runs Twitch predictions or polls. You keep playing; it runs the show.

Built for the AMD Developer Hackathon (Act 2). Every model call — the fast reflex classifier, the reasoning planner, and the vision layer that reads the screen — runs on Fireworks AI on AMD Instinct MI300X, with automatic failover to a self-hosted model on AMD ROCm. There is no non-AMD inference anywhere in the system.

showrunner/
├── director-core/
│   ├── main.py                    <- FastAPI entry point
│   ├── context_bus.py             <- rolling fused-context window
│   ├── reflex.py                  <- fast per-event classifier
│   ├── planner.py                 <- slower reasoning agent
│   ├── vision.py                  <- CV: OBS screenshots -> VLM -> vision_event
│   ├── prediction.py              <- prediction lifecycle (open/lock/resolve/cancel)
│   ├── policy.py                  <- cooldowns, toggles, kill switch, autonomy
│   ├── llm_client.py              <- shared OpenAI-compatible caller + failover
│   ├── prompts.py                 <- all prompt templates, versioned
│   ├── decision_log.py            <- SQLite log of every decision made
│   ├── config.py                  <- env-based configuration
│   ├── mock_client.py             <- inject synthetic events for local testing
│   ├── eval_vision.py             <- vision accuracy benchmark (12-frame set)
│   ├── showrunner-bridge.js       <- Firebot custom script that owns everything
│   ├── director-core.spec         <- PyInstaller build spec
│   ├── StreamSounds/              <- drop .mp3/.wav/.ogg here; filename = sound id
│   ├── requirements.txt
│   └── .env.example
├── README.md                       <- this file

Status. Ships with all effect categories disabled by default and AUTOMATION_PAUSED_ON_START=True (shadow mode). Every decision is logged with its reasoning; nothing touches your stream until you explicitly opt in per category and lift the pause.

---

What it does

Sees two things.

• Chat. A fast reflex model tags every message (hype / laughter / confusion / question / toxic / backseat / neutral).
• The screen. A vision-language model (Kimi-K2 on Fireworks, or a local Qwen2.5-VL on failover) reads OBS frames every ~1 s through a cheap local frame-diff gate, and classifies on-screen game state into boss_fight_start / boss_defeated / player_death / victory_screen / cutscene / normal.

Acts seven ways.

• Switch OBS scenes · play sound effects · post chat messages · show on-screen overlays
• Open Twitch predictions — auto-resolves via CV when it sees the win/loss screen
• Open Twitch polls
• Run a chat-message poll ("type 1 or 2 to vote") — works on any account, no Affiliate / channel points required

Thinks in two tiers. Cheap reflex on every event → escalates only "interesting" moments to the reasoning planner. A local frame-diff gate skips static frames so the vision model isn't spammed.

Safety. Shadow mode, per-action toggles, cooldowns, kill switch (API + Firebot hotkey), one-prediction-at-a-time, autonomous-resolve confirmation call, cancel-refund on timeout, full decision log. Every guardrail is plain, auditable Python — nothing safety-critical is left to a prompt.

Resilience (AMD model story). A single AI backend mode (fireworks / self_hosted / failover) drives all three layers. On failover, every call tries Fireworks first and auto-retries on a self-hosted multimodal model (Ollama / vLLM on AMD ROCm) if the cloud errors. A Fireworks outage no longer stops the show — it degrades to local AMD.

---

Prerequisites

• Firebot v5+ — https://firebot.app/ (Windows/Mac)
• OBS Studio with the built-in WebSocket server enabled (Tools → WebSocket Server Settings → Enable, note the port/password). Only needed if you turn the vision layer on.
• A Fireworks AI API key — https://fireworks.ai/api-keys. This powers reflex, planner, and vision by default. All three are served on AMD Instinct MI300X.

Optional:

• Ollama (https://ollama.com) with a multimodal model pulled (e.g. qwen2.5-vl or llama3.2-vision) — used when MODEL_MODE=self_hosted or as the failover target in failover mode. Runs on AMD ROCm if you have it.
• Python 3.10+ — only needed if you plan to develop against the source instead of running the packaged director-core.exe.

---

Install — the streamer path (packaged, no terminal)

This is how the tool is meant to be used day-to-day. Firebot launches director-core.exe, hands it its settings, and stops it when Firebot closes. You never open a terminal or edit a .env file.

1. Put the packaged files somewhere permanent, e.g. C:\Showrunner\:

   C:\Showrunner\
   ├── director-core.exe
   └── StreamSounds\        (drop your .mp3/.wav files here)
   

   These live in director-core/dist/ after a build (see the developer path below), or grab them from the release you were given.

2. Install the bridge into Firebot. Firebot → Settings → Scripts → Manage Startup Scripts — the dialog shows the scripts folder. Copy director-core/showrunner-bridge.js into it, click Add New Script, and select the file.

3. Fill in the script's settings form (Firebot's own UI — no .env):

   • Auto-launch Director Core → ✅ on
   • Director Core .exe location → point at C:\Showrunner\director-core.exe
   • Fireworks API Key — your fw_...
   • OBS WebSocket host / port / password (leave the password blank if OBS's authentication toggle is off)
   • Director Style / Current Game / PC Specs
   • Enable each action category you want live (all off by default)
   • Boss-Fight Reaction Mode: chat for now (works on any account), prediction or poll once your channel reaches Twitch Affiliate
   • Start in Shadow Mode → ✅ leave on for your first stream

4. Restart Firebot. Firebot's log should show [Showrunner] Connected to Director Core. — you're done. Read TESTING.md before disabling shadow mode.

Kill switch: three independent ways to stop everything, by design.

• Firebot hotkey with a Fetch URL effect → POST http://localhost:8765/control/pause
• API: curl -X POST http://localhost:8765/control/pause
• Just close Firebot — the bridge stops the child process cleanly.

---

Install — the developer path (from source)

Use this if you're editing the Python code. The bridge auto-detects an already-running dev instance and connects to it instead of spawning a duplicate, so you can leave Firebot open while iterating.

cd director-core
python -m venv show
show/Scripts/activate         # macOS/Linux: source show/bin/activate
pip install -r requirements.txt

cp .env.example .env
# edit .env — at minimum set FIREWORKS_API_KEY

Run it:

python -m uvicorn main:app --host 0.0.0.0 --port 8765 --reload

Health check:

curl http://localhost:8765/status

To ship a packaged build (single .exe, sounds folder, no Python dep):

pip install pyinstaller
pyinstaller director-core.spec
# -> dist/director-core.exe
cp -r StreamSounds dist/

---

Configuration reference

Model backend (one setting, three modes)

MODEL_MODE=fireworks              # fireworks | self_hosted | failover

# Fireworks (primary; served on AMD Instinct MI300X)
FIREWORKS_API_KEY=fw_...
FIREWORKS_REFLEX_MODEL=accounts/fireworks/models/glm-5p2
FIREWORKS_PLANNER_MODEL=accounts/fireworks/models/qwen3p7-plus
VISION_MODEL=accounts/fireworks/models/kimi-k2p6

# Self-hosted / AMD ROCm (used in self_hosted or as failover)
SELFHOSTED_BASE_URL=http://localhost:11434/v1     # or http://<amd-cloud-ip>:8000/v1
SELFHOSTED_MODEL=qwen2.5-vl
SELFHOSTED_API_KEY=                                # blank for Ollama

• fireworks — all three layers use Fireworks (AMD MI300X). Default.
• self_hosted — all three use the one local/AMD model at SELFHOSTED_BASE_URL. Fully offline / cost-free.
• failover — Fireworks first, auto-retry against self-hosted on any error/timeout. Fail-closed (reflex→neutral, planner→empty, vision→skip) only if both fail. For vision failover, the self-hosted model must be multimodal (Qwen2.5-VL, Llama-3.2-Vision, LLaVA-family).

Vision layer

ENABLE_VISION=True
OBS_WS_HOST=localhost
OBS_WS_PORT=4455
OBS_WS_PASSWORD=                  # blank if auth is off in OBS
VISION_SOURCE_NAME=               # blank = current program scene

Tune-knobs (defaults are sensible):

VISION_CAPTURE_INTERVAL_S=1.0     # loop tick
VISION_MIN_VLM_INTERVAL_S=3.0     # hard cap on VLM calls
VISION_DIFF_THRESHOLD=6.0         # frame-diff gate sensitivity
VISION_SUSTAINED_FRAMES=2         # streak before "boss_fight_start" acts
VISION_OPEN_CONFIDENCE=0.6        # min confidence to open a prediction
VISION_RESOLVE_CONFIDENCE=0.75    # min confidence to auto-resolve

Boss-fight reaction

BOSS_POLL_MODE=chat               # chat | prediction | poll

# Predictions/polls (need Twitch Affiliate)
ENABLE_CREATE_PREDICTION=False
ENABLE_CREATE_POLL=False
PREDICTION_AUTONOMY=full          # full | auto_open | propose
PREDICTION_TIMEOUT_S=300          # cancel-refund if no clear outcome

# "Closes soon" chat reminder before a prediction/poll window ends
BET_REMINDER_ENABLED=True
BET_REMINDER_LEAD_S=30

BOSS_POLL_MODE	Reaction on a detected boss fight	Needs Affiliate?
chat (default)	Chat-message poll + on-screen overlay	❌ works on any account
prediction	Real Twitch prediction; auto-resolves via CV	✅
poll	Real Twitch poll (ends itself)	✅

Safety flags — leave these alone until TESTING.md says otherwise

AUTOMATION_PAUSED_ON_START=True   # ← shadow mode
ENABLE_SWITCH_SCENE=False
ENABLE_PLAY_SOUND=False
ENABLE_POST_CHAT=False
ENABLE_OVERLAY_CUE=False

Cooldowns (seconds between repeated same-type actions):

COOLDOWN_SWITCH_SCENE=15
COOLDOWN_PLAY_SOUND=8
COOLDOWN_POST_CHAT=30
COOLDOWN_OVERLAY_CUE=10
COOLDOWN_CREATE_PREDICTION=300
COOLDOWN_CREATE_POLL=180

Streamer profile (shapes chat replies)

STREAMER_STYLE=hype               # hype | chill | competitive | variety
CURRENT_GAME=Minecraft
PC_SPECS=Ryzen 7 7500F, RTX 5070 12GB, 32GB 6400MHz CL36

PC_SPECS is the AI's only answer to "what are your specs?" chat questions — it repeats it verbatim, never guesses.

---

Adding sounds

Drop .mp3 / .wav / .ogg files into StreamSounds/. The filename (minus the extension) is the sound id — no code changes, no restart. The AI is told about them automatically at the next decision. Ships with hype_horn.mp3 and sad_trombone.mp3.

---

HTTP API

GET  /status                                  overall state (models, prediction, vision)
GET  /decisions?limit=50                      last N decisions with reasoning

POST /control/pause                           kill switch — stop all effects
POST /control/resume                          arm effects
POST /control/toggle/{action_type}?enabled=…  per-category enable/disable

POST /control/prediction/confirm              (propose mode) open the stashed proposal
POST /control/prediction/resolve/{outcome}    manually resolve open prediction
POST /control/prediction/cancel               manually cancel + refund

POST /test/mock-event                         inject a synthetic event (see mock_client.py)

---

Two ways to verify it works

1. Vision accuracy (no Firebot, no OBS)

cd director-core
python eval_vision.py <folder-of-screenshots>

Prints each frame's predicted label + confidence and a X/N score. On the bundled 12-frame Minecraft set the current setup scores 10/12 exact, 12/12 acceptable on Kimi-K2 / Fireworks.

2. Full pipeline dry-run (no Firebot, no OBS, no Twitch)

python scratchpad/dry_run.py

Fakes OBS with recorded frames + a mock Firebot bridge on the local WebSocket endpoint, feeds a scripted timeline (boss → boss → death), and asserts that exactly one twitch:create-prediction and one twitch:resolve-prediction reach the mock Firebot. Ends on [PASS] FULL DRY RUN OK.

---

Next step

TEST FULLY before disabling shadow mode. The whole point of this architecture is that annoying the streamer is the expensive failure mode, not the missed opportunity. Prove the reasoning is trustworthy against a whole session of logs before you ever let it touch your broadcast.