ComfyUI AI Story Generator 🎬

An automated story generation system that creates coherent multi-scene narratives with AI-generated images and videos using ComfyUI, Ollama, and state-of-the-art models.

🌟 Features

• AI-Driven Story Generation: Uses Ollama with Qwen3-Coder to create coherent multi-scene story outlines
• Image Generation: Leverages Qwen Image model with Lightning LoRA for fast, high-quality images
• Video Generation: Creates animated videos from images using Wan 2.2 i2v models with 4-step LoRA
• Optimized for Speed: Reduced resolution and step counts for faster generation
• Customizable Video Length: Choose from preset lengths or define custom frame counts
• Automatic Scene Continuity: Maintains character consistency across scenes
• Complete Logging: Saves all prompts, seeds, and settings for reproducibility

🎥 Demo

Generate a 4-scene story with images and videos from a simple text prompt like "Dragon Ball" or "Space Adventure"!

📋 Requirements

Software Dependencies

• Python 3.8+
• ComfyUI (with API server running)
• Ollama (with API server running)

Python Packages

pip install requests

Required Models

ComfyUI Models

Download and place these in your ComfyUI model directories:

Checkpoints/UNet:

• qwen_image_fp8_e4m3fn.safetensors - Qwen Image model
• wan2.2_i2v_high_noise_14B_fp8_scaled.safetensors - Wan 2.2 high noise
• wan2.2_i2v_low_noise_14B_fp8_scaled.safetensors - Wan 2.2 low noise

CLIP:

• qwen_2.5_vl_7b_fp8_scaled.safetensors - Qwen CLIP
• umt5_xxl_fp8_e4m3fn_scaled.safetensors - UMT5 CLIP

VAE:

• qwen_image_vae.safetensors - Qwen VAE
• wan_2.1_vae.safetensors - Wan VAE

LoRA:

• Qwen-Image-Lightning-8steps-V1.0.safetensors - Image generation speedup
• wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors - Video high noise
• wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors - Video low noise

Ollama Model

ollama pull qwen3-coder:30b

🚀 Quick Start

1. Start ComfyUI Server

   # Navigate to your ComfyUI directory and start the server
   python main.py --listen 127.0.0.1 --port 8188

2. Start Ollama Server

   ollama serve

3. Run the Story Generator

   python gpt_comfy_launcher_2.py

4. Follow the Prompts

   • Choose video length (short/medium/long/custom)
   • Enter a story topic (e.g., "Dragon Ball", "Space Adventure")
   • Wait for the magic to happen!

⚙️ Configuration

Edit the following constants in gpt_comfy_launcher_2.py:

COMFY_HOST = "http://127.0.0.1:8188"  # ComfyUI API endpoint
OLLAMA_HOST = "http://127.0.0.1:11434"  # Ollama API endpoint
MODEL = "qwen3-coder:30b"  # Ollama model to use
NUM_SCENES = 4  # Number of scenes to generate

# Speed optimizations
IMAGE_WIDTH = 480  # Image width in pixels
IMAGE_HEIGHT = 480  # Image height in pixels
DEFAULT_VIDEO_LENGTH = 33  # Default frames (~2 seconds at 16fps)
IMAGE_STEPS = 6  # Sampling steps for image generation
VIDEO_STEPS = 3  # Total sampling steps for video generation

📁 Output Structure

Generated content is saved to the story_outputs/ directory:

story_outputs/
├── story_log.json  # Complete log of all prompts, seeds, and settings
└── [Generated files appear in ComfyUI's output folder]

ComfyUI saves files to its own output/ directory (typically ComfyUI/output/):

• Images: scene_01_image_*.png, scene_02_image_*.png, etc.
• Videos: scene_01_video_*.mp4, scene_02_video_*.mp4, etc.

🎨 How It Works

1. Story Outline Generation: Ollama creates a coherent 4-scene story structure
2. Image Prompt Generation: Each scene gets a detailed Stable Diffusion prompt with character continuity
3. Image Generation: ComfyUI creates images using Qwen Image model
4. Video Motion Prompt: AI generates dynamic camera/character motion descriptions
5. Video Generation: ComfyUI animates images using Wan 2.2 i2v models
6. Logging: All prompts, seeds, and settings saved for reproducibility

🎯 Video Length Options

• Short (33 frames / ~2 seconds): Fastest generation, ideal for quick tests
• Medium (49 frames / ~3 seconds): Balanced speed and motion
• Long (81 frames / ~5 seconds): More detailed motion, longer generation time
• Custom: Enter any frame count (typical range: 17-161 frames)

🔧 Troubleshooting

ComfyUI Output Not Found

• Ensure ComfyUI is running and accessible at the configured host/port
• Check that ComfyUI's output/ directory exists and is writable
• The script auto-detects ComfyUI's output folder in common locations

Image to Video Conversion Fails

• Verify all required models are downloaded and in correct directories
• Check ComfyUI's input/ directory exists (script copies images there)
• Ensure sufficient VRAM (video generation requires ~14GB)

Ollama Connection Issues

• Verify Ollama server is running: ollama list
• Check if the model is downloaded: ollama pull qwen3-coder:30b
• Ensure OLLAMA_HOST is set correctly

🎓 Technical Details

Image Generation Pipeline

• Model: Qwen Image (fp8 quantized)
• LoRA: Lightning 8-steps for speedup
• Sampler: Euler with simple scheduler
• CFG Scale: 1.0
• Resolution: 480x480 (configurable)

Video Generation Pipeline

• Model: Wan 2.2 i2v (dual-noise approach)
• LoRA: LightX2V 4-steps for both noise levels
• Sampler: Euler Advanced (2-stage sampling)
• FPS: 16 frames per second
• Length: 33-161 frames (customizable)

📝 License

MIT License - See LICENSE file for details

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

🙏 Acknowledgments

• ComfyUI - Powerful node-based UI for Stable Diffusion
• Ollama - Local LLM inference
• Qwen Team - Image generation models
• Wan Team - Video generation models

📧 Contact

For questions or issues, please open an issue on GitHub.

---

Note: This project requires significant computational resources. A GPU with at least 16GB VRAM is recommended for smooth operation.