| 1 | <div align="center"> |
| 2 | |
| 3 | <picture> |
| 4 | <img src="assets/vimax.svg" width="130px" style="border: none; box-shadow: none;" alt="ViMax Logo"> |
| 5 | </picture> |
| 6 | </div> |
| 7 | <div align="center"> |
| 8 | <h1 align="center">ViMax: Agentic Video Generation</h1> |
| 9 | <p align="center"> |
| 10 | <a href="https://trendshift.io/repositories/15299" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/15299" alt="HKUDS%2FViMax | Trendshift" width="250" height="55"/></a> |
| 11 | <a href="https://trendshift.io/repositories/15299?utm_source=trendshift-badge&utm_medium=badge&utm_campaign=badge-trendshift-15299" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/15299/weekly?language=Python" alt="HKUDS%2FViMax | Trendshift" width="250" height="55"/></a> |
| 12 | </p> |
| 13 | <div align="center"> |
| 14 | </div> |
| 15 | |
| 16 | <p align="center"> |
| 17 | <img src="https://img.shields.io/badge/Python-3.12-00d9ff?style=flat-square&logo=python&logoColor=white&labelColor=1a1a2e"> |
| 18 | <a href="https://github.com/astral-sh/uv"><img src="https://img.shields.io/badge/uv-Ready-ff6b6b?style=flat-square&logo=uv&logoColor=white&labelColor=1a1a2e"></a> |
| 19 | <img src="https://img.shields.io/badge/License-MIT-4ecdc4?style=flat-square&logo=opensourceinitiative&logoColor=white" alt="MIT License"> |
| 20 | <a href="https://github.com/HKUDS/ViMax/releases/tag/v1.2.0"><img src="https://img.shields.io/badge/Version-v1.2.0-2563eb?style=flat-square&labelColor=1a1a2e" alt="ViMax v1.2.0"></a> |
| 21 | <a href='https://www.youtube.com/@AI-Creator-is-here'><img src='https://img.shields.io/badge/YouTube-ff0000?style=flat-square&logo=youtube&logoColor=white&labelColor=1a1a2e' /></a> |
| 22 | <a href='https://arxiv.org/abs/2606.07649'><img src='https://img.shields.io/badge/arXiv-2606.07649-b31b1b?style=flat-square&logo=arxiv&logoColor=white&labelColor=1a1a2e' /></a> |
| 23 | </p> |
| 24 | |
| 25 | <p align="center"> |
| 26 | <a href="./Communication.md"><img src="https://img.shields.io/badge/Feishu-Group-07c160?style=flat-square&logo=lark&logoColor=white&labelColor=1a1a2e"></a> |
| 27 | <a href="./Communication.md"><img src="https://img.shields.io/badge/WeChat-Group-07c160?style=flat-square&logo=wechat&logoColor=white&labelColor=1a1a2e"></a> |
| 28 | <a href="readme.md"><img src="https://img.shields.io/badge/English-1a1a2e?style=flat-square"></a> |
| 29 | <a href="README_ZH.md"><img src="https://img.shields.io/badge/中文版-1a1a2e?style=flat-square"></a> |
| 30 | <a href="#quick-start"><img src="https://img.shields.io/badge/Quick%20Start-Get%20Started%20Now-FFC107?style=flat-square&logo=rocket&logoColor=white&labelColor=1a1a2e"></a> |
| 31 | </p> |
| 32 | |
| 33 | </div> |
| 34 | |
| 35 | --- |
| 36 | |
| 37 | ### 🚨 Current Video Generation Limitations: |
| 38 | - ❌ **Limited to Short Clips** - Most AI tools generate only seconds of footage. <br> |
| 39 | - ❌ **Consistency Chaos** - Characters and scenes change unpredictably across frames. <br> |
| 40 | - ❌ **Visual-Only Focus** - Missing scripts, audio, narrative structure, and storytelling depth. <br> |
| 41 | |
| 42 | ### 💡 ViMax Solution: |
| 43 | 🎬 **Director**, **Screenwriter**, **Producer**, and **Video Generator** **All-in-One**! We're exploring a future where AI becomes a complete creative powerhouse. 💡 Simply input your concept. ViMax autonomously handles the rest. It orchestrates scriptwriting, storyboarding, character creation, and final video generation—all end-to-end. 🚀 |
| 44 | |
| 45 | https://github.com/user-attachments/assets/5bad46b2-8276-4e1d-9480-3522640744b2 |
| 46 | |
| 47 | |
| 48 | |
| 49 | </div> |
| 50 | |
| 51 | --- |
| 52 | |
| 53 | |
| 54 | ### 📰 **News** |
| 55 | |
| 56 | - **2026-07-20** 🚀 **ViMax v1.2.0** introduces the Web UI with named projects, Agent Loop conversations, artifact and storyboard previews, render checkpoints, file uploads, provider settings, and dark mode. |
| 57 | - **2026-07-17** 🎬 OpenRouter GPT Image 2 image generation and Seedance 2.0 Fast video generation support added. |
| 58 | - **2026-06-28** 🛠️ Agent Loop and TUI stability update: stronger LLM retries, persistent render status, landscape image guards, and Script2Video resume fixes. |
| 59 | - **2026-06-09** 📄 Technical report released. |
| 60 | - **2026-06-08** 🤖 Agents Loop + TUI workflow integrated for interactive planning, revision, rendering control, session reuse, and context compaction. |
| 61 | - **2026-06-07** 📖 Novel2Video workflow released. |
| 62 | - **2026-06-01** 🎬 Google Omni video generator support added. |
| 63 | - **2026-03-23** ⚡ MiniMax chat model provider support added. |
| 64 | |
| 65 | --- |
| 66 | |
| 67 | |
| 68 | |
| 69 | ## 📑 Table of Contents |
| 70 | |
| 71 | - [✨ Key Features](#key-features) |
| 72 | - [🔮 Demos](#Video-Demos-Generated-from-Scratch) |
| 73 | - [🚀 Quick Start](#quick-start) |
| 74 | |
| 75 | --- |
| 76 | ## ✨ Key Features |
| 77 | |
| 78 | ViMax is an agentic video creation framework that connects narrative planning, visual consistency, image generation, video generation, and final assembly in one extensible workflow. |
| 79 | |
| 80 | - **Idea2Video** — turn a short concept into structured stories, characters, scripts, storyboards, shots, and a finished video. |
| 81 | - **Script2Video** — convert an explicit screenplay into controllable multi-scene, multi-shot video while preserving its creative intent. |
| 82 | - **Novel2Video** — adapt long-form fiction into episodic visual narratives with narrative compression, character tracking, and scene planning. |
| 83 | - **AutoCameo** — place a person or pet from a reference photo into generated stories while maintaining a consistent appearance. |
| 84 | - **Agent Loop and TUI** — discuss ideas, revise plans, resume sessions, review text artifacts, and control rendering from one interactive workspace. |
| 85 | - **Web UI** — manage named projects, work with the ViMax Agent, upload source files, inspect artifacts and storyboard progress, preview renders, and configure providers from the browser. |
| 86 | - **Consistent production pipeline** — coordinate references, first frames, camera continuity, and final assembly end to end. |
| 87 | - **Parallelized generation** — generate compatible shots and media assets concurrently to accelerate multi-shot video production. |
| 88 | |
| 89 | --- |
| 90 | |
| 91 | ## 🔮Video Demos Generated from Scratch |
| 92 | |
| 93 | <table> |
| 94 | <tr> |
| 95 | |
| 96 | <td align="center" width="33%"> |
| 97 | <video src="https://github.com/user-attachments/assets/c2fb27b0-218c-4976-b3d6-2abf8ea06be7" controls width="100%"></video> |
| 98 | </td> |
| 99 | <td align="center" width="33%"> |
| 100 | <video src="https://github.com/user-attachments/assets/bfa566a8-688d-4d53-a9e2-6cedeb4a399d" controls width="100%"></video> |
| 101 | </td> |
| 102 | <td align="center" width="33%"> |
| 103 | <video src="https://github.com/user-attachments/assets/49f61134-4f78-4285-9a9e-bb5e3e0c4abf" controls width="100%"></video> |
| 104 | </td> |
| 105 | </tr> |
| 106 | <tr> |
| 107 | <td align="center" width="33%"> |
| 108 | <video src="https://github.com/user-attachments/assets/a950f449-a15c-449b-a1b8-c393951aa9be" controls width="100%"></video> |
| 109 | </td> |
| 110 | <td align="center" width="33%"> |
| 111 | <video src="https://github.com/user-attachments/assets/bb3ff0fd-9433-4806-886a-3f77b61d06ec" controls width="100%"></video> |
| 112 | </td> |
| 113 | <td align="center" width="33%"> |
| 114 | <video src="https://github.com/user-attachments/assets/2624a3f0-9f66-4fa4-b527-45c0ea0353fc" controls width="100%"></video> |
| 115 | </td> |
| 116 | </tr> |
| 117 | |
| 118 | <tr> |
| 119 | <td align="center" width="33%"> |
| 120 | <video src="https://github.com/user-attachments/assets/5dbb80f7-aff0-4211-940c-a898f91fb80c" controls width="100%"></video> |
| 121 | </td> |
| 122 | <td align="center" width="33%"> |
| 123 | <video src="https://github.com/user-attachments/assets/cc0b0bcd-e7db-4839-950b-0b03949637bd" controls width="100%"></video> |
| 124 | </td> |
| 125 | <td align="center" width="33%"> |
| 126 | <video src="https://github.com/user-attachments/assets/85919b59-80f0-461a-af7e-a93d3fb412fc" controls width="100%"></video> |
| 127 | </td> |
| 128 | </tr> |
| 129 | |
| 130 | |
| 131 | |
| 132 | |
| 133 | |
| 134 | |
| 135 | |
| 136 | |
| 137 | |
| 138 | |
| 139 | |
| 140 | |
| 141 | |
| 142 | |
| 143 | |
| 144 | |
| 145 | |
| 146 | |
| 147 | </table> |
| 148 | |
| 149 | --- |
| 150 | |
| 151 | ### 🖥️ **ViMax Web UI** |
| 152 | |
| 153 | <div align="center"> |
| 154 | <img src="assets/vimax-web-ui.png" width="100%" alt="ViMax Web UI workspace"> |
| 155 | </div> |
| 156 | |
| 157 | The Web UI brings agent conversation, project artifacts, storyboard previews, and render progress into one visual workspace. |
| 158 | |
| 159 | --- |
| 160 | |
| 161 | ### 🎯 **End-to-End Video Creation Engine** |
| 162 | |
| 163 | **The Challenges**: |
| 164 | |
| 165 | - 🌅 **Reference Images**: Time-consuming acquisition, organization, and alignment of reference frames that accurately capture characters, objects, positions, and environments. |
| 166 | |
| 167 | - 🫠 **Consistency Check**: Sometimes, the image generator may generate unusable images even if it is given the correct characters, position, environment reference image and prompts. |
| 168 | |
| 169 | - 📄 **Scripts Generation**: Professional and high-quality videos need to have rich information density and structured design. |
| 170 | |
| 171 | - 📝 **Storyboard Design**: Converting stories into visual narratives requires expertise in cinematography, scene composition, and visual storytelling that most creators lack. |
| 172 | |
| 173 | - 🎬 **Shot Design**: Creating coherent camera sequences with proper angles, transitions, and pacing while maintaining narrative flow across complex scenes. |
| 174 | |
| 175 | - 🎨 **Development Delays**: Ensuring character appearances, environments, and artistic style remain consistent across hundreds of shots in long-form content. |
| 176 | |
| 177 | - ⏱️ **Production Efficiency**: Traditional video creation involves multiple specialists and lengthy workflows, creating barriers for independent creators and rapid prototyping. |
| 178 | |
| 179 | - 🎥 **Scaling AI Generated Video**: AI-generated videos are usually only a few seconds long, high-quality long videos at the minute or even hour level require complex cross-scene continuity and multi-storyboards design and processing capabilities. |
| 180 | |
| 181 | |
| 182 | **ViMAX**: eliminates these production bottlenecks by automating the entire video creation pipeline from narrative input to final video output. |
| 183 | |
| 184 | --- |
| 185 | |
| 186 | |
| 187 | ### 🔥 **Why ViMax?** |
| 188 | |
| 189 | | 🧠 **Effortless Production** | 🚀 **Complete Creative Freedom** | 🔊 **Audio and Video Binding** | 🎨 **Professional Quality** | 🤩 **Interactive Video** |
| 190 | |:---:|:---:|:---:|:---:|:---:| |
| 191 | | One-Prompt to Finished Video | From Any Narrative to Reality | Synchronized Storytelling | Movie-Grade Output | Make Your Own Cameo Video |
| 192 | | Skip the technical complexity—just describe your vision and let ViMax handle script generation, storyboarding, shot design, reference management, and consistency validation | No creative limits—whether it's a trailer, short story, novel chapter, or original concept, ViMax intelligently structures narratives and designs cinematography to bring any idea to life | Seamlessly integrate character voice, and sound effects with visual content to create immersive experiences where audio and video work in perfect harmony | Automated quality control ensures character consistency, proper scene composition, and professional visual standards across every frame of your video | Interact in your own short stories by uploading your photo—ViMax intelligently integrates you as a character with consistent appearance and natural interactions throughout the entire video |
| 193 | |
| 194 | ViMax now also includes an **Agents Loop + TUI** workflow for interactive planning, revision, rendering control, session reuse, and context compaction while preserving the original direct pipeline entrypoints. |
| 195 | |
| 196 | |
| 197 | --- |
| 198 | |
| 199 | ### ☄️ **Roadmap** |
| 200 | |
| 201 | - ✅ 🖥️ **Web frontend workspace with artifact, storyboard, and render previews** |
| 202 | - ✅ 🤖 **Agent Loop + TUI interactive workflow** |
| 203 | - ✅ 🎬 **Seedance 2.0 Fast video generator support** |
| 204 | - ✅ 🖼️ **GPT Image 2 image generator support** |
| 205 | |
| 206 | --- |
| 207 | |
| 208 | |
| 209 | |
| 210 | |
| 211 | ## 🚀Quick Start |
| 212 | |
| 213 | ### 🖥️ **Environment** |
| 214 | |
| 215 | ``` |
| 216 | OS: Linux, Windows |
| 217 | ``` |
| 218 | |
| 219 | ### 📥 **Clone and Install** |
| 220 | We use uv to manage the environment. For uv installation, please refer to the https://docs.astral.sh/uv/getting-started/installation/. |
| 221 | ```bash |
| 222 | git clone https://github.com/HKUDS/ViMax.git |
| 223 | cd ViMax |
| 224 | uv sync |
| 225 | ``` |
| 226 | |
| 227 | |
| 228 | <details> |
| 229 | <summary><strong>Agent TUI</strong></summary> |
| 230 | |
| 231 | ViMax also provides a minimal TUI for interactive agent-based video creation. Create a private local configuration from the tracked example first: |
| 232 | |
| 233 | ```bash |
| 234 | cp configs/agent.example.yaml configs/agent.local.yaml |
| 235 | ``` |
| 236 | |
| 237 | Then configure the model and API key information in `configs/agent.local.yaml`, including the LLM, image generator, and video generator, as shown below. |
| 238 | ```yaml |
| 239 | llm: |
| 240 | model_provider: openai |
| 241 | model: <YOUR_LLM_MODEL> |
| 242 | base_url: <YOUR_LLM_BASE_URL> |
| 243 | api_key: <YOUR_API_KEY> |
| 244 | |
| 245 | image: |
| 246 | model: <YOUR_IMAGE_MODEL> |
| 247 | base_url: <YOUR_IMAGE_BASE_URL> |
| 248 | api_key: <YOUR_API_KEY> |
| 249 | |
| 250 | video: |
| 251 | model: <YOUR_VIDEO_MODEL> |
| 252 | base_url: <YOUR_VIDEO_BASE_URL> |
| 253 | api_key: <YOUR_API_KEY> |
| 254 | ``` |
| 255 | |
| 256 | Then, start the TUI from the ViMax root directory: |
| 257 | ```bash |
| 258 | vimax tui |
| 259 | ``` |
| 260 | |
| 261 | Start a new session or resume an existing one: |
| 262 | ```bash |
| 263 | vimax tui new |
| 264 | vimax tui resume |
| 265 | vimax tui resume <session_id> |
| 266 | ``` |
| 267 | |
| 268 | You can also keep `configs/agent.local.yaml` empty and provide the same values through environment variables, such as `VIMAX_LLM_API_KEY`, `VIMAX_IMAGE_API_KEY`, and `VIMAX_VIDEO_API_KEY`. |
| 269 | |
| 270 | </details> |
| 271 | |
| 272 | <details> |
| 273 | <summary><strong>Web UI</strong></summary> |
| 274 | |
| 275 | The Web UI uses the same ViMax agent runtime, sessions, tools, and private `configs/agent.local.yaml` configuration as the TUI. Node.js 18 or newer is required. |
| 276 | |
| 277 | From the `ViMax` repository root, install the frontend dependencies once and start the local server: |
| 278 | |
| 279 | ```bash |
| 280 | cd web |
| 281 | npm install |
| 282 | npm run dev |
| 283 | ``` |
| 284 | |
| 285 | Open [http://127.0.0.1:4173](http://127.0.0.1:4173) in your browser. The Web UI supports named projects, agent conversations, slash commands, artifact and render inspection, storyboard previews, file uploads, and provider settings. |
| 286 | |
| 287 | The server binds to `127.0.0.1` by default. When ViMax runs on a remote machine, forward the port from your local computer: |
| 288 | |
| 289 | ```bash |
| 290 | ssh -N -L 4173:127.0.0.1:4173 <user>@<server> |
| 291 | ``` |
| 292 | |
| 293 | Use `VIMAX_WEB_PORT` when another local port is required: |
| 294 | |
| 295 | ```bash |
| 296 | VIMAX_WEB_PORT=4174 npm run dev |
| 297 | ``` |
| 298 | |
| 299 | </details> |
| 300 | |
| 301 | <details> |
| 302 | <summary><strong>Usage</strong></summary> |
| 303 | |
| 304 | main_idea2video.py is used to convert your ideas into videos. |
| 305 | You need to configure the model and API key information in the configs/idea2video.yaml file, including three parts—the chat model, the image generator, and the video generator, as shown below |
| 306 | ```yaml |
| 307 | chat_model: |
| 308 | init_args: |
| 309 | model: google/gemini-2.5-flash-lite-preview-09-2025 |
| 310 | model_provider: openai |
| 311 | api_key: <YOUR_API_KEY> |
| 312 | base_url: https://openrouter.ai/api/v1 |
| 313 | |
| 314 | image_generator: |
| 315 | class_path: tools.ImageGeneratorNanobananaGoogleAPI |
| 316 | init_args: |
| 317 | api_key: <YOUR_API_KEY> |
| 318 | |
| 319 | video_generator: |
| 320 | class_path: tools.VideoGeneratorVeoGoogleAPI |
| 321 | init_args: |
| 322 | api_key: <YOUR_API_KEY> |
| 323 | |
| 324 | working_dir: .working_dir/idea2video |
| 325 | ``` |
| 326 | |
| 327 | Then, provide a simple yet thoughtful idea and the corresponding creative requirements in main_idea2video.py. |
| 328 | ```bash |
| 329 | idea = \ |
| 330 | """ |
| 331 | If a cat and a dog are best friends, what would happen when they meet a new cat? |
| 332 | """ |
| 333 | user_requirement = \ |
| 334 | """ |
| 335 | For children, do not exceed 3 scenes. |
| 336 | """ |
| 337 | style = "Cartoon" |
| 338 | ``` |
| 339 | |
| 340 | main_script2video.py generates a video based on a specific script. |
| 341 | You similarly need to set up the API configuration in configs/script2video.yaml file. Then, provide a scene script and the corresponding creative requirements in main_script2video.py, as shown below. |
| 342 | ```python |
| 343 | script = \ |
| 344 | """ |
| 345 | EXT. SCHOOL GYM - DAY |
| 346 | A group of students are practicing basketball in the gym. The gym is large and open, with a basketball hoop at one end and a large crowd of spectators at the other end. John (18, male, tall, athletic) is the star player, and he is practicing his dribble and shot. Jane (17, female, short, athletic) is the assistant coach, and she is helping John with his practice. The other students are watching the practice and cheering for John. |
| 347 | John: (dribbling the ball) I'm going to score a basket! |
| 348 | Jane: (smiling) Good job, John! |
| 349 | John: (shooting the ball) Yes! |
| 350 | ... |
| 351 | """ |
| 352 | user_requirement = \ |
| 353 | """ |
| 354 | Fast-paced with no more than 20 shots. |
| 355 | """ |
| 356 | style = "Animate Style" |
| 357 | ``` |
| 358 | |
| 359 | </details> |
| 360 |