Generate 10-second 1080p videos with audio from text or images

MAGI-2 Preview

Magi-2 Preview is a 114B-parameter unified audio-video generation model that
activates just 6B parameters per token. Built on MagiMoE and co-designed across
architecture, systems, and data, it explores an efficient path to scaling video
generation. The architecture, the training system built around it, and the data
pipeline are described in MAGI-2 Preview: Scaling Video Generation Models
Efficiently; the weights are on Hugging
Face at sand-ai/MAGI-2-preview.

This repository is the inference code. It generates video from a text prompt
(T2V) or from a prompt plus a still image (I2V), with sound generated alongside
the video and muxed into the output file. Clips are 10 seconds long, which is
the only duration the model currently supports. Generation runs in two stages:
magi2_preview denoises the clip at low resolution, and magi2_refiner takes
that result up to 1080p.

Requirements

• NVIDIA Hopper GPUs, 8 of them.
• Python 3.12 and a recent CUDA toolkit.
• ffmpeg on PATH, to mux the audio track. Without it the video is still
  written, just silently.

Setup

Docker

The published image, sandai/magi-2-preview,
already has the dependencies built, including the ones that need a compiler:

docker pull sandai/magi-2-preview:latest
docker run --gpus all -it -v /path/to/ckpt:/workspace/ckpt sandai/magi-2-preview:latest

There is a tag per commit as well, sandai/magi-2-preview:<commit>. Name that
one when reporting a result, because latest moves; the image also records what
it was built from in /etc/magi2-build-info.

Building it yourself is only necessary to change a dependency version, or to
work somewhere the registry is not reachable:

docker build -t magi-2-preview:local .

From source

pip install -r requirements.txt

MAGI-2 also needs MagiAttention
and MagiCompiler. The pinned
revisions are recorded as build args in the Dockerfile.

Checkpoints

Neither transformer has been step-distilled, so the denoising step count is
where most of the wall-clock time goes. That is the difference between the
release that exists today and the one that follows:

Everything the base model loads lives in that one repository, roughly 307 GB in
total. Weights are not bundled with the code; download them into ckpt/ in the
repository root, which is gitignored:

pip install huggingface_hub
hf download sand-ai/MAGI-2-preview --local-dir ckpt

The directory names in that repository are the ones the configs already expect,
so nothing needs renaming afterwards:

ckpt/
├── preview/                            # preview stage: 56 safetensors shards + index
├── refiner/                            # refiner stage: 3 shards + index
├── text_encoder/                       # text encoder
├── vae/                                # video VAE
│   └── Wan2.2_VAE.pth
├── stable-audio-open-1.0/              # audio VAE
└── turbo_vae/                          # fast VAE decoder
    ├── TurboV3-Wan22-TinyShallow_7_7.json
    └── checkpoint.ckpt

Each directory name below links to the folder it comes from in the Hugging Face
repository:

The configs under configs/ reference these as
${MAGI2_CKPT_ROOT}/<name>, and that variable defaults to <repo>/ckpt. To keep
weights somewhere else, point it at them rather than editing the configs:

export MAGI2_CKPT_ROOT=/data/magi2-weights

Prompt enhancement

The captions the model was trained on are long and structured, so a short hand-
written prompt underuses it. The pipeline can optionally rewrite the input
through a prompt-enhancement (PE) step before encoding:

• text-to-video uses inference/prompt_enhancement/prompts/t2v.md
• image-to-video (when a first-frame image is provided) uses
  inference/prompt_enhancement/prompts/i2v.md

Both templates ask an instruction-following LLM for a structured JSON caption of
the 10-second clip; the result is then rendered to readable Markdown and passed
downstream. The reference client in
inference/prompt_enhancement/ talks to an
OpenAI-compatible endpoint - set API_KEY in
enhancer.py to enable it, or
subclass LLMClient for your own provider. Leave API_KEY empty to skip PE and
use the raw prompt as-is.

assets/ ships example prompts that are already PE-enhanced
(sample_000.txt and sample_002.txt). You can still generate without running
PE - leave API_KEY empty and pass any prompt you like.

Running inference

scripts/run_demo.sh launches
inference/pipeline/entry.py under torchrun on
every visible GPU. It generates at 1080p, with seed 42 and the batch in
assets/demo_samples.json:

bash scripts/run_demo.sh
SAMPLES=my_samples.json bash scripts/run_demo.sh   # a different batch
OUTPUT_DIR=output/run7 bash scripts/run_demo.sh

The script takes SAMPLES, OUTPUT_DIR, SEED and MASTER_PORT from the
environment. Videos land in $OUTPUT_DIR/sample_000.mp4 and up, numbered by
position in the batch.

A samples file is a JSON array with one entry per video. An entry carries its
prompt inline as prompt or as a path in prompt_file, and a first frame in
image; leaving image out makes it a T2V entry. The shipped batch runs the
three stills in assets/ as I2V, the same three prompts again as T2V, and
assets/sample_enhanced_t2v.json.

For a single clip, call the entry point directly:

torchrun --nproc_per_node=8 inference/pipeline/entry.py \
    --prompt "a red fox in snow" --output output/

It also takes --prompt-file, --image, --seed, --config, --output-width
/ --output-height, --num-inference-steps, --refiner-num-inference-steps
and --deterministic. Of these only --seed, --samples and --output are
reachable through run_demo.sh.

1080p runs configs/magi2_refiner.json: the
preview stage generates 512x896 and the refiner takes that to 1088x1920.
magi2_refiner.json extends
magi2_preview.json and carries only what the
refiner stage adds, so a shared setting is edited in one place.

1080p is a delivery tier, not the shape that gets generated. The VAE stride
constrains every generated dimension to a multiple of 16, so the tier generates
1088 wide rather than 1080, and the video is written at that generated shape.
Pass --output-width and --output-height to have the finished video rescaled
to an exact size, 1080x1920 the way the reference delivers the tier.

Four environment variables decide where each large component sits between
phases: MAGI2_TEXT_ENC_OFFLOAD_MODE, MAGI2_PREVIEW_OFFLOAD_MODE,
MAGI2_REFINER_OFFLOAD_MODE and MAGI2_VAE_OFFLOAD_MODE, each one of cpu,
gpu or roundtrip. The preview and the refiner default to roundtrip, staged
in and out around the stage that needs them, because at 1080p neither fits on an
80GB card next to the other's activations.

Decoding uses the distilled turbo decoder from ckpt/turbo_vae, a temporal
sliding window that runs on one rank per video. MAGI2_DETERMINISTIC=1, or
--deterministic, makes the MoE scatter and the attention kernels bit-exact at
some cost in speed. MAGI2_SAVE_LATENT_PATH writes the post-refiner latent of
each sample to that directory.