Generate 10-second 1080p videos with audio from text or images

MAGI-2 Preview

Magi-2 Preview is a 114B-parameter unified audio-video generation model that activates just 6B parameters per token. Built on MagiMoE and co-designed across architecture, systems, and data, it explores an efficient path to scaling video generation. The architecture, the training system built around it, and the data pipeline are described in MAGI-2 Preview: Scaling Video Generation Models Efficiently; the weights are on Hugging Face at sand-ai/MAGI-2-preview.

This repository is the inference code. It generates video from a text prompt (T2V) or from a prompt plus a still image (I2V), with sound generated alongside the video and muxed into the output file. Clips are 10 seconds long, which is the only duration the model currently supports. Generation runs in two stages: magi2_preview denoises the clip at low resolution, and magi2_refiner takes that result up to 1080p.

Requirements

• NVIDIA Hopper GPUs, 8 of them.
• Python 3.12 and a recent CUDA toolkit.
• ffmpeg on PATH, to mux the audio track. Without it the video is still written, just silently.

Setup

Docker

The published image, sandai/magi-2-preview, already has the dependencies built, including the ones that need a compiler:

docker pull sandai/magi-2-preview:latest
docker run --gpus all -it -v /path/to/ckpt:/workspace/ckpt sandai/magi-2-preview:latest

There is a tag per commit as well, sandai/magi-2-preview:<commit>. Name that one when reporting a result, because latest moves; the image also records what it was built from in /etc/magi2-build-info.

Building it yourself is only necessary to change a dependency version, or to work somewhere the registry is not reachable:

docker build -t magi-2-preview:local .

From source

pip install -r requirements.txt

MAGI-2 also needs MagiAttention and MagiCompiler. The pinned revisions are recorded as build args in the Dockerfile.

Checkpoints

Neither transformer has been step-distilled, so the denoising step count is where most of the wall-clock time goes. That is the difference between the release that exists today and the one that follows:

Everything the base model loads lives in that one repository, roughly 307 GB in total. Weights are not bundled with the code; download them into ckpt/ in the repository root, which is gitignored:

pip install huggingface_hub
hf download sand-ai/MAGI-2-preview --local-dir ckpt

The directory names in that repository are the ones the configs already expect, so nothing needs renaming afterwards:

ckpt/
├── preview/                            # preview stage: 56 safetensors shards + index
├── refiner/                            # refiner stage: 3 shards + index
├── text_encoder/                       # text encoder
├── vae/                                # video VAE
│   └── Wan2.2_VAE.pth
├── stable-audio-open-1.0/              # audio VAE
└── turbo_vae/                          # fast VAE decoder
    ├── TurboV3-Wan22-TinyShallow_7_7.json
    └── checkpoint.ckpt

Each directory name below links to the folder it comes from in the Hugging Face repository:

The configs under configs/ reference these as ${MAGI2_CKPT_ROOT}/<name>, and that variable defaults to <repo>/ckpt. To keep weights somewhere else, point it at them rather than editing the configs:

export MAGI2_CKPT_ROOT=/data/magi2-weights

Prompt enhancement

The captions the model was trained on are long and structured, so a short hand- written prompt underuses it. The pipeline can optionally rewrite the input through a prompt-enhancement (PE) step before encoding:

• text-to-video uses inference/prompt_enhancement/prompts/t2v.md
• image-to-video (when a first-frame image is provided) uses inference/prompt_enhancement/prompts/i2v.md

Both templates ask an instruction-following LLM for a structured JSON caption of the 10-second clip; the result is then rendered to readable Markdown and passed downstream. The reference client in inference/prompt_enhancement/ talks to an OpenAI-compatible endpoint - set API_KEY in enhancer.py to enable it, or subclass LLMClient for your own provider. Leave API_KEY empty to skip PE and use the raw prompt as-is.

assets/ ships example prompts that are already PE-enhanced (sample_000.txt and sample_002.txt). You can still generate without running PE - leave API_KEY empty and pass any prompt you like.

Running inference

scripts/run_demo.sh launches inference/pipeline/entry.py under torchrun on every visible GPU. It generates at 1080p, with seed 42 and the batch in assets/demo_samples.json:

bash scripts/run_demo.sh
SAMPLES=my_samples.json bash scripts/run_demo.sh   # a different batch
OUTPUT_DIR=output/run7 bash scripts/run_demo.sh

The script takes SAMPLES, OUTPUT_DIR, SEED and MASTER_PORT from the environment. Videos land in $OUTPUT_DIR/sample_000.mp4 and up, numbered by position in the batch.

A samples file is a JSON array with one entry per video. An entry carries its prompt inline as prompt or as a path in prompt_file, and a first frame in image; leaving image out makes it a T2V entry. The shipped batch runs the three stills in assets/ as I2V, the same three prompts again as T2V, and assets/sample_enhanced_t2v.json.

For a single clip, call the entry point directly:

torchrun --nproc_per_node=8 inference/pipeline/entry.py \
    --prompt "a red fox in snow" --output output/

It also takes --prompt-file, --image, --seed, --config, --output-width / --output-height, --num-inference-steps, --refiner-num-inference-steps and --deterministic. Of these only --seed, --samples and --output are reachable through run_demo.sh.

1080p runs configs/magi2_refiner.json: the preview stage generates 512x896 and the refiner takes that to 1088x1920. magi2_refiner.json extends magi2_preview.json and carries only what the refiner stage adds, so a shared setting is edited in one place.

1080p is a delivery tier, not the shape that gets generated. The VAE stride constrains every generated dimension to a multiple of 16, so the tier generates 1088 wide rather than 1080, and the video is written at that generated shape. Pass --output-width and --output-height to have the finished video rescaled to an exact size, 1080x1920 the way the reference delivers the tier.

Four environment variables decide where each large component sits between phases: MAGI2_TEXT_ENC_OFFLOAD_MODE, MAGI2_PREVIEW_OFFLOAD_MODE, MAGI2_REFINER_OFFLOAD_MODE and MAGI2_VAE_OFFLOAD_MODE, each one of cpu, gpu or roundtrip. The preview and the refiner default to roundtrip, staged in and out around the stage that needs them, because at 1080p neither fits on an 80GB card next to the other's activations.

Decoding uses the distilled turbo decoder from ckpt/turbo_vae, a temporal sliding window that runs on one rank per video. MAGI2_DETERMINISTIC=1, or --deterministic, makes the MoE scatter and the attention kernels bit-exact at some cost in speed. MAGI2_SAVE_LATENT_PATH writes the post-refiner latent of each sample to that directory.

scripts/run_demo.sh

#!/usr/bin/env bash
# Generate the demo batch in assets/demo_samples.json with a single model load.
#
#   bash scripts/run_demo.sh                       # 1080p
#   SAMPLES=my_samples.json bash scripts/run_demo.sh

set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."

RESOLUTION=${RESOLUTION:-1080p}
OUTPUT_DIR=${OUTPUT_DIR:-output/demo}
# 10s is the only duration the model supports.
SECONDS_PER_VIDEO=10
SEED=${SEED:-42}

export PYTHONPATH="${PWD}:${PYTHONPATH:-}"
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export NCCL_ALGO="^NVLS"
export OMP_NUM_THREADS=64

GPUS=$(nvidia-smi -L 2>/dev/null | wc -l)
[ "$GPUS" -eq 0 ] && GPUS=8

SAMPLES=${SAMPLES:-assets/demo_samples.json}

echo "[demo] $(date +%T) resolution=$RESOLUTION, ${SECONDS_PER_VIDEO}s each, samples=$SAMPLES"

exec torchrun --nnodes=1 --nproc_per_node="$GPUS" --rdzv-backend=c10d \
    --rdzv-endpoint="localhost:${MASTER_PORT:-29500}" \
    inference/pipeline/entry.py \
    --resolution "$RESOLUTION" --seconds "$SECONDS_PER_VIDEO" --seed "$SEED" \
    --samples "$SAMPLES" --output "$OUTPUT_DIR"

assets/demo_samples.json

[
  {"prompt_file": "assets/sample_000.txt", "image": "assets/sample_000.jpeg"},
  {"prompt_file": "assets/sample_001.txt", "image": "assets/sample_001.jpeg"},
  {"prompt_file": "assets/sample_000.txt"},
  {"prompt_file": "assets/sample_001.txt"},
  {"prompt_file": "assets/sample_enhanced_t2v.json"}
]

assets/sample_enhanced_t2v.json

{
  "global_layer": {
    "context": "A single continuous take inside a small corner cafe on a rainy afternoon.",
    "description": "A barista lowers a cup onto a saucer and speaks one line to the customer while the camera pushes in.",
    "aesthetics": {
      "style": "naturalistic documentary cinematography",
      "contrast": "low to moderate with open shadow detail",
      "saturation": "restrained true-to-life saturation",
      "color_scheme": "warm amber interior against cool grey window light",
      "mood_atmosphere": "calm and unhurried"
    },
    "audio_baseline": {
      "ambience": "steady rain against glass, low espresso machine hum, occasional cup clatter",
      "dialogue": {
        "language": "English",
        "speaker_tags": ["<subject 1>"]
      }
    },
    "objects_static": [
      {
        "object_id": "<object 1>",
        "description": "a white ceramic cup resting on a matching saucer",
        "shape_and_color": "rounded white cup holding a pale crema surface",
        "texture": "smooth glazed ceramic",
        "relative_size": "small, about one third the width of the barista's hand",
        "position": "on the wooden counter in the lower right third of frame",
        "orientation": "handle turned toward the customer"
      }
    ],
    "video_metadata": {
      "duration": 10.0,
      "aspect_ratio": "16:9"
    },
    "lighting_baseline": {
      "conditions": "soft overcast daylight through a large window with warm pendant fill",
      "direction": "key from the window at camera left, warm fill from overhead pendants",
      "shadows": "soft edged with retained detail",
      "source_consistency": "window and pendant sources hold the same intensity and color for the full clip"
    },
    "environment_baseline": {
      "background_setting": "a small cafe interior with a wooden counter, shelves of stacked cups, and a rain streaked window behind"
    },
    "alive_subjects_static": [
      {
        "subject_id": "<subject 1>",
        "description": "a barista working behind the counter",
        "position": "behind the counter at frame center",
        "orientation": "facing camera, angled slightly to frame right",
        "visual_attributes": {
          "gender": "female",
          "ethnicity": "East Asian",
          "age_group": "late twenties",
          "facial_features": "round face, dark eyes, hair tied back in a low bun",
          "clothing": "dark green apron over a white shirt with rolled sleeves",
          "appearance_details": "a thin silver ring on the right hand"
        }
      }
    ],
    "camera_cinematography": {
      "overall_camera_style": "single continuous slow push-in on a stabilized head",
      "default_depth_of_field": "shallow, the barista sharp and the shelves softly separated",
      "lens_focal_length": "50mm"
    }
  },
  "dynamic_layer": {
    "timeline_segments": [
      {
        "segment_basic_info": {
          "timestamp_range": "00:00.0-00:05.0",
          "segment_description": "The barista lowers the cup onto the saucer with both hands and releases it, and the camera begins a slow push-in.",
          "active_background": "rain runs down the window behind her in continuous vertical streaks",
          "camera_delta": {
            "focus": "held on the barista's hands and face",
            "camera_position": "chest height, one meter back from the counter edge",
            "composition": "medium shot, barista at center, cup in the lower right third",
            "camera_movement": {
              "type": "push-in",
              "direction": "forward along the lens axis",
              "intensity": "slow and even"
            }
          }
        },
        "objects": [
          {
            "object_id": "<object 1>",
            "timestamp": "00:00.0-00:05.0",
            "number_of_objects": 1,
            "dynamic_state": {
              "state_change": "the cup travels from her hands down onto the saucer and comes to rest level and full",
              "motion_detail": "a short controlled descent that settles without a wobble, the crema surface swaying once and flattening"
            },
            "spatial_position": {
              "location": "on the counter in the lower right third",
              "relative_size": "small",
              "orientation": "handle turned toward the customer"
            }
          }
        ],
        "alive_subjects": [
          {
            "subject_id": "<subject 1>",
            "timestamp": "00:00.0-00:05.0",
            "number_of_subjects": 1,
            "spatial_position": {
              "location": "behind the counter at frame center",
              "relative_size": "upper body fills the central third",
              "orientation": "facing camera, shoulders square to the counter"
            },
            "action": {
              "primary_action": "setting the cup down and releasing it",
              "body_configuration": "both elbows tucked close, forearms lowering together, spine upright",
              "motion_detail": "fingers open away from the cup one at a time, hands withdraw to the counter edge and rest there",
              "interaction": "both hands hold the saucer rim, then leave it fully supported on the counter",
              "facial_expression": "eyes down on the cup, mouth relaxed"
            }
          }
        ],
        "text_render": [],
        "causal_events": [
          {
            "event_id": "event 1",
            "timestamp_range": "00:02.4-00:04.0",
            "trigger": "the barista releases the saucer",
            "effect_outcome": "the cup stays level on the counter and a thin ribbon of steam rises straight from the surface",
            "physics": "the saucer takes the full weight of the cup and the liquid surface returns to level"
          }
        ],
        "audio": {
          "dialogue_lines": [],
          "ambience_deltas": [
            {
              "timestamp_range": "00:00.0-00:05.0",
              "ambience": "rain and machine hum hold steady under a soft ceramic contact at the moment of placement"
            }
          ],
          "special_audio_events": [
            {
              "timestamp": "00:02.4",
              "sound_description": "a short soft ceramic tap as the cup meets the saucer",
              "visual_sync": "matches the frame where the cup base settles"
            }
          ]
        }
      },
      {
        "segment_basic_info": {
          "timestamp_range": "00:05.0-00:10.0",
          "segment_description": "The barista lifts her gaze to the customer, speaks one line, and holds a small closing smile as the push-in settles.",
          "active_background": "rain continues down the window at the same rate",
          "camera_delta": {
            "focus": "shifts from the cup to the barista's eyes and holds",
            "camera_position": "chest height, now half a meter from the counter edge",
            "composition": "medium close-up, barista at center, cup still visible in the lower right corner",
            "camera_movement": {
              "type": "push-in",
              "direction": "forward along the lens axis, decelerating to a stop",
              "intensity": "slowing to rest by the final second"
            }
          }
        },
        "objects": [
          {
            "object_id": "<object 1>",
            "timestamp": "00:05.0-00:10.0",
            "number_of_objects": 1,
            "dynamic_state": {
              "state_change": "the cup remains at rest on the saucer, still full",
              "motion_detail": "only the steam moves, drifting upward and thinning"
            },
            "spatial_position": {
              "location": "on the counter in the lower right corner",
              "relative_size": "small",
              "orientation": "handle turned toward the customer"
            }
          }
        ],
        "alive_subjects": [
          {
            "subject_id": "<subject 1>",
            "timestamp": "00:05.0-00:10.0",
            "number_of_subjects": 1,
            "spatial_position": {
              "location": "behind the counter at frame center",
              "relative_size": "head and shoulders fill the central half",
              "orientation": "facing camera directly"
            },
            "action": {
              "primary_action": "looking up and speaking one line, then holding still",
              "body_configuration": "chin lifts, shoulders stay level, hands rest on the counter edge",
              "motion_detail": "gaze rises to camera, lips articulate the line, then the mouth closes into a small smile that holds",
              "interaction": "addresses the customer standing at the counter off camera",
              "facial_expression": "attentive, warming into a brief smile"
            }
          }
        ],
        "text_render": [],
        "causal_events": [],
        "audio": {
          "dialogue_lines": [
            {
              "timestamp": "00:05.6-00:07.4",
              "speaker": "<subject 1>",
              "text": "Careful, it's still very hot.",
              "delivery": "warm and low, unhurried, close to the microphone"
            }
          ],
          "ambience_deltas": [
            {
              "timestamp_range": "00:07.4-00:10.0",
              "ambience": "the machine hum drops away and the rain remains alone under the final hold"
            }
          ],
          "special_audio_events": []
        }
      }
    ]
  }
}