Diffusion models have delivered strong results in image synthesis over recent years. The research community is now pursuing the more challenging task of video generation. This task strictly generalizes the image setting (an image can be viewed as a one-frame video), and it is substantially harder for two main reasons:

1. It introduces additional requirements for temporal consistency across frames, which naturally calls for more world knowledge to be encoded in the model.
2. Relative to text or images, collecting large quantities of high-quality, high-dimensional video data is more difficult, even more so for text-video pairs.

🥑 Required Pre-read: Please make sure you have read the previous blog on “What are Diffusion Models?” for image generation before continue here.

Video Generation Modeling from Scratch

We first review methods for designing and training diffusion video models from scratch, meaning approaches that do not rely on pre-trained image generators.

Parameterization & Sampling Basics

Here we adopt a slightly different variable convention than the previous post, while keeping the underlying mathematics unchanged. Let $\mathbf{x} \sim q_\text{real}$ denote a data point sampled from the real data distribution. We then add Gaussian noise in small increments over time, producing a sequence of noisy variants of $\mathbf{x}$, denoted $\{\mathbf{z}_t \mid t =1 \dots, T\}$, where the noise level increases as $t$ grows, with the final state being $q(\mathbf{z}_T) \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$. This noise-injection forward process is Gaussian. Let $\alpha_t, \sigma_t$ specify a differentiable noise schedule for this Gaussian process:

To express $q(\mathbf{z}_t \vert \mathbf{z}_s)$ for $0 \leq s < t \leq T$, we obtain:

Let the log signal-to-noise ratio be $\lambda_t = \log[\alpha^2_t / \sigma^2_t]$. The DDIM (Song et al. 2020) update can then be written as:

A special $\mathbf{v}$-prediction ($\mathbf{v} = \alpha_t \boldsymbol{\epsilon} - \sigma_t \mathbf{x}$) parameterization, introduced by Salimans & Ho (2022), has been shown to help reduce color shift in video generation when compared with $\boldsymbol{\epsilon}$-parameterization.

The $\mathbf{v}$-parameterization is derived via a trick using an angular coordinate. First, define $\phi_t = \arctan(\sigma_t / \alpha_t)$, which implies $\alpha_\phi = \cos\phi, \sigma_t = \sin\phi, \mathbf{z}_\phi = \cos\phi \mathbf{x} + \sin\phi\boldsymbol{\epsilon}$. The velocity of $\mathbf{z}_\phi$ can be expressed as:

From this, we can infer:

The DDIM update rule is modified accordingly:

Under the $\mathbf{v}$-parameterization, the model is trained to predict $\mathbf{v}_\phi = \cos\phi\boldsymbol{\epsilon} -\sin\phi\mathbf{x} = \alpha_t\boldsymbol{\epsilon} - \sigma_t\mathbf{x}$.

For video generation, the diffusion model often must perform multiple upsampling steps, either to extend video duration or to increase the frame rate. This requires the ability to sample a second video $\mathbf{x}^b$ conditioned on the first $\mathbf{x}^a$, $\mathbf{x}^b \sim p_\theta(\mathbf{x}^b \vert \mathbf{x}^a)$, where $\mathbf{x}^b$ may be an autoregressive extension of $\mathbf{x}^a$, or alternatively the missing in-between frames for a video $\mathbf{x}^a$ generated at a low frame rate.

Sampling $\mathbf{x}_b$ must condition on $\mathbf{x}_a$ in addition to its own corresponding noisy variable. Video Diffusion Models (VDM; Ho & Salimans, et al. 2022) proposed reconstruction guidance, which uses an adjusted denoising model so that sampling $\mathbf{x}^b$ can be properly conditioned on $\mathbf{x}^a$:

where $\hat{\mathbf{x}}^a_\theta (\mathbf{z}_t), \hat{\mathbf{x}}^b_\theta (\mathbf{z}_t)$ are reconstructions of $\mathbf{x}^a, \mathbf{x}^b$ produced by the denoising model. $w_r$ is a weighting factor, and using a large $w_r >1$ is found to improve sample quality. Note that the same reconstruction guidance approach can also condition on low-resolution videos, enabling samples to be extended to high resolution.

Model Architecture: 3D U-Net & DiT

As in text-to-image diffusion modeling, U-net and Transformer architectures remain two common architecture choices. A sequence of diffusion video modeling papers from Google builds on U-net variants, while the more recent Sora model from OpenAI uses a Transformer-based design.

VDM (Ho & Salimans, et al. 2022) follows a standard diffusion setup, but modifies the architecture for video. Specifically, it extends the 2D U-net to operate on 3D data (Cicek et al. 2016), where each feature map is a 4D tensor indexed by frames x height x width x channels. This 3D U-net is factorized across space and time, meaning each layer operates on either the spatial dimensions or the temporal dimension, but not both at once:

• Processing Space:
  • Each 2D convolution layer from the original 2D U-net is expanded into a space-only 3D convolution. Concretely, 3x3 convolutions become 1x3x3 convolutions.
  • Each spatial attention block continues to apply attention over space, with the first axis (frames) treated as the batch dimension.
• Processing Time:
  • A temporal attention block is inserted after each spatial attention block. It applies attention over the first axis (frames) while treating the spatial axes as the batch dimension. The relative position embedding tracks frame order. This temporal attention block is critical for capturing strong temporal coherence.

Imagen Video (Ho, et al. 2022) is built as a cascade of diffusion models to improve video quality, and it is scaled to output 1280x768 videos at 24 fps. The Imagen Video system comprises the following components, for a total of 7 diffusion models:

• A frozen T5 text encoder that produces text embeddings used as conditioning inputs.
• A base video diffusion model.
• A cascade of interleaved spatial and temporal super-resolution diffusion models, consisting of 3 TSR (Temporal Super-Resolution) and 3 SSR (Spatial Super-Resolution) modules.

The base denoising model performs spatial operations over all frames jointly using shared parameters, and then a temporal layer mixes activations across frames to better model temporal coherence. This approach is found to outperform frame-autoregressive alternatives.

Both SSR and TSR models condition on upsampled inputs that are concatenated channel-wise with noisy data $\mathbf{z}_t$. SSR upsamples by bilinear resizing, while TSR upsamples by repeating frames or inserting blank frames to be filled.

Imagen Video also applies progressive distillation to accelerate sampling, where each distillation iteration halves the required number of sampling steps. In their experiments, all 7 video diffusion models were distilled to only 8 sampling steps per model, with no noticeable reduction in perceptual quality.

To scale more effectively, Sora (Brooks et al. 2024) uses the DiT (Diffusion Transformer) architecture, which operates on spacetime patches of video and image latent codes. Visual input is represented as a sequence of spacetime patches, which serve as the Transformer input tokens.

Adapting Image Models to Generate Videos

Another major strategy for diffusion-based video modeling is to “inflate” a pre-trained text-to-image diffusion model by inserting temporal layers. With this approach, one can either fine-tune only the newly introduced layers on video data, or avoid additional training altogether. Because the model inherits prior knowledge from text-image pairs, it can reduce the amount of required text-video paired data.

Fine-tuning on Video Data

Make-A-Video (Singer et al. 2022) augments a pre-trained diffusion image model with a temporal dimension, using three main components:

1. A base text-to-image model trained on text-image pair data.
2. Spatiotemporal convolution and attention layers that extend the network to handle the temporal dimension.
3. A frame interpolation network for generating higher frame rates.

The final video inference scheme can be written as:

where:

• $\mathbf{x}$ is the input text.
• $\hat{\mathbf{x}}$ is the BPE-encoded text.
• $\text{CLIP}_\text{text}(.)$ is the CLIP text encoder, $\mathbf{x}_e = \text{CLIP}_\text{text}(\mathbf{x})$.
• $P(.)$ is the prior, generating image embedding $\mathbf{y}_e$ given text embedding $\mathbf{x}_e$ and BPE encoded text $\hat{\mathbf{x}}$ : $\mathbf{y}_e = P(\mathbf{x}_e, \hat{\mathbf{x}})$. This component is trained on text-image pair data and is not fine-tuned on video data.
• $D^t(.)$ is the spatiotemporal decoder that produces a sequence of 16 frames, where each frame is a low-resolution 64x64 RGB image $\hat{\mathbf{y}}_l$.
• $\uparrow_F(.)$ is the frame interpolation network, which increases the effective frame rate by interpolating between generated frames. This model is fine-tuned for predicting masked frames for video upsampling.
• $\text{SR}_h(.), \text{SR}^t_l(.)$ are the spatial and spatiotemporal super-resolution models, which increase resolution to 256x256 and 768x768, respectively.
• $\hat{\mathbf{y}}_t$ is the final generated video.

Spatiotemporal SR layers include pseudo-3D convolution layers and pseudo-3D attention layers:

• Pseudo-3D convo layer : Each spatial 2D convo layer (initialized from the pre-trained image model) is followed by a temporal 1D layer (initialized as the identity function). Conceptually, the 2D convo layer first produces multiple frames, after which the frames are reshaped into a video clip.
• Pseudo-3D attention layer: After each (pre-trained) spatial attention layer, a temporal attention layer is stacked to approximate a full spatiotemporal attention layer.

These layers can be represented as:

where the input tensor is $\mathbf{h} \in \mathbb{R}^{B\times C \times F \times H \times W}$ (corresponding to batch size, channels, frames, height and weight); $\circ T$ swaps temporal and spatial dimensions; $\text{flatten}(.)$ is a matrix operator that converts $\mathbf{h}$ into $\mathbf{h}’ \in \mathbb{R}^{B \times C \times F \times HW}$; and $\text{flatten}^{-1}(.)$ reverses that conversion.

During training, different Make-A-Video components are trained independently:

1. Decoder $D^t$, prior $P$, and the two super-resolution components $\text{SR}_h, \text{SR}^t_l$ are first trained on images only, without paired text.
2. Next, the new temporal layers are introduced, initialized as the identity function, and then fine-tuned on unlabeled video data.

Tune-A-Video (Wu et al. 2023) inflates a pre-trained image diffusion model to enable one-shot video tuning. Given a video containing $m$ frames, $\mathcal{V} = \{v_i \mid i = 1, \dots, m\}$, paired with a descriptive prompt $\tau$, the goal is to generate a new video $\mathcal{V}^*$ from a slightly edited and related prompt $\tau^*$. For instance, $\tau$ = "A man is skiing" can be extended to $\tau^*$="Spiderman is skiing on the beach". Tune-A-Video is intended for object editing, background replacement, and style transfer.

In addition to inflating the 2D convo layers, the Tune-A-Video U-Net integrates an ST-Attention (spatiotemporal attention) block to encourage temporal consistency by querying relevant positions in previous frames. Given latent features for the current frame $v_i$, previous frames $v_{i-1}$, and the first frame $v_1$, these are projected into query $\mathbf{Q}$, key $\mathbf{K}$, and value $\mathbf{V}$. ST-attention is defined as:

The Gen-1 model (Esser et al. 2023) by Runway focuses on editing an input video according to text instructions. It decomposes conditioning for generation by separating structure and content of a video $p(\mathbf{x} \mid s, c)$, although performing a clean separation between these aspects is challenging.

• Content $c$ refers to the video’s appearance and semantics. For conditional editing, it is sampled from text. A CLIP embedding of a frame serves as a strong content representation and remains largely orthogonal to structural properties.
• Structure $s$ captures geometry and dynamics, including object shapes, locations, and temporal changes, and $s$ is sampled from the input video. Depth estimation or other task-specific side information (for example, human body pose or face landmarks for human video synthesis) can also be used.

The architectural modifications in Gen-1 are relatively standard: a 1D temporal convo layer is added after each 2D spatial convo layer in residual blocks, and a 1D temporal attention block is added after each 2D spatial attention block in attention blocks. During training, the structure variable $s$ is concatenated with the diffusion latent variable $\mathbf{z}$, while the content variable $c$ is provided through the cross-attention layer. At inference time, the CLIP embedding is transformed via a prior that converts CLIP text embeddings into CLIP image embeddings.

Video LDM (Blattmann et al. 2023) first trains a LDM (Latent diffusion models) image generator. It then fine-tunes the model to generate videos by adding a temporal dimension. Fine-tuning is applied only to the newly inserted temporal layers on encoded image sequences. In Video LDM (See Fig. 10), the temporal layers $\{l^i_\phi \mid i = \ 1, \dots, L\}$ are interleaved with the existing spatial layers $l^i_\theta$, which remain frozen during fine-tuning. In other words, only the newly introduced parameters $\phi$ are updated, while the pre-trained image backbone parameters $\theta$ are kept fixed. The Video LDM pipeline first generates key frames at low fps, then applies two stages of latent frame interpolation to increase fps.

An input sequence of length $T$ is treated as a batch of images (that is, $B \cdot T$) for the base image model $\theta$, and is then reshaped into video form for the $l^i_\phi$ temporal layers. A skip connection combines the temporal-layer output $\mathbf{z}’$ and the spatial output $\mathbf{z}$ using a learned merging parameter $\alpha$. In practice, two types of temporal mixing layers are used: (1) temporal attention and (2) residual blocks built on 3D convolutions.

A remaining limitation is that the LDM pretrainined autoencoder is exposed only to images, not videos. Using it directly for video generation can therefore introduce flickering artifacts due to weak temporal coherence. To address this, Video LDM adds extra temporal layers in the decoder and fine-tunes them on video data using a patch-wise temporal discriminator built from 3D convolutions, while keeping the encoder unchanged so the pretrained LDM remains reusable. During this temporal decoder fine-tuning stage, the frozen encoder processes each frame independently, and temporally consistent reconstructions across frames are enforced using a video-aware discriminator.

Like Video LDM, Stable Video Diffusion (SVD; Blattmann et al. 2023) also builds on an LDM backbone with temporal layers inserted after every spatial convolution and attention layer, but SVD fine-tunes the full model. Training video LDMs proceeds in three stages:

1. Text-to-image pretraining is essential and improves both quality and prompt adherence.
2. Video pretraining is beneficial as a separate stage and should ideally be conducted at larger scale on a curated dataset.
3. High-quality video finetuning uses a smaller, pre-captioned video dataset with high visual fidelity.

SVD places particular emphasis on the central importance of dataset curation to model performance. The authors run a cut-detection pipeline to increase the number of cuts per video, and then apply three different captioning models: (1) CoCa for the mid-frame, (2) V-BLIP for a video-level caption, and (3) an LLM-based captioning stage that leverages the first two captions. They then iteratively refine the video datasets by removing clips with limited motion (filtered via low optical-flow scores computed at 2 fps), excessive text content (using optical character recognition to identify videos containing large amounts of text), or generally low aesthetic quality (annotating the first, middle, and last frames of each clip with CLIP embeddings, then computing aesthetic scores and text-image similarity scores). The experiments demonstrate that applying filtering to obtain a higher-quality dataset improves model quality, even when the resulting dataset is substantially smaller.

A key challenge in approaches that first generate distant key frames and then apply interpolation with temporal super-resolution is preserving strong temporal consistency at high quality. Lumiere (Bar-Tal et al. 2024) instead uses a space-time U-Net (STUNet) architecture that generates the full temporal extent of a video in a single pass, eliminating reliance on TSR (temporal super-resolution) components. STUNet downsamples video along both the temporal and spatial dimensions, so the expensive computation is performed in a compact spatiotemporal latent space.

STUNet inflates a pretrained text-to-image U-Net so that it can downsample and upsample videos in both the temporal and spatial dimensions. Convolution-based blocks are composed of pretrained text-to-image layers, followed by a factorized space-time convolution. Attention-based blocks at the coarsest U-Net level include the pretrained text-to-image layers, followed by temporal attention. Additional training is performed only on the newly introduced layers.

User study results are reported using the 2AFC protocol (two-alternative forced choice), in which users select the better option from a random pair of images generated by two different models.

![[Pasted image 20240403102834.png|400]]

Training-Free Adaptation

Somewhat surprisingly, a pretrained text-to-image model can be adapted to produce videos without any training 🤯.

If we naively sample a sequence of latent codes at random and then decode them into a video as a sequence of images, there is no guarantee of temporal consistency in objects and semantics. Text2Video-Zero (Khachatryan et al. 2023) supports zero-shot, training-free video generation by augmenting a pretrained image diffusion model with two mechanisms designed to improve temporal consistency:

1. Sample the latent-code sequence with motion dynamics to maintain temporal consistency of the global scene and background.
2. Reprogram frame-level self-attention by introducing a cross-frame attention mechanism, where each frame attends to the first frame, thereby preserving the context, appearance, and identity of the foreground object.

The procedure for sampling a sequence of latent variables, $\mathbf{x}^1_T, \dots, \mathbf{x}^m_T$, using motion information is described below:

1. Define a direction $\boldsymbol{\delta} = (\delta_x, \delta_y) \in \mathbb{R}^2$ to control global scene and camera motion; by default, we set $\boldsymbol{\delta} = (1, 1)$. Also define a hyperparameter $\lambda > 0$ that controls the magnitude of global motion.
2. Randomly sample the latent code for the first frame, $\mathbf{x}^1_T \sim \mathcal{N}(0, I)$.
3. Run $\Delta t \geq 0$ DDIM backward update steps using the pretrained image diffusion model (for example, the Stable Diffusion (SD) model in the paper), and obtain the corresponding latent code $\mathbf{x}^1_{T’}$ where $T’ = T - \Delta t$.
4. For each frame in the latent-code sequence, apply the corresponding motion translation via a warping operation defined by $\boldsymbol{\delta}^k = \lambda(k-1)\boldsymbol{\delta}$ to obtain $\tilde{\mathbf{x}}^k_{T’}$.
5. Finally, apply DDIM forward steps to all $\tilde{\mathbf{x}}^{2:m}_{T’}$ to obtain $\mathbf{x}^{2:m}_T$.

In addition, Text2Video-Zero replaces the self-attention layer in a pretrained SD model with a new cross-frame attention mechanism that references the first frame. The intent is to preserve the foreground object’s appearance, shape, and identity throughout the generated video.

Optionally, a background mask can be applied to further smooth and improve background consistency. Suppose we obtain a foreground mask $\mathbf{M}_k$ for the $k$-th frame using an existing method. Background smoothing then merges the actual latent code and the warped latent code at diffusion step $t$, with respect to the background matrix:

where $\mathbf{x}^k_t$ is the actual latent code and $\tilde{\mathbf{x}}^k_t$ is the warped latent code on the background. $\alpha$ is a hyperparameter, and the paper sets $\alpha=0.6$ in the experiments.

Text2video-zero can also be combined with ControlNet, where the pretrained ControlNet copy branch is applied per frame on each $\mathbf{x}^k_t$ for $k = 1, \dots, m$ at each diffusion time-step $t = T , \dots, 1$, and the ControlNet branch outputs are added to the skip connections of the main U-Net.

ControlVideo (Zhang et al. 2023) targets video generation conditioned on a text prompt $\tau$ and a motion sequence (for example, depth or edge maps), $\mathbf{c} = \{c^i\}_{i=0}^{N-1}$. It is adapted from ControlNet, with three additional mechanisms:

1. Cross-frame attention: Introduces fully cross-frame interactions within self-attention modules. It enables interactions among all frames by mapping latent frames at all time steps into $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ matrices. This differs from Text2Video-zero, which configures all frames to attend only to the first frame.
2. Interleaved-frame smoother: Uses frame interpolation on alternating frames to reduce flicker. At each time step $t$, the smoother interpolates either the even frames or the odd frames to smooth their associated three-frame clips. Note that the number of frames decreases over time after smoothing steps.
3. Hierarchical sampler: Uses a hierarchical sampling strategy to produce long videos with temporal consistency under memory constraints. A long video is divided into multiple short clips, and a key frame is selected for each clip. The model first pre-generates these key frames with full cross-frame attention to ensure long-term coherence, and then synthesizes each short clip sequentially conditioned on the key frames.

Citation

Cited as:

Weng, Lilian. (Apr 2024). Diffusion Models Video Generation. Lil’Log. https://lilianweng.github.io/posts/2024-04-12-diffusion-video/.

Or

@article{weng2024video,
  title   = "Diffusion Models Video Generation.",
  author  = "Weng, Lilian",
  journal = "lilianweng.github.io",
  year    = "2024",
  month   = "Apr",
  url     = "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/"
}

References

[1] Cicek et al. 2016. “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation.”

[2] Ho & Salimans, et al. “Video Diffusion Models.” 2022 | webpage

[3] Bar-Tal et al. 2024 “Lumiere: A Space-Time Diffusion Model for Video Generation.”

[4] Brooks et al. “Video generation models as world simulators.” OpenAI Blog, 2024.

[5] Zhang et al. 2023 “ControlVideo: Training-free Controllable Text-to-Video Generation.”

[6] Khachatryan et al. 2023 “Text2Video-Zero: Text-to-image diffusion models are zero-shot video generators.”

[7] Ho, et al. 2022 “Imagen Video: High Definition Video Generation with Diffusion Models.”

[8] Singer et al. “Make-A-Video: Text-to-Video Generation without Text-Video Data.” 2022.

[9] Wu et al. “Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation.” ICCV 2023.

[10] Blattmann et al. 2023 “Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models.”

[11] Blattmann et al. 2023 “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets.”

[12] Esser et al. 2023 “Structure and Content-Guided Video Synthesis with Diffusion Models.”

[13] Bar-Tal et al. 2024 “Lumiere: A Space-Time Diffusion Model for Video Generation.”