Welcome! This repository contains the implementation of the paper "GenCompositor: Generative Video Compositing with Diffusion Transformer"
Keywords: Diffusion Models, Video Editing, Video Compositing
Shuzhou Yang1, Xiaoyu Li2‡, Xiaodong Cun3, Guangzhi Wang2, Lingen Li4, Ying Shan2, Jian Zhang1✉
1SECE, Peking University 2ARC Lab, Tencent PCG 3GVC Lab, Great Bay University 4The Chinese University of Hong Kong ‡Project Lead ✉Corresponding Author
📖 Table of Contents
- [2026/02/06] 📢 📢 We released the training code today, HAPPY CHINESE NEW YEAR🎉🎉🎊🎊🎁🎁
- [2026/01/26] 📢 📢 GenCompositor has been accepted by ICLR 2026❗️
- [2025/09/02] 📢 📢 GenCompositor is released, the first generative video compositing framework.
- Release inference code
- Release gradio demo
- Release GenCompositor checkpoints (based on CogVideoX-5B)
- Release training code
- Dataset curation code
Video compositing combines live-action footage to create video production, serving as a crucial technique in video creation and film production.
Traditional pipelines require intensive labor efforts and expert collaboration, resulting in lengthy production cycles and high manpower costs.
To address this issue, we automate this process with generative models, called generative video compositing.
This new task strives to adaptively inject identity and motion information of foreground video to the target video in an interactive manner, allowing users to customize the size, motion trajectory, and other attributes of the dynamic elements added in final video.
Specifically, we designed a novel Diffusion Transformer (DiT) pipeline based on its intrinsic properties.
To maintain consistency of the target video before and after editing, we revised a light-weight DiT-based background preservation branch with masked token injection.
As to inherit dynamic elements from other sources, a DiT fusion block is proposed using full self-attention, along with a simple yet effective foreground augmentation for training.
Besides, for fusing background and foreground videos with different layouts based on user control, we developed a novel position embedding, named Extended Rotary Position Embedding (ERoPE).
Finally, we curated a dataset comprising 61K sets of videos for our new task, called VideoComp.
This data includes complete dynamic elements and high-quality target videos. Experiments demonstrate that our method effectively realizes generative video compositing, outperforming existing possible solutions in fidelity and consistency.
Environment Requirement 🌍
- CUDA 12.4
- PyTorch 2.5.0
- Python 3.10.16
- VRAM >= 40GB
Clone the repo:
git clone https://github.com/TencentARC/GenCompositor.git
We recommend you first use conda to create virtual environment, and install needed libraries. For example:
cd ./GenCompositor
conda create -n gencompositor python=3.10 -y
conda activate gencompositor
pip install -r requirements.txt
Then, you can install diffusers (implemented in this repo) with:
cd ./diffusers
pip install -e .
After that, you can install required ffmpeg thourgh:
conda install -c conda-forge ffmpeg -y
Checkpoints Download ⬇️
Checkpoints of GenCompositor can be downloaded from here. The ckpts folder contains
- GenCompositor pretrained checkpoints for CogVideoX-5b-I2V
- pretrinaed CogVideoX-5b-I2V checkpoint from HuggingFace.
You can download the checkpoints, and put the checkpoints to the ckpts folder by:
mkdir ckpts
cd ckpts
huggingface-cli download --resume-download TencentARC/GenCompositor --local-dir ./
You also need to download the base model CogVideoX-5B-I2V by:
git lfs install
cd ckpts
git clone https://huggingface.co/THUDM/CogVideoX-5b-I2V
Finally, you need to download checkpoints of sam2 for foreground segmentation:
cd ckpts
wget -c https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt
The checkpoints structure should be like:
|-- ckpts
|-- branch
|-- config.json
|-- diffusion_pytorch_model.safetensors
|-- CogVideoX-5b-I2V
|-- scheduler
|-- transformer
|-- vae
|-- ...
|-- model
|-- pytorch_model
|-- transformer
|-- latest
|-- random_states_0.pkl
|-- random_states_1.pkl
|-- random_states_2.pkl
|-- random_states_3.pkl
|-- random_states_4.pkl
|-- random_states_5.pkl
|-- random_states_6.pkl
|-- random_states_7.pkl
|-- scheduler.bin
|-- zero_to_fp32.py
|-- sam2.1_hiera_large.pt
Gradio Demo 🤗
You can easily utlize our model through gradio demo:
cd app
CUDA_VISIBLE_DEVICES=0 python app.py
Open your browser at http://localhost:7860
Inference 📜
Alternatively, you can also inference with the script:
cd infer
# Adjust the resolution, frame number of both background and foreground videos to the default value, and specify foreground element to be injected.
bash preprocess_bg_fg_videos.sh
# Drag trajectory (We recommend to perform this interactive step on your local host❗️)
## (1)In the pop-up window, hold down the left mouse button and drag to specify the movement trajectory.
## (2)Press the "ESC" key to exit and automatically output final binary mask video. Getting mask video (sora15_traj.mp4) based on the specified trajectory txt file (sora15_traj.txt).
python get_movemask.py --rescale 0.4 --fg_video_path "../assets/fg/element/fg_butterfly.mp4" --video_path "../assets/bg/source/sora15.mp4" --file_path "../assets/usr_mask/sora15_traj.txt" --usr_mask_path "../assets/usr_mask/sora15_traj.mp4"
# Video compositing (GenCompositor starts work❗️)
python testinput.py --fg_video_path "../assets/fg/element/fg_butterfly.mp4" --video_path "../assets/bg/source/sora15.mp4" --mask_path "../assets/usr_mask/sora15_traj.mp4" --output_path "./output.mp4" --num_inference_steps 20
Train Scripts 🧠
Curate your VideoComp dataset as:|-- VideoComp
|-- filtered_mask
|-- 40029.mp4
|-- 40031.mp4
|-- ...
|-- filtered_masked_video
|-- 40029.mp4
|-- 40031.mp4
|-- ...
|-- inpainted_sum
|-- 40029.mp4
|-- 40031.mp4
|-- ...
|-- GTs
|-- 40029.mp4
|-- 40031.mp4
|-- ...
|-- fg
|-- 40029.mp4
|-- 40031.mp4
|-- ...
|-- white_video.mp4
Modify this line of code to input your own WANDB API to visualize training loss curve:
os.environ["WANDB_API_KEY"] = "INPUT_YOUR_API_HERE"
Then, train with the script:
cd train
bash train.sh
Curation Scripts 🎞
We provide an end-to-end pipeline to build the VideoComp dataset from raw source videos, following the two-stage process (data curation and data filtering) described in Sect. B of our paper. Given a folder of source videos, the pipeline automatically produces the three video samples required for training: the source video, the foreground video (dynamic element centered in each frame), and the mask video (preserving the original motion trajectory).
The pipeline leverages advanced models as tools: a vision-language model (Qwen3.6, served via vLLM) for labeling and Grounded SAM2 for segmentation.
The Labeling stage requires a running Qwen3.6 vision-language endpoint that can take video input. Deploy it with vLLM following the official guide, e.g.
vllm serve Qwen/Qwen3.6-27B --port 9000 --limit-mm-per-prompt video=2Then point the scripts to the served endpoint via environment variables (defaults shown):
export VOE_QWEN_VL_URL="http://<HOST>:9000/v1" # OpenAI-compatible base url export VOE_QWEN_VL_MODEL="Qwen3.6-27B" # served model nameThe labeling scripts call this endpoint through an OpenAI-compatible Chat Completions API with
video_url(file://) inputs, so any vLLM-served Qwen3.6 vision-language model works.
Stage 1 · Labeling. We feed the sampled video frames to QWen to obtain a detailed description, and then ask it to identify the most prominent dynamic object in the video, returning a set of nouns (or NULL if none exists). The results are cached as sidecar files under output/_labels/.
# Identify the most prominent dynamic element of each source video.
# Cases with no significant object (QWen returns NULL) are marked as skipped.
cd gencomp_data
python gencomp_label.py --video_dir /path/to/source_videos --workers 8
Stage 2 · Segmentation. Based on the QWen label, we employ Grounded SAM2 to segment the element throughout the video. We save its mask video (which shows the original motion trajectory) and its foreground video. Note that when saving the foreground video, we center the dynamic element in each frame, eliminating its global position and trajectory information. This allows the resulting trajectory to be controlled entirely by the mask video rather than the foreground video.
# Segment the labeled element, then save the mask video and the centered foreground video.
cd gencomp_data
python gencomp_segment.py --video_dir /path/to/source_videos
Data Filtering is applied automatically during the two stages following our principles: cases where QWen returns NULL are excluded; for videos with multiple elements, only the one with the highest probability is selected; and suboptimal cases where elements have incomplete or excessively fragmented structures (e.g., touching the frame boundary or being too small) are filtered out. We recommend a final manual screening to remove visually unappealing samples.
The pipeline outputs the dataset directly into the structure expected by training:
|-- output
|-- filtered_masked_video # source videos
|-- 40029.mp4
|-- ...
|-- filtered_mask # mask videos (original trajectory)
|-- 40029.mp4
|-- ...
|-- fg # foreground videos (centered, 576x576)
|-- 40029.mp4
|-- ...
|-- _labels # QWen labeling results (sidecar cache)
|-- 40029.json
|-- ...
Both stages cache their results, so reruns will skip finished cases. Use
--forceto recompute, or--limit N/--single <name>to process a subset for a quick trial. The foreground canvas size can be adjusted via theVE_GC_FG_SIZEenvironment variable (default576).
@inproceedings{
yang2026gencompositor,
title={GenCompositor: Generative Video Compositing with Diffusion Transformer},
author={Shuzhou Yang and Xiaoyu Li and Xiaodong Cun and Guangzhi Wang and Lingen Li and Ying Shan and Jian Zhang},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=ynim5u2N4i}
}
Our code is modified based on diffusers and CogVideoX, thanks to all the contributors!
We are pleased to support the open source community. For complete license details, see LICENSE and NOTICE.
For any questions, feel free to email.