PanoWorld: Real-World Panoramic Generation

Quick try: demo assets in assets/demo/ — see Inference below.

Updates

• [2026-07] 🎉 Initial release of code. Training code is coming soon.

Introduction

In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.

Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA). Then, a three-stage training pipeline is introduced to progressively optimize each component.

To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.

Environment Setup

Prerequisites

• OS: Linux (tested on Ubuntu 22.04)
• GPU: CUDA-compatible GPU with ≥20GB VRAM
• CUDA: 12.8 or higher
• Python: 3.10
• FFmpeg: For video processing

Step 1: Create Conda Environment

Action Model inference uses the PanoWorld conda env (see inference_action.sh).

git clone https://github.com/Insta360-Research-Team/PanoWorld.git
cd PanoWorld

bash scripts/setup_panoworld_env.sh
conda activate PanoWorld

Or install manually:

conda create -n PanoWorld python=3.10 -y
conda activate PanoWorld
pip install -e .
export PYTHONPATH="$(pwd)${PYTHONPATH:+:${PYTHONPATH}}"

Dependencies are listed in requirements.txt. After Step 1, continue with Step 2–3 (base model + PanoWorld checkpoints) before running inference_action.sh.

Step 2: Download Base Model (Wan2.2-TI2V-5B)

PanoWorld is built upon the Wan-AI Wan2.2-TI2V-5B video diffusion model.

# Download using provided script
python scripts/download_wan2.2.py

# Or manually download from Hugging Face
# Visit: https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B
# Download to: models/Wan-AI/Wan2.2-TI2V-5B/

Step 3: Download Panoworld Models

Download the 480p or 720p checkpoints:

# Option 1: Using our download script
python scripts/download_panoworld_models.py

# Option 2: Manual download from Hugging Face
# Visit: https://huggingface.co/Insta360-Research/PanoWorld

Final model directory structure:

models/
├── Wan-AI/
│   └── Wan2.2-TI2V-5B/
├── lora/
│   ├── 480p/480p_lora.safetensors
│   └── 720p/720p_lora.safetensors
├── action/
│   ├── 480p/480p_action.safetensors
│   └── 720p/720p_action.safetensors
└── casual/
    ├── dmd_chunkwise/model.pt
    └── rolling_dmd_480p_161/model.pt

Step 4: Install Causal-Forcing Package (Optional)

The Causal-Forcing stage requires additional dependencies:

Please refer to Causal-Forcing/README.md for installation instructions.

World360 Dataset

World360 comprises 120,000 high-quality sequences, unifying 70,000 curated real-world clips with 50,000 high-fidelity simulations from AirSim360, and introduces diverse multi-altitude aerial trajectories with precise camera poses and depth information.

Output: Each dataset generates:

• 81-frame & 161-frame panoramic video.
• Camera trajectory csv file
• PNG image sequence

Inference

Bundled demo assets live under assets/demo/. Resolution-specific cases are under 480/ and 720/; each case folder contains a 2:1 equirectangular panorama (input.jpg), text prompt (prompt.txt), and camera trajectory (pose.txt). 720p cases also include a reference video (reference_gen_joint_step2000.mp4).

Camera-Controlled Video Generation

High-quality panoramic I2V with camera trajectory control.
Unified entry: inference_action.sh

Output: {output}/gen_video.mp4 (single sample) or gen_video_{i}.mp4 (batch).

# --- Demo: image + prompt + synthetic forward motion ---
./inference_action.sh \
  --resolution 480 \
  --image assets/demo/input.jpg \
  --prompt "$(cat assets/demo/prompt.txt)" \
  --motion forward \
  --output ./inference_output/demo_forward

# --- Demo case with recorded pose (480p) ---
CASE=assets/demo/480/case2_waterway_slice000
./inference_action.sh \
  --resolution 480 \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_action

# --- Demo case with recorded pose (720p) ---
CASE=assets/demo/720/case1_waterway_slice706
./inference_action.sh \
  --resolution 720 \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_action

# 480p — single sample from test CSV
./inference_action.sh \
  --resolution 480 \
  --prompt_nums 1 \
  --output ./inference_output/demo_480p

# 720p — native 1408×704, upscaled to 1440×720
./inference_action.sh \
  --resolution 720 \
  --prompt_nums 1 \
  --output ./inference_output/demo_720p

# Or via top-level wrapper
RESOLUTION=480 PROMPT_NUMS=1 ./inference_preview.sh
RESOLUTION=720 PROMPT_NUMS=1 ./inference_preview.sh

python Action-Model/infer_action.py --help

Flag	Description
--resolution	480 (960×480) or 720 (1408×704 → 1440×720)
--image / --prompt	Demo mode: panoramic image + text prompt (2:1 image)
--motion	forward / backward / left / right / up /down /, or a pose file (demo mode)
--output	Output directory
--output_filename	Default gen_video.mp4
--prompt_path	CSV with columns video, short_prompt, pose_path (batch mode)
--prompt_nums	Number of CSV rows to run (batch mode)

Default test CSVs: data_test.csv (480p), data_test_720p.csv (720p).

---

Causal Forcing Stage

Real-time panoramic generation with Causal Forcing on Wan2.2 5B.
Entry: Casual-forcing/inference_causal.sh (conda env: causal_forcing).

Output: {output}/causal_video.mp4

conda activate causal_forcing
cd Casual-forcing

# --- Quick demo (root assets, 480p case1) ---
./inference_causal.sh \
  --image ../assets/demo/input.jpg \
  --prompt "$(cat ../assets/demo/prompt.txt)" \
  --output ../assets/demo/out_short

# --- Demo case with recorded pose ---
CASE=../assets/demo/480/case2_waterway_slice000
./inference_causal.sh \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_pose

# --- 161-frame long video (Rolling Forcing) ---
CASE=../assets/demo/480/case1_park_slice002
./inference_causal.sh \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --frames 161 \
  --output ${CASE}/out_long

# --- Synthetic motion presets ---
./inference_causal.sh \
  --image ../assets/demo/input.jpg \
  --prompt "$(cat ../assets/demo/prompt.txt)" \
  --motion forward \
  --output ../assets/demo/out_forward

python infer_causal.py --help

Flag	Description
--frames	81 (short, default) or 161 (long / Rolling Forcing)
--motion	forward / backward / left / right, or a pose file (.txt / .csv / .npy)
--output	Output directory; video is always causal_video.mp4

---

Memory Preview

Coming soon. Long-range memory-augmented 720p inference (161-frame) will be released in a future update.

---

Training

PanoWorld training is organized in stages:

Stage 1  Video LoRA     →  output/lora_480p | lora_720p
Stage 2  Action Model   →  output/action_480p | action_720p  (requires Stage 1)
Stage 3  Memory         →  coming soon
Stage 4  Causal Forcing →  Casual-forcing/logs/  (480p, see below)

Use RESOLUTION=480|720 to switch presets (Matrix-3D convention). Config: configs/resolution.yaml.

Training CSVs must include panoramic video paths and text prompts. Override defaults with DATA_CSV=/path/to/your.csv.

Resolution	Native train	Output	LoRA rank
480	960×480	960×480	64
720	1408×704	1440×720 (inference upscale)	256

---

Stage 1 — Video LoRA

Fine-tune Wan2.2 I2V with LoRA for panoramic video generation (81 frames, no camera control module).

# 480p
RESOLUTION=480 bash scripts/train/01_video_lora.sh

# 720p (recommended for Action Model 720p)
RESOLUTION=720 bash scripts/train/01_video_lora.sh

Checkpoints are saved under {OUTPUT}/checkpoints/ with latest.json for resume.

Legacy wrappers: scripts/train/01_video_lora_480p.sh, 01_video_lora_720p.sh.

---

Stage 2 — Action Model

Freeze LoRA and train the cam_self_attn camera-control module (trajectory-conditioned I2V). Requires Stage 1 LoRA.

# After Stage 1 completes
RESOLUTION=480 bash scripts/train/02_action.sh
RESOLUTION=720 bash scripts/train/02_action.sh

# Warm-start from a simulation-pretrained action checkpoint (optional)
INIT_ACTION=/path/to/simulation_action.safetensors \
  RESOLUTION=720 bash scripts/train/02_action.sh

Checkpoints: {OUTPUT}/checkpoints/. Use the latest action weights with inference_action.sh for preview inference.

720p project-specific script (hardcoded paths): Action-Model/train_720p.sh.

---

Stage 3 — Memory

Coming soon.

---

Stage 4 — Causal Forcing

See Casual-forcing/README.md for installation and training.

Checkpoint Layout

output/
├── lora_480p/checkpoints/
├── lora_720p/checkpoints/
├── action_480p/checkpoints/
└── action_720p/checkpoints/
Casual-forcing/logs/          # Causal Forcing training outputs

Project Structure

PanoWorld/
├── assets/demo/                # Demo inputs (480/720 cases: input, prompt, pose)
│   ├── 480/                    # 960×480 cases
│   └── 720/                    # 1408×704 cases (+ reference_gen_*.mp4)
├── inference_action.sh         # Action Model unified entry → gen_video.mp4
├── Action-Model/               # Action Model Python scripts (480p / 720p)
│   └── infer_action.py         # unified CLI
├── Casual-forcing/             # Causal Forcing inference & training
│   ├── infer_causal.py         # unified CLI → causal_video.mp4
│   └── inference_causal.sh
├── diffsynth/                  # Shared DiffSynth modules
├── scripts/train/              # Staged training launchers (01–04)
├── models/                     # Wan2.2 base + PanoWorld checkpoints
│   ├── lora/                   # Video LoRA (480p / 720p)
│   ├── action/                 # Action Model (480p / 720p)
│   └── casual/                 # Causal Forcing inference weights
├── inference_preview.sh        # RESOLUTION=480|720 wrapper
└── configs/resolution.yaml     # 480 / 720 presets

Acknowledgments

We thank the following projects for their inspiring work, our code is partially based on the code from these projects:

• Wan-AI: Base video diffusion model

• UCPE: Camera-controlled text-to-video generation.

• Causal-Forcing: Causal-Forcing distillation for fast diffusion models

Citation

If you find Panoworld useful for your research, please cite:

@article{
}