v1.1.0

cjpais/Handyv1.1.0May 7, 2024by FrankLeeeee

AI Summary

v1.1.0 focuses on infrastructure and architecture, releasing an automated data processing pipeline and improving the ST-DiT model to support diverse resolutions, aspect ratios, and advanced video editing capabilities.

Key Highlights

Automated data processing pipeline (scene cutting, filtering, captioning)
Improved ST-DiT architecture (rope, qk norm, longer text)
Support for any resolution, aspect ratio, and duration
Image and video conditioning support
Video editing capabilities (animating images, connecting videos)

New Features

Data processing pipeline v1.1
Improved ST-DiT architecture
Flexible resolution and duration support
Image and video conditioning
Video editing features

Full Release Notes

📍 Open-Sora 1.1 released
- 🌠 Model weights are available [here](https://github.com/hpcaitech/Open-Sora). It is trained on 0s~15s, 144p to 720p, various aspect ratios videos. See our [report 1.1](https://github.com/hpcaitech/docs/report_02.md) for more discussions.
- 🔧 Data processing pipeline v1.1 is released. An automatic [processing pipeline](https://github.com/hpcaitech/Open-Sora#data-processing) from raw videos to (text, video clip) pairs is provided, including scene cutting, filtering(aesthetic, optical flow, OCR, etc.), captioning managing. With this tool, you can easily build your video dataset.
- ✅ Improved ST-DiT architecture includes rope positional encoding, qk norm, longer text length, etc.
- ✅ Support training with any resolution, aspect ratio, and duration (including images).
- ✅ Support image and video conditioning and video editing, and thus support animating images, connecting videos, etc.

Visit the [Open-Sora Gallery](https://hpcaitech.github.io/Open-Sora/) to view more samples.