Starstruk Logo MarkStarstruk

v1.1.0

hpcaitech/Open-Sorav1.1.0May 7, 2024by FrankLeeeee

AI Summary

Open-Sora V1.1.0 releases model weights trained on 0s~15s videos at 144p to 720p, improved ST-DiT architecture with rope positional encoding, and a complete data processing pipeline.

Key Highlights

  • Model weights available (trained on 0s~15s, 144p to 720p, various aspect ratios)
  • Data processing pipeline v1.1 with scene cutting, filtering, captioning
  • Improved ST-DiT architecture with rope positional encoding, qk norm, longer text length
  • Support training with any resolution, aspect ratio, and duration (including images)
  • Support image and video conditioning and video editing

New Features

  • Training with any resolution, aspect ratio, and duration
  • Image and video conditioning
  • Video editing capabilities
  • Image animation
  • Video connecting

Full Release Notes

📍 Open-Sora 1.1 released
- 🌠 Model weights are available [here](https://github.com/hpcaitech/Open-Sora). It is trained on 0s~15s, 144p to 720p, various aspect ratios videos. See our [report 1.1](https://github.com/hpcaitech/docs/report_02.md) for more discussions.
- 🔧 Data processing pipeline v1.1 is released. An automatic [processing pipeline](https://github.com/hpcaitech/Open-Sora#data-processing) from raw videos to (text, video clip) pairs is provided, including scene cutting, filtering(aesthetic, optical flow, OCR, etc.), captioning managing. With this tool, you can easily build your video dataset.
- ✅ Improved ST-DiT architecture includes rope positional encoding, qk norm, longer text length, etc.
- ✅ Support training with any resolution, aspect ratio, and duration (including images).
- ✅ Support image and video conditioning and video editing, and thus support animating images, connecting videos, etc.

Visit the [Open-Sora Gallery](https://hpcaitech.github.io/Open-Sora/) to view more samples.