HunyuanVideo with LoRA Support 🎬

⚠️ This is a work in progress! The model and features will be updated regularly. LoRA training support is coming soon!

Turn your text descriptions into videos using HunyuanVideo, now with support for custom LoRA files! LoRAs are like style plugins that help you customize how your videos look without changing the main model.

What’s This All About? ✨

This is a text-to-video AI model that lets you:

Create videos just by describing what you want to see
Use custom LoRA files to add your own style or characters
Control various aspects of how your video turns out

Think of it like having an AI video creator that you can teach new styles!

How It Works 🎥

Under the hood, this uses HunyuanVideo - a powerful AI model that turns text into videos. We’ve added support for LoRA files, which are like special add-ons that can make the videos look more like what you want. For example, you could use a LoRA trained on anime art to make your videos look more animated!

The cool part is that even if your LoRA was only trained on still images, it can still create smooth-moving videos. Pretty neat, right?

What You Can Control 🎮

When creating your video, you can adjust things like:

Your text description of what you want to see
Your custom LoRA file (must be .safetensors format)
How strongly your LoRA affects the final video
Video size (width and height)
How many frames you want
Video speed (frames per second)
Video quality settings

Current Limits ⚠️

Since this is a work in progress, there are some limitations:

Videos can’t be bigger than 1536x1024
Maximum length is 300 frames
You need to use .safetensors format for LoRA files
Bigger videos take longer to make
Might need a beefy computer for larger videos

Coming Soon! 🚀

I’m (zsxkib) working on adding LoRA training directly to Replicate! This means you’ll be able to:

Train your own LoRAs right here on Replicate
Use them immediately for video generation
Share them with others

Stay tuned for updates!

Credits and Thanks 📚

This builds on the amazing work by Tencent’s HunyuanVideo team:

@misc{kong2024hunyuanvideo,
      title={HunyuanVideo: A Systematic Framework For Large Video Generative Models}, 
      author={Weijie Kong, et al.},
      year={2024},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Special thanks to Jukka Seppänen (@Kijaidesign) for creating the fantastic ComfyUI implementation that makes this all possible. His ComfyUI nodes are the backbone of this project!

Follow me on Twitter/X @zsakib_ for updates on LoRA training and other cool features!