animatediff

animatediff

AnimateDiff,Stable Diffusion AI video generation plugin,converts static images into dynamic animations

Visit Site →

Animatediff Official Website, Stable Diffusion AI Video Generation Plugin, Convert Static Images to Animated Videos

Animatediff Introduction

AnimateDiff can convert static images into animated videos. This method is suitable for various types of images and requires no specific adjustments. Through a trained motion modeling module, it can directly convert personalized T2I models into animations. This method can also be extended to the fields of 3D cartoons and 2D animations. It can serve as a simple and effective baseline for personalized animation generation.To use AnimateDiff, you need to install the SD plugin and AnimateDiff model.

GitHub Open Source Project Address:https://github.com/guoyww/animatediff

WebUI Plugin Address:https://github.com/continue-revolution/sd-webui-animatediff

arXivPaper Address:https://arxiv.org/abs/2307.04725

AnimateDiff uses a control module to influence the Stable Diffusion model. It is trained through various short video clips. The control module can adjust the image generation process to produce images that look like the video clips it has learned. Simply put, the training of the SD model is done through a large number of images. AnimateDiff, on the other hand, trains the connection between images through a large number of short videos (video frames' fluidity), combining with the SD model, it can fine-tune each generated image through the AnimateDiff model, and finally compile them into a short video.

AnimateDiff: Enhanced Animation Generation Experience

– Advances in text-to-image models and personalized technology enable everyone to manifest their imagination into high-quality images at a lower cost. AnimateDiff inserts an action modeling module into the text-to-image model to avoid damaging the original text-to-image fine-tuning model's generation ability and to learn reasonable motion priors from video data. During training, the model structure is expanded to accept video format data, and the original text-to-image model weights are kept unchanged during parameter updates, allowing the network weights after training to be plug-and-play with different versions of customized fine-tuning models without any optimization for the corresponding model. During inference, the action modeling module is directly inserted into the corresponding customized model, and the combined model denoises the sampled noise sequence to generate the corresponding animation segment.

In AnimateDiff, the authors propose a framework for extending any customized text-to-image model for animation generation, which can generate corresponding animation segments while maintaining the original quality of the customized model. It adds dynamic luster to the colorful pebbles.

AnimateDiff is a plugin for quickly generating non-flickering animations in Stable Diffusion. Developed by the Chinese University of Hong Kong team, this open-source AI video generation method is based on the open infrastructure of Stable Diffusion. After the model was open-sourced in August, the quality of AI video generation was improved by several levels in just one month. For example, the following video was generated from 0 using Animatediff, requiring only a prompt word, allowing users to describe the scene at a specific second in the video!

Plugin Installation Tutorial

There are three methods for installing the Stable Diffusion plugin:

1. Start “Stable Diffusion”, click “Extensions”—“Downloadable”—then click “Load Extension List”, search for the keyword “AnimateDiff”, click “Install”, and restart “Stable Diffusion” after installation.

Install from the website by copying the GitHub URL, pasting the plugin URL into “Extension's Git Repository URL”, and clicking “Install”. Restart “Stable Diffusion” after installation.

Plugin GitHub URL: https://github.com/continue-revolution/sd-webui-animatediff.git

3. Install the plugin package. Download the installation package of the plugin from GitHub and install it to the “extensions” folder in the “Stable Diffusion” directory. (Installation package provided at the end.)

 

Model Download

After installing the plugin, you also need to downloadone animation modelandeight control shot LoRA models(the control shot models can be downloaded, but the animation model must be downloaded), the animation model download the latest “mm_sd_v15_v2.ckpt” is enough, as it is the latest and the most stable with the best effect.

After downloading the model, you need to place the model in the corresponding folder. The “mm_sd_v15_v2.ckpt” model should be placed in the “…\sd-webui-aki-v4.2\extensions\sd-webui-animatediff\model” folder, and the LoRA model in the “…\sd-webui-aki-v4.2\models\lora” folder.

After installing the plugin and placing the models in the corresponding folders, restart Stable Diffusion to see the “AnimateDiff” plugin.

Online Experience Animatediff

1. Open:

 https://openxlab.org.cn/apps/detail/Masbfca/AnimateDiff

2. Select Base DreamBooth Model and Motion Module sequentially (you can skip this step, use the default configuration directly, or adjust it yourself)

3. Fill in Prompt and Negative Prompt

Example: Prompt masterpiece, best quality, 1girl, solo, cherry blossoms, hanami, pink flower, white flower, spring season, wisteria, petals, flower, plum blossoms, outdoors, falling petals, white hair, black eyes

Negative Prompt worst quality, low quality, nsfw, logo

4. Click Generate, wait for about one minute, and obtain the generated animation video