Deforum is an animated film creation program that uses Stable Diffusion. All you need to do is give the Deforum Stable Diffusion prompts and camera movement settings.

We’ll walk through the process of creating this deforum video.

This article is good for those who have never made a deforum video before. You will find in this article:

How to install the deforum addon on AUTOMATIC1111 Stable Diffusion.

The fundamental settings.

Step-by-step instructions of making your first deforum video.

1. How to Install Deforum Stable Diffusion?

You will use the deforum extension of AUTOMATIC1111 GUI. This GUI is compatible with Windows, Mac, and Google Colab.

1.1. Use Deforum on Google Colab

Check the Deforum extension before starting AUTOMATIC1111 if you are using the Colab notebook from the Quick Start Guide.

After finishing the setup step, you should see the Deforum tab in the AUTOMATIC1111 GUI.

1.2. Installing Deforum on Windows

If you run AUTOMATIC1111 locally on Windows, follow these steps to install deforum:

Step 1: Open the command prompt (cmd) and navigate to stable-diffusion-webui. If you follow the installation instructions on the website, run

cd%userprofile%\stable-diffusion-webui

Step 2: Run the following command to install the deforum extension.

git clone https://github.com/deforum-art/deforum-for-automatic1111-webui extensions/deforum

After restarting the AUTOMATIC1111 GUI, you should see the Deforum tab.

1.3. Installing Deforum on Mac

If you run AUTOMATIC1111 locally on your Mac, follow these steps to install Deforum Stable Diffusion online.

Step 1: Launch the Terminal application. Navigate to the stable-diffusion-webui installation directory. Run the following if you followed the site’s instructions.

cd /webui-stable-diffusion

Step 2: Run the following command to install the deforum extension.

git clone https://github.com/deforum-art/deforum-for-automatic1111-webui extensions/deforum

After restarting the AUTOMATIC1111 GUI, you should see the Deforum tab.

2. How to Generate a Test Video?

This step is optional, but it will show you where to find the settings you will use.

Step 1: Navigate to the Deforum page in the AUTOMATIC1111 GUI.

Step 2: Select the Keyframes tab.

On the bottom part of the page, you’ll see a Motion tab. You will configure the camera settings here.

The maximum frame rate is the number of frames in your video. A higher number lengthens the video.

You can leave the default values as they are.

Step 3: Select the Prompts tab. You will see a list of prompts, each with a number in front of them. The number means the time frame that the prompt becomes effective.

It will use the first prompt at the beginning of the video for the prompts following. The second prompt will be used at the 30th frame, the third prompt at the 60th frame, and the fourth prompt at the 90th frame.

You can leave the prompts as they are.

Step 4: Click Generate to begin creating a video.

Step 5: Once completed, click the button above the Generate button to view the video.

By clicking the three vertical dots in the bottom right corner, you may save the movie to your local storage. Alternatively, you may find your movie in the img2img-images folder in the output directory.

3. Deforum Stable Diffusion Basic Settings (with examples)

We’ll start with the two most crucial settings.

• Motions (2D and 3D)
• Prompts

Before going through the step-by-step examples of making videos, it’s vital to have a fundamental idea of what Deforum Stable Diffusion free can do.

3.1. Motion Settings

Motion settings are among the most frequently used parameters in Deforum. Modifying them and the instructions will result in a nice film. As a result, you will thoroughly understand how motion settings function and what they can achieve.

Let’s look at the two most common animation modes.

• 2D – consider the pictures as 2D and apply modifications such as magnification and rotation to create the illusion of motion.
• 3D – consider the photos to be a glimpse of a 3D scene. In any 3D process, you can move the camera’s viewpoint.

3.2. 2D Motion Settings

2D Zoom

Zoom in or out of the image using the zoom feature. To zoom in, use a zoom value more than one and less than one.

The faster the zoom, the farther the value is from 1.

The zoom is set to center by default. Transform Center X and Transform Center Y can be used to control the focus.

2D Angle

To rotate the photos, use 2D Angle. A positive value makes the picture rotate counterclockwise, whereas a negative value causes the image to rotate clockwise.

A higher number makes the picture spin quicker.

By default, the picture is rotated around its center. Control the center of rotation by using Transform Center X and Transform Center Y. 

2D Translation X

To shift the picture sideways, use Translation X. A positive value moves the picture to the right, whereas a negative value moves it to the left.

2D Translation Y

To shift the camera up and down, use Translation Y. A positive value will move the image down, while a negative value will move the image up.

2D Transform Center

You Transform Center to change the zoom focus point and/or rotation.

The default value for both X and Y is 0.5, and it’s the image’s center. The top left corner is (X, Y) = (0, 0), whereas the bottom right corner is (1, 1). Look at the diagram for other common locations.

You can enter numbers less than 0 or more than 1. They’ll be outside the frame.

2D Perspective flip

Perspective flip applies 3D-like modifications to the image to create some interesting effects.

To make these choices available, choose to Enable perspective flip.

3.3. 3D motion settings

3D motion differs from 2D motion. Pretend that you are carrying a camera. You can move and rotate that camera in whatever direction you want.

3D Translation X

The camera is moved sideways using Translation X. A positive number makes the camera shift to the right. A negative number makes the camera shift to the left.

Also Read: Stable Diffusion Lora: What are they and how to use them?

3D Translation Y

Translation Y makes the camera move up and down. A positive value raises the camera. A negative value lowers the camera.

3D Translation Z

Translation Z in 3D is the same as zooming in 2D.

3D rotation X

Rotation X turns the camera around on the X-axis.

3D rotation Y

Rotation Y makes the camera rotate around the Y-axis.

3D rotation Z

Rotation Z makes the camera rotate around the Z-axis.

Motion schedule

The motion parameters are put alongside the form.

frame1: (value1), frame2: (value2), frame3: (value3), and so on.

Each element contains two numbers: the frame number that the motion takes effect and the motion’s value. Each entry’s frame and value must be separated by a colon, and the value must be bracketed.

• You must always have an entry for frame 0.
• You can input it multiple times if you want.

It’s vital to notice that having two or more entries signifies interpolation between the two frames.

For example, in zoom, the following formula indicates gradually raising the zoom value from 1 to 1.02 over the first 100 frames and then dropping it back to 1 over the next 100 frames.

0:(1), 100:(1.02), 200:(1)

If you want a new zoom setting to be applied beginning with the 100th frame, type something like:

0:(1), 99:(1), 100:(1.02), 150:(1.02), 151:(1), 200:(1)

The zoom effect stays only between frames 100 and 150 using this formula.

Each setting has its unique motion schedule.

• Zoom: 0:(1)
• Angle: 0:(0)
• Center X Transform: 0:(0.5)
• Center Y Transform: 0:(1)
• Translation X: 0:(0)
• Translation Y: 0:(5), 60:(0).
• You cannot just write 180:(5). Write 0:(0), 180:(5). The 0th frame must be the initial entry.

Step 1: Generate an Initial Image

In a deforum video, one of the few aspects you have complete control is the first image. It’s also likely the most crucial since it sets the tone and color scheme for the remainder of the animation.

Take your time in the txt2img tab to create a solid starting image.

• I used the following prompt in this example.

Casino, key art, sprinting, palm palms, highly detailed, digital painting, artstation, concept art, cinematic lighting, crisp focus, artwork by gaston bussiere alphonse mucha

• And then there’s this negative prompt.

distorted, disfigured

• Set the seed to a random number (-1).
• I created a photorealistic illustration style using the Protogen v2.2 model.

Once you find an image you like, make a note of the seed value.

Step 2: Generate the First Segment of the Video

• In the Prompts tab, input the prompt. In the default prompts, I reused the second prompt. The prompts are as follows:

{“0”: “portrait of henry cavill as tom cruise, casino, key art, sprinting, palm trees, highly detailed, digital painting, artstation, concept art, cinematic lighting, sharp focus, illustration, by gaston bussiere alphonse mucha –neg deformed, disfigured”,

    “50”: “anthropomorphic clean dog, surrounded by fractals, epic angle and pose, symmetrical, 3d, depth of field, ruan jia and fenghua zhong”}

• In the 50th frame, the prompt changes to describing a dog.
• Go to the Run tab now.
• Select the Protogen model.
• Set the seed 2020548858. Fixing the seed allows you to start with the same image each time, so you can continue working on the same movie.

Because my Tom Cruise is facing left in the first photograph, having the camera go right is a wonderful touch. We will use the 3D animation model.

• Select the 3D Animation mode from the Keyframes tab.
• Set the maximum number of frames to 100. This helps make enough frames to view the first two prompts.
• Set the following in the Motion tab down below:
• Translation X to 0:(2). This is used to move the camera to the right.
• Translation Z to 0:(1.75). This is used to zoom in at a slower rate.
• Keep the rest 0:(0) to do nothing.
• To begin creating the video, click Generate.

Step 3: Add the Next Prompt

Now, under the txt2img tab, brainstorm the following prompt:

It would be a good idea to switch to a space scene. This concludes the deforum prompt.

{“0”: “portrait of henry cavill as tom cruise, casino, key art, sprinting, palm trees, highly detailed, digital painting, artstation, concept art, cinematic lighting, sharp focus, illustration, by gaston bussiere alphonse mucha –neg deformed, disfigured”,

    “50”: “anthropomorphic clean dog, surrounded by fractals, epic angle and pose, symmetrical, 3d, depth of field, ruan jia and fenghua zhong”,

    “90”: “giant floating space station, futuristic, star war style, highly detailed, beautiful machine aesthetic, in space, galaxies, dark deep space <lora:epiNoiseoffset_v2:1> –neg bad art, amateur”}

(I used the epi_noiseoffset LoRA model modifier in the third prompt. See the LoRA tutorial for details.)

• Setting the following parameters
• Max frames to 250.
• Rotation 3D X to 0:(0), 70:(0), 71:(0.5). This adds a rotation adjustment at frame 71.
• The rest of the settings remain unchanged. The final motion parameters are listed below.
• Press the Generate button.

We now have the finished video.

Typically, you will spend much time tinkering with the motion and prompts to create the final best result.

You can repeat this process as many times as you want.

4. Tips for the Best Deforum Stable Diffusion video

• Prompts with a large subject work more efficiently than scenes with many small objects.
• Small details will change regularly. This is how image-to-image communication works. As a result, patterns (such as fractals) or inventive themes tend to function better as second and later suggestions.
• If you encounter artifacts during a prompt transition, try adjusting the frame of the prompt by a few frames.
• Using the Output option Make GIF, create an animated gif.
• Use the erase Imgs option in the Output settings to automatically erase the intermediate pictures and preserve only the video.
• Use the output option Add soundtrack to add a soundtrack.

5. Conclusion

Above is our best practice of using Deforum Stable Diffusion. We hope this guide will help you get to know and get started with this top-notch image generator. Don’t hesitate to break the barrier to immerse you in the new world of digital technology.