M2M using image sequences

[toyxyz]

I'm trying to create an animation using multi-controlnet.
After using the ControlNet M2M script, I found it difficult to match the frames, so I modified the script slightly to allow image sequences to be input and output. I'm very new to Python, so I don't know if I modified it correctly, but it works as intended. Using this script I can img2img a sequence of images from multiple controlnets.

But I can't quite figure out how to change the input image of Img2Img, not controlnet. I want to put a sequence of images in a specific path into the input field of Img2Img one after another. Is there anyone who can give me some advice?

Animation created using a modified M2M script.

a.2_1.mp4

The ControlNet M2M code I modified is shown below.

import copy
import os
import shutil

import cv2
import gradio as gr
import modules.scripts as scripts

from modules import images
from modules.processing import process_images
from modules.shared import opts
from PIL import Image
from pathlib import Path


def get_all_frames_from_path(path):
    if not os.path.isdir(path):
        return None
    frame_list = []
    for filename in sorted(os.listdir(path)):
        if filename.endswith(".jpg") or filename.endswith(".png"):
            img_path = os.path.join(path, filename)
            img = cv2.imread(img_path)
            if img is not None:
                frame_list.append(img)
    return frame_list

def get_min_frame_num(video_list):
    min_frame_num = -1
    for video in video_list:
        if video is None:
            continue
        else:
            frame_num = len(video)
            print(frame_num)
            if min_frame_num < 0:
                min_frame_num = frame_num
            elif frame_num < min_frame_num:
                min_frame_num = frame_num
    return min_frame_num

class Script(scripts.Script):  
    
    def title(self):
        return "controlnet m2m_TEST"

    def show(self, is_img2img):
        return True

    def ui(self, is_img2img):

        ctrls_group = ()
        max_models = opts.data.get("control_net_max_models_num", 1)
        
        input_list = []

        with gr.Group():
            with gr.Accordion("ControlNet-M2M", open = True): 
                with gr.Column():
                    for i in range(max_models):
                        input_path = gr.Textbox(label=f"ControlNet-{i}", placeholder="image sequence path")
                        input_list.append(input_path)
                    tone_image_path = gr.Textbox(label=f"Color_Image", placeholder="image sequence path")
                    output_path = gr.Textbox(label=f"Output_path", placeholder="Output path")

        ctrls_group += tuple(input_list) + (tone_image_path, output_path)

        return ctrls_group


    def run(self, p, *args):

        path = p.outpath_samples
        
        output_path = args[-1]  # get the last argument, which is the output path
        
        if os.path.isdir(output_path):
            shutil.rmtree(output_path)
        os.mkdir(output_path)
        
        tmp_dir = path + "/tmp/" 
        if os.path.isdir(tmp_dir):
            shutil.rmtree(tmp_dir)
        os.mkdir(tmp_dir)
        
        
        video_num = opts.data.get("control_net_max_models_num", 1)

        image_list = [get_all_frames_from_path(image) for image in args[:video_num]]
        
        color_image_list = [get_all_frames_from_path(image) for image in args[-2]]
        

        frame_num = get_min_frame_num(image_list)
        if frame_num > 0:
            output_image_list = []
            for frame in range(frame_num):
                copy_p = copy.copy(p)
                copy_p.control_net_input_image = []
                for video in image_list:
                    if video is None:
                        continue
                    copy_p.control_net_input_image.append(video[frame])
                proc = process_images(copy_p)
                img = proc.images[0]
                images.save_image(img, output_path, f"output_{frame}")
                copy_p.close()


        else:
            proc = process_images(p)
        
        return proc

[LonicaMewinsky]

This is what I do:

copy_p.init_images = [yourPILimage] * p.batch_size

Looks awesome so far regardless!

[toyxyz]

Thank you! I'll try.

[toyxyz]

Thanks to Lonica Mewinsky's advice, I succeeded in automatically putting color image sequences into Img2Img inputs. thank you! I'll have to test it out to see if I can get the results I want.

[lyonnel4]

Hi. I'd like to use your script version (I prefer to work with sequences of images instead of videos too).
I understand some parts of your script but I don't know enough about Python or Stable Diffusion to understand the advice of Lonica Mewinsky and apply it to the code.
Could you explain what you change please ?

[toyxyz]

Here! I'm not familiar with github so I uploaded the script to gumroad. : https://toyxyz.gumroad.com/l/jydvk

[OedoSoldier]

You can check my script: https://github.com/OedoSoldier/enhanced-img2img

It can not only input image sequences with ControlNet but also mask sequences, captions from files, etc. And multi-frame rendering is also incluede.

[toyxyz]

Thank you! It looks great.

[toyxyz]

With the help of many wonderful people, I was able to add the functionality I wanted to the script!

Below is an animation created by inputting openpose motion and color image sequences separately.

Script : https://toyxyz.gumroad.com/l/jydvk

s._2.mp4

Color image / openpose motion

[supwang]

Set Directory for detected maps auto saving in Controlnet settings.

Thank you. But using this method can only generate one piece at a time.
I use OPENPOSE and currently generate images directly from videos in bulk.

Could you add mask functionality to your script?

[luthis1124]

I was looking for a way to do this, and eventually gave up and decided I'd get around to writing an extension to do it. I know it is possible because I did it by accident once and have never been able to replicate it.

[supwang]

I was looking for a way to do this, and eventually gave up and decided I'd get around to writing an extension to do it. I know it is possible because I did it by accident once and have never been able to replicate it.

Do you mean generating openpose png sequence detectmap?

[luthis1124]

Generating an openpose sequence, yes, actually if you scroll down a bit you'll see where I asked this question.

[supwang]

Generating an openpose sequence, yes, actually if you scroll down a bit you'll see where I asked this question.

I solved the problem.
You can use OpenPose v1.7.0 to generate png sequence.
https://github.com/CMU-Perceptual-Computing-Lab/openpose/releases/download/v1.7.0/openpose-1.7.0-binaries-win64-gpu-python3.7-flir-3d_recommended.zip
Command: bin\OpenPoseDemo.exe --video examples\media\test.mp4 --write_images examples\media\images --disable_blending --model_pose COCO --alpha_pose 1

I saw you asking the same question on Hugging Face, and I already replied to you. :)

[jAyATJaY]

Ben racking my brain trying to figure out why the script shows up as a duplicate of the default M2M when I install it. All of the dependencies are met. I can't figure it out.

[luthis1124]

I'm really struggling to get the script to work. It seems like the intended use is to do a first run to get the pose images and then use those as input? But when I put any folder in ControlNet-0 I get IndexError: list index out of range. I guess I'm not actually understanding how to use this step by step.

I must have read through your gumroad page 20 times trying to figure out what I'm doing wrong. Any advice?

[toyxyz]

I'm really struggling to get the script to work. It seems like the intended use is to do a first run to get the pose images and then use those as input? But when I put any folder in ControlNet-0 I get IndexError: list index out of range. I guess I'm not actually understanding how to use this step by step.

I must have read through your gumroad page 20 times trying to figure out what I'm doing wrong. Any advice?

First you need to prepare the image sequence for controlnet. The name must be numbered in order, such as a-000, a-001. When using a color image sequence, prepare the same number as the controlnet image.

And feed the first color image to the img2img input.

Now enter the path of the image sequences you have prepared.

The result is:

[luthis1124]

Great thank you! I managed to get it working. Kind of. I was making a few mistakes, most importantly not converting .bmp to .png which is why I was getting list index error. Also I was using batch, not just single img2img.
So please correct me if I'm wrong, this script takes as input:
one single initial starting image ie the first image that was used to get poses (OR is this only the colour image?)
one folder for pose images
optionally a folder for colour images

So I just have 2 more questions:
How to generate all the pose images from pngs? I can do it one at a time but there must be a way to do them as a batch, because I definitely have done it before by accident.

So this script will ONLY use a single image (the initial image) as a base to generate all the next images? Because it seemed like it's the same as batch, like it would use each frame + colour + pose + previous frame to generate. But as written here it is just colour + pose + previous frame.

[luthis1124]

Actually, reading through the script it looks like you're using the first image to find the rest of the set so, I just need to know how to generate the openpose images now.

[toyxyz]

There are two ways to prepare this image for the open pose. Using the preprocessor provided by the extension, and using a 3D model with a similar appearance, as in my case. Same goes for my script. If you entered the normal human image sequence into Controlnet 0, set the preprocessor to open pose, and if you prepared an open pose image, set it to none.

This script uses a different color image for every frame if the user inputs the path of the color image sequence.(A) Otherwise, the single image entered in img2img is used for the whole. (B)

An option to mix methods A and B is also provided. (Blend color image)

[toyxyz]

Actually, reading through the script it looks like you're using the first image to find the rest of the set so, I just need to know how to generate the openpose images now.

If you input a normal human image sequence and set the preprocessor to open pose, you get the following result.

[luthis1124]

Thank you for continuing to answer by questions!
I just need to know how to generate these:

To use here:

From your example steps above.

[toyxyz]

Those pose images were rendered using this 3D model for Blender. https://toyxyz.gumroad.com/l/ciojz