[Feature Request] S2P markers and Painting mode

[bropines]

Since the Issuses are now in discussions I will duplicate my wish here.

Quite a complex and multifaceted idea. The authors of ControlNet have an AI for painting images, Style2Paint.
It had a wonderful mechanic "Poke with color and the neural network will understand that it needs to color this part in blue." It would be nice to adapt that algorithm to Control net, and even better to create an add-on like Style2paint for control net.

The principle of operation is quite simple and is taken from S2P.

1. Upload a picture.
2. We mark it with colors
3. We give the data to the control net.
   At this point, there is one but. CN works as an image to image and cannot somehow transfer text or mathematical data to the neural network for clarification. And here we have two ways. Just train the model, recognize the colors of the markers.
   Make a separate algorithm for passing marker data to stable diffusion algorithms.
4. Get the colorized image

[bropines]

https://github.com/lllyasviel/style2paints/tree/master/V5_preview#supporting-color-scribbles

I think this is more feasible in CN 1.2

[alitan]

I believe Inpaint sketch is doing that partially no?

[bropines]

In fact, yes, but SEPA was originally developed as a powerful application for content coloring, and then the campaign was simply thrown into the trash. Maybe based on CN they will be able to make a tool of similar quality.