batch data , token and map

[Wtandford]

hello ! How to save batch data and how to ensure the correspondence between token and map

[wooyeolBaek]

@Wtandford
Sorry for the late reply. Even for a single image, too many intermediate results are saved at each layer and timestep, so I haven’t yet implemented attention map extraction and storage for batch data. Therefore, I recommend using this approach only for inference on a single image for now. Regarding token and map matching, the prompt is processed through the tokenizer and text encoder, resulting in a text embedding with 77 tokens. To keep things simple, I didn’t fetch tokens directly from the pipeline. Instead, I re-tokenized the prompt to match the tokens.

[shashankg7]

Hi @wooyeolBaek! Any suggestions to extend this approach to get attention map for batched data?

[wooyeolBaek]

@shashankg7
I was planning to extend it to batched data using AttnProcess2_0, but I've been busy lately, so it might take a while before I can get to it. Since AttnProcess2_0 can process batched data simultaneously, unlike AttnProcess, it should be sufficient to modify register_cross_attention_hook in utils.py and lines 767-772 in modules.py.

[wooyeolBaek]

@shashankg7
I've also added feature to support batch operations. So feel free to refer to it if needed.