AttributeError: 'RoboPaligemma' object has no attribute 'image_processor'

[stephonjou]

bash scripts/run.sh configs/calvin_finetune/finetune_paligemma_cont-lstm-post_full-ft_text_vision_wd=0_ws-8_act-10.json

get the following error:
[rank0]: Traceback (most recent call last):
[rank0]: File "main.py", line 350, in
[rank0]: experiment(variant=configs)
[rank0]: File "main.py", line 153, in experiment
[rank0]: image_preprocess = model.model.image_processor
[rank0]: File "/home/zxlong10/.conda/envs/robovlms/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1709, in getattr
[rank0]: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
[rank0]: AttributeError: 'RoboPaligemma' object has no attribute 'image_processor'

[cheng-haha]

same issue. can the author fix this bug?

[lixinghang12]

Sorry for the typo, please modify line 8 of robovlms/model/backbone/robopaligemma.py

RoboVLMs/robovlms/model/backbone/robopaligemma.py

Line 8 in 8ccb05d

return self.model.processor

from return self.model.processor to return self.processor. We will update the repo as well.

[Ericonaldo]

Should be fixed now.

[cheng-haha]

It's trainable. But repeated warnings abound:
"You are passing both text and images to PaliGemmaProcessor. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add <image> tokens in the very beginning of your text and <bos> token after that. For this call, we will infer how many images each text has and add special tokens.
You are passing both text and images to PaliGemmaProcessor. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add <image> tokens in the very beginning of your text and <bos> token after that. For this call, we will infer how many images each text has and add special tokens.
You are passing both text and images to PaliGemmaProcessor. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add <image> tokens in the very beginning of your text and <bos> token after that. For this call, we will infer how many images each text has and add special tokens.
You are passing both text and images to PaliGemmaProcessor. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add <image> tokens in the very beginning of your text and <bos> token after that. For this call, we will infer how many images each text has and add special tokens.
You are passing both text and images to PaliGemmaProcessor. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add <image> tokens in the very beginning of your text and <bos> token after that. For this call, we will infer how many images each text has and add special tokens.
"

[lixinghang12]

@cheng-haha Since we utilize PaliGemmaProcessor as an image / text processor and concat the two modalities in the backbone, so that the processing precedure is different from the original Processor which utilizes <image> as a placeholder and replace it with the image token sequences. You can ignore the warning.