SAM mask-generation - crops_n_layers

[kusstox]

System Info

If I increase in a mask-generation pipeline the "crops_n_layers" to > 0. I run into issues with the batch size:

from PIL import Image
from transformers import pipeline

relative_image_path = "data\image.png"
raw_image = Image.open(relative_image_path)

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
outputs = generator(raw_image, points_per_batch=64, crops_n_layers=1)

Error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File c:\Users\LabUser\Documents\repos\test-classification\main.py:4
      [1](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:1) # %%
      [3](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:3) generator = pipeline("mask-generation", model="facebook/sam-vit-huge", device=0)
----> [4](file:///C:/Users/LabUser/Documents/repos/test-classification/main.py:4) outputs = generator([raw_image], points_per_batch=64, crops_n_layers=1)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\mask_generation.py:166, in MaskGenerationPipeline.__call__(self, image, num_workers, batch_size, *args, **kwargs)
    [128](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:128) def __call__(self, image, *args, num_workers=None, batch_size=None, **kwargs):
    [129](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:129)     """
    [130](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:130)     Generates binary segmentation masks
    [131](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:131) 
   (...)
    [164](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:164) 
    [165](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:165)     """
--> [166](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:166)     return super().__call__(image, *args, num_workers=num_workers, batch_size=batch_size, **kwargs)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\base.py:1282, in Pipeline.__call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   [1278](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1278) if can_use_iterator:
   [1279](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1279)     final_iterator = self.get_iterator(
   [1280](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1280)         inputs, num_workers, batch_size, preprocess_params, forward_params, postprocess_params
   [1281](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1281)     )
-> [1282](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1282)     outputs = list(final_iterator)
   [1283](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1283)     return outputs
   [1284](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1284) else:

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\pt_utils.py:124, in PipelineIterator.__next__(self)
    [121](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:121)     return self.loader_batch_item()
    [123](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:123) # We're out of items within a batch
--> [124](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:124) item = next(self.iterator)
    [125](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:125) processed = self.infer(item, **self.params)
    [126](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:126) # We now have a batch of "inferred things".

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\pt_utils.py:269, in PipelinePackIterator.__next__(self)
    [266](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:266)             return accumulator
    [268](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:268) while not is_last:
--> [269](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:269)     processed = self.infer(next(self.iterator), **self.params)
    [270](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:270)     if self.loader_batch_size is not None:
    [271](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/pt_utils.py:271)         if isinstance(processed, torch.Tensor):

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\base.py:1208, in Pipeline.forward(self, model_inputs, **forward_params)
   [1206](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1206)     with inference_context():
   [1207](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1207)         model_inputs = self._ensure_tensor_on_device(model_inputs, device=self.device)
-> [1208](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1208)         model_outputs = self._forward(model_inputs, **forward_params)
   [1209](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1209)         model_outputs = self._ensure_tensor_on_device(model_outputs, device=torch.device("cpu"))
   [1210](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/base.py:1210) else:

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\pipelines\mask_generation.py:233, in MaskGenerationPipeline._forward(self, model_inputs, pred_iou_thresh, stability_score_thresh, mask_threshold, stability_score_offset)
    [230](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:230) original_sizes = model_inputs.pop("original_sizes").tolist()
    [231](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:231) reshaped_input_sizes = model_inputs.pop("reshaped_input_sizes").tolist()
--> [233](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:233) model_outputs = self.model(**model_inputs)
    [235](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:235) # post processing happens here in order to avoid CPU GPU copies of ALL the masks
    [236](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/pipelines/mask_generation.py:236) low_resolution_masks = model_outputs["pred_masks"]

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\torch\nn\modules\module.py:1736, in Module._wrapped_call_impl(self, *args, **kwargs)
   [1734](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1734)     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   [1735](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1735) else:
-> [1736](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1736)     return self._call_impl(*args, **kwargs)

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\torch\nn\modules\module.py:1747, in Module._call_impl(self, *args, **kwargs)
   [1742](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1742) # If we don't have any hooks, we want to skip the rest of the logic in
   [1743](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1743) # this function, and just call forward.
   [1744](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1744) if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   [1745](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1745)         or _global_backward_pre_hooks or _global_backward_hooks
   [1746](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1746)         or _global_forward_hooks or _global_forward_pre_hooks):
-> [1747](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1747)     return forward_call(*args, **kwargs)
   [1749](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1749) result = None
   [1750](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/torch/nn/modules/module.py:1750) called_always_called_hooks = set()

File c:\Users\LabUser\Documents\repos\test-classification\env\lib\site-packages\transformers\models\sam\modeling_sam.py:1371, in SamModel.forward(self, pixel_values, input_points, input_labels, input_boxes, input_masks, image_embeddings, multimask_output, attention_similarity, target_embedding, output_attentions, output_hidden_states, return_dict, **kwargs)
   [1368](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1368)     input_labels = torch.ones_like(input_points[:, :, :, 0], dtype=torch.int, device=input_points.device)
   [1370](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1370) if input_points is not None and image_embeddings.shape[0] != input_points.shape[0]:
-> [1371](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1371)     raise ValueError(
   [1372](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1372)         "The batch size of the image embeddings and the input points must be the same. ",
   [1373](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1373)         "Got {} and {} respectively.".format(image_embeddings.shape[0], input_points.shape[0]),
   [1374](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1374)         " if you want to pass multiple points for the same image, make sure that you passed ",
   [1375](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1375)         " input_points of shape (batch_size, point_batch_size, num_points_per_image, 3) and ",
   [1376](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1376)         " input_labels of shape (batch_size, point_batch_size, num_points_per_image)",
   [1377](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1377)     )
   [1379](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1379) sparse_embeddings, dense_embeddings = self.prompt_encoder(
   [1380](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1380)     input_points=input_points,
   [1381](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1381)     input_labels=input_labels,
   [1382](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1382)     input_boxes=input_boxes,
   [1383](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1383)     input_masks=input_masks,
   [1384](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1384) )
   [1386](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1386) low_res_masks, iou_predictions, mask_decoder_attentions = self.mask_decoder(
   [1387](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1387)     image_embeddings=image_embeddings,
   [1388](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1388)     image_positional_embeddings=image_positional_embeddings,
   (...)
   [1394](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1394)     output_attentions=output_attentions,
   [1395](file:///C:/Users/LabUser/Documents/repos/test-classification/env/lib/site-packages/transformers/models/sam/modeling_sam.py:1395) )

ValueError: ('The batch size of the image embeddings and the input points must be the same. ', 'Got 5 and 1 respectively.', ' if you want to pass multiple points for the same image, make sure that you passed ', ' input_points of shape (batch_size, point_batch_size, num_points_per_image, 3) and ', ' input_labels of shape (batch_size, point_batch_size, num_points_per_image)')

@Rocketknight1 @amyeroberts @qubvel

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run:

from PIL import Image
from transformers import pipeline

relative_image_path = "data\image.png"
raw_image = Image.open(relative_image_path)

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
outputs = generator(raw_image, points_per_batch=64, crops_n_layers=1)

Expected behavior

It should support this change in parameter

[Rocketknight1]

Hi @kusstox, this seems like a bug, yes! The argument is probably correctly supported in the SAM model itself, but the pipeline code gets multiple outputs when it's used and this causes confusion regarding the true batch size.

We'll try to get to it when we can, but if you or anyone else wants to try making a PR to fix it, feel free!

[sambhavnoobcoder]

i'll give this a look @Rocketknight1