MeanAveragePrecision update method does not accept torchvision.datapoints.BoundingBox arguments

[ryanhunter237]

🐛 Bug

When the preds or target arguments to MeanAveragePrecision.update method are not of the type list[dict[str,Tensor]] a ValueError is raised, but this precludes dictionary values of type torchvision.datapoints.BoundingBox which inherit from Tensor.

The ValueError is raised in the function torchmetrics.detection.helpers._input_validator. I think this could be fixed by changing all the lines like

if any(type(target[ivn]) is not Tensor for target in targets):

to be something like

if not all(isinstance(target[ivn], Tensor) for target in targets):

One could also validate that the BoundingBox.format is the same as MeanAveragePrecision.box_format whenever a BoundingBox class is passed to the update method.

To Reproduce

from torch import tensor
from torchmetrics.detection import MeanAveragePrecision
from torchvision.datapoints import BoundingBox, BoundingBoxFormat

preds = [
    dict(
        boxes=tensor([[1.0, 2.0, 1.0, 2.0]]),
        scores=tensor([0.1]),
        labels=tensor([0]),
    )
]

bounding_box = BoundingBox(
    data=tensor([[1.0, 2.0, 1.0, 2.0]]),
    format=BoundingBoxFormat.XYXY,
    spatial_size=(100,100)
)
target = [
    dict(
        boxes=bounding_box,
        labels=tensor([0]),
    )
]

metric = MeanAveragePrecision(iou_type='bbox', box_format='xyxy')
metric.update(preds, target)

Expected behavior

MeanAveragePrecision.update should accept torchvision.datapoints.BoundingBox arguments.

Environment

python 3.11.4
pytorch 2.0.1
torchmetrics 1.2.0
torchvision 0.15.2

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Hi @ryanhunter237, thanks for raising this issue.
I created PR #2180 that will change the checks in the input validator to use isinstance instead as you propose. I tried locally to run the example you provide and it should work. That said, we are not going to officially support the datapoint interface from torchvision before it is stable. To just run you example with the newest version I had to make changes to the import path, class name and argument names.

from torch import tensor
from torchmetrics.detection import MeanAveragePrecision
from torchvision.tv_tensors import BoundingBoxes, BoundingBoxFormat

preds = [
    dict(
        boxes=tensor([[1.0, 2.0, 1.0, 2.0]]),
        scores=tensor([0.1]),
        labels=tensor([0]),
    )
]

bounding_box = BoundingBoxes(
    data=tensor([[1.0, 2.0, 1.0, 2.0]]),
    format=BoundingBoxFormat.XYXY,
    canvas_size=(100,100)
)
target = [
    dict(
        boxes=bounding_box,
        labels=tensor([0]),
    )
]

metric = MeanAveragePrecision(iou_type='bbox', box_format='xyxy')
metric.update(preds, target)

When it is stable we will reconsider adding testing and checking such that it is official supported.