"dataloader.test.batch_size" > 1 when evaluation is not supported!

[wanghao9610]

Hi:
I want to set "dataloader.test.batch_size = 16" (default is 1) to accelerate the evaluation speed. But, I get a much smaller mAP results, e.g. 0.59 vs 41.5 mAP. Is it the upper stream(detectron2)'s issue? Could you fix the issue?

[rentainhe]

Hi: I want to set "dataloader.test.batch_size = 16" (default is 1) to accelerate the evaluation speed. But, I get a much smaller mAP results, e.g. 0.09 vs 41.5 mAP. Is it the upper stream(detectron2)'s issue? Could you fix the issue?

We will check this problem later~

[rentainhe]

Hi: I want to set "dataloader.test.batch_size = 16" (default is 1) to accelerate the evaluation speed. But, I get a much smaller mAP results, e.g. 0.09 vs 41.5 mAP. Is it the upper stream(detectron2)'s issue? Could you fix the issue?

We will check this problem this weekend, sry for the long time waiting

[rentainhe]

Hi: I want to set "dataloader.test.batch_size = 16" (default is 1) to accelerate the evaluation speed. But, I get a much smaller mAP results, e.g. 0.09 vs 41.5 mAP. Is it the upper stream(detectron2)'s issue? Could you fix the issue?

I was wondering which model you are used in this situation~ @wanghao9610

When setting test.batch_size > 1, the images of different image size will be padding into the same size as a batch input. And the padding part should be masked, however, for convenience, we ignored this situation before and used the same mask for each test image which may raise this issue, we will try to fix it.

[rentainhe]

Hi: I want to set "dataloader.test.batch_size = 16" (default is 1) to accelerate the evaluation speed. But, I get a much smaller mAP results, e.g. 0.09 vs 41.5 mAP. Is it the upper stream(detectron2)'s issue? Could you fix the issue?

You can try this solution if you're in a hurry by modifying the model like:

if self.training:
    batch_size, _, H, W = images.tensor.shape
    img_masks = images.tensor.new_ones(batch_size, H, W)
    for img_id in range(batch_size):
        img_h, img_w = batched_inputs[img_id]["instances"].image_size
        img_masks[img_id, :img_h, :img_w] = 0
else:
    batch_size, _, H, W = images.tensor.shape
    img_masks = images.tensor.new_ones(batch_size, H, W)
    for img_id in range(batch_size):
        img_h, img_w = images.image_sizes[img_id]
        img_masks[img_id, :(img_h-1), :(img_w-1)] = 0

[wanghao9610]

Thanks for your immediate reply~ I will try your solution.

[wanghao9610]

@rentainhe Hello, I have tried your provided solution, but I got the same much lower mAP result. Have you try it successfully?

[rentainhe]

@rentainhe Hello, I have tried your provided solution, but I got the same much lower mAP result. Have you try it successfully?

I was wondering which model you're testing under this situation @wanghao9610

I tried this on DINO-R50 with test-batch=2 each GPU, got 49.2AP

[wanghao9610]

I run this on DN_DETR-R50 with you provided weight (dn_detr_r50_50ep.pth), setting "dataloader.test.batch_size=16" with 2gpus, got 0.59 AP. My tesing command as below: python ./tools/train_net.py --config-file projects/dn_detr/configs/dn_detr_r50_50ep.py --num-gpus 2 --eval-only train.init_checkpoint=work_dirs/dn_detr_r50_50ep/dn_detr_r50_50ep.pth train.output_dir=work_dirs/dn_detr_r50_50ep dataloader.test.batch_size=16. You could try this command and you will may get the result as same as my.

[rentainhe]

I run this on DN_DETR-R50 with you provided weight (dn_detr_r50_50ep.pth), setting "dataloader.test.batch_size=16" with 2gpus, got 0.59 AP. My tesing command as below: python ./tools/train_net.py --config-file projects/dn_detr/configs/dn_detr_r50_50ep.py --num-gpus 2 --eval-only train.init_checkpoint=work_dirs/dn_detr_r50_50ep/dn_detr_r50_50ep.pth train.output_dir=work_dirs/dn_detr_r50_50ep dataloader.test.batch_size=16. You could try this command and you will may get the result as same as my.

I will try to reproduce this issue later~

[ihavision]

Same problem. I also got <1% AP with DINO-R50 and dataloader.test.batch_size=8 on single GPU. The mask patch provided above doesn't make a difference.

[theneotopia]

Same problem, still not fixed yet.
Experimented with DETR-R50 (converted) on a single RTX 3090 GPU, I got 0.9675 AP with test.batch_size=16, 7.37 AP with test.batch_size=4, 18.79 AP with test.batch_size=2, and 42.0 AP with test.batch_size=1.
It seems that the evaluator only works correctly when the test.batch_size is set to 1.

[persistorkwj]

Isn't this BUG fixed yet?