DINO evaluation on custom dataset crushes #171
Comments
|
Do you evaluate the COCO pre-trained models on your custom dataset? You need to train a new model for your dataset. |
|
@SlongLiu I'm not evaluating, I'm training your pre-trained weights on a custom class with |
|
@SlongLiu any updates? were you able to reproduce the issue? |
|
As I do not know the custom dataset you used. I cannot reproduce the error. However, the error is raised by cocoapi. There are some similar problems in other projects, which may be helpful. One possible reason: facebookresearch/detectron2#1570 (comment)
|
|
@SlongLiu it does not matter what custom dataset you use as far as it stays custom and not COCO in order to reproduce the error. |
|
We did not meet the problem in our other custom datasets. You seem to have other costume codes, such as the I do not think the problem is caused by our project, as our custom dataset design inherits from detectron2. Hence, I suggest you refer to other similar issues in the project, which may be helpful. |
|
@SlongLiu the custom code in coco_detr_graffiti.py is not altering anything, the issue is reproducible with |
|
Thanks for the question, I will check it out soon. Could you provide more details about the commands you used? As well as other modified codes if there are. |
|
@SlongLiu sure, the command I used is The config is the same as above, except for the dataloader: And the |
|
Thanks for the details. I tested the model with the modification you provided. I find you still infer models on your custom dataset in your latest experiments. Although changing the line The lines 57-59: dataloader.train.dataset.names = 'graffiti_train'
dataloader.test.dataset.names = 'graffiti_test'
dataloader.evaluator.dataset_name = 'graffiti_val'When I direct ran the shell you provided, it raised an error: Traceback (most recent call last):
File "tools/train_net_satellit_graffiti.py", line 262, in <module>
args=(args,),
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/engine/launch.py", line 82, in launch
main_func(*args)
File "tools/train_net_satellit_graffiti.py", line 249, in main
print(do_test(cfg, model))
File "tools/train_net_satellit_graffiti.py", line 136, in do_test
model, instantiate(cfg.dataloader.test), instantiate(cfg.dataloader.evaluator)
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/config/instantiate.py", line 67, in instantiate
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/config/instantiate.py", line 67, in <dictcomp>
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/config/instantiate.py", line 83, in instantiate
return cls(**cfg)
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/build.py", line 241, in get_detection_dataset_dicts
dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names]
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/build.py", line 241, in <listcomp>
dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names]
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/catalog.py", line 58, in get
return f()
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/datasets/coco.py", line 500, in <lambda>
DatasetCatalog.register(name, lambda: load_coco_json(json_file, image_root, name))
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/datasets/coco.py", line 69, in load_coco_json
coco_api = COCO(json_file)
File "/home/liushilong/anaconda3/envs/q2x/lib/python3.7/site-packages/pycocotools/coco.py", line 84, in __init__
with open(annotation_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/data/kamila/data/test/annotations/instances_default.json'It means the code still runs on your custom dataset. (The dataset registration process ran successfully.) When annotating the lines 57-59, the inference runs successfully on COCO without any error. I suggest stepping into the error lines, i.e., File "/opt/conda/lib/python3.8/site-packages/pycocotools/coco.py", line 327, in loadRes
assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \to compare the difference between the ground truth ids and model-predicted ids. |
|
@SlongLiu It is pretty obvious my code crushes if you run it, because you don't have the data I have. This is what error Let me now repeat myself. What I want to do: I want to use your pre-trained weights to fine-tune them on my custom data with one class 'graffiti'. I do not want to evaluate a trained model, and I do not want to run inference on COCO. I want to see the evaluation of the model during the training every n iterations as in What I did: As stated in the tutorial, I added What happened: Apparently, instead of evaluating the performance of the model on my custom class I'm currently training the model on, it is being evaluated on COCO classes, which I do not need because when I'm fine-tuning on my custom class and custom data, I need to see performance of my custom class. Can you tell me now what am I doing wrong? Is it not the way to achieve what I want? @rentainhe it would be really nice if you could help out here because I opened the issue a month ago and there's still no improvement whatsoever. |
|
Hello, I want to explain what my experiments show:
You said the model was crushed on COCO as well in your experiments. The reason is that you still inferred the model on your custom dataset. The model can work well on COCO. Hence, the reason for your error is your custom dataset. |
|
@SlongLiu Of course, the model can work well on COCO, but this is not what I need. |
|
are there any updates? @rentainhe |
When fine-tuning with following config:
I'm getting an exception:
The text was updated successfully, but these errors were encountered: