Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename and update default metrics #359

Open
blumenstiel opened this issue Jan 9, 2025 · 2 comments
Open

Rename and update default metrics #359

blumenstiel opened this issue Jan 9, 2025 · 2 comments

Comments

@blumenstiel
Copy link
Collaborator

Currently, mIoU is named Multiclass_Jaccard_Index. I would like to rename it to mIoU, because (1) mIoU is the much more common word (I had to google Multiclass_Jaccard_Index the first time I used TerraTorch) and (2) it is shorter which is much better for displaying the results in MLFlow/WanB etc.

Also, we could think about dropping the micro metrics. I see the risk that not every user knows the differences between macro and micro metrics and could report the false one (most of the time micro is higher, so I would rather report this one if I don't know the differences). We can add a parameter to the task to add other metrics like micro mIoU.

Any other metrics that we should change? Any opinion from a user perspective?
@paolo-fraccaro @fmartiescofet @daniszw @romeokienzler @Joao-L-S-Almeida

@fmartiescofet
Copy link
Contributor

I agree that renaming it to mIoU would probably make it much easier for the user to understand what it actually is.

Regarding dropping the micro metrics I understand the point of removing them so people don't get confused but I would suggest to try to make it that people can customize and add new metrics to the reported ones in case they want some specific metric. With the current setup that's not possible unless they modify or create a new task.

@blumenstiel
Copy link
Collaborator Author

Agree, making additinal metrics configureable would be good (as a list of string (names)/dict(classes)?). Their are only a limited numbers of metrics per task from torchmetrics, so it should be easy to do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants