-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Description
🚀 Feature
Create the metric MulticlassAUROC to allow for the AUROC metric to be used in multi-class problem settings. Or,
Expand the AUROC metric to support multi-class data, which would also directly solve this AUROC bug that instead gives a random value when used in multi-class problems: #3303
Motivation
AUROC is a useful metric for model performance, but the current AUROC metric is not made for multi-class problems.
An out of the box working multi-class AUROC metric would be great.
Pitch
According to this post on the PyTorch Lightning forum, implementation would just require wrapping existing functionality together.
Implementing this metric would lower the barrier of its usage, instead of needing to be aware of the different metrics available that could be combined to effectively get his metric.
Alternatives
- Use sklearn's implementation of AUROC:
sklearn.metrics.roc_auc_score.
This slows down the training, however, due to the need of transferring prediction and target data to CPU. - Implement manually by combining multiclass auc and a multiclass roc calculation
Additional context
- Relevant discussion: https://forums.pytorchlightning.ai/t/pytorch-lightning-auroc-value-for-multi-class-seems-to-be-completely-off-compared-to-sklearn-using-it-wrong/61/6
- Relevant AUROC bug when used for multi-class: AUROC metric should throw an error when used for multi-class problems #3303