Add Expected Calibration Error metrics and derived

[vfdev-5]

🚀 Feature

Idea is to provide Expected Calibration Error metrics as coded in Baal : https://github.com/ElementAI/baal/blob/master/src/baal/utils/metrics.py#L88

cc @Dref360

[oaimtiaz]

Hi! I am interested in helping out with this, but I am slightly confused as to what is expected. Should we be implementing these metrics again anew or using the Baal implementations in this project?

[vfdev-5]

Hi @oaimtiaz, idea is to implement ECE and MulticlassECE from scratch and using our formalism of metric's implementation : https://pytorch.org/ignite/metrics.html#how-to-create-a-custom-metric
Implementation in Baal can be a sort of reference and shouldn't be reused.
Do not hesitate to ask other question if need other details.

[oaimtiaz]

Thanks! I'll get started on this right away!

[vfdev-5]

@oaimtiaz any updates on this ?

[oaimtiaz]

Hello, I have made a bit of progress on the project, however I am currently stuck at one part. I do not believe I will be able to complete this task. Apologies.

…

On Wed, Nov 4, 2020 at 4:45 PM vfdev ***@***.***> wrote: @oaimtiaz <https://github.com/oaimtiaz> any updates on this ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1009 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APPJRKD27HYL3UZYA2ENJ2LSOHKR3ANCNFSM4MY2GQDA> .