Remove automatic reset after `compute()` in the Metric's API?

[oterobaguer]

🚀 Feature

a) Remove automatic reset after compute() in the Metric API.
b) Option to automatic log running averages instead of batch-wise results

Motivation

Some of the metrics normally used, such as 'Recall', 'Intersection over Union',... etc, make no sense if calculated for a small set of images. For example, in binary classification, a batch in which there is no positive case returns 0. However, running averages are much better in these cases. Lightning metrics are automatically reset after calling compute(). Why is this behavior? isn't better to allow calling compute() many times, allowing for example, to compute running averages in between. If at some point a reset is needed the reset() method can always be called.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

@oterobaguer currently the reset is called because processes are synced whenever compute is called. Since we sync by doing a all_gather operation by default, the metric states on all processes are being accumulated, which means that if we do not do reset in between calls to compute then we would basically end up accumulating metric states multiple times.

[SkafteNicki]

cc: @teddykoker
Should we provide some way of computing running accumulated values?

[oterobaguer]

That would be great

…

On Thu, Nov 19, 2020 at 3:59 PM Nicki Skafte ***@***.***> wrote: cc: @teddykoker <https://github.com/teddykoker> Should we provide some way of computing running accumulated values? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4641 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK6MM4B6SZHVAJFFRKY5O5DSQUXEJANCNFSM4TTN72UQ> .

[teddykoker]

Sorry I missed this. In hindsight i think it would make sense to not call reset() on compute() see comment here: #5193 (comment)

[SkafteNicki]

Fixed by #5409. Closing now.