Does AccuracyMetric compute accumulated or mean values of each batch?

[xiadingZ]

there are two version of calculate accuracy:

1. calculate accuracy of each batch then do mean operation in validation_epoch_end
2. accumulate each batch's correct, total values then compute accuracy once in validation_epoch_end

which ones does AccuracyMetric do?

[awaelchli]

The 2nd.
And it does that also across gpus.
https://pytorch-lightning.readthedocs.io/en/latest/metrics.html

[xiadingZ]

1. In ddp mode, If i want to implement validation_epoch_end, Does this code do the operation correctly? All Metric class has this rule?

    def validation_step(self, batch, batch_idx):
        ...
        vid_acc = self.accuracy_video(video_labels_hat, video_labels)
        confusion_metric = self.confusion_metric(video_labels_hat, video_labels)
        ...


    def validation_epoch_end(self, outputs):
        vid_acc = self.accuracy_video.compute()
        vid_confusion_metric = self.confusion_metric.compute()

        self.log('val_acc', val_acc, on_epoch=True, prog_bar=True, sync_dist=True)

2. does it mean that I have to create multiple AccuracyMetric instances for each variable in each mode, such as train_vid_acc=AccuracyMetric(), val_vid_acc=AccuracyMetric(), test_vid_acc=AccuracyMetric(), train_img_acc=AccuracyMetric()...?

[awaelchli]

You can call .compute() as it is shown in the docs:
https://pytorch-lightning.readthedocs.io/en/latest/metrics.html
Is this what you mean by "accumulated"?

2. Yes you have to create multiple, because they track the internal state of that particular metric.

[xiadingZ]

1. yes. my accumulated is what .compute() does. As in docs, it call self.log in training_step and training_epoch_end, but I only call it in validation_epoch_end, will this also get the synced value of Metric?

[SkafteNicki]

Yes, you can call it in validation_epoch_end. When compute is called it will sync the Metric, regardless of which method the call is happening in.