Inconsistent accuracy with pl.metrics.Accuracy() across PL 1.1.8 and PL 1.2.x

[kielnino]

🐛 Bug

I have a simple binary segmentation model and train it to segment objects in an image. I measure the accuracy with pl.metrics.Accuracy(). After I switched from PL 1.1.8 to PL 1.2.x without any code-changes the accuracy-values where different (see also my discussion-topic).

I tried to reproduce the problem and even If I seed everything and use synthetic data the problem persist. I have a BoringModel-Colab which uses the same input-image over and over again to overfit the network on purpose.

If I run this with PL 1.1.8 for 3 epochs, I get

loss=0.738, val_binary_segmentation_accuracy=0.375

DATALOADER:0 TEST RESULTS
{'fake_test_acc': 0.7458580136299133}

and with PL 1.2.5 I get

loss=0.738, val_binary_segmentation_accuracy=0.444

DATALOADER:0 TEST RESULTS
{'fake_test_acc': 0.7458580136299133}

As the loss and the test-accuracy (which is also just the loss) is the same, I supect the inconsistency to be in the metric

Please reproduce using the BoringModel

https://colab.research.google.com/drive/1eRgcdQvNWzcEed2eTj8paDnnQ0qplXAh?usp=sharing

To Reproduce

Run the Colab once with:

! pip install torchtext==0.8.0 torchvision==0.8.0 torch==1.7.1
! pip install pytorch-lightning==1.1.8

and one with

 ! pip install pytorch-lightning==1.2.5

Expected behavior

In this case, where everything is seeded and I show the network just the same image, I would expect the accuracy to be the same.

[ananthsub]

After 1.2, you must reset your metrics explicitly - See this for more context: #5409

Changing your validation epoch from this

    def validation_epoch_end(self, outputs) -> None:
        torch.stack([x['x'] for x in outputs]).mean()
        self.log('val_binary_segmentation_accuracy', self.seg_val_metric.compute(), prog_bar=True)

to this

    def validation_epoch_end(self, outputs) -> None:
        torch.stack([x['x'] for x in outputs]).mean()
        self.log('val_binary_segmentation_accuracy', self.seg_val_metric.compute(), prog_bar=True)
        self.seg_val_metric.reset()

should work

[kielnino]

You are right, thank you for pointing this out. Now the accuracies are equal.

But the documentation states:

From v1.2 onward compute() will no longer automatically call reset(), and it is up to the user to reset metrics between epochs, except in the case where the metric is directly passed to LightningModules self.log.

which I do.

[ananthsub]

@kielnino do you see this issue if you do this?

    def validation_step(self, batch, batch_idx):
        x, y = batch
        output = self.base(x)
        loss = self.loss(y, output)

        self.log('val_loss', loss, on_epoch=True)

        true_labels = y.view(y.size(0), -1).int()
        pred_labels = (torch.sigmoid(output).view(output.size(0), -1) > 0.5).int()
        self.seg_val_metric(pred_labels, true_labels)
        self.log('val_binary_segmentation_accuracy', self.seg_val_metric, on_epoch=True)

        return {"x": loss}

[kielnino]

This also works. I just see that it is exactly the same in the documentation. I was not paying attention. Thank you very much.