[Discussion] There should be a Metrics package

[Darktex]

🚀 Feature

Introduce a metrics package that contains easy to use reference implementations of metrics people care about. The package should take care of the following:

1. Logging and printing in the right format. The metric object should know how it wants to be printed, depending on the available surface (terminal vs notebook vs TensorBoard). For example, a ConfusionMatrix metric can use Pandas for terminal (where it will print a nicely tabulated representation) and maybe a color-coded map for notebooks and TensorBoard.
2. Handle the actual computation in all cases: cpu, single-gpu, single-box and multi-box DDP.
3. Support plugging into a MetricsReporter of some sort that will generate a full report of all the metrics you care about.

This would be a rather large change, and I'm not sure what is the best way to do it. This issue is really meant to spur discussion on the topic. The actual solution might require some stuff to be landed on PyTorch, and that's fine.

Motivation

Metrics are a big component of reproducibility. They satisfy all the requirements you can think about to justify standardizing them:

1. They are very hard to test. Kinda reminds me of the barber paradox: who measures the metrics? :) The only way you test them is really by checking corner cases and depending on someone else's implementation (eg sklearn's), hoping you did your piping right.
2. They are unsexy to implement so nobody wants to really spend time dealing with them. They are boilerplate, but necessary boilerplate.
3. They are ambiguous. Even something as trivial as a variance is actually not trivial to compare! Do you divide by n or n-1? (aka: do you use Bessel's correction?). Numpy defaults to not using it, MATLAB and PyTorch default to using it. It's unsurprising to see threads like this.
4. They are quite hard to implement for all cases: how do we compute a multi-node, multi-GPU ROC AUC score?

Pitch

I think Lightning should take a page from Ignite's book and add a metrics package.

I also propose that Lightning take care of the following:

1. Pretty printing. Lightning is the interface to Tensorboard and the loggers, so it should bridge the gap. For example, torch should provide something similar to the sklearn.metrics package: given a tensor x and a tensor y, compute f1 score or whatever. Lightning should wrap that to handle attaching the result on Tensorboard, printing a metrics report table, etc.
2. Attaching to events. PyTorch has no notion of events, so Lightning should handle that. This is essentially akin to writing the validation_step() and validation_end() steps for each metric, so that they can be computed efficiently per-batch.
3. Handle DDP (not sure about this one). On the one hand, it's breaking the abstraction that PyTorch computes and Lightning wraps, but on the other hand I don't see a way out: how you compute the metrics will necessarily change once you introduce DDP into the mix. For example, an accuracy can just keep a running average and how many samples it has seen so far, but something like ROC AUC will necessarily require that all processes store their predictions.

In this proposal, the API for the LightningModule will be simplified significantly. For example, something like this:

class CoolSystem(pl.LightningModule):

    def __init__(self):
        super(CoolSystem, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_idx, training_metrics):
        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
        loss = F.cross_entropy(y_hat, y)
        return {'loss': loss,}

    def configure_metrics(self):
        # OPTIONAL
        training_metrics = MetricsLog([Accuracy(), Loss()])
        eval_metrics = MetricsLog([F1Score('micro'), F1Score('macro'), F1Score(None), Accuracy(), PrecisionAtRecall(80)])
        return {'train_metrics': training_metrics, 'eval_metrics': eval_metrics}

    def configure_optimizers(self):
        # REQUIRED
        # can return multiple optimizers and learning_rate schedulers
        # (LBFGS it is automatically supported, no need for closure function)
        return torch.optim.Adam(self.parameters(), lr=0.02)

    @pl.data_loader
    def train_dataloader(self):
        # REQUIRED
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

    @pl.data_loader
    def val_dataloader(self):
        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

Alternative implementation

If we find a way to have all metrics computation code done in PyTorch, even for DDP, it would be highly preferable I think. I just don't know if it's possible - maybe if we formulate metrics as a layer of sorts we might be able to do that? All standard layers have a state that persists & gets updated across batches (its weights :D ) so maybe we can implement metrics as a sort-of nn.Module?

Additional context

There is a separate discussion about providing the underlying muscle for this directly in torch (see pytorch/pytorch#22439).

[williamFalcon]

Love it!
@srush thoughts?

[srush]

So I like the API and agree this would be nice to factor out. However I don't think that lightning should supply the metrics. Outside of classification, this is a really tricky thing to get right and often requires lots of domain engineering. Just as with the dataloaders, you don't want to own them, but facilitate and super clean API.

Random, why even specify CrossEntropy in training? Isn't that just another metric?

[oplatek]

Hi,

I may contribute WER because I am interested in using it.
There is a reference implementation for Pytorch (https://github.com/1ytic/pytorch-edit-distance), I can also easily implement it in numpy for or plain Python.

However, I have a few general questions based on the discussion above:

My needs vary:

• Sometimes I need metric per batch - e.g. loss
• Sometimes I need metric per a training example (can be hacked by batch_size==1) - e.g for filtering problematic examples while debugging the model
• Most of the time I need metric per epoch
  - For example in case of WER you may accumulate edit_distance per batch and numwords per batch and just sum(edit_distances) / sum(numwords) - I wanted to demonstrate that WER computation differs (slightly) for batch and Dataset.
• Quite often I need to compute more time-consuming metrics at the end of training/on-demand - e.g. evaluate the final system with the trained NN

My question:
Based on my understanding pytorch-lightning does not support events i.e end_of_Epoch, every_N_iteration, end_of_training, etc.
Does this proposal (or any other feature in PyTorch lightning) support triggering metrics
1. at different occasions(start_epoch, end_epoch, 4000iteration, etc)
2. on different objects (training example, batch, dataset(loader))?

[Darktex]

So I like the API and agree this would be nice to factor out. However I don't think that lightning should supply the metrics. Outside of classification, this is a really tricky thing to get right and often requires lots of domain engineering. Just as with the dataloaders, you don't want to own them, but facilitate and super clean API.

I agree with the extensibility principle, but I'd also like each community to have their own standardized metrics to help with reproducibility in research (maybe Lightning supplies the classification ones as an example, and then maybe the image community will build the ones for the tasks they care about eg segmentation, pose estimation, etc. Same for the speech community which may build WER and so on). I think it's fine if they live in submodules, or even externally as long as the discoverability problem is tackled. I like submodules more because they are more "official" and this pushes people to collaborate on one reference implementation rather than have too many.

Random, why even specify CrossEntropy in training? Isn't that just another metric?

Fair enough :) Yep, it's definitely another metric.

[srush]

Sure, I think we agree. Like a dataloader like torch-text https://github.com/pytorch/text is not a submodule right? Its an external package.

I agree with the extensibility principle, but I'd also like each community to have their own standardized metrics to help with reproducibility in research (maybe Lightning supplies the classification ones as an example, and then maybe the image community will build the ones for the tasks they care about eg segmentation, pose estimation, etc. Same for the speech community which may build WER and so on).

The reason I am against this is for the same reason! There are a million implementations of BLEU in NLP exactly because everyone wants to standardize it to "their" version.

[Darktex]

Ignite has convinced me that the computation part (DDP included) could be done at the Torch layer (more info in this comment there). Even if people want to reimplement their own BLEU, we'd probably want to land a reference implementation in TorchText. You are free to avoid it if you don't like it, but it's also a pain to write your own so I think you'd do it only if you either don't trust its code, or it's not flexible enough for you, or it's not fast enough/uses too much memory. I'm willing to bet reimplementations will stop if something better is there. Isn't it the whole point of Lightning to stop people for writing their own training loops? :D I see it as very much the same thing.

Assuming the PyTorch API in the proposal happens, then maybe there is no API change that Lightning would have to undergo: just write your own update() and compute() in the validation_step() and validation_end() methods and pass them to the dicts to attach to TensorBoard? That would take care of the pretty printing and the boilerplate would seem rather minimal:

class CoolSystem(pl.LightningModule):

    def __init__(self):
        super(CoolSystem, self).__init__()
        # not the best model...
        self.l1 = torch.nn.Linear(28 * 28, 10)
        self.train_macro_f1 = nn.F1Score("macro")
        self.validation_macro_f1 = nn.F1Score("macro")

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_idx):
        # REQUIRED
        x, y = batch
        y_hat = self.forward(x)
        loss = F.cross_entropy(y_hat, y)
        self.train_macro_f1.update(y_hat, y)  # void method, just update state
        m = self.train_macro_f1()  # forward calls compute(), takes no args. Compute from state
        tensorboard_logs = {'train_loss': loss, 'train_macro_f1': m}
        return {'loss': loss, 'log': tensorboard_logs}

    def validation_step(self, batch, batch_idx):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)
        self.validation_macro_f1.update(y_hat, y)
        return {'val_loss': F.cross_entropy(y_hat, y)}

    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        tensorboard_logs = {'val_loss': avg_loss, 'val_macro_f1': self.validation_macro_f1()}
        return {'avg_val_loss': avg_loss, 'log': tensorboard_logs}

    def configure_optimizers(self):
        # REQUIRED
        # can return multiple optimizers and learning_rate schedulers
        # (LBFGS it is automatically supported, no need for closure function)
        return torch.optim.Adam(self.parameters(), lr=0.02)

    @pl.data_loader
    def train_dataloader(self):
        # REQUIRED
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

    @pl.data_loader
    def val_dataloader(self):
        # OPTIONAL
        return DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

There is some boilerplate, but it's not the end of the world

[Borda]

I would not add another object which would hold metric values, it seems as duplicating logger functionality... What about defining just a list of metrics to be computed (function or callable class) and these metric will be automatically added to output dictionary?
cc: @PyTorchLightning/core-contributors ^^ opinion about all above?

[oplatek]

I like the way Ignites metrics expose their API as previously discussed.

Opinion:
Organizing the evaluation of the metric at the right moments is often a bigger issue than implementing the metric itself.

Motivation for Solution:
Metrics "Evaluation" is in a similar situation as "a training loop" before PL.
I suggest mimic the approach of how a LightningModule simplified the training loop.
I really like about PL how PL breaks the messy training loop and defines "callback/event" mechanism in the LightningModule.

Solution: EvaluationPlan (pseudo) implementation

class EvaluationPlan():
      def __init__(self, **kwargs):
              pass    # todo think it through

      dev setup_train_epoch(self, trainer, module):
              pass    # E.g. init metric for evaluating epoch

      dev setup_train_batch(self, trainer, module):
              pass    # E.g. init metric for collecting data during training epoch

      dev eval_train_epoch(self, trainer, module):
              pass    # Compute final metric value for epoch and log it

       .... 
       # more useful events 
       ....

      def setup_training(self, trainer, module):
             pass # init metrics for the whole training - collect data in any method above

      def eval_training(self, trainer, module):
             pass  # compute & log metric for the whole training 

       .... 
       # every supported event need to be implemented in a base class
       ....