training curves look fragmented / not clear what is "step"

[undertherain]

❓ train log looks fragmented / docs not clear on how to log multiple parameters

The docs show how single metric is logged, what is the right way to log multiple metrics?

Also examples show the dictionary of metrics being returned as results of training_step and validation_step functions - can't logging be done from there automatically, and if logging is done manually why do we need to return those dicts?

Code

    def training_step(self, batch, batch_idx):
        s1, s2, target = batch
        logits = self(s1, s2)
        loss = F.cross_entropy(logits, target)
        acc = accuracy(logits, target)
        self.log('train_loss', loss)
        self.log('train_acc', acc)
        result = OrderedDict({
            'train_loss': loss,
            'train_acc': acc,
        })
        return result

    def validation_step(self, batch, batch_idx):
        s1, s2, target = batch

        logits = self(s1, s2)
        loss = F.cross_entropy(logits, target)
        acc = accuracy(logits, target)
        self.log('val_loss', loss)
        self.log('val_acc', acc)
        result = OrderedDict({
            'val_loss': loss,
            'val_acc': acc,
        })
        return result

What's your environment?

using PL 0.9.1rc4 and wandb 0.10.4

[rohitgr7]

the step is the global_step used for training.

to log multiple metrics at once you can use log_dict.

Also, you need to return those dicts only if you need them in training_epoch_end else just return loss from training_step that needs to be minimized.

[ananyahjha93]

@undertherain the docs mention here that self.log defaults to on_step for training_step and on_epoch for validation_step. The gap in your training metrics graph is because the x axis plots global step like @rohitgr7 mentioned.

https://pytorch-lightning.readthedocs.io/en/stable/lightning_module.html

[undertherain]

Thanks!
what would be the right way to have contiguous training loss per training steps on one chart and training loss per epoch on another chart? And same for validation

if i set both on_step=True, on_epoch=True -
I'm getting additional charts of for each epoch - which I don't really need
The loss through epochs is still rendered against global step...
And the val_loss us inly per epoch or per step for each individual epoch in those additional charts :-\

[rohitgr7]

I'm getting additional charts of for each epoch - which I don't really need

this must be happening in case of validation and not in case of training right? I am not sure if is it intended behavior or a bug here.

[borisdayma]

@undertherain step increases at each logging on W&B for pytorch-lightning integration (due to #2015).
You just need to set the x-axis as global_step.

[george-gca]

@rohitgr7 what should I do in this case? Just quoting my question in another post, don't know if I should just copy/paste it here. But log_dict is throwing error.

#2783 (comment)

[undertherain]

hm, neither way I could get a view with val loss per step across more than one epoch....

and btw is there any place where it is formally defined what is step and global_step?

[george-gca]

@undertherain what are you trying to achieve exactly? Have 2 plots of loss, one with steps being epoch, another one with steps being batch?

[undertherain]

want to achieve

1. Understanding what the parameters do and what's step vs global_step
2. Why with exact same parameters to log(), and I made sure I explicitly set all default values as well - plots look very differently on wandb. and no, that's not fixable with choosing axis
   say, if I choose on_step=True and on_epoch=False for both train and validation - I can get per epoch axis for train loss on wandb, but not for validation loss
3. plot continuous val loss across steps for several epochs. plot per epoch loss for validation without having individual chart for every epoch on wandb

[george-gca]

1. I believe it is confusing in the docs the usage of on_step=True, on_epoch=True. I don't use them directly, since it states in the docs that their values are already set depending on where you call the log function from. Also, global_step seems like a concept from wandb, not from pytorch. I would guess that step is the processing of a batch, and global_step of a whole epoch.
2. No idea.
3. I managed to make it work on tensorboard, maybe my solution works for you too (presuming that the behavior of all logs must be similar). I wanted to achieve this:
   
   where the x axis is the epoch.

In training_step I return a dict of the losses, being key the name of the loss and value the calculated value. Since I wanted to log the loss value per epoch, not per batch, I don't do the logging in training_step. I presume that if you wanted to log per step, you should use self.log_dict(losses_dict, prog_bar=True, logger=True) here. Note that the keys must be always the same.

For logging the metrics per epoch, I return in validation_step a dict with the metrics, the same way as in training_step. Then I implemented validation_epoch_end. Inside I check if it is not the sanity check, then I use a builtin method that aggregates the values for the metrics for all batches in that epoch:

def validation_epoch_end(self, outputs):
    if not self.trainer.running_sanity_check:
        for metrics_dict in outputs:
            self.logger.agg_and_log_metrics(metrics_dict, step=self.current_epoch)

Hope it helps.

[borisdayma]

The next release of pytorch-lightning should be clearer as wandb won't have a global_step any longer.
You can test it on the master branch.