No Callbacks for Validation Batch Step - How To Get Progress of Validation?

[0x6b756d6172]

🚀 Feature

The Callbacks has two functions on_batch_start and on_batch_end. The documentation and code execution shows that these are only called for training batches, not validaton (or test). I am building a Callback for my own logging/dashboarding via Streamlit and I have a requirement to track the progress of both the train batch steps and validation batch steps during training. I see that there is a validation progress bar in the console during training - how is this implemented (if not via a Callback)?

Current Code

class MyCallback(pl.Callback):
    def on_train_start(self, trainer, pl_module):
        self.train_batch_count = 0
        self.train_batch_total = len(trainer.train_dataloader)

    def on_batch_end(self, train, pl_module):
        self.train_batch_count += 1
        percent_complete = self.train_batch_count / self.train_batch_total
        # display on screen

    def on_epoch_end(self, train, pl_module):
        self.train_batch_count = 0

[Borda]

hello, as you stated you want to have a logger, not a callback, you can write a custom logger...
Moreover, it would be welcome to add Streamlit.io as Lightning loggers, what do you think?

[0x6b756d6172]

Hi, @Borda, thanks for the reply. I looked over the documentation for Loggers and it will indeed be helpful for my future work. However, looking at LightiningLoggerBase, I don't see anything for capturing the epoch or batch progress for training (i.e. Epoch 1 of 5 or Batch 1 of 200) as I am doing above so that I can use Streamlit's streamlit.progress to render a progress bar of the batch, training and validation iterations.

I do think a Streamlit logger would be an excellent addition to Lightning but I am still in the early exploratory phase of using Streamlit myself and it currently seems like it would require a bit of work to make a useful generic dashboard a la Tensorboard. I'll continue experimenting on my end to see where it goes.

[awaelchli]

In case this is still relevant to you, we now also have callbacks for validatation and test batches!
See https://pytorch-lightning.readthedocs.io/en/latest/callbacks.html#callback-base

[awaelchli]

Closing this. As far as i can tell, with the recently introduced new callback methods (validation_batch_start etc) it is now possible to track the progress as you desired.

[awaelchli]

https://pytorch-lightning.readthedocs.io/en/latest/pytorch_lightning.callbacks.base.html?highlight=callback#callback-base

[SouthMemory]

https://pytorch-lightning.readthedocs.io/en/latest/callbacks.html#callback-base

good work!