Support save_hyperparameters() to checkpoints without sending to logger

[timothybrooks]

🚀 Feature

self.save_hyperparameters() is an awesome simple way to save input arguments with checkpoints such that when loading a checkpoint the module will be constructed with the same parameters. The method also seems to log the hyperparameters to the experiment logger, which is not always desired. It would be great to add a flag to turn this on or off.

Pitch

Add a log=True flag, which is true by default, to save_hyperparameters(). This way hyperparameters could be saved but not logged by calling self.save_hyperparameters(log=False).

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[timothybrooks]

I still believe this should be fixed. Logging hyperparameters and saving them to a checkpoint serve different purposes, and saving them should not necessitate logging them.

I do not wish to log hyperparams, but I want to save them as part of the checkpoint. If this will not be fixed as I suggested, is there another way to achieve this while still leveraging ligtning's checkpointing logic?

[ananthsub]

Could you describe why it's not desirable for the logger to save hyperparameters? Which logger do you use? Could you extend the class to override the log_hyperarameters method and disable it?

[timothybrooks]

Sure, thanks for asking.

I am using the W&B logger. The lightning module hyperparameters constitute a portion of my configuration, but I also have configuration parameters for setup, data, training, as well as other custom parts of my experiment. I use W&B to log my entire configuration at the start of my experiment, which has a nice hierarchical structure that makes it easy to track configurations with the W&B interface. I call log_hyperparameters myself to do this.

I also call save_hyperparameters so that they are saved and loaded with checkpoints. Unfortunately this also logs all input arguments to the module. I do not wish to log these because 1) they are redundant with my logging of the entire configuration, 2) they do not have a hierarchical structure like my logging of the entire structure which makes them harder to interpret, 3) I have many input arguments and not all are user configurable or intended to be logged.

I believe a flag could be set in save_hyperparameters that is then checked here:
https://github.com/PyTorchLightning/pytorch-lightning/blob/1302766f83a5fc861ae77ef6f8798cd0b1d606c7/pytorch_lightning/trainer/trainer.py#L507

[maxwells-daemons]

For anyone else having this issue, here's a workaround I've been using.

class WandbManualHparamsLogger(pl.loggers.WandbLogger):
    @pl.utilities.rank_zero_only
    def log_hyperparams(self, params, skip=True):
        if not skip:
            super().log_hyperparams(params)

All hyperparameter logging internal to Lightning is skipped, and you can invoke log_hyperparams(..., skip=False) to log them manually.

It would be nice if this were available by default though.

[edgarriba]

@timothybrooks does this solution work for you ? Would be nice if any of you guys could send a PR with this.

For anyone else having this issue, here's a workaround I've been using.

class WandbManualHparamsLogger(pl.loggers.WandbLogger):
    @pl.utilities.rank_zero_only
    def log_hyperparams(self, params, skip=True):
        if not skip:
            super().log_hyperparams(params)

All hyperparameter logging internal to Lightning is skipped, and you can invoke log_hyperparams(..., skip=False) to log them manually.

It would be nice if this were available by default though.

[kaushikb11]

Picking this up, once the LoggerConnector Refactor PR is merged.