LightningCLI creates wandb experiment when using --help

[awaelchli]

Bug description

Wandb creates an experiment directly when instantiating the lightning.pytorch.loggers.WandbLogger object:
https://github.com/Lightning-AI/lightning/blob/cc3800aac3e91d0a741047fd26584b1de0cfedc4/src/lightning/pytorch/loggers/wandb.py#L353-L356

This is normally no problem when used inside of a regular script, but it is a bit awkward when used in combination with e.g. the LightningCLI. For example, if you just query the --help command, not intending to actually run the scritp fully, it will still create a run in the wandb dashboard.

What version are you seeing the problem on?

2.00, 2.1.0dev

How to reproduce the bug

import torch
from lightning.pytorch import LightningModule
from lightning.pytorch.cli import LightningCLI
from lightning.pytorch.loggers import WandbLogger


class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def training_step(self, batch, batch_idx):
        loss = self.layer(batch).sum()
        return {"loss": loss}


if __name__ == "__main__":
    LightningCLI(
        BoringModel,
        trainer_defaults=dict(logger=WandbLogger(project="test"))
    )

Then running (just --help)

python repro.py --help

logs me in to wandb and creates a run.

Error messages and logs

wandb: Currently logged in as: awaelchli. Use `wandb login --relogin` to force relogin
wandb: wandb version 0.15.0 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.13.9
wandb: Run data is saved locally in ./wandb/run-20230426_231331-gmlnuzsr
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run fiery-butterfly-5
wandb: ⭐️ View project at REDACTED
wandb: 🚀 View run at REDACTED

Environment

Current environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow): LightningCLI
#- PyTorch Lightning Version (e.g., 1.5.0): master (2.1.0dev)
#- Lightning App Version (e.g., 0.5.2): x
#- PyTorch Version (e.g., 2.0): 2.0
#- Python version (e.g., 3.9): 3.10
#- OS (e.g., Linux): MacOS
#- CUDA/cuDNN version: x
#- GPU models and configuration: x
#- How you installed Lightning(`conda`, `pip`, source): pip
#- Running environment of LightningApp (e.g. local, cloud): x

More info

No response

cc @awaelchli @morganmcg1 @borisdayma @scottire @parambharat @carmocca @mauvilsa

[awaelchli]

My bad, this is actually not directly related to Lightning CLI (editing the description and labels now). It happens because the wandb logger initializes the experiment here in init (I forgot):
https://github.com/Lightning-AI/lightning/blob/cc3800aac3e91d0a741047fd26584b1de0cfedc4/src/lightning/pytorch/loggers/wandb.py#L353-L356

I need to find a solution for this in that logger, because it is indeed very awkward when used in combination with lightning cli.

[awaelchli]

This logic here

if _WANDB_GREATER_EQUAL_0_12_10: 
     wandb.require("service") 
     _ = self.experiment 

in wandb is only relevant for the strategies that use mp.multiprocessing (ddp_spawn). It was fixed like this in the past because ddp_spawn was actually the default multi-gpu strategy. But with Lightning 2.0, we changed that to subprocess ddp, and there the above logic wouldn't be necessary. We need to make a decision whether it is ok to remove this logic again. Or I'm not sure how we can support both worlds.

[mauvilsa]

The __init__ of the logger will not run for --help if you set the default as dict(logger=lazy_instance(WandbLogger, project="test")) (see class-type-defaults) or as a dict with class_path and init_args,.

[awaelchli]

Smart! This is really helpful!