WandbLogger : use random name instead of project as default name

[ierezell]

Proposed refactor

Use the wandb behavior (creating a random name) if no name is passed for the run or just let it None and wandb will deal with creating the name.

Motivation

I got 20 runs with all the same name which is not easy to navigate through

Pitch

Use a random name or name with a date or anything that would distinguish two runs.

Additional context

It's here: https://github.com/Lightning-AI/lightning/blob/9d02ad761c6c9360cba96e339d340c0cc2e572a8/src/pytorch_lightning/loggers/wandb.py#L300

cc @tchaton @rohitgr7 @awaelchli @morganmcg1 @borisdayma @scottire @manangoel99

[tshu-w]

Hi, this modification was introduced in this #12604 (comment), and I have also discussed the modification aspect of combining with third-party loggers at #12028 (comment), and currently PL logger is not perfect.

[kabouzeid]

Something must've changed in 1.7.0 though. The names of the runs in the wandb dashboard are now no longer randomly generated.

[kotchin]

This small change has been deteriorating a simple unique identification method provided by WandB which helps distinguishing multiple runs. I understand there are bigger changes in which this fits, but I do wonder if this specific change is required? It's particularly helpful to get an automatic name generated.

[hogru]

Can confirm that after updating to PL 1.7 the former random wandb run names (e.g., “noble-snowball-37”) all have the same (project) name. Not useful (for me). Since 1.7 introduced other bugs/changes with wandb, I am back to 1.6.5

[manangoel99]

Hi! Engineer from W&B here. In PL 1.7 this change of using the project as a default run name instead of random wandb run names (I love those too) was added because using the default None value lead to a broken directiry structure for PL Logs. More discussion here.
This is because of the interface between the PL logger and WandbLogger requires some adjustment which we are looking into it.
For now, there is a workaround which is call wandb.init before your initialization of the WandbLogger. This will initialize a run with a random name and the WandbLogger will use this run automatically. For e.g.

import wandb
from pytorch_lightning.loggers import WandbLogger

wandb.init(project='my-project')
logger = WandbLogger(project='my-project')

I hope this helps.

[hogru]

Thank you, @manangoel99, that's good to know. For now, I stay with 1.6.5 mainly for this reason: #14021 but once it's resolved, and I adapt my code to work with some other minor changes this workaround will be useful (I had it this way anyway until recently, but I think wandb (?) issues a warning that a run is already existing. Does not hurt, just wanted to cleanse the console output.

[awaelchli]

We can try to revert it but keep in mind that there are efforts in standardizing things across loggers which may contradict certain logger specific usage patterns, so this back and forth of naming conventions will (I predict) always happen as people disagree in how folders and experiments should be named.

For example, in wandb, there is a difference between a version and a run name, whereas with the tensorboard logger the version is sort of the same concept as the run name.

The best we can do is make things configurable and hope we choose good defaults. Note that this is all not at all trivial. If we did this

wandb.init(project='my-project')
logger = WandbLogger(project='my-project')

directly inside the initialization of your wandb logger, it would create an experiment with a random name (great!). But you will see N experiments created when running DDP, which is not at all desired.

I hope this adds some context, and I will see what can be done.

[ierezell]

Hello @awaelchli,

For my use case, your solution (wandb.init first) is perfectly fine and it's easy enough so everyone can do it and it let you focus on bigger problems.
So we can close the issue if you want.

For more complex problems, (like DDP) I guess the user will have to make more small hacks (like only logging on rank 0 )) but it seems reasonable not to break the whole "PR loggers API".

Another solution is to use Faker with something like

from faker import Faker
fake = Faker()
name = fake.name().split()[0] + "_" + fake.color_name()

Which will give a name and a color (like John_DarkBlue) which mimics the wandb behavior.

It's three lines but is still usable by many and is not too much overhead.

Hope it helps someone.
Have a great day

[hogru]

Thank you, @ierezell, for mentioning Faker, I did not know that. I think, I (we) cannot expect that a single API (in this case pytorch lightning) can cover all features/cases of all loggers it wants to integrate with. Neither are the features identical, nor the semantics/conventions/assumptions.

But… it would be nice to have transparency/documentation about what works with what in which ways. The combination of frameworks is (for me at least) often the big issue (in this case, on the API level, sometimes on more abstract levels). The websites/documentation of those frameworks sometimes suggest “with only 3 lines of code you can integrate x and y”. And yes, that's true, and I genuinely understand this from a marketing perspective in a changing and growing field, but if you really want to dig deeper, it's sometimes not as easy. Ok, enough whining, I guess it's a plea for some fine-print in the documentation / release notes about the conditions that apply when you intend to integrate two frameworks/packages/utils.