Using log_gpu_memory with MLFLow logger causes an exception.

[dscarmo]

🐛 Bug

Using log_gpu_memory with MLFLow logger causes an error. It appears the name of the metric is not supported by MLFLow.

MlflowException: Invalid metric name: 'gpu_id: 0/memory.used (MB)'. Names may only contain alphanumerics, underscores (_), dashes (-), periods (.), spaces ( ), and slashes (/).

To Reproduce

I reproduced the bug with the BoringModel, in the link bellow:
https://colab.research.google.com/drive/1P8uhSfjvYhKPMyRZH-QmfbOUOfnePy6G?usp=sharing

Expected behavior

log_gpu_memory should log gpu memory correctly when using an MLFlow logger.

Environment

Colab environment:

• CUDA:
  • GPU:
    • Tesla T4
  • available: True
  • version: 10.1
• Packages:
  • numpy: 1.18.5
  • pyTorch_debug: False
  • pyTorch_version: 1.6.0+cu101
  • pytorch-lightning: 1.0.3
  • tqdm: 4.41.1
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.6.9
  • version: Proposal for help #1 SMP Thu Jul 23 08:00:38 PDT 2020

[dscarmo]

This could be fixed removing the parenthesis from the string (as in the linked pr), but requires discussion if you guys want to change this for all loggers.

[awaelchli]

would [MB] work?

[awaelchli]

or should MLFlowLogger sanitize keys automatically by removing the unsupported characters?

[dscarmo]

Seems like MLFlow only wants: "[...] Names may only contain alphanumerics, underscores (_), dashes (-), periods (.), spaces ( ), and slashes (/)".

Maybe make the GPU memory key different only if MLFlow is being used? Sanitizing mlflow metrics with a warning might also be an option.

[awaelchli]

I see sanitization directly in the MLFlowLogger as a better long term solution. We already do sanitization for other loggers (for hyperparameters).

[dscarmo]

I changed the PR to remove parenthesis in the mlflow logger, however i don't know if it makes sense to give a warning, since the metric name for log_gpu_memory is not defined by the user. Should the removing of parenthesis be silent?

[rohitgr7]

Maybe do a PR on mlflow github too to support these or do sanitization process 😂