Skip to content

log_gpu_memory='min_max' leads to error in parsing metrics keys #9010

@cosw0t

Description

@cosw0t

🐛 Bug

log_gpu_memory='min_max' raises:

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py in _log_gpus_metrics(self)
    227     def _log_gpus_metrics(self):
    228         for key, mem in self.gpus_metrics.items():
--> 229             gpu_id = int(key.split("/")[0].split(":")[1])
    230             if gpu_id in self.trainer.accelerator_connector.parallel_device_ids:
    231                 self.trainer.lightning_module.log(key, mem, prog_bar=False, logger=True, on_step=True, on_epoch=False)

To Reproduce

https://colab.research.google.com/drive/13Q_YzybNXDRL7gBZsluIRqfBIq21G3Et?usp=sharing

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedOpen to be worked on

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions