Skip to content

fix logged keys in mlflow logger #4412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Nov 10, 2020
Merged

fix logged keys in mlflow logger #4412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e6c1f05
[#4411] fix gpu_log_memory with mlflow logger
dscarmo Oct 28, 2020
83af360
sanitize parenthesis instead of removing for all loggers
dscarmo Oct 28, 2020
d397d3d
apply regex for mlflow key sanitization
dscarmo Oct 28, 2020
c657486
replace ',' with '.' typo
dscarmo Oct 28, 2020
57d5028
Merge branch 'master' into fix_log_gpu_memory_mlflow
rohitgr7 Oct 28, 2020
cedfbe8
add single warning and test
dscarmo Nov 10, 2020
34030c4
Merge branch 'master' into fix_log_gpu_memory_mlflow
dscarmo Nov 10, 2020
4fa1628
Merge branch 'master' into fix_log_gpu_memory_mlflow
tchaton Nov 10, 2020
e37f6d0
Merge branch 'master' into fix_log_gpu_memory_mlflow
tchaton Nov 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions pytorch_lightning/loggers/mlflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
MLflow
------
"""
import re
import warnings
from argparse import Namespace
from time import time
from typing import Any, Dict, Optional, Union
Expand Down Expand Up @@ -151,6 +153,13 @@ def log_metrics(self, metrics: Dict[str, float], step: Optional[int] = None) ->
if isinstance(v, str):
log.warning(f'Discarding metric with string value {k}={v}.')
continue

new_k = re.sub("[^a-zA-Z0-9_/. -]+", "", k)
if k != new_k:
warnings.warn(("MLFlow only allows '_', '/', '.' and ' ' special characters in metric name.\n",
f"Replacing {k} with {new_k}."))
k = new_k

self.experiment.log_metric(self.run_id, k, v, timestamp_ms, step)

@rank_zero_only
Expand Down
3 changes: 2 additions & 1 deletion tests/loggers/test_mlflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ def test_mlflow_logger_dirs_creation(tmpdir):
assert set(os.listdir(tmpdir / exp_id)) == {run_id, 'meta.yaml'}

model = EvalModelTemplate()
trainer = Trainer(default_root_dir=tmpdir, logger=logger, max_epochs=1, limit_val_batches=3)
trainer = Trainer(default_root_dir=tmpdir, logger=logger, max_epochs=1, limit_val_batches=3,
log_gpu_memory=True)
trainer.fit(model)
assert set(os.listdir(tmpdir / exp_id)) == {run_id, 'meta.yaml'}
assert 'epoch' in os.listdir(tmpdir / exp_id / run_id / 'metrics')
Expand Down