SWA steps with lr scheduler step interval #6571
Description
🐛 Bug
I'm currently working on a custom implementation of SWA with cyclic learning rates and was wondering about the replacement procedure in the current implementation.
If an lr scheduler is already configured, only the scheduler key is exchanged for the SWALR scheduler which expects to be stepped at the end of the epoch. I was wondering if this is correct if the original lr scheduler was configured with interval=step ?
Please reproduce using the BoringModel
To Reproduce
Use following BoringModel and post here
Expected behavior
The SWALR scheduler is only stepped at the end of the epoch and the configuration is exchanged too.
_scheduler_config = _get_default_scheduler_config()
assert _scheduler_config["interval"] == "epoch"
_scheduler_config["scheduler"] = self._swa_scheduler
if trainer.lr_schedulers:
lr_scheduler = trainer.lr_schedulers[0]["scheduler"]
rank_zero_warn(f"Swapping lr_scheduler {lr_scheduler} for {self._swa_scheduler}")
trainer.lr_schedulers[0] = _scheduler_config
else:
trainer.lr_schedulers.append(_scheduler_config)
Environment
Note: Bugs with code are solved faster ! Colab Notebook should be made public !
-
IDE: Please, use our python bug_report_model.py template. -
Colab Notebook: Please copy and paste the output from our environment collection script (or fill out the checklist below manually).
You can get the script and run it with:
wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env_details.py before running it.
python collect_env_details.py
- PyTorch Version (e.g., 1.0):
- OS (e.g., Linux):
- How you installed PyTorch (
conda,pip, source): - Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version:
- GPU models and configuration:
- Any other relevant information: