Skip to content

Fix cosine LR scheduler for warmup #2312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Fix cosine LR scheduler for warmup #2312

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9a9633e
Fix cosine LR scheduler for warmup
sinahmr Oct 24, 2024
772e3b7
Fix cosine LR scheduler for warmup and cycles
sinahmr Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions timm/scheduler/cosine_lr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,25 +82,25 @@ def _get_lr(self, t: int) -> List[float]:
if t < self.warmup_t:
lrs = [self.warmup_lr_init + t * s for s in self.warmup_steps]
else:
if self.warmup_prefix:
t = t - self.warmup_t
t_i = self.t_initial
if self.warmup_prefix and t < self.t_initial:
t -= self.warmup_t
t_i -= self.warmup_t

if self.cycle_mul != 1:
i = math.floor(math.log(1 - t / self.t_initial * (1 - self.cycle_mul), self.cycle_mul))
t_i = self.cycle_mul ** i * self.t_initial
t_curr = t - (1 - self.cycle_mul ** i) / (1 - self.cycle_mul) * self.t_initial
t_curr = t - (1 - self.cycle_mul ** i) / (1 - self.cycle_mul) * t_i
else:
i = t // self.t_initial
t_i = self.t_initial
t_curr = t - (self.t_initial * i)
t_curr = t - (t_i * i)

gamma = self.cycle_decay ** i
lr_max_values = [v * gamma for v in self.base_values]
k = self.k_decay

if i < self.cycle_limit:
lrs = [
self.lr_min + 0.5 * (lr_max - self.lr_min) * (1 + math.cos(math.pi * t_curr ** k / t_i ** k))
self.lr_min + 0.5 * (lr_max - self.lr_min) * (1 + math.cos(math.pi * t_curr ** k / (t_i * self.cycle_mul ** i) ** k))
for lr_max in lr_max_values
]
else:
Expand Down