Lr_finder based on the validation loss rather than training loss

[thisisyichenzhang]

What is your question?

For the default LR Range Test in PyTorch lightning, i.e., "lr_finder", is the reported loss curve based on training loss, test loss, or generalization loss? For me, it would be more reasonable to select the learning rate based on the test loss rather than training loss.

I noticed that there is a "val_dataloader" and "train_dataloader" argument in "lr_finder", but it is not clear what they are doing.

What's your environment?

• OS: iOS
• Packaging: pip
• Version: 0.7.6

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Currently the LR finder only has support for logging the train loss (as a function of the learning rate). The reason that the lr_find accepts a val_dataloader is due to compatibility with the rest of lightning.

I was planning on expanding the functionality of the lr finder to be used on the validation set. This will give better estimates of the learning rate to use, with the penalty of the search taking longer time.

[thisisyichenzhang]

@SkafteNicki Thank you for the information! Do you know how soon we may expect to see this new functionality? I plan to write my own implementation for my current project if it takes a while to happen. Otherwise, I would wait to see the update.

[SkafteNicki]

I really don't know. I will be part of a major refactoring of the two tuner algorithms we currently have in lightning (learning rate finder, batch size scaler), so I may take so time. But I am working on it ;)

[thisisyichenzhang]

Thank you!

[lgvaz]

Can I just ask why you think it's more reasonable to select the learning rate based on the test loss rather than training loss?

Making hyperameters decisions based on your test set means you're overfitting to that

[thisisyichenzhang]

Good question @lgvaz . Sorry, I was not very strict with my terminology. But by writing test loss, I meant the loss on the validation set.
Specifically, the data is split into training set (for training), validation set (for hyperparameter tuning), and test set( for testing the generalization performance). So picking hyperparameter on the validation set does not overfit the test set.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[SgtWhiskeyjack]

I really don't know. I will be part of a major refactoring of the two tuner algorithms we currently have in lightning (learning rate finder, batch size scaler), so I may take so time. But I am working on it ;)

@SkafteNicki Hi Nicki. I'm curious, has there been any progress on this? I'm very interested in using a LR finder based on the validation loss.

[SavvaVyatkin]

We ran into this problem at work. Here is the solution we ended up implementing: https://github.com/SavvaVyatkin/LRFinder/tree/main and the explanation for it: https://savvavy.wordpress.com/2024/07/20/pytorch-lightning-learning-rate-finder/
Feel free to use it.