Skip to content

[1/2] Add support for early stopping during training epoch end #6944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Apr 28, 2021
Merged

[1/2] Add support for early stopping during training epoch end #6944

merged 18 commits into from
Apr 28, 2021

Conversation

@ananthsub
Copy link
Contributor

@ananthsub ananthsub commented Apr 10, 2021 •
edited
Loading

What does this PR do?

Fixes #7033
Part 1

In a follow up PR, we can remove https://github.com/PyTorchLightning/pytorch-lightning/blob/5bd3cd5f712b65d38812b27cf957261bb06b83c5/pytorch_lightning/trainer/training_loop.py#L152-L160

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

@codecov
Copy link

codecov bot commented Apr 10, 2021 •
edited
Loading

Codecov Report

Merging #6944 (3b6b606) into master (cbc6e30) will decrease coverage by 4%.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #6944    +/-   ##
=======================================
- Coverage      91%     87%    -4%     
=======================================
  Files         199     199            
  Lines       12783   12805    +22     
=======================================
- Hits        11659   11160   -499     
- Misses       1124    1645   +521

@ananthsub ananthsub mentioned this pull request Apr 10, 2021
Closed
ananthsub
ananthsub commented Apr 10, 2021
View reviewed changes
Copy link
Contributor Author

@ananthsub ananthsub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is blocked on #6921 as the hook ordering is off, meaning we can't access the trainer's logger metrics as expected. the callback for on_train_epoch_end runs before training_epoch_end in the lightning module

@ananthsub ananthsub force-pushed the feat-early-stop-train branch from 04ae832 to dc55fd5 Compare April 15, 2021 05:22
@pep8speaks
Copy link

pep8speaks commented Apr 15, 2021 •
edited
Loading

Hello @ananthsub! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-04-28 08:59:19 UTC

@ananthsub ananthsub mentioned this pull request Apr 15, 2021
Closed
@ananthsub ananthsub added this to the 1.3 milestone Apr 15, 2021
@ananthsub ananthsub marked this pull request as ready for review April 15, 2021 06:42
awaelchli
awaelchli reviewed Apr 16, 2021
View reviewed changes
Copy link
Contributor

@awaelchli awaelchli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a follow up PR, we can remove

Have you already tried it? Does it go without complications?

@ananthsub ananthsub mentioned this pull request Apr 17, 2021
Merged
11 tasks
@ananthsub
Copy link
Contributor Author

ananthsub commented Apr 17, 2021 •
edited
Loading

@awaelchli see #7069 for the followup.

@ananthsub ananthsub changed the title Add support for early stopping during training epoch end [1/2] Add support for early stopping during training epoch end Apr 17, 2021
carmocca
carmocca reviewed Apr 26, 2021
View reviewed changes
@mergify mergify bot removed the has conflicts label Apr 26, 2021
Borda
Borda reviewed Apr 26, 2021
View reviewed changes
awaelchli
awaelchli approved these changes Apr 27, 2021
View reviewed changes
tchaton
tchaton approved these changes Apr 27, 2021
View reviewed changes
Copy link
Contributor

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

tests/callbacks/test_early_stopping.py
def _epoch_end(self) -> None:
losses = [8, 4, 2, 3, 4, 5, 8, 10]
val_loss = losses[self.current_epoch]
self.log('abc', torch.tensor(val_loss))
Copy link
Contributor

@tchaton tchaton Apr 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we log with different keys on val/train for the test ?

Borda
Borda reviewed Apr 27, 2021
View reviewed changes
tests/callbacks/test_early_stopping.py Outdated
"callbacks, expected_stop_epoch, check_on_train_epoch_end, accelerator, num_processes",
[
([EarlyStopping(monitor='abc'), EarlyStopping(monitor='cba', patience=3)], 3, None, 1),
([EarlyStopping(monitor='abc'), EarlyStopping(monitor='cba', patience=3)], 3, False, None, 1),
Copy link
Collaborator

@Borda Borda Apr 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carmocca id we simplify each EarlyStop we can fit to single line :P

Copy link
Contributor

@carmocca carmocca Apr 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean?

@ananthsub ananthsub force-pushed the feat-early-stop-train branch from 1a48461 to a59c78a Compare April 28, 2021 08:06
@carmocca carmocca merged commit 947d1cb into Lightning-AI:master Apr 28, 2021
Borda
Borda reviewed Apr 28, 2021
View reviewed changes
tests/callbacks/test_early_stopping.py
Comment on lines +415 to +424
([EarlyStopping('abc'), EarlyStopping('cba', patience=3)], 3, False, None, 1),
([EarlyStopping('cba', patience=3), EarlyStopping('abc')], 3, False, None, 1),
pytest.param([EarlyStopping('abc'), EarlyStopping('cba', patience=3)], 3, False, 'ddp_cpu', 2, **_NO_WIN),
pytest.param([EarlyStopping('cba', patience=3), EarlyStopping('abc')], 3, False, 'ddp_cpu', 2, **_NO_WIN),
([EarlyStopping('abc', **_ES_CHECK), EarlyStopping('cba', **_ES_CHECK_P3)], 3, True, None, 1),
([EarlyStopping('cba', **_ES_CHECK_P3), EarlyStopping('abc', **_ES_CHECK)], 3, True, None, 1),
pytest.param([EarlyStopping('abc', **_ES_CHECK),
EarlyStopping('cba', **_ES_CHECK_P3)], 3, True, 'ddp_cpu', 2, **_NO_WIN),
pytest.param([EarlyStopping('cba', **_ES_CHECK_P3),
EarlyStopping('abc', **_ES_CHECK)], 3, True, 'ddp_cpu', 2, **_NO_WIN),
Copy link
Collaborator

@Borda Borda Apr 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean this @carmocca from #6944 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Borda Borda Borda left review comments

@tchaton tchaton tchaton approved these changes

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@kaushikb11 kaushikb11 Awaiting requested review from kaushikb11

@SeanNaren SeanNaren Awaiting requested review from SeanNaren

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

+2 more reviewers

@carmocca carmocca carmocca approved these changes

@awaelchli awaelchli awaelchli approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

callback refactor

Projects

None yet

Milestone

pl:1.3

Development

Successfully merging this pull request may close these issues.

Support early stopping during training inside of the early stopping callback

6 participants

@ananthsub @pep8speaks @awaelchli @Borda @carmocca @tchaton