[WIP] Rename overfit_pct to overfit_batches (and fix) and val_percent_check and test_percent_check (and fix) (#2213)

* fixed percent check for val/test

* fixed percent check for val/test

* fixed percent check for val/test

* fixed percent check for val/test

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* overfit_pct now uses train loaders for val and test and does not shuffle

* add on fit_start on fit_end hooks

* add on fit_start on fit_end hooks

* add on fit_start on fit_end hooks

Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: Rohit Gupta <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>

Author: 4 people

CHANGELOG.md

@@ -21,6 +21,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- Added overfit_batches, limit_xxx_batches flags (overfit now uses training set for all three) ([#2213](https://github.com/PyTorchLightning/pytorch-lightning/pull/2213))
+- Added metric Base classes ([#1326](https://github.com/PyTorchLightning/pytorch-lightning/pull/1326), [#1877](https://github.com/PyTorchLightning/pytorch-lightning/pull/1877))
+- Added Sklearn metrics classes ([#1327](https://github.com/PyTorchLightning/pytorch-lightning/pull/1327))
+- Added Native torch metrics ([#1488](https://github.com/PyTorchLightning/pytorch-lightning/pull/1488))
 - Added metrics 
   * Base classes ([#1326](https://github.com/PyTorchLightning/pytorch-lightning/pull/1326), [#1877](https://github.com/PyTorchLightning/pytorch-lightning/pull/1877))
   * Sklearn metrics classes ([#1327](https://github.com/PyTorchLightning/pytorch-lightning/pull/1327))
@@ -54,6 +58,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Deprecated
 
+- Deprecated `overfit_pct`, `val_percent_check`, `test_percent_check` ([#2213](https://github.com/PyTorchLightning/pytorch-lightning/pull/2213))
 - Deprecated `ModelCheckpoint`'s attributes `best` and `kth_best_model` ([#1799](https://github.com/PyTorchLightning/pytorch-lightning/pull/1799))
 - Dropped official support/testing for older PyTorch versions <1.3 ([#1917](https://github.com/PyTorchLightning/pytorch-lightning/pull/1917))
 

docs/source/debugging.rst

@@ -48,12 +48,19 @@ Make model overfit on subset of data
 A good debugging technique is to take a tiny portion of your data (say 2 samples per class),
 and try to get your model to overfit. If it can't, it's a sign it won't work with large datasets.
 
-(See: :paramref:`~pytorch_lightning.trainer.trainer.Trainer.overfit_pct`
+(See: :paramref:`~pytorch_lightning.trainer.trainer.Trainer.overfit_batches`
 argument of :class:`~pytorch_lightning.trainer.trainer.Trainer`)
 
 .. testcode::
 
-    trainer = Trainer(overfit_pct=0.01)
+    # use only 1% of training data (and use the same training Dataloader (with shuffle off) in val and test)
+    trainer = Trainer(overfit_batches=0.01)
+
+    # or overfit a number of batches
+    trainer = Trainer(overfit_batches=0.01)
+
+With this flag, the train, val, and test sets will all be the same train set. We will also replace the sampler
+in the training set to turn off shuffle for you.
 
 Print a summary of your LightningModule
 ---------------------------------------

docs/source/fast_training.rst

@@ -56,17 +56,17 @@ If you don't want to check 100% of the training/validation/test set (for debuggi
     # DEFAULT
     trainer = Trainer(
         train_percent_check=1.0,
-        val_percent_check=1.0,
-        test_percent_check=1.0
+        limit_val_batches=1.0,
+        limit_test_batches=1.0
     )
 
     # check 10%, 20%, 30% only, respectively for training, validation and test set
     trainer = Trainer(
         train_percent_check=0.1,
-        val_percent_check=0.2,
-        test_percent_check=0.3
+        limit_val_batches=0.2,
+        limit_test_batches=0.3
     )
 
-.. note:: ``train_percent_check``, ``val_percent_check`` and ``test_percent_check`` will be overwritten by ``overfit_pct`` if ``overfit_pct`` > 0. ``val_percent_check`` will be ignored if ``fast_dev_run=True``.
+.. note:: ``train_percent_check``, ``limit_val_batches`` and ``limit_test_batches`` will be overwritten by ``overfit_batches`` if ``overfit_batches`` > 0. ``limit_val_batches`` will be ignored if ``fast_dev_run=True``.
 
-.. note:: If you set ``val_percent_check=0``, validation will be disabled.
+.. note:: If you set ``limit_val_batches=0``, validation will be disabled.

pytorch_lightning/callbacks/progress.py

@@ -98,6 +98,7 @@ def total_val_batches(self) -> int:
         elif not self.trainer.disable_validation:
             is_val_epoch = trainer.current_epoch % trainer.check_val_every_n_epoch == 0
             total_val_batches = trainer.num_val_batches if is_val_epoch else 0
+            total_val_batches = sum(total_val_batches)
         return total_val_batches
 
     @property
@@ -111,6 +112,7 @@ def total_test_batches(self) -> int:
             total_test_batches = len(self.trainer.test_dataloaders)
         else:
             total_test_batches = self.trainer.num_test_batches
+            total_test_batches = sum(total_test_batches)
         return total_test_batches
 
     def disable(self):

pytorch_lightning/trainer/__init__.py

@@ -433,6 +433,40 @@ def on_train_end(self, trainer, pl_module):
     # default used by the Trainer
     trainer = Trainer(gradient_clip_val=0.0)
 
+
+limit_test_batches
+^^^^^^^^^^^^^^^^^^
+
+How much of test dataset to check.
+
+Example::
+
+    # default used by the Trainer
+    trainer = Trainer(limit_test_batches=1.0)
+
+    # run through only 25% of the test set each epoch
+    trainer = Trainer(limit_test_batches=0.25)
+
+    # run for only 10 batches
+    trainer = Trainer(limit_test_batches=10)
+
+limit_val_batches
+^^^^^^^^^^^^^^^^^
+
+How much of validation dataset to check.
+Useful when debugging or testing something that happens at the end of an epoch.
+
+Example::
+
+    # default used by the Trainer
+    trainer = Trainer(limit_val_batches=1.0)
+
+    # run through only 25% of the validation set each epoch
+    trainer = Trainer(limit_val_batches=0.25)
+
+        # run for only 10 batches
+    trainer = Trainer(limit_val_batches=10)
+
 log_gpu_memory
 ^^^^^^^^^^^^^^
 Options:
@@ -652,29 +686,28 @@ def on_train_end(self, trainer, pl_module):
 
 overfit_pct
 ^^^^^^^^^^^
-Uses this much data of all datasets (training, validation, test).
+
+.. warning:: .. deprecated:: 0.8.0.
+
+    Use `overfit_batches`. Will remove 1.0.0.
+
+overfit_batches
+^^^^^^^^^^^^^^^
+Uses this much data of the training set. If will use the same training set for validation and testing.
+If the training Dataloaders(shuffle=True), Lightning will automatically disable it.
+
 Useful for quickly debugging or trying to overfit on purpose.
 
 Example::
 
     # default used by the Trainer
-    trainer = Trainer(overfit_pct=0.0)
-
-    # use only 1% of the train, test, val datasets
-    trainer = Trainer(overfit_pct=0.01)
+    trainer = Trainer(overfit_batches=0.0)
 
-    # equivalent:
-    trainer = Trainer(
-        train_percent_check=0.01,
-        val_percent_check=0.01,
-        test_percent_check=0.01
-    )
-
-See Also:
-    - `train_percent_check`_
-    - `val_percent_check`_
-    - `test_percent_check`_
+    # use only 1% of the train set (and use the train set for val and test)
+    trainer = Trainer(overfit_batches=0.01)
 
+    # overfit on 10 of the same batches
+    trainer = Trainer(overfit_batches=10)
 
 precision
 ^^^^^^^^^
@@ -829,39 +862,7 @@ def on_train_end(self, trainer, pl_module):
 test_percent_check
 ^^^^^^^^^^^^^^^^^^
 
-How much of test dataset to check.
-
-Example::
-
-    # default used by the Trainer
-    trainer = Trainer(test_percent_check=1.0)
-
-    # run through only 25% of the test set each epoch
-    trainer = Trainer(test_percent_check=0.25)
-
-val_check_interval
-^^^^^^^^^^^^^^^^^^
-
-How often within one training epoch to check the validation set.
-Can specify as float or int.
-
-- use (float) to check within a training epoch
-- use (int) to check every n steps (batches)
-
-.. code-block:: python
-
-    # default used by the Trainer
-    trainer = Trainer(val_check_interval=1.0)
-
-Example::
-
-    # check validation set 4 times during a training epoch
-    trainer = Trainer(val_check_interval=0.25)
-
-    # check validation set every 1000 training batches
-    # use this when using iterableDataset and your dataset has no length
-    # (ie: production cases with streaming data)
-    trainer = Trainer(val_check_interval=1000)
+.. warning:: deprecated in v0.8.0 please use `limit_test_batches`. Will remove in 1.0.0
 
 track_grad_norm
 ^^^^^^^^^^^^^^^
@@ -955,20 +956,36 @@ def tbptt_split_batch(self, batch, split_size):
                 # do your own splitting on the batch
                 return splits
 
+val_check_interval
+^^^^^^^^^^^^^^^^^^
 
-val_percent_check
-^^^^^^^^^^^^^^^^^
+How often within one training epoch to check the validation set.
+Can specify as float or int.
 
-How much of validation dataset to check.
-Useful when debugging or testing something that happens at the end of an epoch.
+- use (float) to check within a training epoch
+- use (int) to check every n steps (batches)
 
-Example::
+.. code-block:: python
 
     # default used by the Trainer
-    trainer = Trainer(val_percent_check=1.0)
+    trainer = Trainer(val_check_interval=1.0)
+
+Example::
+
+    # check validation set 4 times during a training epoch
+    trainer = Trainer(val_check_interval=0.25)
+
+    # check validation set every 1000 training batches
+    # use this when using iterableDataset and your dataset has no length
+    # (ie: production cases with streaming data)
+    trainer = Trainer(val_check_interval=1000)
+
+
+val_percent_check
+^^^^^^^^^^^^^^^^^
+
+.. warning:: deprecated in v0.8.0 please use `limit_val_batches`. Will remove in 1.0.0
 
-    # run through only 25% of the validation set each epoch
-    trainer = Trainer(val_percent_check=0.25)
 
 weights_save_path
 ^^^^^^^^^^^^^^^^^