Merge branch 'master' into add-checkpoint-to-callback-on_save_checkpoint

Author: carmocca

.github/CONTRIBUTING.md

@@ -237,7 +237,7 @@ We welcome any useful contribution! For your convenience here's a recommended wo
 
 #### How can I help/contribute?
 
-All types of contributions are welcome - reporting bugs, fixing documentation, adding test cases, solving issues, and preparing bug fixes. 
+All types of contributions are welcome - reporting bugs, fixing documentation, adding test cases, solving issues, and preparing bug fixes.
 To get started with code contributions, look for issues marked with the label [good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) or chose something close to your domain with the label [help wanted](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22). Before coding, make sure that the issue description is clear and comment on the issue so that we can assign it to you (or simply self-assign if you can).
 
 #### Is there a recommendation for branch names?
@@ -323,14 +323,6 @@ run our/your test with
 python -m pytest tests/..../...py::test_explain_what_is_being_tested --verbose --capture=no
 ```
 
-#### How to contribute bugfixes/features?
-
-Currently we have separate streams/branches for bugfixes/features and release from the default branch (`master`).
-Bugfixes should land in this `master` branch and features should land in `release/X.y-dev`.
-This means that when starting your contribution and creating a branch according to question 2) you should start this new branch from master or future release dev branch.
-Later in PR creation also pay attention to properly set the target branch, usually the starting (base) and target branch are the same.
-
-_Note, that this flow may change after the 1.2 release as we will adjust releasing strategy._
 
 #### How to fix PR with mixed base and target branches?
 
@@ -339,7 +331,7 @@ Do not panic, the solution is very straightforward and quite simple.
 All you need to do are these two steps in arbitrary order:
    - Ask someone from Core to change the base/target branch to the correct one
    - Rebase or cherry-pick your commits onto the correct base branch...
-     
+
 Let's show how to deal with the git...
 the sample case is moving a PR from `master` to `release/1.2-dev` assuming my branch name is `my-branch`
 and the last true master commit is `ccc111` and your first commit is `mmm222`.
@@ -354,7 +346,7 @@ and the last true master commit is `ccc111` and your first commit is `mmm222`.
      #  so open one and cherry-pick your last commits from `my-branch-backup`
      #  resolve all eventual conflict as the new base may contain different code
      # when all done, push back to the open PR
-     git push -f 
+     git push -f
      ```
    * **Rebasing way**, see more about [rebase onto usage](https://womanonrails.com/git-rebase-onto)
      ```bash

CHANGELOG.md

@@ -9,6 +9,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- Added a way to print to terminal without breaking up the progress bar ([#5470](https://github.com/PyTorchLightning/pytorch-lightning/pull/5470))
+
 
 - Added `checkpoint` parameter to callback's `on_save_checkpoint` hook ([#6072](https://github.com/PyTorchLightning/pytorch-lightning/pull/6072))
 
@@ -21,15 +23,51 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Removed
 
+- Removed support for passing a bool value to `profiler` argument of Trainer ([#6164](https://github.com/PyTorchLightning/pytorch-lightning/pull/6164))
+
+
+- Removed deprecated Trainer argument `enable_pl_optimizer` and `automatic_optimization` ([#6163](https://github.com/PyTorchLightning/pytorch-lightning/pull/6163))
+
+
+- Removed deprecated metrics ([#6161](https://github.com/PyTorchLightning/pytorch-lightning/pull/6161))
+    * from `pytorch_lightning.metrics.functional.classification` removed `to_onehot`, `to_categorical`, `get_num_classes`, `roc`, `multiclass_roc`, `average_precision`, `precision_recall_curve`, `multiclass_precision_recall_curve`
+    * from `pytorch_lightning.metrics.functional.reduction` removed `reduce`, `class_reduce`	
+
+
+- Removed deprecated `ModelCheckpoint` arguments `prefix`, `mode="auto"` ([#6162](https://github.com/PyTorchLightning/pytorch-lightning/pull/6162))
+
+
+### Fixed
+
+- Made the `Plugin.reduce` method more consistent across all Plugins to reflect a mean-reduction by default ([#6011](https://github.com/PyTorchLightning/pytorch-lightning/pull/6011))
+
+
+- Move lightning module to correct device type when using LightningDistributedWrapper ([#6070](https://github.com/PyTorchLightning/pytorch-lightning/pull/6070))
+
+
+- Do not print top-k verbose log with `ModelCheckpoint(monitor=None)` ([#6109](https://github.com/PyTorchLightning/pytorch-lightning/pull/6109))
+
+
+- Expose DeepSpeed loss parameters to allow users to fix loss instability ([#6115](https://github.com/PyTorchLightning/pytorch-lightning/pull/6115))
+
+
+- Fixed epoch level schedulers not being called when `val_check_interval < 1.0` ([#6075](https://github.com/PyTorchLightning/pytorch-lightning/pull/6075))
+
+
+## [1.2.1] - 2021-02-23
 
 ### Fixed
 
+- Fixed incorrect yield logic for the amp autocast context manager ([#6080](https://github.com/PyTorchLightning/pytorch-lightning/pull/6080))
+- Fixed priority of plugin/accelerator when setting distributed mode ([#6089](https://github.com/PyTorchLightning/pytorch-lightning/pull/6089))
+- Fixed error message for AMP + CPU incompatibility ([#6107](https://github.com/PyTorchLightning/pytorch-lightning/pull/6107))
+
 
 ## [1.2.0] - 2021-02-18
 
 ### Added
 
-- Added `DataType`, `AverageMethod` and `MDMCAverageMethod` enum in metrics ([#5657](https://github.com/PyTorchLightning/pytorch-lightning/pull/5689)
+- Added `DataType`, `AverageMethod` and `MDMCAverageMethod` enum in metrics ([#5657](https://github.com/PyTorchLightning/pytorch-lightning/pull/5689))
 - Added support for summarized model total params size in megabytes ([#5590](https://github.com/PyTorchLightning/pytorch-lightning/pull/5590))
 - Added support for multiple train loaders ([#1959](https://github.com/PyTorchLightning/pytorch-lightning/pull/1959))
 - Added `Accuracy` metric now generalizes to Top-k accuracy for (multi-dimensional) multi-class inputs using the `top_k` parameter ([#4838](https://github.com/PyTorchLightning/pytorch-lightning/pull/4838))

azure-pipelines.yml

@@ -66,8 +66,9 @@ jobs:
         pip list
       displayName: 'Install dependencies'
 
-    - script: |
+    - bash: |
         python tests/collect_env_details.py
+        python -c "import torch ; mgpu = torch.cuda.device_count() ; assert mgpu >= 2, f'GPU: {mgpu}'"
       displayName: 'Env details'
 
     - bash: |
@@ -76,7 +77,7 @@ jobs:
         ls -l legacy/checkpoints/
       displayName: 'Get legacy checkpoints'
 
-    - script: |
+    - bash: |
         python -m coverage run --source pytorch_lightning -m pytest pytorch_lightning tests -v --durations=50
       displayName: 'Testing: standard'
 
@@ -90,11 +91,11 @@ jobs:
         codecov --token=$(CODECOV_TOKEN) --flags=gpu,pytest --name="GPU-coverage" --env=linux,azure
       displayName: 'Statistics'
 
-    - script: |
+    - bash: |
         python -m pytest benchmarks pl_examples -v --maxfail=2 --durations=0
       displayName: 'Testing: extended'
 
-    - script: |
+    - bash: |
         python setup.py install --user --quiet
         bash pl_examples/run_ddp-example.sh
         pip uninstall -y pytorch-lightning

docs/source/advanced/multi_gpu.rst

@@ -690,9 +690,9 @@ DeepSpeed
 .. note::
     The DeepSpeed plugin is in beta and the API is subject to change. Please create an `issue <https://github.com/PyTorchLightning/pytorch-lightning/issues>`_ if you run into any issues.
 
-`DeepSpeed <https://github.com/microsoft/DeepSpeed>`_ offers additional CUDA deep learning training optimizations, similar to `FairScale <https://github.com/facebookresearch/fairscale>`_. DeepSpeed offers lower level training optimizations, and useful efficient optimizers such as `1-bit Adam <https://www.deepspeed.ai/tutorials/onebit-adam/>`_.
-Using the plugin, we were able to **train model sizes of 10 Billion parameters and above**, with a lot of useful information in this `benchmark <https://github.com/huggingface/transformers/issues/9996>`_ and the DeepSpeed `docs <https://www.deepspeed.ai/tutorials/megatron/>`_.
-We recommend using DeepSpeed in environments where speed and memory optimizations are important (such as training large billion parameter models). In addition, we recommend trying :ref:`sharded` first before trying DeepSpeed's further optimizations, primarily due to FairScale Sharded ease of use in scenarios such as multiple optimizers/schedulers.
+`DeepSpeed <https://github.com/microsoft/DeepSpeed>`_ is a deep learning training optimization library, providing the means to train massive billion parameter models at scale.
+Using the DeepSpeed plugin, we were able to **train model sizes of 10 Billion parameters and above**, with a lot of useful information in this `benchmark <https://github.com/huggingface/transformers/issues/9996>`_ and the DeepSpeed `docs <https://www.deepspeed.ai/tutorials/megatron/>`_.
+DeepSpeed also offers lower level training optimizations, and efficient optimizers such as `1-bit Adam <https://www.deepspeed.ai/tutorials/onebit-adam/>`_. We recommend using DeepSpeed in environments where speed and memory optimizations are important (such as training large billion parameter models).
 
 To use DeepSpeed, you first need to install DeepSpeed using the commands below.
 
@@ -706,7 +706,7 @@ Additionally if you run into any issues installing m4py, ensure you have openmpi
 .. note::
     Currently ``resume_from_checkpoint`` and manual optimization are not supported.
 
-    DeepSpeed only supports single optimizer, single scheduler.
+    DeepSpeed currently only supports single optimizer, single scheduler within the training loop.
 
 ZeRO-Offload
 """"""""""""

docs/source/common/hyperparameters.rst

@@ -167,9 +167,6 @@ improve readability and reproducibility.
             def train_dataloader(self):
                 return DataLoader(mnist_train, batch_size=self.hparams.batch_size)
 
-    .. warning:: Deprecated since v1.1.0. This method of assigning hyperparameters to the LightningModule
-        will no longer be supported from v1.3.0. Use the ``self.save_hyperparameters()`` method from above instead.
-
 
 4.  You can also save full objects such as `dict` or `Namespace` to the checkpoint.
 

docs/source/common/optimizers.rst

@@ -300,8 +300,6 @@ override the :meth:`optimizer_step` function.
 
 For example, here step optimizer A every 2 batches and optimizer B every 4 batches
 
-.. note:: When using Trainer(enable_pl_optimizer=True), there is no need to call `.zero_grad()`.
-
 .. testcode::
 
     def optimizer_zero_grad(self, current_epoch, batch_idx, optimizer, opt_idx):

docs/source/extensions/metrics.rst

@@ -23,7 +23,7 @@ provided input.
 .. warning::
     From v1.2 onward ``compute()`` will no longer automatically call ``reset()``,
     and it is up to the user to reset metrics between epochs, except in the case where the
-    metric is directly passed to ``LightningModule``s ``self.log``.
+    metric is directly passed to ``LightningModule``'s ``self.log``.
 
 These metrics work with DDP in PyTorch and PyTorch Lightning by default. When ``.compute()`` is called in
 distributed mode, the internal state of each metric is synced and reduced across each process, so that the
@@ -478,7 +478,7 @@ binary/multi-label inputs as 2-class (multi-dimensional) multi-class inputs.
 
 For these cases, the metrics where this distinction would make a difference, expose the
 ``is_multiclass`` argument. Let's see how this is used on the example of 
-:class:`~pytorch_lightning.metrics.classification.StatScores` metric.
+:class:`~pytorch_lightning.metrics.StatScores` metric.
 
 First, let's consider the case with label predictions with 2 classes, which we want to
 treat as binary.
@@ -530,86 +530,86 @@ Class Metrics (Classification)
 Accuracy
 ~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.Accuracy
+.. autoclass:: pytorch_lightning.metrics.Accuracy
     :noindex:
 
 AveragePrecision
 ~~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.AveragePrecision
+.. autoclass:: pytorch_lightning.metrics.AveragePrecision
     :noindex:
 
 AUC
 ~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.AUC
+.. autoclass:: pytorch_lightning.metrics.AUC
     :noindex:
 
 AUROC
 ~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.AUROC
+.. autoclass:: pytorch_lightning.metrics.AUROC
     :noindex:
 
 ConfusionMatrix
 ~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.ConfusionMatrix
+.. autoclass:: pytorch_lightning.metrics.ConfusionMatrix
     :noindex:
 
 F1
 ~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.F1
+.. autoclass:: pytorch_lightning.metrics.F1
     :noindex:
 
 FBeta
 ~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.FBeta
+.. autoclass:: pytorch_lightning.metrics.FBeta
     :noindex:
 
 IoU
 ~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.IoU
+.. autoclass:: pytorch_lightning.metrics.IoU
     :noindex:
 
 Hamming Distance
 ~~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.HammingDistance
+.. autoclass:: pytorch_lightning.metrics.HammingDistance
     :noindex:
 
 Precision
 ~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.Precision
+.. autoclass:: pytorch_lightning.metrics.Precision
     :noindex:
 
 PrecisionRecallCurve
 ~~~~~~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.PrecisionRecallCurve
+.. autoclass:: pytorch_lightning.metrics.PrecisionRecallCurve
     :noindex:
 
 Recall
 ~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.Recall
+.. autoclass:: pytorch_lightning.metrics.Recall
     :noindex:
 
 ROC
 ~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.ROC
+.. autoclass:: pytorch_lightning.metrics.ROC
     :noindex:
 
 
 StatScores
 ~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.classification.StatScores
+.. autoclass:: pytorch_lightning.metrics.StatScores
     :noindex:
 
 
@@ -654,7 +654,7 @@ confusion_matrix [func]
 dice_score [func]
 ~~~~~~~~~~~~~~~~~
 
-.. autofunction:: pytorch_lightning.metrics.functional.classification.dice_score
+.. autofunction:: pytorch_lightning.metrics.functional.dice_score
     :noindex:
 
 
@@ -735,7 +735,7 @@ stat_scores [func]
 stat_scores_multiple_classes [func]
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. autofunction:: pytorch_lightning.metrics.functional.classification.stat_scores_multiple_classes
+.. autofunction:: pytorch_lightning.metrics.functional.stat_scores_multiple_classes
     :noindex:
 
 
@@ -762,49 +762,49 @@ Class Metrics (Regression)
 ExplainedVariance
 ~~~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.regression.ExplainedVariance
+.. autoclass:: pytorch_lightning.metrics.ExplainedVariance
     :noindex:
 
 
 MeanAbsoluteError
 ~~~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.regression.MeanAbsoluteError
+.. autoclass:: pytorch_lightning.metrics.MeanAbsoluteError
     :noindex:
 
 
 MeanSquaredError
 ~~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.regression.MeanSquaredError
+.. autoclass:: pytorch_lightning.metrics.MeanSquaredError
     :noindex:
 
 
 MeanSquaredLogError
 ~~~~~~~~~~~~~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.regression.MeanSquaredLogError
+.. autoclass:: pytorch_lightning.metrics.MeanSquaredLogError
     :noindex:
 
 
 PSNR
 ~~~~
 
-.. autoclass:: pytorch_lightning.metrics.regression.PSNR
+.. autoclass:: pytorch_lightning.metrics.PSNR
     :noindex:
 
 
 SSIM
 ~~~~
 
-.. autoclass:: pytorch_lightning.metrics.regression.SSIM
+.. autoclass:: pytorch_lightning.metrics.SSIM
     :noindex:
 
 
 R2Score
 ~~~~~~~
 
-.. autoclass:: pytorch_lightning.metrics.regression.R2Score
+.. autoclass:: pytorch_lightning.metrics.R2Score
     :noindex:
 
 Functional Metrics (Regression)
@@ -873,7 +873,7 @@ NLP
 bleu_score [func]
 -----------------
 
-.. autofunction:: pytorch_lightning.metrics.functional.nlp.bleu_score
+.. autofunction:: pytorch_lightning.metrics.functional.bleu_score
     :noindex:
 
 ********
@@ -883,5 +883,5 @@ Pairwise
 embedding_similarity [func]
 ---------------------------
 
-.. autofunction:: pytorch_lightning.metrics.functional.self_supervised.embedding_similarity
+.. autofunction:: pytorch_lightning.metrics.functional.embedding_similarity
     :noindex:

docs/source/starter/new-project.rst

@@ -737,7 +737,7 @@ Lightning has many tools for debugging. Here is an example of just a few of them
 .. testcode::
 
     # Profile your code to find speed/memory bottlenecks
-    Trainer(profiler=True)
+    Trainer(profiler="simple")
 
 ---------------