Merge branch 'master' into rich/default

Author: kaushikb11

.azure-pipelines/gpu-tests.yml

@@ -43,6 +43,7 @@ jobs:
         lspci | egrep 'VGA|3D'
         whereis nvidia
         nvidia-smi
+        which python && which pip
         python --version
         pip --version
         pip list
@@ -51,7 +52,8 @@ jobs:
     - bash: |
         python -c "fname = 'requirements/extra.txt' ; lines = [line for line in open(fname).readlines() if 'horovod' not in line] ; open(fname, 'w').writelines(lines)"
         pip install fairscale==0.4.0
-        pip install deepspeed==0.5.4
+        pip install deepspeed==0.5.7
+        pip install bagua-cuda102==0.9.0
         pip install . --requirement requirements/devel.txt
         pip list
       displayName: 'Install dependencies'

.circleci/config.yml

@@ -37,10 +37,7 @@ references:
     run:
       name: Make Documentation
       command: |
-        # First run the same pipeline as Read-The-Docs
-        # apt-get update && apt-get install -y cmake
-        # using: https://hub.docker.com/r/readthedocs/build
-        # we need to use py3.7 ot higher becase of an issue with metaclass inheritence
+        # the image uses python 2.7 by default, force a different version
         pyenv global 3.7.3
         python --version
         pip install -r requirements/docs.txt

.github/BECOMING_A_CORE_CONTRIBUTOR.md

@@ -1,69 +1,65 @@
-# How to become a core contributor
+# How to Become a Core Contributor
 
 Thanks for your interest in joining the Lightning team! We’re a rapidly growing project which is poised to become the go-to framework for DL researchers!
-We're currently recruiting for a team of 5 core maintainers.
 
-As a core maintainer you will have a strong say in the direction of the project. Big changes will require a majority of maintainers to agree.
+As a core maintainer, you will have a strong say in the direction of the project. Big changes will require a majority of maintainers to agree.
+Our development is fully open, so you can still raise your voice just by commenting on issues and pull requests! Doing so is a big step in becoming part of core.
 
-## Code of conduct
+## Code of Conduct
 
-First and foremost, you'll be evaluated against [these core values](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md). Any code we commit or feature we add needs to align with those core values.
+First and foremost, you'll be evaluated against [these core values](CONTRIBUTING.md). Any code we commit or feature we add needs to align with those core values.
+Any collaboration and communication must adhere to our [code of conduct](CODE_OF_CONDUCT.md).
 
-## The bar for joining the team
+## The Bar for Joining the Team
 
 Lightning is being used to solve really hard problems at the top AI labs in the world. As such, the bar for adding team members is extremely high. Candidates must have solid engineering skills, have a good eye for user experience, and must be a power user of Lightning and PyTorch.
 
 With that said, the Lightning team will be diverse and a reflection of an inclusive AI community. You don't have to be an engineer to contribute! Scientists with great usability intuition and PyTorch ninja skills are welcomed!
 
 ## Responsibilities:
 
-The responsibilities mainly revolve around 3 things.
+Here, we describe general expectations from core contributors:
 
-### Github issues
+### Github Issues
 
-- Here we want to help users have an amazing experience. These range from questions from new people getting into DL to questions from researchers about doing something esoteric with Lightning
-  Often, these issues require some sort of bug fix, document clarification or new functionality to be scoped out.
+- Our community is the main motivation for our work. Help them have an amazing experience. Issues range from answering questions from new people getting into deep learning to helping researchers doing something esoteric.
+  They often require some sort of bug fix, document clarification, or new functionality to be scoped out. You can help them solve their issues and guide them to completion.
 
-- To become a core member you must resolve at least 10 Github issues which align with the API design goals for Lightning. By the end of these 10 issues I should feel comfortable in the way you answer user questions
-  Pleasant/helpful tone.
+- Weigh in on discussions in a timely fashion. Most importantly, on the RFCs (request for comments) that will shape the future of Lightning.
+  There are some big decisions which the project must make. For these, we expect core contributors to have something meaningful to add, especially if it’s their area of expertise.
 
-- Can abstract from that issue or bug into functionality that might solve other related issues or makes the platform more flexible.
+- Propose your own RFCs that align with the API design goals for Lightning.
+
+- Identify opportunities from an issue or bug that can solve other related issues or make the framework more flexible.
 
 - Don’t make users feel like they don’t know what they’re doing. We’re here to help and to make everyone’s experience delightful.
 
-### Pull requests
+- Help out with critical bugs. Nobody likes bugs so you'll be a hero if you fix them!
+
+### Pull Requests (PRs)
 
-- Here we need to ensure the code that enters Lightning is high quality. For each PR we need to:
-- Make sure code coverage does not decrease
-- Documents are updated
-- Code is elegant and simple
-- Code is NOT overly engineered or hard to read
-- Ask yourself, could a non-engineer understand what’s happening here?
-- Make sure new tests are written
-- Is this NECESSARY for Lightning? There are some PRs which are just purely about adding engineering complexity which have no place in Lightning.
-  Guidance
-- Some other PRs are for people who are wanting to get involved and add something unnecessary. We do want their help though! So don’t approve the PR, but direct them to a Github issue that they might be interested in helping with instead!
-- To be considered for core contributor, please review 10 PRs and help the authors land it on master. Once you've finished the review, ping me
-  for a sanity check. At the end of 10 PRs if your PR reviews are inline with expectations described above, then you can merge PRs on your own going forward,
-  otherwise we'll do a few more until we're both comfortable :)
+- Pull requests are the evolutionary mechanism of Lightning, so quality is extremely important. Make sure contributors adhere to the guidelines described in the [contributing section](CONTRIBUTING.md#Pull-request).
 
-### Project directions
+- Some PRs are from people who want to get involved and try to add something unnecessary. We do want their help though! So don’t approve the PR, but direct them to a Github issue that they might be interested in helping with instead!
 
-There are some big decisions which the project must make. For these I expect core contributors to have something meaningful to add if it’s their area of expertise.
+- Provide strong and valuable feedback during reviews. This is expected both when reviewing community PRs as well as PRs from other core contributors.
+  Even if you are not part of core yet, you can still review and approve PRs. This will show us your abilities.
 
 ### Diversity
 
-Lightning should reflect the broader community it serves. As such we should have scientists/researchers from
-different fields contributing!
+Lightning should reflect the broader community it serves. As such we should have scientists/researchers from different fields contributing!
+
+### Community
 
-The first 5 core contributors will fit this profile. Thus if you overlap strongly with experiences and expertise as someone else on the team, you might have to wait until the next set of contributors are added.
+We have an active [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ) community, where questions are asked daily.
+This is a great way to show off your Lightning and PyTorch knowledge, and help out others.
+There's also [GitHub discussions](https://github.com/PyTorchLightning/pytorch-lightning/discussions).
 
-### Summary: Requirements to apply
+## Applying
 
-The goal is to be inline with expectations for solving issues by the last one so you can do them on your own. If not, I might ask you to solve a few more specific ones.
+There are no precise targets for becoming a core contributor. In the past, community members have become core after fitting the previous expectations consistently.
+We are on the lookout for new people to join, however, if you feel like you meet the expectations already and we haven't reached out to you yet, feel free to ping us privately on [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)!.
 
-- Solve 10+ Github issues.
-- Create 5+ meaningful PRs which solves some reported issue - bug,
-- Perform 10+ PR reviews from other contributors.
+## Employment
 
-If you want to be considered, ping me on [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ).
+You can also become a [Grid.ai](https://www.grid.ai) employee or intern and work on Lightning. To get started, you can email `[email protected]` with your resume or check out our [open job postings](https://boards.greenhouse.io/gridai).

.github/CODEOWNERS

@@ -32,6 +32,7 @@
 /pytorch_lightning/plugins              @tchaton @SeanNaren @awaelchli @justusschock
 /pytorch_lightning/profiler             @williamfalcon @tchaton @borda @carmocca
 /pytorch_lightning/profiler/pytorch.py  @nbcsm @guotuofeng
+/pytorch_lightning/strategies           @tchaton @SeanNaren @awaelchli @justusschock @kaushikb11
 /pytorch_lightning/trainer              @williamfalcon @borda @tchaton @SeanNaren @carmocca @awaelchli @justusschock @kaushikb11
 /pytorch_lightning/trainer/connectors   @tchaton @SeanNaren @carmocca @borda
 /pytorch_lightning/tuner                @SkafteNicki @borda @awaelchli
@@ -52,6 +53,7 @@
 /tests/helpers/boring_model.py          @williamfalcon @tchaton @borda
 
 /.github/CODEOWNERS              @williamfalcon
+/SECURITY.md                     @williamfalcon
 /README.md                       @williamfalcon @edenlightning @borda
 /setup.py                        @williamfalcon @borda
 /pytorch_lightning/__about__.py  @williamfalcon @borda

.github/CONTRIBUTING.md

@@ -1,6 +1,7 @@
 # Contributing
 
-Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices that the amazing PyTorch team rolls out!
+Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices
+and integrations that the amazing PyTorch team and other research organization rolls out!
 
 If you are new to open source, check out [this blog to get started with your first Open Source contribution](https://devblog.pytorchlightning.ai/quick-contribution-guide-86d977171b3a).
 
@@ -9,12 +10,12 @@ If you are new to open source, check out [this blog to get started with your fir
 Simplify the API as much as possible from the user perspective.
 Any additions or improvements should minimize the things the user needs to remember.
 
-For example: One benefit of the validation_step is that the user doesn't have to remember to set the model to .eval().
+For example: One benefit of the `validation_step` is that the user doesn't have to remember to set the model to .eval().
 This helps users avoid all sorts of subtle errors.
 
 ## Lightning Design Principles
 
-We encourage all sorts of contributions you're interested in adding! When coding for lightning, please follow these principles.
+We encourage all sorts of contributions you're interested in adding! When coding for Lightning, please follow these principles.
 
 ### No PyTorch Interference
 
@@ -102,7 +103,7 @@ _**Note**, even if you do not find the solution, sending a PR with a test coveri
 
 Want to keep Lightning healthy? Love seeing those green tests? So do we! How to we keep it that way? We write tests! We value tests contribution even more than new features.
 
-Most of the tests in PyTorch Lightning train a trial MNIST model under various trainer conditions (ddp, ddp2+amp, etc...). The tests expect the model to perform to a reasonable degree of testing accuracy to pass. Want to add a new test case and not sure how? [Talk to us!](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)
+Most of the tests in PyTorch Lightning train a random `BoringModel` under various trainer conditions (ddp, ddp2+amp, etc...). Want to add a new test case and not sure how? [Talk to us!](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)
 
 ______________________________________________________________________
 
@@ -125,7 +126,7 @@ In case you adding new dependencies, make sure that they are compatible with the
 ### Coding Style
 
 1. Use f-strings for output formation (except logging when we stay with lazy `logging.info("Hello %s!", name)`.
-1. You can use `pre-commit` to make sure your code style is correct.
+1. You can use [pre-commit](https://pre-commit.com/) to make sure your code style is correct.
 
 ### Documentation
 
@@ -134,7 +135,7 @@ Moreover, we set Google style to follow with type convention.
 
 - [Napoleon formatting with Google style](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)
 - [ReStructured Text (reST)](https://docs.pylonsproject.org/projects/docs-style-guide/)
-- [Paragraph-level markup](https://www.sphinx-doc.org/en/1.5/markup/para.html)
+- [Paragraph-level markup](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#paragraphs)
 
 See following short example of a sample function taking one position string and optional
 
@@ -152,12 +153,14 @@ def my_func(param_a: int, param_b: Optional[float] = None) -> str:
     Return:
         sum of both numbers
 
-    Example:
+    Example::
+
         Sample doctest example...
         >>> my_func(1, 2)
         3
 
-    .. note:: If you want to add something.
+    Note:
+        If you want to add something.
     """
     p = param_b if param_b else 0
     return str(param_a + p)
@@ -170,6 +173,7 @@ Run these commands
 ```bash
 git submodule update --init --recursive
 pip install -r requirements/docs.txt
+make clean
 cd docs
 make html
 ```
@@ -194,6 +198,7 @@ To setup a local development environment, install both local and test dependenci
 ```bash
 python -m pip install ".[dev, examples]"
 python -m pip install pre-commit
+pre-commit install
 ```
 
 Additionally, for testing backward compatibility with older versions of PyTorch Lightning, you also need to download all saved version-checkpoints from the public AWS storage. Run the following script to get all saved version-checkpoints:
@@ -232,21 +237,42 @@ python -m pytest -v tests/trainer/test_trainer_cli.py::test_default_args
 
 We welcome any useful contribution! For your convenience here's a recommended workflow:
 
-0. Think about what you want to do - fix a bug, repair docs, etc. If you want to implement a new feature or enhance an existing one, start by opening a GitHub issue to explain the feature and the motivation. Members from core-contributors will take a look (it might take some time - we are often overloaded with issues!) and discuss it. Once an agreement was reached - start coding.
-1. Start your work locally (usually until you need our CI testing).
+1. Think about what you want to do - fix a bug, repair docs, etc. If you want to implement a new feature or enhance an existing one.
+
+   - Start by opening a GitHub issue to explain the feature and the motivation.
+     In the case of features, ask yourself first - Is this NECESSARY for Lightning? There are some PRs that are just
+     purely about adding engineering complexity which has no place in Lightning.
+   - Core contributors will take a look (it might take some time - we are often overloaded with issues!) and discuss it.
+   - Once an agreement was reached - start coding.
+
+1. Start your work locally.
+
    - Create a branch and prepare your changes.
-   - Tip: do not work with your master directly, it may become complicated when you need to rebase.
+   - Tip: do not work on your master branch directly, it may become complicated when you need to rebase.
    - Tip: give your PR a good name! It will be useful later when you may work on multiple tasks/PRs.
+
 1. Test your code!
+
    - It is always good practice to start coding by creating a test case, verifying it breaks with current behaviour, and passes with your new changes.
    - Make sure your new tests cover all different edge cases.
-   - Make sure all exceptions are handled.
-1. Create a "Draft PR" which is clearly marked, to let us know you don't need feedback yet.
+   - Make sure all exceptions raised are tested.
+   - Make sure all warnings raised are tested.
+
+1. If your PR is not ready for reviews, but you want to run it on our CI, open a "Draft PR" to let us know you don't need feedback yet.
+
 1. When you feel ready for integrating your work, mark your PR "Ready for review".
+
    - Your code should be readable and follow the project's design principles.
-   - Make sure all tests are passing.
-   - Make sure you add a GitHub issue to your PR.
-1. Use tags in PR name for following cases:
+   - Make sure all tests are passing and any new code is tested for (coverage!).
+   - Make sure you link the GitHub issue to your PR.
+   - Make sure any docs for that piece of code are updated, or added.
+   - The code should be elegant and simple. No over-engineering or hard-to-read code.
+
+   Do your best but don't sweat about perfection! We do code-review to find any missed items.
+   If you need help, don't hesitate to ping the core team on the PR.
+
+1. Use tags in PR name for the following cases:
+
    - **\[blocked by #<number>\]** if your work is dependent on other PRs.
    - **\[wip\]** when you start to re-edit your work, mark it so no one will accidentally merge it in meantime.
 
@@ -303,7 +329,7 @@ Here is the process to create a new test
 - 0. Optional: Follow tutorials !
 - 1. Find a file in tests/ which match what you want to test. If none, create one.
 - 2. Use this template to get started !
-- 3. Use `BoringModel and derivates to test out your code`.
+- 3. Use **BoringModel and derivates to test out your code**.
 
 ```python
 # TEST SHOULD BE IN YOUR FILE: tests/..../...py
@@ -335,7 +361,7 @@ def test_explain_what_is_being_tested(tmpdir):
 run our/your test with
 
 ```bash
-python -m pytest tests/..../...py::test_explain_what_is_being_tested --verbose --capture=no
+python -m pytest tests/..../...py::test_explain_what_is_being_tested -v --capture=no
 ```
 
 #### How to fix PR with mixed base and target branches?

.github/mergify.yml

@@ -66,4 +66,4 @@ pull_request_rules:
     actions:
       request_reviews:
         teams:
-          - "@PyTorchLightning/core-contributors"
+          - "@PyTorchLightning/core-lightning"

.github/set-min-requirements.py

@@ -2,8 +2,8 @@
     "requirements.txt",
     "requirements/extra.txt",
     "requirements/loggers.txt",
-    "requirements/test.txt",
     "requirements/examples.txt",
+    # "requirements/test.txt",  # Don't use old testing packages
 )
 
 

.github/workflows/ci_dockers.yml

@@ -17,6 +17,10 @@ on: # Trigger the workflow on push or pull request, but only for the master bran
       - ".github/workflows/events-nightly.yml"
       - "setup.py"
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
+  cancel-in-progress: ${{ ! (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release/')) }}
+
 jobs:
   build-PL:
     runs-on: ubuntu-20.04

.github/workflows/ci_pkg-install.yml

@@ -7,6 +7,10 @@ on:  # Trigger the workflow on push or pull request, but only for the master bra
   pull_request:
     branches: [master, "release/*"]
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
+  cancel-in-progress: ${{ ! (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/heads/release/')) }}
+
 jobs:
 
   install:
@@ -16,7 +20,7 @@ jobs:
       # max-parallel: 6
       matrix:
         os: [ubuntu-20.04, macOS-10.15, windows-2019]
-        python-version: [3.6, 3.9]
+        python-version: [3.7, 3.9]
 
     steps:
       - uses: actions/checkout@v2