[RFC] Drop testing in unlikely settings in CI

[akihironitta]

Proposed change in CI

In #11217 (comment), @Borda suggests dropping testing with Python 3.9 and "oldest" because it may be very unlikely that someone uses the latest (supported) Python version with the oldest packages.

... in general, if someone is using almost the latest python I don't think he would assist on the oldest packages...

If we decide to do this, are there any other rare environments?
e.g. should we exclude the oldest Python version (3.7) with the latest packages, too?

Motivation

• Reduces our time spent on debugging something happening in environments that are rarely used.
• Saves computational resources.

Pitch

Exclude either/both of the following envs from ci_test-full.yml:

• cpu (*, 3.9, oldest, stable)
• cpu (*, 3.7, latest, stable)

Additional context

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @carmocca @akihironitta @Borda

[carmocca]

I don't think there's a strong motivation for this considering we aren't at the job limit (currently), they are not a bottleneck, and the checks are green.

However, if you still think this is worth doing, I'd advise not removing:

cpu (*, 3.7, latest, stable)

In the past, I've worked on environments where the Python version can't be upgraded but you are free to choose your library dependencies.

[akihironitta]

I'd advise not removing:
cpu (*, 3.7, latest, stable)
In the past, I've worked on environments where the Python version can't be upgraded but you are free to choose your library dependencies.

That totally makes sense!

---

• Pro: We don't have to look at any future issues with cpu (*, 3.9, oldest, stable) in the future by removing it.
• Con: Somewhat small number of users may experience some issues with PyTorch 1.7 and Python 3.9 used in the future without us knowing it.

@carmocca @Borda I'm now not sure if we should drop it because of the con above. Should we?

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!