Always download MNIST from PyTorch mirror

[equaeghe]

🐛 Describe the bug

Torchvision by default downloads MNIST from the original server. The owner there explicitly requests

 Please refrain from accessing these files from automated scripts with high frequency. Make copies!

Currently, MNIST is already available on the PyTorch mirror: see #3544 and #3549. It seems that the polite thing to do is to never download from the original server.

Versions

Collecting environment information...
PyTorch version: 1.9.0a0+gitd69c22d
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Gentoo/Linux (x86_64)
GCC version: (Gentoo 11.2.0 p1) 11.2.0
Clang version: 12.0.1
CMake version: version 3.20.5
Libc version: glibc-2.33

Python version: 3.9 (64-bit runtime)
Python platform: Linux-5.14.12-gentoo-b-x86_64-AMD_Ryzen_7_PRO_4750U_with_Radeon_Graphics-with-glibc2.33
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.1
[pip3] numpydoc==1.1.0
[pip3] torch==1.9.0a0+gitd69c22d
[pip3] torchvision==0.10.0a0
[conda] Could not collect

cc @pmeier

[pmeier]

Please refrain from accessing these files from automated scripts with high frequency. Make copies!

I agree with the author there, but we are not doing that. The only time we actually hitting the server is on a daily basis to check that the files are available. Even than we have a rate limitation and we are only doing head request to keep the traffic minimal. Thus, the instruction is actually for the users. If you use our builtin torchvision.datasets.MNIST(..., download=True) you don't need to worry, since we will cache the dataset on disk for further usage.

Furthermore, AFAIK the author only authorized us to use the mirror in case the original server is unavailable for some reason. cc @fmassa. Note that MNIST is a special case, because it is the only dataset for which we have a fallback download server.

In conclusion, I appreciate your concern, but we already do everything we can to limit our access to the MNIST server. Thus, I'm closing this.

[equaeghe]

 Thus, the instruction is actually for the users.

But users never see this instruction and may redownload it multiple times, e.g., if they regularly clean their downloads or often use fresh environments (VMs or other types).

If you use our builtin torchvision.datasets.MNIST(..., download=True) you don't need to worry, since we will cache the dataset on disk for further usage.

Am I correct in thinking that this way every user needing MNIST will download it once, as happened to me? Given that the number of users can be large, then avoiding ‘high frequency’ seems a hard requirement to meet. (My concern is about a downstream library that uses torchvision and I would prefer it to respect the MNIST author's request.)

Furthermore, AFAIK the author only authorized us to use the mirror in case the original server is unavailable for some reason.

Oh, that seems like strange restriction. I've seen MNIST elsewhere and didn't realize the author might claim some kind of database right, as that is not mentioned on their website. But if this is the case I understand the current approach.

[NicolasHug]

Furthermore, AFAIK the author only authorized us to use the mirror in case the original server is unavailable for some reason.

Oh, that seems like strange restriction. I've seen MNIST elsewhere and didn't realize the author might claim some kind of database right, as that is not mentioned on their website. But if this is the case I understand the current approach.

I think this is the key bit here. If we don't have permission to put our mirror first, then there isn't much we can do.

But if we're allowed to, I would agree that it would be something good to consider. While we (the torchvision package) "only" downloads such datasets a few times a day, the fact that we're exposing such API will necessarily increase the number of download by a lot, just because we're making it very easy and convenient to do so.

[pmeier]

I think this issue will be mitigated quite a bit in the future by having one central place to store all the datasets. Depending on your setup, it might be quicker to simply do something like datasets.MNIST("/tmp/mnist", download=True) since the download is fairly quick, but of course strains the server. After the datasets rework, users no longer need to provide a root directory, so they would need to manually delete the files in order for them to get re-downloaded.