torchtext.legacy.datasets.IWSLT is unusable due to outdated URL

[chrisyeh96]

🐛 Bug

The IWSLT dataset's URL was updated some time in late 2020, as mentioned in #1091. When torchtext v0.9.0 updated the IWSLT datasets to use the new dataset URL on Google Drive (see #1115), the corresponding torchtext.legacy.datasets.IWSLT dataset was not updated to the new URL.

Consequently, using torchtext.legacy.datasets.IWSLT causes torchtext to download an HTML page with a 404 message, instead of the actual dataset. This leads to the error: OSError: Not a gzipped file.

To Reproduce

Code

from torchtext.legacy import data, datasets
f = data.Field()
datasets.IWSLT.splits(exts=('.de', '.en'), fields=(f, f))

Output

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/usr/lib/python3.7/tarfile.py in gzopen(cls, name, mode, fileobj, compresslevel, **kwargs)
   1645         try:
-> 1646             t = cls.taropen(name, mode, fileobj, **kwargs)
   1647         except OSError:

------⬍ 12 frames------
OSError: Not a gzipped file (b'<!')

During handling of the above exception, another exception occurred:

ReadError                                 Traceback (most recent call last)
/usr/lib/python3.7/tarfile.py in gzopen(cls, name, mode, fileobj, compresslevel, **kwargs)
   1648             fileobj.close()
   1649             if mode == 'r':
-> 1650                 raise ReadError("not a gzip file")
   1651             raise
   1652         except:

ReadError: not a gzip file

Note that switching to earlier versions of torchtext (e.g., v0.9 or v0.8) don't help, because that does not resolve the underlying 3rd-party URL issue.

Environment Info

• torchtext v0.10.0
• System: tested on Google Colab and local machine, details unimportant

[parmeet]

Thanks @chrisyeh96 for raising the issue. Unfortunately, the legacy code-base is no longer maintained.

Please refer to the release notes and migration guide for additional guidance. Also we have IWSLT available as Iterable datasets. Please refer to the documentation for IWSLT2016 and IWSLT2017.

[yuvaraj91]

@parmeet Thanks for notifying, I myself am facing this issue. I have gone through the migration guide, however it only shows for a single language. For a translation dataset like Multi30k and IWSLT2016, is there an example on how to achieve the similar iterator output as the legacy implementation?

I know on the PyTorch website there are two guides, but this one (https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html) is downloading the data instead of using the torchtext in-built dataset ; and this one (https://pytorch.org/tutorials/beginner/translation_transformer.html) is too complicated for loading the dataset.

[FlorientHuang]

I met this problem while studying and now I know why it went wrong...

[GuassTesla]

I have the same problem when using Multi30k.

[Nayef211]

@GuassTesla this should be fixed by #1816