Can't download IWSLT dataset to Google Colab

[senadkurtisi]

I am experimenting with an implementation of the "Attention is All You Need" paper.
This is the implementation.
And I am using the Google Colab to be able to use the GPU.
But the code for downloading the dataset using PyTorch results in error.

This is the code I used:

from torchtext import data, datasets

import spacy
spacy_de = spacy.load('de')
spacy_en = spacy.load('en')

def tokenize_de(text):
    return [tok.text for tok in spacy_de.tokenizer(text)]

def tokenize_en(text):
    return [tok.text for tok in spacy_en.tokenizer(text)]

BOS_WORD = '<s>'
EOS_WORD = '</s>'
BLANK_WORD = "<blank>"
SRC = data.Field(tokenize=tokenize_de, pad_token=BLANK_WORD)
TGT = data.Field(tokenize=tokenize_en, init_token = BOS_WORD, 
                 eos_token = EOS_WORD, pad_token=BLANK_WORD)

MAX_LEN = 100
train, val, test = datasets.IWSLT.splits(
    exts=('.de', '.en'), fields=(SRC, TGT), 
    filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and 
        len(vars(x)['trg']) <= MAX_LEN)
MIN_FREQ = 2
SRC.build_vocab(train.src, min_freq=MIN_FREQ)
TGT.build_vocab(train.trg, min_freq=MIN_FREQ)

And this is the error I got:

OSError                                   Traceback (most recent call last)
/usr/lib/python3.6/tarfile.py in gzopen(cls, name, mode, fileobj, compresslevel, **kwargs)
   1644         try:
-> 1645             t = cls.taropen(name, mode, fileobj, **kwargs)
   1646         except OSError:

12 frames
OSError: Not a gzipped file (b'<!')

During handling of the above exception, another exception occurred:

ReadError                                 Traceback (most recent call last)
/usr/lib/python3.6/tarfile.py in gzopen(cls, name, mode, fileobj, compresslevel, **kwargs)
   1647             fileobj.close()
   1648             if mode == 'r':
-> 1649                 raise ReadError("not a gzip file")
   1650             raise
   1651         except:

ReadError: not a gzip file

Is this know issue? Or am I missing something?

[zhangguanheng66]

Have you resolved the issue? I think someone reported the issue here #1091

[senadkurtisi]

No I haven't resolved it. But

Have you resolved the issue? I think someone reported the issue here #1091

I haven't resolved it myself, but I tried the solution to which you redirected me and I was able to download it. Thank you!

[zhangguanheng66]

Fixed by #1115

[diegoquintanav]

@senadkurtisi did you manage to fix the error? if so, what did you do? thanks!

[parmeet]

You can also try IWSLT dataset in the latest release where you can find options to download by year, language_pair etc.
https://pytorch.org/text/stable/datasets.html#machine-translation

[Matrix-Jian]

I met the same problem and fixed it now. I thought that the url of dataset in the function: 'datasets.IWSLT.splits' is outdated and it lead to download an error file 'de-en.tgz'. So I download the IWSLT2016 datasets from here. I found the file 'de-en.tgz' in the path: '2016-01/2016-01/texts/de/en/' after unzipping. And replace 'de-en.tgz' in './data/iwslt/' with it. Finally, run the code again.

[yuvaraj91]

I am facing the same problem on MacOs, may I know where is the './data' folder location? @Matrix-Jian