Multi30K dataset link is broken

[xuzhao9]

The link to Multi30K dataset at http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz is broken:

text/torchtext/datasets/multi30k.py

Line 16 in 73bf4fa

"train": r"http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz",

[parmeet]

Yupp, looks like the server is down :(.

[muskbing]

Yupp, looks like the server is down :(.

Any solutions?

[2401ch]

I meet the same problem, is there any solution?

[2401ch]

[Nayef211]

@chenghan1995 @muskbing unfortunately, we're not responsible for hosting the datasets. I'd recommend waiting for their server to come back up or reaching out directly to the organization that hosts the dataset. In this case this would be the University of Sheffield.

cc @parmeet I wonder if you know of a way to get in contact with the team that hosts this dataset?

[muskbing]

@chenghan1995 @muskbing unfortunately, we're not responsible for hosting the datasets. I'd recommend waiting for their server to come back up or reaching out directly to the organization that hosts the dataset. In this case this would be the University of Sheffield.

cc @parmeet I wonder if you know of a way to get in contact with the team that hosts this dataset?

I've send email to the owner of the dataset
https://www.statmt.org/wmt16/multimodal-task.html#task1

email address is:lspecia@gmail.com

But there is no response.

I wander is there anyone who has the data file

[neychev]

Found a local copy of the dataset and uploaded it to github (it's rather small). For now it is available via this link: https://github.com/neychev/small_DL_repo/tree/master/datasets/Multi30k

Just in case, all rights belong to the original authors of the dataset, this is only a temporal copy for convenience.

[muskbing]

Found a local copy of the dataset and uploaded it to github (it's rather small). For now it is available via this link: https://github.com/neychev/small_DL_repo/tree/master/datasets/Multi30k

Just in case, all rights belong to the original authors of the dataset, this is only a temporal copy for convenience.

Thanks bro, you're really awesome

[neychev]

Please, refer to the next answer with updated example

Example code to make it work (tested on Colab):

!pip install torchdata
!mkdir -p ~/.torchtext/cache/Multi30k
!wget -P ~/.torchtext/cache/Multi30k https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz
!wget -P ~/.torchtext/cache/Multi30k https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz
!wget -P ~/.torchtext/cache/Multi30k https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt16_task1_test.tar.gz

from torchtext.datasets import Multi30k
train_iter = Multi30k(split="train")

[rrmina]

Found a local copy of the dataset and uploaded it to github (it's rather small). For now it is available via this link: https://github.com/neychev/small_DL_repo/tree/master/datasets/Multi30k

Just in case, all rights belong to the original authors of the dataset, this is only a temporal copy for convenience.

Thank you! This worked for train and valid but not for test :( .

The test file being downloaded by torchtext (and torchdata) are from http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/mmt16_task1_test.tar.gz

Does anyone have the mmt16_task1_test.tar.gz file for the meantime? Thanks in advance!!!

P.S. I was able to work around the 'test' issue by making another tar.gz from the contents of mmt_task1_test2016.tar.gz, and changing the 'test' hash in torchtext sources but I assume that other users may not be able to this

[Nayef211]

Example code to make it work (tested on Colab):

!pip install torchdata

!mkdir -p ~/.torchtext/cache/Multi30k

!wget -P ~/.torchtext/cache/Multi30k https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz
!wget -P ~/.torchtext/cache/Multi30k https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz
!wget -P ~/.torchtext/cache/Multi30k https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt_task1_test2016.tar.gz


# now everything works as intended
from torchtext.datasets import Multi30k
train_iter = Multi30k(split="train")

Just wanted to mention another approach to get Multi30k working with the data you are hosting @neychev. Rather than downloading the data directly using wget we can programmatically modify the URLs that each split of the dataset is being dowloaded from as follows:

from torchtext.datasets import multi30k, Multi30k

# Update URLs to point to data stored by user
multi30k.URL["train"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz"
multi30k.URL["valid"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz"
multi30k.URL["test"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt_task1_test2016.tar.gz"

# Update hash since there is a discrepancy between user hosted test split and that of the test split in the original dataset 
multi30k.MD5["test"] = "876a95a689a2a20b243666951149fd42d9bfd57cbbf8cd2c79d3465451564dd2"

dp = Multi30k(split='train')

As @rrmina mentioned earlier, this approach still doesn't work with the test split. If I try to print the contents of the test split, I don't get any outputs. @neychev do you happen to know what the discrepancy is for mmt16_task1_test.tar.gz between the original test split and the one you host?

As a next step, I also plan to update our Multi30k dataset implementation so we can rely on the data stored in https://github.com/neychev/small_DL_repo/tree/master/datasets/Multi30k until the dataset in the original server is restored. This way we don't need to rely on any of the above hacks to get this dataset working. 😄

[neychev]

Thanks, @Nayef211, @rrmina !

No idea what's exactly wrong with the data, the files above were located in ~/.torchtext/cache/Multi30k of one of my students.

I've tried to simply rename the archive (according to the name in torchtext docs) and files in it and change MD5 to the correct one and it seems to work.

Including the approach suggested by @Nayef211, which is way more elegant, the final algorithm should be the following:

from torchtext.datasets import multi30k, Multi30k

# Update URLs to point to data stored by user
multi30k.URL["train"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz"
multi30k.URL["valid"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz"
multi30k.URL["test"] = "https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/mmt16_task1_test.tar.gz"

# Update hash since there is a discrepancy between user hosted test split and that of the test split in the original dataset 
multi30k.MD5["test"] = "6d1ca1dba99e2c5dd54cae1226ff11c2551e6ce63527ebb072a1f70f72a5cd36"

data_train = Multi30k(split='train')
data_val = Multi30k(split='valid')
data_test = Multi30k(split='test')

Test data has 1000 sentences, which seems correct.