experimental.dataset WikiText2, WikiText103, PennTreeBank, WMTNewsCrawl

[rmz59]

• Update word language modeling datasets [HELP WANTED] Re-write datasets in torchtext #742

[zhangguanheng66]

To be consistent with text classification datasets, please call the transform func in __getitem__ func

[rmz59]

It might be hard to call the transform func in __getitem__. For dataset like WikiText2, the raw data wiki.train.tokens is stored as a multi-line txt file, and self.__getitem__[i] is expeced to output i-th token. Therefore, pre-processing is required to concat and tokenize the multi-line file. Can you advise if we need an additional preprocess func?

[rmz59]

I cant' move transforms into __getitem__, because tokenizer must be applied before __getitem__. Otherwise, the unit test https://github.com/pytorch/text/blob/master/test/data/test_builtin_datasets.py#L54-L57 will fail here.

Possible solution

• Move tokenizer out of transforms
• OR: split transforms into two parts - global transforms / token-level transforms.

Please advise how I can move forward. Thanks.

[zhangguanheng66]

Looks good to me and very close to the dataset abstraction. Only made a few suggestions for changes.

[zhangguanheng66]

Could you also consolidate WMTNewsCrawl here into word language modeling datasets?

[zhangguanheng66]

Since you change the order of vocab and tokenizer, will this be a BC breaking?

[rmz59]

Yes, it's a possible BC breaking if people use args instead of kwargs. I restored the previous order of vocab and tokenizer in commit f433b40 above.

[zhangguanheng66]

Great. Let me know when you are done with the revision. I will have another look.

[zhangguanheng66]

I thought we agreed to go by tokenizer, root, vocab in order.

[zhangguanheng66]

@z-Runmin Please kindly let us know if you need a review.

[rmz59]

Not sure why unittest_windows failed to install torchtext

[zhangguanheng66]

Not sure why unittest_windows failed to install torchtext

@peterjc123 Could you take a look at here? The master branch is green.

[zhangguanheng66]

and docs for single_line

[zhangguanheng66]

LGTM. Add only two comments. Once the CI tests are fixed, we can merge the PR. Then, I will switch my BERT pipeline to the new datasets.

[zhangguanheng66]

And could you add a test for WMTNewsCrawl dataset? like

text/test/data/test_builtin_datasets.py

Line 79 in e709553

def test_penntreebank(self):

[peterjc123]

Not sure why unittest_windows failed to install torchtext

@peterjc123 Could you take a look at here? The master branch is green.

@z-Runmin @zhangguanheng66 You might need to rebase your branch on master.

[peterjc123]

Looks like the wmt dataset is so large that it couldn't be downloaded within the time limit (30 min).

[zhangguanheng66]

Looks like the wmt dataset is so large that it couldn't be downloaded within the time limit (30 min).

OK, let me remove the test.

[zhangguanheng66]

Add some minor changes. Will merge after CI tests pass.