[HELP WANTED] Re-write datasets in torchtext

[zhangguanheng66]

As mentioned in #664, we are working on a new dataset abstraction. The new datasets will be more compatible with pytorch core library and capable of out-of-box libraries (like SentencePiece BPE).

We have landed several datasets in torchtext.experimental.datasets folder to test the new abstraction. Now, we want some help from open-source community. Please sign up here and contribute PRs to re-write those datasets in torchtext. Those datasets in torchtext/experimental/datasets/text_classification should be some good examples to follow. Some datasets but not limited include:

• Word language modeling datasets. Already in the experimental folder but need some update to the latest abstraction.
• IMDB
• Text classification datasets (AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull)
• Translation
• Question-answer datasets (e.g. SQuAD here)
• EnWik9, which's already good
• Sequence tagging dataset in Experimental sequence tagging datasets #805

Here is a checklist that you may consider to migrate a dataset:

1. Set up a IterableDataset with an iterator to read raw data link
2. Cache out the raw dataset and create a transform pipeline link
   • pick up a tokenizer
   • generate the vocab object if not provided by users
   • attach tokenizer + vocab + totensor to the transform pipeline
3. Add unit tests link
   • check the total length of the dataset
   • check the content of the dataset

Free free to ping me and @cpuhrsch if you want to check some ideas for this. Thanks in advance.

[akurniawan]

hi @zhangguanheng66 I'm interested to work on the translation dataset migration

[zhangguanheng66]

@akurniawan Thanks. There is a draft branch link I wrote 5 months ago and was close to finish. You may start from there and consider some changes in the latest abstraction. Let us know if you have any questions.

[rmz59]

@zhangguanheng66 willing to work with the Word language modeling datasets.

[zhangguanheng66]

@zhangguanheng66 Thanks. Please follow the instructions above to re-write the datasets.

[akurniawan]

@zhangguanheng66 I tried to import sequence tagging dataset in #805 , seems not in the list you mentioned above, so I don't know if it's a priority for now. Let me know what you think

[zhangguanheng66]

@zhangguanheng66 I tried to import sequence tagging dataset in #805 , seems not in the list you mentioned above, so I don't know if it's a priority for now. Let me know what you thought.

@akurniawan Thanks. I have updated the list above.

[Nayef211]

Closing as all datasets mentioned in this issue have now been added