Migrate datasets to build on top of torchdata datapipes

[parmeet]

🚀 Feature

Motivation

https://github.com/pytorch/data#why-composable-data-loading

user-experience: TorchData datasets enable new functional API, auto-sharding, and snapshotting support out-of-the-box. They also enable standard flow-control like batching, collation, shuffling, bucketing, and mapping/transformation using user-defined functions and transforms (UDFs).

Maintenance: By relying on TorchData, we no longer have to maintain low level functionality like downloading, extracting, caching, file/steam parsing, etc.

Reference
Examples: https://github.com/facebookexternal/torchdata/tree/main/examples/text
TorchData: https://github.com/facebookexternal/torchdata

Backlog of datasets

• AG_NEWS migrate AG_NEWS to datapipes. #1498
• AmazonReviewFull migrate Amazon Review Full to datapipes. #1499
• AmazonReviewPolarity Migrating AmazonReviewPolarity to datapipes #1490
• DBpedia migrate DBPedia to datapipes. #1500
• SogouNews migrate SogouNews to datapipes. #1503
• YelpReviewFull migrate YelpReviewFull to datapipes. #1507
• YelpReviewPolarity migrate YelpReviewPolarity to datapipes. #1509
• YahooAnswers migrate YahooAnswers to datapipes. #1508
• CoNLL2000Chunking migrate CONLL 2000 to datapipes. #1515
• UDPOS migrate UDPOS to datapipes. #1535
• IWSLT2016 migrate IWSLT2016 to datapipes. #1545
• IWSLT2017 migrate IWSLT2017 to datapipes. #1547
• Multi30K migrate Multi30k to datapipes. #1536
• SQuAD1 migrate SQUAD1 to datapipes. #1513
• SQuAD2 Migrate squad2 to datapipes #1514
• PennTreebank Migrating PennTreebank to datapipes #1511
• WikiText103 Migrate WikiText103 to datapipes #1518
• WikiText2 Migrate WikiText2 to datapipes #1519
• EnWik9 Migrating EnWik9 to datapipes #1511 #1512
• IMDB Migrate IMDB to datapipes #1531
• SST2 Migrate SST2 from experimental to datasets folder #1538
• CC-100 add CC100 #1562

Contributing

Please leave a message below if you plan to work on particular dataset(s) to avoid duplication of efforts. Also please link to the corresponding PRs.

cc: @Nayef211 , @abhinavarora , @erip , @ejguan , @VitalyFedyunin

[erip]

This sounds great! Logistically, how do we want to organize the PRs? Should we have a branch which adds torchdata as a dependency, we branch from that branch and target that branch for our PRs?

[Nayef211]

@erip That's a good point. Right now we have torchdata as an optional dependency since it is not yet available on pypi (torchdata is in prototype phase until the next Pytorch release). I think going forward we still want to keep it as an optional dependency until torchdata is released as beta.

What do you think @parmeet @abhinavarora?

[parmeet]

Right, agreed with @Nayef211 . Let's continue keeping optional dependency. I am working on PR to migrate AmazonReviewPolarity (#1490) to ensure nothing breaks in the CI when doing the migration. Once this lands, we can follow the same to migrate other datasets.

[erip]

Great, so individual PRs instead of one big one -- got it! I'll pick a couple soon and start digging.

[mthrok]

I guess it will become necessary to use some skip if logic when dealing with optional dependency.
In CI, I recommend to make them fail unless explicitly allowed to skip.
We had an issue where CUDA tests were unintentionally skipped because of wrong CI configuration and permissive skipIf.

Here is the update I applied recently to avoid that pytorch/audio#2127

[erip]

I'll keep a moving list here. To start, I'll take...

• AG_NEWS
• Amazon review full
• DBPedia
• Sogou News
• YelpReviewFull
• YelpReviewPolarity
• YahooAnswers
• CoNLL2000Chunking
• SQUAD1
• SQUAD2
• UDPOS (blocked by CoNLL2000Chunking)
• IWSLT2016
• IWSLT2017
• Multi30K

[parmeet]

Hey @erip, thank you for picking the datasets :).

I'll keep a moving list here. To start, I'll take...

Since we already have a backlog checklist in the the issue, you may constraint this list to the datasets you pick-up and checkmark it directly in the backlog items once the PR lands.

• AmazonReviewPolarity Parmeet has finished this

you may skip the mention for this dataset. I will check mark it once I land the PR :)

[parmeet]

AmazonRiviewPolarity [WIP] #1490. Note that this PR shall also resolve issues with existing CI dataset tests.

[parmeet]

I guess it will become necessary to use some skip if logic when dealing with optional dependency. In CI, I recommend to make them fail unless explicitly allowed to skip. We had an issue where CUDA tests were unintentionally skipped because of wrong CI configuration and permissive skipIf.

Here is the update I applied recently to avoid that pytorch/audio#2127

Actually in this case, the problem is beyond just unit-testing. The package builds (conda, wheels, docs) also rely on availability of torchdata, because it is imported inside the modules. As of now, the import is conditioned on availability. For unit-testing, we are installing torchdata from source. Perhaps we could do the same for package builds?