Skip to content
This repository was archived by the owner on Sep 10, 2025. It is now read-only.

Cache extraction for AmazonReviewPolarity #1527

Merged
merged 6 commits into from
Jan 20, 2022
Merged

Cache extraction for AmazonReviewPolarity #1527

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e11217e
caching extracted files
parmeet Jan 17, 2022
a5eb508
Merge branch 'main' of github.com:pytorch/text into fix_extraction_cache
parmeet Jan 18, 2022
c09eb57
fix caching
parmeet Jan 18, 2022
27e94a7
remove comparison code
parmeet Jan 18, 2022
4b5bcd9
remove mapping of path
parmeet Jan 19, 2022
36d5370
fix directory creation
parmeet Jan 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions torchtext/datasets/amazonreviewpolarity.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,14 @@ def AmazonReviewPolarity(root: str, split: Union[Tuple[str], str]):
raise ModuleNotFoundError("Package `torchdata` not found. Please install following instructions at `https://github.com/pytorch/data`")

url_dp = IterableWrapper([URL])
cache_dp = url_dp.on_disk_cache(
cache_compressed_dp = url_dp.on_disk_cache(
filepath_fn=lambda x: os.path.join(root, _PATH), hash_dict={os.path.join(root, _PATH): MD5}, hash_type="md5"
)
cache_dp = GDriveReader(cache_dp).end_caching(mode="wb", same_filepath_fn=True)
cache_dp = FileOpener(cache_dp, mode="b")
extracted_files = cache_dp.read_from_tar()
filter_extracted_files = extracted_files.filter(lambda x: _EXTRACTED_FILES[split] in x[0])
return filter_extracted_files.parse_csv().map(fn=lambda t: (int(t[0]), ' '.join(t[1:])))
cache_compressed_dp = GDriveReader(cache_compressed_dp).end_caching(mode="wb", same_filepath_fn=True)

cache_decompressed_dp = cache_compressed_dp.on_disk_cache(filepath_fn=lambda x: os.path.join(root, _EXTRACTED_FILES[split]))
cache_decompressed_dp = FileOpener(cache_decompressed_dp, mode="b").read_from_tar().filter(lambda x: _EXTRACTED_FILES[split] in x[0])
cache_decompressed_dp = cache_decompressed_dp.end_caching(mode="wb", same_filepath_fn=True)
Copy link
Contributor

@Nayef211 Nayef211 Jan 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: let's try to be consistent with our usage of single vs double quotes for strings in the torchtext repo. Maybe it's worth adding coding style preferences to the contribution guidelines.

Copy link
Contributor Author

@parmeet parmeet Jan 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good point. I do not have a specific preference. Not sure, if there is a convention across PyTorch board. @mthrok Any thoughts/suggestions here?

Copy link
Contributor

@mthrok mthrok Jan 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We rely on black to do the formatting, which prefers double quote. Recently @jdsgomes has set up a lint job for audio.

Copy link
Contributor

@Nayef211 Nayef211 Jan 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can add setting up a lint job for torchtext as a followup item.


data_dp = FileOpener(cache_decompressed_dp, mode='b')
return data_dp.parse_csv().map(fn=lambda t: (int(t[0]), ' '.join(t[1:])))