TorchText tests failed when multiprocessing

[ejguan]

🐛 Describe the bug

I think the test from TorchText starts failing after pytorch/pytorch#74984 is landed into core and shipped into nightly.

Should we either revert the change or fix this problem ASAP considering the closing branch cut for TorchText?
We can let TorchText team to remove all lambda function in their dataset if needed.

Versions

main

Progress

• Replacing lambda usages in TorchText
  Replacing lambda functions with regular functions in all datasets pytorch/text#1718
• Add ability to use dill in DataLoaderV1
  [DataLoader] Adding ability to use dill to pass DataPipes in mutiprocessing pytorch/pytorch#77288
• Add ability to use dill in DataLoaderV2 (TODO)

[ejguan]

See the domain tests in my PR https://github.com/pytorch/data/pull/394/checks

[VitalyFedyunin]

Thank you for reporting, lets fix it forward in DataLoader1 and DataLoader2

[NivekT]

If users upgrade torch but not torchtext, this will break in the next release. To address that, it seems like we definitely need to make changes to both versions DataLoader to switch to use dill for serialization.

I can also submit a PR to torchtext and switch its lambda usages to non-lambda function instead, since we cannot expect all users to have dill installed. Do you think that will be appropriate as well?

[VitalyFedyunin]

@NivekT totally makes sense to update torchtext. As parallel topic: Should we consider dill as hard dependency for torchdata?

[NicolasHug]

If that helps, we have opened pytorch/vision#5938 so that you can check whether recent changes break the torchvision datapipes when dill is installed.

Re torchtext's useof lambdas: I mentioned to @parmeet yesterday that the use of lambdas will break users' code when doing multi-processing on Windows, so I believe they are planning on fixing this (would need Parmeet's confirmation though).

Regarding making dill a hard-dependency: doesn't dill silently replace pickle automatically?

[ejguan]

Regarding making dill a hard-dependency: doesn't dill silently replace pickle automatically?

Yeah but technically, we can revert the change by doing dill.extend(use_dill=False). https://github.com/pytorch/pytorch/blob/50efa3a6e8ff8a29f7a0cb08cf3e3e5ed42f38fe/torch/utils/data/_utils/serialization.py#L5-L15

[NicolasHug]

Do I understand correctly that this revert will only affect torchdata code? I.e. installing dill will still silently replace pickle in the rest of the users environement?

[ejguan]

Install won't affect. It only affects the pickle when import dill. So, I think always reverting the change every time thattorchdata finishes using dill should remove impact for other part of code.