WandBlogger doesn't log when working with TPU cores

[rusty-electron]

🐛 Bug

I am trying to run a resnet200d model for document classification on Kaggle TPUs using python-lightning. I have set up WandB logger and connected it to my wandb account. The training starts well but at 50 steps the logger complains that It can't log as the process was called from a different pid. See error image below:

I have tried to use options like sync_dist=True or rank_zero_only=True (in the self.log calls) but it still doesn't work.

To Reproduce

You can try running the mentioned kaggle notebook:
https://www.kaggle.com/rustyelectron/documentclassification-pytorch-tpu-wandb/

Wait till about 50 steps to get the error.

Expected behavior

The wandblogger should work normally and log data.

Environment

• CUDA:
  • GPU:
  • available: False
  • version: None
• Packages:
  • numpy: 1.19.5
  • pyTorch_debug: True
  • pyTorch_version: 1.7.0a0+7e71a98
  • pytorch-lightning: 1.2.10
  • tqdm: 4.62.1
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.7.10

Additional context

None

cc @kaushikb11 @awaelchli @morganmcg1 @AyushExel @borisdayma @scottire

[Programmer-RD-AI]

hi,

Can you check these issues?

https://forums.pytorchlightning.ai/t/tpu-failures-in-colab/310
wandb/wandb#1994
#1062
https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498
https://wandb.ai/site/articles/pytorch-lightning-with-weights-biases
https://wandb.ai/cayush/pytorchlightning/reports/Use-Pytorch-Lightning-with-Weights-Biases--Vmlldzo2NjQ1Mw

Regards,

[Programmer-RD-AI]

and can you make the Kaggle notebook public please so I can check the notebook?

Regards,

[rusty-electron]

@Programmer-RD-AI sorry, 🤦 its public now. Ok, I will take a look at the links you provided.

[Programmer-RD-AI]

It ok check the links and see if they solve the issue

Regards

[rusty-electron]

@Programmer-RD-AI I checked the links but it didn't help, they are not related to my issue.

I must mention that this is an issue only when I use pytorch-lightning==1.2.10. I have also tried v1.1.8 and this issue doesn't show up in the latter version. I am not using the latter version as I ran into a different issue with it.

[Programmer-RD-AI]

hi, v1.1.8 and v1.2.10 are old versions can you try updating it to 1.4.9 or 1.4.8 it sometimes works.

Regards

[rusty-electron]

I tried installing the newer versions on kaggle environments but it didn't work with the torch xla made for TPUs, do you know which pytorch-lightning versions are compatible with which versions of torch-xla? something like a compatibility chart. I found that most notebooks on kaggle were using these versions.

[Programmer-RD-AI]

hi,
I found some resources about TPU with torch-xla.
https://pytorch.org/xla/release/1.9/index.html
#4829
https://pytorch-lightning.readthedocs.io/en/latest/advanced/tpu.html

Hope the above resources help.

With best regards.

[Programmer-RD-AI]

hi I found some more resources,

https://towardsdatascience.com/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4
https://issueexplorer.com/issue/pytorch/xla/3080
https://githubmemory.com/repo/pytorch/xla/issues?cursor=Y3Vyc29yOnYyOpK5MjAyMS0wNS0xMlQxNDo1MDoyNSswODowMM41CF8K&pagination=next&page=2
https://www.kaggle.com/tanlikesmath/xlm-roberta-pytorch-xla-tpu

With best regards.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!