import error with protobuf==4.21.0

[peterdudfield]

🐛 Bug

import error when using new protobuf version

To Reproduce

pip install pytorch_lightning protobuf

python

import pytorch_lightning as pl

Results

import pytorch_lightning as pl
/usr/local/lib/python3.9/site-packages/pytorch_lightning/__init__.py:30: in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
/usr/local/lib/python3.9/site-packages/pytorch_lightning/callbacks/__init__.py:26: in <module>
    from pytorch_lightning.callbacks.pruning import ModelPruning
/usr/local/lib/python3.9/site-packages/pytorch_lightning/callbacks/pruning.py:[31](https://github.com/openclimatefix/nowcasting_forecast/runs/6607770425?check_suite_focus=true#step:4:32): in <module>
    from pytorch_lightning.core.lightning import LightningModule
/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/__init__.py:16: in <module>
    from pytorch_lightning.core.lightning import LightningModule
/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py:40: in <module>
    from pytorch_lightning.loggers import LightningLoggerBase, LoggerCollection
/usr/local/lib/python3.9/site-packages/pytorch_lightning/loggers/__init__.py:18: in <module>
    from pytorch_lightning.loggers.tensorboard import TensorBoardLogger
/usr/local/lib/python3.9/site-packages/pytorch_lightning/loggers/tensorboard.py:26: in <module>
    from torch.utils.tensorboard import SummaryWriter
/usr/local/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:10: in <module>
    from .writer import FileWriter, SummaryWriter  # noqa: F401
/usr/local/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py:9: in <module>
    from tensorboard.compat.proto.event_pb2 import SessionLog
/usr/local/lib/python3.9/site-packages/tensorboard/compat/proto/event_pb2.py:17: in <module>
    from tensorboard.compat.proto import summary_pb2 as tensorboard_dot_compat_dot_proto_dot_summary__pb2
/usr/local/lib/python3.9/site-packages/tensorboard/compat/proto/summary_pb2.py:17: in <module>
    from tensorboard.compat.proto import tensor_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__pb2
/usr/local/lib/python3.9/site-packages/tensorboard/compat/proto/tensor_pb2.py:16: in <module>
    from tensorboard.compat.proto import resource_handle_pb2 as tensorboard_dot_compat_dot_proto_dot_resource__handle__pb2
/usr/local/lib/python3.9/site-packages/tensorboard/compat/proto/resource_handle_pb2.py:16: in <module>
    from tensorboard.compat.proto import tensor_shape_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__shape__pb2
/usr/local/lib/python3.9/site-packages/tensorboard/compat/proto/tensor_shape_pb2.py:[36](https://github.com/openclimatefix/nowcasting_forecast/runs/6607770425?check_suite_focus=true#step:4:37): in <module>
    _descriptor.FieldDescriptor(
/usr/local/lib/python3.9/site-packages/google/protobuf/descriptor.py:[56](https://github.com/openclimatefix/nowcasting_forecast/runs/6607770425?check_suite_focus=true#step:4:57)0: in __new__
    _message.Message._CheckCalledFromGeneratedFile()
E   TypeError: Descriptors cannot not be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E   
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Expected behavior

No import error

Environment

• PyTorch Lightning Version (e.g., 1.5.0): pytorch-lightning-1.6.3
• PyTorch Version (e.g., 1.10): torch-1.11.0
• Python version (e.g., 3.9): 3.9
• OS (e.g., Linux): from docker image python:3.9-slim
• CUDA/cuDNN version:
• GPU models and configuration: CPU
• How you installed PyTorch (conda, pip, source): pip
• If compiling from source, the output of torch.__config__.show():
• Any other relevant information: I also get this error on my macbook

Additional context

Filed on this github action

cc @tchaton @rohitgr7

[peterdudfield]

Could be related to this - protocolbuffers/protobuf#10051

[akihironitta]

It seems so. For anyone seeing this issue, a quick workaround is to downgrade protobuf version as shown in the error:

pip install "protobuf<4.21.0"

---

EDIT on June 9, 2022:
Upgrading tensorboard to 2.9.1 or later should also work:

pip install tensorboard -U

[adamjstewart]

This is an issue with tensorboard, not with pytorch-lightning, correct? Can someone open an issue with tensorboard?

[ashnair1]

Tensorboard devs have pinned protobuf - tensorflow/tensorboard@fd4f5ff

[akihironitta]

(To protect our CI from errors like this for stability, ) This issue is not something we could've avoided with our current dependency freezing logic using dependabot because Lightning doesn't have a direct dependency to protobuf and tensorboard releases didn't set the upper version of protobuf.

[akihironitta]

Quickly checked our CI jobs and saw a few failures due to this issue, but they all seem to work fine now possibly because tensorboard pinned protobuf version already.