Can't trace any model with pytorch-transformers 1.2

[Bycob]

🐛 Bug

I get an error whenever I try to trace any model from pytorch-transformers 1.2.0.
When I roll back to 1.1 everything is fine.

To Reproduce

from pytorch_transformers import BertModel
import torch

model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
model.to('cuda')
model.eval()
ids = torch.LongTensor([[1, 2, 3]]).cuda()
tok = torch.zeros_like(ids)
att = torch.ones_like(ids)
torch.jit.trace(model, (ids, tok, att, ids))

This script produces the following error:

Traceback (most recent call last):
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 545, in run_mod_and_filter_tensor_outputs
    outs = wrap_retval(mod(*_clone_inputs(inputs)))
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
RuntimeError: r ASSERT FAILED at /pytorch/aten/src/ATen/core/jit_type.h:142, please report a bug to PyTorch. (expect at /pytorch/aten/src/ATen/core/jit_type.h:142)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f273aa91441 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f273aa90d7a in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
frame #2: std::shared_ptr<c10::DimensionedTensorType const> c10::Type::expect<c10::DimensionedTensorType const>() + 0x140 (0x7f27397ff810 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #3: torch::jit::fuser::compileKernel(torch::jit::fuser::KernelSpec const&, torch::jit::fuser::ArgSpec const&, std::vector<long, std::allocator<long> > const&, c10::Device) + 0xa5a (0x7f27397fbdca in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #4: torch::jit::fuser::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&, std::string*) + 0x5b0 (0x7f2739803c20 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #5: torch::jit::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x13 (0x7f2739733bc3 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #6: <unknown function> + 0xb2b066 (0x7f273973d066 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #7: <unknown function> + 0xa8ebe6 (0x7f27396a0be6 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #8: torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x22 (0x7f273969c202 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #9: <unknown function> + 0xa7685d (0x7f273968885d in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
frame #10: <unknown function> + 0x457617 (0x7f277a2cd617 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x130d0c (0x7f2779fa6d0c in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #14: python3() [0x4fbfce]
frame #16: python3() [0x574db6]
frame #20: python3() [0x4ec2e3]
frame #22: python3() [0x4fbfce]
frame #24: python3() [0x574db6]
frame #27: python3() [0x5401ef]
frame #30: python3() [0x4ec3f7]
frame #33: python3() [0x5401ef]
frame #35: python3() [0x53fc97]
frame #37: python3() [0x53fc97]
frame #39: python3() [0x60cb42]
frame #44: __libc_start_main + 0xf0 (0x7f279df7d830 in /lib/x86_64-linux-gnu/libc.so.6)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "session.py", line 10, in <module>
    torch.jit.trace(model, (ids, tok, att, ids))
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 702, in trace
    _check_trace([example_inputs], func, executor_options, traced, check_tolerance, _force_outplace)
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/autograd/grad_mode.py", line 43, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 583, in _check_trace
    traced_outs = run_mod_and_filter_tensor_outputs(module, inputs, 'trace')
  File "/home/louisj/.local/lib/python3.5/site-packages/torch/jit/__init__.py", line 551, in run_mod_and_filter_tensor_outputs
    ' with test inputs.\nException:\n' + indent(str(e)))
torch.jit.TracingCheckError: Tracing failed sanity checks!
Encountered an exception while running the trace with test inputs.
Exception:
	r ASSERT FAILED at /pytorch/aten/src/ATen/core/jit_type.h:142, please report a bug to PyTorch. (expect at /pytorch/aten/src/ATen/core/jit_type.h:142)
	frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f273aa91441 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
	frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f273aa90d7a in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libc10.so)
	frame #2: std::shared_ptr<c10::DimensionedTensorType const> c10::Type::expect<c10::DimensionedTensorType const>() + 0x140 (0x7f27397ff810 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #3: torch::jit::fuser::compileKernel(torch::jit::fuser::KernelSpec const&, torch::jit::fuser::ArgSpec const&, std::vector<long, std::allocator<long> > const&, c10::Device) + 0xa5a (0x7f27397fbdca in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #4: torch::jit::fuser::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&, std::string*) + 0x5b0 (0x7f2739803c20 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #5: torch::jit::runFusion(long, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x13 (0x7f2739733bc3 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #6: <unknown function> + 0xb2b066 (0x7f273973d066 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #7: <unknown function> + 0xa8ebe6 (0x7f27396a0be6 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #8: torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 0x22 (0x7f273969c202 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #9: <unknown function> + 0xa7685d (0x7f273968885d in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch.so.1)
	frame #10: <unknown function> + 0x457617 (0x7f277a2cd617 in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
	frame #11: <unknown function> + 0x130d0c (0x7f2779fa6d0c in /home/louisj/.local/lib/python3.5/site-packages/torch/lib/libtorch_python.so)
	<omitting python frames>
	frame #14: python3() [0x4fbfce]
	frame #16: python3() [0x574db6]
	frame #20: python3() [0x4ec2e3]
	frame #22: python3() [0x4fbfce]
	frame #24: python3() [0x574db6]
	frame #27: python3() [0x5401ef]
	frame #30: python3() [0x4ec3f7]
	frame #33: python3() [0x5401ef]
	frame #35: python3() [0x53fc97]
	frame #37: python3() [0x53fc97]
	frame #39: python3() [0x60cb42]
	frame #44: __libc_start_main + 0xf0 (0x7f279df7d830 in /lib/x86_64-linux-gnu/libc.so.6)

Environment

• OS: Ubuntu 16.04.6 LTS
• Python version: Python 3.5.2
• PyTorch version: '1.1.0'
• PyTorch Transformers version (or branch): '1.2.0'
• Using GPU ? yes
• Distributed of parallel setup ? no
• Any other relevant information:
  I installed pytorch-transformers with pip:

pip3 install --user pytorch-transformers==1.2

[LysandreJik]

Hi, thank you for reporting this. I can reproduce it on my side. It seems to be a problem relative to the model being on cuda, as it doesn't fail if you don't put the model/ids on cuda.

This doesn't fail:

from pytorch_transformers import BertModel
import torch

model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
model.eval()
ids = torch.LongTensor([[1, 2, 3]])
tok = torch.zeros_like(ids)
att = torch.ones_like(ids)
torch.jit.trace(model, (ids, tok, att, ids))

[Bycob]

Traced models on cpu may not be convertible to cuda due to hard coded tensor creation in torchscript. I tried a while ago and it wasn't working, and I found an issue (#1010) referencing a similar problem.

[thomwolf]

Yes, with the merge of #1195, the jit tracing issue of #1010 should now be fixed on master.

You test installing from source and see if it solves your issue.

[Bycob]

I ran BERT on GPU after tracing it on CPU, and it works fine! Any information about when this fix will be available from the official distribution?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gitabtion]

Hi, while the model outputs has multi outputs like sequence_output and pooled_output，how to get one of them in C++?