tensorrt 7.0 In function importInstanceNormalization: [8] Assertion failed: !isDynamic(tensor_ptr->getDimensions()) && "InstanceNormalization does not support dynamic inputs!"

[lucasjinreal]

 In function importInstanceNormalization:
[8] Assertion failed: !isDynamic(tensor_ptr->getDimensions()) && "InstanceNormalization does not support dynamic inputs!"

Using tensorrt 7.0 pytorch 1.4 export in opset 11 got above error.

But, wired things is that, previous can be converted when using opset 10,

[arjunbhargava]

I'm having a similar issue, but also with Opset 10. GroupNormalization is exported as an InstanceNormalization (which makes sense as a workaround), however TRT7 seems to introduce this new error. For some reason, a single normalization layer has no issue but one we add another the engine fails to export. I've attached a reproducible minimal error case here:

import numpy as np
import onnx
import onnxruntime as rt
import torch
from torch import nn

import tensorrt as trt

TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE)


class TestModel(nn.Module):
    """Minimal model for reproducing error"""
    def __init__(self):
        super().__init__()
        # Block 1
        self.conv1 = nn.Conv2d(3, 32, 3, 1)
        self.gn = nn.GroupNorm(8, 32, affine=True)
        self.conv2 = nn.Conv2d(32, 32, 3, 1)

        # An additional GN breaks.
        self.gn2 = nn.GroupNorm(2, 32, affine=True)

    def forward(self, x):
        out = self.conv1(x)
        out = self.gn(out)
        out = self.conv2(out)

        out = self.gn2(out) # If we comment out this line, there are no issues
        return out


def export_model(image_width=224, image_height=224):
    """Export test-case ONNX Graph"""

    model_file = "test.onnx"
    unrolled_model = TestModel()
    unrolled_model.eval()

    img = torch.randn(1, 3, image_height, image_width)

    # Run ONNXification
    unrolled_model = unrolled_model.eval()
    torch.onnx.export(
        unrolled_model,
        img,
        model_file,
        verbose=False,
        do_constant_folding=False,
        opset_version=10,
        input_names=['input'],
        output_names=['output']
    )

    # Try to load model and check validity
    onnx_model = onnx.load(model_file)
    onnx.checker.check_model(onnx_model)
    print(onnx.helper.printable_graph(onnx_model.graph))

    # Run unrolled model
    with torch.no_grad():
        outputs = unrolled_model(img)
        outputs = outputs.numpy().squeeze()

    # Run ONNX model
    sess = rt.InferenceSession(model_file)
    input_name = sess.get_inputs()[0].name
    pred_onnx = sess.run(None, {input_name: img.numpy()})
    outputs_onnx = pred_onnx[0].squeeze()

    # Now compare output tensors
    assert np.allclose(outputs, outputs_onnx, atol=1e-03)
    return onnx_model


def GiB(val):
    return val * 1 << 30


def build_engine_onnx(model_file, workspace_GiB=2, max_batch_size=1):
    """Build engine from ONNX file"""
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
    ) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:

        builder.max_workspace_size = GiB(workspace_GiB)
        builder.max_batch_size = max_batch_size
        builder.fp16_mode = False

        with open(model_file, 'rb') as model:
            pars_res = parser.parse(model.read())
        if pars_res:
            print("OK ONNX")
            print('Number of errors: {}'.format(parser.num_errors))
        else:
            print('Number of errors: {}'.format(parser.num_errors))
            for err_idx in range(parser.num_errors):
                error = parser.get_error(err_idx)
                print('Description of the error: {}'.format(error.desc()))
                print('Line where the error occurred: {}'.format(error.line()))
                print('Error code: {}'.format(error.code()))
                print("Model was not parsed successfully")
            del parser
            return None

        engine = builder.build_cuda_engine(network)
        print("ENGINE = ", engine)
        print("NETWORK = ", network)
        return engine


def build_engine(onnx_graph):
    # Try to load model and check validity.
    onnx.checker.check_model(onnx_graph)
    sess = rt.InferenceSession("test.onnx")

    print('Inputs:')
    [print(i) for i in sess.get_inputs()]
    print('Outputs:')
    [print(o) for o in sess.get_outputs()]

    # Build TRT engine.
    engine = build_engine_onnx("test.onnx")
    assert engine is not None
    print("DONE")


if __name__ == "__main__":
    build_engine(export_model())

Environment detail:

 TORCHVISION_VERSION=0.5.0
 CUDNN_VERSION=7.6.3.30-1+cuda10.0
 NCCL_VERSION=2.5.6-1+cuda10.0
 TRT_VERSION=7.0.0.11

[lucasjinreal]

does there any updates for this?

[TengFeiHan0]

@jinfagang did you solve this problem? I am trying to speed up FCOS with TensorRT, then I got this kind of error.

[springyoung]

@TengFeiHan0 same as you.I use onnx2trt to parse my fcos.onnx model, got same error!

[dechunwang]

@jinfagang Any updates? I got the same issue with tensorRT 7.1 , pytorch 1.6
@rajeevsrao Will this to be fixed in the near future?

trtexec --onnx=model.onnx --verbose

[08/12/2020-10:24:43] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 393 for ONNX tensor: 393 [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:179: Reshape_204 [Reshape] outputs: [393 -> (-1, 32, -1)], [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:103: Parsing node: Constant_205 [Constant] [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:125: Constant_205 [Constant] inputs: [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:179: Constant_205 [Constant] outputs: [394 -> (32)], [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:103: Parsing node: Constant_206 [Constant] [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:125: Constant_206 [Constant] inputs: [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:179: Constant_206 [Constant] outputs: [395 -> (32)], [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:103: Parsing node: InstanceNormalization_207 [InstanceNormalization] [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:119: Searching for input: 393 [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:119: Searching for input: 394 [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:119: Searching for input: 395 [08/12/2020-10:24:43] [V] [TRT] ModelImporter.cpp:125: InstanceNormalization_207 [InstanceNormalization] inputs: [393 -> (-1, 32, -1)], [394 -> (32)], [395 -> (32)], ERROR: builtin_op_importers.cpp:1595 In function importInstanceNormalization: [8] Assertion failed: !isDynamic(tensorPtr->getDimensions()) && "InstanceNormalization does not support dynamic inputs!

[rajeevsrao]

@dechunwang @jinfagang yes, the fix will be in our next ONNX parser monthly update. Thanks.

[VincentGu11]

In TRT-7.1.3.4, I transfer FCOS with fpn also get the same problem:
ERROR: builtin_op_importers.cpp:1595 In function importInstanceNormalization:
[8] Assertion failed: !isDynamic(tensorPtr->getDimensions()) && "InstanceNormalization does not support dynamic inputs!"

[olferuk]

Any updates?

[kevinch-nv]

The InstanceNormalizationPlugin has been updated to support dynamic shapes in the latest TRT release (7.2). For those still following this thread please upgrade to TRT 7.2 and try importing your model again.

I will be closing this issue since this has been updated, if anyone is still having trouble with TRT 7.2 feel free to open a new issue.

[VincentGu11]

I use TRT 7.2.2.3 to convert dynamic input InstanceNormalization, I still has the same problem, and my cuda is 10.2.

[kevinch-nv]

@VincentGu11 can you open a new issue about this and attach the model you are having trouble with?