🐛 [Bug] perf gap reduce on BERT

[zewenli98]

Bug Description

Compare the perf of Torch-TRT against ONNX-TRT.

In fp32:

1. Skipping constant folding of embedding layers can reduce engine size. It doesn't affect latency or precision
2. Disabling linear decomposition + adding linear converter doesn't affect latency
3. opt_level=3 or 5 get almost same latency
4. onnx-trt takes much longer in compile time
5. torch-trt is ~2.5% slower than onnx-trt

In fp16:

1. Skipping constant folding of embedding layers can reduce engine size. It doesn't affect latency or precision
2. Disabling linear decomposition + adding linear converter reduces ~18% latency
3. opt_level=3 or 5 get almost same latency
4. onnx-trt takes much longer in compile time
5. torch-trt is ~11% slower than onnx-trt

To Reproduce

run perf_run.py script

[zewenli98]

linear converter PR + fixing broadcast PR can eliminate perf gap on BERT in both fp32 and fp16

[zewenli98]

When embedding layer is constant folded, the engine size will become larger and larger as inputs increase, while the latency is not affected though. Currently, I only observed the issue on BERT but not on other models.

[zewenli98]

engine size issue will follow up in #3710