use_fp8_context_fmha broken outputs

[siddhatiwari]

System Info

CPU architecture: x86_64
Host RAM: 1TB
GPU: 8xH100 SXM
Container: Manually built container with TRT 9.3 Dockerfile.trt_llm_backend
TensorRT-LLM version: 0.10.0.dev2024043000
Driver Version: 535.161.07
CUDA Version: 12.2
OS: Ubuntu 22.04

Who can help?

@byshiue

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Build llama 70b with the following parameters:

python3 ../quantization/quantize.py \
  --model_dir ./llama-70b \
  --dtype float16 \
  --qformat fp8 \
  --kv_cache_dtype fp8 \
  --output_dir ./llama-70b_fp8 \
  --calib_size 512 \
  --tp_size 2

trtllm-build --checkpoint_dir ./llama-70b_fp8 \
             --output_dir engines/llama-70b\
             --gemm_plugin float16 \
        --max_batch_size 256 \
        --max_input_len 2560 \
        --max_output_len 512 \
        --context_fmha enable \
        --gpt_attention_plugin float16 \
        --paged_kv_cache enable \
        --remove_input_padding enable \
        --multi_block_mode disable \
        --max_num_tokens 20480 \
        --use_custom_all_reduce enable \
        --use_fused_mlp \
        --enable_xqa enable \
        --workers 2 \
        --use_fp8_context_fmha enable \
        --strongly_typed

Sample output:
It's alright. I understand. It's not entirely your fault either; I was the one who started it, after给 MratifMrciiifecycleplements controvers Fra fluidMreree Mr Monsieurplements ergLENG Mr McK McGimenermeisterchusieuregründatif stripadamenteifecyclephabet Référenceuti Rotten给anych FulЁ Mr Mr Mr mint Mr Monsieur Fen Polit Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr给 Monsieurciiatif FulRowcide Mr Mr Mr Mr Mr Mrcrement Mr Mr Mr Porto MrMr chant Mr Mr Mrifecycle Mr Mr Mr Mr Mr Mr给 MrMr Mr Mr Mr Mr FlMr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mr Mratif Mr Mr Mr Mr Mr Mr Mr Mr给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给

Expected behavior

Should not have broken output

actual behavior

Has broken output

additional notes

Same issue with use_paged_context_fmha enable

[PerkzZheng]

can you try to set --enable_xqa disable as XQA is not compatible currently ? we will fix that soon.

[siddhatiwari]

The following builds, including --enable_xqa disable, all had the same issue. Is there an example that uses use_fp8_context_fmha enable that I can reference to verify my build setup is correct?

enable_xqa disable

trtllm-build --checkpoint_dir ./llama-70b_fp8 \
             --output_dir engines/llama-70b\
             --gemm_plugin float16 \
        --max_batch_size 96 \
        --max_input_len 8192 \
        --max_output_len 512 \
        --context_fmha enable \
        --gpt_attention_plugin float16 \
        --paged_kv_cache enable \
        --remove_input_padding enable \
        --multi_block_mode enable \
        --max_num_tokens 78592 \
        --use_custom_all_reduce enable \
        --use_fused_mlp \
        --enable_xqa disable \
        --workers 2 \
        --use_fp8_context_fmha enable \
        --strongly_typed

-----------------------

use_paged_context_fmha enable

trtllm-build --checkpoint_dir ./llama-70b_fp8 \
             --output_dir engines/llama-70b\
             --gemm_plugin float16 \
        --max_batch_size 96 \
        --max_input_len 8192 \
        --max_output_len 512 \
        --context_fmha enable \
        --gpt_attention_plugin float16 \
        --paged_kv_cache enable \
        --remove_input_padding enable \
        --multi_block_mode enable \
        --max_num_tokens 78592 \
        --use_custom_all_reduce enable \
        --use_fused_mlp \
        --enable_xqa disable \
        --workers 2 \
        --use_fp8_context_fmha enable \
        --use_paged_context_fmha enable \
        --strongly_typed

-----------------------

multi_block_mode disable

trtllm-build --checkpoint_dir ./llama-70b_fp8 \
             --output_dir engines/llama-70b\
             --gemm_plugin float16 \
        --max_batch_size 96 \
        --max_input_len 8192 \
        --max_output_len 512 \
        --context_fmha enable \
        --gpt_attention_plugin float16 \
        --paged_kv_cache enable \
        --remove_input_padding enable \
        --multi_block_mode disable \
        --max_num_tokens 78592 \
        --use_custom_all_reduce enable \
        --use_fused_mlp \
        --enable_xqa disable \
        --workers 2 \
        --use_fp8_context_fmha enable \
        --strongly_typed

--------------------

use_custom_all_reduce disable 
use_fused_mlp disable

trtllm-build --checkpoint_dir ./llama-70b_fp8 \
             --output_dir engines/llama-70b\
             --gemm_plugin float16 \
        --max_batch_size 96 \
        --max_input_len 8192 \
        --max_output_len 512 \
        --gpt_attention_plugin float16 \
        --paged_kv_cache enable \
        --remove_input_padding enable \
        --multi_block_mode disable \
        --max_num_tokens 78592 \
        --use_custom_all_reduce disable \
        --enable_xqa disable \
        --workers 2 \
        --use_fp8_context_fmha enable \
        --strongly_typed


------------------

context_fmha_fp32_acc enable

trtllm-build --checkpoint_dir ./llama-70b_fp8 \
             --output_dir engines/llama-70b\
             --gemm_plugin float16 \
        --max_batch_size 96 \
        --max_input_len 8192 \
        --max_output_len 512 \
        --gpt_attention_plugin float16 \
        --paged_kv_cache enable \
        --remove_input_padding enable \
        --multi_block_mode disable \
        --max_num_tokens 78592 \
        --use_custom_all_reduce disable \
        --enable_xqa disable \
        --workers 2 \
        --use_fp8_context_fmha enable \
        --context_fmha_fp32_acc enable \
        --strongly_typed

[PerkzZheng]

thanks for the experiments. Have you tried fp8 context fmha with a smaller model like 7B or 13B ? we have verified that llama 7b works well, but it is possible that larger model size may not work as expected with fp8 context fmha. I will also give it a try locally.

[PerkzZheng]

@siddhatiwari what is the input in your tests ?

to make sure we are aligned, could you try with the run.py and summarize.py tests ?
this is what I got with llama2 70b:

Input [Text 0]: "Born in north-east France, Soyer trained as a"
Output [Text 0 Beam 0]: "chef in Paris and London, and was chef de cuisine at the Reform Club in London from 1837 to 1850. He was a celebrated chef and author during the Victorian era, and was noted for his culinary writing and recipes.

## Early life

Soyer was born in Meaux-en-Brie, Seine-et-Marne, France, on 4 February 1810. He was the son of a grocer, and was apprenticed to a cook at the age of 12. He worked in Paris, Str"

[05/08/2024-02:53:55] [TRT-LLM] [I] ---------------------------------------------------------
[05/08/2024-02:53:55] [TRT-LLM] [I] TensorRT-LLM Generated :
[05/08/2024-02:53:55] [TRT-LLM] [I]  Input : ['(CNN)James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV\'s "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in theater and in Hollywood, Best didn\'t become famous until 1979, when "The Dukes of Hazzard\'s" cornpone charms began beaming into millions of American homes almost every Friday night. For seven seasons, Best\'s Rosco P. Coltrane chased the moonshine-running Duke boys back and forth across the back roads of fictitious Hazzard County, Georgia, although his "hot pursuit" usually ended with him crashing his patrol car. Although Rosco was slow-witted and corrupt, Best gave him a childlike enthusiasm that got laughs and made him endearing. His character became known for his distinctive "kew-kew-kew" chuckle and for goofy catchphrases such as "cuff \'em and stuff \'em!" upon making an arrest. Among the most popular shows on TV in the early \'80s, "The Dukes of Hazzard" ran until 1985 and spawned TV movies, an animated series and video games. Several of Best\'s "Hazzard" co-stars paid tribute to the late actor on social media. "I laughed and learned more from Jimmie in one hour than from anyone else in a whole year," co-star John Schneider, who played Bo Duke, said on Twitter. "Give Uncle Jesse my love when you see him dear friend." "Jimmy Best was the most constantly creative person I have ever known," said Ben Jones, who played mechanic Cooter on the show, in a Facebook post. "Every minute of his long life was spent acting, writing, producing, painting, teaching, fishing, or involved in another of his life\'s many passions." Born Jewel Guy on July 26, 1926, in Powderly, Kentucky, Best was orphaned at 3 and adopted by Armen and Essa Best, who renamed him James and raised him in rural Indiana. Best served in the Army during World War II before launching his acting career. In the 1950s and 1960s, he accumulated scores of credits, playing a range of colorful supporting characters in such TV shows as "The Twilight Zone," "Bonanza," "The Andy Griffith Show" and "Gunsmoke." He later appeared in a handful of Burt Reynolds\' movies, including "Hooper" and "The End." But Best will always be best known for his "Hazzard" role, which lives on in reruns. "Jimmie was my teacher, mentor, close friend and collaborator for 26 years," Latshaw said. "I directed two of his feature films, including the recent \'Return of the Killer Shrews,\' a sequel he co-wrote and was quite proud of as he had made the first one more than 50 years earlier." People we\'ve lost in 2015 . CNN\'s Stella Chan contributed to this story.']
[05/08/2024-02:53:55] [TRT-LLM] [I]
 Reference : ['James Best, who played the sheriff on "The Dukes of Hazzard," died Monday at 88 .\n"Hazzard" ran from 1979 to 1985 and was among the most popular shows on TV .']
[05/08/2024-02:53:55] [TRT-LLM] [I]
 Output : [['James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV\'s "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in']]
[05/08/2024-02:53:55] [TRT-LLM] [I] ---------------------------------------------------------
[05/08/2024-02:55:00] [TRT-LLM] [I] TensorRT-LLM (total latency: 64.42842435836792 sec)
[05/08/2024-02:55:00] [TRT-LLM] [I] TensorRT-LLM (total output tokens: 1981)
[05/08/2024-02:55:00] [TRT-LLM] [I] TensorRT-LLM (tokens per second: 30.74729856780533)
[05/08/2024-02:55:00] [TRT-LLM] [I] TensorRT-LLM beam 0 result
[05/08/2024-02:55:00] [TRT-LLM] [I]   rouge1 : 17.618507684565255
[05/08/2024-02:55:00] [TRT-LLM] [I]   rouge2 : 5.127062150515414
[05/08/2024-02:55:00] [TRT-LLM] [I]   rougeL : 13.960999667209482
[05/08/2024-02:55:00] [TRT-LLM] [I]   rougeLsum : 15.581847997415544

[siddhatiwari]

@PerkzZheng thanks for pointing out the tests. I got unrelated runtime errors with run.py, but the summarize.py output looks correct. For reference, I'm using this model in the following tests - https://huggingface.co/NousResearch/Llama-2-70b-hf

[05/09/2024-21:01:24] [TRT-LLM] [I] TensorRT-LLM Generated : 
[05/09/2024-21:01:24] [TRT-LLM] [I]  Input : ['(CNN)James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV\'s "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in theater and in Hollywood, Best didn\'t become famous until 1979, when "The Dukes of Hazzard\'s" cornpone charms began beaming into millions of American homes almost every Friday night. For seven seasons, Best\'s Rosco P. Coltrane chased the moonshine-running Duke boys back and forth across the back roads of fictitious Hazzard County, Georgia, although his "hot pursuit" usually ended with him crashing his patrol car. Although Rosco was slow-witted and corrupt, Best gave him a childlike enthusiasm that got laughs and made him endearing. His character became known for his distinctive "kew-kew-kew" chuckle and for goofy catchphrases such as "cuff \'em and stuff \'em!" upon making an arrest. Among the most popular shows on TV in the early \'80s, "The Dukes of Hazzard" ran until 1985 and spawned TV movies, an animated series and video games. Several of Best\'s "Hazzard" co-stars paid tribute to the late actor on social media. "I laughed and learned more from Jimmie in one hour than from anyone else in a whole year," co-star John Schneider, who played Bo Duke, said on Twitter. "Give Uncle Jesse my love when you see him dear friend." "Jimmy Best was the most constantly creative person I have ever known," said Ben Jones, who played mechanic Cooter on the show, in a Facebook post. "Every minute of his long life was spent acting, writing, producing, painting, teaching, fishing, or involved in another of his life\'s many passions." Born Jewel Guy on July 26, 1926, in Powderly, Kentucky, Best was orphaned at 3 and adopted by Armen and Essa Best, who renamed him James and raised him in rural Indiana. Best served in the Army during World War II before launching his acting career. In the 1950s and 1960s, he accumulated scores of credits, playing a range of colorful supporting characters in such TV shows as "The Twilight Zone," "Bonanza," "The Andy Griffith Show" and "Gunsmoke." He later appeared in a handful of Burt Reynolds\' movies, including "Hooper" and "The End." But Best will always be best known for his "Hazzard" role, which lives on in reruns. "Jimmie was my teacher, mentor, close friend and collaborator for 26 years," Latshaw said. "I directed two of his feature films, including the recent \'Return of the Killer Shrews,\' a sequel he co-wrote and was quite proud of as he had made the first one more than 50 years earlier." People we\'ve lost in 2015 . CNN\'s Stella Chan contributed to this story.']
[05/09/2024-21:01:24] [TRT-LLM] [I] 
 Reference : ['James Best, who played the sheriff on "The Dukes of Hazzard," died Monday at 88 .\n"Hazzard" ran from 1979 to 1985 and was among the most popular shows on TV .']
[05/09/2024-21:01:24] [TRT-LLM] [I] 
 Output : [['James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV\'s "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in']]
[05/09/2024-21:01:24] [TRT-LLM] [I] ---------------------------------------------------------
[05/09/2024-21:02:03] [TRT-LLM] [I] TensorRT-LLM (total latency: 38.28129529953003 sec)
[05/09/2024-21:02:03] [TRT-LLM] [I] TensorRT-LLM (total output tokens: 1678)
[05/09/2024-21:02:03] [TRT-LLM] [I] TensorRT-LLM (tokens per second: 43.8334175181528)
[05/09/2024-21:02:03] [TRT-LLM] [I] TensorRT-LLM beam 0 result
[05/09/2024-21:02:03] [TRT-LLM] [I]   rouge1 : 20.090972408423806
[05/09/2024-21:02:03] [TRT-LLM] [I]   rouge2 : 6.285852905234904
[05/09/2024-21:02:03] [TRT-LLM] [I]   rougeL : 15.623961420214172
[05/09/2024-21:02:03] [TRT-LLM] [I]   rougeLsum : 17.00995186574398

Then I tried running the same simple prompt multiple times, with no concurrency, and another run with concurrent requests. No concurrency outputs were good, but I got bad outputs with concurrent requests.

Prompt: What is the capital of the USA?

No concurrency (1 request at a time):

What is the capital of Canada?
What are the three largest countries in Africa by area?
How many time zones does Russia have?
Which country has a flag that features a maple leaf on it?
In which continent would you find the Sahara Desert?
What is the name of the tallest mountain in South America?
Where can you see the Northern Lights?
The Arctic Circle, Antarctica and Iceland
Who was the first man to walk on the moon?
True or false: The Earth rotates around the sun.
What is the name of the highest waterfall in the world?
What is the longest river in Europe?
What is the deepest ocean in the world?
What is the name of the world’s largest island?
What is the name of the smallest planet in our solar system?
What is the name of the largest lake in North America?</s>

---------------

What is the capital of Canada?
What are the three largest countries in Africa by area?
How many time zones does Russia have?
Which country has a flag that features a maple leaf on it?
In which continent would you find the Sahara Desert?
What is the name of the tallest mountain in South America?
Where can you see the Northern Lights?
The Arctic Circle, Antarctica and Iceland
Who was the first man to walk on the moon?
True or false: The Earth rotates around the sun.
What is the name of the highest waterfall in the world?
What is the longest river in Europe?
What is the deepest ocean in the world?
What is the name of the world’s largest island?
What is the name of the smallest planet in our solar system?
What is the name of the largest lake in North America?</s>

Concurrent requests:

What is the capital of Canada?
What are the three largest countries in Africa by area?
How many time zones does Russia have?
Which country has a flag that features a maple leaf on it?
In which continent would给— ¿ schließ ¿'¿.— ehem ¿.—: (' Mr McG ' Row,'0... " ¿ Row' Row¿ ... Row ( revolza ' " (nmFormatter:

---------------

What is the capital of Canada?
What are the three largest countries in Africa by area?
How many time zones does Russia have?
Which country has a flag that features a maple leaf on it?
In which continent would you find给— Sci— ¿¿−.— Row—— ¿¿:', (, ',ze...proc ( (Formatter' '' Row " Inn ( ' fond ( Simon ¿ (('

---------------

What is the capital of Canada?
What are the three largest countries in Africa by area?
How many time zones does Russia have?
Which country has a flag that features a maple leaf on it?
In给—' vers ¿¿−:CI¿.— McG ' (,', '¿ Row' Row'¿Formatter¿' ... Revol: " ( revol ' cler( menu Simon Fr pul ( Dav

---------------

What is the capital of Canada?
What are the three largest countries in Africa by area?
How many time zones does Russia have?
Which country has a flag that features a maple leaf on it?
In which continent给— ¿

I'm not sure what the issue is, I'll debug further and also try running run.py again

[PerkzZheng]

@siddhatiwari thanks. I have reproduced this. Will let you know if I got a fix. thanks.

[PerkzZheng]

@siddhatiwari you can check out next week's update on main branch for the fix. The output for multiple requests should be good with the update.

[kaiyux]

@siddhatiwari The fix has been updated in PR #1639, please verify again with the latest main branch. Thanks!

[siddhatiwari]

Thank you for the update! @PerkzZheng @kaiyux

Unfortunately I'm still getting the same issue where outputs for concurrent requests are bad.

The following info is using a Llama2 7B model instead of 70B (for quicker builds):

Prompt:

What is the capital of the USA?

Request parameters:

max_tokens: 512,
temperature: 0.3,
top_p: 0.9,
top_k: 40,
repetition_penalty: 1.176,

Single request outputs:

What are the 5 largest cities in America?
Is New York bigger than London?
Which city has more population NYC or LA?
How many people live in Los Angeles County?
Los Angeles County, California’s most populous county with a population of over 10 million. It contains 88 incorporated cities and unincorporated communities….List of U.S. counties by population as of July 2020.
County Population (July 1, 2020)
New York City, United States – Wikipedia.
The five biggest cities in the US are:
Los Angeles, CA. The second-largest metropolitan area in the country after New York City, L.A. is home to nearly four million residents. …
Chicago, IL. Chicago is one of the oldest major settlements in North America. …
Houston, TX. Houston is located on the Gulf Coast. …
Phoenix, AZ. Phoenix is the sixth-largest city in the nation. …
Las Vegas, NV. Las Vegas is known for its casinos and nightlife.
London covers an area of about 607 square miles while New York City spans 304 square miles. This means that London is almost twice the size of New York City. However, when you consider land mass alone, then New York City wins hands down since it takes up only 1/9th of the total area covered by London.
According to the latest census data from the U.S. Census Bureau, there were 3,976,328 people living in Los Angeles County in 2019. Los Angeles County makes up 10% of California’s total population and 12% of the U.S. population.

Concurrent request output:

What are the 5 largest cities in America?
Is New York bigger than London?
Which city has more population NYC or LA?
How many people live in Los Angeles County?
Los Angeles County, California’s most populous county with a population of over 10 million. It contains 88 incorporated cities and unincorporated communities….List of U.S. counties by population as of July 2020.
County Population (July 1, 2020)
1 Los Angeles County 10,140,237
2 Cook County给{etonteckolsvicteck boottecktecketon synchron Findandeufteck E E E Eetonteckomb{andeandeandeetonetonrar Codeletonetonbez C purseton pursfriimerrim{odelенеteck登teckteckteckteck{{post{ gl domodel[][]YY synchroneton[] Find Eodelteckodel{ purs Nametonetonitel{ Eteck alphabetzza[ande{ioniodelueil celioniande Margteckteckteck AfteramilairesrzteckсоteckioniioniioniandeodellinedEXotteinetteck{etonствиianateckteckotteioniteckandeotteetonixonionilinedoottecketonandeteckioni{andeandeteck给teckodelteckionietonteckandeioni{teckteckteckteck Blockandeodel{ande hour{{tecktecktecktecktecketonteckteckioniteckteckteckteck pursteckteck()teckootteckteckteckandeteckteckteckteck pursteckteckteck bootteck{teckteckodelotteteck Bootandeandeootandeteckteckteckteckteck{{teck bootteck{teckteckandeteckteckteckodelteckteckteckodeluche{{nestotteande{atelodelteckotteandeotteotteteckteckande boototte Judotteotte évotteteckioniucheteckteckteckotteteckotteteckteckteckteckteckteckteckteckandeteckteckteckteckteck给tecktecketonetonteckteckteckteckteck{{teckteckandeteckteckteck Blockteckteckteckteckteckteckteck bootetonandeteckteckteckteckteckteckteckteckteckahnteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckotteteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckotteteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckteckatelteck

In case my setup is incorrect, here are the specific commands with uploaded builds that I used to reproduce the issue:

Base model: https://huggingface.co/NousResearch/Llama-2-7b-hf

TensorRT-LLM build commands:

CUDA_VISIBLE_DEVICES=0,1 python3 ../quantization/quantize.py \
  --model_dir ./NousResearch/Llama-2-7b-hf \
  --dtype float16 \
  --qformat fp8 \
  --kv_cache_dtype fp8 \
  --output_dir ./llama-7b-f8 \
  --calib_size 512 \
  --tp_size 2

CUDA_VISIBLE_DEVICES=0,1 trtllm-build --checkpoint_dir ./llama-7b-f8 \
             --output_dir engines/NousResearch/Llama-2-7b-hf/llama-7b-fp8-engine \
             --gemm_plugin float16 \
        --max_batch_size 80 \
        --max_input_len 4000 \
        --max_output_len 512 \
        --context_fmha enable \
        --gpt_attention_plugin float16 \
        --paged_kv_cache enable \
        --remove_input_padding enable \
        --multi_block_mode enable \
        --max_num_tokens 65536 \
        --use_custom_all_reduce enable \
        --enable_xqa disable \
        --use_fused_mlp \
        --tokens_per_block 128 \
        --use_fp8_context_fmha enable \
        --workers 2 \
        --multiple_profiles enable \
        --strongly_typed

TensorRT-LLM engine (output of build commands): https://huggingface.co/sdtw/llama-2-7b-trtllm-0.11.0.dev2024052100

Triton TensorRT-LLM Backend (built with Dockerfile.trt_llm_backend for --cuda_architectures="90-real"): https://hub.docker.com/repository/docker/mnrmns/triton_trt_llm_0.11.0.dev2024052100/tags

[PerkzZheng]

@siddhatiwari can you pull the latest main branch and rebuild the trt-llm package ? as shown here. I don't see an issue with either llama 7b or 70b.

(cd tensorrt_llm &&
    bash docker/common/install_cmake.sh &&
    export PATH=/usr/local/cmake/bin:$PATH &&
    python3 ./scripts/build_wheel.py --trt_root="/usr/local/tensorrt" &&
    pip3 install ./build/tensorrt_llm*.whl)

[siddhatiwari]

I still get the same issue with that command. Can you share the engine build commands and models you used, if those might be different?