Can't build whisper engines with past two releases

[MahmoudAshraf97]

System Info

• CPU architecture: x86_64
• CPU/Host memory size: 512GB
• GPU properties
  • GPU name: 2xNVIDIA H100
• Libraries
  • TensorRT-LLM branch or tag: v0.16.0dev2024111900 or higher
  • Versions of TensorRT, Modelopt, CUDA, cuBLAS, etc. used: the ones in the requirements

Who can help?

@byshiue

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

trtllm-build  --checkpoint_dir trt/whisper_large-v2_weights_int8/encoder \
              --output_dir trt/whisper_large-v2_int8_2/encoder \
              --kv_cache_type paged \
              --moe_plugin disable \
              --enable_xqa enable \
              --max_batch_size 16 \
              --gemm_plugin disable \
              --bert_attention_plugin float16 \
              --max_input_len 3000 \
              --max_seq_len 3000 \
              --log_level debug \
              --enable_debug_output

Expected behavior

Engine builds successfully and fast

actual behavior

Whisper Large-V2 encoder hangs when building the engine
this is the last thing printed before it hangs: [11/27/2024-23:00:09] [TRT] [V] Disabling unused tactic source: JIT_CONVOLUTIONS
I tried waiting for around an hour but it still didn't complete and nvidia-smi memory usage for the process was constant

Tiny model builds fine but it's much slower than older versions

additional notes

None

[hello-11]

@MahmoudAshraf97 Could you use the latest version of TrtLLM?

[MahmoudAshraf97]

I tested 2024111900 and 2024112600,at the time of opening this issue, nothing new was released since

[yuekaizhang]

@MahmoudAshraf97 Sorry, just confirmed that it's a bug on H100. Will update here later.

[MahmoudAshraf97]

I tried 0.16.0.dev2024121000 and it's not been solved

[yuekaizhang]

@MahmoudAshraf97 Sorry, we're working on the issue. Would you mind using fp16 rather than int8 woq for now?

[MahmoudAshraf97]

@yuekaizhang I'm sticking to v0.15 for now, in the mean time this issue regarding fetching the encoder output from the executor response is also a priority so we can have word timestamps
#2338 (comment)