Benchmark AWQ and SmoothQuant within vLLM ecosystem

[namgyu-youn]

Overview

SmoothQuant and AWQ operate using a similar intuition, scaling up weights and scaling down activations. Therefore, benchmarks can share many logics within the vLLM ecosystem, using vllm/lm-eval and vllm/benchmarks/benchmark_latency.

Ideally, the following user guide and docs can be updated focusing on vLLM:

• AWQ: torchao/prototype/awq/example.py & torchao/prototype/awq/README.md
• SmoothQuant: torchao/prototype/smoothquant/example.py & torchao/prototype/smoothquant/README.md

Related Issue/PR

• Make SmoothQuant more General #2728 (comment)
• AWQ and SmoothQuant mit-han-lab/llm-awq#130

Resources

• https://blog.squeezebits.com/vllm-vs-tensorrtllm-2-towards-optimal-batching-for-llm-serving-31349
• https://vllm-ascend.readthedocs.io/en/latest/developer_guide/evaluation/using_lm_eval.html
• https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_latency.py
• How to benchmark vLLM a short tutorial vllm-project/vllm#7181

[namgyu-youn]

@jerryzh168 Hi, I have a question while addressing this issue.

For the benchmark, we need three metrics: 1) latency/throughput (using TTFT and TPOT), 2) memory usage, 3) accuracy (perplexity; ppl). We previously agreed on fully-customized datasets (tasks) for calibration flow in #2728 (comment), but wondering if get_calib_dataset() can substitute the calibration flow in TransformerEvalWrapper() although there is accuracy loss.

My reasoning is mostly about OSS (HF/vLLM) support:

1. lm_eval is only designed to evaluate accuracy (ppl), not evaluating 1) and 2), and is not compatible with vLLM/HF.
2. Based on the above, we should produce our own benchmark for latency and memory usage, making too complex.
3. Unlike lm_eval, the Hugging Face (HF) format is a widely used case, opening tickets to HF - Benchmarks and vLLM - Benchmark.

I am not certain how much the dataset in calibration flow determines its accuracy, but how about using get_calib_dataset() instead of lm_eval same as here? Finding another wrapper (lm_eval -> vLLM/HF) is a little bit weird, I think.

[jerryzh168]

@namgyu-youn please take a look at https://github.com/pytorch/ao/tree/main/.github/scripts/torchao_model_releases, you can add smoothquant there as well.

for your questions,

1. lm_eval is only designed to evaluate accuracy (ppl), not evaluating 1) and 2), and is not compatible with vLLM/HF.

lm_eval can use vllm as backend as well IIRC, but we are not using it yet, I haven't tested it with torchao models, but we can try that to see if it can speedup things, it is compatible with HF

2. Based on the above, we should produce our own benchmark for latency and memory usage, making too complex.

3. Unlike lm_eval, the Hugging Face (HF) format is a widely used case, opening tickets to HF - Benchmarks and vLLM - Benchmark.

please see the linked script

[namgyu-youn]

@namgyu-youn please take a look at https://github.com/pytorch/ao/tree/main/.github/scripts/torchao_model_releases, you can add smoothquant there as well.

Thanks for the leading; the linked script (following) resolves everything I have been wondering.

ao/.github/scripts/torchao_model_releases/quantize_and_upload.py

Lines 670 to 699 in 4dffb40

	if "AWQ" in quant:
	# awq will use torchao API directly
	assert quant == "AWQ-INT4", "Only support AWQ-INT4 for now"
	model = AutoModelForCausalLM.from_pretrained(
	model_to_quantize,
	device_map="auto",
	torch_dtype=torch.bfloat16,
	)
	tokenizer = AutoTokenizer.from_pretrained(model_id)
	base_config = Int4WeightOnlyConfig(group_size=128)
	quant_config = AWQConfig(base_config, step="prepare")
	quantize_(
	model,
	quant_config,
	)
	TransformerEvalWrapper(
	model=model,
	tokenizer=tokenizer,
	max_seq_length=max_seq_length,
	).run_eval(
	tasks=tasks,
	limit=calibration_limit,
	)
	quant_config = AWQConfig(base_config, step="convert")
	quantize_(model, quant_config)
	quantized_model = model
	quant_config = AWQConfig(base_config, step="prepare_for_loading")
	quantized_model.config.quantization_config = TorchAoConfig(quant_config)