sequence length is longer than the specified maximum sequence length for this model (2842 > 2048). Running this sequence through the model will result in indexing errors

[jinz2014]

Hi

The command produces the error shown below. Thanks for your suggestion.

python run_clm_no_trainer.py
    --model EleutherAI/gpt-j-6B \
    --dataset NeelNanda/pile-10k \
    --seed 0 \
    --quantize \
    --approach weight_only \
    --woq_algo GPTQ \
    --woq_bits 4 \
    --woq_scheme asym \
    --woq_group_size 128 \
    --gptq_pad_max_length 2048 \
    --gptq_use_max_length \
    --gptq_debug 

sequence length is longer than the specified maximum sequence length for this model (2842 > 2048). Running this sequence through the model will result in indexing errors

[yiliu30]

Hi @jinz2014 , as we have enabled this example on both the 2.x APIs and 3.x APIs, could you please provide the folder path of the example so that we can replicate the issue?

[jinz2014]

Sorry. The folder path of the example is https://github.com/intel/neural-compressor/tree/master/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/llm

Thanks.

[yiliu30]

Thanks for providing the details, it seems to be a warning rather than an error.
Our implementation will double-check the input length and select partial input if it exceeds model_max_length.
If you are interested in the details, please refer to the code.

neural-compressor/neural_compressor/adaptor/torch_utils/gptq.py

Lines 341 to 349 in 4239d36

	elif batch[0].shape[-1] > unified_length:
	i = random.randint(0, batch[0].shape[-1] - unified_length - 1)
	j = i + unified_length
	batch_final = []
	for item in batch:
	if isinstance(item, torch.Tensor) and item.shape.__len__() == 2:
	batch_final.append(item[:, i:j])
	else:
	batch_final.append(item)

[yiliu30]

Hi @jinz2014, since we haven't received a response for an extended period. we'll assume the issue has been resolved. Feel free to reopen it if needed. Thank you