[SYCL] Support newer non linear quantization

[qnixsynapse]

cc: @abhilash1910 @airMeng

Also important!

[airMeng]

we haven't supported iq3_xxs, but you can try to implement yourself based on CUDA code. Feel free to ping us here if you encounter any issues

https://github.com/ggerganov/llama.cpp/blob/15499eb94227401bdc8875da6eb85c15d37068f7/ggml-sycl.cpp#L14970-L14972

[qnixsynapse]

I see. Thanks. Should I close this then?

[airMeng]

Also important!

Thank you for informing!

I see. Thanks. Should I close this then?

up to you

[qnixsynapse]

I am reopening this. I may/may not work on this since I am not a C person. I wish I could set the label to "help-wanted".

[qnixsynapse]

@ggerganov Thank you!

[airMeng]

#5862

Can you try this PR and report the results?

[qnixsynapse]

Got a ggml assert while trying to test an IQ3_XXS quantized gguf:

...
...
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
...
...
GGML_ASSERT: llama.cpp/ggml-sycl.cpp:14097: to_fp16_sycl != nullptr

The other non IQ quants looking good although I didn't understand the warning that I am getting about ext_intel_free_memory.

[airMeng]

what model are you using? can you share a link?

[qnixsynapse]

This time I used Eris 7B to test IQ3.

And for linear quants I used Mistral 7B.

[NeoZhangJianyu]

Got a ggml assert while trying to test an IQ3_XXS quantized gguf:

...
...
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
...
...
GGML_ASSERT: llama.cpp/ggml-sycl.cpp:14097: to_fp16_sycl != nullptr

The other non IQ quants looking good although I didn't understand the warning that I am getting about ext_intel_free_memory.

export/set ZES_ENABLE_SYSMAN=1 to support ext_intel_free_memory() to get free memory.
It's not important if your GPU is not shared.