LLAVA v1.6

[enrico07]

Hello,

I was wondering if it is compatible with LLAVA v1.6 34B. If it is so, how can I implement it? Should I use the same code of LLAVA v1.5?

[Robzz]

Seems like it's compatible, I just tried running the llava-v1.6-mistral-7b.Q5_K_M.gguf model using the LLaVa 1.5 example code and it's been working fine so far.

Edit: I gave a 34B model a try (using the Q4_K_M quants from the ollama models library) a try and got a segfault, needs investigation.

[enrico07]

I tried to use the 34B version. What I got at the beginning is that the model had a lot of hallucinations at the end like strange comments, emojis and also sometimes it appends the chats between the system and the user.

I solved it by controlling the output of the model by asking in the prompt to add a special token at the end and using the "stop" parameter to stop writing after that token.

My doubt is that if the right structure of LLAVA-v1.6 prompt is used and if the new technique of LLAVA-v1.6 of splitting the images into grids is exploited.