Improve pre-processed image size for QwenVL

[ngxson]

Extracted the discussion from:

• llama: store mrope data in KV cell #16825 (comment)
• llama: store mrope data in KV cell #16825 (comment)
• llama: store mrope data in KV cell #16825 (comment)

In order for the bbox to be correct, I'm thinking about:

• Implement the correct max_pixels / min_pixels from the original config
• Pad right/bottom corner of the image if we require to upscale (to multiple of 2*patch_size). This will guarantee that the x/y coordinates stay unchanged. But we also need to check if the original implementation actually use this strategy or not.

CC @broadbit-hu @theo77186

[pwilkin]

I think this might be the key method from the reference implementation:

def smart_resize(
    height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280
):
    """Rescales the image so that the following conditions are met:

    1. Both dimensions (height and width) are divisible by 'factor'.

    2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].

    3. The aspect ratio of the image is maintained as closely as possible.

    """
    if max(height, width) / min(height, width) > 200:
        raise ValueError(
            f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
        )
    h_bar = round(height / factor) * factor
    w_bar = round(width / factor) * factor
    if h_bar * w_bar > max_pixels:
        beta = math.sqrt((height * width) / max_pixels)
        h_bar = max(factor, math.floor(height / beta / factor) * factor)
        w_bar = max(factor, math.floor(width / beta / factor) * factor)
    elif h_bar * w_bar < min_pixels:
        beta = math.sqrt(min_pixels / (height * width))
        h_bar = math.ceil(height * beta / factor) * factor
        w_bar = math.ceil(width * beta / factor) * factor
    return h_bar, w_bar

[ngxson]

Yes, indeed the functions which handles the max_pixels / min_pixels is pretty much the same across multiple models: https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20max_pixels&type=code

[pwilkin]

@ngxson I checked the code for 2.5 VL and it explicitly references the 2 VL image processor.

[ngxson]

Note that it will be not just an improvement to QwenVL, but other models that use the max_pixels / min_pixels strategy will also reuse the same code from this fix

[tarruda]

On master branch:

On this branch (I meant to comment in the PR but ended up doing it here):

This is qwen3-vl-8b-instruct UD-Q4_K_XL quant from unsloth (mmproj-F32).

Using this test page, but I changed it in two ways:

• Applied this fix
• Modified the request to return structured output in the format required by the parsing code.

Both using these inference parameter:

{
  "temperature": 0,
  "top_k": 1,
  "max_tokens": 24000,
  "repeat_penalty": 1,
  "cache_prompt": false,
  "samplers": [
    "top_k",
    "temperature"
  ]
}

[tarruda]

Dummy invoice I used above:

[tarruda]

Uploaded my modified test page: https://gist.github.com/tarruda/09dcbc44c2be0cbc96a4b9809942d503

[ngxson]

I think your test script have problems. It's better to draw the bbox using HTML canvas as it will be pixel-accurate. Don't trust CSS pixel

Also, try uploading a raw JPG/PNG without modifications. The image you provided was a SVG.

[tarruda]

@ngxson did you use my version of the test page? https://gist.github.com/tarruda/09dcbc44c2be0cbc96a4b9809942d503

From your screenshot, it seems like the coordinates were not adjusted on the client.

This is the result with Qwen3-VL-32B-Instruct q8_0 on master branch, seems pretty accurate to me:

Here's the result on #16878

The original dummy invoice was svg, I converted to png using PIL.
Here's the same png file I tested on:

[tarruda]

@ngxson also my llama-server script:

#!/bin/sh -e

model=Qwen3-VL-32B-Instruct-GGUF/q8_0/Qwen3-VL-32B-Instruct-Q8_0.gguf
mmproj=Qwen3-VL-32B-Instruct-GGUF/q8_0/mmproj-F32.gguf

llama-server --no-mmap --no-warmup --model $model --mmproj $mmproj --ctx-size 131072 --jinja -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --host 0.0.0.0