bug: Continue downloading of partial file doesn't work

[freelerobot]

Cortex version

v75

Describe the Bug

1. cortex pull llama3.1 onnx version

2. download times out half way (2.2 GB out of 5+ GB downloaded)

3. ctrl-c to force quit download (stuck in that state)

4. pull llama3.1 again

(base) PS C:\Users\n\cortexcpp-nightly\models> cortex-nightly.exe pull llama3.1
Select an option
1. 8b-onnx
5. 8b-gguf
6. onnx
7. gguf

Select an option (1-4): 1

Validating download items, please wait..
Start downloading: genai_config.json
genai_config.json is already downloaded!
Re-download? [Y/n]: Y
Re-downloading..
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1740  100  1740    0     0   5134      0 --:--:-- --:--:-- --:--:--  5147
Start downloading: model.onnx
model.onnx is already downloaded!
Re-download? [Y/n]: Y
Re-downloading..
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68  100    68    0     0    212      0 --:--:-- --:--:-- --:--:--   211

5. Program correctly notices .data was not fully downloaded.
6. I resume download

Start downloading: model.onnx.data
Found unfinished download! Additional 69.00 B need to be downloaded.
Continue download [Y/n]: Y
Resuming download..
** Resuming transfer from byte position -1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    68    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl_easy_perform() failed: Requested range was not delivered by the server
Start downloading: model.yml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   893  100   893    0     0   2632      0 --:--:-- --:--:-- --:--:--  2642
Start downloading: special_tokens_map.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68  100    68    0     0    215      0 --:--:-- --:--:-- --:--:--   215
Start downloading: tokenizer.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68  100    68    0     0    216      0 --:--:-- --:--:-- --:--:--   217
Start downloading: tokenizer_config.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    68  100    68    0     0    213      0 --:--:-- --:--:-- --:--:--   213
Model llama3.1 downloaded successfully!

7. download finishes really quickly, like 2 seconds.
8. run the model. doesn't work.

(base) PS C:\Users\n\cortexcpp-nightly\models> cortex-nightly.exe run llama3.1
Starting server ...
20240921 08:25:51.227000 UTC 19540 INFO  Host: 127.0.0.1 Port: 3928
 - main.cc:32
Server started
gguf_init_from_file: failed to open '': 'Invalid argument'
{"timestamp":1726907151,"level":"ERROR","function":"LoadModel","line":186,"message":"llama.cpp unable to load model","model":""}
Model is not loaded yet!

Ignoring the fact that Cortex tried to run this with llamacpp.

ls \llama3.1 shows model.yml is 2.25GB
However, actual .data binary is 5.31 GB

What is your OS?

• MacOS
• Windows
• Linux

But this should be a OS agnostic issue. Expected to occur on mac as well

What engine are you running?

• cortex.llamacpp (default)
• cortex.tensorrt-llm (Nvidia GPUs)
• cortex.onnx (NPUs, DirectML)

[namchuai]

@0xSage , just for confirmation, could you please test the feature with llamacpp model only? I think onnx is not really ready atm and might cause the issue model load failed.

[freelerobot]

@namchuai GGUF also buggy:

Reproduce

1. stop download at ~95%

Resuming download..
** Resuming transfer from byte position 51707904
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1121  100  1121    0     0   3313      0 --:--:-- --:--:-- --:--:--  3326
 98 4643M   98 4576M    0     0  10.9M      0  0:07:02  0:06:56  0:00:06 3856k

2. resume download

(base) PS C:\Users\n\cortexcpp-nightly\models> cortex-nightly.exe pull llama3.1
Select an option
1. 8b-onnx
2. 8b-gguf
3. onnx
4. gguf

Select an option (1-4): 4
Validating download items, please wait..
Start downloading: model.gguf
Found unfinished download! Additional 4.58 GB need to be downloaded.
Continue download [Y/n]: Y
Resuming download..
** Resuming transfer from byte position -1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1113  100  1113    0     0   3873      0 --:--:-- --:--:-- --:--:--  3878
  0 4692M    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl_easy_perform() failed: Requested range was not delivered by the server

3. Get ERROR: Requested range was not delivered by the server

4. But model.yaml resumed downloading, when it shouldn't.

Start downloading: model.yml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   893  100   893    0     0   2459      0 --:--:-- --:--:-- --:--:--  2460
Model llama3.1 downloaded successfully!

5. Get success message:

Model llama3.1 downloaded successfully!

6. Model doesn't run (gguf file incomplete)

(base) PS C:\Users\n\cortexcpp-nightly\models> cd .\llama3.1\
(base) PS C:\Users\n\cortexcpp-nightly\models\llama3.1> ls

    Directory: C:\Users\n\cortexcpp-nightly\models\llama3.1

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           9/23/2024 12:20 PM     4852260864 model.gguf              >>>>> wrong file size, should be 4.92++
-a---           9/23/2024 12:21 PM            893 model.yml

[namchuai]

Interesting. I did not face that issue. Mine was successfully restarting the download. Let me investigate this.

[nguyenhoangthuan99]

I also tested, in my machine, but It still works fine

[freelerobot]

@namchuai yeah its a flaky issue. Happens more often on my windows (1/3 times) than mac. Have you tried reproducing on windows?

There are 2 distinct issues:

1. The calculation of remaining filesize is wrong: Found unfinished download! Additional 4.58 GB need to be downloaded. (this shows the full file size, but there were only ~<1GB remaining)
2. curl_easy_perform() failed: Requested range was not delivered by the server. What is this from - seems like a distinct error?

[namchuai]

1. The calculation of remaining filesize is wrong: Found unfinished download! Additional 4.58 GB need to be downloaded. (this shows the full file size, but there were only ~<1GB remaining)
   -> The calculation is based on two below things and we just subtract it.

• The remote file size (total, getting via content-length) header query.
• The local file size (downloaded).

Based on this, my guess is the content-length is somehow return an incorrect value. The local file size should be reliable enough.

2. curl_easy_perform() failed: Requested range was not delivered by the server. What is this from - seems like a distinct error?
   -> So, in order for resume download to be able to work, the server provides the file has to support byte range download. i.e. client only want to start download from a specific number of bytes.

The error is from libcurl. It indicates that the server does not support range request. Interesting thing is, as you said, it's flaky. Honestly, I haven't experienced this while downloading models.

I don't think we have the best way to handle this, I think we only can start download over.

[namchuai]

Ignore my previous comment. I found the issue. Will fire PR shortly! Thanks for reporting this bug!