Can I use multiple models with llama-server and select one with "models" field in POST request?

[valensys]

Hello, everyone.

With "ollama serve" I can run multiple models at once and select one of them with "models" field in body of POST request. Like:

{ "prompt": "Tell about python", "model": "t-pro-2.0" }

Can I do the same with llama-server somehow? Thank you.

[0x9334]

@valensys Could you try eternal-zoo?

[iansmathew]

Hello. llama-swap is what you're looking for.

You can set up a config file to serve multiple models and have them automatically load and unload with a ttl.

[yazon]

@valensys or Flexllama: https://github.com/yazon/flexllama

[LMSSonos]

Hello. llama-swap is what you're looking for.

Thanks for that, looks exactly what I was looking for. IMHO it still would be nice to have to this kind of functionality integrated into llama.cpp. Just moved from ollama to llama.cpp to keep the whole setup minimalistic and this just adds again another container / level ;-) But will do for the moment ;-) Thanks

[iansmathew]

@LMSSonos Agreed that it would be nice to have llama-server be able to load a gateway server.

this just adds again another container / level

If by container, you mean a docker container, I would add llama-swap to the same container you run llama-server on. If on the other hand you mean it's more steps involved, I would argue that llama-swap makes things a lot easier in fact. But yes.. it is one more dependency to add.

Here's my current configurations to help get you set up quicker:

~/.zshrc

# a function alias that includes the path to my config file 
# and additionally passes any arguments that I would want to add
llama-swap() {
    command llama-swap -config ~/dev/LLM/llama-swap-config.yaml "$@"
}

# an alias to quickly open my llama-swap-config in my IDE
alias llama-swap-settings='code ~/dev/LLM/llama-swap-config.yaml'

llama-swap-config.yaml

macros:
  "llama-server": |
    /opt/homebrew/bin/llama-server
    --port ${PORT}

models:
  "unsloth-gemma-3-27b":
    cmd: |
      ${llama-server}
      -hf unsloth/gemma-3-27b-it-GGUF:Q4_K_XL
      -a unsloth-gemma-3-27b
      --mlock
      --jinja
      -ngl 99
      --threads 10
      --ctx-size 32768
      --temp 1.0
      --min-p 0.0
      --top-p 0.95
      --top-k 64
      --repeat-penalty 1.0

  "unsloth-DeepSeek-R1-0528-Qwen3-8B":
    cmd: |
      ${llama-server}
      -hf unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF:Q8_K_XL
      -a unsloth-DeepSeek-R1-0528-Qwen3-8B
      --mlock
      --jinja
      -ngl 99
      --threads 10
      --ctx-size 32768
      --temp 0.6
      --top-p 0.95

  "unsloth-Qwen3-Coder-30B":
    cmd: |
      ${llama-server}
      -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q6_K
      -a unsloth-Qwen3-Coder-30B
      --mlock
      --jinja
      -ngl 99
      --threads 10
      --ctx-size 16384
      --temp 0.7
      --min-p 0.0
      --top-p 0.80
      --top-k 20
      --repeat-penalty 1.05
      -fa

  "unsloth-Qwen3-30B-Thinking":
    cmd: |
      ${llama-server}
      -hf unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:Q4_K_XL
      -a unsloth-Qwen3-30B-Thinking
      --mlock
      --jinja
      -ngl 99
      --threads 10
      --ctx-size 32768
      --temp 0.7
      --min-p 0.0
      --top-p 0.80
      --top-k 20
      --repeat-penalty 1.05
      
  "Qwen-Qwen3-Embedding-0.6B":
    cmd: |
      ${llama-server}
      -hf Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0 \
      -a Qwen-Qwen3-Embedding-0.6B \
      --embedding \
      --pooling cls \
      -ub 8192

# Set groups so that embedding models are always running.
# But other models unload when a new model loads. 
groups:
  "single-run-models":
    swap: true
    exclusive: true
    members:
      - "unsloth-gemma-3-27b"
      - "unsloth-Qwen3-30B-Thinking"
      - "unsloth-DeepSeek-R1-0528-Qwen3-8B"
      - "unsloth-Qwen3-Coder-30B"

  "embedding-models":
    swap: false
    exclusive: false
    members:
      - Qwen-Qwen3-Embedding-0.6B
      

This lets me enter a single command -> llama-swap in my terminal and start using all my models straight away from any configured clients.
Be aware that sometimes you might have to set your API url in the clients to http://localhost:8080/v1 or http://localhost:8080.
Additionally, be sure to use the model names that you have specified in your -a alias argument in your config.yaml. I keep my aliases the same name as the model name to clear up any confusion.

llama-swap will soon be coming out with a pre-loading feature to allow select models to start up immediately when llama-swap in run. Currently models are only loaded when the first API request is made to a particular model.

[LMSSonos]

@iansmathew thanks for sharing - I was referring docker containers, but saw that according to llama-swap that should work as well

[LMSSonos]

@iansmathew thanks again, I was able to make everything work, had to recompile for arm64 but llama-swap works now. Just personally hope the server has now the new UI #14839 they will integrate it somehow, because the way you can switch model in ollama is nicer than having 2 UIs with llama-swap and llama-cpp ;-)

Anyway, solution works and I'm happy with it. Thanks again