epic: llama.cpp params are settable via API call or `model.yaml`

[dan-menlo]

Goal

• Cortex can handle all llama.cpp params correctly
• Model running params (i.e. POST /v1/models/<model_id>/start)
• Inference params (i.e. POST /chat/completions)
• Function Calling, eg for llama.cpp

Tasklist

I am using this epic to aggregate all llama.cpp params issues, including llama3.1 function calling + tool use

• https://github.com/janhq/cortex.cpp/issues/618d
• feat: Support response_format and structured JSON responses. jan#3785
• feat: Cortex supports Function Calling #295
• [WIP] add support for function calling #354 (prev PR with function calling/tool use)
• min/max range for discrete int/float params, to prevent out-of-bounds errors (e.g. fix: missing type number for slider input right panel jan#3609)
• bug: Duplicate BOS Token in Hugging Face Chat Templates #618

model.yaml

• model.yaml as optional? (i.e. depend on GGUF params)
  • See @tikikun's comment in epic: llama.cpp params are settable via API call or model.yaml #1151 (comment)
• model.yaml should be well documented with approrpriate naming conventions
  • Model loading params?
  • Inference params?
  • To exclude engine params?

Out-of-scope:

• TensorRT-LLM
• ONNX
• feat: Return logits_prob in chat_completion #1163

Related

• epic: Essential, advanced parameters are missing jan#3140
• feat: Jan UI allows user to change llama.cpp params (through Cortex) jan#3508

[nguyenhoangthuan99]

Generally, I will break down this epic into tasks:

• fully support function calling: linking a comment from llamacpp's contributor.
• Request body parameters: we need to add more option that supported in llama.cpp chat completion, load model to the request body. Need to Pr in cortex.llamacpp. Currently, most of popular params for load model and chat completion are supported in cortex.llamacpp.
• Response body: add an option to return log probs -> need to modify cortex.llamacpp source, this task will need more effort because it related to inference implementation, need to carefully if not it will break inference or performance degrade.

Out-of-scope:

• Speech api support
• Image api support

model.yaml
From my side all information for running model should be in 1 file, because 1 model file can only run with 1 engine and user can do modify/tunning every parameters needed to run a model in the same place, it's more convenient.

Function calling
According to this comment, we can see that function calling is just a more complicated chat template that ask model to find the proper function and params to answer the input question. And there is no standard way to do it, each model has different training process so the prompt for each model is also different.

Function calling is essentially an advanced form of prompt engineering. It involves crafting a specialized prompt that instructs the model to identify appropriate functions and their parameters based on the input question. However, there's no universal approach to implementing this feature, as each model has undergone unique training processes, necessitating model-specific prompting strategies.
Developing a generic function calling feature presents significant challenges:

• Model variability: Different models require distinct prompting techniques, making it difficult to create a one-size-fits-all solution.
  Extensive experimentation: Even for a single model (e.g., llama3.1 - 8B), substantial testing is required to optimize performance across various scenarios.
• User-defined functions: Since users will define custom functions, it's challenging to ensure that a preset system prompt will work effectively for all possible function definitions.
• Quality assurance: Maintaining consistent output quality across diverse models and user-defined functions is extremely difficult.
  Unpredictable responses: The complexity of the task increases the likelihood of unexpected or incorrect outputs.

Given these challenges, it's crucial to approach the implementation of a generalized function calling feature with caution. The goal of supporting every model and every user-defined function is likely unattainable due to the inherent variability and complexity involved. Instead, it may be more practical to focus on optimizing the feature for specific, well-defined use cases or a limited set of models.

I also check chat-gpt, mistral, groq, ... they also support function calling but the different is they do this feature for their own models.
llama.cpp also haven't support function calling yet since it is not really useful for normal user, and developer can do it themselves with better result.