We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent 564fa63 commit e703e04Copy full SHA for e703e04
README.md
@@ -62,7 +62,7 @@ python optillm.py
62
63
- Set the `OPENAI_API_KEY` env variable to a placeholder value
64
- e.g. `export OPENAI_API_KEY="no_key"`
65
-- Run `./llama-server -m path_to_model` to start the server with the specified model
+- Run `./llama-server -c 4096 -m path_to_model` to start the server with the specified model and a context length of 4096 tokens
66
- Run `python3 optillm.py --base_url base_url` to start the proxy
67
- e.g. for llama.cpp, run `python3 optillm.py --base_url http://localhost:8080/v1`
68
0 commit comments