Add flake app to run openai proxy #4612
Conversation
Works with: nix run .#llama-server-openai-proxy If merged, then you can run both the server and proxy via two commands: nix run github:ggerganov/llama.cpp#llama-server nix run github:ggerganov/llama.cpp#llama-server-openai-proxy
| program = "${ | ||
| (let pythonWithPkgs = pkgs.python3.withPackages (ps: with ps; [ flask requests ]); | ||
| in pkgs.writeShellScriptBin "run-openai-proxy" '' | ||
| ${pythonWithPkgs}/bin/python3 ${self}/examples/server/api_like_OAI.py |
There was a problem hiding this comment.
This could've been a makeWrapper. I don't know if api_like_OAI.py takes any command line arguments, but makeWrapper would have passed those on
| (let pythonWithPkgs = pkgs.python3.withPackages (ps: with ps; [ flask requests ]); | ||
| in pkgs.writeShellScriptBin "run-openai-proxy" '' | ||
| ${pythonWithPkgs}/bin/python3 ${self}/examples/server/api_like_OAI.py | ||
| '') | ||
| }/bin/run-openai-proxy"; |
There was a problem hiding this comment.
Can't suggest anything concrete right now, but it might be a good thing to do some form of buildPythonApplication because its probably less affected by PYTHONPATH contamination.
| type = "app"; | ||
| program = "${self.packages.${system}.default}/bin/llama-server"; | ||
| }; | ||
| apps.llama-server-openai-proxy = { |
There was a problem hiding this comment.
I was going to mention this in #4605, but we wouldn't have had to manually wrap these in apps if we instead exposed them (in the overlay and) in packages/legacyPackages, and if we ensured that meta.mainPrograms are set correctly (with writeShellScriptBin they are)
There was a problem hiding this comment.
You could now add this to scope.nix: https://github.com/ggerganov/llama.cpp/blob/4e1d2510e445bb98c221c7b1d2ded20ffae60cab/.devops/nix/scope.nix#L7-L12
This way the attribute would show up as llamaPackages.llama-server-openai-proxy in nixpkgs with the flake's overlays.default applied, as well as in the flake's legacyPackages.${system}.llamaPackages.
SomeoneSerge
added
the
nix
|
I think this might not be needed since the default |
Yes, I'm thinking we should probably deprecate |
3 participants
Works with:
nix run .#llama-server-openai-proxy
If merged, then you can run both the server and proxy via two commands:
nix run github:ggerganov/llama.cpp#llama-server
nix run github:ggerganov/llama.cpp#llama-server-openai-proxy
Note: This is likely not ideal or the best way to do this, but it works and can hopefully spark useful discussion to get this feature into the llama.cpp flake 😃