Add gix-upload-pack crate for use in server deployments

[jonasgrosch]

Add Git upload-pack server implementation

Overview

This PR introduces gix-upload-pack, a high-performance, drop-in replacement for Git's native git-upload-pack command. The implementation attempts to provide full protocol compatibility with Git clients while leveraging the gitoxide ecosystem for improved performance and memory safety.

Key Features

Protocol Support

• Protocol v0/v1 & v2: Complete implementation with stateful/stateless negotiation
• HTTP Backend: Full git-http-backend integration with --http-backend-info-refs alias
• Stateless RPC: Support for HTTP smart transport protocol

Core Functionality

• Reference Advertisement: Comprehensive ref listing with capabilities
• Object Negotiation: Multi-ack algorithms (basic and detailed)
• Pack Generation: Efficient streaming with sideband support
• Shallow Clones: Complete shallow repository support
• Object Filtering: Blob size, tree depth, and other filters

Command-Line Interface

• Drop-in Replacement: Compatible argument parsing with git-upload-pack
• Advanced Options: Timeout, strict mode, comprehensive help
• HTTP Integration: --advertise-refs and --http-backend-info-refs (hidden alias)

Architecture

gix-upload-pack/
├── src/
│   ├── main.rs              # CLI with clap argument parsing
│   ├── server/              # Core server implementation  
│   ├── protocol/            # Protocol v1 and v2 handlers
│   ├── services/            # Packet I/O and object services
│   └── config.rs            # Server configuration
├── tests/compatibility.rs   # Extensive compatibility tests
└── examples/               # Usage examples

Compatibility

Git Feature Parity

• ✅ All Git protocol versions (v0, v1, v2)
• ✅ HTTP smart transport integration
• ✅ Shallow operations (--depth, --since, --not)
• ✅ Object filtering (--filter=blob:limit, --filter=tree:depth)
• ✅ Multi-ack negotiation and sideband communication
• ✅ Reference filtering and tag peeling

Command Examples

gix-upload-pack /path/to/repo.git
gix-upload-pack --stateless-rpc --advertise-refs /path/to/repo
gix-upload-pack --http-backend-info-refs /path/to/repo.git  # Hidden alias
gix-upload-pack --timeout=300 --strict /path/to/repo.git

Testing

• Compatibility Tests: Direct comparison with native Git output
• Protocol Tests: All protocol variations and edge cases
• Integration Tests: End-to-end scenarios with real Git clients

Dependencies

Built on gitoxide ecosystem (gix, gix-protocol, gix-pack, etc.) with minimal external dependencies (clap, thiserror, bstr).

Usage

use gix_upload_pack::{Server, ServerOptions};

let options = ServerOptions::default();
let mut server = Server::new("/path/to/repo", options)?;
server.serve(stdin.lock(), stdout.lock())?;

This implementation attempts an almost byte-for-byte compatibility with Git's native upload-pack while offering the performance and safety benefits of the Rust/gitoxide ecosystem.

[jonasgrosch]

Tests currently depend on git to be installed and tokio existing as a submodule to run the tests on. I could probably find an alternative approach for the compatibility test and fixtures that drop these dependencies.

[Byron]

This is quite a biggie to drop on me like that, 8k lines, apparently created in 4 days judging by commit dates, that's quite a super-human feat.

Could you resolve the conflict by recreating Cargo.lock so CI can run?

From there I think we'd need to discuss how to tackle this - my disposition here is that it's unmergeable, but I would want to try to understand it better as well to see if something can or should be salvaged.

Even if this wasn't written by AI, I'd have trouble merging it as it would take me a lot of time to understand everything and become comfortable maintaining it, as testing, despite present, probably isn't anywhere near where it would have to be.

[jonasgrosch]

That's more than fair - I will gladly fix the conflict and fill you in with more details.
I'm working on a better iteration at the moment anyway as when I dove a bit deeper into the other gix-crates I see some redundancies. I would have also loved to contribute more into the rest of the codebase rather than isolating it in a single sub-crate but I was looking for feedback if you are interested including this in the gix ecosystem at all. If not the path for me to publish it as it's own repo is a bit easier than scattering the code around in your project.
There are definitely some parts that could be tighter integrated with the protocol and transport crate.

A little bit of background: I'm developing a pure rust based git backend - serveable via ssh and http. While I love your project it is pretty client focused at the moment, so I was missing critical git plumbing services to not need to call into c via libgit or rpc call into native git upload-pack and receive-pack. I thought that this fits here pretty well and could benefit the project as it basically fills some gaps that are included in upstream git, even if not actively used a lot by the average git user. Let me work on this a bit more in the next couple of days and you can think if this makes sense for you. I'm also working on a git receive-pack implementation and hook system that I can offer to contribute if that's a path you are open for with this crate.

I'm also happy to commit as a regular contributor and maintainer on this repo so you won't be left alone with it.

Happy yo hear your general thoughts on it or any advice you can give.

[Byron]

Thanks for the background, let's find a solution.

At this state, not integrating more tightly with the other crates (as in sprinkling server code around) might be a good thing, despite my initial vision to have more code-sharing for when the time comes. gix-protocol was always meant to be restructured to capture the server-side as well and be written so it's the 'core' that can be used in both sync and asynchronous contexts just like it's done now.

Also, I appreciate you wanting to contribute the implementation to the project and to keep maintaining it.

How about we move the crate into its very own repository, give you all the rights there so you can work independently, and see how over time it will develop? Maybe from there parts can be moved into the existing or even new plumbing crates with all the tests and diligence that gitoxide requires?

If acceptable, we can see to what extend the crate names are suitable. I'd want to avoid mixed library and application crates, and also wonder if it really is compatible to git-upload-pack or should rather have another name if it is not. And even if it is compatible I wonder if it has to be, after all this is a standalone server if I see that correctly? gix-serve comes to mind for the binary part.

Let's stop here and hear your thoughts.

[Byron]

CC @EliahKagan Maybe a Codeowner file can also be used to allow PRs to specific crates to be merged by the author without assigning broad repository merge permissions right away. Your perspective on how to handle this would be appreciated as well.

[jonasgrosch]

The reason why I shaped the crate the way I did is to mirror the way git structures it and bundles it in a git sub command with "git upload-pack" which is basically just an io-binary streaming the commands in and sending the packs out given the "wants" and "haves" the client announces with all the protocol wrapping. Git pull even accepts a binary path for upload-pack to execute a push locally or ssh (https://git-scm.com/docs/git-pull#Documentation/git-pull.txt---upload-packupload-pack).

Thinking about it that might be too strict of an approach and a "gix-serve" could probably handle the fetch and ls-refs fom upload-pack together with the push side of things that I'm currently building with receive-pack. the main problem with that approach on first glance seems to be that git upstream decides to route to the services on ssh/http protocol level respectively e.g. (https://git-scm.com/docs/git-http-backend).

I wanted to avoid coupling everything together too tightly and tackle it in a plumbing approach that would keep the individual serve binaries compatible with the rest of the git ecosystem. Currently my perspective is that "gix-serve" would require me to also add a transport layers on top and drop the low level plumbing approach. Also receive-pack and upload-pack seem to not share too much other than the basic packet-line and pack protocol, plus parts of the capability negotiation phase(with different capabilities and even version support).

That said - I'm more than happy to iterate over this and let it evolve on my forked repo for now with also my understanding of the other gix crates and principles evolving while I'm building receive-pack.
Testing will mature over time - I purposefully avoided unit tests to keep iteration speed quick during the initial development and rather test it e2e for full parity with the git upload-pack command (which I believe I achieved). I will improve the way I use the fixtures in the e2e test, add unit tests over the time to lock the individual parts in once the module structure was derived and considered appropriate. It might even be very interesting to have some tests where gix fetch is directly talking to this but I'm currently not sure how it would be possible without adding the transport layer on top.