[WIP] Add script to generate tests cases for `gix-diff` #2197

cruessler · 2025-09-28T18:05:57Z

This PR is about 75–80 % done (hopefully). It still needs comments and there’s also a couple of TODO comments that I want to go through. I already wanted to open it at this early stage, kind of as a sneak preview of the things I’m currently working on. :-)

What’s the context and what are the goals of this PR?

We assume that most people would expect the output of gix-diff to be identical to that of git diff. Therefore, we want an easy way to compare the output of gix-diff and git diff on a large corpus of diffs to make sure they match.

This PR tries to make as much of that process scripted and as easily reproducible as possible.

There is a repository that contains “[t]ools for experimenting [with] diff "slider" heuristics”: diff-slider-tools. We can use diff-slider-tools to, for any git repository, generate a list of locations where a diff between two files is not unambiguous.

This PR creates a tool that takes this a list of locations generated by diff-slider-tools and turns it into test cases that can be run as part of gix-diff’s test suite.

This enables us to, whenever we want, run large-scale tests comparing gix-diff to gix diff, hopefully uncovering any edge case we might have missed in our slider heuristic and making sure we conform to our users’ expectations.

Usage

Follow these instructions to generate a file containing sliders: https://github.com/mhagger/diff-slider-tools/blob/b59ed13d7a2a6cfe14a8f79d434b6221cc8b04dd/README.md?plain=1#L122-L146

Run create-diff-cases to create the script in gix-diff/tests/fixtures/. The script which will be called make_diff_for_sliders_repo.sh.

# run inside `gitoxide`
cargo run --package internal-tools -- create-diff-cases --sliders-file $DIFF_SLIDER_TOOLS/corpus/git.sliders --worktree-dir $DIFF_SLIDER_TOOLS/corpus/git.git/ --destination-dir gix-diff/tests/fixtures/

Run cargo test sliders -- --nocapture inside gix-diff/tests to run the actual tests.

Implementation details

The preamble to make_diff_for_sliders_repo.sh comes from a similar script in gix-diff: https://github.com/cruessler/gitoxide/blob/f0ceafa62ff519484e016789c50fa49a79819a80/gix/tests/fixtures/make_diff_repos.sh#L1-L4.

The repo created by make_diff_for_sliders_repo.sh follows a few conventions:

Each two subsequent commits form a pair that corresponds to a diff in the source repository that is ambiguous.
The commit message of the first commit in a pair consists of the path to the file this is a diff of. It comes from the blob id of the file in the source repository representing the old/before state. That way, the original file name in the source repository doesn’t matter and we don’t have to encode it anywhere.

The test runs assertions for each pair of commits. It reads the baseline the comes from git diff and compares it to the output of gix-diff.

Open questions

Do we want to go for an approach that maps pairs of commits to individual test cases? That way, we would be able to run the tests in parallel, something that’s not possible given the current setup. The current setup runs tests sequentially and stops at the first failure.

Byron

Thanks so much for making this happen! Once the slider problem is solved for us and validated, it's such a major leap for the diff-quality in gitoxide.

There is a major question I have that isn't mentioned inline: Is it easy to read/understand the output of the pretty assertion that compares both diffs using the internal hunk format? How does that fare in comparison to diffing unified diffs?

tests/it/src/commands/create_diff_cases.rs

gix-diff/tests/diff/blob/slider.rs

This obviates the need to create a git history which considerably simplifies the test as it can just direclty read files instead.

This makes visually interpreting differences between the two versions significantly easier.

cruessler · 2025-10-07T14:03:44Z

There is a major question I have that isn't mentioned inline: Is it easy to read/understand the output of the pretty assertion that compares both diffs using the internal hunk format? How does that fare in comparison to diffing unified diffs?

This is two different diffs, the first one uses the internal hunk format, the second one uses unified diffs. I think the second one is significantly easier to understand, so I changed the test to use that format in the last commit.

Using the internal hunk format

Using unified diffs

Byron

Thanks, this looks so much simpler now!

The diffs are also much easier to understand, yet I am wondering: would an actual unified diff be better to diff against another unified diff? That would retain diff lines, and I think the common term for it is 'interdiff'.

Byron · 2025-10-08T04:48:39Z

tests/it/src/commands/create_diff_cases.rs

+        let mut blocks: Vec<String> = vec![format!(
+            r#"#!/usr/bin/env bash
+# TODO:
+# `git diff --no-index` returns 1 when there's differences, but 1 is treated as an error by the


I'd keep the set -eu… and do git diff … || true or something like that, assuming that it will keep working.

Thanks for the suggestion, changed!

Byron · 2025-10-08T04:49:11Z

tests/it/src/commands/create_diff_cases.rs

+
+            blocks.push(format!(
+                r#"git diff --no-index "$ROOT/{asset_dir}/{old_blob_id}.commit" "$ROOT/{asset_dir}/{new_blob_id}.commit" > .git/{old_blob_id}-{new_blob_id}.baseline
+cp "$ROOT/{asset_dir}/{old_blob_id}.commit" assets/


Are these .commits?

Thanks for the suggestion, changed to .blob.

cruessler · 2025-10-08T07:57:45Z

The diffs are also much easier to understand, yet I am wondering: would an actual unified diff be better to diff against another unified diff? That would retain diff lines, and I think the common term for it is 'interdiff'.

That’s possible and it’s easier than I thought it would be. This is the result:

cruessler added 5 commits September 28, 2025 19:58

Add create-diff-cases

f27ecaf

Add DiffHunkRecorder

bc6bef2

Add Baseline for parsing git diff baseline

3b89374

Add assertion

99a6bb6

Thanks clippy

f0ceafa

Byron reviewed Oct 7, 2025

View reviewed changes

tests/it/src/commands/create_diff_cases.rs Outdated Show resolved Hide resolved

gix-diff/tests/diff/blob/slider.rs Outdated Show resolved Hide resolved

cruessler added 2 commits October 7, 2025 15:35

Use git diff --no-index to create baseline

6eaafda

This obviates the need to create a git history which considerably simplifies the test as it can just direclty read files instead.

Use textual diff in test

653a353

This makes visually interpreting differences between the two versions significantly easier.

Parse hunk header without regex

41cfdba

Byron reviewed Oct 8, 2025

View reviewed changes

cruessler added 3 commits October 8, 2025 09:27

Enable set -eu -o pipefail

20c7288

Replace .commit by .blob

28f9fe8

Use unified diff in test

1060c4c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[WIP] Add script to generate tests cases for `gix-diff` #2197

[WIP] Add script to generate tests cases for `gix-diff` #2197

Uh oh!

cruessler commented Sep 28, 2025 •

edited

Loading

Uh oh!

Byron left a comment

Uh oh!

Uh oh!

Uh oh!

cruessler commented Oct 7, 2025

Uh oh!

Byron left a comment

Uh oh!

Byron Oct 8, 2025

Uh oh!

cruessler Oct 8, 2025

Uh oh!

Byron Oct 8, 2025

Uh oh!

cruessler Oct 8, 2025

Uh oh!

cruessler commented Oct 8, 2025

Uh oh!

Uh oh!

Uh oh!

[WIP] Add script to generate tests cases for gix-diff #2197

Are you sure you want to change the base?

[WIP] Add script to generate tests cases for gix-diff #2197

Uh oh!

Conversation

cruessler commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What’s the context and what are the goals of this PR?

Usage

Implementation details

Open questions

Uh oh!

Byron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cruessler commented Oct 7, 2025

Using the internal hunk format

Using unified diffs

Uh oh!

Byron left a comment

Choose a reason for hiding this comment

Uh oh!

Byron Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

cruessler Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Byron Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

cruessler Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

cruessler commented Oct 8, 2025

Uh oh!

Uh oh!

[WIP] Add script to generate tests cases for `gix-diff` #2197

[WIP] Add script to generate tests cases for `gix-diff` #2197

cruessler commented Sep 28, 2025 •

edited

Loading