On Brotli and non-deterministic compression

[IvanUkhov]

Hello,

In version 0.4.23, brotli was updated from 7 to 8. I am wondering if anybody has experienced the same problem that the compression is no longer deterministic: compressing the same file multiple times results in different binary files.

[NobodyXu]

Hello, do you have a minimal reproduction code + data we can try?

[IvanUkhov]

I have been trying to create some kind of minimal example. This is my first try:

https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=56e76f50afde96d319b9788d25c71f19

When I run, it gives two different checksums:

I am currently not sure if this is the same problem as what I encountered in my actual code, but let us start here. How should one explain two different compressed files in this case?

[NobodyXu]

Hmm that's strange.

Is there some kinds of pattern in your checksum output?

[NobodyXu]

It could be a bug in brotli, to test it you can patch compression-codecs to use a different brotli version and see if it still occurs

[IvanUkhov]

My current understanding is that the larger the input, the more variations one gets. I suspect that it was not present in 0.4.22 and started to happen from 0.4.23 onwards, which is when brotli was upgraded to 8. But it is not that I tested every version.

[NobodyXu]

You can patch compression-codecs to use brotli 7, you don't have to test every version.

[IvanUkhov]

OK, so I activated fs for tokio in async-compression/Cargo.toml and added this to async-compression/tests/brotli.rs:

use std::collections::HashSet;

use async_compression::Level;
use async_compression::tokio::bufread::BrotliEncoder;
use tokio::io::{AsyncReadExt, BufReader};
use tokio::fs::File;

#[tokio::test]
async fn main() {
    std::fs::write("data.json", [0u8; 1024]).unwrap();
    let mut seen: HashSet<usize> = Default::default();
    loop {
        let uncompressed = BufReader::new(File::open("data.json").await.unwrap());
        let mut encoder = BrotliEncoder::with_quality(uncompressed, Level::Precise(8));
        let mut compressed = vec![];
        encoder.read_to_end(&mut compressed).await.unwrap();
        let checksum = compressed.into_iter().fold(0usize, |sum, value| sum.wrapping_add(value as usize));
        seen.insert(checksum);
        if seen.len() > 1 {
            println!("{seen:?}");
            assert!(false);
        }
    }
}

It fails with both brotli 7 and 8 in compression-codecs/Cargo.toml.

[IvanUkhov]

Checked out 7a33560 (0.4.22), and it has not failed so far.

[NobodyXu]

It'd be great if you can use git bisect to find out which commits causes this failure, so we can fix this

Note that #398 is about to merge soon and I'd prefer the it to be merged first, to avoid merge conflicts.

Maybe I should merge it tomorrow given that there's no objection so far.

[IvanUkhov]

According to my manual bisection, it appeared in 0.4.32 0.4.31:

https://github.com/Nullus157/async-compression/compare/74f5eb11ace53716be9298b19a59e2d4c5953eeb..fcd9e8771048a1a150c6fcac3c100e93fe627558

[IvanUkhov]

I am a bit confused. My original bug is still reproducible with 0.4.31. So it might be two different things after all.

[IvanUkhov]

To summarize, my original problem it due to changes in 0.4.23, and the problem uncovered by my toy example is due to changes in 0.4.32 0.4.31, and those seem to be two different problems.

I will keep on poking regarding the former, but I would appreciate if you could confirm the existence of the latter. If you just add the test I wrote above and try to run it, it should eventually fail. One note, though, is that it seems to be platform dependent or something. I was not able to reproduce anything on my MacBook, while I could do so on GitHub with ubuntu-latest and on a Linux virtual machine on Google Cloud Platform. And well, on the Rust Playground.