Add a tolerance for pixel perfect tests

[huguesdevimeux]

Basically the title.

The problem with pixel-perfect tests is that they are too perfect. Then, we spend so much time and energy in debugging a lot of small things (that are actually small bugs, I guess) but that didn't prevent the correct functioning of manim and were not worth the amount of time spent.

There are plenty of examples; I'm pretty sure every community dev has encountered at least one issue related to a single pixel differing, causing the entire test to fail.

My proposal would be to set a tolerance to a certain amount of pixel, like, if the frames differ from less than X pixels (or Y% pixels) then the assertion is not triggered and the test passes.
I propose to set Y to 10%.

I know that it may allow some bugs to go into manim; but the idea is not to waste too much time in debugging small and annoying things.

thoughts ?

[ToxicGLaDOS]

I haven't had this experience personally, but it seems like a dangerous door to open. After all, if a change causes an unexpected change in a test (even a small one) who knows what it will do in other use cases? It might be the case that the change causes bigger issues in a different scene.

At the very least I think the tests should display a warning or something if it isn't pixel perfect so we know there's something wrong/different.

[leotrs]

I agree, I think setting a tolerance is a slippery slope. Also, @huguesdevimeux can we close this now that #335 is in progress?

[huguesdevimeux]

Yep. Closing this (but this is unrelated to #315, I close this as we didn't reach a consensus).