Add updater for expected results of integration tests

[benmss]

Currently any changes that create a legitimate difference in Macaron's results and the expected results used in the integration tests must be handled manually. An automatic method for updating the expected results files would save considerable time moving forwards.

[behnazh-w]

Yes please! Looking forward to the PR 👍

[nathanwn]

Can we utilize syrupy for this purpose? It already supports comparing complex JSON structures and overwriting existing results.

[behnazh-w]

Can we utilize syrupy for this purpose? It already supports comparing complex JSON structures and overwriting existing results.

We have our own custom scripts to process the JSON outputs. We trim the non-deterministic parts and only check the parts that are crucial. So, I doubt Syrupy being a good solution.

All we need to do really is to update the JSON outputs by copying over the files if an update flag is provided.

[nathanwn]

If we have non-deterministic parts in the expected result files, then can't we just "eliminate those non-deterministic parts" from these expected result files? These files can store just the "deterministic subset" of the JSON result that Macaron produces, and not the whole result. This way we may be able to delegate both comparing and updating to surupy and remove the burden of maintaining the compare script.
Here by "eliminate the non-deterministic parts", I mean both (1) literally removing certain things that we don't need to check for and (2) forcing certain things to be deterministic when Macaron outputs them, such as the order of lists in the output. Point (2) also helps to minimize the diff of these expected output files when we commit them, making it easier to inspect and code review.

[behnazh-w]

This way we may be able to delegate both comparing and updating to surupy and remove the burden of maintaining the compare script.

Removing certain values from the output and storing it as intermediate files for testing also needs maintenance and can be even more complex. I don't see the benefit here since we already have comparison scripts.

[nathanwn]

Yep. You're right that in that case, we need to cherry-pick things from the current output, and that part requires maintenance. Something I am wondering is whether the compare script or the cherry-picking will require more maintenance long-term.

[tromai]

Can this issue be closed as #764 has been merged 🤔 ?