Mark (provisionally) test_writing and test_logging as xfails

[huguesdevimeux]

(xfail means that we expect those tests to fail, or we could skip them)

These tests are slowing down A LOT the development.
the test_writing use latex things, and in some random time, they suddenly became OS-dependent, after x time when there is no particular reason. This test is not stable at all. It's blocking #315, #335 , #198 , and probably others. As all the latex related things are unstable, I don't think they should be tested as this is a PURE nightmare to debut them.

The test_logging used a character-per-character check (which is not bad) but this over the whole .log, which is also a nightmare to debug as one has to deal with the whitespace and all of this stuff. These tests will be easily re-implemented once #365 is done.

So I think, according to our new test policy, "massive amount of tests, massive amount of PR" (ok I made up this), we should get rid of tests that are not stable and very hard to debug, seeing that in addition, these tests are not vital.

Tell your opinion here; please.

[Aathish04]

the test_writing use latex things, and in some random time, they suddenly became OS-dependent,

Do you think it would be too much of a hassle to keep different reference data for different operating systems? I really don't want to "get rid" of a test completely, but if it's not worth the effort, then there's not much we can do...

The test_logging used a character-per-character check (which is not bad) but this over the whole .log, which is also a nightmare to debug

I don't think this is too much of an issue currently. The expected log only has to be updated very rarely. The main problem I'm trying to sort out is not due to the test itself, but due to some strange behaviour of the testing mechanism on MacOS (see our discussion in #227 ). I say we leave test_logging as it is for now, fix the underlying reason for the failure, which doesn't seem to be due to the test itself, but due to some bug in the testing mechanism, and implement the JSON log tests once they are ready.

In the event that doing either of these takes too much time (say, longer than a 3 days or a week), let's mark them as xfail and come back to them later, perhaps right before we release v1.0.0 or something.

[huguesdevimeux]

Do you think it would be too much of a hassle to keep different reference data for different operating systems? I really don't want to "get rid" of a test completely, but if it's not worth the effort, then there's not much we can do...

We can try this. I will implement a such thing.

I don't think this is too much of an issue currently. The expected log only has to be updated very rarely. The main problem I'm trying to sort out is not due to the test itself, but due to some strange behaviour of the testing mechanism on MacOS (see our discussion in #227 ). I say we leave test_logging as it is for now, fix the underlying reason for the failure, which doesn't seem to be due to the test itself, but due to some bug in the testing mechanism, and implement the JSON log tests once they are ready.

Yes, so could you try to fix the test_logging in #335? I think this is related to the dir generation in temp dirs. (the test works on windows, but not on other OS, so I think someone exclusive to windows is used to create the output directories.)

[Aathish04]

Is this closed by #335 ?

[huguesdevimeux]

Yep. closing