Direction of diff (use of +/-) for strings intentional?

[ctheune]

This is on pytest 3.4.2, I don't think other environmental parameters are relevant.

I've noticed a few times that I'm struggling with the +/- usage of long string diffs (-v) as that I never can immediately understand what was expected and what was found.

First, pytest documentation shows the same style of writing assertions as I do, in the form of:

    assert my_result == 'bob'

So the left hand would hold the value the UUT gave me and the right hand is what the test is expecting it to be. If you use this for long strings and -v for showing diffs this ends up in the following:

    assert app.mailer_mock.messages[0][2] == """\
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Subject: Re: Hilfe
To: dh@example.com
From: support@flyingcircus.io
In-Reply-To: 12345
Message-ID: <msg@flyingcircus.io>

Dear Customer,

we could not find a valid SLA for you. If you are sure to have one,
please check if you provided the correct PIN.

Best regards,
the Flying Circus support team
"""

Output:

AssertionError: assert 'Content-Type...upport team\n' == 'Content-Type:...upport team\n'
    Content-Type: text/plain; charset="utf-8"
    Content-Transfer-Encoding: 7bit
    MIME-Version: 1.0
    Subject: Re: Hilfe
    To: dh@example.com
  + From: support@flyingcircus.io
    In-Reply-To: 12345
    Message-ID: <msg@flyingcircus.io>
  - From: support@flyingcircus.io

    Dear Customer,

    we could not find a valid SLA for you. If you are sure to have one,
    please check if you provided the correct PIN.

    Best regards,
    the Flying Circus support team

When looking at that output, I would like to interpret it as "the + shows me what the test found that was there but not expected" and "the - shows me what the test expected but is missing".

Obviously this comes down to the +/- being a convention that implies that the left hand side shows the expected value and the right hand side the result of the UUT. However, writing it down the other way makes the test IMHO less readable and is also known as "yoda expressions" and usually frowned upon [citation needed].

This issue is probably more about discussion and understanding intent and stance from pytest's perspective and likely won't change the way its reported. Although I could also imagine adding a configuration knob for this behaviour.

[pytestbot]

GitMate.io thinks possibly related issues are #2435 (f), #325 (use py.test + xdist to parallelise selenium tests, but it failed), and #954 (How to use skipif-decorators plus parametrized-skipif on a test-function?).

[RonnyPfannschmidt]

as far as i understand,/vaugely remember the intent here is to support the common spelling on assert expected == computed

@flub and @benjaminp may remember details on when and how it was chosen without digging trough the project history

[flub]

Huh, I always thought it was computed == expected, at least that's how I tend to write my tests. It matches Python APIs like isinstance(my_thing, ExpectedThing) as well. And according to @ctheune our docs even seem to agree on this.

To be honest, I don't think there was ever much thought put into the choice. The code did difflib.ndiff(left, right) because that's just the I-didnt-think-about-this way to write it and the output looked sane. I've also always read the output as "this is the patch needed to make the output match the expectation", but appreciate this is pretty subjective.

I'm kind of tempted to say that it's too late for this anyway, I'm even tempted to argue against a configuration knob as then the output of pytest is dependent on some hidden setting somewhere. So when you're in a new project or just see a paste somewhere you get to guess which way they used this, if you even realised this existed in the first place.

[RonnyPfannschmidt]

from my pov making this one sane is a good reason for a major version bump
lets document and wire up the sane expectation, then look at timeframes

after all its ore of an ux issue, not a consumed api as far as i understod

[ctheune]

@flub I understand that thought about leaving it as-is, however, from a UI perspective I think it might be better in the long run to admit mistakes, fix them and move on. If the majority thinks that the UI issue isn't worthwhile and I'm the onlly one bothered by the current state of affairs then I won't hold a grudge against that decision. ;)

However, if nobody actually relies on the usability because it's broken, then you could also just fix it and accept that the current confusing state of affairs might be a bit longer confusing for a transition period until the expectation settles for the new / "better" version ... not sure whether a major version bump is rectified by this, but I guess at least a minor number would definitely be in order ...

[benjaminp]

I don't recall there being any good reason for this initially. Proposed change seems good to me.

[The-Compiler]

I've also always found the diff outputs confusing in some way, and usually looked at the expected text (or even printed it) to make things clearer to me. I've never put much thought into it, but this might have been why!

FWIW I also do assert foo() == expected and would expect (hah!) most people to do it that way in pytest. self.assertEqual(expected, actual) is probably common with unittest.py though, probably because JUnit does it that way.

[flub]

Well, seems like there's a quorum for the change!

[nicoddemus]

@The-Compiler

I've never put much thought into it, but this might have been why!

Exactly what I was thinking before I reached your comment. 😆

I'm definitely 👍 about changing this, this always bothered me and I never realized the reason, like @The-Compiler said. I also don't think we need a major version bump either, it is an improvement on how we output things, we just need a clear changelog entry and a note to the docs.