Use character columns instead of UTF-8 columns in the diagnostics printer

[ahoppen]

Fixes a crash and improves the printing of errors on lines that contain multi-byte UTF-8 characters.

Fixes #2507
rdar://123409748

[ahoppen]

@swift-ci Please test

[bnbarham]

I assume you don't think this should be part of the location converter?

[ahoppen]

No, I think it’s quite specific to this use case.

[ahoppen]

@swift-ci Please test Windows

[Matejkob]

I'm not so sure if considering emojis as one character is correct. Are they two-character symbols?

I believe in this test:

func testEmojiInSourceCode() {
    let source = """
      let 👨‍👩‍👧‍👦 = ;
      """

    let expectedOutput = """
      1 │ let 👨‍👩‍👧‍👦 = ;
        │         ╰─ error: expected expression in variable
      """

    assertStringsEqualWithDiff(annotate(source: source), expectedOutput)
}

the diagnostic should point to ; not the space between = and ;. I can not test it right now, but I think if we change the variable name to regular characters like so:

func testEmojiInSourceCode() {
    let source = """
      let e = ;
      """

    let expectedOutput = """
      1 │ let e = ;
        │         ╰─ error: expected expression in variable
      """

    assertStringsEqualWithDiff(annotate(source: source), expectedOutput)
}

the annotation will be pointing to semi column.

[ahoppen]

The problem is that, as far as I know, it depends on the font that renders the font. A font can render an emoji as a single-width character, a double width character, anything in between or however it likes, really. And as far as I know there’s no Unicode specification for it and my thought that assuming that 👨‍👩‍👧‍👦 is rendered as a single character is closer to the truth than assuming that it gets rendered as 25 characters.

[Matejkob]

Ran the emoji test, and indeed, the annotation shifts on difficult Xcode font (also in the terminal), highlighting the complexity of emoji rendering in code.

Your explanation about the unpredictable rendering of emojis due to font differences is super insightful, thanks!

Out of curiosity, I've built this C code:

#include <stdio.h>

int main() {
    printf("Hello, World!🤯")
    return 0;
}

and the annotation in the diagnostic method produced by gcc is shifted as well:

[al45tair]

The problem is that, as far as I know, it depends on the font that renders the font. A font can render an emoji as a single-width character, a double width character, anything in between or however it likes, really. And as far as I know there’s no Unicode specification for it and my thought that assuming that 👨‍👩‍👧‍👦 is rendered as a single character is closer to the truth than assuming that it gets rendered as 25 characters.

This is mainly to do with terminal emulator behaviour rather than the font (though the font can certainly make things worse if it so desires); generally speaking terminal emulators try to match the behaviour of the wcwidth() function on their respective platform, and if that supports Unicode, it tends to just use the narrow or wide property from the East Asian Width table without doing anything much more complicated. That usually goes wrong for cases where the base character is narrow but some Emoji sequence is in play (so that the result is wide).