Fix #488 use non-ascii punctuation, not letters

[cmcaine]

The instructions specify that the only input letters will be the 26 english language letters, but we were testing with unlauts.

I think it's worth changing the test rather than the instructions because it's ambiguous whether adding diacritics to a letter makes it a new letter or not.

I did think about adding a bonus test that makes a decision one way or the other (you could use unicode's NFKD transformation1 to match letters, ignoring diacritics, or to require each unique glyph you could maybe use the NFC transformation + maybe splitting to graphemes (depending on if unicode has a codepoint for all "letters" in the alphabet)), but it's a bit complicated.

[SaschaMann]

These aren't combined characters, and the instructions don't specify that the inputs are restricted to English letters. #488 was incorrect at the time and after #556 that hasn't changed, though #556 made it less explicit.

The previous text defined the alphabet for the pangrams to be "ASCII letters a to z, inclusive," not the inputs.

[cmcaine]

The instructions now say:

For this exercise we only use the basic letters used in the English alphabet: a to z.

Which doesn't specify that it is talking about only input or only output.

I'm okay with us revising the instructions instead of the tests, but I do think we should revise at least one of the two.

Re: combined characters, I'm referring to how u-with-unlaut can be one unicode codepoint or it can be two (u codepoint and the combining unlaut codepoint).

[SaschaMann]

Re: combined characters, I'm referring to how u-with-unlaut can be one unicode codepoint or it can be two (u codepoint and the combining unlaut codepoint).

I know, I've ran into that issue ages ago on the rust track with some other exercise. But in this case, it's the single codepoint character. Or do you think it's a bad idea that this is visually ambigious?

I'm okay with us revising the instructions instead of the tests, but I do think we should revise at least one of the two.

I think that's a better option, I've opened exercism/problem-specifications#2240

[cmcaine]

Re: combined characters, I think normalisation is required to solve the exercise correctly for e.g. German or French sentences. Otherwise solutions will give different results depending on whether the glyph was written with combining diacritics or as a single codepoint, which I think is weird.

julia> "u\u0308" |> collect
2-element Vector{Char}:
 'u': ASCII/Unicode U+0075 (category Ll: Letter, lowercase)
 '̈': Unicode U+0308 (category Mn: Mark, nonspacing)

julia> using Unicode

julia> Unicode.normalize("u\u0308") |> collect
1-element Vector{Char}:
 'ü': Unicode U+00FC (category Ll: Letter, lowercase)

julia> occursin("ü", "u\u0308")
false

[cmcaine]

I've added a commit that changes the testcase to use an arabic language pangram so that we can avoid specifying any behaviour around latin characters with diacritcs.

[SaschaMann]

That's a nice solution to that issue

[SaschaMann]

Thanks!