Recommend avoiding invalid surrogate pairs

[GULPF]

Unicode code points in the range U+D800 to U+DFFF are used for encoding UTF-16 surrogate pairs, and can not occur on their own in a valid Unicode sequence. However, the \uHHHH escape sequence can be used to construct such a an invalid Unicode sequence, e.g "\uDEAD".

The interpretation of invalid Unicode sequences like "\uDEAD" is not specified in the JSON, JSON5 or ES5 (I think) specs. However, it is specified in the ES6 spec:

A code unit that is in the range 0xD800 to 0xDFFF, but is not part of a surrogate pair, is interpreted as a code point with the same value.

This also matches the behavior of the JSON5 reference implementation. I think it's best if this is also clarified in the JSON5 spec.

[jordanbtucker]

Thanks. RFC 8259 addresses this, so I agree that the JSON5 spec should address this too.

[ell1e]

May I ask why U+.... is limited to the UTF-16 range and surrogates are used in the first place? If that was changed, then any sort of escaping would also automatically become moot because even the higher code points could be put in directly without surrogate workarounds

[jordanbtucker]

@ell1e Unicode escapes are limited to UTF-16 code units because that's the syntax available in ES5.

[ell1e]

Oh, how unfortunate. I get if you want to stick to it since JSON traditionally mapped to valid JavaScript, but I would suggest at least considering to change this if an opportunity presents itself (new revision of the spec, if that is ever done). After all, technically it shouldn't be too hard to write a JavaScript parser that reads this, even when ES5 itself wouldn't accept the according escape.

[ell1e]

Just to explain this further, I feel like there are a lot of headaches in the world because of UTF-16 surrogates. E.g. the ES6 specification leads to confusing surrogating-the-surrogates in languages like Python 3 which use surrogates already for implementing broken values like in WTF-8 which is needed to retain broken surrogates in e.g. broken UTF-16 implementations like windows in an UTF-32 string: in Python 3, I believe any broken surrogates will themselves need to be treated as binary and remapped with other surrogates if it is supposed to be a non-lossy conversion. (At least I'm quite sure that is what CPython will do internally, if you decode Unicode with error handling being surrogate escaping.) That sort of surrogate-hacking around broken surrogates is just mind-bending to me, and it confuses me why we would want more of that in existence.

Edit: obviously, the above isn't a technical problem since this will be handled transparently, so it works just fine. I just find it conceptually so weird that I'm wondering, why would you even want to encourage people to keep using broken UTF-16 surrogates?

I am therefore also suggesting that it might be better to reject broken UTF-16 surrogates as invalid due to this, because if one really wants them they can be encoded with \x.... anyway and I feel like \U+... should suggest something that is actually valid, meaningful Unicode. But ideally, I just think \U+... should be an unbounded value for all possible code points and surrogates just disallowed entirely, because then this whole mess wouldn't be required for people to just input the Unicode character they want.

But that's just my opinion, obviously!

[jordanbtucker]

@ell1e You make some good points. I'm wondering why the ECMAScript Specification decided to keep unmatched surrogates rather than reject them as invalid, and whether their reasoning should pertain to JSON5.

This restriction does not exist in the JSON spec either, but that may be because so many JSON documents already exist in the wild, and adding that restriction would invalidate some JSON documents, causing interoperability issues.

This technically would be a breaking change, but it would still allow JSON5 to be backward compatible with ES5 and JSON.

I'll have to think about this some more.

[ell1e]

Ouch, I never noticed these broken code-points-but-actually-UTF-16 actually were in original JSON, I assumed \U+... was a JSON5 addition. Well that is a bummer. And I checked the spec, it says:

To escape an extended character that is not in the Basic Multilingual
Plane, the character is represented as a twelve-character sequence,
encoding the UTF-16 surrogate pair. So, for example, a string
containing only the G clef character (U+1D11E) may be represented as
"\uD834\uDD1E".

And that's kind of all on it.

Honestly, it feels like they just mixed up unicode character and utf-16 encoding (which I think was common before utf-32 became more adopted as the "true" representation) and as if they didn't even think about the special case of invalid surrogates. As didn't Windows, or anyone else doing UTF-16 without thinking about how to integrate surrogates properly when they were added... oh well.

I guess if it's broken in JSON that's kind of a point for just sticking with it 🤷‍♀️

[jordanbtucker]

Oh yeah. JSON5 is a superset of JSON, so if it's allowed in JSON it needs to be allowed in JSON5. Adding this restriction would actually break compatibility with JSON. So it's a no go.

[ell1e]

Yeah I agree. Ignore all I said 😂

Edit: I mean to be fair, JSON doesn't really say it's allowed either. But in practice probably most implementations will allow it, so probably too late to try to curb it now.

[jordanbtucker]

Yeah I agree. Ignore all I said 😂

Your points are still valid, so I think it would be good to add a recommendation to avoid invalid surrogate pairs, similar to how avoiding duplicate names and using RFC 3339 formatted dates are recommendations, but not hard requirements.