From 7780cab8dcaaa042c31b0588deed3d408af0d210 Mon Sep 17 00:00:00 2001 From: Solomon Rutzky Date: Sat, 13 Jul 2019 14:59:46 -0400 Subject: [PATCH] Fix Supplementary Character / Surrogate Pair info (no code changes) (#7221) Terminology and info regarding Supplementary Characters / Surrogate Pairs is either incorrect, or at least incomplete (which then leads to incorrect statements and/or code). 1. Introduce the term "Supplementary Character" since that is often what we are dealing with, not "surrogate pairs" since that is an encoding-specific concept (UTF-16 only). 2. Add comment re: Supplementary Character code point range, which helps to explain the `elif high > 0x10 then Invalid` condition (line 173). 3. Fix URI for Unicode Standard PDF, Chapter 3, and specify the name of the section (i.e. "Surrogates") instead of the section number (i.e. 3.8) since the section number was 3.7 but is now 3.8 (line 174). 4. Add comment for definition of a valid "surrogate pair" because why make the reader guess or have to go look it up when it will never change? (line 175) 5. Correct and expand comment with example long Unicode escape sequence (line 64): `"\UDEADBEEF"` is _not_ a valid escape sequence. Usage of the `\U` escape has been misstated from the very beginning, both in this documentation as well as the C# Specification documentation, and the language references for "String" for both F# and C#: 1. `\U` is used to specify a Unicode code point (or UTF-32 code unit, which maps 1:1 with all Unicode code points, hence they are synonymous), not surrogate pairs. Hence the valid range is `00000000` - `0010FFFF`, hence the first two digits are static `0`s, and the third digit can only ever be a `0` or `1`. This escape sequence can specify either a BMP character or a Supplementary character. Supplementary characters are then encoded as a surrogate pair in UTF-16 only, not in UTF-8 or UTF-32. If you want to specify an actual surrogate pair, then use the `\u` escape, e.g. `\uD83D\uDC7D` == `\U0001F47D`. 2. Even if you could specify a surrogate pair using `\U`, "DEADBEEF" is not valid. U+DEAD is a valid surrogate, _but_ it's a low surrogate code point and cannot be specified first in the pair (meaning, at best one could use `\UxxxxDEAD`). Also, U+BEEF is _not_ a valid surrogate code point, high or low. Surrogate code points are in the range of U+D800 to U+DFFF. For more info, please see: https://sqlquantumleap.com/2019/06/26/unicode-escape-sequences-across-various-languages-and-platforms-including-supplementary-characters/#fsharp --- src/fsharp/lexhelp.fs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/fsharp/lexhelp.fs b/src/fsharp/lexhelp.fs index e966678486e..fb66ac5f85a 100644 --- a/src/fsharp/lexhelp.fs +++ b/src/fsharp/lexhelp.fs @@ -60,7 +60,8 @@ type lexargs = applyLineDirectives: bool pathMap: PathMap } -/// possible results of lexing a long unicode escape sequence in a string literal, e.g. "\UDEADBEEF" +/// possible results of lexing a long Unicode escape sequence in a string literal, e.g. "\U0001F47D", +/// "\U000000E7", or "\UDEADBEEF" returning SurrogatePair, SingleChar, or Invalid, respectively type LongUnicodeLexResult = | SurrogatePair of uint16 * uint16 | SingleChar of uint16 @@ -169,7 +170,9 @@ let unicodeGraphLong (s:string) = if high = 0 then SingleChar(uint16 low) // invalid encoding elif high > 0x10 then Invalid - // valid surrogate pair - see http://www.unicode.org/unicode/uni2book/ch03.pdf, section 3.7 *) + // valid supplementary character: code points U+10000 to U+10FFFF + // valid surrogate pair: see http://www.unicode.org/versions/latest/ch03.pdf , "Surrogates" section + // high-surrogate code point (U+D800 to U+DBFF) followed by low-surrogate code point (U+DC00 to U+DFFF) else let codepoint = high * 0x10000 + low let hiSurr = uint16 (0xD800 + ((codepoint - 0x10000) / 0x400))