[WIP] Update to Unicode 16.0.0

[linusg]

Please review this very carefully, I have no idea what I'm doing :)

Closes #77.
Closes #530.

[linusg]

Someone needs to tell me how to implement these, my brain is not capable of writing this kind of obfuscated C code.

[bnoordhuis]

Caveat emptor, I'm not 100% sure but I suspect you can merge the RUN_TYPE_U_EXT2 case with RUN_TYPE_LF_EXT2 and update the guard to:

if (is_lower != (type - RUN_TYPE_U_EXT2))
    break;

RUN_TYPE_U_EXT2 should immediately precede RUN_TYPE_LF_EXT2 in the enum declaration for that to work (not the case right now.)

For RUN_TYPE_U_EXT3, the code should probably look like this:

case RUN_TYPE_U_EXT3:
    res[0] = c - code + case_conv_ext[data >> 8];
    res[1] = case_conv_ext[(data >> 4) & 0x0f];
    res[2] = case_conv_ext[data & 0x0f];
    return 3;

[linusg]

Copy and pasted from the one below except there is no F component

[bnoordhuis]

That means the if statement below is dead code now, right? Less specific subsumes more specific.

It's probably okay if you swap them. The code itself looks correct at a superficial glance.

[bnoordhuis]

Caveat emptor, I'm not 100% sure but I suspect you can merge the RUN_TYPE_U_EXT2 case with RUN_TYPE_LF_EXT2 and update the guard to:

if (is_lower != (type - RUN_TYPE_U_EXT2))
    break;

RUN_TYPE_U_EXT2 should immediately precede RUN_TYPE_LF_EXT2 in the enum declaration for that to work (not the case right now.)

For RUN_TYPE_U_EXT3, the code should probably look like this:

case RUN_TYPE_U_EXT3:
    res[0] = c - code + case_conv_ext[data >> 8];
    res[1] = case_conv_ext[(data >> 4) & 0x0f];
    res[2] = case_conv_ext[data & 0x0f];
    return 3;

[bnoordhuis]

That means the if statement below is dead code now, right? Less specific subsumes more specific.

It's probably okay if you swap them. The code itself looks correct at a superficial glance.

[bnoordhuis]

With some clever rearranging of the enum values, you can avoid code duplication by folding the three cases into one:

case RUN_TYPE_L:
case RUN_TYPE_LF:
case RUN_TYPE_U:
    te->type += RUN_TYPE_L_EXT - RUN_TYPE_L;
    te->ext_len = 1;
    te->ext_data[0] = te->data;
    break;

[bnoordhuis]

@linusg do you plan on revisiting this?

FWIW, I don't think this fully closes out #77 until the regex engine is taught about unicode emoji (tr51). Ex. /\p{Basic_Emoji}/ with the challenge that emoji are variable-length encodings.

I did a proof of concept where I pregenerate the regex bytecode for Basic_Emoji, Emoji_Keycap_Sequence, etc. and memcpy it when compiling regular expressions but that results in really bloated bytecode.

[linusg]

I won't get around to it until sometime next week at the earliest, if someone else wants to pick this up before then that's fine by me :)

[bnoordhuis]

No rush. I was just wondering if I should adopt it.

[linusg]

I didn't find the time for this yet and likely won't in the coming weeks either - if you or someone else wants to finish it please feel free, I'd love to have this merged!

[bnoordhuis]

I took a slightly different tack because I wanted to understand why 15.1 behaved differently from 16.0 but the update to 15.1 was done in 416ab66, followed immediately afterwards by an update to 16.0 in 243b968. Thanks anyway for the pull request!

[linusg]

Thank you, much appreciated!