Allow UTF-16 String access

[Sytten]

Hi there!

I am currently working a on StringDecoder implementation for llrt.
One thing that I currently test for is the following:

const decoder = new StringDecoder('utf16le');
let res = "";
res += decoder.write(Buffer.of(0x3d, 0xd8));
res += decoder.write(Buffer.of(0x61, 0x00));
res += decoder.end();
expect(res).toEqual("\uD83Da");

The 0x3d, 0xd8 is an invalid high surrogate for 0x61, 0x00, the UTF-16 spec says that in such a case we should keep the high surrogate as is. But there is no way to encode that in a UTF-8 C String.
Node likely has direct access to the UTF-16 String of V8.

This is an edge case I know, but having direct access to the underlining string would also reduce the amount of copies when going back/forth between the engine and the caller.

[bnoordhuis]

What is the ask here? A new C API JS_NewUncheckedTwoByteString(JSContext *ctx, const uint16_t *str, size_t len)? Would you like quickjs to support WTF-8 somehow?

direct access to the underlining string

Then we could never change the internal representation again (and never add things like, say, cons strings.)

Node likely has direct access to the UTF-16 String of V8.

No, V8 doesn't allow that either. You can ask it to copy out the string as a two-byte buffer though (two-byte because the result isn't guaranteed well-formed UTF-16.)

[Sytten]

Maybe WTF-8 would be good, I found another bug in our Buffer implementation.
When doing Buffer.from("ababc", "utf16le"), it should not convert the string to UTF-8 before decoding it but we don't have access to the data.

I am guessing we would not need if had a built-in ArrayBufferView that could be retrieved from all objects including String, right now we mock in the Rust by checking all the types that can convert to it.

Otherwise yes I think JS_NewUncheckedTwoByteString would work and we can handle that on the rust side.

[Sytten]

Other issue, not being able to write a high surrogate is a problem:

EDIT: I managed to fix it by keeping the high surrogate in the buffer instead of flushing it early

[Sytten]

I managed to do a good enough job with the string decoder but yeah having direct 2 bytes representation copy would be great.

[Sytten]

Looking into it further, V8 offers a ValueView over the String (https://v8.github.io/api/head/classv8_1_1String_1_1ValueView.html).
That allows the node code to read and write into the buffer of the string. The reading code is responsible to check if the string is one or two bytes.

[bnoordhuis]

String::ValueView is read-only, not read-write. It's also kind of dangerous; easy to introduce use-after-free errors by invalidating the backing string.

(And inefficient because it has to prep/flatten the string first but that's true of most string APIs in V8.)

[Sytten]

Right, I see that the write api is String::NewFromTwoByte, so yeah I am guessing what we were considering earlier would work. As long as we have a way to read and write the string as "two bytes" unchecked we can deal with the abstraction on the rust side.

[Sytten]

@bnoordhuis Great! Thanks for that, though we also need a way to read it similar to JS_ToCString, maybe JS_ToTwoByteString?

[bnoordhuis]

That's a taller order because it means encoding to UTF-16 when the string is a one-byte string.