embedding: string encoding-related functions NYI

[zmike]

These functions exist in CEF and require various encoding magicks to work properly.

cef_string_wide_cmp()
cef_string_ascii_to_utf16()
cef_string_ascii_to_wide()
cef_string_utf8_to_wide()
cef_string_wide_to_utf8()

[SimonSapin]

Maybe one possibility for dealing with "wide" characters is to check std::mem::size_of::<libc::types::os::arch::c95::wchar_t>(), unsafely transmute *wchar_t to *u16 or *u32, then assume that it’s meant to be UTF-16 (with surrogate pairs) or UTF-32.

Other than that, the principle is the same as for existing functions in https://github.com/servo/servo/blob/master/ports/cef/string.rs

• slice::raw::buf_as_slice gives you a &[T] slice from a *T raw pointer and a uint length

• &[u8], &[u16], and &[u32] all implement the .cmp(other) method

• str::from_utf8 checks that a &[u8] is valid UTF-8 and gives you an Option<&str>

• String:: from_utf8_lossy replaces invalid sequences with U+FFFD

• .utf16_units().collect::<Vec<u16>>() converts (UTF-8) str to UTF-16

• String::from_utf16 or String::from_utf16_lossy convert from UTF-16

• .chars().map(|c| c as u32).collect::<Vec<u32>>() converts (UTF-8) str to UTF-32

• .iter().map(|&c| std::char::from_u32(c).unwrap_or('\uFFFD')).collect::<String>() converts &[u32] UTF-32 to UTF-8, lossily

• Since u7 is not a thing, let’s assume that "ascii" means bytes (&[u8]) "that we promise are never ever gonna be over 0x7F". You can:

  • Decide to take it to the letter and panic! the task if they get it wrong: bytes.iter().map(|&byte| { assert!(byte <= 0x7F); byte as char })
  • Decide they’re lying, and "ascii" really means ISO/IEC 8859-1:1998 a.k.a. Latin-1: bytes.iter().map(|&byte| byte as char)
  • Decide they’re lying, and "ascii" really means UTF-8 (see above)

  Both map() return something that implements Iterator<char>. Call .collect::<Vec<char>> to get a Vec<char>. &[char] can be safely transmuted to &[u32] which represents UTF-32. Or call .collect::<String> to get an UTF-8 String on which you can call .utf16_units().collect::<Vec<u16>>() to get UTF-16. Make Utf16CodeUnits generic and give it a public constructor. rust-lang/rust#19042 will avoid the intermediate UTF-8 conversion.

• The search bar at the top of http://doc.servo.org/std/index.html can be very useful

Does this help?