Dangerously vague meaning of .len() and .truncate() on strings

[mzabaluev]

There is a number of operations on the built-in string slices and String that are specified terms of length or the number of unspecified collection items.
Notably, the implementation of Collection method len and String's method truncate. The documentation of these methods does not say whether the length is in bytes or UTF-8 codepoints; in practice it's bytes. This is hinted, but not said explicitly, in the general description of str.

Contrastingly, many popular programming environments such as Java, C#/CLI, and Qt, have similar string operations in terms of UTF-16 characters (UTF-16 is not free of the same issues, but it still generally works as wide-char Unicode for the masses, unless you are into dead or obscure scripts or emoji chat software). These operations are so familiar that many people would use them without looking up their precise definition, and in case of Rust, they may end up being wrong even if the documentation gave all possible warning. Their code will compile and work until it meets its first non-ASCII string, which in some sad cases might not happen until after shipping. Double grief if a mistakenly interpreted value passes into unsafe code and causes hard-to-debug trouble, putting a stain on the image of Rust as a safe language for the developers concerned.

I think this problem could best be mitigated by careful API design. I've got the following suggestions:

• Deprecate truncate in favor of truncate_bytes, and add truncate_chars alongside.
• Move len out of the Collection trait into a new subtrait SizedCollection, which standard strings will not implement. The byte length for strings will be always one method call away behind as_bytes(), which makes the intent explicit. This trait split will also allow implementations of linked lists without the explicitly maintained size counter, and maybe enable some clever lock-free concurrent collections in the future.

[aturon]

cc @alexcrichton @brson @huonw @nikomatsakis

See this PR for the rationale for the current design. I don't think the documentation has been updated to make this design completely clear.

[mzabaluev]

Good that the convention has been thought out, but no amount of documentation will change the fact that "string length" means "length in something that represents Unicode character code points to a stunningly practical degree" to a huge number of practicing programmers around. as_bytes/into_bytes could be a reasonable escape hatch.

[mahkoh]

They will also believe that char is a one byte object, int is four bytes, and that as_bytes returns a utf16 representation of the string.

[mzabaluev]

@mahkoh: somewhat true, but that is easily caught by the compiler (making int a distinct type from both i32 and i64, not merely an alias, is a wonderful idea, BTW). String length miscalculations can only be caught at runtime, and due to the "ASCII divide", they may not become apparent in pre-release testing, or even in early deployment.

[mahkoh]

It can only be caught in safe code. That a program fails in a predictable way because the developer forgot to read the documentation seems like the correct behavior. Rust should not make its built-in types harder to use in order to support people who already write broken code that assumes that UTF-16 is fixed width.

"length in something that represents Unicode character code points to a stunningly practical degree"

The number of code points is a useless metric outside of very few applications. I don't see any application in which this would be more useful than the number of bytes or the number of graphemes. I assume that much more serious logic errors will become apparent in code that depends on len() having this meaning and that doesn't even have tests that test non-ASCII input. Especially in FFI code that passes pointers to the string around because the programmer clearly isn't aware of the internal representation of strings.

[alexcrichton]

While it is true that one can mix up what truncate does in terms of bytes or characters, I don't think there's a way to get around the fact that a string needs to have a length, and that method really does need to be called len (in Rust style).

On one hand we want Rust strings to be valid utf-8 at all time, along with all that it entails. This involves having all function be "unicode aware" in terms of character boundaries, multi-byte characters, etc. It also means that we need to modify apis in some places to generate an error when certain contracts are violated. For example, today all functions fail if something operates in the middle of a multi-byte character.

On the other hand, Rust strings should be usable to those who are not utf-8 gurus. Coming from any other language I need to be able to quickly inspect a strings and operate on it. To me this is a very important aspect which needs to be retained when working with strings.

These two goals are, of course, at odds, however. A usable string does not necessarily do the right thing in the "easy case", and the "correct unicode string" is often quite difficult to use (and not necessarily ergonomic). Today we try to strike a balance between these two goals, but we have to remember that these are both at odds and we need to find the right middle-ground.

[thestinger]

I think the only sensible definitions of len are the length in bytes (that's the unit used for all slicing / indexing operations, as it's transparently UTF-8) or the true length of the string as defined by Unicode. It would need to take into account zero-width and double-width code points, so it would be O(n) and would involve table lookups. That would be usable for stuff like monospace text alignment, but I think the current definition is better because it matches the API for efficient slicing and so on.

[mzabaluev]

Understood; let's close it then.