Skip to content

std.unicode fixes #3987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Dec 28, 2019
Merged

std.unicode fixes #3987

merged 5 commits into from
Dec 28, 2019

Conversation

@daurnimator
Copy link
Contributor

@daurnimator daurnimator commented Dec 28, 2019

Bug fixes, cleaner and faster :)

@daurnimator
std: fix unicode encoding of astral plane codepoints to utf16
a81c0ba
@daurnimator
std: add warm up phase to benchmark
8b72eed 
In my tests, whatever ran first was getting much better numbers.
Additionally, add alignment requirements so that comparison is fair.
@daurnimator
std: optimise utf8ByteSequenceLength
5843a6e 
Also tested (but not as fast):
```zig
pub fn utf8ByteSequenceLength(first_byte: u8) !u3 {
    const len = @clz(u8, ~first_byte);
    if (len == 0) return 1;
    if (len < 4) return @intcast(u3, len);
    return error.Utf8InvalidStartByte;
}
```
@daurnimator daurnimator added the standard library This issue involves writing Zig code for the standard library. label Dec 28, 2019
@daurnimator daurnimator mentioned this pull request Dec 28, 2019
Closed
@data-man
Copy link
Contributor

data-man commented Dec 28, 2019

Proposal:

pub const CodePoint = u21;

@daurnimator
Copy link
Contributor Author

daurnimator commented Dec 28, 2019

Proposal:

pub const CodePoint = u21;

I thought about that but decided I'd like to see how #3806 panned out first.

@daurnimator
std: simplify utf8ToUtf16Le
ab60654 
Also faster, on my machine unicode/throughput_test.zig now gives e.g.
> original utf8ToUtf16Le: elapsed: 1048 ns (0 ms)
> new utf8ToUtf16Le: elapsed: 971 ns (0 ms)
@daurnimator
std: unicode codepoints are 21 bits
edb5deb
@andrewrk andrewrk merged commit cb02125 into ziglang:master Dec 28, 2019
@andrewrk
Copy link
Member

andrewrk commented Dec 28, 2019

Nice!

@daurnimator daurnimator deleted the std.unicode-fixes branch December 28, 2019 23:16
@daurnimator daurnimator mentioned this pull request Nov 11, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

standard library This issue involves writing Zig code for the standard library.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@daurnimator @data-man @andrewrk