You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
⚡️ Speed up function encoded_tokens_len by 39% in PR #231 (remove-tiktoken)
Here is an optimized version of your code.
The multiplication and conversion to int are very fast, but calling `len()` on a Python string first computes the length.
To minimize overhead, we can use integer arithmetic to avoid the float operations in `len(s)*0.3`. Multiplying by 0.3 is equivalent to multiplying by 3 and integer dividing by 10.
Here's the optimized code.
This avoids floating point multiplication and `int()` casting, and is slightly faster.
All comments and signatures are preserved.
0 commit comments