Revert ts5.9 changes for Uint8Array

[paulmillr]

⚙ Compilation target

es2022

⚙ Library

es2022, dom

Missing / Incorrect Definition

1. User sees weird error. "Uint16Array is not assignable to Uint16Array". First thoughts: WTF? What does this even mean? "But I don't use SharedArrayBuffer in my project!!"

TS2345: Argument of type 'Uint16Array' is not assignable to parameter of type 'Uint16Array<ArrayBuffer>'.
  Type 'ArrayBufferLike' is not assignable to type 'ArrayBuffer'.
    Type 'SharedArrayBuffer' is not assignable to type 'ArrayBuffer'.
      Types of property '[Symbol.toStringTag]' are incompatible.
        Type '"SharedArrayBuffer"' is not assignable to type '"ArrayBuffer"'.

2. User heads to blog post and sees "explicitly writing out Uint8Array<ArrayBuffer> rather than a plain Uint8Array"
3. User applies the changes.
4. The changes make library unusable in TS <=5.6.
5. User reverts to older TS to ensure his library doesn't require "latest ts".
6. There is zero other documentation about resolving the problem, so it's unclear how to solve it. It's also not obvious that () => new Uint8Array, () => Uint8Array.from([]) and () => Uint8Array.of() now all return subtyped uint8array, which fails in TS <=5.6

---

Previous issue text

ts5.7 change which made uint8arrays generic was okay.

ts5.9 introduced absolutely horrific change which breaks Uint8Array-heavy projects.

It seems like some changes made types too strict. It's unclear at which phase - perhaps during inference (i'm using infer). I'm now constantly getting those errors and it's really unhelpful - it's unclear what to do. Blog post recommendations (https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-9.html#libdts-changes) make arrays subtype-specific. It seems like this is a wrong way to fix things. Users should not be forced to always consume ArrayBuffer-based type arrays. It's perfectly fine to use either ArrayBuffer or SharedArrayBuffer based arrays, the changes are minor. The defaults should be able to consume both types!

// type Poly = Uint16Array
TS2345: Argument of type 'Poly' is not assignable to parameter of type 'Uint16Array<ArrayBuffer>'.
  Type 'ArrayBufferLike' is not assignable to type 'ArrayBuffer'.
    Type 'SharedArrayBuffer' is not assignable to type 'ArrayBuffer'.
      Types of property '[Symbol.toStringTag]' are incompatible.
        Type '"SharedArrayBuffer"' is not assignable to type '"ArrayBuffer"'.

Sample Code

I've spent a lot of time but was not able to produce a standalone example which reproduces the issue. There are a bunch of types at play and it's unclear which exact type causes the issue.

The code which produces errors is in commit cb76058 of https://github.com/paulmillr/noble-post-quantum

git clone https://github.com/paulmillr/noble-post-quantum
cd noble-post-quantum
git checkout cb76058
npm install
npm run build

[MartinJohns]

This is intentional and documented. They mention a way to fix it, which you didn't use.

https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-9.html#libdts-changes

[paulmillr]

@MartinJohns Nope. As I mentioned above:

The "fix" mentioned in blog posts makes arrays subtype-specific. The goal is for my code to consume any typed array - both ArrayBuffer and SharedArrayBuffer based. I am not using any "specific" methods. It seems like type inference screws up somewhere and makes stuff strict.

Changing this line from type Poly = Int32Array to type Poly = Int32Array<any> fixes things but it seems to be wrong. Shouldn't Int32Array<any> be the default?

https://github.com/paulmillr/noble-post-quantum/blob/cb76058c3328cb5ec09c007824ecd8de3f345dc4/src/ml-dsa.ts#L77

[Renegade334]

The problem, as you've identified, is where in-types and out-types collide. The array buffer view constructors almost always return a T<ArrayBuffer> by design; if that type is inferred and is then carried forward to type, say, a function parameter, then the change in variance can lead to issues.

These cases now require explicit annotations in certain places, to avoid inferring a narrowed type from a newly constructed array buffer view where narrowing is not desirable. In this instance, two changes are all that's needed for the project to build:

--- a/src/ml-dsa.ts
+++ b/src/ml-dsa.ts
@@ -75,7 +75,7 @@ export const PARAMS: Record<string, DSAParam> = {
 
 // NOTE: there is a lot cases where negative numbers used (with smod instead of mod).
 type Poly = Int32Array;
-const newPoly = (n: number) => new Int32Array(n);
+const newPoly = (n: number): Int32Array => new Int32Array(n);
 
 const { mod, smod, NTT, bitsCoder } = genCrystals({
   N,
diff --git a/src/ml-kem.ts b/src/ml-kem.ts
index fbd9a6a..18c8994 100644
--- a/src/ml-kem.ts
+++ b/src/ml-kem.ts
@@ -46,7 +46,7 @@ const { mod, nttZetas, NTT, bitsCoder } = genCrystals({
   Q,
   F,
   ROOT_OF_UNITY,
-  newPoly: (n: number) => new Uint16Array(n),
+  newPoly: (n: number): Uint16Array => new Uint16Array(n),
   brvBits: 7,
   isKyber: true,
 });

ts5.9 introduced absolutely horrific change which breaks Uint8Array-heavy projects.

This is only semi-accurate; these changes already existed from TS 5.7, but only for es2024 targets, which is where the changes to ArrayBuffer and SharedArrayBuffer made those interfaces diverge (#59417). Reverting the change which made this apply to all targets in TS 5.9 would only kick this can down the road until users start adopting es2024, and reverting the original change altogether isn't an option for the reasons detailed there.

[paulmillr]

The fix is reasonable, simple, and makes sense. Thanks.

However, it's completely non-obvious. If it's non-obvious to me (a lib author), then to my users it could be even worse. They would just slip on Uint8Array and call it a day. Or they would slip on Uint8Array and get more type errors in some cases.

Is there any way to make this kind of fix more documented? The blog post really does a bad job in this regard. Something like "Writing fn = () => new Uint8Array() would always return specific Uint8Array since ts5.9" would work.

[paulmillr]

Applied this fix in other libs -- just for the record, those are the diffs of changes that I thought are correct vs what's correct as per comment above

• paulmillr/noble-curves@878621b
• paulmillr/noble-hashes@46bf893
• paulmillr/noble-secp256k1@c00b81b

[paulmillr]

Ok it gets worse. Generic typed arrays aren't supported before ts 5.7. So a project with ts 5.0 can't compile fixes recommended by ts 5.9.

Ts 5.0 stuff doesn't work in 5.9. Ts 5.9 stuff (subtype) doesn't work in 5.0.

The docs must definitely be updated to accomodate for this, listing proper fixes.

[jakebailey]

TS 5.0 is well out of support, no? (See also, https://github.com/DefinitelyTyped/DefinitelyTyped?tab=readme-ov-file#support-window, which is as conservative as we get)

[paulmillr]

Instead of ts 5.0 I should have said "any ts <= 5.6". Which msft seems to support.

[paulmillr]

To reiterate, what happens:

1. User sees weird error. "Uint16Array is not assignable to Uint16Array". First thoughts: WTF? What does this even mean? "But I don't use SharedArrayBuffer in my project!!"

TS2345: Argument of type 'Uint16Array' is not assignable to parameter of type 'Uint16Array<ArrayBuffer>'.
  Type 'ArrayBufferLike' is not assignable to type 'ArrayBuffer'.
    Type 'SharedArrayBuffer' is not assignable to type 'ArrayBuffer'.
      Types of property '[Symbol.toStringTag]' are incompatible.
        Type '"SharedArrayBuffer"' is not assignable to type '"ArrayBuffer"'.

2. User heads to blog post and sees "explicitly writing out Uint8Array<ArrayBuffer> rather than a plain Uint8Array"
3. User applies the changes.
4. The changes make library unusable in TS <=5.6.
5. User reverts to older TS to ensure his library doesn't require "latest ts".
6. There is zero other documentation about resolving the problem, so it's unclear how to solve it. It's also not obvious that () => new Uint8Array, () => Uint8Array.from([]) and () => Uint8Array.of() now all return subtyped uint8array, which fails in TS <=5.6

[pnappa]

Yeah, I hit a similar cascade of changes due to the Uint8Array update.

For the record, I'm not for reverting the changes per se, I'm just wishing to highlight the additional work this required for developers (and as a consequence, politely request that the ramifications of these effects on downstream users is considered more thoroughly).

Our problem was a result of various Uint8Array and Buffer usages in NodeJS libraries & our code - some old libraries expect/return Buffer, and some expect/return Uint8Array.

I essentially had to rip out all Buffer usages, and insert a shim to convert them to/from Uint8Array, where expected. It's an unfortunate load of work required, which required a bunch of package updates, tsconfig updates, and removal of dependencies to resolve this. It's also an annoying performance downside (by way of having to go through constructors to get the right type) just to satisfy the types without casting.

This all came about because a developer was using VSCode which automatically uses the latest TS version, whilst the codebase was still a couple minor versions behind. Not wanting to resist the future, I had to update the codebase to 5.9, causing all these changes (which would have had to happen sooner or later, given this Uint8Array change).

Oh well! Life in the JS ecosystem 😄

[Nerivec]

Quick note, this also affects Buffer and other related types that use the same mechanism. It makes a mess with narrowing, and node built-ins (& external deps...).
Some of the related issues were marked "Needs investigation" while others "Working as intended"...
#62168
#61793
Seems the whole thing is rather "uncertain" at the moment...

[RyanCavanaugh]

Effectively ~all of these errors people are encountering represent problems that really could have induced actual runtime errors. For better or worse we do generally treat "This was not actually reliably working in prior versions" lib situations as bugs. The docs can always do better and I'd be happy to see a useful update to the appropriate location, but there's not a lot of universally-applicable guidance we can give in these situations since the exact manifestation of type errors can vary quite a lot depending on what's going on.