[vm/ffi] Array should expose an iterable

[dcharkes]

Pointers expose asTypedList which yields a TypedData which can be iterated over. Arrays do not.

We should either create an Array.asTypedList set of extensions, or provide another way to iterate over an Array's content. (Bonus: Arrays already carry a lenght, so .asTypedList() does not require a length argument.)

@timsneath thanks for reporting!

[Sunbreak]

Any plan?

[maks]

@dcharkes in the meantime is there any chance to document somewhere how to deal with ffi Arrays in structs. Maybe in the existing samples?

If it wasn't for my happening to find this comment (thank you @Sunbreak !!! 🙏🏻 ) I would have had no idea how to deal with getting a string out of a inline Array in a C struct I'm working with - the Dart docs for ffi Array are not really helpful at the moment.

[lrhn]

Consider exposing it as a (fixed-length) List, maybe List<X> get elements;.
(I'm assuming it has to be an extension getter, so the Dart type X can depend on the native array element type.)

[Sunbreak]

Here is the use case

https://github.com/woodemi/twaindsm.dart/blob/ecc1f3d242bbdca56e71ea75ce0f6d88fd165b42/lib/twaindsm.dart#L517-L518

@ffi.Packed(2)
class TW_IMAGEINFO extends ffi.Struct {
  @ffi.Array.multi([8])
  external ffi.Array<TW_INT16> BitsPerSample;
}

https://github.com/woodemi/twaindsm.dart/blob/ecc1f3d242bbdca56e71ea75ce0f6d88fd165b42/lib/structs.dart#L80-L84

extension TWImageInfo on TW_IMAGEINFO {
  Map<String, Object> toMap() => {
      'BitsPerSample': spreadBitsPerSample(),
    };

  Iterable<int> spreadBitsPerSample() sync* {
    for (var i = 0; i < 8; i++) {
      yield BitsPerSample[i];
    }
  }
}

---

With List<X> get elements it would be

extension TWImageInfo on TW_IMAGEINFO {
  Map<String, Object> toMap() => {
      'BitsPerSample': BitsPerSample.elements,
    };
}

[shikharish]

@dcharkes I would like to work on this issue. Please can you give me an overview about it?

[dcharkes]

Consider exposing it as a (fixed-length) List, maybe List<X> get elements;. (I'm assuming it has to be an extension getter, so the Dart type X can depend on the native array element type.)

@lrhn Do you suggest List<X> instead of Iterable<X> so that people can query the length and have O(1) random access?

Array's of ints and floats

I think for Array<Int8> (and the other int and floating types) asTypedList is more general, it implements list.

/// Bounds checking indexing methods on [Array]s of [Int8].
@Since('2.13')
extension Int8Array on Array<Int8> {
  external int operator [](int index);

  external void operator []=(int index, int value);

  /// Creates a typed list view backed by memory this Array is a view on.
  ///
  /// Uses the array dimensions for determining the list.
  @Since('3.4')
  external Int8List get elements;
}

@shikharish

• sdk/lib/ffi/ffi.dart The API needs to be introduced here.
• sdk/lib/_internal/vm/lib/ffi_patch.dart The implementations for ints and floats can be introduced here.
• (and the code generation file that generates part of those files)

@patch
extension Int8Array on Array<Int8> {
  Int8List get elements {
    final length = _nestedDimensionsFlattened;
    if(_typedDataBase is Pointer) {
      return (_typedDataBase as Pointer).cast<Int8>().asTypedList(length);
    }
    assert(_typedDataBase is TypedData);
    return Int8List.sublistView(_typeDataBase as TypedData, 0, length);
  }
}

Make sure to write a bunch of tests which exercise things with different base offsets etc. I think you can currently only write tests which use Structs, because that's the only way to obtain Arrays currently.

Arrays of Structs, Unions, Arrays, and Pointers

For Array<MyStruct> we can't use TypedData. So we'd need to return a class that implements List<MyStruct> (and throws on modification).

The API would be somewhat similar:

/// Bounds checking indexing methods on [Array]s of [Struct].
@Since('2.13')
extension StructArray<T extends Struct> on Array<T> {
  /// This extension method must be invoked on a receiver of type `Pointer<T>`
  /// where `T` is a compile-time constant type.
  external T operator [](int index);
  
  /// Creates a typed list view backed by memory this Array is a view on.
  ///
  /// Uses the array dimensions for determining the list.
  @Since('3.4')
  external List<T> get elements;
}

The implementation would need to return object that implements the List interface, but we don't want to actually fabricate the list.

@patch
extension StructArray<T extends Struct> on Array<T> {
  @patch
  external List<T> get elements {
    return _StructArrayList<T>(this);
  }
}

class _StructArrayList<T extends Struct> with UnmodifiableListMixin {
  final Array<T> _array;

  @override
  T operator [](int index) => _array[index];

  // all the other methods that need to be implemented such as 
}

All of the implementation should be possible in the two files mentioned before.

This is a rough sketch of the implementation, I might have missed some details. 😄 Let me know if you can make it work or if you have more questions.

[lrhn]

@lrhn Do you suggest List<X> instead of Iterable<X> so that people can query the length and have O(1) random access?

Yes. Because we can, and if we don't, we can expect users to do .toList() on the iterable all the time.
Which is exactly what we don't want.

[shikharish]

@dcharkes Thank you for the detailed explanation. Will make a PR soon.

I had a question, which is a bit off-topic. Is there something like "array to pointer decay" in Dart? How are arrays passed as parameters?