What to do with VBuffer?

[eerhardt]

VBuffer is a very critical, fundamental type in ML.NET. And as such, it has some critical performance considerations and characteristics it needs to adhere to.

However, it is not as intuitive of a type as it could be. We even wrote a Care and Feeding doc that describes some common pitfalls and other things to be aware of when using the type.

At its core, VBuffer has 2 responsibilities:

1. It is a vector that can be either dense or sparse.
2. It is a reusable/cached "buffer" to allow minimal allocations and garbage collections.

We should do some investigation into what types of improvements we can make to this type.

• Is there something "base class library"-ish that can be introduced in .NET that would make this type better?
  • Potentially can we use ArrayPool to take care of the "buffer"-ness? (see https://github.com/dotnet/corefx/issues/4547)
  • Are there other BCL types we could use or introduce?
• Are there alternative API designs that we should consider/incorporate here to make the type more intuitive and less error prone by users?
  • i.e. should some members be encapsulated?
  • Should we split VBuffer into two types SparseVector and DenseVector?

/cc @TomFinley @Zruty0

[Zruty0]

As for the ArrayPool, there are some problems with this approach:

• Since a pair of buffers (VBuffer) is a struct, this could cloud the ownership problem. It's not a huge deal in our codebase, since cloning VBuffers is always viewed as a coding mistake.
• As of now, we don't really 'release' arrays, but just lake GC take them when it feels like.
• We would still have to keep track of 'logical size' of the array that we rented.

[Zruty0]

To recap an online discussion: we should separate the two concepts of VBuffer:

• Serving as a polymorphous 'dense or sparse' vector
• Retaining arrays if possible and tracking 'capacity' and 'logical length' of arrays.

For the first one, the plan is to get rid of polymorphous behavior, and formally separate into SparseVector and DenseVector

For the second one, the current approach is fine (i.e. DenseVector<T> will be a struct with two fields: T[] Values and int Length)

[markusweimer]

Retaining arrays if possible and tracking 'capacity' and 'logical length' of arrays.

Isn't this something where Span<T> could help? That is also designed to reuse the memory of differently-sized, array-ish accessors, right?

Serving as a polymorphous 'dense or sparse' vector

The closest thing to that in the BCL is Tensor<T>. Do we have ideas on how to merge VBuffer with that?

[ericstj]

You're DenseVector<T> actually sounds very much like Memory<T> except Memory<T> can also support an offset and be backed by a native pointer.

Another scenario we have for this is wrapping memory that's owned by someone else. Consider the TensorFlow case. TF will allocate tensors (and unfortunately doesn't seem to provide an hook for callback allocation or pre-allocation of output tensors) and the ownership of these tensors is transferred to the caller. For Python they are able to wrap the output tensor in an ndarray since ndarray permits wrapping memory: https://docs.scipy.org/doc/numpy-1.13.0/user/c-info.how-to-extend.html#c.PyArray_SimpleNewFromData.

Unfortunately since the tensor/vector type in ML.NET is VBuffer<T>, VBuffer<T> only exposes managed arrays, and managed arrays must be .net arrays on the managed heap we cannot do this.

To address this scenario consider changing the type that ML.NET exposes for tensor/vector data to something that can wrap a native pointer (like Memory/Span) or modify VBuffer itself so that it can represent data backed with a native pointer. I started experimenting with this idea a few months back in with Tensor.

I do think we can try to merge the ideas of VBuffer with Tensor. We may find that both have good ideas and we end up with a hybrid.

[KrzysztofCwalina]

I quickly searched through current ML.NET APIs and it does not seem like there are any APIs that return VBuffer (except some inconsequential factory methods). And so it seems like we could use Span<T> instead of Memory<T>. If we indeed could use Span<T>, I think it would be a non-trivial perf win. Not only because of Span<T> being faster than a wrapper over arrays, but also because it could allow us to avoid heap allocations and pooling in many cases, i.e. we would stackalloc Span<T>.