What is the official `similar` interface?

[johnmyleswhite]

While discussing #10525, it became clear that NullableArray{Int, 1} is an anomaly with regard to some of the assumptions made about AbstractArray{T, N}. Unlike Array{Int, 1}, which has an eltype of Int, NullableArray{Int, 1} has an eltype of Nullable{Int}. It is, to my knowledge, the only AbstractArray type whose first type parameter is not equal to its eltype. (This, of course, could be changed.)

This distinction affects how one interprets something like

x = NullableArray([1, 2, 3], [false, false, false])

similar(x, Float64, 3)

versus something like

x = NullableArray([1, 2, 3], [false, false, false])

similar(x, Nullable{Float64}, 3)

In the first variant, a call to similar(x, T, dims) produces a new object of typeof(x) <: AbstractArray{T, N} whose first type parameter is the passed argument T.

In the second variant, a call to similar(x, T, dims) produces a new object of typeof(x) <: AbstractArray{T, N} whose first type parameter is eltype(T).

More generally, this raises the issue: should the second argument to similar refer to the first type parameter of the resulting AbstractArray or should it refer to the eltype of the resulting AbstractArray? Or should we require all AbstractArray objects to match the type-parameterization assumptions that the first type is always the eltype and the second is always the order of the array?

[tkelman]

Worth thinking about whether #11557 ties into this too? It's seeming like using eltype in as many places as we do right now isn't as general as one might like.

[mbauman]

My stance on this is fairly firm: I think that what you choose to be the parameters of an AbstractArray subtype are implementation details specific to that type. They don't matter. You don't even need to parameterize an AbstractArray subtype (e.g., type BitArray{N} <: DenseArray{Bool,N}, or SparseMatrix{Tv,Ti} <: AbstractArray{Tv,2}). The fact that we typically do so in the same order as AbstractArray is just a code style convention.

What matters is what you report are the parameters to AbstractArray{T,N}; T must be its eltype, and N must be ndims. It's actually not possible to write methods that generically extract out the type parameters of an arbitrary leaf type, and that's a good thing (yes, I see the irony here with respect to #11547 😄).

So, for similar, I think that it'd be very confusing to allow specifying anything other than the AbstractArray's eltype because it's part of the generic AbstractArray interface. Going back to BitArray, similar(falses(10), Float64) is simply Array(Float64, 10) (even before #10525). I think that similar simply means "give me a chunk of mutable storage of a certain size that can support this element type."

[StefanKarpinski]

I pretty much agree with everything @mbauman says. It's a little strange that

similar(NullableArray([1], [false]), Float64)

produce an Array{Float64,1} but it seems right.

[mbauman]

Diagonal is an interesting case for similar in terms of its mutability (once setindex! is defined for its diagonal elements): #10001 (comment)

Is returning a Diagonal from similar mutable enough? Or does the entire array need to be mutable to satisfy the requirements of similar? What if the dimensions aren't square? Should it return a full array in that case? If so, then it'd be type unstable to return a Diagonal in the square case.

[johnmyleswhite]

I mostly agree with what @mbauman is saying, but I think that specific behavior is really weird. In particular, I think it's odd that similar(NullableArray([1], [false]), Float64) produces an Array, but similar(NullableArray([1], [false]), Nullable{Float64}) produces a NullableArray. What should similar(NullableArray([1], [false]), Nullable{Nullable{Float64}}) produce?

Alternatively, why are we saying that Array{Nullable{Float64}, 1) should never be in the image of similar(NullableArray, ...)} It seems strange to me that you could ever get an array back, but certain arrays could never be produced.

[tkelman]

Is returning a Diagonal from similar mutable enough?

I don't see how the thinking here could be any different than for SparseMatrixCSC, where similar returning anything other than another SparseMatrixCSC is just asking for an OutOfMemoryError.

What if the dimensions aren't square?

It looks like we don't have non-square Diagonal matrices right now, since the size is not explicitly stored as a field right now, it's assumed to always be square:

julia/base/linalg/diagonal.jl

Line 25 in ce90a6d

size(D::Diagonal) = (length(D.diag),length(D.diag))

#8240 would provide an equivalent though.

[mbauman]

I think similar is just spelled wrong for the way we've been using it. If I remember right, I think its name led to a lot of my initial confusion on its purpose. Perhaps a better spelling would be alloc, or maybe even AbstractArray. I see it as a way to ask for an AbstractArray in such a way that an existing array can say something about what kind of array is returned. So @johnmyleswhite: similar(NullableArray([1], [false]), Nullable{Nullable{Float64}}) should yield a NullableArray{Nullable{Float64}}.

[nalimilan]

Indeed, it helps to think about NullableArray{T} as being an efficient but fully compatible replacement for Array{Nullable{T}}.

[StefanKarpinski]

This is a good point: when similar was first introduced, we could not call abstract types. Now it might make more sense for these to be methods of abstract types like AbstractArray.

[tkelman]

What does call on an abstract type even mean? I guess the answer is "whatever we want it to," but that doesn't instill much confidence.

[StefanKarpinski]

"Give me a reasonable value of this type". For example, you could write Integer(x) to get a reasonable kind of Integer for x – e.g. if x is Float64 then an Int, if x is BigFloat then a BigInt. Admittedly this is a little different since you're not converting a single value. I'm not saying it's what we should do, but it's worth considering as something that we now have the ability to do.

[mbauman]

I was just spit-balling, but I'm liking it more and more. It certianly solves @johnmyleswhite's initial question: it'd be more obvious that AbstractArray(A, T, dims) gives you some subtype of AbstractArray{T}.

[tkelman]

I'm liking it less and less. It's really "give me a reasonable value of a reasonably chosen concrete subtype of this abstract type" which is a bit too ambiguous for my liking. Especially if there start being odd mapping rules from the concrete type of A (combined with T in some interdependent way) to the concrete type of the output. If that mapping is always the identity then at least it's consistent and maybe not too bad, but anything else would feel fishy.

[StefanKarpinski]

+1 about the identity mapping of element types (unless, of course, a different element type is asked for).