sparse setindex and stored zeros

[mauro3]

During construction of sparse matrices and when using setindex!, entries with values 0 are not included/dropped from the underlying storage. This is no good for certain applications. In particular when preallocating sparse matrices.

For example we currently got:

julia> sparse([1],[1], [0.], 5 ,5)
5x5 sparse matrix with 0 Float64 entries:

Instead

julia> sparse([1],[1], [1.], 5 ,5)
5x5 sparse matrix with 1 Float64 entries:
        [1, 1]  =  0.0

or

julia> sparse([1],[1], [1.], 5 ,5, keepzeros=true)
5x5 sparse matrix with 1 Float64 entries:
        [1, 1]  =  0.0

My use case are preallocated Jacobian matrices when solving PDEs. Usually the pattern of potential non-zeros in the Jacobian is known, so it can be preallocated. However, when repeatedly filling the Jacobian during the calculation there is no guarantee that its underlaying storage will be kept fixed.

Maybe there could be a sparse type for fixed size sparse matrix, say SparseMatrixCSCfixed. Constructed with sparse([1],[1], [1.], 5 ,5, keepzeros=true). Then setindex! would not drop zeros and error when trying to assign to a not-allocated entry.

[mlubin]

I agree that a keepzeros flag would be useful. But anyway if you're preallocating why not just use dummy nonzero values?

[mauro3]

Yes, I use ones to preallocate, which is fine. But the other problem does not have a workaround: if I update the Jacobian J[i,j]=someval and someval happens to be zero at that iteration then the whole sparse matrix is redone, only to be restored to the same state once someval is non-zero again.

[JeffBezanson]

dup of JuliaLang/LinearAlgebra.jl#60

Looks like it is possible to have explicit zeros by calling the SparseMatrixCSC constructor directly.

[mauro3]

Thanks for pointing out that issue. I am aware that zeros can be stored in a sparse matrix, but not easily as setindex! will just purge that entry from the matrix. I think there should be a way to keep them when using standard indexing. The Julian way to do this would be with a new type.

[mlubin]

I don't want to rehash JuliaLang/LinearAlgebra.jl#60, but I believe the conclusion is that explicit zeros are allowed in SparseMatrixCSC because they're generally harmless, but that high-level interfaces like getindex are free to remove them. J[i,j]=someval in a loop is potentially a very inefficient way to update a sparse matrix anyway. If you're aiming for efficiency you have to work at the level of CSC format.

[JeffBezanson]

That's true; if you want (1) fast changes to individual elements, (2) keeping zeros in place, it seems the best thing to do is directly manipulate the representation.

[tkelman]

The Julian way to do this would be with a new type.

I don't think a new type would be a good idea for dealing with this, at least not with the current design. You'd have to reimplement a lot of methods for any new type, or introduce an AbstractCSC level to the hierarchy. Maybe a boolean type parameter could work, but that could also get pretty messy.

I think the best solution at the moment would be to add a specific setexplicitzero! method for this. I think there's already a non-exported helper function that can be used to do this, but I don't remember exactly which one.

[ViralBShah]

New type is definitely not a good idea. The best thing for now is perhaps to have new methods for this. We could even have a module with various methods that store their zeros.

[mauro3]

Thanks for all the input. So I suggest two things:

• add a keyword to the constructors keepzeros to force keeping zeros
• mention in docs that zeros are dropped in constructors and setindex!

Let me know if that is good and I can make a pull request for those two points

Also, I just discovered this:

julia> sparse([1,1], [2,2], [1,-1])
1x2 sparse matrix with 1 Int64 entries:
        [1, 2]  =  0

Is this a bug?

[ViralBShah]

I guess the only constructor where this makes sense is sparse(). We can certainly do that.

Yes - that is a bug. Could you file a separate issue for that?

[mauro3]

keepzeros also applies to spdiagm.

Filed issue #9928

[ViralBShah]

I feel that a better approach may be to have AbstractSparseMatrixCSC, and have all the methods implemented on the abstract type. The concrete types can be SparseMatrixCSC (default without stored zeros) and SparseMatrixCSCStoredZeros. They will share largely the same implementation, except that there will be special methods where stored zeros have to be treated differently.

I prefer the tags to be in the type system, rather than in the function arguments or flags in the data structure, and multiple dispatch will do the rest for us. The code will have to be refactored to use accessor methods, but I feel that overall, this is the better approach.

[tkelman]

I thought the majority opinion in #9928 was that no one actually thinks removing structural nonzeros with values numerically equal to zero is all that important.