Remove hard-coded `Vector` fields and add vectors constructors

[theogf]

Right now kernels like LinearKernel have two problems: they can only take Real arguments (no way to pass kernel.c) as an argument and also they do not allow for different AbstractVector types to be stored.
The interested use case is when using GPUs, one cannot do kernelmatrix on CuArrays without this.

struct LinearKernel{Tc<:Real} <: SimpleKernel
    c::Vector{Tc}

    function LinearKernel(c::Real)
        @check_args(LinearKernel, c, c >= zero(c), "c ≥ 0")
        return new{typeof(c)}([c])
    end
end

should be

struct LinearKernel{Tc<:Real,Vc<:AbstractVector{<:Tc}} <: SimpleKernel
    c::Vc
    function LinearKernel(c::V) where {T<:Real, V<:AbstractVector{T}}
        @check_args(LinearKernel, first(c), first(c) >= zero(c), "c ≥ 0")
        return new{T,V}(c)
    end
    function LinearKernel(c::Real)
        @check_args(LinearKernel, c, c >= zero(c), "c ≥ 0")
        C = [c]
        return new{eltype(C),typeof(C)}(C)
    end
end

[devmotion]

Or we could just not make it a vector at all but a scalar 😛

[devmotion]

If we stick with vectors (e.g. because of Flux/Zygote), in the example I guess LinearKernel(c::Real) could be a simple outer constructor LinearKernel(c::Real) = LinearKernel([c]).

[theogf]

But then how do I do my inplace updates! 😆. Also nobody like scalars! Especially GPUs!

[devmotion]

One could eg use ParameterHandling and perform in-place updates of the resulting vector 😉 But then you would still have to go back and reconstruct the kernel.

[willtebbutt]

Did we decide that Refs were a bad idea? i.e. we shouldn't be using Ref(5.0) doesn't play nicely with Functors.jl?

[theogf]

Ref is not captured by Flux.params AFAICT

[devmotion]

I still think one could consider dropping support for Flux.params and switching to scalar parameters instead. Both Ref and Vector seem a bit unnatural.

[willtebbutt]

Ref is not captured by Flux.params AFAICT

:(

I still think one could consider dropping support for Flux.params and switching to scalar parameters instead. Both Ref and Vector seem a bit unnatural.

I can definitely see the appeal of this, but I would really like this ecosystem to be compatible with the Flux one e.g. so that you can map the inputs to a GP through a DNN defined using Flux. I'm not saying that this is necessarily a good idea in general, see [1], but I'd at least like to avoid ruling it out.

If we didn't use the Functors API, is there another way to interop nicely with Flux?

[1] - https://arxiv.org/abs/2102.12108

[theogf]

One could eg use ParameterHandling and perform in-place updates of the resulting vector But then you would still have to go back and reconstruct the kernel.

You mentionned ParameterHandling and it looks very nice and it seems the only requirement on our side would be to implement the flatten approach. However that means any additional operations on the kernels would need to follow the ParameterHandling approach no?

[willtebbutt]

Not necessarily. You can implemented flatten for all of the things in your model, but my preferred approach is to use ParameterHandling.jl to make it easy to write a function which builds your model.

For example, see here.

This is a different style from the one that you prefer though @theogf .

[devmotion]

I haven't thought this through but maybe it is possible to write a custom Flux layer that contains only the parameter vector and a function that builds the kernel:

struct KernelLayer{P,F}
    params::P
    model::F
end

Functors.@functor KernelLayer (params,)

(f::KernelLayer)(x, y) = f.model(f.params)(x, y)
...

[theogf]

Not necessarily. You can implemented flatten for all of the things in your model, but my preferred approach is to use ParameterHandling.jl to make it easy to write a function which builds your model.

I think I got this part. However I wanted to say that I am also happy with the destructure approach of Flux/Functors as well, which seems "similar" to ParameterHandling.

My point is that if we can have params, reconstruct = flatten(my_very_complicated_kernel) I am also happy with that :)

[willtebbutt]

I haven't thought this through but maybe it is possible to write a custom Flux layer that contains only the parameter vector and a function that builds the kernel:

Interesting -- how are you imagining this interacting with Flux @devmotion ? I don't think I'm seeing where you're going with this yet.

[devmotion]

As I said, I didn't spend much thought on it. But the main idea would be that instead of forcing all kernels to be mutable we just have one kernel that just wraps the parameters of a possibly complicated kernel in a Flux-compatible way (e.g. as a Vector) and includes a function that constructs the kernel from the parameters. Due to the functor definition, Flux would only work with and update the parameters.

[theogf]

As I said, I didn't spend much thought on it. But the main idea would be that instead of forcing all kernels to be mutable we just have one kernel that just wraps the parameters of a possibly complicated kernel in a Flux-compatible way (e.g. as a Vector) and includes a function that constructs the kernel from the parameters. Due to the functor definition, Flux would only work with and update the parameters.

Ok so that's like a complementary idea right? We would need a conversion from any kernel to a KernelLayer to be then used with any optimization tool?
The conversion could be handled in back-end by ParameterHandling or something handcrafted.