Performance issue with models running on CPU

[findmyway]

Problem

@norci mentioned in JuliaReinforcementLearning/ReinforcementLearningZoo.jl#87 (comment) that, there may be some potential performance improvements with algorithms running on CPU only.

Currently the experience buffer is using CircularArrayBuffer to store data. When doing batch updating, we use select_last_dim function to create a view. But according to the doc: Copying-data-is-not-always-bad, it may be faster to turn the view into an Array first before feeding it into Flux models.

Initial investigation shows that, by transforming the SubArray into Array, the average time per step of experiment E`JuliaRL_BasicDQN_CartPole` will decrease from ~0.00128 to ~0.00107. When the model is more complex, the improvement becomes larger.

Models on GPU are not affected

Note that models on GPU will not be affected, since SubArray will be automatically converted to Array first:

https://github.com/JuliaGPU/CUDA.jl/blob/f31cbe22b4baba872a48bcb48e9f60e712f653fc/src/array.jl#L206

And we have already forced the SubArray of Array to be converted into CuArray instead of SubArray of CuArray here:

ReinforcementLearningCore.jl/src/utils/device.jl

Lines 16 to 32 in a94544c

	send_to_device(
	::Val{:gpu},
	x::Union{
	SubArray{<:Any,<:Any,<:Union{CircularArrayBuffer,ElasticArray}},
	Base.ReshapedArray{<:Any,<:Any,<:SubArray{<:Any,<:Any,<:CircularArrayBuffer}},
	SubArray{
	<:Any,
	<:Any,
	<:Base.ReshapedArray{
	<:Any,
	<:Any,
	<:SubArray{<:Any,<:Any,<:Union{CircularArrayBuffer,ElasticArray}},
	},
	},
	ElasticArray,
	},
	) = CuArray(x)

Possible Solutions

1. Nothing to change (But need to document it somewhere)

Users need to manually add a layer in models to convert the SubArray into Array first when working in CPU only devices .

2. Automacally convert SubArray of Array into Array in the send_to_host function.

This is the easiest way. But I think it breaks the meaning of send_to_host. Afterall, the SubArray of Array is already in CPU.

[norci]

Let's treat SubArray as something special, so we don't care it's on the host side or not.

what about

julia> send_to_device(::Val{:cpu}, x::SubArray) = Array(x)

julia> send_to_device(Val(:cpu), @view (1:10)[[1,2,3]])
3-element Array{Int64,1}:
 1
 2
 3

[findmyway]

Yeah, that's solution#2 I mentioned above. Let me have a think to make sure it doesn't bring in unexpected side effects.

[findmyway]

After reading the source code of Adapt.jl, I think the send_to_device function needs redesign. Things changed a little since Adapt.jl@v1.