Action Space Meaning

[zsunberg]

If I use

ReinforcementLearningBase.get_actions(env::MyEnv) = [-1, 1]

the DQN example will throw an out of bounds error. I think this is because of the -1 action. Is this the intended behavior? Do actions need to be natural numbers, or can they be numbers that have physical meaning for the problem?

Complete code to reproduce the error: https://gist.github.com/zsunberg/a6fdebed95c518ba4e06e1d5d782a30a

[findmyway]

Hi @zsunberg Thanks for reporting.

No, it's not the intended behavior.

The return of get_actions can be of arbitrary vectors that have physical meaning. Some policies handled it correctly. But the problem here is that the DQN algorithm requires that actions need to be of discrete integers starting from 1. And DQN (and many other algorithms working discrete action spaces ) doesn't do this kind of check currently.

I'll add a checker and review all the existing algorithms to make sure acion space is handled correctly later.

[findmyway]

To be more specific. Here the return of agent(env) is -1 or 1. And then they are put into the experience replay buffer. But the DQNLearner expects that actions are natural numbers.

One solution that comes up to my mind is to encode the actions at runtime. But doing so will have a little overhead.

Another solution is to wrap the env implicitly for all the environments with an Action Space of AbstractVector.

[findmyway]

JuliaReinforcementLearning/ReinforcementLearningBase.jl-Archive#88 Improved the ActionTransformedEnv to support encoding actions.

JuliaReinforcementLearning/ReinforcementLearningCore.jl#143 Split the run into two steps to allow injecting checkers.

The next step is to implicitly convert environments in the checker. An example is

https://github.com/JuliaReinforcementLearning/ReinforcementLearningBase.jl/pull/88/files#diff-49de12fd0d5a5f0d909b2db8e40f9add27b9590d5a746ad7d76d50b2603de719R10-R13

That may need a bit of work. I'll work on it at the end of this week.

[zsunberg]

It seems like the performance penalty for mapping is not too huge. Perhaps instead of introducing ActionTransformedEnv, you could, by default, assume that you need to do the mapping, and if solvers want to eek out all of the performance, they can specialize on action spaces that are of type Range or Base.OneTo.

[findmyway]

It seems like the performance penalty for mapping is not too huge.

Yes, I agree. Thanks for your suggestions. 👍