POMDPs.generate_s, POMDPs.generate_sr... and POMPDs.gen, PIMPDs.sampletype.

[miguzca]

Hi all!,

Thank you for the nice framework!. I am super new to Julia, and I am trying to run an implementation of a project that was done a while a go, but it seems that the implementation does not run in new versions of POMDPs framework and I am trying to figure out how to make it run. The implementation is provided here (https://github.com/biy001/UAV-autonomous-landing/blob/master/UAVLanding.jl) and I have noticed that there are mainly issues in the way "generate_s", "generate_o" and "generate_sr" are implemented in new versions of the framework. In current versions, apparently you have to use "POMDPs.gen" and return a NamedTuple with next state (sp), observation (o), and reward (r). So, basically I have modified for example

function POMDPs.generate_s(pp::UAVchaseProblem, s::MDPState, a::MDPAction, rng::AbstractRNG)
    p = mdp(pp)
    # calculate target state
    target_dt_distance = p.dt*p.target_velocity
    ## sensor_noise = Base.rand(Normal(0, p.target_std), 2)
    target_dynamic_cov = Matrix(p.target_std*Diagonal{Float64}(I, 2))
    sensor_noise = rand(rng, MvNormal(target_dynamic_cov))
    ## sensor_noise = SVector(0,0)
    curr_targ_true = s.targetPose_true + target_dt_distance
    if if_in_FOV(pp, s)
        curr_targ = curr_targ_true + sensor_noise
    else
        curr_targ = s.targetPose + target_dt_distance + sensor_noise # next_target_pos(p, s.targetPose)
    end
    # calculate UAV state
    curr_angle = s.uavHeading + a.angle
    xy_dt_distance = p.dt*a.xy_speed*SVector(cosd(curr_angle), sind(curr_angle)) # careful
    z_dt_distance = p.dt*a.z_speed
    xyz_dt_distance = SVector(xy_dt_distance[1], xy_dt_distance[2], z_dt_distance)
    curr_pos = s.uavPose + xyz_dt_distance
    return MDPState(curr_pos, curr_angle, curr_targ, curr_targ_true)
end

with

function POMDPs.gen(pp::UAVchaseProblem, s::MDPState, a::MDPAction, rng::AbstractRNG)
    p = mdp(pp)
    # calculate target state
    target_dt_distance = p.dt*p.target_velocity
    ## sensor_noise = Base.rand(Normal(0, p.target_std), 2)
    target_dynamic_cov = Matrix(p.target_std*Diagonal{Float64}(I, 2))
    sensor_noise = rand(rng, MvNormal(target_dynamic_cov))
    ## sensor_noise = SVector(0,0)
    curr_targ_true = s.targetPose_true + target_dt_distance
    if if_in_FOV(pp, s)
        curr_targ = curr_targ_true + sensor_noise
    else
        curr_targ = s.targetPose + target_dt_distance + sensor_noise # next_target_pos(p, s.targetPose)
    end
    # calculate UAV state
    curr_angle = s.uavHeading + a.angle
    xy_dt_distance = p.dt*a.xy_speed*SVector(cosd(curr_angle), sind(curr_angle)) # careful
    z_dt_distance = p.dt*a.z_speed
    xyz_dt_distance = SVector(xy_dt_distance[1], xy_dt_distance[2], z_dt_distance)
    curr_pos = s.uavPose + xyz_dt_distance
    return (sp=MDPState(curr_pos, curr_angle, curr_targ, curr_targ_true))
end

and

function POMDPs.generate_sr(pp::UAVchasePOMDP, s::MDPState, a::MDPAction, rng::AbstractRNG)
    sp = generate_s(pp, s, a, rng)
    return sp, reward(pp,a, sp)
end

with

function POMDPs.gen(pp::UAVchasePOMDP, s::MDPState, a::MDPAction, rng::AbstractRNG) #(**)
    #here generate_s is called in the old implementation.
    sp=@gen(:sp)(pp,s,a,rng) #Seems to be calling recursively the implementation we are doing here. (*)
    return (sp=sp, r=reward(pp,a, sp))
end

But I get an "StackOverflowError" because it seems to be looping (calling recursively (*) and (**)). So I guess I just do not know how to call the different implementations of "POMDPs.gen", could you please explain me how to do it?. Similarly, there is an issue with

POMDPs.sampletype(::Type{MDPStateDistribution}) = MDPState

And I have modified the code like this

function sampletype end

sampletype(d::Any) = sampletype(typeof(d))
sampletype(t::Type) = throw(MethodError(sampletype, (t,)))

But I do not think that is going to work (so far it does not show any errors there). Do you have any ideas about how to solve these issues? :).

Thank you very much in advance.!

[zsunberg]

Hi @miguzca ,

I really apologize for letting this slip through the cracks! There is a simple solution: just delete the second method of gen that you wrote to return sp and r. When someone calls @gen(:sp, :r)(...) it will take the sp from your gen implementation and the automatically fill in the reward using your reward function. In your code above to replace generate_sr, you have essentially repeated what the @gen macro already does 😄 .

Another alternative is to just define transition (using an ImplicitDistribution) and reward separately. You only really need to define gen if there is a huge efficiency gain by generating the state, reward, and observation simultaneously. That would look something like this.

function POMDPs.transition(pp::UAVchaseProblem, s::MDPState, a::MDPAction)
    return ImplicitDistribution() do rng
        # put all your code that was in `gen` here
        return MDPState(curr_pos, curr_angle, curr_targ, curr_targ_true)
    end
end

I don't think sampletype should be needed anymore, but feel free to elaborate if you keep running into problems.

[miguzca]

Dear Zachary,

Thank you very much for your reply!. I have tried the first solution you proposed (that I have tried as well in the past) and I got this error message

ERROR: LoadError: AssertionError: gen(m::Union{MDP,POMDP}, ...) must return a NamedTuple; got a MDPState Stacktrace:

but as shown in the original post I am returning

return (sp=MDPState(curr_pos, curr_angle, curr_targ, curr_targ_true))
so it do not understand what is wrong.

Then, I have tried sol 2, and I have implemented transition as suggested

`function POMDPs.transition(pp::UAVchaseProblem, s::MDPState, a::MDPAction

return ImplicitDistribution() do rng
    # put all your code that was in `gen` here
    p = mdp(pp)
    # calculate target state
    target_dt_distance = p.dt*p.target_velocity
    ## sensor_noise = Base.rand(Normal(0, p.target_std), 2)
    target_dynamic_cov = Matrix(p.target_std*Diagonal{Float64}(I, 2))
    sensor_noise = rand(rng, MvNormal(target_dynamic_cov))
    ## sensor_noise = SVector(0,0)
    curr_targ_true = s.targetPose_true + target_dt_distance
    if if_in_FOV(pp, s)
       curr_targ = curr_targ_true + sensor_noise
    else
       curr_targ = s.targetPose + target_dt_distance + sensor_noise # next_target_pos(p, s.targetPose)
    end
    # calculate UAV state
    curr_angle = s.uavHeading + a.angle
    xy_dt_distance = p.dt*a.xy_speed*SVector(cosd(curr_angle), sind(curr_angle)) # careful
    z_dt_distance = p.dt*a.z_speed
    xyz_dt_distance = SVector(xy_dt_distance[1], xy_dt_distance[2], z_dt_distance)
    curr_pos = s.uavPose + xyz_dt_distance
    return MDPState(curr_pos, curr_angle, curr_targ, curr_targ_true)
end

end
`
But I get another error with observation

ERROR: LoadError: MethodError: no method matching observation(::UAVchasePOMDP, ::MDPState) Closest candidates are: observation(::POMDP, ::Any, ::Any) at /home/user/.julia/packages/POMDPs/PFpk1/src/pomdp.jl:61 observation(::POMDP, ::Any, ::Any, ::Any) at /home/user/.julia/packages/POMDPs/PFpk1/src/pomdp.jl:64 observation(::FullyObservablePOMDP, ::Any, ::Any) at /home/user/.julia/packages/POMDPModelTools/PYObx/src/fully_observable_pomdp.jl:22 ...

I think that is because I need to implement observation(::UAVchasePOMDP, ::MDPState). In the old implementation,
observation is implemented using generate_o,

`
function POMDPs.observation(pp::UAVchasePOMDP, s::MDPState, a::MDPAction, sp::MDPState, rng::AbstractRNG)

return generate_o(pp, s, a, sp, rng) # not sure if this function is needed for anything

end
`

so I have tried to implement it like

`
function POMDPs.observation(pp::UAVchasePOMDP, s::MDPState, a::MDPAction, rng::AbstractRNG)

 return @gen(:sp, :o, :r)(pp, s, a, rng)

end
however, this is not going to help me with the error above, because a need a method receiving justobservation(::UAVchasePOMDP, ::MDPState)`. I think there are a lot of basic things that I need to learn first 😄, I have tried to use the ImplicitDistribution for rng, but for the MDPAction I would not know how to do. Do you have any idea what I should do?.

Thank you very much in advance.

Kind regards.

[zsunberg]

but as shown in the original post I am returning

return (sp=MDPState(curr_pos, curr_angle, curr_targ, curr_targ_true))
so it do not understand what is wrong.

Ah, return (sp = x) will not create a NamedTuple; it will just assign sp to x and return it. Instead you need return (sp=x,) with the comma to show that it is a tuple and not just a parentheses. But, in any case, I think implementing transition like you have is better.

For the observation, you should be able to do something like

function POMDPs.observation(m::UAVChasePOMDP, sp::MDPState)
    ImplicitDistribution do rng
        # put all your code from generate_o in here
        return o
    end
end

Does this section of the documentation help? https://juliapomdp.github.io/POMDPs.jl/latest/dynamics/

[miguzca]

Dear Zachary,

Ah ok, I have used the comma for the gen and it solved that issue, however for the observation part actually that is my question (although I think I did not state it correctly). I had this implementation

function POMDPs.observation(m::UAVChasePOMDP, sp::MDPState)
    ImplicitDistribution do rng
        # put all your code from generate_o in here
        return o
    end
end

function POMDPs.observation(m::UAVChasePOMDP, s::MDPState)
    ImplicitDistribution do rng
        # a, or action MDPAction is missing
        return @gen(:sp, :o, :r)(m, s, a, rng)
    end
end

But given that the method receives just m and s, the action a (that I would need to call gen) is missing,
given that I am not passing it as a parameter of observation like in the old implementation.

function POMDPs.observation(pp::UAVchasePOMDP, s::MDPState, a::MDPAction, sp::MDPState, rng::AbstractRNG)
    return generate_o(pp, s, a, sp, rng) 
end

Btw, thank you also for the link with the documentation, I will have a look at it as well.

[zsunberg]

Oh, I see. I thought your observation only depended on sp because I did not read your post closely. Take a closer look at the MethodError stack trace - the methods of observation with more arguments call the ones with fewer. You should be able to implement any one of

observation(m, sp)
observation(m, a, sp)
observation(m, s, a, sp)

and it will work. Also note that transition and observation should never have an rng argument since they return a distribution that will be sampled from later.