ARDESPOT model formulation

[Etwari3]

Hello,

I have been trying to formulate a continuous state POMDP for the ARDESPOT solver. I constructed continuous states as range values and the transitions function on these states. I wondered whether you could use a continuous state transition for this solver. Because I checked with Discrete POMDPs, and it works fine. Can you please give me some advice (attached is the model I created)?

Thanks
ARDESPOTmodel - Jupyter Notebook.pdf

[lassepe]

Hi @Etwari3,

ARDESPOT uses the generative POMDPs.jl interface and does not internally attempt to enumerate all states. Hence, it can operate on problem definitions that do not implement POMDPs.states at all. Therefore, in order to define a continuous state version of your problem, you can simply drop the definition of POMDPs.states for your problem as we also do in the partially-observable mountain car example. In fact, even if you provide POMDPs.states, it should just be ignored by ARDESPOT.jl.

Looking at your Jupyter Notebook, the only thing that stops ARDESPOT from operating on the continuous state space is that your transition function implicitly implicitly discretizes the state space due to the use of, e.g., sp = rand(rng, 0.1:0.25). If instead you were sample from a uniform distribution, e.g, sp = (0.25 - 0.1) * rand(rng) + 0.1, this issue should be fixed.

[Etwari3]

Hello @lassepe

Thank you for the explanation. It was very helpful in understanding the solver. However, I could not fix the error I was getting. Can you please give me some direction? I have revised my code according to your explanation.

Thank you
ARDESPOTmodel.pdf

[lassepe]

Your implementation of POMDPs.transition does not quite match the interface requirements. In particular:

• POMDPs.transition does not return a sample but a distribution object which one can sample from. To facilitate generative models, we provide the ImplicitDistribution helper type. Please consult this section on how to use it.
• POMDPs.transitions has requires no rng as input. The correct signature is POMDPs.transition(m::POMDP, state, action). You can think of the arguments as the problem data m and variables a and s that distribution of the next state is conditioned on.

[Etwari3]

@lassepe ,
Hello, I have one more question. I have fixed the transition and was able to run the model. However, I wanted to use the solver for gen function instead of transition. I keep getting "no method matching transition."
I formulated the gen function for a discrete POMDP problem so that I can easily convert it to an equivalent continuous POMDP. My question is do you have to strictly formulate discrete POMDP using transition or you can use gen function? If you only use transition, how do you formulate continuous state POMDP?
Thank you.

[lassepe]

The only reason to ever implement the POMDPs.gen interface is the following (quote from the docs):

gen is intended only for the case where two or more of the next state, observation, and reward need to be generated at the same time. If the state transition model can be separated from the reward and observation models, you should implement transition with an ImplicitDistribution instead of gen. See also the "What is the difference between transition, gen, and @gen?" FAQ.

In principle, you can use either gen or transition irrespective of whether your problem is continuous or discrete. However, implementing the more "low-level" gen is discouraged because it provides less flexibility: gen generates joint samples of (sp, o, r, info) --- hence, a solver that only wants to retrieve a sample of the next state ( sp) may waste computation by having to compute o, r, info as well. This wasted computation may sometimes be avoided if the Julia compiler is smart-enough to skip their computation but relying on that can be overly optimistic and there is no guarantee.

By implementing POMDPs.transition, you don't rely on this compiler optimization: a solver that just wants to sample sp has direct access to a function that does only that.

[Etwari3]

@lassepe ,

Thank you.