Terminal States in GenerativeBeliefMDP

[FlyingWorkshop]

Porting this conversation from email @zsunberg, @mykelk.

Is there another way to implement @gen in GenerativeBeliefMDP. Currently, the function samples a state from the belief and checks whether the sampled state is terminal during generation. This makes it hard to check and handle termination before generation (which is a useful behavior in practice).

[zsunberg]

At one level higher, what are you trying to do? i.e. why is checking and handling termination before generation "a useful behavior in practice"? or what would you do if you could check it before next state generation? (maybe link to some code where there is a problem)

[FlyingWorkshop]

This is useful when you want to rollout trajectories and stop at a terminal state (see snippet).

[zsunberg]

I see. I will think about a permanent fix for GenerativeBeliefMDP

For now, to get your code to run, I'd recommend this:

function (s::PolicySampler)(pomdp::POMDP)
    return unique(stepthrough(pomdp, s.policy, s.updater, "bp", max_steps=s.n))
end

or similar. I think that will do what you want.

[zsunberg]

The behavior was improved in #559

[zsunberg]

I spoke to my PhD students who had a similar issue, and they decided to use problems where the observation indicates that the state must be terminal so that the belief is always terminal when the state is terminal.