(feat) Add trajectory replay for headless mode #6215
Conversation
|
Demo: Step 1. Run OpenHands and generate a trajectory workspace_base="./workspace"
save_trajectory_path="./traj/demo.json"With command: Step 2. Run OpenHands with trajectory replay workspace_base="./workspace2"
replay_trajectory_path="./traj/demo.json"With command: Step 3. Check the result 
FYI: example trajectory |
| self.replay_events is not None | ||
| and self.replay_index < len(self.replay_events) | ||
| and isinstance(self.replay_events[self.replay_index], Action) | ||
| and self.replay_events[self.replay_index].source != EventSource.USER |
There was a problem hiding this comment.
Ah I see! This has to work just fine... I do wonder though, what could break it?
- delegation? because in that case, the controller itself handles AgentDelegateAction and creates a MessageAction for the new guy, and puts in the stream; the trajectory must have saved the old one too. I don't think it needs to be in scope of this PR, though.
- MessageActions with source user, which happened after the initial task?
There was a problem hiding this comment.
I bet delegation won't work; I am not even sure if trajectory properly records delegation.
MessageActions with source user, which happened after the initial task?
yeah that is just not possible with headless mode, but it would be interesting when we enable this in GUI mode as well... I'd love to have that functionality working, so that people can just upload trajectory to replay some recorded events first and then start working with agents.
Oh actually headless mode does allow interactive inputs, that would be interesting to test out as the next step.
There was a problem hiding this comment.
I bet delegation won't work; I am not even sure if trajectory properly records delegation.
I did think of it here! Though I didn't test this use case after the last changes (and for a long time really).
There was a problem hiding this comment.
MessageActions with source user, which happened after the initial task?
Tested in headless mode and it just worked!
Steps 0-5 were from replay
There was a problem hiding this comment.
So it continues normally. But it can't replay the new one, together with that user message? 😅
There was a problem hiding this comment.
Yes, I mean, at this end of the run in your image, the event stream has all events, those first 5 steps plus 3 other steps. The agent history has them too. If we save trajectory now, traj2.json should contain all of them, including the user message in the middle. Can we replay this traj2.json?
There was a problem hiding this comment.
💭 no, because "wait_for_response": true from agent message would trigger a AWAITING_USER_INPUT state.
A workaround is to manually fix the trajectory and change AWAITING_USER_INPUT to false. And that works!
A hack in the code is to somehow not AWAITING_USER_INPUT if there's a next action to replay. (Not implemented)
There was a problem hiding this comment.
I agree, I wouldn't hack this. Maybe a next iteration of this feature, when we accept user messages, could take care of it because the user message should (?) change the agent state.
There was a problem hiding this comment.
Yeah it needs some design in the next iteration. I am thinking that the agent state space should be a subset under replay mode. AWAITING_USER_INPUT is not a valid state during replay; it's only valid after a replay (or equivalently, without replay).
End-user friendly description of the problem this fixes or functionality that this introduces
Add trajectory replay feature for headless mode.
Give a summary of what the PR does, explaining any non-trivial design decisions
To test it out, add
replay_trajectory_pathtoconfig.toml, and thenTODOs in this PR:
trajectories_pathconfig tosave_trajectory_path#6216 is mergedadd tests(non-trivial, will skip in this PR)TODOs in next PR:
Link of any specific issues this addresses
#6049
To run this PR locally, use the following command: