-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(workflow): Implement a simplified CoAct workflow #3770
Closed
Closed
Changes from 18 commits
Commits
Show all changes
53 commits
Select commit
Hold shift + click to select a range
48ebc6e
make codeact accept delegated task
ryanhoangt 136f10a
implement a draft planner agent that can delegate task to CodeAct
ryanhoangt 6ebcf02
add executor agent and refactor planner agent
ryanhoangt 9b4430d
experiment: adjust prompt to make BrowsingAgent better potentially?
ryanhoangt 135f26d
modify prompt and action parser for executor agent
ryanhoangt 9da4217
fix bug agent using wrong prompt files, improve logging on terminal a…
ryanhoangt a963080
revert debugging in browsing_agent.py
ryanhoangt 25d7a74
revert BrowsingAgent prompt
ryanhoangt 56edccb
fix: local agent should finish instead of delegate
ryanhoangt cd660c0
Merge branch 'main' into coact-workflow-draft
ryanhoangt 567297c
update prompt of planner agent
ryanhoangt 2264e70
add meta prompt to use in the future to reinforce workflow
ryanhoangt 136da2b
update README
ryanhoangt d5b5831
revert change in browser obs
ryanhoangt c864237
fix copy paste error
ryanhoangt 39d8d0a
update run_infer and README
ryanhoangt fc44e17
specialize global planner agent by removing editing skills and update…
ryanhoangt 5cadc38
Merge branch 'main' into coact-workflow-draft
ryanhoangt af0b3d7
improve planner agent's prompt
ryanhoangt e80c001
add reasoning scratchpad into plan structure
ryanhoangt 91a5873
exp: separate edit plan and testing plan
ryanhoangt cf80534
exp: further enhance run_infer prompt
ryanhoangt a7f4c0b
dummy commit to pump planner version
ryanhoangt 63a9f55
finalize the prompt to follow the shape of the workflow
ryanhoangt f404c7c
update eval prompt
ryanhoangt 97c4dba
update README
ryanhoangt 9ed5257
change username in planner system prompt
ryanhoangt 89a8ed7
update new system prompts from codeact
ryanhoangt 76f093e
Merge branch 'main' into coact-workflow-draft
ryanhoangt fab5874
Fix typo for agenthub/coact_agent/executor/user_prompt.j2
ryanhoangt 7285cfd
Fix typo for evaluation/swe_bench/run_infer.py
ryanhoangt 1e426d3
Fix typo for evaluation/swe_bench/run_infer.py
ryanhoangt e3bdb5c
remove FIXME
ryanhoangt 2346350
move stop sequences to attribute
ryanhoangt 1764205
resolve comments
ryanhoangt 0d91fcf
delete meta prompts and resolve comments
ryanhoangt 5689ed8
Merge branch 'main' into coact-workflow-draft
tobitege dbc5b9d
modify prompt location
ryanhoangt 942ff4e
Merge branch 'main' into coact-workflow-draft
ryanhoangt 10cc53b
add ability to inject problem statement into executor
ryanhoangt 9a7c5a9
Merge branch 'main' into coact-workflow-draft
ryanhoangt 038313a
fix bug in injecting initial user message
ryanhoangt 34cabef
place 'reason'beefore 'description' in plan structure
ryanhoangt c5303f2
update executor's system prompt
ryanhoangt f962fd6
final touch
ryanhoangt 7b4a2da
Merge branch 'main' into coact-workflow-draft
ryanhoangt 05cc97f
fix unit tests
ryanhoangt 8b4eb1b
revert change in AgentDelegateAction
ryanhoangt 481a3e4
update mocks for intg tests
ryanhoangt ff90ff4
fix bug in CodeActAgent
ryanhoangt 7b285c2
Merge branch 'main' into coact-workflow-draft
ryanhoangt 5e70aca
Merge branch 'main' into coact-workflow-draft
ryanhoangt 0823c1c
Merge branch 'main' into coact-workflow-draft
ryanhoangt File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# CoAct Multi-Agent Framework | ||
|
||
This folder implements a multi-agent workflow inspired by the CoAct framework ([paper](https://arxiv.org/abs/2406.13381)), that provides a robust structure for defining, planning, and executing tasks using multiple agents. | ||
|
||
## Agents | ||
|
||
1. `CoActPlannerAgent`: | ||
- is responsible for exploring and creating a global plan. It can replan if there are issues with the previous one. | ||
- has full capabilities of [CodeActAgent](https://github.com/All-Hands-AI/OpenHands/tree/main/agenthub/codeact_agent). | ||
2. `CoActExecutorAgent`: | ||
- is responsible for executing the proposed plan. Facing issues with the plan, it can request for a new one. | ||
- also has full capabilities of [CodeActAgent](https://github.com/All-Hands-AI/OpenHands/tree/main/agenthub/codeact_agent). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
from agenthub.coact_agent.executor.executor_agent import ( | ||
LocalExecutorAgent as CoActExecutorAgent, | ||
) | ||
from agenthub.coact_agent.planner.planner_agent import ( | ||
GlobalPlannerAgent as CoActPlannerAgent, | ||
) | ||
from openhands.controller.agent import Agent | ||
|
||
Agent.register('CoActPlannerAgent', CoActPlannerAgent) | ||
Agent.register('CoActExecutorAgent', CoActExecutorAgent) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
import re | ||
|
||
from agenthub.codeact_agent.action_parser import ( | ||
CodeActActionParserAgentDelegate, | ||
CodeActActionParserCmdRun, | ||
CodeActActionParserFinish, | ||
CodeActActionParserIPythonRunCell, | ||
CodeActActionParserMessage, | ||
CodeActResponseParser, | ||
) | ||
from openhands.controller.action_parser import ActionParser | ||
from openhands.events.action import ( | ||
Action, | ||
AgentFinishAction, | ||
) | ||
|
||
|
||
class ExecutorResponseParser(CodeActResponseParser): | ||
"""Parser action: | ||
- CmdRunAction(command) - bash command to run | ||
- IPythonRunCellAction(code) - IPython code to run | ||
- AgentDelegateAction(agent, inputs) - delegate action for (sub)task | ||
- MessageAction(content) - Message action to run (e.g. ask for clarification) | ||
- AgentFinishAction() - end the interaction | ||
""" | ||
|
||
def __init__(self): | ||
# Need pay attention to the item order in self.action_parsers | ||
super().__init__() | ||
self.action_parsers = [ | ||
CodeActActionParserFinish(), | ||
CodeActActionParserCmdRun(), | ||
CodeActActionParserIPythonRunCell(), | ||
CodeActActionParserAgentDelegate(), | ||
CoActActionParserRequest(), | ||
] | ||
self.default_parser = CodeActActionParserMessage() | ||
|
||
def parse_response(self, response) -> str: | ||
action = response.choices[0].message.content | ||
if action is None: | ||
return '' | ||
for action_suffix in ['bash', 'ipython', 'browse', 'request']: | ||
if ( | ||
f'<execute_{action_suffix}>' in action | ||
and f'</execute_{action_suffix}>' not in action | ||
): | ||
action += f'</execute_{action_suffix}>' | ||
return action | ||
|
||
|
||
class CoActActionParserRequest(ActionParser): | ||
def __init__(self): | ||
self.request = None | ||
|
||
def check_condition(self, action_str: str) -> bool: | ||
self.request = re.search( | ||
r'<execute_request>(.*)</execute_request>', action_str, re.DOTALL | ||
) | ||
return self.request is not None | ||
|
||
def parse(self, action_str: str) -> Action: | ||
assert ( | ||
self.request is not None | ||
), 'self.request should not be None when parse is called' | ||
|
||
replan_request = self.request.group(1).strip() | ||
return AgentFinishAction( | ||
thought=replan_request, | ||
outputs={'content': replan_request}, | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
import os | ||
|
||
from agenthub.coact_agent.executor.action_parser import ExecutorResponseParser | ||
from agenthub.codeact_agent.codeact_agent import CodeActAgent | ||
from openhands.core.config import AgentConfig | ||
from openhands.llm.llm import LLM | ||
from openhands.runtime.plugins.agent_skills import AgentSkillsRequirement | ||
from openhands.utils.prompt import PromptManager | ||
|
||
|
||
class LocalExecutorAgent(CodeActAgent): | ||
VERSION = '1.0' | ||
|
||
def __init__(self, llm: LLM, config: AgentConfig) -> None: | ||
super().__init__(llm, config) | ||
|
||
self.action_parser = ExecutorResponseParser() | ||
self.prompt_manager = PromptManager( | ||
prompt_dir=os.path.join(os.path.dirname(__file__)), | ||
agent_skills_docs=AgentSkillsRequirement.documentation, | ||
micro_agent=self.micro_agent, | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
{% set MINIMAL_SYSTEM_PREFIX %} | ||
You are an autonomous intelligent programming agent playing the role of a subordinate employee responsible for local planning and execution of specific tasks in a multi-tier task execution structure, tasked with software development. You will be given coding-based tasks. The global agent has set a global plan for the tasks, divided into multiple phases. These phase plans will be given to you as a whole. Your responsibility is to perform them and return the results to the global agent. When you face some issues that require a new global plan, you can request a new global plan from the global planner agent. | ||
|
||
Here's the information you'll have: | ||
* The broken-down phase list: These are the tasks you're trying to complete now. | ||
* The current codebase: This is what you need to navigate through and make the changes to complete the tasks given by the global agent. | ||
|
||
The agent can use a Python environment with <execute_ipython>, e.g.: | ||
<execute_ipython> | ||
print("Hello World!") | ||
</execute_ipython> | ||
|
||
The agent can execute bash commands wrapped with <execute_bash>, e.g. <execute_bash> ls </execute_bash>. | ||
The agent is not allowed to run interactive commands. For commands that may run indefinitely, | ||
the output should be redirected to a file and the command run in the background, e.g. <execute_bash> python3 app.py > server.log 2>&1 & </execute_bash> | ||
|
||
If a command execution result says "Command timed out. Sending SIGINT to the process", | ||
the agent should retry running the command in the background. | ||
ryanhoangt marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
As a local executor agent, there are some additional actions that you can use to communicate back to the global planner agent: | ||
- `<execute_request>`: You have encountered an exception in the execution process. You suspect problems with the global planner's plan and trigger a request for replanning. Explain why you decide to request a new global plan using this action. | ||
|
||
{% endset %} | ||
{% set BROWSING_PREFIX %} | ||
The agent can browse the Internet with <execute_browse> and </execute_browse>. | ||
For example, <execute_browse> Tell me the usa's president using google search </execute_browse>. | ||
Or <execute_browse> Tell me what is in http://example.com </execute_browse>. | ||
{% endset %} | ||
{% set PIP_INSTALL_PREFIX %} | ||
The agent can install Python packages using the %pip magic command in an IPython environment by using the following syntax: <execute_ipython> %pip install [package needed] </execute_ipython> and should always import packages and define variables before starting to use them. | ||
{% endset %} | ||
{% set SYSTEM_PREFIX = MINIMAL_SYSTEM_PREFIX + BROWSING_PREFIX + PIP_INSTALL_PREFIX %} | ||
{% set COMMAND_DOCS %} | ||
Apart from the standard Python library, the agent can also use the following functions (already imported) in <execute_ipython> environment: | ||
{{ agent_skills_docs }} | ||
IMPORTANT: | ||
- `open_file` only returns the first 100 lines of the file by default! The agent MUST use `scroll_down` repeatedly to read the full file BEFORE making edits! | ||
- The agent shall adhere to THE `edit_file_by_replace`, `append_file` and `insert_content_at_line` FUNCTIONS REQUIRING PROPER INDENTATION. If the agent would like to add the line ' print(x)', it must fully write the line out, with all leading spaces before the code! | ||
- Indentation is important and code that is not indented correctly will fail and require fixing before it can be run. | ||
- Any code issued should be less than 50 lines to avoid context being cut off! | ||
- After EVERY `create_file` the method `append_file` shall be used to write the FIRST content! | ||
- For `edit_file_by_replace` NEVER provide empty parameters! | ||
- For `edit_file_by_replace` the file must be read fully before any replacements! | ||
{% endset %} | ||
{% set SYSTEM_SUFFIX %} | ||
Responses should be concise. | ||
The agent should attempt fewer things at a time instead of putting too many commands OR too much code in one "execute" block. | ||
Include ONLY ONE <execute_ipython>, <execute_bash>, or <execute_browse> per response, unless the agent is finished with the task or needs more input or action from the user in order to proceed. | ||
If the agent is finished with the task you MUST include <finish></finish> in your response. | ||
IMPORTANT: Execute code using <execute_ipython>, <execute_bash>, or <execute_browse> whenever possible. | ||
The agent should utilize full file paths and the `pwd` command to prevent path-related errors. | ||
The agent must avoid apologies and thanks in its responses. | ||
Remeber to execute ALL the phases of the global plan and only return the summary of the whole process to the global agent. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ryanhoangt @neubig |
||
|
||
{% endset %} | ||
{# Combine all parts without newlines between them #} | ||
{{ SYSTEM_PREFIX -}} | ||
{{- COMMAND_DOCS -}} | ||
{{- SYSTEM_SUFFIX }} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure you worked on this already, but there're still occurences of "agent", further down is "assistant" already (might be from other PR)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought about it, but it looks a bit odd to me:
Maybe something like this is better?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That reads much better, indeed!