Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Docker image build failing for SWE-Bench evaluation #4164

Closed
2 tasks done
sani903 opened this issue Oct 2, 2024 · 2 comments · Fixed by #4154
Closed
2 tasks done

[Bug]: Docker image build failing for SWE-Bench evaluation #4164

sani903 opened this issue Oct 2, 2024 · 2 comments · Fixed by #4154
Labels
bug Something isn't working evaluation Related to running evaluations with OpenHands severity:medium Affecting multiple users

Comments

@sani903
Copy link

sani903 commented Oct 2, 2024

Is there an existing issue for the same bug?

Describe the bug

The Sandbox image build is failing for SWE-Bench instances testing.
I was previously working with OH version 0.9.4 which did not have this issue but upon pulling 0.9.7, this problem started with most instances. For example: astropy__astropy-7606, django__django-15375, django__django-13112, django__django-14855

I am using the Docker images (such as xingyaoww/sweb.eval.x86_64.django_s_django-13112:latest).

Current OpenHands version

0.9.7

Installation and Configuration

Cloned OpenHands and ran evaluation with "./evaluation/swe_bench/scripts/run_infer.sh <model_config>"
I stored the instance IDs in config.toml

Model and Agent

No response

Operating System

Linux

Reproduction Steps

No response

Logs, Errors, Screenshots, and Additional Context

Screenshot 2024-10-01 at 11 10 46 PM
@sani903 sani903 added the bug Something isn't working label Oct 2, 2024
@neubig
Copy link
Contributor

neubig commented Oct 2, 2024

This might be duplicate with #4157 ?

@sani903
Copy link
Author

sani903 commented Oct 2, 2024

This might be duplicate with #4157 ?

As per the logs in that issue, while the error was related to building the Sandbox Image, the error there was an API error
[File "/home/gneubig/.cache/pypoetry/virtualenvs/openhands-ai-02VxP7eD-py3.11/lib/python3.11/site-packages/docker/api/image.py", line 251, in inspect_image return self._result( ^^^^^^^^^^^^^ File "/home/gneubig/.cache/pypoetry/virtualenvs/openhands-ai-02VxP7eD-py3.11/lib/python3.11/site-packages/docker/api/client.py", line 281, in _result self._raise_for_status(response) File "/home/gneubig/.cache/pypoetry/virtualenvs/openhands-ai-02VxP7eD-py3.11/lib/python3.11/site-packages/docker/api/client.py", line 277, in _raise_for_status raise create_api_error_from_http_exception(e) from e](https://github.com/All-Hands-AI/OpenHands/issues/4164)

Here I am able to get the image, but then a particular mamba command is failing. So it might be related, but I thought this was a different problem as the error stemmed from docker.py instead

SmartManoj added a commit to SmartManoj/Kevin that referenced this issue Oct 2, 2024
@mamoodi mamoodi added evaluation Related to running evaluations with OpenHands severity:medium Affecting multiple users labels Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working evaluation Related to running evaluations with OpenHands severity:medium Affecting multiple users
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants