Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: refactor method to properly release vllm instance resources #1012

Merged
merged 2 commits into from
Nov 29, 2024

Conversation

e7217
Copy link
Contributor

@e7217 e7217 commented Nov 28, 2024

I have investigated the related issue(#1003), and it seems that the root cause of the problem is the complexity involved in removing the allocated resources.

The issue did not occur when only a single task was performed, as shown below:

...
- node_type: generator
  ...
  modules:
    - module_type: vllm
      llm: meta-llama/Llama-3.2-1B-instruct
      temperature: 0.1
      max_tokens: 128
      max_model_len: 200
      gpu_memory_utilization: 0.3

However, after adding another scenario, the error occurred. It seems that the error arose during the process of reinitializing the vllm model when the following change was made:

temperature: [ 0.1, 1.0 ]

I have addressed this issue in the code, and I would appreciate it if you could review it.

This PR only addresses the changes related to the destructor, and I will add further details regarding the issue on the issue page.

Thank you.

references

Copy link
Contributor

@vkehfdl1 vkehfdl1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much!

@vkehfdl1 vkehfdl1 enabled auto-merge (squash) November 29, 2024 09:06
@vkehfdl1 vkehfdl1 merged commit c9072e3 into Marker-Inc-Korea:main Nov 29, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants