Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize GitHub data cache generation #17

Merged
merged 7 commits into from
Dec 3, 2024

Conversation

neubig
Copy link
Contributor

@neubig neubig commented Dec 3, 2024

This PR improves the performance of the GitHub data cache generation by parallelizing the comment fetching process.

Changes:

  • Process items in parallel batches of 10 to avoid rate limiting
  • Use Promise.all to fetch comments for multiple issues/PRs simultaneously
  • Add better progress reporting showing which batch is being processed

Performance improvements:

  • Initial fetch: 4.70s (was 4.82s)
  • Processing items: 12.40s (was 86.83s, 7x faster!)
  • Sorting: 0.00s (unchanged)
  • Total time: 17.11s (was 91.65s, 5.4x faster overall!)

Copy link

vercel bot commented Dec 3, 2024

@openhands-agent is attempting to deploy a commit to the all-hands-ai team on Vercel, but is not a member of this team. To resolve this issue, you can:

  • Make your repository public. Collaboration is free for open source and public repositories.
  • Add @openhands-agent as a member. A Pro subscription is required to access Vercel's collaborative features.
    • If you're the owner of the team, click here and add @openhands-agent as a member.
    • If you're the user who initiated this build request, click here to request access.
    • If you're already a member of the all-hands-ai team, make sure that your Vercel account is connected to your GitHub account.

To read more about collaboration on Vercel, click here.

@neubig neubig merged commit 41542ff into main Dec 3, 2024
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants