Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/Add bullmq redis for message queue processing #3568

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

HenryHengZJ
Copy link
Contributor

@HenryHengZJ HenryHengZJ commented Nov 25, 2024

How it works:

  • Ensure you have redis running locally (or have access to an instance), update your .env with the settings (refer to .env.sample for settings)
  • Open 2 terminals
  • On 1st terminal: pnpm start to start the server
  • On 2nd terminal: pnpm run start-worker to start the worker
  • Navigate to http://localhost:3000/admin/queues to view bullmq dashboard

Tasks

  • BullMQ: Agentflow/Chatflow
  • BullMQ: Upsert Vector
  • BullMQ: Upsert/Refresh/Process/Insert Doc Store
  • Cache Pool move to Redis when Queue mode
  • Rate Limit to Redis when Queue mode
  • Remove singleton design, refractor to open/close connection for every actions
  • Proper logging when Queue mode

HenryHengZJ and others added 9 commits November 25, 2024 15:19
# Conflicts:
#	packages/server/src/Interface.ts
#	packages/server/src/utils/buildChatflow.ts
#	packages/server/src/utils/index.ts
#	pnpm-lock.yaml
# Conflicts:
#	pnpm-lock.yaml
# Conflicts:
#	packages/server/src/commands/start.ts
#	pnpm-lock.yaml
@HenryHengZJ HenryHengZJ marked this pull request as ready for review January 3, 2025 22:47
# Conflicts:
#	packages/server/package.json
#	packages/server/src/commands/start.ts
#	packages/server/src/index.ts
#	packages/server/src/services/documentstore/index.ts
#	packages/server/src/utils/upsertVector.ts
#	pnpm-lock.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant