Concurrency inside of a given process

[SlovakianCanon]

I've been trying all kinds of patterns to achieve what I need, and I've had only limited success so far.

Let's say I have a job type that's very CPU-intensive. I want to be able to process multiple jobs of that type at the same time, so I scaled horizontally and created multiple worker instances in different servers.

The one particular constraint I have is that for each user (< 50 users, let's say 30 for this example) there must not be more than one job running at any given time. So if user1, user2, user3, and user4 all start a job at the same time, fine they can all be executed in parallel by the servers. But if user1 created 4 jobs in a row, I need to wait for job1 to finish before job2 starts (because the result of job1 can influence the execution of job2).

To achieve this, the only way I know of is to use one queue per user, with a concurrency of 1. The problem with this approach is that if I register a worker for each of those queues, and I have 30 queues, there's the potential of running 30 jobs at once on a server (one job in each queue). Since these jobs are CPU-intensive, they basically bring the server down to a crawl.

The fix seems easy in theory: limit the concurrency inside a given server to N, ideally the number of cores. So if there is only one server with two cores, and 4 users register jobs in 4 queues, the first two will be picked up and executed, and the third one will wait for the first one to finish.

Is there a way to do this? I feel like I tried everything but maybe there's a simpler solution I didn't think about.

[manast]

The case you are describing is not very uncommon however it is a bit complex in practice. I do not think this is feasible currently with BullMQ. I have in the drawing board the concept of "demuxers" that basically works like this, you have N "pseudo-queues" that are all connected to a demux that picks in a round robin fashion jobs from the N "pseudo queues" and are processed by M workers connected to the demux. However this feature is still in design stage and will take some time (months) before it can be used in production...

[duncanb-q]

For us, we use the same groupKey every time so it doesn't necessarily have to be different for different groupKeys (although that is cool :-) )

[xiaosha007]

ok, so different concurrency factors depending on the group key. This would need to be implemented as a new feature, it is not possible to do at the moment.

I need this feature as well...It would be great if we can use the concurrent feature based on groupKey

[everhardt]

I second that, only I wouldn't want to define that on the worker level, but on the queue level, so that if I have multiple workers, it is still guaranteed that only one job of that groupKey will run at the same time.

[manast]

@everhardt concurrency per groupKey is doable but it is costly to implement, so a feature like this will most certainly only be implemented in BullMQ Pro if there is enough interest.

[everhardt]

I understand @manast, although I would prefer to stay on the open source code. I'll explore a solution with a redis based locking mechanism first (like redlock).

[BMaD-Chillyo]

This answer is a bit late but it sounds like Flows is what you're looking for.