#368 allow scheduler parallelism

[shokanson-extensiv]

See issue #368 for discussion.

Hangfire.Core 1.8.12 introduces parallelism support for DelayedJobScheduler and RecurringJobScheduler. That support requires the storage layer to allow batched retrieval of scheduled/recurring jobs, which is done by overriding the List-returning GetFirstByLowestScoreFromSet.

Hangire.Core's sheduler parallelism is implemented fully within the DelayedJobScheduler and RecurringJobScheduler methods:

1. acquires a distributed scheduler or recurring-jobs lock, respectively
2. calls GetFirstByLowestScoreFromSet to retrieve the List of IDs for the to-be-enqueued jobs
3. performs a Parallel.ForEach over the List of IDs
   i. uses the MaxDegreeOfParallelism configured when Hangfire is started
   ii. enqueues each job, during which it acquires a distributed job lock, changes the job's state, and releases the distributed lock

The point of describing this is to note that the scope of the distributed scheduler and recurring-jobs locks don't change whether parallelism is used or not; as such, the database behavior for those locks will be the same either way.

For the distributed job locks, the difference between sequential vs. parallel will be as follows:

• for sequential: the scheduler will only have one job lock active at a time
• for parallel: there may be as many as MaxDegreeOfParallelism job locks active at one time; however, because these locks are represented as distinct rows in the locks table, there won't be blocking on those records

In either case (sequential or parallel enqueueing), workers will dequeue jobs as they always have.

Here are some results of profiling the scheduler performance with and without batching and parallelism. The PostgreSQL server is an out-of-the box installation running on a Windows machine with a 12-processor Intel i7 x64. The Hangfire server is an 8-processor Intel i7 x64.

Batching On?	Batch Size	Parallelism Degree	Sec/Batch	% Improvement
n	1000	n/a	10.6
y	1000	1	10.1	4.7%
y	1000	2	8.2	22.6%
y	1000	3	7.8	26.4%
y	1000	5	6.4	39.6%
y	1000	10	4.3	59.4%
y	1000	20	3.9	63.2%
y	1000	50	3.8	64.2%
y	1000	100	3.9	63.2%

On this hardware/software configuration, I hit the "knee" of the curve at parallelism degree 20. Your mileage may vary.