Concurrent Block Processing

[tiger5226]

In order to increase the performance of the processing in chainquery by increasing the number of cores that can be used, a reordering of the flow is needed. Currently we tap out 4 core machines. However, that is the max Chainquery can make use of without parallel block processing. @nikooo777 had an 8 core machine that was only using around 60% of the capacity of the machine. I wanted to put my thoughts on this down in writing for others to comment on.

• The main way to improve core usage is to process blocks in parallel. Since the chain is sequential, this is not a small lift. Processing of Blocks and the Transactions in them need to be decoupled. The sequential dependencies in the blockchain are actually amongst transactions rather than blocks. In a any give block most transactions are not dependant on transactions within x blocks from the block they are in. This is a major area of improvement. If we could process those transactions in parallel we could sync more and faster.

• We need to be able to track dependencies to make this process efficient. Currently to enable concurrent transaction processing within a block, we use a reprocessing queue. So if a transaction fails to process due to a dependant transaction in the block not being processed yet, we add it to the back of the queue and "try again later" so to speak. This works well and is efficient for a specific block. However, if we want to be able to be "working" on say 5 blocks at a time, this becomes very inefficient as we could be uselessly trying to process a transaction in the queue hundreds of times, when in reality we should just wait until it's dependant transaction is processed.

• By tracking dependencies, we can then do more work than possible on most machines. This means we need to make our go routines more efficient. Currently we concurrently process, inputs, outputs and transactions. They all have their own set of workers. Currently, this is not an issue as at most we usually have around 23-25 routines going. This will not be efficient if we are looking at 5 blocks. Each one of these "Types" should use the same set of workers, thereby increasing the efficiency of the concurrency. We should have a more generic worker library.

• In order to process this many things concurrently, we need a channel based, robust, communication mechanism to synchronize this information between routines. For example, when a transaction that is depended upon finishes I need a mechanism to notify a routine that the depending transaction can now be processed. This is to avoid a type of polling mechanism we have with our queue.

• Since the database is our bottleneck, we can also take advantage of the parallel block processing by batching inputs, outputs and transaction addresses for any given transaction. The indices are the most expensive part of inserting data into chainquery. They are powerful and help us with very fast queries, but it also slows down the initial processing of blocks since each piece of data is currently inserted one by one. Batching will allow us to insert all the data in one go and update the index once.