Enhance transaction receipt confirmations for EVM chains

[EnriqueL8]

This is the next step in FIR-18 hyperledger/firefly-fir#18

Reworking Receipt handler to use new block listener

Existing Behaviour

Today in FireFly, the EVMConnect plugin will submit transactions to the blockchain connector and then wait for a receipt notification to be returned by the connector. The blockchain connector keeps track of all the transactions it cares about, so the ones that were submitted to it. Internally this connector has a block indexer that is fetching new blocks from the chain. It then filters out the transactions from those blocks and cross references with the transactions that were submitted. If it finds any, it will submit a notification back to FireFly Core with the status of that transaction (Successful, Failed). Unfortunately these notifications are not guaranteed to be in the order of the blockchain so some operations might get confirmed before others.

The websocket delivering these confirmations is best effort so only will attempt delivery to the current connected clients.

This diagram attempts to show the current behavior:

Suggested Changes

As explained in the FIR-18 , the benefits and necessity to process transactions in the order of the block on the blockchain drives this piece of work to use the new block listener in FireFly core to confirm operations/transactions that have been submitted!

The ides is to create new block listener in the blockchain connector and listen to those events in FireFly core through a websocket. Then the Ethereum plugin can process those events and update the relevant operations in FireFly. This would guarantee that FireFly processes the transactions in the order of the blocks in the chain and only acknowledges to the blockchain connector the next block once it has finished processing all the transactions in the current block it received.

There is a sticky point in this approach. Today the block event form the block listener only includes the TX hashes, but we need the transaction receipts to confirm that the transaction went through correctly.

Example of an event:

{
    "batchNumber": 1,
    "events": [
        {
            "blockHash": "0x06b0a191be7106dfe4b5dce6fd24956803d2904f6be4091c7c2bb12702cab7c2",
            "blockNumber": "33",
            "listenerId": "0194ad9b-d1ec-0420-d1f4-b641148375dd",
            "listenerName": "block",
            "listenerType": "blocks",
            "parentHash": "0x5fac4b9a3537336e4d65852095c19bc10446f56256cfdd580de23ecaeb7d95d1",
            "streamId": "0194ad53-59a5-ba4b-3122-211bfe33f466",
            "subId": "0194ad9b-d1ec-0420-d1f4-b641148375dd",
            "transactionHashes": [
                "0x85ce4e471e73fd78420a685db0f65271376a3ca25dac7d0bdd00553e74c8bc2b"
            ]
        }
    ]
}

FireFly now needs a wait to get the transaction receipts for all the transaction it cares about, two problems:

How does FireFly know it cares about that TX just having the hash?

Today the Operation ID is passed to the transaction manager when submitting a transaction and returned back on the BlockchainReceiptNotification so FireFly can correlate that back. So a trivial thing would be that for each TX Hash we call GetTransactionStatus (using the hash instead of the ID, as it today) that returns both the receipt and the header containing the Operation ID.
This is expensive especially with a large amount of transactions and most of time it might be transactions that FireFly does not care about. This call today calls the node with eth_getTransactionReceipt, expensive performance hit on the node!

A better approach would be to store the TX Hash in FireFly Core that is returned when submitting a transaction to the connector! This changes a lot of internal plumbing code around storing an operation + FF transaction. We can then fetch if a transaction exists with that hash when we process the block events! But we still need the receipts...

How does FireFly fetch the TX receipt in a performant way?

One thought would be to extend the current Block Listener events to also provide the receipts for all the transactions within the block. There is a handy RPC method etc_getBlockReceipts that retrieves all the receipts for that block. This would still affect performance as the blocks are only kept in memory today and not persisted so re-indexing will cause the blockchain connector to fetch receipts for transactions it has already processed or ones it doesn't care about! How large would the events be when indexing a public chain? How about rate limiting or size limits from an JSONRPC provider?

The simplest solution is once we know we care about a specific TX then one by one go and fetch the receipt. This can generate a lot of load on the blockchain node.

Would appreciate any thoughts @peterbroadhurst @awrichar @Chengxuan @matthew1001 !

[peterbroadhurst]

A better approach would be to store the TX Hash in FireFly Core that is returned when submitting a transaction to the connector!

This specific item needs to be elaborated on and decided on from a DB transaction and thread consistency perspective, before we could make the move. Today these are completely separate threads writing the correlating piece of information, and the transaction ID. That could be done through code experimentation, or up-front design, to work out how big the change is.

Noting there's a slack thread here digging into lots of details: https://discord.com/channels/905194001349627914/1333762986157932555