Block caching optimizations between WASM executions

[abourget]

Problem

• on Solana, the mere decoding of the protobuf block takes multiple milliseconds. This bytestream can be decoded multiple times: one for each module that has the block as an input.
• Ideally, the VM that decoded the protobuf message into a Rust struct would keep it for future use downstream. This however breaks determinism, and is akin to have a global variable.

Previous issues we had with determinism and global leaking happened BETWEEN blocks. We would reuse the VM between the different blocks, and so things would leak between segments (if the segments were different, the state of memory would be different). In this case, we would be looking at reusing the VM only during the execution of the tree of modules WITHIN the scope of a single block.

An idea would be to have a special function, per-chain, that is used to decode the large block model, and store it in a global (for the duration of a block). This function would be called once per execution of the module tree, the first time such a block appears in a module input. The module code would then get that block from the cache if it was decoded already, or decode it. This wouldn't prevent the allocation of the big block in the module that needs it. Only the time it takes to decode it and lay it out in the program's memory. The program would assume that a) the global would always be clean for a new block, b) that the block input would always be the same, c) making the cached struct or a new run at decoding the Block one and the same. The runtime would have no way to validate that no one has modified the struct representing the Block within the WASM environment, so there is a risk that decoding the Block message from bytes (managed by the Substreams runtime), and reading from the cache, be indeterminist.

Because this feature risks indeterminism vs the current mode, we need to introduce a declarative feature flag system in the spkg (and manifest), as well as in the Request. This would be needed to conserve the expectation of determinism (and lift that expectation when the flag is there) in the context of Subservices (deployable units) sent to The Graph Network.

/cc @billettc