Reusable Substreams modules with rewired inputs (use:)

[abourget]

Context

Currently, we can reuse a module while overriding its params and initialBlocks.

However, there's no easy way to declare a completely generic module (ex: a mapper that converts a proto:sf.substreams.sink.database.v1.DatabaseChanges to type: proto:sf.substreams.entity.v1.EntityChanges) and use it on the output on our module, because we cannot override its "input".

The idea is to be able to swap a module's input with another input that has the same type.

ex

specVersion: v0.1.0
package:
  name: byac
  version: v0.1.0

imports:
  converter: spkg.io/streamingfast/converter-v1.0.2

modules:
...
  - name: db_out
    kind: map
    initialBlock: 12287507
    inputs:
      - map: map_events
    output:
      type: proto:sf.substreams.sink.database.v1.DatabaseChanges

  - name: graph_out
    use: converter:dbout_to_graphout
    inputs:
      - map: db_out

The line:

    use: converter:dbout_to_graphout

in here would make it use the code from the imported module, but rewire the inputs and outputs, making sure that they are of the same type.

The converter author has a substreams.yaml like this:

specVersion: v0.1.0
package:
  name: converter
  version: v1.0.2

modules:
...
  - name: example_dbout  # write some code here that does nothing, just to link things.
    kind: map
    inputs:
       source: sf.substreams.v1.Clock
    output:
      type: proto:sf.substreams.sink.database.v1.DatabaseChanges

  - name: dbout_to_graphout
    kind: map
    inputs:
      - map: example_dbout
    output:
      type: proto:sf.substreams.entity.v1.EntityChanges

TODO

The substreams client will do all the required remapping for that feature.

• Implement the use attribute in the yaml parser
• make sure that a module in substreams.yaml that has use can only specify (override) the following attributes:
  inputs, initialBlockand the (not yet implemented)blockIndex(the params can always be set in theparams` section of the manifest)
• Also validate that the overridden inputs are similar to the previous one (same number, same types)
• Implement all the input rewire logic so that the generated modules are tweaked as requested
• Document the usage (with the example in this issue)

[ArnaudBger]

#432