🚧 | Y5: Implement existing BFT consensus in subnets

[jsoares]

Description

We intend to rapidly experiment with the hierarchical consensus architecture by integrating an existing, production-ready BFT implementation as a possible subnet consensus protocol. After analysing several options, we decided on Tendermint as a proof-of-concept.

This project will, therefore, consist of implementing, documenting, and evaluating Tendermint as a consensus layer for Eudico subnets. Tendermint Core will be used in the service-based model, rather than in the traditional ABCI-based model.

In our approach, Tendermint Core acts as a message system providing BFT total order broadcast of input messages without any modifications of the Filecoin state. Each Eudico node interacts with a trusted sidecar Tendermint node A Eudico node sends input messages to the corresponding Tendermint node; Tendermint validates them and ensures that the messages are recorded on every Tendermint node in the same order. The Eudico node then retrieves Tendermint blocks, perform static and semantic validation, and generates a Filecoin block with the verified messages, which can be applied directly.

Scope

1. Explore and test the expressiveness and power of Eudico’s design and interfaces (e.g., Consensus interface) by adding support for the Tendermint consensus protocol.
2. Improve Eudico characteristics by adding a BFT-type consensus protocol with better characteristics and justified properties in addition to already i plemented and used consensus protocols (e.g., PoW).
3. Research Eudico's “consensus upgradeability” and improve it if necessary.
4. Explore Filecoin design and Eudico implementation pitfalls for (potential) future integration with MirBFT protocols.

Dependencies

• 🚧 | B1: Hierarchical consensus PoC #4

Resources

• Project spec
• Tendermint
• Implementation

Talks

2022-02-07 ConsensusLab Team Week 22Q1: Y5 update

Demos

2022-02-10 Y5 Tendermint subnet (Denis)

2022-02-21 Y5 Tendermint fault injection (Denis)

2022-02-28 Y5 Tendermint fault injection (Denis)

[jsoares]

2022-02-21 Meeting Notes

Updates

• Deployed a Tendermint testnet with 4 nodes
• Found a simple way to add a new node to Tendermint testned, but! It also should be supported on the app layer. That will be the next step if necessary
• Improved the implementation: found a bad cache that should support Tendermint connections
• Performed two early internal demos: Tendermint in subnet, Tendermint in root net with fault injection
• Merged the main eudico/filecoin branches

Discussion

• How do the state transfers of Eudico and Tendermint cooperate?
• Does someone need to have exclusive control?
• Eudico block exchange:
• Exchange block headers
• Check in Tendermint whether it has been verified
• Pick up the messages and check inclusion
• Are we wasting bandwidth? It seems exchange is taking place at multiple levels.
• This only happens during catch-up — otherwise you get it through gossipsub once.
• [...]
• Not a fundamental issue — implementation of MirBFT will sidestep it.

Up next

• Understand how to use itests and start using it
• Implement basic integration tests. We will get an answer to the question is it possible to use itests for consensus testing
• Implement basic integration tests
• Make a PR

[jsoares]

2022-02-28

✋ Attendees

• @dnkolegov
• @adlrocha
• @matejpavlovic
• @jsoares
• @LefKok
• @vukolic

📣 Updates

• New demo: 4 Tendermint nodes (including 1 faulty node), 4 Eudico nodes, fault injection, root subnet with Tendermint consensus
• Created a README document for Tendermint implementation
• Created a checklist for a new consensus protocol implementation in Eudico
• Still trying to adapt and run itests for Eudico, can't catch a bug, asked Alfonso for help

🧵 Discussion

• Timeline recap
  • Main blocker right now is itests. If we can get this right, it will also work for future consensus protocols and allow us to write different kinds of tests (inc. benchmarks).
    • Alfonso aiming to unblock this by tomorrow. Will have sync call.
  • Planning to ramp up work on Y3. There are synergies between the two, inc. the upcoming testing work. Let's make sure the timeline for Y5 does not slide.
• Consensus as a service
  • Different number of Eudico nodes and ordering service nodes source of confusion in the past. Would it make sense to do this intentionally?
  • What is a subnet? What is an Eudico node?
    • One could define a subnet as using an external ordering service. Whoever joins the subnet would implicitly trust external orderer.
  • Endorsers and orderers in Hyperledger Fabric
    • Incentives?
  • Blockchain bridge integration in our design? E.g. Tendermint node is Cosmos network. Cryptoeconomic model.
    • We also discussed having consensusless subnets in the past. Having a bridge just means some of the transactions coming from other networks.
    • Aspirational. We should have a design that doesn't preclude this but not an immediate goal.
    • Running MirBFT node as a separate protocol makes more sense in this model.
      • What's the MirBFT node identity?
      • Counterpoint: Shipping two binaries comes with operational issues and is also more resource-intensive. It would be nice to have a single one.
      • For subnets, we're doing everything as goroutines in the same process.
      • Avoid duplicating stack as we're doing with Tendermint

🎯 Up next

• The main goal is run itests for Eudico

[jsoares]

2022-03-07

✋ Attendees

• @adlrocha
• @dnkolegov
• @jsoares
• @matejpavlovic
• @vukolic

📣 Updates

• Worked on itests
• Added a basic test for Eudico PoW consensus
• Added a basic test for Eudico delegated consensus

🧵 Discussion

• How do we stand on reconfiguration?
  • Does the Tendermint implementation support joining/leaving subnets?
  • Right now, the node set-up is statically configured. We don't join or leave networks.
  • But we support Tendermint as both root and subnet consensus (see early February demo), even simultaneously.
  • This isn't a requirement for Y3 completion, but we'll need it for Mir.

🎯 Up next

• Add a test for Tendermint
• Clean Eudico itests code
• Continue adapting itests to Eudico

[jsoares]

2022-03-14

✋ Attendees

• @adlrocha
• @dnkolegov
• @jsoares
• @matejpavlovic
• @vukolic

📣 Updates

• Created the PR for Tendermint code
• Created two issues related to Consensus interface and delegated consensus block delay
• Scripts and tests don't need Tendermint repo anymore

🧵 Discussion

• Alfonso's found an issue:

We support the use of the same Tendermint blockchain by several subnets, right? And here we are registering a new subnet to the consensus. The assumption here is that only the owner of the node can tailor this registration message, right? Why I ask is because I am wondering how critical would it be for a rogue node to register a subnet he is not part of (or that doesn't even exist yet).

• Yes, you are right. We register a Tendermint blockchain offset for a new subnet. Suppose we ran a Tendermint blockchain and a root subnet, then root's subnet offset may be 3 blocks. And then, after an hour, we decided to register a new subnet, its offset is 1000. If we don't use offsets then the second subnet will have to access all previous Tendermint blocks. So, yes, it is critical. A rogue node can register a network earlier than it would be registered, e.g. at epoch h. Then when the network is instantiated by real participants at epoch H they will have to process all blocks from epoch h to epoch H.

• Way forward

  • We're missing
    • decoupling subnets from Tendermint network (i.e., one Tendermint network/instance per Eudico subnet)
    • refactoring
    • benchmarking
    • misc fixes
  • We're on track for EOM
  • Non-Y5 items on Denis's plate (not critical for EoM, nice to have)
    • B1 itests for subnets
    • Y3 interfaces

🎯 Up next

• Fix issues Alfonso has found, refactor the code according to the comments, test it

[jsoares]

2022-03-21

✋ Attendees

• @adlrocha
• @dnkolegov
• @jsoares
• @LefKok
• @matejpavlovic
• @vukolic

📣 Updates

• Removed unnecesarry features from the codebase
• Added acceptance tests to the README
• Implemented a benchmark tool
• Merged Tandermint consensus PR
• Did code review
• Added 3 issues and fixed 1 issue

🧵 Discussion

• We can write a guest post in the "Decentralized Thoughts" blog
  • Marko reaching out
  • Focus: hierarchical consensus; start from paper and research blog, make it more technical
• What kind of demo should I provide? Offline? What scenarios should I demonstrate?
  • April: Eudico Garden, cross-subnet transaction
  • Benchmark plan dependent on availability of tooling (pending Y2 outcome)
    • We could probably use traditional load generation tools.
    • We need to check whether API communication becomes the bottleneck (saturate API instead of consensus)
    • Demo benchmarking in May (go beyond Tendermint)

🎯 Up next

• Implementing itests for subnet
• Demo

[jsoares]

2022-03-28

✋ Attendees

• @dnkolegov
• @jsoares
• @matejpavlovic
• @vukolic

📣 Updates

• Implemented a basic flow (happy path) covering the most of subnet features for different consensus pairs for one Eudico host: pow-pow, pow-tendermint, delegated-tendermint, etc.
• Found several bugs, added several improvements.

🎯 Up next

• Adapt tests for deleg-deleg and filcns-* consensus pairs
• Look at Tendermint setup in Eudico-Garden

[jsoares]

The project wrapped up with the implementation demoed in the initial B3 project demo. See respective issue for more details: #6.

🥳