feat(code): BlockSync MVP (#440)

* feat(code): Add dummy `BlockSync` actor

* Add `/block_sync` pubsub channel

* Rename `broadcast` to `publish`

* Use `eyre` instead of boxed `Error`

* Have peers publish their status every time they move to a new height/round

* Fix bug where `StartRound` effect was performed twice

* Move status handling into BlockSync actor

* Less noise

* Move status update logic onto BlockSync actor

* Decouple GossipConsensus actor from Consensus and BlockSync actors

* Push latest hight from consensus to blocksync

* Revert "Decouple GossipConsensus actor from Consensus and BlockSync actors"

This reverts commit 0a59ada.

* feat(code): BlockSync RPC (#457)

* Define RPC behaviour for BlockSync

* Hook BlockSync behaviour into networking layer

* Revert to encoding-agnostic networking layer

* Hook BlockSync actor to BlockSync behaviour

* Fix unwraps

* Remove round from status

* code(feat): Add block store (#458)

* Add block store for blocksync testing

* Sync helper get stored block at height from host

* Add block store pruning

* Update Cargo.lock

* Add blocksync protocol initial implementation, wip

* Add config flag for maximum number of blocks in store

* Ask for the block only from peer at lower height.

* Fix mistake in host, should return ReceivedProposedValue and not
ProposedValue.

* When processing the synced block use the on_ handlers instead of
apply_driver_input.

* When storing the decision look for full proposal at proposal round and
NOT consensus round

* Change max_retain_blocks in config.toml to match the default.

* Set voting power for all vals to 1 for easier debugging.

* Use on_vote instead of apply_driver_input so our own commit is stored
and used on decided operations.

* Fix spelling.

* Remove lower number of peers (was used for testing).

* Store the commits in a set to avoid duplicates.

* Return if proposal from non-proposer.

* Add TODO for potential min block height in Status

* Change max_retain_blocks default to 100.

* Small cleanup

* Work around libp2p bug where a node cannot publish to a topic if restarted too quickly

See libp2p/rust-libp2p#5097

* Cleanup

* Ensure validator set is always sorted properly, even after being deserialized from genesis

* Update spawn.fish

* Move BlockSyncState to blocksync crate

* Add batched request for blocks

* Fix bug where only a single block would be sent back

* Revert block batching

* Cleanup

* Remove outdated comment

* Post-merge fixes

* Post-merge fix

* Cleanup

* chore(code): Extract logic from `BlockSync` actor and into `blocksync` crate (#487)

* chore(code): Extract BlockSync logic from actor and into `blocksync` crate

* Cleanup

* Fix doc

* Fix logs

* Use custom timeout per test

* Add timeouts to BlockSync requests (#490)

* feat(code/blocksync): Add timeouts to BlockSync requests

* Fix parsing of blocksync config

* Randomize choice of peer to sync from (#492)

* Add basic integration test (#493)

* Re-enable mempool in integration test

* Add config option `blocksync.enabled`

* Send back actual block bytes instead of just the block hash

* Fix example config file

* test(code/blocksync): Introduce small DSL for expressing test scenarios (#496)

* Introduce small DSL for expressing test scenarios

* Add basic integration test for BlockSync

* Update fish script

* Allow overriding the transport used in tests via a `MALACHITE_TRANSPORT` env variable

* Update test

* Add doc comments

* Use TCP by default in tests

Can be overriden locally with `MALACHITE_TRANSPORT=quic`

* Run integration tests sequentially

* Run integration tests in release mode

* Fix coverage job

* Try again

* Do not panic when tracing subscribers are set twice

* Enable basic optimizations in debug mode

* Update MSRV to 1.82

* feat(code/blocksync): Only request sync from peers which have the requested block in their store (#495)

* feat(code/blocksync): Only request sync from peers which have the requested block in their store

For this, we extend the status with the earliest block height available
in the store, to ensure we only request a block from peers which have
told us they have it.

* Remove `blocksync::State::random_peer_at_or_above()` method

* Update clippy msrv

* Add metrics

* Ensure Consensus and BlockSync actors never fail

---------

Co-authored-by: Anca Zamfir <[email protected]>

.github/workflows/coverage.yml

@@ -9,15 +9,22 @@ on:
       - specs/quint/**
       - .github/workflows/coverage.yml
 
+env:
+  CARGO_TERM_COLOR: always
+  CARGO_INCREMENTAL: 0
+  CARGO_PROFILE_DEV_DEBUG: 1
+  CARGO_PROFILE_RELEASE_DEBUG: 1
+  RUST_BACKTRACE: short
+  CARGO_NET_RETRY: 10
+  RUSTUP_MAX_RETRIES: 10
+
 jobs:
   integration:
     name: Integration
     runs-on: ubuntu-latest
     defaults:
       run:
         working-directory: code
-    env:
-      CARGO_TERM_COLOR: always
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -36,17 +43,16 @@ jobs:
         with:
           toolchain: nightly
           components: llvm-tools-preview
-      - name: Install cargo-nextest
-        uses: taiki-e/install-action@cargo-nextest
       - name: Install cargo-llvm-cov
         uses: taiki-e/install-action@cargo-llvm-cov
       - name: Generate code coverage
         run: |
-          cargo llvm-cov nextest \
+          cargo llvm-cov test \
             --workspace \
             --exclude malachite-test-mbt \
             --ignore-filename-regex crates/cli \
             --all-features \
+            --jobs 1 \
             --ignore-run-fail \
             --lcov \
             --output-path lcov.info

.github/workflows/rust.yml

@@ -15,6 +15,7 @@ on:
 
 env:
   CARGO_INCREMENTAL: 0
+  CARGO_TERM_COLOR: always
   CARGO_PROFILE_DEV_DEBUG: 1
   CARGO_PROFILE_RELEASE_DEBUG: 1
   RUST_BACKTRACE: short
@@ -23,7 +24,7 @@ env:
 
 jobs:
   test:
-    name: Test
+    name: Unit Tests
     runs-on: ubuntu-latest
     defaults:
       run:
@@ -47,10 +48,38 @@ jobs:
           cache-workspaces: "code"
       - name: Install cargo-nextest
         uses: taiki-e/install-action@cargo-nextest
-      - name: Build code
-        run: cargo nextest run --workspace --all-features --no-run
       - name: Run tests
-        run: cargo nextest run --workspace --all-features --no-fail-fast --failure-output final
+        run: cargo nextest run --workspace --all-features --no-fail-fast --failure-output final --test-threads 1 --exclude malachite-starknet-test --exclude malachite-discovery-test
+
+  integration:
+    name: Integration Tests
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: code
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v3
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Setup Node
+        uses: actions/setup-node@v3
+        with:
+          node-version: "18"
+      - name: Install Quint
+        run: npm install -g @informalsystems/quint
+      - name: Setup Rust toolchain
+        uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          cache-workspaces: "code"
+      - name: Install cargo-nextest
+        uses: taiki-e/install-action@cargo-nextest
+      - name: Run Starknet tests
+        run: cargo nextest run --all-features --no-fail-fast --failure-output final --test-threads 1 -p malachite-starknet-test
+      - name: Run Discovery tests
+        run: cargo nextest run --all-features --no-fail-fast --failure-output final --test-threads 1 -p malachite-discovery-test
 
   clippy:
     name: Clippy

README.md

@@ -6,7 +6,7 @@
 [![Code coverage][coverage-image]][coverage-link]
 [![Apache 2.0 Licensed][license-image]][license-link]
 ![Rust Stable][rustc-image]
-![Rust 1.74+][rustc-version]
+![Rust 1.82+][rustc-version]
 [![Quint 0.18][quint-version]][quint-repo]
 
 Tendermint consensus in Rust
@@ -21,7 +21,7 @@ The repository is split in three areas, each covering one of the important areas
 
 ## Requirements
 
-- Rust v1.74+ ([rustup.rs](https://rustup.rs))
+- Rust v1.82+ ([rustup.rs](https://rustup.rs))
 - Quint v0.18+ ([github.com](https://github.com/informalsystems/quint))
 
 ## License
@@ -46,6 +46,6 @@ Unless required by applicable law or agreed to in writing, software distributed
 [license-image]: https://img.shields.io/badge/license-Apache_2.0-blue.svg
 [license-link]: https://github.com/informalsystems/hermes/blob/master/LICENSE
 [rustc-image]: https://img.shields.io/badge/Rust-stable-orange.svg
-[rustc-version]: https://img.shields.io/badge/Rust-1.74+-orange.svg
+[rustc-version]: https://img.shields.io/badge/Rust-1.82+-orange.svg
 [quint-version]: https://img.shields.io/badge/Quint-0.18-purple.svg
 [quint-repo]: https://github.com/informalsystems/quint