fix-doc: Bugfix and Readme update (#60)

* feat: Allow building docker compose ps cmd without -f flag

* fix(templates): Remove MEV key from LH-Ropsten .env template

* fix: Bad checks for docker compose plugin availability

* fix: Use -f flag for 'compose ps' on docker compose checks

* fix: compose plugin not being installed on root user

Fixed setup docker scripts for Ubuntu 20.04/22.04. Compose plugin was being installed on the current user, but sedge is currently executing everything using sudo

* fix: Track Sync stoping early

There are false responses from the nodes, mostly at they start up time. Fixed stoping the tracking after consecutive synced results

* doc: Update README

Changes:
- Add table with mev-boost support information
- Add more instructions and descriptions about the tool

* chore: Update CHANGELOG

* doc(readme): Add table with OS support for dependency installation

* fix: TrackSync flags handle

CHANGELOG.md

@@ -6,20 +6,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.2] - 2022-07-21
+
+### Added
+- Table with mev-boost support information on the Readme
+- Table with OS support for dependency installation
+- More instructions and descriptions about the tool on the Readme
+
+### Changed 
+- Update Roadmap.
+
+### Fixed
+- Fix bad checks for compose availability.
+- Stop running Lighthouse with mev-boost settings on Ropsten. Official and stable docker image doesn't support mev-boost yet.
+- Fix compose installation. Now compose is installed on root user, as sedge runs everything using sudo currently.
+- Fix track sync of nodes stoping early.
+
 ## [0.1.1] - 2022-07-20
 ### Changed
-- Update Roadmap
+- Update Roadmap.
 
 ### Fixed
-- Fix error when trying to generate jwtsecret on unexisting folder
+- Fix error when trying to generate jwtsecret on unexisting folder.
 
 ## [0.1.0] - 2022-07-15
 ### Added
 - Create cli tool able to set up a Ethereum based validator in an on-premise way.
 - Generate `docker-compose` scripts and `.env` files for selected clients with the `cli` command.
 - Generate keystore folder with the cli using `keys` command. The inner tool used for this is the [staking-deposit-cli](https://github.com/ethereum/staking-deposit-cli) tool.
 - Generate `jwtsecret` for post-merge networks.
-- Test coverage (unit tests)
-- Integrate Kiln network
-- Integrate Ropsten network
-- Integrate MEV-Boost as an option
+- Test coverage (unit tests).
+- Integrate Kiln network.
+- Integrate Ropsten network.
+- Integrate MEV-Boost as an option.

README.md

@@ -64,6 +64,11 @@ sudo cp $GOPATH/bin/sedge /usr/local/bin/
 sudo cp sedge/build/sedge /usr/local/bin/
 ```
 
+#### Download binary from release page
+1. Download correct binary for the machine you want to use for Sedge, i.e choose the correct target OS/Arch.
+2. Set binary as executable with `chmod +x <binary>`. Replace `<binary>` with the name of the downloaded binary.
+3. (Optional) Put the binary on path with `cp <binary> /usr/local/bin/sedge`
+
 ### Dependencies
 `sedge` dependencies are `docker` with `docker compose` plugin, but if you don't have those installed, `sedge` will show instructions to install them, or install them for you.
 
@@ -77,7 +82,59 @@ With `sedge cli` you can go through the entire workflow setup:
 
 Between steps 4 and 5 you can generate the validator(s) keystore folder using `sedge keys`. 
 
+The entire process is interactive, although you can use the `-y` flag to run Sedge without prompts.
+
 Check all the options and flags with `sedge cli --help`. More instructions or guides about sedge's features will come soon!
+
+### Run a validator with mev-boost on Ropsten
+
+The following command takes care of the setup of a Teku consensus and validator nodes on Ropsten with a random execution client:
+
+```
+sedge cli --network ropsten -c teku
+```
+
+The `--network` flag allow you to choose the target network for the setup. Check out supported networks below. Default network is mainnet.
+
+The `-c/-v` flag is to select the desired consensus/validator client for the setup. If you only use one of those flags, then the same client pair will be used for consensus and validator nodes.
+
+There is also a `-e` flag to select the execution client. The default behavior is to choose a randomized client, that's why if we skip the `-e` flag this time, a randomized execution client will be used.
+
+mev-boost is a default setting as long as Sedge supports mev-boost for the selected client and network. If you don't want to use mev-boost in this case, then add the `--no-mev-boost` flag to the command.
+
+### Configuration file
+When you run Sedge for the first time, it generates a `.sedge.yml` on your HOME directory. This file should look like this:
+
+```yaml
+dependencies:
+  - docker
+
+executionClients:
+  - geth
+  - nethermind
+
+consensusClients:
+  - lighthouse
+  - lodestar
+  - prysm
+  - teku
+
+validatorClients:
+  - lighthouse
+  - lodestar
+  - prysm
+  - teku
+
+logs:
+  logLevel: info
+```
+
+If you want to know what Sedge does in every step, then just read the logs. Sedge is logging every step. Some of the applied steps or commands are shown on the debug logs only. To see these logs you need to replace `info` for `debug` on the `logLevel` field.
+
+You can modify the clients there to customize Sedge's random selection of clients. If you remove a client there, it won't be randomly selected. Be careful not to add an unsupported client, if this client is choosed you will face an error. Sedge knows very well which clients it supports.
+
+You don't need to modify the `dependencies` field at all. If you alter it, Sedge may not work as expected.
+
 ## 🔥 What can you do with sedge today?
 
 - Select an execution, consensus and validator node (manually or automatically) and generate a `docker-compose` script with production-tested configurations to run the setup you want.
@@ -86,6 +143,10 @@ Check all the options and flags with `sedge cli --help`. More instructions or gu
 
 > The setup is currently designed to start all three nodes required to run a validator (execution, consensus and validator node). Soon `sedge` will let you directly connect to a public or remote node. The execution and consensus nodes will be executed first, and the validator node will be executed automatically after those nodes are synced, giving you time to prepare the keystore file and make the deposit for your staked ether.
 
+If you are familiar with `docker`, `docker compose`, and the validator setup, then you can use Sedge to generate a base docker-compose script with the recommended settings, stop Sedge instead of letting it execute the script, and then edit the script as much as you want. Is a lot more easier than doing everything from scratch!
+
+> Although Sedge supports several clients, is still on beta. Some settings may not work because -at least on the testnets- the clients are constantly evolving. Please let us know any issues you encounter!
+
 ## Supported networks and clients
 
 ### Mainnet
@@ -106,6 +167,38 @@ Check all the options and flags with `sedge cli --help`. More instructions or gu
 |            | Prysm      | Prysm      |
 |            | Teku       | Teku       |
 
+### Ropsten
+
+| Execution  | Consensus  | Validator  |
+| ---------- | ---------- | ---------- |
+| Geth       | Lighthouse | Lighthouse |
+| Nethermind | Lodestar   | Lodestar   |
+|            | Prysm      | Prysm      |
+|            | Teku       | Teku       |
+
+
+### CL clients with Mev-Boost
+
+| Client     | Mev-Boost | Networks   |
+| ---------- | --------- | ---------- |
+| Lighthouse | no*       | Ropsten    |
+| Lodestar   | no        | -          |
+| Prysm      | no        | -          |
+| Teku       | yes       | Ropsten    |
+
+> Settings for Lighthouse with mev-boost are quite ready, we are waiting for an official and stable Lighthouse docker image with mev-boost support
+## Supported Linux flavours for dependency installation
+
+| OS             | Versions                |
+| -------------- | ----------------------- |
+| Ubuntu         | 22.04,21.10,21.04,20.04 |
+| Debian         | 11,10,9,8               |
+| Fedora         | 35,34                   |
+| CentOS         | 8                       |
+| Arch           | -                       |
+| Amazon Linux 2 | -                       |
+| Alpine         | 3.15,3.14,3.14.3        |
+
 ## ✅ Roadmap
 The following roadmap covers the main features and ideas we want to implement but doesn't cover everything we are planning for this tool. Stay in touch if you are interested, a lot of improvements are coming in the next two months. Please note that this Roadmap is continually changing until version 1.0.
 
@@ -124,6 +217,7 @@ The following roadmap covers the main features and ideas we want to implement bu
 
 ### Version 0.3
 - [ ] Enable use of public execution and consensus nodes
+- [ ] Integrate Goerli network
 - [ ] Include monitoring tool for alerting, tracking validator balance, and tracking sync progress and status of nodes
 
 ### Version 0.4

cli/cli_test.go

@@ -275,6 +275,8 @@ func buildCliTestCase(
 		data: []posmoni.EndpointSyncStatus{
 			{Endpoint: inspectExecutionUrl + ":" + ports["ELApi"], Synced: true},
 			{Endpoint: inspectConsensusUrl + ":" + ports["CLApi"], Synced: true},
+			{Endpoint: inspectExecutionUrl + ":" + ports["ELApi"], Synced: true},
+			{Endpoint: inspectConsensusUrl + ":" + ports["CLApi"], Synced: true},
 		},
 	}
 

cli/cli_utils.go

@@ -203,6 +203,17 @@ func runAndShowContainers(services []string) error {
 		return fmt.Errorf(configs.DockerEngineOffError, err)
 	}
 
+	// Check that compose plugin is installed with docker running 'docker compose ps'
+	dockerComposePsCMD := commands.Runner.BuildDockerComposePSCMD(commands.DockerComposePsOptions{
+		Path: filepath.Join(generationPath, configs.DefaultDockerComposeScriptName),
+	})
+	log.Debugf(configs.RunningCommand, dockerComposePsCMD.Cmd)
+	dockerComposePsCMD.GetOutput = true
+	_, err := commands.Runner.RunCMD(dockerComposePsCMD)
+	if err != nil {
+		return fmt.Errorf(configs.DockerComposeOffError, err)
+	}
+
 	// Run docker-compose script
 	upCMD := commands.Runner.BuildDockerComposeUpCMD(commands.DockerComposeUpOptions{
 		Path:     filepath.Join(generationPath, configs.DefaultDockerComposeScriptName),
@@ -303,17 +314,31 @@ func trackSync(m MonitoringTool, elPort, clPort string, wait time.Duration) erro
 	statuses := m.TrackSync(done, []string{consensusUrl}, []string{executionUrl}, wait)
 
 	var esynced, csynced bool
+	// Threshold to stop tracking, to avoid false responses
+	times := 0
 	for s := range statuses {
 		if s.Error != nil {
 			return fmt.Errorf(configs.TrackSyncError, s.Endpoint, s.Error)
 		}
-		esynced = esynced || (s.Synced && s.Endpoint == executionUrl)
-		csynced = csynced || (s.Synced && s.Endpoint == consensusUrl)
+
+		if s.Endpoint == executionUrl {
+			esynced = s.Synced
+		} else if s.Endpoint == consensusUrl {
+			csynced = s.Synced
+		}
+
 		if esynced && csynced {
-			// Stop tracking
-			done <- struct{}{}
-			log.Info(configs.NodesSynced)
-			break // statuses channel might still have data before closing done channel
+			times++
+			// Stop tracking after consecutive synced reports
+			if times == 3 {
+				// Stop tracking
+				done <- struct{}{}
+				log.Info(configs.NodesSynced)
+				break // statuses channel might still have data before closing done channel
+			}
+		} else {
+			// Restart threshold
+			times = 0
 		}
 	}
 

cli/down.go

@@ -30,7 +30,7 @@ import (
 var downCmd = &cobra.Command{
 	Use:   "down [flags]",
 	Short: "Shutdown sedge running containers",
-	Long:  `Shutdown sedge running containers using docker-compose CLI. Shortcut for 'docker compose -f <script> down'`,
+	Long:  `Shutdown sedge running containers using docker compose CLI. Shortcut for 'docker compose -f <script> down'`,
 	Run: func(cmd *cobra.Command, args []string) {
 		if err := utils.PreCheck(generationPath); err != nil {
 			log.Fatal(err)

configs/errors.go

@@ -37,6 +37,7 @@ const (
 	DependencyNotInstalledError       = "dependency %s is not installed on host machine"
 	DependenciesMissingError          = "required dependencies are missing. Please install them and try again. Dependencies can be installed using 'sedge cli' command"
 	DockerEngineOffError              = "it seems docker engine is not running. Please start it and try again. Error: %v"
+	DockerComposeOffError             = "it seems docker compose plugin is not installed. Please install it and try again. Error: %v"
 	DockerComposeScriptNotFoundError  = "docker-compose script not found at %s. Please run 'sedge cli' command to generate it or check the script path. By default, it should be located at %s"
 	ScriptIsNotRunningError           = "services of docker-compose script provided are not running. Error: %v"
 	GettingLogsError                  = "failed to get logs for services %s. Error: %v"

internal/pkg/commands/cmd_runner_unix.go

@@ -54,6 +54,7 @@ func (cr *UnixCMDRunner) BuildDockerPSCMD(options DockerPSOptions) Command {
 func (cr *UnixCMDRunner) BuildDockerComposePSCMD(options DockerComposePsOptions) Command {
 	flags := ""
 	name := ""
+
 	if options.Services {
 		log.Debug(`Command "docker compose ps" built with "--service" flag.`)
 		flags += " --services"

internal/utils/checks.go

@@ -62,8 +62,8 @@ a. error
 Error if any
 */
 func PreCheck(generationPath string) error {
-	// Check that docker and docker-compose are installed
-	pending := CheckDependencies([]string{"docker", "docker-compose"})
+	// Check docker is installed
+	pending := CheckDependencies([]string{"docker"})
 	for _, dependency := range pending {
 		log.Errorf(configs.DependencyNotInstalledError, dependency)
 	}
@@ -80,6 +80,17 @@ func PreCheck(generationPath string) error {
 		return fmt.Errorf(configs.DockerEngineOffError, err)
 	}
 
+	// Check that compose plugin is installed with docker running 'docker compose ps'
+	dockerComposePsCMD := commands.Runner.BuildDockerComposePSCMD(commands.DockerComposePsOptions{
+		Path: filepath.Join(generationPath, configs.DefaultDockerComposeScriptName),
+	})
+	log.Debugf(configs.RunningCommand, dockerComposePsCMD.Cmd)
+	dockerComposePsCMD.GetOutput = true
+	_, err = commands.Runner.RunCMD(dockerComposePsCMD)
+	if err != nil {
+		return fmt.Errorf(configs.DockerComposeOffError, err)
+	}
+
 	// Check if docker-compose script was generated
 	file := generationPath + "/" + configs.DefaultDockerComposeScriptName
 	if _, err := os.Stat(file); os.IsNotExist(err) {

internal/utils/checks_test.go

@@ -83,7 +83,6 @@ func TestPreCheck(t *testing.T) {
 		runner          commands.CommandRunner
 		isErr           bool
 		noDocker        bool
-		noCompose       bool
 	}{
 		{
 			name:            "Ok",
@@ -138,9 +137,6 @@ func TestPreCheck(t *testing.T) {
 			if !tc.noDocker {
 				dPath = test.CreateFakeDep(t, "docker")
 			}
-			if !tc.noCompose {
-				dcPath = test.CreateFakeDep(t, "docker-compose")
-			}
 
 			commands.InitRunner(func() commands.CommandRunner {
 				return tc.runner

templates/envs/ropsten/validator/lighthouse.tmpl

@@ -8,5 +8,4 @@ VL_INSTANCE_NAME=LighthouseValidator
 VL_IMAGE_VERSION={{if .VlImage}}{{.VlImage}}{{else}}sigp/lighthouse:latest{{end}}
 KEYSTORE_DIR={{.KeystoreDir}}
 VL_DATA_DIR={{.VlDataDir}}
-MEV=true
 {{ end }}

templates/setup/linux/docker/ubuntu_20.04.sh

@@ -23,7 +23,7 @@ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
 
 # Install docker-compose
 
-DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
-mkdir -p $DOCKER_CONFIG/cli-plugins
-curl -SL "https://github.com/docker/compose/releases/download/v2.6.1/docker-compose-$(uname -s)-$(uname -m)" -o $DOCKER_CONFIG/cli-plugins/docker-compose
-chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
+DOCKER_CONFIG=${DOCKER_CONFIG:-/root/.docker}
+sudo mkdir -p $DOCKER_CONFIG/cli-plugins
+sudo curl -SL "https://github.com/docker/compose/releases/download/v2.6.1/docker-compose-$(uname -s)-$(uname -m)" -o $DOCKER_CONFIG/cli-plugins/docker-compose
+sudo chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

templates/setup/linux/docker/ubuntu_22.04.sh

@@ -24,6 +24,6 @@ sudo apt-get install -y docker-ce
 # Install docker-compose
 
 DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
-mkdir -p $DOCKER_CONFIG/cli-plugins
-curl -SL "https://github.com/docker/compose/releases/download/v2.6.1/docker-compose-$(uname -s)-$(uname -m)" -o $DOCKER_CONFIG/cli-plugins/docker-compose
-chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
+sudo mkdir -p $DOCKER_CONFIG/cli-plugins
+sudo curl -SL "https://github.com/docker/compose/releases/download/v2.6.1/docker-compose-$(uname -s)-$(uname -m)" -o $DOCKER_CONFIG/cli-plugins/docker-compose
+sudo chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose