Merge master to t3 0523 (#3085)

* fix signers share computation to use only non-harmony nodes (#3008)

* Add LastEpochInCommittee at prestaking epoch too (#3014)

* write validator stats at the end of prestaking epoch (#3021)

* write validator stats at the end of prestaking epoch

* also need to write block reward accumulator

* [build] enable static build by default (#2885)

scripts/go_executable_build.sh -s (static build, default)
scripts/go_executable_build.sh -S (non-static build)

Signed-off-by: Leo Chen <[email protected]>

* Code formatting fix based on `staticcheck` (#3023)

* fix some static check issues

* more fixes according to staticcheck

* add nil check on getHeader

* Add more log info

* [rpc] Revert change to v2 GetBalance RPC (#3037)

* Cherrypick crosslink related fixes from t3 to master. (#3045)

* Optimize crosslink verification logic and add more cache (#3032)

* make crosslink broadcast smarter and more efficient (#3036)

* adjust catch up speed to be a sane number

* Fix crosslink broadcast condition (#3041)

* Tx pool error report fix & local build fix (#3046)

* [tx_list] Add transaction after you can fetch tx cost

* [tx_pool] Add error sink reports for removed tx

* Correct log levels for known transactions
* Remove redundant hash to `enqueue` and `add` signatures
* Update tests for signature changes

* [build] Fix local build for MacOs

* [tx_pool] Correct typo & logging level for known tx

* [config] Create Testnet config

* [node.sh] update testnet network config

* [config] correct testnet config

* [main] Enable beacon sync for explorer nodes (#3059)

* [apr] should not use snapshot from last epoch, use current epoch snapshot (#3051)

* use snapshot total delegation instead of current total delegation

* should not use snapshot from last epoch, use current epoch snapshot (#3049)

Co-authored-by: Rongjian Lan <[email protected]>

Co-authored-by: Rongjian Lan <[email protected]>

* cherrypick log level change

* [rpc] Add active status back to Validator information
[validator] Fix typo for ValidatorRPCEnhanced
[Makefile] Change make exe to build non-static by default

* [network] Added caching to node server (#3048)

* [network] Added caching to node server

* [network] renamed two singleflight.Group

* [network] added two error handling

* [network] added forget logic for single flight

* [network] changed single flight to lru cache solution

* [network] adjusted the cache size

* [network] changed cache size to 10000

* Transaction api improvements (#3057)

* [node] Add option to broadcast invalid tx (on by default)

* update main args to take in broadcast invalid tx
* update node.sh for new broadcast invalid tx option

* [rpc] Do not submit transaction if tx has been finalized

* Support legacy behavior and not return errors on tx submission

* [main] make broadcasting invalid tx default false

* [node] Fix formatting

* Add timeouts for the explorer HTTP service

* [test] State trans unit test (#3043)

* completing create-validator tests

generalizing create validator tests, started with edit validator tests

one round completed before major refactor

* [test] finished coding test TestCheckDuplicateFields

* [test] finished debug TestCheckDuplicateFields

* [test] Added TestDescription_EnsureLength

* [test] added test in validator_test.go TestUpdateDescription

* [test] Added test TestComputed_String

* [test] refactored TestValidatorSanityCheck and added some message in error message returned from sanity check

* [test] removed test index for debugging

* [test] Added test TestValidatorWrapper_SanityCheck. Added value check for TestMarshalUnmarshalValidator. Fixed rlp usage defect in UnmarshalValidator

* [test] added test TestVerifyBLSKeys

* [test] added test TestContainsHarmonyBLSKeys

* [test] added test TestCreateValidatorFromNewMsg

* [test] fixed some tests errors

* [test] added test TestUpdateValidatorFromEditMsg, last editted the test file, finished testing validator_test.go

* [staking] added numeric.Dec.Copy method

* [staking] added CommissionRates.Copy

* [staking] add nil case handle to copy method and related test case

* [test] added nil case for commissionRates.Copy test

* [staking] finished CreateValidator.Copy and related test case

* [staking] added EditValidator.Copy method and related test case

* [test] added zero test cases for Copy method

* [staking] implemented Delegate.Copy and related unit test case

* [staking] added Undelegate.Copy and CollectReward.Copy method. Implemented corresponding test cases

* [test] added two more simple unit tests

* [test] solving the merge conflict with deep copy fix

* [test] added schedule for sharding

* [test] refactored double_sign_test.go

* [test] comment out all test cases for now.

* [test] added record Copy method for testing

* [test] added the first test case in TestVerify

* [test] finished test TestVerify

* [test] Added test TestVerify, TestApplySlashRate, TestSetDifference

* [test] half through TestDelegatorSlashApply

* [test] fix the problem cause by unsafe usage of big.Int

* [staking] added the copy methods to validatorWrapper

* [test] added test cases for copy method for staking/validatorWrapper

* [test] added test case TestApply

* [test] added test case TestRate

* [test] fix the golint errors

* [test] removed commented out unit test code

* [test] remote the empty line in imports

* [test] moved copy methods to stake_testing module

* [test] removed usage of interface stateDB

* [test] removed empty lines in imports

* [test] fixed golint package name issues

* [test] removed a todo when writing code

* [test] moved record.Copy to staking/slash/test

* [test] add some changes

* [test] added prototypes in staketest module

* [test] fix the golint issue in staketest

* [test] make prototype more beautiful

* [test] refactored test TestCheckDuplicateFields

* [test] add createValidator test data

* [test] added positive test case for VerifyAndCreateValidatorFromMsg

* [test] added create validator test

* [test] added the positive test case for TestVerifyAndEditValidatorFromMsg

* [test] added the tests case TestVerifyAndEditValidatorFromMsg

* [test] fix one of the error is not triggered as expected

* [test] more changes in test

* [test] fix the positive test data for TestVerifyAndDelegateFromMsg

* [test] Fixed two comment golint errors

* [test] added delegate tests

* [test] added a make function. WIP

* [test] added undelegate positive test

* [test] added negative test cases for undelegate

* [test] added positive test. not passed. Need to add assert method in test suites

* [test] added equal function to check Validator Equality

* [test] added equals for staketest

* [test] replaced deep equal with new equal methods:

* [test] removed unused codes

* [test] Finishing touch

* [test] fix comment golint issue

* [test] removed dead code in staking_verifier_test.go

Co-authored-by: Ganesha Upadhyaya <[email protected]>

* changed the sync port to dns port in flag

* [network] Limit client sync connection (#3071)

* [network] limit client connected peers
* Changed signature and added tests

* Set secure permissions on generated blskey pass file in node.sh

* Fix spellin gtypo in node.sh

* fix latest param handling in api v1

* added comment for the isGreaterThanLatestBlockNum check

* fix a nil pointer crash due to previous revert

Signed-off-by: Leo Chen <[email protected]>

* fix nil pointer crash

Signed-off-by: Leo Chen <[email protected]>

* fix nil pointer of block

Signed-off-by: Leo Chen <[email protected]>

* [rpc] add viewID and epoch to RPCMarshalBlock (#3009)

* [rpc] add viewID and epoch to RPCMarshalBlock

* add apt-get update

* Do not rollback current block on failed block sync (#3101)

Co-authored-by: Ganesha Upadhyaya <[email protected]>
Co-authored-by: Rongjian Lan <[email protected]>
Co-authored-by: Jacky Wang <[email protected]>
Co-authored-by: Janet Liang <[email protected]>
Co-authored-by: Daniel Van Der Maden <[email protected]>
Co-authored-by: Janet Liang <[email protected]>
Co-authored-by: Sebastian Johnsson <[email protected]>
Co-authored-by: hsiung <[email protected]>
Co-authored-by: Dennis Won <[email protected]>
Co-authored-by: Yishuang Chen <[email protected]>

Dockerfile

@@ -16,6 +16,7 @@ ENV LD_LIBRARY_PATH=${BLS_DIR}/lib:${MCL_DIR}/lib
 ENV GIMME_GO_VERSION="1.14.1"
 ENV PATH="/root/bin:${PATH}"
 
+RUN apt-get update -y
 RUN apt install libgmp-dev libssl-dev curl git \
 psmisc dnsutils jq make gcc g++ bash tig tree sudo vim \
 silversearcher-ag unzip emacs-nox nano bash-completion -y

Makefile

@@ -16,7 +16,7 @@ libs:
 	make -C $(TOP)/bls BLS_SWAP_G=1 -j8
 
 exe:
-	./scripts/go_executable_build.sh
+	./scripts/go_executable_build.sh -S
 
 race:
 	./scripts/go_executable_build.sh -r

api/service/explorer/service.go

@@ -102,7 +102,13 @@ func (s *Service) Run() *http.Server {
 
 	// Do serving now.
 	utils.Logger().Info().Str("port", GetExplorerPort(s.Port)).Msg("Listening")
-	server := &http.Server{Addr: addr, Handler: s.router}
+	server := &http.Server{
+		Addr:         addr,
+		Handler:      s.router,
+		ReadTimeout:  5 * time.Second,
+		WriteTimeout: 10 * time.Second,
+		IdleTimeout:  120 * time.Second,
+	}
 	go func() {
 		if err := server.ListenAndServe(); err != nil {
 			utils.Logger().Warn().Err(err).Msg("server.ListenAndServe()")

api/service/syncing/syncing.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"encoding/hex"
 	"fmt"
+	"math/rand"
 	"reflect"
 	"sort"
 	"strconv"
@@ -36,6 +37,11 @@ const (
 	verifyHeaderBatchSize    uint64 = 100  // block chain header verification batch size
 	SyncLoopFrequency               = 1    // unit in second
 	LastMileBlocksSize              = 50
+
+	// after cutting off a number of connected peers, the result number of peers
+	// shall be between numPeersLowBound and numPeersHighBound
+	numPeersLowBound  = 3
+	numPeersHighBound = 5
 )
 
 // SyncPeerConfig is peer config to sync.
@@ -225,6 +231,10 @@ func (peerConfig *SyncPeerConfig) GetBlocks(hashes [][]byte) ([][]byte, error) {
 
 // CreateSyncConfig creates SyncConfig for StateSync object.
 func (ss *StateSync) CreateSyncConfig(peers []p2p.Peer, isBeacon bool) error {
+	// limit the number of dns peers to connect
+	randSeed := time.Now().UnixNano()
+	peers = limitNumPeers(peers, randSeed)
+
 	utils.Logger().Debug().
 		Int("len", len(peers)).
 		Bool("isBeacon", isBeacon).
@@ -237,6 +247,7 @@ func (ss *StateSync) CreateSyncConfig(peers []p2p.Peer, isBeacon bool) error {
 		ss.syncConfig.CloseConnections()
 	}
 	ss.syncConfig = &SyncConfig{}
+
 	var wg sync.WaitGroup
 	for _, peer := range peers {
 		wg.Add(1)
@@ -263,6 +274,34 @@ func (ss *StateSync) CreateSyncConfig(peers []p2p.Peer, isBeacon bool) error {
 	return nil
 }
 
+// limitNumPeers limits number of peers to release some server end sources.
+func limitNumPeers(ps []p2p.Peer, randSeed int64) []p2p.Peer {
+	targetSize := calcNumPeersWithBound(len(ps), numPeersLowBound, numPeersHighBound)
+	if len(ps) <= targetSize {
+		return ps
+	}
+
+	r := rand.New(rand.NewSource(randSeed))
+	r.Shuffle(len(ps), func(i, j int) { ps[i], ps[j] = ps[j], ps[i] })
+
+	return ps[:targetSize]
+}
+
+// Peers are expected to limited at half of the size, capped between lowBound and highBound.
+func calcNumPeersWithBound(size int, lowBound, highBound int) int {
+	if size < lowBound {
+		return size
+	}
+	expLen := size / 2
+	if expLen < lowBound {
+		expLen = lowBound
+	}
+	if expLen > highBound {
+		expLen = highBound
+	}
+	return expLen
+}
+
 // GetActivePeerNumber returns the number of active peers
 func (ss *StateSync) GetActivePeerNumber() int {
 	if ss.syncConfig == nil {

api/service/syncing/syncing_test.go

@@ -1,9 +1,12 @@
 package syncing
 
 import (
+	"fmt"
+	"reflect"
 	"testing"
 
 	"github.com/harmony-one/harmony/api/service/syncing/downloader"
+	"github.com/harmony-one/harmony/p2p"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -49,3 +52,69 @@ func TestCreateStateSync(t *testing.T) {
 		t.Error("Unable to create stateSync")
 	}
 }
+
+func TestLimitPeersWithBound(t *testing.T) {
+	tests := []struct {
+		size    int
+		expSize int
+	}{
+		{0, 0},
+		{1, 1},
+		{3, 3},
+		{4, 3},
+		{7, 3},
+		{8, 4},
+		{10, 5},
+		{11, 5},
+		{100, 5},
+	}
+	for _, test := range tests {
+		ps := makePeersForTest(test.size)
+
+		res := limitNumPeers(ps, 1)
+
+		if len(res) != test.expSize {
+			t.Errorf("result size unexpected: %v / %v", len(res), test.expSize)
+		}
+		if err := checkTestPeerDuplicity(res); err != nil {
+			t.Error(err)
+		}
+	}
+}
+
+func TestLimitPeersWithBound_random(t *testing.T) {
+	ps1 := makePeersForTest(100)
+	ps2 := makePeersForTest(100)
+	s1, s2 := int64(1), int64(2)
+
+	res1 := limitNumPeers(ps1, s1)
+	res2 := limitNumPeers(ps2, s2)
+	if reflect.DeepEqual(res1, res2) {
+		t.Fatal("not randomized limit peer")
+	}
+}
+
+func makePeersForTest(size int) []p2p.Peer {
+	ps := make([]p2p.Peer, 0, size)
+	for i := 0; i != size; i++ {
+		ps = append(ps, p2p.Peer{
+			IP: makeTestPeerIP(i),
+		})
+	}
+	return ps
+}
+
+func checkTestPeerDuplicity(ps []p2p.Peer) error {
+	m := make(map[string]struct{})
+	for _, p := range ps {
+		if _, ok := m[p.IP]; ok {
+			return fmt.Errorf("duplicate ip")
+		}
+		m[p.IP] = struct{}{}
+	}
+	return nil
+}
+
+func makeTestPeerIP(i interface{}) string {
+	return fmt.Sprintf("%v", i)
+}

cmd/harmony/main.go

@@ -72,6 +72,7 @@ var (
 	onlyLogTps  = flag.Bool("only_log_tps", false, "Only log TPS if true")
 	dnsZone     = flag.String("dns_zone", "", "if given and not empty, use peers from the zone (default: use libp2p peer discovery instead)")
 	dnsFlag     = flag.Bool("dns", true, "[deprecated] equivalent to -dns_zone t.hmny.io")
+	dnsPort     = flag.String("dns_port", "9000", "port of dns node")
 	//Leader needs to have a minimal number of peers to start consensus
 	minPeers = flag.Int("min_peers", 32, "Minimal number of Peers in shard")
 	// Key file to store the private key
@@ -110,8 +111,9 @@ var (
 	revertTo       = flag.Int("revert_to", 0, "The revert will rollback all blocks until and including block number revert_to")
 	revertBeacon   = flag.Bool("revert_beacon", false, "Whether to revert beacon chain or the chain this node is assigned to")
 	// Blacklist of addresses
-	blacklistPath   = flag.String("blacklist", "./.hmy/blacklist.txt", "Path to newline delimited file of blacklisted wallet addresses")
-	webHookYamlPath = flag.String(
+	blacklistPath      = flag.String("blacklist", "./.hmy/blacklist.txt", "Path to newline delimited file of blacklisted wallet addresses")
+	broadcastInvalidTx = flag.Bool("broadcast_invalid_tx", false, "Broadcast invalid transactions to sync pool state (default: false)")
+	webHookYamlPath    = flag.String(
 		"webhook_yaml", "", "path for yaml config reporting double signing",
 	)
 	// aws credentials
@@ -290,7 +292,7 @@ func readMultiBLSKeys(consensusMultiBLSPriKey *multibls.PrivateKey, consensusMul
 		os.Exit(100)
 	}
 
-	keyFiles := []os.FileInfo{}
+	var keyFiles []os.FileInfo
 	legacyBLSFile := true
 
 	if len(awsEncryptedBLSKeyFiles) > 0 {
@@ -460,6 +462,7 @@ func setupConsensusAndNode(nodeConfig *nodeconfig.ConfigType) *node.Node {
 	chainDBFactory := &shardchain.LDBFactory{RootDir: nodeConfig.DBDir}
 
 	currentNode := node.New(myHost, currentConsensus, chainDBFactory, blacklist, *isArchival)
+	currentNode.BroadcastInvalidTx = *broadcastInvalidTx
 
 	switch {
 	case *networkType == nodeconfig.Localnet:
@@ -474,9 +477,9 @@ func setupConsensusAndNode(nodeConfig *nodeconfig.ConfigType) *node.Node {
 		currentNode.SyncingPeerProvider = node.NewLocalSyncingPeerProvider(
 			6000, uint16(selfPort), epochConfig.NumShards(), uint32(epochConfig.NumNodesPerShard()))
 	case *dnsZone != "":
-		currentNode.SyncingPeerProvider = node.NewDNSSyncingPeerProvider(*dnsZone, syncing.GetSyncingPort(*port))
+		currentNode.SyncingPeerProvider = node.NewDNSSyncingPeerProvider(*dnsZone, syncing.GetSyncingPort(*dnsPort))
 	case *dnsFlag:
-		currentNode.SyncingPeerProvider = node.NewDNSSyncingPeerProvider("t.hmny.io", syncing.GetSyncingPort(*port))
+		currentNode.SyncingPeerProvider = node.NewDNSSyncingPeerProvider("t.hmny.io", syncing.GetSyncingPort(*dnsPort))
 	default:
 		currentNode.SyncingPeerProvider = node.NewLegacySyncingPeerProvider(currentNode)
 

consensus/consensus_service.go

@@ -418,7 +418,7 @@ func (consensus *Consensus) UpdateConsensusInformation() Mode {
 		consensus.Decider = decider
 	}
 
-	committeeToSet := &shard.Committee{}
+	var committeeToSet *shard.Committee
 	epochToSet := curEpoch
 	hasError := false
 	curShardState, err := committee.WithStakingEnabled.ReadFromDB(

consensus/leader.go

@@ -262,20 +262,19 @@ func (consensus *Consensus) onCommit(msg *msg_pb.Message) {
 		return
 	}
 
+	viewID := consensus.viewID
+
 	quorumIsMet := consensus.Decider.IsQuorumAchieved(quorum.Commit)
 	if !quorumWasMet && quorumIsMet {
 		logger.Info().Msg("[OnCommit] 2/3 Enough commits received")
-		go func(viewID uint64) {
-			consensus.getLogger().Info().Msg("[OnCommit] Starting Grace Period")
-			// Always wait for 2 seconds as minimum grace period
-			time.Sleep(2 * time.Second)
-			if n := time.Now(); n.Before(consensus.NextBlockDue) {
-				// Sleep to wait for the full block time
-				time.Sleep(consensus.NextBlockDue.Sub(n))
-			}
+
+		next := consensus.NextBlockDue
+		consensus.getLogger().Info().Msg("[OnCommit] Starting Grace Period")
+		time.AfterFunc(2*time.Second, func() {
+			<-time.After(time.Until(next))
 			logger.Info().Msg("[OnCommit] Commit Grace Period Ended")
 			consensus.commitFinishChan <- viewID
-		}(consensus.viewID)
+		})
 
 		consensus.msgSender.StopRetry(msg_pb.MessageType_PREPARED)
 	}

consensus/votepower/roster.go

@@ -143,13 +143,9 @@ func AggregateRosters(
 					AccommodateHarmonyVote: *voteCard,
 					ShardID:                roster.ShardID,
 				}
-				if _, ok := result[voteCard.EarningAccount]; ok {
-					result[voteCard.EarningAccount] = append(
-						result[voteCard.EarningAccount], voterID,
-					)
-				} else {
-					result[voteCard.EarningAccount] = []VoteOnSubcomittee{voterID}
-				}
+				result[voteCard.EarningAccount] = append(
+					result[voteCard.EarningAccount], voterID,
+				)
 			}
 		}
 	}

core/block_validator.go

@@ -36,7 +36,7 @@ import (
 // BlockValidator is responsible for validating block headers, uncles and
 // processed state.
 //
-// BlockValidator implements Validator.
+// BlockValidator implements validator.
 type BlockValidator struct {
 	config *params.ChainConfig     // Chain configuration options
 	bc     *BlockChain             // Canonical block chain

core/blockchain.go

@@ -102,9 +102,9 @@ type CacheConfig struct {
 // block. The Blockchain manages chain imports, reverts, chain reorganisations.
 //
 // Importing blocks in to the block chain happens according to the set of rules
-// defined by the two stage Validator. Processing of blocks is done using the
+// defined by the two stage validator. Processing of blocks is done using the
 // Processor which processes the included transaction. The validation of the state
-// is done in the second part of the Validator. Failing results in aborting of
+// is done in the second part of the validator. Failing results in aborting of
 // the import.
 //
 // The BlockChain also helps in returning blocks from **any** chain included
@@ -170,7 +170,7 @@ type BlockChain struct {
 }
 
 // NewBlockChain returns a fully initialised block chain using information
-// available in the database. It initialises the default Ethereum Validator and
+// available in the database. It initialises the default Ethereum validator and
 // Processor.
 func NewBlockChain(
 	db ethdb.Database, cacheConfig *CacheConfig, chainConfig *params.ChainConfig,

core/staking_verifier.go

@@ -111,9 +111,8 @@ func VerifyAndCreateValidatorFromMsg(
 	wrapper.Delegations = []staking.Delegation{
 		staking.NewDelegation(v.Address, msg.Amount),
 	}
-	zero := big.NewInt(0)
-	wrapper.Counters.NumBlocksSigned = zero
-	wrapper.Counters.NumBlocksToSign = zero
+	wrapper.Counters.NumBlocksSigned = big.NewInt(0)
+	wrapper.Counters.NumBlocksToSign = big.NewInt(0)
 	wrapper.BlockReward = big.NewInt(0)
 	maxBLSKeyAllowed := shard.ExternalSlotsAvailableForEpoch(epoch) / 3
 	if err := wrapper.SanityCheck(maxBLSKeyAllowed); err != nil {
@@ -167,7 +166,7 @@ func VerifyAndEditValidatorFromMsg(
 
 	snapshotValidator, err := chainContext.ReadValidatorSnapshot(wrapper.Address)
 	if err != nil {
-		return nil, errors.WithMessage(err, "Validator snapshot not found.")
+		return nil, errors.WithMessage(err, "validator snapshot not found.")
 	}
 	rateAtBeginningOfEpoch := snapshotValidator.Validator.Rate