diff --git a/README.md b/README.md index 466f8afdc..a6681e66d 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,8 @@ [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) [![Tests](https://github.com/lidofinance/lido-oracle/workflows/Tests/badge.svg?branch=daemon_v2)](https://github.com/lidofinance/lido-oracle/actions) -Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update reports to the Lido smart contracts. +Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update +reports to the Lido smart contracts. ## How it works @@ -15,7 +16,8 @@ There are 3 modules in the oracle: ### Accounting module -Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck validators and processes user withdrawal requests. +Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck +validators and processes user withdrawal requests. Also Accounting module makes decision to turn on/off the bunker. **Flow** @@ -37,7 +39,8 @@ The frame includes these stages: ### Ejector module -Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user withdrawals. +Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user +withdrawals. **Flow** @@ -58,6 +61,7 @@ Only Oracle: - Memory - 8 GB Oracle + KAPI: + - vCPU - 4 - Memory - 16 GB @@ -65,7 +69,8 @@ Oracle + KAPI: ### Execution Client Node -To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node. +To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on +historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node. Oracle needs two weeks of archived data. | Client | Tested | Notes | @@ -77,19 +82,21 @@ Oracle needs two weeks of archived data. ### Consensus Client Node -Also, to calculate some metrics for bunker mode Oracle needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node. +Also, to calculate some metrics for bunker mode Oracle +needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node. -| Client | Tested | Notes | -|---------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------| -| [Lighthouse](https://lighthouse.sigmaprime.io/) | 🟢 | Use `--reconstruct-historic-states` param | -| [Lodestar](https://lodestar.chainsafe.io) | 🔴 | Not tested yet | -| [Nimbus](https://nimbus.team) | 🔴 | Not tested yet | -| [Prysm](https://github.com/prysmaticlabs/prysm) | 🟢 | Use
`--grpc-max-msg-size=104857600`
`--enable-historical-state-representation=true`
`--slots-per-archive-point=1024`
params | -| [Teku](https://docs.teku.consensys.net) | 🟢 | Use
`--data-storage-mode=archive`
`--data-storage-archive-frequency=1024`
`--reconstruct-historic-states=true`
params | +| Client | Tested | Notes | +|-------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------| +| [Lighthouse](https://lighthouse.sigmaprime.io/) | 🟢 | Use `--reconstruct-historic-states` param | +| [Lodestar](https://lodestar.chainsafe.io) | 🔴 | Not tested yet | +| [Nimbus](https://nimbus.team) | 🔴 | Not tested yet | +| [Prysm](https://github.com/prysmaticlabs/prysm) | 🟢 | Use
`--grpc-max-msg-size=104857600`
`--enable-historical-state-representation=true`
`--slots-per-archive-point=1024`
params | +| [Teku](https://docs.teku.consensys.net) | 🟢 | Use
`--data-storage-mode=archive`
`--data-storage-archive-frequency=1024`
`--reconstruct-historic-states=true`
params | ### Keys API Service -This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in database. +This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in +database. [Lido Keys API repository.](https://github.com/lidofinance/lido-keys-api) @@ -102,9 +109,11 @@ Pull the image using the following command: docker pull lidofinance/oracle:{tag} ``` -Where `{tag}` is a version of the image. You can find the latest version in the [releases](https://github.com/lidofinance/lido-oracle/releases) +Where `{tag}` is a version of the image. You can find the latest version in +the [releases](https://github.com/lidofinance/lido-oracle/releases) **OR**\ -You can build it locally using the following command (make sure build it from latest [release](https://github.com/lidofinance/lido-oracle/releases)): +You can build it locally using the following command (make sure build it from +latest [release](https://github.com/lidofinance/lido-oracle/releases)): ```bash docker build -t lidofinance/oracle . @@ -124,16 +133,17 @@ Full variables list could be found [here](https://github.com/lidofinance/lido-or and your environment is ready to run the oracle. ## Run the oracle + 1. By default, the oracle runs in *dry mode*. It means that it will not send any transactions to the Ethereum network. - To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable: + To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable: ``` MEMBER_PRIV_KEY={value} ``` - Where `{value}` is a private key of the Oracle member account or: + Where `{value}` is a private key of the Oracle member account or: ``` MEMBER_PRIV_KEY_FILE={path} ``` - Where `{path}` is a path to the private key of the Oracle member account. + Where `{path}` is a path to the private key of the Oracle member account. 2. Run the container using the following command: ```bash @@ -203,6 +213,7 @@ In manual mode all sleeps are disabled and `ALLOW_REPORTING_IN_BUNKER_MODE` is T | `CACHE_PATH` | Directory to store cache for CSM module | False | `.` | ### Mainnet variables + > LIDO_LOCATOR_ADDRESS=0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb > ALLOW_REPORTING_IN_BUNKER_MODE=False @@ -281,9 +292,12 @@ Special metrics for ejector oracle: Special metrics for CSM oracle: -| Metric name | Description | Labels | -|-----------------------------------|---------------------------------------------|--------| -| TBD | TBD | | +| Metric name | Description | Labels | +|---------------------------------|----------------------------------------|--------| +| csm_current_frame_range_l_epoch | Left epoch of the current frame range | | +| csm_current_frame_range_r_epoch | Right epoch of the current frame range | | +| csm_unprocessed_epochs_count | Unprocessed epochs count | | +| csm_min_unprocessed_epoch | Minimum unprocessed epoch | | # Development diff --git a/src/constants.py b/src/constants.py index e3dc6b67d..0d4afe7f1 100644 --- a/src/constants.py +++ b/src/constants.py @@ -1,5 +1,7 @@ +from src.types import Gwei + # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#misc -FAR_FUTURE_EPOCH = 2 ** 64 - 1 +FAR_FUTURE_EPOCH = 2**64 - 1 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#time-parameters-1 MIN_VALIDATOR_WITHDRAWABILITY_DELAY = 2**8 SHARD_COMMITTEE_PERIOD = 256 @@ -9,23 +11,34 @@ # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#rewards-and-penalties PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX = 3 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#gwei-values -EFFECTIVE_BALANCE_INCREMENT = 2 ** 0 * 10 ** 9 -MAX_EFFECTIVE_BALANCE = 32 * 10 ** 9 +EFFECTIVE_BALANCE_INCREMENT = 2**0 * 10**9 +MAX_EFFECTIVE_BALANCE = Gwei(32 * 10**9) # https://github.com/ethereum/consensus-specs/blob/dev/specs/capella/beacon-chain.md#execution -MAX_WITHDRAWALS_PER_PAYLOAD = 2 ** 4 +MAX_WITHDRAWALS_PER_PAYLOAD = 2**4 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#withdrawal-prefixes ETH1_ADDRESS_WITHDRAWAL_PREFIX = '0x01' +# https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#withdrawal-prefixes +COMPOUNDING_WITHDRAWAL_PREFIX = '0x02' # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#validator-cycle -MIN_PER_EPOCH_CHURN_LIMIT = 2 ** 2 -CHURN_LIMIT_QUOTIENT = 2 ** 16 +MIN_PER_EPOCH_CHURN_LIMIT = 2**2 +CHURN_LIMIT_QUOTIENT = 2**16 +# https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#validator-cycle +MIN_PER_EPOCH_CHURN_LIMIT_ELECTRA = Gwei(2**7 * 10**9) +MAX_PER_EPOCH_ACTIVATION_EXIT_CHURN_LIMIT = Gwei(2**8 * 10**9) # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#time-parameters SLOTS_PER_HISTORICAL_ROOT = 8192 +# https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#gwei-values +MIN_ACTIVATION_BALANCE = Gwei(2**5 * 10**9) +MAX_EFFECTIVE_BALANCE_ELECTRA = Gwei(2**11 * 10**9) + +LIDO_DEPOSIT_AMOUNT = MIN_ACTIVATION_BALANCE + # Local constants -GWEI_TO_WEI = 10 ** 9 -SHARE_RATE_PRECISION_E27 = 10 ** 27 +GWEI_TO_WEI = 10**9 +SHARE_RATE_PRECISION_E27 = 10**27 TOTAL_BASIS_POINTS = 10000 MAX_BLOCK_GAS_LIMIT = 30_000_000 -UINT64_MAX = 2 ** 64 - 1 +UINT64_MAX = 2**64 - 1 diff --git a/src/main.py b/src/main.py index 34028ce03..b6b2a7c7d 100644 --- a/src/main.py +++ b/src/main.py @@ -24,14 +24,13 @@ KeysAPIClientModule, LidoValidatorsProvider, FallbackProviderModule, - LazyCSM + LazyCSM, ) from src.web3py.middleware import metrics_collector from src.web3py.types import Web3 from src.web3py.contract_tweak import tweak_w3_contracts - logger = logging.getLogger(__name__) @@ -42,22 +41,10 @@ def main(module_name: OracleModule): 'variables': { **build_info, 'module': module_name, - 'ACCOUNT': variables.ACCOUNT.address if variables.ACCOUNT else 'Dry', - 'LIDO_LOCATOR_ADDRESS': variables.LIDO_LOCATOR_ADDRESS, - 'CSM_MODULE_ADDRESS': variables.CSM_MODULE_ADDRESS, - 'FINALIZATION_BATCH_MAX_REQUEST_COUNT': variables.FINALIZATION_BATCH_MAX_REQUEST_COUNT, - 'EL_REQUESTS_BATCH_SIZE': variables.EL_REQUESTS_BATCH_SIZE, - 'MAX_CYCLE_LIFETIME_IN_SECONDS': variables.MAX_CYCLE_LIFETIME_IN_SECONDS, + **variables.PUBLIC_ENV_VARS, }, }) - ENV_VARIABLES_INFO.info({ - "ACCOUNT": str(variables.ACCOUNT.address) if variables.ACCOUNT else 'Dry', - "LIDO_LOCATOR_ADDRESS": str(variables.LIDO_LOCATOR_ADDRESS), - "CSM_MODULE_ADDRESS": str(variables.CSM_MODULE_ADDRESS), - "FINALIZATION_BATCH_MAX_REQUEST_COUNT": str(variables.FINALIZATION_BATCH_MAX_REQUEST_COUNT), - "EL_REQUESTS_BATCH_SIZE": str(variables.EL_REQUESTS_BATCH_SIZE), - "MAX_CYCLE_LIFETIME_IN_SECONDS": str(variables.MAX_CYCLE_LIFETIME_IN_SECONDS), - }) + ENV_VARIABLES_INFO.info(variables.PUBLIC_ENV_VARS) BUILD_INFO.info(build_info) logger.info({'msg': f'Start healthcheck server for Docker container on port {variables.HEALTHCHECK_SERVER_PORT}'}) diff --git a/src/modules/accounting/accounting.py b/src/modules/accounting/accounting.py index 6369d9409..e897b7abf 100644 --- a/src/modules/accounting/accounting.py +++ b/src/modules/accounting/accounting.py @@ -39,7 +39,6 @@ from src.services.bunker import BunkerService from src.types import BlockStamp, Gwei, ReferenceBlockStamp, StakingModuleId, NodeOperatorGlobalIndex, FinalizationBatches from src.utils.cache import global_lru_cache as lru_cache -from src.utils.exception import IncompatibleException from src.variables import ALLOW_REPORTING_IN_BUNKER_MODE from src.web3py.types import Web3 from src.web3py.extensions.lido_validators import StakingModule @@ -58,7 +57,7 @@ class Accounting(BaseModule, ConsensusModule): - Send extra data Contains stuck and exited validators count by each node operator. """ - COMPATIBLE_ONCHAIN_VERSIONS = [(1, 1), (2, 2)] + COMPATIBLE_ONCHAIN_VERSIONS = [(1, 1), (2, 2), (2, 3)] def __init__(self, w3: Web3): self.report_contract: AccountingOracleContract = w3.lido_contracts.accounting_oracle @@ -174,10 +173,8 @@ def _calculate_report(self, blockstamp: ReferenceBlockStamp): # or in the `execute_module` method if consensus_version == 1: report_data = self._calculate_report_v1(blockstamp) - elif consensus_version == 2: - report_data = self._calculate_report_v2(blockstamp) else: - raise IncompatibleException("Consensus version is not supported") + report_data = self._calculate_report_latest_version(consensus_version, blockstamp) self._update_metrics(report_data) return report_data @@ -216,11 +213,14 @@ def get_updated_modules_stats( def _get_consensus_lido_state(self, blockstamp: ReferenceBlockStamp) -> tuple[ValidatorsCount, ValidatorsBalance]: lido_validators = self.w3.lido_validators.get_lido_validators(blockstamp) - count = len(lido_validators) - total_balance = Gwei(sum(int(validator.balance) for validator in lido_validators)) + validators_count = len(lido_validators) + active_balance = sum(int(validator.balance) for validator in lido_validators) + pending_deposits = self.w3.lido_validators.calculate_pending_deposits_sum(lido_validators) + total_balance = Gwei(active_balance + pending_deposits) - logger.info({'msg': 'Calculate lido state on CL. (Validators count, Total balance in gwei)', 'value': (count, total_balance)}) - return ValidatorsCount(count), ValidatorsBalance(total_balance) + logger.info( + {'msg': f'Calculate Lido state on CL. {validators_count=}, {active_balance=}, {pending_deposits=}, {total_balance=} (Gwei)'}) + return ValidatorsCount(validators_count), ValidatorsBalance(total_balance) def _get_finalization_data(self, blockstamp: ReferenceBlockStamp) -> tuple[FinalizationShareRate, FinalizationBatches]: simulation = self.simulate_full_rebase(blockstamp) @@ -228,7 +228,11 @@ def _get_finalization_data(self, blockstamp: ReferenceBlockStamp) -> tuple[Final frame_config = self.get_frame_config(blockstamp) is_bunker = self._is_bunker(blockstamp) - share_rate = simulation.post_total_pooled_ether * SHARE_RATE_PRECISION_E27 // simulation.post_total_shares + share_rate = ( + simulation.post_total_pooled_ether * SHARE_RATE_PRECISION_E27 // simulation.post_total_shares + if simulation.post_total_shares + else 0 + ) logger.info({'msg': 'Calculate shares rate.', 'value': share_rate}) withdrawal_service = Withdrawal(self.w3, blockstamp, chain_config, frame_config) @@ -366,13 +370,13 @@ def _calculate_report_v1(self, blockstamp: ReferenceBlockStamp) -> ReportData: extra_data_part_v1 = self._calculate_extra_data_report_v1(blockstamp) return self._combine_report_parts(1, blockstamp, rebase_part, modules_part, wq_part, extra_data_part_v1) - def _calculate_report_v2(self, blockstamp: ReferenceBlockStamp) -> ReportData: + def _calculate_report_latest_version(self, consensus_version: int, blockstamp: ReferenceBlockStamp) -> ReportData: rebase_part = self._calculate_rebase_report(blockstamp) modules_part = self._get_newly_exited_validators_by_modules(blockstamp) wq_part = self._calculate_wq_report(blockstamp) - extra_data_part_v2 = self._calculate_extra_data_report_v2(blockstamp) - return self._combine_report_parts(2, blockstamp, rebase_part, modules_part, wq_part, extra_data_part_v2) + extra_data_part = self._calculate_extra_data_report(blockstamp) + return self._combine_report_parts(consensus_version, blockstamp, rebase_part, modules_part, wq_part, extra_data_part) # fetches validators_count, cl_balance, withdrawal_balance, el_vault_balance, shares_to_burn def _calculate_rebase_report(self, blockstamp: ReferenceBlockStamp) -> RebaseReport: @@ -397,7 +401,7 @@ def _calculate_extra_data_report_v1(self, blockstamp: ReferenceBlockStamp) -> Ex orl.max_node_operators_per_extra_data_item, ) - def _calculate_extra_data_report_v2(self, blockstamp: ReferenceBlockStamp) -> ExtraData: + def _calculate_extra_data_report(self, blockstamp: ReferenceBlockStamp) -> ExtraData: stuck_validators, exited_validators, orl = self._get_generic_extra_data(blockstamp) return ExtraDataServiceV2.collect( stuck_validators, diff --git a/src/modules/csm/checkpoint.py b/src/modules/csm/checkpoint.py index ba6f94a2c..410d09736 100644 --- a/src/modules/csm/checkpoint.py +++ b/src/modules/csm/checkpoint.py @@ -3,25 +3,25 @@ from dataclasses import dataclass from itertools import batched from threading import Lock -from typing import Iterable, Sequence +from typing import Iterable, Sequence, TypeGuard from src import variables from src.constants import SLOTS_PER_HISTORICAL_ROOT -from src.metrics.prometheus.csm import CSM_UNPROCESSED_EPOCHS_COUNT, CSM_MIN_UNPROCESSED_EPOCH +from src.metrics.prometheus.csm import CSM_MIN_UNPROCESSED_EPOCH, CSM_UNPROCESSED_EPOCHS_COUNT from src.modules.csm.state import State from src.providers.consensus.client import ConsensusClient -from src.providers.consensus.types import BlockAttestation +from src.providers.consensus.types import BlockAttestation, BlockAttestationEIP7549 from src.types import BlockRoot, BlockStamp, EpochNumber, SlotNumber, ValidatorIndex from src.utils.range import sequence from src.utils.timeit import timeit +from src.utils.types import hex_str_to_bytes from src.utils.web3converter import Web3Converter logger = logging.getLogger(__name__) lock = Lock() -class MinStepIsNotReached(Exception): - ... +class MinStepIsNotReached(Exception): ... @dataclass @@ -103,7 +103,9 @@ def _is_min_step_reached(self): return False -type Committees = dict[tuple[str, str], list[ValidatorDuty]] +type Slot = str +type CommitteeIndex = str +type Committees = dict[tuple[Slot, CommitteeIndex], list[ValidatorDuty]] class FrameCheckpointProcessor: @@ -229,18 +231,49 @@ def _prepare_committees(self, epoch: EpochNumber) -> Committees: def process_attestations(attestations: Iterable[BlockAttestation], committees: Committees) -> None: for attestation in attestations: - committee_id = (attestation.data.slot, attestation.data.index) - committee = committees.get(committee_id, []) - att_bits = _to_bits(attestation.aggregation_bits) - for index_in_committee, validator_duty in enumerate(committee): - validator_duty.included = validator_duty.included or _is_attested(att_bits, index_in_committee) + committee_offset = 0 + for committee_idx in get_committee_indices(attestation): + committee = committees.get((attestation.data.slot, committee_idx), []) + att_bits = hex_bitlist_to_list(attestation.aggregation_bits)[committee_offset:][: len(committee)] + for index_in_committee in get_set_indices(att_bits): + committee[index_in_committee].included = True + committee_offset += len(committee) -def _is_attested(bits: Sequence[bool], index: int) -> bool: - return bits[index] +def get_committee_indices(attestation: BlockAttestation) -> list[CommitteeIndex]: + if is_eip7549_attestation(attestation): + return [str(i) for i in get_set_indices(hex_bitvector_to_list(attestation.committee_bits))] + return [attestation.data.index] -def _to_bits(aggregation_bits: str) -> Sequence[bool]: +def is_eip7549_attestation(attestation: BlockAttestation) -> TypeGuard[BlockAttestationEIP7549]: + # @see https://eips.ethereum.org/EIPS/eip-7549 + has_committee_bits = getattr(attestation, "committee_bits") is not None + has_zero_index = attestation.data.index == "0" + if has_committee_bits and not has_zero_index: + raise ValueError(f"Got invalid {attestation=}") + return has_committee_bits and has_zero_index + + +def get_set_indices(bits: Sequence[bool]) -> list[int]: + """Returns indices of truthy values in the supplied sequence""" + return [i for (i, bit) in enumerate(bits) if bit] + + +def hex_bitvector_to_list(bitvector: str) -> list[bool]: + bytes_ = hex_str_to_bytes(bitvector) + return _bytes_to_bool_list(bytes_) + + +def hex_bitlist_to_list(bitlist: str) -> list[bool]: + bytes_ = hex_str_to_bytes(bitlist) + if not bytes_ or bytes_[-1] == 0: + raise ValueError(f"Got invalid {bitlist=}") + bitlist_len = int.from_bytes(bytes_, "little").bit_length() - 1 + return _bytes_to_bool_list(bytes_, count=bitlist_len) + + +def _bytes_to_bool_list(bytes_: bytes, count: int | None = None) -> list[bool]: + count = count if count is not None else len(bytes_) * 8 # copied from https://github.com/ethereum/py-ssz/blob/main/ssz/sedes/bitvector.py#L66 - att_bytes = bytes.fromhex(aggregation_bits[2:]) - return [bool((att_bytes[bit_index // 8] >> bit_index % 8) % 2) for bit_index in range(len(att_bytes) * 8)] + return [bool((bytes_[bit_index // 8] >> bit_index % 8) % 2) for bit_index in range(count)] diff --git a/src/modules/csm/csm.py b/src/modules/csm/csm.py index ff7e52d43..d989d91ff 100644 --- a/src/modules/csm/csm.py +++ b/src/modules/csm/csm.py @@ -60,7 +60,7 @@ class CSOracle(BaseModule, ConsensusModule): 3. Calculate the share of each CSM node operator excluding underperforming validators. """ - COMPATIBLE_ONCHAIN_VERSIONS = [(1, 1)] + COMPATIBLE_ONCHAIN_VERSIONS = [(1, 1), (1, 2)] report_contract: CSFeeOracleContract module_id: StakingModuleId diff --git a/src/modules/ejector/ejector.py b/src/modules/ejector/ejector.py index db2e7fe81..80ddf5da1 100644 --- a/src/modules/ejector/ejector.py +++ b/src/modules/ejector/ejector.py @@ -1,25 +1,24 @@ import logging -from functools import reduce +import math -from more_itertools import ilen from web3.exceptions import ContractCustomError from web3.types import Wei from src.constants import ( + EFFECTIVE_BALANCE_INCREMENT, FAR_FUTURE_EPOCH, - MAX_EFFECTIVE_BALANCE, MAX_WITHDRAWALS_PER_PAYLOAD, MIN_VALIDATOR_WITHDRAWABILITY_DELAY, ) from src.metrics.prometheus.business import CONTRACT_ON_PAUSE +from src.metrics.prometheus.duration_meter import duration_meter from src.metrics.prometheus.ejector import ( - EJECTOR_VALIDATORS_COUNT_TO_EJECT, - EJECTOR_TO_WITHDRAW_WEI_AMOUNT, EJECTOR_MAX_WITHDRAWAL_EPOCH, + EJECTOR_TO_WITHDRAW_WEI_AMOUNT, + EJECTOR_VALIDATORS_COUNT_TO_EJECT, ) -from src.metrics.prometheus.duration_meter import duration_meter from src.modules.ejector.data_encode import encode_data -from src.modules.ejector.types import ReportData, EjectorProcessingState +from src.modules.ejector.types import EjectorProcessingState, ReportData from src.modules.submodules.consensus import ConsensusModule, InitialEpochIsYetToArriveRevert from src.modules.submodules.oracle_module import BaseModule, ModuleExecuteDelay from src.modules.submodules.types import ZERO_HASH @@ -29,19 +28,20 @@ from src.services.exit_order_v2.iterator import ValidatorExitIteratorV2 from src.services.prediction import RewardsPredictionService from src.services.validator_state import LidoValidatorStateService -from src.types import BlockStamp, EpochNumber, ReferenceBlockStamp, NodeOperatorGlobalIndex +from src.types import BlockStamp, EpochNumber, Gwei, NodeOperatorGlobalIndex, ReferenceBlockStamp from src.utils.cache import global_lru_cache as lru_cache from src.utils.validator_state import ( + compute_activation_exit_epoch, + get_activation_exit_churn_limit, + get_validator_churn_limit, + get_max_effective_balance, is_active_validator, is_fully_withdrawable_validator, is_partially_withdrawable_validator, - compute_activation_exit_epoch, - compute_exit_churn_limit, ) from src.web3py.extensions.lido_validators import LidoValidator from src.web3py.types import Web3 - logger = logging.getLogger(__name__) @@ -62,7 +62,8 @@ class Ejector(BaseModule, ConsensusModule): 3. Decode lido validators into bytes and send report transaction """ - COMPATIBLE_ONCHAIN_VERSIONS = [(1, 1), (1, 2)] + + COMPATIBLE_ONCHAIN_VERSIONS = [(1, 1), (1, 2), (1, 3)] AVG_EXPECTING_WITHDRAWALS_SWEEP_DURATION_MULTIPLIER = 0.5 @@ -75,7 +76,7 @@ def __init__(self, w3: Web3): self.validators_state_service = LidoValidatorStateService(w3) def refresh_contracts(self): - self.report_contract = self.w3.lido_contracts.validators_exit_bus_oracle + self.report_contract = self.w3.lido_contracts.validators_exit_bus_oracle # type: ignore def execute_module(self, last_finalized_blockstamp: BlockStamp) -> ModuleExecuteDelay: report_blockstamp = self.get_blockstamp_for_report(last_finalized_blockstamp) @@ -117,9 +118,9 @@ def get_validators_to_eject(self, blockstamp: ReferenceBlockStamp) -> list[tuple EJECTOR_TO_WITHDRAW_WEI_AMOUNT.set(to_withdraw_amount) logger.info({'msg': 'Calculate to withdraw amount.', 'value': to_withdraw_amount}) - expected_balance = self._get_total_expected_balance(0, blockstamp) + expected_balance = self._get_total_expected_balance([], blockstamp) - consensus_version = self.w3.lido_contracts.validators_exit_bus_oracle.get_consensus_version(blockstamp.block_hash) + consensus_version = self.get_consensus_version(blockstamp) validators_iterator = iter(self.get_validators_iterator(consensus_version, blockstamp)) validators_to_eject: list[tuple[NodeOperatorGlobalIndex, LidoValidator]] = [] @@ -129,8 +130,11 @@ def get_validators_to_eject(self, blockstamp: ReferenceBlockStamp) -> list[tuple while expected_balance < to_withdraw_amount: gid, next_validator = next(validators_iterator) validators_to_eject.append((gid, next_validator)) - validator_to_eject_balance_sum += self._get_predicted_withdrawable_balance(next_validator) - expected_balance = self._get_total_expected_balance(len(validators_to_eject), blockstamp) + validator_to_eject_balance_sum + validator_to_eject_balance_sum += self.w3.to_wei(self._get_predicted_withdrawable_balance(next_validator), "gwei") + expected_balance = ( + self._get_total_expected_balance([v for (_, v) in validators_to_eject], blockstamp) + + validator_to_eject_balance_sum + ) except StopIteration: pass @@ -141,7 +145,7 @@ def get_validators_to_eject(self, blockstamp: ReferenceBlockStamp) -> list[tuple 'validators_to_eject_count': len(validators_to_eject), }) - if consensus_version != 1: + if self.get_consensus_version(blockstamp) != 1: forced_validators = validators_iterator.get_remaining_forced_validators() if forced_validators: logger.info({'msg': 'Eject forced to exit validators.', 'len': len(forced_validators)}) @@ -149,7 +153,7 @@ def get_validators_to_eject(self, blockstamp: ReferenceBlockStamp) -> list[tuple return validators_to_eject - def _get_total_expected_balance(self, vals_to_exit: int, blockstamp: ReferenceBlockStamp): + def _get_total_expected_balance(self, vals_to_exit: list[Validator], blockstamp: ReferenceBlockStamp): chain_config = self.get_chain_config(blockstamp) validators_going_to_exit = self.validators_state_service.get_recently_requested_but_not_exited_validators(blockstamp, chain_config) @@ -165,7 +169,7 @@ def _get_total_expected_balance(self, vals_to_exit: int, blockstamp: ReferenceBl rewards_speed_per_epoch = self.prediction_service.get_rewards_per_epoch(blockstamp, chain_config) logger.info({'msg': 'Calculate average rewards speed per epoch.', 'value': rewards_speed_per_epoch}) - withdrawal_epoch = self._get_predicted_withdrawable_epoch(blockstamp, len(validators_going_to_exit) + vals_to_exit + 1) + withdrawal_epoch = self._get_predicted_withdrawable_epoch(blockstamp, validators_going_to_exit + vals_to_exit) logger.info({'msg': 'Withdrawal epoch', 'value': withdrawal_epoch}) EJECTOR_MAX_WITHDRAWAL_EPOCH.set(withdrawal_epoch) @@ -203,23 +207,16 @@ def is_reporting_allowed(self, blockstamp: ReferenceBlockStamp) -> bool: @lru_cache(maxsize=1) def _get_withdrawable_lido_validators_balance(self, on_epoch: EpochNumber, blockstamp: BlockStamp) -> Wei: lido_validators = self.w3.lido_validators.get_lido_validators(blockstamp=blockstamp) - - def get_total_withdrawable_balance(balance: Wei, validator: Validator) -> Wei: - if is_fully_withdrawable_validator(validator, on_epoch): - return Wei(balance + self._get_predicted_withdrawable_balance(validator)) - - return balance - - result = reduce( - get_total_withdrawable_balance, - lido_validators, - Wei(0), + return Wei( + sum( + self._get_predicted_withdrawable_balance(v) + for v in lido_validators + if is_fully_withdrawable_validator(v, on_epoch) + ) ) - return result - - def _get_predicted_withdrawable_balance(self, validator: Validator) -> Wei: - return self.w3.to_wei(min(int(validator.balance), MAX_EFFECTIVE_BALANCE), 'gwei') + def _get_predicted_withdrawable_balance(self, validator: Validator) -> Gwei: + return Gwei(min(int(validator.balance), get_max_effective_balance(validator))) @lru_cache(maxsize=1) def _get_total_el_balance(self, blockstamp: BlockStamp) -> Wei: @@ -232,11 +229,23 @@ def _get_total_el_balance(self, blockstamp: BlockStamp) -> Wei: def _get_predicted_withdrawable_epoch( self, blockstamp: ReferenceBlockStamp, - validators_to_eject_count: int, + validators_to_eject: list[Validator], ) -> EpochNumber: """ Returns epoch when all validators in queue and validators_to_eject will be withdrawn. """ + spec = self.w3.cc.get_config_spec() + + if blockstamp.ref_epoch < int(spec.ELECTRA_FORK_EPOCH): + return self._get_predicted_withdrawable_epoch_pre_electra(blockstamp, validators_to_eject) + + return self._get_predicted_withdrawable_epoch_post_electra(blockstamp, validators_to_eject) + + def _get_predicted_withdrawable_epoch_pre_electra( + self, + blockstamp: ReferenceBlockStamp, + validators_to_eject: list[Validator], + ) -> EpochNumber: max_exit_epoch_number, latest_to_exit_validators_count = self._get_latest_exit_epoch(blockstamp) activation_exit_epoch = compute_activation_exit_epoch(blockstamp.ref_epoch) @@ -247,10 +256,32 @@ def _get_predicted_withdrawable_epoch( churn_limit = self._get_churn_limit(blockstamp) - epochs_required_to_exit_validators = (validators_to_eject_count + latest_to_exit_validators_count) // churn_limit + epochs_required_to_exit_validators = (len(validators_to_eject) + 1 + latest_to_exit_validators_count) // churn_limit return EpochNumber(max_exit_epoch_number + epochs_required_to_exit_validators + MIN_VALIDATOR_WITHDRAWABILITY_DELAY) + def _get_predicted_withdrawable_epoch_post_electra( + self, + blockstamp: ReferenceBlockStamp, + validators_to_eject: list[Validator], + ) -> EpochNumber: + per_epoch_churn = get_activation_exit_churn_limit(self._get_total_active_balance(blockstamp)) + activation_exit_epoch = compute_activation_exit_epoch(blockstamp.ref_epoch) + state_view = self.w3.cc.get_state_view(blockstamp.state_root) + + if state_view.earliest_exit_epoch < activation_exit_epoch: + earliest_exit_epoch = activation_exit_epoch + exit_balance_to_consume = per_epoch_churn + else: + earliest_exit_epoch = state_view.earliest_exit_epoch + exit_balance_to_consume = state_view.exit_balance_to_consume + + exit_balance = sum(self._get_predicted_withdrawable_balance(v) for v in validators_to_eject) + balance_to_process = max(0, exit_balance - exit_balance_to_consume) + additional_epochs = math.ceil(balance_to_process / per_epoch_churn) + + return EpochNumber(earliest_exit_epoch + additional_epochs + MIN_VALIDATOR_WITHDRAWABILITY_DELAY) + @lru_cache(maxsize=1) def _get_latest_exit_epoch(self, blockstamp: ReferenceBlockStamp) -> tuple[EpochNumber, int]: """ @@ -285,36 +316,56 @@ def _get_latest_exit_epoch(self, blockstamp: ReferenceBlockStamp) -> tuple[Epoch @lru_cache(maxsize=1) def _get_sweep_delay_in_epochs(self, blockstamp: ReferenceBlockStamp) -> int: """Returns amount of epochs that will take to sweep all validators in chain.""" + + if self.get_consensus_version(blockstamp) in (1, 2): + return self._get_sweep_delay_in_epochs_pre_pectra(blockstamp) + return self._get_sweep_delay_in_epochs_post_pectra(blockstamp) + + def _get_sweep_delay_in_epochs_pre_pectra(self, blockstamp: ReferenceBlockStamp) -> int: chain_config = self.get_chain_config(blockstamp) - total_withdrawable_validators = self._get_total_withdrawable_validators(blockstamp) + + total_withdrawable_validators = len(self._get_withdrawable_validators(blockstamp)) + logger.info({'msg': 'Calculate total withdrawable validators.', 'value': total_withdrawable_validators}) full_sweep_in_epochs = total_withdrawable_validators / MAX_WITHDRAWALS_PER_PAYLOAD / chain_config.slots_per_epoch return int(full_sweep_in_epochs * self.AVG_EXPECTING_WITHDRAWALS_SWEEP_DURATION_MULTIPLIER) - def _get_total_withdrawable_validators(self, blockstamp: ReferenceBlockStamp) -> int: - total_withdrawable_validators = ilen(filter(lambda validator: ( - is_partially_withdrawable_validator(validator) or - is_fully_withdrawable_validator(validator, blockstamp.ref_epoch) - ), self.w3.cc.get_validators(blockstamp))) - + def _get_sweep_delay_in_epochs_post_pectra(self, blockstamp: ReferenceBlockStamp) -> int: + # This version is intended for use with Pectra, but we do not currently take into account pending withdrawal + # requests. It would require a large amount of pending withdrawal requests to significantly impact sweep + # duration. Roughly every 512 requests adds one more epoch to sweep duration in the current state. + # On the other side, to consider pending withdrawals it is necessary to fetch the beacon state and query the + # EIP-7002 predeployed contract, which adds complexity with limited improvement for predictions. + chain_config = self.get_chain_config(blockstamp) + total_withdrawable_validators = len(self._get_withdrawable_validators(blockstamp)) logger.info({'msg': 'Calculate total withdrawable validators.', 'value': total_withdrawable_validators}) - return total_withdrawable_validators + slots_to_sweep = math.ceil(total_withdrawable_validators / MAX_WITHDRAWALS_PER_PAYLOAD) + full_sweep_in_epochs = math.ceil(slots_to_sweep / chain_config.slots_per_epoch) + return math.ceil(full_sweep_in_epochs * self.AVG_EXPECTING_WITHDRAWALS_SWEEP_DURATION_MULTIPLIER) + + def _get_withdrawable_validators(self, blockstamp: ReferenceBlockStamp) -> list[Validator]: + return [ + v + for v in self.w3.cc.get_validators(blockstamp) + if is_partially_withdrawable_validator(v) or is_fully_withdrawable_validator(v, blockstamp.ref_epoch) + ] @lru_cache(maxsize=1) def _get_churn_limit(self, blockstamp: ReferenceBlockStamp) -> int: - total_active_validators = self._get_total_active_validators(blockstamp) - churn_limit = compute_exit_churn_limit(total_active_validators) + total_active_validators = len(self._get_active_validators(blockstamp)) + logger.info({'msg': 'Calculate total active validators.', 'value': total_active_validators}) + churn_limit = get_validator_churn_limit(total_active_validators) logger.info({'msg': 'Calculate churn limit.', 'value': churn_limit}) return churn_limit - def _get_total_active_validators(self, blockstamp: ReferenceBlockStamp) -> int: - total_active_validators = reduce( - lambda total, validator: total + int(is_active_validator(validator, blockstamp.ref_epoch)), - self.w3.cc.get_validators(blockstamp), - 0, - ) - logger.info({'msg': 'Calculate total active validators.', 'value': total_active_validators}) - return total_active_validators + # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#get_total_active_balance + def _get_total_active_balance(self, blockstamp: ReferenceBlockStamp) -> Gwei: + active_validators = self._get_active_validators(blockstamp) + return Gwei(max(EFFECTIVE_BALANCE_INCREMENT, sum(int(v.validator.effective_balance) for v in active_validators))) + + @lru_cache(maxsize=1) + def _get_active_validators(self, blockstamp: ReferenceBlockStamp) -> list[Validator]: + return [v for v in self.w3.cc.get_validators(blockstamp) if is_active_validator(v, blockstamp.ref_epoch)] def is_main_data_submitted(self, blockstamp: BlockStamp) -> bool: processing_state = self._get_processing_state(blockstamp) diff --git a/src/modules/submodules/consensus.py b/src/modules/submodules/consensus.py index 00ed876bb..29807c1ad 100644 --- a/src/modules/submodules/consensus.py +++ b/src/modules/submodules/consensus.py @@ -87,6 +87,10 @@ def _get_consensus_contract_members(self, blockstamp: BlockStamp): consensus_contract = self._get_consensus_contract(blockstamp) return consensus_contract.get_members(blockstamp.block_hash) + @lru_cache(maxsize=1) + def get_consensus_version(self, blockstamp: BlockStamp): + return self.report_contract.get_consensus_version(blockstamp.block_hash) + @lru_cache(maxsize=1) def get_chain_config(self, blockstamp: BlockStamp) -> ChainConfig: consensus_contract = self._get_consensus_contract(blockstamp) diff --git a/src/modules/submodules/oracle_module.py b/src/modules/submodules/oracle_module.py index 6fb22b900..c6cab4782 100644 --- a/src/modules/submodules/oracle_module.py +++ b/src/modules/submodules/oracle_module.py @@ -25,7 +25,6 @@ from src import variables from src.types import SlotNumber, BlockStamp, BlockRoot - logger = logging.getLogger(__name__) @@ -57,42 +56,30 @@ def run_as_daemon(self): logger.debug({'msg': 'Startup new cycle.'}) self.cycle_handler() - @timeout(variables.MAX_CYCLE_LIFETIME_IN_SECONDS) def cycle_handler(self): - blockstamp = self._receive_last_finalized_slot() - - if blockstamp.slot_number > self._slot_threshold: - if self.w3.lido_contracts.has_contract_address_changed(): - clear_global_cache() - self.refresh_contracts() - result = self.run_cycle(blockstamp) - - if result is ModuleExecuteDelay.NEXT_FINALIZED_EPOCH: - self._slot_threshold = blockstamp.slot_number - else: - logger.info({ - 'msg': 'Skipping the report. Wait for new finalized slot.', - 'slot_threshold': self._slot_threshold, - }) - - logger.info({'msg': f'Cycle end. Sleep for {variables.CYCLE_SLEEP_IN_SECONDS} seconds.'}) - time.sleep(variables.CYCLE_SLEEP_IN_SECONDS) + self._cycle() + self._sleep_cycle() - def _receive_last_finalized_slot(self) -> BlockStamp: - block_root = BlockRoot(self.w3.cc.get_block_root('finalized').root) - block_details = self.w3.cc.get_block_details(block_root) - bs = build_blockstamp(block_details) - logger.info({'msg': 'Fetch last finalized BlockStamp.', 'value': asdict(bs)}) - ORACLE_SLOT_NUMBER.labels('finalized').set(bs.slot_number) - ORACLE_BLOCK_NUMBER.labels('finalized').set(bs.block_number) - return bs - - def run_cycle(self, blockstamp: BlockStamp) -> ModuleExecuteDelay: + @timeout(variables.MAX_CYCLE_LIFETIME_IN_SECONDS) + def _cycle(self): + """ + Main cycle logic: fetch the last finalized slot, refresh contracts if necessary, + and execute the module's business logic. + """ # pylint: disable=too-many-branches - logger.info({'msg': 'Execute module.', 'value': blockstamp}) - try: - result = self.execute_module(blockstamp) + blockstamp = self._receive_last_finalized_slot() + + # Check if the blockstamp is below the threshold and exit early + if blockstamp.slot_number <= self._slot_threshold: + logger.info({ + 'msg': 'Skipping the report. Waiting for new finalized slot.', + 'slot_threshold': self._slot_threshold, + }) + return + + self.refresh_contracts_if_address_change() + self.run_cycle(blockstamp) except IsNotMemberException as exception: logger.error({'msg': 'Provided account is not part of Oracle`s committee.'}) raise exception @@ -119,12 +106,28 @@ def run_cycle(self, blockstamp: BlockStamp) -> ModuleExecuteDelay: logger.error({'msg': 'IPFS provider error.', 'error': str(error)}) except ValueError as error: logger.error({'msg': 'Unexpected error.', 'error': str(error)}) - else: - # if there are no exceptions, then pulse - pulse() - return result - return ModuleExecuteDelay.NEXT_SLOT + @staticmethod + def _sleep_cycle(): + """Handles sleeping between cycles based on the configured cycle sleep time.""" + logger.info({'msg': f'Cycle end. Sleeping for {variables.CYCLE_SLEEP_IN_SECONDS} seconds.'}) + time.sleep(variables.CYCLE_SLEEP_IN_SECONDS) + + def _receive_last_finalized_slot(self) -> BlockStamp: + block_root = BlockRoot(self.w3.cc.get_block_root('finalized').root) + block_details = self.w3.cc.get_block_details(block_root) + bs = build_blockstamp(block_details) + logger.info({'msg': 'Fetch last finalized BlockStamp.', 'value': asdict(bs)}) + ORACLE_SLOT_NUMBER.labels('finalized').set(bs.slot_number) + ORACLE_BLOCK_NUMBER.labels('finalized').set(bs.block_number) + return bs + + def run_cycle(self, blockstamp: BlockStamp): + logger.info({'msg': 'Execute module.', 'value': blockstamp}) + result = self.execute_module(blockstamp) + pulse() + if result is ModuleExecuteDelay.NEXT_FINALIZED_EPOCH: + self._slot_threshold = blockstamp.slot_number @abstractmethod def execute_module(self, last_finalized_blockstamp: BlockStamp) -> ModuleExecuteDelay: @@ -140,3 +143,9 @@ def execute_module(self, last_finalized_blockstamp: BlockStamp) -> ModuleExecute def refresh_contracts(self): """This method called if contracts addresses were changed""" raise NotImplementedError('Module should implement this method.') # pragma: no cover + + def refresh_contracts_if_address_change(self): + # Refresh contracts if the address has changed + if self.w3.lido_contracts.has_contract_address_changed(): + clear_global_cache() + self.refresh_contracts() diff --git a/src/providers/consensus/client.py b/src/providers/consensus/client.py index 0c39d2f64..0bb7a132c 100644 --- a/src/providers/consensus/client.py +++ b/src/providers/consensus/client.py @@ -1,11 +1,14 @@ from http import HTTPStatus from typing import Literal, cast -from json_stream.base import TransientStreamingJSONObject # type: ignore +from json_stream.base import TransientAccessException, TransientStreamingJSONObject # type: ignore from src.metrics.logging import logging from src.metrics.prometheus.basic import CL_REQUESTS_DURATION from src.providers.consensus.types import ( + BeaconStateView, + BlockAttestation, + BlockAttestationResponse, BlockDetailsResponse, BlockHeaderFullResponse, BlockHeaderResponseData, @@ -13,16 +16,15 @@ Validator, BeaconSpecResponse, GenesisResponse, - SlotAttestationCommittee, BlockAttestation, + SlotAttestationCommittee, ) from src.providers.http_provider import HTTPProvider, NotOkResponse -from src.types import BlockRoot, BlockStamp, SlotNumber, EpochNumber +from src.types import BlockRoot, BlockStamp, SlotNumber, EpochNumber, StateRoot from src.utils.dataclass import list_of_dataclasses from src.utils.cache import global_lru_cache as lru_cache logger = logging.getLogger(__name__) - LiteralState = Literal['head', 'genesis', 'finalized', 'justified'] @@ -51,7 +53,7 @@ class ConsensusClient(HTTPProvider): API_GET_SPEC = 'eth/v1/config/spec' API_GET_GENESIS = 'eth/v1/beacon/genesis' - def get_config_spec(self): + def get_config_spec(self) -> BeaconSpecResponse: """Spec: https://ethereum.github.io/beacon-APIs/#/Config/getSpec""" data, _ = self._get(self.API_GET_SPEC) if not isinstance(data, dict): @@ -108,7 +110,10 @@ def get_block_details(self, state_id: SlotNumber | BlockRoot) -> BlockDetailsRes return BlockDetailsResponse.from_response(**data) @lru_cache(maxsize=256) - def get_block_attestations(self, state_id: SlotNumber | BlockRoot) -> list[BlockAttestation]: + def get_block_attestations( + self, + state_id: SlotNumber | BlockRoot, + ) -> list[BlockAttestation]: """Spec: https://ethereum.github.io/beacon-APIs/#/Beacon/getBlockAttestations""" data, _ = self._get( self.API_GET_BLOCK_ATTESTATIONS, @@ -117,14 +122,14 @@ def get_block_attestations(self, state_id: SlotNumber | BlockRoot) -> list[Block ) if not isinstance(data, list): raise ValueError("Expected list response from getBlockAttestations") - return [BlockAttestation.from_response(**att) for att in data] + return [BlockAttestationResponse.from_response(**att) for att in data] @list_of_dataclasses(SlotAttestationCommittee.from_response) def get_attestation_committees( self, blockstamp: BlockStamp, epoch: EpochNumber | None = None, - index: int | None = None, + committee_index: int | None = None, slot: SlotNumber | None = None ) -> list[SlotAttestationCommittee]: """Spec: https://ethereum.github.io/beacon-APIs/#/Beacon/getEpochCommittees""" @@ -132,24 +137,48 @@ def get_attestation_committees( data, _ = self._get( self.API_GET_ATTESTATION_COMMITTEES, path_params=(blockstamp.state_root,), - query_params={'epoch': epoch, 'index': index, 'slot': slot}, + query_params={'epoch': epoch, 'index': committee_index, 'slot': slot}, force_raise=self.__raise_on_prysm_error ) except NotOkResponse as error: if self.PRYSM_STATE_NOT_FOUND_ERROR in error.text: - data = self._get_attestation_committees_with_prysm(blockstamp, epoch, index, slot) + data = self._get_attestation_committees_with_prysm( + blockstamp, + epoch, + committee_index, + slot, + ) else: raise error return cast(list[SlotAttestationCommittee], data) @lru_cache(maxsize=1) def get_state_block_roots(self, state_id: SlotNumber) -> list[BlockRoot]: + streamed_json = cast(TransientStreamingJSONObject, self._get( + self.API_GET_STATE, + path_params=(state_id,), + stream=True, + )) + return list(streamed_json['data']['block_roots']) + + @lru_cache(maxsize=1) + def get_state_view(self, state_id: SlotNumber | StateRoot) -> BeaconStateView: + """Spec: https://ethereum.github.io/beacon-APIs/#/Debug/getStateV2""" streamed_json = cast(TransientStreamingJSONObject, self._get( self.API_GET_STATE, path_params=(state_id,), stream=True, )) - return list(streamed_json['data']['block_roots']) + view = {} + data = streamed_json['data'] + try: + # NOTE: Keep in mind: the order is important, see TransientStreamingJSONObject. + view['slot'] = int(data['slot']) + view['exit_balance_to_consume'] = int(data['exit_balance_to_consume']) + view['earliest_exit_epoch'] = int(data['earliest_exit_epoch']) + except TransientAccessException: + pass + return BeaconStateView.from_response(**view) @lru_cache(maxsize=1) def get_validators(self, blockstamp: BlockStamp) -> list[Validator]: @@ -159,6 +188,12 @@ def get_validators(self, blockstamp: BlockStamp) -> list[Validator]: @list_of_dataclasses(Validator.from_response) def get_validators_no_cache(self, blockstamp: BlockStamp, pub_keys: str | tuple | None = None) -> list[dict]: """Spec: https://ethereum.github.io/beacon-APIs/#/Beacon/getStateValidators""" + logger.info({ + 'msg': 'Getting validators...', + 'url': self.API_GET_VALIDATORS, + 'slot_number': blockstamp.slot_number, + 'state_root': blockstamp.state_root, + }) try: data, _ = self._get( self.API_GET_VALIDATORS, @@ -168,6 +203,7 @@ def get_validators_no_cache(self, blockstamp: BlockStamp, pub_keys: str | tuple ) if not isinstance(data, list): raise ValueError("Expected list response from getStateValidators") + logger.info({'msg': f'Fetched {len(data)} validators'}) return data except NotOkResponse as error: if self.PRYSM_STATE_NOT_FOUND_ERROR in error.text: @@ -175,7 +211,7 @@ def get_validators_no_cache(self, blockstamp: BlockStamp, pub_keys: str | tuple raise error - PRYSM_STATE_NOT_FOUND_ERROR = 'State not found: state not found in the last' + PRYSM_STATE_NOT_FOUND_ERROR = 'State not found' def __raise_on_prysm_error(self, errors: list[Exception]) -> Exception | None: """ diff --git a/src/providers/consensus/types.py b/src/providers/consensus/types.py index 7b257b34c..60ad7b97b 100644 --- a/src/providers/consensus/types.py +++ b/src/providers/consensus/types.py @@ -1,8 +1,10 @@ from dataclasses import dataclass from enum import Enum +from typing import Literal, Protocol -from src.types import BlockHash, BlockRoot, StateRoot +from src.types import BlockHash, BlockRoot, Gwei, SlotNumber, StateRoot from src.utils.dataclass import Nested, FromResponse +from src.constants import FAR_FUTURE_EPOCH @dataclass @@ -12,6 +14,7 @@ class BeaconSpecResponse(FromResponse): SECONDS_PER_SLOT: str DEPOSIT_CONTRACT_ADDRESS: str SLOTS_PER_HISTORICAL_ROOT: str + ELECTRA_FORK_EPOCH: str = str(FAR_FUTURE_EPOCH) @dataclass @@ -75,16 +78,29 @@ class Checkpoint: @dataclass class AttestationData(Nested, FromResponse): slot: str - index: str + index: str | Literal["0"] beacon_block_root: BlockRoot source: Checkpoint target: Checkpoint @dataclass -class BlockAttestation(Nested, FromResponse): +class BlockAttestationResponse(Nested, FromResponse): aggregation_bits: str data: AttestationData + committee_bits: str | None = None + + +class BlockAttestationPhase0(Protocol): + aggregation_bits: str + data: AttestationData + + +class BlockAttestationEIP7549(BlockAttestationPhase0): + committee_bits: str + + +type BlockAttestation = BlockAttestationPhase0 | BlockAttestationEIP7549 @dataclass @@ -150,3 +166,13 @@ class SlotAttestationCommittee(FromResponse): index: str slot: str validators: list[str] + + +@dataclass +class BeaconStateView(Nested, FromResponse): + """A view to BeaconState with only the required keys presented""" + + slot: SlotNumber + # This fields are new in Electra, so here are default values for backward compatibility. + exit_balance_to_consume: Gwei = Gwei(0) + earliest_exit_epoch: int = 0 diff --git a/src/providers/execution/contracts/lido.py b/src/providers/execution/contracts/lido.py index ed49f6ee1..301e61fd6 100644 --- a/src/providers/execution/contracts/lido.py +++ b/src/providers/execution/contracts/lido.py @@ -89,7 +89,7 @@ def _handle_oracle_report( state_override: dict[ChecksumAddress, CallOverrideParams] = { accounting_oracle_address: { # Fix: insufficient funds for gas * price + value - 'balance': Wei(10**18), + 'balance': Wei(100 * 10**18), # Fix: Sanity checker uses `lastProcessingRefSlot` from AccountingOracle to # properly process negative rebase sanity checks. Since current simulation skips call to AO, # setting up `lastProcessingRefSlot` directly. diff --git a/src/providers/keys/client.py b/src/providers/keys/client.py index cb227f4ed..ae0ba65a6 100644 --- a/src/providers/keys/client.py +++ b/src/providers/keys/client.py @@ -1,11 +1,10 @@ from time import sleep -from typing import cast +from typing import cast, TypedDict, List from src.metrics.prometheus.basic import KEYS_API_REQUESTS_DURATION, KEYS_API_LATEST_BLOCKNUMBER from src.providers.http_provider import HTTPProvider, NotOkResponse from src.providers.keys.types import LidoKey, KeysApiStatus from src.types import BlockStamp, StakingModuleAddress -from src.utils.dataclass import list_of_dataclasses from src.utils.cache import global_lru_cache as lru_cache @@ -17,6 +16,12 @@ class KAPIClientError(NotOkResponse): pass +class ModuleOperatorsKeys(TypedDict): + keys: List[LidoKey] + module: dict + operators: list + + class KeysAPIClient(HTTPProvider): """ Lido Keys are stored in different modules in on-chain and off-chain format. @@ -51,17 +56,18 @@ def _get_with_blockstamp(self, url: str, blockstamp: BlockStamp, params: dict | raise KeysOutdatedException(f'Keys API Service stuck, no updates for {self.backoff_factor * self.retry_count} seconds.') @lru_cache(maxsize=1) - @list_of_dataclasses(LidoKey.from_response) - def get_used_lido_keys(self, blockstamp: BlockStamp) -> list[dict]: + def get_used_lido_keys(self, blockstamp: BlockStamp) -> list[LidoKey]: """Docs: https://keys-api.lido.fi/api/static/index.html#/keys/KeysController_get""" - return cast(list[dict], self._get_with_blockstamp(self.USED_KEYS, blockstamp)) + return list(map(lambda x: LidoKey.from_response(**x), self._get_with_blockstamp(self.USED_KEYS, blockstamp))) @lru_cache(maxsize=1) - def get_module_operators_keys(self, module_address: StakingModuleAddress, blockstamp: BlockStamp) -> dict: + def get_module_operators_keys(self, module_address: StakingModuleAddress, blockstamp: BlockStamp) -> ModuleOperatorsKeys: """ Docs: https://keys-api.lido.fi/api/static/index.html#/operators-keys/SRModulesOperatorsKeysController_getOperatorsKeys """ - return cast(dict, self._get_with_blockstamp(self.MODULE_OPERATORS_KEYS.format(module_address), blockstamp)) + data = cast(dict, self._get_with_blockstamp(self.MODULE_OPERATORS_KEYS.format(module_address), blockstamp)) + data['keys'] = [LidoKey.from_response(**k) for k in data['keys']] + return cast(ModuleOperatorsKeys, data) def get_status(self) -> KeysApiStatus: """Docs: https://keys-api.lido.fi/api/static/index.html#/status/StatusController_get""" diff --git a/src/providers/keys/types.py b/src/providers/keys/types.py index 371fc9861..797c80b9b 100644 --- a/src/providers/keys/types.py +++ b/src/providers/keys/types.py @@ -1,7 +1,9 @@ from dataclasses import dataclass +from typing import cast, Self from eth_typing import ChecksumAddress, HexStr +from src.types import NodeOperatorId from src.utils.dataclass import FromResponse @@ -9,10 +11,17 @@ class LidoKey(FromResponse): key: HexStr depositSignature: HexStr - operatorIndex: int + operatorIndex: NodeOperatorId used: bool moduleAddress: ChecksumAddress + @classmethod + def from_response(cls, **kwargs) -> Self: + response_lido_key = super().from_response(**kwargs) + lido_key: Self = cast(Self, response_lido_key) + lido_key.key = HexStr(lido_key.key.lower()) # pylint: disable=no-member + return lido_key + @dataclass class KeysApiStatus(FromResponse): diff --git a/src/services/bunker.py b/src/services/bunker.py index 6060d310a..9e2791373 100644 --- a/src/services/bunker.py +++ b/src/services/bunker.py @@ -16,6 +16,7 @@ from src.services.bunker_cases.types import BunkerConfig from src.services.safe_border import filter_slashed_validators from src.types import BlockStamp, ReferenceBlockStamp, Gwei +from src.utils.web3converter import Web3Converter from src.web3py.types import Web3 @@ -70,8 +71,11 @@ def is_bunker_mode( logger.info({"msg": "Bunker ON. CL rebase is negative"}) return True + cl_spec = self.w3.cc.get_config_spec() + consensus_version = self.w3.lido_contracts.accounting_oracle.get_consensus_version(blockstamp.block_hash) + web3_converter = Web3Converter(chain_config, frame_config) high_midterm_slashing_penalty = MidtermSlashingPenalty.is_high_midterm_slashing_penalty( - blockstamp, frame_config, chain_config, all_validators, lido_validators, current_report_cl_rebase, last_report_ref_slot + blockstamp, consensus_version, cl_spec, web3_converter, all_validators, lido_validators, current_report_cl_rebase, last_report_ref_slot ) if high_midterm_slashing_penalty: logger.info({"msg": "Bunker ON. High midterm slashing penalty"}) diff --git a/src/services/bunker_cases/abnormal_cl_rebase.py b/src/services/bunker_cases/abnormal_cl_rebase.py index ef74a7fca..7c8ffbc3c 100644 --- a/src/services/bunker_cases/abnormal_cl_rebase.py +++ b/src/services/bunker_cases/abnormal_cl_rebase.py @@ -6,7 +6,7 @@ from web3.contract.contract import ContractEvent from web3.types import EventData -from src.constants import MAX_EFFECTIVE_BALANCE, EFFECTIVE_BALANCE_INCREMENT +from src.constants import EFFECTIVE_BALANCE_INCREMENT, MIN_ACTIVATION_BALANCE from src.modules.submodules.types import ChainConfig from src.providers.consensus.types import Validator from src.providers.keys.types import LidoKey @@ -93,6 +93,7 @@ def _calculate_lido_normal_cl_rebase(self, blockstamp: ReferenceBlockStamp) -> G self.lido_keys, last_report_all_validators ) + # Calculate mean sum of effective balance for all validators and Lido validators (ACTIVE only) mean_sum_of_all_effective_balance = AbnormalClRebase.get_mean_sum_of_effective_balance( last_report_blockstamp, blockstamp, last_report_all_validators, self.all_validators ) @@ -216,10 +217,11 @@ def _get_lido_validators_balance_with_vault( Get Lido validator balance with withdrawals vault balance """ real_cl_balance = AbnormalClRebase.calculate_validators_balance_sum(lido_validators) + pending_deposits_sum = LidoValidatorsProvider.calculate_pending_deposits_sum(lido_validators) withdrawals_vault_balance = int( self.w3.from_wei(self.w3.lido_contracts.get_withdrawal_balance_no_cache(blockstamp), "gwei") ) - return Gwei(real_cl_balance + withdrawals_vault_balance) + return Gwei(real_cl_balance + pending_deposits_sum + withdrawals_vault_balance) def _get_withdrawn_from_vault_between_blocks( self, prev_blockstamp: BlockStamp, ref_blockstamp: ReferenceBlockStamp @@ -289,7 +291,7 @@ def calculate_validators_count_diff_in_gwei( validators_diff = len(ref_validators) - len(prev_validators) if validators_diff < 0: raise ValueError("Validators count diff should be positive or 0. Something went wrong with CL API") - return Gwei(validators_diff * MAX_EFFECTIVE_BALANCE) + return Gwei(validators_diff * MIN_ACTIVATION_BALANCE) @staticmethod def get_mean_sum_of_effective_balance( diff --git a/src/services/bunker_cases/midterm_slashing_penalty.py b/src/services/bunker_cases/midterm_slashing_penalty.py index 721238378..b9417ac60 100644 --- a/src/services/bunker_cases/midterm_slashing_penalty.py +++ b/src/services/bunker_cases/midterm_slashing_penalty.py @@ -5,25 +5,28 @@ EPOCHS_PER_SLASHINGS_VECTOR, MIN_VALIDATOR_WITHDRAWABILITY_DELAY, PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX, - EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE + EFFECTIVE_BALANCE_INCREMENT, + MAX_EFFECTIVE_BALANCE, ) -from src.modules.submodules.types import FrameConfig, ChainConfig -from src.providers.consensus.types import Validator +from src.providers.consensus.types import Validator, BeaconSpecResponse from src.types import EpochNumber, Gwei, ReferenceBlockStamp, FrameNumber, SlotNumber from src.utils.validator_state import calculate_total_active_effective_balance +from src.utils.web3converter import Web3Converter from src.web3py.extensions.lido_validators import LidoValidator - logger = logging.getLogger(__name__) +type SlashedValidatorsFrameBuckets = dict[tuple[FrameNumber, EpochNumber], list[LidoValidator]] + class MidtermSlashingPenalty: @staticmethod def is_high_midterm_slashing_penalty( blockstamp: ReferenceBlockStamp, - frame_config: FrameConfig, - chain_config: ChainConfig, + consensus_version: int, + cl_spec: BeaconSpecResponse, + web3_converter: Web3Converter, all_validators: list[Validator], lido_validators: list[LidoValidator], current_report_cl_rebase: Gwei, @@ -45,7 +48,7 @@ def is_high_midterm_slashing_penalty( # Put all Lido slashed validators to future frames by midterm penalty epoch future_frames_lido_validators = MidtermSlashingPenalty.get_lido_validators_with_future_midterm_epoch( - blockstamp.ref_epoch, frame_config, lido_validators + blockstamp.ref_epoch, web3_converter, lido_validators ) # If no one Lido in current not withdrawn slashed validators @@ -58,16 +61,21 @@ def is_high_midterm_slashing_penalty( total_balance = calculate_total_active_effective_balance(all_validators, blockstamp.ref_epoch) # Calculate sum of Lido midterm penalties in each future frame - frames_lido_midterm_penalties = MidtermSlashingPenalty.get_future_midterm_penalty_sum_in_frames( - blockstamp.ref_epoch, all_slashed_validators, total_balance, future_frames_lido_validators, - ) + if consensus_version in (1, 2): + frames_lido_midterm_penalties = MidtermSlashingPenalty.get_future_midterm_penalty_sum_in_frames_pre_electra( + blockstamp.ref_epoch, all_slashed_validators, total_balance, future_frames_lido_validators + ) + else: + frames_lido_midterm_penalties = MidtermSlashingPenalty.get_future_midterm_penalty_sum_in_frames_post_electra( + blockstamp.ref_epoch, cl_spec, all_slashed_validators, total_balance, future_frames_lido_validators, + ) max_lido_midterm_penalty = max(frames_lido_midterm_penalties.values()) logger.info({"msg": f"Max lido midterm penalty: {max_lido_midterm_penalty}"}) # Compare with calculated frame CL rebase on pessimistic strategy # and whether they will cover future midterm penalties, so that the bunker is better to be turned on than not frame_cl_rebase = MidtermSlashingPenalty.get_frame_cl_rebase_from_report_cl_rebase( - frame_config, chain_config, current_report_cl_rebase, blockstamp, last_report_ref_slot + web3_converter, current_report_cl_rebase, blockstamp, last_report_ref_slot ) if max_lido_midterm_penalty > frame_cl_rebase: return True @@ -129,13 +137,13 @@ def get_possible_slashed_epochs(validator: Validator, ref_epoch: EpochNumber) -> @staticmethod def get_lido_validators_with_future_midterm_epoch( ref_epoch: EpochNumber, - frame_config: FrameConfig, + web3_converter: Web3Converter, lido_validators: list[LidoValidator], - ) -> dict[FrameNumber, list[LidoValidator]]: + ) -> SlashedValidatorsFrameBuckets: """ Put validators to frame buckets by their midterm penalty epoch to calculate penalties impact in each frame """ - buckets: dict[FrameNumber, list[LidoValidator]] = defaultdict(list[LidoValidator]) + buckets: SlashedValidatorsFrameBuckets = defaultdict(list[LidoValidator]) for validator in lido_validators: if not validator.validator.slashed: # We need only slashed validators @@ -144,22 +152,24 @@ def get_lido_validators_with_future_midterm_epoch( if midterm_penalty_epoch <= ref_epoch: # We need midterm penalties only from future frames continue - frame_number = MidtermSlashingPenalty.get_frame_by_epoch(midterm_penalty_epoch, frame_config) - buckets[frame_number].append(validator) + frame_number = web3_converter.get_frame_by_epoch(midterm_penalty_epoch) + frame_ref_slot = SlotNumber(web3_converter.get_frame_first_slot(frame_number) - 1) + frame_ref_epoch = web3_converter.get_epoch_by_slot(frame_ref_slot) + buckets[(frame_number, frame_ref_epoch)].append(validator) return buckets @staticmethod - def get_future_midterm_penalty_sum_in_frames( + def get_future_midterm_penalty_sum_in_frames_pre_electra( ref_epoch: EpochNumber, all_slashed_validators: list[Validator], total_balance: Gwei, - per_frame_validators: dict[FrameNumber, list[LidoValidator]], + per_frame_validators: SlashedValidatorsFrameBuckets, ) -> dict[FrameNumber, Gwei]: """Calculate sum of midterm penalties in each frame""" per_frame_midterm_penalty_sum: dict[FrameNumber, Gwei] = {} - for frame_number, validators_in_future_frame in per_frame_validators.items(): - per_frame_midterm_penalty_sum[frame_number] = MidtermSlashingPenalty.predict_midterm_penalty_in_frame( + for (frame_number, _), validators_in_future_frame in per_frame_validators.items(): + per_frame_midterm_penalty_sum[frame_number] = MidtermSlashingPenalty.predict_midterm_penalty_in_frame_pre_electra( ref_epoch, all_slashed_validators, total_balance, @@ -169,7 +179,7 @@ def get_future_midterm_penalty_sum_in_frames( return per_frame_midterm_penalty_sum @staticmethod - def predict_midterm_penalty_in_frame( + def predict_midterm_penalty_in_frame_pre_electra( ref_epoch: EpochNumber, all_slashed_validators: list[Validator], total_balance: Gwei, @@ -187,6 +197,55 @@ def predict_midterm_penalty_in_frame( ) return Gwei(penalty_in_frame) + @staticmethod + def get_future_midterm_penalty_sum_in_frames_post_electra( + ref_epoch: EpochNumber, + cl_spec: BeaconSpecResponse, + all_slashed_validators: list[Validator], + total_balance: Gwei, + per_frame_validators: SlashedValidatorsFrameBuckets, + ) -> dict[FrameNumber, Gwei]: + """Calculate sum of midterm penalties in each frame""" + per_frame_midterm_penalty_sum: dict[FrameNumber, Gwei] = {} + for (frame_number, frame_ref_epoch), validators_in_future_frame in per_frame_validators.items(): + per_frame_midterm_penalty_sum[frame_number] = MidtermSlashingPenalty.predict_midterm_penalty_in_frame_post_electra( + ref_epoch, + frame_ref_epoch, + cl_spec, + all_slashed_validators, + total_balance, + validators_in_future_frame + ) + + return per_frame_midterm_penalty_sum + + @staticmethod + def predict_midterm_penalty_in_frame_post_electra( + report_ref_epoch: EpochNumber, + frame_ref_epoch: EpochNumber, + cl_spec: BeaconSpecResponse, + all_slashed_validators: list[Validator], + total_balance: Gwei, + midterm_penalized_validators_in_frame: list[LidoValidator] + ) -> Gwei: + """Predict penalty in frame""" + penalty_in_frame = 0 + for validator in midterm_penalized_validators_in_frame: + midterm_penalty_epoch = MidtermSlashingPenalty.get_midterm_penalty_epoch(validator) + bound_slashed_validators = MidtermSlashingPenalty.get_bound_with_midterm_epoch_slashed_validators( + report_ref_epoch, all_slashed_validators, EpochNumber(midterm_penalty_epoch) + ) + + if frame_ref_epoch < int(cl_spec.ELECTRA_FORK_EPOCH): + penalty_in_frame += MidtermSlashingPenalty.get_validator_midterm_penalty( + validator, len(bound_slashed_validators), total_balance + ) + else: + penalty_in_frame += MidtermSlashingPenalty.get_validator_midterm_penalty_electra( + validator, bound_slashed_validators, total_balance + ) + return Gwei(penalty_in_frame) + @staticmethod def get_validator_midterm_penalty( validator: LidoValidator, @@ -208,6 +267,28 @@ def get_validator_midterm_penalty( return Gwei(penalty) + @staticmethod + def get_validator_midterm_penalty_electra( + validator: LidoValidator, + bound_slashed_validators: list[Validator], + total_balance: Gwei, + ) -> Gwei: + """ + Calculate midterm penalty for particular validator + https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#modified-process_slashings + """ + # We don't know validators effective balances on the moment of slashing, + # so we assume that it was at least `effective_balance` + slashings = Gwei(sum(int(v.validator.effective_balance) for v in bound_slashed_validators)) + adjusted_total_slashing_balance = min( + slashings * PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX, total_balance + ) + effective_balance = int(validator.validator.effective_balance) + penalty_per_effective_balance_increment = adjusted_total_slashing_balance // (total_balance // EFFECTIVE_BALANCE_INCREMENT) + effective_balance_increments = effective_balance // EFFECTIVE_BALANCE_INCREMENT + penalty = penalty_per_effective_balance_increment * effective_balance_increments + return Gwei(penalty) + @staticmethod def get_bound_with_midterm_epoch_slashed_validators( ref_epoch: EpochNumber, @@ -228,27 +309,21 @@ def is_bound(v: Validator) -> bool: @staticmethod def get_frame_cl_rebase_from_report_cl_rebase( - frame_config: FrameConfig, - chain_config: ChainConfig, + web3_converter: Web3Converter, report_cl_rebase: Gwei, curr_report_blockstamp: ReferenceBlockStamp, last_report_ref_slot: SlotNumber ) -> Gwei: """Get frame rebase from report rebase""" - last_report_ref_epoch = EpochNumber(last_report_ref_slot // chain_config.slots_per_epoch) + last_report_ref_epoch = web3_converter.get_epoch_by_slot(last_report_ref_slot) epochs_passed_since_last_report = curr_report_blockstamp.ref_epoch - last_report_ref_epoch frame_cl_rebase = ( - (report_cl_rebase / epochs_passed_since_last_report) * frame_config.epochs_per_frame + (report_cl_rebase / epochs_passed_since_last_report) * web3_converter.frame_config.epochs_per_frame ) return Gwei(int(frame_cl_rebase)) - @staticmethod - def get_frame_by_epoch(epoch: EpochNumber, frame_config: FrameConfig) -> FrameNumber: - """Get oracle report frame index by epoch""" - return FrameNumber((epoch - frame_config.initial_epoch) // frame_config.epochs_per_frame) - @staticmethod def get_midterm_penalty_epoch(validator: Validator) -> EpochNumber: """https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#slashings""" diff --git a/src/services/exit_order/iterator.py b/src/services/exit_order/iterator.py index bf398352a..d350f1236 100644 --- a/src/services/exit_order/iterator.py +++ b/src/services/exit_order/iterator.py @@ -7,8 +7,7 @@ from src.metrics.prometheus.duration_meter import duration_meter from src.modules.submodules.types import ChainConfig from src.services.exit_order.iterator_state import ExitOrderIteratorStateService, NodeOperatorPredictableState -from src.types import ReferenceBlockStamp, NodeOperatorGlobalIndex, StakingModuleId, NodeOperatorId - +from src.types import ReferenceBlockStamp, NodeOperatorGlobalIndex, StakingModuleId from src.utils.validator_state import get_validator_age from src.web3py.extensions.lido_validators import LidoValidator from src.web3py.types import Web3 @@ -154,9 +153,13 @@ def _operator_stake_weight( We prefer to exit validators which operators with high stake weight first. Operators who have stake weight less than `operator_network_penetration_threshold` will have the same weight """ + if not total_predictable_validators_count: + return 0 + stake_volume = operator_state.predictable_validators_count / total_predictable_validators_count if stake_volume > operator_network_penetration_threshold: return operator_state.predictable_validators_total_age + return 0 @staticmethod @@ -173,5 +176,5 @@ def operator_index_by_validator( ) -> NodeOperatorGlobalIndex: return ( StakingModuleId(staking_module_id[validator.lido_id.moduleAddress]), - NodeOperatorId(validator.lido_id.operatorIndex), + validator.lido_id.operatorIndex, ) diff --git a/src/utils/blockstamp.py b/src/utils/blockstamp.py index 897847806..01234076a 100644 --- a/src/utils/blockstamp.py +++ b/src/utils/blockstamp.py @@ -31,5 +31,5 @@ def _build_blockstamp_data( "state_root": slot_details.message.state_root, "block_number": BlockNumber(int(execution_payload.block_number)), "block_hash": execution_payload.block_hash, - "block_timestamp": Timestamp(int(execution_payload.timestamp)) + "block_timestamp": Timestamp(int(execution_payload.timestamp)), } diff --git a/src/utils/build.py b/src/utils/build.py index e969e5e98..ff2c9af0c 100644 --- a/src/utils/build.py +++ b/src/utils/build.py @@ -1,16 +1,12 @@ import json -import os UNKNOWN_BUILD_INFO = {"version": "unknown", "branch": "unknown", "commit": "unknown"} def get_build_info() -> dict: path = "./build-info.json" - if os.path.exists(path): + try: with open(path, "r") as f: - try: - build_info = json.load(f) - except json.JSONDecodeError: - return UNKNOWN_BUILD_INFO - return build_info - return UNKNOWN_BUILD_INFO + return json.load(f) + except (FileNotFoundError, json.JSONDecodeError): + return UNKNOWN_BUILD_INFO diff --git a/src/utils/types.py b/src/utils/types.py index 08dd5d85d..b1e3457c8 100644 --- a/src/utils/types.py +++ b/src/utils/types.py @@ -5,5 +5,15 @@ def bytes_to_hex_str(b: bytes) -> HexStr: return HexStr('0x' + b.hex()) -def hex_str_to_bytes(hex_str: HexStr) -> bytes: - return bytes.fromhex(hex_str[2:]) +def hex_str_to_bytes(hex_str: str) -> bytes: + return bytes.fromhex(hex_str[2:]) if hex_str.startswith("0x") else bytes.fromhex(hex_str) + + +def is_4bytes_hex(s: str) -> bool: + if not s.startswith("0x"): + return False + + try: + return len(bytes.fromhex(s[2:])) == 4 + except ValueError: + return False diff --git a/src/utils/validator_state.py b/src/utils/validator_state.py index 56449f290..04ac7e4b6 100644 --- a/src/utils/validator_state.py +++ b/src/utils/validator_state.py @@ -1,14 +1,18 @@ from typing import Sequence from src.constants import ( - MAX_EFFECTIVE_BALANCE, + CHURN_LIMIT_QUOTIENT, + COMPOUNDING_WITHDRAWAL_PREFIX, + EFFECTIVE_BALANCE_INCREMENT, ETH1_ADDRESS_WITHDRAWAL_PREFIX, - SHARD_COMMITTEE_PERIOD, FAR_FUTURE_EPOCH, - EFFECTIVE_BALANCE_INCREMENT, + MAX_EFFECTIVE_BALANCE_ELECTRA, + MAX_PER_EPOCH_ACTIVATION_EXIT_CHURN_LIMIT, MAX_SEED_LOOKAHEAD, + MIN_ACTIVATION_BALANCE, MIN_PER_EPOCH_CHURN_LIMIT, - CHURN_LIMIT_QUOTIENT, + MIN_PER_EPOCH_CHURN_LIMIT_ELECTRA, + SHARD_COMMITTEE_PERIOD, ) from src.providers.consensus.types import Validator from src.types import EpochNumber, Gwei @@ -41,15 +45,24 @@ def is_partially_withdrawable_validator(validator: Validator) -> bool: Check if `validator` is partially withdrawable https://github.com/ethereum/consensus-specs/blob/dev/specs/capella/beacon-chain.md#is_partially_withdrawable_validator """ - has_max_effective_balance = int(validator.validator.effective_balance) == MAX_EFFECTIVE_BALANCE - has_excess_balance = int(validator.balance) > MAX_EFFECTIVE_BALANCE + max_effective_balance = get_max_effective_balance(validator) + has_max_effective_balance = int(validator.validator.effective_balance) == max_effective_balance + has_excess_balance = int(validator.balance) > max_effective_balance return ( - has_eth1_withdrawal_credential(validator) + has_execution_withdrawal_credential(validator) and has_max_effective_balance and has_excess_balance ) +def has_compounding_withdrawal_credential(validator: Validator) -> bool: + """ + Check if ``validator`` has an 0x02 prefixed "compounding" withdrawal credential. + https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-has_compounding_withdrawal_credential + """ + return validator.validator.withdrawal_credentials[:4] == COMPOUNDING_WITHDRAWAL_PREFIX + + def has_eth1_withdrawal_credential(validator: Validator) -> bool: """ Check if ``validator`` has an 0x01 prefixed "eth1" withdrawal credential. @@ -58,13 +71,21 @@ def has_eth1_withdrawal_credential(validator: Validator) -> bool: return validator.validator.withdrawal_credentials[:4] == ETH1_ADDRESS_WITHDRAWAL_PREFIX +def has_execution_withdrawal_credential(validator: Validator) -> bool: + """ + Check if ``validator`` has a 0x01 or 0x02 prefixed withdrawal credential. + https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-has_execution_withdrawal_credential + """ + return has_compounding_withdrawal_credential(validator) or has_eth1_withdrawal_credential(validator) + + def is_fully_withdrawable_validator(validator: Validator, epoch: EpochNumber) -> bool: """ Check if `validator` is fully withdrawable - https://github.com/ethereum/consensus-specs/blob/dev/specs/capella/beacon-chain.md#is_fully_withdrawable_validator + https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#modified-is_fully_withdrawable_validator """ return ( - has_eth1_withdrawal_credential(validator) + has_execution_withdrawal_credential(validator) and EpochNumber(int(validator.validator.withdrawable_epoch)) <= epoch and Gwei(int(validator.balance)) > Gwei(0) ) @@ -74,7 +95,10 @@ def is_validator_eligible_to_exit(validator: Validator, epoch: EpochNumber) -> b """ Check if `validator` can exit. Verify the validator has been active long enough. - https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#voluntary-exits + https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#voluntary-exits + + The validator can only exit if it has no pending withdrawals in the queue. + This method don't take partial withdrawals into account because Lido protocol doesn't support partial withdrawals. """ active_long_enough = int(validator.validator.activation_epoch) + SHARD_COMMITTEE_PERIOD <= epoch return active_long_enough and not is_on_exit(validator) @@ -112,5 +136,26 @@ def compute_activation_exit_epoch(ref_epoch: EpochNumber): return ref_epoch + 1 + MAX_SEED_LOOKAHEAD -def compute_exit_churn_limit(active_validators_count: int): +def get_validator_churn_limit(active_validators_count: int): return max(MIN_PER_EPOCH_CHURN_LIMIT, active_validators_count // CHURN_LIMIT_QUOTIENT) + + +# @see https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-get_activation_exit_churn_limit +def get_activation_exit_churn_limit(total_active_balance: Gwei) -> Gwei: + return min(MAX_PER_EPOCH_ACTIVATION_EXIT_CHURN_LIMIT, get_balance_churn_limit(total_active_balance)) + + +# @see https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-get_balance_churn_limit +def get_balance_churn_limit(total_active_balance: Gwei) -> Gwei: + churn = max(MIN_PER_EPOCH_CHURN_LIMIT_ELECTRA, total_active_balance // CHURN_LIMIT_QUOTIENT) + return Gwei(churn - churn % EFFECTIVE_BALANCE_INCREMENT) + + +def get_max_effective_balance(validator: Validator) -> Gwei: + """ + Get max effective balance for ``validator``. + https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#new-get_max_effective_balance + """ + if has_compounding_withdrawal_credential(validator): + return MAX_EFFECTIVE_BALANCE_ELECTRA + return MIN_ACTIVATION_BALANCE diff --git a/src/variables.py b/src/variables.py index ec13ae368..942561c2f 100644 --- a/src/variables.py +++ b/src/variables.py @@ -87,22 +87,66 @@ def check_all_required_variables(): errors = check_uri_required_variables() - if LIDO_LOCATOR_ADDRESS in (None, ''): + if not LIDO_LOCATOR_ADDRESS: errors.append('LIDO_LOCATOR_ADDRESS') return errors def check_uri_required_variables(): - errors = [] - if '' in EXECUTION_CLIENT_URI: - errors.append('EXECUTION_CLIENT_URI') - if '' in CONSENSUS_CLIENT_URI: - errors.append('CONSENSUS_CLIENT_URI') - if '' in KEYS_API_URI: - errors.append('KEYS_API_URI') - return errors + required_uris = { + 'EXECUTION_CLIENT_URI': EXECUTION_CLIENT_URI, + 'CONSENSUS_CLIENT_URI': CONSENSUS_CLIENT_URI, + 'KEYS_API_URI': KEYS_API_URI, + } + return [name for name, uri in required_uris.items() if '' in uri] def raise_from_errors(errors): if errors: raise ValueError("The following variables are required: " + ", ".join(errors)) + + +# All non-private env variables to the logs in main +PUBLIC_ENV_VARS = { + 'ACCOUNT': 'Dry' if ACCOUNT is None else ACCOUNT.address, + 'LIDO_LOCATOR_ADDRESS': LIDO_LOCATOR_ADDRESS, + 'CSM_MODULE_ADDRESS': CSM_MODULE_ADDRESS, + 'FINALIZATION_BATCH_MAX_REQUEST_COUNT': FINALIZATION_BATCH_MAX_REQUEST_COUNT, + 'EL_REQUESTS_BATCH_SIZE': EL_REQUESTS_BATCH_SIZE, + 'CSM_ORACLE_MAX_CONCURRENCY': CSM_ORACLE_MAX_CONCURRENCY, + 'TX_GAS_ADDITION': TX_GAS_ADDITION, + 'EVENTS_SEARCH_STEP': EVENTS_SEARCH_STEP, + 'MIN_PRIORITY_FEE': MIN_PRIORITY_FEE, + 'MAX_PRIORITY_FEE': MAX_PRIORITY_FEE, + 'PRIORITY_FEE_PERCENTILE': PRIORITY_FEE_PERCENTILE, + 'DAEMON': DAEMON, + 'SUBMIT_DATA_DELAY_IN_SLOTS': SUBMIT_DATA_DELAY_IN_SLOTS, + 'CYCLE_SLEEP_IN_SECONDS': CYCLE_SLEEP_IN_SECONDS, + 'ALLOW_REPORTING_IN_BUNKER_MODE': ALLOW_REPORTING_IN_BUNKER_MODE, + 'HTTP_REQUEST_TIMEOUT_EXECUTION': HTTP_REQUEST_TIMEOUT_EXECUTION, + 'HTTP_REQUEST_TIMEOUT_CONSENSUS': HTTP_REQUEST_TIMEOUT_CONSENSUS, + 'HTTP_REQUEST_RETRY_COUNT_CONSENSUS': HTTP_REQUEST_RETRY_COUNT_CONSENSUS, + 'HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS_CONSENSUS': HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS_CONSENSUS, + 'HTTP_REQUEST_TIMEOUT_KEYS_API': HTTP_REQUEST_TIMEOUT_KEYS_API, + 'HTTP_REQUEST_RETRY_COUNT_KEYS_API': HTTP_REQUEST_RETRY_COUNT_KEYS_API, + 'HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS_KEYS_API': HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS_KEYS_API, + 'HTTP_REQUEST_TIMEOUT_IPFS': HTTP_REQUEST_TIMEOUT_IPFS, + 'HTTP_REQUEST_RETRY_COUNT_IPFS': HTTP_REQUEST_RETRY_COUNT_IPFS, + 'PROMETHEUS_PORT': PROMETHEUS_PORT, + 'PROMETHEUS_PREFIX': PROMETHEUS_PREFIX, + 'HEALTHCHECK_SERVER_PORT': HEALTHCHECK_SERVER_PORT, + 'MAX_CYCLE_LIFETIME_IN_SECONDS': MAX_CYCLE_LIFETIME_IN_SECONDS, + 'CACHE_PATH': str(CACHE_PATH), +} + +PRIVATE_ENV_VARS = { + 'EXECUTION_CLIENT_URI': EXECUTION_CLIENT_URI, + 'CONSENSUS_CLIENT_URI': CONSENSUS_CLIENT_URI, + 'KEYS_API_URI': KEYS_API_URI, + 'GW3_ACCESS_KEY': GW3_ACCESS_KEY, + 'GW3_SECRET_KEY': GW3_SECRET_KEY, + 'PINATA_JWT': PINATA_JWT, + 'MEMBER_PRIV_KEY': MEMBER_PRIV_KEY, +} + +assert not set(PRIVATE_ENV_VARS.keys()).intersection(set(PUBLIC_ENV_VARS.keys())) diff --git a/src/web3py/extensions/lido_validators.py b/src/web3py/extensions/lido_validators.py index 6e813e20b..768da009a 100644 --- a/src/web3py/extensions/lido_validators.py +++ b/src/web3py/extensions/lido_validators.py @@ -3,19 +3,18 @@ from enum import Enum from typing import TYPE_CHECKING -from eth_typing import ChecksumAddress +from eth_typing import ChecksumAddress, HexStr from web3.module import Module +from src.constants import FAR_FUTURE_EPOCH, LIDO_DEPOSIT_AMOUNT from src.providers.consensus.types import Validator from src.providers.keys.types import LidoKey from src.types import BlockStamp, StakingModuleId, NodeOperatorId, NodeOperatorGlobalIndex, StakingModuleAddress from src.utils.dataclass import Nested from src.utils.cache import global_lru_cache as lru_cache - logger = logging.getLogger(__name__) - if TYPE_CHECKING: from src.web3py.types import Web3 # pragma: no cover @@ -172,6 +171,16 @@ def merge_validators_with_keys(keys: list[LidoKey], validators: list[Validator]) return lido_validators + @staticmethod + def calculate_pending_deposits_sum(lido_validators: list[LidoValidator]) -> int: + # NOTE: Using 32 ETH as a default validator pending balance is OK for the current protocol implementation. + # It must be changed in case of validators consolidation feature implementation. + return sum( + LIDO_DEPOSIT_AMOUNT + for validator in lido_validators + if int(validator.balance) == 0 and int(validator.validator.activation_epoch) == FAR_FUTURE_EPOCH + ) + @lru_cache(maxsize=1) def get_lido_validators_by_node_operators(self, blockstamp: BlockStamp) -> ValidatorsByNodeOperator: merged_validators = self.get_lido_validators(blockstamp) @@ -190,7 +199,7 @@ def get_lido_validators_by_node_operators(self, blockstamp: BlockStamp) -> Valid for validator in merged_validators: global_no_id = ( staking_module_address[validator.lido_id.moduleAddress], - NodeOperatorId(validator.lido_id.operatorIndex), + validator.lido_id.operatorIndex, ) if global_no_id in no_validators: @@ -204,15 +213,28 @@ def get_lido_validators_by_node_operators(self, blockstamp: BlockStamp) -> Valid return no_validators @lru_cache(maxsize=1) - def get_module_validators_by_node_operators(self, module_address: StakingModuleAddress, blockstamp: BlockStamp) -> ValidatorsByNodeOperator: - """Get module validators by querying the KeysAPI for the module keys""" + def get_module_validators_by_node_operators( + self, + module_address: StakingModuleAddress, + blockstamp: BlockStamp + ) -> ValidatorsByNodeOperator: + """ + Get module validators by querying the KeysAPI for the module keys. + + Args: + module_address (StakingModuleAddress): The address of the staking module. + blockstamp (BlockStamp): The block timestamp for querying validators. + + Returns: + ValidatorsByNodeOperator: A mapping of node operator IDs to their corresponding validators. + """ + # Fetch module operator keys from the KeysAPI kapi = self.w3.kac.get_module_operators_keys(module_address, blockstamp) if (kapi_module_address := kapi['module']['stakingModuleAddress']) != module_address: raise ValueError(f"Module address mismatch: {kapi_module_address=} != {module_address=}") operators = kapi['operators'] - keys = {k['key']: k for k in kapi['keys']} + keys = {k.key: k for k in kapi['keys']} validators = self.w3.cc.get_validators(blockstamp) - module_id = StakingModuleId(int(kapi['module']['id'])) # Make sure even empty NO will be presented in dict @@ -220,14 +242,15 @@ def get_module_validators_by_node_operators(self, module_address: StakingModuleA (module_id, NodeOperatorId(int(operator['index']))): [] for operator in operators } + # Map validators to their corresponding node operators for validator in validators: - lido_key = keys.get(validator.validator.pubkey) + lido_key = keys.get(HexStr(validator.validator.pubkey)) if not lido_key: continue - global_id = (module_id, lido_key['operatorIndex']) + global_id = (module_id, lido_key.operatorIndex) no_validators[global_id].append( LidoValidator( - lido_id=LidoKey.from_response(**lido_key), + lido_id=lido_key, **asdict(validator), ) ) diff --git a/src/web3py/types.py b/src/web3py/types.py index c21e890eb..7f206cd46 100644 --- a/src/web3py/types.py +++ b/src/web3py/types.py @@ -1,14 +1,13 @@ from web3 import Web3 as _Web3 - from src.providers.ipfs import IPFSProvider from src.web3py.extensions import ( - LidoContracts, - TransactionUtils, + CSM, ConsensusClientModule, KeysAPIClientModule, + LidoContracts, LidoValidatorsProvider, - CSM + TransactionUtils, ) diff --git a/tests/factory/bitarrays.py b/tests/factory/bitarrays.py new file mode 100644 index 000000000..a9d8f3032 --- /dev/null +++ b/tests/factory/bitarrays.py @@ -0,0 +1,60 @@ +from typing import Sequence + +from pydantic import BaseModel +from pydantic_factories import ModelFactory + + +class BitList(BaseModel): + __root__: bytes + + def hex(self) -> str: + return f"0x{self.__root__.hex()}" + + +class BitListFactory(ModelFactory): + __model__ = BitList + + @classmethod + def build( + cls, + factory_use_construct: bool = False, + set_indices: list[int] = [], + bits_count: int = 0, + **kwargs, + ) -> BitList: + bit_list: list[bool] = [] + for n in sorted(set_indices): + while len(bit_list) < n: + bit_list += [False] + bit_list += [True] + + model = cls._get_model() + return model( + __root__=get_serialized_bytearray( + bit_list, + bits_count=bits_count or len(bit_list), + extra_byte=True, + ) + ) + + +def get_serialized_bytearray(value: Sequence[bool], bits_count: int, extra_byte: bool) -> bytearray: + """ + Serialize a sequence either into a Bitlist or a Bitvector + @see https://github.com/ethereum/py-ssz/blob/main/ssz/utils.py#L223 + """ + + if extra_byte: + # Serialize Bitlist + as_bytearray = bytearray(bits_count // 8 + 1) + else: + # Serialize Bitvector + as_bytearray = bytearray((bits_count + 7) // 8) + + for i in range(bits_count): + as_bytearray[i // 8] |= value[i] << (i % 8) + + if extra_byte: + as_bytearray[bits_count // 8] |= 1 << (bits_count % 8) + + return as_bytearray diff --git a/tests/factory/configs.py b/tests/factory/configs.py index 029e4395a..6ccb411bf 100644 --- a/tests/factory/configs.py +++ b/tests/factory/configs.py @@ -3,7 +3,7 @@ from src.providers.consensus.types import ( BeaconSpecResponse, SlotAttestationCommittee, - BlockAttestation, + BlockAttestationResponse, AttestationData, Checkpoint, ) @@ -61,9 +61,10 @@ class SlotAttestationCommitteeFactory(Web3Factory): class BlockAttestationFactory(Web3Factory): - __model__ = BlockAttestation + __model__ = BlockAttestationResponse aggregation_bits = "0x" + committee_bits = None data = AttestationData( slot="0", index="0", diff --git a/tests/factory/no_registry.py b/tests/factory/no_registry.py index eee95c6c5..785c633fa 100644 --- a/tests/factory/no_registry.py +++ b/tests/factory/no_registry.py @@ -3,13 +3,15 @@ from typing import Any from faker import Faker +from hexbytes import HexBytes from pydantic_factories import Use -from src.constants import FAR_FUTURE_EPOCH +from src.constants import EFFECTIVE_BALANCE_INCREMENT, FAR_FUTURE_EPOCH, MAX_EFFECTIVE_BALANCE, MIN_ACTIVATION_BALANCE from src.providers.consensus.types import Validator, ValidatorState from src.providers.keys.types import LidoKey +from src.types import Gwei +from src.web3py.extensions.lido_validators import LidoValidator, NodeOperator, StakingModule from tests.factory.web3_factory import Web3Factory -from src.web3py.extensions.lido_validators import StakingModule, LidoValidator, NodeOperator faker = Faker() @@ -17,12 +19,32 @@ class ValidatorStateFactory(Web3Factory): __model__ = ValidatorState + withdrawal_credentials = "0x01" exit_epoch = FAR_FUTURE_EPOCH + @classmethod + def build(cls, **kwargs: Any): + if 'pubkey' not in kwargs: + kwargs['pubkey'] = HexBytes(faker.binary(length=48)).hex() + return super().build(**kwargs) + class ValidatorFactory(Web3Factory): __model__ = Validator + @classmethod + def build_pending_deposit_vals(cls, **kwargs: Any): + return cls.build( + balance=str(0), + validator=ValidatorStateFactory.build( + activation_eligibility_epoch=str(FAR_FUTURE_EPOCH), + activation_epoch=str(FAR_FUTURE_EPOCH), + exit_epoch=str(FAR_FUTURE_EPOCH), + effective_balance=str(0), + ), + **kwargs, + ) + class LidoKeyFactory(Web3Factory): __model__ = LidoKey @@ -53,6 +75,19 @@ def build_with_activation_epoch_bound(cls, max_value: int, **kwargs: Any): validator=ValidatorStateFactory.build(activation_epoch=str(faker.pyint(max_value=max_value - 1))), **kwargs ) + @classmethod + def build_pending_deposit_vals(cls, **kwargs: Any): + return cls.build( + balance=str(0), + validator=ValidatorStateFactory.build( + activation_eligibility_epoch=str(FAR_FUTURE_EPOCH), + activation_epoch=str(FAR_FUTURE_EPOCH), + exit_epoch=str(FAR_FUTURE_EPOCH), + effective_balance=str(0), + ), + **kwargs, + ) + @classmethod def build_not_active_vals(cls, epoch, **kwargs: Any): return cls.build( @@ -60,7 +95,7 @@ def build_not_active_vals(cls, epoch, **kwargs: Any): activation_epoch=str(faker.pyint(min_value=epoch + 1, max_value=FAR_FUTURE_EPOCH)), exit_epoch=str(FAR_FUTURE_EPOCH), ), - **kwargs + **kwargs, ) @classmethod @@ -70,7 +105,7 @@ def build_active_vals(cls, epoch, **kwargs: Any): activation_epoch=str(faker.pyint(min_value=0, max_value=epoch - 1)), exit_epoch=str(faker.pyint(min_value=epoch + 1, max_value=FAR_FUTURE_EPOCH)), ), - **kwargs + **kwargs, ) @classmethod @@ -80,7 +115,18 @@ def build_exit_vals(cls, epoch, **kwargs: Any): activation_epoch='0', exit_epoch=str(faker.pyint(min_value=1, max_value=epoch)), ), - **kwargs + **kwargs, + ) + + @classmethod + def build_with_balance(cls, balance: float, meb: int = MAX_EFFECTIVE_BALANCE, **kwargs: Any): + return cls.build( + balance=balance, + validator=ValidatorStateFactory.build( + effective_balance=min(balance - balance % EFFECTIVE_BALANCE_INCREMENT, meb), + withdrawal_credentials="0x01" if meb == MAX_EFFECTIVE_BALANCE else "0x02", + ), + **kwargs, ) diff --git a/tests/modules/accounting/bunker/conftest.py b/tests/modules/accounting/bunker/conftest.py index 68b01e80e..c6df7477a 100644 --- a/tests/modules/accounting/bunker/conftest.py +++ b/tests/modules/accounting/bunker/conftest.py @@ -9,6 +9,7 @@ from src.services.bunker_cases.abnormal_cl_rebase import AbnormalClRebase from src.services.bunker_cases.types import BunkerConfig from src.types import BlockNumber, BlockStamp, ReferenceBlockStamp +from tests.modules.ejector.test_exit_order_state_service import FAR_FUTURE_EPOCH def simple_ref_blockstamp(block_number: int) -> ReferenceBlockStamp: @@ -25,7 +26,9 @@ def simple_key(pubkey: str) -> LidoKey: return key -def simple_validator(index, pubkey, balance, slashed=False, withdrawable_epoch='', exit_epoch='100500') -> Validator: +def simple_validator( + index, pubkey, balance, slashed=False, withdrawable_epoch='', exit_epoch='100500', activation_epoch="0" +) -> Validator: return Validator( index=str(index), balance=str(balance), @@ -36,7 +39,7 @@ def simple_validator(index, pubkey, balance, slashed=False, withdrawable_epoch=' effective_balance=str(32 * 10**9), slashed=slashed, activation_eligibility_epoch='', - activation_epoch='0', + activation_epoch=activation_epoch, exit_epoch=exit_epoch, withdrawable_epoch=withdrawable_epoch, ), @@ -134,6 +137,7 @@ def _get_withdrawal_vault_balance(blockstamp: BlockStamp): 31: 2 * 10**18, 33: 2 * 10**18, 40: 2 * 10**18, + 50: 2 * 10**18, } return balance[blockstamp.block_number] @@ -199,6 +203,14 @@ def _get_validators(state: ReferenceBlockStamp, _=None): simple_validator(4, '0x04', 32 * 10**9), simple_validator(5, '0x05', (32 * 10**9) + 824112), ], + 50: [ + simple_validator(4, '0x00', balance=0, activation_epoch=FAR_FUTURE_EPOCH), + simple_validator(1, '0x01', 32 * 10**9), + simple_validator(2, '0x02', 32 * 10**9), + simple_validator(3, '0x03', (32 * 10**9) + 333333), + simple_validator(4, '0x04', balance=0, activation_epoch=FAR_FUTURE_EPOCH), + simple_validator(5, '0x05', (32 * 10**9) + 824112), + ], 1000: [ simple_validator(0, '0x00', 32 * 10**9), simple_validator(1, '0x01', 32 * 10**9), diff --git a/tests/modules/accounting/bunker/test_bunker.py b/tests/modules/accounting/bunker/test_bunker.py index a3340d1c2..2174bd4ea 100644 --- a/tests/modules/accounting/bunker/test_bunker.py +++ b/tests/modules/accounting/bunker/test_bunker.py @@ -51,6 +51,7 @@ def test_true_when_cl_rebase_is_negative( is_high_midterm_slashing_penalty: Mock, ) -> None: bunker.w3.lido_contracts.get_accounting_last_processing_ref_slot = Mock(return_value=ref_blockstamp) + bunker.w3.cc.get_config_spec = Mock() bunker.get_cl_rebase_for_current_report = Mock(return_value=-1) result = bunker.is_bunker_mode( @@ -79,6 +80,8 @@ def test_true_when_high_midterm_slashing_penalty( is_abnormal_cl_rebase: Mock, ) -> None: bunker.w3.lido_contracts.get_accounting_last_processing_ref_slot = Mock(return_value=ref_blockstamp) + bunker.w3.lido_contracts.accounting_oracle.get_consensus_version = Mock() + bunker.w3.cc.get_config_spec = Mock() bunker.get_cl_rebase_for_current_report = Mock(return_value=0) is_high_midterm_slashing_penalty.return_value = True result = bunker.is_bunker_mode( @@ -105,6 +108,8 @@ def test_true_when_abnormal_cl_rebase( is_abnormal_cl_rebase: Mock, ) -> None: bunker.w3.lido_contracts.get_accounting_last_processing_ref_slot = Mock(return_value=ref_blockstamp) + bunker.w3.lido_contracts.accounting_oracle.get_consensus_version = Mock() + bunker.w3.cc.get_config_spec = Mock() bunker.get_cl_rebase_for_current_report = Mock(return_value=0) is_high_midterm_slashing_penalty.return_value = False is_abnormal_cl_rebase.return_value = True @@ -133,6 +138,8 @@ def test_no_bunker_mode_by_default( is_abnormal_cl_rebase: Mock, ) -> None: bunker.w3.lido_contracts.get_accounting_last_processing_ref_slot = Mock(return_value=ref_blockstamp) + bunker.w3.lido_contracts.accounting_oracle.get_consensus_version = Mock() + bunker.w3.cc.get_config_spec = Mock() bunker.get_cl_rebase_for_current_report = Mock(return_value=0) is_high_midterm_slashing_penalty.return_value = False is_abnormal_cl_rebase.return_value = False diff --git a/tests/modules/accounting/bunker/test_bunker_abnormal_cl_rebase.py b/tests/modules/accounting/bunker/test_bunker_abnormal_cl_rebase.py index a85c46288..a004d6847 100644 --- a/tests/modules/accounting/bunker/test_bunker_abnormal_cl_rebase.py +++ b/tests/modules/accounting/bunker/test_bunker_abnormal_cl_rebase.py @@ -90,6 +90,7 @@ def test_is_abnormal_cl_rebase( @pytest.mark.parametrize( ("blockstamp", "expected_rebase"), [ + (simple_ref_blockstamp(50), 512000000), (simple_ref_blockstamp(40), 420650924), (simple_ref_blockstamp(20), 140216974), (simple_ref_blockstamp(123), 1120376622), @@ -234,6 +235,7 @@ def test_calculate_cl_rebase_between_blocks( @pytest.mark.parametrize( ("blockstamp", "expected_result"), [ + (simple_ref_blockstamp(50), 98001157445), (simple_ref_blockstamp(40), 98001157445), (simple_ref_blockstamp(20), 77999899300), ], diff --git a/tests/modules/accounting/bunker/test_bunker_medterm_penalty.py b/tests/modules/accounting/bunker/test_bunker_medterm_penalty.py deleted file mode 100644 index a7fc0a4c4..000000000 --- a/tests/modules/accounting/bunker/test_bunker_medterm_penalty.py +++ /dev/null @@ -1,459 +0,0 @@ -import pytest - -from src.modules.submodules.consensus import FrameConfig -from src.modules.submodules.types import ChainConfig -from src.providers.consensus.types import Validator, ValidatorStatus, ValidatorState -from src.services.bunker_cases.midterm_slashing_penalty import MidtermSlashingPenalty -from src.types import EpochNumber, ReferenceBlockStamp - - -def simple_blockstamp( - block_number: int, -) -> ReferenceBlockStamp: - return ReferenceBlockStamp(f"0x{block_number}", block_number, '', block_number, 0, block_number, block_number // 32) - - -def simple_validators( - from_index: int, - to_index: int, - slashed=False, - withdrawable_epoch="8192", - exit_epoch="7892", - effective_balance=str(32 * 10**9), -) -> list[Validator]: - validators = [] - for index in range(from_index, to_index + 1): - validator = Validator( - index=str(index), - balance=effective_balance, - status=ValidatorStatus.ACTIVE_ONGOING, - validator=ValidatorState( - pubkey=f"0x{index}", - withdrawal_credentials='', - effective_balance=str(32 * 10**9), - slashed=slashed, - activation_eligibility_epoch='', - activation_epoch='0', - exit_epoch=exit_epoch, - withdrawable_epoch=withdrawable_epoch, - ), - ) - validators.append(validator) - return validators - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("blockstamp", "all_validators", "lido_validators", "report_cl_rebase", "expected_result"), - [ - ( - # no one slashed - simple_blockstamp(0), - simple_validators(0, 50), - simple_validators(0, 9), - 0, - False, - ), - ( - # no one Lido slashed - simple_blockstamp(0), - [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], - simple_validators(0, 9), - 0, - False, - ), - ( - # Lido slashed, but midterm penalty is not in the future - simple_blockstamp(1500000), - [ - *simple_validators(0, 49), - *simple_validators(50, 99, slashed=True, exit_epoch="16084", withdrawable_epoch="16384"), - ], - simple_validators(50, 99, slashed=True, exit_epoch="16084", withdrawable_epoch="16384"), - 0, - False, - ), - ( - # one day since last report, penalty greater than report rebase - simple_blockstamp(225 * 32), - [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], - simple_validators(50, 99, slashed=True), - 49 * 32 * 10**9, - True, - ), - ( - # three days since last report, penalty greater than frame rebase - simple_blockstamp(3 * 225 * 32), - [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], - simple_validators(50, 99, slashed=True), - 3 * 49 * 32 * 10**9, - True, - ), - ( - # one day since last report,penalty equal report rebase - simple_blockstamp(225 * 32), - [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], - simple_validators(50, 99, slashed=True), - 50 * 32 * 10**9, - False, - ), - ( - # one day since last report, penalty less report rebase - simple_blockstamp(225 * 32), - [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], - simple_validators(50, 99, slashed=True), - 51 * 32 * 10**9, - False, - ), - ], -) -def test_is_high_midterm_slashing_penalty( - blockstamp, all_validators, lido_validators, report_cl_rebase, expected_result -): - chain_config = ChainConfig( - slots_per_epoch=32, - seconds_per_slot=12, - genesis_time=0, - ) - frame_config = FrameConfig( - initial_epoch=EpochNumber(0), - epochs_per_frame=EpochNumber(225), - fast_lane_length_slots=0, - ) - - result = MidtermSlashingPenalty.is_high_midterm_slashing_penalty( - blockstamp, frame_config, chain_config, all_validators, lido_validators, report_cl_rebase, 0 - ) - assert result == expected_result - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("validator", "ref_epoch", "expected_result"), - [ - # slashing epoch is first epoch and it's determined - (simple_validators(0, 0, slashed=True)[0], EpochNumber(225), [0]), - # slashing epoch is not first epoch and it's determined - ( - simple_validators(0, 0, slashed=True, exit_epoch="16084", withdrawable_epoch="16384")[0], - EpochNumber(225), - [8192], - ), - # slashing epoch is not determined - ( - simple_validators(0, 0, slashed=True, exit_epoch="16380", withdrawable_epoch="16384")[0], - EpochNumber(225), - list(range(226)), - ), - # slashing epoch is not determined and ref epoch is not last epoch in first frame - ( - simple_validators(0, 0, slashed=True, exit_epoch="16380", withdrawable_epoch="16384")[0], - EpochNumber(16000), - list(range(7808, 8193)), - ), - ], -) -def test_get_possible_slashed_epochs(validator, ref_epoch, expected_result): - result = MidtermSlashingPenalty.get_possible_slashed_epochs(validator, ref_epoch) - - assert result == expected_result - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("ref_epoch", "future_midterm_penalty_lido_slashed_validators", "expected_result"), - [ - (225, {}, {}), - ( - # the same midterm epoch - 225, - simple_validators(0, 9, slashed=True), - {18: simple_validators(0, 9, slashed=True)}, - ), - ( - # midterm frames in past - 100500, - simple_validators(0, 9, slashed=True), - {}, - ), - ( - # different midterm epochs in different frames - 225, - [ - *simple_validators(0, 9, slashed=True), - *simple_validators(10, 59, slashed=True, withdrawable_epoch="8417"), - ], - { - 18: simple_validators(0, 9, slashed=True), - 19: simple_validators(10, 59, slashed=True, withdrawable_epoch="8417"), - }, - ), - ], -) -def test_get_per_frame_lido_validators_with_future_midterm_epoch( - ref_epoch, future_midterm_penalty_lido_slashed_validators, expected_result -): - frame_config = FrameConfig( - initial_epoch=EpochNumber(0), - epochs_per_frame=EpochNumber(225), - fast_lane_length_slots=0, - ) - - result = MidtermSlashingPenalty.get_lido_validators_with_future_midterm_epoch( - EpochNumber(ref_epoch), - frame_config, - future_midterm_penalty_lido_slashed_validators, - ) - - assert result == expected_result - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("ref_epoch", "per_frame_validators", "all_slashed_validators", "active_validators_count", "expected_result"), - [ - (225, {}, [], 100, {}), - ( - # one is slashed - 225, - {18: simple_validators(0, 0, slashed=True)}, - simple_validators(0, 0, slashed=True), - 100, - {18: 0}, - ), - ( - # all are slashed - 225, - {18: simple_validators(0, 99, slashed=True)}, - simple_validators(0, 99, slashed=True), - 100, - {18: 100 * 32 * 10**9}, - ), - ( - # slashed in different frames with determined slashing epochs - 225, - { - 18: simple_validators(0, 9, slashed=True), - 19: simple_validators(10, 59, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), - }, - [ - *simple_validators(0, 9, slashed=True), - *simple_validators(10, 59, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), - ], - 100, - {18: 10 * 32 * 10**9, 19: 50 * 32 * 10**9}, - ), - ( - # slashed in different epochs in different frames without determined shasling epochs - 225, - { - 18: [ - *simple_validators(0, 5), - *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), - ], - 19: [ - *simple_validators(10, 29, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), - *simple_validators(30, 59, slashed=True, exit_epoch="8417", withdrawable_epoch="8419"), - ], - }, - [ - *simple_validators(0, 5), - *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), - *simple_validators(10, 29, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), - *simple_validators(30, 59, slashed=True, exit_epoch="8417", withdrawable_epoch="8419"), - ], - 100, - {18: 10 * 32 * 10**9, 19: 50 * 32 * 10**9}, - ), - ], -) -def test_get_future_midterm_penalty_sum_in_frames( - ref_epoch, per_frame_validators, all_slashed_validators, active_validators_count, expected_result -): - result = MidtermSlashingPenalty.get_future_midterm_penalty_sum_in_frames( - EpochNumber(ref_epoch), all_slashed_validators, active_validators_count * 32 * 10**9, per_frame_validators - ) - - assert result == expected_result - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("ref_epoch", "all_slashed_validators", "total_balance", "validators_in_frame", "expected_result"), - [ - (225, [], 100 * 32 * 10**9, [], 0), - ( - # one is slashed - 225, - simple_validators(0, 0, slashed=True), - 100 * 32 * 10**9, - simple_validators(0, 0, slashed=True), - 0, - ), - ( - # all are slashed - 225, - simple_validators(0, 99, slashed=True), - 100 * 32 * 10**9, - simple_validators(0, 99, slashed=True), - 100 * 32 * 10**9, - ), - ( - # several are slashed - 225, - simple_validators(0, 9, slashed=True), - 100 * 32 * 10**9, - simple_validators(0, 9, slashed=True), - 10 * 9 * 10**9, - ), - ( - # slashed in different epochs in different frames without determined slashing epochs - 225, - [ - *simple_validators(0, 5, slashed=True), - *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), - ], - 100 * 32 * 10**9, - [ - *simple_validators(0, 5, slashed=True), - *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), - ], - 10 * 9 * 10**9, - ), - ], -) -def test_predict_midterm_penalty_in_frame( - ref_epoch, all_slashed_validators, total_balance, validators_in_frame, expected_result -): - result = MidtermSlashingPenalty.predict_midterm_penalty_in_frame( - EpochNumber(ref_epoch), all_slashed_validators, total_balance, validators_in_frame - ) - - assert result == expected_result - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("bounded_slashings_count", "active_validators_count", "expected_penalty"), - [ - (1, 500000, 0), - (100, 500000, 0), - (1000, 500000, 0), - (5000, 500000, 0), - (10000, 500000, 1000000000), - (20000, 500000, 3000000000), - (50000, 500000, 9000000000), - ], -) -def test_get_midterm_penalty(bounded_slashings_count, active_validators_count, expected_penalty): - result = MidtermSlashingPenalty.get_validator_midterm_penalty( - simple_validators(0, 0)[0], bounded_slashings_count, active_validators_count * 32 * 10**9 - ) - - assert result == expected_penalty - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("ref_epoch", "slashed_validators", "midterm_penalty_epoch", "expected_bounded"), - [ - ( - # slashing epoch is determined - 225, - simple_validators(0, 9), - 4096, - simple_validators(0, 9), - ), - ( - # slashing epoch is not determined - EpochNumber(16000), - simple_validators(0, 0, exit_epoch="16380", withdrawable_epoch="16384"), - 12288, - simple_validators(0, 0, exit_epoch="16380", withdrawable_epoch="16384"), - ), - ], -) -def test_get_bound_with_midterm_epoch_slashed_validators( - ref_epoch, slashed_validators, midterm_penalty_epoch, expected_bounded -): - result = MidtermSlashingPenalty.get_bound_with_midterm_epoch_slashed_validators( - EpochNumber(ref_epoch), slashed_validators, EpochNumber(midterm_penalty_epoch) - ) - - assert result == expected_bounded - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("lido_validators", "ref_epoch", "expected_len"), - [ - ( - # no one slashed - simple_validators(0, 9), - EpochNumber(20000000), - 0, - ), - ( - # slashed and withdrawable epoch greater than ref_epoch - simple_validators(0, 9, slashed=True), - EpochNumber(0), - 10, - ), - ( - # slashed and withdrawable epoch less than ref_epoch - simple_validators(0, 9, slashed=True), - EpochNumber(20000000), - 0, - ), - ], -) -def test_get_slashed_validators_with_impact_to_midterm_penalties(lido_validators, ref_epoch, expected_len): - result = MidtermSlashingPenalty.get_slashed_validators_with_impact_on_midterm_penalties(lido_validators, ref_epoch) - assert len(result) == expected_len - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("report_cl_rebase", "blockstamp", "last_report_ref_slot", "expected_result"), - [ - (5 * 32 * 10**9, simple_blockstamp(225 * 32), 0, 5 * 32 * 10**9), - (7 * 5 * 32 * 10**9, simple_blockstamp(7 * 225 * 32), 0, 5 * 32 * 10**9), - ], -) -def test_get_frame_cl_rebase_from_report_cl_rebase(report_cl_rebase, blockstamp, last_report_ref_slot, expected_result): - chain_config = ChainConfig( - slots_per_epoch=32, - seconds_per_slot=12, - genesis_time=0, - ) - frame_config = FrameConfig( - initial_epoch=EpochNumber(0), - epochs_per_frame=EpochNumber(225), - fast_lane_length_slots=0, - ) - result = MidtermSlashingPenalty.get_frame_cl_rebase_from_report_cl_rebase( - frame_config, chain_config, report_cl_rebase, blockstamp, last_report_ref_slot - ) - - assert result == expected_result - - -@pytest.mark.unit -@pytest.mark.parametrize( - ("epoch", "expected_frame"), - [(EpochNumber(0), 0), (EpochNumber(224), 0), (EpochNumber(225), 1), (EpochNumber(449), 1), (EpochNumber(450), 2)], -) -def test_get_frame_by_epoch(epoch, expected_frame): - frame_config = FrameConfig( - initial_epoch=EpochNumber(0), - epochs_per_frame=EpochNumber(225), - fast_lane_length_slots=0, - ) - frame_by_epoch = MidtermSlashingPenalty.get_frame_by_epoch(epoch, frame_config) - assert frame_by_epoch == expected_frame - - -@pytest.mark.unit -def test_get_midterm_slashing_epoch(): - result = MidtermSlashingPenalty.get_midterm_penalty_epoch(simple_validators(0, 0)[0]) - assert result == 4096 diff --git a/tests/modules/accounting/bunker/test_bunker_midterm_penalty.py b/tests/modules/accounting/bunker/test_bunker_midterm_penalty.py new file mode 100644 index 000000000..05eef9319 --- /dev/null +++ b/tests/modules/accounting/bunker/test_bunker_midterm_penalty.py @@ -0,0 +1,847 @@ +from unittest.mock import Mock + +import pytest + +from src.constants import MAX_EFFECTIVE_BALANCE_ELECTRA, MAX_EFFECTIVE_BALANCE +from src.modules.submodules.consensus import FrameConfig +from src.modules.submodules.types import ChainConfig +from src.providers.consensus.types import Validator, ValidatorStatus, ValidatorState +from src.services.bunker_cases.midterm_slashing_penalty import MidtermSlashingPenalty +from src.types import EpochNumber, ReferenceBlockStamp, Gwei +from src.utils.web3converter import Web3Converter + + +def simple_blockstamp( + block_number: int, +) -> ReferenceBlockStamp: + return ReferenceBlockStamp(f"0x{block_number}", block_number, '', block_number, 0, block_number, block_number // 32) + + +def simple_validators( + from_index: int, + to_index: int, + slashed=False, + withdrawable_epoch="8192", + exit_epoch="7892", + effective_balance=str(32 * 10**9), +) -> list[Validator]: + validators = [] + for index in range(from_index, to_index + 1): + validator = Validator( + index=str(index), + balance=effective_balance, + status=ValidatorStatus.ACTIVE_ONGOING, + validator=ValidatorState( + pubkey=f"0x{index}", + withdrawal_credentials='', + effective_balance=effective_balance, + slashed=slashed, + activation_eligibility_epoch='', + activation_epoch='0', + exit_epoch=exit_epoch, + withdrawable_epoch=withdrawable_epoch, + ), + ) + validators.append(validator) + return validators + + +TEST_ELECTRA_FORK_EPOCH = 450 + + +@pytest.fixture(params=[TEST_ELECTRA_FORK_EPOCH]) +def spec_with_electra(request): + # sets the electra fork epoch to the test value for calculating the penalty + return Mock(ELECTRA_FORK_EPOCH=request.param) + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("blockstamp", "all_validators", "lido_validators", "report_cl_rebase", "expected_result"), + [ + ( + # no one slashed + simple_blockstamp(0), + simple_validators(0, 50), + simple_validators(0, 9), + 0, + False, + ), + ( + # no one Lido slashed + simple_blockstamp(0), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(0, 9), + 0, + False, + ), + ( + # Lido slashed, but midterm penalty is not in the future + simple_blockstamp(1500000), + [ + *simple_validators(0, 49), + *simple_validators(50, 99, slashed=True, exit_epoch="16084", withdrawable_epoch="16384"), + ], + simple_validators(50, 99, slashed=True, exit_epoch="16084", withdrawable_epoch="16384"), + 0, + False, + ), + ( + # one day since last report, penalty greater than report rebase + simple_blockstamp(225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 49 * 32 * 10**9, + True, + ), + ( + # three days since last report, penalty greater than frame rebase + simple_blockstamp(3 * 225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 3 * 49 * 32 * 10**9, + True, + ), + ( + # one day since last report,penalty equal report rebase + simple_blockstamp(225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 50 * 32 * 10**9, + False, + ), + ( + # one day since last report, penalty less report rebase + simple_blockstamp(225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 51 * 32 * 10**9, + False, + ), + ], +) +def test_is_high_midterm_slashing_penalty_pre_electra( + blockstamp, all_validators, lido_validators, report_cl_rebase, expected_result +): + cl_spec = Mock() + chain_config = ChainConfig( + slots_per_epoch=32, + seconds_per_slot=12, + genesis_time=0, + ) + frame_config = FrameConfig( + initial_epoch=EpochNumber(0), + epochs_per_frame=EpochNumber(225), + fast_lane_length_slots=0, + ) + web3_converter = Web3Converter(chain_config, frame_config) + + result = MidtermSlashingPenalty.is_high_midterm_slashing_penalty( + blockstamp, 2, cl_spec, web3_converter, all_validators, lido_validators, report_cl_rebase, 0 + ) + assert result == expected_result + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("blockstamp", "all_validators", "lido_validators", "report_cl_rebase", "expected_result"), + [ + ( + # no one slashed + simple_blockstamp(0), + simple_validators(0, 50), + simple_validators(0, 9), + 0, + False, + ), + ( + # no one Lido slashed + simple_blockstamp(0), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(0, 9), + 0, + False, + ), + ( + # Lido slashed, but midterm penalty is not in the future + simple_blockstamp(1500000), + [ + *simple_validators(0, 49), + *simple_validators(50, 99, slashed=True, exit_epoch="16084", withdrawable_epoch="16384"), + ], + simple_validators(50, 99, slashed=True, exit_epoch="16084", withdrawable_epoch="16384"), + 0, + False, + ), + ( + # one day since last report, penalty greater than report rebase + simple_blockstamp(225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 49 * 32 * 10**9, + True, + ), + ( + # three days since last report, penalty greater than frame rebase + simple_blockstamp(3 * 225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 3 * 49 * 32 * 10**9, + True, + ), + ( + # one day since last report,penalty equal report rebase + simple_blockstamp(225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 50 * 32 * 10**9, + False, + ), + ( + # one day since last report, penalty less report rebase + simple_blockstamp(225 * 32), + [*simple_validators(0, 49), *simple_validators(50, 99, slashed=True)], + simple_validators(50, 99, slashed=True), + 51 * 32 * 10**9, + False, + ), + ], +) +def test_is_high_midterm_slashing_penalty_post_electra( + blockstamp, spec_with_electra, all_validators, lido_validators, report_cl_rebase, expected_result +): + chain_config = ChainConfig( + slots_per_epoch=32, + seconds_per_slot=12, + genesis_time=0, + ) + frame_config = FrameConfig( + initial_epoch=EpochNumber(0), + epochs_per_frame=EpochNumber(225), + fast_lane_length_slots=0, + ) + web3_converter = Web3Converter(chain_config, frame_config) + result = MidtermSlashingPenalty.is_high_midterm_slashing_penalty( + blockstamp, + 3, + spec_with_electra, + web3_converter, + all_validators, + lido_validators, + report_cl_rebase, + 0, + ) + assert result == expected_result + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("validator", "ref_epoch", "expected_result"), + [ + # slashing epoch is first epoch and it's determined + (simple_validators(0, 0, slashed=True)[0], EpochNumber(225), [0]), + # slashing epoch is not first epoch and it's determined + ( + simple_validators(0, 0, slashed=True, exit_epoch="16084", withdrawable_epoch="16384")[0], + EpochNumber(225), + [8192], + ), + # slashing epoch is not determined + ( + simple_validators(0, 0, slashed=True, exit_epoch="16380", withdrawable_epoch="16384")[0], + EpochNumber(225), + list(range(226)), + ), + # slashing epoch is not determined and ref epoch is not last epoch in first frame + ( + simple_validators(0, 0, slashed=True, exit_epoch="16380", withdrawable_epoch="16384")[0], + EpochNumber(16000), + list(range(7808, 8193)), + ), + ], +) +def test_get_possible_slashed_epochs(validator, spec_with_electra, ref_epoch, expected_result): + result = MidtermSlashingPenalty.get_possible_slashed_epochs(validator, ref_epoch) + + assert result == expected_result + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("ref_epoch", "future_midterm_penalty_lido_slashed_validators", "expected_result"), + [ + (225, {}, {}), + ( + # the same midterm epoch + 225, + simple_validators(0, 9, slashed=True), + {(18, 4049): simple_validators(0, 9, slashed=True)}, + ), + ( + # midterm frames in past + 100500, + simple_validators(0, 9, slashed=True), + {}, + ), + ( + # different midterm epochs in different frames + 225, + [ + *simple_validators(0, 9, slashed=True), + *simple_validators(10, 59, slashed=True, withdrawable_epoch="8417"), + ], + { + (18, 4049): simple_validators(0, 9, slashed=True), + (19, 4274): simple_validators(10, 59, slashed=True, withdrawable_epoch="8417"), + }, + ), + ], +) +def test_get_per_frame_lido_validators_with_future_midterm_epoch( + ref_epoch, spec_with_electra, future_midterm_penalty_lido_slashed_validators, expected_result +): + chain_config = ChainConfig( + slots_per_epoch=32, + seconds_per_slot=12, + genesis_time=0, + ) + frame_config = FrameConfig( + initial_epoch=EpochNumber(0), + epochs_per_frame=EpochNumber(225), + fast_lane_length_slots=0, + ) + web3_converter = Web3Converter(chain_config, frame_config) + + result = MidtermSlashingPenalty.get_lido_validators_with_future_midterm_epoch( + EpochNumber(ref_epoch), + web3_converter, + future_midterm_penalty_lido_slashed_validators, + ) + + assert result == expected_result + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("ref_epoch", "per_frame_validators", "all_slashed_validators", "active_validators_count", "expected_result"), + [ + (225, {}, [], 100, {}), + ( + # one is slashed + 225, + {(18, 4050): simple_validators(0, 0, slashed=True)}, + simple_validators(0, 0, slashed=True), + 100, + {18: 0}, + ), + ( + # all are slashed + 225, + {(18, 4050): simple_validators(0, 99, slashed=True)}, + simple_validators(0, 99, slashed=True), + 100, + {18: 100 * 32 * 10**9}, + ), + ( + # slashed in different frames with determined slashing epochs + 225, + { + (18, 4050): simple_validators(0, 9, slashed=True), + (19, 4725): simple_validators(10, 59, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + }, + [ + *simple_validators(0, 9, slashed=True), + *simple_validators(10, 59, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + ], + 100, + {18: 10 * 32 * 10**9, 19: 50 * 32 * 10**9}, + ), + ( + # slashed in different epochs in different frames without determined shasling epochs + 225, + { + (18, 4050): [ + *simple_validators(0, 5), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + (19, 4725): [ + *simple_validators(10, 29, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + *simple_validators(30, 59, slashed=True, exit_epoch="8417", withdrawable_epoch="8419"), + ], + }, + [ + *simple_validators(0, 5), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + *simple_validators(10, 29, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + *simple_validators(30, 59, slashed=True, exit_epoch="8417", withdrawable_epoch="8419"), + ], + 100, + {18: 10 * 32 * 10**9, 19: 50 * 32 * 10**9}, + ), + ], +) +def test_get_future_midterm_penalty_sum_in_frames_pre_electra( + ref_epoch, per_frame_validators, all_slashed_validators, active_validators_count, expected_result +): + result = MidtermSlashingPenalty.get_future_midterm_penalty_sum_in_frames_pre_electra( + EpochNumber(ref_epoch), all_slashed_validators, active_validators_count * 32 * 10**9, per_frame_validators + ) + + assert result == expected_result + + +@pytest.mark.unit +@pytest.mark.parametrize( + ( + "ref_epoch", + "spec_with_electra", + "per_frame_validators", + "all_slashed_validators", + "active_validators_count", + "expected_result", + ), + [ + (225, 225, {}, [], 50000, {}), + ( + # one is slashed before electra + 225, + 4500, + {(18, 4049): simple_validators(0, 0, slashed=True)}, + simple_validators(0, 0, slashed=True), + 50000, + {18: 0}, + ), + ( + # one is slashed after electra + 225, + 225, + {(18, 4049): simple_validators(0, 0, slashed=True)}, + simple_validators(0, 0, slashed=True), + 50000, + {18: 1_920_000}, + ), + ( + # all are slashed before electra + 225, + 4500, + {(18, 4049): simple_validators(0, 99, slashed=True)}, + simple_validators(0, 99, slashed=True), + 50000, + {18: 0}, + ), + ( + # all are slashed after electra + 225, + 225, + {(18, 4049): simple_validators(0, 99, slashed=True)}, + simple_validators(0, 99, slashed=True), + 50000, + {18: 19_200_000_000}, + ), + ( + # slashed in different frames with determined slashing epochs in different forks + 225, + 4500, + { + (18, 4049): simple_validators(0, 0, slashed=True), + (19, 4724): simple_validators(10, 59, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + }, + [ + *simple_validators(0, 0, slashed=True), + *simple_validators(10, 59, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + ], + 50000, + {18: 0, 19: 4_896_000_000}, + ), + ( + # slashed in different epochs in different frames without determined slashing epochs in different forks + 225, + 4500, + { + (18, 4049): [ + *simple_validators(0, 5), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + (19, 4724): [ + *simple_validators(10, 29, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + *simple_validators(30, 59, slashed=True, exit_epoch="8417", withdrawable_epoch="8419"), + ], + }, + [ + *simple_validators(0, 5), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + *simple_validators(10, 29, slashed=True, exit_epoch="8000", withdrawable_epoch="8417"), + *simple_validators(30, 59, slashed=True, exit_epoch="8417", withdrawable_epoch="8419"), + ], + 50000, + {18: 0, 19: 5_760_000_000}, + ), + ], + indirect=["spec_with_electra"], +) +def test_get_future_midterm_penalty_sum_in_frames_post_electra( + ref_epoch, + spec_with_electra, + per_frame_validators, + all_slashed_validators, + active_validators_count, + expected_result, +): + result = MidtermSlashingPenalty.get_future_midterm_penalty_sum_in_frames_post_electra( + EpochNumber(ref_epoch), + spec_with_electra, + all_slashed_validators, + active_validators_count * 32 * 10**9, + per_frame_validators, + ) + + assert result == expected_result + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("ref_epoch", "all_slashed_validators", "total_balance", "validators_in_frame", "expected_result"), + [ + (225, [], 100 * 32 * 10**9, [], 0), + ( + # one is slashed + 225, + simple_validators(0, 0, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 0, slashed=True), + 0, + ), + ( + # all are slashed + 225, + simple_validators(0, 99, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 99, slashed=True), + 100 * 32 * 10**9, + ), + ( + # several are slashed + 225, + simple_validators(0, 9, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 9, slashed=True), + 10 * 9 * 10**9, + ), + ( + # slashed in different epochs in different frames without determined slashing epochs + 225, + [ + *simple_validators(0, 5, slashed=True), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + 100 * 32 * 10**9, + [ + *simple_validators(0, 5, slashed=True), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + 10 * 9 * 10**9, + ), + ], +) +def test_predict_midterm_penalty_in_frame_pre_electra( + ref_epoch, all_slashed_validators, total_balance, validators_in_frame, expected_result +): + result = MidtermSlashingPenalty.predict_midterm_penalty_in_frame_pre_electra( + EpochNumber(ref_epoch), all_slashed_validators, total_balance, validators_in_frame + ) + + assert result == expected_result + + +@pytest.mark.unit +@pytest.mark.parametrize( + ( + "ref_epoch", + "is_after_electra", + "all_slashed_validators", + "total_balance", + "validators_in_frame", + "expected_result", + ), + [ + # BEFORE ELECTRA + (225, False, [], 100 * 32 * 10**9, [], 0), + ( + # one is slashed + 225, + False, + simple_validators(0, 0, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 0, slashed=True), + 0, + ), + ( + # all are slashed + 225, + False, + simple_validators(0, 99, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 99, slashed=True), + 100 * 32 * 10**9, + ), + ( + # several are slashed + 225, + False, + simple_validators(0, 9, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 9, slashed=True), + 10 * 9 * 10**9, + ), + ( + # slashed in different epochs in different frames without determined slashing epochs + 225, + False, + [ + *simple_validators(0, 5, slashed=True), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + 100 * 32 * 10**9, + [ + *simple_validators(0, 5, slashed=True), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + 10 * 9 * 10**9, + ), + # AFTER ELECTRA + (225, True, [], 100 * 32 * 10**9, [], 0), + ( + # one is slashed + 225, + True, + simple_validators(0, 0, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 0, slashed=True), + 960_000_000, + ), + ( + # all are slashed + 225, + True, + simple_validators(0, 99, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 99, slashed=True), + 100 * 32 * 10**9, + ), + ( + # several are slashed + 225, + True, + simple_validators(0, 9, slashed=True), + 100 * 32 * 10**9, + simple_validators(0, 9, slashed=True), + 96_000_000_000, + ), + ( + # slashed in different epochs in different frames without determined slashing epochs + 225, + True, + [ + *simple_validators(0, 5, slashed=True), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + 100 * 32 * 10**9, + [ + *simple_validators(0, 5, slashed=True), + *simple_validators(6, 9, slashed=True, exit_epoch="8192", withdrawable_epoch="8197"), + ], + 96_000_000_000, + ), + ], +) +def test_predict_midterm_penalty_in_frame_post_electra( + ref_epoch, + is_after_electra, + all_slashed_validators, + total_balance, + validators_in_frame, + expected_result, + spec_with_electra, +): + result = MidtermSlashingPenalty.predict_midterm_penalty_in_frame_post_electra( + report_ref_epoch=EpochNumber(ref_epoch), + frame_ref_epoch=EpochNumber( + spec_with_electra.ELECTRA_FORK_EPOCH if is_after_electra else spec_with_electra.ELECTRA_FORK_EPOCH - 1 + ), + cl_spec=spec_with_electra, + all_slashed_validators=all_slashed_validators, + total_balance=total_balance, + midterm_penalized_validators_in_frame=validators_in_frame, + ) + + assert result == expected_result + + +# 50% active validators with 2048 EB and the rest part with 32 EB +half_electra = [ + *simple_validators(0, 250_000, effective_balance=str(MAX_EFFECTIVE_BALANCE)), + *simple_validators(250_001, 500_000, effective_balance=str(MAX_EFFECTIVE_BALANCE_ELECTRA)), +] +# 20% active validators with 2048 EB and the rest part with 32 EB +part_electra = [ + *simple_validators(0, 10_000, effective_balance=str(MAX_EFFECTIVE_BALANCE_ELECTRA)), + *simple_validators(10_001, 500_000, effective_balance=str(MAX_EFFECTIVE_BALANCE)), +] + +one_32eth = simple_validators(0, 0, effective_balance=str(MAX_EFFECTIVE_BALANCE)) +one_2048eth = simple_validators(0, 0, effective_balance=str(MAX_EFFECTIVE_BALANCE_ELECTRA)) + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("bounded_slashed_validators", "active_validators", "expected_penalty"), + [ + (one_32eth, half_electra, 5888), + (one_2048eth, half_electra, 378_080), + (one_32eth, part_electra, 84_928), + (one_2048eth, part_electra, 5_436_832), + (100 * one_32eth, half_electra, 590_752), + (100 * one_2048eth, half_electra, 37_809_216), + (100 * one_32eth, part_electra, 8_495_072), + (100 * one_2048eth, part_electra, 543_686_016), + (10_000 * one_32eth, half_electra, 59_076_896), + (10_000 * one_2048eth, half_electra, 3_780_922_816), + (10_000 * one_32eth, part_electra, 849_509_408), + (10_000 * one_2048eth, part_electra, 32_000_000_000), + ], + ids=[ + "1 bounded slashing with 32 EB, half active validators with 2048 EB and the rest part with 32 EB", + "1 bounded slashing with 2048 EB, half active validators with 2048 EB and the rest part with 32 EB", + "1 bounded slashing with 32 EB, 10% active validators with 2048 EB and the rest part with 32 EB", + "1 bounded slashing with 2048 EB, 10% active validators with 2048 EB and the rest part with 32 EB", + "100 bounded slashing with 32 EB, half active validators with 2048 EB and the rest part with 32 EB", + "100 bounded slashing with 2048 EB, half active validators with 2048 EB and the rest part with 32 EB", + "100 bounded slashing with 32 EB, 10% active validators with 2048 EB and the rest part with 32 EB", + "100 bounded slashing with 2048 EB, 10% active validators with 2048 EB and the rest part with 32 EB", + "10_000 bounded slashing with 32 EB, half active validators with 2048 EB and the rest part with 32 EB", + "10_000 bounded slashing with 2048 EB, half active validators with 2048 EB and the rest part with 32 EB", + "10_000 bounded slashing with 32 EB, 10% active validators with 2048 EB and the rest part with 32 EB", + "10_000 bounded slashing with 2048 EB, 10% active validators with 2048 EB and the rest part with 32 EB", + ], +) +def test_get_validator_midterm_penalty_electra(bounded_slashed_validators, active_validators, expected_penalty): + result = MidtermSlashingPenalty.get_validator_midterm_penalty_electra( + validator=simple_validators(0, 0)[0], + bound_slashed_validators=bounded_slashed_validators, + total_balance=Gwei(sum(int(v.validator.effective_balance) for v in active_validators)), + ) + + assert result == expected_penalty + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("bounded_slashings_count", "active_validators_count", "expected_penalty"), + [ + (1, 500000, 0), + (100, 500000, 0), + (1000, 500000, 0), + (5000, 500000, 0), + (10000, 500000, 1000000000), + (20000, 500000, 3000000000), + (50000, 500000, 9000000000), + ], +) +def test_get_validator_midterm_penalty(bounded_slashings_count, active_validators_count, expected_penalty): + result = MidtermSlashingPenalty.get_validator_midterm_penalty( + simple_validators(0, 0)[0], bounded_slashings_count, active_validators_count * 32 * 10**9 + ) + + assert result == expected_penalty + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("ref_epoch", "slashed_validators", "midterm_penalty_epoch", "expected_bounded"), + [ + ( + # slashing epoch is determined + 225, + simple_validators(0, 9), + 4096, + simple_validators(0, 9), + ), + ( + # slashing epoch is not determined + EpochNumber(16000), + simple_validators(0, 0, exit_epoch="16380", withdrawable_epoch="16384"), + 12288, + simple_validators(0, 0, exit_epoch="16380", withdrawable_epoch="16384"), + ), + ], +) +def test_get_bound_with_midterm_epoch_slashed_validators( + ref_epoch, slashed_validators, midterm_penalty_epoch, expected_bounded +): + result = MidtermSlashingPenalty.get_bound_with_midterm_epoch_slashed_validators( + EpochNumber(ref_epoch), slashed_validators, EpochNumber(midterm_penalty_epoch) + ) + + assert result == expected_bounded + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("lido_validators", "ref_epoch", "expected_len"), + [ + ( + # no one slashed + simple_validators(0, 9), + EpochNumber(20000000), + 0, + ), + ( + # slashed and withdrawable epoch greater than ref_epoch + simple_validators(0, 9, slashed=True), + EpochNumber(0), + 10, + ), + ( + # slashed and withdrawable epoch less than ref_epoch + simple_validators(0, 9, slashed=True), + EpochNumber(20000000), + 0, + ), + ], +) +def test_get_slashed_validators_with_impact_to_midterm_penalties(lido_validators, ref_epoch, expected_len): + result = MidtermSlashingPenalty.get_slashed_validators_with_impact_on_midterm_penalties(lido_validators, ref_epoch) + assert len(result) == expected_len + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("report_cl_rebase", "blockstamp", "last_report_ref_slot", "expected_result"), + [ + (5 * 32 * 10**9, simple_blockstamp(225 * 32), 0, 5 * 32 * 10**9), + (7 * 5 * 32 * 10**9, simple_blockstamp(7 * 225 * 32), 0, 5 * 32 * 10**9), + ], +) +def test_get_frame_cl_rebase_from_report_cl_rebase(report_cl_rebase, blockstamp, last_report_ref_slot, expected_result): + chain_config = ChainConfig( + slots_per_epoch=32, + seconds_per_slot=12, + genesis_time=0, + ) + frame_config = FrameConfig( + initial_epoch=EpochNumber(0), + epochs_per_frame=EpochNumber(225), + fast_lane_length_slots=0, + ) + web3_converter = Web3Converter(chain_config, frame_config) + result = MidtermSlashingPenalty.get_frame_cl_rebase_from_report_cl_rebase( + web3_converter, report_cl_rebase, blockstamp, last_report_ref_slot + ) + + assert result == expected_result + + +@pytest.mark.unit +def test_get_midterm_slashing_epoch(): + result = MidtermSlashingPenalty.get_midterm_penalty_epoch(simple_validators(0, 0)[0]) + assert result == 4096 diff --git a/tests/modules/accounting/test_accounting_module.py b/tests/modules/accounting/test_accounting_module.py index c0124b555..a69effa17 100644 --- a/tests/modules/accounting/test_accounting_module.py +++ b/tests/modules/accounting/test_accounting_module.py @@ -6,6 +6,7 @@ from web3.types import Wei from src import variables +from src.constants import LIDO_DEPOSIT_AMOUNT from src.modules.accounting import accounting as accounting_module from src.modules.accounting.accounting import Accounting from src.modules.accounting.accounting import logger as accounting_logger @@ -21,7 +22,6 @@ from tests.factory.configs import ChainConfigFactory, FrameConfigFactory from tests.factory.contract_responses import LidoReportRebaseFactory from tests.factory.no_registry import LidoValidatorFactory, StakingModuleFactory -from tests.web3_extentions.test_lido_validators import blockstamp @pytest.fixture(autouse=True) @@ -101,13 +101,18 @@ def test_get_updated_modules_stats(accounting: Accounting): @pytest.mark.usefixtures("lido_validators") def test_get_consensus_lido_state(accounting: Accounting): bs = ReferenceBlockStampFactory.build() - validators = LidoValidatorFactory.batch(10) + validators = [ + *[LidoValidatorFactory.build_pending_deposit_vals() for _ in range(3)], + *[LidoValidatorFactory.build_not_active_vals(bs.ref_epoch) for _ in range(3)], + *[LidoValidatorFactory.build_active_vals(bs.ref_epoch) for _ in range(2)], + *[LidoValidatorFactory.build_exit_vals(bs.ref_epoch) for _ in range(2)], + ] accounting.w3.lido_validators.get_lido_validators = Mock(return_value=validators) count, balance = accounting._get_consensus_lido_state(bs) assert count == 10 - assert balance == sum((int(val.balance) for val in validators)) + assert balance == sum((int(val.balance) for val in validators)) + 3 * LIDO_DEPOSIT_AMOUNT @pytest.mark.unit diff --git a/tests/modules/accounting/test_validator_state.py b/tests/modules/accounting/test_validator_state.py index 5fe11fb59..050154189 100644 --- a/tests/modules/accounting/test_validator_state.py +++ b/tests/modules/accounting/test_validator_state.py @@ -86,7 +86,7 @@ def validator(index: int, exit_epoch: int, pubkey: HexStr, activation_epoch: int lido_id=LidoKey( key=pubkey, depositSignature="", - operatorIndex=-1, + operatorIndex=NodeOperatorId(-1), used=True, moduleAddress="", ), diff --git a/tests/modules/csm/test_checkpoint.py b/tests/modules/csm/test_checkpoint.py index 952ac2390..08bc24a23 100644 --- a/tests/modules/csm/test_checkpoint.py +++ b/tests/modules/csm/test_checkpoint.py @@ -1,5 +1,5 @@ from copy import deepcopy -from typing import Iterator, cast +from typing import cast from unittest.mock import Mock import pytest @@ -16,6 +16,7 @@ from src.providers.consensus.client import ConsensusClient from src.providers.consensus.types import BeaconSpecResponse, BlockAttestation, SlotAttestationCommittee from src.utils.web3converter import Web3Converter +from tests.factory.bitarrays import BitListFactory from tests.factory.configs import ( BeaconSpecResponseFactory, BlockAttestationFactory, @@ -254,12 +255,12 @@ def test_checkpoints_processor_process_attestations(mock_get_attestation_committ attestation = cast(BlockAttestation, BlockAttestationFactory.build()) attestation.data.slot = 0 attestation.data.index = 0 - attestation.aggregation_bits = '0x' + 'f' * 32 + attestation.aggregation_bits = BitListFactory.build(set_indices=[i for i in range(32)]).hex() # the same but with no included attestations in bits attestation2 = cast(BlockAttestation, BlockAttestationFactory.build()) attestation2.data.slot = 0 attestation2.data.index = 0 - attestation2.aggregation_bits = '0x' + '0' * 32 + attestation2.aggregation_bits = BitListFactory.build(set_indices=[]).hex() process_attestations([attestation, attestation2], committees) for index, validators in enumerate(committees.values()): for validator in validators: diff --git a/tests/modules/csm/test_processing_attestation.py b/tests/modules/csm/test_processing_attestation.py new file mode 100644 index 000000000..542f0ea08 --- /dev/null +++ b/tests/modules/csm/test_processing_attestation.py @@ -0,0 +1,199 @@ +from itertools import chain +from unittest.mock import Mock + +import pytest + +from src.modules.csm.checkpoint import ( + get_committee_indices, + hex_bitlist_to_list, + hex_bitvector_to_list, + is_eip7549_attestation, + process_attestations, +) +from src.providers.consensus.types import BlockAttestation + + +@pytest.mark.unit +def test_hex_bitvector_to_list(): + bits = hex_bitvector_to_list("0x00") + assert bits == [False] * 8 + + bits = hex_bitvector_to_list("00") + assert bits == [False] * 8 + + bits = hex_bitvector_to_list("50") + assert bits == [ + # 0 + False, + False, + False, + False, + # 5 little-endian + True, + False, + True, + False, + ] + + bits = hex_bitvector_to_list("0x3174") + assert bits == [ + # 1 little-endian + True, + False, + False, + False, + # 3 little-endian + True, + True, + False, + False, + # 4 little-endian + False, + False, + True, + False, + # 7 little-endian + True, + True, + True, + False, + ] + + +@pytest.mark.unit +def test_hex_bitlist_to_list(): + bits = hex_bitlist_to_list("0x000000000000000000001000000000000010001000000000000000000000000020") + assert len(bits) == 261 + assert [i for (i, v) in enumerate(bits) if v] == [84, 140, 156] + + with pytest.raises(ValueError, match="invalid bitlist"): + hex_bitlist_to_list("0x000000000000000000001000000000000010001000000000000000000000000000") + + bits = hex_bitlist_to_list("0x01") + assert bits == [] + + +@pytest.mark.unit +def test_attested_indices_pre_electra(): + committees = { + ("42", "20"): [Mock(index=20000 + i) for i in range(130)], + ("42", "22"): [Mock(index=22000 + i) for i in range(131)], + } + process_attestations( + [ + Mock( + data=Mock(slot="42", index="20"), + aggregation_bits="0000000000000000000030", + committee_bits=None, + ), + Mock( + data=Mock(slot="42", index="22"), + aggregation_bits="0004000c", + committee_bits=None, + ), + ], + committees, # type: ignore + ) + vals = [v for v in chain(*committees.values()) if v.included is True] + assert [v.index for v in vals] == [20084, 22010, 22026] + + +@pytest.mark.unit +def test_attested_indices_post_electra(): + committees = { + ("42", "20"): [Mock(index=20000 + i) for i in range(130)], + ("42", "22"): [Mock(index=22000 + i) for i in range(131)], + ("17", "12"): [Mock(index=12000 + i) for i in range(999)], + } + process_attestations( + [ + Mock( + data=Mock(slot="42", index="0"), + aggregation_bits="0x000000000000000000001000000000000010001000000000000000000000000020", + committee_bits="0x0000500000000000", + ), + Mock( + data=Mock(slot="17", index="0"), + aggregation_bits="0x0000000000000000000030", + committee_bits="0x0010", + ), + ], + committees, # type: ignore + ) + vals = [v for v in chain(*committees.values()) if v.included is True] + assert [v.index for v in vals] == [20084, 22010, 22026, 12084] + + +@pytest.mark.unit +def test_derive_attestation_version(): + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits=None) + assert not is_eip7549_attestation(att) + + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits="") + assert is_eip7549_attestation(att) + + att: BlockAttestation = Mock(data=Mock(index="1"), aggregation_bits="", committee_bits="") + with pytest.raises(ValueError, match="invalid attestation"): + assert is_eip7549_attestation(att) + + +@pytest.mark.unit +def test_get_committee_indices_pre_electra(): + att: BlockAttestation = Mock( + data=Mock(index="0"), + aggregation_bits="", + committee_bits=None, + ) + assert get_committee_indices(att) == ["0"] + + att: BlockAttestation = Mock( + data=Mock(index="42"), + aggregation_bits="", + committee_bits=None, + ) + assert get_committee_indices(att) == ["42"] + + att: BlockAttestation = Mock( + data=Mock(index="42"), + aggregation_bits="", + committee_bits="0xff", + ) + with pytest.raises(ValueError, match="invalid attestation"): + get_committee_indices(att) + + +@pytest.mark.unit +def test_get_committee_indices_post_electra(): + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits="") + assert get_committee_indices(att) == [] + + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits="0x0000000000000000") + assert get_committee_indices(att) == [] + + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits="0x0100000000000000") + assert get_committee_indices(att) == ["0"] + + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits="0xffffff0000000000") + assert get_committee_indices(att) == [str(n) for n in range(24)] + + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits="0x0000500000000000") + assert get_committee_indices(att) == ["20", "22"] + + att: BlockAttestation = Mock(data=Mock(index="0"), aggregation_bits="", committee_bits="0x5ff2990000000000") + assert get_committee_indices(att) == [ + "0", + "1", + "2", + "3", + "4", + "6", + "9", + "12", + "13", + "14", + "15", + "16", + "19", + "20", + "23", + ] diff --git a/tests/modules/ejector/test_ejector.py b/tests/modules/ejector/test_ejector.py index cbc267cc9..8c6792646 100644 --- a/tests/modules/ejector/test_ejector.py +++ b/tests/modules/ejector/test_ejector.py @@ -5,14 +5,22 @@ from web3.exceptions import ContractCustomError from src import constants -from src.constants import MAX_EFFECTIVE_BALANCE +from src.constants import ( + EFFECTIVE_BALANCE_INCREMENT, + MAX_EFFECTIVE_BALANCE, + MAX_EFFECTIVE_BALANCE_ELECTRA, + MAX_SEED_LOOKAHEAD, + MIN_ACTIVATION_BALANCE, + MIN_VALIDATOR_WITHDRAWABILITY_DELAY, +) from src.modules.ejector import ejector as ejector_module from src.modules.ejector.ejector import Ejector from src.modules.ejector.ejector import logger as ejector_logger from src.modules.ejector.types import EjectorProcessingState from src.modules.submodules.oracle_module import ModuleExecuteDelay from src.modules.submodules.types import ChainConfig, CurrentFrame -from src.types import BlockStamp, ReferenceBlockStamp +from src.providers.consensus.types import BeaconStateView +from src.types import BlockStamp, Gwei, ReferenceBlockStamp from src.utils import validator_state from src.web3py.extensions.contracts import LidoContracts from src.web3py.extensions.lido_validators import NodeOperatorId, StakingModuleId @@ -136,9 +144,8 @@ def test_no_validators_to_eject( result = ejector.get_validators_to_eject(ref_blockstamp) assert result == [], "Unexpected validators to eject" - ejector.w3.lido_contracts.validators_exit_bus_oracle.get_consensus_version = Mock(return_value=2) - with monkeypatch.context() as m: + ejector.get_consensus_version = Mock(return_value=2) val_iter = iter(SimpleIterator([])) val_iter.get_remaining_forced_validators = Mock(return_value=[]) m.setattr( @@ -160,14 +167,13 @@ def test_simple( ): ejector.get_chain_config = Mock(return_value=chain_config) ejector.w3.lido_contracts.withdrawal_queue_nft.unfinalized_steth = Mock(return_value=200) - ejector.w3.lido_contracts.validators_exit_bus_oracle.get_contract_version = Mock(return_value=1) ejector.prediction_service.get_rewards_per_epoch = Mock(return_value=1) - ejector._get_sweep_delay_in_epochs = Mock(return_value=ref_blockstamp.ref_epoch) + ejector._get_sweep_delay_in_epochs = Mock(return_value=0) ejector._get_total_el_balance = Mock(return_value=100) ejector.validators_state_service.get_recently_requested_but_not_exited_validators = Mock(return_value=[]) ejector._get_withdrawable_lido_validators_balance = Mock(return_value=0) - ejector._get_predicted_withdrawable_epoch = Mock(return_value=50) + ejector._get_predicted_withdrawable_epoch = Mock(return_value=ref_blockstamp.ref_epoch + 50) ejector._get_predicted_withdrawable_balance = Mock(return_value=50) validators = [ @@ -177,6 +183,7 @@ def test_simple( ] with monkeypatch.context() as m: + ejector.get_consensus_version = Mock(return_value=1) m.setattr( ejector_module.ExitOrderIterator, "__iter__", @@ -185,9 +192,8 @@ def test_simple( result = ejector.get_validators_to_eject(ref_blockstamp) assert result == [validators[0]], "Unexpected validators to eject" - ejector.w3.lido_contracts.validators_exit_bus_oracle.get_consensus_version = Mock(return_value=2) - with monkeypatch.context() as m: + ejector.get_consensus_version = Mock(return_value=2) val_iter = iter(SimpleIterator(validators[:2])) val_iter.get_remaining_forced_validators = Mock(return_value=validators[2:]) m.setattr( @@ -225,17 +231,97 @@ def test_is_contract_reportable(ejector: Ejector, blockstamp: BlockStamp) -> Non @pytest.mark.unit -def test_get_predicted_withdrawable_epoch(ejector: Ejector) -> None: +def test_get_predicted_withdrawable_epoch_pre_electra(ejector: Ejector) -> None: + ejector.w3.cc = Mock() + ejector.w3.cc.get_config_spec = Mock(return_value=Mock(ELECTRA_FORK_EPOCH=FAR_FUTURE_EPOCH)) ejector._get_latest_exit_epoch = Mock(return_value=[1, 32]) ejector._get_churn_limit = Mock(return_value=2) ref_blockstamp = ReferenceBlockStampFactory.build(ref_epoch=3546) - result = ejector._get_predicted_withdrawable_epoch(ref_blockstamp, 2) + result = ejector._get_predicted_withdrawable_epoch(ref_blockstamp, [Mock()] * 2) assert result == 3808, "Unexpected predicted withdrawable epoch" - result = ejector._get_predicted_withdrawable_epoch(ref_blockstamp, 4) + result = ejector._get_predicted_withdrawable_epoch(ref_blockstamp, [Mock()] * 4) assert result == 3809, "Unexpected predicted withdrawable epoch" +class TestPredictedWithdrawableEpochPostElectra: + @pytest.fixture + def ref_blockstamp(self) -> ReferenceBlockStamp: + return ReferenceBlockStampFactory.build( + ref_slot=10_000_000, + ref_epoch=10_000_000 // 32, + ) + + @pytest.mark.unit + def test_earliest_exit_epoch_is_old(self, ejector: Ejector, ref_blockstamp: ReferenceBlockStamp) -> None: + ejector._get_total_active_balance = Mock(return_value=int(2048e9)) + ejector.w3.cc.get_state_view = Mock( + return_value=BeaconStateView( + slot=ref_blockstamp.slot_number, + earliest_exit_epoch=ref_blockstamp.ref_epoch, + exit_balance_to_consume=Gwei(0), + ) + ) + result = ejector._get_predicted_withdrawable_epoch( + ref_blockstamp, + [LidoValidatorFactory.build_with_balance(MIN_ACTIVATION_BALANCE)] * 1, + ) + assert result == ref_blockstamp.ref_epoch + (1 + MAX_SEED_LOOKAHEAD) + MIN_VALIDATOR_WITHDRAWABILITY_DELAY + + @pytest.mark.unit + def test_exit_fits_exit_balance_to_consume(self, ejector: Ejector, ref_blockstamp: ReferenceBlockStamp) -> None: + ejector._get_total_active_balance = Mock(return_value=int(2048e9)) + ejector.w3.cc.get_state_view = Mock( + return_value=BeaconStateView( + slot=ref_blockstamp.slot_number, + earliest_exit_epoch=ref_blockstamp.ref_epoch + 10_000, + exit_balance_to_consume=Gwei(int(256e9)), + ) + ) + result = ejector._get_predicted_withdrawable_epoch( + ref_blockstamp, + [LidoValidatorFactory.build_with_balance(129e9, meb=MAX_EFFECTIVE_BALANCE_ELECTRA)] * 1, + ) + assert result == ref_blockstamp.ref_epoch + 10_000 + MIN_VALIDATOR_WITHDRAWABILITY_DELAY + + @pytest.mark.unit + def test_exit_exceeds_balance_to_consume(self, ejector: Ejector, ref_blockstamp: ReferenceBlockStamp) -> None: + ejector._get_total_active_balance = Mock(return_value=2048e9) + ejector.w3.cc.get_state_view = Mock( + return_value=BeaconStateView( + slot=ref_blockstamp.slot_number, + earliest_exit_epoch=ref_blockstamp.ref_epoch + 10_000, + exit_balance_to_consume=Gwei(1), + ) + ) + result = ejector._get_predicted_withdrawable_epoch( + ref_blockstamp, + [LidoValidatorFactory.build_with_balance(512e9, meb=MAX_EFFECTIVE_BALANCE_ELECTRA)] * 1, + ) + assert result == ref_blockstamp.ref_epoch + 10_000 + 4 + MIN_VALIDATOR_WITHDRAWABILITY_DELAY + + @pytest.mark.unit + def test_exit_exceeds_churn_limit(self, ejector: Ejector, ref_blockstamp: ReferenceBlockStamp) -> None: + ejector._get_total_active_balance = Mock(return_value=2048e9) + ejector.w3.cc.get_state_view = Mock( + return_value=BeaconStateView( + slot=ref_blockstamp.slot_number, + earliest_exit_epoch=ref_blockstamp.ref_epoch, + exit_balance_to_consume=Gwei(0), + ) + ) + result = ejector._get_predicted_withdrawable_epoch( + ref_blockstamp, + [LidoValidatorFactory.build_with_balance(512e9, meb=MAX_EFFECTIVE_BALANCE_ELECTRA)] * 1, + ) + assert result == ref_blockstamp.ref_epoch + (1 + MAX_SEED_LOOKAHEAD) + 3 + MIN_VALIDATOR_WITHDRAWABILITY_DELAY + + @pytest.fixture(autouse=True) + def _patch_ejector(self, ejector: Ejector): + ejector.w3.cc = Mock() + ejector.w3.cc.get_config_spec = Mock(return_value=Mock(ELECTRA_FORK_EPOCH=0)) + + @pytest.mark.unit def test_get_total_active_validators(ejector: Ejector) -> None: ref_blockstamp = ReferenceBlockStampFactory.build(ref_epoch=3546) @@ -248,12 +334,39 @@ def test_get_total_active_validators(ejector: Ejector) -> None: ] ) - assert ejector._get_total_active_validators(ref_blockstamp) == 100 + assert len(ejector._get_active_validators(ref_blockstamp)) == 100 + + +@pytest.mark.unit +def test_get_total_active_balance(ejector: Ejector) -> None: + ejector._get_active_validators = Mock(return_value=[]) + assert ejector._get_total_active_balance(Mock()) == EFFECTIVE_BALANCE_INCREMENT + ejector._get_active_validators.assert_called_once() + + ejector._get_active_validators = Mock( + return_value=[ + LidoValidatorFactory.build_with_balance(Gwei(32 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(33 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9)), + ] + ) + assert ejector._get_total_active_balance(Mock()) == Gwei(95 * 10**9) + ejector._get_active_validators.assert_called_once() + + ejector._get_active_validators = Mock( + return_value=[ + LidoValidatorFactory.build_with_balance(Gwei(32 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(99 * 10**9), meb=MAX_EFFECTIVE_BALANCE_ELECTRA), + ] + ) + assert ejector._get_total_active_balance(Mock()) == Gwei(162 * 10**9) + ejector._get_active_validators.assert_called_once() @pytest.mark.unit @pytest.mark.usefixtures("consensus_client", "lido_validators") -def test_get_withdrawable_lido_validators( +def test_get_withdrawable_lido_validators_balance( ejector: Ejector, ref_blockstamp: ReferenceBlockStamp, monkeypatch: pytest.MonkeyPatch, @@ -275,7 +388,7 @@ def test_get_withdrawable_lido_validators( ) result = ejector._get_withdrawable_lido_validators_balance(42, ref_blockstamp) - assert result == 42 * 10**9, "Unexpected withdrawable amount" + assert result == 42, "Unexpected withdrawable amount" ejector._get_withdrawable_lido_validators_balance(42, ref_blockstamp) ejector.w3.lido_validators.get_lido_validators.assert_called_once() @@ -283,22 +396,29 @@ def test_get_withdrawable_lido_validators( @pytest.mark.unit def test_get_predicted_withdrawable_balance(ejector: Ejector) -> None: - validator = LidoValidatorFactory.build(balance="0") + validator = LidoValidatorFactory.build_with_balance(Gwei(0)) result = ejector._get_predicted_withdrawable_balance(validator) assert result == 0, "Expected zero" - validator = LidoValidatorFactory.build(balance="42") + validator = LidoValidatorFactory.build_with_balance(Gwei(42)) result = ejector._get_predicted_withdrawable_balance(validator) - assert result == 42 * 10**9, "Expected validator's balance in gwei" + assert result == 42, "Expected validator's balance in gwei" - validator = LidoValidatorFactory.build(balance=str(MAX_EFFECTIVE_BALANCE + 1)) + validator = LidoValidatorFactory.build_with_balance(Gwei(MAX_EFFECTIVE_BALANCE + 1)) result = ejector._get_predicted_withdrawable_balance(validator) - assert result == MAX_EFFECTIVE_BALANCE * 10**9, "Expect MAX_EFFECTIVE_BALANCE" + assert result == MAX_EFFECTIVE_BALANCE, "Expect MAX_EFFECTIVE_BALANCE" + + validator = LidoValidatorFactory.build_with_balance( + Gwei(MAX_EFFECTIVE_BALANCE + 1), + meb=MAX_EFFECTIVE_BALANCE_ELECTRA, + ) + result = ejector._get_predicted_withdrawable_balance(validator) + assert result == MAX_EFFECTIVE_BALANCE + 1, "Expect MAX_EFFECTIVE_BALANCE + 1" @pytest.mark.unit @pytest.mark.usefixtures("consensus_client") -def test_get_sweep_delay_in_epochs( +def test_get_sweep_delay_in_epochs_pre_electra( ejector: Ejector, ref_blockstamp: ReferenceBlockStamp, chain_config: ChainConfig, @@ -306,6 +426,7 @@ def test_get_sweep_delay_in_epochs( ) -> None: ejector.w3.cc.get_validators = Mock(return_value=LidoValidatorFactory.batch(1024)) ejector.get_chain_config = Mock(return_value=chain_config) + ejector.get_consensus_version = Mock(return_value=1) with monkeypatch.context() as m: m.setattr( @@ -340,6 +461,102 @@ def test_get_sweep_delay_in_epochs( assert result == 1, "Unexpected sweep delay in epochs" +@pytest.mark.unit +@pytest.mark.usefixtures("consensus_client") +def test_get_sweep_delay_in_epochs_post_electra( + ejector: Ejector, + chain_config: ChainConfig, + monkeypatch: pytest.MonkeyPatch, +) -> None: + ejector.get_chain_config = Mock(return_value=chain_config) + ejector.get_consensus_version = Mock(return_value=3) + ejector.w3.cc = Mock() + + ejector.w3.cc.get_validators = Mock(return_value=[]) + delay = ejector._get_sweep_delay_in_epochs(Mock(ref_epoch=0)) + assert delay == 0, "Unexpected sweep delay in epochs" + + ejector.w3.cc.get_validators = Mock(return_value=[LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9))] * 3) + with monkeypatch.context() as m: + m.setattr( + ejector_module, + "is_fully_withdrawable_validator", + Mock(return_value=False), + ) + delay = ejector._get_sweep_delay_in_epochs(Mock(ref_epoch=0)) + assert delay == 0, "Unexpected sweep delay in epochs" + + ejector.w3.cc.get_validators = Mock( + return_value=[ + LidoValidatorFactory.build_with_balance(Gwei(32 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(33 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9)), + ], + ) + with monkeypatch.context() as m: + m.setattr( + ejector_module, + "is_fully_withdrawable_validator", + Mock(return_value=False), + ) + delay = ejector._get_sweep_delay_in_epochs(Mock(ref_epoch=0)) + assert delay == 1, "Unexpected sweep delay in epochs" + + ejector.w3.cc.get_validators = Mock( + return_value=[ + LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9)), + ], + ) + with monkeypatch.context() as m: + m.setattr( + ejector_module, + "is_fully_withdrawable_validator", + Mock(return_value=True), + ) + delay = ejector._get_sweep_delay_in_epochs(Mock(ref_epoch=0)) + assert delay == 1, "Unexpected sweep delay in epochs" + + ejector.w3.cc.get_validators = Mock( + return_value=[ + LidoValidatorFactory.build_with_balance(Gwei(32 * 10**9)), + LidoValidatorFactory.build_with_balance(Gwei(33 * 10**9)), + ] + * 513, + ) + with monkeypatch.context() as m: + m.setattr( + ejector_module, + "is_fully_withdrawable_validator", + Mock(return_value=True), + ) + delay = ejector._get_sweep_delay_in_epochs(Mock(ref_epoch=0)) + assert delay == 2, "Unexpected sweep delay in epochs" + + +@pytest.mark.unit +def test_get_withdrawable_validators(ejector: Ejector, monkeypatch) -> None: + ejector.w3.cc = Mock() + ejector.w3.cc.get_validators = Mock( + return_value=[ + LidoValidatorFactory.build_with_balance(Gwei(32 * 10**9), index=1), + LidoValidatorFactory.build_with_balance(Gwei(33 * 10**9), index=2), + LidoValidatorFactory.build_with_balance(Gwei(31 * 10**9), index=3), + ], + ) + + with monkeypatch.context() as m: + m.setattr( + ejector_module, + "is_fully_withdrawable_validator", + Mock(return_value=False), + ) + withdrawable = ejector._get_withdrawable_validators(Mock()) + + assert [v.index for v in withdrawable] == [2] + + @pytest.mark.usefixtures("contracts") def test_get_total_balance(ejector: Ejector, blockstamp: BlockStamp) -> None: ejector.w3.lido_contracts.get_withdrawal_balance = Mock(return_value=3) diff --git a/tests/modules/ejector/test_exit_order_iterator.py b/tests/modules/ejector/test_exit_order_iterator.py index 41aa37224..a0f497098 100644 --- a/tests/modules/ejector/test_exit_order_iterator.py +++ b/tests/modules/ejector/test_exit_order_iterator.py @@ -14,12 +14,12 @@ @pytest.mark.unit def test_predicates(): - def v(module_address, operator, index, activation_epoch) -> LidoValidator: + def v(module_address, operator: int, index, activation_epoch) -> LidoValidator: validator = object.__new__(LidoValidator) validator.lido_id = object.__new__(LidoKey) validator.validator = object.__new__(ValidatorState) validator.lido_id.moduleAddress = module_address - validator.lido_id.operatorIndex = operator + validator.lido_id.operatorIndex = NodeOperatorId(operator) validator.index = index validator.validator.activation_epoch = activation_epoch return validator @@ -75,12 +75,12 @@ def v(module_address, operator, index, activation_epoch) -> LidoValidator: @pytest.mark.unit def test_decrease_node_operator_stats(): - def v(module_address, operator, index, activation_epoch) -> LidoValidator: + def v(module_address, operator: int, index, activation_epoch) -> LidoValidator: validator = object.__new__(LidoValidator) validator.lido_id = object.__new__(LidoKey) validator.validator = object.__new__(ValidatorState) validator.lido_id.moduleAddress = module_address - validator.lido_id.operatorIndex = operator + validator.lido_id.operatorIndex = NodeOperatorId(operator) validator.index = index validator.validator.activation_epoch = activation_epoch return validator diff --git a/tests/modules/submodules/test_oracle_module.py b/tests/modules/submodules/test_oracle_module.py index cb302bb81..0eaabc791 100644 --- a/tests/modules/submodules/test_oracle_module.py +++ b/tests/modules/submodules/test_oracle_module.py @@ -1,4 +1,4 @@ -from unittest.mock import Mock +from unittest.mock import Mock, patch, MagicMock from typing import Type import pytest @@ -92,41 +92,51 @@ def _throw_on_third_call(): @pytest.mark.parametrize( "ex", [ - DecoratorTimeoutError, - NoActiveProviderError, - RequestsConnectionError, - NotOkResponse, - NoSlotsAvailable, - SlotNotFinalized, - InconsistentData, - KeysOutdatedException, + DecoratorTimeoutError("Fake exception"), + NoActiveProviderError("Fake exception"), + RequestsConnectionError("Fake exception"), + NotOkResponse(status=500, text="Fake exception"), + NoSlotsAvailable("Fake exception"), + SlotNotFinalized("Fake exception"), + InconsistentData("Fake exception"), + KeysOutdatedException("Fake exception"), ], + ids=lambda param: f"{type(param).__name__}", ) -def test_run_cycle_no_fail_on_retryable_error(oracle: BaseModule, ex: Type[Exception]): - def _throw_with(*args): - if ex is NotOkResponse: - raise ex(status=500, text="Fake exception") # type: ignore - raise ex("Fake exception") - - oracle.execute_module = Mock(side_effect=_throw_with) - - ret = oracle.run_cycle(ReferenceBlockStampFactory.build()) - assert ret is ModuleExecuteDelay.NEXT_SLOT +def test_cycle_no_fail_on_retryable_error(oracle: BaseModule, ex: Exception): + oracle.w3.lido_contracts = MagicMock() + with patch.object( + oracle, "_receive_last_finalized_slot", return_value=MagicMock(slot_number=1111111) + ), patch.object(oracle.w3.lido_contracts, "has_contract_address_changed", return_value=False), patch.object( + oracle, "execute_module", side_effect=ex + ): + oracle._cycle() + # test node availability + with patch.object(oracle, "_receive_last_finalized_slot", side_effect=ex): + oracle._cycle() @pytest.mark.unit @pytest.mark.parametrize( "ex", [ - IsNotMemberException, - IncompatibleOracleVersion, + IsNotMemberException("Fake exception"), + IncompatibleOracleVersion("Fake exception"), ], + ids=lambda param: f"{type(param).__name__}", ) -def test_run_cycle_fails_on_critical_exceptions(oracle: BaseModule, ex: Type[Exception]): - def _throw_with(*args): - raise ex("Fake exception") - - oracle.execute_module = Mock(side_effect=_throw_with) - - with pytest.raises(ex, match="Fake exception"): - oracle.run_cycle(ReferenceBlockStampFactory.build()) +def test_run_cycle_fails_on_critical_exceptions(oracle: BaseModule, ex: Exception): + oracle.w3.lido_contracts = MagicMock() + with patch.object( + oracle, "_receive_last_finalized_slot", return_value=MagicMock(slot_number=1111111) + ), patch.object(oracle.w3.lido_contracts, "has_contract_address_changed", return_value=False), patch.object( + oracle, "execute_module", side_effect=ex + ), pytest.raises( + type(ex), match="Fake exception" + ): + oracle._cycle() + # test node availability + with patch.object(oracle, "_receive_last_finalized_slot", side_effect=ex), pytest.raises( + type(ex), match="Fake exception" + ): + oracle._cycle() diff --git a/tests/providers/consensus/test_consensus_client.py b/tests/providers/consensus/test_consensus_client.py index c58aaf4f0..770fd8f67 100644 --- a/tests/providers/consensus/test_consensus_client.py +++ b/tests/providers/consensus/test_consensus_client.py @@ -71,6 +71,19 @@ def test_get_validators(consensus_client: ConsensusClient): assert validator_by_pub_key[0] == validator +@pytest.mark.integration +@pytest.mark.skip(reason="Too long to complete in CI") +def test_get_state_view(consensus_client: ConsensusClient): + state_view = consensus_client.get_state_view("head") + assert state_view.slot > 0 + + spec = consensus_client.get_config_spec() + epoch = state_view.slot // 32 + if epoch >= int(spec.ELECTRA_FORK_EPOCH): + assert state_view.earliest_exit_epoch != 0 + assert state_view.exit_balance_to_consume >= 0 + + @pytest.mark.unit def test_get_returns_nor_dict_nor_list(consensus_client: ConsensusClient): consensus_client._get_without_fallbacks = Mock(return_value=(1, None)) diff --git a/tests/utils/test_build.py b/tests/utils/test_build.py index 8decb00aa..66b357b13 100644 --- a/tests/utils/test_build.py +++ b/tests/utils/test_build.py @@ -7,11 +7,10 @@ class TestGetBuildInfo(unittest.TestCase): - @patch('os.path.exists', return_value=True) @patch( 'builtins.open', new_callable=mock_open, read_data='{"version": "1.0.0", "branch": "main", "commit": "abc123"}' ) - def test_get_build_info_success(self, mock_open_file, mock_exists): + def test_get_build_info_success(self, mock_open_file): """Test that get_build_info successfully reads from the JSON file.""" expected_build_info = {"version": "1.0.0", "branch": "main", "commit": "abc123"} @@ -19,17 +18,14 @@ def test_get_build_info_success(self, mock_open_file, mock_exists): build_info = get_build_info() # Assertions - mock_exists.assert_called_once_with("./build-info.json") mock_open_file.assert_called_once_with("./build-info.json", "r") self.assertEqual(build_info, expected_build_info, "Build info should match the data from the file") - @patch('os.path.exists', return_value=False) - def test_get_build_info_file_not_exists(self, mock_exists): + def test_get_build_info_file_not_exists(self): """Test that get_build_info returns UNKNOWN_BUILD_INFO when the file does not exist.""" build_info = get_build_info() # Assertions - mock_exists.assert_called_once_with("./build-info.json") self.assertEqual(build_info, UNKNOWN_BUILD_INFO, "Should return UNKNOWN_BUILD_INFO when file doesn't exist") @patch('os.path.exists', return_value=True) @@ -41,6 +37,5 @@ def test_get_build_info_json_decode_error(self, mock_open_file, mock_exists): build_info = get_build_info() # Assertions - mock_exists.assert_called_once_with("./build-info.json") mock_open_file.assert_called_once_with("./build-info.json", "r") self.assertEqual(build_info, UNKNOWN_BUILD_INFO, "Should return UNKNOWN_BUILD_INFO when JSON decode fails") diff --git a/tests/utils/test_types.py b/tests/utils/test_types.py index 78bdcefd3..d0339f5ee 100644 --- a/tests/utils/test_types.py +++ b/tests/utils/test_types.py @@ -1,6 +1,6 @@ import pytest -from src.utils.types import bytes_to_hex_str, hex_str_to_bytes +from src.utils.types import bytes_to_hex_str, hex_str_to_bytes, is_4bytes_hex @pytest.mark.unit @@ -12,6 +12,26 @@ def test_bytes_to_hex_str(): @pytest.mark.unit def test_hex_str_to_bytes(): + assert hex_str_to_bytes("") == b"" + assert hex_str_to_bytes("00") == b"\x00" + assert hex_str_to_bytes("000102") == b"\x00\x01\x02" assert hex_str_to_bytes("0x") == b"" assert hex_str_to_bytes("0x00") == b"\x00" assert hex_str_to_bytes("0x000102") == b"\x00\x01\x02" + + +@pytest.mark.unit +def test_is_4bytes_hex(): + assert is_4bytes_hex("0x00000000") + assert is_4bytes_hex("0x02000000") + assert is_4bytes_hex("0x02000000") + assert is_4bytes_hex("0x30637624") + + assert not is_4bytes_hex("") + assert not is_4bytes_hex("0x") + assert not is_4bytes_hex("0x00") + assert not is_4bytes_hex("0x01") + assert not is_4bytes_hex("0x01") + assert not is_4bytes_hex("0xgg") + assert not is_4bytes_hex("0x111") + assert not is_4bytes_hex("0x02000000ff") diff --git a/tests/utils/test_validator_state_utils.py b/tests/utils/test_validator_state_utils.py index 677763b3c..b0553cddf 100644 --- a/tests/utils/test_validator_state_utils.py +++ b/tests/utils/test_validator_state_utils.py @@ -1,21 +1,31 @@ -from pydantic.class_validators import validator import pytest +from pydantic.class_validators import validator -from src.constants import FAR_FUTURE_EPOCH, EFFECTIVE_BALANCE_INCREMENT -from src.providers.consensus.types import Validator, ValidatorStatus, ValidatorState +from src.constants import ( + EFFECTIVE_BALANCE_INCREMENT, + FAR_FUTURE_EPOCH, + MAX_EFFECTIVE_BALANCE_ELECTRA, + MIN_ACTIVATION_BALANCE, +) +from src.providers.consensus.types import Validator, ValidatorState, ValidatorStatus from src.types import EpochNumber, Gwei from src.utils.validator_state import ( + calculate_active_effective_balance_sum, calculate_total_active_effective_balance, - is_on_exit, + compute_activation_exit_epoch, + get_balance_churn_limit, + get_max_effective_balance, get_validator_age, - calculate_active_effective_balance_sum, - is_validator_eligible_to_exit, - is_fully_withdrawable_validator, - is_partially_withdrawable_validator, + has_compounding_withdrawal_credential, has_eth1_withdrawal_credential, - is_exited_validator, + has_execution_withdrawal_credential, is_active_validator, - compute_activation_exit_epoch, + is_exited_validator, + is_fully_withdrawable_validator, + is_on_exit, + is_partially_withdrawable_validator, + is_validator_eligible_to_exit, + get_activation_exit_churn_limit, ) from tests.factory.no_registry import ValidatorFactory from tests.modules.accounting.bunker.test_bunker_abnormal_cl_rebase import simple_validators @@ -144,6 +154,24 @@ def test_is_on_exit(exit_epoch, expected): assert actual == expected +@pytest.mark.unit +@pytest.mark.parametrize( + "withdrawal_credentials, expected", + [ + ('0x02ba', True), + ('02ab', False), + ('0x00ba', False), + ('00ba', False), + ], +) +def test_has_compounding_withdrawal_credential(withdrawal_credentials, expected): + validator = ValidatorFactory.build() + validator.validator.withdrawal_credentials = withdrawal_credentials + + actual = has_compounding_withdrawal_credential(validator) + assert actual == expected + + @pytest.mark.unit @pytest.mark.parametrize( "withdrawal_credentials, expected", @@ -164,18 +192,44 @@ def test_has_eth1_withdrawal_credential(withdrawal_credentials, expected): @pytest.mark.unit @pytest.mark.parametrize( - "withdrawable_epoch, balance, epoch, expected", + "wc, expected", [ - (176720, 32 * (10**10), 176722, True), - (176722, 32 * (10**10), 176722, True), - (176723, 32 * (10**10), 176722, False), - (176722, 0, 176722, False), + ('0x01ba', True), + ('01ab', False), + ('0x00ba', False), + ('00ba', False), + ('0x02ba', True), + ('02ab', False), + ('0x00ba', False), + ('00ba', False), + ], +) +def test_has_execution_withdrawal_credential(wc, expected): + validator = ValidatorFactory.build() + validator.validator.withdrawal_credentials = wc + + actual = has_execution_withdrawal_credential(validator) + assert actual == expected + + +@pytest.mark.unit +@pytest.mark.parametrize( + "withdrawable_epoch, wc, balance, epoch, expected", + [ + (176720, '0x01ba', 32 * (10**10), 176722, True), + (176722, '0x01ba', 32 * (10**10), 176722, True), + (176723, '0x01ba', 32 * (10**10), 176722, False), + (176722, '0x01ba', 0, 176722, False), + (176720, '0x02ba', 32 * (10**10), 176722, True), + (176722, '0x02ba', 32 * (10**10), 176722, True), + (176723, '0x02ba', 32 * (10**10), 176722, False), + (176722, '0x02ba', 0, 176722, False), ], ) -def test_is_fully_withdrawable_validator(withdrawable_epoch, balance, epoch, expected): +def test_is_fully_withdrawable_validator(withdrawable_epoch, wc, balance, epoch, expected): validator = ValidatorFactory.build() validator.validator.withdrawable_epoch = withdrawable_epoch - validator.validator.withdrawal_credentials = '0x01ba' + validator.validator.withdrawal_credentials = wc validator.balance = balance actual = is_fully_withdrawable_validator(validator, EpochNumber(epoch)) @@ -187,10 +241,13 @@ def test_is_fully_withdrawable_validator(withdrawable_epoch, balance, epoch, exp "effective_balance, add_balance, withdrawal_credentials, expected", [ (32 * 10**9, 1, '0x01ba', True), + (MAX_EFFECTIVE_BALANCE_ELECTRA, 1, '0x02ba', True), (32 * 10**9, 1, '0x0', False), (32 * 10**8, 0, '0x01ba', False), + (MAX_EFFECTIVE_BALANCE_ELECTRA, 0, '0x02ba', False), (32 * 10**9, 0, '0x', False), (0, 0, '0x01ba', False), + (0, 0, '0x02ba', False), ], ) def test_is_partially_withdrawable(effective_balance, add_balance, withdrawal_credentials, expected): @@ -221,6 +278,22 @@ def test_is_validator_eligible_to_exit(activation_epoch, exit_epoch, epoch, expe assert actual == expected +@pytest.mark.unit +@pytest.mark.parametrize( + "wc, expected", + [ + ('0x01ba', MIN_ACTIVATION_BALANCE), + ('0x02ba', MAX_EFFECTIVE_BALANCE_ELECTRA), + ('0x0', MIN_ACTIVATION_BALANCE), + ], +) +def test_max_effective_balance(wc, expected): + validator = ValidatorFactory.build() + validator.validator.withdrawal_credentials = wc + result = get_max_effective_balance(validator) + assert result == expected + + class TestCalculateTotalEffectiveBalance: @pytest.fixture def validators(self): @@ -296,3 +369,41 @@ def test_skip_ongoing(self, validators: list[Validator]): def test_compute_activation_exit_epoch(): ref_epoch = 3455 assert 3460 == compute_activation_exit_epoch(ref_epoch) + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("total_active_balance", "expected_limit"), + ( + (0, 128e9), + (32e9, 128e9), + (2 * 32e9, 128e9), + (1024 * 32e9, 128e9), + (512 * 1024 * 32e9, 256e9), + (1024 * 1024 * 32e9, 512e9), + (2000 * 1024 * 32e9, 1000e9), + (3300 * 1024 * 32e9, 1650e9), + ), +) +def test_get_balance_churn_limit(total_active_balance: Gwei, expected_limit: Gwei): + actual_limit = get_balance_churn_limit(total_active_balance) + assert actual_limit == expected_limit, "Unexpected balance churn limit" + + +@pytest.mark.unit +@pytest.mark.parametrize( + ("total_active_balance", "expected_limit"), + ( + (0, 128e9), + (32e9, 128e9), + (2 * 32e9, 128e9), + (1024 * 32e9, 128e9), + (512 * 1024 * 32e9, 256e9), + (1024 * 1024 * 32e9, 256e9), + (2000 * 1024 * 32e9, 256e9), + (3300 * 1024 * 32e9, 256e9), + ), +) +def test_compute_exit_balance_churn_limit(total_active_balance: Gwei, expected_limit: Gwei): + actual_limit = get_activation_exit_churn_limit(total_active_balance) + assert actual_limit == expected_limit, "Unexpected exit churn limit" diff --git a/tests/web3_extentions/test_lido_validators.py b/tests/web3_extentions/test_lido_validators.py index 2a28b37ab..f775373f3 100644 --- a/tests/web3_extentions/test_lido_validators.py +++ b/tests/web3_extentions/test_lido_validators.py @@ -2,6 +2,7 @@ import pytest +from src.constants import LIDO_DEPOSIT_AMOUNT from src.modules.accounting.types import BeaconStat from src.web3py.extensions.lido_validators import CountOfKeysDiffersException from tests.factory.blockstamp import ReferenceBlockStampFactory @@ -38,6 +39,16 @@ def test_get_lido_validators(web3, lido_validators, contracts): assert v.lido_id.key == v.validator.pubkey +@pytest.mark.unit +def test_calc_pending_deposits_sum(web3, lido_validators, contracts): + lido_validators = LidoValidatorFactory.batch(30) + lido_validators.extend(LidoValidatorFactory.build_pending_deposit_vals() for _ in range(5)) + + pending_deposits_sum = web3.lido_validators.calculate_pending_deposits_sum(lido_validators) + + assert pending_deposits_sum == 5 * LIDO_DEPOSIT_AMOUNT + + @pytest.mark.unit def test_kapi_has_lesser_keys_than_deposited_validators_count(web3, lido_validators, contracts): validators = ValidatorFactory.batch(10)