diff --git a/README.md b/README.md index 4513c2a12..2516cb22b 100644 --- a/README.md +++ b/README.md @@ -149,18 +149,18 @@ If you made changes in both indy-plenum and indy-node, you need to do the follow - Indy-plenum is based on [RBFT](https://pakupaku.me/plaublin/rbft/5000a297.pdf) protocol - Please have a look at documents and diagrams in [docs/source](docs/source) folder - Please have a look at documents and diagrams in Plenum's [docs](https://github.com/hyperledger/indy-plenum/tree/master/docs) folder, or on https://indy.readthedocs.io/projects/plenum : - - [Technical Overview of Plenum](https://github.com/hyperledger/indy-plenum/blob/master/docs/main.md) - - [Plenum Consensus Algorithm Diagram](https://github.com/hyperledger/indy-plenum/blob/master/docs/diagrams/consensus-protocol.png) - - [Glossary](https://github.com/hyperledger/indy-plenum/blob/master/docs/glossary.md) - - [Storages](https://github.com/hyperledger/indy-plenum/blob/master/docs/storage.md) - - [Request Handling](https://github.com/hyperledger/indy-plenum/blob/master/docs/request_handling.md) - - [Catchup](https://github.com/hyperledger/indy-plenum/blob/master/docs/catchup.md) - - [Catchup Diagram](https://github.com/hyperledger/indy-plenum/blob/master/docs/diagrams/catchup-procedure.png) - - [Plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/plugins.md) + - [Technical Overview of Plenum](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/main.md) + - [Plenum Consensus Algorithm Diagram](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/diagrams/consensus-protocol.png) + - [Glossary](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/glossary.md) + - [Storages](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/storage.md) + - [Request Handling](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/request_handling.md) + - [Catchup](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/catchup.md) + - [Catchup Diagram](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/diagrams/catchup-procedure.png) + - [Plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md) - Relationship between Entities and Transactions: [relationship diagram](docs/source/relationship-diagram.png) - Supported transactions and their format: [transactions](docs/source/transactions.md) - Supported requests (write, read) and their format: [requests](docs/source/requests.md) -- [Network roles and permissions](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0) +- [Network roles and permissions](https://github.com/hyperledger/indy-node/blob/master/docs/source/auth_rules.md) - [Indy file folder structure guideline](docs/source/indy-file-structure-guideline.md) - [Helper Scripts](docs/source/helper-scripts.md) - [Pool Upgrade](docs/source/pool-upgrade.md) diff --git a/build-scripts/ubuntu-1604/build-3rd-parties.sh b/build-scripts/ubuntu-1604/build-3rd-parties.sh index 987b60af0..542a94814 100755 --- a/build-scripts/ubuntu-1604/build-3rd-parties.sh +++ b/build-scripts/ubuntu-1604/build-3rd-parties.sh @@ -42,4 +42,4 @@ function build_from_pypi { # build_from_pypi build_from_pypi timeout-decorator 0.4.0 - +build_from_pypi distro 1.3.0 diff --git a/build-scripts/ubuntu-1604/prepare-package.sh b/build-scripts/ubuntu-1604/prepare-package.sh index 1378eade4..d9672e7c6 100755 --- a/build-scripts/ubuntu-1604/prepare-package.sh +++ b/build-scripts/ubuntu-1604/prepare-package.sh @@ -39,6 +39,7 @@ sed -i -r "s~indy-plenum-[a-z]+~indy-plenum~" "$repo/setup.py" echo -e "Adapt the dependencies for the Canonical archive" sed -i "s~python-dateutil~python3-dateutil~" "$repo/setup.py" sed -i "s~timeout-decorator~python3-timeout-decorator~" "$repo/setup.py" +sed -i "s~distro~python3-distro~" "$repo/setup.py" # create manifest file repourl=$(git --git-dir $repo/.git --work-tree $repo config --get remote.origin.url) diff --git a/docs/source/auth_rules.md b/docs/source/auth_rules.md index 6aedd48e0..8ad85fed3 100644 --- a/docs/source/auth_rules.md +++ b/docs/source/auth_rules.md @@ -4,10 +4,12 @@ | NYM |`role` |`` | TRUSTEE | TRUSTEE|Adding new TRUSTEE| | NYM |`role` |`` | STEWARD | TRUSTEE|Adding new STEWARD| | NYM |`role` |`` | TRUST_ANCHOR| TRUSTEE, STEWARD|Adding new TRUST_ANCHOR| +| NYM |`role` |`` | NETWORK_MONITOR| TRUSTEE, STEWARD|Adding new NETWORK_MONITOR| | NYM |`role` |`` |`` | TRUSTEE, STEWARD, TRUST_ANCHOR| Adding new Identity Owner| | NYM |`role` | TRUSTEE |`` | TRUSTEE | Blacklisting Trustee| | NYM |`role` | STEWARD |`` | TRUSTEE | Blacklisting Steward| | NYM |`role` | TRUST_ANCHOR |`` | TRUSTEE | Blacklisting Trust anchor| +| NYM |`role` | NETWORK_MONITOR|`` | TRUSTEE, STEWARD | Blacklisting user with NETWORK_MONITOR role| | NYM |`verkey`|`*`|`*`| Owner of this nym | Key Rotation| | SCHEMA |`*`|`*`|`*`| TRUSTEE, STEWARD, TRUST_ANCHOR | Adding new Schema| | SCHEMA |`*`|`*`|`*`| No one can edit existing Schema | Editing Schema| @@ -25,7 +27,7 @@ | POOL_UPGRADE |`action`|`start`|`cancel`|TRUSTEE| Canceling upgrade procedure| | POOL_RESTART |`action`|`*`|`*`|TRUSTEE| Restarting pool command| | POOL_CONFIG |`action`|`*`|`*`|TRUSTEE| Pool config command (like a `read only` option)| -| VALIDATOR_INFO |`*`|`*`|`*`| TRUSTEE, STEWARD| Getting validator_info from pool| +| VALIDATOR_INFO |`*`|`*`|`*`| TRUSTEE, STEWARD, NETWORK_MONITOR| Getting validator_info from pool| ### Also, there is a some optional rules for case if in config option ANYONE_CAN_WRITE is set to True: @@ -52,5 +54,3 @@ |REVOC_REG_DEF|`*`|`*`|`*`| Only owners can edit existing REVOC_REG_DEF| Editing REVOC_REG_DEF| |REVOC_REG_ENTRY|`*`|`*`|`*`| Only the owner of the corresponding REVOC_REG_DEF can create new REVOC_REG_ENTRY| Adding new REVOC_REG_ENTRY| |REVOC_REG_ENTRY|`*`|`*`|`*`| Only owners can edit existing REVOC_REG_ENTRY| Adding new REVOC_REG_ENTRY| - - diff --git a/docs/source/requests-new.md b/docs/source/requests-new.md index 57cdc4e0b..38cd3594c 100644 --- a/docs/source/requests-new.md +++ b/docs/source/requests-new.md @@ -31,7 +31,7 @@ then have a look at [transactions](transactions.md). [indy-sdk](https://github.com/hyperledger/indy-sdk) expects the format as specified below. -See [roles and permissions](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0) on the roles and who can create each type of transactions. +See [roles and permissions](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md) on the roles and who can create each type of transactions. ## Base Client-to-Node and Node-to-Node serialization @@ -404,7 +404,7 @@ The format of each request-specific data for each type of request. ### NYM Creates a new NYM record for a specific user, trust anchor, steward or trustee. -Note that only trustees and stewards can create new trust anchors and trustee can be created only by other trusties (see [roles](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0)). +Note that only trustees and stewards can create new trust anchors and trustee can be created only by other trusties (see [roles](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md)). The request can be used for creation of new DIDs, setting and rotation of verification key, setting and changing of roles. @@ -425,7 +425,7 @@ creation of new DIDs, setting and rotation of verification key, setting and chan - 2 (STEWARD) - 101 (TRUST_ANCHOR) - A TRUSTEE can change any Nym's role to None, this stopping it from making any writes (see [roles](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0)). + A TRUSTEE can change any Nym's role to None, this stopping it from making any writes (see [roles](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md)). - `verkey` (base58-encoded string; optional): diff --git a/docs/source/requests.md b/docs/source/requests.md index a92363ed9..86c84f711 100644 --- a/docs/source/requests.md +++ b/docs/source/requests.md @@ -31,7 +31,7 @@ then have a look at [transactions](transactions.md). [indy-sdk](https://github.com/hyperledger/indy-sdk) expects the format as specified below. -See [roles and permissions](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0) on the roles and who can create each type of transactions. +See [roles and permissions](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md) on the roles and who can create each type of transactions. ## Common Request Structure @@ -337,7 +337,7 @@ The format of each request-specific data for each type of request. ### NYM Creates a new NYM record for a specific user, trust anchor, steward or trustee. -Note that only trustees and stewards can create new trust anchors and trustee can be created only by other trusties (see [roles](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0)). +Note that only trustees and stewards can create new trust anchors and trustee can be created only by other trusties (see [roles](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md)). The request can be used for creation of new DIDs, setting and rotation of verification key, setting and changing of roles. @@ -358,7 +358,7 @@ creation of new DIDs, setting and rotation of verification key, setting and chan - "2" (STEWARD) - "101" (TRUST_ANCHOR) - A TRUSTEE can change any Nym's role to None, this stopping it from making any writes (see [roles](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0)). + A TRUSTEE can change any Nym's role to None, this stopping it from making any writes (see [roles](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md)). - `verkey` (base58-encoded string, possibly starting with "~"; optional): @@ -534,7 +534,7 @@ So, if the Schema needs to be evolved, a new Schema with a new version or name n Dictionary with Schema's data: - - `attr_names`: array of attribute name strings + - `attr_names`: array of attribute name strings (125 attributes maximum) - `name`: Schema's name string - `version`: Schema's version string diff --git a/docs/source/transactions.md b/docs/source/transactions.md index b860797fd..4ccce2422 100644 --- a/docs/source/transactions.md +++ b/docs/source/transactions.md @@ -38,7 +38,7 @@ where key is a sequence number of the transaction and value is the serialized tr - All transactions are serialized to MsgPack format - All transactions (both transaction log and merkle tree hash stores) are stored in a LevelDB - One can use the `read_ledger` script to get transactions for a specified ledger in a readable format (JSON) -- See [roles and permissions](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0) for a list of roles and they type of transactions they can create. +- See [roles and permissions](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md) for a list of roles and they type of transactions they can create. Below you can find the format and description of all supported transactions. @@ -170,7 +170,7 @@ Please note that all these metadata fields may be absent for genesis transaction #### NYM Creates a new NYM record for a specific user, trust anchor, steward or trustee. -Note that only trustees and stewards can create new trust anchors and a trustee can be created only by other trustees (see [roles](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0)). +Note that only trustees and stewards can create new trust anchors and a trustee can be created only by other trustees (see [roles](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md)). The transaction can be used for creation of new DIDs, setting and rotation of verification key, setting and changing of roles. @@ -191,7 +191,7 @@ creation of new DIDs, setting and rotation of verification key, setting and chan - 2 (STEWARD) - 101 (TRUST_ANCHOR) - A TRUSTEE can change any Nym's role to None, thus stopping it from making any further writes (see [roles](https://docs.google.com/spreadsheets/d/1TWXF7NtBjSOaUIBeIH77SyZnawfo91cJ_ns4TR-wsq4/edit#gid=0)). + A TRUSTEE can change any Nym's role to None, thus stopping it from making any further writes (see [roles](https://github.com/hyperledger/indy-node/blob/master/docs/auth_rules.md)). - `verkey` (base58-encoded string, possibly starting with "~"; optional): diff --git a/indy_common/auth.py b/indy_common/auth.py index 0f86eff7f..735dc806f 100644 --- a/indy_common/auth.py +++ b/indy_common/auth.py @@ -4,7 +4,7 @@ from indy_common.constants import OWNER, POOL_UPGRADE, TRUST_ANCHOR, NYM, \ POOL_CONFIG, SCHEMA, CLAIM_DEF, \ - POOL_RESTART, VALIDATOR_INFO + POOL_RESTART, VALIDATOR_INFO, NETWORK_MONITOR from indy_common.roles import Roles logger = getlogger() @@ -71,7 +71,7 @@ def generate_auth_map(valid_roles, anyone_can_write=None): class Authoriser: - ValidRoles = (TRUSTEE, STEWARD, TRUST_ANCHOR, None) + ValidRoles = (TRUSTEE, STEWARD, TRUST_ANCHOR, NETWORK_MONITOR, None) auth_map = None diff --git a/indy_common/authorize/auth_map.py b/indy_common/authorize/auth_map.py index 5dde27a0f..67594f022 100644 --- a/indy_common/authorize/auth_map.py +++ b/indy_common/authorize/auth_map.py @@ -1,7 +1,7 @@ from indy_common.authorize.auth_actions import AuthActionAdd, AuthActionEdit from indy_common.authorize.auth_constraints import AuthConstraint, AuthConstraintOr from indy_common.constants import TRUST_ANCHOR, POOL_CONFIG, VALIDATOR_INFO, POOL_UPGRADE, POOL_RESTART, NODE, \ - CLAIM_DEF, SCHEMA, NYM, ROLE + CLAIM_DEF, SCHEMA, NYM, ROLE, NETWORK_MONITOR from plenum.common.constants import TRUSTEE, STEWARD, VERKEY @@ -17,6 +17,10 @@ field=ROLE, value=TRUST_ANCHOR) +addNewNetworkMonitor = AuthActionAdd(txn_type=NYM, + field=ROLE, + value=NETWORK_MONITOR) + addNewIdentityOwner = AuthActionAdd(txn_type=NYM, field=ROLE, @@ -38,6 +42,11 @@ old_value=TRUST_ANCHOR, new_value='') +blacklistingNetworkMonitor = AuthActionEdit(txn_type=NYM, + field=ROLE, + old_value=NETWORK_MONITOR, + new_value='') + sameRoleTrustee = AuthActionEdit(txn_type=NYM, field=ROLE, old_value=TRUSTEE, @@ -53,6 +62,11 @@ old_value=TRUST_ANCHOR, new_value=TRUST_ANCHOR) +sameRoleNetworkMonitor = AuthActionEdit(txn_type=NYM, + field=ROLE, + old_value=NETWORK_MONITOR, + new_value=NETWORK_MONITOR) + sameRoleNone = AuthActionEdit(txn_type=NYM, field=ROLE, old_value='', @@ -175,13 +189,16 @@ addNewSteward.get_action_id(): AuthConstraint(TRUSTEE, 1), addNewTrustAnchor.get_action_id(): AuthConstraintOr([AuthConstraint(TRUSTEE, 1), AuthConstraint(STEWARD, 1)]), + addNewNetworkMonitor.get_action_id(): AuthConstraintOr([AuthConstraint(STEWARD, 1), + AuthConstraint(TRUSTEE, 1)]), addNewIdentityOwner.get_action_id(): AuthConstraintOr([AuthConstraint(TRUSTEE, 1), AuthConstraint(STEWARD, 1), AuthConstraint(TRUST_ANCHOR, 1)]), blacklistingTrustee.get_action_id(): AuthConstraint(TRUSTEE, 1), blacklistingSteward.get_action_id(): AuthConstraint(TRUSTEE, 1), blacklistingTrustAnchor.get_action_id(): AuthConstraint(TRUSTEE, 1), - + blacklistingNetworkMonitor.get_action_id(): AuthConstraintOr([AuthConstraint(STEWARD, 1), + AuthConstraint(TRUSTEE, 1)]), sameRoleTrustee.get_action_id(): AuthConstraint(role='*', sig_count=1, need_to_be_owner=True), @@ -194,6 +211,9 @@ sameRoleNone.get_action_id(): AuthConstraint(role='*', sig_count=1, need_to_be_owner=True), + sameRoleNetworkMonitor.get_action_id(): AuthConstraint(role="*", + sig_count=1, + need_to_be_owner=True), keyRotation.get_action_id(): AuthConstraint(role='*', sig_count=1, need_to_be_owner=True), @@ -220,7 +240,8 @@ poolRestart.get_action_id(): AuthConstraint(TRUSTEE, 1), poolConfig.get_action_id(): AuthConstraint(TRUSTEE, 1), validatorInfo.get_action_id(): AuthConstraintOr([AuthConstraint(TRUSTEE, 1), - AuthConstraint(STEWARD, 1)])} + AuthConstraint(STEWARD, 1), + AuthConstraint(NETWORK_MONITOR, 1)])} anyoneCanWriteMap = {anyoneCanAddNYM.get_action_id(): AuthConstraint(role='*', sig_count=1), diff --git a/indy_common/config.py b/indy_common/config.py index 37af80164..526016a2a 100644 --- a/indy_common/config.py +++ b/indy_common/config.py @@ -103,3 +103,5 @@ PACKAGES_TO_HOLD = ['indy-plenum', 'indy-node', 'python3-indy-crypto', 'libindy-crypto'] authPolicy = LOCAL_AUTH_POLICY + +SCHEMA_ATTRIBUTES_LIMIT = 125 diff --git a/indy_common/constants.py b/indy_common/constants.py index a23e532e0..559ebda75 100644 --- a/indy_common/constants.py +++ b/indy_common/constants.py @@ -126,6 +126,9 @@ TRUST_ANCHOR = Roles.TRUST_ANCHOR.value TRUST_ANCHOR_STRING = 'TRUST_ANCHOR' +NETWORK_MONITOR = Roles.NETWORK_MONITOR.value +NETWORK_MONITOR_STRING = 'NETWORK_MONITOR' + # client transaction types NODE = IndyTransactions.NODE.value NYM = IndyTransactions.NYM.value diff --git a/indy_common/roles.py b/indy_common/roles.py index 1472b099a..52f3ab44d 100644 --- a/indy_common/roles.py +++ b/indy_common/roles.py @@ -11,6 +11,7 @@ class Roles(Enum): TRUSTEE = Roles.TRUSTEE.value STEWARD = Roles.STEWARD.value TRUST_ANCHOR = "101" + NETWORK_MONITOR = "201" def __str__(self): return self.name diff --git a/indy_common/test/auth/conftest.py b/indy_common/test/auth/conftest.py index 8d1ef9924..911949e3e 100644 --- a/indy_common/test/auth/conftest.py +++ b/indy_common/test/auth/conftest.py @@ -9,7 +9,7 @@ from indy_node.persistence.idr_cache import IdrCache from plenum.common.constants import STEWARD, TRUSTEE -from indy_common.constants import TRUST_ANCHOR, LOCAL_AUTH_POLICY +from indy_common.constants import TRUST_ANCHOR, LOCAL_AUTH_POLICY, NETWORK_MONITOR from plenum.common.exceptions import UnauthorizedClientRequest from plenum.test.helper import randomOperation from plenum.test.testing_utils import FakeSomething @@ -49,7 +49,9 @@ def idr_cache(): verkey="steward_identifier_verkey", isCommitted=False) cache.set("trust_anchor_identifier", 3, int(time.time()), role=TRUST_ANCHOR, verkey="trust_anchor_identifier_verkey", isCommitted=False) - cache.set(OTHER_IDENTIFIER, 4, int(time.time()), role='OtherRole', + cache.set("network_monitor_identifier", 4, int(time.time()), role=NETWORK_MONITOR, + verkey="network_monitor_identifier_verkey", isCommitted=False) + cache.set(OTHER_IDENTIFIER, 5, int(time.time()), role='OtherRole', verkey="other_verkey", isCommitted=False) return cache @@ -64,7 +66,9 @@ def write_auth_req_validator(idr_cache): return validator -@pytest.fixture(scope='module', params=["trustee_identifier", "steward_identifier", "trust_anchor_identifier", OTHER_IDENTIFIER]) +@pytest.fixture(scope='module', params=["trustee_identifier", "steward_identifier", + "trust_anchor_identifier", "network_monitor_identifier", + OTHER_IDENTIFIER]) def identifier(request): return request.param diff --git a/indy_common/test/auth/test_auth_nym_with_new_auth_map.py b/indy_common/test/auth/test_auth_nym_with_new_auth_map.py index 91e3c161b..f77383df6 100644 --- a/indy_common/test/auth/test_auth_nym_with_new_auth_map.py +++ b/indy_common/test/auth/test_auth_nym_with_new_auth_map.py @@ -2,7 +2,7 @@ from indy_common.authorize.auth_actions import AuthActionAdd, AuthActionEdit from plenum.common.constants import TRUSTEE, STEWARD, VERKEY -from indy_common.constants import ROLE, NYM, TRUST_ANCHOR +from indy_common.constants import ROLE, NYM, TRUST_ANCHOR, NETWORK_MONITOR @pytest.fixture(scope='module', params=[True, False]) @@ -37,6 +37,15 @@ def test_make_trust_anchor(write_request_validation, req, is_owner): is_owner=is_owner)]) +def test_make_network_monitor(write_request_validation, req, is_owner): + authorized = req.identifier in ("trustee_identifier", "steward_identifier") + assert authorized == write_request_validation(req, + [AuthActionAdd(txn_type=NYM, + field=ROLE, + value=NETWORK_MONITOR, + is_owner=is_owner)]) + + def test_remove_trustee(write_request_validation, req, is_owner): authorized = (req.identifier == "trustee_identifier") assert authorized == write_request_validation(req, @@ -67,6 +76,16 @@ def test_remove_trust_anchor(write_request_validation, req, is_owner): is_owner=is_owner)]) +def test_remove_network_monitor(write_request_validation, req, is_owner): + authorized = req.identifier in ("trustee_identifier", "steward_identifier") + assert authorized == write_request_validation(req, + [AuthActionEdit(txn_type=NYM, + field=ROLE, + old_value=NETWORK_MONITOR, + new_value='', + is_owner=is_owner)]) + + def test_change_verkey(write_request_validation, req, is_owner): authorized = is_owner assert authorized == write_request_validation(req, @@ -75,3 +94,43 @@ def test_change_verkey(write_request_validation, req, is_owner): old_value="_verkey".format(req.identifier), new_value='new_value', is_owner=is_owner)]) + + +def test_same_role_trustee(write_request_validation, req, is_owner): + authorized = is_owner + assert authorized == write_request_validation(req, + [AuthActionEdit(txn_type=NYM, + field=ROLE, + old_value=TRUSTEE, + new_value=TRUSTEE, + is_owner=is_owner)]) + + +def test_same_role_steward(write_request_validation, req, is_owner): + authorized = is_owner + assert authorized == write_request_validation(req, + [AuthActionEdit(txn_type=NYM, + field=ROLE, + old_value=STEWARD, + new_value=STEWARD, + is_owner=is_owner)]) + + +def test_same_role_trust_acnhor(write_request_validation, req, is_owner): + authorized = is_owner + assert authorized == write_request_validation(req, + [AuthActionEdit(txn_type=NYM, + field=ROLE, + old_value=TRUST_ANCHOR, + new_value=TRUST_ANCHOR, + is_owner=is_owner)]) + + +def test_same_role_network_monitor(write_request_validation, req, is_owner): + authorized = is_owner + assert authorized == write_request_validation(req, + [AuthActionEdit(txn_type=NYM, + field=ROLE, + old_value=NETWORK_MONITOR, + new_value=NETWORK_MONITOR, + is_owner=is_owner)]) diff --git a/indy_common/test/auth/test_auth_validator_info_with_new_auth_map.py b/indy_common/test/auth/test_auth_validator_info_with_new_auth_map.py index 2503c85e8..d7c951a29 100644 --- a/indy_common/test/auth/test_auth_validator_info_with_new_auth_map.py +++ b/indy_common/test/auth/test_auth_validator_info_with_new_auth_map.py @@ -3,7 +3,7 @@ def test_pool_restart_add_action(write_request_validation, is_owner, req): - authorized = req.identifier in ("trustee_identifier", "steward_identifier") + authorized = req.identifier in ("trustee_identifier", "steward_identifier", "network_monitor_identifier") assert authorized == write_request_validation(req, [AuthActionAdd(txn_type=VALIDATOR_INFO, field='some_field', diff --git a/indy_common/types.py b/indy_common/types.py index 82f154fed..341f25001 100644 --- a/indy_common/types.py +++ b/indy_common/types.py @@ -2,6 +2,7 @@ from copy import deepcopy from hashlib import sha256 +from indy_common.config import SCHEMA_ATTRIBUTES_LIMIT from indy_common.constants import TXN_TYPE, ATTRIB, GET_ATTR, \ DATA, GET_NYM, GET_SCHEMA, GET_CLAIM_DEF, ACTION, \ POOL_UPGRADE, POOL_CONFIG, \ @@ -80,7 +81,8 @@ class SchemaField(MessageValidator): (SCHEMA_VERSION, VersionField(components_number=(2, 3,), max_length=VERSION_FIELD_LIMIT)), (SCHEMA_ATTR_NAMES, IterableField( LimitedLengthStringField(max_length=NAME_FIELD_LIMIT), - min_length=1)), + min_length=1, + max_length=SCHEMA_ATTRIBUTES_LIMIT)), ) diff --git a/indy_node/server/request_handlers/action_req_handlers/pool_restart_handler.py b/indy_node/server/request_handlers/action_req_handlers/pool_restart_handler.py index aaaa20b69..18116e16e 100644 --- a/indy_node/server/request_handlers/action_req_handlers/pool_restart_handler.py +++ b/indy_node/server/request_handlers/action_req_handlers/pool_restart_handler.py @@ -1,5 +1,46 @@ +from indy_common.authorize.auth_actions import AuthActionAdd +from indy_common.authorize.auth_request_validator import WriteRequestValidator +from indy_node.server.request_handlers.action_req_handlers.utils import generate_action_result +from indy_node.server.restarter import Restarter + +from indy_common.constants import POOL_RESTART, ACTION + +from plenum.common.request import Request +from plenum.common.txn_util import get_request_data +from plenum.server.database_manager import DatabaseManager from plenum.server.request_handlers.handler_interfaces.action_request_handler import ActionRequestHandler +from stp_core.common.log import getlogger + +logger = getlogger() class PoolRestartHandler(ActionRequestHandler): - pass + + def __init__(self, database_manager: DatabaseManager, + write_req_validator: WriteRequestValidator, + restarter: Restarter): + super().__init__(database_manager, POOL_RESTART, None) + self.write_req_validator = write_req_validator + self.restarter = restarter + + def static_validation(self, request: Request): + self._validate_request_type(request) + + def dynamic_validation(self, request: Request): + self._validate_request_type(request) + action = request.operation.get(ACTION) + self.write_req_validator.validate(request, + [AuthActionAdd(txn_type=POOL_RESTART, + field=ACTION, + value=action)]) + + def process_action(self, request: Request): + self._validate_request_type(request) + identifier, req_id, operation = get_request_data(request) + logger.debug("Transaction {} with type {} started" + .format(req_id, request.txn_type)) + self.restarter.handleRestartRequest(request) + result = generate_action_result(request) + logger.debug("Transaction {} with type {} finished" + .format(req_id, request.txn_type)) + return result diff --git a/indy_node/server/request_handlers/action_req_handlers/utils.py b/indy_node/server/request_handlers/action_req_handlers/utils.py new file mode 100644 index 000000000..b8e03c0db --- /dev/null +++ b/indy_node/server/request_handlers/action_req_handlers/utils.py @@ -0,0 +1,8 @@ +from plenum.common.request import Request +from plenum.common.types import f + + +def generate_action_result(request: Request): + return {**request.operation, **{ + f.IDENTIFIER.nm: request.identifier, + f.REQ_ID.nm: request.reqId}} diff --git a/indy_node/server/request_handlers/action_req_handlers/validator_info_handler.py b/indy_node/server/request_handlers/action_req_handlers/validator_info_handler.py index 9c899ba84..04854890a 100644 --- a/indy_node/server/request_handlers/action_req_handlers/validator_info_handler.py +++ b/indy_node/server/request_handlers/action_req_handlers/validator_info_handler.py @@ -1,5 +1,47 @@ +from indy_common.authorize.auth_actions import AuthActionAdd +from indy_common.authorize.auth_request_validator import WriteRequestValidator +from indy_common.constants import VALIDATOR_INFO +from indy_node.server.request_handlers.action_req_handlers.utils import generate_action_result +from plenum.common.constants import DATA +from plenum.common.request import Request +from plenum.common.txn_util import get_request_data +from plenum.server.database_manager import DatabaseManager + from plenum.server.request_handlers.handler_interfaces.action_request_handler import ActionRequestHandler +from stp_core.common.log import getlogger + +logger = getlogger() class ValidatorInfoHandler(ActionRequestHandler): - pass + def __init__(self, database_manager: DatabaseManager, + write_req_validator: WriteRequestValidator, + info_tool): + super().__init__(database_manager, VALIDATOR_INFO, None) + self.write_req_validator = write_req_validator + self.info_tool = info_tool + + def static_validation(self, request: Request): + pass + + def dynamic_validation(self, request: Request): + self._validate_request_type(request) + self.write_req_validator.validate(request, + [AuthActionAdd(txn_type=VALIDATOR_INFO, + field='*', + value='*')]) + + def process_action(self, request: Request): + self._validate_request_type(request) + identifier, req_id, operation = get_request_data(request) + logger.debug("Transaction {} with type {} started" + .format(req_id, request.txn_type)) + result = generate_action_result(request) + result[DATA] = self.info_tool.info + result[DATA].update(self.info_tool.memory_profiler) + result[DATA].update(self.info_tool._generate_software_info()) + result[DATA].update(self.info_tool.extractions) + result[DATA].update(self.info_tool.node_disk_size) + logger.debug("Transaction {} with type {} finished" + .format(req_id, request.txn_type)) + return result diff --git a/indy_node/server/request_handlers/config_batch_handler.py b/indy_node/server/request_handlers/config_batch_handler.py new file mode 100644 index 000000000..65323f3f2 --- /dev/null +++ b/indy_node/server/request_handlers/config_batch_handler.py @@ -0,0 +1,27 @@ +from indy_node.server.pool_config import PoolConfig + +from indy_common.constants import CONFIG_LEDGER_ID +from indy_node.server.upgrader import Upgrader +from plenum.common.txn_util import is_forced +from plenum.server.batch_handlers.batch_request_handler import BatchRequestHandler +from plenum.server.database_manager import DatabaseManager + + +class ConfigBatchHandler(BatchRequestHandler): + def __init__(self, database_manager: DatabaseManager, + upgrader: Upgrader, pool_config: PoolConfig): + super().__init__(database_manager, CONFIG_LEDGER_ID) + self.upgrader = upgrader + self.pool_config = pool_config + + def commit_batch(self, txn_count, state_root, txn_root, pp_time, prev_result): + committed_txns = super().commit_batch(txn_count, state_root, txn_root, pp_time, prev_result) + for txn in committed_txns: + # Handle POOL_UPGRADE or POOL_CONFIG transaction here + # only in case it is not forced. + # If it is forced then it was handled earlier + # in applyForced method. + if not is_forced(txn): + self.upgrader.handleUpgradeTxn(txn) + self.pool_config.handleConfigTxn(txn) + return committed_txns diff --git a/indy_node/server/request_handlers/config_req_handlers/config_write_request_handler.py b/indy_node/server/request_handlers/config_req_handlers/config_write_request_handler.py new file mode 100644 index 000000000..9819e7b42 --- /dev/null +++ b/indy_node/server/request_handlers/config_req_handlers/config_write_request_handler.py @@ -0,0 +1,14 @@ +from indy_common.types import Request +from plenum.common.txn_util import append_txn_metadata +from plenum.server.request_handlers.handler_interfaces.write_request_handler import WriteRequestHandler + + +class ConfigWriteRequestHandler(WriteRequestHandler): + + def apply_request(self, request: Request, batch_ts, prev_result): + self._validate_request_type(request) + txn = append_txn_metadata(self._req_to_txn(request), + txn_time=batch_ts) + self.ledger.append_txns_metadata([txn]) + (start, _), _ = self.ledger.appendTxns([txn]) + return start, txn diff --git a/indy_node/server/request_handlers/config_req_handlers/node_upgrade_handler.py b/indy_node/server/request_handlers/config_req_handlers/node_upgrade_handler.py deleted file mode 100644 index 675a22fb7..000000000 --- a/indy_node/server/request_handlers/config_req_handlers/node_upgrade_handler.py +++ /dev/null @@ -1,5 +0,0 @@ -from plenum.server.request_handlers.handler_interfaces.write_request_handler import WriteRequestHandler - - -class NodeUpgradeHandler(WriteRequestHandler): - pass diff --git a/indy_node/server/request_handlers/config_req_handlers/pool_config_handler.py b/indy_node/server/request_handlers/config_req_handlers/pool_config_handler.py index f6bc03f00..84fa75912 100644 --- a/indy_node/server/request_handlers/config_req_handlers/pool_config_handler.py +++ b/indy_node/server/request_handlers/config_req_handlers/pool_config_handler.py @@ -1,5 +1,43 @@ -from plenum.server.request_handlers.handler_interfaces.write_request_handler import WriteRequestHandler +from indy_node.server.pool_config import PoolConfig +from indy_common.authorize.auth_actions import AuthActionEdit +from indy_common.authorize.auth_request_validator import WriteRequestValidator +from indy_common.constants import POOL_CONFIG, CONFIG_LEDGER_ID, ACTION +from indy_node.server.request_handlers.config_req_handlers.config_write_request_handler import ConfigWriteRequestHandler +from plenum.common.request import Request +from plenum.server.database_manager import DatabaseManager -class PoolConfigHandler(WriteRequestHandler): - pass + +class PoolConfigHandler(ConfigWriteRequestHandler): + + def __init__(self, database_manager: DatabaseManager, + write_request_validator: WriteRequestValidator, + pool_config: PoolConfig): + super().__init__(database_manager, POOL_CONFIG, CONFIG_LEDGER_ID) + self.write_request_validator = write_request_validator + self.pool_config = pool_config + + def static_validation(self, request: Request): + self._validate_request_type(request) + + def dynamic_validation(self, request: Request): + self._validate_request_type(request) + action = '*' + status = '*' + self.write_request_validator.validate(request, + [AuthActionEdit(txn_type=self.txn_type, + field=ACTION, + old_value=status, + new_value=action)]) + + def apply_forced_request(self, req: Request): + super().apply_forced_request(req) + txn = self._req_to_txn(req) + self.pool_config.handleConfigTxn(txn) + + # Config handler don't use state for any validation for now + def update_state(self, txn, prev_result, is_committed=False): + pass + + def gen_state_key(self, txn): + pass diff --git a/indy_node/server/request_handlers/config_req_handlers/pool_upgrade_handler.py b/indy_node/server/request_handlers/config_req_handlers/pool_upgrade_handler.py index a39167e1a..21886c130 100644 --- a/indy_node/server/request_handlers/config_req_handlers/pool_upgrade_handler.py +++ b/indy_node/server/request_handlers/config_req_handlers/pool_upgrade_handler.py @@ -1,5 +1,119 @@ -from plenum.server.request_handlers.handler_interfaces.write_request_handler import WriteRequestHandler +from indy_common.authorize.auth_actions import AuthActionAdd, AuthActionEdit +from indy_node.server.request_handlers.config_req_handlers.config_write_request_handler import ConfigWriteRequestHandler +from indy_node.utils.node_control_utils import NodeControlUtil +from indy_common.config_util import getConfig -class PoolUpgrade(WriteRequestHandler): - pass +from indy_common.constants import CONFIG_LEDGER_ID, POOL_UPGRADE, \ + ACTION, CANCEL, START, SCHEDULE, PACKAGE, APP_NAME, REINSTALL + +from indy_common.authorize.auth_request_validator import WriteRequestValidator +from indy_node.server.upgrader import Upgrader +from plenum.common.constants import FORCE, VERSION, NAME +from plenum.common.exceptions import InvalidClientRequest +from plenum.common.request import Request +from plenum.common.txn_util import get_request_data, get_payload_data +from plenum.server.database_manager import DatabaseManager +from plenum.server.pool_manager import TxnPoolManager + + +class PoolUpgradeHandler(ConfigWriteRequestHandler): + + def __init__(self, database_manager: DatabaseManager, + upgrader: Upgrader, + write_request_validator: WriteRequestValidator, + pool_manager: TxnPoolManager): + super().__init__(database_manager, POOL_UPGRADE, CONFIG_LEDGER_ID) + self.upgrader = upgrader + self.write_request_validator = write_request_validator + self.pool_manager = pool_manager + + def static_validation(self, request: Request): + self._validate_request_type(request) + identifier, req_id, operation = get_request_data(request) + action = operation.get(ACTION) + if action not in (START, CANCEL): + raise InvalidClientRequest(identifier, req_id, + "{} not a valid action". + format(action)) + if action == START: + schedule = operation.get(SCHEDULE, {}) + force = operation.get(FORCE) + force = str(force) == 'True' + isValid, msg = self.upgrader.isScheduleValid( + schedule, self.pool_manager.getNodesServices(), force) + if not isValid: + raise InvalidClientRequest(identifier, req_id, + "{} not a valid schedule since {}". + format(schedule, msg)) + + def dynamic_validation(self, request: Request): + self._validate_request_type(request) + identifier, req_id, operation = get_request_data(request) + status = '*' + pkt_to_upgrade = operation.get(PACKAGE, getConfig().UPGRADE_ENTRY) + if pkt_to_upgrade: + currentVersion, cur_deps = self.curr_pkt_info(pkt_to_upgrade) + if not currentVersion: + raise InvalidClientRequest(identifier, req_id, + "Packet {} is not installed and cannot be upgraded". + format(pkt_to_upgrade)) + if all([APP_NAME not in d for d in cur_deps]): + raise InvalidClientRequest(identifier, req_id, + "Packet {} doesn't belong to pool".format(pkt_to_upgrade)) + else: + raise InvalidClientRequest(identifier, req_id, "Upgrade packet name is empty") + + targetVersion = operation[VERSION] + reinstall = operation.get(REINSTALL, False) + if not Upgrader.is_version_upgradable(currentVersion, targetVersion, reinstall): + # currentVersion > targetVersion + raise InvalidClientRequest(identifier, req_id, "Version is not upgradable") + + action = operation.get(ACTION) + # TODO: Some validation needed for making sure name and version + # present + txn = self.upgrader.get_upgrade_txn( + lambda txn: get_payload_data(txn).get( + NAME, + None) == operation.get( + NAME, + None) and get_payload_data(txn).get(VERSION) == operation.get(VERSION), + reverse=True) + if txn: + status = get_payload_data(txn).get(ACTION, '*') + + if status == START and action == START: + raise InvalidClientRequest( + identifier, + req_id, + "Upgrade '{}' is already scheduled".format( + operation.get(NAME))) + if status == '*': + auth_action = AuthActionAdd(txn_type=POOL_UPGRADE, + field=ACTION, + value=action) + else: + auth_action = AuthActionEdit(txn_type=POOL_UPGRADE, + field=ACTION, + old_value=status, + new_value=action) + self.write_request_validator.validate(request, + [auth_action]) + + def apply_forced_request(self, req: Request): + super().apply_forced_request(req) + txn = self._req_to_txn(req) + self.upgrader.handleUpgradeTxn(txn) + + def curr_pkt_info(self, pkg_name): + if pkg_name == APP_NAME: + return Upgrader.getVersion(), [APP_NAME] + return NodeControlUtil.curr_pkt_info(pkg_name) + + # Config handler don't use state for any validation for now + def update_state(self, txn, prev_result, is_committed=False): + pass + + def gen_state_key(self, txn): + pass diff --git a/indy_node/server/request_handlers/domain_req_handlers/claim_def_handler.py b/indy_node/server/request_handlers/domain_req_handlers/claim_def_handler.py index c80be5cc9..7f4569d2d 100644 --- a/indy_node/server/request_handlers/domain_req_handlers/claim_def_handler.py +++ b/indy_node/server/request_handlers/domain_req_handlers/claim_def_handler.py @@ -4,21 +4,22 @@ from indy_common.state import domain from indy_common.constants import CLAIM_DEF, REF, SCHEMA -from indy_node.server.request_handlers.write_request_handler import WriteRequestHandler from plenum.common.constants import DOMAIN_LEDGER_ID from plenum.common.exceptions import InvalidClientRequest from plenum.common.request import Request from plenum.common.txn_util import get_request_data + from plenum.server.database_manager import DatabaseManager +from plenum.server.request_handlers.handler_interfaces.write_request_handler import WriteRequestHandler class ClaimDefHandler(WriteRequestHandler): def __init__(self, database_manager: DatabaseManager, write_request_validator: WriteRequestValidator): - super().__init__(database_manager, CLAIM_DEF, DOMAIN_LEDGER_ID, - write_request_validator) + super().__init__(database_manager, CLAIM_DEF, DOMAIN_LEDGER_ID) + self.write_request_validator = write_request_validator def static_validation(self, request: Request): pass diff --git a/indy_node/server/request_handlers/domain_req_handlers/nym_handler.py b/indy_node/server/request_handlers/domain_req_handlers/nym_handler.py index 3d3c8c78e..d1b9fc78f 100644 --- a/indy_node/server/request_handlers/domain_req_handlers/nym_handler.py +++ b/indy_node/server/request_handlers/domain_req_handlers/nym_handler.py @@ -6,7 +6,6 @@ from indy_common.state import domain from indy_common.constants import NYM from indy_common.auth import Authoriser -from indy_node.server.request_handlers.write_request_handler import WriteRequestHandler from ledger.util import F from plenum.common.constants import ROLE, TARGET_NYM, VERKEY, DOMAIN_LEDGER_ID, TXN_TIME @@ -16,6 +15,7 @@ from plenum.common.types import f from plenum.server.database_manager import DatabaseManager from plenum.server.request_handlers.utils import nym_to_state_key, get_nym_details +from plenum.server.request_handlers.handler_interfaces.write_request_handler import WriteRequestHandler class NymHandler(WriteRequestHandler): @@ -23,7 +23,8 @@ class NymHandler(WriteRequestHandler): def __init__(self, database_manager: DatabaseManager, write_request_validator: WriteRequestValidator): - super().__init__(database_manager, NYM, DOMAIN_LEDGER_ID, write_request_validator) + super().__init__(database_manager, NYM, DOMAIN_LEDGER_ID) + self.write_request_validator = write_request_validator def static_validation(self, request: Request): self._validate_request_type(request) diff --git a/indy_node/server/request_handlers/domain_req_handlers/schema_handler.py b/indy_node/server/request_handlers/domain_req_handlers/schema_handler.py index 739d7987e..ac18d8f6a 100644 --- a/indy_node/server/request_handlers/domain_req_handlers/schema_handler.py +++ b/indy_node/server/request_handlers/domain_req_handlers/schema_handler.py @@ -6,20 +6,21 @@ from indy_common.req_utils import get_write_schema_name, get_write_schema_version from indy_node.server.request_handlers.read_req_handlers.get_schema_handler import GetSchemaHandler -from indy_node.server.request_handlers.write_request_handler import WriteRequestHandler from plenum.common.constants import DOMAIN_LEDGER_ID from plenum.common.exceptions import InvalidClientRequest from plenum.common.request import Request from plenum.common.txn_util import get_request_data from plenum.server.database_manager import DatabaseManager +from plenum.server.request_handlers.handler_interfaces.write_request_handler import WriteRequestHandler class SchemaHandler(WriteRequestHandler): def __init__(self, database_manager: DatabaseManager, get_schema_handler: GetSchemaHandler, write_request_validator: WriteRequestValidator): - super().__init__(database_manager, SCHEMA, DOMAIN_LEDGER_ID, write_request_validator) + super().__init__(database_manager, SCHEMA, DOMAIN_LEDGER_ID) + self.write_request_validator = write_request_validator self.get_schema_handler = get_schema_handler def static_validation(self, request: Request): diff --git a/indy_node/server/request_handlers/pool_req_handlers/__init__.py b/indy_node/server/request_handlers/pool_req_handlers/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/indy_node/server/request_handlers/pool_req_handlers/node_handler.py b/indy_node/server/request_handlers/pool_req_handlers/node_handler.py new file mode 100644 index 000000000..119c901a4 --- /dev/null +++ b/indy_node/server/request_handlers/pool_req_handlers/node_handler.py @@ -0,0 +1,79 @@ +from copy import deepcopy + +from indy_common.constants import NODE + +from indy_common.authorize.auth_actions import AuthActionAdd, AuthActionEdit +from indy_common.authorize.auth_request_validator import WriteRequestValidator +from plenum.common.constants import DATA, SERVICES, VALIDATOR, TARGET_NYM, ALIAS, BLS_KEY_PROOF +from plenum.server.database_manager import DatabaseManager +from plenum.server.request_handlers.node_handler import NodeHandler as PNodeHandler + + +class NodeHandler(PNodeHandler): + + def __init__(self, database_manager: DatabaseManager, bls_crypto_verifier, + write_request_validator: WriteRequestValidator): + super().__init__(database_manager, bls_crypto_verifier) + self.write_request_validator = write_request_validator + + def _is_steward(self, nym, is_committed: bool = True): + return self.database_manager.idr_cache.hasSteward(nym, is_committed) + + def _auth_error_while_adding_node(self, request): + origin = request.identifier + operation = request.operation + data = operation.get(DATA, {}) + error = self._data_error_while_validating(data, skip_keys=False) + if error: + return error + + if self._steward_has_node(origin): + return "{} already has a node".format(origin) + error = self._is_node_data_conflicting(data) + if error: + return "existing data has conflicts with " \ + "request data {}. Error: {}".format(operation.get(DATA), error) + self.write_request_validator.validate(request, + [AuthActionAdd(txn_type=NODE, + field=SERVICES, + value=data.get(SERVICES, [VALIDATOR]))]) + + def _auth_error_while_updating_node(self, request): + origin = request.identifier + is_trustee = self.database_manager.idr_cache.hasTrustee(origin, isCommitted=False) + if not is_trustee: + error = super()._auth_error_while_updating_node(request) + if error: + return error + origin = request.identifier + operation = request.operation + node_nym = operation.get(TARGET_NYM) + + data = operation.get(DATA, {}) + error = self._data_error_while_validating_update(data, node_nym) + if error: + return error + + is_steward_of_node = self._is_steward_of_node( + origin, node_nym, is_committed=False) + + node_info = self._get_node_data(node_nym, is_committed=False) + data = deepcopy(data) + data.pop(ALIAS, None) + for k in data: + if k == BLS_KEY_PROOF: + continue + old_val = node_info.get(k, None) if node_info else None + new_val = data[k] + if k == SERVICES: + if not old_val: + old_val = [] + if not new_val: + new_val = [] + if old_val != new_val: + self.write_request_validator.validate(request, + [AuthActionEdit(txn_type=NODE, + field=k, + old_value=old_val, + new_value=new_val, + is_owner=is_steward_of_node)]) diff --git a/indy_node/server/request_handlers/write_request_handler.py b/indy_node/server/request_handlers/write_request_handler.py deleted file mode 100644 index 8f38e972f..000000000 --- a/indy_node/server/request_handlers/write_request_handler.py +++ /dev/null @@ -1,12 +0,0 @@ -from indy_common.authorize.auth_request_validator import WriteRequestValidator -from plenum.server.database_manager import DatabaseManager -from plenum.server.request_handlers.handler_interfaces.write_request_handler import \ - WriteRequestHandler as PWriteRequestHandler - - -class WriteRequestHandler(PWriteRequestHandler): - - def __init__(self, database_manager: DatabaseManager, txn_type, ledger_id, - write_request_validator: WriteRequestValidator): - super().__init__(database_manager, txn_type, ledger_id) - self.write_request_validator = write_request_validator diff --git a/indy_node/server/upgrader.py b/indy_node/server/upgrader.py index e53456601..f167b8971 100644 --- a/indy_node/server/upgrader.py +++ b/indy_node/server/upgrader.py @@ -375,7 +375,6 @@ async def _sendUpgradeRequest(self, when, version, upgrade_id, failTimeout, pkg_ upgrade_id=upgrade_id, reason="problems in communication with node control service") self._unscheduleAction() - self._actionFailedCallback() else: logger.info("Waiting {} minutes for upgrade to be performed".format(failTimeout)) timesUp = partial(self._declareTimeoutExceeded, when, version, upgrade_id) @@ -449,7 +448,7 @@ def isScheduleValid(self, schedule, node_srvs, force) -> (bool, str): return True, '' times = sorted(times) for i in range(len(times) - 1): - diff = (times[i + 1] - times[i]).seconds + diff = (times[i + 1] - times[i]).total_seconds() if diff < self.config.MinSepBetweenNodeUpgrades: return False, 'time span between upgrades is {} ' \ 'seconds which is less than specified ' \ diff --git a/indy_node/server/validator_info_tool.py b/indy_node/server/validator_info_tool.py index 52900b46a..dc297390a 100644 --- a/indy_node/server/validator_info_tool.py +++ b/indy_node/server/validator_info_tool.py @@ -16,7 +16,7 @@ def info(self): ts_str = "{}".format(time.strftime( "%A, %B %{0}d, %Y %{0}I:%M:%S %p %z".format('#' if os.name == 'nt' else '-'), time.localtime(info["timestamp"]))) - info.update({"Update time": ts_str}) + info.update({"Update_time": ts_str}) if 'Node_info' in info: if 'Metrics' in info['Node_info']: std_ledgers = [POOL_LEDGER_ID, DOMAIN_LEDGER_ID, CONFIG_LEDGER_ID] diff --git a/indy_node/test/catchup/test_batch_rejected_and_later_ordered_on_catchup.py b/indy_node/test/catchup/test_batch_rejected_and_later_ordered_on_catchup.py index 0474e528a..f1271e2e4 100644 --- a/indy_node/test/catchup/test_batch_rejected_and_later_ordered_on_catchup.py +++ b/indy_node/test/catchup/test_batch_rejected_and_later_ordered_on_catchup.py @@ -53,7 +53,7 @@ def test_batch_rejected_on_catchup_start_can_be_ordered_before_ledgers_sync( disable_transport_batching): """ Verifies that a batch rejected due to catch-up start can be successfully - re-applied and ordered later before ledgers synchronization without any + ordered later before ledgers synchronization without any warnings. In the test we perform stashing / unstashing messages and patching / @@ -75,8 +75,7 @@ def test_batch_rejected_on_catchup_start_can_be_ordered_before_ledgers_sync( since it is not in participating mode. 11. The slow node receives LEDGER_STATUS messages from the other nodes. 12. Prior to ledgers synchronization the slow node processes the stashed - ORDER messages. When it is processing the ORDER message from the master - replica, it re-applies the batch and orders it. + Commits messages and orders batch. 13. The slow node synchronizes its ledgers and completes the catch-up. """ @@ -86,8 +85,6 @@ def test_batch_rejected_on_catchup_start_can_be_ordered_before_ledgers_sync( slow_node.nodeIbStasher.delay(cDelay(300)) slow_node.start_catchup = MethodType(patched_start_catchup, slow_node) - send_random_requests(looper, sdk_pool_handle, sdk_wallet_trust_anchor, 1) - no_more_catchups_needed_call_times_before = \ slow_node.spylog.count(Node.no_more_catchups_needed.__name__) on_batch_rejected_call_times_before = \ @@ -97,6 +94,8 @@ def test_batch_rejected_on_catchup_start_can_be_ordered_before_ledgers_sync( process_ordered_call_times_before = \ slow_node.spylog.count(Node.processOrdered.__name__) + send_random_requests(looper, sdk_pool_handle, sdk_wallet_trust_anchor, 1) + slow_node.start_catchup() def check_catchup_done(): @@ -119,7 +118,7 @@ def check_catchup_done(): - on_batch_created_call_times_before == 1 assert process_ordered_call_times_after \ - - process_ordered_call_times_before == 2 # one per replica + - process_ordered_call_times_before == 1 # one per replica last_2_process_ordered_results = \ [call.result for call in slow_node.spylog.getAll(Node.processOrdered.__name__)[-2:]] diff --git a/indy_node/test/conftest.py b/indy_node/test/conftest.py index 33bde3a0e..55fd594ea 100644 --- a/indy_node/test/conftest.py +++ b/indy_node/test/conftest.py @@ -35,7 +35,7 @@ from plenum.test.conftest import sdk_pool_handle as plenum_pool_handle, sdk_pool_data, sdk_wallet_steward, \ sdk_wallet_handle, sdk_wallet_data, sdk_steward_seed, sdk_wallet_client, sdk_wallet_trustee, \ sdk_trustee_seed, trustee_data, sdk_client_seed, poolTxnClientData, poolTxnClientNames, \ - sdk_wallet_stewards, create_node_and_not_start + sdk_wallet_stewards, create_node_and_not_start, sdk_wallet_handle Logger.setLogLevel(logging.NOTSET) diff --git a/indy_node/test/helper.py b/indy_node/test/helper.py index 30005c2c0..e4cda3102 100644 --- a/indy_node/test/helper.py +++ b/indy_node/test/helper.py @@ -21,7 +21,6 @@ logger = getlogger() - @spyable(methods=[Upgrader.processLedger]) class TestUpgrader(Upgrader): pass @@ -50,7 +49,9 @@ def __init__(self, *args, **kwargs): def init_upgrader(self): return TestUpgrader(self.id, self.name, self.dataLocation, self.config, - self.configLedger) + self.configLedger, + actionFailedCallback=self.postConfigLedgerCaughtUp, + action_start_callback=self.notify_upgrade_start) def init_domain_req_handler(self): return Node.init_domain_req_handler(self) diff --git a/indy_node/test/nym_txn/conftest.py b/indy_node/test/nym_txn/conftest.py new file mode 100644 index 000000000..549a54307 --- /dev/null +++ b/indy_node/test/nym_txn/conftest.py @@ -0,0 +1,6 @@ +import pytest + + +@pytest.fixture(scope="function", params=[False, True]) +def with_verkey(request): + return request.param diff --git a/indy_node/test/nym_txn/test_demote_network_monitor.py b/indy_node/test/nym_txn/test_demote_network_monitor.py new file mode 100644 index 000000000..19e2c6018 --- /dev/null +++ b/indy_node/test/nym_txn/test_demote_network_monitor.py @@ -0,0 +1,78 @@ +import pytest +from indy import did + +from indy_common.constants import NETWORK_MONITOR +from indy_node.test.validator_info.helper import sdk_get_validator_info +from plenum.common.constants import STEWARD_STRING +from plenum.common.exceptions import RequestRejectedException +from plenum.test.helper import sdk_sign_and_submit_op, sdk_get_and_check_replies +from plenum.test.pool_transactions.helper import sdk_add_new_nym + + +def test_network_monitor_suspension_by_another_steward(looper, + sdk_pool_handle, + sdk_wallet_steward, + sdk_wallet_trustee, + sdk_wallet_handle, + with_verkey): + new_steward_did, new_steward_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_trustee[0], "{}")) + new_network_monitor_did, new_network_monitor_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_steward[0], "{}")) + + """Adding new steward""" + sdk_add_new_nym(looper, sdk_pool_handle, + sdk_wallet_trustee, 'newSteward', STEWARD_STRING, verkey=new_steward_verkey, dest=new_steward_did) + + """Adding NETWORK_MONITOR role by first steward""" + op = {'type': '1', + 'dest': new_network_monitor_did, + 'role': NETWORK_MONITOR, + 'verkey': new_network_monitor_verkey} + req = sdk_sign_and_submit_op(looper, sdk_pool_handle, (sdk_wallet_handle, new_steward_did), op) + sdk_get_and_check_replies(looper, [req]) + + """Check that get_validator_info command works for NETWORK_MONITOR role""" + sdk_get_validator_info(looper, (sdk_wallet_handle, new_network_monitor_did), sdk_pool_handle) + + """Blacklisting network_monitor by new steward""" + op = {'type': '1', + 'dest': new_network_monitor_did, + 'role': None} + if with_verkey: + op['verkey'] = new_network_monitor_verkey + req = sdk_sign_and_submit_op(looper, sdk_pool_handle, (sdk_wallet_handle, new_steward_did), op) + if with_verkey: + with pytest.raises(RequestRejectedException): + sdk_get_and_check_replies(looper, [req]) + else: + sdk_get_and_check_replies(looper, [req]) + with pytest.raises(RequestRejectedException): + sdk_get_validator_info(looper, (sdk_wallet_handle, new_network_monitor_did), sdk_pool_handle) + + +def test_network_monitor_suspension_by_itself(looper, + sdk_pool_handle, + sdk_wallet_steward, + sdk_wallet_handle, + with_verkey): + new_network_monitor_did, new_network_monitor_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_steward[0], "{}")) + + """Adding NETWORK_MONITOR role by steward""" + op = {'type': '1', + 'dest': new_network_monitor_did, + 'role': NETWORK_MONITOR, + 'verkey': new_network_monitor_verkey} + req = sdk_sign_and_submit_op(looper, sdk_pool_handle, (sdk_wallet_handle, sdk_wallet_steward[1]), op) + sdk_get_and_check_replies(looper, [req]) + + """Blacklisting network_monitor by itself""" + op = {'type': '1', + 'dest': new_network_monitor_did, + 'role': None} + if with_verkey: + op['verkey'] = new_network_monitor_verkey + req = sdk_sign_and_submit_op(looper, sdk_pool_handle, (sdk_wallet_handle, new_network_monitor_did), op) + with pytest.raises(RequestRejectedException): + sdk_get_and_check_replies(looper, [req]) \ No newline at end of file diff --git a/indy_node/test/nym_txn/test_nym_blacklisting.py b/indy_node/test/nym_txn/test_nym_blacklisting.py new file mode 100644 index 000000000..3491112a6 --- /dev/null +++ b/indy_node/test/nym_txn/test_nym_blacklisting.py @@ -0,0 +1,82 @@ +import pytest +from indy import did + +from indy_common.constants import TRUST_ANCHOR_STRING +from plenum.common.constants import TRUSTEE_STRING, STEWARD_STRING +from plenum.common.exceptions import RequestRejectedException +from plenum.test.helper import sdk_get_and_check_replies, sdk_sign_and_submit_op +from plenum.test.pool_transactions.helper import sdk_add_new_nym + + +def test_steward_suspension_by_another_trustee(looper, + sdk_pool_handle, + sdk_wallet_trustee, + sdk_wallet_handle, + with_verkey): + new_trustee_did, new_trustee_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_trustee[0], "{}")) + new_steward_did, new_steward_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_trustee[0], "{}")) + + """Adding new steward""" + sdk_add_new_nym(looper, sdk_pool_handle, + sdk_wallet_trustee, 'newSteward', STEWARD_STRING, verkey=new_steward_verkey, dest=new_steward_did) + + """Adding new trustee""" + sdk_add_new_nym(looper, sdk_pool_handle, + sdk_wallet_trustee, 'newTrustee', TRUSTEE_STRING, verkey=new_trustee_verkey, dest=new_trustee_did) + + """Blacklisting new steward by new trustee""" + op = {'type': '1', + 'dest': new_steward_did, + 'role': None} + if with_verkey: + op['verkey'] = new_steward_verkey + req = sdk_sign_and_submit_op(looper, sdk_pool_handle, (sdk_wallet_handle, new_trustee_did), op) + if with_verkey: + with pytest.raises(RequestRejectedException): + sdk_get_and_check_replies(looper, [req]) + else: + sdk_get_and_check_replies(looper, [req]) + + +def test_steward_cannot_create_trust_anchors_after_demote (looper, + sdk_pool_handle, + sdk_wallet_trustee, + sdk_wallet_handle): + new_steward_did, new_steward_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_trustee[0], "{}")) + new_ta_did, new_ta_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_trustee[0], "{}")) + new_ta_2_did, new_ta_2_verkey = looper.loop.run_until_complete( + did.create_and_store_my_did(sdk_wallet_trustee[0], "{}")) + + """Adding new steward""" + sdk_add_new_nym(looper, sdk_pool_handle, + sdk_wallet_trustee, + 'newSteward', + STEWARD_STRING, + verkey=new_steward_verkey, dest=new_steward_did) + + """Adding new TA""" + sdk_add_new_nym(looper, sdk_pool_handle, + (sdk_wallet_handle, new_steward_did), + 'newSteward', + TRUST_ANCHOR_STRING, + verkey=new_ta_verkey, dest=new_ta_did) + + """Blacklisting new steward by trustee""" + op = {'type': '1', + 'dest': new_steward_did, + 'role': None} + req = sdk_sign_and_submit_op(looper, sdk_pool_handle, sdk_wallet_trustee, op) + sdk_get_and_check_replies(looper, [req]) + + """Try to add new TA by previous demoted steward""" + with pytest.raises(RequestRejectedException): + sdk_add_new_nym(looper, sdk_pool_handle, + (sdk_wallet_handle, new_steward_did), + 'newSteward', + TRUST_ANCHOR_STRING, + verkey=new_ta_2_verkey, dest=new_ta_2_did) + diff --git a/indy_node/test/pool_config/test_pool_config.py b/indy_node/test/pool_config/test_pool_config.py index ecd66b285..898a2c46c 100644 --- a/indy_node/test/pool_config/test_pool_config.py +++ b/indy_node/test/pool_config/test_pool_config.py @@ -13,7 +13,7 @@ from plenum.common.constants import VERSION from plenum.test.pool_transactions.helper import sdk_add_new_nym -from indy_node.test.upgrade.conftest import validUpgrade, nodeIds +from indy_node.test.upgrade.conftest import validUpgrade, nodeIds, pckg def sdk_pool_bad_config_sent(looper, sdk_pool_handle, sdk_wallet_trustee, change_writes, diff --git a/indy_node/test/request_handlers/test_pool_upgrade_handler.py b/indy_node/test/request_handlers/test_pool_upgrade_handler.py new file mode 100644 index 000000000..c369f02f4 --- /dev/null +++ b/indy_node/test/request_handlers/test_pool_upgrade_handler.py @@ -0,0 +1,112 @@ +import pytest +from indy_common.constants import POOL_UPGRADE, ACTION, START, PACKAGE, APP_NAME, REINSTALL +from indy_node.server.request_handlers.config_req_handlers.pool_upgrade_handler import PoolUpgradeHandler +from plenum.common.constants import VERSION, TXN_PAYLOAD, TXN_PAYLOAD_DATA +from plenum.common.exceptions import InvalidClientRequest + +from plenum.common.request import Request +from plenum.common.util import randomString +from plenum.test.testing_utils import FakeSomething + + +@pytest.fixture(scope='function') +def pool_upgrade_request(): + return Request(identifier=randomString(), + reqId=5, + operation={ + 'type': POOL_UPGRADE, + ACTION: START, + PACKAGE: 'smth' + }) + + +@pytest.fixture(scope='function') +def pool_upgrade_handler(): + return PoolUpgradeHandler(None, + FakeSomething(), + FakeSomething(), + FakeSomething()) + + +def test_pool_upgrade_static_validation_fails_action(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_request.operation[ACTION] = 'smth' + with pytest.raises(InvalidClientRequest) as e: + pool_upgrade_handler.static_validation(pool_upgrade_request) + e.match('not a valid action') + + +def test_pool_upgrade_static_validation_fails_schedule(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_handler.pool_manager.getNodesServices = lambda: 1 + pool_upgrade_handler.upgrader.isScheduleValid = lambda schedule, node_srvs, force: (False, '') + with pytest.raises(InvalidClientRequest) as e: + pool_upgrade_handler.static_validation(pool_upgrade_request) + e.match('not a valid schedule since') + + +def test_pool_upgrade_static_validation_passes(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_handler.pool_manager.getNodesServices = lambda: 1 + pool_upgrade_handler.upgrader.isScheduleValid = lambda schedule, node_srvs, force: (True, '') + pool_upgrade_handler.static_validation(pool_upgrade_request) + + +def test_pool_upgrade_dynamic_validation_fails_pckg(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_request.operation[PACKAGE] = '' + with pytest.raises(InvalidClientRequest) as e: + pool_upgrade_handler.dynamic_validation(pool_upgrade_request) + e.match('Upgrade packet name is empty') + + +def test_pool_upgrade_dynamic_validation_fails_not_installed(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_handler.curr_pkt_info = lambda pkt: (None, None) + with pytest.raises(InvalidClientRequest) as e: + pool_upgrade_handler.dynamic_validation(pool_upgrade_request) + e.match('is not installed and cannot be upgraded') + + +def test_pool_upgrade_dynamic_validation_fails_belong(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_handler.curr_pkt_info = lambda pkt: ('1.1.1', ['some_pckg']) + with pytest.raises(InvalidClientRequest) as e: + pool_upgrade_handler.dynamic_validation(pool_upgrade_request) + e.match('doesn\'t belong to pool') + + +def test_pool_upgrade_dynamic_validation_fails_upgradable(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_handler.curr_pkt_info = lambda pkt: ('1.1.1', [APP_NAME]) + pool_upgrade_request.operation[VERSION] = '1.1.1' + pool_upgrade_request.operation[REINSTALL] = False + with pytest.raises(InvalidClientRequest) as e: + pool_upgrade_handler.dynamic_validation(pool_upgrade_request) + e.match('Version is not upgradable') + + +def test_pool_upgrade_dynamic_validation_fails_scheduled(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_handler.curr_pkt_info = lambda pkt: ('1.1.1', [APP_NAME]) + pool_upgrade_request.operation[VERSION] = '1.1.1' + pool_upgrade_request.operation[REINSTALL] = True + pool_upgrade_handler.upgrader.get_upgrade_txn = \ + lambda predicate, reverse: \ + {TXN_PAYLOAD: {TXN_PAYLOAD_DATA: {ACTION: START}}} + + with pytest.raises(InvalidClientRequest) as e: + pool_upgrade_handler.dynamic_validation(pool_upgrade_request) + e.match('is already scheduled') + + +def test_pool_upgrade_dynamic_validation_passes(pool_upgrade_handler, + pool_upgrade_request): + pool_upgrade_handler.curr_pkt_info = lambda pkt: ('1.1.1', [APP_NAME]) + pool_upgrade_request.operation[VERSION] = '1.1.1' + pool_upgrade_request.operation[REINSTALL] = True + pool_upgrade_handler.upgrader.get_upgrade_txn = \ + lambda predicate, reverse: \ + {TXN_PAYLOAD: {TXN_PAYLOAD_DATA: {}}} + pool_upgrade_handler.write_request_validator.validate = lambda a, b: 0 + pool_upgrade_handler.dynamic_validation(pool_upgrade_request) diff --git a/indy_node/test/schema/test_send_schema.py b/indy_node/test/schema/test_send_schema.py index caf65c81e..bb50f4c22 100644 --- a/indy_node/test/schema/test_send_schema.py +++ b/indy_node/test/schema/test_send_schema.py @@ -1,7 +1,10 @@ import pytest +from indy_common.config import SCHEMA_ATTRIBUTES_LIMIT from indy_node.test.api.helper import validate_write_reply, sdk_write_schema_and_check -from plenum.common.exceptions import RequestRejectedException +from plenum.common.exceptions import RequestRejectedException, RequestNackedException +from plenum.common.util import randomString +from plenum.config import NAME_FIELD_LIMIT def test_send_schema_multiple_attrib(looper, sdk_pool_handle, @@ -48,3 +51,37 @@ def test_can_not_send_same_schema(looper, sdk_pool_handle, ex_info.match( "can have one and only one SCHEMA with name business and version 1.8" ) + + +def test_schema_maximum_attrib(looper, sdk_pool_handle, + sdk_wallet_trust_anchor): + attribs = [] + for i in range(SCHEMA_ATTRIBUTES_LIMIT): + attribs.append(randomString(NAME_FIELD_LIMIT)) + + sdk_write_schema_and_check( + looper, sdk_pool_handle, + sdk_wallet_trust_anchor, + attribs, + "business1", + "1.9" + ) + + +def test_schema_over_maximum_attrib(looper, sdk_pool_handle, + sdk_wallet_trust_anchor): + attribs = [] + for i in range(SCHEMA_ATTRIBUTES_LIMIT + 1): + attribs.append('attrib' + str(i)) + + with pytest.raises(RequestNackedException) as ex_info: + sdk_write_schema_and_check( + looper, sdk_pool_handle, + sdk_wallet_trust_anchor, + attribs, + "business2", + "2.0" + ) + ex_info.match( + "length should be at most {}".format(SCHEMA_ATTRIBUTES_LIMIT) + ) diff --git a/indy_node/test/tools/test_nsreplay.py b/indy_node/test/tools/test_nsreplay.py index 514cecdf6..98fa37e16 100644 --- a/indy_node/test/tools/test_nsreplay.py +++ b/indy_node/test/tools/test_nsreplay.py @@ -78,6 +78,11 @@ def tconf(tconf): return tconf +@pytest.yield_fixture(scope="session", autouse=True) +def warncheck(warnfilters): + pass + + def test_end_to_end_replay(looper, tdir, tdirWithPoolTxns, diff --git a/indy_node/test/txn_validation/test_pool_upgrade_validation.py b/indy_node/test/txn_validation/test_pool_upgrade_validation.py index a4e091377..f6a482517 100644 --- a/indy_node/test/txn_validation/test_pool_upgrade_validation.py +++ b/indy_node/test/txn_validation/test_pool_upgrade_validation.py @@ -7,7 +7,7 @@ from indy_node.test.upgrade.helper import loweredVersion, sdk_ensure_upgrade_sent from indy_common.constants import JUSTIFICATION, JUSTIFICATION_MAX_SIZE -from indy_node.test.upgrade.conftest import validUpgrade, nodeIds +from indy_node.test.upgrade.conftest import validUpgrade, nodeIds, pckg def testPoolUpgradeFailsIfVersionIsLowerThanCurrent( diff --git a/indy_node/test/upgrade/conftest.py b/indy_node/test/upgrade/conftest.py index a0f9c903d..668c57007 100644 --- a/indy_node/test/upgrade/conftest.py +++ b/indy_node/test/upgrade/conftest.py @@ -61,7 +61,12 @@ def mock_get_info_from_package_manager(package): @pytest.fixture(scope='function', params=[(EXT_PKT_NAME, EXT_PKT_VERSION), (APP_NAME, None)]) -def validUpgrade(nodeIds, tconf, monkeypatch, request): +def pckg(request): + return request.param + + +@pytest.fixture(scope='function') +def validUpgrade(nodeIds, tconf, monkeypatch, pckg): schedule = {} unow = datetime.utcnow().replace(tzinfo=dateutil.tz.tzutc()) startAt = unow + timedelta(seconds=100) @@ -70,10 +75,10 @@ def validUpgrade(nodeIds, tconf, monkeypatch, request): schedule[i] = datetime.isoformat(startAt) startAt = startAt + timedelta(seconds=acceptableDiff + 3) - patch_packet_mgr_output(monkeypatch, request.param[0], request.param[1]) + patch_packet_mgr_output(monkeypatch, pckg[0], pckg[1]) - return dict(name='upgrade-{}'.format(randomText(3)), version=bumpedVersion(request.param[1]), - action=START, schedule=schedule, timeout=1, package=request.param[0], + return dict(name='upgrade-{}'.format(randomText(3)), version=bumpedVersion(pckg[1]), + action=START, schedule=schedule, timeout=1, package=pckg[0], sha256='db34a72a90d026dae49c3b3f0436c8d3963476c77468ad955845a1ccf7b03f55') diff --git a/indy_node/test/upgrade/test_broken_connection_control_tool.py b/indy_node/test/upgrade/test_broken_connection_control_tool.py new file mode 100644 index 000000000..e33bceff9 --- /dev/null +++ b/indy_node/test/upgrade/test_broken_connection_control_tool.py @@ -0,0 +1,66 @@ +import dateutil +import dateutil.tz +import pytest + +from datetime import datetime, timedelta +from copy import deepcopy +from indy_common.constants import START +from indy_node.test.upgrade.conftest import patch_packet_mgr_output, EXT_PKT_NAME, EXT_PKT_VERSION + +from indy_node.server.upgrader import Upgrader +from indy_node.test.upgrade.helper import sdk_ensure_upgrade_sent, bumpedVersion +from plenum.test.helper import randomText + +from stp_core.common.log import getlogger + +delta = 2 +logger = getlogger() + + +@pytest.fixture(scope='function') +def pckg(): + return (EXT_PKT_NAME, EXT_PKT_VERSION) + + +@pytest.fixture(scope="module") +def tconf(tconf): + old_delta = tconf.MinSepBetweenNodeUpgrades + tconf.MinSepBetweenNodeUpgrades = delta + yield tconf + tconf.MinSepBetweenNodeUpgrades = old_delta + + +@pytest.fixture(scope='function') +def skip_functions(): + # Do this to prevent exceptions because of node_control_tool absence + old_action_failed = deepcopy(Upgrader._action_failed) + + Upgrader._action_failed = \ + lambda self, version, scheduled_on, upgrade_id, external_reason: 1 + yield + Upgrader._action_failed = old_action_failed + + +def test_node_doesnt_retry_upgrade(looper, nodeSet, validUpgrade, nodeIds, + sdk_pool_handle, sdk_wallet_trustee, tconf, + skip_functions): + schedule = {} + unow = datetime.utcnow().replace(tzinfo=dateutil.tz.tzutc()) + startAt = unow + timedelta(seconds=delta) + for i in nodeIds: + schedule[i] = datetime.isoformat(startAt) + startAt = startAt + timedelta(seconds=delta) + validUpgrade['schedule'] = schedule + + # Emulating connection problems + for node in nodeSet: + node.upgrader.retry_limit = 0 + + # Send upgrade + sdk_ensure_upgrade_sent(looper, sdk_pool_handle, + sdk_wallet_trustee, validUpgrade) + looper.runFor(len(nodeIds) * delta) + + # Every node, including bad_node, tried to upgrade only once + for node in nodeSet: + assert node.upgrader.spylog.count(Upgrader.processLedger.__name__) == 1 diff --git a/indy_node/test/upgrade/test_node_control_tool_performs_migrations.py b/indy_node/test/upgrade/test_node_control_tool_performs_migrations.py index 3c3cc2d27..95f690f98 100644 --- a/indy_node/test/upgrade/test_node_control_tool_performs_migrations.py +++ b/indy_node/test/upgrade/test_node_control_tool_performs_migrations.py @@ -1,14 +1,16 @@ import multiprocessing import os -from stp_core.loop.eventually import eventually -from indy_node.test.upgrade.helper import NodeControlToolExecutor as NCT, sendUpgradeMessage, nodeControlGeneralMonkeypatching - +from indy_node.utils.migration_tool import _get_current_platform +from stp_core.loop.eventually import eventually +from indy_node.test.upgrade.helper import NodeControlToolExecutor as NCT, sendUpgradeMessage, \ + nodeControlGeneralMonkeypatching m = multiprocessing.Manager() whitelist = ['Unexpected error in _upgrade test'] + def testNodeControlPerformsMigrations(monkeypatch, tdir, looper, tconf): msg = 'test' stdout = 'teststdout' @@ -32,4 +34,8 @@ def checkMigration(): sendUpgradeMessage(msg) looper.run(eventually(checkMigration)) finally: - nct.stop() \ No newline at end of file + nct.stop() + + +def test_get_current_platform(): + _get_current_platform() diff --git a/indy_node/test/upgrade/test_pool_upgrade_same_time_different_days.py b/indy_node/test/upgrade/test_pool_upgrade_same_time_different_days.py new file mode 100644 index 000000000..955779edf --- /dev/null +++ b/indy_node/test/upgrade/test_pool_upgrade_same_time_different_days.py @@ -0,0 +1,24 @@ +from copy import deepcopy + +from datetime import datetime, timedelta + +import dateutil +import dateutil.tz + +from indy_node.test.upgrade.helper import sdk_ensure_upgrade_sent + + +def test_pool_upgrade_same_time_different_days(looper, tconf, nodeSet, + validUpgrade, sdk_pool_handle, + sdk_wallet_trustee, nodeIds): + day_in_sec = 24 * 60 * 60 + upgr1 = deepcopy(validUpgrade) + schedule = {} + unow = datetime.utcnow().replace(tzinfo=dateutil.tz.tzutc()) + startAt = unow + timedelta(seconds=day_in_sec) + for i in nodeIds: + schedule[i] = datetime.isoformat(startAt) + startAt = startAt + timedelta(seconds=day_in_sec) + upgr1['schedule'] = schedule + + sdk_ensure_upgrade_sent(looper, sdk_pool_handle, sdk_wallet_trustee, upgr1) diff --git a/indy_node/test/validator_info/test_validator_info.py b/indy_node/test/validator_info/test_validator_info.py index fcf128715..51db58859 100644 --- a/indy_node/test/validator_info/test_validator_info.py +++ b/indy_node/test/validator_info/test_validator_info.py @@ -189,9 +189,9 @@ def test_validator_info_file_metrics_count_all_ledgers_field_valid(node): def test_validator_info_update_date_field_valid(info): - assert "Update time" in info + assert "Update_time" in info import time import datetime - from_str = time.mktime(datetime.datetime.strptime(info["Update time"], + from_str = time.mktime(datetime.datetime.strptime(info["Update_time"], "%A, %B %d, %Y %I:%M:%S %p %z").timetuple()) assert int(from_str) == info["timestamp"] diff --git a/indy_node/utils/migration_tool.py b/indy_node/utils/migration_tool.py index 4ca5a0ff7..2128d1534 100644 --- a/indy_node/utils/migration_tool.py +++ b/indy_node/utils/migration_tool.py @@ -3,7 +3,7 @@ import importlib import os import pkgutil -import platform +import distro import subprocess import time from pathlib import Path @@ -128,8 +128,7 @@ def _compare_migration_scripts(migration1, migration2): def _get_current_platform(): - uname = platform.uname() - version = uname.version - if 'Ubuntu' in version: + name = distro.linux_distribution()[0] + if 'Ubuntu' in name or 'ubuntu' in name: return 'Ubuntu' raise Exception('Platform is not supported') diff --git a/scripts/validator-info b/scripts/validator-info index 7eade6dd4..07843f464 100755 --- a/scripts/validator-info +++ b/scripts/validator-info @@ -269,9 +269,18 @@ class TransactionsStats(BaseUnknown): else: return "" + def items(self): + return dict(self.val).items() + def __iter__(self): return iter(self.val) + def __getitem__(self, key): + return self.val[key] + + def __setitem__(self, key, value): + self.val[key] = value + class AverageStats(BaseStats): shema = [ @@ -550,21 +559,22 @@ def nagios(vstats): if "running" == running: state = '0' - lines = [ - "{} {}_Total_Config_Transactions config_transactions={} {} Total Config Transactions".format( - state,vstats['Node_info']['Name'],vstats['Node_info']['Metrics']['transaction-count']['config'],vstats['Node_info']['Name']) - ] + [ - "{} {}_Total_Ledger_Transactions ledger_transactions={} {} Total Ledger Transactions".format( - state,vstats['Node_info']['Name'],vstats['Node_info']['Metrics']['transaction-count']['ledger'],vstats['Node_info']['Name']) - ] + [ - "{} {}_Total_Pool_Transactions pool_transactions={} {} Total Pool Transactions".format( - state,vstats['Node_info']['Name'],vstats['Node_info']['Metrics']['transaction-count']['pool'],vstats['Node_info']['Name']) - ] + [ - "{} {}_Read_Transactions_per_second read_transactions_per_second={} {} Read Transactions/Second".format( - state,vstats['Node_info']['Name'],vstats['Node_info']['Metrics']['average-per-second']['read-transactions'],vstats['Node_info']['Name']) + lines = ["{} {}_Total_{}_Transactions config_transactions={} {} Total {} Transactions".format( + state, + vstats['Node_info']['Name'], + ledger_name.title(), + count, + vstats['Node_info']['Name'], + ledger_name.title()) + for ledger_name, count in vstats['Node_info']['Metrics']['transaction-count'].items()] + + lines += ["{} {}_Read_Transactions_per_second read_transactions_per_second={} {} Read Transactions/Second".format( + state,vstats['Node_info']['Name'],vstats['Node_info']['Metrics']['average-per-second']['read-transactions'], + vstats['Node_info']['Name']) ] + [ "{} {}_Write_Transactions_per_second write_transactions_per_second={} {} Write Transactions/Second".format( - state,vstats['Node_info']['Name'],vstats['Node_info']['Metrics']['average-per-second']['write-transactions'],vstats['Node_info']['Name']) + state,vstats['Node_info']['Name'],vstats['Node_info']['Metrics']['average-per-second']['write-transactions'], + vstats['Node_info']['Name']) ] + [ "{} {}_Number_of_Validators number_of_validators={} {} Number of Validators".format( state,vstats['Node_info']['Name'],vstats['Pool_info']['Total_nodes_count'],vstats['Node_info']['Name']) diff --git a/setup.py b/setup.py index 2f9a1cf80..88b35881d 100644 --- a/setup.py +++ b/setup.py @@ -36,7 +36,7 @@ LOG_DIR = os.path.join(BASE_DIR, "log") CONFIG_FILE = os.path.join(BASE_DIR, "indy_config.py") -tests_require = ['pytest==3.3.1', 'pytest-xdist==1.22.1', 'python3-indy==1.6.1.dev683'] +tests_require = ['pytest==3.3.1', 'pytest-xdist==1.22.1', 'python3-indy==1.6.8', 'pytest-asyncio==0.8.0'] setup( name='indy-node-dev', @@ -56,9 +56,10 @@ data_files=[( (BASE_DIR, ['data/nssm_original.exe']) )], - install_requires=['indy-plenum-dev==1.6.646', + install_requires=['indy-plenum-dev==1.6.665', 'python-dateutil', - 'timeout-decorator==0.4.0'], + 'timeout-decorator==0.4.0', + 'distro==1.3.0'], setup_requires=['pytest-runner'], extras_require={ 'tests': tests_require