Merge master into PytestBenchmarkTesting

CHANGELOG.md

@@ -5,14 +5,40 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog][keepachangelog],
 and this project adheres to [Semantic Versioning][semver].
 
+
 ## [Unreleased]
 
 ### Added
 
+
+- Implement removal and rotation of keys [(561)]
+
 ### Changed
 
+Transition to the newest version of TUF [(561)]
+
 ### Fixed
 
+
+[561]: https://github.com/openlawlibrary/taf/pull/561
+
+
+## [0.33.1]
+
+### Added
+
+### Changed
+
+### Fixed
+
+- Run validation with --no-deps when pushing ([579])
+- Do not update last validated commit if pushing to a branch other than the default branch ([577])
+- Fix determining from which commit the update should start if the auth repo is in front of all target repos ([577])
+
+[579]: https://github.com/openlawlibrary/taf/pull/579
+[577]: https://github.com/openlawlibrary/taf/pull/577
+
+
 ## [0.33.0]
 
 ### Added
@@ -1385,7 +1411,8 @@ and this project adheres to [Semantic Versioning][semver].
 
 [keepachangelog]: https://keepachangelog.com/en/1.0.0/
 [semver]: https://semver.org/spec/v2.0.0.html
-[unreleased]: https://github.com/openlawlibrary/taf/compare/v0.33.0...HEAD
+[unreleased]: https://github.com/openlawlibrary/taf/compare/v0.33.1...HEAD
+[0.33.1]: https://github.com/openlawlibrary/taf/compare/v0.33.0...v0.33.1
 [0.33.0]: https://github.com/openlawlibrary/taf/compare/v0.32.4...v0.33.0
 [0.32.4]: https://github.com/openlawlibrary/taf/compare/v0.32.3...v0.32.4
 [0.32.3]: https://github.com/openlawlibrary/taf/compare/v0.32.2...v0.32.3

docs/architecture.md

@@ -160,13 +160,11 @@ To validate commits that could be decades old without being obstructed by expire
 
 This module encapsulates the `GitRepository` class, a high-level abstraction over Git operations, designed to interface directly with Git repositories at the filesystem level. The `GitRepository` class serves as an intermediary, enabling programmatic access to Git actions including: creating branches, working with commits, and working with remotes. It leverages [`pygit2`](https://www.pygit2.org/) for some of the interactions with Git. Other interactions use direct shell command execution via subprocess for operations not covered by `pygit2` or where direct command invocation is preferred for efficiency or functionality reasons.
 
-### `taf/repository_tool.py`
+### `taf/tuf/repository`
 
-Contains a `Repository` class, which is a wrapper around TUF's repository, making it simple to execute important updates, like
-adding new signing keys, updating and signing metadata files and extracting information about roles, keys,
-delegations and targets.
-
-NOTE: Long-term plan is to rework this part of the codebase. This is necessary to transition to the newest version of TUF, since it is relying on parts which no longer exist in newer TUF.
+Contains a `MetadataRepository` class, which is an implementation of TUF's `Repository` class for editing metadata.
+It simplifies the execution of important updates such as adding new signing keys, updating and signing metadata
+files, and extracting information about roles, keys, delegations, and targets.
 
 ### `taf/auth_repo.py`
 

docs/developers/repository-classes.md

@@ -1,18 +1,18 @@
 # Repositories
 
-As a tool focused on creation and secure update of Git repositories (authentication repositories and their
-targets), TAF contains classes and functions which strive to make integration with Git and TUF as easy as possible.
-`GitRepository` can be seen as a wrapper around git calls which make it possible to interact with the actual `Git`
-repository located on the file system. E.g. to create a new branch, list commits, push to the remote etc.
-On the other hand, `Repository` class contained by the `repository_tool` module can instantiate a TUF repository,
-provided that the directory passed to it contains metadata files expected to be found in such a repository. It also
-implements important TUF concepts, such as adding a new delegated role, determine which role is responsible for which
-target file, add TUF targets etc. An authentication repository can be seen as a Git repository which is also a TUF repository - it
-contains TUF's metadata and target files and a `.git` folder. TAF's `auth_repo` module's `AuthenticationRepository`
-class follows that logic and is derived from the two previously mentioned base classes. Finally, `repositoriesdb`
-is a module inspired by TUF's modules like `keysdb`, which deals with instantiation of repositories and stores the
-created classes inside a "database" - a dictionary which maps authentication repositories and their commits
-to lists of their target repositories at certain revisions.
+As a tool focused on the creation and secure update of Git repositories (authentication repositories and their
+targets), TAF contains classes and functions that strive to make integration with Git and TUF as simple as possible.
+`GitRepository` acts as a wrapper around Git calls, enabling interaction with the actual `Git` repository on the file
+system, e.g., creating a new branch, listing, creating, and pushing commits, etc. Conversely, the `MetadataRepository`
+class in `tuf/repository.py` extends TUF's `Repository` class, an abstract class for metadata modifying implementations.
+It provides implementations of crucial TUF concepts, such as adding a new delegated role, determining which role is
+responsible for which target file, and adding TUF targets etc. An authentication repository can be seen as a Git
+repository that is also a TUF repository. It contains TUF's metadata and target files and a `.git` folder. TAF's
+`auth_repo` module's `AuthenticationRepository` class follows that logic and is derived from the two previously
+mentioned base classes. Finally, `repositoriesdb` is a module inspired by TUF's modules like `keysdb`, which deals with
+the instantiation of repositories and stores the created classes inside a "database" - a dictionary which maps
+authentication repositories and their commits to lists of their target repositories at certain revisions. Note: the
+concept of databases has been removed from TUF and removal of `repositoriesdb` is also planned in case of TAF.
 
 ## GitRepository
 
@@ -66,44 +66,40 @@ repo.commit_empty('An example message')
 repo.push()
 ```
 
-## Repository tool's `Repository`
+## Implementation of TUF's `Repository` class (`tuf/repository/MetadataRepository`)
+
+This class extends TUF's repository interface, providing features for executing metadata updates, such as
+adding new signing keys, updating and signing metadata files, and extracting information about roles,
+keys, delegations, and targets. It can be used to create a new TUF repository, retrieve information about
+a TUF repository, or update its metadata files. TAF's implementation of the repository class follows the
+convention of separating metadata and target files into directories named `metadata` and `target`:
 
-This class can be seen as a wrapper around a TUF repository, making it simple to execute important updates, like
-adding new signing keys, updating and signing metadata files and extracting information about roles, keys,
-delegations and targets. It is instantiated by passing file system path which corresponds to a directory containing
-all files and folders that a TUF repository expects. That means that `metadata` and `targets` folders have to exist
-and that a valid `root.json` file needs to be found inside `metadata`. So:
 ```
 - repo_root
   - metadata
     - root.json
   - targets
 ```
-Optionally, `name` attribute can also be specified during instantiation. It will be used to set name of the TUF's
-repository instance. This value is set to `default` if not provided. If more than one repository is to be used
-at the same time, it is important to set distinct names.
-
-TUF repository is instantiated lazily the first time it is needed. This object is not meant to be used directly.
-The main purpose of TAF's repository class is to group operations which enable valid update of TUF metadata and acquiring
-information like can a key be used to sign a certain metadata file or finding roles that are linked with
-the provided public key. To set up a new repository or add a new signing key, it is recommended to use the
-`developer_tool` module since it contains full implementations of these complex functionalities. Functionalities
-like updating targets and signing metadata or updating a metadata's expiration date are fully covered by repository
-class's methods and can be used directly. These include:
-- `update_timestamp_keystores`, `update_snapshot_keystores` (`update_rolename_keystores`) and `update_role_keystores` (for delegated roles)
--`update_timestamp_yubikeys`, `update_snapshot_yubikeys` (`update_rolename_yubikeys`) and `update_role_yubikeys` (for delegated roles)
-
-If `added_targets_data` or `removed_targets_data` is passed in when calling these methods (only applicable to
-`targets` and delegated target roles), information about target files will be updated and the corresponding metadata
-file will be signed. Its expiration date will be updated too. If there is targets data or if the called method
-corresponds to a non-targets role, the metadata file's expiration will still be updated and the file will be signed.
+
+It is instantiated by providing the repository's path. Unlike the previous implementation, which was based on an
+older version of TUF, this repository does not have, nor does it need, a name. The class can be instantiated
+regardless of whether there are `metadata` files located at `path/metadata`. In fact, it is possible to read the
+metadata and target files from mediums other than the local file system. TUF enables such flexibility by allowing
+custom implementations of the `StorageBackendInterface`. These implementations can redefine how metadata and target
+files are read and written. To instantiate a `MetadataRepository` class with a custom storage interface, use the
+`storage` keyword argument. If not specified, TUF's default `FilesystemBackend` will be used. The other available
+option is `GitStorageBackend`. This implementation loads data from a specific commit if the commit is specified,
+or from the filesystem if the commit is `None`, by extending `FilesystemBackend`.
+
+This class is used extensively to implement API functions.
 
 
 ## `AuthenticationRepository`
 
-This class is derived from both `GitRepository` and TAF's `Repository`. Authentication repositories are expected
-to contain TUF metadata and target files, but are also Git repositories. It is important to note that only files
-inside the `targets` folder are tracked and secured by TUF.
+This class is derived from `GitRepository`, and indirectly from `MetadataRepository`. Authentication repositories are
+expected to contain TUF metadata and target files, but are also Git repositories. It is important to note that only
+files inside the `targets` folder are tracked and secured by TUF.
+
 
 Instances of the `AuthenticationRepository` are created by passing the same arguments as to `GitRepository` (`library_dir`, `name`, `urls`, `custom`, `default_branch`, `allow_unsafe` and `path` which can replace `library_dir` and `name` combination), as well as some optional additional arguments:
 - `conf_directory_root` - path to the directory where the `last_validated_commit` will be stored.

setup.py

@@ -1,7 +1,7 @@
 from setuptools import find_packages, setup
 
 PACKAGE_NAME = "taf"
-VERSION = "0.33.0"
+VERSION = "0.33.1"
 AUTHOR = "Open Law Library"
 AUTHOR_EMAIL = "[email protected]"
 DESCRIPTION = "Implementation of archival authentication"
@@ -31,6 +31,7 @@
     "freezegun==0.3.15",
     "jsonschema==3.2.0",
     "jinja2==3.1.*",
+    "pytest-mock==3.14.*",
 ]
 
 yubikey_require = ["yubikey-manager==5.5.*"]
@@ -54,13 +55,13 @@
         "cattrs>=23.1.2",
         "click==8.*",
         "colorama>=0.3.9",
-        "oll-tuf==0.20.0.dev2",
-        "cryptography==38.0.*",
-        "securesystemslib==0.25.*",
+        "tuf==5.*",
+        "cryptography==43.0.*",
+        "securesystemslib==1.*",
         "loguru==0.7.*",
         'pygit2==1.9.*; python_version < "3.11"',
         'pygit2==1.14.*; python_version >= "3.11"',
-        "pyOpenSSL==22.1.*",
+        "pyOpenSSL==24.2.*",
         "logdecorator==2.*",
     ],
     "extras_require": {

taf/api/api_workflow.py

@@ -0,0 +1,118 @@
+from contextlib import contextmanager
+from pathlib import Path
+from typing import Dict, List, Optional, Union
+
+from taf.api.utils._conf import find_keystore
+from taf.auth_repo import AuthenticationRepository
+from taf.constants import DEFAULT_RSA_SIGNATURE_SCHEME
+from taf.exceptions import PushFailedError, TAFError
+from taf.keys import load_signers
+from taf.log import taf_logger
+from taf.messages import git_commit_message
+from taf.constants import METADATA_DIRECTORY_NAME
+
+
+@contextmanager
+def transactional_execution(auth_repo):
+    initial_commit = auth_repo.head_commit_sha()
+    try:
+        yield
+    except PushFailedError:
+        pass
+    except Exception:
+        auth_repo.reset_to_commit(initial_commit, hard=True)
+        raise
+
+
+@contextmanager
+def manage_repo_and_signers(
+    auth_repo: AuthenticationRepository,
+    roles: Optional[List[str]] = None,
+    keystore: Optional[Union[str, Path]] = None,
+    scheme: Optional[str] = DEFAULT_RSA_SIGNATURE_SCHEME,
+    prompt_for_keys: Optional[bool] = False,
+    paths_to_reset_on_error: Optional[List[Union[str, Path]]] = None,
+    load_roles: Optional[bool] = True,
+    load_parents: Optional[bool] = False,
+    load_snapshot_and_timestamp: Optional[bool] = True,
+    commit: Optional[bool] = True,
+    push: Optional[bool] = True,
+    commit_key: Optional[str] = None,
+    commit_msg: Optional[str] = None,
+    no_commit_warning: Optional[bool] = True,
+):
+    """
+    A context manager that loads all signers and adds them to the specified authentication repository's
+    signers cache. This allows for the execution of other methods without having to update the
+    signers cache manually. Optionally, at the end, the context manager commits and pushes all changes made
+    to the authentication repository and handles cleanup in case of an error.
+
+    Arguments:
+        auth_repo (AuthenticationRepository): Already instantiated authentication repository.
+        roles (Optional[List[str]]): List of roles that are expected to be updated.
+        keystore (Optional[Union[str, Path]]): Path to the keystore containing signing keys.
+        scheme (Optional[str]): The signature scheme.
+        prompt_for_keys (Optional[bool]): If True, prompts for keys if not found. Defaults to False.
+        paths_to_reset_on_error (Optional[List[Union[str, Path]]]): Paths to reset if an error occurs.
+        load_roles (Optional[bool]): If True, loads signing keys of the roles specified using the argument of the same name.
+        load_parents (Optional[bool]): If true, loads sining keys of the specified roles' parents.
+        load_snapshot_and_timestamp (Optional[bool]): If True, loads snapshot and timestamp signing keys.
+        commit (Optional[bool]): If True, commits changes to the repository.
+        push (Optional[bool]): If True, pushes changes to the remote repository.
+        commit_key (Optional[str]): Commit key from `messages.py`
+        commit_msg (Optional[str]): The message to use for commits.
+        no_commit_warning (Optional[bool]): If True, suppresses warnings when not committing.
+    """
+    try:
+        roles_to_load = set()
+        if roles:
+            unique_roles = set(roles)
+            if load_roles:
+                roles_to_load.update(unique_roles)
+            if load_parents:
+                roles_to_load.update(auth_repo.find_parents_of_roles(unique_roles))
+        if load_snapshot_and_timestamp:
+            roles_to_load.add("snapshot")
+            roles_to_load.add("timestamp")
+        if roles_to_load:
+            if not keystore:
+                keystore_path = find_keystore(auth_repo.path)
+            else:
+                keystore_path = Path(keystore)
+            loaded_yubikeys: Dict = {}
+            for role in roles_to_load:
+                if not auth_repo.check_if_keys_loaded(role):
+                    keystore_signers, yubikey_signers = load_signers(
+                        auth_repo,
+                        role,
+                        loaded_yubikeys=loaded_yubikeys,
+                        keystore=keystore_path,
+                        scheme=scheme,
+                        prompt_for_keys=prompt_for_keys,
+                    )
+                    auth_repo.add_signers_to_cache({role: keystore_signers})
+                    auth_repo.add_signers_to_cache({role: yubikey_signers})
+        yield
+        if commit and auth_repo.something_to_commit():
+            if not commit_msg and commit_key:
+                commit_msg = git_commit_message(commit_key)
+            auth_repo.commit_and_push(commit_msg=commit_msg, push=push)
+        elif not no_commit_warning:
+            taf_logger.log("NOTICE", "\nPlease commit manually\n")
+
+    except PushFailedError:
+        raise
+    except Exception as e:
+        taf_logger.error(f"An error occurred: {e}")
+        if not paths_to_reset_on_error:
+            paths_to_reset_on_error = [METADATA_DIRECTORY_NAME]
+        elif METADATA_DIRECTORY_NAME not in paths_to_reset_on_error:
+            paths_to_reset_on_error.append(METADATA_DIRECTORY_NAME)
+
+        if auth_repo.is_git_repository and paths_to_reset_on_error:
+            # restore metadata, leave targets as they might have been modified by the user
+            # TODO flag for also resetting targets?
+            # also update the CLI error handling]
+            auth_repo.restore([str(path) for path in paths_to_reset_on_error])
+
+        raise TAFError from e