Transition to the newest version of TUF #561
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Remove unused pyopenssl Signed-off-by: Lukas Puehringer <[email protected]>
Implements basic primitives, defined by the python-tuf Repository abstraction, to read and edit metadata on disk, handling version and expiry bumps, and signature creation, and facilitating snapshot and timestamp creation. And adds exemplary API methods that use these primitives while preserving consistent repo states: - create - add_target_files - add_keys Can be tested with: ``` PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest --noconftest taf/tests/tuf/ ``` More detailed usage docs + migration path TBD... Signed-off-by: Lukas Puehringer <[email protected]>
The original design aimed at separating the concepts of delegation (adding public keys) and signing (using private keys). Since the MetadataRepository assumes that metadata can be signed rightaway after edit (e.g. after having added a delegation), which in turn requires private keys to be available, we might as well conflate these two concepts. The advantage is that the signer cache does not have to be managed independently and is more likely to stay in sync with the delegations. Signed-off-by: Lukas Puehringer <[email protected]>
Signed-off-by: Lukas Puehringer <[email protected]>
This should really happen upstream (see linked issue) Signed-off-by: Lukas Puehringer <[email protected]>
YkSigner provides a minimal compatibility layer over `taf.yubikey`
module functions for use with MetadataRepository.
Even though a yubikey signer implementation (HSMSigner) based on
pykcs11 is available in securesystemslib, YkSigner was added for the
following reasons:
- TAF requires rsa support for yubikeys, but HSMSigner only supports
ecdsa. Adding rsa support to HSMSigner, or providing a custom
pykcs11-based RSAHSMSigner is feasible, and seems desirable, but
requires more effort than this YkSigner did.
- TAF provides a few additional features, like setting up a Yubikey,
changing pins, etc., which will not be added to securesystemslib.
This means the current Yubikey infrastructure based on yubikey-manager
needs to be preserved for the time being. Thus it made sense to
re-use the existing implementation for YkSigner.
- YkSigner show-cases the new Signer API and might be used as blue print
for future Signer implementations in TAF.
This commit adds basic tests with fake and real Yubikey:
```
REAL_YK=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
pytest --noconftest taf/tests/tuf/ taf/tests/tuf/test_yk.py -s
```
Signed-off-by: Lukas Puehringer <[email protected]>
This allows running previously added YkSigner tests, but breaks other things, which need change anyway in the course of upgrading to latest tuf/securesystemslib. Signed-off-by: Lukas Puehringer <[email protected]>
Add alternative TUF metadata repo implementation (WIP)
…gated role lookup
…n dates to the new repository class
…ort for delegations
…ass and added tests
The idea being we cover all the key cli commands with tests. - `click` supports CLI tests [1]. To get started, we initialize a `CliRunner` and invoke the taf command that we want to test. Thankfully, testing is relatively straightforward. In cases where the CLI expects a user input, such as pressing ENTER or a "[y/N]" answer, `runner` supports an `input` param that gets passed into the subprocess stdin. Moreover, we can take the output of the cli test and assert the print/logging statements that occurred, which is really cool. This should make adding new cli tests relatively easy. - When asserting CLI output, such as logging statements (when the command began and when it finished executing), with the `caplog` built-in pytest fixture, things get funky since we use `loguru` instead of built-in python logging module. To resolve, we patch the `caplog` fixture in `conftest.py` to point to the `loguru` module. Added a docstring explaining it in more detail in `conftest.py` - Added ~14 cli tests that should cover all the important flows that we use. I managed to get most of them working, but a couple of them seem to be having slight issues with asserts and expected states. I've added comments to those to debug easier. - Since cli tests share a lot of the fixtures that `test_api.conftest` has, slightly re-organized the `test_api` module to avoid duplicating code/functions. The existing tests are now in their own subdirectory (e.g. `test_api\dependencies\api\` `test_api\roles\api`, etc.), while the newly added tests are in the sibling `cli` directory (e.g. `test_api\dependencies\cli\`...). The nice thing is is that this is complementary to the api functions, so when adding a new api test, we can easily add a cli test. [1] - https://click.palletsprojects.com/en/stable/testing/
n-dusan
reviewed
Dec 23, 2024
There was a problem hiding this comment.
I've just finished reviewing the testing framework and added 14 new CLI tests. 4 are currently failing. Could you take a look when you have a moment? I’m happy to help troubleshoot together if you’d like. I’ll push my commit to your branch so you can see the changes. Once those tests are passing, we can merge the PR. Also, your docstrings are fantastic. They really improve the code’s readability.
After merging, I’ll focus on updating the platform to the latest TAF.
taf/tests/test_api/test_targets.py
Outdated
Comment on lines
218
to
260
| # def test_remove_target_repository_when_not_on_filesystem( | ||
| # auth_repo_when_add_repositories_json: AuthenticationRepository, | ||
| # library: Path, | ||
| # keystore_delegations: str, | ||
| # ): | ||
| # repo_path = str(library / "auth") | ||
| # initial_commits_num = len(auth_repo_when_add_repositories_json.list_commits()) | ||
| # namespace = library.name | ||
| # target_repo_name = f"{namespace}/target4" | ||
| # repositories_json = repositoriesdb.load_repositories_json( | ||
| # auth_repo_when_add_repositories_json | ||
| # ) | ||
| # assert repositories_json is not None | ||
| # repositories = repositories_json["repositories"] | ||
| # assert target_repo_name in repositories | ||
| # remove_target_repo( | ||
| # str(repo_path), | ||
| # target_repo_name, | ||
| # keystore_delegations, | ||
| # push=False, | ||
| # ) | ||
| # # verify repositories.json was updated and that changes were committed | ||
| # # then validate the repository | ||
| # # target repo should not be in the newest repositories.json | ||
| # repositories_json = repositoriesdb.load_repositories_json( | ||
| # auth_repo_when_add_repositories_json | ||
| # ) | ||
| # assert repositories_json is not None | ||
| # repositories = repositories_json["repositories"] | ||
| # assert target_repo_name not in repositories | ||
| # commits = auth_repo_when_add_repositories_json.list_commits() | ||
| # # this function is expected to commit twice | ||
| # assert len(commits) == initial_commits_num + 2 | ||
| # assert commits[1].message.strip() == git_commit_message( | ||
| # "remove-target", target_name=target_repo_name | ||
| # ) | ||
| # assert commits[0].message.strip() == git_commit_message( | ||
| # "remove-from-delegated-paths", target_name=target_repo_name | ||
| # ) | ||
| # delegated_paths = auth_repo_when_add_repositories_json.get_paths_of_role( | ||
| # "delegated_role" | ||
| # ) | ||
| # assert target_repo_name not in delegated_paths |
There was a problem hiding this comment.
question: if taf targets remove-repo isn't supported, should we create an issue?
taf/tests/test_api/test_roles.py
Outdated
Comment on lines
157
to
158
| # TODO enable when remove role is reimplemented | ||
| # def test_remove_role_when_no_targets(auth_repo_with_delegations: AuthenticationRepository, roles_keystore: str): |
There was a problem hiding this comment.
question: if remove roles isn't reimplemented, should we create an issue?
There was a problem hiding this comment.
Sure, I'll create one
taf/api/roles.py
Outdated
| @@ -779,7 +775,7 @@ def remove_role( | |||
| if parent_role is None: | |||
| taf_logger.error("Role is not among delegated roles") | |||
| return | |||
| parent_role_obj = _role_obj(parent_role, auth_repo) | |||
| parent_role_obj = auth_repo._role_obj(parent_role, auth_repo) | |||
There was a problem hiding this comment.
Makes sense. Do we create an issue?
taf/git.py
Outdated
Comment on lines
1469
to
1472
| def restore( | ||
| self, subdirectories: str | ||
| ) -> None: | ||
| self._git(f"restore {' '.join(subdirectories)}") |
|
By the way, should we consider doing a minor release of the current TAF changes off master before merging this PR? It's a big rewrite, so having a stable release in place might be a good idea |
renatav
force-pushed
the
feature/tuf-repositoty
branch
from
173ccbf to
005b293
Compare
|
@n-dusan Updated the tests and merge master. Ready for another review |
renatav
force-pushed
the
feature/tuf-repositoty
branch
from
54ca094 to
f313992
Compare
renatav
force-pushed
the
feature/tuf-repositoty
branch
from
f313992 to
503653b
Compare
It was a convenience method used to figure out which paths from repositories.json match which role. Early exit function if repo is bare (to signal that it's currently not supported)
n-dusan
force-pushed
the
feature/tuf-repositoty
branch
from
bc1806f to
408f9bd
Compare
This is because file hashes get calculated differently on Windows vs Linux if line endings are first saved as LF then get converted to CRLF on Windows. To resolve, normalize all line endings to LF.
…get files, even if no targets were modified
n-dusan
approved these changes
Jan 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description (e.g. "Related to ...", etc.)
Closes #274
Also implemented/addressed:
Closes #501
Closes #555
Closes #560
Code review checklist (for code reviewer to complete)