Merge branch 'feature/tuf-repositoty' into tuf-upgrade

openlawlibrary · Oct 29, 2024 · ec294a8 · ec294a8
2 parents 25371d7 + 4e157e3
commit ec294a8
Show file tree

Hide file tree

Showing 54 changed files with 2,973 additions and 1,352 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -159,25 +159,26 @@ jobs:
       - name: Install dependencies
         run: |
           pip install .[yubikey]
+          pip install .[executable]
           pip install pyinstaller
 
       - name: Build and test standalone executable (Linux)
         if: matrix.os == 'ubuntu-latest'
         run: |
-          pyinstaller --onefile --hidden-import=yubikey_manager --name taf-linux -y taf/tools/cli/taf.py
+          pyinstaller --onefile --hidden-import=yubikey_manager --hidden-import=lxml --collect-submodules taf.tools --name taf-linux -y taf/tools/cli/taf.py
           chmod +x dist/taf-linux
           ./dist/taf-linux --help | grep "TAF Command Line Interface" || { echo "Error: Expected text not found in the executable output"; exit 1; }
 
       - name: Build and test standalone executable (Windows)
         if: matrix.os == 'windows-latest'
         run: |
-          pyinstaller --onefile --hidden-import=yubikey_manager  --name taf-windows.exe -y taf/tools/cli/taf.py
+          pyinstaller --onefile --hidden-import=yubikey_manager --hidden-import=lxml --collect-submodules taf.tools --name taf-windows.exe -y taf/tools/cli/taf.py
           ./dist/taf-windows.exe --help | Select-String "TAF Command Line Interface" -quiet
 
       - name: Build and test standalone executable (macOS)
         if: matrix.os == 'macos-latest'
         run: |
-          pyinstaller --onefile --hidden-import=yubikey_manager  --name taf-macos -y taf/tools/cli/taf.py
+          pyinstaller --onefile --hidden-import=yubikey_manager --hidden-import=lxml --collect-submodules taf.tools --name taf-macos -y taf/tools/cli/taf.py
           ./dist/taf-macos --help | grep "TAF Command Line Interface" || { echo "Error: Expected text not found in the executable output"; exit 1; }
 
       - name: Upload standalone executable (Linux)

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,73 @@ and this project adheres to [Semantic Versioning][semver].
 
 ### Added
 
+### Changed
+
+### Fixed
+
+## [0.32.0] - 10/23/2024
+
+### Added
+
+
+### Changed
+
+
+### Fixed
+
+- Fix specification of pygit2 version depending on the Python version [(558)]
+- Fix validation and listing targets of an auth repo that does not contain `mirrors.json` [(558)]
+
+[558]: https://github.com/openlawlibrary/taf/pull/558
+
+
+## [0.31.2] - 10/16/2024
+
+### Added
+
+- Added a function for exporting `keys-description.json` ([550])
+- Added support for cloning a new dependency when adding it to `dependencies.json` if it is not on disk ([550])
+- Clean up authentication repository if an error occurs while running a cli command ([550])
+
+### Changed
+
+- Return a non-zero exit code with `sys.exit` when updater fails ([550])
+- Rework addition of a new role and target repositories. Use `custom.json` files ([550])
+
+
+### Fixed
+
+- Minor `conf init` and detection of the authentication repository fixes ([550])
+- Replace `info` logging calls with `notice` in API functions ([550])
+- Use `mirrors.json` urls when cloning dependencies ([551])
+
+
+[551]: https://github.com/openlawlibrary/taf/pull/551
+[550]: https://github.com/openlawlibrary/taf/pull/550
+
+
+## [0.31.1] - 10/03/2024
+
+### Added
+
+### Changed
+
+### Fixed
+
+- Fix `load_repositories` following a rework needed to support parallelization ([547])
+- Fix `clone_from_disk` ([547])
+- Fix pre-push hook ([547])
+
+[547]: https://github.com/openlawlibrary/taf/pull/547
+
+
+## [0.31.0] - 09/28/2024
+
+### Added
+
+
+- Added lxml to taf pyinstaller to execute arbitrary python scripts ([535])
+- Added support for execution of executable files within the scripts directories ([529])
 - Added yubikey_present parameter to keys description (Can be specified when generating keys) ([508])
 - Removed 2048-bit key restriction [494]
 - Allow for the displaying of varied levels of log and debug information based on the verbosity level ([493])
@@ -29,9 +96,11 @@ and this project adheres to [Semantic Versioning][semver].
 - Added --bare tags for repository cloning and updating ([459])
 - Added workflow to build standalone executable of TAF ([447])
 
-
 ### Changed
 
+- If in detached head state or an older branch, do not automatically checkout the newest one without force ([543])
+- Move validation of the last validated commit to the pipeline from the update handler ([543])
+- Default verbosity to 0 (NOTICE) level; add notice level update outcome logging ([538])
 - Raise a more descriptive error if `pygit2` repository cannot be instantiated  ([485], [489])
 - Enhanced commit_and_push for better error logging and update the last validated commit ([469])
 - Generate public key from private key if .pub file is missing ([462])
@@ -40,8 +109,24 @@ and this project adheres to [Semantic Versioning][semver].
 
 ### Fixed
 
+- Handle invalid last validated commit ([543])
+- Fixes to executing taf handler scripts from a pyinstaller executable ([535])
+- Fix `persisent` and `transient` NoneType error when running taf handlers ([535])
+- Fix update status when a target repo was updated and the auth repo was not ([532])
+- Fix merge-commit which wasn't updating the remote-tracking branch ([532])
+- Fix removal of additional local commits ([532])
+- Fix top-level authentication repository update to correctly update child auth repos ([528])
 - Fix setup role when specifying public keys in keys-description ([511])
+- `check_if_repositories_clean` error now returns a list of repositories which aren't clean, instead of a single repository ([525])
+
 
+[543]: https://github.com/openlawlibrary/taf/pull/543
+[538]: https://github.com/openlawlibrary/taf/pull/538
+[535]: https://github.com/openlawlibrary/taf/pull/535
+[532]: https://github.com/openlawlibrary/taf/pull/532
+[529]: https://github.com/openlawlibrary/taf/pull/529
+[528]: https://github.com/openlawlibrary/taf/pull/528
+[525]: https://github.com/openlawlibrary/taf/pull/525
 [511]: https://github.com/openlawlibrary/taf/pull/511
 [508]: https://github.com/openlawlibrary/taf/pull/508
 [504]: https://github.com/openlawlibrary/taf/pull/504
@@ -89,6 +174,18 @@ and this project adheres to [Semantic Versioning][semver].
 [391]: https://github.com/openlawlibrary/taf/pull/391
 [389]: https://github.com/openlawlibrary/taf/pull/389
 
+## [0.30.3] - 08/29/2024
+
+### Added
+
+### Changed
+
+### Fixed
+
+- Move `yubikey_utils` module to include it in wheel ([516])
+
+[516]: https://github.com/openlawlibrary/taf/pull/516
+
 ## [0.30.2] - 08/20/2024
 
 ### Added
@@ -1222,7 +1319,11 @@ and this project adheres to [Semantic Versioning][semver].
 
 [keepachangelog]: https://keepachangelog.com/en/1.0.0/
 [semver]: https://semver.org/spec/v2.0.0.html
-[unreleased]: https://github.com/openlawlibrary/taf/compare/v0.30.2...HEAD
+[unreleased]: https://github.com/openlawlibrary/taf/compare/v0.31.2...HEAD
+[0.32.0]: https://github.com/openlawlibrary/taf/compare/v0.31.2...v0.32.0
+[0.31.2]: https://github.com/openlawlibrary/taf/compare/v0.31.1...v0.31.2
+[0.31.1]: https://github.com/openlawlibrary/taf/compare/v0.31.0...v0.31.1
+[0.31.0]: https://github.com/openlawlibrary/taf/compare/v0.30.2...0.31.0
 [0.30.2]: https://github.com/openlawlibrary/taf/compare/v0.30.1...v0.30.2
 [0.30.1]: https://github.com/openlawlibrary/taf/compare/v0.30.0...v0.30.1
 [0.30.0]: https://github.com/openlawlibrary/taf/compare/v0.29.1...v0.30.0

diff --git a/setup.py b/setup.py
@@ -1,9 +1,7 @@
 from setuptools import find_packages, setup
-from importlib.util import find_spec
-import sys
 
 PACKAGE_NAME = "taf"
-VERSION = "0.30.2"
+VERSION = "0.32.0"
 AUTHOR = "Open Law Library"
 AUTHOR_EMAIL = "[email protected]"
 DESCRIPTION = "Implementation of archival authentication"
@@ -24,6 +22,8 @@
     "freezegun==0.3.15",
 ]
 
+executable_require = ["lxml"]
+
 dev_require = ["bandit>=1.6.0", "black>=19.3b0", "pre-commit>=1.18.3"]
 
 tests_require = [
@@ -35,11 +35,6 @@
 
 yubikey_require = ["yubikey-manager==5.1.*"]
 
-# Determine the appropriate version of pygit2 based on the Python version
-if sys.version_info >= (3, 11):
-    pygit2_version = "pygit2==1.14.1"
-elif sys.version_info >= (3, 7) and sys.version_info < (3, 11):
-    pygit2_version = "pygit2==1.9.*"
 
 kwargs = {
     "name": PACKAGE_NAME,
@@ -63,14 +58,17 @@
         "cryptography>=40.0.0",
         "securesystemslib==1.*",
         "loguru==0.7.*",
-        pygit2_version,
+        'pygit2==1.9.*; python_version < "3.11"',
+        'pygit2==1.14.*; python_version >= "3.11"',
+        "pyOpenSSL==22.1.*",
         "logdecorator==2.*",
     ],
     "extras_require": {
         "ci": ci_require,
         "test": tests_require,
         "dev": dev_require,
         "yubikey": yubikey_require,
+        "executable": executable_require,
     },
     "tests_require": tests_require,
     "entry_points": {
@@ -97,14 +95,4 @@
     ],
 }
 
-
-try:
-    tests_exist = find_spec("taf.tests")
-except ModuleNotFoundError:
-    tests_exist = False  # type: ignore
-if tests_exist:
-    kwargs["entry_points"]["pytest11"] = (
-        ["taf_yubikey_utils = taf.tests.yubikey_utils"],
-    )
-
 setup(**kwargs)
diff --git a/taf/api/conf.py b/taf/api/conf.py
@@ -1,8 +1,10 @@
 from shutil import Error, copytree
+import shutil
 from typing import Optional
 from pathlib import Path
 from taf.api.keystore import generate_keys
 from taf.log import taf_logger
+from taf.utils import read_input_dict
 
 
 def init(
@@ -17,11 +19,11 @@ def init(
         taf_directory = Path(".taf")
 
     if taf_directory.exists() and taf_directory.is_dir():
-        taf_logger.info(".taf directory already exists.")
+        taf_logger.log("NOTICE", ".taf directory already exists.")
     else:
         # Create the .taf directory
         taf_directory.mkdir(exist_ok=True)
-        taf_logger.info("Generated .taf directory")
+        taf_logger.log("NOTICE", "Generated .taf directory")
 
     # Create the config.toml file
     config_file_path = taf_directory / "config.toml"
@@ -32,7 +34,15 @@ def init(
     keystore_directory.mkdir(exist_ok=True)
 
     # If any of these parameters exist you can assume the user wants to generate keys
-    if not keystore and not roles_key_infos:
+
+    # check if keystore already exists
+    roles_key_infos_dict = read_input_dict(roles_key_infos)
+    keystore = (
+        keystore or (roles_key_infos and roles_key_infos_dict.get("keystore")) or None
+    )
+    should_generate_keys = False
+    keystore_path = Path(keystore) if keystore else None
+    if not keystore:
         # Prompt the user if they want to run the generate_keys function
         while True:
             use_keystore = (
@@ -43,9 +53,9 @@ def init(
             if use_keystore in ["y", "n"]:
                 should_generate_keys = use_keystore == "y"
                 break
-    if should_generate_keys or (keystore and not roles_key_infos):
-        # First check if the user already specified keystore
-        if not keystore:
+
+        if should_generate_keys:
+            # First check if the user already specified keystore
             copy_keystore = (
                 input(
                     "Do you want to load an existing keystore from another location? [y/N]: "
@@ -61,32 +71,33 @@ def init(
                     keystore_path = Path(keystore_input)
                     if keystore_path.exists() and keystore_path.is_dir():
                         keystore = keystore_input  # Assign the string path to the keystore variable
+                        should_generate_keys = (
+                            False  # no need to generate keys, they will be copied
+                        )
                         break
                     else:
                         taf_logger.error(
                             f"Provided keystore path {keystore} is invalid."
                         )
-        # Check if keystore is specified now. If so copy the keys
-        if keystore:
-            try:
-                copytree(keystore, keystore_directory, dirs_exist_ok=True)
-                taf_logger.info(
-                    f"Copied keystore from {keystore} to {keystore_directory}"
-                )
-            except FileNotFoundError:
-                taf_logger.error(f"Provided keystore path {keystore} not found.")
-            except Error as e:
-                taf_logger.error(f"Error occurred while copying keystore: {e}")
+    # Check if keystore is specified now. If so copy the keys
+    if keystore and keystore_path and keystore_path.is_dir():
+        try:
+            copytree(keystore, keystore_directory, dirs_exist_ok=True)
+            taf_logger.log(
+                "NOTICE", f"Copied keystore from {keystore} to {keystore_directory}"
+            )
+        except FileNotFoundError:
+            taf_logger.error(f"Provided keystore path {keystore} not found.")
+        except Error as e:
+            taf_logger.error(f"Error occurred while copying keystore: {e}")
 
-        # If there is no keystore path specified, ask for keys description and generate keys
-        elif not roles_key_infos:
-            roles_key_infos = input(
-                "Enter the path to the keys description JSON file (can be left empty): "
-            ).strip()
-            if not roles_key_infos:
-                roles_key_infos = "."
-    if roles_key_infos:
-        generate_keys(taf_directory, str(keystore_directory), roles_key_infos)
-        taf_logger.info(
-            f"Successfully generated keys inside the {keystore_directory} directory"
+    if should_generate_keys:
+        generate_keys(keystore_directory, roles_key_infos)
+        taf_logger.log(
+            "NOTICE",
+            f"Successfully generated keys inside the {keystore_directory} directory",
         )
+
+    if roles_key_infos is not None and Path(roles_key_infos).is_file():
+        infos_config_path = (taf_directory / Path(roles_key_infos).name).absolute()
+        shutil.copy(str(roles_key_infos), str(infos_config_path))