Integrating the RootKey Feature into the pre-release develop branch (A…

…zure#568) * initial util and types * remove ECC and change untrusted to disabled * Remove EC conditional from rootkeypackage json schema * build rootkeypackage_utils * Add missing doxygen docs in rootkeypackage_types.h * Fix fn name prefix and add Cleanup * Parse root key package + UTs (Azure#255) * parse bad json pkg * add test for valid protected props * add err code for failed parse of pkg * add parse module, scaffolding * parse version and published * implement most of parsing * add ParseRootKey * revert CMAKE_VERBOSE_MAKEFILE * Complete parsing, add test docs, UTs complete with hardcoded hashs and sigs * add test for verify rootkey rsa params, fix parse rootkey * factor out cleanup into component cleanup functions * Make valgrind clean--fix TestRSAKeyPair memory leak * transfer ownership of kidStrHandle and rsa_modulus only after VECTOR_push_back succeeds * replace all MISSING_REQUIRED_PROPERTY with more specific ERC * rename _Free to _DeInit for RootKey and RootKeyPackage_Hash * rename _Cleanup to _Destroy * replace outlier with ADUC_RootKeyPackage_Hash_DeInit that was supposed to replace it * separate parse of signature and hashes, using RSxxx and SHAversion, respectively * Report DeploymentInPRogress before update manifest signature valid (Azure#256) * Refactor _workflow_parse and create workflow_parse_peek_unprotected_workflow_properties * Report DeploymentInProgress before update manifest signature valid * mv and update rootkeyfiles and add rootkey.json package (Azure#272) * mv and update rootkeyfiles and add rootkey.json package * keytype to keyType and parse number instead of string for 'e' RSA exponent * clang format * Added parsing of the protected properties into a json string for hash… (Azure#274) * Added parsing of the protected properties into a json string for hashing later on * Added granular no mem erc * Implementation of the RootKeyUtility (Azure#279) * parse bad json pkg * add test for valid protected props * add err code for failed parse of pkg * add parse module, scaffolding * parse version and published * implement most of parsing * add ParseRootKey * revert CMAKE_VERBOSE_MAKEFILE * Initial commit * Complete parsing, add test docs, UTs complete with hardcoded hashs and sigs * add test for verify rootkey rsa params, fix parse rootkey * factor out cleanup into component cleanup functions * Make valgrind clean--fix TestRSAKeyPair memory leak * Integrated new work and added stub functions * transfer ownership of kidStrHandle and rsa_modulus only after VECTOR_push_back succeeds * replace all MISSING_REQUIRED_PROPERTY with more specific ERC * Pre-merge commit * Reworking and remerging * Finished implementation and prepared for unit tests * Initial commit for checking into store * Added changes for unit tests * Changing build files and unit tests to reflect latest chagnes * Fixed a few bugs, added some functionality * Added parsing of the protected properties into a json string for hashing later on * Moved to using root key utils root keys * reworked key creation to follow normal flow for checks * reworking names * Unifying result code types and added documentation * updated logs and results * Fixed unit tests, added functionality for RootKeyUtility, and packaged it up nice * Removed musings and miscellaneous comments * responded to feedback and added comments * Responded to more feedback * Fixed init/deinit process for keys Co-authored-by: jw-msft <[email protected]> * Add rootkey package download (Azure#275) * Add copyright header to rootkeypackage_parse.c * add GetLastPathSegmentOfUrl in new url_utils * parse rootKeyPackageUrl from propupd json, add call to download fr adu core ifc * impl DO rootkey downloader * add download ut * Add rootkeypackage test app * Move https_proxy_utils into url_utils * Rename to GetPathFileName and rmv dep on libcurl (Azure#285) * Rev deviceUpdate Model and ContractModel to 3 for adu-m rootkey support (Azure#383) * Rev deviceUpdate Model and ContractModel to 3 for adu-m rootkey support * rmv typo, dup word * Added valid tests and fixed bugs exposed by the full unit tests (Azure#395) * Added valid tests and fixed bugs exposed by the full unit tests * Removed unused errors * Add cmake fn to add test root keys only for debug builds (Azure#438) - Add embed_test_root_keys_if_applicable cmake function into new security.cmake - Invoke embed_test_root_keys_if_applicable instead of unconditionally including test root keys - Ran cmake-format.sh - Verified the following using `objdump -s out/src/utils/root_key_utils/CMakeFiles/root_key_utils.dir/src/root_key_lists.c.o`: - `--type=Debug` has 2 retail *and 2 test keys* - `--type=RelWithDebInfo` has only 2 retail keys - `--type=Relelease` has only 2 retail keys - `--type=MinSizeRel` has only 2 retail keys * Added code to create the rootkey store when it does not exist (Azure#448) * Initial integration of rootkeypackage_utils * fix swap of out params when parse unprotected params * pass rootkeypkgurl through to download * Fixed bug where rootkeystore was not created at startup * Repsonded to PR feedback --------- Co-authored-by: jw-msft <[email protected]> * report ERCs str, rootkey workflow/dnld fixes, add rootkey telemetry (Azure#489) * report ERCs str, rootkey workflow/dnld fixes, add rootkey telemetry * fix static analysis errors * Add enforcement of disabledSigningKeys rootkey pkg (Azure#495) * Add RootKey disabledSigningKeys support * clang format jws_utils.c * rmv stray commented line * rmv unnecessary malloc+memcpy for CONSTBUFFER_Create * check NULL from GetEvpMdFromShaAlg * add libxml2-dev for codeql github action * Revert "add libxml2-dev for codeql github action" This reverts commit 37215cf. * catch exceptions from DO when download rootkey pkg (Azure#528) * catch exceptions from DO when download rootkey pkg * fix cppcheck err for free of possibly uninitialized var * Disabled RootKey/SigningKey Tests and Fix UB crash (Azure#536) * add prod disabled rootkey unit test, no signatures * fix crash rootkey pkg parse not zero after malloc * add test for ReloadPkg and 0 disabled keys * add disabled signingKey GetDisabledSigningKey UT * remove unused fn replaced by CryptoUtils GenPubKey usage * Add isTest support to rootkey pkg schema (Azure#542) * add isTest boolean property to rootkey pkg schema * add isTest to rootkey pkg protected props * fail on mismatch isTest and e2etest agent * include result in report failure * Resolve rootkey download work folder properly after rebase from develop branch. * fix type conv issues found in yocto build * fix another yocto build issue * fix double -> time_t * Adds the ability to create a service e2e compliant test agent (Azure#567) * Creating service side agent e2e build process * Ban strncpy with compilation error, replaced by ADUC_Safe_StrCopyN (Azure#560) * Ban strncpy with compilation error on use * Added changed result code on success * fixed build errors * Integrating OpenSSL 3 Changes and Fixing Pipelines for Ubuntu 22.04 (Azure#544) * Added new sdk update * Changing around apt deployment test to check error * Upgrading to latest C-SDK for OpenSSL integration * Added install instructions for OpenSSL 3.0 in build dependencies * Initial commit with openssl integration * forgot a file * Added support for OpenSSL 3 and added build configs * Adding support for installing non platform standard openssl installations * fixing minor errors * Openssl installation script * Adding additional content for openssl migration * Fixed errors * Adding better #def guards * test * fixing bugs * Adding Ubuntu 22.04 build * pipeline reenabling * removed weird wording * removing enforced openssl requirements * Addressing windows build failures * Removed openssl 1.1.1w installation on ubuntu 22.04 * Corrected the version of Ubuntu 22.04 ARM64 and AMD64 image_offer param * Upgrading the root key modulus work to include openssl 3.0 work * Fixed openssl 3.0 incompatible error * Revert "fixed build errors" This reverts commit 2656c0b. * fixed build errors * Revert "Integrating OpenSSL 3 Changes and Fixing Pipelines for Ubuntu 22.04 (Azure#544)" This reverts commit 5947c46. * fixed double definition * fixed ubuntu 22.04 build errors * format mistake * Added OpenSSL 3.0 version of the CryptoUtils_GeneratePublicKey function and fixed some minor exit function issues * fixing debug issues * fixing debug issues * build error * build. issues. help. --------- Co-authored-by: jw-msft <[email protected]> * Ban strncpy with compilation error, replaced by ADUC_Safe_StrCopyN (Azure#560) (Azure#569) * Ban strncpy with compilation error on use Co-authored-by: jw-msft <[email protected]> * Fixed merge errors (Azure#572) * Added target_link_dosdk for rootkeypackage * nonsense * Fixed and addressed windows system build errors * fixed errors on linux caused by windows differences * Fixed errors in the root_key_utils get_root_keys function * Fixed test errors * fixed jws utils uts * Added position independent code signifier to prevent bad module integrations --------- Co-authored-by: jw-msft <[email protected]>
AndreRicardo-Zoetis · Nov 18, 2023 · 22deada · 22deada
1 parent 0fb456a
commit 22deada
Show file tree

Hide file tree

Showing 141 changed files with 10,239 additions and 1,847 deletions.
diff --git a/.gitignore b/.gitignore
@@ -29,6 +29,10 @@ cscope.out
 cscope.in.out
 cscope.po.out
 
+### openssl output
+*.sig
+*.pem
+
 ### Python Module Files
 *__pycache__
 *__pycache__/*

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -212,23 +212,36 @@ else ()
         ADUC_STEP_HANDLERS
         "microsoft/swupdate_v2,microsoft/swupdate,microsoft/script,microsoft/apt,microsoft/simulator"
         CACHE STRING "The list of step handlers.")
-
-    set (
-        ADUC_STEP_HANDLERS
-        "microsoft/swupdate_v2,microsoft/swupdate,microsoft/script,microsoft/apt,microsoft/simulator"
-        CACHE STRING "The list of step handlers.")
 endif ()
 
+set (
+    ADUC_ROOTKEY_STORE_PATH
+    "${ADUC_DATA_FOLDER}/rootkeystore/"
+    CACHE STRING "Path to the folder containing the information for the local store of root keys")
+
+set (
+    ADUC_ROOTKEY_STORE_PACKAGE_PATH
+    "${ADUC_ROOTKEY_STORE_PATH}/rootkeys.json"
+    CACHE STRING "Path to the folder containing the local root keys")
+
 set (
     ADUC_COMMANDS_FIFO_NAME
     "${ADUC_DATA_FOLDER}/du-commands.fifo"
     CACHE STRING "The named-pipe for commands IPC.")
 
+set (
+    ADUC_ROOTKEY_PKG_URL_OVERRIDE
+    ""
+    CACHE
+        STRING
+        "An override URL to use for the public rootkey package. Leave empty to use the URL provided in the C2D message."
+)
+
 #
 # Starting from version 0.8.1, Device Update Agent must support only one version of the update manifest
 # based on the base dtmi model that the Agent announces when connecting to the IoT Hub.
 #
-# For this version, "dtmi:azure:iot:deviceUpdateModel;2" is using the manifest version verison 5.
+# For this version, "dtmi:azure:iot:deviceUpdateModel;3" is using the manifest version 5.
 #
 set (SUPPORTED_UPDATE_MANIFEST_VERSION_MIN 4)
 set (SUPPORTED_UPDATE_MANIFEST_VERSION_MAX 5)
@@ -464,6 +477,9 @@ option (ADUC_BUILD_PACKAGES "Build the ADU Agent packages" OFF)
 option (ADUC_INSTALL_DAEMON "Install the ADU Agent as a daemon" ON)
 option (ADUC_REGISTER_DAEMON "Register the ADU Agent daemon with the system" ON)
 option (ADUC_TRACE_TARGET_DEPS "Trace target dependencies" OFF)
+option (ADUC_USE_TEST_ROOT_KEYS "Include test root keys" OFF)
+option (ADUC_ENABLE_E2E_TESTING "Enable e2e test pipeline settings" OFF)
+option (ADUC_ENABLE_SRVC_E2E_TESTING "Enable service side test agent settings" OFF)
 
 ### End CMake Options
 

diff --git a/scripts/build.sh b/scripts/build.sh
@@ -31,11 +31,14 @@ build_packages=false
 platform_layer="linux"
 trace_target_deps=false
 step_handlers="microsoft/apt,microsoft/script,microsoft/simulator,microsoft/swupdate,microsoft/swupdate_v2"
+use_test_root_keys=false
+srvc_e2e_agent_build=false
 build_type=Debug
 adu_log_dir=""
 default_log_dir=/var/log/adu
 output_directory=$root_dir/out
 build_unittests=false
+enable_e2e_testing=false
 declare -a static_analysis_tools=()
 log_lib="zlog"
 install_prefix=/usr/local
@@ -50,6 +53,7 @@ print_help() {
     echo "                                      Options: Release Debug RelWithDebInfo MinSizeRel"
     echo "-d, --build-docs                      Builds the documentation."
     echo "-u, --build-unit-tests                Builds unit tests."
+    echo "--enable-e2e-testing                  Enables settings for the E2E test pipelines."
     echo "--build-packages                      Builds and packages the client in various package formats e.g debian."
     echo "-o, --out-dir <out_dir>               Sets the build output directory. Default is out."
     echo "-s, --static-analysis <tools...>      Runs static analysis as part of the build."
@@ -225,6 +229,9 @@ while [[ $1 != "" ]]; do
     -u | --build-unit-tests)
         build_unittests=true
         ;;
+    --enable-e2e-testing)
+        enable_e2e_testing=true
+        ;;
     --build-packages)
         build_packages=true
         ;;
@@ -262,6 +269,12 @@ while [[ $1 != "" ]]; do
     --trace-target-deps)
         trace_target_deps=true
         ;;
+    --use-test-root-keys)
+        use_test_root_keys=true
+        ;;
+    --build-service-e2e-agent)
+        srvc_e2e_agent_build=true
+        ;;
     --log-lib)
         shift
         if [[ -z $1 || $1 == -* ]]; then
@@ -347,6 +360,15 @@ library_dir=${output_directory}/lib
 cmake_bin="${cmake_dir_path}/bin/cmake"
 shellcheck_bin="${work_folder}/deviceupdate-shellcheck"
 
+if [[ $srvc_e2e_agent_build == "true" ]]; then
+    warn "BUILDING SERVICE E2E AGENT NEVER USE FOR PRODUCTION"
+    echo "Additionally implies: "
+    echo " --enable-e2e-testing , --use-test-root-keys, --build-packages"
+    use_test_root_keys=true
+    enable_e2e_testing=true
+    build_packages=true
+fi
+
 # Output banner
 echo ''
 header "Building ADU Agent"
@@ -360,6 +382,7 @@ bullet "Log directory: $adu_log_dir"
 bullet "Logging library: $log_lib"
 bullet "Output directory: $output_directory"
 bullet "Build unit tests: $build_unittests"
+bullet "Enable E2E testing: $enable_e2e_testing"
 bullet "Build packages: $build_packages"
 bullet "CMake: $cmake_bin"
 bullet "CMake version: $(${cmake_bin} --version | grep version | awk '{ print $3 }')"
@@ -370,17 +393,20 @@ if [[ ${#static_analysis_tools[@]} -eq 0 ]]; then
 else
     bullet "Static analysis: " "${static_analysis_tools[@]}"
 fi
+bullet "Include Test Root Keys: $use_test_root_keys"
 echo ''
 
 CMAKE_OPTIONS=(
     "-DADUC_BUILD_DOCUMENTATION:BOOL=$build_documentation"
     "-DADUC_BUILD_UNIT_TESTS:BOOL=$build_unittests"
     "-DADUC_BUILD_PACKAGES:BOOL=$build_packages"
     "-DADUC_STEP_HANDLERS:STRING=$step_handlers"
+    "-DADUC_ENABLE_E2E_TESTING=$enable_e2e_testing"
     "-DADUC_LOG_FOLDER:STRING=$adu_log_dir"
     "-DADUC_LOGGING_LIBRARY:STRING=$log_lib"
     "-DADUC_PLATFORM_LAYER:STRING=$platform_layer"
     "-DADUC_TRACE_TARGET_DEPS=$trace_target_deps"
+    "-DADUC_USE_TEST_ROOT_KEYS=$use_test_root_keys"
     "-DCMAKE_BUILD_TYPE:STRING=$build_type"
     "-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON"
     "-DCMAKE_LIBRARY_OUTPUT_DIRECTORY:STRING=$library_dir"