[CI] add code coverage by grcov

We need to track the code coverage to make sure the basic testing is
in place to exercise all the code.

We are adding grcov to the CI pipeline to collect and aggregate code
coverage. The coverage result is uploaded to codecov.io. The dashboard
can help visualize the breakdown of test coverage.

Closes: #1420
Approved by: bob-wilson

.circleci/config.yml

@@ -51,6 +51,19 @@ commands:
             sudo apt-get clean
             sudo rm -r /var/lib/apt/lists/*
             rustup component add clippy rustfmt
+  install_code_coverage_deps:
+    steps:
+      - run:
+          name: Install grcov and lcov
+          command: |
+            sudo apt-get update
+            sudo apt-get install lcov
+            cargo install --force grcov
+  install_rust_nightly_toolchain:
+    steps:
+      - run:
+          name: Install nightly toolchain for features not in beta/stable
+          command: rustup install nightly
   build_setup:
     steps:
       - checkout
@@ -138,6 +151,22 @@ jobs:
             cargo install --force cargo-audit
             cargo audit
       - build_teardown
+  code-coverage:
+    description: Run code coverage
+    executor: build-executor
+    steps:
+      - build_setup
+      - install_code_coverage_deps
+      - install_rust_nightly_toolchain
+      - run:
+          name: Setup code coverage output
+          command: echo "export CODECOV_OUTPUT=codecov" >> $BASH_ENV
+      - run:
+          name: Run code coverage
+          command: ./scripts/coverage_report.sh . $CODECOV_OUTPUT --batch
+      - run:
+          name: Upload result to codecov.io
+          command: bash <(curl -s https://codecov.io/bash) -f $CODECOV_OUTPUT/lcov.info;
   terraform:
     executor: terraform-executor
     steps:
@@ -203,3 +232,4 @@ workflows:
               only: master
     jobs:
       - audit
+      - code-coverage

codecov.yml

@@ -0,0 +1,18 @@
+---
+codecov:
+  require_ci_to_pass: true
+
+ignore:
+  - testsuite/*
+  - x/*
+
+coverage:
+  # range for color spectrum display, red=50%, green=80%
+  range: "50...80"
+  round: down
+  precision: 1
+
+  status:
+    project: true
+    patch: true
+    changes: false

scripts/coverage_report.sh

@@ -84,7 +84,9 @@ echo "Cleaning project..."
 echo "Running tests..."
 while read -r line; do
         dirline=$(realpath $(dirname "$line"));
-        (cd "$dirline" && pwd && cargo xtest)
+        # Don't fail out of the loop here. We just want to run the test binary
+        # to collect its profile data.
+        (cd "$dirline" && pwd && cargo xtest || true)
 done < <(find "$TEST_DIR" -name 'Cargo.toml')
 
 # Make the coverage directory if it doesn't exist