Instrument Code in CI/CD Pipeline, Execute Integration tests Later Against Deployed Instrumented Code

[ericdraney]

I would like to instrument my code in my yml ci/cd pipeline, have the instrumented assemblies deployed to an environment. Unit tests will be run as part of that pipeline. I would then like to exercise the deployed and instrumented code via integration tests in a separate pipeline and collect code coverage metrics for the deployed code in step 1. What approach would I explore to make that happen?

[SteveGilham]

Yes, that's just the Instrument now, test later, collect coverage after that mode of operation, with the expected level of other things going on in the middle. If this is all on the same machine and the test process has write access to the folder the coverage report was placed in, you're all good.

In the case where instrumenting is done on one (possibly transient virtual) machine and testing on another, the absolute path of the coverage.xml file should be in a location that is writeable by the test process.

So, for example, if the instrumentation puts the coverage file as, say, C:\Users\Public\Documents\coverage.xml on the one machine, then so long as the C:\Users\Public\Documents\ folder on the test machine is writeable by the test process, the raw *.<digits>.acv files will be generated there. To control data collection, use the presence of the control file in the equivalent location.

To collect the data as a report, assemble the original report created during instrumentation and all the *.<digits>.acv files from the test process together in the corresponding location on possibly a third machine, then perform the collect step there.

Variants:

1. It should be possible to write to a common network share e.g. define coverage as \\fileserver\documents\coverage.xml, so the copy from the test machine happens as part of the test. I've not tested this, so have no clear measure of how the speed of network write operations will impact performance of the test process.
2. If you're not using a test runner with an aggressive shutdown policy, it would be possible to use the Instrument now, test later, classic mode instead, copying the initial generated coverage report and control files to the test machine, where the report will be filled in on test termination, with the usual limitations (NCover or partially filled OpenCover format reports only); and then use the PowerShell capabilities to fill in the gaps and transform to a different report format, with no third step/machine required.

[SteveGilham]

One thing I forgot - you'll also need at least the altcover.recorder.g.dll file from the instrumented files on the third machine - the collect process looks at information written into that file for the path to the coverage report.

[ericdraney]

Steve, thanks for the quick response. Quick question does Altcover have the ability to instrument the assemblies within a pipeline packaged artifact. Today when I instrument the deployable artifacts that are packages as a .zip file altcover drops a coverage.xml file where I designate and the altcover.recorder.g.dll but it appears that the assemblies themselves inside of the .zip remain untouched.

[ericdraney]

Also I'm having a hard time understanding the purpose of the --save parameter in the instrumentation step. Following the instrument now, test later, collect coverage I would assume I want to generate that .pdb but for whatever reason having a hard time grasping the value add.

[SteveGilham]

To the first point, there's no unpackaging done - it's assumed that the files are deployed into a directory from which they could be executed. This could be either before being packaged, or after being deployed from a package, just so long as the debug symbols are available (embedded, as .pdb files in the same directory, or at the original intermediate build location stored in the assembly).

To the second, the original, for .NET Framework, implementation c2010 just supported the "classic" mode whereby the recorder assembly stores the coverage visit data in memory and then uses it to rewrite the coverage file using the ProcessExit handler, which takes advantage of the couple of seconds' grace that allows to do the work. When porting to the - then still quite new - cross-platform .Net Core in 2017, I found that, although it was documented to have an indefinite grace period for the ProcessExit, the dotnet test operation took it upon itself to force a much shorter (100ms ) one, insufficient to allow any reasonably size coverage file to be reported.

The --save parameter added in response instructs the recorder to write the raw coverage data to one or more files, doing no more than flushing any open files to disk in ProcessExit; leaving the processing to the new collect phase. The parameter sets this in motion by the simple mechanism of creating the control file at the same time as the template coverage file; the recorder then uses its presence or absence as a toggle for how it should direct the coverage data, using a FileSystemWatch handler to support the pause/resume behaviour.

--save parameter has a

[ericdraney]

Where we arrived:

1. Instrument the code in the initial pipeline and deploy it to environment X.
2. On the second pipeline that contains our integration tests we use PowerShell remote to execute
   altcover runner --collect -r <path to instrumented folder>
   on the environment we deployed to. We assume that will flush the metrics and start collection.
3. Then execute the tests on the second pipeline.
4. Execute an IISReset on the remote machine to write the coverage collection from memory to disk. Unsure if there is a way outside of this to force the flushing of collected coverage to disk. Sounds like we may need the --save parameter in our initial altcover runner call? Or am I thinking about this the wrong way?

[SteveGilham]

In this distributed system, the text value of [absolute path to coverage reportfile] on the instrumentation machine being an accessible location on all the machines the key to the whole thing.

For the current set-up, instrumenting without --save would be fine; you could just create the [absolute path to coverage reportfile].acv control file manually on the test machine to instruct the execution to save the coverage data to file as required. In either case, deployment should include a copy of the [absolute path to coverage reportfile] from the instrumentation process at the exact same path on the test machine at the point in time when the coverage data are being processed.

Depending on the metrics you are gathering, what output format you want, and whether your test runner places a time limit on ProcessExit handling, just running the instrumented files may be enough; use --save and altcover runner --collect to get the full spectrum (or use the various APIs or PowerShell cmdlets to similar effect).

Of course a simpler approach (the use case implicitly assumed) would be to defer instrumentation until after test deployment, but before running tests. As you already have altcover on the test machine to do the collect function, would not involve any configuration changes there. Using altcover --inplace will put the instrumented files in the deployed location, where they can be run as before.

---

Going into more implementation detail -

The function of the --save parameter is to purely create the file [absolute path to coverage reportfile].acv - the control file, which is used as a way of passing one bit of information to the coverage recording process, toggling whether to process coverage data at program exit, or to write it to file.

When the instrumented code executes, the recorder assembly looks at the [absolute path to coverage reportfile] which has been written into it, and appends .acv, to find the control file. If it exists, it will create one or more [absolute path to coverage reportfile].[number].acv files containing a compressed record of the branch and sequence point visits. If not, the visit data will be stored in memory.

When the process running the instrumented code terminates, if the control file never existed during the run, the (NCover or OpenCover format) empty coverage report [absolute path to coverage reportfile] is read and filled in with the branch and sequence point visit counts dumped from memory (ignoring all the derived quantities in the OpenCover format). If the control file existed at any point, then any remaining data in I/O buffers is simply flushed to disk.

The function of altcover runner --collect is to read [absolute path to coverage reportfile] from the recorder, then read all the the [absolute path to coverage reportfile].[number].acv files and process them into the zero coverage original report, which bypasses any process exit time constraints, and handles more elaborate output formats than the basic XML ones.

[SteveGilham]

Release 8.8.10 offers the --portable option, which puts the coverage report, control file, and intermediate data files in the same directory as the recorder assembly, so allowing the whole instrumented application to be moved between filing systems.

[ericdraney]

@SteveGilham we have decided to pursue another path as the security hurdles were too much trying to collect coverage from a pipeline on different subnet. So what we have today is in our build pipeline we are instrumenting the code using:
altcover -i={pathtodropinstrumentedassemblies} -t=Microsoft
We can see that the code is getting instrumented but don't have a coverage.xml file.
We package the instrumented assemblies in a .zip file and overwrite our deployable zip to Octopus with the instrumented version.
We deploy the code to our environment lets just say e:/release1234 out on our server, in this case its a windows service so we recycle the service to use the new (instrumented) code.
We then execute as a post deploy step our test suite against instrumented code.
I'm still confused in this case what I need to execute in altcover pre-test run to flush coverage details and what I need to execute post test suite in altcover to gather the metrics. (I do understand that I likely need to bounce the service to stop the collection.)
I think I can then push the metric details to ADO with a yml step but unsure how to flush and collect the metrics, it sounds like runner mode is what I need but the documentation is a little hard to understand.

[SteveGilham]

The coverage template file will appear in whatever is the process current working directory by default. However for this sort of scenario, using the --portable option, which moves the report (and the temporary coverage data files) to be co-located with the instrumented files (or, more precisely the recorder assembly added to the instrumented files), wherever that might happen to be.

The line

altcover -i={pathtodropinstrumentedassemblies} -t=Microsoft

concerns me, as -i is for the as-built files; -o overrides the the default relative folder for the instrumented files, unless you're also using --inplace.

The sort of process I would expect goes something like

altcover -i <as built location> -o <where to pick up from> --save --portable [...filters, other options...]

or perhaps

altcover -i <as built location> -o <anywhere else> --inplace --save --portable [...filters, other options...]

Then you can zip/tar/whatever up the -o folder in the first case (or -i for the --inplace option) and drop it where you need to do testing, and run the tests.

After the tests are run, the you can do altcover runner --collect in the test location and retrieve the report file, or zip/whatever up (at least) the report file, the AltCover.Recorder.g.dll assembly and all the .acv files, and run the altcover runner --collect operation on what you fetch back.

[ericdraney]

All seems to work fine through the instrumentation process. I get a FileLoad exception when starting up the instrumented version of the windows service. This is going to be fun to figure out.