Instrumentation-Based Test Coverage Measurement

[notJoon]

tl;dr

This RFC proposes replacing the current (not merged yet) test coverage mechanism—which relies on indirect execution information via Run() from Machine function—with a more precise and maintainable approach based on code instrumentation. By automatically inserting coverage tracking code into the test source, this can overcome several current issues:

• Imprecise execution tracking: Directly reading executed lines from the Machine (especially, Op Codes) is inaccurate.
• Tight coupling with Machine: The current approach heavily depends on the Machine, making maintenance difficult.
• Heavy static analysis: Additional manual static analysis is needed to adjust coverage data.

Using instrumrntation, we generate additional code during the build process that records execution hits at predefined points, thereby decoupling coverage collection from the runtime Machine and simplifying both maintenence and accuracy.

Background

The current PR (#2616) implements coverage measurement by indirectly retrieving execution information. This method, however, has three major drawbacks:

1. Imprecise Line Tracking:
   Using Op Code to capture executed lines from the Machine.Run provides only approximate coverage data.

2. Strong Dependency on the Machine:
   The current design introduces a tight coupling between the coverage system and the Machine implementation, leading to maintenance issues as the Machine evolves.

3. Complex Post-Processing:
   To correct and reconcile coverage data, significant manual static analysis is performed, increasing both development and testing overhead.

Proposed Approach

I'll switch current implementation to an instrumentation-based method for measuring test coverage. The idea is to perform source-level instrumentation before complication to inject code that record coverage events.

How It Works

1. Parsing and Instrumentation
   Before tests are run, the source code (e.g., *_test.gno and *_filetest.gno files) is parsed into and AST using package like go/ast and go/parser.
   For each function declaration (or other code blocks are needed), we insert a call to a coverage tracking function.
2. Injected Coverage Call
   For example, a call like:

testing.CoverageHit("filename:FunctionName:line")

is inserted at the beginning of each function. The unique ID includes the file name, function name, and the line number of the function declaration.

3. Post-Test Reporting:
   After running tests, the coverage data is collected and reported, providing precise information on which code paths were executed.

Instrumentation Flow

Below is an diagram to illustrate the process:

             +-----------------------------+
             |       Test Source Files     |
             | (e.g., *_test.gno, etc.)    |
             +-------------+---------------+
                           |
                           v
             +-----------------------------+
             |       AST Parsing           |
             |  (using go/ast, go/parser)  |
             +-------------+---------------+
                           |
                           v
             +-----------------------------+
             |  Instrumentation Insertion  |
             +-------------+---------------+
                           |
                           v
             +-----------------------------+
             |   Code Generation           |
             |  (reconstruct source code)  |
             +-------------+---------------+
                           |
                           v
             +-----------------------------+
             |  Compile & Run Tests        |
             |  (instrumented code)        |
             +-------------+---------------+
                           |
                           v
             +-----------------------------+
             |  Global Coverage Data       |
             |  Collection (runtime hits)  |
             +-------------+---------------+
                           |
                           v
             +-----------------------------+
             |  Generate Coverage Report   |
             +-----------------------------+

Prior Arts and References

Several established tools use instrumentation for coverage measurement. Examples includes:

• coverage.py (Python): https://coverage.readthedocs.io/en/7.6.12/
• -C instrument-coverage flag (Rust): https://doc.rust-lang.org/rustc/instrument-coverage.html

Conclusion

Switching to an instrumentation-based approach for test coverage measurement addresses the key issues faced with the current implementation:

• Enhanced Accuracy: Direct insertion of coverage tracking calls provides precise execution data.
• Decoupling: Removing the dependency on the Machine simplifies maintenance.
• Reduced Analysis Overhead: The approach minimizes the need for post-hoc static analysis and manual correction.

---

Appendix

A. Example with Code

Suppose we have a simple function defined as follows in a file named math.gno

package math

// addNums returns the sum of two numbers.
func addNums(a, b int) int {
    return a + b
}

And a corresponding test file math_test.gno:

package math

import "testing"

func TestAddNums(t *testing.T) {
    result := addNums(2, 3)
    if result != 5 {
        t.Errorf("expected 5, got %d", result)
    }
}

Instrumentation Process

Before compiling and running tests, our instrumentation tool will parse the source and insert a coverage call at the beginning of addNums. For example, it will modify the function to look like:

package math

import "testing"

// addNums returns the sum of two numbers.
func addNums(a, b int) int {
    // Instrumentation: record a hit for this function.
    testing.CoverageHit("math.gno:addNums:3")
    return a + b
}

Here, the unique ID "math.gno:addNums:3" encodes the file name, function name, and the line number where the function starts (line 3 in this case).

Runtime Collection and Reporting

At runtime, each time addNums is called during testing, the instrumentation call:

testing.CoverageHit("math.gno:addNums:3")

updates a global map in the testing package. For example, the global state might look like:

// In testing package:
var coverageData = map[string]int{
    "math.gno:addNums:3": 1, // Function hit once during TestAddNums
}

After the tests finish, a coverage report can be generated by iterating over this map, showing how many times each instrumented point was executed.