diff --git a/README.md b/README.md index 031b2f01f..bde468511 100644 --- a/README.md +++ b/README.md @@ -14,3 +14,25 @@ benchmark, actual users should benefit. In order to achieve this, it should: - Be accessible to the public and useful to browser engineers. It should run in every modern browser by visiting a normal web page. It should run relatively quickly, while providing enough test coverage to be reflective of the real-world Web. + +# Score and Test Measurement + +Each test can contain several steps who contributed to the test duration. Note that the prepare step of a test is _unmeasured_ and thus does not contribute to the score. All following test +steps are **measured** and summed up in the test step time. + + + +Each step consists of the following phases: + +
+
Sync:
+
Time spent in synchronous JS execution.
+
Async:
+
Time spent in asynchronous JS like Promise or setTimeout tasks scheduled by the sync-step.
+
+ +## Score + +The final benchmark score is calculated based of the inverse of [geomean](https://en.wikipedia.org/wiki/Geometric_mean) of all tests. We average the score over multiple iterations with the [arithmetic mean](https://en.wikipedia.org/wiki/Arithmetic_mean). By using the geomean to combine the test durations we maintain the invariant that relative improvements are favoured equally amongst all tests, even though they can have vastly different durations. + + diff --git a/resources/measurement-timeline.svg b/resources/measurement-timeline.svg new file mode 100644 index 000000000..865583cfe --- /dev/null +++ b/resources/measurement-timeline.svg @@ -0,0 +1,88 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Iteration:Iteration 1...Iteration N
Test:TodoMVC-JavaScript-ES5...Perf-DashboardTodoMVC-......
Steps:PrepareAddingCompleting...Prepare...Prepare...
SyncAsyncSyncAsync
+ + \ No newline at end of file diff --git a/resources/score.svg b/resources/score.svg new file mode 100644 index 000000000..d1f51b63f --- /dev/null +++ b/resources/score.svg @@ -0,0 +1,198 @@ + + + + + + + + + + Score + + + + + = + + 1 + n + + + + ∑ + + i + = + 1 + + n + + + + + IterationScore + + i + + + + + + + + + IterationScore + + i + + + + + = + + 1000 + + + IterationGeoMeanTime + + i + + + + + + + + + + IterationGeoMeanTime + + i + + + + + = + + + + + ∏ + + j + = + 1 + + m + + + + + TestTime + + + i + j + + + + m + + + + + + + + + TestTime + + + i + j + + + + + + = + + + ∑ + k + + + + + TestStepTime + + + i + j + k + + + + + + + + + + TestStepTime + + + i + j + k + + + + + + = + + + TestStepSyncTime + + + i + j + k + + + + + + + TestStepAsyncTime + + + i + j + k + + + + + + + + + \ No newline at end of file