Scoring Strategies

[MeritedHobbit]

Summarizing the scoring strategies

Assume we have 4 test level/buckets - L0, L1, L2, L3 (in decreasing order of priority/importance, L3 being not important at all)

Assume a typical test case distribution across the levels,
L0 = (T0, T1),
L1 = (T2, T3, ..T7),
L2 = (T8, .......... T15)
L3 -> (T16...remaining).
Why levels? - CSPs always want to know how compliant the product is. While the expectation typically is that we are 100% compliant post-production, that's often not a mandate during early validation stages (or pre-production). Nevertheless, an indication of L0 test cases failing during pre-production would permit raising the alarm bells.

The considered scoring & reporting schemes so far...

1. Simple count of test cases (P/F) = Score.
   Grade = Score/(Executed Test Cases) * 100.
   Overall Grade = Score/(Available Test Cases) * 100

2. Level weighted scoring. Let w0, w1, w2, w3 be the per test case weight of test cases under L0, L1, L2 & L3 respectively.
   Score = Weighted sigma of all passed test cases.
   Grade = Score / (Weighted sigma of all executed test cases) * 100.
   Overall Grade = Score / (Weighted sigma of all available test cases) * 100.

3. Normalized level weighted scoring. Similar to above but,
   w0 + w1 + w2 + w3 = 100. &
   Weight per test case is (weight of the level it belongs to) / (total number of test cases under that level).
   For the above example, if w0=50, w1=36, w2=14, w3=0 then weight of T0 = 50/2 = 25 = weight of T1. Similarly weight of T2 = 36/6 = 6 = weight of T3..

4. So far, we have assumed that the weight of every test case within a level is identical. In the eg. of 3rd scheme, all L1 test cases had a weight of 6 (a total of 36). But we could potential have differing weights per test case within the level. In which case, 36 would be distributed proportionally.

5. Taking the above further would be weighting every test step of OCPTV. Maybe once we standardize one of the above, we can explore this. Or maybe we don't have to standardize scoring in the first place :).

Why weighted scoring schemes? - permits the partnership (Vendor+CSP) to see an organic progress towards 100% compliance before going into production.