Date filtering bugs - Overview tab - scan count discrepancy

[jcf2]

There seems to be a discrepancy between the table and bar portions of the Overview when date filtering with "last 15 days" is used (this might possibly be a sign of a general problem, but it's much easier to verify manually in this case because of the low numbers).

Example (situation from 8/26 report, development version): currently, with "15 days" window selected, the number of scan sessions is 10, from 3 sites:

Setting "split" and "by site" in the bar chart, specific scan counts are

Issues:

• The UI site has 1 V1 scan session and 6 individual scans, which seems fine. This corresponds to the Heat Map, which shows one session (11051) with 6 scans.
• UC is definitely problematic. It has 1 V1 + 5 V3 scan sessions in the table, and yet only 12 combined scans in the bar plot (normally would expect 3-6 per session in almost all sessions). Furthermore, in the bar plot all 12 count as V3, which is inconsistent with the table reporting a V1 session as well. The current Heat Map plot of UC with "last 15 days" shows two V3 scan sessions, with 6 scans each.
• UM is also seemingly problematic. . It says 0 V1 + 3 V3 sessions in the table, and shows a total of only 3 scans, all V3, in the bar plot. It is technically possible for each session to have only one scan, but this would be highly unusual for all three sessions. I can't get this to display in the Heat Map.

[jcf2]

Another cross-check for the scan counts is the Completions tab, which currently (8/26/24 with "15 day" selection) has

Those look reasonable and they suggest that the number of scans for UC should be 6 x 6 = 36 (vs. 12 shown), UI should be 1 x 6 = 6 (as shown), and UM should be 6 + 3 + 4 = 13 (vs. 3 shown).

[mepearson]

With 15 day scan, we are pulling in some images that don't yet have qc records.

[jcf2]

I see -- so they are being filtered, intentionally or not, if they have no quality rating? I would say that we would want to include all scans, regardless of rating (or lack of rating). That's particularly true here, since we want the complete count, but I think it probably should apply in general.

[mepearson]

We are filtering on subject id based on acquisition week. so imaging records are 1 per subject visit. We then merge that data with the ratings data, where there are 0-6 records (1 per scan) for each subject / visit.

One option is to create a record for every subject, and if they don't have an actual record we create a 'no rating' (or 'no data') for that particular scan.

[mepearson]

We also don't break the bar chart apart by scan type, though could if desired.

[jcf2]

Yes, that seems like a good solution -- if no ratings entry exists, then instead of omitting, use a default type value (n/a) and keep the imaging record. This could be a general solution everywhere, since we don't want to hide/miss those scans, and thus might perhaps be done once as part of importing the ratings table, if that makes it simpler? Incidentally, it would be useful to list such cases on Discrepancies, e.g. as "Scans with no quality rating" (I can add this to #45 as a reminder).

I like the thought of splitting the bar chart by scan type, too... I can set up a new issue as a reminder of this (#47)

[mepearson]

Solution to this issue: add missing rating records to data ETL

[mepearson]

Have added the missing records and a toggle to the bar chart to determine whether or not to keep the unavailable data.

also fixed the bugs that cause it to break if the data is filtered to no records.