Detect and optionally fix <=0.1 fluorescence values (fl?_max)

[sdespina]

Events are missing when plotting fluorescent values using the logarithmic scale option.
Whenever the FL value of an event is negative, the event is completely omitted from the output graph. Results are therefore under-represented in the graphs

[paulmueller]

The general issue is described and tracked in DC-analysis/dclab#101. The task here would be to implement the UI that allows users to remove <=0.1 ==0.1 values from fluorescence max data.

Tasks:

• increment the dclab version in setup.py once Create a new check for negative fluorescence values dclab#101 is part of a release
• verify that the fl?_max checks show up when running the checks
• add a checkbox (which defaults to true) for setting <=0 values to 1 (or add an offset depending on how the discussion in Create a new check for negative fluorescence values dclab#101 goes)
• add a checkbox (which defaults to true) for "reparing" the fluorescence traces (see Create a new check for negative fluorescence values dclab#101 (comment))
  1. Determine the global offset by fitting a Poisson distribution or taking a median or whatever from all traces (see next comments)
  2. Determine the global offset by taking the traces for the 100 lowest peaks (or all peaks with 0.1) and computing their minimum.
  3. Only correct offsets that are <0 (as per @chrherold's comment in Create a new check for negative fluorescence values dclab#101 (comment)). Do not edit trace data (see comments below)
  4. Add this offset to all the traces
  5. Write the offset as an additional meta data keyword ["fluorescence"]: baseline ? offset (which first has to be introduced in dclab)
  6. Recompute fl_max for all events events that are 0.1 (we can just extract them, because we have the fl_pos feature) by adding the known offset to the trace data at that position.

[paulmueller]

Comments for @maxschloegel:

• Locate main.ui and add the checkbox somewhere on the right hand side. The checkbox should have a toolTip that explains what it does (you can edit the toolTip in designer).
• The user should be notified before conversion/compression if/which datasets will receive a repair for the fl?_max issue.
• You do not have to expect the fl?_max_ctc feature, only the fl?_max feature. This is because the cross-talk-corrected (ctc) features are usually created within dclab/Shape-Out2 and are not part of any raw data files. DCKit is for raw data files only.
• You will have to install dclab in editable mode (pip install -e path/to/root/of/dclab), because Create a new check for negative fluorescence values dclab#101 is not part of any release yet.
• DCKit always writes additional logs to the produced .rtdc files. The log files should have an entry stating by how much the baseline of each trace was shifted.
• Regarding (i) from above: You will probably have to look at example data to get an idea how the fluorescence data is usually distributed. I think something like a median should work for finding the baseline in the fluorescence trace data. (see comment below)
• Regarding (ii) from above: If there are still fluorescence values <0 when the baseline is >0, maybe it would still make sense to lift the baseline to a higher value. We would have to find out how the new version of Shape-In determines this baseline. (see comment below)

[paulmueller]

I just realized that the baseline should actually always be around zero. (see comment below) This setting is adjusted during RT-DC device maintenance for each sensor. This means we could still check for a non-zero baseline, but we should not modify any trace data, only issue a warning (maybe?). (see below)

[paulmueller]

That also means that the original issue (negative fl_max values) cannot be solved here, because those are absolutely expected for a basline at zero. This issue is now only about fl_max values that were set to 0.1. The other issue (data points cropped in log view) is actually just a visualization issue (see DC-analysis/dclab#56).

[paulmueller]

OK, now I got everything together.

1. "Baseline" is defined as the value in the trace from which the peak max is measured. This is always "0"!
2. "Offset" is defined as the lowest value in all traces, which should be set to zero. Sometimes this is not the case and we want to fix this here.
3. Negative peak maxima are the result of a faulty offset.
4. Actually, none of the fluorescence values (all of them, not only the maximum) should be positive (by definition).
5. We are not running into crosstalk-correction issues, because the offset is really small compared to actual fluorescence signals.

When we have determined the correct offset, we don't manipulate the trace data, but we write the offset to the meta data. We do however update the fl?_max features. We do not update the fl?_width features (the peaks are usually not triangular but rectangular, so we don't really make a mistake).

[maxschloegel]

@paulmueller I am currently at the part where I adjust the fl?_max values using the offset. Currently there is not a well-defined way to do this, right? RTDC_HDF5 does not support item assignment and once you suggested to use the workaround by converting to a dictionary -> change values -> convert back to HDF5. I could somehow implement this in DCKit, but it feels more appropriate to implement it in dclab, namely in the initialization-function of RTDC_Dict.
What is your opinion on that?

[paulmueller]

You are correct, there is no well-defined way of doing this. I don't like the idea that people can just change features of an RTDCBase, because this can make things very confusing.

The problem with RTDC_Dict Is that you would have to load the entire dataset into memory (Unless you base the feature dictionary on an open HDF5 file, which could work).

I would suggest the following: After compression/conversion, open the dataset with h5py and override the fl?_max features. There is only one problem with this approach: The new .hdf5 file will be larger, because the old fl?_max data will still be present (although not reachable). This could be resolved by repacking the dataset. I think this is the best solution, because it requires the least amount of lines of code. The user has to wait a little longer, but I think that is acceptable.

What do you think?

[maxschloegel]

Thanks for your input!
I will implement it like you suggested.
In the end I will run it for a normal dataset to see how long it takes.