Move pixel_replace step

[stscijgbot-jp]

Issue JP-3566 was created on JIRA by David Law:

At present, the pixel_replace step happens in spec2, after photometric calibration and prior to writing out the cal.fits files.  There are two downsides to this: First, any new outliers detected in spec3 won't be able to run through pixel replacement prior to resampling/cubebuild into the final data products.  Second, any users wishing to try running with/without pixel replacement need to run both the spec2 and spec3 pipeline.

We should therefore move the pixel_replace step to immediately before resample/cube_build, in both spec2 and spec3 pipelines.  This would happen after producing the cal files, so that the cal files for a given exposure do not contain any replaced values.  If turned on for spec2, the s2d/s3d files would have thus been pixel-replaced prior to resampling.  If turned on for spec3, the s2d/s3d files would likewise have been pixel-replaced prior to resampling.  Essentially, while pixel_replace is a discrete step it can be thought of as an option to resample/cubebuild, either replacing just prior to resampling or not depending how the param ref files are set up.

Thoughts?  Tyler Pauly Greg Sloan Melanie Clarke 

[stscijgbot-jp]

Comment by Greg Sloan on JIRA:

I agree with the reasoning. We should move pixel_replace so that the cal.fits files are saved before pixels are replaced.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

I agree as well - I think this is a good first step toward making missing pixel handling more robust.  This would save interpolation until the last minute possible with the current implementation, and make it much easier to generate products with and without pixel replacement for comparison.

For more future improvements, we might want to consider moving the interpolation even further down in the processing, during spectral extraction and/or resampling.  It might be possible to make a more robust spatial profile to fit, for example, if all input data is available to the algorithm, or if a PSF model is provided for optimal extraction.

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

Howard Bushouse or Greg Sloan I am updating the regression tests. I updated test_miri_lrs_slit_spec2.py to do the pixel_replacement and not skip it - since that is how the default pipeline is run.
I want to make sure the s2d results are the expected results. When a pixel is a nan in the input to resample it seems that the resulting s2d pixel value is 0 . The CON plane has a value of 0 in this case. I wanted to make sure that to be consistent with other products these pixels should be a NAN. This would be a nice JP-ticket but I thought since I saw this I would ask.

[stscijgbot-jp]

Comment by Greg Sloan on JIRA:

Seems like NaNs should stay NaNs. Zeroes can be interpreted as valid data. But I want to be clear on what is happening. Is pixel_replace writing "0", or is resample_spec doing that?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Jane Morrison Greg Sloan The issue of zero vs. NaN in resampled images is the subject of the long-standing ticket JP-3281, which was just discussed at the JP coordination meeting yesterday. They come out zero in resampled images because of the setting of the FILLVAL=INDEF parameter in resample. Once the default gets changed for that to NaN, then resampled pixels that have no contribution from inputs will be set to NaN.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

will it still be possible to run the pixel_replace step on a level 2a exposure when its default place in the pipeline is in Stage 3? I have used the step to clean up data for troubleshooting/helpdesk work, it would be nice (but not critically important) to still be able to do that.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Sarah Kendrew Yes, it's getting moved and copied, such that it'll be available in both spec2 and spec3 pipelines, right before resampling is done. That way it can be applied to individual exposures in spec2, as well as the combined data in spec3.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #8409

 

The pixel_replace step is now called from the Spec2 pipeline after the cal/calints files have been created, but before the resample_spec/cube_build steps are called, so it will affect s2d and s3d products. It is also now called from the Spec3 pipeline right before calling the resample_spec/cube_build steps, so it will not affect crf/crfints products, but will affect combined s2d/s3d products. It is also now called from the TSO3 pipeline right before calling the extract_1d step, so it will affect x1dints products.

[stscijgbot-jp]

Comment by David Law on JIRA:

LG2M.  Tested that pix replace can now run in both spec2 and spec3, does so in the expected locations (e.g., doesn't affect cal files), and can write out files if told to do so.  Also tested that the spec3 version is successfully replacing both previously-known bad pixel and those identified by the spec3 outlier detection routine.  Resulting spectra look as expected (pixreplace.png attached), being mostly identical the results pre-move, but now also fixing a few small issues from the outlier detection step not previously fixed.

[stscijgbot-jp]

Comment by David Law on JIRA:

Test above was for MIRI MRS, also tested for MIRI LRS (which I think are the only two modes using pixel_replace currently?).  LRS had one slight issue due to https://jira.stsci.edu/browse/JP-3447, but using exptime weighting avoids this and successfully shows that the pixel_replace change is working.