Comment by Paul Goudfrooij on JIRA:

One way to implement the proposed option would be to let the user input a rectangular subarray of the first (reference) individual dithered image, then let the script translate that to a region in (RA, DEC) space using the WCS, then translate the latter into pixel subarrays of the other individual dithered images where the sky level is to be measured. (Rounding to integer pixels will be OK in the latter step, though interpolating using fractional edge pixels would be preferred.)

Comment by Misty Cracraft on JIRA:

There have been several questions about matching and background subtraction lately, so maybe this does warrant another discussion in the calwg. The current usage of this step is to find common background levels for matching between images in a mosiac, so it looks at the overlap region between individual images. So far as I understand, this is not about completely removing the true background of a single image, only to find what levels to subtract from multiple images to get them to match up in a mosaic using the automatic pipeline. The discussions have so far focused on what is needed to run in the automatic pipeline, not the full amount of parameters someone might want to run in an 'off-line' version.

Anton Koekemoer Do we need to set up a Calwg discussion to talk about all the background questions that have come up recently?

Comment by Anton Koekemoer on JIRA:

Just noting that general discussion relevant to background subtraction questions took place in JWST Cal WG meeting 2020-08-25

I'd be happy to discuss this specific ticket further in a subsequent Cal WG meeting, if necessary.

Comment by Anton Koekemoer on JIRA:

Added this particular ticket to the items for discussion at JWST Cal WG meeting 2020-09-29

Comment by Anton Koekemoer on JIRA:

This item was discussed at JWST Cal WG meeting 2020-09-29; setting the "calwg" priority label accordingly.

The following questions came up, which would be useful to have answers to in order for us to proceed, if Paul Goudfrooij can provide further clarification please:

• There was some lack of clarity about the drivers behind this ticket, since skymatch should, in principle at least, be able to accommodate regions with spatially variable emission.
• So, if skymatch is not working properly in these cases then this sounded more like a bug fix, since by design it should be able to accommodate this.
• On the other hand if skymatch is working correctly, and this is a request for providing more flexibility to run it offline, then that could potentially be addressed by exposing more parameters.

Comment by Paul Goudfrooij on JIRA:

Hi there, 

the reason why I issued the ticket is that at least at that time, skymatch did not return correct relative sky levels for the different dithers. The simulated images all had the same sky level, but skymatch returned different sky levels for the different dithers.  While when I ran iterative statistics on a specific region of the sky in the different dithers, I did get the same sky levels. If this is a bug in the sense that the algorithm should actually determine sky levels from the same sky regions (rather than the full footprints of the individual dithers), then yes it should be treated as a bug.

However, my thought was that perhaps, the default option of skymatch is to do the statistics on the full footprints of the individual dithers, in which case I was arguing that there should at least be an option to do the sky match on a specific, user-supplied region on the sky.

Comment by Anton Koekemoer on JIRA:

thanks for the clarification! As far as I understand it, at least if it follows the astrodrizzle approach, the implementation of skymatch uses overlapping footprints between pairs of images (starting with, eg, the pair with the largest overlap, then working down from there), and would match them based on the statistics in that overlapping footprint.

There might however still possibly be a bug even with that implementation (or the implementation might not be quite as described).

Maybe as a next step (since the pipeline has had many improvements recently), would you mind just trying your test again please, seeing how skymatch performs with the current version of the pipeline, and let us know here whether you think there might still be an issue or whether you think it looks like it's now resolved?

Comment by David Law on JIRA:

Paul Goudfrooij Just following up to see whether you think this is still relevant?

Comment by Paul Goudfrooij on JIRA:

Hi David Law - thanks for the prompt, I will look at this again after the HST deadline.

Comment by Paul Goudfrooij on JIRA:

I have now tried running skymatch with method = 'local', 'match', and 'global+match'. In most cases, it works very well. However, there is one dataset where I cannot get the sky levels to match without leaving residual sky level offsets between the different tiles/dithers, even when playing with parameters clip, lower, upper, lsigma, and usigma. I would strongly recommend introducing an option to let users input their own sky levels for each input image. That could be through a header keyword (say 'SKY_USER') in the input images, or by inputting the sky levels in a list file similar to that already used in the 'use_custom_catalogs' and 'catfile' options of tweakreg.

EDIT: I just tried an alternative method for this problematic dataset, namely to first do an off-line sky level adjustment to get all input images at the same "real" background level, followed by setting skymatch.skip = True in the call to Image3Pipeline. That worked much better than anything I tried with the current optional parameters for skymatch, so there is a workaround to address this issue, albeit one that requires more work for the user.

Comment by David Law on JIRA:

Seems reasonable to me to allow users to pass in a custom set of values to subtract, calculated in some offline manner that might be arbitrarily complicated for a given data set.  The simplest way of doing this might be to add a skymethod='user', and then read values from an input list that must match the input list of FITS files.

Comment by Ned Molter on JIRA:

Hi all, I put together a basic implementation of this: #9053 

Would you please check if this works for your desired use-cases?

Paul and David, I also requested your reviews on the PR, any feedback is much appreciated!

Comment by Mihai Cara on JIRA:

Paul Goudfrooij

Here is an example of the same algorithm (as implemented in stsci.skypac): https://github.com/spacetelescope/drizzlepac-notebooks/blob/master/notebooks/sky_matching/labeled_local_globalmatch_match.gif That image of the Horsehead nebula was obtained from input images with very little overlap (10-20% maybe) and with a complex background in the overlap regions. I do not know what set you had for which this didn't work, but if JWST images had low incidence of cosmic rays, I would suggest turning off clip, lower, upper, lsigma, and usigma altogether. These are helpful for mostly uniform backgrounds since one can clip away stars, cosmic rays, etc. For complex backgrounds and few CRs, I would also suggest using skystat set to "mean" or maybe median/midpt.