nirspec ifu data fails to load in cubeviz with MJD errors

[Jdaviz-Triage-Bot]

Reporter: Brian Cherinka

Error loading cal_ver 1.2 NRS_IFU data into Cubeviz.  Gives the following error:

Input values did not match the format class mjd: TypeError: Input values for mjd class must be finite doubles"

Steps to reproduce:

1. Go to https://masttest.stsci.edu/viz/jwst/spectra?&uri=mast:JWST/product/jw00626-o054_t017_nirspec_g140h-f100lp_s3d.fits
2. Error occurs during the Cubeviz.load_data step

Blocked by:

• SpectralCube to specutils transition in cubeviz #547

🐱

---

DISCLAIMER: This issue was autocreated by the Jdaviz Issue Creation Bot on behalf of the reporter. If any information is incorrect, please contact Duy Nguyen

[havok2063]

Pasting example of full traceback

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/time/core.py in _get_time_fmt(self, val, val2, format, scale, precision, in_subfmt, out_subfmt)
    429             try:
--> 430                 return cls(val, val2, scale, precision, in_subfmt, out_subfmt)
    431             except UnitConversionError:

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/time/formats.py in __init__(self, val1, val2, scale, precision, in_subfmt, out_subfmt, from_jd)
    148         else:
--> 149             val1, val2 = self._check_val_type(val1, val2)
    150             self.set_jds(val1, val2)

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/time/formats.py in _check_val_type(self, val1, val2)
    424         if val1.dtype.kind == 'f':
--> 425             val1, val2 = super()._check_val_type(val1, val2)
    426         elif (not orig_val2_is_none

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/time/formats.py in _check_val_type(self, val1, val2)
    262         if not (ok1 and ok2):
--> 263             raise TypeError('Input values for {} class must be finite doubles'
    264                             .format(self.name))

TypeError: Input values for mjd class must be finite doubles

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-21-b27e462e12cf> in <module>
----> 1 c.load_data(s3d[1])

~/Work/git/havok2063/jdaviz/jdaviz/core/helpers.py in load_data(self, data, parser_reference, **kwargs)
     39 
     40     def load_data(self, data, parser_reference=None, **kwargs):
---> 41         self.app.load_data(data, parser_reference=parser_reference, **kwargs)
     42 
     43     @property

~/Work/git/havok2063/jdaviz/jdaviz/app.py in load_data(self, file_obj, parser_reference, **kwargs)
    304                 # If the parser returns something other than known, assume it's
    305                 #  a message we want to make the user aware of.
--> 306                 msg = parser(self, file_obj, **kwargs)
    307 
    308                 if msg is not None:

~/Work/git/havok2063/jdaviz/jdaviz/configs/cubeviz/plugins/parsers.py in parse_data(app, file_obj, data_type, data_label)
     50 
     51         with fits.open(file_obj) as hdulist:
---> 52             _parse_hdu(app, hdulist, file_name=data_label or file_name)
     53 
     54     # If the data types are custom data objects, use explicit parsers. Note

~/Work/git/havok2063/jdaviz/jdaviz/configs/cubeviz/plugins/parsers.py in _parse_hdu(app, hdulist, file_name)
    122         if any(x in hdu.name.lower() for x in EXT_TYPES['flux']):
    123             app.add_data_to_viewer('flux-viewer', data_label)
--> 124             app.add_data_to_viewer('spectrum-viewer', data_label)
    125 
    126 

~/Work/git/havok2063/jdaviz/jdaviz/app.py in add_data_to_viewer(self, viewer_reference, data_path, clear_other_data, ext)
    664         if data_id is not None:
    665             data_ids.append(data_id)
--> 666             self._update_selected_data_items(viewer_item['id'], data_ids)
    667         else:
    668             raise ValueError(

~/Work/git/havok2063/jdaviz/jdaviz/app.py in _update_selected_data_items(self, viewer_id, selected_items)
    979                                               viewer_id=viewer_id,
    980                                               sender=self)
--> 981             self.hub.broadcast(add_data_message)
    982 
    983         # Remove any deselected data objects from viewer

~/anaconda3/envs/havokve/lib/python3.8/site-packages/glue/core/hub.py in broadcast(self, message)
    213             logging.getLogger(__name__).info("Broadcasting %s", message)
    214             for subscriber, handler in self._find_handlers(message):
--> 215                 handler(message)
    216 
    217     def __getstate__(self):

~/Work/git/havok2063/jdaviz/jdaviz/configs/specviz/plugins/unit_conversion/unit_conversion.py in _on_viewer_data_changed(self, msg)
     75             return
     76 
---> 77         self._viewer_data = self.app.get_data_from_viewer('spectrum-viewer')
     78 
     79         self.dc_items = [data.label

~/Work/git/havok2063/jdaviz/jdaviz/app.py in get_data_from_viewer(self, viewer_reference, data_label, cls, include_subsets)
    424                         #  collapsed via the defined statistic
    425                         if cls == Spectrum1D:
--> 426                             layer_data = layer_data.get_object(cls=cls,
    427                                                                statistic=statistic)
    428                         else:

~/anaconda3/envs/havokve/lib/python3.8/site-packages/glue/core/data.py in get_object(self, cls, **kwargs)
    287         handler, _ = data_translator.get_handler_for(cls)
    288 
--> 289         return handler.to_object(self, **kwargs)
    290 
    291     @property

~/anaconda3/envs/havokve/lib/python3.8/site-packages/glue_astronomy/translators/spectrum1d.py in to_object(self, data_or_subset, attribute, statistic)
    145             [attribute] if not hasattr(attribute, '__len__') else attribute)
    146 
--> 147         return Spectrum1D(**data_kwargs, **kwargs)

~/Work/git/havok2063/specutils/specutils/spectra/spectrum1d.py in __init__(self, flux, spectral_axis, wcs, velocity_convention, rest_value, redshift, radial_velocity, bin_specification, **kwargs)
    284                     spec_axis = self.wcs.pixel_to_world(np.arange(self.flux.shape[-1]))
    285                 else:
--> 286                     spec_axis = self.wcs.spectral.pixel_to_world(np.arange(self.flux.shape[-1]))
    287             else:
    288                 spec_axis = self.wcs.pixel_to_world(np.arange(self.flux.shape[-1]))

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/wcs/wcsapi/high_level_api.py in pixel_to_world(self, *pixel_arrays)
    243 
    244         # Cache the classes and components since this may be expensive
--> 245         components = self.low_level_wcs.world_axis_object_components
    246         classes = self.low_level_wcs.world_axis_object_classes
    247 

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/wcs/wcsapi/fitswcs.py in world_axis_object_components(self)
    320     @property
    321     def world_axis_object_components(self):
--> 322         return self._get_components_and_classes()[0]
    323 
    324     @property

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/wcs/wcsapi/fitswcs.py in _get_components_and_classes(self)
    412 
    413                 earth_location = EarthLocation(*self.wcs.obsgeo[:3], unit=u.m)
--> 414                 obstime = Time(self.wcs.mjdobs, format='mjd', scale='utc',
    415                                location=earth_location)
    416                 observer_location = SkyCoord(earth_location.get_itrs(obstime=obstime))

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/time/core.py in __init__(self, val, val2, format, scale, precision, in_subfmt, out_subfmt, location, copy)
   1524                 self._set_scale(scale)
   1525         else:
-> 1526             self._init_from_vals(val, val2, format, scale, copy,
   1527                                  precision, in_subfmt, out_subfmt)
   1528             self.SCALES = TIME_TYPES[self.scale]

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/time/core.py in _init_from_vals(self, val, val2, format, scale, copy, precision, in_subfmt, out_subfmt)
    370 
    371         # Parse / convert input values into internal jd1, jd2 based on format
--> 372         self._time = self._get_time_fmt(val, val2, format, scale,
    373                                         precision, in_subfmt, out_subfmt)
    374         self._format = self._time.name

~/anaconda3/envs/havokve/lib/python3.8/site-packages/astropy/time/core.py in _get_time_fmt(self, val, val2, format, scale, precision, in_subfmt, out_subfmt)
    436                 # easier for user to see what is wrong.
    437                 if len(formats) == 1:
--> 438                     raise ValueError(
    439                         f'Input values did not match the format class {format}:'
    440                         + os.linesep

ValueError: Input values did not match the format class mjd:
TypeError: Input values for mjd class must be finite doubles

[pllim]

How do you download that FITS file with direct URL? I don't want to bother with keyring and stuff.

[pllim]

The error comes straight out of astropy. I fear WCS is invalid in some way but there is no way to check if we cannot get to the file.

[havok2063]

Either of the s3d files located in the shared box folder will produce the error. I'm not sure what the policy is for sharing box links on github, so I put in the slack instead.

[pllim]

I see these warnings:

WARNING: No spectral axis found; header may be non-compliant. [spectral_cube.cube_utils]
WARNING:astropy:No spectral axis found; header may be non-compliant.

Is there a way to check if this file is compliant according to what we are supposed to support? I'll have to defer to @rosteen or @javerbukh here.

Not sure how relevant, but this is what is being passed into the glue-astronomy translator at https://github.com/glue-viz/glue-astronomy/blob/4c5e6e78bcaa20d1723510f06bbe3070f2b1642a/glue_astronomy/translators/spectrum1d.py#L147 :

.../glue_astronomy/translators/spectrum1d.py in to_object(self, data_or_subset, attribute, statistic)
    145             [attribute] if not hasattr(attribute, '__len__') else attribute)
    146 
--> 147         return Spectrum1D(**data_kwargs, **kwargs)

data_kwargs = {'flux': <Quantity [1.64488238e+14, 1.35582303e+14, 1.35576540e+14, 1.35446969e+14,
                                  1.32581253e+14, 1.43669877e+14, 1.43669877e+14, 1.43669877e+14, ...
                                  2.01576405e+14, 2.02250882e+14] MJy / sr>,
               'mask': None}
kwargs = {'wcs': ...}

where wcs is:

WCS Keywords

Number of WCS axes: 1
CTYPE : 'WAVE'
CRVAL : 5.77931382762185e-07
CRPIX : 1.0
PC1_1  : 1.0
CDELT : 4.999999888241291e-09
NAXIS : 950}

[ibusko]

@pllim is correct: the error comes from astropy, but because the file's primary header is missing a MJD-OBS keyword. The parser sets it's value to an empty string, which in turn gets converted to a NaN. That generates the reported exception.

Note that the ultimate cause is the use of SpectralCube to do the parsing. SpectralCube resorts to astropy to build the WCS instance. This likely will be fixed with the migration away from SpectralCube (the file reads correctly with Spectrum1D).

[pllim]

Reposting from JIRA -- A lot of discussion on Slack but my version of the event in tl;dr form is as follows; You can do one of the following:

1. Insert MJD-OBS into the FITS header so astropy.wcs can find it so Spectrum1D can read in the cube into Cubeviz. This can happen in pipeline level or special handling by specutils by converting DATE-OBS or DATE-BEG (or whatever you deem the equivalent) into MJD-OBS during parsing.
2. Refactor Cubeviz parser in Jdaviz to handle ASDF/GWCS and forget about the FITS WCS problem. (RECOMMENDED)

---

This is how you can load this particular file into Cubeviz by grabbing its ASDF/GWCS. This code will need to be worked into the parser natively. But before the good stuff, we need this patch in Jdaviz to work around #588 or you will get traceback from the Collapse plugin on load (unrelated to this issue):

--- a/jdaviz/configs/default/plugins/collapse/collapse.py
+++ b/jdaviz/configs/default/plugins/collapse/collapse.py
@@ -68,7 +68,7 @@ class Collapse(TemplateMixin):
                                     if x.label == event['new']))

         # Also set the spectral min and max to default to the full range
-        cube = self._selected_data.get_object(cls=SpectralCube)
+        cube = self._selected_data.get_object(cls=Spectrum1D)
         self.selected_subset = "None"
         self.spectral_min = cube.spectral_axis[0].value
         self.spectral_max = cube.spectral_axis[-1].value

The good stuff you can copy and paste into a notebook to see for yourself (using Jdaviz with the above patch):

from asdf.fits_embed import AsdfInFits
from glue.core import Component, Data

from jdaviz import Cubeviz

cubeviz = Cubeviz(verbosity='warning')
cubeviz.app

filename = 'jw00619-o094_t001_miri_ch1-long_s3d.fits'
data_label = 'my_cube'

with AsdfInFits.open(filename) as af:
    bunit = af.tree['meta']['bunit_data']
    data_arr = af.tree['data']
    w = af.tree['meta']['wcs']

    data = Data(label=data_label)
    data.coords = w
    comp = Component.autotyped(data_arr, units=bunit)
    data.add_component(component=comp, label='data')

    cubeviz.app.add_data(data, data_label)
    cubeviz.app.add_data_to_viewer('flux-viewer', data_label)

[pllim]

Alas, Pure GWCS support is blocked by glue-viz/glue-astronomy#59 !