Merge branch 'release/3.2.x' of https://github.com/GeminiDRSoftware/D…

…RAGONS into release/3.2.x

LICENSE

@@ -6,7 +6,7 @@ whether written by AURA or third parties.
 
 -----
 
-Copyright (C) 2014-2021 Association of Universities for Research in Astronomy, Inc. (AURA)
+Copyright (C) 2014-2024 Association of Universities for Research in Astronomy, Inc. (AURA)
 
 Redistribution and use in source and binary forms, with or without modification,
 are permitted provided that the following conditions are met:

README.md

@@ -11,18 +11,18 @@
 </table>
 
 # Current Status
-**The stable version is v3.1.0.**  
+**The stable version is v3.2.0.**  
 
 
-[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7776065.svg)](https://doi.org/10.5281/zenodo.7776065) DRAGONS is distributed as a conda package, *dragons*, and it is 
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10841622.svg)](https://doi.org/10.5281/zenodo.10841622) DRAGONS is distributed as a conda package, *dragons*, and it is 
 included in the conda *gemini* stack.
 
-Version 3.1 is recommend for the reduction of **imaging** data from Gemini's
-current facility instruments: GMOS, NIRI, Flamingos-2, and GSAOI, and for the
-reduction of GMOS longslit spectroscopy data.
+Version 3.2 is recommend for the reduction of **imaging** data from Gemini's
+current facility instruments: GMOS, NIRI, Flamingos-2, and GSAOI, for the
+reduction of GMOS longslit spectroscopy data, and the reduction of GHOST data.
 
 To reduce other types of Gemini spectroscopy data, please continue to use 
-the [Gemini IRAF package](https://gemini.edu/observing/phase-iii/reducing-data/gemini-iraf-data-reduction-software).
+the [Gemini IRAF package](https://www.gemini.edu/observing/phase-iii/reducing-data/gemini-iraf-data-reduction-software).
 
 To install DRAGONS:
 
@@ -38,7 +38,7 @@ $ conda config --add channels http://astroconda.gemini.edu/public
 ```
 
 
-A list of changes since 3.0 can be found in the [Change Logs](https://dragons.readthedocs.io/en/v3.1.0/changes.html).
+A list of changes since 3.1 can be found in the [Change Logs](https://dragons.readthedocs.io/en/v3.2.0/changes.html).
 
 ---
 # What is DRAGONS
@@ -51,9 +51,9 @@ reduction of Gemini data.
 ---
 
 # Documentation
-Documentation on DRAGONS v3.1 is available on "readthedocs" at:
+Documentation on DRAGONS v3.2 is available on "readthedocs" at:
 
-* https://dragons.readthedocs.io/en/v3.1.0/
+* https://dragons.readthedocs.io/en/v3.2.0/
 
 There your will find manuals for Astrodata and the Recipe System, and hands-on
 tutorials on reducing Gemini imaging data with DRAGONS.
@@ -69,12 +69,12 @@ manuals.
 # Setting up a development environment
 
 To run checkouts, first set up a development conda environment.  This is what
-we are using at this time for the `master` branch and the `release/3.1.x`, 
-`release/3.2.x` branches.
+we are using at this time for the `master` branch and the `release/3.2.x` 
+branches.
 
 ```
-$ conda create -n dgdev3.10_20240201 python=3.10 astropy=6 astroquery matplotlib numpy psutil pytest python-dateutil requests scikit-image scipy sextractor sqlalchemy ds9 gwcs specutils sphinx sphinx_rtd_theme bokeh holoviews cython future astroscrappy=1.1 fitsverify imexam
-$ conda activate dgdev3.10_20240201
+$ conda create -n dgdev3.10_20240401 python=3.10 astropy=6 astroquery matplotlib numpy psutil pytest python-dateutil requests scikit-image scipy sextractor sqlalchemy ds9 gwcs specutils sphinx sphinx_rtd_theme bokeh holoviews cython future astroscrappy=1.1 fitsverify imexam
+$ conda activate dgdev3.10_20240401
 $ pip install git+https://github.com/GeminiDRSoftware/GeminiObsDB.git@release/1.0.x
 $ pip install git+https://github.com/GeminiDRSoftware/GeminiCalMgr.git@release/1.1.x
 ```

astrodata/doc/.readthedocs.yaml

@@ -27,4 +27,6 @@ sphinx:
 python:
    install:
    - requirements: requirements.txt
-   - requirements: doc/requirements.txt
+   - requirements: doc/requirements.txt
+   - method: pip
+     path: .

astrodata/doc/cheatsheet.rst

@@ -13,7 +13,7 @@ Cheat Sheet
 A data package is available for download if you wish to run the examples
 included in this cheat sheet.  Download it at:
 
-    `<http://www.gemini.edu/sciops/data/software/datapkgs/ad_usermanual_datapkg-v1.tar>`_
+    `<https://www.gemini.edu/sciops/data/software/datapkgs/ad_usermanual_datapkg-v1.tar>`_
 
 To unpack::
 

doc/DRAGONS/changes.rst

@@ -16,10 +16,12 @@ New Features
 ------------
 
 Full support for the reduction of GHOST data.
-  This is based on the external GHOSTDR package, with improvements and
-  changes to the names and scope of primitives to better align with the
+  This is based on the external GHOSTDR package, with important improvements.
+  Includes changes to the names and scope of primitives to better align with the
   other instrument recipes.
 
+Support for new GMOS-S CCDs installed in late 2023.
+
 Improvements
 ------------
 **astrodata.wcs**
@@ -28,7 +30,16 @@ Improvements
 
 * Support for reading and writing tabular wavelength information to/from FITS.
 
-**geminidr**
+**astrodata.provenance**
+
+* Renamed the ``PROVHISTORY`` table to ``HISTORY``, and changed wording in the
+code from "provenance history" to simply "history".
+
+**astrodata.fits**
+
+* Support reading ASCII tables when opening FITS files with astrodata
+
+**geminidr.core**
 
 * Creation of new ``skip_primitive`` parameter, e.g.,
   ``reduce -p skyCorrectFromSlit:skip_primitive=True`` which allows any
@@ -40,51 +51,47 @@ Improvements
   ``reduce -p ADUToElectrons:write_outputs=True`` which will write to disk
   the outputs of the primitive.
 
+* Allow input files to ``shiftImages`` to recognize tabs or multiple
+  whitespaces as the delimiter
+
+**geminidr.gsaoi***
+
+* Modification to the `nostack` science recipe to not "store" the image but
+  rather continue and detect sources in the images in anticipation of the likely
+  stacking that will follow later.  The output images will have the
+  `_sourcesDetected` suffix rather than the `_image` suffix.
+
+**recipe_system.cal_service**
+
+* Whitespace now allowed in directory paths (if quoted), e.g.,
+  ``databases = "~/.my dragons/dragons.db"``
+
 
 Bug fixes
 ---------
-**geminidr**
+**geminidr.core**
 
 * Set default ``calculateSensitivity.bandpass`` parameter to 0.001 nm to
   better handle pure spectra in flux density units.
 
-* Allow ``display`` to handle non-standard extention names, which did not
+* Allow ``display`` to handle non-standard extension names, which did not
   work as intended.
 
+**geminidr.gmos**
+
+* Fix the QE model selection for the GMOS-S EEV CDDs.
 
 **recipe_system**
 
 * Set the ``engineering`` flag to False for all data stored in the local
   calibration database, to ensure that it can be retrieved.
 
+Compatibility
+-------------
+**geminidr.interactive**
 
-3.1.1
-=====
-
-Improvements
-------------
-
-**astrodata.provenance**
-
-* Renamed the ``PROVHISTORY`` table to ``HISTORY``, and changed wording in the
-code from "provenance history" to simply "history".
-
-**geminidr.core**
-
-* Allow input files to ``shiftImages`` to recognize tabs or multiple
-  whitespaces as the delimiter
-
-**recipe_system.cal_service**
-
-* Whitespace now allowed in directory paths (if quoted), e.g.,
-  ``databases = "~/.my dragons/dragons.db"``
-
-Bug Fixes
----------
-
-**geminidr.gmos**
+* The interactive tools are now compatible with and require bokeh v3 and above.
 
-* Fix the QE model selection for the GMOS-S EEV CDDs.
 
 
 3.1.0

doc/DRAGONS/conf.py

@@ -294,6 +294,7 @@ def setup(app):
 .. |GNIRSImgTut| replace:: :raw-html:`<a href="http://GNIRSImg-DRTutorial.readthedocs.io/en/{v}">GNIRS Keyhole Imaging Data Reduction Tutorial</a>`
 
 .. |GMOSLSTut| replace:: :raw-html:`<a href="http://GMOSLS-DRTutorial.readthedocs.io/en/{v}">GMOS Longslit Data Reduction Tutorial</a>`
+.. |GHOSTTut| replace:: :raw-html:`<a href="http://GHOST-DRTutorial.readthedocs.io/en/{v}">GHOST Data Reduction Tutorial</a>`
 
 .. |ADMaster| replace:: :raw-html:`<a href="http://astrodata.readthedocs.io/en/{v}">Astrodata Manual</a>`
 .. |ADCheat| replace:: :raw-html:`<a href="http://astrodata.readthedocs.io/en/{v}/cheatsheet.html">AstroData Cheat Sheet</a>`

doc/DRAGONS/dragonsteam.rst

@@ -10,13 +10,14 @@ Current DRAGONS Team
 ====================
 (Alphabetical order)
 
+* Dan Avner
 * Daniel Berke
 * Paul Hirst
-* David Jones
 * Kathleen Labrie
 * Chris Simpson
 * Olesja Smirnova
 * Monika Soraisam
+* D.J. Teal
 * James Turner
 * Bill Vacca
 

doc/DRAGONS/index.rst

@@ -34,8 +34,9 @@ Tutorials - Reducing data with DRAGONS
 * |GSAOIImgTut|
 * |GNIRSImgTut|
 
-**Longslit Spectroscopy**
+**Spectroscopy**
 
+* |GHOSTTut|
 * |GMOSLSTut|
 
 DRAGONS Manuals

doc/DRAGONS/releasenotes.rst

@@ -6,6 +6,37 @@
 Release Notes
 *************
 
+V3.2.0
+======
+
+This release adds:
+
+* Fully integrated support for GHOST
+* Support for the new GMOS-S CCDs that were installed in late 2023
+
+The release also includes a number of other improvements and bug fixes.
+See the :ref:`changes` for all the details.
+
+With this release, DRAGONS offers support for:
+
+Science Quality reduction
+   * GMOS imager
+   * NIRI imager
+   * GSAOI imager
+   * F2 imager
+   * GMOS longslit spectrograph (including nod-and-shuffle)
+   * GHOST spectrograph
+
+For imaging and GMOS longslit spectroscopy, this software should be used
+instead of the Gemini IRAF package.
+
+To install DRAGONS, please follow the installation guide provided in the
+Recipe System User Manual:
+
+  |RSUserInstall|.
+
+
+
 V3.1.0
 ======
 

gemini_instruments/f2/adclass.py

@@ -14,7 +14,6 @@ class AstroDataF2(AstroDataGemini):
     __keyword_dict = dict(central_wavelength='WAVELENG',
                           disperser='GRISM',
                           dispersion='DISPERSI',
-                          focal_plane_mask='MOSPOS',
                           lyot_stop='LYOT',
                           )
 
@@ -521,20 +520,40 @@ def group_id(self):
     @astro_data_descriptor
     def lyot_stop(self):
         """
-        Returns the LYOT filter used for the observation.  This works around
+        Returns the LYOT stop used for the observation.  This works around
         inconsistencies in the header keywords.
 
         Returns
         -------
         str
-            LYOT filter name, or None
+            LYOT stop name, or None
 
         """
         lyot = self.phu.get('LYOT', None)
         if lyot:
             return lyot
         return self.phu.get('LYOTPOS', None)
 
+    @astro_data_descriptor
+    def focal_plane_mask(self, stripID=False, pretty=False):
+        """
+        Returns the focal plane mask used for the observation. This is generally
+        the MASKNAME header. For imaging data, MASKNAME sometimes has strange
+        values, so we check MOSPOS and if MOSPOS is 'Open' we return that
+        for consistency.
+
+        The stripID and pretty arguments are ignored.
+
+        Returns
+        -------
+        str
+            focal plane mask name, or None
+
+        """
+        mospos = self.phu.get('MOSPOS', None)
+        maskname = self.phu.get('MASKNAME', None)
+        return mospos if mospos == 'Open' else maskname
+
     @returns_list
     @astro_data_descriptor
     def nominal_photometric_zeropoint(self):

gemini_instruments/test/lut_descriptors.py

@@ -158,6 +158,12 @@
         ('telescope_x_offset', -0.0324777684697),
         ('telescope_y_offset', 1.52706222735),
     ),
+    ('F2', 'S20150622S0008.fits'): (
+        ('focal_plane_mask', 'GS2011SQ600-02'),
+    ),
+    ('F2', 'S20130719S0463.fits'): (
+        ('focal_plane_mask', 'Open'),
+    ),
     ('GMOS', 'N20160605S0099.fits'): (
         ('airmass', 1.269),
         ('ao_seeing', None),
@@ -298,7 +304,7 @@
         ('nod_count', None),
         ('shuffle_pixels', None),
         ('nominal_atmospheric_extinction', 0.002299999999999991),
-        ('nominal_photometric_zeropoint', [None, None, None, None]),
+        ('nominal_photometric_zeropoint', [27.693869869509008, 27.68460698159726, 27.728983802095854, 27.707914081527598]),
         ('non_linear_level', [65535, 65535, 65535, 65535]),
         ('object', 'V4077 Sgr'),
         ('observation_class', 'acq'),

recipe_system/doc/rs_ProgManual/.readthedocs.yaml

@@ -27,4 +27,6 @@ sphinx:
 python:
    install:
    - requirements: requirements.txt
-   - requirements: doc/requirements.txt
+   - requirements: doc/requirements.txt
+   - method: pip
+     path: .

recipe_system/doc/rs_ProgManual/conf.py

@@ -435,4 +435,6 @@ def setup(app):
 .. |RSUser|  replace:: :raw-html:`<a href="http://dragons-recipe-system-users-manual.readthedocs.io/en/{v}/">Recipe System Users Manual</a>`
 .. |astrodataprog| replace:: :raw-html:`<a href="https://astrodata.readthedocs.io/en/{v}/progmanual/index.html">Astrodata Programmer Manual</a>`
 
+.. |DRAGONS| replace:: :raw-html:`<a href="http://dragons.readthedocs.io/en/{v}/">DRAGONS</a>`
+
 """.format(v = rtdurl)

recipe_system/doc/rs_ProgManual/index.rst

@@ -10,6 +10,8 @@ Recipe System Programmers Manual
 
    PIPE-USER-108_RSProgManual
 
+[Return to the main |DRAGONS| page.]
+
 .. toctree::
    :maxdepth: 2
    :caption: Table of Contents

recipe_system/doc/rs_UsersManual/conf.py

@@ -298,4 +298,6 @@ def setup(app):
 .. |astrodataprog| replace:: :raw-html:`<a href="https://astrodata.readthedocs.io/en/{v}/progmanual/index.html">Astrodata Programmer Manual</a>`
 .. |RSProg|  replace:: :raw-html:`<a href="http://dragons-recipe-system-programmers-manual.readthedocs.io/en/{v}/">Recipe System Programmers Manual</a>`
 
+.. |DRAGONS| replace:: :raw-html:`<a href="http://dragons.readthedocs.io/en/{v}/">DRAGONS</a>`
+
 """.format(v = rtdurl)

recipe_system/doc/rs_UsersManual/index.rst

@@ -11,6 +11,8 @@ Recipe System User Manual
 
    PIPE-USER-109_RSUserManual
 
+[Return to the main |DRAGONS| page.]
+
 .. toctree::
    :maxdepth: 2
    :caption: Table of Contents