Skip to content

Commit

Permalink
update outputs section and advanced features
Browse files Browse the repository at this point in the history
  • Loading branch information
@mschwamb
mschwamb committed Jan 9, 2025
1 parent 174d3e6 commit 7cf0e3f
Show file tree
Hide file tree
Showing 2 changed files with 78 additions and 59 deletions.
25 changes: 22 additions & 3 deletions docs/advanced.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ To implement the magnitude limit (remove detections of objects fainter than 22 m
Only one of these filters may be implemented at once.


Specifying Alernative Versions of the Auxiliaryy Files Used in the Ephemeris Generator
Specifying Alernative Versions of the Auxiliary Files Used in the Ephemeris Generator
-----------------------------------------------------------------------------------------

For backwards compability and to enable new version of the files to be run as well, users can override the default filenames and download locations of the :ref:`auxiliary files<auxfiles>` used by ``Sorcha``'s bult-in :ref:`ephemeris generator<ephemeris_gen>`. These :ref:`configs`:: variables are added to a new auxiliary ([AUXILIARY]) section::
Expand Down Expand Up @@ -147,8 +147,27 @@ For backwards compability and to enable new version of the files to be run as we
Advanced Output Options
-----------------------------------

We recommend that you do not change the decimal place precision and instead leave ``Sorcha`` to output the full value
to machine precision, but there may be reasons why you need to reduce the size of the output.
Custom Outputs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By setting the value of the output_columns configuration file keyword to a comma-separated list of column names, you may
specify your own custom output, using this page as a reference for potential column names.

For example, you could state this in your configuration file to get the object ID, position and magnitude only::

[OUTPUT]
output_columns = ObjID,RA_deg,Dec_deg,trailedSourceMag

.. warning::
If you are choosing to specify the column names in this way, please perform a quick test-run first to ensure your column names are correct before
embarking on any long runs. As we allow for user-written code and add-ons to add new column names, we do not error-handle the column names until
late in the code, upon output.


Specifying the Decimal Precision for the Photometric and Astromeitc Values
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By default, no rounding is performed on any of the output values. We recommend that you do not change the decimal place precision and instead leave ``Sorcha`` to output the full value to machine precision, but there may be reasons why you need to reduce the size of the output.

In the [OUTPUT] section of the :ref:`configs`, you can set the decimal precision for the astrometry outputs::

Expand Down
112 changes: 56 additions & 56 deletions docs/outputs.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,33 +3,17 @@
Outputs
==================

Sorcha Output
----------------------
.. attention::
Use the **-o** flag on the command line to specify where ``Sorcha`` should be saving any output and log files (the file path).

``Sorcha`` produces an output file describing each predicted observation the survey will record of the input simulated objects,
with a row for each predicted detection and a column for each parameter to be calculated. This output file can be in several formats
set by the output_format configuration file keyword.

Additionally, the output columns can be set to either "basic" or "all" settings (described below) using the output_columns configuration file keyword.
Alternatively, you may specify the columns you wish to be output.
.. attention::
Use the **-t** flag on the command line to specify the filename stem for all the ``Sorcha`` output files and logs.

The format of any output from ``Sorcha`` will look something like::

ObjID,fieldMJD_TAI,fieldRA_deg,fieldDec_deg,RA_deg,Dec_deg,astrometricSigma_deg,optFilter,PSFMag,trailedSourceMag,PSFMagSigma,trailedSourceMagSigma,fiveSigmaDepth_mag,fiveSigmaDepthAtSource
S1000000a,61769.320619,163.87542090842982,-18.84327137012991,164.03771300000017,-17.58257500000004,2.9880927198448093e-06,r,19.667095021023798,19.655534004675797,0.006775654132479691,0.006755926588113991,23.86356436464961,23.839403736057715
S1000000a,61769.332335,163.87542090842982,-18.84327137012991,164.03840499999956,-17.583782000000177,3.0580983448792015e-06,i,19.654439857054346,19.651499866857677,0.008648382870172588,0.00861644095296432,23.50948086026021,23.485408367730255
S1000000a,61773.283672,163.33185289781585,-17.478349047859123,164.25272700000096,-17.970833000000166,2.8628267283501646e-06,g,19.605094385361397,19.59913996244041,0.004573058990569846,0.004562676340629368,24.412081324532746,24.40274105573913
S1000000a,61773.304607,163.33185289781585,-17.478349047859123,164.2535509999998,-17.972800999999485,2.8619239276501636e-06,r,19.60417845127433,19.610463241887746,0.005414938113316873,0.005396964439230442,24.142184414583568,24.132798535794453
S1000000a,61780.286672,163.70205228035468,-18.10471138055092,164.4364500000006,-18.561287999999216,3.106487369364405e-06,i,19.50224387218658,19.49961057650898,0.00996299590797273,0.009945212307287087,23.1343489868631,23.13059981155987
S1000000a,61780.310927,163.70205228035468,-18.10471138055092,164.4365160000002,-18.56311500000129,3.0899264531165437e-06,z,19.506070321795203,19.506622970072044,0.01126449135209172,0.011237007559280756,22.968207967454678,22.964441345175853
S1000000a,61781.239134,163.95033588103914,-18.031113105727716,164.44201499999986,-18.63119400000105,3.2223774034283947e-06,i,19.50028114807821,19.494448387335947,0.01214406799779637,0.01212132996202541,22.85013563621249,22.84858482288965
S1000000a,61781.263141,163.95033588103914,-18.031113105727716,164.4419770000004,-18.63294700000159,3.042088583360277e-06,z,19.486562767073988,19.47832341807803,0.011723502868190884,0.011688663662533069,22.899894717824814,22.898283896399494
S1000000a,61789.27659,164.99043640246796,-19.09523631317997,164.29665099999988,-19.110176000000447,2.8895553381860802e-06,z,19.376978135088684,19.359651855968583,0.008079363622311368,0.00805998568672928,23.293210067462763,23.293123719813384
Output Formats
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The configuration file keyword output_format allows ``Sorcha`` to output files in CSV, SQLite3 or HDF5 formats. For example::
Output File Formats
----------------------------
The :ref:`configuration file<configs>` keyword output_format in the OUTPUT section allows ``Sorcha`` to output files in CSV, SQLite3 or HDF5 formats. For example::

[OUTPUT]
# The options: csv, sqlite3, hdf5
Expand All @@ -43,26 +27,27 @@ The configuration file keyword output_format allows ``Sorcha`` to output files i
with a number (due to a limitation in PyTables).


Output Rounding
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By default, no rounding is performed on any of the output values. If you wish to round
output values, this can be done separately for magnitude and position values using the following
configuration file keywords::
Detections File
----------------------

[OUTPUT]
position_decimals = 7
magnitude_decimals = 3
``Sorcha`` produces a detections file describing each predicted survey detection of the input small body populations,
with a row for each predicted detection and a column for each parameter calculated.


Additionally, the output columns of the detections file can be set to either "basic" or "all" settings (described below) using the output_columns :ref:`configuration file<configs>` keyword.

.. _basic::

Basic Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The "basic" output includes the columns most relevant to general photometry and detection purposes. This is declared
in the configuration file like so::
in the :ref:`configuration file<configs>` like so::

[OUTPUT]
output_columns = basic

The column names, formats, and descriptions are as follows:
Detections File: Basic Output Column Names, Formats, and Descriptions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

+------------------------------------+--------------+----------------------------------------------------------------------------------+
| Keyword | Format | Description |
Expand Down Expand Up @@ -108,16 +93,40 @@ The column names, formats, and descriptions are as follows:
The object_linked column only appears if the :ref:`linking filter<linking>` is on and the user has requested that observations of unlinked objects should not be dropped.


All Output
.. warning::
If you are writing to a HDF5 file that you plan to access using the PyTables library, note that your object IDs cannot begin
with a number (due to a limitation in PyTables).


Example Detections File in Basic Format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block::
ObjID,fieldMJD_TAI,fieldRA_deg,fieldDec_deg,RA_deg,Dec_deg,astrometricSigma_deg,optFilter,PSFMag,trailedSourceMag,PSFMagSigma,trailedSourceMagSigma,fiveSigmaDepth_mag,fiveSigmaDepthAtSource
S1000000a,61769.320619,163.87542090842982,-18.84327137012991,164.03771300000017,-17.58257500000004,2.9880927198448093e-06,r,19.667095021023798,19.655534004675797,0.006775654132479691,0.006755926588113991,23.86356436464961,23.839403736057715
S1000000a,61769.332335,163.87542090842982,-18.84327137012991,164.03840499999956,-17.583782000000177,3.0580983448792015e-06,i,19.654439857054346,19.651499866857677,0.008648382870172588,0.00861644095296432,23.50948086026021,23.485408367730255
S1000000a,61773.283672,163.33185289781585,-17.478349047859123,164.25272700000096,-17.970833000000166,2.8628267283501646e-06,g,19.605094385361397,19.59913996244041,0.004573058990569846,0.004562676340629368,24.412081324532746,24.40274105573913
S1000000a,61773.304607,163.33185289781585,-17.478349047859123,164.2535509999998,-17.972800999999485,2.8619239276501636e-06,r,19.60417845127433,19.610463241887746,0.005414938113316873,0.005396964439230442,24.142184414583568,24.132798535794453
S1000000a,61780.286672,163.70205228035468,-18.10471138055092,164.4364500000006,-18.561287999999216,3.106487369364405e-06,i,19.50224387218658,19.49961057650898,0.00996299590797273,0.009945212307287087,23.1343489868631,23.13059981155987
S1000000a,61780.310927,163.70205228035468,-18.10471138055092,164.4365160000002,-18.56311500000129,3.0899264531165437e-06,z,19.506070321795203,19.506622970072044,0.01126449135209172,0.011237007559280756,22.968207967454678,22.964441345175853
S1000000a,61781.239134,163.95033588103914,-18.031113105727716,164.44201499999986,-18.63119400000105,3.2223774034283947e-06,i,19.50028114807821,19.494448387335947,0.01214406799779637,0.01212132996202541,22.85013563621249,22.84858482288965
S1000000a,61781.263141,163.95033588103914,-18.031113105727716,164.4419770000004,-18.63294700000159,3.042088583360277e-06,z,19.486562767073988,19.47832341807803,0.011723502868190884,0.011688663662533069,22.899894717824814,22.898283896399494
S1000000a,61789.27659,164.99043640246796,-19.09523631317997,164.29665099999988,-19.110176000000447,2.8895553381860802e-06,z,19.376978135088684,19.359651855968583,0.008079363622311368,0.00805998568672928,23.293210067462763,23.293123719813384
.. _full::

Full Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The 'all' output option includes all columns from the basic output, as well as those relevant to ephemeris generation for each
predicted detection, and some of the input orbital and physical parameters of each simulated object. This is declared
in the configuration file like so::
predicted detection, and some of the input orbital and physical parameters of each simulated object. All columns within the pandas databframe at the end of the ``Sorcha`` run are written out. This is declared in the :ref:`configuration file<configs>` like so::

[OUTPUT]
output_columns = all

The column names, formats, and descriptions are as follows
Detections File: Full Output Column Names, Formats, and Descriptions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

+------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+
| Keyword | Format | Description |
Expand Down Expand Up @@ -236,32 +245,24 @@ The column names, formats, and descriptions are as follows
.. note::
All positions, positions, and velocities are in respect to J2000.

Custom Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
By setting the value of the output_columns configuration file keyword to a comma-separated list of column names, you may
specify your own custom output, using this page as a reference for potential column names.

For example, you could state this in your configuration file to get the object ID, position and magnitude only::
.. note::
All columns in the comple physicalx parameters file will also be included in the full output.

[OUTPUT]
output_columns = ObjID,RA_deg,Dec_deg,trailedSourceMag

.. warning::
If you are choosing to specify the column names in this way, please perform a quick test-run first to ensure your column names are correct before
embarking on any long runs. As we allow for user-written code and add-ons to add new column names, we do not error-handle the column names until
late in the code, upon output.

If you are writing to a HDF5 file that you plan to access using the PyTables library, note that your object IDs cannot begin
with a number (due to a limitation in PyTables).

Additional Outputs
Optional Outputs
----------------------

Ephemeris Output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Optionally (with the **-ew** flag set at the command line), an ephemeris file of all detections near the
Optionally (with the **--ew** flag set at the command line), an ephemeris file of all detections near the
field can be generated to a separate file, which can then be provided back to ``Sorcha`` as an optional external ephemeris file with the **-er** flag.
More information can be found on this functionality, including the output columns, in the :ref:`Ephemeris Generation<ephemeris_gen>` section of the documentation.

The format of the outputted ephemeris file is controlled by the **eph_format** configuration keyword in the Inputs section of the configuration file::
The format of the outputted ephemeris file is controlled by the **eph_format** configuration keyword in the Inputs section of the :ref:`configuration file<configs>`e::

[INPUT]
ephemerides_type = external
Expand All @@ -271,9 +272,9 @@ The format of the outputted ephemeris file is controlled by the **eph_format** c
Users should note that output produced by reading in a previously-generated ephemeris file will be in a different order than the output produced when running the ephemeris generator within ``Sorcha``.
This is simply a side-effect of how ``Sorcha`` reads in ephemeris files and does not affect the actual content of the output.

Statistics Output
Statistics (Tally) File
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``Sorcha`` can also output a statistics or "tally" file which contains an overview of the ``Sorcha`` output for each object and filter. Minimally, this
``Sorcha`` can also output a statistics or "tally" file (if specified uisng the **--st flag) which contains an overview of the ``Sorcha`` output for each object and filter. Minimally, this
file lists the number of observations for each object in each filter, along with the minimum, maximum and median apparent magnitude and the minimum and maximum
phase angle. If the :ref:`linking filter<linking>` is on, this file also contains information on whether and when the object was linked by SSP.
Expand Down Expand Up @@ -304,5 +305,4 @@ The columns in the statistics file are as follows:
+------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+

.. note::
Unless the user has specified **drop_unlinked = False** in the configuration file, the object_linked column will read TRUE for all objects. To see which objects were not linked by ``Sorcha``, this
variable must be set to False.
Unless the user has specified **drop_unlinked = False** in the :ref:`configuration file<configs>`, the object_linked column will read TRUE for all objects. To see which objects were not linked by ``Sorcha``, this variable must be set to False.

0 comments on commit 7cf0e3f

Please sign in to comment.