Contributing to desidatamodel¶

Introduction¶

This page is about contributing to the DESI data model documentation; it is not about the data model itself.

Examples and Other Documents¶

examples

Directory Tree¶

Please follow these rules when creating or updating directories within the data model.

All directories must have an index.rst file.
The title of every index.rst file contains only the name of the directory it is in.
Each index.rst contains the following items in its toctree:
1. Links to index.rst files in any subdirectories.
2. Links to files in that directory.

Every file must have a title. For example:

===================
fibermap-EXPID.fits
===================

Code setup¶

To build the data model locally, you first need to install the following:

pip install sphinx-toolbox sphinx-rtd-theme

Building the Documents¶

To build all the documents, in your git clone directory:

cd doc
make html

Once the build is complete, you can open the file _build/html/index.html in a browser, e.g.: file:///home/user/desidatamodel/doc/_build/html/index.html. If you have installed the sphinx-rtd-theme Python package, the docs will be formatted using the ReadTheDocs theme as they will appear at https://desidatamodel.readthedocs.io In macOS, there is a shortcut for this:

open _build/html/index.html

Sphinx will often print warnings and claim that the “build succeeded” when in fact there were syntax errors that break the output. You must pay attention to the warnings and fix them!

Also note that Sphinx builds documents incrementally. That is, if you run make html, change one file, and then run make html again, it will only rebuild the changed file. Normally this is fine, but if the change causes the directory tree to change, for example, adding a file to a table of contents, then the entire document tree should be rebuilt. This can be done by simply cleaning up first:

make clean
make html

Code Tests¶

desidatamodel also includes unit tests; you can run these locally before opening a PR using:

pytest py/desidatamodel/test

Units¶

We encourage the documentation of units as well as types. Although not every FITS file specifies units, we want units to be documented anyway. FITS images that have units should have a BUNIT header keyword. FITS table columns that have units should have a TUNITxx keyword. For the purposes of documentation though, we want the units to be specified, even if they don’t actually appear in the file being documented.

Units should follow the FITS Standard, in particular following Section 4.3, and Tables 3, 4, and 5 in that document.

You can test units for validity by using Astropy Units. This package already supports the FITS Standard. The desidatamodel package itself already uses this internally. In fact, we have added some units that DESI considers acceptable, even if they do not strictly follow the FITS Standard.

Here are some examples of units that are used in this data model, as well as a few common gotchas.

Unit	FITS Standard?	Comment
`um`	Yes	Micrometers, \(\mu m\).
`Angstrom`	Yes	Ångström.
`photon`	Yes	Number of photons.
`count`	Yes	Number of counts, usually electrons.
`adu`	Yes	Closely related to `counts`.
`deg`	Yes	Degrees.
`arcsec`	Yes	Seconds of arc. Not time!
`mag`	Yes	Standard astronomical magnitude. Not the same as a `maggie`.
`pc`	Yes	Parsec.
`Jy`	Yes	Jansky.
`10**-17 erg/(s cm2 Angstrom)`	Yes	Common unit of spectrophotometric flux.
`10**+34 (s2 cm4 Angstrom2) / erg2`	Yes	Inverse variance of flux.
`A`	Yes, but…	`A` is the unit for amperes not Ångström.
`maggie`	No, but OK	Standard prefix is also OK: `nanomaggie`. `nanomaggy` is also OK.
`mgy`	No, but OK	Abbreviation for `maggie`.
`electron/Angstrom`	No, but OK	Used in some calibration files.
`ergs`	No	`erg`, not `ergs`.
`sec`	No	`s` for seconds, not `sec`. Even though `arcsec` is OK.
`1/deg^2`	No	`1` is not a unit. Use `deg^-2` instead. `deg**-2` is also OK.

Tips and Tests¶

You can browse some examples.

Cross-Referencing¶

To a file¶

Here is how you make a direct link to a file:

Here is how you make a :doc:`direct link to a file <examples/spPlate>`

Note that the link must take into account the directory structure. So for example, if you’re in the directory DESI_SPECTRO_SIM/PIXPROD/NIGHT and want to refer to a file in DESI_SPECTRO_DATA/NIGHT, the link has to have the form:

:doc:`link to real data <../../../DESI_SPECTRO_DATA/NIGHT/real_data_file>`

or:

:doc:`link to real data </DESI_SPECTRO_DATA/NIGHT/real_data_file>`

That is, you can use a relative or absolute path.

Within a file¶

You can also cross-reference sections within files, however the notation is somewhat different. There are two methods. The first involves creating an explicit reference point in the target document. For example, in the spPlate file we referenced above, we can label HDU5:

.. _spplate-hdu5-plugmap:

HDU5
----

EXTNAME = PLUGMAP

Then we use :ref: to cross-reference that label. Here’s a link to that section of the spPlate file:

:ref:`section of the spPlate file <spplate-hdu5-plugmap>`

Note however, that this label must be globally unique!

Alternatively, one can use “raw” ReStructuredText constructions. Here’s a link to another section of the spPlate file we already linked to above:

link to another `section of the spPlate file`_

.. _`section of the spPlate file`: examples/spPlate.html#hdu6

Note this time that the section name may be upper case (HDU6), but the HTML anchor is lower case #hdu6.

To a HDU¶

The latter form can be used to create cross references to individual HDUs in other files. This would have the (strict!) form:

HDU5
----

See `HDU1 in some-other-file <../../some-other-file.html#hdu2`_.

In other words, with this notation, the data model for HDU2 in some-other-file will be used as the data model for HDU5 in the file with the cross-reference.

Environment variables¶

Here is how to highlight an ENVIRONMENT_VARIABLE:

Here is how to highlight an :envvar:`ENVIRONMENT_VARIABLE`

Optional Keywords and Columns¶

Sometimes HDU in a file might not have all of the header keywords or a table might not have all the columns described in the data model. Sometimes this is expected, and these items should be marked as optional, so we can focus on required items that might be missing. The optional notation leverages reStructuredText’s footnote notation. Here is an example using table columns:

Required Data Table Columns
~~~~~~~~~~~~~~~~~~~~~~~~~~~

========= ======== ===== ===========
Name      Type     Units Description
========= ======== ===== ===========
target    char[20]
OPT1 [1]_ int16
V_mag     float32  mag
vdisp     float64  km/s
OPT2 [1]_ float32
========= ======== ===== ===========

.. [1] Optional

Note how .. [1] Optional is added to the bottom. This keeps Sphinx from complaining about undefined footnotes, but also makes it easy for humans to see what this notation means. .. [1] Optional only needs to be added once per file, not once per table.

Optional HDUs¶

This is a work in progress.

Strings¶

Depending on how data sets are collated, it sometimes happens that sets of strings may be written out to FITS files with different lengths.

For example, data files A and B are supposedly identical (same columns, same types, etc.). However data file A has a string-valued column NAME that has values from the set {'one', 'two', 'three'}, while in data file B the same column has values from the set {'one', 'two', 'six'}. When written out, file A has the NAME column represented as char[5] (5A in FITS notation), while file B has the same column represented as char[3] (3A).

To account for differences like this, one can use:

Required Data Table Columns
~~~~~~~~~~~~~~~~~~~~~~~~~~~

========= ======== ===== ===========
Name      Type     Units Description
========= ======== ===== ===========
...
NAME      char[*]
...
========= ======== ===== ===========

to represent the fact that the actual length of the string doesn’t matter.