Add better Alignment/tilt angle validations

[uermel]

Post migration we have the opportunity to better validate alignment files and metadata concerning acquisition.

In general one should think about collection/angle/alignment metadata in cryoET the following way:

1. MDOC and Frames

The mdoc file and frames are raw data from the microscope. The mdoc file is created simultaneously to acquisition of the frames. mdoc contains acquisition metadata. As such, there should be a 1:1 mapping of mdoc-sections to frame files.

Useful validations would be:

• number of mdoc sections == number of frame files on bucket (I believe we have this)
• frame files names match those in mdoc file
• tilt axis angle from mdoc file ~approximately (+/- 10 deg) matches that in tilt series metadata (if not we'll have to fix the mdoc as part of frame import scope)
• tilt axis angle from mdoc file ~approximately (+/- 10 deg) matches those in alignment files and metadata
• P1: number of sub-frames listed in mdoc file matches frame files

2. RAWTLT and Tiltseries

Frames are aligned and assembled into tilt series during motion correction. In order to communicate the tilt series to downstream programs, the rawtlt file is typically used. As such, the rawtlt file is metadata of the tiltseries, and there should be exactly as many angles in it as there are sections in the tilt series. As the name implies, they should be the raw angles from acquisition.

Useful validations are:

• number of rawtlt angles == number of z-sections in tilt series
• number of rawtlt angles <= number of mdoc sections (raise warning if not exactly equal)
• all rawtlt angles are ~approximately (+/- 1e-2) appearing in mdoc file (only do this validation if is_aligned is not set)
• the same validations as above, but against our metadata files and not the data itself.

3. Alignments

IMOD-style ( TLT, XF, XTILT files )

Imod stores alignments in columnar files, with tilt angles (after alignment) stored in TLT, translation and tilt axis angle stored in XF, and x-tilt stored in XTILT files. IMOD alignments should have one entry per section in the tilt series, and each columnar file should have the same number of rows.

Useful validations are:

• # rows of TLT == # rows in XF == # rows in XTILT == # sections in tilt series == number of sections in portal metadata
• # rows in TLT <= number of sections in mdoc (raise warning if not exactly equal)
• TLT, XTILT have 1 column
• XF has 6 columns

AreTomo-style (ALN)

The ALN file contains the same information as the IMOD-style alignment, but stores everything in a more complex format. Briefly, it stores the total size of the tilt series in pixel, the indeces of the excluded images (DarkFrames), and global and local alignment information in two columnar sections.

Useful validations are:

• Total size == size of tilt series == size in portal metadata
• Total size <= number of sections in mdoc (raise warning if not exactly equal)
• Internal consistency of ALN file