CUMULO Project

This repository was created during the practical course "Creation of Deep Learning Methods" in the winter semester 2021 at the Technical University of Munich. Its purpose was to use equivariant deep learning architectures to predict cloud types using the CUMULO dataset. Equivariant deep learning architectures are explained here: https://arxiv.org/abs/1911.08251.

The repository is based on https://github.com/FrontierDevelopmentLab/CUMULO. The U-Net architecture and training script were adapted from https://github.com/LobellLab/weakly_supervised.

Installation

The installation has been tested with python 3.8.5.

cd cumulo
pip install -e .
pip install -r requirements.txt

Demo

This demo first preprocesses the nc files by filtering artefacts and finding files without labels. Then, the dataset statistics are calculated with 16 tiles per nc file. Next, a model is trained on the demo data, excluding nc_files without labels. Last, the trained model is used to predict the demo data and the predictions get evaluated. This is just a demo and just demonstrates the use of the commands. It does not produce meaningful results.

cd cumulo
python scripts/preprocessing/filter_artefacts_and_nolabels.py --path ./demo_data --removed_path ./artefacts
python scripts/preprocessing/calculate_statistics.py --path ./demo_data --tile_number 16
python scripts/pipeline/train.py --model_path ./models/demo_training --data_path ./demo_data --model unet --epoch_number 5 --nc_exclude_path ./demo_data/no_labels.pkl --demo 
python scripts/pipeline/predict.py --flagfile ./models/demo_training/flagfile.txt --output_path ./models/demo_training/predictions --prediction_number 2

Structure and Examples

All examples assume that this repository is the current folder.

scripts

• preprocessing

  • calculate_statistics.py: Can be used to calculate mean, variance and class weights for a given dataset. Example: python scripts/preprocessing/calculate_statistics.py --path <data_path> --sample_number 10 --tile_number 16 to use 16 tiles of 10 nc files for calculating the statistic.

  • filter_artefacts_and_nolabels.py: The CUMULO dataset contains nc files where missing data was filled up with the nearest available data. This script can be used to filter these artefacts (vertical lines with equal values). There also exist nc files without labels. This script creates a list of these files which can be used to exclude them from training. Example:python scripts/preprocessing/filter_artefacts_and_nolabels.py --path <data_path> --removed_path <other_path> to sort nc files with artefacts into 'other_path' and to save a list of nc files without labels in 'data_path'.

• pipeline

  • train.py: Can be used to train a model on the CUMULO dataset. All training parameters are given and saved as flags from the abseil library. Example: python scripts/pipeline/train.py --model_path <train_path> --data_path <data_path> --model equi --rot 4 to train an equivariant U-Net (4 discrete rotations) on the data at 'data_path' and save the model and all training information at 'train_path'. Training with the iresnet model is still work-in-progress and throws an error in evaluation and prediction. Set mask_weight = 0 and class_number = 8 if training without cloud mask and only with labeled pixels. Set auto_weight > 0 and class_number = 22 (1 for cloud mask + 8 for cloud classes + 13 for autoencoder) if training with additional autoencoder loss.

  • predict.py: Can be used to generate predictions using a trained model and to evaluate these predictions, generating images, ROCs, histograms, evaluation reports and confusion matrices. Example: python scripts/pipeline/predict.py --flagfile <path_to_flagfile.txt> --output_path <output_path> --prediction_number 50 to generate and evaluate predictions of 50 nc files from the test set of the trained model which is described by the flagfile.txt in its training folder and save all results at 'output_path'

• visualization

  • dataset.py: Can be used to convert nc files to images.
  • metrics.py: Can be used to visualize metrics which are saved during training of models.
  • models_to_table.py: Can be used to merge the results of multiple training evaluations into a single LaTeX table. Requires that the predictions of all trainings have been saved in a 'predictions' folder in the respective training folder.
  • predictions.py: Can be used to visualize .npz predictions (e.g. training examples generated by train.py).

cumulo

• data

  • Contains the CumuloDataset which is used to receive tiles during training.

• models

  • iresnet.py: Copied from https://github.com/FrontierDevelopmentLab/CUMULO. For details, see https://arxiv.org/abs/1902.02767.
  • unet.py: Adapted U-Net implementation from https://github.com/LobellLab/weakly_supervised.
  • unet_equi.py: Same architecture as unet.py, but with equivariant modules from the e2cnn library.

• utils

  • basics.py: Simple functions for reading nc files, processing labels, ...
  • evaluation.py: Functions for performing the evaluation tasks (generating histograms, ROCs, evaluation reports, ...).
  • iresnet_utils.py: Functions necessary for the iresnet, all copied from https://github.com/FrontierDevelopmentLab/CUMULO. This is still work-in-progress.
  • training.py: Functions used during training or by the CumuloDataset (e.g. tile extraction, dataset statistics, ...)
  • visualization.py: Functions used to convert network predictions into RGB images.

Future Work

As indicated, the IResNet model integration into the trainings pipeline is still work-in-progress. The UNets produce worse results than the IResNet results from https://arxiv.org/abs/1911.04227. Therefore, the next steps would be to reproduce the IResNet results and understand why it is better in learning the cloud classes. Implementing an equivariant version of the IResNet may also improve the results.

The equivariant UNet architecture could still be extended with equivariance to mirroring.