Skip to content

TECA 4.0.0
Compare
Choose a tag to compare
@burlen burlen released this 27 Jan 15:29
· 773 commits to develop since this release
4.0.0
2430670
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Documentation

  1. A major overhaul of the command line application section of the user guide including the addition of examples.
  2. Publish batch scripts illustrating running TECA at scale in the new TECA_examples repo.
  3. Giving tutorials and publishing the materials in the new TECA_tutorials repo
  4. Updates to the installation section of the TECA User's Guide](https://teca.readthedocs.io/en/latest/installation.html)

Data Model Improvements

  1. Added support for Arakawa C Grids in teca_arakawa_c_grid
  2. Added support for logically Cartesian so called curvilinear grids in teca_curvilinear_mesh
  3. Refactored mesh related class hierarchy so that common codes such as array accessing and I/O live in teca_mesh
  4. Added support for face and edge centered mesh based data.

I/O Capabilities

  1. Added reader for WRF simulation teca_wrf_reader
  2. Add support for writing logically Cartesian curvilinear meshes in teca_cartesian_mesh_writer.
  3. Added a new NetCDF based output format for tabular data to the teca_table_writer.
  4. Added support for reading tabular CSV files to the teca_table_reader. This enables the tabular outputs such as TC tracks etc saved from TECA apps to be stored in a format ingestible by other tools such as Python and Excel without the need to convert from TECA's internal binary format.
  5. Added versioning and error checking to TECA's internal binary serialization format across all datasets. This enables us to catch version differences and handle bad or corrupted files gracefully.
  6. use of NetCDF parallel 4 (i.e. MPI collective I/O) for writing results. this enables the use of any number of files with any number of ranks.

Execution Patterns

  1. Implement a new streaming mode reduction where data is incrementally reduced as it becomes available. This parallelizes the reduction step and reduces the memory overhead.
  2. Introducing a new MPI parallel approach to scan the time axis. This has substantial benefit when there are a large number of files.
  3. expose MPI aware thread load balancing to Python. This was used in the teca_pytorch_algorithm to automatically load balance the OpenMP backend of PyTorch.
  4. implement GPU load balancing strategy in the teca_pytorch_algorithm.
  5. Enable process groups to be excluded from execution. This lets a pipeline run on a subset of MPI_COMM_WORLD.

Algorithmic Capabilities

  1. Added teca_pytorch_algorithm a base class that handle tasks common to interfacing to PyTroch when developing Machine Learning based detectors.
  2. Added teca_deeplab_ar_detect a new PyTorch based Machine Learning based AR detector.
  3. Added teca_valid_value_mask an algorithm that generates a mask identifying the presence of NetCDF _FillValue values in arrays. Down stream algorithms use the mask to handle _FillValue's in an algorithm appropriate manner.
  4. Added teca_temporal_reduction an algorithm that implements transformations from one time resolution to another. The implementation includes min, max, and average operators and supports daily, monthly, and seasonal intervals.
  5. Added teca_vertical_reduction an algorithm that converts 3D data to 2D by applying a reduction in the vertical spatial dimension. This is a base class that contains code common to vertical reductions.
  6. Added teca_integrated_vapor_transport a vertical reduction that computes IVT from horizontal wind vector and specific humidity.
  7. An improved floating point differencing algorithm was developed and a number of codes were updated to use it.

Command Line Applications

  1. Added teca_integrated_vapor_transport command line application for computing IVT.
  2. Added teca_restripe command line application for re-organizing NetCDF datasets.
  3. Added teca_deeplab_ar_detector command line application detecting AR's using machine learning.
  4. Integrated IVT calculations into the teca_nayesian_ar_detector.
  5. Normalized names and meaning of command line options across command line applications

Python Capabilities

  1. A polymorphic redesigned the teca_python_algorithm makes it easier to use.
  2. Handle numpy scalar types
  3. Expose more features such as MPI aware thread load balancing, calendaring, profiling, and file manipulation utilities.

Testing

  1. Added testing infrastructure and tests for command line applications
  2. Deployed testing on Ubuntu 18.04, Fedora 31, Fedora 32, and Mac OS with xcode 12.2.

Bug fixes

More than 50 bug fixes were reported.

Assets 2
Loading