Merged develop in

NOAA-GFDL · Apr 18, 2024 · 80b4695 · 80b4695
2 parents fdd658a + b8c8c53
commit 80b4695
Show file tree

Hide file tree

Showing 111 changed files with 1,530 additions and 2,851 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -386,7 +386,7 @@ jobs:
           name: run tests
           command: |
             . venv/bin/activate
-            cd driver
+            cd pace
             GT_CACHE_ROOT=$(pwd)/.gt_cache MPIRUN_CALL="mpirun -n 6 --mca btl_vader_single_copy_mechanism none" make test_mpi
       - save_gt_cache:
           key: v1-gt_cache_driver
@@ -409,9 +409,9 @@ jobs:
           name: run tests
           command: |
             . venv/bin/activate
-            cd driver
+            cd pace
             mpirun -n 6 --mca btl_vader_single_copy_mechanism none \
-              python3 -m mpi4py -m pace.driver.run examples/configs/baroclinic_c12_orch_cpu.yaml
+              python3 -m mpi4py -m pace.run examples/configs/baroclinic_c12_orch_cpu.yaml
 
   test_main:
     machine:

diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,30 @@
+---
+name: Bug Report
+about: Create a bug report to help us improve
+title: ''
+labels: Bug
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is
+
+**To Reproduce**
+Steps to reproduce the behavior
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**System Environment**
+Describe the system environment, include:
+- OS: [e.g. RHEL 7.2]
+- Backend used [e.g. dace:cpu]
+- Environment variables set
+- Compiler(s): Type and version [e.g. Intel 19.1]
+- MPI type, and version (e.g. MPICH, Cray MPI, openMPI)
+- netCDF Version: For both C and Fortran
+- Configure options: Any additional flags, or macros passed to configure
+- If this bug came from a model run, which model
+
+**Additional context**
+Add any other context about the problem.  If applicable, include where any files
+that help describe, or reproduce the problem exist.
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -0,0 +1,19 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: 'enhancement'
+assignees: ''
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
diff --git a/.github/ISSUE_TEMPLATE/support_request.md b/.github/ISSUE_TEMPLATE/support_request.md
@@ -0,0 +1,14 @@
+---
+name: Support request
+about: Request for help
+title: ''
+labels: 'question'
+assignees: ''
+---
+
+**Is your question related to a problem? Please describe.**
+A clear and concise description of what the problem is.
+
+**Describe what you have tried**
+A clear and concise description of what steps you have taken.  Include command
+lines, and any messages from the command.
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,21 @@
+**Description**
+Include a summary of the change and which issue is fixed. Please also include
+relevant motivation and context. List any dependencies that are required for
+this change.
+
+Fixes # (issue)
+If this is a hotfix to a released version, please specify it.
+
+**How Has This Been Tested?**
+Please describe the tests that you ran to verify your changes. Please also note
+any relevant details for your test configuration (e.g. compiler, OS).  Include
+enough information so someone can reproduce your tests.
+
+**Checklist:**
+- [ ] My code follows the style guidelines of this project
+- [ ] I have performed a self-review of my own code
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I have made corresponding changes to the documentation
+- [ ] My changes generate no new warnings
+- [ ] Any dependent changes have been merged and published in downstream modules
+- [ ] New check tests, if applicable, are included
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -23,7 +23,7 @@ Before submitting this PR, please make sure:
 - [ ] You have followed the coding standards guidelines established at [Code Review Checklist](https://drive.google.com/file/d/1R0nqOxfYnzaSdoYdt8yjx5J482ETI2Ft/view?usp=sharing).
 - [ ] Docstrings and type hints are added to new and updated routines, as appropriate
 - [ ] All relevant documentation has been updated or added (e.g. README, CONTRIBUTING docs)
-- [ ] For each public change and fix in `pace-util`, HISTORY has been updated
+- [ ] For each public change and fix in `pace`, HISTORY has been updated
 - [ ] Unit tests are added or updated for non-stencil code changes
 
 Additionally, if this PR contains code authored by new contributors:

diff --git a/.github/workflows/main_unit_tests.yml b/.github/workflows/main_unit_tests.yml
@@ -28,4 +28,4 @@ jobs:
             git clone -b store_files https://github.com/mlee03/pace.git tmp && mv tmp/*.nc . && rm -rf tmp
         - name: Run all main tests
           run: |
-            pytest -x tests/main
+            python -m pytest -x tests/main
diff --git a/.github/workflows/main_unit_tests_mpich.yaml b/.github/workflows/main_unit_tests_mpich.yaml
@@ -0,0 +1,26 @@
+name: "Main unit tests with mpich"
+on:
+  pull_request:
+      types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
+
+# cancel running jobs if theres a newer push
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  main_unit_tests:
+    runs-on: ubuntu-latest
+    container:
+      image: ghcr.io/noaa-gfdl/pace_mpich:3.8
+    steps:
+        - name: Checkout Pace repository
+          uses: actions/checkout@v4
+          with:
+            submodules: 'recursive'
+        - name: setup env and run tests (test)
+          run: |
+            cp /home/scripts/setup_env.sh . && chmod +x setup_env.sh
+            cp /home/scripts/run_tests.sh . && chmod +x run_tests.sh
+            ./setup_env.sh
+            ./run_tests.sh
diff --git a/.github/workflows/main_unit_tests_openmpi.yaml b/.github/workflows/main_unit_tests_openmpi.yaml
@@ -0,0 +1,26 @@
+name: "Main unit tests with openmpi"
+on:
+  pull_request:
+      types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
+
+# cancel running jobs if theres a newer push
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  main_unit_tests:
+    runs-on: ubuntu-latest
+    container:
+      image: ghcr.io/noaa-gfdl/pace_openmpi:3.8
+    steps:
+        - name: Checkout Pace repository
+          uses: actions/checkout@v4
+          with:
+            submodules: 'recursive'
+        - name: setup env and run tests
+          run: |
+            cp /home/scripts/setup_env.sh . && chmod +x setup_env.sh
+            cp /home/scripts/run_tests.sh . && chmod +x run_tests.sh
+            ./setup_env.sh
+            ./run_tests.sh
diff --git a/.gitignore b/.gitignore
@@ -2,9 +2,9 @@
 # Test outputs
 results.xml
 *.zarr
-driver/examples/*.json
-driver/examples/**/*.json
-driver/examples/comm
+examples/*.json
+examples/**/*.json
+examples/comm
 20*-*-*-*-*-*.json
 *.pkl
 

diff --git a/.gitmodules b/.gitmodules
@@ -1,10 +1,13 @@
 
 [submodule "NDSL"]
 	path = NDSL
-	url = [email protected]:NOAA-GFDL/NDSL.git
+	url = https://github.com/NOAA-GFDL/NDSL.git
+	branch = develop
 [submodule "pyFV3"]
 	path = pyFV3
-	url = [email protected]:NOAA-GFDL/PyFV3.git
+	url = https://github.com/NOAA-GFDL/PyFV3.git
+	branch = develop
 [submodule "pySHiELD"]
 	path = pySHiELD
-	url = [email protected]:NOAA-GFDL/PySHiELD.git
+	url = https://github.com/NOAA-GFDL/PySHiELD.git
+	branch = develop
diff --git a/.jenkins/baroclinic_initialization.sh b/.jenkins/baroclinic_initialization.sh
@@ -62,7 +62,7 @@ export VIRTUALENV=${JENKINS_DIR}/../venv_driver
 ${JENKINS_DIR}/install_virtualenv.sh ${VIRTUALENV}
 source ${VIRTUALENV}/bin/activate
 
-CMD="srun python3 ${PACE_DIR}/driver/examples/baroclinic_init.py ${JENKINS_DIR}/driver_configs/${experiment}.yaml"
+CMD="srun python3 ${PACE_DIR}/examples/baroclinic_init.py ${JENKINS_DIR}/driver_configs/${experiment}.yaml"
 run_command "${CMD}" Job${action} ${scheduler_script} $minutes
 if [ $? -ne 0 ] ; then
   exitError 1510 ${LINENO} "problem while executing script ${script}"
@@ -71,7 +71,7 @@ fi
 module load sarus
 sarus pull elynnwu/pace:latest
 echo "####### generating figures..."
-srun -C gpu --partition=debug --account=go31 --time=00:05:00 sarus run --mount=type=bind,source=${PACE_DIR},destination=/work elynnwu/pace:latest python /work/driver/examples/plot_baroclinic_init.py /work/output.zarr ${experiment} pt -1
+srun -C gpu --partition=debug --account=go31 --time=00:05:00 sarus run --mount=type=bind,source=${PACE_DIR},destination=/work elynnwu/pace:latest python /work/examples/plot_baroclinic_init.py /work/output.zarr ${experiment} pt -1
 mkdir -p ${ARTIFACT_ROOT}/${experiment}
 echo "####### moving figures..."
 cp *.png ${ARTIFACT_ROOT}/${experiment}/.

diff --git a/.jenkins/driver_performance.sh b/.jenkins/driver_performance.sh
@@ -33,7 +33,7 @@ cat << EOF > run.daint.slurm
 set -x
 export OMP_NUM_THREADS=12
 export FV3_DACEMODE=BuildAndRun
-srun python -m pace.driver.run ${JENKINS_DIR}/driver_configs/baroclinic_c192_6ranks.yaml
+srun python -m pace.run ${JENKINS_DIR}/driver_configs/baroclinic_c192_6ranks.yaml
 EOF
 launch_job run.daint.slurm 3600
 

diff --git a/.jenkins/install_virtualenv.sh b/.jenkins/install_virtualenv.sh
@@ -35,7 +35,7 @@ cd ${PACE_DIR}
 python3 -m pip install $wheel_command -r ${PACE_DIR}/requirements_dev.txt -c ${PACE_DIR}/constraints.txt
 # have to be installed in non-develop mode because the directory where this gets built from
 # gets deleted before the tests run on daint
-python3 -m pip install ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/driver ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/dsl ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/fv3core ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/physics ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/stencils ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/util ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/external/gt4py -c ${PACE_DIR}/constraints.txt
+python3 -m pip install ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/driver ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/ndsl ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/pyFV3 ${PACE_VENV_INSTALL_PREFIX} ${PACE_DIR}/pySHiELD ${PACE_DIR}/external/gt4py -c ${PACE_DIR}/constraints.txt
 cd ${workdir}
 
 deactivate
diff --git a/.jenkins/run_compare_fortran.sh b/.jenkins/run_compare_fortran.sh
@@ -55,7 +55,7 @@ cat << EOF > run.daint.slurm
 ########################################################
 set -x
 export OMP_NUM_THREADS=12
-srun python -m pace.driver.run ${JENKINS_DIR}/driver_configs/${NAMELIST}.yaml
+srun python -m pace.run ${JENKINS_DIR}/driver_configs/${NAMELIST}.yaml
 EOF
 
 launch_job run.daint.slurm 2400

diff --git a/.jenkins/run_diff_rank.sh b/.jenkins/run_diff_rank.sh
@@ -29,7 +29,7 @@ cat << EOF > run.daint.slurm
 ########################################################
 set -x
 export OMP_NUM_THREADS=12
-srun python -m pace.driver.run ${JENKINS_DIR}/driver_configs/baroclinic_c192_54ranks.yaml
+srun python -m pace.run ${JENKINS_DIR}/driver_configs/baroclinic_c192_54ranks.yaml
 EOF
 
 launch_job run.daint.slurm 29000
@@ -55,7 +55,7 @@ cat << EOF > run.daint.slurm
 ########################################################
 set -x
 export OMP_NUM_THREADS=12
-srun python -m pace.driver.run ${JENKINS_DIR}/driver_configs/baroclinic_c192_6ranks.yaml
+srun python -m pace.run ${JENKINS_DIR}/driver_configs/baroclinic_c192_6ranks.yaml
 EOF
 
 launch_job run.daint.slurm 29000

diff --git a/.jenkins/test_driver.sh b/.jenkins/test_driver.sh
@@ -3,4 +3,4 @@
 JENKINS_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
 
 PACE_IMAGE="driver_image" make -C ${JENKINS_DIR}/.. build
-docker run --rm driver_image make -C /pace/driver test test_mpi
+docker run --rm driver_image make -C /driver/pace/driver test test_mpi
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -18,13 +18,10 @@ repos:
     rev: v0.812
     hooks:
     - id: mypy
-      name: mypy-driver
+      name: mypy-pace
+      files: pace
       args: [--config-file, setup.cfg]
-      files: driver
-      exclude: |
-        (?x)^(
-        tests/main/driver/__init__.py |
-        )$
+
 -   repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v2.3.0
     hooks:

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -113,7 +113,7 @@ Turns into
 - Internal functions that are likely to be inlined into a larger stencil do not need this if it will just be removed in the near-term.
 
 ### GT4Py stencils
-We interface to `gt4py.cartesian.gtscript.stencil` through pace.dsl.stencil, specifically the FrozenStencil, that allows us to minimize runtime overhead in calling stencils.
+We interface to `gt4py.cartesian.gtscript.stencil` through ndsl.dsl.stencil, specifically the FrozenStencil, that allows us to minimize runtime overhead in calling stencils.
 
 
 ```python

diff --git a/Dockerfile b/Dockerfile
@@ -2,12 +2,16 @@ FROM python:3.8.13-bullseye@sha256:2a01d88a1684e6d7f08030cf5ae73b536926c64076cab
 
 RUN apt-get update && apt-get install -y make \
     software-properties-common \
+    libgeos-dev \
     libopenmpi3 \
     libopenmpi-dev \
     libboost-all-dev \
     libhdf5-serial-dev \
     netcdf-bin \
     libnetcdf-dev \
+    libproj-dev \
+    proj-bin \
+    proj-data \
     python3 \
     python3-pip
 
@@ -22,5 +26,26 @@ COPY . /pace
 RUN cd /pace && \
     pip3 install -r /pace/requirements_dev.txt -c /pace/constraints.txt
 
+RUN cd / && \
+    git clone https://github.com/ai2cm/fv3net
+
+ENV CFLAGS="-I/usr/include -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H=1"
+
+RUN python3 -m pip install \
+    numpy==1.21.2 \
+    netCDF4==1.5.7 \
+    mpi4py==3.1.1 \
+    matplotlib==3.5.2 \
+    ipyparallel==8.4.1 \
+    jupyterlab==3.4.4 \
+    shapely==1.8.5 \
+    cartopy==0.18.0 \
+    jupyterlab_code_formatter==1.5.2 \
+    isort==5.10.1 \
+    black==22.3.0 \
+    /fv3net/external/vcm
+
+ENV PYTHONPATH=/fv3net/external/fv3viz:/pace/external/gt4py/src
+
 ENV OMPI_ALLOW_RUN_AS_ROOT=1
 ENV OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1