Code

Initial ICI changes#238

Code

Draft

hannahnesser wants to merge 114 commits intodevgeoschem/integrated_methane_inversion:devfrom ICIgeoschem/integrated_methane_inversion:ICICopy head branch name to clipboard

+2,130-596

Commits

Commits on Mar 19, 2024

• Creating default config, python environment, and GCC environment settings

  

  Hannah Nesser

  committed
  882ce86

  Copy full SHA for 882ce86

Commits on Mar 20, 2024

• Changing DataPathTROPOMI to DataPathObs (and adding Species to config_required for sanitize_input_yaml)

  

  Hannah Nesser

  committed
  466fefc

  Copy full SHA for 466fefc

• Changes to generalize supporting other clusters, including adding Species, UseScheduler, SchedulerType, DataPathObs, and PythonEnvType/PythonEnvPath

  

  Hannah Nesser

  committed
  4afa68e

  Copy full SHA for 4afa68e

• Default species for Harvard should be CH4

  

  Hannah Nesser

  committed
  8b7e200

  Copy full SHA for 8b7e200

• Generalizing scheduler options

  

  Hannah Nesser

  committed
  beb0651

  Copy full SHA for beb0651

Commits on Mar 21, 2024

• Moving conda environment activation into environment files

  

  Hannah Nesser

  committed
  277bcf3

  Copy full SHA for 277bcf3

• Removing conda environment specification from config file

  

  Hannah Nesser

  committed
  a217e2e

  Copy full SHA for a217e2e

• Removing conda environment specification from config file

  

  Hannah Nesser

  committed
  e8312be

  Copy full SHA for e8312be

• Moving conda environment activation into environment files

  

  Hannah Nesser

  committed
  5d3abb3

  Copy full SHA for 5d3abb3

• Removing UseSlurm option and replacing with UseScheduler/SchedulerType

  

  Hannah Nesser

  committed
  08023af

  Copy full SHA for 08023af

• Switching from UseSlurm to UseScheduler/SchedulerType

  

  Hannah Nesser

  committed
  a8576c4

  Copy full SHA for a8576c4

Commits on Mar 22, 2024

• Adding PythonEnv option

  

  Hannah Nesser

  committed
  e8b13e7

  Copy full SHA for e8b13e7

• Adding PythonEnv to variables required for local simulation

  

  Hannah Nesser

  committed
  639a1f4

  Copy full SHA for 639a1f4

• Removing python loading from GEOS-Chem environment

  

  Hannah Nesser

  committed
  55bd28c

  Copy full SHA for 55bd28c

• Creating separate environment file for python loading

  

  Hannah Nesser

  committed
  5c05ed4

  Copy full SHA for 5c05ed4

• Adding separate statement for loading Python env

  

  Hannah Nesser

  committed
  0960227

  Copy full SHA for 0960227

• Switching conda activate to source

  

  Hannah Nesser

  committed
  fb1e776

  Copy full SHA for fb1e776

• Handling python environment loading for AWS (now hardcoded?) and local versions

  

  Hannah Nesser

  committed
  fdf9e33

  Copy full SHA for fdf9e33

• Added reference to PythonEnv since htat's now specified along with GEOSChemEnv for local simulations

  

  Hannah Nesser

  committed
  cdad798

  Copy full SHA for cdad798

Commits on Apr 2, 2024

• Changes related to generalizing the IMI for multiple clusters

  Show description for 035b444
  hannahnesser

  committed
  035b444

  Copy full SHA for 035b444

• Amending config files to include PythonEnv and simplify SchedulerType treatment

  hannahnesser

  committed
  6e6980e

  Copy full SHA for 6e6980e

• These files contained non-default uses of sbatch. As a result, we replaced sbatch with an if/elif/else statement that depends on SchedulerType and adjusts the call to the scheduler within the scrip…

  Show description for 59227ab
  hannahnesser

  committed
  59227ab

  Copy full SHA for 59227ab

• Removed reference to UseScheduler

  hannahnesser

  committed
  9eea594

  Copy full SHA for 9eea594

• Documentation changes consistent with the changes described in a previous commit

  hannahnesser

  committed
  b0ef56b

  Copy full SHA for b0ef56b

• Adding tmux option to SchedulerType

  hannahnesser

  committed
  d9d7c62

  Copy full SHA for d9d7c62

• Adding species argument

  hannahnesser

  committed
  e6bfae8

  Copy full SHA for e6bfae8

• Removing only aws requirementS

  hannahnesser

  committed
  656b013

  Copy full SHA for 656b013

Commits on Apr 3, 2024

• Modified to handle tmux

  hannahnesser

  committed
  a2b24b4

  Copy full SHA for a2b24b4

Commits on Apr 11, 2024

• Switched to standard SBATCH headers

  

  Hannah Nesser

  committed
  6436535

  Copy full SHA for 6436535

Commits on Apr 12, 2024

• Adding in corrections for PBS

  

  Hannah Nesser

  committed
  01c522a

  Copy full SHA for 01c522a

• Changes to SBATCH to PBS conversion

  

  Hannah Nesser

  committed
  4279240

  Copy full SHA for 4279240

• Moved SBATCH to PBS conversion to common.sh

  

  Hannah Nesser

  committed
  17d4dda

  Copy full SHA for 17d4dda

• Added units to mem to avoid confusion in conversion from SBATCH to PBS

  

  Hannah Nesser

  committed
  262ad9e

  Copy full SHA for 262ad9e

• Added units to mem to avoid confusion in conversion from SBATCH to PBS

  

  Hannah Nesser

  committed
  211932d

  Copy full SHA for 211932d

• Switched condaenv options to PythonEnv

  

  Hannah Nesser

  committed
  b29b878

  Copy full SHA for b29b878

Commits on Jun 7, 2024

• Minor syntax fix

  

  Hannah Nesser

  committed
  5318cb8

  Copy full SHA for 5318cb8

• Mostly changes to get replacement of SBATCH to PBS working

  

  Hannah Nesser

  committed
  d0d6a41

  Copy full SHA for d0d6a41

• Adding print statement to check

  

  Hannah Nesser

  committed
  bc91e5e

  Copy full SHA for bc91e5e

• Continued bug fixes to awk

  

  Hannah Nesser

  committed
  f133892

  Copy full SHA for f133892

Commits on Jun 8, 2024

• Continued debugging

  

  Hannah Nesser

  committed
  288d049

  Copy full SHA for 288d049

Commits on Jun 24, 2024

• Continued bug fixes

  

  Hannah Nesser

  committed
  6c829e6

  Copy full SHA for 6c829e6

• Adding node request to sbatch

  

  Hannah Nesser

  committed
  5d50a2a

  Copy full SHA for 5d50a2a

• Removing nodes options, which will now be passed with sbatch -N directly

  

  Hannah Nesser

  committed
  adbe07e

  Copy full SHA for adbe07e

Commits on Jun 25, 2024

• Changing default format of time request to HH:MM:SS for consistency between PBS/SBATCH requirements; changed default resource request to include a unit term (gb) for compatibility between PBS/SBATC…

  Show description for b9839a2
  

  Hannah Nesser

  committed
  b9839a2

  Copy full SHA for b9839a2

Commits on Jun 26, 2024

• Removed BlendedTROPOMI variable and replaced it with satellite_str (specified as BlendedTROPOMI, TROPOMI, or Other), added species variable, and removed all references to TROPOMI/methane where poss…

  Show description for 338d10a
  

  Hannah Nesser

  committed
  338d10a

  Copy full SHA for 338d10a

• Removing references to TROPOMI and correcting function names to match the generic function names from the satellite_operator file

  

  Hannah Nesser

  committed
  dded98c

  Copy full SHA for dded98c

• Removing references to TROPOMI and correcting function names to match the generic function names from the satellite_operator file

  

  Hannah Nesser

  committed
  ecdbe13

  Copy full SHA for ecdbe13

Commits on Jun 27, 2024

• Switched HourlyCH4 to HourlySpecies, changed ch4_run.template to run.template, and added in new arguments for jacobian.py (species and name changes for variables)

  

  Hannah Nesser

  committed
  a6ed832

  Copy full SHA for a6ed832

• Changed HourlyCH4 to HourlySpecies and ch4_run.template to run.template

  

  Hannah Nesser

  committed
  ad5303e

  Copy full SHA for ad5303e

• Changed tropomi_cache --> satellite_cache

  

  Hannah Nesser

  committed
  4b55f6b

  Copy full SHA for 4b55f6b

• Moving away from specific TROPOMI/CH4 references (mostly in variable names, but

  Show description for 694dfd5
  

  Hannah Nesser

  committed
  694dfd5

  Copy full SHA for 694dfd5

• Changes mostly to variable names to move away from TROPOMI-specific and methane-

  Show description for 9156a77
  

  Hannah Nesser

  committed
  9156a77

  Copy full SHA for 9156a77

• Removing TROPOMI specific references

  Show description for 2f83ab5
  

  Hannah Nesser

  committed
  2f83ab5

  Copy full SHA for 2f83ab5

• - Added species argument to average_satellite_observations

  Show description for ac3385a
  

  Hannah Nesser

  committed
  ac3385a

  Copy full SHA for ac3385a

• Adding functions: mixing_ratio_conv_factor, species_molar_mass, read_tropomi,

  Show description for dfce7e6
  

  Hannah Nesser

  committed
  dfce7e6

  Copy full SHA for dfce7e6

• Change BlendedTROPOMI to SatelliteProduct, change HourlyCH4 to HourlySpecies, and add UseBCsForRestart

  

  Hannah Nesser

  committed
  9a33d72

  Copy full SHA for 9a33d72

• While this script is implicitly very TROPOMI-specific, we still remained TROPOMI and

  Show description for 1ab74f2
  

  Hannah Nesser

  committed
  1ab74f2

  Copy full SHA for 1ab74f2

• - Change ch4_run.template to use the species variable from the config file

  Show description for 2873f6c
  

  Hannah Nesser

  committed
  2873f6c

  Copy full SHA for 2873f6c

• Changed data directory name from data_TROPOMI to data_satellite; there's still a reference to the integrated_methane_inversion in one of the AWS functions, but I'm not sure what to do with this

  

  Hannah Nesser

  committed
  49fc3e1

  Copy full SHA for 49fc3e1

• Changed ch4_run.template to <species>_run.template

  

  Hannah Nesser

  committed
  771b962

  Copy full SHA for 771b962

• Changed ch4_run.template to <species>_run.template

  

  Hannah Nesser

  committed
  e128bda

  Copy full SHA for e128bda

• Move UseBCsForRestart from setup.sh to config file

  

  Hannah Nesser

  committed
  46b6dd3

  Copy full SHA for 46b6dd3

• Switched from HourlyCH4 to HourlySpecies and from ch4_run.template to <species>_run.template

  

  Hannah Nesser

  committed
  0156426

  Copy full SHA for 0156426

• Changed name of TROPOMI data directory from data_TROPOMI to data_satellite

  

  Hannah Nesser

  committed
  af530d4

  Copy full SHA for af530d4

• Changed HourlyCH4 to HourlySpecies and ch4_run.template to <species>_run.template

  

  Hannah Nesser

  committed
  dafd54c

  Copy full SHA for dafd54c

Commits on Jun 28, 2024

• Added species argument to main options

  

  Hannah Nesser

  committed
  79127be

  Copy full SHA for 79127be

• Changed tropomiCache to satelliteCache and addded Species and SatelliteProduct arguments to the call to jacobian.py

  

  Hannah Nesser

  committed
  74d79df

  Copy full SHA for 74d79df

• Removed bug check print statements and fixed a few PBS specific bugs

  

  Hannah Nesser

  committed
  262a478

  Copy full SHA for 262a478

• Corrected calls to write_boundary_conditions to include new arguments

  

  Hannah Nesser

  committed
  275295b

  Copy full SHA for 275295b

• Changed tropomiCache to satelliteCache, data_TROPOMI default storage to data_satellite, and added if statements to the TROPOMI download section to only download TROPOMI if it is a methane inversion

  

  Hannah Nesser

  committed
  0013b7f

  Copy full SHA for 0013b7f

• Changed HourlyCH4 to HourlySpecies and BlendedTROPOMI to SatelliteProdcut

  

  Hannah Nesser

  committed
  3bc4444

  Copy full SHA for 3bc4444

• Changed HourlyCH4 to HourlySpecies

  

  Hannah Nesser

  committed
  2150008

  Copy full SHA for 2150008

• Changed <species>_run.template to run.template

  

  Hannah Nesser

  committed
  02090d7

  Copy full SHA for 02090d7

• Generalizing name of run template

  

  Hannah Nesser

  committed
  6a6f6de

  Copy full SHA for 6a6f6de

• Changed ch4_run.template to just run.template

  

  Hannah Nesser

  committed
  fd0c727

  Copy full SHA for fd0c727

• Local changes

  

  Hannah Nesser

  committed
  732e551

  Copy full SHA for 732e551

• Changing back to default config files

  

  Hannah Nesser

  committed
  9c4bc72

  Copy full SHA for 9c4bc72

• Merging dev and ICI branches before moving the ICI branch to a sub branch of dev

  

  Hannah Nesser

  committed
  d96bf30

  Copy full SHA for d96bf30

• Moving sbatch requests to use the submit_job function

  

  Hannah Nesser

  committed
  a8baba5

  Copy full SHA for a8baba5

• Adding False argument for the new viz Boolean

  

  Hannah Nesser

  committed
  f8a67e6

  Copy full SHA for f8a67e6

• Adding species arguments

  

  Hannah Nesser

  committed
  ae02566

  Copy full SHA for ae02566

• Removing specific CH4 references and replacing with {species} and doing the same for TROPOMI > satellite

  

  Hannah Nesser

  committed
  56eea96

  Copy full SHA for 56eea96

Commits on Jul 1, 2024

• Fixing comment string

  

  Hannah Nesser

  committed
  2086e7c

  Copy full SHA for 2086e7c

• Updating to include new flags from dev branch

  

  Hannah Nesser

  committed
  d24cb3e

  Copy full SHA for d24cb3e

• Removing condaEnv/condaFile refs and replacing with PythonEnv per the PBS requirements

  

  Hannah Nesser

  committed
  5243ef9

  Copy full SHA for 5243ef9

• Adjusting for SBATCH options that were previously not caught by convert_sbatch_to_pbs

  

  Hannah Nesser

  committed
  2864204

  Copy full SHA for 2864204

• Adding a check to see if the PBS -l site=needed option has been previously added

  

  Hannah Nesser

  committed
  9665bb4

  Copy full SHA for 9665bb4

• Printing out any SBATCH options not caught by the conversion script

  

  Hannah Nesser

  committed
  8a8b167

  Copy full SHA for 8a8b167

• Removed print statement that didn't really work

  

  Hannah Nesser

  committed
  15c139a

  Copy full SHA for 15c139a

Commits on Jul 2, 2024

• Removing activation of python, which we get instead from a designated python environment

  

  Hannah Nesser

  committed
  02e3747

  Copy full SHA for 02e3747

• Switched tabs to spaces for readability

  

  Hannah Nesser

  committed
  2190404

  Copy full SHA for 2190404

• Changed tabs to spaces and changed hard coded GEOSCHEM_VERSION to the variable specified in run_imi.sh

  

  Hannah Nesser

  committed
  2e1d6bc

  Copy full SHA for 2e1d6bc

• Changes to allow submit_job to take a SaveOutput true/false boolean

  

  Hannah Nesser

  committed
  32efc58

  Copy full SHA for 32efc58

• Added SaveOut option to submit_job

  

  Hannah Nesser

  committed
  ac774b0

  Copy full SHA for ac774b0

• Added SaveOut option to submit_job

  

  Hannah Nesser

  committed
  208ffe8

  Copy full SHA for 208ffe8

• Looks like I accidentally deletd a fi?

  

  Hannah Nesser

  committed
  62cf5c4

  Copy full SHA for 62cf5c4

• Added a missing fi

  

  Hannah Nesser

  committed
  7d5698f

  Copy full SHA for 7d5698f

• Updated config for Pleiades

  

  Hannah Nesser

  committed
  192eadf

  Copy full SHA for 192eadf

Commits on Jul 3, 2024

• Resolving merge conflicts with bugfix/jacobian-perturbation-fix

  

  Hannah Nesser

  committed
  b936c5b

  Copy full SHA for b936c5b

• Switching from Simulation CPUs/Memory to Requested CPUs/Memory

  

  Hannah Nesser

  committed
  94d65ee

  Copy full SHA for 94d65ee

• Removing commented out vestige

  

  Hannah Nesser

  committed
  35c4a09

  Copy full SHA for 35c4a09

• Removing lingering references to TROPOMI

  

  Hannah Nesser

  committed
  958b1ea

  Copy full SHA for 958b1ea

• Updating options to better match bugfix/jacobian-perturbation-fix config files

  

  Hannah Nesser

  committed
  69ec97a

  Copy full SHA for 69ec97a

• UseEmisSF and UseOHSF seem to have been removed as options

  

  Hannah Nesser

  committed
  acf033f

  Copy full SHA for acf033f

• - Bug fix for BCs being used in all simulations, not just regional simulations

  Show description for a156273
  

  Hannah Nesser

  committed
  a156273

  Copy full SHA for a156273

• Beginning to generalize HEMCO tracer changes for not just methane

  

  Hannah Nesser

  committed
  4d30edb

  Copy full SHA for 4d30edb

• Generalized function to be a function of species

  

  Hannah Nesser

  committed
  da7fe89

  Copy full SHA for da7fe89

• Changed from tabs to spaces for readability and removed portions of code (UseEmisSF/UseOHSF if statemetns and also default analyticalInv setting) that were removed in the jacobian bug fix branch

  

  Hannah Nesser

  committed
  e951e42

  Copy full SHA for e951e42

• Updating

  

  Hannah Nesser

  committed
  c9ca0c3

  Copy full SHA for c9ca0c3

• Merge resolution with dev branch

  

  Hannah Nesser

  committed
  4c87fbc

  Copy full SHA for 4c87fbc

Commits on Oct 22, 2024

• Merge remote-tracking branch 'origin/dev'

  laestrada

  committed
  03fd4ea

  Copy full SHA for 03fd4ea

Commits on Oct 23, 2024

• Merge remote-tracking branch 'origin/dev'

  laestrada

  committed
  6767732

  Copy full SHA for 6767732

Commits on Oct 31, 2024

• Merge remote-tracking branch 'origin/dev'

  laestrada

  committed
  e64b99b

  Copy full SHA for e64b99b

Commits on Nov 2, 2024

• Resolving merge conflicts

  

  Hannah Nesser

  committed
  c754c5a

  Copy full SHA for c754c5a