Skip to content

Commit

Permalink
Feature/parallelize preview: corresponding pr for updating imi previe…
Browse files Browse the repository at this point in the history
…w to parallelize (#31)

* update preview to accept cpu count param

* take advantage of multiple cores for preview when using sbatch

* add commit for imi preview changes

* set default vscode formatter to black
  • Loading branch information
laestrada authored Feb 7, 2022
1 parent 9c55038 commit 8c95aa8
Show file tree
Hide file tree
Showing 3 changed files with 20 additions and 3 deletions.
3 changes: 3 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"python.formatting.provider": "black"
}
2 changes: 1 addition & 1 deletion PostprocessingScripts/CH4_TROPOMI_INV
Submodule CH4_TROPOMI_INV updated 1 files
+264 −137 imi_preview.py
18 changes: 16 additions & 2 deletions setup_ch4_inversion.sh
Original file line number Diff line number Diff line change
Expand Up @@ -541,8 +541,22 @@ if "$DoPreview"; then
state_vector_path=${MyPath}/${RunName}/StateVector.nc
preview_dir=${MyPath}/${RunName}/${runDir}
tropomi_cache=${MyPath}/${RunName}/data_TROPOMI
python ${InversionPath}/PostprocessingScripts/CH4_TROPOMI_INV/imi_preview.py $config_path $state_vector_path $preview_dir $tropomi_cache

preview_file=${InversionPath}/PostprocessingScripts/CH4_TROPOMI_INV/imi_preview.py

# if running end to end script with sbatch then use
# sbatch to take advantage of multiple cores
if "$UseSlurm"; then
# set number of cores to run preview with
if "$isAWS"; then
sed -i -e "s:#SBATCH -c 8:#SBATCH -c ${cpu_count}:g" ${InversionPath}/PostprocessingScripts/CH4_TROPOMI_INV/imi_preview.py
fi
cd ${InversionPath}/PostprocessingScripts/CH4_TROPOMI_INV/
chmod +x $preview_file
sbatch -W $preview_file $config_path $state_vector_path $preview_dir $tropomi_cache $cpu_count; wait;
cd $runDir
else
python $preview_file $config_path $state_vector_path $preview_dir $tropomi_cache $cpu_count
fi
printf "=== DONE RUNNING IMI PREVIEW ===\n"

# Escape condition for DOFS threshold? Read diagnostics file for expectedDOFS variable
Expand Down

0 comments on commit 8c95aa8

Please sign in to comment.