[Bug]: prov log doesn't include complete traceback

[tomvothecoder]

What happened?

Example of incomplete log (prov dir): https://portal.nersc.gov/project/e3sm/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/prov/e3sm_diags_run.log

Example of complete log (manually copied from terminal): https://portal.nersc.gov/project/e3sm/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/prov/complete_run.log

What did you expect to happen? Are there are possible answers you came across?

I don't think the Python log handlers are correctly streaming and/or concatenating the output to a single log file. It seems like the outputs are being overwritten instead.

Minimal Complete Verifiable Example (MVCE)

No response

Relevant log output

2025-01-06 10:55:36,294 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: T
2025-01-06 10:55:42,931 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:55:43,286 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:55:43,835 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:55:44,276 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global.json
2025-01-06 10:55:44,920 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global.png
2025-01-06 10:55:44,921 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: U
2025-01-06 10:55:52,669 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:55:53,020 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:55:53,567 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:55:53,870 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global.json
2025-01-06 10:55:54,481 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global.png
2025-01-06 10:55:54,481 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: RELHUM
2025-01-06 10:56:07,151 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:56:07,496 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:56:08,101 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:56:08,403 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global.json
2025-01-06 10:56:09,074 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global.png
2025-01-06 10:56:09,074 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: OMEGA
2025-01-06 10:56:24,167 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:56:24,508 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:56:24,985 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:56:25,289 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global.json
2025-01-06 10:56:25,938 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global.png
2025-01-06 10:56:25,938 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: T
2025-01-06 10:56:47,765 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:56:48,303 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:56:49,629 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:56:49,929 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-T-ANN-global.json
2025-01-06 10:56:50,560 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-T-ANN-global.png
2025-01-06 10:56:50,560 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: U
2025-01-06 10:57:14,053 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:57:14,557 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:57:15,864 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:57:16,157 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-U-ANN-global.json
2025-01-06 10:57:16,786 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-U-ANN-global.png
2025-01-06 10:57:16,786 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: OMEGA
2025-01-06 10:57:46,578 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:57:47,099 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:57:48,418 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:57:48,704 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global.json
2025-01-06 10:57:49,369 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global.png
2025-01-06 10:57:49,369 [INFO]: meridional_mean_2d_driver.py(run_diag:60) >> Variable: RELHUM
2025-01-06 10:58:17,731 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:58:18,239 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-01-06 10:58:19,524 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/xesmf/smm.py:131: UserWarning: Input array is not C_CONTIGUOUS. Will affect performance.
  warnings.warn('Input array is not C_CONTIGUOUS. ' 'Will affect performance.')

2025-01-06 10:58:19,818 [INFO]: io.py(_save_data_metrics_and_plots:77) >> Metrics saved in /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global.json
2025-01-06 10:58:20,439 [INFO]: utils.py(_save_plot:91) >> Plot saved in: /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global.png
2025-01-06 10:58:20,440 [INFO]: main.py(create_viewer:132) >> lat_lon /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/viewer
2025-01-06 10:58:21,022 [WARNING]: warnings.py(_showwarnmsg:112) >> /global/u2/v/vo13/mambaforge/envs/eamxx_1024/lib/python3.12/site-packages/e3sm_diags/plot/taylor_diagram.py:93: UserWarning: No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
  ax.legend(loc="center left", bbox_to_anchor=(1, 0.5))

2025-01-06 10:58:25,823 [INFO]: main.py(create_viewer:135) >> [('Latitude-Longitude contour maps', 'lat_lon/index.html'), ('Table', 'table/index.html'), ('Taylor Diagram', 'taylor/index.html'), ('CMIP6 Comparison', 'cmip6/index.html')]
2025-01-06 10:58:25,823 [INFO]: main.py(create_viewer:132) >> zonal_mean_xy /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/viewer
2025-01-06 10:58:25,978 [INFO]: main.py(create_viewer:135) >> ('Zonal mean line plots', 'zonal_mean_xy/index.html')
2025-01-06 10:58:25,978 [INFO]: main.py(create_viewer:132) >> zonal_mean_2d /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/viewer
2025-01-06 10:58:26,145 [INFO]: main.py(create_viewer:135) >> ('Pressure-Latitude zonal mean contour plots', 'zonal_mean_2d/index.html')
2025-01-06 10:58:26,146 [INFO]: main.py(create_viewer:132) >> zonal_mean_2d_stratosphere /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/viewer
2025-01-06 10:58:26,240 [INFO]: main.py(create_viewer:135) >> ('Pressure-Latitude zonal mean contour plots (Stratosphere)', 'zonal_mean_2d_stratosphere/index.html')
2025-01-06 10:58:26,240 [INFO]: main.py(create_viewer:132) >> polar /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/viewer
2025-01-06 10:58:26,418 [INFO]: main.py(create_viewer:135) >> ('Polar contour maps', 'polar/index.html')
2025-01-06 10:58:26,418 [INFO]: main.py(create_viewer:132) >> meridional_mean_2d /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/viewer
2025-01-06 10:58:26,495 [INFO]: main.py(create_viewer:135) >> ('Pressure-Longitude meridional mean contour plots', 'meridional_mean_2d/index.html')
2025-01-06 10:58:26,500 [INFO]: e3sm_diags_driver.py(main:392) >> Viewer HTML generated at /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/892-bottleneck/eamxx_decadal_1996_1107_edv3/viewer/index.html

Anything else we need to know?

No response

Environment

Latest e3sm_diags

[chengzhuzhang]

I noticed this too. And the issue with repeated logger messages is gone!

[tomvothecoder]

And the issue with repeated logger messages is gone!

Nice! I don't think anything changed with the logger code. Hope it doesn't randomly appear again.