How to get the right time dimension of variable using series analysis wrapper?

[weiwilliam]

I would like to calculate the series analysis with the dataset below as an example using METplus 3.1 and MET 9.1 on Orion.
fcst files:
2015122100.nc
2015122106.nc
2015122112.nc
2015122118.nc
obs files:
20151221.nc (with 4 cycles available)

I've tried to setup METplus configuration file as below.
VALID_BEG=2015122100
VALID_END=2015122118
VALID_INCREMENT=6H
FCST_VAR1_LEVELS = "(0,0,,)"
OBS_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},,)"
But it looks like {valid?fmt=%Y%m%d_%H%M%S} only use the VALID_BEG to generate the time dimension.
Is there any approach to make it find the right time dimension in obs dataset?

[georgemccabe]

Hi @weiwilliam,

If you were to upgrade to using METplus 4.0, then you would be able to get the correct valid times from this configuration. This version is available on Orion. Setup instructions can be found here https://dtcenter.org/community-code/metplus/metplus-4-0-existing-builds. You will need to set the following in your METplus config file:

SERIES_ANALYSIS_RUNTIME_FREQ = RUN_ONCE_PER_INIT_OR_VALID

This will run the tool once per valid time and substitute the correct valid time into OBS_VAR1_LEVELS for each run.

I recommend upgrading to v4.0, but if you must use METplus 3.1, please let me know. I believe you can configure the tool using that version to process these files by making a few changes including looping by initialization time, providing a forecast lead list, and modifying the filename templates.

[georgemccabe]

I don't have access to Orion, but I can have someone who does pull the files for me if that is easier for you. Another option is to upload the files to our FTP. Instructions on how to do that can be found here: #954

Here is the GitHub issue for this work: #1166
I will update this issue with more information once we come up with a good solution to handle this situation.

[weiwilliam]

Got it. I've created the tarball and uploaded it to FTP site.
It's test_data.tgz under Wei_data.
Under data folder, you'll find two subfolders.
'fv3' keeps 4 fcst files and 'cams' keeps 1 obs file for the 4 cycles of 20151221
I also include the correct log file that found 4 fcst files and 1 obs file under 'logs'.
Thank you.

[georgemccabe]

It appears that there is something wrong with that tarfile. I get an error when I uncompress it and I only see the cams data, not the fv3 data. Could you regenerate the tarfile and upload it again?

[weiwilliam]

I uploaded the uncompressed folder. Please check if there is any problem, thanks.

[georgemccabe]

Thanks, I was able to get all of the data. Once the work is completed I will update this discussion with information on how to set up your config files.

[georgemccabe]

Hi @weiwilliam, I am going to mark this discussion as answered and lock it to prevent commenting. Please follow GitHub issue #1166 for this work for any updates related to the enhancements needed to fix this issue.

[georgemccabe]

Hi @weiwilliam , I am unlocking this discussion to give you an update. I was able to process your data with SeriesAnalysis. The changes required to get this to work are included in pull request #1353 and will be available in the METplus 4.1.0-beta5 release that will be created within a week. Here are a few of the settings that I changed from the config file you provided to get it to work:

SERIES_ANALYSIS_RUNTIME_FREQ = RUN_ONCE
FCST_VAR1_LEVELS = "(0,0,*,*)"
FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = fv3_aods_{valid?fmt=%Y%m%d%H}_ll.nc
OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = cams_aods_{valid?fmt=%Y%m%d}.nc

I also removed SERIES_ANALYSIS_CONFIG_FILE because you do not need to set this anymore. Leaving it unset will use the wrapped config file provided with the repository.

Please let me know if you have any other issues or questions.

[georgemccabe]

Hi @weiwilliam, I am just following up on this issue. Were you able to test to see if these changes work for your use case?

[weiwilliam]

Hi @georgemccabe, I haven't got a chance to test it.
Which platform has METplus 4.1.0-beta5?
I checked that is not available to load on Orion a few days ago.

[jprestop]

Hi @weiwilliam.

I have just installed METplus-4.1.0-beta5 on Orion. I have updated our Existing Builds page to reflect this update.

For quick reference, you can load the following to access this beta5 release:

module use /apps/contrib/modulefiles
module load metplus/4.1.0-beta5

Please let us know how it goes when you test the changes. Thanks!

[weiwilliam]

Hi @georgemccabe and @jprestop , it works. Appreciate that you make this available. Thanks!