MET 8.1: Frequency of Superior Performance with TC-Stat

[WillySedlacek-16WS]

Good afternoon,

I'm wondering how to appropriately use the TC-Stat summary job in MET 8.1 to obtain frequency of superior performance for a list of a-deck models. I'm verifying eight amodels, using the following job summary statement in the config file:

"-job summary -line_type TCMPR -column TK_ERR -by AMODEL,LEAD"

I hoped this would give me each amodel's frequency of superior performance for TK_ERR at each lead time. For example, the frequency that a given model's 12-hour track forecast error was better than that of the other seven models. However, instead, the FSP_TOTAL, FSP_BEST, and FSP_TIES columns are all filled with 0s. How should I change my job summary statement to get the desired result?

Please let me know if I can provide any additional information or clarification.

Thank you!
-Willy

[JohnHalleyGotway]

Hi Willy, I see you have a question about computing the Frequency of Superior Performance (FSP) for the tc_stat tool in MET version 8.1. As you note, these columns are included in the "summary" job type for tc_stat.

I've attached a sample data file, that's output from the tc_pairs tool, to demonstrate:
alal2010_V8.1.tcst.txt
Note that I added a ".txt" suffix to this file so that GitHub would enable me to upload it.

I tried running a simple command line job like you did, and also got all 0's in the FSP columns:

/usr/local/met-8.1.1/bin/tc_stat -lookin alal2010_V8.1.tcst.txt -job summary -column TK_ERR -by AMODEL

The key detail here is that FSP can only be computed when you have exactly the same cases for each model. To enable the event equalization logic, you need to add the "-event_equal TRUE" job command option (it's false by default). So adding that I get:

/usr/local/met-8.1.1/bin/tc_stat -lookin alal2010_V8.1.tcst.txt -job summary -column TK_ERR -by AMODEL -event_equal TRUE
DEBUG 2: Job 1 used 0 out of 0 lines read.

But note that all of the lines are discarded because there are no cases common to ALL of the models present. Next, instead of looking at all the models present, I'll only look at 2 of them that have a lot of cases.

/usr/local/met-8.1.1/bin/tc_stat -lookin alal2010_V8.1.tcst.txt -job summary -column TK_ERR -by AMODEL -event_equal TRUE -amodel AHWI,BCLP
DEBUG 1: [Source 1] Lookin: alal2010_V8.1.tcst.txt
DEBUG 2: 
DEBUG 2: Processing Job 1: -job summary -amodel AHWI -amodel BCLP -event_equal true -column TK_ERR -by AMODEL -out_alpha 0.05000 
JOB_LIST: -job summary -amodel AHWI -amodel BCLP -line_type TCMPR -event_equal true -column TK_ERR -by AMODEL -out_alpha 0.05000 
COL_NAME: COLUMN AMODEL TOTAL VALID      MEAN MEAN_NCL  MEAN_NCU     STDEV MIN      P10      P25      P50       P75       P90       MAX       IQR     RANGE        SUM TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES     FSP
SUMMARY:  TK_ERR   AHWI   576    64 119.95974 83.30427 156.61521 149.61692   0 10.17486 24.43622 59.01506 172.72925 277.877   697.46641 148.29303 697.46641 7677.42348     NA     NA        64       19        4 0.29688
SUMMARY:  TK_ERR   BCLP   576    64 119.74625 72.67151 166.82099 192.14534   0  0        4.01666 28.47715 130.75262 388.33615 884.95536 126.73596 884.95536 7663.76011     NA     NA        64       41        4 0.64062

And that does compute FSP values of .29688 and .64602 for the AHWI and BCLP models, respectively. Those add up to .9429 which means that "ties" occur at a rate of .0571.

So I suspect that you'll need to add the "-event_equal TRUE" job command option. If all of the cases get tossed out, consider using "-amodel" to filter down to a subset of models for which you have plenty of cases.