ICP and MesoVICT

  • About
  • News
  • MesoVICT Cases
  • MesoVICT Model Re-Runs
  • ICP Cases
  • Meetings
  • Software
  • References
  • MesoVICT Papers
  • Working Groups
  • Participants
  • Contact

MesoVICT: Mesoscale Verification Intercomparison over Complex Terrain (Phase 2 of the ICP)

About the ICP and MesoVICT

The spatial forecast verification Inter-Comparison Project (ICP) and follow-on MesoVICT project sifted through the maze of newly proposed methods for verifying primarily high-resolution forecasts against analysis products on the same regular grid, as well as to help forge a consistent set of cases to test methods that may be proposed in the future so as to have a consistent set of results that can be cross compared.

Papers describing the original ICP: Ahijevych et al. (2009; doi: 10.1175/2009WAF2222298.1), Gilleland et al. (2009; doi: 10.1175/2009WAF2222269.1) and Gilleland et al. (2010; doi: 10.1175/2010BAMS2819.1).

Papers describing MesoVICT: Dorninger et al. (2013; doi: 10.5065/D6416V21) and Dorninger et al. (2018; doi: 10.1175/BAMS-D-17-0164.1), and Gilleland et al. (2020; 10.1175/MWR-D-19-0256.1; for the more recently added idealized test cases).

Although the project officially ended with the final workshop that took place 8 - 9 July 2019 in Vienna, a new set of idealized test cases have more recently been developed. These test cases are aimed at methods that attempt to inform about spatial-alignment errors. The paper Gilleland et al. (2020, doi: 10.1175/MWR-D-19-0256.1) provides a description of these cases and the various test sets can be obtained from this web page under the "MesoVICT Cases" tab in both R and netCDF formats.

News

The 2nd International Verification Challenge to develop and demonstrate the best new verification metric using non-traditional observations is open for entries for a few more months. Forecast verification is evolving beyond traditional metrics for basic weather variables to make use of many new sources of data from new sensors, social media, photos and other smart phone data, and user-relevant variables such as energy output, crop yield, and so on, to assess forecast quality. To encourage the development of forecast verification approaches that use new sources and types of observations, the World Meteorological Organization's Joint Working Group on Forecast Verification Research is conducting a challenge to develop and demonstrate new forecast verification metrics that make use of non-traditional observations. The deadline for entries is 30 April 2021. The winner will be announced in June 2021 and will receive all expense paid attendance and a keynote address at the 8th International Verification Methods Workshop.

For more information and an entry form, please visit https://community.wmo.int/news/2nd-international-verification-challenge or write to verifchallenge@bom.gov.au.

New spatial-alignment summary measures for forecast performance introduced in Gilleland (2021) are now included in version 0.6-6 of the R package SpatialVx, as well as a circle histogram function (rose diagram). The additional G function will be available with the soon to be released version 0.8, which also includes new functions from Gregor Skok.

New test cases introduced in Gilleland (2017) and Gilleland et al. (2020) are now available (in R and netCDF formats) under the MesoVICT Cases tab.

The final MesoVICT workshop was held 8 - 9 July 2019 in Vienna, Austria. Please visit the workshop's website at https://mesovict.univie.ac.at/.

Contents

An example of how to read some of the test cases into R

For further model data that go beyond the MesoVICT project please visit the World Data Center for Climate in Hamburg, Germany, where all model output (and other data) from the DPHASE project are stored.

MesoVICT cases

For groups wishing to provide model re-runs of the test cases below (i.e., test newer model versions on these dates), please contact the organizing committee. Note that our purpose is not to verify your models (this is a meta-verification project), but it may be possible to work with individual participants who might be interested in doing so. In such cases, it will not be possible to store the data on this website, but we will be happy to provide a link to the data.

Information

Please refer to the Technical Note for the project to learn about the test cases for MesoVICT. Note that COSMO2 06 run is not available for the core case, so we will use the 00 run for the core case (for this model only) instead; the CMC GEMH is only available at 06, which is why we chose to use the 06 runs generally. For all other cases, we will use the 06 runs. See also Bauer et al. (2011) for more information about the data sets generally.

The figure below shows a schematic diagram of the tiered case structure for the project.

Vienna Enhanced Resolution Analysis (VERA)

Description of the VERA cases file format (pdf). The lon/lat coordinates are in this text file.

Topography information is given in VERA_topography.dat, with a description of this file in Description_of_VERA_topography_file_format.pdf

The six VERA test case analyses are each available in a separate zip file linked below:

Back to top

Point Observations: Joint D-PHASE - COPS (JDC) data set

Description of the JDC cases file format (pdf).

The six JDC test case point observation files are linked below:

Back to top

Model Output

Description of model output file format for all models (pdf).

Note that COSMO2 06 hour runs are not available for cases 1 and 6. Because case 1 is a core case, please use COSMO2 00 hour run for this case instead; as though it were the 06 hour run.

MeteoSwiss plans to recalculate these cases with the current version of their models (experimental COSMO-1, operational COSMO-2, and COSMO-7, which have grid meshes of 1.1, 2.2 and 6.6 km, respectively). These re-run cases should be available in the coming months.

Canadian high-resolution model: CMC GEMH (GEM-LAM) model output from Environment Canada

Back to top

COSMO2 06 Hour Runs

COSMO2 00 Hour Runs

Back to top

VERA ensembles

Description of VERA ensemble File Format

VERA_ensemble_8km_coordinates_lam_phi.txt

COSMO-LEPS (CLEPS)

Description of the CLEPS file formats.

Back to top

New Geometric Test Cases

New test cases introduced in Gilleland (2017) and Gilleland et al. (2019) are now available (in R and netCDF format) under the MesoVICT Cases tab.

Gilleland, E., 2017. A new characterization in the spatial verification framework for false alarms, misses, and overall patterns. Weather Forecast., 32 (1), 187 - 198, doi: 10.1175/WAF-D-16-0134.1.

Gilleland, E., G. Skok, B. G. Brown, B. Casati, M. Dorninger, M. P. Mittermaier, N. Roberts, and L. J. Wilson, 2019. A novel set of verification test fields with application to distance measures. Accepted to Monthly Weather Review (doi: 10.1175/MWR-D-19-0256.1; temporarily available at: pdf).

netCDF format

R format (links to each set of test cases below): use

data( "[path]/filename.R" )

Each R object is an array. For example, to get case C3, use:

Circles[ , , 3 ].

Back to top

Model Re-Runs

List of output variables for atmospheric model re-runs

ISPRA re-runs of all MesoVICT cases are now available. Re-runs have been conducted using the current operational version of the ISPRA meteorological models belonging to the SIMM forecasting system (fully updated version of the NWP model used during MAP D-PHASE), namely the hydrostatic BOLAM model (with horizontal grid spacing of 0.07o and 0.1o ) and the non-hydrostatic MOLOCH model (with horizontal grid spacing of 0.00225o), and they can be used for any additional experiments in the framework of MesoVICT. To obtain them, please contact Marco Casaioli or Stefano Mariani at simm-pre-meteo@isprambiente.it

The new COSMO-1 model re-runs provided by MeteoSwiss are available. Runs have been conducted for the core case and cases 2 and 6. Da ta can be accessed via the University of Vienna on request. Please contact Manfred.Dorninger@univie.ac.at

ICP cases

The primary ICP cases are also available via the ICPtestcases R package (ICPtestcases_1.0.tar.gz). After unzipping and untarring, use the command

R CMD INSTALL ICPtestcases -l library

to install the package on your system (in a folder called library, which needs to exist in before calling this command).

The test cases for the first phase of the ICP included geometric idealized, perturbed real, and real cases. There were some additional cases as well, but they were not the primary focus. These cases are available and described below.

Information

Please refer to Ahijevych et al. (2009) for a thorough description of the ICP test cases. Some information is given below.

We had three sets of test cases picked out for phase 1 of this project. Thanks to Mike Baldwin, Beth Ebert, Barbara Casati and others for helping to obtain them, and to David Ahijevych for creating the fake test cases. All of the cases are included for download by clicking on the link to this file: Cases20081023.tar.gz

Once extracted you will have the following folders inside the folder called Cases: 'sp2005,' 'nimrod,' 'MeteoSwiss,' and 'Fake.' Inside 'Fake' are further folders for the geometric and perturbed cases. Many of these test cases are also available with the R package SpatialVx.

Initial Test Cases from the Storm Prediction Center/National Severe Storms Laboratory (SPC/NSSL) Spring 2005 Program (sp2005)

We have cases from the WRF Spring 2005 program (see Kain et al., 2008 Weather Forecast., 23:931--952) including stage II analyses and three versions of WRF referred to as 'wrf2,' 'wrf4ncar' and 'wrf4ncep.' The fields are hourly accumulated precipitation in hundredths-of-an-inch. Please convert to mm by multiplying by 0.254 so that we all use the same units. All of these models have been interpolated onto the same 501 x 601 grid. Specifically, the NCEP g240 grid, which is a "4-km grid," but note that the grid is irregular (i.e., not always 4-km at each grid square). The longitude and latitude information is provided in the sp2005 folder as 'g240LatNorth.txt' and 'g240LonEast.txt.'

In the 'figures' folder you will find postscript and pdf graphs in the 'ps' and 'pdf' folders, respectively, of the first 9 cases chosen from this Spring program for this inter-comparison. Note that there are a couple extra cases in the 'st2,' 'wrf2' and 'wrf4ncar' folders. We did not have these dates for the NCEP model, so we will exclude them, but they are still provided as they may make for nice cases. All files are in ASCII format. See the README file in this folder for more information.

The cases we will start with are for verification times (00Z issue times): 4/26, 5/13, 5/14, 5/18, 5/19, 5/25, 6/1, 6/3 and 6/4.

The entire 32 Spring 2005 ARW and NMM cases studied in Davis et al (2009), Gilleland et al (2010) and Gilleland (2013) are available here (spc2005agg24hLead00UTC.tar.gz).

Nimrod Test Cases

Please acknowledge the UK Met Office (see the README file in the 'Nimrod' folder. These are the same cases used in the Casati et al (2004) paper (Meteorol. Appl. 11:141--154), and are described therein. We do not presently have lon/lat info for these cases (sorry).

MeteoSwiss Test Cases

Daniel Leuenberger at MeteoSwiss has offered two cases we could consider using. He asks that we acknowledge their source, and use them only for research purposes (see the 'README' file in the 'MeteoSwiss' folder). The cases give high-resolution aLMo radar data over Switzerland, along with hourly forecasts for a 24-hour period. One case also has results on a 7-km grid corresponding to a lower resolution version of the model. The files are in netCDF format, and IDL code to read them is included. One case has a postscript file that shows the hourly fields and basic verification statistics. The other case has a postscript file for the 24-hr totals, plus gif files for 6h totals.

Fake Test Cases

Fake test cases are based on the same grid as the sp2005 cases. There are both idealized geometric cases as well as perturbed real cases. See the README files for each case for more information.

Past MesoVICT Meetings

The final MesoVICT workshop was held 8 - 9 July 2019 in Vienna, Austria. The workshop's website: https://mesovict.univie.ac.at/.

There was a MesoVICT session at the WMO Joint Working Group for Forecast Verification (JWGFVR) International Verification Workshop held in Berlin 8 - 11 May 2017.

The 2nd MesoVICT Workshop. 21 - 23 September 2016, Bologna, Italy.

MesoVICT session (NWP7) at the 16th EMS Annual Meeting and 11th European Conference on Applied Climatology (ECAC) Trieste, Italy, 12 - 16 September 2016.

The 2015 meeting was held as a session at the 15th European Meteorological Society (EMS) annual meeting that took place 7 - 11 September 2015 in Sofia, Bulgaria. Our session was part of the Numerical Weather Prediction programme group (NWP6).

  • Bundel et al. INtercomparison of Spatial vErification methods for COSMO Terrain (INSPECT): Preliminary results (pdf)
  • Casati et al. An intensity-scale score to assess the added value of enhanced resolution (pdf)
  • Dorninger. Towards a simple and easy to interpret wind verification scheme (pdf)
  • Mittermaier, M. Exploring the impact of neighbourhood size and height stratification in neighbourhood-based methods using single-site observations (pdf, pptx)
  • Weniger, M. and P. Friederichs, A wavelet approach to the verification of probabilistic spatial fields (pdf)

The MesoVICT kick-off meeting was held 2 - 3 October 2014 in Vienna, Austria.

First Planning/information meeting for the second phase of the ICP (SM9) was held on Tuesday, 10 September 2013 at the 13th EMS and 11th ECAM Conference, Reading, UK. Relevant talks from the meeting: Overview of ICP and phase 2 by Marion Mittermaier and Introduction to the next set of test cases by Manfred Dorninger.

Past ICP Meetings

Fall 2009 Workshop: 24-25 August 2009 at NCAR's Center Green Campus, Boulder, Colorado.

Spring 2008 workshop held in Boulder, Colorado 14 - 15 April 2008.

20 February 2007 Planning Meeting (includes slides from the various talks).

Preliminary Results

An old link to some early results that may still be useful (mostly from past meeting presentations).

We will provide links to available software for performing spatial forecast verification from this page as they become available. All references listed below can be found in-full by clicking on the references tab above.

SpatialVx

A new R package called SpatialVx is in the works. It already has functions to do many of the newly proposed methods, but there are many yet to do. Keep checking for new updates.

Traditional Verification

The following book has a nice appendix by Matt Pocernich on verification software (pp. 231--240). Also see Forecast Verification Issues, Methods and FAQ. From these, it is clear that several software tools for traditional verification are available. Listed here are only tools that were developed specifically at NCAR.

Jolliffe, I.T. and Stephenson, D.B., Forecast Verification: A practioner's guide in atmospheric sciences, Second Edition, Wiley-Balckwell, Chichester, West Sussex, U.K., 274 pp.

R package verification by Matt Pocernich. From your R session, you can install the package onto your machine with the install.packages function. You can then load it into your R session using the library function.

Model Evaluation Tools (MET) user's home page

SpatialVx contains some functionality for doing the traditional forecast verification methods.

Features-based Approaches

Method for Object-based Diagnostic Evaluation (MODE) tool (Davis et al., 2006). Available through MET user's home page.

SpatialVx has some functionality for doing the method proposed by Davis et al. (2006), as well as the SAL technique (Wernli et al., 2008). In particular, it is possible to do either convolution thresholding, or just thresholding to identify features (or objects) as connected grid points above the threshold. It also has functions to calculate properties of single features (major/minor axis angle/length, aspect ratio, area, centroid) and properties for matched features (e.g., intersection area, area ratio, centroid distance, angle difference, as well as numerous binary image metrics; see below). More is on tap for future releases.

Distance-based Approaches

Baddeley's Delta Metric: R code for computing Baddeley's delta metric for binary images (see, e.g., Gilleland, 2011) and the Hausdorff distance can be found in the R package spatstat and SpatialVx (via spatstat) as well as in MET. The mean-error distance (MED, see Gilleland 2017), Pratt's Figure of Merit, centoid distance, and partial Hausdorff distance (see Gilleland et al. 2020 for all of these) are available in SpatialVx and MET. The forecast quality index (FQI, see Venugopal et al. 2005, Basu et al. 2003, and Gilleland 2020) is available in SpatialVx.

Gβ, Gβ,IL and G2,β,IL (see Gilleland 2020) are available in SpatialVx.

SpatialVx additionally contains the minimum separation distance. See the help file for locperf for more on all of these distance-based measures.

MatLab code for calculating FQI (Venugopal et al, 2005; Basu et al, 2003).

Fields Deformation Approaches

The optical flow procedure of Marzban and Sandgathe (2010) is available in SpatialVx. Some rudimentary image warping functionality is available in SpatialVx (version >= 0.7).

Neighborhood-Based Approaches (summarized in Ebert, 2008)

The R package SpatialVx has functions for doing most of the neighborhood methods.

Scale Decomposition Approaches

Intensity-Scale (IS) (Casati et al, 2004). Available in the R package verification (above under traditional verification). Also available from the Model Evaluation Tools (MET) software.

SpatialVx has the wavelet denoising (more a neighborhood type approach), and wavelet decomposition approaches of Briggs and Levine (1997), as well as the intensity-scale techniques of Casati et al. (2004) and Casati (2009) largely via the R package waveslim.

Other

Validation Toolbox

Software from UC Irvine's Center for Hydrology and Remote Sensing (CHRS) Department of Civil and Environmental Engineering:

http://amir.eng.uci.edu/downloads/ValidationToolbox.zip

Includes: Performance Metrics for Evaluation of Remote Sensing Observations and Climate Model Simulations: A simple and easy to use Validation Toolbox (MATLAB source code) that can be used for validation of gridded data including satellite observations, reanalysis data, and weather and climate model simulations. In addition to the commonly used categorical indices, the toolbox includes the Volumetric Hit Index (VHI), Volumetric False Alarm Ration (VFAR), Volumetric Missed Index (VMI), and Volumetric Critical Success Index (VCSI).

References

The list of references was getting to be too long to list here. Please find them at this link.

Many of the references are also available in BibTeX format, where the cite key follows the format: AuthorLastNameYear (one author; e.g., Casati2009), Author1LastNameAuthor2LastNameYear (two authors; e.g., BriggsLevine1997), Author1LastNameEtAlYear (more than two authors; e.g., AhijevychEtAl2009). If the previous does not identify a unique paper in the list (e.g., Davis et al., 2006), then a lower case letter a, b, c, ... is appended to the end (e.g., DavisEtAl2006a and DavisEtAl2006b).

Dorninger, M., E. Gilleland, B. Casati, M. P. Mittermaier, E. E. Ebert, B. G. Brown, and L. Wilson, 2018: The set-up of the Mesoscale Verification Inter-Comparison over Complex Terrain project. Bull. Amer. Meteorol. Soc., 99 (9), 1887 - 1906, doi: 10.1175/BAMS-D-17-0164.1.

Dorninger, M., M. P. Mittermaier, E. Gilleland, E. E. Ebert, B. G. Brown, and L. J. Wilson, 2013: MesoVICT: Mesoscale Verification Inter-Comparison over Complex Terrain. NCAR Technical Note NCAR/TN-505+STR, 23 pp, DOI: http://dx.doi.org/10.5065/D6416V21.

Geiß, S., 2015: Comparison of spatial verification methods. Bachelor thesis. Ludwig-Maximilians-University Munich Meteorological Institute Munich, Supervisor: Christian Keil, 55 pp. (pdf)

Gilleland, E., 2021. Novel measures for summarizing high-resolution forecast performance. Advances in Statistical Climatology, Meteorology and Oceanography, 7 (1), 13 - 34, doi:10.5194/ascmo-7-13-2021.

Gilleland, E., 2017: A new characterization in the spatial verification framework for false alarms, misses, and overall patterns. Weather Forecast. 32 (1), 187 - 198, DOI: 10.1175/WAF-D-16-0134.1.

Kloiber, S., 2017: Verification in Complex Terrain with Ensemble-Analysis. M. S. thesis, Dep. of Meteorology and Geophysics, University of Vienna, 74 pp.

Mariani, S. and M. Casaioli, 2018: Effects of model domain extent and horizontal grid size on contiguous rain area (CRA) analysis: A MesoVICT study. Meteorologische Zeitschrift, 27 (6), 481 - 502, doi: 10.1127/metz/2018/0897.

Radanovics, S., J.-P. Vidal, and E. Sauquet, 2018: Spatial verification of ensemble precipitation: An ensemble version of SAL. Weather and Forecasting, 33, 1001 - 1020, doi: 10.1175/WAF-D-17-0162.1.

Skok, G., and V. Hladnik, 2018: Verification of Gridded Wind Forecasts in Complex Alpine Terrain: A New Wind Verification Methodology Based on the Neighborhood Approach. Mon. Wea. Rev., 146, 63 - 75, https://doi.org/10.1175/MWR-D-16-0471.1.

We have divided the project into working groups to focus on specific tiers of MesoVICT (see image below). The groups and their leaders are listed below. Participants may work in more than one of these groups, if desired, but note that they all overlap with WG1 and WG3 with WG2.

Working Group 1 (Core Case)

Lead: Eric Gilleland

Brief description: Working Group 1 will focus on the core case (the minimal amount). For those participants merely looking to learn about the various existing methods, this working group would be the appropriate one to join.

Working Group 2 (Core Case + Tier 1)

Lead: Marion Mittermaier

Brief description: In addition to the core case (overlapping with WG1), Working Group 2 will focus on the wind variable and ensembles of forecast models.

Working Group 3 (Core Case + Tier 1 to Tier 3)

Lead: Manfred Dorninger

Brief description: In addition to the core case and tier 1, this group will fully investigate the use of ensembles of observations, as well as other models and variables.

Working Group 4 (NWP model re-runs)

Lead: Stefano Mariani (stefano "dot" mariani "at " isprambiente.it)

Brief description: Working Group 4 is for those planning to do NWP model re-runs in order to test newer models for the same cases.

The list of participants under ICP phase 1 below was created post priori from the list of authors of papers in the ICP special collection of Weather and Forecasting. Please contact us with any errors.

MesoVICT Participants*

Organizing committee in bold.

Amir AghaKouchak Barbara G. Brown Randy G. Bullock
Anastasia Bundel Barbara Casati George C. Craig
Alex Dekmyn Manfred Dorninger Izuchukwu Ebenebe
Elizabeth E. Ebert Jian Fan Eric Gilleland
Christian Keil Chiara Marsigli Caren Marzban
Marion P. Mittermaier Andrea Montani Jason E. Nachamkin
Phu Nguyen Tiziana Paccagnella Sabrina Radanovics
Nigel Roberts Francis Schubiger Scott Sellars
Gregor Skok Geert Smet Maria Stefania Tesini
Susanne Theis Emiel van der Plas Bert van Schaeybroeck
Volker Wulfmeyer Laurence Wilson
*The above list is an initial list of folks who have already expressed interest in participating and is subject to change. Please let Eric Gilleland know if your name is there, but should not be, or if your name is not on the list, but should be. Thank you!
ICP phase 1 Participants
David A. Ahijevych Keith F. Brill Barbara G. Brown
Randy G. Bullock Barbara Casati George C. Craig
Christopher A. Davis Elizabeth E. Ebert Neil I. Fox
Eric Gilleland John Halley Gotway Christiane Hofmann
John S. Kain Christian Keil Steven A. Lack
Valliappa Lakshmanan Nicholas Lederer George L. Limpert
Finn Lindgren Johan Lindström Hilary Lyons
Caren Marzban Fedor Mesinger Marion P. Mittermaier
Jason E. Nachamkin Nigel Roberts Scott Sandgathe
Heini Wernli Matthias Zimmer

Contact Information

MesoVICT Committee Members

Eric Gilleland (webmaster)

Marion Mittermaier (marion.mittermaier " at " metoffice.gov.uk)

Barb Brown

Barbara Casati

Manfred Dorninger

Beth Ebert

Laurie Wilson (Lawrence.Wilson " at " ec.gc.ca)

Subscribe to or unsubscribe from the project email list

Subscribe

To subscribe to the email list, please do both of the following:

(1) Send an email to Eric Gilleland giving your name, institution, contact information, and whether or not you plan to participate in MesoVICT, and if not, why you want to be on the list.

(2) Using your same email address from step (1), sign up here.*

Note: the reason for step (1) above is to verify who you are in order to protect against spam. If your identity cannot be easily verified, your subscription request will be ignored.

Unsubscribe

To unsubscribe, go to the same web page, and follow the instructions there to unsubscribe. No need to send Eric an email to unsubscribe.


MesoVICT is a project of the WMO Joint Working Group on Forecast Verification

*Any information collected is used solely to determine the legitimacy of subscription requests (e.g., to protect against spam). Email addresses are added to a controlled list (only subscribers may send messages). The list is intended primarily for participants in the ICP, but others wishing to keep updated on the progress of the project may also subscribe. If you have any trouble subscribing (or unsubscribing) contact the webmaster (Eric Gilleland).

The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.