Spatial Forecast Verification Methods Inter-Comparison Project

24-25 August 2009 at NCAR's Center Green Campus, Boulder, Colorado.

The goal of the workshop is to delve further into the various methods by applying them to other test cases, as well as to discuss potential future directions for the project.

The primary discussion topic for this meeting is: "Next Steps and Future Directions for the ICP." Possible specific topics include:

• Next Test Cases: What other real test cases should be applied to the inter-comparison. Cloud and wind forecasts have been suggested in addition to the Meteo Swiss cases already available from the ICP test cases.
• Design of Experiment: The methods have all been compared on real and contrived cases, but how well do they represent what is needed in the forecast verification community? What other types of cases are needed in order to get a complete comparison?
• Mixing Categories: Most of the methods can be categorized into one of four general types of methods. To obtain a complete picture of forecast performance, it may prove useful to combine some of the already proposed methods. To some extent, this has already been done. Further, some of the methods, even classes of methods, may essentially provide the same information. To what extent is this true?
• New methods?: What new methods have been proposed since the special collection of Wea. Forecasting, and where do they belong? Are there other existing methods from other areas (e.g., image processing, medicine, etc.) that might be adapted for use in forecast verification?
• Future papers: Is there interest in making a contributed book on spatial forecast verification?
• Space and time domains: Some work has begun on incorporating the time component. Up to now, the ICP has been focused on strictly spatial verification, where timing errors might be discovered in the sense of spatial displacement. However, more direct assessments of the temporal component are still desired. Should the ICP plunge into this domain? Are there enough new methods here to warrant a comparison?

All sessions wil be held at NCAR's Center Green campus in the main auditorium (south). There will be one 2-hour ReadyTalk session (Discussion I in the schedule below) to discuss the next steps and future directions of the ICP. One next step is clearly to analyze more varied fields, but which fields? The meetings will run from the afternoon of 24 August and all day on 25 August. Specific agenda is subject to change (pdf).

Venue

The workshop will be held at the National Center for Atmospheric Research (NCAR) Center Green campus in the northeast part of Boulder, Colorado. The Center Green Campus has an on-site cafeteria where breakfast and lunch may be purchased. The meetings will be in the main auditorium (south).

Remote Attendence

For those who wish to attend, but cannot make it to Boulder, we will have one 2-hour ReadyTalk session to discuss the next steps and future directions of the ICP. Please email Barbara Brown (bgb " at " ucar "dot" edu) if you would like to attend the meeting in this way.

Lodging

Arrangements have been made with the Best Western Golden Buff Lodge for a special lodging rate of $84.00 per night. The Golden Buff is located at 1725 28th Street, Boulder, CO 80301, and is on the NCAR shuttle route and near to many shopping areas and restaurants. You can make reservations with them at bwgoldenbuff@yahoo.com or at 303-442-7450 (1-800-999-2833). Use Group # 536321 to get the special rate.

Transportation

NCAR provides a regular shuttle service that has a stop across the street from the Golden Buff Lodge, at 28th Street and Canyon Blvd. Shuttle schedules and maps can be found at http://www.fin.ucar.edu/sass/transport/shuttserv.html.

Our third ICP meeting took place 24-25 Aug 2009 at the Center Green Campus of NCAR in Boulder, Colorado. A welcome and brief history of the project, as well as goals for the present meeting were given by Eric Gilleland.

David Ahijevych and Eric Gilleland then introduced a game for the meeting. Participants in attendance were given results from having applied some of the methods to test cases. The participants were not allowed to see the fields (or even know what type of fields they were) for which they had results. In all, there were four sets of cases based on two types of fields: (i) 2.2 km and 7 km precipitation fields provided by MeteoSwiss, as found in the ICP test cases, and (ii) curl and divergence fields derived from wind fields. The participants generally found the activitiy to be useful for learning more about the techniques. It was decided that such a game could prove useful, but needs to have a few kinks worked out. The forecast and observed wind fields were never shown (they are now available from this page under the "ICP Game" tab). In general, it was difficult for the participants to analyze the results of some of the methods, which may have been a lack of understanding of the methods, or it could indicate that the methods are simply difficult to interpret. A possible future activity would be to play the game where persons heavily knowledgeable about the techniques are tasked with analyzing the forecast based on their own results' techniques. Techniques used in the game were chosen purely out of time constraints pertaining to methods that were readily carried out by Ahijevych and Gilleland.

After the game, a few more participants joined the meeting from remote locations via ReadyTalk. An approximately two-hour discussion ensued. Jason Nachamkin graciously agreed to take notes, and also wrote them up for us. Click here for a slightly edited version of these summary notes. A list of participants at the meeting is also included therein.

The next day consisted of presentations, results from the previous day's game, and some further discussion. Valliappa Lakshmanan presented a very new method that appears to be straightforward, easily and efficiently implemented, and powerful in its interpretations. He cited the ICP's test cases and special collection of papers as having a great impact on the quick development of the method. His slides can be viewed from the "Agenda" tab of this web page, and a draft paper is available at: http://cimms.ou.edu/~lakshman/Papers/gmmverif.pdf (the paper has now been submitted to Wea. Forecasting for the ICP special collection).

Jason Nachamkin presented results from his composite method with a comparison of the approach with the neighborhood methods; in particular the FSS. He found that the composite approach shares much in common with the neighborhood approaches. In Gilleland et al. (2009a) of the special collection, the method is classified as a features-based approach because it identifies features, but also notes that it does not fit very well in the category.

Other presentations included David Ahijevych's talk on traditional verification results compared with subjective results for the "real" test cases. The results are the most recent results that have been included in the corresponding submitted paper for the special collection.

Finally, Chris Davis and Eric Gilleland presented results from MODE and Image Warping, resp.

At this meeting, we played a game where participants were given results from some fo the methods, and asked to evaluate the forecast performance without looking at the fields from which the results were derived. In fact, they were not even told what sort of fields they were. Because we only had limited time and ability to garner the various results and plan the exercise, and because it is the first attempt at such a scheme, we will refrain from discussing the results here. Nevertheless, the fields used and the results shown to the participants can be obtained from here.

In particular, the Meteo Swiss cases can be obtained from the netCDF files: 200508180000_MFC2.nc, 200508180000_MFC7.nc, 200508180000_SRN2.nc and 200508180000_SRN7.nc. Here, the observations are the files with MFC in the file name, and the verification fields are the ones with SRN in the name. The 2 refers to the 2.2 km resolution cases, and the 7 to the 7 km resolution cases. The associated variograms are in the files vg2.2.pdf and vg7.pdf where the verification field variograms are "X" and "T" in the graphs (note that only variograms over the entire field, and not just where precipitation occurs, are presented). For the traditional verification results, the fine refers to the 2.2 km resolution, and large to the 7 km case.

We also gave results from wind curl and divergence fields. The netCDF files: curl_V20050712_120000_L00.nc, 3.0M curl_V20050712_120000_L36.nc, div_V20050712_120000_L00.nc and div_V20050712_120000_L36.nc give the fields used to make the results. The MODE output can be viewed in the postscript files MODEcurl.ps and MODEdiv.ps, resp. Images of the entire fields (without maps) can be viewed in the curlImages.pdf and divImages.pdf files. The variograms for the fields can be viewed in the files: Acurl00Bcurl36.pdf and Cdiv00Ddiv36.pdf, where "A" and "C" are the corresponding verification fields, and "B" and "D" are from the forecast fields.