Spatial Forecast Verification Methods

SpatialVx

  • About
  • Content
  • Installation
  • User's Group
  • Updates

SpatialVx is an R package for performing spatial forecast verification. Most of the recently proposed methods are or will be included.

Note: This package is still under development and has not been thoroughly tested. Furthermore, major changes may still take place to the existing code.

See the Content tab above to find out what major methods are currently available. More coming soon.

Installation is easy if you're familiar with R. In either case, see the Installation tab above to learn how to install the package.

The Updates tab shows the progression of the package, and gives more details about what has been included. This will be re-set upon release of version 1.0-0 (to be the first official release).

When a solid foundation of all the methods to be included is finished, then a user's manual, quick-start guide and tutorial will be created, and made available under the Getting Started tab.

A LinkedIn user's group is available. Updates will be posted to this group.

The following table lists the major methods currently available in SpatialVx. See the help file for the primary function for more details on a particular method.

Method NameFunctionsDescription
Significance Testing
Field Signficance spatbiasFS, LocSig, MCdof Simulation-based field significance testing
Spatial Prediction Comparison Test spct (regular or irregular grids) and for regular grids only: lossdiff (find the loss differential field for a specific loss function), empiricalVG.lossdiff (estimate the variogram for the loss differential field), flossdiff (fit a parametric variogram model to the empirical variogram for the loss differential field), and summary to finally do the test. Test for competing forecasts that accounts for spatial dependence
Filter Methods: Neighborhood/Smoothing Filter
Fractions Skill Score (FSS) hoods2d Compares event frequencies within neighborhoods
Fuzzy Logic hoods2d Compares event frequencies within neighborhoods with alternative definitions for hits, misses, false alarms and correct negatives
Joint Distribution hoods2d Compares event frequencies within neighborhoods with alternative definitions for hits, misses, false alarms and correct negatives
Minimum Coverage hoods2d Compares event frequencies within neighborhoods
Multiple Event Contingency Table hoods2d Compares event frequencies within neighborhoods of the forecast against binary events in the verification field
Practically Perfect Hindcast pphindcast2d Compares event frequencies within neighborhoods of the forecast against binary events in the verification field
Pragmatic hoods2d Compares event frequencies within neighborhoods of the forecast against binary events in the verification field
Upscaling upscale2d Computes traditional forecast verification on smoothed fields
Wavelet De-Noising wavePurifyVx De-noises the verification set by applying a discrete wavelet transform to each field, setting small wavelet coefficients to zero and then re-constructing the fields
Filter Methods: Scale Separation/Band-pass Filters
Intenisty Scale (IS) waveIS Applies summary measures to detail fields of the binary verification set
Structure Function structurogram, structurogram.matrix Calculates the structure function for different spatial separations
Variogram variogram.matrix Variability at different spatial separations (special case of the structure function)
Wavelet Details waverify2d Applies summary measures to detail fields of the verification set
Spatial Dissimilarity (and other) Summary Measures
Gβ, G Gbeta, TheBigG New dissimilarity measures introduced by Gilleland (2021).
Baddeley's binary image metric locmeasures2d Modification of the Hausdorff metric that takes the Lp-norm of the difference in distance maps for the verification set
Forecast Quality Index FQI, UIQI Image metric informing about both location and intensity errors
Geometric characterizations Aindex, Cindex, Sindex Summary indices of shapes and textures of each field
Hausdorff metric locperf, locmeasures2d Maximum difference in distance maps for the verification set
Mean Error Distance locperf, locmeasures2d The mean of shortest distances between events in the forecast to events in the verification set
Mean Square Error Distance locperf, locmeasures2d The mean of squared shortest distances between events in the forecast to events in the verification set
Minimum Separation Between Boundaries locperf, locmeasures2d Smallest value of the distance map from one field over the subset where events occur in the other field
modified Hausdorff Measure locperf, locmeasures2d Modification of the Hausdorff metric that compares the maximum of the mean error distance
partial Hausdorff Measure locperf, locmeasures2d Modification of the Hausdorff metric that compares the k-th largest difference in distance maps for the verification set
Pratt's Figure of Merit locperf, locmeasures2d Compares binary images
Zhu's metric metrV linear combination of two measures: (i) the amount of overlap between events in two fields and (ii) the mean error distance
Image Moments imomenter Yields useful information about an object or field, such as the centroid, orientation angle, etc.
Field Displacement: Deformation Methods
Image Warping iwarper, warper Rudimentary image warping functions.
Optical Flow (OF) OF Deform the forecast field to better match the observed field using gradient information
Rigid Transformation rigider, rigidTransform Calculate the optimal rigid transformation (translations and rotations)
Field Displacement: Feature-based Methods
Cluster Analysis clusterer, CSIsamples Cluster analysis applied to the locations and possibly also the intensities
Method of Object-based Diagnostic Evaluation (MODE) Identification of features: convthresh, threshfac and threshsizer. As of version 0.3 (to be released), these three functions are deprecated, having been combined into the single function, FeatureFinder.

Merging and/or matching features: centmatch, deltamm, minboundmatch, MergeForce.

Analyzing features: FeatureMatchAnalyzer, FeatureAxis, FeatureProps, FeatureComps, FeatureTable and interester.
MODE has numerous possibilities whereby one attempts to (i) identify, (ii) possibly merge and (iii) match features between fields, and then apply summaries (e.g., the above field displacement methods).
Structure, Amplitude, Location (SAL) saller (see also functions for identification of features for MODE above) Compares the distributions of identified features in terms of structure, amplitude and location
Composite compositer (see also functions for identification of features for MODE above) Places features onto a relative grid so that they all share the same centroid.
Shape Analysis hiw, distill and summary (see also functions for identification of features for MODE above) Identifies landmarks (boundary points) for each feature, and sets up an object that can be analyzed by functions from the shapes package. Also finds the translation SS and intensity SS values for every pair of features between fields.
Distributional
Variography variographier Variography measure
Anomoly Correlation Coefficient (ACC) ACC Correlation using a climatology instead of the mean
Other
Bias-adjusted Gilbert Skill Score vxstats Adjusted traditional GSS to account for spatial bias
Gaussian Mixture Model (GMM) gmm2d Fits a Gaussian Mixture Model to each field and makes comparisons using the estimated parameters (this is a cross between a field summary, because it is applied to the entire field, and a feature-based approach because it informs about specific features whereby the number of features is determined a priori)
Geographic Box-Plot GeoBoxPlot Box-plots that allow for incorporation of differing areas of grid squares when aggregating over space.
S1 Score S1 A comparison of gradients between two fields

SpatialVx is a new R package with functions to perform many of the spatial forecast verification methods compared in the ICP (as well as some others).

Installation

First, you must have R installed on your computer. To install R, go to the R project home page (http://www.r-project.org).

To install SpatialVx, simply open an R session, and from the R prompt, type:

update.packages(checkBuilt = TRUE)
install.packages("SpatialVx")


The first line is to ensure that you have the most recent build of the packages on which SpatialVx depends (if you have any of them previously installed, which is not entirely unlikely given the numerous dependencies). If you do not have the correct permissions, you may have to have your systems administrator install the package for you. Alternatively, you may be able to provide an alternate directory to which you can install R packages. For example, say you have permissions to install packages to the directory "[path]/library". Then replace the second line above with

install.packages("SpatialVx", lib = "[path]/library")

Once the package is successfully installed, you must load it into your R session in order to use it (this must be done each time you begin a new R session). This can be done with

library(SpatialVx)

Or, if you installed it to a directory where R does not know to search, then, supposing it is installed in [path]/library, use:

library(SpatialVx, lib.loc = "[path]/library")

To see how to cite SpatialVx, type from the R prompt:

citation("SpatialVx")




*Any information collected is used solely to determine the legitimacy of subscription requests (e.g., to protect against spam). Email addresses are added to a controlled list (only subscribers may send messages). The list is intended primarily for participants in the ICP, but others wishing to keep updated on the progress of the project may also subscribe. If you have any trouble subscribing (or unsubscribing) contact the webmaster (Eric Gilleland).



The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.