Spatial Forecast Verification Methods
SpatialVx
- About
- Content
- Installation
- User's Group
- Updates
SpatialVx is an R package for performing spatial forecast verification. Most of the recently proposed methods are or will be included.
Note: This package is still under development and has not been thoroughly tested. Furthermore, major changes may still take place to the existing code.
See the Content tab above to find out what major methods are currently available. More coming soon.
Installation is easy if you're familiar with R. In either case, see the Installation tab above to learn how to install the package.
The Updates tab shows the progression of the package, and gives more details about what has been included. This will be re-set upon release of version 1.0-0 (to be the first official release).
When a solid foundation of all the methods to be included is finished, then a user's manual, quick-start guide and tutorial will be created, and made available under the Getting Started tab.
A LinkedIn user's group is available. Updates will be posted to this group.
The following table lists the major methods currently available in SpatialVx. See the help file for the primary function for more details on a particular method.
| Method Name | Functions | Description | |
| Field Signficance | spatbiasFS, LocSig, MCdof | Simulation-based field significance testing | |
| Spatial Prediction Comparison Test | spct (regular or irregular grids) and for regular grids only: lossdiff (find the loss differential field for a specific loss function), empiricalVG.lossdiff (estimate the variogram for the loss differential field), flossdiff (fit a parametric variogram model to the empirical variogram for the loss differential field), and summary to finally do the test. | Test for competing forecasts that accounts for spatial dependence | |
| Fractions Skill Score (FSS) | hoods2d | Compares event frequencies within neighborhoods | |
| Fuzzy Logic | hoods2d | Compares event frequencies within neighborhoods with alternative definitions for hits, misses, false alarms and correct negatives | |
| Joint Distribution | hoods2d | Compares event frequencies within neighborhoods with alternative definitions for hits, misses, false alarms and correct negatives | |
| Minimum Coverage | hoods2d | Compares event frequencies within neighborhoods | |
| Multiple Event Contingency Table | hoods2d | Compares event frequencies within neighborhoods of the forecast against binary events in the verification field | |
| Practically Perfect Hindcast | pphindcast2d | Compares event frequencies within neighborhoods of the forecast against binary events in the verification field | |
| Pragmatic | hoods2d | Compares event frequencies within neighborhoods of the forecast against binary events in the verification field | |
| Upscaling | upscale2d | Computes traditional forecast verification on smoothed fields | |
| Wavelet De-Noising | wavePurifyVx | De-noises the verification set by applying a discrete wavelet transform to each field, setting small wavelet coefficients to zero and then re-constructing the fields | |
| Intenisty Scale (IS) | waveIS | Applies summary measures to detail fields of the binary verification set | |
| Structure Function | structurogram, structurogram.matrix | Calculates the structure function for different spatial separations | |
| Variogram | variogram.matrix | Variability at different spatial separations (special case of the structure function) | |
| Wavelet Details | waverify2d | Applies summary measures to detail fields of the verification set | |
| Gβ, G | Gbeta, TheBigG | New dissimilarity measures introduced by Gilleland (2021). | |
| Baddeley's binary image metric | locmeasures2d | Modification of the Hausdorff metric that takes the Lp-norm of the difference in distance maps for the verification set | |
| Forecast Quality Index | FQI, UIQI | Image metric informing about both location and intensity errors | |
| Geometric characterizations | Aindex, Cindex, Sindex | Summary indices of shapes and textures of each field | |
| Hausdorff metric | locperf, locmeasures2d | Maximum difference in distance maps for the verification set | |
| Mean Error Distance | locperf, locmeasures2d | The mean of shortest distances between events in the forecast to events in the verification set | |
| Mean Square Error Distance | locperf, locmeasures2d | The mean of squared shortest distances between events in the forecast to events in the verification set | |
| Minimum Separation Between Boundaries | locperf, locmeasures2d | Smallest value of the distance map from one field over the subset where events occur in the other field | |
| modified Hausdorff Measure | locperf, locmeasures2d | Modification of the Hausdorff metric that compares the maximum of the mean error distance | |
| partial Hausdorff Measure | locperf, locmeasures2d | Modification of the Hausdorff metric that compares the k-th largest difference in distance maps for the verification set | |
| Pratt's Figure of Merit | locperf, locmeasures2d | Compares binary images | |
| Zhu's metric | metrV | linear combination of two measures: (i) the amount of overlap between events in two fields and (ii) the mean error distance | |
| Image Moments | imomenter | Yields useful information about an object or field, such as the centroid, orientation angle, etc. | |
| Image Warping | iwarper, warper | Rudimentary image warping functions. | |
| Optical Flow (OF) | OF | Deform the forecast field to better match the observed field using gradient information | |
| Rigid Transformation | rigider, rigidTransform | Calculate the optimal rigid transformation (translations and rotations) | |
| Cluster Analysis | clusterer, CSIsamples | Cluster analysis applied to the locations and possibly also the intensities | |
| Method of Object-based Diagnostic Evaluation (MODE) | Identification of features: convthresh, threshfac and threshsizer. As of version 0.3 (to be released), these three functions are deprecated, having been combined into the single function, FeatureFinder. Merging and/or matching features: centmatch, deltamm, minboundmatch, MergeForce. Analyzing features: FeatureMatchAnalyzer, FeatureAxis, FeatureProps, FeatureComps, FeatureTable and interester. |
MODE has numerous possibilities whereby one attempts to (i) identify, (ii) possibly merge and (iii) match features between fields, and then apply summaries (e.g., the above field displacement methods). | |
| Structure, Amplitude, Location (SAL) | saller (see also functions for identification of features for MODE above) | Compares the distributions of identified features in terms of structure, amplitude and location | |
| Composite | compositer (see also functions for identification of features for MODE above) | Places features onto a relative grid so that they all share the same centroid. | |
| Shape Analysis | hiw, distill and summary (see also functions for identification of features for MODE above) | Identifies landmarks (boundary points) for each feature, and sets up an object that can be analyzed by functions from the shapes package. Also finds the translation SS and intensity SS values for every pair of features between fields. | |
| Variography | variographier | Variography measure | |
| Anomoly Correlation Coefficient (ACC) | ACC | Correlation using a climatology instead of the mean | |
| Bias-adjusted Gilbert Skill Score | vxstats | Adjusted traditional GSS to account for spatial bias | |
| Gaussian Mixture Model (GMM) | gmm2d | Fits a Gaussian Mixture Model to each field and makes comparisons using the estimated parameters (this is a cross between a field summary, because it is applied to the entire field, and a feature-based approach because it informs about specific features whereby the number of features is determined a priori) | |
| Geographic Box-Plot | GeoBoxPlot | Box-plots that allow for incorporation of differing areas of grid squares when aggregating over space. | |
| S1 Score | S1 | A comparison of gradients between two fields | |
SpatialVx is a new R package with functions to perform many of the spatial forecast verification methods compared in the ICP (as well as some others).
Installation
First, you must have R installed on your computer. To install R, go to the R project home page (http://www.r-project.org).
To install SpatialVx, simply open an R session, and from the R prompt, type:
update.packages(checkBuilt = TRUE)
install.packages("SpatialVx")
The first line is to ensure that you have the most recent build of the packages on which SpatialVx depends (if you have any of them previously installed, which is not entirely unlikely given the numerous dependencies). If you do not have the correct permissions, you may have to have your systems administrator install the package for you. Alternatively, you may be able to provide an alternate directory to which you can install R packages. For example, say you have permissions to install packages to the directory "[path]/library". Then replace the second line above with
install.packages("SpatialVx", lib = "[path]/library")
Once the package is successfully installed, you must load it into your R session in order to use it (this must be done each time you begin a new R session). This can be done with
library(SpatialVx)
Or, if you installed it to a directory where R does not know to search, then, supposing it is installed in [path]/library, use:
library(SpatialVx, lib.loc = "[path]/library")
To see how to cite SpatialVx, type from the R prompt:
citation("SpatialVx")
Link to LinkedIn user's group.
The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
