Introduction
As part of restoration of Greater Everglades ecosystems, managers and scientists are relying on ecological models to aid in decision making. Economic and ecological stakes are high and at some point in the process an astute model user should be concerned about how reliable a model is. No model is a perfect representation of reality. To ensure reliability, models should be tested and improved as part of their development and application. Ecological models are simplifications of complex natural systems and in ecosystem restoration are used to increase our understanding of how ecosystems work and how they respond to human activities. Tradeoffs must be made, and uncertainties dealt with to make models work in a management-driven timeframe, and within budget limitations. It is important to make these tradeoffs and uncertainties explicit. That still does not answer the question, "Do the models work?"
The issue of testing and improving models has been discussed in the literature with three general themes: the importance of doing it (Mitro 2001, Rykiel 1996), the impossibility of doing it (Oreskes et al. 1994), and the confusion over doing it (Mitro 2001). We agree with the importance of testing and improving models, and the purpose of this publication is to discuss briefly the impossibility and try to reduce confusion.
Most confusion in testing and improving models comes from imprecise use of terminology for the process of testing and improving (Mitro 2001). Three distinct terms are used frequently and interchangeably. In the case of ecological modeling, these terms, verification, validation, and calibration, have been defined to allow for consistent use (Rykiel 1996).
Components of Testing and Improving Models
Rykiel (1996) recognized three components to testing and improving ecological models. The first two, is the model internally correct (are the parameters, equations, and code true) and does the model perform well (model output compared to field data), are components of testing the model. The third component, improving the model, is a distinct but necessarily integrated task. The terms verification, validation, and calibration are related respectively to these three components. Confusion over terminology used to describe the testing of models stems from the fact that the terms verification and validation are often used interchangeably in everyday language However, these terms have been given specific meanings when used to describe ecological models (Mitro 2001). The use of verification and validation also has been confounded by the assertion that models cannot be either verified or validated (Oreskes et al. 1994). There may be some theoretical or philosophical validity to this, but this does not address the pragmatic importance of determining reliability of models for decision-making purposes (Mitro 2001).
Verification and Validation
Rykiel (1996) defines verification as it pertains to modeling as "a demonstration that the modeling formalism is correct." We can determine that any calculations, inputs, or computer code are correct or true. However, models that use these calculations are based on parameters which are not completely known so they cannot be taken as absolute truth (Oreskes et al. 1994). In contrast, the definition of validation, "a demonstration that a model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model," refers to its performance (Rykiel 1996). Validation compares simulated system output with real system observations using data not used in model development.
Calibration
Rykiel (1996) defines calibration as "the estimation and adjustment of model parameters and constants to improve the agreement between model output and a data set." Models contain variables and constants that need to be parameterized. Calibration improves the parameterization of a model. Calibration can occur as part of either verification or validation.
Reliability
Reliable models provide the confidence upon which to base management decisions (Holling 1978). Validation, verification, and calibration are integral processes in creating reliable models. There are no completely objective and accepted methods for testing and improving models. Agreeing on definitions of components of testing and evaluation is the first step in creating reliable models. Establishing conventions and standards for verification, validation, and calibration is the next step. It is important to note that determination of reliability of models cannot be separated form the context (knowledge base, purpose and consequences) in which models are used.
References
Holling, C. S. 1978. Adaptive Environmental Assessment and Management. John Wiley and Sons. New York, NY.
Mitro M. G. 2001. Ecological model testing: verification, validation, or neither? Bulletin of the Ecological Society of America, 82: 235–236.
Oreskes N, Shrader-Frechette K, Belitz K. Feb 1994. Verification, validation, and confirmation of numerical models in the earth sciences. Science, 263: 641–646.
Rykiel, E. J., Jr. 1996. Testing ecological models: the meaning of validation. Ecological Modeling, 90: 229–244.