Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the SWAT model
Introduction
Due to advancements in the understanding of natural processes and their interactions, and due to the advancements in software engineering and the increased computational power, environmental modeling tools have become more complex over the past decades (e.g. Arnold et al., 1998, Rossman, 2009, DHI, 2009). In general, such complex simulators contain many parameters, most of which cannot be measured directly and can only be inferred by calibration to observed system responses (Yapo et al., 1998, Vrugt et al., 2002). Consequently, parameter estimation has become a major issue, which may limit the applicability of complex simulators (van Griensven et al., 2006). A manual calibration of a model with a large number of parameters is very tedious and time consuming (Vrugt et al., 2003). On the other hand, the efficiency of automatic calibration algorithms is reduced when the number of parameters is large (Duan et al., 1992). In fact, it is not feasible to include all the model parameters in the calibration process (Bekele and Nicklow, 2007, Nossent et al., 2011). In order to support the choice of which model parameters should be the focus of calibration, and which ones could be instead excluded from calibration (and set to ‘default’ values), Global Sensitivity Analysis (GSA) is becoming popular in environmental modeling practices (e.g. Muleta and Nicklow, 2005, van Werkhoven et al., 2009, Norton, 2015, Pianosi et al., 2016a, Pianosi et al., 2016b). GSA indeed allows for the identification of the parameters that have the largest influence on a set of model performance metrics (so called ‘factor prioritization’) and the identification of non-influential parameters (‘factor fixing’) (Saltelli et al., 2008, Nossent et al., 2011). Other uses of GSA include the understanding and the interpretation of the model behavior, the prioritization of efforts for uncertainty reduction and the model simplification (Nossent et al., 2011, Pianosi et al., 2016a, Pianosi et al., 2016b).
The Soil and Water Assessment Tool (SWAT) (Arnold et al., 1998) is a particular example of a relatively complex environmental simulator, which has been widely applied all over the world for watershed management purposes (e.g. Gassman et al., 2010, van Griensven et al., 2012, Bressiani et al., 2015). In SWAT, different watershed processes, including surface runoff, groundwater flow, plant growth, and pesticide and nutrient conversion and transport, are controlled by a large number of parameters (more than 100). Even when some of these parameters can be fixed a priori, calibration of SWAT remains quite challenging given the relatively large number of parameters (26 in our case) that are typically left to be varied simultaneously. Therefore, GSA is often applied prior to the calibration process to identify the most influential parameters and the non-influential ones (Cibin et al., 2010, Nossent et al., 2011, Leta et al., 2015).
Many different GSA methods have been developed (Sobol', 1990, Saltelli et al., 2000, van Griensven et al., 2006, Borgonovo, 2007, Pianosi and Wagener, 2015). Among them, the most well-established and widely-applied one is probably the variance-based method of Sobol' (Sobol', 1990; applications to environmental models include Pappenberger et al., 2008, van Werkhoven et al., 2008, Nossent et al., 2011, Rosolem et al., 2012, Gan et al., 2014). In general, variance-based methods seek to measure sensitivity to an uncertain input (parameter) using the contribution of that input to the total variance of the model output (a metric of model performance, in the context of model calibration). A well-known merit of variance-based methods is their ability to quantify the individual parameter contribution and the contribution resulting from parameter interactions, independently from assumptions on the form of the input-output relation (e.g. linearity and additivity). Moreover, variance-based sensitivity indices are easy to interpret, as they represent the fraction of the output variance caused by the variation of an input (Saltelli, 2002b).
Variance-based GSA methods use the variance - i.c. the second moment-as a measure of the output uncertainty, and as Saltelli (2002b) underlined, “implicitly assume that this moment is sufficient to describe the output variability”. However, it has been recognized that the variance does not adequately represent output uncertainty when the model output is highly-skewed or multi-modal (Liu et al., 2006, Borgonovo et al., 2011, Pianosi and Wagener, 2015). To overcome this limitation, moment-independent GSA measures have been developed (Liu et al., 2006, Borgonovo, 2007, Pianosi and Wagener, 2015). These methods -also known as density-based methods-use the entire output distribution to fully characterize the output uncertainty and to quantify the relative influence of the uncertain parameters. The main advantage of these methods, as compared to variance-based ones, is that they do not use a specific moment of the output distribution to measure the output variability and, therefore, are applicable regardless of its shape (e.g. symmetric or highly-skewed).
Pianosi and Wagener (2015) have proposed a moment-independent GSA method, called PAWN. It measures sensitivity based on the difference between the unconditional output distribution, obtained when all the parameters are free to vary, and the conditional output distribution, obtained when one of the parameters is fixed. Hereby, a Cumulative Distribution Function (CDF) is used to characterize the output distribution, whereas other density-based methods (e.g. the entropy-based method (Liu et al., 2006) and the δ-sensitivity measure (Borgonovo, 2007)) used the Probability Density Function (PDF). The main advantage of the PAWN method is that approximating CDFs by using empirical distributions of the data sample is much easier than approximating PDFs, because, it does not require any parameter tuning. This facilitates the analysis of the robustness and the convergence of the estimated sensitivity indices (Pianosi and Wagener, 2015).
In Pianosi and Wagener (2015), the PAWN method was tested on a simple conceptual hydrological model with only 5 parameters. To further investigate its effectiveness and efficiency, it is necessary to apply it to a more complex simulator with a higher number of parameters, such as SWAT, and to compare its results with those of another GSA method. The main objective of this paper is therefore to evaluate and compare the application of the Sobol' and PAWN methods to a SWAT model. In particular, the two methods will be compared in terms of the rate of convergence of the respective sensitivity indices, and their results for parameter ranking and screening. To this end, 26 parameters of a SWAT model of the upstream sub-catchment of the River Zenne (Belgium) are analysed. As model outputs for sensitivity analysis, we consider two performance metrics for simulating daily river flows at the catchment outlet: the Nash-Sutcliffe efficiency (NSE) (Nash and Sutcliffe, 1970) and the mean error (ME). In performing parameter screening, we propose to calculate the sensitivity index of a “dummy parameter”, which has no influence on the model output. The sensitivity index of this dummy parameter is used as a threshold to identify non-influential parameters. It is calculated numerically using sample data, without adding the dummy parameter explicitly to the model. The “dummy parameter approach” provides a practical way to sensibly define a threshold for screening, which is an unresolved issue increasingly discussed in recent GSA literature for both Sobol' and PAWN (e.g. Sarrazin et al., 2016). However, for the PAWN method, in particular, its effectiveness can be demonstrated by validating the screening results using the two-sample Kolmogorov-Smirnov statistical test (Smirnov, 1948).
Section snippets
The variance-based Sobol' method
Sobol' (1990) is a “global, quantitative and model free” GSA method (Saltelli, 2002b), which also works properly for non-linear and non-monotonic models. In this method, the contributions of each parameter to the total model output variance, either by variation of the parameter itself or by interactions with other parameters, are quantified and expressed as Sobol' sensitivity indices. These indices provide a quantitative measure of the importance of the parameters and can be used for both
The distributions of the performance measures
As mentioned above, two different performance measures are considered for the GSA: the NSE and the ME. The 9000 random sample generated for Sobol' application (i.e. model performance against random parameter matrices M1) are used to obtain the empirical distributions of the NSE and ME. As shown in Fig. 1, the distribution of NSE is negatively-skewed, while the distribution of ME is slightly bi-modal with a second (small) peak on the right. The statistical analysis, based on the KS test (
Discussion
The results of this study are used to compare the Sobol' and PAWN methods for the global sensitivity analysis to 26 parameters of the SWAT model. However, it should be noted that the comparison is performed for a specific case study (River Zenne, Belgium). Previous studies, for example Cibin et al. (2010), pointed out the effects of contrasting climate conditions and flow regimes on the Sobol' sensitivity analysis results of SWAT models. Moreover, many other choices made in the experimental
Conclusions
In this paper, we compared the application of two GSA techniques, the variance-based Sobol' method and the density-based PAWN method, to the analysis of 26 parameters of the SWAT model, a hydrological model widely-used for water quality and quantity simulations. The comparison was performed in terms of convergence rate and parameter ranking and screening results. Moreover, the use of a “dummy parameter” approach as a viable option to set a threshold value for parameter screening was
Software/data availability
The PAWN method is implemented in the SAFE Matlab/Octave Toolbox for GSA (Pianosi et al., 2015). SAFE is freely available for non commercial purposes at www.bristol.ac.uk/cabot/-resources/safe-toolbox/.
The Soil and Water Assessment Tool (SWAT) (Arnold et al., 1998) is a public domain environmental simulator. The SWAT model as developed by Leta (Leta, 2013, Leta et al., 2015) for the River Zenne (Belgium) is used in this study.
Acknowledgment
The authors would like to thank the Flanders Hydraulics Research for supporting and coordinating the project of “Development of conceptual models for an integrated river basin management”. We also thank Dr. Olkeba Tolessa Leta for setting up the SWAT model for the River Zenne (Belgium). This work is partially supported by a University of Bristol Alumni Postgraduate Scholarship to Fanny Sarrazin. Partial support for Francesca Pianosi and Thorsten Wagener was provided by the Natural Environment
References (63)
- et al.
Multi-objective automatic calibration of SWAT using NSGA-II
J. Hydrology
(2007) A new uncertainty importance measure
Reliab. Eng. Syst. Saf.
(2007)- et al.
A comprehensive evaluation of various sensitivity analysis methods: a case study with a hydrological model
Environ. Model. Softw.
(2014) - et al.
Importance measures in global sensitivity analysis of nonlinear models
Reliab. Eng. Syst. Saf.
(1996) - et al.
Assessment of the different sources of uncertainty in a SWAT model of the River Senne (Belgium)
Environ. Model. Softw.
(2015) - et al.
Sensitivity and uncertainty analysis coupled with automatic calibration for a distributed watershed model
J. Hydrology
(2005) - et al.
River flow forecasting through conceptual models part I—a discussion of principles
J. hydrology
(1970) An introduction to sensitivity assessment of simulation models
Environ. Model. Softw.
(2015)- et al.
Sobol' sensitivity analysis of a complex environmental model
Environ. Model. Softw.
(2011) - et al.
Multi-method global sensitivity analysis of flood inundation models
Adv. Water Resour.
(2008)
PAWN: a simple and efficient method for Global Sensitivity Analysis based on cumulative distribution functions
Environ. Model. Softw.
A Matlab toolbox for global sensitivity analysis
Environ. Model. Softw.
Sensitivity analysis of environmental models: a systematic review with practical workflow
Environ. Modelling& Softw.
Global sensitivity measures from given data
Eur. J. Oper. Res.
Making best use of model evaluations to compute sensitivity indices
Comput. Phys. Commun.
Global sensitivity analysis of environmental models: convergence and validation
Environ. Model. Softw.
Addressing ten questions about conceptual rainfall–runoff models with global sensitivity analyses in R
J. Hydrology
On the distribution of points in a cube and the approximate evaluation of integrals
USSR Comput. Math. Math. Phys.
Uniformly distributed sequences with an additional uniform property
USSR Comput. Math. Math. Phys.
Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates
Math. Comput. Simul.
Sensitivity-guided reduction of parametric dimensionality for multi-objective calibration of watershed models
Adv. Water Resour.
A global sensitivity analysis tool for the parameters of multi-variable catchment models
J. hydrology
Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis
Environ. Model. Softw.
Multi-objective global optimization for hydrologic models
J. hydrology
A symptotic theory of certain” goodness of fit” criteria based on stochastic processes
Ann. Math. statistics
Sensitivity measures, ANOVA-like techniques and the use of bootstrap
J. Stat. Comput. Simul.
Large area hydrologic modeling and assessment part I: model development1
J. Am. Water Resour. Assoc.
Soil and water assessment tool
Moment independent importance measures: new results and analytical test cases
Risk Anal.
A common rationale for global sensitivity measures and their estimation
Risk Anal.
Review of soil and water assessment tool (SWAT) applications in Brazil: challenges and prospects
Int. J. Agric. Biol. Eng.
Cited by (115)
Modeling the impact of climate change on streamflow in glacier/snow-fed northern Tianshan basin
2023, Journal of Hydrology: Regional StudiesResearch on predicting the diffusion of toxic heavy gas sulfur dioxide by applying a hybrid deep learning model to real case data
2023, Science of the Total EnvironmentApplication of an integrated catchment-lake model approach for simulating effects of climate change on lake inputs and biogeochemistry
2023, Science of the Total EnvironmentA sensitivity analysis of a FAO-56 dual crop coefficient-based model under various field conditions
2023, Environmental Modelling and SoftwareWatershed model parameter estimation in low data environments
2023, Journal of Hydrology: Regional Studies