Abstract
In this paper, we exploit micro data from the ECB survey of professional forecasters to examine the link between the characteristics of macroeconomic density forecasts (such as their location, spread, skewness, and tail risk) and density forecast performance. Controlling for the effects of common macroeconomic shocks, we apply cross-sectional and fixed effect panel regressions linking such density characteristics and density forecast performance. Our empirical results suggest that many macroeconomic experts could systematically improve their density performance by correcting a downward bias in their variances. Aside from this shortcoming in the second moment characteristics of the individual densities, other higher moment features, such as skewness or variation in the degree of probability mass given to the tails of the predictive distributions, tend—as a rule—not to contribute significantly to enhancing individual density forecast performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The micro dataset underlying our study can be downloaded from the ECB website at http://www.ecb.europa.eu/stats/prices/indic/forecast/html/index.en.html. ECB (2014a) provides a recent overview of the survey drawing on the fifteen years of experience since its inception.
Garrat et al. (2013) combine the predictive densities from VAR models in order to derive probability forecasts that can quantitatively underpin central bank warnings about low inflation outcomes during the great recession. An interesting direction for future research may be the extent to which such model-based analysis can be combined with the corresponding indicators from surveyed densities such as those derived in the study by Andrade et al. (2011).
See Chart 10 in ECB (2014b).
In the ECB SPF dataset, as discussed in Bowles et al. (2010), the forecast horizon (\(\tau \)) in each survey is set 1 and 2 years ahead of the latest observed outcome and therefore differs across variables due to differing publication lags for the release of official HICP and GDP statistics. For example, the 1-year ahead GDP forecast refers to the annual growth rate two quarters after the survey quarter, while the equivalent HICP forecast refers to the annual inflation rate approximately 11 months after the survey month. For notational convenience, we nonetheless refer to these variable 1 and 2 year ahead rolling horizon forecasts as \(H=1\) and \(H=2\), respectively.
In our sample, the maximum number of bins used has differed for GDP and inflation variables, with the former comprising 24 bins ranging from \(-\)6.0 to +4.9 and the latter comprising 14 bins ranging from \(-\)2.0 to +3.9. The width of the bins in the ECB SPF has been held constant at 0.4 for both GDP growth and inflation.
Aside from this practical problem confronting its use with survey data, the log score is a valid and attractive measure of density performance. In particular, it is also a strictly proper scoring rule.
As discussed in Diebold et al. (1998), “there is no way to rank two incorrect density forecasts such that all users will agree with the ranking” (p. 866). While the RPS provides an attractive and meaningful ranking of the densities, particularly for the type of surveyed data that we employ, our study is still very much conditional on this choice.
We have also checked the sensitivity of our analysis to other possible assumptions of equating the open intervals with one closed interval or spreading the probability uniformly over two closed intervals. However, reflecting the relatively low utilization of open intervals in the panel dataset, this has no noticeable impact on the main empirical results we report in Sect. 5.
However, we have also recomputed all our results using the 1st available vintage for both GDP and inflation. As expected, we find no noticeable impact on our main findings.
A close inspection of the SPF data from the 2009Q1 survey shows a pilling up of probability mass in the interval \(<-\)1.0 % for the GDP forecasts while the reported point forecasts were located well below this interval reflecting the strength of the deterioration in the euro area outlook. Such a pilling up of probability mass suggests that respondents may have been unable to report their true subjective densities because they were not supplied with a sufficient range of outcomes in the survey questionnaire. We therefore conducted all our analysis also excluding the forecasts from the 2009Q1survey. Overall, we find very similar results, suggesting that the measurement error associated with this survey round has not unduly influenced our findings.
In Sect. 3, we propose a panel regression linking density performance with these estimated moments. Given the need to estimate density moments from the underlying survey data, it is important to consider the potential complications for our subsequent empirical analysis that can arise as a result of measurement error [see, for example, (Curtin 2010)]. In particular, following Grilliche and Hausmann (1986), we control for stable sources of measurement error and check the robustness of our results by comparing both levels and first differences of the moments in the regressions.
In our empirical analysis, we primarily focus on the absolute skewness rather than the actual skewness in order to focus attention on the hypothesis that more skewed distributions are associated with better density forecast performance (i.e., lower scores). However, in Figs. 2 and 3 discussed below, we also report actual skewness and discuss the extent of positive and negative asymmetries in the SPF distributions.
The equivalent figures for \(H=2\) provide a very similar impression.
The estimated variance provides a measure of ex ante uncertainty as perceived by SPF forecasters. In a recent study, Clements (2012) has examined the relationship between ex ante uncertainty and ex post or realized uncertainty.
We have also considered the GMM estimator of which FGLS is a special case and found very similar results. Also, as an alternative way to control for the impact of aggregate shocks, we considered the use of a group mean estimator using the Common Correlated Effects for Pooled regressions (CCEP estimator) due to Peseran (2006). We do this by augmenting the estimation regression with cross section averages of independent and dependent variables and found very similar results.
Throughout the paper, we measure point forecast accuracy using the estimated mean of the density forecast as estimated by Eq. (2). However, in the SPF survey, in addition to the reported density forecasts, respondents also report “point forecasts” for equivalent variables and horizons and these can differ from the estimated means. At the suggestion of an anonymous referee, we have therefore checked the sensitivity of our results to this choice and recomputed all our empirical analysis but using these point forecasts. Overall, we do not find that this choice has impacted the main conclusions we derive. This is consistent with a methodological survey of the ECB SPF conducted in 2009 in which a large majority of participants reported that their point forecasts correspond to the means of their reported densities. The results of this methodological survey are available at http://www.ecb.europa.eu/stats/prices/indic/forecast/html/index.en.html.
Our approach has some parallels with the work of Mitchell and Wallis (2011) who focus on density evaluation with probability integral transforms (pits). They suggest regressing the pits on explanatory variables in order to gain insights into the factors that might explain good and bad performance. Indeed, more generally, inspection of the pits can provide insights into the sources of density forecast performance such as a too high or low mean/variance or an excessively small/high tail probability mass relative to the true data generating process.
Neglecting risks is not problematic per se. For example, the literature on rational inattention (see Sims 2003) has demonstrated that it may not be optimal to plan for rare (low probability) events such as earthquakes and financial disaster. Within our framework, it is the neglect of risks that have some significant chance of occurring that can be associated with a worse density performance. In a forecasting context, neglecting the possibility of certain outcomes may also be fully rational if those outcomes are not feasible or possible given the underlying economic reality. However, assigning a zero probability to outcomes that have some chance of occurring will tend to be penalized by strictly proper scoring rules such as the log score or the RPS. Indeed for the log score, the penalty for giving zero probability to an event that subsequently materializes is infinite and as a result a forecaster seeking to maximize performance should only assign a zero probability to an event or outcome that is truly impossible (e.g., unemployment rates that are less than 0 %). In the case of the discrete distributions of the SPF, this would tend to imply always having some very small positive probability in any outcome ranges that are technically feasible. In the case of the RPS, as highlighted in Sect. 2, the penalty for neglecting risks that materialize is bounded given that the RPS itself is bounded.
In practice, autocorrelations greater than 4 tend to be very small and can be discarded in the FGLS procedure. For the 1-year ahead horizon, this is entirely consistent with what would be expected for a four quarter ahead forecast sampled at a quarterly frequency but it is more surprising for the two-year ahead forecast.
In the estimation of the adjusted error variance matrix, we also considered possible correlation in the errors across individual forecasters that might arise due to common aggregate shocks in line with the panel analysis of point forecasts in Keane and Runkle (1990). However, we did not find these correlations to be important. A likely explanation for this finding is that the impact of common shocks is adequately captured through the inclusion of the fixed effects time dummies.
An alternative would be the consideration of estimation using instrumental variables but we did not pursue this because, aside from a lack of any obvious choice of instruments for density moments that we are using, the measured correlation between the regressors and the estimated residuals was always very close to zero.
We estimate the regression with asymmetries only in first differences given that the variance is always positive in levels, while in a first difference specification, we are able to distinguish between positive and negative changes in the variance.
The approach to balancing the panel builds on Genre et al. (2013) which uses a pooled regression to measure the degree of persistence in the deviation of an individual point forecasts from the average forecasts. This provides an update for the location of the density. The updated density is then centered on the bin containing this updated forecast and the associated probabilities are derived using the most recently observed probabilities that were submitted. Kenny et al. (2013, pp. 17–19), provide a more complete description of this procedure.
Of course, heterogeneity in ex post performance does not imply that it is easy to identify good forecasters ex ante. In line with this, for the case of point forecasts using the US SPF, D’Agostino et al. (2012) find limited evidence for the idea that the best forecasters are “innately” better than others.
Given the dependence of the RPS performance metric on the maximum number of bins used, and given that this differs for the GDP growth and inflation densities, some caution is warranted in drawing strong inference based on the size of the estimated coefficients.
The relative insensitivity of our findings with respect to the choice of how the density spread is measured also tends to rule out measurement error as a key driver of the results. In computing the IQR, we extract the 25th and 75th percentiles of the individual densities under the assumption that probabilities are uniformly distributed within each interval.
The panel results using the IQR also confirm this systematic negative relationship. Additionally, the panel estimations were run over a shorter sample that excluded the most recent period influenced by the financial crisis (i.e., only with outcomes up to 2007Q4). This smaller sample also yields a systematic negative relation suggesting that our results are not driven by the crisis period alone.
In the first difference regressions, the F-statistics from the tests of the pooling of coefficients are 0.0161, 0.0117, 0.0428, and 0.0545 for GDP (\(H=1\)), GDP (\(H=2\)), Inflation (\(H=1\)), and Inflation (\(H=2\)), respectively. As was the case with the levels regression, these are never significant and thus tend to confirm the validity of the pooled model with homogenous parameters.
At the request of an anonymous referee, we have also conducted additional robustness checks by considering the inclusion of the IQR in the panel regression rather than the variance and obtained very similar results.
References
Andrade P, Ghysels E, Idier J (2011) Tails of inflation forecasts and tales of monetary policy. Working paper no. 407, Banque de France
Boero G, Smith J, Wallis KF (2008) Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters. Int J Forecast 24(2008):354–367
Boero G, Smith J, Wallis KF (2011) Scoring rules and survey density forecasts. Int J Forecast 27(2):379–393
Boero G, Smith J, Wallis KF (2012) The measurement and characteristics of professional forecasters’ uncertainty. In: Paper presented at the 32nd annual international symposium on forecasting, Boston, 24–27 June 2012.
Bowles C, Friz R, Genre V, Kenny G, Meyler A, Rautanen T (2010) An evaluation of the growth and unemployment rate forecasts in the ECB SPF. J Bus Cycle Meas Anal 2010(2):63–90
Casillas-Olvera G, Bessler DA (2006) Probability forecasting and central bank accountability. J Policy Model 28(2):223–234
Clements MP (2006) Evaluating the survey of professional forecasters probability distributions of expected inflation based on the derived event probability forecasts. Emp Econ 31:49–64
Clements MP (2010) Explanations of inconsistencies in survey respondent’s forecasts. Eur Econ Rev 54(4):536–549
Clements MP (2012) Subjective and ex post forecast uncertainty. Warwick economic research papers no. 995, The University of Warwick
Conflitti C (2011) Measuring uncertainty and disagreement in the European survey of professional forecasters. J Bus Cycle Meas Anal 2:69–103
Corradi V, Swanson NR (2006) Predictive density evaluation. In: Granger Clive WJ, Elliot Graham, Timmerman Allan (eds) Handbook of economic forecasting. Elsevier, Amsterdam, pp 197–284
Curtin R (2010) Inflation expectations: theoretical models and empirical tests. In: Sinclair P (ed) Inflation expectations. Routledge International Studies in Money and Banking, Routledge
D’Agostino A, McQuinn K, Whelan K (2012) Are some forecasters really better than others? J Mon Cred Bank 44(4):715–732
D’Amico S, Orphanides A (2008) Uncertainty and disagreement in economic forecasting. Finance and economics discussion series 2008–56, Board of Governors of the Federal Reserve System (US)
Davies A, Lahiri K (1995) A new framework for analyzing survey forecasts using three-dimensional panel data. J Econom 68:205–227
Davies A, Lahiri K (1999) Re-examining the rational expectations hypothesis using panel data on multi-period forecasts. In: Hsiao C, Pesaran MH, Lahiri K, Lee LF (eds) Analysis of panels and limited dependent variable models. Cambridge University Press, Cambridge, pp 226–254
Diebold FX, Gunther T, Tay A (1998) Evaluating density forecasts with application to financial risk management. Int Econ Rev 39(4):863–883
Diebold FX, Tay AS, Wallis KF (1999) Evaluating density forecasts of inflation: the survey of professional forecasters. In: Engle R, White H (eds) Cointegration, causality and forecasting: a Festschrift in honour of Clive W. J. Granger. Oxford University Press, Oxford
ECB (2014a) Fifteen years of the ECB survey of professional forecasters, ECB Monthly Bulletin, January 2014
ECB (2014b) Results of a second special questionnaire for participants in the ECB survey of professional forecasters, January 2014. http://www.ecb.europa.eu/stats/prices/indic/forecast/html/index.en.html
Engelberg J, Manski CF, Williams J (2009) Comparing the point predictions and subjective probability distributions of professional forecasters. J Bus Econ Stat 27(1):30–41
Epstein ES (1969) A scoring system for probability forecasts of ranked categories. J App Metrop 8:985–987
Garrat T, Mitchell J, Vahey S (2013) Probability forecasting for inflation warnings from the US Federal Reserve. In Paper presented at the 9th Annual joint Cirano-Cireq Workshop on data revisions in economic forecasting and policy, Montreal, 10–11 Oct 2013
Genre V, Kenny G, Meyler A, Timmermann A (2013) Combining expert forecasts: can anything beat the simple average? Int J Forecast 29(1):108–121
Giordani P, Söderlind P (2003) Inflation forecast uncertainty. Eur Econ Rev 47(6):1037–1059
Giordani P, Söderlind P (2006) Is there evidence of pessimism and doubt in subjective distributions? Implications for the equity premium puzzle. J Econ Dyn Control 30(6):1027–1043
Gneiting T, Raftery AE (2007) Strictly proper scoring rules: prediction and estimation. J Am Stat Assoc 102(477):359–378
Grilliche Z, Hausmann J (1986) Errors in variables in panel data. J Econom 1986(31):93–118
Hsiao C (2003) Analysis of panel data, 2nd edn. Cambridge University Press, Cambridge
Kahneman D, Tversky A (2000) Choices, values and frames. Cambridge University Press, Russel Sage Foundation, Cambridge
Keane MP, Runkle DE (1990) Testing the rationality of price forecasts: new evidence from panel data. Am Econ Rev 80(4):714–735
Kenny G, Kostka T, Masera F (2013) Can macroeconomists forecast risk? Event-based evidence from the ECB SPF, ECB working paper no. 1540
Kenny G, Kostka T, Masera F (2014) How informative are the subjective density forecasts of Macroeconomists? J Forecast 33(3): 163–185
Knüppel M, Schultefrankenfeld G (2012) How informative are central bank assessments of macroeconomic risks. Int J Cent Bank 8(3):87–139
Lahiri K, Teigland C, Zaporowski M (1988) Interest rates and the subjective probability distribution of inflation forecasts. J Mon Credit Bank 20(2):233–248
Lahiri K, Wang JG (2007) The value of probability forecast as predictors of economic downturns. Appl Econ Lett 14(14):11–14
Lahiri K, Wang JG (2013) Evaluating probability forecasts for GDP declines using alternative methodologies. Int J Forecast 29(2013):175–190
Leeper EM (2003) An inflation reports, NBER working paper no. 10089, Nov 2003
Mitchell J, Wallis K (2011) Evaluating density forecasts: Forecast combinations and model mixtures, calibration and sharpness. J Appl Econom 26(6):1023–1040
Peseran MH (2006) Estimation and inference in heterogenous panels with a multifactor error structure. Econometrica 74(4):967–1012
Rich R, Tracy J (2010) The relationship among expected inflation, disagreement, and uncertainty: evidence from matched point and density forecasts. Rev Econ Stat 92(1):200–207
Rich R, Song J, Tracey J (2012) The measurement and behaviour of uncertainty: evidence from the ECB survey of professional forecasters. Federal Reserve Bank of New York, staff report no 588
Sims Christopher A (2003) Implications of rational inattention. J Mon Econ 50(3):665–690
Tay AS, Wallis KF (2000) Density forecasting: a survey. J Forecast 19:235–254. Reprinted in (2002) A companion to economic forecasting. Clements MP, Hendry DF (eds) Oxford, Blackwell, pp. 45–68
Wooldridge J (2002) Econometric analysis of cross section and panel data, 1st edn. MIT Press, Cambridge
Zarnowitz V, Lambros LA (1987) Consensus and uncertainty in economic prediction. J Polit Econ 95(3):591–621
Acknowledgments
The authors would like to thank Malte Knüppel, Sylvain Leduc, James Mitchell, Robert Rich, Shaun Vahey, and Ken Wallis for useful comments on earlier drafts of this paper as well as participants at the EABCN conference “Judgement and combination in Forecasting and Policy Models,” London 20–21 March 2014, the 2nd CesIfo conference on Macroeconomics and Survey Data in Munich, 11 and 12 November 2011 and participants in the Eurosystem’s Working Group on Forecasting meeting in Bratislava on 22–23 September 2011. In addition, the paper has benefitted from substantial comments from four anonymous referees. The opinions expressed in this paper are those of the authors and do not necessarily reflect the views of the ECB or the Eurosystem. Any errors are the sole responsibility of the authors.
Author information
Authors and Affiliations
Corresponding author
Additional information
The opinions expressed in this paper are those of the authors and do not necessarily reflect the views of the ECB or the Eurosystem. Any errors are the sole responsibility of the authors.
Rights and permissions
About this article
Cite this article
Kenny, G., Kostka, T. & Masera, F. Density characteristics and density forecast performance: a panel analysis. Empir Econ 48, 1203–1231 (2015). https://doi.org/10.1007/s00181-014-0815-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00181-014-0815-9