Evaluating Density Forecasts with Applications to ESPF

This paper evaluates density forecasts using micro data from the ESP forecast (ESPF), a monthly survey of Japanese professional forecasters. The ESPF has collected individual density forecasts since June 2008. We employ two approaches, Probability Integral Transform (PIT) and Ranked Probability Score (RPS). First, we apply Berkowitz’s (2001) test to individual density forecasts produced every June. We fail to reject the independency in FY 2010 and 2011 real GDP growth rates. As for CPI inflation rates, we reject the independency in all the samples during FY 2008 to 2011, but fail to reject it if the sample is limited to a half with better forecast performance. The result may ensure individual densities coincide with unobserved true data generation process of the actual outcomes. Second, we calculate RPS, following Kenny, Kostka, and Masera (2012), and compare the Mean Probability Distribution (MPD), the average of individual densities, with three benchmarks -- Uniform, Normal and Naïve distributions -- and individual density forecasts. The MPD turns out to be a “good” density: it beats the benchmarks in most cases and ranks about fifth out of around 35 participants every year. Subjective judgments added to the MPD are likely to deteriorate the performance in the case of CPI inflation rate, but to improve in the case of real GDP growth rate.