Implications of the Regional Earthquake Likelihood Models test of earthquake forecasts in California

The Regional Earthquake Likelihood Models (RELM) test was the first competitive comparison of prospective earthquake forecasts. The test was carried out over 5 years from 1 January 2006 to 31 December 2010 over a region that included all of California. The test area was divided into 7682 0.1°x0.1° spatial cells. Each submitted forecast gave the predicted numbers of earthquakes N_emi larger than M=4.95 in 0.1 magnitude bins for each cell. In this paper we present a method that separates the forecast of the number of test earthquakes from the forecast of their locations. We first obtain the number N_em of forecast earthquakes in magnitude bin m. We then determine the conditional probability λ_emi=N_emi/N_em that an earthquake in magnitude bin m will occur in cell i. The summation of λ_emi over all 7682 cells is unity. A random (no skill) forecast gives equal values of λ_emi for all spatial cells and magnitude bins. The skill of a forecast, in terms of the location of the earthquakes, is measured by the success in assigning large values of λ_emi to the cells in which earthquakes occur and low values of λ_emi to the cells where earthquakes do not occur. Thirty-one test earthquakes occurred in 27 different combinations of spatial cells i and magnitude bins m, we had the highest value of λ_emi for that mi cell. We evaluate the performance of eleven submitted forecasts in two ways. First, we determine the number of mi cells for which the forecast λ_emi was the largest, the best forecast is the one with the highest number. Second, we determine the mean value of λ_emi for the 27 mi cells for each forecast. The best forecast has the highest mean value of λ_emi. The success of a forecast during the test period is dependent on the allocation of the probabilities λemi between the mi cells, since the sum over the mi cells is unity. We illustrate the forecast distributions of λ_emi and discuss their differences. We conclude that the RELM test was successful in illustrating the choices required when a forecast of the location of a future earthquake is made.