A Proposal to Evaluate Earthquake Predictions

A few years ago, I was active on the sci.geo.earthquakes forum evaluating
earthquake predictions. Recently Petra encouraged me to begin again this exercise to evaluate earthquake predictions.

When someone made a prediction with clearly defined paramters (time window, magnitude window, and space window), I would compute, based on earthquake catalogs, what the probability was of the event happening by chance. Then I would follow up when the window closed to see if one or more hits were made.

For a period of time, quite a bit of excitement was generated on the when Dennis Gentry hit on 11 out of 12 predictions. However, since his predictions were sometimes for events that had a 90% or better chance of happening randomly, it took 12 predictions before he approached being statistically signifcant at the 95% level. However, he then missed on the next few predictions relegating him to the status of a mere mortal.

With a background in science, engineering, and statistics, I am interested in evaluating earthquake predictions. I do not do this by going back and looking at a predictor's data. I have no way to know if this has been doctored. What I do is ask of those who claim to predict earthquakes is to notify me when they have a prediction. What they must provide is three windows:

Time window: Preferably UTC

Magnitude window: After an event, the largest of the commonly used magnitudes.

Space window: Clearly defined region. I prefer a "rectangular" or circular region.

Please note that the windows specified are rigorously adhered to. For example if someone predicts a magnitude 4.0 or above and a 3.9 occurs, this is a failure. If you complain that there should be some tolerance, then the person making the prediction must add on the tolerance and specify, say, that the predicted event(s) will be 3.5 and above. But notice that there is a price to this because the wider the window(s), the higher the probability that a background event will satisfy the prediction.

Dennis and Bob Shannon were good to work with because they posted predictions with clearly defined windows. Dennis' predictions typically used as his regions a set of concentric circles center on his house in Southern California. For each ring he would post his predicted magnitude range.

I would encourage anyone who believes they can predict earthquakes to join this experiment. This is the way it works: A predictor has to e-mail me (AlanJones@stny.rr.com) a prediction before the time window starts. I use a method I've developed to state what I think is the probability of this event happening by chance. To do this I used catalogs from 1960 to the present. Say a prediction of a magnitude 4.0 or greater is made for a region within 200 kilometers of downtown Los Angeles within a specified 10-day region. I then examine the catalogs for all the 10-day periods for that region since 1960. My method also eliminates the problem caused by clustering. In addition, if there has been a significant earthquake in Southern California recently, I add on the probability of an after-shock based on a formula developed by Lucy Jones and Paul Reasonberg. We then wait until the time window closes and evaluate if an event has happened which fulfills the prediction. A preliminary reading is obtained from the immediate broadcast of events but for a "final" decision I typically wait one week until the QED has been published by the (National Earthquake Information Center).

Let's take an example. Say a predictor, let's call her Sarah, makes a series of predictions of events and, for simplicity, say I evaluate each as having a 80% chance of happening randomly. If 80% of Sarah's predictions are correct, than she has done no better than chance. On the other hand, if I evaluate each of her predictions as having, say, only a 10% chance of happening and Sarah hits on 80% of them, then there is probably something to her method.

One cannot evaluate predictions by saying that a person hits on 80% of his/her predictions without evaluating what the probability is of each happening by chance. If I predict a magnitude 3.0 or larger in California every day for the next year, I will probably have a hit the majority of days. Does this mean I can predict earthquakes. Certainly not.

It is also impossible to evaluate most of the "predictions" made on this Web site. They say such things as, "Look for increased activity in the north-west Pacific over the next few days." Since activity is constantly going up and down, a prediction like this is likely to succeed about 50% of the time. Also, how does one evaluate such a prediction. If there are an increased number of small events but smaller number of large events, does this satisfy the prediction? What if it is the other way around?

If you truly believe you can predict earthquakes, then you should develop a database to relate their signals to the size and location of events and use these data to make future predictions. If these new predictions work out, then you are ready to go public and have your predictions tested.

Alan Jones
AlanJones@stny.rr.com
www.geol.binghamton.edu/faculty/jones