The hazards of evaluating predictions
Posted by Roger Hunter on October 24, 2009 at 13:18:23:

Hi all;

I've lately had a running battle with Petra. It's been my fault for issuing progress reports on her evaluation. She didn't understand why her result went from impossibly high to below average in one step.

I tried to explain it was due to a programming error but she preferred to believe it was because I was a tool of TPTB assigned to discrediting her.

In an attempt to settle the matter I offered to redo the entire evaluation from the start. New database, updated program, no predictions eliminated even though they should be.

So I downloaded everything in the ANSS catalog from mag 2 up, converted it to computer readable format and reran her evaluation on 216 predictions for 2007.

Once again I got a very significant answer from the first pass which used the entire database for probability values. But when I shortened the timespan that hign value dropped considerably.

What this tells me is that the longer time span is not reflecting current events. Essentially, the seismicity pattern has changed over time, leading to underestimates of probability. This causes a higher score for hits.

Additionally, most of her hits were multiples, with several quakes falling in the predicted window, indicating that she had predicted quakes where quakes were already happening.

The result I settled on was from a database of 10 years, from 1997 to 2006. The final score was 1.595 standard deviations which is significant at the 94.4% level.

This is better than the original run but still below the target 99% level.

I would suggest that her method may have potential for success but is not there yet.

Roger


Follow Ups:
     ● predicting aftershocks - John Vidale  17:16:27 - 10/25/2009  (76163)  (0)