Logout 
Portal 
Forum
Search
Members
Help 
12-30-2013, 08:40 PM
FOR FUTURE REFERENCE
These are the rules I follow when evaluating predictions. I do not claim that there are no other ways to evaluate nor do I claim to be the only evaluator around.
Roger
-------------------------------------------------------------------
Principles of Earthquake Prediction Evaluation.
Prediction requirements
1) Date
The prediction must specify the date when the quake will happen. If a date range is given, the quake must happen within that range. Near-miss dates may be specified but will receive smaller scores based on probability of success. If times are given, time zone must be specified.
2) Location
Location must be given in an unambiguous manner. Center point and radius, lat-lon boundaries or areas which have defined borders such as states are acceptable.
3) Magnitude
Magnitudes must be given, usually as a range such as 5.0 to 9.9. A single value such as 5 would be taken as 5.0 to 5.9 whereas 4.5 would allow no other answer. NOTE: Predictions for quakes smaller than 4.5 will not be evaluated.
4) Probability
The expected probability should be the statistical chance of success, not the predictor's subjective confidence level. It may be omitted as it will be calculated as part of the evaluation.
Practices which are not allowed.
1) Predictions must be made before the expected date
2) A series of predictions for the same location may not have consecutive dates.
3) Predictions may not overlap in both date and location.
Evaluation procedures
1) Predictions from only one source at a time will be evaluated. The exception is when a group jointly contribute to predictions.
2) Earthquakes from only one source will be examined. The source may be selected for special situations but not on a case by case basis. Sufficient time must elapse before evaluation begins since quake parameters may be revised while new information is coming to the centers.
3) Predictions will be judged strictly on what is said. "Close enough" is not allowed except when near-miss regions are defined in advance.
4) Predictions will be awarded the first quake which fits all the parameters as stated.
5) Each quake can satisfy only one prediction and will not be considered for other predictions.
6) If a quake has been given multiple magnitudes the largest value will be used.
7) Aftershocks will be accepted if the prediction was made before the main shock happened.
8) Probability is necessary in determining significance. It will be assigned in one of two ways. When all predictions have the same probability, as when predictions are essentially identical except for dates, the probability will be determnined by the Jones Observed Probability method and that value will apply to all such predictions. If predictions are all different, values will be computed by the Jones method for each prediction.
Significance calculations
There are two tests of significance which may be used, depending on the circumstances.
If all predictions have the same probability the z-binomial test may be used. This is a standard test of significance and may be found on the VassarStats website as an interactive applet.
If each prediction has a different probability the method devised By Dr Jones will be used. This procedure assigns a score to each prediction. If the prediction is correct the score is -ln(p) and if it's incorrect the score is ln(1-p), where ln is the natural log and p is the probability of success. Each prediction has an expected value which is -p*ln(p) + (1-p)*ln(1-p) and a variance which is p*(1-p)*(ln((p)*(1-p)))^2.
These values are summed and the standard deviation is the square root of the variance. The final answer is (score-expected)/standard deviation. This quantity is normally distributed so the significance is determined by comparing it to a table of the normal distribution.
The Jones Observed Probability method
The problem with earthquake catalogs is that the quakes fall into clusters caused by swarms and aftershock sequences. Thus simple probability based on the number of days covered by the predictions can give very misleading answers. The usual procedure is to decluster the catalog by various methods.
The Jones method has the advantage of simplicity and while it is not perfect, it greatly reduces the clustering influence, particularly if long windows are being predicted. The procedure is as follows:
1) Select the time span of interest.
2) Divide the time span into consecutive windows of the predicted length
3) Compute the total number of windows in the time span
4) Count the number of windows containing one or more quakes
5) Divide the number of windows containing quakes by the total number of windows. The answer is the probability that such a window will contain a quake.
The time span to be used is a matter of opinion. A long time span gives a long term average answer whereas a shorter time can better suit changing circumstances. Care must be taken that the time span does not exceed the catalog's reliable period inasmuch as network capability increases with time.
These are the rules I follow when evaluating predictions. I do not claim that there are no other ways to evaluate nor do I claim to be the only evaluator around.
Roger
-------------------------------------------------------------------
Principles of Earthquake Prediction Evaluation.
Prediction requirements
1) Date
The prediction must specify the date when the quake will happen. If a date range is given, the quake must happen within that range. Near-miss dates may be specified but will receive smaller scores based on probability of success. If times are given, time zone must be specified.
2) Location
Location must be given in an unambiguous manner. Center point and radius, lat-lon boundaries or areas which have defined borders such as states are acceptable.
3) Magnitude
Magnitudes must be given, usually as a range such as 5.0 to 9.9. A single value such as 5 would be taken as 5.0 to 5.9 whereas 4.5 would allow no other answer. NOTE: Predictions for quakes smaller than 4.5 will not be evaluated.
4) Probability
The expected probability should be the statistical chance of success, not the predictor's subjective confidence level. It may be omitted as it will be calculated as part of the evaluation.
Practices which are not allowed.
1) Predictions must be made before the expected date
2) A series of predictions for the same location may not have consecutive dates.
3) Predictions may not overlap in both date and location.
Evaluation procedures
1) Predictions from only one source at a time will be evaluated. The exception is when a group jointly contribute to predictions.
2) Earthquakes from only one source will be examined. The source may be selected for special situations but not on a case by case basis. Sufficient time must elapse before evaluation begins since quake parameters may be revised while new information is coming to the centers.
3) Predictions will be judged strictly on what is said. "Close enough" is not allowed except when near-miss regions are defined in advance.
4) Predictions will be awarded the first quake which fits all the parameters as stated.
5) Each quake can satisfy only one prediction and will not be considered for other predictions.
6) If a quake has been given multiple magnitudes the largest value will be used.
7) Aftershocks will be accepted if the prediction was made before the main shock happened.
8) Probability is necessary in determining significance. It will be assigned in one of two ways. When all predictions have the same probability, as when predictions are essentially identical except for dates, the probability will be determnined by the Jones Observed Probability method and that value will apply to all such predictions. If predictions are all different, values will be computed by the Jones method for each prediction.
Significance calculations
There are two tests of significance which may be used, depending on the circumstances.
If all predictions have the same probability the z-binomial test may be used. This is a standard test of significance and may be found on the VassarStats website as an interactive applet.
If each prediction has a different probability the method devised By Dr Jones will be used. This procedure assigns a score to each prediction. If the prediction is correct the score is -ln(p) and if it's incorrect the score is ln(1-p), where ln is the natural log and p is the probability of success. Each prediction has an expected value which is -p*ln(p) + (1-p)*ln(1-p) and a variance which is p*(1-p)*(ln((p)*(1-p)))^2.
These values are summed and the standard deviation is the square root of the variance. The final answer is (score-expected)/standard deviation. This quantity is normally distributed so the significance is determined by comparing it to a table of the normal distribution.
The Jones Observed Probability method
The problem with earthquake catalogs is that the quakes fall into clusters caused by swarms and aftershock sequences. Thus simple probability based on the number of days covered by the predictions can give very misleading answers. The usual procedure is to decluster the catalog by various methods.
The Jones method has the advantage of simplicity and while it is not perfect, it greatly reduces the clustering influence, particularly if long windows are being predicted. The procedure is as follows:
1) Select the time span of interest.
2) Divide the time span into consecutive windows of the predicted length
3) Compute the total number of windows in the time span
4) Count the number of windows containing one or more quakes
5) Divide the number of windows containing quakes by the total number of windows. The answer is the probability that such a window will contain a quake.
The time span to be used is a matter of opinion. A long time span gives a long term average answer whereas a shorter time can better suit changing circumstances. Care must be taken that the time span does not exceed the catalog's reliable period inasmuch as network capability increases with time.