Evaluation encore une fois
Posted by Roger Musson on April 20, 2001 at 08:54:25:

I don't know, I turn my back for a few days and a huge thread appears on systems for scoring predictions and evaluating them. I can't follow it all because some of the posts seem to have got chewed up.

You can argue for years about how big the fall-off margin should be and how rapidly your points should decrease if you don't hit the parameters on the nail, and you'll never come to a final agreement because the result is arbitrary whatever it is.

The only approach that makes sense is to use a system in which the precise details of the scoring system are not important. You do this by running a random prediction sequence in parallel with the real predictions. Score both of them the same way. The degree of success of the predictor is measured by the extent to which he/she does better than the dummy predictions.

If you make the scoring system really generous to the predictor, it's generous to the dummy predictions as well and so the predictor has to do really well to be better than chance. In that case any scoring system will do, and you can save a lot of argument.

After all, at the end of the day, what you want to know is not whether the predictor scores 100 points or 1000 points, but whether the results are better than chance. THAT is the thing that has to be evaluated.


Follow Ups:
     ● Re: Evaluation encore une fois - Canie  13:56:54 - 4/20/2001  (6883)  (1)
        ● Re: Evaluation encore une fois - michael  15:27:35 - 4/20/2001  (6893)  (0)
     ● Re: Evaluation encore une fois - Roger Hunter  13:06:51 - 4/20/2001  (6880)  (1)
        ● Re: Evaluation encore une fois - Roger Musson  07:53:32 - 4/21/2001  (6907)  (0)
     ● Re: Evaluation encore une fois - michael  09:50:22 - 4/20/2001  (6875)  (2)
        ● Re: Evaluation encore une fois - Roger Hunter  13:17:16 - 4/20/2001  (6882)  (1)
           ● Earthquake Prediction Registry - Under Construction - michael  16:17:47 - 4/20/2001  (6899)  (1)
              ● Re: Earthquake Prediction Registry - Under Construction - michael  16:19:30 - 4/20/2001  (6900)  (1)
                 ● Re: Earthquake Prediction Registry - Under Construction - Roger Hunter  20:19:49 - 4/20/2001  (6905)  (0)
        ● Re: Evaluation encore une fois - Don in Hollister  10:29:28 - 4/20/2001  (6878)  (1)
           ● Learning - michael  15:30:11 - 4/20/2001  (6894)  (0)