Brian;

There are two tests of significance which may be used, depending on the circumstances.

If all predictions have the same probability the z-binomial test may be used. This is a standard test of significance and may be found on the VassarStats website as an interactive applet.

If each prediction has a different probability the method devised By Dr Jones will be used. This procedure assigns a score to each prediction. If the prediction is correct the score is -ln(p) and if it's incorrect the score is ln(1-p), where ln is the natural log and p is the probability of success. Each prediction has an expected value which is -p*ln(p) + (1-p)*ln(1-p) and a variance which is p*(1-p)*(ln((p)*(1-p)))^2.

These values are summed and the standard deviation is the square root of the variance. The final answer is (score-expected)/standard deviation. This quantity is normally distributed so the significance is determined by comparing it to a table of the normal distribution.

Clear?

Roger