Roger and Michael,
I have been off today enjoying a beautiful 80 F day in the mountains of Colorado. I'll
get back to you on the other questions as soon as I have a bit more time. Still
working on that question you gave me Roger, and should have some answers this
weekend.
As for the request that we make regarding the predictor's confidence in his prediction,
Roger is right. This is an "out " for the predictor. It is not actually used in any evaluation,
just as additional comments are not actually used, but may be helpful in determining
exactly what the predictor thought. The other reason for the confidence question is
that we hope it will cause the predictor to begin thinking about questions like - how
often am I right? What is the probability this event will occur (even if my technique
has no validity). We want predictors to think about these issues. This is our way
of trying to bring some social consciousness to the process.
Making a prediction should be a learning experience for all concerned. That is our
real goal in taking predictions.