Chris,

"do you have a better idea?"

No, I do not. And it was not my opinion that evaluations cannot be done when the probabilities of predictions vary -- that statement was made repeatedly by the evaluator himself.

"You want scientists to take some method or prediction record on faith?"

No, I do not. To the contrary, my opinion was very clear that if Shan or others do not describe clearly their method, then there is no way to evaluate the predictions. In that I agreed with John Vidale, for example, that there is no way to determine whether or not someone is (inadvertently or not) choosing parameters associated with large quakes in order to get aftershocks.

Chris, from your response I see you are saying "well, there is nothing better, so this will have to do" though the "this" does not meet any standard of mathematical soundness.

That's a little like sending a shuttle up next week with bad O-rings and broken tiles, because nothing else is available.

"I usually ignore your posts where you go on and on about some detail like this"

That is no problem, but the "some detail like this" happens to be a fundamental issue. You are saying that because there are no valid evaluations available, and because you do not want to take predictions on faith, you will resort to using invalid evaluation methods.

Rather, as a scientist, you should simply state that those predictions are not objectively evaluatable. Certainly I regard them as unevaluatable, and I most definitely do not take any predictions on faith.

"...I guess all my 3D fault representations are correct...beyond change or question, because not only are they 3D, they are in color, and some of them are animated!"

Your work is not the issue at all, except in that you seemed to be comparing your effort to convince people to publish it, with the efforts of predictors to get people to take their predictions seriously. That was your comparison, not mine -- I stated at the outset that I was not commenting upon your work.

I assume you meant that if the predictors do not make the same kind of substantive effort you are making to convince people, they will be stuck with nothing but an invalid "prediction evaluation" that never lets them show their predictions are worth anything. If that is what you meant, I agree wholeheartedly.