They claim that "laypeople" can predict if the study is reproducible, but they only got a 59% accuracy. It's technically true, but it's too close to 50%. I can't find how many of the 27 studies there "reproducible" and how many no. Specially because half of the subject were first year students of psychology that should have a minimal idea of the subject.
And the 67% when the people is informed about the strength of evidence is even less impressive. How much accuracy would have a parrot that just repeat the information?
To be fair they translate the strength to the evidence from a numeric scale to a simple words scale, and the participants should translate it back to a numeric scale. The real question is how good is people doing this task with random numbers, without additional information like the description of the study.
And the 67% when the people is informed about the strength of evidence is even less impressive. How much accuracy would have a parrot that just repeat the information?
To be fair they translate the strength to the evidence from a numeric scale to a simple words scale, and the participants should translate it back to a numeric scale. The real question is how good is people doing this task with random numbers, without additional information like the description of the study.