> I think there is a very good chance that some effect holds up.
Huge RCTs with many thousands of subjects only sometimes find effects for SSRIs which are the first thing doctors prescribe for depression after telling you to go get some exercise and sleep better. This study has N=21, isn't an RCT, and isn't even claiming to be studying efficacy. In terms of result robustness, this is about as strong as posting a poll in your team Slack channel.
Anyway, the study wasn't even focused on efficacy! It says so in the paper. But that didn't stop Stanford PR from putting the supposed efficacy in the headline anyway to get clicks/shares.
> Huge RCTs with many thousands of subjects only sometimes find effects for SSRIs.
This has everything to do with small effect size for SSRIs and nothing to do with this study. Doctor's prescribe SSRI's not because they are especially effective but because they are especially safe. For example MAOI's have much higher effect sizes than SSRI's. Also I'm pretty sure every single RCTs with thousands of subjects regarding SSRI efficacy has found an effect, just not always a large one, but I'd love to see a study as a counter example.
But on this to this study
What do you think a control group remission rate would be 20%, with ECT it's 48%. They had a remission rate of 90%. An effect that size is most likely a true effect or outright fraud. What is the probability this happened by accident? My probability is rusty but that seems like that would be highly unlikely. It's gotta be less than 1/1000.
If you gave a pill you thought might help people lose weight to 21 people and 90% of them last 80 lbs. Would you think "hey there's no control group", or "holy shit I probably found something".
If the entire effect were guaranteed to be attributable to the magnet intervention with 48% as the baseline likelihood of remission, the p value is ~0.0003.
That's just math.
The question is how believable it is that the entire effect is attributable to the magnet intervention.
Exciting research results announced in university press releases turn about to be something cool infrequently to put it mildly (also true for small studies, also true for medical results, also true for self-reported studies, etc), so my prior is that most or all of the effect will go away.
I've seen lots of studies come and go but I don't remember any which demonstrated as dramatic a change as this one.
This study showed a 30 point improvement on the MADRS, that's from 35 -> 5. That's insane, the typical anti-depressant move a patient 3 points.
What happened? 21 people got this technique applied to them, and 19 of them achieved remission. The average person in this study went from severely depressed to not depressed in 5 days. I'm a big fan of Gelman and the garden of forking paths, but you can't fork your way to a result like that.
Something interesting happened here, even if it's not the therapy.
Huge RCTs with many thousands of subjects only sometimes find effects for SSRIs which are the first thing doctors prescribe for depression after telling you to go get some exercise and sleep better. This study has N=21, isn't an RCT, and isn't even claiming to be studying efficacy. In terms of result robustness, this is about as strong as posting a poll in your team Slack channel.
Anyway, the study wasn't even focused on efficacy! It says so in the paper. But that didn't stop Stanford PR from putting the supposed efficacy in the headline anyway to get clicks/shares.