I think I asked this last time I saw a Statwing analysis, but are you guys accounting for multiple comparisons anywhere? The actual p-values are quite robust here, but I can generate some pretty spurious analysis by just running every comparison at once.
We still don't account for them (except in the context of ANOVA post hocs).
The thinking is that (1) we're similar to other stats tools in that it's incumbent on the user to account for that, (2) a 'practical' version of accounting for multiple comparisons is to just be aware of them (as per your "p-values are quite robust here" comment), and (3) eventually this will be a really cool opportunity for us to stand out, and we do plan on eventually accounting for them--we just haven't really been able to prioritize it at this point.
Thanks for the feedback, we're very happy to have that comment brought up quite a bit, it's definitely really important, especially given the goal of democratizing data analysis.
I think I just always bring it up since your tool moves directly to data mining operations and I know my presentation of so many results will be highly likely to have false positives.
Yeah.. I thought the same. People reporting stats to the public at large lack incentives to adjust for multiple testing: most of their audience do not know enough to care; and to them, the value of the product improves with the number of 'insights' found. It is perverse but real.
