I think there is something to be said for when one first discovers Bayes; not least because its often presented like this article as a sort of enlightenment. Perhaps the finest work on the topic is probability theory by Jaynes; if the underpinning of the Bayesian sect was to have a bible that would be it and BDA3 would be its practical counterpart. I say this both owning and having read both and having invested a a lot of time purveying, learning and applying the Bayesian perspective.
What Bayes makes possible, from an applied statistics perspective, is a kind of unification of a large variety of modelling approaches into a single framework fitted in the same way: simple regression yes, but also hierarchical models, mixtures, pooled, partially pooled, hurdle, regularised, horse shoe, and so on. So when you learn Stan or PyMC3 or Nimble or whatever, you're enabled to go forth and make myriad custom models: this is powerful, and it is enough to respect Bayes.
Various results show that lots of other models can in principle be expressed in a Bayesian way given the right prior; hinting in some theoretical sense that Bayes is a universal modelling approach: the panacea you've been looking for young statistician.
However, Bayes has many epistemological problems and for the actually interested reader see here for a summary:
For those, like myself, that are out there (like Breiman was) trying to actually solve real world problems; you find yourself quickly limited by the Bayesian approach. The data generating process of the real world is totally not obivious MOST of the time but you are forced as a Bayesian to pretend otherwise. As much as I love problems where Bayes does work well; they are fairly few and far between for me.
So as a closing comment; lets not "Bayes all the things", as tempting as it is. It is in many respects the first part of the journey for many avid evangelists and self confessed "how I became a Bayesian" converts, but its not the end all.
> The mind doesn’t always work like Bayes Theorem prescribes. There’s lots of things Bayes can’t explain. Don’t try to fit everything you see into Bayes rule. But wherever beliefs are concerned, it’s a good model to use. Find the best beliefs and calibrate them!
The one concern I'm really interested by, and don't understand, is this:
> There are thus lots of reasons to believe that we do not think and should not think in a Bayesian way.
Can you give an example that can serve as motivation to read the entire paper linked?
Section 6.2 gives a very concise list of problems.
For further reading have a go at para-consistent logic for more total upheaval of the Jaynesian view of the centrality of predicate logic as the principal language of science (of which Bayes is a probabilistic relaxation according to Jaynes).
I didn't allege it wasn't a problem for Frequentists in a similar situation. I'd add that the modelling world is very far from partitioned into Bayesians and Frequentists.
What Bayes makes possible, from an applied statistics perspective, is a kind of unification of a large variety of modelling approaches into a single framework fitted in the same way: simple regression yes, but also hierarchical models, mixtures, pooled, partially pooled, hurdle, regularised, horse shoe, and so on. So when you learn Stan or PyMC3 or Nimble or whatever, you're enabled to go forth and make myriad custom models: this is powerful, and it is enough to respect Bayes.
Various results show that lots of other models can in principle be expressed in a Bayesian way given the right prior; hinting in some theoretical sense that Bayes is a universal modelling approach: the panacea you've been looking for young statistician.
However, Bayes has many epistemological problems and for the actually interested reader see here for a summary:
https://plato.stanford.edu/entries/epistemology-bayesian/
There are thus lots of reasons to believe that we do not think and should not think in a Bayesian way.
For other mind expanding papers consider Andrew Gelman (very prominent Bayesian) and Cosmo Shalizi (of 3 toed sloth fame):
https://arxiv.org/abs/1006.3868
and Breiman (of random forest fame):
https://projecteuclid.org/euclid.ss/1009213726
For those, like myself, that are out there (like Breiman was) trying to actually solve real world problems; you find yourself quickly limited by the Bayesian approach. The data generating process of the real world is totally not obivious MOST of the time but you are forced as a Bayesian to pretend otherwise. As much as I love problems where Bayes does work well; they are fairly few and far between for me.
So as a closing comment; lets not "Bayes all the things", as tempting as it is. It is in many respects the first part of the journey for many avid evangelists and self confessed "how I became a Bayesian" converts, but its not the end all.