I see that you work in particle physics. I don't think particle physics is representative of science in general. My impression is that particle physics is seen as sexy and attracts a larger proportion of better researchers than many other fields for that reason.
The stagnation in fluid dynamics (my general field) seems fairly obviously linked to incentives. Pumping out papers is seen as more important than doing a good job as far as I can tell. In the past few years I think I've made a lot of progress in my subfield by simply compiling tons of data from the open literature [1], which apparently no one thought to do on the scale that I've been doing. (It's a lot of work up front, which would lead to a delay in publishing results.) In the process I've found that a lot of what's written in review articles and books on the subject is obviously wrong. This is not a sign of a healthy field! And I don't think you can do this in particle physics, but I think you can in many fields of engineering.
My experience in research comes from starting in very fundamental fluid mechanics (vortex dynamics and instabilities), moving into biomedical (fundamental and applied) and turbulence (fundamental) and now working in applied stuff.
I think part of the problem is that the field is stagnating partly because of the amazing work done in the last century ticked soooo many of the boxes. It was really the golden period ending with Prandtl and Karman - like the end of the ultimate share house. A lot of the tree has been stripped bare by the 70s and most of the advancements have been refinements and application specific. Obviously computational techniques have exploded and experimental methods have improved and there are still new discoveries, but in terms of impact, outside of microfluids and the never-ending grant gift of the turbulence closure model we are entering a dry season where grants are very much application specific and the fundamental physics is simply not rewarded. We used to have a bet what year the fluid mechanics chair will no longer be a thing at universities. It doesn't help that the fluids community is not super tight knit (lots of beef!)
I also 100% agree with you in terms of papers vs good job - but that is science in general nowadays - it will always be a tiny percentage that actually progress the field, a bunch that try and fail as research is want to do and a majority just try and stay relevant by pumping out rubbish.
Could you elaborate on some of the obviously wrong stuff that you've seen? I'm curious. I have an ME background but strayed away from it into electronics, then oddly enough ended up in a fluids heavy business.
This paper is on the different varieties of liquid jet breakup, e.g., for applications like fire hose streams and fuel sprays. At low speeds you have more regular breakup but this changes to various types of less regular breakup at higher speeds. (There are photos of each type in the preprint.) These varieties are called "regimes" in the literature. Because models are typically only valid in a particular regime, it's important to identify the correct regime. This is often done with a "regime diagram".
There's a lot I could write about what was wrong before. The easiest thing to do is compare figure 3 (old regime diagram, p. 5) and figure 4 (my new diagram, p. 13). These are in exactly the same coordinates but don't resemble each other much. If science were working correctly then the change would not be anywhere near as dramatic as it was.
Now, with the small amount of data past researchers used to construct these diagrams, they couldn't see that the diagram was wrong. The data was simply too sparse to see the big picture. But once you start adding tons of data it becomes dead obvious that the diagrams you see in textbooks and review articles are wrong. So I can't blame previous researchers that much, but compiling open data is something that should happen regularly. The current academic system does not incentivize data compilation, at least in engineering.
The most recent study mentioned in the paper used a grand total of 11 data points and claimed to have enough resolution to move some of the established boundaries slightly. This actually is a regression. The first study to construct a regime diagram had 63 data points. Mine has roughly 1200, and I still want more data! 11 data points is not scientifically acceptable, but it was enough to get a publication.
(I've since revised and extended this paper but I can't upload the new version yet due to the publisher's policies. One example: I determined that "turbulent dripping" is probably not possible so I removed it from the diagram. ;-)
I could list more if you want. There's no shortage of problems. But keep in mind that these problems are usually only obvious after you get enough data.
The stagnation in fluid dynamics (my general field) seems fairly obviously linked to incentives. Pumping out papers is seen as more important than doing a good job as far as I can tell. In the past few years I think I've made a lot of progress in my subfield by simply compiling tons of data from the open literature [1], which apparently no one thought to do on the scale that I've been doing. (It's a lot of work up front, which would lead to a delay in publishing results.) In the process I've found that a lot of what's written in review articles and books on the subject is obviously wrong. This is not a sign of a healthy field! And I don't think you can do this in particle physics, but I think you can in many fields of engineering.
[1] https://github.com/btrettel/pipe-jet-breakup-data