I think it's just harder to get a clear signal of the value that each manager is delivering. Sure there are some headed for promotion, but deciding between an average and underperforming manager can be a tougher call. And getting rid of one just means that all their ICs get reassigned to some other manager, but the immediate impact to projects/deliverables is probably pretty minimal in the short term at least.
Changing managers throws a team into mayhem for the short term. But if the manager wasn't getting results anyway, it obviously pays off very quickly.
I think like any profession, there is a spectrum, and the better you understand the skills, the clearer you can assess them. It takes a little longer for a manager's performance to play out, but underperformance is just as clear as undelivered or bad code. If the team is unhappy and results are inconsistent, the manager is to blame (fairly or not, which is part of the job).
