This is bound to happen in any industry where supply of talent exceeds demand AND the powers-that-be can get away with collusion (either by virtue of being 'private' or simply being powerful and thus not give a shit).
So what if one talented employee decided to quit one crappy company 'A' AND is unable to find work at the non-poach pact company 'B'? The place he quit will always find another willing sucker 'coz the supply is ever present (with, at most, a minor reduction in quality).
When supply exceeds demand, you do not need collusion, because there are unemployed workers who you can just hire at your target rate. If that is not enough money to entice them to work for you, then it would still not be enough even if you colluded.
What collusion allows is for the employers to maintain these low prices, even when demand exceed supply.
So what if one talented employee decided to quit one crappy company 'A' AND is unable to find work at the non-poach pact company 'B'? The place he quit will always find another willing sucker 'coz the supply is ever present (with, at most, a minor reduction in quality).