The point is more so that the pin unlocks a key on your local device and that key is much stronger than the password the typical user would select. Plus it is site specific in a way that your typical user does not do with passwords.

So it's making a system weaker against offline attacks if someone steals your hardware in exchange for making it stronger against phishing. This is probably the correct tradeoff for most people.