I buy more than I return so the "pain in the butt" cost is still lower. Plus Amazon makes it really, really easy to return stuff, you just print the label and drop it off.
Good point. Returns usually are exception conditions, so one need only factor in a fraction of the "cost" of a return when making most purchases. However, when I order two sizes of the same shirt, I obviously will be returning one of them.