I think that for any travel there is some critical time when it doesn't matter how long it is as long it is < 24h and more than a long commute. You are wasting a whole day for that leg of the trip anyway no matter if the flight is 4 or 10 hours. So why waste money.
It makes some difference but I agree with your basic point. Especially as you have to add 2 hours or so buffer for check-in, security, etc.--especially for an international flight.
And if you're looking for more generalized "reduced pain of travel," the reality is that really comfortable seating arrangements and decent food/drink can be delivered on a sub-sonic widebody for a lot less $$ than it would cost you to travel the route on a hypothetical supersonic passenger jet.
Transpacific is probably where the difference would matter the most, assuming the supersonic jet had that kind of range (which is sort of doubtful). But it's worth noting that, even with today's jets, some really long haul routes that would significantly cut travel time for some don't make economic sense and have been eliminated (e.g. New York-Singapore).
There is a big difference between a 4 and 10 hour flight. For a 4 day weekend, I wouldn't take a 10hr trip but 4 is doable. Of course, I live 20 minutes from an international airport.
