The problem is that for many other people, this one feature is the only reason to even have Signal - they won't bother with it if it can't be used to message random people already on their contact list.

And then once those people drop it, you have to do so as well if you want to keep talking to them.