We dilute, pervert, change, invert, transform and reshape the meaning of words all the time, throughout history and across languages. I don't see anything special about the words we use to describe sex and gender.
We can change "war" to mean "peace", "freedom" to mean "slavery", "ignorance" to mean "strength", and "man" to mean "woman". We shouldn't, though. Some meanings are worth preserving, because they have deep legal implications.
FWIW, I try to use "male/males" to refer to sex, and "man/men" to refer to gender, since AFAICT, there are no terms that clearly refer to gender.