If everybody's convinced by AI safety arguments, then the game matrix changes and the dominant strategy is to just not do it. You lose if anyone does it including yourself. It works for the same reason that nobody sets off a giant cobalt bomb and kills the planet.[1]

Our unfortunate situation right now is that we don't understand the physics well enough to really be confident of consequences.

http://www.nuclearfiles.org/menu/key-issues/nuclear-weapons/...