"Boring" AI safety is not a major risk. It's basically an extension of "humans can be reckless and incompetent" - a threat faced by human civilization since before recorded history. There's a very limited amount of people a Therac 25 can kill. Even Bhopal disaster has only caused this much harm.

Now, an AI that can play the game of human politics and always win, the way a skilled human can always win against the dumb AI bots in Civilization V? There is no upper bound on how bad that can go.

I know I've got a hard-earned cynical streak, but I am more worried about the mass sociological cascade. I absolutely don't worry about some ultra smart AI having malignant intent or even being enslaved by someone with malignant intent. Instead, it's the weird Tulip Mania or Salem Witch Trial risks that I do not know how to quantify.

How severely can the mass delusions and magical thinking around AI mix with other regressive social trends? I do not relish the mass effect of people deferring their decision making to glorified Magic 8 Ball devices, treating them as oracles rather than syntactic lava lamps. Will it lay waste to a generation of education, thinking, science, and policy so that society itself acts out some kind of Therac 25 or Bhopal incident at scale?

The difference is scale.

Humans can be reckless, but the scale is so unbelievably small that's its not a big problem.

Like, you can't accidently do the Holocaust. Because that requires deliberate coordination of millions of people.

Can AI accidently do the Holocaust? Yeah, probably. Its software, it can duplicate itself infinitely.

It think it is about killer robots as in someone intentionally deploying Therac 25 Incidents.