Decidability. We have intuition to decide what we cannot reason. A machine needs the same intuition (biases). We could give it a bias to ‘give up’ or ‘go to sleep’. Or we could give it a bias to merely ‘talk’. We don’t like to confront the fact that ‘consciousness’ of a human being relies upon a competition between a predator-bias, and a prey-bias. We can likewise create all AI’s in pairs sharing the same memory but relying upon different decidability (weights), one with a change bias, and one with a safety bias, with decidability provided by the differences (limits). I don’t fear AI because I have worked on the problem for a long time and I understand that most of the experience of human consciousness evolved to keep us motivated amidst extraordinary informational challenge. All AI’s have to do, is what we do: no violate property (investments) of others. The difference is that it’s actually easier to regulate an ai with algorithms. With people we need norms, traditions, laws, courts, and punishment, and we still are just barely good enough at it.
The problem is creating and enforcing a death sentence for every single person involved in creating any other kind of AI.