David (all)
It is quite possible to reduce human categories, relations and values to a correspondent, consistent, coherent, and commensurable set of references.
And as far as I know we can also simulate the same changes in state within the machine that we experience via our reward systems (emotions).
Now, just as some people can’t imitate your actions, sympathize with your thinking, and empathize with your feelings, because the differences in frames are too hard to overcome, I suspect that machines will ‘get it wrong’ with some of us now and then.
Some people want to work with the neural network architecture. And NN’s are exceptional at reducing inputs to symbols (objects, relations, values). And in my opinion the NN ceases being useful(cost effective and controllable) at the point at which we develop symbols (objects, relations, values).
Beyond the construction of symbols, the game engine (algorithmic) architecture appears to be a better solution.
We can create transparency, auditability, and yes, conscience, as long as we use such an architecture. For example, imagine that you could only act upon what you could vocalize?
I’ll also stay on message that the fundamental problem any intelligence faces is taking action to conduct tests. Ergo the primary problem with ‘superintelligence’ or even ‘general intelligence’ as I understand it, (and I have been at this a long time), are the same as the primary problem with knowledge creation in any other field: cost, logistics, and permission to organize a network of actions.
There don’t seem to be many other people in AI that have both an understanding of computer science, an understanding of economics, and the economics of the increasingly complex problem of experimentation whether personal, political, entrepreneurial, technical, or
It is, believe it or not, within one lifetime, possible for a human being of adequate ability and time to comprehend the limits of each and every major discipline. And it is equally possible to ‘keep up’ with the current status of those disciplines.
As far as I know computers will not mine much that is good out of the existing base of knowldge.
The primary difference is that machies will be able to think and act faster than us in work capacities.
In creativity? Creativity isn’t a process of reasoning but free association, and experimentation.
The problem in every field today is that where it took one person to solve a scientific problem a century ago, it takes armies of them today.
The problem is cost.