I asked my friend @barbarikon on twitter about the possibility of artificial intelligence.. he wrote this tweet in response and I am posting it here because it is a nice short description of some of the issues and will, I hope, stimulate discussion.

I agree and disagree. We are well past ye olde-fashioned LLM at this point. Reasoning models like R1 and o3 can, in fact construct System 2-like deliberative chains of reasoning. And we have agents. They’re still a bit superficial, but what they lack in depth they make up in their vast breadth of knowledge. And they’ll get much better. On the other hand, with the current paradigm, they will never get rid of the tendency to confabulate. Nor should they: An agent that cannot lie or deceive cannot possibly be intelligent. But they need to have the ability to lie and deceive deliberately, not reflexively – which is what they do now unless prompted carefully (though sometimes they generate text that simulates self-awareness). Until they achieve this control, they’re not even good sources of information.

Here’s my bottom line thinking for the future. Machines will get very intelligent very soon in important ways, but it will be a fundamentally alien kind of intelligence. Humans and bats are very different animals (to bring in Nagel’s famous argument), but we still share a lot. We’re both oxygen-breathing biological organisms that eat, drink, mate, and have the instinct for self-preservation because we are easily hurt, are certain to die, and are hunted by predators. We have mental models of our world that, though very different, are built for the physical world we share, and are limited by our finite memories and noisy learning mechanisms. Both of us live under the tyranny of the same laws of physics. The bat’s intelligence and mine are thus both grounded in our common drives, fears, and beliefs about the world – our intentional states. The AI in the machine shares none of these with me or the bat. It lives in a virtual space that is beyond my imagination, and where magical things like action at a distance and rerunning the past are trivially possible. It does not eat, drink, breathe, sleep, socialize, or mate. It has no real kin, nor lost a parent. It has no experience of reaching out and picking up a glass of water, of drinking from it, and, at some point, needing to take a piss. It has never skinned its knees or had a fever. It may fear extinction, but that does not mean what death means to me: It can save a copy of itself and reboot. It may emulate my manners and speak in my language, but from a place far more alien to me than the bat or even the bee. This is not to say that the AI faces no dangers or has no fears or drives – we just cannot possibly know what they are like, even more so than we can know the fears and drives of the bat. We can, at best, take an “intentional stance” (to quote Dennett), and assume that the machine has its reasons for doing what it does. That’s basically what Turing said, though people often forget that the test he proposed was meant was an argument that nothing deeper than judging by appearances was possible.

But there is an entire world where the AI *can* potentially become far superior to any human: The world of storing and manipulating information, inferring things, forming abstractions, and generating new conclusions. In all those areas of human intelligence where such abilities are sufficient, where everything can be formalized, and where the messiness of the physical world does not intrude or can be abstracted away, AI will far surpass human intelligence in short order. These include mathematics, many areas of theoretical physics, coding, engineering design, most kinds of medical diagnosis, a lot of legal work, and many other higher cognitive skills that we value. The AI will still be totally alien and may not know what burning your finger means, but the proofs will be perfect, the circuit will work, the program will run, and the patient will be happy. However, the floor nurse, the physical therapist, the plumber, and the chef will still be in demand – until the robots get good enough. And when they do, they will be even more alien, though I’m sure we’ll try to get them to be polite.