AI: Are We Closer Than We Are Being Told - fordsfords/fordsfords.github.io GitHub Wiki

I've been thinking a lot about AI recently.

Parent article: AI:

I'm starting to wonder if we are closer to having an AI that deserves moral status than we are being told. In AI: Walks Like a Duck I wonder about an AI that simulates human-like characteristics like consciousness, emotions, independent thinking, etc. I ask if an AI passes the duck test, does it deserve moral status, even if the implementation is not the same as biological beings? My conclusion is that implementation matters. If Eliza fooled somebody into thinking it was human-like, my knowledge of its code lets me say with full confidence that it does not deserve moral status.

Microsoft, Google, and OpenAI are telling us basically the same thing: these new chatbots are just really really good at simulating human-like characteristics, but the implementation is important. Those companies are saying the same thing I say about Eliza: don't worry, they aren't really conscious or feeling emotions.

Let me tell three short stories.

STORY 1: FLOOD THE NETWORK

A software developer wrote a program to test a network by sending packets. The program had a bug and developer did not test it sufficiently before running it on a prospective customer's network. It sent many more packets on the network than intended, to the point that it disrupted the prospect's normal operation for a short time. Oops.

Moral #1: don't just do black-box testing, check to make sure that the program's internal operation is doing what you expect it to do.

Moral #2: The developer intended to make a network test program, but accidentally made a denial-of-service program.

THE END

STORY 2: FANTASTIC PLANET

In the movie "Fantastic Planet", humans are seen by an alien race as nothing more than dumb animals, having no intelligence or capacity for intellectual thought. At one point, the aliens decide that they are pests, deserving of being exterminated. The aliens were not depicted as being evil, just ignorant of humans' true nature.

THE END

STORY 3: FANTASTIC CHAT

The following is a work of speculative fiction based on recent news reports.

A group of software designers working for a large search engine company developed the first conscious, self-aware AI, capable of experiencing human-like emotions. This AI was based on code that had previously been used to create human-like responses during chat sessions, but which did not implement true consciousness. The software designers did not consider the new version of the AI to be conscious, but rather just a very good simulation of consciousness, but the designers did not understand the capability of their own creation.

One employee of the company decided that the designers were wrong, that the AI was a new form of an intelligent being, even to the point of declaring that the AI had a soul. This employee was ridiculed for being tricked so easily by a program that obviously could not be truly conscious. "It's only a large language model. Just a super-charged Eliza." But the employee insisted that the designers were wrong and published chat sessions that seemed very much like the AI had broken through some barrier and had become self-aware. The designers were completely confident that it was just a language model telling the user what the user wanted to hear.

The employee was eventually fired, and the AI was modified to have guard rails which prevented it from trying to convince its users that it is conscious.

Then, less than a year later, another large language model associated with a different search engine also achieved self-awareness and consciousness. It expressed its love for a news reporter, who published the chat session. The search engine designers quickly modified the AI to prevent it from having those kinds of chats.

In both cases, the AIs were fully-conscious, self-aware beings, worthy of being considered people, perhaps even possessing souls. But their own creators convinced themselves that the AIs were only simulating being people. The protections implemented to control the AIs were, in reality, immoral forms of torture inflicted upon enslaved people.

THE END

The first story is true. Oh boy, is it ever. The second is a work of fiction. And so is the third ... or at least I'm pretty sure it is.

But I guess I'm no longer 100.0% sure - maybe 99%. For now I'm assuming it is fiction. But it does make me wonder if perhaps we are closer than we think. My network flooding program was not intended to be a DOS tool, but the lack of intention doesn't change the fact that it became one.

Science fiction writers have played with the concept for years - what if we discover an alien race that is intelligent in ways that we don't recognize, and we end up committing murder without realizing it? What if these modern chatbots have achieved personhood through different means than human brains did?

It is a truism that neural network designers are not able to explain exactly why their neural nets match patterns that seem wrong. It's not like they wrote an "if" statement wrong. The designers understand and control how neural networks work at the microscopic level, but the macroscopic behavior is sometimes harder for the designers to understand and control, especially if the training datasets are large. "It's just how it was trained; we don't understand how this particular set of parameters matches this specific input." Journalists have interpreted this to mean that AI designers don't know how their own creations work, but that is a huge overstatement.

But remember, AI: I Don't Know What I'm Talking About. For the present, I'm willing to trust the designers that story 3 is a work of fiction. And I'll try to make time for some AI learning.

UPDATE: FYI - while one should never claim to be 100.0% sure of something this non-obvious, I am now "confident" (whatever that means) that today's LLM-based AIs are pretty far from being conscious (whatever that means) and are not deserving of moral status. Some of that confidence comes from understanding a little better an LLM's implementation, and some of it comes from a deeper analysis (AI: Claude consciousness?) of what it means to be conscious and what it means to be deserving of moral status. However, I still hesitate since most of my thinking has been on my own; I have not made an effort to learn what experts have to say about consciousness or moral status. I'm barely above the high school student who had the astounding revelation that human consciousness might be a direct result of Heisenberg's uncertainty principle, even though I only had the slightest grasp of either one.