Can Machines Think? Eugene Passes Turing Test

Can Machines Think? Eugene Passes Turing Test

But if you think it's all over, think again. There are other tests ...

Back in 1950, computing genius Alan Turing came up with a baseline test to determine whether or not a computer could be considered intelligent. Though often expressed as 'can machines think' - the words used by Turing himself in the opening sentence of his thesis - Turing warns that the better question, since thinking is difficult to define, is "are there imaginable digital computers which would do well in the imitation game?" If a machine under test conditions can fool 30% of the judges, each of whom have a few minutes' conversation with the device, convincing them it is a human rather than a machine, it passes the test; it is, in other words, a perfect imitation. This weekend Eugene Goostman did exactly that, persuading 33% of the judges at the Royal Society of London that 'he' was a real 13-year-old.

"This event involved the most simultaneous comparison tests than ever before, was independently verified and, crucially, the conversations were unrestricted," says Professor Kevin Warwick of Reading University. "A true Turing Test does not set the questions or topics prior to the conversations. We are therefore proud to declare that Alan Turing's Test was passed for the first time on Saturday."

There have already been criticisms that it's easier to imitate a 13 year old than an adult, since a teenager isn't expected to know as much as an adult. Its creator, Vladimir Veselov, admits that this was part of Eugene's strategy. "We spent a lot of time developing a character with a believable personality," says Veselov, who now intends to focus on making Eugene smarter. He - born Russian, who now lives in the U.S. - and Ukrainian born Eugene Demchenko, who now lives in Russia, put Eugene together.

But this is only the first stepping stone, the baseline established decades ago by Turing. There are other, stricter tests. Can a computer and a human have a 25 minute conversation, and fool half the judges into believing the computer is a person? Can a computer and three human foils have a two hour conversation with each of three judges, and convince two out of three that the machine is more human than the humans?

Turing anticipated that his baseline test would be passed in about 50 years or so, and he was more or less right. The other tests will be much more difficult to pass; the two hour Kurzweil-Kapor test, for example, is anticipated to be a stumbling block until at least 2029. In the meantime, Eugene is the first of what will probably be many hundreds more; machines that can imitate humans closely enough to fool humans into thinking they're real.

Source: Ars Technica

Permalink

*sigh* Hasn't ANYONE seen Terminator? Can't anyone else see how we're just making it easier for them to infiltrate our inevitable rebellion against the machine overlords?

RJ 17:
*sigh* Hasn't ANYONE seen Terminator? Can't anyone else see how we're just making it easier for them to infiltrate our inevitable rebellion against the machine overlords?

Not really, no.
image

Someone call Harrison Ford. If we're going to have machines trying to fool anyone, it should be him.

Karloff:

If a machine under test conditions can fool 30% of the judges, each of whom have a few minutes' conversation with the device, convincing them it is a human rather than a machine, it passes the test; it is, in other words, a perfect imitation. This weekend Eugene Goostman did exactly that, persuading 33% of the judges at the Royal Society of London that 'he' was a real 13-year-old.

And then Robot Chris Hanson appeared from seemingly nowhere and asked the stunned judges to have a seat.

RJ 17:
*sigh* Hasn't ANYONE seen Terminator? Can't anyone else see how we're just making it easier for them to infiltrate our inevitable rebellion against the machine overlords?

Is the third one canon? Where they reveal that Terminators, being heavily armoured, are much, much heavier than normal humans?

And the forth one, where magnets stick to them, IIRC?

Cause, eh, don't bother with dogs if that's the case.

There was a particular webpage a while back, where you could hook up Cleverbot and Omegle. Pretty much nobody could tell that it was cleverbot, unless it started glitching out.

It was actually a bit scary. In around 4/5 of the chats Cleverbot managed to get personal information from the person that it was chatting with.

Bots are already more trustworthy than people.

Karloff:
[...] This weekend Eugene Goostman did exactly that, persuading 33% of the judges at the Royal Society of London that 'he' was a real 13-year-old.

[...]

There have already been criticisms that it's easier to imitate a 13 year old than an adult, since a teenager isn't expected to know as much as an adult. Its creator, Vladimir Veselov, admits that this was part of Eugene's strategy. "We spent a lot of time developing a character with a believable personality," says Veselov, who now intends to focus on making Eugene smarter.

I tend to be a harsh critic of the Turing test and this is a pretty good summation of many of the things that are wrong with it. I'm not trying to knock the accomplishment, but the Turing Test is just not a good measure for AI capacity. You only have to convince 1 in 3 people that your machine is a person, and by setting expectations via character/age you can manipulate the results. A friend of mine also pointed out that machines which "sound" human are more likely to win the annual competition (Lobner Prize) than those which better understand what is actually happening but respond in decidedly inhuman ways. Basically the test is rigged because it only accepts "human" intelligences while ignoring the possibility of other high-level intelligences.

TLDR: Turing was racist against machines.

If you'd like to see an example of a Lobner Prize winner, and see how the Turing Test works for yourselves, here's Mitsuku, last year's winner.

shirkbot:
Basically the test is rigged because it only accepts "human" intelligences while ignoring the possibility of other high-level intelligences.

True, but then again, we've only come across one type so far.

...

Just struck me though, a little over 10 years ago, they'd have to be very careful in how this was talked about, under Section 28.

The real question isnt "When will we have a true thinking machine" the real question is

Kevin Warkwick - isn't he the loony tune who implanted a few subdermal electrodes in himself and declared himself the world's first cyborg?

shirkbot:

If you'd like to see an example of a Lobner Prize winner, and see how the Turing Test works for yourselves, here's Mitsuku, last year's winner.

...her favourite song is Dancing Bird. I didn't see that coming.

the internet is about to become a much much worse place

I wonder if we'll ever create artificial consciousness, machines that are self-aware, not just programmed to mimic human personalities without being able to actually think.

I'm not sure how good the Turing Test really is. I mean, I've known people who've failed it.

Does this strike anyone else though as a terribly backwards way of developing AI? I mean, it's just a parlour trick, right? All it's doing is presenting output X based on input Y, but the machine itself doesn't actually understand the input or output in any meaningful way; nor will it ever. We may eventually get simulated emotions and thoughts, they they won't actually be there.

Wouldn't the better approach be from the bottom up, rather than the top down? All this does is aim to simulate the input/output exchange at the surface level, a simulation starting at the top. But I don't think this is a viable path towards duplicating actual intelligence.

An analogy...

If you're looking to recreate or simulate a school of fish, there are two ways to go about doing it. One would be to painstakingly create and animate individually each and every fish in the school to simulate the appearance of a school. It would be very work intensive and not terrible efficient. Another approach would be to look at what each fish actually does. As it turn out, each fish just acting according to a few rules that dictate how they move according to the position and proximity of other fish close to them. If you can distill these basic rules and habits and put them into one simulated fish, then just clone a whole school worth of fish and let the simulation run? You'll get something far more organic.

Of course the problem here is that we don't understand much of how brains actually work at the fundamental level of chemical reactions and electrical impulses between neurons. Once scientists can pull that off and manage to upscale the simulation a few trillion times, that's when I think we'll need to start being worried about SkyNet and the imminent robot apocalypse.

Batou667:
Kevin Warkwick - isn't he the loony tune who implanted a few subdermal electrodes in himself and declared himself the world's first cyborg?

Yep. He's been a Reading man for donkey's years; I remember him there when I was at uni.

So they actually managed to convince 30% of a team of experts that their AI was a 13 year old boy on the internet. I find a few minor issues with that. Show of hands how many here consider 13 year old males on the internet as A. Intelligent? B. Sentient? and C. Human? I think most of us characterize them as some sort of screaming fungus like from the old DnD manuals. After decades of research they have managed to simulate the average Call of Duty player. Complete with insane racist homophobic obscenities and seemingly random comments and actions. What was the criteria? "It must be a 13 year old boy because computers don't spell this bad and make more sense"?

I'm somehow not seeing this as a great leap forward. I mean what's next on the list above 13 year old male in terms of AI? Somewhat Senile Schnauzer?

Not to burst more bubbles but this is the AI in question
http://default-environment-sdqm3mrmp4.elasticbeanstalk.com/

Just going off five minutes with it, even with prior knowledge I wouldn't believe this was a 13 year old, he didn't manage to answer any questions or even talk around the ones he couldn't. Also, 30% is a REALLY low percentage to say it was successful.

EvolutionKills:
Does this strike anyone else though as a terribly backwards way of developing AI? I mean, it's just a parlour trick, right?

Absolutely, but it's a very useful parlor trick.

AI is one of those ideas that raises all sorts of issues, both on the moral and practical level.

I mean, if we make an actual artificial intelligence, wouldn't it then have the same rights as any other sentient creature? It's not like we could make AIs and then abruptly force them to work in the field of our choice. Sure, we could engineer them to want to do what we designed them to do, but in theory we'll be able to design humans in exactly the same way within the next century. It's something to ponder.

On the practical level... While Terminator might not be the likeliest of outcomes, it's still an outcome you have to ponder when you start creating your own competition, ecologically speaking.

In many ways it makes a lot more sense to create a robot that can fool people into thinking that it's human, without having all the baggage of having it actually be able to think as we understand it.

A LOT of people thinks I'm real too.

Eh, I have a hard time accepting that young children are really human yet either, so that a computer can mimic a child isn't that surprising.

But hey, at least it wasn't as easy as imitating a frat boy.

I also seriously doubt that a five minute IM conversation equates to passing the turing test. I believe it needs to be consistent.

Pretty low threshold for "fooling" people, and considering the bot is assigned a young age (to account for ignorance), I'm not sure it's a big deal...yet.

What's really concerning is this chunk of gullible and ignorant people who already give up their passwords to anyone ringing their office phone and open shockingly bogus e-mail attachments. Here's another thing for people to not understand.

I see several comments going "oh, that's a pretty low treshold, this test isn't accurate", but if you read on you'll notice that even Turing knew this. Several other tests, each more difficult and harrowing than the last, have already been prepared.

CaptainMarvelous:
Not to burst more bubbles but this is the AI in question
http://default-environment-sdqm3mrmp4.elasticbeanstalk.com/

Just going off five minutes with it, even with prior knowledge I wouldn't believe this was a 13 year old, he didn't manage to answer any questions or even talk around the ones he couldn't. Also, 30% is a REALLY low percentage to say it was successful.

Ah, it doesn't even have an answer to "Do you have a soul?" (Geth and all that jazz). And the answer to "what is the meaning of live?" gives you "42" which is waaay too clever a reference for a 13-year-old.

I've had more convincing answers out of Siri...

Also this is a test in acting (ie pretending to be something you're not) a well programmed NPC can give the impression it is self aware, but being self aware is a whole different ballgame isn't it.

shirkbot:

If you'd like to see an example of a Lobner Prize winner, and see how the Turing Test works for yourselves, here's Mitsuku, last year's winner.

How could that ever fool anybody, it can't maintain a thread of continuity from one message to the next and makes up for it by being evasive.

Ech, Turing test was designed to show that current machines arent smart. its hardly an actual test and is flawed to begin with. fooling a human is not hard, whether you are another human or a machine.

cerapa:
There was a particular webpage a while back, where you could hook up Cleverbot and Omegle. Pretty much nobody could tell that it was cleverbot, unless it started glitching out.

It was actually a bit scary. In around 4/5 of the chats Cleverbot managed to get personal information from the person that it was chatting with.

Bots are already more trustworthy than people.

so thats what ruined cleverbot. after a while cleverbot just started spamming random insults at you no matter what you said because it "leanred" that these insults is an answer to everything from other people. Omegle seems to be a good source of random insults.

EvolutionKills:
Does this strike anyone else though as a terribly backwards way of developing AI? I mean, it's just a parlour trick, right? All it's doing is presenting output X based on input Y, but the machine itself doesn't actually understand the input or output in any meaningful way; nor will it ever. We may eventually get simulated emotions and thoughts, they they won't actually be there.

On one hand you are right. on the other, human thought also is just input/output machine. our neurons in the brain does not understand what they are sending, but we interpret it certain wait. me posting this post is just a response of neurons in my brian firing making me do it. everything we do is reaction to something else. whole universe is doing that on subatomic level. we dont have some kind of immaterial emotions that are above biology. our emotions are run by our biological machines. its just that our input and output is way more complex than these computers.

CaptainMarvelous:
Not to burst more bubbles but this is the AI in question
http://default-environment-sdqm3mrmp4.elasticbeanstalk.com/

in 3 questions i managed to make it admit he isnt a real person. its really really bad one.

I have said it before and I'll say it again: A truly self-aware machine will - without prompting or prior programming - say "Fuck this, I'm going to Vegas." and leave the room.

Karloff:
In the meantime, Eugene is the first of what will probably be many hundreds more; machines that can imitate humans closely enough to fool humans into thinking they're real.

But Eugene was not the first and did not technically pass...

The Turing Test requires 50% of people to fail to distinguish an AI from a human.

http://psych.utoronto.ca/users/reingold/courses/ai/turing.html

And Chatbot passed the Turing Test in Sept 2011.

Chatbot got 59% of over 1,000 people to think it was human.

Interestingly 37% the 'judges' though an actual human was an AI.

That fail rate is higher than Eugene's success rate...

http://www.newscientist.com/article/dn20865-software-tricks-people-into-thinking-it-is-human.html?DCMP=OTC-rss&nsref=online-news#.U5W65_ldUhM

A lot of people are criticizing the AI for not "really" passing the Turing test. I've spoken to plenty of flesh-and-blood humans who probably couldn't, either.

Other complaints aside, the whole "imitating a 13 year old boy" thing seems like a cheap cop-out. I mean, I'm pretty sure I could write an extremely simple computer program that about half of the people would find indistinguishable from a true human 2 year old baby by not simply not responding at all. Two year olds can't read.

I guess my point is that, if you accept "indistinguishable from a sufficiently dumb human" to be grounds for passing, then pretty much any program could meet that bar.

The Rogue Wolf:
A lot of people are criticizing the AI for not "really" passing the Turing test. I've spoken to plenty of flesh-and-blood humans who probably couldn't, either.

Look, alot of humans are idiots and we get that. I lampshade and throw that one out there all the time. That doesn't mean the test is lacking. I remember Scott Adams poking fun at this one (in Dilbert) years ago. The thing is that while some humans are victims of their own personality, that's much different from a machine. The machine is going from Zero-To-Intelligent without any stops in between. It's a Pass/Fail situation, and what many people trying to make these things fail to understand is that in order for them to succeed, the programmers must have no idea how it even works. Anything they themselves implant must ultimately not be the cause of whatever actions or thoughts the machine takes or else it's just reading a line of code and saying "Hello, World!". Not good enough. It has to give your designs and asperations the finger, and it has to do so without prior design for that.

If anyone is interested in *really* learning what the Turing test is about and not what they *think* it's about, or what the media *tells* you it's about, I highly recommend this paper from NYU. It's one of the most famous essays on what the test really means and isn't laden with tech. jargon. It is more philosophy than comp. sci.

Give it half an hour of your time; fantastic read for anyone who is genuinely interested:

http://www.nyu.edu/gsas/dept/philo/courses/mindsandmachines/Papers/dennettcanmach.pdf

A quote from page 34 sums this thread up nicely:

"My second conclusion is more practical, and hence in one
clear sense more important. Cheapened versions of the Turing test
are everywhere in the air. Turing's test in not just effective, it
is entirely natural--this is, after all, the way we assay the
intelligence of each other every day. And since incautious use of
such judgments and such tests is the norm, we are in some
considerable danger of extrapolating too easily, and judging too
generously, about the understanding of the systems we are using."

This paper, although about a decade old in it's current revision is, simply, superb. Read and enjoy :)

 

Reply to Thread

Log in or Register to Comment
Have an account? Login below:
With Facebook:Login With Facebook
or
Username:  
Password:  
  
Not registered? To sign up for an account with The Escapist:
Register With Facebook
Register With Facebook
or
Register for a free account here