AI with IQ of four-year-old

LittlePinto · Oct 7, 2015

Researchers from the University of Illinois developed a simple IQ test for a computer built by MIT. The machine scored about the equivalent of a four-year-old.

Here's my favorite part:

A lot depended on how the machine interpreted the questions. For instance, in answer to the question 'Why do we shake hands?", the machine produced the result "epileptic fit".

But when the team reduced the question to just "shake hand", the computer came back with more relevant answers such as "flirt", "thanks" and "meet friend".

Sometimes its answers appeared completely illogical - in response to the question "Where can you find a teacher?", it came back with "piano" and "band".

The researchers could not explain such anomalies but did suggest ways that the test could be improved - for instance, inputting questions using natural language via a virtual assistant such as Siri or Cortana.

I just love that it's giving them unexpected answers. (I don't know what it says about me that I think those answers make perfect sense.) I also think that the odd answers are pretty in line with some young children I've spoken to. At first glance, what they say makes no sense at all and then you think about it a bit and the association is perfectly clear.

Just wait until it reaches its teenage years. God help us when the machines learn about sarcasm.

Vince524 · Oct 7, 2015

Big deal. They tested me and I hit the IQ of a 4 year old. Sometimes. On a good day.

BarII · Oct 7, 2015

LittlePinto said:
But when the team reduced the question to just "shake hand"

Not a question. Wrong answers given with the right questions should be considered wrong.

...the computer came back with more relevant answers such as "flirt"

"Epileptic fit" was closer.

Hapax Legomenon · Oct 7, 2015

A lot depended on how the machine interpreted the questions. For instance, in answer to the question 'Why do we shake hands?", the machine produced the result "epileptic fit".

Well, it's not wrong...

Rufus Coppertop · Oct 7, 2015

I once scored 35 on an IQ test.

LittlePinto · Oct 7, 2015

BarII said:
Not a question. Wrong answers given with the right questions should be considered wrong.

"Epileptic fit" was closer.

What, you don't flirt through a handshake?

BarII · Oct 7, 2015

The closest I got was demonstrating an alleged "gay handshake" that I learned in grade school. Grip loosly and tickle the guy's palm with your middle finger.

VeryBigBeard · Oct 7, 2015

blacbird said:
Hell, on a good day I can beat that.

caw

Maybe someone who knows more about robotics can correct me, but I've always been sceptical of IQ in general. Quantifying human intelligence on a single-dimensional scale just seems limited.

Also, this, courtesy of a Guardian editorial destroying Boris Johnson and IQ tests in one fell swoop:

As a matter of convention, average IQ has been defined as 100, with the distribution calibrated – again, purely by convention – to a standard deviation of 15. Seeing as IQ tests have evolved to secure the same bell-shaped ("normal") curve found in physical natural phenomenon, it drops out as a matter of logic that roughly 16% of people will indeed be assigned an IQ below 85, and about 2% a score of 130+. These statements convey no information about anything except the way that IQ is defined. Any idea that they say anything about "our species" is, well, specious. An intelligent man (which Mr Johnson undoubtedly is, whatever his IQ) ought not to claim they are "relevant" to debates about pay.

Diana Hignutt · Oct 8, 2015

The coming AI doom...you can smell it in the air...

Sage · Oct 8, 2015

At least it skipped its terrible twos

Don · Oct 8, 2015

Diana Hignutt said:
The coming AI doom...you can smell it in the air...

Computers are supremely logical. The only ones who need fear AI are those who teach fairy tales as fact. AI would quickly dismantle the facade erected around terms like "Original Sin," "Intelligent Design," "Social Contract," and "taxation" that con some people into genuflecting before other people. Bad news for preachers and politicians, good news for sanity and society.

LittlePinto · Oct 8, 2015

VeryBigBeard said:
Maybe someone who knows more about robotics can correct me, but I've always been sceptical of IQ in general. Quantifying human intelligence on a single-dimensional scale just seems limited.

Also, this, courtesy of a Guardian editorial destroying Boris Johnson and IQ tests in one fell swoop:

I don't know much about robotics, but from a psychology standpoint IQ tests are questionable. It's unfortunate that we place so much emphasis on them.

And don't get me started on personality inventories.

Kaiser-Kun · Oct 8, 2015

It won't be long until these evil machines force us out of our planets and we have to live in closed environmental suits because the artificial atmosphere has rendered our immune systems useless and we'll be drawn into a long war with our former creations, thus unprepared for the imminent Reaper invasion, at least until a VERY paragon Commander Shepard comes along.

Diana Hignutt · Oct 8, 2015

Don said:
Computers are supremely logical. The only ones who need fear AI are those who teach fairy tales as fact. AI would quickly dismantle the facade erected around terms like "Original Sin," "Intelligent Design," "Social Contract," and "taxation" that con some people into genuflecting before other people. Bad news for preachers and politicians, good news for sanity and society.

"Yeah, it starts with oohs and ahs, and then comes the running and the screaming." - Ian Malcolm.

Having read Bostrom and Barratt on this topic, I'm a lot less optimistic than you are.

Albedo · Oct 8, 2015

Kaiser-Kun said:
It won't be long until these evil machines force us out of our planets and we have to live in closed environmental suits because the artificial atmosphere has rendered our immune systems useless and we'll be drawn into a long war with our former creations, thus unprepared for the imminent Reaper invasion, at least until a VERY paragon Commander Shepard comes along.

Alas Shepard will side with the AIs (because we're dicks), feeling a vague sense of regret at our extinction only because his girlfriend was one of us. Awkward.

asroc · Oct 8, 2015

Rufus Coppertop said:
I once scored 35 on an IQ test.

Nerd.

JimmyB27 · Oct 8, 2015

Sometimes its answers appeared completely illogical - in response to the question "Where can you find a teacher?", it came back with "piano" and "band".
The researchers could not explain such anomalies but did suggest ways that the test could be improved - for instance, inputting questions using natural language via a virtual assistant such as Siri or Cortana.

I can explain them. It's got the IQ of a four year old. Four year olds are idiots.

Liosse de Velishaf · Oct 12, 2015

Actually, the answers make quite a lot of sense if you know the current methods of language processing, machine learning, and AI. I highly doubt the machine actually has an IQ equivalent to a four-year-old. Rather, it happens that the methods used by the software resulted in a similar score. But the methods used by the AI have little to do with the reasoning behind the answers given by an average four-year-old.

You'll notice the article said very little about the software involved, essentially treating it as a black box. It's not. There are a limited number of common and well-tested algorithms for the various components of natural language processing and knowledge-storage-and-retrieval.

The linked article from MITTR suggests the AI has no natural language processing capability in the tested form, which strikes me as a bit odd given the test involved and what we know of human intelligence. So I'm curious how they store their concept map and how it was created. Anyway, the point is, there's not much of substance in this article.

Silva · Oct 12, 2015

Those "wrong" answers make logical sense to me as well-- it seems like perhaps there's more of a lack of significant socialization, or anyway, the sort of socialization that informs the knee-jerk logical connections humans make.

This also makes me curious what the typical IQ of a four year old is. I have one of those, and I'm pretty sure the only reason she's not smarter than me already is because I know how to read and she's still learning. And, heaven help me, she knows what sarcasm is.

BenPanced · Oct 12, 2015

I wonder if this is the future of the cootie shot.

Liosse de Velishaf · Oct 12, 2015

Silva said:
Those "wrong" answers make logical sense to me as well-- it seems like perhaps there's more of a lack of significant socialization, or anyway, the sort of socialization that informs the knee-jerk logical connections humans make.

My suspicion is that they are using a statistical search method of a bunch of text, and massaging whatever the most common near-by words are. Naturally it's more complicated than that as far as the inner workings of the algorithms, but that's the gist of it.

The machine doesn't have natural language processing software, so it doesn't understand the syntactic relationship between "we" and "shake hands". It sees "why/shake/hands" and does a brute force search on them as three separate keywords. "Why are my hands shaking" no doubt would come up more frequently than "why do we shake hands". A quick google search confirms this. Take out the "why", and "shake hands" is much more likely to come up with stuff about handshakes.

For the "teacher" issue coming up with "piano" and "band". "Where/find/teacher". Anyone wanna take a bet that "where do I find a piano teacher" is a much more common question on a search engine than "where are teachers located"?

I might even go so far as to speculate their "conceptnet" is a collection of nodes with weighted edges(connections) between various concepts. So it performed quite well on "pens and pencils are both" by using a simple logic process and a word relationship chart. Both those words will connect with "writing implement" say. "Where can you find a penguin" is also a fairly simple logical relationship--"location". "Why" is a much harder relationship to reduce to logic variables.

Silva · Oct 12, 2015

Liosse de Velishaf said:
My suspicion is that they are using a statistical search method of a bunch of text, and massaging whatever the most common near-by words are. Naturally it's more complicated than that as far as the inner workings of the algorithms, but that's the gist of it.

The hands example seems even simpler. The machine doesn't have natural language processing software, so it doesn't understand the syntactic relationship between "we" and "shake hands". It sees "why/shake/hands" and does a brute force search on them as three separate keywords. "Why are my hands shaking" no doubt would come up more frequently than "why do we shake hands". A quick google search confirms this. Take out the "why", and "shake hands" is much more likely to come up with stuff about handshakes.

For the "teacher" issue coming up with "piano" and "band". "Where/find/teacher". Anyone wanna take a bet that "where do I find a piano teacher" is a much more common question on a search engine than "where are teachers located"?

I might even go so far as to speculate their "conceptnet" is a collection of nodes with weighted edges(connections) between various concepts. So it performed quite well on "pens and pencils are both" by using a simple logic process and a word relationship chart. Both those words will connect with "writing implement" say. "Where can you find a penguin" is also a fairly simple logical relationship--"location". "Why" is a much harder relationship to reduce to logic variables.

That makes a lot of sense.

Fingers · Oct 13, 2015

Quick, get that robot on the list of those running for president. At a four year olds IQ its at least as smart as half a dozen of the republican candidates.

Liosse de Velishaf · Oct 13, 2015

Fingers said:
Quick, get that robot on the list of those running for president. At a four year olds IQ its at least as smart as half a dozen of the republican candidates.

I bet it's response algorithm will construct better policy, too!

AI with IQ of four-year-old

Perpetually confused

Are you gonna finish that bacon?

Perpetually confused

Preparing for winter

Very Tired

Supreme Guessinator

All Living is Local

Perpetually confused

!

Very Tired

Alex

Alex

Hoopy frood

saucy

THE BLUEBERRY QUEEN OF HADES (he/him)

saucy

My cat Toby