Jay McClelland: Neural Networks and the Emergence of Cognition

Lex Fridman (01:36.140)

And the author said that the study of the nervous system

Jay McClelland (01:42.060)

was only of peripheral interest.

Lex Fridman (01:44.540)

It wasn't going to tell us anything about the mind.

Lex Fridman (01:48.420)

And I didn't agree with that.

Lex Fridman (01:51.980)

I always felt, oh, look, I'm a physical being.

Jay McClelland (01:58.840)

From dust to dust, you know,

Lex Fridman (02:01.300)

ashes to ashes, and somehow I emerged from that.

Lex Fridman (02:06.580)

So that's really interesting.

Lex Fridman (02:08.000)

So there was a sense with cognitive psychology

Jay McClelland (02:11.700)

that in understanding the neuronal structure of things,

Lex Fridman (02:17.220)

you're not going to be able to understand the mind.

Lex Fridman (02:20.020)

And then your sense is if we study these neural networks,

Lex Fridman (02:23.700)

we might be able to get at least very close

Jay McClelland (02:25.860)

to understanding the fundamentals of the human mind.

Lex Fridman (02:28.260)

Yeah.

Jay McClelland (02:29.300)

I used to think, or I used to talk about the idea

Lex Fridman (02:32.580)

of awakening from the Cartesian dream.

Lex Fridman (02:36.620)

So Descartes, you know, thought about these things, right?

Lex Fridman (02:41.620)

He was walking in the gardens of Versailles one day,

Lex Fridman (02:46.260)

and he stepped on a stone.

Lex Fridman (02:48.020)

And a statue moved.

Lex Fridman (02:52.180)

And he walked a little further,

Lex Fridman (02:53.540)

he stepped on another stone, and another statue moved.

Lex Fridman (02:55.900)

And he, like, why did the statue move

Lex Fridman (02:59.300)

when I stepped on the stone?

Lex Fridman (03:00.540)

And he went and talked to the gardeners,

Lex Fridman (03:02.900)

and he found out that they had a hydraulic system

Jay McClelland (03:06.780)

that allowed the physical contact with the stone

Lex Fridman (03:10.660)

to cause water to flow in various directions,

Jay McClelland (03:12.780)

which caused water to flow into the statue

Lex Fridman (03:14.780)

and move the statue.

Lex Fridman (03:15.880)

And he used this as the beginnings of a theory

Lex Fridman (03:22.840)

about how animals act.

Lex Fridman (03:28.260)

And he had this notion that these little fibers

Lex Fridman (03:33.320)

that people had identified that weren't carrying the blood,

Jay McClelland (03:37.400)

you know, were these little hydraulic tubes

Lex Fridman (03:39.880)

that if you touch something, there would be pressure,

Lex Fridman (03:42.160)

and it would send a signal of pressure

Lex Fridman (03:43.700)

to the other parts of the system,

Lex Fridman (03:46.240)

and that would cause action.

Lex Fridman (03:49.200)

So he had a mechanistic theory of animal behavior.

Lex Fridman (03:54.260)

And he thought that the human had this animal body,

Lex Fridman (04:00.080)

but that some divine something else

Jay McClelland (04:03.740)

had to have come down and been placed in him

Lex Fridman (04:06.960)

to give him the ability to think, right?

Lex Fridman (04:10.560)

So the physical world includes the body in action,

Lex Fridman (04:15.680)

but it doesn't include thought according to Descartes, right?

Lex Fridman (04:19.480)

And so the study of physiology at that time

Lex Fridman (04:22.920)

was the study of sensory systems and motor systems

Lex Fridman (04:26.400)

and things that you could directly measure

Lex Fridman (04:30.080)

when you stimulated neurons and stuff like that.

Lex Fridman (04:33.640)

And the study of cognition was something that, you know,

Lex Fridman (04:38.160)

was tied in with abstract computer algorithms

Lex Fridman (04:41.160)

and things like that.

Lex Fridman (04:43.320)

But when I was an undergraduate,

Jay McClelland (04:45.080)

I learned about the physiological mechanisms.

Lex Fridman (04:48.720)

And so when I'm studying cognitive psychology

Jay McClelland (04:51.240)

as a first year PhD student, I'm saying,

Lex Fridman (04:53.760)

wait a minute, the whole thing is biological, right?

Lex Fridman (04:56.800)

You know?

Lex Fridman (04:57.980)

You had that intuition right away.

Jay McClelland (04:59.600)

That always seemed obvious to you.

Lex Fridman (05:00.880)

Yeah, yeah.

Jay McClelland (05:03.000)

Isn't that magical, though,

Lex Fridman (05:04.440)

that from just a little bit of biology can emerge

Lex Fridman (05:08.240)

the full beauty of the human experience?

Lex Fridman (05:10.840)

Why is that so obvious to you?

Jay McClelland (05:13.200)

Well, obvious and not obvious at the same time.

Lex Fridman (05:18.160)

And I think about Darwin in this context, too,

Jay McClelland (05:20.400)

because Darwin knew very early on

Lex Fridman (05:25.040)

that none of the ideas that anybody had ever offered

Jay McClelland (05:29.260)

gave him a sense of understanding

Lex Fridman (05:31.240)

how evolution could have worked.

Lex Fridman (05:36.440)

But he wanted to figure out how it could have worked.

Lex Fridman (05:40.520)

That was his goal.

Lex Fridman (05:42.560)

And he spent a lot of time working on this idea

Lex Fridman (05:48.440)

and reading about things that gave him hints

Lex Fridman (05:52.320)

and thinking they were interesting but not knowing why

Lex Fridman (05:54.640)

and drawing more and more pictures of different birds

Jay McClelland (05:57.520)

that differ slightly from each other and so on, you know.

Lex Fridman (06:00.400)

And then he figured it out.

Lex Fridman (06:03.400)

But after he figured it out, he had nightmares about it.

Lex Fridman (06:06.960)

He would dream about the complexity of the eye

Lex Fridman (06:10.000)

and the arguments that people had given

Lex Fridman (06:12.720)

about how ridiculous it was to imagine

Jay McClelland (06:16.200)

that that could have ever emerged

Lex Fridman (06:19.120)

from some sort of, you know, unguided process, right?

Jay McClelland (06:24.700)

That it hadn't been the product of design.

Lex Fridman (06:28.400)

And so he didn't publish for a long time,

Jay McClelland (06:32.000)

in part because he was scared of his own ideas.

Lex Fridman (06:35.440)

He didn't think they could possibly be true.

Lex Fridman (06:40.960)

But then, you know, by the time

Lex Fridman (06:44.640)

the 20th century rolls around, we all,

Jay McClelland (06:49.480)

you know, we understand that,

Lex Fridman (06:52.640)

many people understand or believe

Jay McClelland (06:55.560)

that evolution produced, you know, the entire

Lex Fridman (06:59.720)

range of animals that there are.

Jay McClelland (07:03.520)

And, you know, Descartes's idea starts to seem

Lex Fridman (07:06.400)

a little wonky after a while, right?

Jay McClelland (07:08.240)

Like, well, wait a minute.

Lex Fridman (07:11.200)

There's the apes and the chimpanzees and the bonobos

Jay McClelland (07:15.380)

and, you know, like, they're pretty smart in some ways.

Lex Fridman (07:18.360)

You know, so what?

Jay McClelland (07:20.560)

Oh, you know, somebody comes up,

Lex Fridman (07:22.040)

oh, there's a certain part of the brain

Jay McClelland (07:23.680)

that's still different.

Lex Fridman (07:24.520)

They don't, you know, there's no hippocampus

Jay McClelland (07:26.680)

in the monkey brain.

Lex Fridman (07:28.720)

It's only in the human brain.

Jay McClelland (07:31.160)

Huxley had to do a surgery in front of many, many people

Lex Fridman (07:34.240)

in the late 19th century to show to them

Jay McClelland (07:36.240)

there's actually a hippocampus in the chimpanzee's brain.

Lex Fridman (07:40.320)

You know, so the continuity of the species

Jay McClelland (07:45.800)

is another element that, you know,

Lex Fridman (07:49.640)

contributes to this sort of, you know, idea

Jay McClelland (07:56.240)

that we are ourselves a total product of nature.

Lex Fridman (08:01.920)

And that, to me, is the magic and the mystery,

Lex Fridman (08:06.960)

how nature could actually, you know,

Lex Fridman (08:11.880)

give rise to organisms that have the capabilities

Jay McClelland (08:16.880)

that we have.

Lex Fridman (08:20.140)

So it's interesting because even the idea of evolution

Jay McClelland (08:23.020)

is hard for me to keep all together in my mind.

Lex Fridman (08:27.100)

So because we think of a human time scale,

Jay McClelland (08:30.180)

it's hard to imagine, like, the development

Lex Fridman (08:33.620)

of the human eye would give me nightmares too.

Jay McClelland (08:36.220)

Because you have to think across many, many, many

Lex Fridman (08:38.500)

generations, and it's very tempting to think about

Jay McClelland (08:41.860)

kind of a growth of a complicated object

Lex Fridman (08:44.720)

and it's like, how is it possible for such a thing

Lex Fridman (08:49.300)

to be built?

Lex Fridman (08:50.140)

Because also, me, from a robotics engineering perspective,

Jay McClelland (08:53.260)

it's very hard to build these systems.

Lex Fridman (08:55.340)

How can, through an undirected process,

Lex Fridman (08:58.620)

can a complex thing be designed?

Lex Fridman (09:00.940)

It seems not, it seems wrong.

Jay McClelland (09:03.460)

Yeah, so that's absolutely right.

Lex Fridman (09:05.620)

And I, you know, a slightly different career path

Jay McClelland (09:08.700)

that would have been equally interesting to me

Lex Fridman (09:10.600)

would have been to actually study the process

Jay McClelland (09:15.900)

of embryological development flowing on

Lex Fridman (09:21.380)

into brain development and the exquisite sort of laying

Jay McClelland (09:29.300)

down of pathways and so on that occurs in the brain.

Lex Fridman (09:32.300)

And I know the slightest bit about that is not my field,

Lex Fridman (09:35.780)

but there are, you know, fascinating aspects

Lex Fridman (09:43.860)

to this process that eventually result in the, you know,

Jay McClelland (09:49.780)

the complexity of various brains.

Lex Fridman (09:54.020)

At least, you know, one thing we're,

Jay McClelland (09:59.860)

in the field, I think people have felt for a long time,

Lex Fridman (10:02.580)

in the study of vision, the continuity between humans

Lex Fridman (10:07.420)

and nonhuman animals has been second nature

Lex Fridman (10:11.020)

for a lot longer.

Jay McClelland (10:12.340)

I was having, I had this conversation with somebody

Lex Fridman (10:16.180)

who is a vision scientist and he was saying,

Jay McClelland (10:17.940)

oh, we don't have any problem with this.

Lex Fridman (10:19.900)

You know, the monkey's visual system

Lex Fridman (10:21.500)

and the human visual system, extremely similar

Lex Fridman (10:26.300)

up to certain levels, of course, they diverge after a while.

Lex Fridman (10:29.760)

But the first, the visual pathway from the eye

Lex Fridman (10:34.860)

to the brain and the first few layers of cortex

Jay McClelland (10:41.940)

or cortical areas, I guess one would say,

Lex Fridman (10:45.340)

are extremely similar.

Jay McClelland (10:49.180)

Yeah, so on the cognition side is where the leap

Lex Fridman (10:52.340)

seems to happen with humans,

Jay McClelland (10:54.220)

that it does seem we're kind of special.

Lex Fridman (10:56.660)

And that's a really interesting question

Jay McClelland (10:58.500)

when thinking about alien life

Lex Fridman (11:00.260)

or if there's other intelligent alien civilizations

Lex Fridman (11:03.100)

out there, is how special is this leap?

Lex Fridman (11:06.000)

So one special thing seems to be the origin of life itself.

Jay McClelland (11:09.260)

However you define that, there's a gray area.

Lex Fridman (11:11.820)

And the other leap, this is very biased perspective

Jay McClelland (11:14.820)

of a human, is the origin of intelligence.

Lex Fridman (11:19.700)

And again, from an engineer perspective,

Jay McClelland (11:22.060)

it's a difficult question to ask.

Lex Fridman (11:24.420)

An important one is how difficult is that leap?

Lex Fridman (11:27.940)

How special were humans?

Lex Fridman (11:30.060)

Did a monolith come down?

Jay McClelland (11:32.380)

Did aliens bring down a monolith

Lex Fridman (11:33.740)

and some apes had to touch a monolith to get it?

Lex Fridman (11:38.100)

That's a lot like Descartes idea, right?

Lex Fridman (11:41.620)

Exactly, but it just seems one heck of a leap

Jay McClelland (11:46.620)

to get to this level of intelligence.

Lex Fridman (11:48.540)

Yeah, and so Chomsky argued that some genetic fluke occurred

Jay McClelland (12:00.660)

100,000 years ago and just happened

Lex Fridman (12:04.420)

that some human, some hominin predecessor of current humans

Jay McClelland (12:13.060)

had this one genetic tweak that resulted in language.

Lex Fridman (12:20.380)

And language then provided this special thing that separates us

Jay McClelland (12:29.580)

from all other animals.

Lex Fridman (12:36.340)

I think there's a lot of truth to the value and importance

Jay McClelland (12:39.420)

of language, but I think it comes along

Lex Fridman (12:43.420)

with the evolution of a lot of other related things related

Jay McClelland (12:48.940)

to sociality and mutual engagement with others

Lex Fridman (12:53.980)

and establishment of, I don't know,

Jay McClelland (13:01.420)

rich mechanisms for organizing and understanding

Lex Fridman (13:07.020)

of the world, which language then plugs into.

Jay McClelland (13:12.940)

Right, so language is a tool that

Lex Fridman (13:16.580)

allows you to do this kind of collective intelligence.

Lex Fridman (13:18.980)

And whatever is at the core of the thing that

Lex Fridman (13:21.660)

allows for this collective intelligence is the main thing.

Lex Fridman (13:25.300)

And it's interesting to think about that one fluke, one

Lex Fridman (13:29.460)

mutation could lead to the first crack opening of the door

Jay McClelland (13:36.220)

to human intelligence.

Lex Fridman (13:38.100)

All it takes is one.

Jay McClelland (13:39.420)

Evolution just kind of opens the door a little bit,

Lex Fridman (13:41.540)

and then time and selection takes care of the rest.

Jay McClelland (13:45.860)

You know, there's so many fascinating aspects

Lex Fridman (13:48.180)

to these kinds of things.

Lex Fridman (13:49.180)

So we think of evolution as continuous, right?

Lex Fridman (13:54.180)

We think, oh, yes, OK, over 500 million years,

Jay McClelland (13:58.700)

there could have been this relatively continuous changes.

Lex Fridman (14:04.860)

And but that's not what anthropologists,

Jay McClelland (14:12.420)

evolutionary biologists found from the fossil record.

Lex Fridman (14:15.620)

They found hundreds of millions of years of stasis.

Lex Fridman (14:24.380)

And then suddenly a change occurs.

Lex Fridman (14:27.060)

Well, suddenly on that scale is a million years or something,

Jay McClelland (14:32.420)

or even 10 million years.

Lex Fridman (14:33.940)

But the concept of punctuated equilibrium

Jay McClelland (14:38.860)

was a very important concept in evolutionary biology.

Lex Fridman (14:44.140)

And that also feels somehow right about the stages

Jay McClelland (14:53.860)

of our mental abilities.

Lex Fridman (14:55.220)

We seem to have a certain kind of mindset at a certain age.

Lex Fridman (14:59.220)

And then at another age, we look at that four year old

Lex Fridman (15:04.260)

and say, oh, my god, how could they have thought that way?

Lex Fridman (15:07.180)

So Piaget was known for this kind of stage theory

Lex Fridman (15:10.140)

of child development, right?

Lex Fridman (15:11.580)

And you look at it closely, and suddenly those stages

Lex Fridman (15:14.780)

are so discreet and it transitions.

Lex Fridman (15:17.140)

But the difference between the four year old and the seven

Lex Fridman (15:19.380)

year old is profound.

Lex Fridman (15:20.820)

And that's another thing that's always interested me

Lex Fridman (15:24.300)

is how something happens over the course of several years

Jay McClelland (15:29.340)

of experience where at some point

Lex Fridman (15:31.140)

we reach the point where something

Jay McClelland (15:33.940)

like an insight or a transition or a new stage of development

Lex Fridman (15:37.620)

occurs.

Lex Fridman (15:38.180)

And these kinds of things can be understood

Lex Fridman (15:45.180)

in complex systems research.

Lex Fridman (15:47.620)

And so evolutionary biology, developmental biology,

Lex Fridman (15:55.860)

cognitive development are all things

Jay McClelland (15:57.820)

that have been approached in this kind of way.

Lex Fridman (15:59.980)

Yeah.

Jay McClelland (16:01.140)

Just like you said, I find both fascinating

Lex Fridman (16:03.940)

those early years of human life, but also

Jay McClelland (16:07.180)

the early minutes, days from the embryonic development

Lex Fridman (16:13.140)

to how from embryos you get the brain.

Jay McClelland (16:17.460)

That development, again, from an engineer perspective,

Lex Fridman (16:20.900)

is fascinating.

Lex Fridman (16:22.020)

So it's not.

Lex Fridman (16:22.740)

So the early, when you deploy the brain to the human world

Lex Fridman (16:27.420)

and it gets to explore that world and learn,

Lex Fridman (16:29.340)

that's fascinating.

Lex Fridman (16:30.460)

But just like the assembly of the mechanism

Lex Fridman (16:33.340)

that is capable of learning, that's amazing.

Jay McClelland (16:36.700)

The stuff they're doing with brain organoids

Lex Fridman (16:39.660)

where you can build many brains and study

Jay McClelland (16:42.660)

that self assembly of a mechanism from the DNA material,

Lex Fridman (16:48.300)

that's like, what the heck?

Jay McClelland (16:51.780)

You have literally biological programs

Lex Fridman (16:55.300)

that just generate a system, this mushy thing that's

Jay McClelland (17:00.580)

able to be robust and learn in a very unpredictable world

Lex Fridman (17:05.660)

and learn seemingly arbitrary things,

Jay McClelland (17:08.340)

or a very large number of things that enable survival.

Lex Fridman (17:14.100)

Yeah.

Jay McClelland (17:15.060)

Ultimately, that is a very important part

Lex Fridman (17:19.980)

of the whole process of understanding

Jay McClelland (17:22.380)

this emergence of mind from brain kind of thing.

Lex Fridman (17:27.780)

And the whole thing seems to be pretty continuous.

Lex Fridman (17:29.900)

So let me step back to neural networks

Lex Fridman (17:32.620)

for another brief minute.

Jay McClelland (17:35.220)

You wrote parallel distributed processing books

Lex Fridman (17:37.940)

that explored ideas of neural networks in the 1980s

Jay McClelland (17:42.100)

together with a few folks.

Lex Fridman (17:43.180)

But the books you wrote with David Romelhart,

Jay McClelland (17:47.220)

who is the first author on the back propagation

Lex Fridman (17:50.380)

paper with Jeff Hinton.

Lex Fridman (17:52.460)

So these are just some figures at the time

Lex Fridman (17:54.420)

that we're thinking about these big ideas.

Lex Fridman (17:57.020)

What are some memorable moments of discovery

Lex Fridman (18:00.380)

and beautiful ideas from those early days?

Jay McClelland (18:04.620)

I'm going to start sort of with my own process in the mid 70s

Lex Fridman (18:13.140)

and then into the late 70s when I met Jeff Hinton

Lex Fridman (18:18.820)

and he came to San Diego and we were all together.

Lex Fridman (18:25.380)

In my time in graduate schools, I've already described to you,

Jay McClelland (18:30.300)

I had this sort of feeling of, OK, I'm

Lex Fridman (18:33.500)

really interested in human cognition,

Lex Fridman (18:35.540)

but this disembodied sort of way of thinking about it

Lex Fridman (18:40.100)

that I'm getting from the current mode of thought about it

Jay McClelland (18:44.740)

isn't working fully for me.

Lex Fridman (18:47.060)

And when I got my assistant professorship,

Jay McClelland (18:52.260)

I went to UCSD and that was in 1974.

Lex Fridman (18:58.460)

Something amazing had just happened.

Jay McClelland (19:00.860)

Dave Romelhart had written a book together

Lex Fridman (19:03.620)

with another man named Don Norman

Lex Fridman (19:06.220)

and the book was called Explorations in Cognition.

Lex Fridman (19:09.940)

And it was a series of chapters exploring

Jay McClelland (19:14.780)

interesting questions about cognition,

Lex Fridman (19:17.780)

but in a completely sort of abstract, nonbiological kind

Jay McClelland (19:22.900)

of way.

Lex Fridman (19:23.420)

And I'm saying, gee, this is amazing.

Jay McClelland (19:25.420)

I'm coming to this community where people can get together

Lex Fridman (19:28.980)

and feel like they've collectively exploring ideas.

Lex Fridman (19:35.100)

And it was a book that had a lot of, I don't know,

Lex Fridman (19:39.820)

lightness to it.

Lex Fridman (19:40.980)

And Don Norman, who was the more senior figure

Lex Fridman (19:47.220)

to Romelhart at that time who led that project,

Jay McClelland (19:51.220)

always created this spirit of playful exploration of ideas.

Lex Fridman (19:55.820)

And so I'm like, wow, this is great.

Lex Fridman (19:58.300)

But I was also still trying to get from the neurons

Lex Fridman (1:00:03.740)

with each other properly anymore.

Jay McClelland (1:00:05.740)

They can't relate the pictures to the words either.

Lex Fridman (1:00:07.740)

They can't do word picture matching.

Lex Fridman (1:00:09.740)

But they've lost the conceptual grounding

Lex Fridman (1:00:12.740)

from either modality of input.

Lex Fridman (1:00:15.740)

And so that's why it's called semantic dementia.

Lex Fridman (1:00:19.740)

The very semantics is disintegrating.

Lex Fridman (1:00:22.740)

And we understand this in terms of our idea

Lex Fridman (1:00:27.740)

that distributed representation, a pattern of activation,

Jay McClelland (1:00:31.740)

represents the concepts, really similar ones.

Lex Fridman (1:00:33.740)

As you degrade them, they start being,

Jay McClelland (1:00:36.740)

you lose the differences.

Lex Fridman (1:00:40.740)

So the difference between the dog and the goat

Jay McClelland (1:00:42.740)

is no longer part of the pattern anymore.

Lex Fridman (1:00:44.740)

And since dog is really familiar,

Jay McClelland (1:00:47.740)

that's the thing that remains.

Lex Fridman (1:00:49.740)

And we understand that in the way the models work and learn.

Lex Fridman (1:00:52.740)

But Rumelhart underwent this condition.

Lex Fridman (1:00:57.740)

So on the one hand, it's a fascinating aspect

Jay McClelland (1:01:00.740)

of parallel distributed processing to be.

Lex Fridman (1:01:03.740)

It reveals this sort of texture of distributed representation

Jay McClelland (1:01:08.740)

in a very nice way, I've always felt.

Lex Fridman (1:01:11.740)

But at the same time, it was extremely poignant

Jay McClelland (1:01:13.740)

because this is exactly the condition

Lex Fridman (1:01:16.740)

that Rumelhart was undergoing.

Lex Fridman (1:01:18.740)

And there was a period of time when he was this man

Lex Fridman (1:01:22.740)

who had been the most focused, goal directed, competitive,

Jay McClelland (1:01:35.740)

thoughtful person who was willing to work for years

Lex Fridman (1:01:41.740)

to solve a hard problem, he starts to disappear.

Lex Fridman (1:01:48.740)

And there was a period of time when it was hard for any of us

Lex Fridman (1:01:57.740)

to really appreciate that he was sort of, in some sense,

Jay McClelland (1:02:00.740)

not fully there anymore.

Lex Fridman (1:02:04.740)

Do you know if he was able to introspect

Lex Fridman (1:02:07.740)

the solution of the understanding mind?

Lex Fridman (1:02:14.740)

I mean, this is one of the big scientists that thinks about this.

Lex Fridman (1:02:19.740)

Was he able to look at himself and understand the fading mind?

Lex Fridman (1:02:24.740)

You know, we can contrast Hawking and Rumelhart in this way.

Lex Fridman (1:02:31.740)

And I like to do that to honor Rumelhart

Lex Fridman (1:02:33.740)

because I think Rumelhart is sort of like the Hawking

Jay McClelland (1:02:36.740)

of cognitive science to me in some ways.

Lex Fridman (1:02:40.740)

Both of them suffered from a degenerative condition.

Jay McClelland (1:02:45.740)

In Hawking's case, it affected the motor system.

Lex Fridman (1:02:49.740)

In Rumelhart's case, it's affecting the semantics.

Lex Fridman (1:02:54.740)

And not just the pure object semantics,

Lex Fridman (1:03:01.740)

but maybe the self semantics as well.

Lex Fridman (1:03:04.740)

And we don't understand that.

Lex Fridman (1:03:06.740)

Concepts broadly.

Lex Fridman (1:03:08.740)

So I would say he didn't.

Lex Fridman (1:03:13.740)

And this was part of what, from the outside,

Jay McClelland (1:03:16.740)

was a profound tragedy.

Lex Fridman (1:03:18.740)

But on the other hand, at some level, he sort of did

Jay McClelland (1:03:22.740)

because there was a period of time when it finally was realized

Lex Fridman (1:03:28.740)

that he had really become profoundly impaired.

Jay McClelland (1:03:32.740)

This was clearly a biological condition.

Lex Fridman (1:03:35.740)

It wasn't just like he was distracted that day or something like that.

Lex Fridman (1:03:39.740)

So he retired from his professorship at Stanford

Lex Fridman (1:03:44.740)

and he became, he lived with his brother for a couple years

Lex Fridman (1:03:51.740)

and then he moved into a facility for people with cognitive impairments.

Lex Fridman (1:04:00.740)

One that many elderly people end up in when they have cognitive impairments.

Lex Fridman (1:04:06.740)

And I would spend time with him during that period.

Lex Fridman (1:04:12.740)

This was like in the late 90s, around 2000 even.

Lex Fridman (1:04:16.740)

And we would go bowling and he could still bowl.

Lex Fridman (1:04:25.740)

And after bowling, I took him to lunch and I said,

Lex Fridman (1:04:32.740)

where would you like to go?

Lex Fridman (1:04:34.740)

You want to go to Wendy's?

Lex Fridman (1:04:35.740)

And he said, nah.

Lex Fridman (1:04:37.740)

And I said, okay, well, where do you want to go?

Lex Fridman (1:04:38.740)

And he just pointed.

Lex Fridman (1:04:40.740)

He said, turn here.

Lex Fridman (1:04:41.740)

So he still had a certain amount of spatial cognition

Lex Fridman (1:04:44.740)

and he could get me to the restaurant.

Lex Fridman (1:04:47.740)

And then when we got to the restaurant, I said,

Lex Fridman (1:04:51.740)

what do you want to order?

Lex Fridman (1:04:53.740)

And he couldn't come up with any of the words,

Lex Fridman (1:04:56.740)

but he knew where on the menu the thing was that he wanted.

Lex Fridman (1:04:59.740)

So it's, you know, and he couldn't say what it was,

Lex Fridman (1:05:04.740)

but he knew that that's what he wanted to eat.

Lex Fridman (1:05:07.740)

And so it's like it isn't monolithic at all.

Lex Fridman (1:05:14.740)

Our cognition is, you know, first of all, graded in certain kinds of ways,

Lex Fridman (1:05:21.740)

but also multipartite and there's many elements to it and things,

Lex Fridman (1:05:27.740)

certain sort of partial competencies still exist

Jay McClelland (1:05:31.740)

in the absence of other aspects of these competencies.

Lex Fridman (1:05:36.740)

So this is what always fascinated me about what used to be called

Jay McClelland (1:05:43.740)

cognitive neuropsychology, you know,

Lex Fridman (1:05:46.740)

the effects of brain damage on cognition.

Lex Fridman (1:05:49.740)

But in particular, this gradual disintegration part.

Lex Fridman (1:05:53.740)

You know, I'm a big believer that the loss of a human being that you value

Jay McClelland (1:05:59.740)

is as powerful as, you know, first falling in love with that human being.

Lex Fridman (1:06:03.740)

I think it's all a celebration of the human being.

Lex Fridman (1:06:06.740)

So the disintegration itself too is a celebration in a way.

Lex Fridman (1:06:10.740)

Yeah, yeah.

Lex Fridman (1:06:12.740)

But just to say something more about the scientist

Lex Fridman (1:06:17.740)

and the backpropagation idea that you mentioned.

Lex Fridman (1:06:22.740)

So in 1982, Hinton had been there as a postdoc and organized that conference.

Lex Fridman (1:06:34.740)

He'd actually gone away and gotten an assistant professorship

Lex Fridman (1:06:37.740)

and then there was this opportunity to bring him back.

Lex Fridman (1:06:41.740)

So Jeff Hinton was back on a sabbatical.

Jay McClelland (1:06:45.740)

San Diego.

Lex Fridman (1:06:46.740)

And Rommelhard and I had decided we wanted to do this, you know,

Jay McClelland (1:06:52.740)

we thought it was really exciting and the papers on the interactive activation model

Lex Fridman (1:06:58.740)

that I was telling you about had just been published

Lex Fridman (1:07:00.740)

and we both sort of saw a huge potential for this work and Jeff was there.

Lex Fridman (1:07:06.740)

And so the three of us started a research group,

Jay McClelland (1:07:11.740)

which we called the PDP Research Group.

Lex Fridman (1:07:13.740)

And several other people came.

Jay McClelland (1:07:17.740)

Francis Crick, who was at the Salk Institute, heard about it from Jeff

Lex Fridman (1:07:22.740)

because Jeff was known among Brits to be brilliant

Lex Fridman (1:07:27.740)

and Francis was well connected with his British friends.

Lex Fridman (1:07:30.740)

So Francis Crick came.

Jay McClelland (1:07:32.740)

That's a heck of a group of people, wow.

Lex Fridman (1:07:34.740)

And Paul Spolensky was one of the other postdocs.

Jay McClelland (1:07:40.740)

He was still there as a postdoc.

Lex Fridman (1:07:41.740)

And a few other people.

Lex Fridman (1:07:45.740)

But anyway, Jeff talked to us about learning

Lex Fridman (1:07:56.740)

and how we should think about how, you know, learning occurs in a neural network.

Lex Fridman (1:08:06.740)

And he said, the problem with the way you guys have been approaching this

Lex Fridman (1:08:12.740)

is that you've been looking for inspiration from biology

Jay McClelland (1:08:17.740)

to tell you what the rules should be for how the synapses should change

Lex Fridman (1:08:22.740)

the strengths of their connections, how the connections should form.

Jay McClelland (1:08:27.740)

He said, that's the wrong way to go about it.

Lex Fridman (1:08:30.740)

What you should do is you should think in terms of

Lex Fridman (1:08:36.740)

how you can adjust connection weights to solve a problem.

Lex Fridman (1:08:44.740)

So you define your problem and then you figure out

Lex Fridman (1:08:49.740)

how the adjustment of the connection weights will solve the problem.

Lex Fridman (1:08:54.740)

And Rumelhart heard that and said to himself, okay,

Lex Fridman (1:09:01.740)

so I'm going to start thinking about it that way.

Lex Fridman (1:09:04.740)

I'm going to essentially imagine that I have some objective function,

Jay McClelland (1:09:11.740)

some goal of the computation.

Lex Fridman (1:09:14.740)

I want my machine to correctly classify all of these images.

Lex Fridman (1:09:19.740)

And I can score that.

Lex Fridman (1:09:21.740)

I can measure how well they're doing on each image.

Lex Fridman (1:09:24.740)

And I get some measure of error or loss, it's typically called in deep learning.

Lex Fridman (1:09:30.740)

And I'm going to figure out how to adjust the connection weights

Lex Fridman (1:09:35.740)

so as to minimize my loss or reduce the error.

Lex Fridman (1:09:41.740)

And that's called, you know, gradient descent.

Lex Fridman (1:09:47.740)

And engineers were already familiar with the concept of gradient descent.

Lex Fridman (1:09:53.740)

And in fact, there was an algorithm called the delta rule

Jay McClelland (1:09:58.740)

that had been invented by a professor in the electrical engineering department

Lex Fridman (1:10:07.740)

at Stanford, Bernie Widrow and a collaborator named Hoff.

Jay McClelland (1:10:11.740)

I never met him.

Lex Fridman (1:10:13.740)

So gradient descent in continuous neural networks

Jay McClelland (1:10:19.740)

with multiple neuron like processing units was already understood

Lex Fridman (1:10:26.740)

for a single layer of connection weights.

Jay McClelland (1:10:29.740)

We have some inputs over a set of neurons.

Lex Fridman (1:10:32.740)

We want the output to produce a certain pattern.

Jay McClelland (1:10:35.740)

We can define the difference between our target

Lex Fridman (1:10:38.740)

and what the neural network is producing.

Lex Fridman (1:10:41.740)

And we can figure out how to change the connection weights to reduce that error.

Lex Fridman (1:10:44.740)

So what Romilhar did was to generalize that

Lex Fridman (1:10:49.740)

so as to be able to change the connections from earlier layers of units

Lex Fridman (1:10:53.740)

to the ones at a hidden layer between the input and the output.

Lex Fridman (1:10:58.740)

And so he first called the algorithm the generalized delta rule

Lex Fridman (1:11:03.740)

because it's just an extension of the gradient descent idea.

Lex Fridman (1:11:08.740)

And interestingly enough, Hinton was thinking that this wasn't going to work very well.

Lex Fridman (1:11:15.740)

So Hinton had his own alternative algorithm at the time

Jay McClelland (1:11:20.740)

based on the concept of the Boltzmann machine that he was pursuing.

Lex Fridman (1:11:24.740)

So the paper on the Boltzmann machine came out in,

Jay McClelland (1:11:27.740)

learning in Boltzmann machines came out in 1985.

Lex Fridman (1:11:31.740)

But it turned out that back prop worked better than the Boltzmann machine learning algorithm.

Lex Fridman (1:11:37.740)

So this generalized delta algorithm ended up being called back propagation, as you say, back prop.

Lex Fridman (1:11:44.740)

Yeah. And probably that name is opaque to me.

Lex Fridman (1:11:50.740)

What does that mean?

Lex Fridman (1:11:53.740)

What it meant was that in order to figure out what the changes you needed to make

Jay McClelland (1:11:59.740)

to the connections from the input to the hidden layer,

Lex Fridman (1:12:03.740)

you had to back propagate the error signals from the output layer

Jay McClelland (1:12:10.740)

through the connections from the hidden layer to the output

Lex Fridman (1:12:15.740)

to get the signals that would be the error signals for the hidden layer.

Lex Fridman (1:12:20.740)

And that's how Rumelhart formulated it.

Lex Fridman (1:12:22.740)

It was like, well, we know what the error signals are at the output layer.

Jay McClelland (1:12:25.740)

Let's see if we can get a signal at the hidden layer

Lex Fridman (1:12:28.740)

that tells each hidden unit what its error signal is essentially.

Lex Fridman (1:12:32.740)

So it's back propagating through the connections

Lex Fridman (1:12:37.740)

from the hidden to the output to get the signals to tell the hidden units

Lex Fridman (1:12:41.740)

how to change their weights from the input.

Lex Fridman (1:12:43.740)

And that's why it's called back prop.

Jay McClelland (1:12:47.740)

Yeah. But so it came from Hinton having introduced the concept of, you know,

Lex Fridman (1:12:54.740)

define your objective function, figure out how to take the derivative

Lex Fridman (1:12:59.740)

so that you can adjust the connections so that they make progress towards your goal.

Lex Fridman (1:13:04.740)

So stop thinking about biology for a second

Lex Fridman (1:13:06.740)

and let's start to think about optimization and computation a little bit more.

Lex Fridman (1:13:12.740)

So what about Jeff Hinton?

Jay McClelland (1:13:15.740)

You've gotten a chance to work with him in that little thing.

Lex Fridman (1:13:20.740)

The set of people involved there is quite incredible.

Jay McClelland (1:13:24.740)

The small set of people under the PDP flag,

Lex Fridman (1:13:28.740)

it's just given the amount of impact those ideas have had over the years,

Jay McClelland (1:13:32.740)

it's kind of incredible to think about.

Lex Fridman (1:13:34.740)

But, you know, just like you said, like yourself,

Jay McClelland (1:13:38.740)

Jeffrey Hinton is seen as one of the, not just like a seminal figure in AI,

Lex Fridman (1:13:43.740)

but just a brilliant person,

Jay McClelland (1:13:45.740)

just like the horsepower of the mind is pretty high up there for him

Lex Fridman (1:13:49.740)

because he's just a great thinker.

Lex Fridman (1:13:52.740)

So what kind of ideas have you learned from him?

Lex Fridman (1:13:57.740)

Have you influenced each other on?

Jay McClelland (1:13:59.740)

Have you debated over what stands out to you in the full space of ideas here

Lex Fridman (1:14:05.740)

at the intersection of computation and cognition?

Jay McClelland (1:14:09.740)

Well, so Jeff has said many things to me that had a profound impact on my thinking.

Lex Fridman (1:14:18.740)

And he's written several articles which were way ahead of their time.

Jay McClelland (1:14:26.740)

He had two papers in 1981, just to give one example,

Lex Fridman (1:14:37.740)

one of which was essentially the idea of transformers

Lex Fridman (1:14:42.740)

and another of which was an early paper on semantic cognition

Lex Fridman (1:14:49.740)

which inspired him and Rumelhart and me throughout the 80s

Jay McClelland (1:15:01.740)

and, you know, still I think sort of grounds my own thinking

Lex Fridman (1:15:11.740)

about the semantic aspects of cognition.

Jay McClelland (1:15:16.740)

He also, in a small paper that was never published that he wrote in 1977,

Jay McClelland (1:15:25.740)

you know, before he actually arrived at UCSD or maybe a couple years even before that,

Jay McClelland (1:15:29.740)

I don't know, when he was a PhD student,

Lex Fridman (1:15:32.740)

he described how a neural network could do recursive computation.

Lex Fridman (1:15:40.740)

And it was a very clever idea that he's continued to explore over time,

Lex Fridman (1:15:48.740)

which was sort of the idea that when you call a subroutine,

Jay McClelland (1:15:56.740)

you need to save the state that you had when you called it

Lex Fridman (1:16:01.740)

so you can get back to where you were when you're finished with the subroutine.

Lex Fridman (1:16:04.740)

And the idea was that you would save the state of the calling routine

Lex Fridman (1:16:10.740)

by making fast changes to connection weights.

Lex Fridman (1:16:13.740)

And then when you finished with the subroutine call,

Lex Fridman (1:16:19.740)

those fast changes in the connection weights would allow you to go back

Jay McClelland (1:16:23.740)

to where you had been before and reinstate the previous context

Lex Fridman (1:16:27.740)

so that you could continue on with the top level of the computation.

Jay McClelland (1:16:32.740)

Anyway, that was part of the idea.

Lex Fridman (1:16:35.740)

And I always thought, okay, that's really, you know,

Jay McClelland (1:16:38.740)

he had extremely creative ideas that were quite a lot ahead of his time

Lex Fridman (1:16:44.740)

and many of them in the 1970s and early 1980s.

Lex Fridman (1:16:49.740)

So another thing about Geoff Hinton's way of thinking,

Lex Fridman (1:16:57.740)

which has profoundly influenced my effort to understand

Jay McClelland (1:17:05.740)

human mathematical cognition, is that he doesn't write too many equations.

Lex Fridman (1:17:13.740)

And people tell stories like, oh, in the Hinton Lab meetings,

Jay McClelland (1:17:17.740)

you don't get up at the board and write equations

Lex Fridman (1:17:19.740)

like you do in everybody else's machine learning lab.

Lex Fridman (1:17:22.740)

What you do is you draw a picture.

Lex Fridman (1:17:26.740)

And, you know, he explains aspects of the way deep learning works

Jay McClelland (1:17:33.740)

by putting his hands together and showing you the shape of a ravine

Lex Fridman (1:17:38.740)

and using that as a geometrical metaphor for what's happening

Jay McClelland (1:17:45.740)

as this gradient descent process.

Lex Fridman (1:17:47.740)

You're coming down the wall of a ravine.

Jay McClelland (1:17:49.740)

If you take too big a jump, you're going to jump to the other side.

Lex Fridman (1:17:53.740)

And so that's why we have to turn down the learning rate, for example.

Lex Fridman (1:17:59.740)

And it speaks to me of the fundamentally intuitive character of deep insight

Lex Fridman (1:18:12.740)

together with a commitment to really understanding

Jay McClelland (1:18:21.740)

in a way that's absolutely ultimately explicit and clear, but also intuitive.

Lex Fridman (1:18:31.740)

Yeah, there's certain people like that.

Jay McClelland (1:18:33.740)

Here's an example, some kind of weird mix of visual and intuitive

Lex Fridman (1:18:38.740)

and all those kinds of things.

Jay McClelland (1:18:40.740)

Feynman is another example, different style of thinking, but very unique.

Lex Fridman (1:18:44.740)

And when you're around those people, for me in the engineering realm,

Jay McClelland (1:18:48.740)

there's a guy named Jim Keller who's a chip designer, engineer.

Lex Fridman (1:18:52.740)

Every time I talk to him, it doesn't matter what we're talking about.

Jay McClelland (1:18:57.740)

Just having experienced that unique way of thinking transforms you

Lex Fridman (1:19:02.740)

and makes your work much better.

Lex Fridman (1:19:04.740)

And that's the magic.

Lex Fridman (1:19:06.740)

You look at Daniel Kahneman, you look at the great collaborations

Jay McClelland (1:19:10.740)

throughout the history of science.

Lex Fridman (1:19:12.740)

That's the magic of that.

Jay McClelland (1:19:13.740)

It's not always the exact ideas that you talk about,

Lex Fridman (1:19:16.740)

but it's the process of generating those ideas.

Jay McClelland (1:19:19.740)

Being around that, spending time with that human being,

Lex Fridman (1:19:22.740)

you can come up with some brilliant work,

Jay McClelland (1:19:24.740)

especially when it's cross disciplinary as it was a little bit in your case with Jeff.

Lex Fridman (1:19:29.740)

Yeah.

Jay McClelland (1:19:31.740)

Jeff is a descendant of the logician Boole.

Lex Fridman (1:19:38.740)

He comes from a long line of English academics.

Lex Fridman (1:19:43.740)

And together with the deeply intuitive thinking ability that he has,

Lex Fridman (1:19:51.740)

he also has, it's been clear, he's described this to me,

Lex Fridman (1:19:59.740)

and I think he's mentioned it from time to time in other interviews

Lex Fridman (1:20:04.740)

that he's had with people.

Jay McClelland (1:20:06.740)

He's wanted to be able to sort of think of himself as contributing

Lex Fridman (1:20:12.740)

to the understanding of reasoning itself, not just human reasoning.

Lex Fridman (1:20:22.740)

Like Boole is about logic, right?

Lex Fridman (1:20:25.740)

It's about what can we conclude from what else and how do we formalize that.

Lex Fridman (1:20:31.740)

And as a computer scientist, logician, philosopher,

Lex Fridman (1:20:40.740)

the goal is to understand how we derive truths from other,

Jay McClelland (1:20:46.740)

from givens and things like this.

Lex Fridman (1:20:48.740)

And the work that Jeff was doing in the early to mid 80s

Jay McClelland (1:20:57.740)

on something called the Bolton machine was his way of connecting

Lex Fridman (1:21:02.740)

with that Boolean tradition and bringing it into the more continuous,

Jay McClelland (1:21:07.740)

probabilistic graded constraint satisfaction realm.

Lex Fridman (1:21:11.740)

And it was a beautiful set of ideas linked with theoretical physics

Jay McClelland (1:21:20.740)

as well as with logic.

Lex Fridman (1:21:26.740)

And it's always been, I mean, I've always been inspired

Jay McClelland (1:21:31.740)

by the Bolton machine too.

Lex Fridman (1:21:33.740)

It's like, well, if the neurons are probabilistic rather than deterministic

Jay McClelland (1:21:38.740)

in their computations, then maybe this somehow is part of the serendipity

Lex Fridman (1:21:48.740)

or adventitiousness of the moment of insight, right?

Jay McClelland (1:21:53.740)

It might not have occurred at that particular instant.

Lex Fridman (1:21:56.740)

It might be sort of partially the result of a stochastic process.

Lex Fridman (1:22:00.740)

And that too is part of the magic of the emergence of some of these things.

Lex Fridman (1:22:07.740)

Well, you're right with the Boolean lineage and the dream of computer science

Jay McClelland (1:22:11.740)

is somehow, I mean, I certainly think of humans this way,

Lex Fridman (1:22:16.740)

that humans are one particular manifestation of intelligence,

Jay McClelland (1:22:20.740)

that there's something bigger going on and you're hoping to figure that out.

Lex Fridman (1:22:25.740)

The mechanisms of intelligence, the mechanisms of cognition

Jay McClelland (1:22:28.740)

are much bigger than just humans.

Lex Fridman (1:22:30.740)

Yeah. So I think of, I started using the phrase computational intelligence

Jay McClelland (1:22:37.740)

at some point as to characterize the field that I thought, you know,

Lex Fridman (1:22:42.740)

people like Geoff Hinton and many of the people I know at DeepMind

Jay McClelland (1:22:51.740)

are working in and where I feel like I'm, you know,

Lex Fridman (1:23:00.740)

I'm a kind of a human oriented computational intelligence researcher

Jay McClelland (1:23:06.740)

in that I'm actually kind of interested in the human solution.

Lex Fridman (1:23:10.740)

But at the same time, I feel like that's where a huge amount

Jay McClelland (1:23:18.740)

of the excitement of deep learning actually lies is in the idea that,

Lex Fridman (1:23:26.740)

you know, we may be able to even go beyond what we can achieve

Jay McClelland (1:23:32.740)

with our own nervous systems when we build computational intelligences

Lex Fridman (1:23:38.740)

that are, you know, not limited in the ways that we are by our own biology.

Jay McClelland (1:23:46.740)

Perhaps allowing us to scale the very mechanisms of human intelligence

Lex Fridman (1:23:51.740)

just increases power through scale.

Jay McClelland (1:23:55.740)

Yes. And I think that that, you know, obviously that's the,

Lex Fridman (1:24:03.740)

that's being played out massively at Google Brain, at OpenAI

Lex Fridman (1:24:08.740)

and to some extent at DeepMind as well.

Lex Fridman (1:24:11.740)

I guess I shouldn't say to some extent.

Jay McClelland (1:24:14.740)

Just the massive scale of the computations that are used to succeed

Lex Fridman (1:24:22.740)

at games like Go or to solve the protein folding problems

Jay McClelland (1:24:25.740)

that they've been solving and so on.

Lex Fridman (1:24:27.740)

Still not as many synapses and neurons as the human brain.

Lex Fridman (1:24:31.740)

So we still got, we're still beating them on that.

Lex Fridman (1:24:35.740)

We humans are beating the AIs, but they're catching up pretty quickly.

Jay McClelland (1:24:41.740)

You write about modeling of mathematical cognition.

Lex Fridman (1:24:45.740)

So let me first ask about mathematics in general.

Jay McClelland (1:24:49.740)

There's a paper titled Parallel Distributed Processing

Lex Fridman (1:24:53.740)

Approach to Mathematical Cognition where in the introduction

Jay McClelland (1:24:56.740)

there's some beautiful discussion of mathematics.

Lex Fridman (1:25:00.740)

And you referenced there Tristan Needham who criticizes a narrow

Jay McClelland (1:25:05.740)

form of view of mathematics by liking the studying of mathematics

Lex Fridman (1:25:10.740)

as symbol manipulation to studying music without ever hearing a note.

Lex Fridman (1:25:16.740)

So from that perspective, what do you think is mathematics?

Lex Fridman (1:25:20.740)

What is this world of mathematics like?

Jay McClelland (1:25:23.740)

Well, I think of mathematics as a set of tools for exploring

Lex Fridman (1:25:32.740)

idealized worlds that often turn out to be extremely relevant

Jay McClelland (1:25:42.740)

to the real world but need not.

Lex Fridman (1:25:47.740)

But they're worlds in which objects exist with idealized properties

Lex Fridman (1:26:01.740)

and in which the relationships among them can be characterized

Lex Fridman (1:26:07.740)

with precision so as to allow the implications of certain facts

Jay McClelland (1:26:17.740)

to then allow you to derive other facts with certainty.

Lex Fridman (1:26:22.740)

So if you have two triangles and you know that there is an angle

Jay McClelland (1:26:37.740)

in the first one that has the same measure as an angle in the second one

Lex Fridman (1:26:42.740)

and you know that the lengths of the sides adjacent to that angle

Jay McClelland (1:26:47.740)

in each of the two triangles, the corresponding sides adjacent

Lex Fridman (1:26:53.740)

to that angle also have the same measure, then you can then conclude

Jay McClelland (1:26:58.740)

that the triangles are congruent.

Lex Fridman (1:27:02.740)

That is to say they have all of their properties in common.

Lex Fridman (1:27:06.740)

And that is something about triangles.

Lex Fridman (1:27:11.740)

It's not a matter of formulas.

Jay McClelland (1:27:15.740)

These are idealized objects.

Lex Fridman (1:27:18.740)

In fact, we built bridges out of triangles and we understand

Lex Fridman (1:27:26.740)

how to measure the height of something we can't climb by extending

Lex Fridman (1:27:32.740)

these ideas about triangles a little further.

Lex Fridman (1:27:36.740)

And all of the ability to get a tiny speck of matter launched

Lex Fridman (1:27:49.740)

from the planet Earth to intersect with some tiny, tiny little body

Jay McClelland (1:27:56.740)

way out in way beyond Pluto somewhere at exactly a predicted time

Lex Fridman (1:28:02.740)

and date is something that depends on these ideas.

Lex Fridman (1:28:08.740)

And it's actually happening in the real physical world that these ideas

Lex Fridman (1:28:18.740)

make contact with it in those kinds of instances.

Lex Fridman (1:28:27.740)

But there are these idealized objects, these triangles or these distances

Lex Fridman (1:28:32.740)

or these points, whatever they are, that allow for this set of tools

Jay McClelland (1:28:40.740)

to be created that then gives human beings this incredible leverage

Lex Fridman (1:28:47.740)

that they didn't have without these concepts.

Lex Fridman (1:28:51.740)

And I think this is actually already true when we think about just,

Lex Fridman (1:29:01.740)

you know, the natural numbers.

Jay McClelland (1:29:06.740)

I always like to include zero, so I'm going to say the nonnegative integers,

Lex Fridman (1:29:11.740)

but that's a place where some people prefer not to include zero.

Jay McClelland (1:29:17.740)

We like zero here, natural numbers, zero, one, two, three, four, five,

Lex Fridman (1:29:21.740)

six, seven, and so on.

Jay McClelland (1:29:23.740)

Yeah. And because they give you the ability to be exact about

Lex Fridman (1:29:36.740)

how many sheep you have.

Jay McClelland (1:29:38.740)

I sent you out this morning, there were 23 sheep.

Lex Fridman (1:29:41.740)

You came back with only 22. What happened?

Jay McClelland (1:29:44.740)

The fundamental problem of physics, how many sheep you have.

Lex Fridman (1:29:48.740)

It's a fundamental problem of human society that you damn well better

Jay McClelland (1:29:53.740)

bring back the same number of sheep as you started with.

Lex Fridman (1:29:57.740)

And it allows commerce, it allows contracts, it allows the establishment

Jay McClelland (1:30:03.740)

of records and so on to have systems that allow these things to be notated.

Lex Fridman (1:30:10.740)

But they have an inherent aboutness to them that's one in the same time sort of

Jay McClelland (1:30:20.740)

abstract and idealized and generalizable, while on the other hand,

Lex Fridman (1:30:26.740)

potentially very, very grounded and concrete.

Lex Fridman (1:30:30.740)

And one of the things that makes for the incredible achievements of the human mind

Lex Fridman (1:30:41.740)

is the fact that humans invented these idealized systems that leverage

Jay McClelland (1:30:49.740)

the power of human thought in such a way as to allow all this kind of thing to happen.

Lex Fridman (1:30:57.740)

And so that's what mathematics to me is the development of systems for thinking about

Jay McClelland (1:31:06.740)

the properties and relations among sets of idealized objects and

Lex Fridman (1:31:18.740)

the mathematical notation system that we unfortunately focus way too much on

Jay McClelland (1:31:26.740)

is just our way of expressing propositions about these properties.

Lex Fridman (1:31:36.740)

It's just like we're talking with Chomsky in language.

Jay McClelland (1:31:39.740)

It's the thing we've invented for the communication of those ideas.

Lex Fridman (1:31:43.740)

They're not necessarily the deep representation of those ideas.

Lex Fridman (1:31:48.740)

So what's a good way to model such powerful mathematical reasoning, would you say?

Lex Fridman (1:31:57.740)

What are some ideas you have for capturing this in a model?

Jay McClelland (1:32:01.740)

The insights that human mathematicians have had is a combination of the kind of the

Jay McClelland (1:32:10.740)

intuitive kind of connectionist like knowledge that makes it so that something is just like

Jay McClelland (1:32:24.740)

obviously true so that you don't have to think about why it's true.

Jay McClelland (1:32:31.740)

That then makes it possible to then take the next step and ponder and reason and

Jay McClelland (1:32:40.740)

figure out something that you previously didn't have that intuition about.

Lex Fridman (1:32:45.740)

It then ultimately becomes a part of the intuition that the next generation of

Jay McClelland (1:32:54.740)

mathematical thinkers have to ground their own thinking on so that they can extend the ideas even further.

Jay McClelland (1:33:02.740)

I came across this quotation from Henri Poincare while I was walking in the woods with my wife

Jay McClelland (1:33:15.740)

in a state park in Northern California late last summer.

Lex Fridman (1:33:20.740)

And what it said on the bench was it is by logic that we prove but by intuition that we discover.

Lex Fridman (1:33:32.740)

And so what for me the essence of the project is to understand how to bring the intuitive

Jay McClelland (1:33:41.740)

connectionist resources to bear on letting the intuitive discovery arise from engagement in

Jay McClelland (1:33:56.740)

thinking with this formal system.

Lex Fridman (1:33:59.740)

So I think of the ability of somebody like Hinton or Newton or Einstein or Rumelhart or

Jay McClelland (1:34:14.740)

Poincare to Archimedes is another example.

Lex Fridman (1:34:21.740)

So suddenly a flash of insight occurs. It's like the constellation of all of these

Jay McClelland (1:34:31.740)

simultaneous constraints that somehow or other causes the mind to settle into a novel state that

Jay McClelland (1:34:38.740)

it never did before and give rise to a new idea that then you can say, okay, well, now how can I

Lex Fridman (1:34:51.740)

prove this? How do I write down the steps of that theorem that allow me to make it rigorous and certain?

Lex Fridman (1:35:01.740)

And so I feel like the kinds of things that we're beginning to see deep learning systems do of

Jay McClelland (1:35:14.740)

their own accord kind of gives me this feeling of hope or encouragement that ultimately it'll all happen.

Lex Fridman (1:35:34.740)

So in particular as many people now have become really interested in thinking about, you know,

Jay McClelland (1:35:46.740)

neural networks that have been trained with massive amounts of text can be given a prompt and they

Jay McClelland (1:35:55.740)

can then sort of generate some really interesting, fanciful, creative story from that prompt.

Lex Fridman (1:36:05.740)

And there's kind of like a sense that they've somehow synthesized something like novel out of

Jay McClelland (1:36:15.740)

the, you know, all of the particulars of all of the billions and billions of experiences that went

Jay McClelland (1:36:22.740)

into the training data that gives rise to something like this sort of intuitive sense of what would

Jay McClelland (1:36:29.740)

be a fun and interesting little story to tell or something like that. It just sort of wells up out

Jay McClelland (1:36:36.740)

of the letting the thing play out its own imagining of what somebody might say given this prompt as

Jay McClelland (1:36:47.740)

an input to get it to start to generate its own thoughts. And to me that sort of represents the

Jay McClelland (1:36:56.740)

potential of capturing the intuitive side of this.

Lex Fridman (1:37:01.740)

And there's other examples, I don't know if you find them as captivating is, you know, on the

Jay McClelland (1:37:06.740)

DeepMind side with AlphaZero, if you study chess, the kind of solutions that has come up in terms

Jay McClelland (1:37:12.740)

of chess, it is, there's novel ideas there. It feels very like there's brilliant moments of insight.

Lex Fridman (1:37:20.740)

And the mechanism they use, if you think of search as maybe more towards good old fashioned AI and

Jay McClelland (1:37:31.740)

then there's the connection is the neural network that has the intuition of looking at a board,

Jay McClelland (1:37:37.740)

looking at a set of patterns and saying, how good is this set of positions? And the next few

Jay McClelland (1:37:42.740)

positions, how good are those? And that's it. That's just an intuition. Grandmasters have this

Lex Fridman (1:37:49.740)

and understanding positionally, tactically, how good the situation is, how can it be improved

Jay McClelland (1:37:55.740)

without doing this full, like deep search. And then maybe doing a little bit of what human chess

Jay McClelland (1:38:03.740)

players call calculation, which is the search, taking a particular set of steps down the line to

Jay McClelland (1:38:08.740)

see how they unroll. But there is moments of genius in those systems too. So that's another hopeful

Jay McClelland (1:38:16.740)

illustration that from neural networks can emerge this novel creation of an idea.

Jay McClelland (1:38:25.740)

Yes. And I think that, you know, I think Demis Hassabis is, you know, he's spoken about those

Jay McClelland (1:38:34.740)

things. I heard him describe a move that was made in one of the go matches against Lisa Dahl in a

Jay McClelland (1:38:44.740)

very similar way. And it caused me to become really excited to kind of collaborate with some of those

Jay McClelland (1:38:52.740)

people and analyze it at DeepMind. So I think though that what I like to really emphasize here

Jay McClelland (1:39:05.740)

is one part of what I like to emphasize about mathematical cognition at least is that philosophers

Lex Fridman (1:39:15.740)

and logicians going back three or even a little more than 3000 years ago began to develop these

Jay McClelland (1:39:28.740)

formal systems and gradually the whole idea about thinking formally got constructed. And, you know,

Jay McClelland (1:39:45.740)

it's preceded Euclid, certainly present in the work of Thales and others. And I'm not the world's

Jay McClelland (1:39:55.740)

leading expert in all the details of that history, but Euclid's elements were the kind of the touch

Jay McClelland (1:40:03.740)

point of a coherent document that sort of laid out this idea of an actual formal system within which

Jay McClelland (1:40:15.740)

these objects were characterized and the system of inference that allowed new truths to be derived

Jay McClelland (1:40:31.580)

from others was sort of like established as a paradigm. And what I find interesting is the

Jay McClelland (1:40:43.900)

idea that the ability to become a person who is capable of thinking in this abstract formal way

Jay McClelland (1:40:55.020)

is a result of the same kind of immersion in experience thinking in that way that we now

Jay McClelland (1:41:10.060)

begin to think of our understanding of language as being, right? So, we immerse ourselves in a

Jay McClelland (1:41:16.440)

particular language, in a particular world of objects and their relationships and we learn

Jay McClelland (1:41:22.780)

to talk about that and we develop intuitive understanding of the real world. In a similar

Jay McClelland (1:41:30.220)

way, we can think that what academia has created for us, what those early philosophers and their

Jay McClelland (1:41:39.740)

academies in Athens and Alexandria and other places allowed was the development of these

Jay McClelland (1:41:49.780)

schools of thought, modes of thought that then become deeply ingrained and it becomes what it

Jay McClelland (1:42:00.660)

is that makes it so that somebody like Jerry Fodor would think that systematic thought is

Jay McClelland (1:42:11.420)

the essential characteristic of the human mind as opposed to a derived and an acquired characteristic

Jay McClelland (1:42:20.860)

that results from acculturation in a certain mode that's been invented by humans.

Jay McClelland (1:42:28.460)

Would you say it's more fundamental than like language? If we start dancing, if we bring

Jay McClelland (1:42:34.700)

Chomsky back into the conversation, first of all, is it unfair to draw a line between mathematical

Lex Fridman (1:42:43.340)

cognition and language, linguistic cognition?

Jay McClelland (1:42:48.540)

I think that's a very interesting question and I think it's one of the ones that I'm actually very

Jay McClelland (1:42:54.780)

interested in right now, but I think the answer is in important ways, it is important to draw that

Jay McClelland (1:43:06.380)

line, but then to come back and look at it again and see some of the subtleties and interesting

Jay McClelland (1:43:12.540)

aspects of the difference. So if we think about Chomsky himself, he was born into an academic

Jay McClelland (1:43:34.300)

family. His father was a professor of rabbinical studies at a small rabbinical college in

Jay McClelland (1:43:40.220)

Philadelphia. He was deeply enculturated in a culture of thought and reason and brought to the

Jay McClelland (1:43:59.820)

effort to understand natural language, this profound engagement with these formal systems. I

Jay McClelland (1:44:13.260)

think that there was tremendous power in that and that Chomsky had some amazing insights into the

Jay McClelland (1:44:23.420)

structure of natural language, but that, I'm going to use the word but there, the actual intuitive

Jay McClelland (1:44:34.300)

knowledge of these things only goes so far and does not go as far as it does in people like

Jay McClelland (1:44:41.260)

Chomsky himself. And this was something that was discovered in the PhD dissertation of Lyla

Jay McClelland (1:44:48.620)

Gleitman, who was actually trained in the same linguistics department with Chomsky. So what Lyla

Jay McClelland (1:44:55.340)

discovered was that the intuitions that linguists had about even the meaning of a phrase, not just

Jay McClelland (1:45:09.980)

about its grammar, but about what they thought a phrase must mean were very different from the

Jay McClelland (1:45:17.820)

intuitions of an ordinary person who wasn't a formally trained thinker. And well, it recently

Jay McClelland (1:45:27.580)

has become much more salient. I happened to have learned about this when I myself was a PhD student

Jay McClelland (1:45:32.380)

at the University of Pennsylvania, but I never knew how to put it together with all of my other

Jay McClelland (1:45:38.380)

thinking about these things. So I actually currently have the hypothesis that formally

Jay McClelland (1:45:45.820)

trained linguists and other formally trained academics, whether it be linguistics, philosophy,

Jay McClelland (1:45:58.620)

cognitive science, computer science, machine learning, mathematics,

Jay McClelland (1:46:02.940)

have a mode of engagement with experience that is intuitively deeply structured to be more

Jay McClelland (1:46:17.900)

organized around the systematicity and ability to be conformant with the principles of a system

Lex Fridman (1:46:35.580)

than is actually true of the natural human mind without that immersion.

Jay McClelland (1:46:42.300)

That's fascinating. So the different fields and approaches with which you start to study the mind

Jay McClelland (1:46:48.620)

actually take you away from the natural operation of the mind. So it makes it very difficult for you

Jay McClelland (1:46:56.860)

to be somebody who introspects.

Jay McClelland (1:46:59.020)

Yes. And this is where things about human belief and so called knowledge that we consider

Jay McClelland (1:47:16.620)

private, not our business to manipulate in others. We are not entitled to tell somebody else what to

Jay McClelland (1:47:29.500)

believe about certain kinds of things. What are those beliefs? Well, they are the product of this

Jay McClelland (1:47:42.540)

sort of immersion and enculturation. That is what I believe.

Lex Fridman (1:47:51.660)

And that's limiting.

Jay McClelland (1:47:55.020)

It's something to be aware of.

Lex Fridman (1:47:58.140)

Does that limit you from having a good model of cognition?

Jay McClelland (1:48:04.380)

It can.

Lex Fridman (1:48:04.860)

So when you look at mathematical or linguistics, I mean, what is that line then? So is Chomsky

Jay McClelland (1:48:13.660)

unable to sneak up to the full picture of cognition? Are you, when you're focusing on

Lex Fridman (1:48:17.580)

mathematical thinking, are you also unable to do so?

Jay McClelland (1:48:22.940)

I think you're right. I think that's a great way of characterizing it. And

Jay McClelland (1:48:27.180)

I also think that it's related to the concept of beginner's mind and another concept called the

Jay McClelland (1:48:43.580)

expert blind spot. So the expert blind spot is much more prosaic seeming than this point that

Jay McClelland (1:48:53.180)

you were just making. But it's something that plagues experts when they try to communicate

Jay McClelland (1:49:01.260)

their understanding to non experts. And that is that things are self evident to them that

Jay McClelland (1:49:12.540)

they can't begin to even think about how they could explain it to somebody else.

Jay McClelland (1:49:23.180)

Because it's just like so patently obvious that it must be true. And

Lex Fridman (1:49:31.580)

when Kronacker said, God made the natural numbers, all else is the work of man,

Jay McClelland (1:49:47.180)

he was expressing that intuition that somehow or other, the basic fundamentals of discrete

Jay McClelland (1:49:57.980)

quantities being countable and innumerable and indefinite in number was not something that

Jay McClelland (1:50:10.780)

had to be discovered. But he was wrong. It turns out that many cognitive scientists

Lex Fridman (1:50:21.020)

agreed with him for a time. There was a long period of time where the natural

Jay McClelland (1:50:27.580)

numbers were considered to be a part of the innate endowment of core knowledge or to use

Jay McClelland (1:50:35.820)

the kind of phrases that Spelke and Kerry used to talk about what they believe are

Jay McClelland (1:50:41.500)

the innate primitives of the human mind. And they no longer believe that. It's actually

Jay McClelland (1:50:50.700)

been more or less accepted by almost everyone that the natural numbers are actually a cultural

Jay McClelland (1:50:56.620)

construction. And it's so interesting to go back and study those few people who still exist who

Jay McClelland (1:51:04.300)

don't have those systems. So this is just an example to me where a certain mode of thinking

Jay McClelland (1:51:13.100)

about language itself or a certain mode of thinking about geometry and those kinds of

Jay McClelland (1:51:20.940)

relations. So it becomes so second nature that you don't know what it is that you need to teach. And

Jay McClelland (1:51:30.300)

in fact, we don't really teach it all that explicitly anyway. You take a math class,

Jay McClelland (1:51:41.420)

the professor sort of teaches it to you the way they understand it. Some of the students in the

Jay McClelland (1:51:47.420)

class sort of like they get it. They start to get the way of thinking and they can actually do the

Jay McClelland (1:51:52.780)

problems that get put on the homework that the professor thinks are interesting and challenging

Jay McClelland (1:51:57.500)

ones. But most of the students who don't kind of engage as deeply don't ever get. And we think,

Jay McClelland (1:52:08.220)

oh, that man must be brilliant. He must have this special insight. But he must have some

Jay McClelland (1:52:14.300)

some biological sort of bit that's different, that makes him so that he or she could have

Jay McClelland (1:52:20.860)

that insight. Although I don't want to dismiss biological individual differences completely,

Jay McClelland (1:52:31.340)

I find it much more interesting to think about the possibility that it was that difference in the

Jay McClelland (1:52:39.660)

dinner table conversation at the Chomsky house when he was growing up that made it so that he

Jay McClelland (1:52:45.820)

had that cast of mind. Yeah. And there's a few topics we talked about that kind of interconnect

Jay McClelland (1:52:53.580)

because I wonder the better I get at certain things, we humans, the deeper we understand

Jay McClelland (1:52:59.900)

something, what are you starting to then miss about the rest of the world? We talked about David

Lex Fridman (1:53:11.020)

and his degenerative mind. And, you know, when you look in the mirror and wonder how different

Jay McClelland (1:53:19.980)

am I am I cognitively from the man I was a month ago, from the man I was a year ago, like what,

Jay McClelland (1:53:26.780)

you know, if I can, having thought about language of Chomsky for 10, 20 years, what am I no longer

Jay McClelland (1:53:35.980)

able to see? What is in my blind spot? And how big is that? And then to somehow be able to leap back

Jay McClelland (1:53:43.100)

out of your deep, like structure that you form for yourself about thinking about the world,

Jay McClelland (1:53:48.860)

leap back and look at the big picture again, or jump out of the your current way of thinking.

Lex Fridman (1:53:54.780)

And to be able to introspect, like what are the limitations of your mind? How is your mind less

Jay McClelland (1:54:00.860)

powerful than it used to be or more powerful or different, powerful in different ways? So that

Jay McClelland (1:54:06.380)

seems to be a difficult thing to do because we're living, we're looking at the world through the

Jay McClelland (1:54:11.980)

lens of our mind, right? To step outside and introspect is difficult, but it seems necessary

Jay McClelland (1:54:17.980)

if you want to make progress. You know, one of the threads of psychological research that's always

Jay McClelland (1:54:25.020)

been very, I don't know, important to me to be aware of is the idea that our explanations of our

Jay McClelland (1:54:38.620)

own behavior aren't necessarily actually part of the causal process that caused that behavior to

Jay McClelland (1:54:53.980)

occur, or even valid observations of the set of constraints that led to the outcome, but they are

Jay McClelland (1:55:03.980)

post hoc rationalizations that we can give based on information at our disposal about what might

Jay McClelland (1:55:11.660)

have contributed to the result that we came to when asked. And so this is an idea that was

Jay McClelland (1:55:21.340)

introduced in a very important paper by Nisbet and Wilson about, you know, the limits on our ability

Jay McClelland (1:55:29.580)

to be aware of the factors that cause us to make the choices that we make. And, you know, I think

Jay McClelland (1:55:42.940)

it's something that we really ought to be much more cognizant of, in general, as human beings,

Jay McClelland (1:55:54.380)

is that our own insight into exactly why we hold the beliefs that we do and we hold the attitudes

Lex Fridman (1:56:01.500)

and make the choices and feel the feelings that we do is not something that we totally control

Jay McClelland (1:56:12.060)

or totally observe. And it's subject to, you know, our culturally transmitted understanding of what

Jay McClelland (1:56:25.340)

it is that is the mode that we give to explain these things when asked to do so as much as it is

Jay McClelland (1:56:34.780)

about anything else. And so even our ability to introspect and think we have access to our own

Lex Fridman (1:56:42.060)

thoughts is a product of culture and belief, you know, practice.

Lex Fridman (1:56:47.260)

So let me ask you the big question of advice. So you've lived an incredible life in terms of the

Jay McClelland (1:56:57.180)

ideas you've put out into the world, in terms of the trajectory you've taken through your career,

Jay McClelland (1:57:02.540)

through your life. What advice would you give to young people today, in high school, in college,

Lex Fridman (1:57:09.980)

about how to have a career or how to have a life they can be proud of?

Jay McClelland (1:57:16.300)

Finding the thing that you are intrinsically motivated to engage with and then celebrating

Jay McClelland (1:57:27.660)

that discovery is what it's all about. When I was in college, I struggled with that. I had thought

Jay McClelland (1:57:43.020)

I wanted to be a psychiatrist because I think I was interested in human psychology in high school.

Lex Fridman (1:57:50.620)

And at that time, the only sort of information I had that had anything to do with the psyche was,

Jay McClelland (1:57:58.300)

you know, Freud and Erich Fromm and sort of popular psychiatry kinds of things.

Lex Fridman (1:58:03.820)

And so, well, they were psychiatrists, right? So I had to be a psychiatrist.

Lex Fridman (1:58:08.780)

And that meant I had to go to medical school. And I got to college and I find myself taking,

Jay McClelland (1:58:14.700)

you know, the first semester of a three quarter physics class and it was mechanics. And this was

Lex Fridman (1:58:21.820)

so far from what it was I was interested in, but it was also too early in the morning in the winter

Jay McClelland (1:58:26.860)

court semester. So I never made it to the physics class. But I wondered about the rest of my

Jay McClelland (1:58:34.780)

freshman year and most of my sophomore year until I found myself in the midst of this situation where

Jay McClelland (1:58:45.260)

around me there was this big revolution happening. I was at Columbia University in 1968 and

Jay McClelland (1:58:54.220)

the Vietnam War is going on. Columbia is building a gym in Morningside Heights, which is part of

Jay McClelland (1:58:59.580)

Harlem. And people are thinking, oh, the big bad rich guys are stealing the parkland that

Jay McClelland (1:59:06.700)

belongs to the people of Harlem. And, you know, they're part of the military industrial complex,

Jay McClelland (1:59:13.980)

which is enslaving us and sending us all off to war in Vietnam. And so there was a big revolution

Jay McClelland (1:59:20.380)

that involved a confluence of black activism and, you know, SDS and social justice and the whole

Lex Fridman (1:59:27.740)

university blew up and got shut down. And I got a chance to sort of think about

Lex Fridman (1:59:34.780)

why people were behaving the way they were in this context. And I, you know, I happened to have

Jay McClelland (1:59:42.380)

taken mathematical statistics. I happened to have been taking psychology that quarter at just cycle

Jay McClelland (1:59:48.540)

one. And somehow things in that space all ran together in my mind and got me really excited

Jay McClelland (1:59:54.780)

about asking questions about why people, what made certain people go into the buildings and not

Jay McClelland (20:07.540)

to the cognition.

Lex Fridman (20:10.380)

And I realized at one point, I got this opportunity

Jay McClelland (20:15.700)

to go to a conference where I heard a talk by a man named

Lex Fridman (20:18.700)

James Anderson, who was an engineer,

Lex Fridman (20:22.540)

but by then a professor in a psychology department, who

Lex Fridman (20:26.300)

had used linear algebra to create neural network

Jay McClelland (20:32.220)

models of perception and categorization and memory.

Lex Fridman (20:37.540)

And it just blew me out of the water

Jay McClelland (20:41.180)

that one could create a model that was simulating neurons,

Lex Fridman (20:47.940)

not just engaged in a stepwise algorithmic process that

Jay McClelland (20:56.900)

was construed abstractly.

Lex Fridman (20:58.540)

But it was simulating remembering and recalling

Lex Fridman (21:03.540)

and recognizing the prior occurrence of a stimulus

Lex Fridman (21:07.980)

or something like that.

Lex Fridman (21:08.900)

So for me, this was a bridge between the mind and the brain.

Lex Fridman (21:14.900)

And I remember I was walking across campus one day in 1977,

Lex Fridman (21:20.500)

and I almost felt like St. Paul on the road to Damascus.

Lex Fridman (21:25.020)

I said to myself, if I think about the mind in terms

Jay McClelland (21:30.860)

of a neural network, it will help

Lex Fridman (21:32.380)

me answer the questions about the mind

Jay McClelland (21:33.980)

that I'm trying to answer.

Lex Fridman (21:36.100)

And that really excited me.

Lex Fridman (21:38.820)

So I think that a lot of people were

Lex Fridman (21:43.260)

becoming excited about that.

Lex Fridman (21:45.060)

And one of those people was Jim Anderson, who I had mentioned.

Lex Fridman (21:49.980)

Another one was Steve Grossberg, who

Jay McClelland (21:52.140)

had been writing about neural networks since the 60s.

Lex Fridman (21:58.700)

And Jeff Hinton was yet another.

Lex Fridman (22:00.700)

And his PhD dissertation showed up in an applicant pool

Lex Fridman (22:08.780)

to a postdoctoral training program

Jay McClelland (22:11.700)

that Dave and Don, the two men I mentioned before,

Lex Fridman (22:16.220)

Rumelhart and Norman, were administering.

Lex Fridman (22:19.340)

And Rumelhart got really excited about Hinton's PhD dissertation.

Lex Fridman (22:26.140)

And so Hinton was one of the first people

Jay McClelland (22:30.580)

who came and joined this group of postdoctoral scholars

Lex Fridman (22:34.780)

that was funded by this wonderful grant that they got.

Jay McClelland (22:39.340)

Another one who is also well known

Lex Fridman (22:41.900)

in neural network circles is Paul Smolenski.

Jay McClelland (22:45.660)

He was another one of that group.

Lex Fridman (22:47.900)

Anyway, Jeff and Jim Anderson organized a conference

Jay McClelland (22:55.940)

at UCSD where we were.

Lex Fridman (22:59.460)

And it was called Parallel Models of Associative Memory.

Lex Fridman (23:04.540)

And it brought all the people together

Lex Fridman (23:06.380)

who had been thinking about these kinds of ideas

Jay McClelland (23:08.980)

in 1979 or 1980.

Lex Fridman (23:11.780)

And this began to kind of really resonate

Jay McClelland (23:18.820)

with some of Rumelhart's own thinking,

Lex Fridman (23:23.220)

some of his reasons for wanting something

Jay McClelland (23:26.380)

other than the kinds of computation

Lex Fridman (23:28.620)

he'd been doing so far.

Lex Fridman (23:29.980)

So let me talk about Rumelhart now for a minute,

Lex Fridman (23:32.020)

OK, with that context.

Jay McClelland (23:33.060)

Well, let me also just pause because he

Lex Fridman (23:34.820)

said so many interesting things before we go to Rumelhart.

Lex Fridman (23:37.620)

So first of all, for people who are not familiar,

Lex Fridman (23:40.940)

neural networks are at the core of the machine learning,

Jay McClelland (23:43.140)

deep learning revolution of today.

Lex Fridman (23:45.300)

Geoffrey Hinton that we mentioned

Jay McClelland (23:46.700)

is one of the figures that were important in the history

Lex Fridman (23:50.420)

like yourself in the development of these neural networks,

Jay McClelland (23:53.060)

artificial neural networks that are then

Lex Fridman (23:54.820)

used for the machine learning application.

Jay McClelland (23:56.900)

Like I mentioned, the backpropagation paper

Lex Fridman (23:59.300)

is one of the optimization mechanisms

Jay McClelland (24:02.020)

by which these networks can learn.

Lex Fridman (24:05.820)

And the word parallel is really interesting.

Lex Fridman (24:09.580)

So it's almost like synonymous from a computational

Lex Fridman (24:12.940)

perspective how you thought at the time about neural networks

Jay McClelland (24:17.260)

as parallel computation.

Lex Fridman (24:20.140)

Would that be fair to say?

Jay McClelland (24:21.140)

Well, yeah, the parallel, the word parallel in this

Lex Fridman (24:25.580)

comes from the idea that each neuron is

Lex Fridman (24:30.060)

an independent computational unit, right?

Lex Fridman (24:33.540)

It gathers data from other neurons,

Jay McClelland (24:36.420)

it integrates it in a certain way,

Lex Fridman (24:39.340)

and then it produces a result. And it's

Jay McClelland (24:41.660)

a very simple little computational unit.

Lex Fridman (24:44.900)

But it's autonomous in the sense that it does its thing, right?

Jay McClelland (24:51.260)

It's in a biological medium where

Lex Fridman (24:53.380)

it's getting nutrients and various chemicals

Jay McClelland (24:57.340)

from that medium.

Lex Fridman (25:00.300)

But you can think of it as almost like a little computer

Jay McClelland (25:05.820)

in and of itself.

Lex Fridman (25:08.020)

So the idea is that each our brains have, oh, look,

Jay McClelland (25:13.220)

100 or hundreds, almost a billion

Lex Fridman (25:17.100)

of these little neurons, right?

Lex Fridman (25:21.700)

And they're all capable of doing their work at the same time.

Lex Fridman (25:25.500)

So it's like instead of just a single central processor that's

Jay McClelland (25:30.180)

engaged in chug one step after another,

Lex Fridman (25:36.700)

we have a billion of these little computational units

Jay McClelland (25:41.100)

working at the same time.

Lex Fridman (25:42.660)

So at the time that's, I don't know, maybe you can comment,

Jay McClelland (25:45.860)

it seems to me, even still to me,

Lex Fridman (25:49.100)

quite a revolutionary way to think about computation

Jay McClelland (25:52.860)

relative to the development of theoretical computer science

Lex Fridman (25:56.660)

alongside of that where it's very much like sequential computer.

Jay McClelland (26:00.460)

You're analyzing algorithms that are running on a single computer.

Lex Fridman (26:04.340)

You're saying, wait a minute, why don't we

Jay McClelland (26:08.300)

take a really dumb, very simple computer

Lex Fridman (26:11.420)

and just have a lot of them interconnected together?

Lex Fridman (26:14.420)

And they're all operating in their own little world

Lex Fridman (26:16.620)

and they're communicating with each other

Lex Fridman (26:18.620)

and thinking of computation that way.

Lex Fridman (26:21.020)

And from that kind of computation,

Jay McClelland (26:24.540)

trying to understand how things like certain characteristics

Lex Fridman (26:28.580)

of the human mind can emerge.

Jay McClelland (26:31.140)

That's quite a revolutionary way of thinking, I would say.

Lex Fridman (26:35.940)

Well, yes, I agree with you.

Lex Fridman (26:37.500)

And there's still this sort of sense

Lex Fridman (26:44.020)

of not sort of knowing how we kind of get all the way there,

Jay McClelland (26:53.740)

I think.

Lex Fridman (26:54.380)

And this very much remains at the core of the questions

Jay McClelland (26:58.700)

that everybody's asking about the capabilities

Lex Fridman (27:01.060)

of deep learning and all these kinds of things.

Lex Fridman (27:02.940)

But if I could just play this out a little bit,

Lex Fridman (27:07.460)

a convolutional neural network or a CNN,

Jay McClelland (27:11.060)

which many people may have heard of, is a set of,

Lex Fridman (27:19.580)

you could think of it biologically as a set of

Jay McClelland (27:24.900)

collections of neurons.

Lex Fridman (27:27.980)

Each collection has maybe 10,000 neurons in it.

Lex Fridman (27:33.620)

But there's many layers.

Lex Fridman (27:35.740)

Some of these things are hundreds or even

Jay McClelland (27:38.100)

1,000 layers deep.

Lex Fridman (27:39.940)

But others are closer to the biological brain

Lex Fridman (27:43.660)

and maybe they're like 20 layers deep or something like that.

Lex Fridman (27:47.020)

So within each layer, we have thousands of neurons

Jay McClelland (27:52.980)

or tens of thousands maybe.

Lex Fridman (27:54.460)

Well, in the brain, we probably have millions in each layer.

Lex Fridman (27:59.460)

But we're getting sort of similar in a certain way.

Lex Fridman (28:05.940)

And then we think, OK, at the bottom level,

Jay McClelland (28:09.220)

there's an array of things that are like the photoreceptors.

Lex Fridman (28:12.140)

In the eye, they respond to the amount

Jay McClelland (28:14.980)

of light of a certain wavelength at a certain location

Lex Fridman (28:17.900)

on the pixel array.

Lex Fridman (28:21.180)

So that's like the biological eye.

Lex Fridman (28:24.540)

And then there's several further stages going up,

Jay McClelland (28:27.300)

layers of these neuron like units.

Lex Fridman (28:30.460)

And you go from that raw input array of pixels

Jay McClelland (28:36.700)

to the classification, you've actually

Lex Fridman (28:40.820)

built a system that could do the same kind of thing

Jay McClelland (28:44.180)

that you and I do when we open our eyes and we look around

Lex Fridman (28:46.700)

and we see there's a cup, there's a cell phone,

Jay McClelland (28:49.700)

there's a water bottle.

Lex Fridman (28:52.220)

And these systems are doing that now, right?

Lex Fridman (28:54.940)

So they are, in terms of the parallel idea

Lex Fridman (29:00.380)

that we were talking about before,

Jay McClelland (29:02.220)

they are doing this massively parallel computation

Lex Fridman (29:05.540)

in the sense that each of the neurons in each

Jay McClelland (29:08.860)

of those layers is thought of as computing

Lex Fridman (29:12.300)

its little bit of something about the input

Jay McClelland (29:17.740)

simultaneously with all the other ones in the same layer.

Lex Fridman (29:21.980)

We get to the point of abstracting that away

Lex Fridman (29:24.100)

and thinking, oh, it's just one whole vector that's

Lex Fridman (29:27.100)

being computed, one activation pattern that's

Jay McClelland (29:30.460)

computed in a single step.

Lex Fridman (29:32.020)

And that abstraction is useful, but it's still that parallel.

Lex Fridman (29:39.260)

And distributed processing, right?

Lex Fridman (29:41.300)

Each one of these guys is just contributing

Jay McClelland (29:43.180)

a tiny bit to that whole thing.

Lex Fridman (29:45.100)

And that's the excitement that you felt,

Jay McClelland (29:46.700)

that from these simple things, you can emerge.

Lex Fridman (29:50.700)

When you add these level of abstractions on it,

Jay McClelland (29:53.860)

you can start getting all the beautiful things

Lex Fridman (29:56.020)

that we think about as cognition.

Lex Fridman (29:58.260)

And so, OK, so you have this conference.

Jay McClelland (2:00:01.420)

others and things like that. And so suddenly I had a path forward and I had just been wandering

Jay McClelland (2:00:07.260)

around aimlessly. And at the different points in my career, you know, and I think, okay,

Jay McClelland (2:00:12.540)

well, should I take this class or should I just read that book about some idea that I want to

Jay McClelland (2:00:26.780)

understand better, you know, or should I pursue the thing that excites me and interests me or

Jay McClelland (2:00:33.500)

should I, you know, meet some requirement? You know, that's, I always did the latter.

Lex Fridman (2:00:39.340)

So I ended up, my professors in psychology were, thought I was great. They wanted me to go to

Jay McClelland (2:00:46.940)

graduate school. They nominated me for Phi Beta Kappa. And I went to the Phi Beta Kappa ceremony

Lex Fridman (2:00:55.420)

and this guy came up and he said, oh, are you Magna Arsuma? And I wasn't even getting honors

Jay McClelland (2:01:00.940)

based on my grades. They just happened to have thought I was interested enough in ideas to

Jay McClelland (2:01:07.340)

belong to Phi Beta Kappa. So. I mean, would it be fair to say you kind of stumbled around a little

Jay McClelland (2:01:12.780)

bit through accidents of too early morning of classes in physics and so on until you discovered

Jay McClelland (2:01:20.940)

intrinsic motivation, as you mentioned, and then that's it. It hooked you. And then you celebrate

Jay McClelland (2:01:26.380)

the fact that this happens to human beings. Yeah. And what is it that made what I did intrinsically

Jay McClelland (2:01:34.860)

motivating to me? Well, that's interesting and I don't know all the answers to it. And I don't

Jay McClelland (2:01:41.260)

think I want anybody to think that you should be sort of in any way, I don't know, sanctimonious or

Jay McClelland (2:01:52.940)

anything about it. You know, it's like, I really enjoyed doing statistical analysis of data. I

Jay McClelland (2:02:01.100)

really enjoyed running my own experiment, which was what I got a chance to do in the psychology

Jay McClelland (2:02:09.260)

department that chemistry and physics had never, I never imagined that mere mortals would ever do

Jay McClelland (2:02:14.860)

an experiment in those sciences, except one that was in the textbook that you were told to do in

Jay McClelland (2:02:20.220)

lab class. But in psychology, we were already like, even when I was taking psych one, it turned out

Jay McClelland (2:02:26.460)

we had our own rat and we got to, after two set experiments, we got to, okay, do something you

Jay McClelland (2:02:32.140)

think of with your rat. So it's the opportunity to do it myself and to bring together a certain

Jay McClelland (2:02:42.060)

set of things that engaged me intrinsically. And I think it has something to do with why

Jay McClelland (2:02:49.340)

certain people turn out to be profoundly amazing musical geniuses, right? They get immersed in it

Jay McClelland (2:02:59.660)

at an early enough point and it just sort of gets into the fabric. So my little brother had intrinsic

Jay McClelland (2:03:07.020)

motivation for music as we witnessed when he discovered how to put records on the phonograph

Jay McClelland (2:03:15.740)

when he was like 13 months old and recognize which one he wanted to play, not because he could read

Jay McClelland (2:03:21.660)

the labels, because he could sort of see which ones had which scratches, which were the different,

Lex Fridman (2:03:26.780)

you know, oh, that's rapidi espanol. And that's, you know, and, and, and,

Lex Fridman (2:03:31.420)

And he enjoyed that, that connected with him somehow.

Jay McClelland (2:03:33.660)

Yeah. And, and there was something that it fed into and it, you're extremely lucky if you have

Jay McClelland (2:03:40.380)

that and if you can nurture it and can let it grow and let it be, be an important part of your life.

Lex Fridman (2:03:47.420)

Yeah. Those are, those are the two things is like, be attentive enough to,

Jay McClelland (2:03:52.780)

to feel it when it comes, like this is something special. I mean, I don't know. For example,

Jay McClelland (2:03:59.020)

I really like tabular data, like Excel sheets. Like it brings me a deep joy. I don't know how

Jay McClelland (2:04:08.540)

useful that is for anything. That's part of what I'm talking about.

Lex Fridman (2:04:12.220)

Exactly. So there's like a million, not a million, but there's a lot of things

Jay McClelland (2:04:17.180)

like that. For me, you have to hear that for yourself, like be, like realize this is really

Jay McClelland (2:04:23.180)

joyful. But then the other part that you're mentioning, which is the nurture is take time

Lex Fridman (2:04:27.980)

and stay with it, stay with it a while and see where that takes you in life.

Lex Fridman (2:04:33.260)

Yeah. And I think, I think the, the, the motivational engagement results in the

Jay McClelland (2:04:40.060)

immersion that then creates the opportunity to obtain the expertise. So, you know, we could call

Jay McClelland (2:04:47.500)

it the Mozart effect, right? I mean, when I think about Mozart, I think about, you know,

Jay McClelland (2:04:53.340)

the person who was born as the fourth member of the family string quartet, right? And, and they

Jay McClelland (2:05:01.260)

handed him the violin when he was six weeks old. All right, start playing, you know, it's like,

Lex Fridman (2:05:08.220)

and so the, the level of immersion there was, was amazingly profound, but hopefully he also had,

Jay McClelland (2:05:20.220)

you know, some, something, maybe this is where the more sort of the genetic part comes in.

Jay McClelland (2:05:28.300)

Sometimes I think, you know, something in him resonated to the music so that that,

Jay McClelland (2:05:34.860)

the synergy of the combination of that was so powerful. So, so that's what I really considered

Jay McClelland (2:05:40.140)

to be the Mozart effect. It's sort of the, the synergy of something with, with experience that,

Lex Fridman (2:05:47.020)

that then results in the unique flowering of a particular, you know, mind.

Lex Fridman (2:05:51.740)

And so I know my siblings and I are all very different from each other. We've all gone in

Jay McClelland (2:06:01.020)

our own different directions. And, you know, I mentioned my younger brother who was very musical.

Jay McClelland (2:06:07.180)

I had my other younger brother was like this amazing, like intuitive engineer.

Lex Fridman (2:06:11.420)

And my sister, one of my sisters was passionate about, in, you know, water conservation well

Jay McClelland (2:06:23.900)

before it was, you know, such a hugely important issue that it is today. So we all sort of somehow

Jay McClelland (2:06:31.900)

these find a different thing. And I don't, I don't mean to say it isn't tied in with something about,

Jay McClelland (2:06:41.740)

about us biologically, but, but it's also when that happens, where you can find that, then,

Jay McClelland (2:06:47.340)

you know, you can do your thing and you can be excited about it. So people can be excited about

Jay McClelland (2:06:52.140)

fitting people on bicycles, as well as excited about making neural networks, achieve insights

Jay McClelland (2:06:56.780)

into human cognition, right? Yeah. Like for me personally, I've always been excited about

Jay McClelland (2:07:03.260)

love and friendship between humans. And just like the actual experience of it,

Jay McClelland (2:07:10.060)

since I was a child, just observing people around me and also been excited about robots.

Lex Fridman (2:07:16.140)

And there's something in me that thinks I really would love to explore how those two things

Jay McClelland (2:07:21.580)

combine. And it doesn't make any sense. A lot of it is also timing, just to think of your own career

Lex Fridman (2:07:26.940)

and your own life. You found yourself in certain pieces, places that happened to involve some of

Jay McClelland (2:07:33.100)

the greatest thinkers of our time. And so it just worked out that like, you guys developed those

Jay McClelland (2:07:37.820)

ideas. And there may be a lot of other people similar to you, and they were brilliant, and

Jay McClelland (2:07:43.020)

they never found that right connection and place to where they, their ideas could flourish. So

Jay McClelland (2:07:48.460)

it's timing, it's place, it's people. And ultimately the whole ride, you know, it's undirected.

Lex Fridman (2:07:56.460)

Can I ask you about something you mentioned in terms of psychiatry when you were younger?

Jay McClelland (2:08:00.620)

Because I had a similar experience of, you know, reading Freud and Carl Jung and just,

Jay McClelland (2:08:09.580)

you know, those kind of popular psychiatry ideas. And that was a dream for me early on in high

Jay McClelland (2:08:15.420)

school too. Like I hoped to understand the human mind by, somehow psychiatry felt like

Lex Fridman (2:08:24.060)

the right discipline for that. Does that make you sad? That psychiatry is not

Jay McClelland (2:08:31.340)

the mechanism by which you are able to explore the human mind. So for me, I was a little bit

Jay McClelland (2:08:37.500)

disillusioned because of how much prescription medication and biochemistry is involved in the

Jay McClelland (2:08:46.300)

discipline of psychiatry, as opposed to the dream of the Freud like, use the mechanisms of language

Jay McClelland (2:08:53.740)

to explore the human mind. So that was a little disappointing. And that's why I kind of went to

Jay McClelland (2:09:00.540)

computer science and thinking like, maybe you can explore the human mind by trying to build the

Jay McClelland (2:09:04.940)

thing. Yes. I wasn't exposed to the sort of the biomedical slash pharmacological aspects of

Jay McClelland (2:09:14.780)

psychiatry at that point because I dropped out of that whole idea of premed that I never even

Jay McClelland (2:09:22.780)

found out about that until much later. But you're absolutely right. So I was actually a member of the

Jay McClelland (2:09:30.620)

National Advisory Mental Health Council. That is to say the board of scientists who advise the

Jay McClelland (2:09:41.260)

director of the National Institute of Mental Health. And that was around the year 2000. And

Jay McClelland (2:09:47.900)

in fact, at that time, the man who came in as the new director, I had been on this board for a year

Jay McClelland (2:09:56.220)

when he came in, said, okay, schizophrenia is a biological illness. It's a lot like cancer.

Jay McClelland (2:10:08.380)

We've made huge strides in curing cancer. And that's what we're going to do with schizophrenia.

Jay McClelland (2:10:13.100)

We're going to find the medications that are going to cure this disease. And we're not going

Jay McClelland (2:10:18.620)

to listen to anybody's grandmother anymore. And good old behavioral psychology is not something

Jay McClelland (2:10:27.580)

we're going to support any further. And he completely alienated me from the Institute

Lex Fridman (2:10:40.940)

and from all of its prior policies, which had been much more holistic, I think, really at some level.

Lex Fridman (2:10:46.700)

And the other people on the board were like psychiatrists, very biological psychiatrists.

Jay McClelland (2:10:57.100)

It didn't pan out that nothing has changed in our ability to help people with mental illness.

Lex Fridman (2:11:07.020)

And so 20 years later, that particular path was a dead end, as far as I can tell.

Jay McClelland (2:11:14.220)

Well, there's some aspect to, and sorry to romanticize the whole philosophical conversation

Jay McClelland (2:11:20.700)

about the human mind. But to me, psychiatrists, for a time, held the flag of we're the deep thinkers.

Jay McClelland (2:11:29.980)

In the same way that physicists are the deep thinkers about the nature of reality,

Jay McClelland (2:11:34.300)

psychiatrists are the deep thinkers about the nature of the human mind. And I think that flag

Jay McClelland (2:11:38.860)

has been taken from them and carried by people like you. It's like, it's more in the cognitive

Jay McClelland (2:11:44.940)

psychology, especially when you have a foot in the computational view of the world, because you can

Jay McClelland (2:11:50.380)

both build it, you can like, intuit about the functioning of the mind by building little models

Lex Fridman (2:11:56.220)

and be able to see mathematical things and then deploying those models, especially in computers,

Jay McClelland (2:12:00.700)

to say, does this actually work? They do like experiments. And then some combination of

Jay McClelland (2:12:07.180)

neuroscience, where you're starting to actually be able to observe, do certain experiments on

Jay McClelland (2:12:13.500)

human beings and observe how the brain is actually functioning. And there, using intuition, you can

Jay McClelland (2:12:21.260)

start being the philosopher. Like Richard Feynman is the philosopher, cognitive psychologists can

Jay McClelland (2:12:26.940)

become the philosopher, and psychiatrists become much more like doctors. They're like very medical.

Jay McClelland (2:12:32.140)

They help people with medication, biochemistry, and so on. But they are no longer the book writers

Lex Fridman (2:12:39.340)

and the philosophers, which of course I admire. I admire the Richard Feynman ability to do

Lex Fridman (2:12:45.740)

great low level mathematics and physics and the high level philosophy.

Jay McClelland (2:12:52.060)

Yeah, I think it was Fromm and Jung more than Freud that was sort of initially kind of like

Jay McClelland (2:13:00.700)

made me feel like, oh, this is really amazing and interesting and I want to explore it further.

Jay McClelland (2:13:06.620)

I actually, when I got to college and I lost that thread, I found more of it in sociology

Lex Fridman (2:13:15.180)

and literature than I did in any place else. So I took quite a lot of both of those

Jay McClelland (2:13:23.660)

disciplines as an undergraduate. And I was actually deeply ambivalent about

Jay McClelland (2:13:32.860)

the psychology because I was doing experiments after the initial flurry of interest in

Lex Fridman (2:13:40.140)

why people would occupy buildings during an insurrection and consider

Jay McClelland (2:13:44.860)

being so overcommitted to their beliefs. But I ended up in the psychology laboratory running

Jay McClelland (2:13:55.100)

experiments on pigeons. And so I had these profound dissonance between the kinds of issues

Jay McClelland (2:14:03.580)

that would be explored when I was thinking about what I read about in modern British literature

Jay McClelland (2:14:12.060)

versus what I could study with my pigeons in the laboratory. That got resolved when I went

Jay McClelland (2:14:18.700)

to graduate school and I discovered cognitive psychology. And so for me, that was the path

Jay McClelland (2:14:25.340)

out of this sort of like extremely sort of ambivalent divergence between the interest

Jay McClelland (2:14:31.900)

in the human condition and the desire to do actual mechanistically oriented thinking about it. And I

Jay McClelland (2:14:42.700)

think we've come a long way in that regard and that you're absolutely right that nowadays this

Jay McClelland (2:14:50.620)

is something that's accessible to people through the pathway in through computer science or the

Jay McClelland (2:14:57.900)

pathway in through neuroscience. You can get derailed in neuroscience down to the bottom of

Jay McClelland (2:15:08.620)

the system where you might find the cures of various conditions, but you don't get a chance

Jay McClelland (2:15:16.300)

to think about the higher level stuff. So it's in the systems and cognitive neuroscience and

Jay McClelland (2:15:21.100)

computational intelligence, miasma up there at the top that I think these opportunities are most

Jay McClelland (2:15:28.460)

are richest right now. And so yes, I am indeed blessed by having had the opportunity to fall

Jay McClelland (2:15:36.060)

into that space. So you mentioned the human condition, speaking which you happen to be a

Jay McClelland (2:15:44.060)

human being who's unfortunately not immortal. That seems to be a fundamental part of the human

Jay McClelland (2:15:52.140)

condition that this ride ends. Do you think about the fact that you're going to die one day? Are you

Jay McClelland (2:16:00.220)

afraid of death? I would say that I am not as much afraid of death as I am of degeneration. And

Jay McClelland (2:16:15.260)

I say that in part for reasons of having, you know, seen some tragic degenerative situations

Jay McClelland (2:16:24.300)

unfold. It's exciting when you can continue to participate and feel like you're near the place

Jay McClelland (2:16:42.140)

where the wave is breaking on the shore, if you like. And I think about my own future potential.

Jay McClelland (2:16:58.460)

If I were to begin to suffer from Alzheimer's disease or semantic dementia or some other

Jay McClelland (2:17:07.260)

condition, you know, I would sort of gradually lose the thread of that ability. And so one can

Jay McClelland (2:17:17.500)

live on for a decade after, you know, sort of having to retire because one no longer has

Jay McClelland (2:17:28.540)

these kinds of abilities to engage. And I think that's the thing that I fear the most.

Jay McClelland (2:17:34.860)

SL. The losing of that, like the breaking of the wave, the flourishing of the mind,

Jay McClelland (2:17:42.220)

where you have these ideas and they're swimming around and you're able to play with them.

Jay McClelland (2:17:46.220)

RL. Yeah. And collaborate with other people who, you know, are themselves

Lex Fridman (2:17:54.140)

really helping to push these ideas forward. So, yeah.

Jay McClelland (2:17:58.540)

SL. What about the edge of the cliff? The end? I mean, the mystery of it. I mean...

Jay McClelland (2:18:05.260)

RL. The migrated sort of conception of mind and, you know, sort of continuous sort of way of

Jay McClelland (2:18:12.780)

thinking about most things makes it so that, to me, the discreteness of that transition is less

Lex Fridman (2:18:25.020)

apparent than it seems to be to most people.

Jay McClelland (2:18:27.100)

SL. I see. I see. Yeah. Yeah. I wonder, so I don't know if you know the work of Ernest Becker

Lex Fridman (2:18:35.180)

and so on. I wonder what role mortality and our ability to be cognizant of it

Lex Fridman (2:18:42.060)

and anticipate it and perhaps be afraid of it, what role that plays in our reasoning of the world.

Jay McClelland (2:18:49.500)

RL. I think that it can be motivating to people to think they have a limited period left.

Jay McClelland (2:18:55.020)

SL. I think in my own case, you know, it's like seven or eight years ago now that I was

Lex Fridman (2:19:03.580)

sitting around doing experiments on decision making that were

Jay McClelland (2:19:11.660)

satisfying in a certain way because I could really get closure on whether the model fit the data

Jay McClelland (2:19:19.740)

perfectly or not. And I could see how one could test, you know, the predictions in monkeys as well

Jay McClelland (2:19:26.940)

as humans and really see what the neurons were doing. But I just realized, hey, wait a minute,

Jay McClelland (2:19:33.580)

you know, I may only have about 10 or 15 years left here. And I don't feel like I'm getting

Jay McClelland (2:19:40.220)

towards the answers to the really interesting questions while I'm doing this particular level

Jay McClelland (2:19:46.060)

of work. And that's when I said to myself, okay, let's pick something that's hard. So that's when

Jay McClelland (2:19:56.220)

I started working on mathematical cognition. And I think it was more in terms of, well,

Jay McClelland (2:20:03.420)

I got 15 more years possibly of useful life left. Let's imagine that it's only 10.

Jay McClelland (2:20:09.980)

I'm actually getting close to the end of that now, maybe three or four more years.

Lex Fridman (2:20:13.260)

But I'm beginning to feel like, well, I probably have another five after that. So, okay, I'll give

Jay McClelland (2:20:17.900)

myself another six or eight. But a deadline is looming and therefore. It's not going to go on

Jay McClelland (2:20:23.500)

forever. And so, yeah, I got to keep thinking about the questions that I think are the interesting and

Jay McClelland (2:20:31.500)

important ones for sure. What do you hope your legacy is? You've done some incredible work in

Jay McClelland (2:20:37.980)

your life as a man, as a scientist, when the aliens and the human civilization is long gone

Lex Fridman (2:20:46.140)

and the aliens are reading the encyclopedia about the human species. What do you hope is the

Lex Fridman (2:20:51.580)

paragraph written about you? I would want it to sort of highlight

Jay McClelland (2:20:56.780)

a couple things that I was able to see one path that was more exciting to me than the one that

Jay McClelland (2:21:20.540)

seemed already to be there for a cognitive psychologist, but not for any super special

Jay McClelland (2:21:28.860)

reason other than that I'd had the right context prior to that, but that I had gone ahead and

Jay McClelland (2:21:34.540)

followed that lead. And then I forget the exact wording, but I said in this preface that

Jay McClelland (2:21:44.220)

the joy of science is the moment in which a partially formed thought in the mind of one person

Lex Fridman (2:22:01.500)

gets crystallized a little better in the discourse and becomes the foundation

Jay McClelland (2:22:08.540)

of some exciting concrete piece of actual scientific progress. And I feel like that

Jay McClelland (2:22:16.220)

moment happened when Rumelhart and I were doing the interactive activation model and when

Jay McClelland (2:22:21.740)

Rumelhart heard Hinton talk about gradient descent and having the objective function to guide the

Jay McClelland (2:22:29.500)

learning process. And it happened a lot in that period and I sort of seek that kind of

Jay McClelland (2:22:37.980)

thing in my collaborations with my students. So the idea that this is a person who contributed

Jay McClelland (2:22:49.660)

to science by finding exciting collaborative opportunities to engage with other people

Jay McClelland (2:22:55.100)

through is something that I certainly hope is part of the paragraph.

Lex Fridman (2:22:59.740)

And like you said, taking a step maybe in directions that are non obvious. So it's the

Jay McClelland (2:23:08.620)

old Robert Frost road less taken. So maybe because you said like this incomplete initial idea,

Lex Fridman (2:23:16.860)

that step you take is a little bit off the beaten path.

Jay McClelland (2:23:22.140)

If I could just say one more thing here. This was something that really contributed

Jay McClelland (2:23:28.940)

to energizing me in a way that I feel it would be useful to share. My PhD dissertation project

Jay McClelland (2:23:40.060)

was completely empirical experimental project. And I wrote a paper based on the two main

Jay McClelland (2:23:48.460)

experiments that were the core of my dissertation and I submitted it to a journal. And at the end

Jay McClelland (2:23:55.020)

of the paper, I had a little section where I laid out the beginnings of my theory about what I

Jay McClelland (2:24:05.900)

thought was going on that would explain the data that I had collected. And I had submitted the

Jay McClelland (2:24:13.580)

paper to the Journal of Experimental Psychology. So I got back a letter from the editor saying,

Jay McClelland (2:24:20.540)

thank you very much. These are great experiments and we'd love to publish them in the journal.

Lex Fridman (2:24:23.980)

But what we'd like you to do is to leave the theorizing to the theorists and take that part

Jay McClelland (2:24:30.940)

out of the paper. And so I did, I took that part out of the paper. But I almost found myself labeled

Jay McClelland (2:24:42.300)

as a non theorist by this. And I could have succumbed to that and said, okay, well, I guess

Jay McClelland (2:24:50.540)

my job is to just go on and do experiments, right? But that's not what I wanted to do. And so when I

Jay McClelland (2:25:01.500)

got to my assistant professorship, although I continued to do experiments because I knew I had

Jay McClelland (2:25:07.340)

to get some papers out, I also at the end of my first year submitted my first article to

Jay McClelland (2:25:13.740)

Psychological Review, which was the theoretical journal where I took that section and elaborated

Jay McClelland (2:25:18.780)

it and wrote it up and submitted it to them. And they didn't accept that either, but they said,

Jay McClelland (2:25:24.940)

oh, this is interesting. You should keep thinking about it this time. And then that was what got me

Jay McClelland (2:25:29.900)

going to think, okay, you know, so it's not a superhuman thing to contribute to the development

Jay McClelland (2:25:37.500)

of theory. You know, you don't have to be, you can do it as a mere mortal.

Jay McClelland (2:25:43.580)

LB And the broader, I think, lesson is don't succumb to the labels of a particular reviewer.

Lex Fridman (2:25:50.540)

RL Yeah, that's for sure. Or anybody labeling you, right?

Jay McClelland (2:25:55.500)

LB Yeah, exactly. I mean that, yeah, exactly. And especially as you become successful,

Jay McClelland (2:26:01.580)

your labels get assigned to you for that you're successful for that thing.

Lex Fridman (2:26:05.820)

RL Connectionist or cognitive scientist and not a neuroscientist.

Jay McClelland (2:26:09.740)

LB And then you can, you can completely, that's just, that's the stories of the past. You're

Jay McClelland (2:26:15.260)

today a new person that can completely revolutionize in totally new areas. So don't

Jay McClelland (2:26:20.940)

let those labels hold you back. Well, let me ask the big question. When you look at into the,

Jay McClelland (2:26:29.980)

you said it started with Columbia trying to observe these humans and they're doing

Jay McClelland (2:26:34.140)

weird stuff and you want to know why are they doing this stuff. So Zuma even bigger.

Jay McClelland (2:26:38.940)

LB At the hundred plus billion people who've ever lived on earth. Why do you think we're all

Jay McClelland (2:26:47.740)

doing what we're doing? What do you think is the meaning of it all? The big why question.

Jay McClelland (2:26:51.500)

We seem to be very busy doing a bunch of stuff and we seem to be kind of directed towards somewhere.

Lex Fridman (2:26:59.420)

But why?

Jay McClelland (2:27:00.060)

RL Well, I myself think that we make meaning for ourselves and that we find inspiration

Jay McClelland (2:27:13.420)

in the meaning that other people have made in the past. You know, and the great religious thinkers

Jay McClelland (2:27:21.100)

of the first millennium BC and, you know, few that came in the early part of the second millennium,

Jay McClelland (2:27:36.620)

you know, laid down some important foundations for us.

Lex Fridman (2:27:40.460)

But I do believe that, you know, we are an emergent result of a process that happened

Jay McClelland (2:27:54.220)

naturally without guidance and that meaning is what we make of it and that the creation of

Jay McClelland (2:28:05.340)

efforts to reify meaning in like religious traditions and so on is just a part of the

Jay McClelland (2:28:15.260)

expression of that goal that we have to, you know, not find out what the meaning is, but to

Jay McClelland (2:28:26.700)

make it ourselves. And so, to me, it's something that's very personal. It's very individual. It's

Jay McClelland (2:28:40.460)

like meaning will come for you through the particular combination of synergistic elements

Jay McClelland (2:28:50.380)

that are your fabric and your experience and your context and, you know, you should...

Jay McClelland (2:29:04.620)

It's all made in a certain kind of a local context though, right? Here I am at UCSD with this brilliant

Jay McClelland (2:29:12.700)

man, Rommelhart, who's having, you know, these doubts about symbolic artificial intelligence

Jay McClelland (2:29:24.780)

that resonate with my desire to see it grounded in the biology and let's make the most of that,

Jay McClelland (2:29:35.020)

you know? Yeah. And so, from that like little pocket, there's some kind of peculiar little

Jay McClelland (2:29:41.580)

emergent process that then, which is basically each one of us, each one of us humans is a kind of,

Jay McClelland (2:29:49.580)

you know, you think cells and they come together and it's an emergent process that then tells fancy

Jay McClelland (2:29:56.380)

stories about itself and then gets, just like you said, just enjoys the beauty of the stories

Jay McClelland (2:30:03.340)

we tell about ourselves. It's an emergent process that lives for a time, is defined by its local

Jay McClelland (2:30:10.300)

pocket and context in time and space and then tells pretty stories and we write those stories

Jay McClelland (2:30:16.620)

down and then we celebrate how nice the stories are and then it continues because we build stories

Jay McClelland (2:30:21.660)

on top of each other and eventually we'll colonize hopefully other planets, other solar systems,

Jay McClelland (2:30:30.540)

other galaxies and we'll tell even better stories. But it all starts here on Earth. Jay, you're

Jay McClelland (2:30:37.740)

speaking of peculiar emergent processes that lived one heck of a story. You're one of the

Jay McClelland (2:30:47.740)

the great scientists of cognitive science, of psychology, of computation. It's a huge honor

Jay McClelland (2:30:58.460)

you would talk to me today that you spend your very valuable time. I really enjoyed talking with

Jay McClelland (2:31:03.340)

you and thank you for all the work you've done. I can't wait to see what you do next.

Jay McClelland (2:31:06.460)

JL Well, thank you so much and this has been an amazing opportunity for me to let ideas that I've

Jay McClelland (2:31:13.580)

never fully expressed before come out because you asked such a wide range of the deeper questions

Jay McClelland (2:31:20.620)

that we've all been thinking about for so long. So thank you very much for that.

Lex Fridman (2:31:24.700)

RL Thank you. Thanks for listening to this conversation with Jay McClelland.

Jay McClelland (2:31:29.420)

To support this podcast, please check out our sponsors in the description.

Lex Fridman (2:31:32.940)

And now, let me leave you with some words from Jeffrey Hinton. In the long run,

Jay McClelland (2:31:37.980)

curiosity driven research works best. Real breakthroughs come from people focusing

Jay McClelland (2:31:43.260)

on what they're excited about. Thanks for listening and hope to see you next time.

Jay McClelland (30:01.180)

I forgot the name already, but it's

Lex Fridman (30:02.540)

Parallel and Something Associative Memory and so on.

Jay McClelland (30:05.860)

Very exciting, technical and exciting title.

Lex Fridman (30:08.700)

And you started talking about Dave Romerhart.

Lex Fridman (30:11.660)

So who is this person that was so,

Lex Fridman (30:15.140)

you've spoken very highly of him.

Lex Fridman (30:17.220)

Can you tell me about him, his ideas, his mind, who he was

Lex Fridman (30:22.300)

as a human being, as a scientist?

Lex Fridman (30:24.940)

So Dave came from a little tiny town in Western South Dakota.

Lex Fridman (30:31.780)

And his mother was the librarian,

Lex Fridman (30:35.820)

and his father was the editor of the newspaper.

Lex Fridman (30:41.180)

And I know one of his brothers pretty well.

Jay McClelland (30:46.020)

They grew up, there were four brothers,

Lex Fridman (30:49.540)

and they grew up together.

Lex Fridman (30:53.620)

And their father encouraged them to compete with each other

Lex Fridman (30:56.660)

a lot.

Jay McClelland (30:58.420)

They competed in sports, and they competed in mind games.

Lex Fridman (31:04.580)

I don't know, things like Sudoku and chess and various things

Jay McClelland (31:07.860)

like that.

Lex Fridman (31:08.740)

And Dave was a standout undergraduate.

Jay McClelland (31:16.380)

He went at a younger age than most people

Lex Fridman (31:20.260)

do to college at the University of South Dakota

Lex Fridman (31:23.220)

and majored in mathematics.

Lex Fridman (31:24.820)

And I don't know how he got interested in psychology,

Lex Fridman (31:30.140)

but he applied to the mathematical psychology

Lex Fridman (31:33.940)

program at Stanford and was accepted as a PhD student

Jay McClelland (31:37.740)

to study mathematical psychology at Stanford.

Lex Fridman (31:40.340)

So mathematical psychology is the use of mathematics

Jay McClelland (31:46.620)

to model mental processes.

Lex Fridman (31:50.620)

So something that I think these days

Jay McClelland (31:52.620)

might be called cognitive modeling, that whole space.

Lex Fridman (31:55.300)

Yeah, it's mathematical in the sense

Jay McClelland (31:57.940)

that you say, if this is true and that is true,

Lex Fridman (32:05.580)

then I can derive that this should follow.

Lex Fridman (32:08.220)

And so you say, these are my stipulations

Lex Fridman (32:10.300)

about the fundamental principles,

Lex Fridman (32:12.260)

and this is my prediction about behavior.

Lex Fridman (32:15.180)

And it's all done with equations.

Jay McClelland (32:16.780)

It's not done with a computer simulation.

Lex Fridman (32:19.860)

So you solve the equation, and that tells you

Lex Fridman (32:23.220)

what the probability that the subject

Lex Fridman (32:26.620)

will be correct on the seventh trial or the experiment is

Jay McClelland (32:29.380)

or something like that.

Lex Fridman (32:30.540)

So it's a use of mathematics to descriptively characterize

Jay McClelland (32:37.620)

aspects of behavior.

Lex Fridman (32:39.940)

And Stanford at that time was the place

Jay McClelland (32:43.300)

where there were several really, really strong

Lex Fridman (32:48.700)

mathematical thinkers who were also connected with three

Jay McClelland (32:51.500)

or four others around the country who brought

Lex Fridman (32:55.540)

a lot of really exciting ideas onto the table.

Lex Fridman (32:59.220)

And it was a very, very prestigious part

Lex Fridman (33:02.860)

of the field of psychology at that time.

Lex Fridman (33:05.060)

So Rummelhart comes into this.

Lex Fridman (33:08.500)

He was a very strong student within that program.

Lex Fridman (33:13.420)

And he got this job at this brand new university

Lex Fridman (33:19.140)

in San Diego in 1967, where he's one of the first assistant

Jay McClelland (33:24.900)

professors in the Department of Psychology at UCSD.

Lex Fridman (33:30.220)

So I got there in 74, seven years later,

Lex Fridman (33:37.460)

and Rummelhart at that time was still

Lex Fridman (33:43.700)

doing mathematical modeling.

Lex Fridman (33:48.740)

But he had gotten interested in cognition.

Lex Fridman (33:53.180)

He'd gotten interested in understanding.

Lex Fridman (33:58.780)

And understanding, I think, remains,

Lex Fridman (34:04.180)

what does it mean to understand anyway?

Jay McClelland (34:08.260)

It's an interesting sort of curious,

Lex Fridman (34:11.220)

how would we know if we really understood something?

Lex Fridman (34:14.180)

But he was interested in building machines

Lex Fridman (34:18.780)

that would hear a couple of sentences

Lex Fridman (34:21.540)

and have an insight about what was going on.

Lex Fridman (34:23.700)

So for example, one of his favorite things at that time

Jay McClelland (34:26.700)

was, Margie was sitting on the front step

Lex Fridman (34:32.780)

when she heard the familiar jingle of the good humor man.

Jay McClelland (34:38.340)

She remembered her birthday money and ran into the house.

Lex Fridman (34:42.060)

What is Margie doing?

Lex Fridman (34:44.740)

Why?

Lex Fridman (34:47.180)

Well, there's a couple of ideas you could have,

Lex Fridman (34:50.140)

but the most natural one is that the good humor

Lex Fridman (34:53.940)

man brings ice cream.

Jay McClelland (34:55.220)

She likes ice cream.

Lex Fridman (34:57.340)

She knows she needs money to buy ice cream,

Lex Fridman (34:59.940)

so she's going to run into the house and get her money

Lex Fridman (35:02.100)

so she can buy herself an ice cream.

Jay McClelland (35:03.900)

It's a huge amount of inference that

Lex Fridman (35:05.420)

has to happen to get those things to link up

Jay McClelland (35:07.500)

with each other.

Lex Fridman (35:09.500)

And he was interested in how the hell that could happen.

Lex Fridman (35:13.100)

And he was trying to build good old fashioned AI style

Lex Fridman (35:20.620)

models of representation of language and content of things

Jay McClelland (35:30.020)

like has money.

Lex Fridman (35:32.300)

So like formal logic and knowledge bases,

Jay McClelland (35:35.420)

like that kind of stuff.

Lex Fridman (35:36.740)

So he was integrating that with his thinking about cognition.

Jay McClelland (35:40.580)

The mechanisms of cognition, how can they mechanistically

Lex Fridman (35:45.100)

be applied to build these knowledge,

Jay McClelland (35:46.860)

like to actually build something that

Lex Fridman (35:49.860)

looks like a web of knowledge and thereby from there emerges

Jay McClelland (35:54.940)

something like understanding, whatever the heck that is.

Lex Fridman (35:57.740)

Yeah, he was grappling.

Jay McClelland (35:59.940)

This was something that they grappled

Lex Fridman (36:01.700)

with at the end of that book that I was describing,

Jay McClelland (36:04.260)

Explorations in Cognition.

Lex Fridman (36:06.380)

But he was realizing that the paradigm of good old fashioned

Jay McClelland (36:11.220)

AI wasn't giving him the answers to these questions.

Lex Fridman (36:16.140)

By the way, that's called good old fashioned AI now.

Jay McClelland (36:18.700)

It wasn't called that at the time.

Lex Fridman (36:20.540)

Well, it was.

Jay McClelland (36:21.380)

It was beginning to be called that.

Lex Fridman (36:23.180)

Oh, because it was from the 60s.

Jay McClelland (36:24.780)

Yeah, yeah.

Lex Fridman (36:26.380)

By the late 70s, it was kind of old fashioned,

Lex Fridman (36:28.980)

and it hadn't really panned out.

Lex Fridman (36:30.820)

And people were beginning to recognize that.

Lex Fridman (36:34.300)

And Rommelhardt was like, yeah, he's part of the recognition

Lex Fridman (36:37.940)

that this wasn't all working.

Jay McClelland (36:39.580)

Anyway, so he started thinking in terms of the idea

Lex Fridman (36:48.860)

that we needed systems that allowed us to integrate

Jay McClelland (36:52.260)

multiple simultaneous constraints in a way that would

Lex Fridman (36:56.180)

be mutually influencing each other.

Lex Fridman (37:00.100)

So he wrote a paper that just really, first time I read it,

Lex Fridman (37:07.980)

I said, oh, well, yeah, but is this important?

Lex Fridman (37:11.940)

But after a while, it just got under my skin.

Lex Fridman (37:15.180)

And it was called An Interactive Model of Reading.

Lex Fridman (37:18.340)

And in this paper, he laid out the idea

Lex Fridman (37:21.660)

that every aspect of our interpretation of what's

Jay McClelland (37:34.700)

coming off the page when we read at every level of analysis

Lex Fridman (37:40.180)

you can think of actually depends

Jay McClelland (37:42.700)

on all the other levels of analysis.

Lex Fridman (37:45.980)

So what are the actual pixels making up each letter?

Lex Fridman (37:53.940)

And what do those pixels signify about which letters they are?

Lex Fridman (38:00.300)

And what do those letters tell us about what words are there?

Lex Fridman (38:05.540)

And what do those words tell us about what ideas

Lex Fridman (38:09.940)

the author is trying to convey?

Lex Fridman (38:12.540)

And so he had this model where we

Lex Fridman (38:18.860)

have these little tiny elements that represent

Jay McClelland (38:25.940)

each of the pixels of each of the letters,

Lex Fridman (38:29.580)

and then other ones that represent the line segments

Jay McClelland (38:31.780)

in them, and other ones that represent the letters,

Lex Fridman (38:33.900)

and other ones that represent the words.

Lex Fridman (38:36.340)

And at that time, his idea was there's this set of experts.

Lex Fridman (38:43.100)

There's an expert about how to construct a line out of pixels,

Lex Fridman (38:48.420)

and another expert about which sets of lines

Lex Fridman (38:51.700)

go together to make which letters,

Lex Fridman (38:53.260)

and another one about which letters go together

Lex Fridman (38:55.340)

to make which words, and another one about what

Jay McClelland (38:58.020)

the meanings of the words are, and another one about how

Lex Fridman (39:01.460)

the meanings fit together, and things like that.

Lex Fridman (39:04.140)

And all these experts are looking at this data,

Lex Fridman (39:06.220)

and they're updating hypotheses at other levels.

Lex Fridman (39:12.740)

So the word expert can tell the letter expert,

Lex Fridman (39:15.580)

oh, I think there should be a T there,

Jay McClelland (39:17.220)

because I think there should be a word the here.

Lex Fridman (39:20.780)

And the bottom up sort of feature to letter expert

Jay McClelland (39:23.580)

could say, I think there should be a T there, too.

Lex Fridman (39:25.660)

And if they agree, then you see a T, right?

Lex Fridman (39:28.700)

And so there's a top down, bottom up interactive process,

Lex Fridman (39:32.540)

but it's going on at all layers simultaneously.

Lex Fridman (39:34.820)

So everything can filter all the way down from the top,

Lex Fridman (39:37.140)

as well as all the way up from the bottom.

Lex Fridman (39:39.180)

And it's a completely interactive, bidirectional,

Lex Fridman (39:42.700)

parallel distributed process.

Jay McClelland (39:45.180)

That is somehow, because of the abstractions, it's hierarchical.

Lex Fridman (39:48.980)

So there's different layers of responsibilities,

Jay McClelland (39:52.780)

different levels of responsibilities.

Lex Fridman (39:54.700)

First of all, it's fascinating to think about it

Jay McClelland (39:56.620)

in this kind of mechanistic way.

Lex Fridman (39:58.460)

So not thinking purely from the structure

Jay McClelland (40:02.100)

of a neural network or something like a neural network,

Lex Fridman (40:04.980)

but thinking about these little guys

Jay McClelland (40:06.860)

that work on letters, and then the letters come words

Lex Fridman (40:09.860)

and words become sentences.

Lex Fridman (40:11.620)

And that's a very interesting hypothesis

Lex Fridman (40:14.780)

that from that kind of hierarchical structure

Jay McClelland (40:18.420)

can emerge understanding.

Lex Fridman (40:21.580)

Yeah, so, but the thing is, though,

Jay McClelland (40:23.300)

I wanna just sort of relate this

Lex Fridman (40:25.700)

to the earlier part of the conversation.

Jay McClelland (40:28.980)

When Romelhart was first thinking about it,

Lex Fridman (40:31.220)

there were these experts on the side,

Jay McClelland (40:34.620)

one for the features and one for the letters

Lex Fridman (40:36.860)

and one for how the letters make the words and so on.

Lex Fridman (40:39.900)

And they would each be working,

Lex Fridman (40:43.060)

sort of evaluating various propositions about,

Jay McClelland (40:46.580)

you know, is this combination of features here

Lex Fridman (40:48.980)

going to be one that looks like the letter T and so on.

Lex Fridman (40:52.620)

And what he realized,

Lex Fridman (40:56.700)

kind of after reading Hinton's dissertation

Lex Fridman (40:59.380)

and hearing about Jim Anderson's

Lex Fridman (41:03.700)

linear algebra based neural network models

Jay McClelland (41:06.060)

that I was telling you about before

Lex Fridman (41:07.620)

was that he could replace those experts

Jay McClelland (41:10.780)

with neuron like processing units,

Lex Fridman (41:12.660)

which just would have their connection weights

Jay McClelland (41:14.700)

that would do this job.

Lex Fridman (41:16.500)

So what ended up happening was

Jay McClelland (41:20.340)

that Romelhart and I got together

Lex Fridman (41:22.260)

and we created a model

Jay McClelland (41:24.100)

called the interactive activation model of letter perception,

Lex Fridman (41:29.020)

which takes these little pixel level inputs,

Jay McClelland (41:35.980)

constructs line segment features, letters and words.

Lex Fridman (41:41.860)

But now we built it out of a set of neuron

Jay McClelland (41:44.780)

like processing units that are just connected

Lex Fridman (41:47.100)

to each other with connection weights.

Lex Fridman (41:49.540)

So the unit for the word time has a connection

Lex Fridman (41:53.060)

to the unit for the letter T in the first position

Lex Fridman (41:56.180)

and the letter I in the second position, so on.

Lex Fridman (41:59.940)

And because these connections are bi directional,

Jay McClelland (42:05.820)

if you have prior knowledge that it might be the word time

Lex Fridman (42:08.820)

that starts to prime the letters and the features.

Lex Fridman (42:12.020)

And if you don't, then it has to start bottom up.

Lex Fridman (42:14.980)

But the directionality just depends

Jay McClelland (42:17.380)

on where the information comes in first.

Lex Fridman (42:19.460)

And if you have context together

Jay McClelland (42:22.100)

with features at the same time,

Lex Fridman (42:24.260)

they can convergently result in an emergent perception.

Lex Fridman (42:27.740)

And that was the piece of work that we did together

Lex Fridman (42:35.780)

that sort of got us both completely convinced

Jay McClelland (42:41.260)

that this neural network way of thinking

Lex Fridman (42:44.540)

was going to be able to actually address the questions

Jay McClelland (42:48.460)

that we were interested in as cognitive psychologists.

Lex Fridman (42:50.780)

So the algorithmic side, the optimization side,

Jay McClelland (42:53.140)

those are all details like when you first start the idea

Lex Fridman (42:56.460)

that you can get far with this kind of way of thinking,

Jay McClelland (42:59.420)

that in itself is a profound idea.

Lex Fridman (43:01.420)

So do you like the term connectionism

Lex Fridman (43:05.020)

to describe this kind of set of ideas?

Lex Fridman (43:07.740)

I think it's useful.

Jay McClelland (43:10.100)

It highlights the notion that the knowledge

Lex Fridman (43:15.460)

that the system exploits is in the connections

Lex Fridman (43:19.820)

between the units, right?

Lex Fridman (43:21.340)

There isn't a separate dictionary.

Jay McClelland (43:24.780)

There's just the connections between the units.

Lex Fridman (43:27.980)

So I already sort of laid that on the table

Jay McClelland (43:31.980)

with the connections from the letter units

Lex Fridman (43:34.140)

to the unit for the word time, right?

Jay McClelland (43:36.900)

The unit for the word time isn't a unit for the word time

Lex Fridman (43:40.020)

for any other reason than it's got the connections

Jay McClelland (43:43.180)

to the letters that make up the word time.

Lex Fridman (43:46.020)

Those are the units on the input that excited

Jay McClelland (43:48.340)

when it's excited that it in a sense represents

Lex Fridman (43:52.660)

in the system that there's support for the hypothesis

Jay McClelland (43:57.700)

that the word time is present in the input.

Lex Fridman (44:01.860)

But it's not, the word time isn't written anywhere

Jay McClelland (44:07.420)

inside the bottle, it's only written there

Lex Fridman (44:09.620)

in the picture we drew of the model

Lex Fridman (44:11.780)

to say that's the unit for the word time, right?

Lex Fridman (44:14.900)

And if somebody wants to tell me,

Lex Fridman (44:18.620)

well, how do you spell that word?

Lex Fridman (44:21.100)

You have to use the connections from that out

Jay McClelland (44:24.340)

to then get those letters, for example.

Lex Fridman (44:27.780)

That's such a, that's a counterintuitive idea

Jay McClelland (44:31.580)

where humans want to think in this logic way.

Lex Fridman (44:36.220)

This idea of connectionism, it doesn't, it's weird.

Jay McClelland (44:41.580)

It's weird that this is how it all works.

Lex Fridman (44:43.540)

Yeah, but let's go back to that CNN, right?

Jay McClelland (44:46.140)

That CNN with all those layers of neuron

Lex Fridman (44:48.500)

like processing units that we were talking about before,

Jay McClelland (44:51.540)

it's gonna come out and say, this is a cat, that's a dog,

Lex Fridman (44:55.420)

but it has no idea why it said that.

Jay McClelland (44:57.740)

It's just got all these connections

Lex Fridman (44:59.460)

between all these layers of neurons,

Jay McClelland (45:02.060)

like from the very first layer to the,

Lex Fridman (45:04.740)

you know, like whatever these layers are,

Jay McClelland (45:07.900)

they just get numbered after a while

Lex Fridman (45:09.500)

because they, you know, they somehow further in you go,

Jay McClelland (45:13.660)

the more abstract the features are,

Lex Fridman (45:17.200)

but it's a graded and continuous sort of process

Jay McClelland (45:20.320)

of abstraction anyway.

Lex Fridman (45:21.660)

And, you know, it goes from very local,

Jay McClelland (45:24.420)

very specific to much more sort of global,

Lex Fridman (45:28.860)

but it's still, you know, another sort of pattern

Jay McClelland (45:32.020)

of activation over an array of units.

Lex Fridman (45:33.980)

And then at the output side, it says it's a cat

Jay McClelland (45:36.500)

or it's a dog.

Lex Fridman (45:37.380)

And when I open my eyes and say, oh, that's Lex,

Jay McClelland (45:42.460)

or, oh, you know, there's my own dog

Lex Fridman (45:47.620)

and I recognize my dog,

Jay McClelland (45:50.500)

which is a member of the same species as many other dogs,

Lex Fridman (45:53.060)

but I know this one

Jay McClelland (45:54.940)

because of some slightly unique characteristics.

Lex Fridman (45:57.420)

I don't know how to describe what it is

Jay McClelland (46:00.300)

that makes me know that I'm looking at Lex

Lex Fridman (46:02.500)

or at my particular dog, right?

Jay McClelland (46:04.660)

Or even that I'm looking at a particular brand of car.

Lex Fridman (46:07.660)

Like I can say a few words about it,

Lex Fridman (46:09.420)

but I wrote you a paragraph about the car,

Lex Fridman (46:12.820)

you would have trouble figuring out

Lex Fridman (46:14.180)

which car is he talking about, right?

Lex Fridman (46:16.760)

So the idea that we have propositional knowledge

Jay McClelland (46:19.400)

of what it is that allows us to recognize

Lex Fridman (46:23.340)

that this is an actual instance

Jay McClelland (46:25.300)

of this particular natural kind

Lex Fridman (46:27.740)

has always been something that it never worked, right?

Jay McClelland (46:36.540)

You couldn't ever write down a set of propositions

Lex Fridman (46:38.900)

for visual recognition.

Lex Fridman (46:41.540)

And so in that space, it sort of always seemed very natural

Lex Fridman (46:46.260)

that something more implicit,

Jay McClelland (46:51.540)

you don't have access to what the details

Lex Fridman (46:54.060)

of the computation were in between,

Jay McClelland (46:56.500)

you just get the result.

Lex Fridman (46:58.320)

So that's the other part of connectionism,

Jay McClelland (47:00.100)

you cannot, you don't read the contents of the connections,

Lex Fridman (47:04.020)

the connections only cause outputs to occur

Jay McClelland (47:08.060)

based on inputs.

Lex Fridman (47:09.600)

Yeah, and for us that like final layer

Jay McClelland (47:13.700)

or some particular layer is very important,

Lex Fridman (47:16.580)

the one that tells us that it's our dog

Jay McClelland (47:19.500)

or like it's a cat or a dog,

Lex Fridman (47:22.220)

but each layer is probably equally as important

Jay McClelland (47:25.420)

in the grand scheme of things.

Lex Fridman (47:27.280)

Like there's no reason why the cat versus dog

Jay McClelland (47:30.240)

is more important than the lower level activations,

Lex Fridman (47:33.140)

it doesn't really matter.

Jay McClelland (47:34.060)

I mean, all of it is just this beautiful stacking

Lex Fridman (47:36.820)

on top of each other.

Lex Fridman (47:37.660)

And we humans live in this particular layers,

Lex Fridman (47:40.020)

for us it's useful to survive,

Jay McClelland (47:43.400)

to use those cat versus dog, predator versus prey,

Lex Fridman (47:47.860)

all those kinds of things.

Jay McClelland (47:49.180)

It's fascinating that it's all continuous,

Lex Fridman (47:51.260)

but then you then ask,

Jay McClelland (47:53.700)

the history of artificial intelligence, you ask,

Lex Fridman (47:55.940)

are we able to introspect and convert the very things

Jay McClelland (47:59.420)

that allow us to tell the difference between cat and dog

Lex Fridman (48:02.380)

into a logic, into formal logic?

Jay McClelland (48:05.380)

That's been the dream.

Lex Fridman (48:06.620)

I would say that's still part of the dream of symbolic AI.

Lex Fridman (48:10.460)

And I've recently talked to Doug Lenat who created Psych

Lex Fridman (48:19.340)

and that's a project that lasted for many decades

Lex Fridman (48:23.180)

and still carries a sort of dream in it, right?

Lex Fridman (48:28.900)

But we still don't know the answer, right?

Jay McClelland (48:30.700)

It seems like connectionism is really powerful,

Lex Fridman (48:34.840)

but it also seems like there's this building of knowledge.

Lex Fridman (48:38.740)

And so how do we, how do you square those two?

Lex Fridman (48:41.420)

Like, do you think the connections can contain

Jay McClelland (48:44.180)

the depth of human knowledge and the depth

Lex Fridman (48:46.940)

of what Dave Romahart was thinking about of understanding?

Jay McClelland (48:51.500)

Well, that remains the $64 question.

Lex Fridman (48:55.760)

And I...

Jay McClelland (48:58.040)

With inflation, that number is higher.

Lex Fridman (48:59.840)

Okay, $64,000.

Jay McClelland (49:01.800)

Maybe it's the $64 billion question now.

Lex Fridman (49:08.800)

You know, I think that from the emergentist side,

Jay McClelland (49:13.800)

which, you know, I placed myself on.

Lex Fridman (49:23.760)

So I used to sometimes tell people

Jay McClelland (49:26.040)

I was a radical, eliminative connectionist

Lex Fridman (49:29.660)

because I didn't want them to think

Jay McClelland (49:34.420)

that I wanted to build like anything into the machine.

Lex Fridman (49:38.320)

But I don't like the word eliminative anymore

Jay McClelland (49:45.620)

because it makes it seem like it's wrong to think

Lex Fridman (49:51.060)

that there is this emergent level of understanding.

Lex Fridman (49:55.900)

And I disagree with that.

Lex Fridman (50:00.140)

So I think, you know, I would call myself

Jay McClelland (50:02.300)

an a radical emergentist connectionist

Lex Fridman (50:06.920)

rather than eliminative connectionist, right?

Jay McClelland (50:09.500)

Because I want to acknowledge

Lex Fridman (50:12.540)

that these higher level kinds of aspects

Jay McClelland (50:17.540)

of our cognition are real, but they're not,

Lex Fridman (50:26.700)

they don't exist as such.

Lex Fridman (50:29.020)

And there was an example that Doug Hofstadter used to use

Lex Fridman (50:33.580)

that I thought was helpful in this respect.

Jay McClelland (50:36.700)

Just the idea that we can think about sand dunes

Lex Fridman (50:42.980)

as entities and talk about like how many there are even.

Lex Fridman (50:51.420)

But we also know that a sand dune is a very fluid thing.

Lex Fridman (50:56.820)

It's a pile of sand that is capable

Jay McClelland (51:00.740)

of moving around under the wind and reforming itself

Lex Fridman (51:08.860)

in somewhat different ways.

Lex Fridman (51:10.140)

And if we think about our thoughts as like sand dunes,

Lex Fridman (51:13.040)

as being things that emerge from just the way

Jay McClelland (51:19.380)

all the lower level elements sort of work together

Lex Fridman (51:22.460)

and are constrained by external forces,

Jay McClelland (51:26.980)

then we can say, yes, they exist as such,

Lex Fridman (51:29.680)

but they also, we shouldn't treat them

Jay McClelland (51:34.820)

as completely monolithic entities that we can understand

Lex Fridman (51:40.400)

without understanding sort of all of the stuff

Jay McClelland (51:43.820)

that allows them to change in the ways that they do.

Lex Fridman (51:47.540)

And that's where I think the connectionist

Jay McClelland (51:49.220)

feeds into the cognitive.

Lex Fridman (51:52.220)

It's like, okay, so if the substrate

Jay McClelland (51:55.380)

is parallel distributed connectionist, then it doesn't mean

Lex Fridman (52:01.220)

that the contents of thought isn't like abstract

Lex Fridman (52:05.980)

and symbolic, but it's more fluid maybe

Lex Fridman (52:10.340)

than it's easier to capture

Jay McClelland (52:13.060)

with a set of logical expressions.

Lex Fridman (52:15.420)

Yeah, that's a heck of a sort of thing

Jay McClelland (52:17.740)

to put at the top of a resume,

Lex Fridman (52:20.480)

radical, emergentist, connectionist.

Lex Fridman (52:23.500)

So there is, just like you said, a beautiful dance

Lex Fridman (52:26.940)

between that, between the machinery of intelligence,

Jay McClelland (52:30.380)

like the neural network side of it,

Lex Fridman (52:32.340)

and the stuff that emerges.

Jay McClelland (52:34.340)

I mean, the stuff that emerges seems to be,

Lex Fridman (52:40.900)

I don't know, I don't know what that is,

Jay McClelland (52:44.020)

that it seems like maybe all of reality is emergent.

Lex Fridman (52:48.940)

What I think about, this is made most distinctly rich to me

Jay McClelland (52:57.380)

when I look at cellular automata, look at game of life,

Lex Fridman (53:01.340)

that from very, very simple things,

Jay McClelland (53:03.620)

very rich, complex things emerge

Lex Fridman (53:06.780)

that start looking very quickly like organisms

Jay McClelland (53:10.260)

that you forget how the actual thing operates.

Lex Fridman (53:13.620)

They start looking like they're moving around,

Jay McClelland (53:15.620)

they're eating each other,

Lex Fridman (53:16.500)

some of them are generating offspring.

Jay McClelland (53:20.100)

You forget very quickly.

Lex Fridman (53:21.780)

And it seems like maybe it's something

Jay McClelland (53:23.940)

about the human mind that wants to operate

Lex Fridman (53:26.060)

in some layer of the emergent,

Lex Fridman (53:28.460)

and forget about the mechanism

Lex Fridman (53:30.580)

of how that emergence happens.

Lex Fridman (53:32.220)

So it, just like you are in your radicalness,

Lex Fridman (53:35.560)

I'm also, it seems like unfair

Jay McClelland (53:39.020)

to eliminate the magic of that emergent,

Lex Fridman (53:43.040)

like eliminate the fact that that emergent is real.

Jay McClelland (53:48.280)

Yeah, no, I agree.

Lex Fridman (53:49.860)

I'm not, that's why I got rid of eliminative, right?

Jay McClelland (53:53.220)

Eliminative, yeah.

Lex Fridman (53:54.060)

Yeah, because it seemed like that was trying to say

Jay McClelland (53:56.580)

that it's all completely like.

Lex Fridman (54:01.860)

An illusion of some kind, it's not.

Jay McClelland (54:03.380)

Well, who knows whether there isn't,

Lex Fridman (54:06.180)

there aren't some illusory characteristics there.

Lex Fridman (54:08.620)

And I think that philosophically many people

Lex Fridman (54:15.020)

have confronted that possibility over time,

Lex Fridman (54:17.780)

but it's still important to accept it as magic, right?

Lex Fridman (54:26.300)

So, I think of Fellini in this context,

Jay McClelland (54:30.300)

I think of others who have appreciated the role of magic,

Lex Fridman (54:35.300)

the role of magic, of actual trickery

Jay McClelland (54:39.180)

in creating illusions that move us.

Lex Fridman (54:45.820)

And Plato was on to this too.

Jay McClelland (54:47.380)

It's like somehow or other these shadows

Lex Fridman (54:52.620)

give rise to something much deeper than that.

Lex Fridman (54:55.900)

And that's, so we won't try to figure out what it is.

Lex Fridman (55:01.060)

We'll just accept it as given that that occurs.

Jay McClelland (55:04.140)

And, you know, but he was still onto the magic of it.

Lex Fridman (55:08.660)

Yeah, yeah, we won't try to really, really,

Jay McClelland (55:11.900)

really deeply understand how it works.

Lex Fridman (55:14.220)

We'll just enjoy the fact that it's kind of fun.

Jay McClelland (55:16.700)

Okay, but you worked closely with Dave Romo Hart.

Lex Fridman (55:21.940)

He passed away as a human being.

Lex Fridman (55:24.960)

What do you remember about him?

Lex Fridman (55:27.020)

Do you miss the guy?

Jay McClelland (55:28.060)

Absolutely, you know, he passed away 15ish years ago now.

Lex Fridman (55:38.740)

And his demise was actually one of the most poignant

Jay McClelland (55:43.740)

and, you know, like relevant tragedies, relevant to our conversation.

Lex Fridman (55:52.740)

He started to undergo a progressive neurological condition

Jay McClelland (56:03.740)

that isn't far from what we're used to.

Lex Fridman (56:08.740)

A neurological condition that isn't fully understood.

Jay McClelland (56:15.740)

That is to say his particular course isn't fully understood

Lex Fridman (56:23.740)

because, you know, brain scans weren't done at certain stages

Lex Fridman (56:28.740)

and no autopsy was done or anything like that.

Lex Fridman (56:32.740)

The wishes of the family.

Jay McClelland (56:34.740)

We don't know as much about the underlying pathology as we might,

Lex Fridman (56:38.740)

but I had begun to get interested in this neurological condition

Jay McClelland (56:48.740)

that might have been the very one that he was succumbing to

Lex Fridman (56:52.740)

as my own efforts to understand another aspect of this mystery

Jay McClelland (56:57.740)

that we've been discussing while he was beginning

Lex Fridman (57:01.740)

to get progressively more and more affected.

Lex Fridman (57:04.740)

So I'm going to talk about the disorder

Lex Fridman (57:06.740)

and not about Rumelhart for a second, okay?

Jay McClelland (57:09.740)

The disorder is something my colleagues and collaborators

Lex Fridman (57:12.740)

have chosen to call semantic dementia.

Lex Fridman (57:17.740)

So it's a specific form of loss of mind

Lex Fridman (57:23.740)

related to meaning, semantic dementia.

Lex Fridman (57:27.740)

And it's progressive in the sense that the patient loses the ability

Lex Fridman (57:37.740)

to appreciate the meaning of the experiences that they have,

Jay McClelland (57:44.740)

either from touch, from sight, from sound, from language.

Lex Fridman (57:50.740)

They, I hear sounds, but I don't know what they mean kind of thing.

Lex Fridman (57:56.740)

So as this illness progresses, it starts with the patient

Lex Fridman (58:04.740)

being unable to differentiate like similar breeds of dog

Jay McClelland (58:12.740)

or remember the lower frequency unfamiliar categories

Lex Fridman (58:18.740)

that they used to be able to remember.

Lex Fridman (58:21.740)

But as it progresses, it becomes more and more striking

Lex Fridman (58:27.740)

and the patient loses the ability to recognize things like

Jay McClelland (58:36.740)

pigs and goats and sheep and calls all middle sized animals dogs

Lex Fridman (58:42.740)

and can't recognize rabbits and rodents anymore.

Jay McClelland (58:46.740)

They call all the little ones cats

Lex Fridman (58:49.740)

and they can't recognize hippopotamuses and cows anymore.

Jay McClelland (58:53.740)

They call them all horses.

Lex Fridman (58:55.740)

So there was this one patient who went through this progression

Jay McClelland (59:00.740)

where at a certain point, any four legged animal,

Lex Fridman (59:03.740)

he would call it either a horse or a dog or a cat.

Lex Fridman (59:07.740)

And if it was big, he would tend to call it a horse.

Lex Fridman (59:10.740)

If it was small, he'd tend to call it a cat.

Jay McClelland (59:12.740)

Middle sized ones, he called dogs.

Lex Fridman (59:16.740)

This is just a part of the syndrome though.

Jay McClelland (59:19.740)

The patient loses the ability to relate concepts to each other.

Lex Fridman (59:25.740)

So my collaborator in this work, Carolyn Patterson,

Jay McClelland (59:28.740)

developed a test called the pyramids and palm trees test.

Lex Fridman (59:34.740)

So you give the patient a picture of pyramids

Lex Fridman (59:39.740)

and they have a choice which goes with the pyramids,

Lex Fridman (59:42.740)

palm trees or pine trees.

Lex Fridman (59:46.740)

And she showed that this wasn't just a matter of language

Lex Fridman (59:50.740)

because the patient's loss of this ability shows up

Jay McClelland (59:55.740)

whether you present the material with words or with pictures.

Lex Fridman (59:59.740)

The pictures, they can't put the pictures together