Yann LeCun: Dark Matter of Intelligence and Self-Supervised Learning

Lex Fridman (01:22.760)

number of trial and errors for a system to learn anything.

Lex Fridman (01:29.320)

And that's why we don't have self driving cars.

Lex Fridman (01:32.960)

That was a big leap from one to the other.

Yann LeCun (01:34.760)

Okay, so that, to solve difficult problems,

Lex Fridman (01:38.760)

you have to have a lot of human annotation

Yann LeCun (01:42.360)

for supervised learning to work.

Lex Fridman (01:44.080)

And to solve those difficult problems

Yann LeCun (01:45.520)

with reinforcement learning,

Lex Fridman (01:46.680)

you have to have some way to maybe simulate that problem

Yann LeCun (01:50.240)

such that you can do that large scale kind of learning

Lex Fridman (01:52.720)

that reinforcement learning requires.

Yann LeCun (01:54.420)

Right, so how is it that most teenagers can learn

Lex Fridman (01:58.320)

to drive a car in about 20 hours of practice,

Yann LeCun (02:02.280)

whereas even with millions of hours of simulated practice,

Lex Fridman (02:07.400)

a self driving car can't actually learn

Yann LeCun (02:09.220)

to drive itself properly.

Lex Fridman (02:12.120)

And so obviously we're missing something, right?

Lex Fridman (02:13.920)

And it's quite obvious for a lot of people

Lex Fridman (02:15.600)

that the immediate response you get from many people is,

Yann LeCun (02:19.760)

well, humans use their background knowledge

Lex Fridman (02:22.840)

to learn faster, and they're right.

Lex Fridman (02:25.820)

Now, how was that background knowledge acquired?

Lex Fridman (02:28.280)

And that's the big question.

Lex Fridman (02:30.080)

So now you have to ask, how do babies

Lex Fridman (02:34.040)

in the first few months of life learn how the world works?

Yann LeCun (02:37.120)

Mostly by observation,

Lex Fridman (02:38.240)

because they can hardly act in the world.

Lex Fridman (02:40.960)

And they learn an enormous amount

Lex Fridman (02:42.560)

of background knowledge about the world

Yann LeCun (02:43.840)

that may be the basis of what we call common sense.

Lex Fridman (02:47.960)

This type of learning is not learning a task.

Yann LeCun (02:51.280)

It's not being reinforced for anything.

Lex Fridman (02:53.680)

It's just observing the world and figuring out how it works.

Yann LeCun (02:58.400)

Building world models, learning world models.

Lex Fridman (03:01.240)

How do we do this?

Lex Fridman (03:02.120)

And how do we reproduce this in machines?

Lex Fridman (03:04.560)

So self supervised learning is one instance

Yann LeCun (03:09.520)

or one attempt at trying to reproduce this kind of learning.

Lex Fridman (03:13.120)

Okay, so you're looking at just observation,

Lex Fridman (03:16.400)

so not even the interacting part of a child.

Lex Fridman (03:18.720)

It's just sitting there watching mom and dad walk around,

Yann LeCun (03:21.600)

pick up stuff, all of that.

Lex Fridman (03:23.480)

That's what we mean about background knowledge.

Yann LeCun (03:25.520)

Perhaps not even watching mom and dad,

Lex Fridman (03:27.520)

just watching the world go by.

Yann LeCun (03:30.000)

Just having eyes open or having eyes closed

Lex Fridman (03:31.920)

or the very act of opening and closing eyes

Yann LeCun (03:34.480)

that the world appears and disappears,

Lex Fridman (03:36.280)

all that basic information.

Lex Fridman (03:39.120)

And you're saying in order to learn to drive,

Lex Fridman (03:43.160)

like the reason humans are able to learn to drive quickly,

Yann LeCun (03:45.840)

some faster than others,

Lex Fridman (03:47.360)

is because of the background knowledge.

Yann LeCun (03:48.680)

They're able to watch cars operate in the world

Lex Fridman (03:51.760)

in the many years leading up to it,

Yann LeCun (03:53.640)

the physics of basic objects, all that kind of stuff.

Lex Fridman (03:55.760)

That's right.

Yann LeCun (03:56.600)

I mean, the basic physics of objects,

Lex Fridman (03:57.440)

you don't even need to know how a car works, right?

Yann LeCun (04:00.880)

Because that you can learn fairly quickly.

Lex Fridman (04:02.500)

I mean, the example I use very often

Yann LeCun (04:03.840)

is you're driving next to a cliff.

Lex Fridman (04:06.680)

And you know in advance because of your understanding

Yann LeCun (04:10.560)

of intuitive physics that if you turn the wheel

Lex Fridman (04:13.200)

to the right, the car will veer to the right,

Yann LeCun (04:15.080)

will run off the cliff, fall off the cliff,

Lex Fridman (04:17.560)

and nothing good will come out of this, right?

Lex Fridman (04:20.400)

But if you are a sort of tabularized

Lex Fridman (04:23.760)

reinforcement learning system

Yann LeCun (04:25.100)

that doesn't have a model of the world,

Lex Fridman (04:28.160)

you have to repeat falling off this cliff

Yann LeCun (04:30.500)

thousands of times before you figure out it's a bad idea.

Lex Fridman (04:32.800)

And then a few more thousand times

Yann LeCun (04:34.560)

before you figure out how to not do it.

Lex Fridman (04:36.960)

And then a few more million times

Yann LeCun (04:38.480)

before you figure out how to not do it

Lex Fridman (04:39.800)

in every situation you ever encounter.

Lex Fridman (04:42.520)

So self supervised learning still has to have

Lex Fridman (04:45.800)

some source of truth being told to it by somebody.

Lex Fridman (04:50.560)

So you have to figure out a way without human assistance

Lex Fridman (04:54.560)

or without significant amount of human assistance

Yann LeCun (04:56.600)

to get that truth from the world.

Lex Fridman (04:59.100)

So the mystery there is how much signal is there?

Lex Fridman (05:03.980)

How much truth is there that the world gives you?

Lex Fridman (05:06.280)

Whether it's the human world,

Yann LeCun (05:08.160)

like you watch YouTube or something like that,

Lex Fridman (05:10.020)

or it's the more natural world.

Lex Fridman (05:12.960)

So how much signal is there?

Lex Fridman (05:14.920)

So here's the trick.

Yann LeCun (05:16.280)

There is way more signal in sort of a self supervised

Lex Fridman (05:20.120)

setting than there is in either a supervised

Yann LeCun (05:22.500)

or reinforcement setting.

Lex Fridman (05:24.520)

And this is going to my analogy of the cake.

Yann LeCun (05:30.280)

The cake as someone has called it,

Lex Fridman (05:32.320)

where when you try to figure out how much information

Yann LeCun (05:36.000)

you ask the machine to predict

Lex Fridman (05:37.840)

and how much feedback you give the machine at every trial,

Yann LeCun (05:41.040)

in reinforcement learning,

Lex Fridman (05:41.880)

you give the machine a single scalar.

Yann LeCun (05:43.340)

You tell the machine you did good, you did bad.

Lex Fridman (05:45.400)

And you only tell this to the machine once in a while.

Yann LeCun (05:49.640)

When I say you, it could be the universe

Lex Fridman (05:51.440)

telling the machine, right?

Lex Fridman (05:54.120)

But it's just one scalar.

Lex Fridman (05:55.840)

And so as a consequence,

Yann LeCun (05:57.160)

you cannot possibly learn something very complicated

Lex Fridman (05:59.600)

without many, many, many trials

Yann LeCun (06:01.120)

where you get many, many feedbacks of this type.

Lex Fridman (06:04.760)

Supervised learning, you give a few bits to the machine

Yann LeCun (06:08.880)

at every sample.

Lex Fridman (06:11.280)

Let's say you're training a system on recognizing images

Yann LeCun (06:15.720)

on ImageNet with 1000 categories,

Lex Fridman (06:17.680)

that's a little less than 10 bits of information per sample.

Lex Fridman (06:22.180)

But self supervised learning, here is the setting.

Lex Fridman (06:24.640)

Ideally, we don't know how to do this yet,

Lex Fridman (06:26.360)

but ideally you would show a machine a segment of video

Lex Fridman (06:31.680)

and then stop the video and ask the machine to predict

Yann LeCun (06:34.200)

what's going to happen next.

Lex Fridman (06:37.640)

And so we let the machine predict

Lex Fridman (06:38.720)

and then you let time go by

Lex Fridman (06:41.400)

and show the machine what actually happened

Lex Fridman (06:44.340)

and hope the machine will learn to do a better job

Lex Fridman (06:47.920)

at predicting next time around.

Yann LeCun (06:49.400)

There's a huge amount of information you give the machine

Lex Fridman (06:51.580)

because it's an entire video clip

Yann LeCun (06:54.680)

of the future after the video clip you fed it

Lex Fridman (06:59.280)

in the first place.

Lex Fridman (07:00.280)

So both for language and for vision, there's a subtle,

Lex Fridman (07:05.120)

seemingly trivial construction,

Lex Fridman (07:06.920)

but maybe that's representative

Lex Fridman (07:08.520)

of what is required to create intelligence,

Yann LeCun (07:10.620)

which is filling the gap.

Lex Fridman (07:13.720)

So it sounds dumb, but can you,

Yann LeCun (07:19.760)

it is possible you could solve all of intelligence

Lex Fridman (07:22.080)

in this way, just for both language,

Yann LeCun (07:25.280)

just give a sentence and continue it

Lex Fridman (07:28.800)

or give a sentence and there's a gap in it,

Yann LeCun (07:32.080)

some words blanked out and you fill in what words go there.

Lex Fridman (07:35.720)

For vision, you give a sequence of images

Lex Fridman (07:39.200)

and predict what's going to happen next,

Lex Fridman (07:40.960)

or you fill in what happened in between.

Lex Fridman (07:43.840)

Do you think it's possible that formulation alone

Lex Fridman (07:48.600)

as a signal for self supervised learning

Lex Fridman (07:50.980)

can solve intelligence for vision and language?

Lex Fridman (07:53.640)

I think that's the best shot at the moment.

Lex Fridman (07:56.320)

So whether this will take us all the way

Lex Fridman (07:59.120)

to human level intelligence or something,

Yann LeCun (08:01.760)

or just cat level intelligence is not clear,

Lex Fridman (08:04.840)

but among all the possible approaches

Yann LeCun (08:07.340)

that people have proposed, I think it's our best shot.

Lex Fridman (08:09.520)

So I think this idea of an intelligent system

Yann LeCun (08:14.640)

filling in the blanks, either predicting the future,

Lex Fridman (08:18.880)

inferring the past, filling in missing information,

Yann LeCun (08:23.760)

I'm currently filling the blank

Lex Fridman (08:25.200)

of what is behind your head

Lex Fridman (08:26.680)

and what your head looks like from the back,

Lex Fridman (08:30.600)

because I have basic knowledge about how humans are made.

Lex Fridman (08:33.760)

And I don't know what you're going to say,

Lex Fridman (08:36.360)

at which point you're going to speak,

Yann LeCun (08:37.280)

whether you're going to move your head this way or that way,

Lex Fridman (08:38.960)

which way you're going to look,

Lex Fridman (08:40.280)

but I know you're not going to just dematerialize

Lex Fridman (08:42.080)

and reappear three meters down the hall,

Yann LeCun (08:46.280)

because I know what's possible and what's impossible

Lex Fridman (08:49.520)

according to intuitive physics.

Yann LeCun (08:51.160)

You have a model of what's possible and what's impossible

Lex Fridman (08:53.280)

and then you'd be very surprised if it happens

Lex Fridman (08:55.080)

and then you'll have to reconstruct your model.

Lex Fridman (08:57.840)

Right, so that's the model of the world.

Yann LeCun (08:59.600)

It's what tells you, what fills in the blanks.

Lex Fridman (09:02.240)

So given your partial information about the state

Yann LeCun (09:04.960)

of the world, given by your perception,

Lex Fridman (09:08.080)

your model of the world fills in the missing information

Lex Fridman (09:11.360)

and that includes predicting the future,

Lex Fridman (09:13.760)

re predicting the past, filling in things

Yann LeCun (09:16.880)

you don't immediately perceive.

Lex Fridman (09:18.400)

And that doesn't have to be purely generic vision

Yann LeCun (09:22.280)

or visual information or generic language.

Lex Fridman (09:24.340)

You can go to specifics like predicting

Lex Fridman (09:28.920)

what control decision you make when you're driving

Lex Fridman (09:31.120)

in a lane, you have a sequence of images from a vehicle

Lex Fridman (09:35.620)

and then you have information if you record it on video

Lex Fridman (09:39.640)

where the car ended up going so you can go back in time

Lex Fridman (09:43.680)

and predict where the car went

Lex Fridman (09:45.520)

based on the visual information.

Yann LeCun (09:46.680)

That's very specific, domain specific.

Lex Fridman (09:49.440)

Right, but the question is whether we can come up

Yann LeCun (09:51.480)

with sort of a generic method for training machines

Lex Fridman (09:57.000)

to do this kind of prediction or filling in the blanks.

Lex Fridman (09:59.840)

So right now, this type of approach has been unbelievably

Lex Fridman (10:04.720)

successful in the context of natural language processing.

Yann LeCun (10:08.200)

Every modern natural language processing is pre trained

Lex Fridman (10:10.440)

in self supervised manner to fill in the blanks.

Yann LeCun (10:13.720)

You show it a sequence of words, you remove 10% of them

Lex Fridman (10:16.400)

and then you train some gigantic neural net

Yann LeCun (10:17.940)

to predict the words that are missing.

Lex Fridman (10:20.320)

And once you've pre trained that network,

Yann LeCun (10:22.760)

you can use the internal representation learned by it

Lex Fridman (10:26.600)

as input to something that you train supervised

Yann LeCun (10:30.480)

or whatever.

Lex Fridman (10:32.240)

That's been incredibly successful.

Yann LeCun (10:33.400)

Not so successful in images, although it's making progress

Lex Fridman (10:37.600)

and it's based on sort of manual data augmentation.

Yann LeCun (10:42.600)

We can go into this later,

Lex Fridman (10:43.560)

but what has not been successful yet is training from video.

Lex Fridman (10:47.200)

So getting a machine to learn to represent

Lex Fridman (10:49.440)

the visual world, for example, by just watching video.

Yann LeCun (10:52.800)

Nobody has really succeeded in doing this.

Lex Fridman (10:54.800)

Okay, well, let's kind of give a high level overview.

Yann LeCun (10:57.520)

What's the difference in kind and in difficulty

Lex Fridman (11:02.360)

between vision and language?

Lex Fridman (11:03.960)

So you said people haven't been able to really

Lex Fridman (11:08.280)

kind of crack the problem of vision open

Yann LeCun (11:10.480)

in terms of self supervised learning,

Lex Fridman (11:11.960)

but that may not be necessarily

Yann LeCun (11:13.800)

because it's fundamentally more difficult.

Lex Fridman (11:15.840)

Maybe like when we're talking about achieving,

Yann LeCun (11:18.720)

like passing the Turing test in the full spirit

Lex Fridman (11:22.320)

of the Turing test in language might be harder than vision.

Yann LeCun (11:24.920)

That's not obvious.

Lex Fridman (11:26.400)

So in your view, which is harder

Lex Fridman (11:29.440)

or perhaps are they just the same problem?

Lex Fridman (11:31.960)

When the farther we get to solving each,

Yann LeCun (11:34.840)

the more we realize it's all the same thing.

Lex Fridman (11:36.720)

It's all the same cake.

Yann LeCun (11:37.680)

I think what I'm looking for are methods

Lex Fridman (11:40.200)

that make them look essentially like the same cake,

Lex Fridman (11:43.600)

but currently they're not.

Lex Fridman (11:44.800)

And the main issue with learning world models

Yann LeCun (11:48.480)

or learning predictive models is that the prediction

Lex Fridman (11:53.120)

is never a single thing

Yann LeCun (11:55.880)

because the world is not entirely predictable.

Lex Fridman (11:59.240)

It may be deterministic or stochastic.

Yann LeCun (12:00.680)

We can get into the philosophical discussion about it,

Lex Fridman (12:02.960)

but even if it's deterministic,

Yann LeCun (12:05.280)

it's not entirely predictable.

Lex Fridman (12:07.440)

And so if I play a short video clip

Lex Fridman (12:11.760)

and then I ask you to predict what's going to happen next,

Lex Fridman (12:14.160)

there's many, many plausible continuations

Yann LeCun (12:16.360)

for that video clip and the number of continuation grows

Lex Fridman (12:20.520)

with the interval of time that you're asking the system

Yann LeCun (12:23.920)

to make a prediction for.

Lex Fridman (12:26.480)

And so one big question with self supervised learning

Yann LeCun (12:29.880)

is how you represent this uncertainty,

Lex Fridman (12:32.320)

how you represent multiple discrete outcomes,

Lex Fridman (12:35.200)

how you represent a sort of continuum

Lex Fridman (12:37.120)

of possible outcomes, et cetera.

Lex Fridman (12:40.400)

And if you are sort of a classical machine learning person,

Lex Fridman (12:45.200)

you say, oh, you just represent a distribution, right?

Lex Fridman (12:49.120)

And that we know how to do when we're predicting words,

Lex Fridman (12:52.560)

missing words in the text,

Yann LeCun (12:53.720)

because you can have a neural net give a score

Lex Fridman (12:56.840)

for every word in the dictionary.

Yann LeCun (12:58.640)

It's a big list of numbers, maybe 100,000 or so.

Lex Fridman (13:02.480)

And you can turn them into a probability distribution

Yann LeCun (13:05.280)

that tells you when I say a sentence,

Lex Fridman (13:09.880)

the cat is chasing the blank in the kitchen.

Yann LeCun (13:13.000)

There are only a few words that make sense there.

Lex Fridman (13:15.840)

It could be a mouse or it could be a lizard spot

Lex Fridman (13:18.360)

or something like that, right?

Lex Fridman (13:21.560)

And if I say the blank is chasing the blank in the Savannah,

Yann LeCun (13:25.840)

you also have a bunch of plausible options

Lex Fridman (13:27.840)

for those two words, right?

Yann LeCun (13:30.960)

Because you have kind of a underlying reality

Lex Fridman (13:33.640)

that you can refer to to sort of fill in those blanks.

Lex Fridman (13:38.080)

So you cannot say for sure in the Savannah,

Lex Fridman (13:42.040)

if it's a lion or a cheetah or whatever,

Yann LeCun (13:44.480)

you cannot know if it's a zebra or a do or whatever,

Lex Fridman (13:49.560)

wildebeest, the same thing.

Lex Fridman (13:55.360)

But you can represent the uncertainty

Lex Fridman (13:56.840)

by just a long list of numbers.

Yann LeCun (13:58.520)

Now, if I do the same thing with video,

Lex Fridman (14:01.800)

when I ask you to predict a video clip,

Yann LeCun (14:04.360)

it's not a discrete set of potential frames.

Lex Fridman (14:07.400)

You have to have somewhere representing

Yann LeCun (14:10.000)

a sort of infinite number of plausible continuations

Lex Fridman (14:13.520)

of multiple frames in a high dimensional continuous space.

Lex Fridman (14:17.480)

And we just have no idea how to do this properly.

Lex Fridman (14:20.520)

Fine night, high dimensional.

Lex Fridman (14:22.880)

So like you,

Lex Fridman (14:23.720)

It's finite high dimensional, yes.

Yann LeCun (14:25.320)

Just like the words,

Lex Fridman (14:26.240)

they try to get it down to a small finite set

Yann LeCun (14:32.200)

of like under a million, something like that.

Lex Fridman (14:34.240)

Something like that.

Yann LeCun (14:35.080)

I mean, it's kind of ridiculous that we're doing

Lex Fridman (14:38.320)

a distribution over every single possible word

Yann LeCun (14:40.840)

for language and it works.

Lex Fridman (14:42.880)

It feels like that's a really dumb way to do it.

Yann LeCun (14:46.480)

Like there seems to be like there should be

Lex Fridman (14:49.720)

some more compressed representation

Yann LeCun (14:52.920)

of the distribution of the words.

Lex Fridman (14:55.040)

You're right about that.

Lex Fridman (14:56.120)

And so do you have any interesting ideas

Lex Fridman (14:58.880)

about how to represent all of reality in a compressed way

Lex Fridman (15:01.880)

such that you can form a distribution over it?

Lex Fridman (15:03.800)

That's one of the big questions, how do you do that?

Yann LeCun (15:06.200)

Right, I mean, what's kind of another thing

Lex Fridman (15:08.440)

that really is stupid about, I shouldn't say stupid,

Lex Fridman (15:13.080)

but like simplistic about current approaches

Lex Fridman (15:15.560)

to self supervised learning in NLP in text

Yann LeCun (15:19.360)

is that not only do you represent

Lex Fridman (15:21.920)

a giant distribution over words,

Lex Fridman (15:23.840)

but for multiple words that are missing,

Lex Fridman (15:25.680)

those distributions are essentially independent

Yann LeCun (15:27.680)

of each other.

Lex Fridman (15:30.200)

And you don't pay too much of a price for this.

Lex Fridman (15:33.040)

So you can't, so the system in the sentence

Lex Fridman (15:37.840)

that I gave earlier, if it gives a certain probability

Yann LeCun (15:41.280)

for a lion and cheetah, and then a certain probability

Lex Fridman (15:44.800)

for gazelle, wildebeest and zebra,

Yann LeCun (15:51.960)

those two probabilities are independent of each other.

Lex Fridman (15:55.960)

And it's not the case that those things are independent.

Yann LeCun (15:58.040)

Lions actually attack like bigger animals than cheetahs.

Lex Fridman (16:01.480)

So there's a huge independent hypothesis in this process,

Yann LeCun (16:05.960)

which is not actually true.

Lex Fridman (16:07.800)

The reason for this is that we don't know

Lex Fridman (16:09.880)

how to represent properly distributions

Lex Fridman (16:13.000)

over combinatorial sequences of symbols,

Yann LeCun (16:16.240)

essentially because the number grows exponentially

Lex Fridman (16:19.000)

with the length of the symbols.

Lex Fridman (16:21.320)

And so we have to use tricks for this,

Lex Fridman (16:22.760)

but those techniques can get around,

Yann LeCun (16:26.400)

like don't even deal with it.

Lex Fridman (16:27.800)

So the big question is would there be some sort

Yann LeCun (16:31.760)

of abstract latent representation of text

Lex Fridman (16:35.640)

that would say that when I switch lion for gazelle,

Lex Fridman (16:40.680)

lion for cheetah, I also have to switch zebra for gazelle?

Lex Fridman (16:45.480)

Yeah, so this independence assumption,

Yann LeCun (16:48.720)

let me throw some criticism at you that I often hear

Lex Fridman (16:51.160)

and see how you respond.

Lex Fridman (16:52.920)

So this kind of filling in the blanks is just statistics.

Lex Fridman (16:56.000)

You're not learning anything

Yann LeCun (16:58.880)

like the deep underlying concepts.

Lex Fridman (17:01.600)

You're just mimicking stuff from the past.

Yann LeCun (17:05.640)

You're not learning anything new such that you can use it

Lex Fridman (17:08.560)

to generalize about the world.

Yann LeCun (17:11.960)

Or okay, let me just say the crude version,

Lex Fridman (17:14.120)

which is just statistics.

Yann LeCun (17:16.200)

It's not intelligence.

Lex Fridman (17:18.320)

What do you have to say to that?

Lex Fridman (17:19.640)

What do you usually say to that

Lex Fridman (17:20.880)

if you kind of hear this kind of thing?

Yann LeCun (17:22.640)

I don't get into those discussions

Lex Fridman (17:23.960)

because they are kind of pointless.

Lex Fridman (17:26.760)

So first of all, it's quite possible

Lex Fridman (17:28.760)

that intelligence is just statistics.

Yann LeCun (17:30.480)

It's just statistics of a particular kind.

Lex Fridman (17:32.760)

Yes, this is the philosophical question.

Yann LeCun (17:35.480)

It's kind of is it possible

Lex Fridman (17:38.400)

that intelligence is just statistics?

Lex Fridman (17:40.280)

Yeah, but what kind of statistics?

Lex Fridman (17:43.520)

So if you are asking the question,

Yann LeCun (17:47.160)

are the models of the world that we learn,

Lex Fridman (17:50.680)

do they have some notion of causality?

Yann LeCun (17:52.320)

Yes.

Lex Fridman (17:53.400)

So if the criticism comes from people who say,

Yann LeCun (17:57.200)

current machine learning system don't care about causality,

Lex Fridman (17:59.440)

which by the way is wrong, I agree with them.

Yann LeCun (18:04.600)

Your model of the world should have your actions

Lex Fridman (18:06.560)

as one of the inputs.

Lex Fridman (18:09.080)

And that will drive you to learn causal models of the world

Lex Fridman (18:11.400)

where you know what intervention in the world

Yann LeCun (18:15.080)

will cause what result.

Lex Fridman (18:16.720)

Or you can do this by observation of other agents

Yann LeCun (18:19.400)

acting in the world and observing the effect.

Lex Fridman (18:22.520)

Other humans, for example.

Lex Fridman (18:24.240)

So I think at some level of description,

Lex Fridman (18:28.440)

intelligence is just statistics.

Lex Fridman (18:31.680)

But that doesn't mean you don't have models

Lex Fridman (18:35.200)

that have deep mechanistic explanation for what goes on.

Lex Fridman (18:40.080)

The question is how do you learn them?

Lex Fridman (18:41.760)

That's the question I'm interested in.

Yann LeCun (18:44.440)

Because a lot of people who actually voice their criticism

Lex Fridman (18:49.360)

say that those mechanistic model

Yann LeCun (18:51.040)

have to come from someplace else.

Lex Fridman (18:52.640)

They have to come from human designers,

Yann LeCun (18:54.040)

they have to come from I don't know what.

Lex Fridman (18:56.200)

And obviously we learn them.

Yann LeCun (18:59.280)

Or if we don't learn them as an individual,

Lex Fridman (19:01.800)

nature learn them for us using evolution.

Lex Fridman (19:04.920)

So regardless of what you think,

Lex Fridman (19:07.160)

those processes have been learned somehow.

Lex Fridman (19:10.240)

So if you look at the human brain,

Lex Fridman (19:12.920)

just like when we humans introspect

Yann LeCun (19:14.640)

about how the brain works,

Lex Fridman (19:16.320)

it seems like when we think about what is intelligence,

Yann LeCun (19:20.240)

we think about the high level stuff,

Lex Fridman (19:22.440)

like the models we've constructed,

Yann LeCun (19:23.960)

concepts like cognitive science,

Lex Fridman (19:25.560)

like concepts of memory and reasoning module,

Yann LeCun (19:28.720)

almost like these high level modules.

Lex Fridman (19:32.360)

Is this serve as a good analogy?

Yann LeCun (19:35.400)

Like are we ignoring the dark matter,

Lex Fridman (19:40.720)

the basic low level mechanisms?

Yann LeCun (19:43.560)

Just like we ignore the way the operating system works,

Lex Fridman (19:45.800)

we're just using the high level software.

Yann LeCun (19:49.640)

We're ignoring that at the low level,

Lex Fridman (19:52.720)

the neural network might be doing something like statistics.

Yann LeCun (19:56.440)

Like meaning, sorry to use this word

Lex Fridman (19:59.120)

probably incorrectly and crudely,

Yann LeCun (1:00:04.240)

the question is how fundamental is that,

Lex Fridman (1:00:06.000)

the nature of the whole hardware?

Lex Fridman (1:00:08.400)

And then is there any way to shortcut it

Lex Fridman (1:00:11.680)

if it's fundamental?

Yann LeCun (1:00:12.520)

If it's not, if it's most of intelligence,

Lex Fridman (1:00:14.280)

most of the cool stuff we've been talking about

Yann LeCun (1:00:15.920)

is mostly nurture, mostly trained.

Lex Fridman (1:00:18.800)

We figure it out by observing the world.

Yann LeCun (1:00:20.680)

We can form that big, beautiful, sexy background model

Lex Fridman (1:00:24.760)

that you're talking about just by sitting there.

Yann LeCun (1:00:28.880)

Then, okay, then you need to, then like maybe,

Lex Fridman (1:00:34.800)

it is all supervised learning all the way down.

Yann LeCun (1:00:37.840)

Self supervised learning, say.

Lex Fridman (1:00:39.000)

Whatever it is that makes, you know,

Yann LeCun (1:00:41.360)

human intelligence different from other animals,

Lex Fridman (1:00:44.080)

which, you know, a lot of people think is language

Lex Fridman (1:00:46.320)

and logical reasoning and this kind of stuff.

Lex Fridman (1:00:48.720)

It cannot be that complicated because it only popped up

Yann LeCun (1:00:51.000)

in the last million years.

Lex Fridman (1:00:52.840)

Yeah.

Yann LeCun (1:00:54.320)

And, you know, it only involves, you know,

Lex Fridman (1:00:57.840)

less than 1% of our genome might be,

Yann LeCun (1:00:59.640)

which is the difference between human genome

Lex Fridman (1:01:01.200)

and chimps or whatever.

Lex Fridman (1:01:03.360)

So it can't be that complicated.

Lex Fridman (1:01:06.640)

You know, it can't be that fundamental.

Yann LeCun (1:01:08.040)

I mean, most of the complicated stuff

Lex Fridman (1:01:10.880)

already exists in cats and dogs and, you know,

Yann LeCun (1:01:13.640)

certainly primates, nonhuman primates.

Lex Fridman (1:01:17.120)

Yeah, that little thing with humans

Yann LeCun (1:01:18.640)

might be just something about social interaction

Lex Fridman (1:01:22.480)

and ability to maintain ideas

Yann LeCun (1:01:24.000)

across like a collective of people.

Lex Fridman (1:01:28.160)

It sounds very dramatic and very impressive,

Lex Fridman (1:01:30.840)

but it probably isn't mechanistically speaking.

Lex Fridman (1:01:33.400)

It is, but we're not there yet.

Yann LeCun (1:01:34.680)

Like, you know, we have, I mean, this is number 634,

Lex Fridman (1:01:39.480)

you know, in the list of problems we have to solve.

Lex Fridman (1:01:43.400)

So basic physics of the world is number one.

Lex Fridman (1:01:46.880)

What do you, just a quick tangent on data augmentation.

Lex Fridman (1:01:51.600)

So a lot of it is hard coded versus learned.

Lex Fridman (1:01:57.920)

Do you have any intuition that maybe

Yann LeCun (1:02:00.960)

there could be some weird data augmentation,

Lex Fridman (1:02:03.600)

like generative type of data augmentation,

Yann LeCun (1:02:06.200)

like doing something weird to images,

Lex Fridman (1:02:07.680)

which then improves the similarity learning process?

Lex Fridman (1:02:13.120)

So not just kind of dumb, simple distortions,

Lex Fridman (1:02:16.280)

but by you shaking your head,

Yann LeCun (1:02:18.120)

just saying that even simple distortions are enough.

Lex Fridman (1:02:20.880)

I think, no, I think data augmentation

Yann LeCun (1:02:22.800)

is a temporary necessary evil.

Lex Fridman (1:02:26.480)

So what people are working on now is two things.

Yann LeCun (1:02:28.880)

One is the type of self supervised learning,

Lex Fridman (1:02:32.960)

like trying to translate the type of self supervised learning

Yann LeCun (1:02:35.480)

people use in language, translating these two images,

Lex Fridman (1:02:38.680)

which is basically a denoising autoencoder method, right?

Lex Fridman (1:02:41.800)

So you take an image, you block, you mask some parts of it,

Lex Fridman (1:02:47.320)

and then you train some giant neural net

Yann LeCun (1:02:49.520)

to reconstruct the parts that are missing.

Lex Fridman (1:02:52.640)

And until very recently,

Yann LeCun (1:02:56.200)

there was no working methods for that.

Lex Fridman (1:02:59.160)

All the autoencoder type methods for images

Yann LeCun (1:03:01.600)

weren't producing very good representation,

Lex Fridman (1:03:03.720)

but there's a paper now coming out of the fair group

Yann LeCun (1:03:06.600)

at MNL Park that actually works very well.

Lex Fridman (1:03:08.960)

So that doesn't require data augmentation,

Yann LeCun (1:03:12.120)

that requires only masking, okay.

Lex Fridman (1:03:15.000)

Only masking for images, okay.

Yann LeCun (1:03:18.640)

Right, so you mask part of the image

Lex Fridman (1:03:20.280)

and you train a system, which in this case is a transformer

Yann LeCun (1:03:24.560)

because the transformer represents the image

Lex Fridman (1:03:28.400)

as non overlapping patches,

Lex Fridman (1:03:30.880)

so it's easy to mask patches and things like that.

Lex Fridman (1:03:33.320)

Okay, but then my question transfers to that problem,

Lex Fridman (1:03:35.680)

the masking, like why should the mask be square or rectangle?

Lex Fridman (1:03:40.080)

So it doesn't matter, like, you know,

Yann LeCun (1:03:41.600)

I think we're gonna come up probably in the future

Lex Fridman (1:03:44.360)

with sort of ways to mask that are kind of random,

Yann LeCun (1:03:50.480)

essentially, I mean, they are random already, but.

Lex Fridman (1:03:52.920)

No, no, but like something that's challenging,

Yann LeCun (1:03:56.800)

like optimally challenging.

Lex Fridman (1:03:59.400)

So like, I mean, maybe it's a metaphor that doesn't apply,

Lex Fridman (1:04:02.440)

but you're, it seems like there's a data augmentation

Lex Fridman (1:04:06.400)

or masking, there's an interactive element with it.

Yann LeCun (1:04:09.880)

Like you're almost like playing with an image.

Lex Fridman (1:04:12.560)

And like, it's like the way we play with an image

Yann LeCun (1:04:14.720)

in our minds.

Lex Fridman (1:04:15.680)

No, but it's like dropout.

Yann LeCun (1:04:16.680)

It's like Boston machine training.

Lex Fridman (1:04:18.160)

You, you know, every time you see a percept,

Yann LeCun (1:04:23.200)

you also, you can perturb it in some way.

Lex Fridman (1:04:26.840)

And then the principle of the training procedure

Yann LeCun (1:04:31.520)

is to minimize the difference of the output

Lex Fridman (1:04:33.600)

or the representation between the clean version

Lex Fridman (1:04:36.920)

and the corrupted version, essentially, right?

Lex Fridman (1:04:40.280)

And you can do this in real time, right?

Lex Fridman (1:04:42.000)

So, you know, Boston machine work like this, right?

Lex Fridman (1:04:44.240)

You show a percept, you tell the machine

Yann LeCun (1:04:47.400)

that's a good combination of activities

Lex Fridman (1:04:49.840)

or your input neurons.

Lex Fridman (1:04:50.880)

And then you either let them go their merry way

Lex Fridman (1:04:56.560)

without clamping them to values,

Yann LeCun (1:04:58.960)

or you only do this with a subset.

Lex Fridman (1:05:01.120)

And what you're doing is you're training the system

Lex Fridman (1:05:03.520)

so that the stable state of the entire network

Lex Fridman (1:05:07.000)

is the same regardless of whether it sees

Yann LeCun (1:05:08.920)

the entire input or whether it sees only part of it.

Lex Fridman (1:05:12.880)

You know, denoising autoencoder method

Lex Fridman (1:05:14.360)

is basically the same thing, right?

Lex Fridman (1:05:15.880)

You're training a system to reproduce the input,

Yann LeCun (1:05:18.600)

the complete inputs and filling the input

Lex Fridman (1:05:20.480)

and filling the blanks, regardless of which parts

Yann LeCun (1:05:23.400)

are missing, and that's really the underlying principle.

Lex Fridman (1:05:26.280)

And you could imagine sort of, even in the brain,

Yann LeCun (1:05:28.320)

some sort of neural principle where, you know,

Lex Fridman (1:05:30.720)

neurons kind of oscillate, right?

Lex Fridman (1:05:32.800)

So they take their activity and then temporarily

Lex Fridman (1:05:35.520)

they kind of shut off to, you know,

Yann LeCun (1:05:38.040)

force the rest of the system to basically reconstruct

Lex Fridman (1:05:42.120)

the input without their help, you know?

Yann LeCun (1:05:44.800)

And, I mean, you could imagine, you know,

Lex Fridman (1:05:49.040)

more or less biologically possible processes.

Yann LeCun (1:05:51.040)

Something like that.

Lex Fridman (1:05:51.880)

And I guess with this denoising autoencoder

Lex Fridman (1:05:54.960)

and masking and data augmentation,

Lex Fridman (1:05:58.720)

you don't have to worry about being super efficient.

Yann LeCun (1:06:01.160)

You could just do as much as you want

Lex Fridman (1:06:03.960)

and get better over time.

Yann LeCun (1:06:06.160)

Because I was thinking, like, you might want to be clever

Lex Fridman (1:06:08.800)

about the way you do all these procedures, you know,

Lex Fridman (1:06:12.000)

but that's only, it's somehow costly to do every iteration,

Lex Fridman (1:06:16.720)

but it's not really.

Yann LeCun (1:06:17.960)

Not really.

Lex Fridman (1:06:19.280)

Maybe.

Lex Fridman (1:06:20.280)

And then there is, you know,

Lex Fridman (1:06:21.480)

data augmentation without explicit data augmentation.

Yann LeCun (1:06:24.160)

This data augmentation by weighting,

Lex Fridman (1:06:25.600)

which is, you know, the sort of video prediction.

Yann LeCun (1:06:29.320)

You're observing a video clip,

Lex Fridman (1:06:31.480)

observing the, you know, the continuation of that video clip.

Yann LeCun (1:06:36.400)

You try to learn a representation

Lex Fridman (1:06:38.040)

using dual joint embedding architectures

Yann LeCun (1:06:40.240)

in such a way that the representation of the future clip

Lex Fridman (1:06:43.280)

is easily predictable from the representation

Yann LeCun (1:06:45.680)

of the observed clip.

Lex Fridman (1:06:48.600)

Do you think YouTube has enough raw data

Lex Fridman (1:06:52.720)

from which to learn how to be a cat?

Lex Fridman (1:06:56.400)

I think so.

Lex Fridman (1:06:57.760)

So the amount of data is not the constraint.

Lex Fridman (1:07:01.200)

No, it would require some selection, I think.

Lex Fridman (1:07:04.120)

Some selection?

Lex Fridman (1:07:05.400)

Some selection of, you know, maybe the right type of data.

Yann LeCun (1:07:08.480)

You need some.

Lex Fridman (1:07:09.320)

Don't go down the rabbit hole of just cat videos.

Yann LeCun (1:07:11.400)

You might need to watch some lectures or something.

Lex Fridman (1:07:14.600)

No, you wouldn't.

Lex Fridman (1:07:15.720)

How meta would that be

Lex Fridman (1:07:17.480)

if it like watches lectures about intelligence

Lex Fridman (1:07:21.400)

and then learns,

Lex Fridman (1:07:22.240)

watches your lectures in NYU

Lex Fridman (1:07:24.320)

and learns from that how to be intelligent?

Lex Fridman (1:07:26.280)

I don't think that would be enough.

Lex Fridman (1:07:30.080)

What's your, do you find multimodal learning interesting?

Lex Fridman (1:07:33.240)

We've been talking about visual language,

Yann LeCun (1:07:35.080)

like combining those together,

Lex Fridman (1:07:36.440)

maybe audio, all those kinds of things.

Yann LeCun (1:07:38.120)

There's a lot of things that I find interesting

Lex Fridman (1:07:40.400)

in the short term,

Lex Fridman (1:07:41.240)

but are not addressing the important problem

Lex Fridman (1:07:44.080)

that I think are really kind of the big challenges.

Lex Fridman (1:07:46.600)

So I think, you know, things like multitask learning,

Lex Fridman (1:07:48.920)

continual learning, you know, adversarial issues.

Yann LeCun (1:07:54.360)

I mean, those have great practical interests

Lex Fridman (1:07:57.000)

in the relatively short term, possibly,

Lex Fridman (1:08:00.280)

but I don't think they're fundamental.

Lex Fridman (1:08:01.240)

You know, active learning,

Yann LeCun (1:08:02.600)

even to some extent, reinforcement learning.

Lex Fridman (1:08:04.360)

I think those things will become either obsolete

Yann LeCun (1:08:07.920)

or useless or easy

Lex Fridman (1:08:10.800)

once we figured out how to do self supervised

Yann LeCun (1:08:14.880)

representation learning

Lex Fridman (1:08:15.880)

or learning predictive world models.

Lex Fridman (1:08:19.280)

And so I think that's what, you know,

Lex Fridman (1:08:21.480)

the entire community should be focusing on.

Yann LeCun (1:08:24.400)

At least people who are interested

Lex Fridman (1:08:25.400)

in sort of fundamental questions

Yann LeCun (1:08:26.680)

or, you know, really kind of pushing the envelope

Lex Fridman (1:08:28.440)

of AI towards the next stage.

Lex Fridman (1:08:31.440)

But of course, there's like a huge amount of,

Lex Fridman (1:08:33.080)

you know, very interesting work to do

Yann LeCun (1:08:34.360)

in sort of practical questions

Lex Fridman (1:08:35.840)

that have, you know, short term impact.

Yann LeCun (1:08:38.000)

Well, you know, it's difficult to talk about

Lex Fridman (1:08:41.240)

the temporal scale,

Yann LeCun (1:08:42.200)

because all of human civilization

Lex Fridman (1:08:44.240)

will eventually be destroyed

Yann LeCun (1:08:45.400)

because the sun will die out.

Lex Fridman (1:08:48.520)

And even if Elon Musk is successful

Yann LeCun (1:08:50.280)

in multi planetary colonization across the galaxy,

Lex Fridman (1:08:54.520)

eventually the entirety of it

Yann LeCun (1:08:56.560)

will just become giant black holes.

Lex Fridman (1:08:58.920)

And that's gonna take a while though.

Yann LeCun (1:09:02.120)

So, but what I'm saying is then that logic

Lex Fridman (1:09:04.800)

can be used to say it's all meaningless.

Yann LeCun (1:09:07.360)

I'm saying all that to say that multitask learning

Lex Fridman (1:09:11.840)

might be, you're calling it practical

Yann LeCun (1:09:15.400)

or pragmatic or whatever.

Lex Fridman (1:09:17.280)

That might be the thing that achieves something

Yann LeCun (1:09:19.440)

very akin to intelligence

Lex Fridman (1:09:22.560)

while we're trying to solve the more general problem

Yann LeCun (1:09:26.880)

of self supervised learning of background knowledge.

Lex Fridman (1:09:29.400)

So the reason I bring that up,

Yann LeCun (1:09:30.640)

maybe one way to ask that question.

Lex Fridman (1:09:33.040)

I've been very impressed

Yann LeCun (1:09:34.000)

by what Tesla Autopilot team is doing.

Lex Fridman (1:09:36.440)

I don't know if you've gotten a chance to glance

Yann LeCun (1:09:38.320)

at this particular one example of multitask learning,

Lex Fridman (1:09:42.080)

where they're literally taking the problem,

Yann LeCun (1:09:44.960)

like, I don't know, Charles Darwin studying animals.

Lex Fridman (1:09:48.880)

They're studying the problem of driving

Lex Fridman (1:09:51.600)

and asking, okay, what are all the things

Lex Fridman (1:09:53.320)

you have to perceive?

Lex Fridman (1:09:55.000)

And the way they're solving it is one,

Lex Fridman (1:09:57.800)

there's an ontology where you're bringing that to the table.

Lex Fridman (1:10:00.400)

So you're formulating a bunch of different tasks.

Lex Fridman (1:10:02.240)

It's like over a hundred tasks or something like that

Yann LeCun (1:10:04.240)

that they're involved in driving.

Lex Fridman (1:10:06.040)

And then they're deploying it

Lex Fridman (1:10:07.720)

and then getting data back from people that run into trouble

Lex Fridman (1:10:10.480)

and they're trying to figure out, do we add tasks?

Lex Fridman (1:10:12.680)

Do we, like, we focus on each individual task separately?

Lex Fridman (1:10:16.040)

In fact, I would say,

Yann LeCun (1:10:18.280)

I would classify Andre Karpathy's talk in two ways.

Lex Fridman (1:10:20.680)

So one was about doors

Lex Fridman (1:10:22.360)

and the other one about how much ImageNet sucks.

Lex Fridman (1:10:24.720)

He kept going back and forth on those two topics,

Yann LeCun (1:10:28.560)

which ImageNet sucks,

Lex Fridman (1:10:30.000)

meaning you can't just use a single benchmark.

Yann LeCun (1:10:33.040)

There's so, like, you have to have like a giant suite

Lex Fridman (1:10:37.240)

of benchmarks to understand how well your system actually works.

Yann LeCun (1:10:39.880)

Oh, I agree with him.

Lex Fridman (1:10:40.720)

I mean, he's a very sensible guy.

Yann LeCun (1:10:43.880)

Now, okay, it's very clear that if you're faced

Lex Fridman (1:10:47.560)

with an engineering problem that you need to solve

Yann LeCun (1:10:50.480)

in a relatively short time,

Lex Fridman (1:10:51.920)

particularly if you have Elon Musk breathing down your neck,

Lex Fridman (1:10:55.880)

you're going to have to take shortcuts, right?

Lex Fridman (1:10:58.640)

You might think about the fact that the right thing to do

Lex Fridman (1:11:02.560)

and the longterm solution involves, you know,

Lex Fridman (1:11:04.520)

some fancy self supervised running,

Lex Fridman (1:11:06.560)

but you have, you know, Elon Musk breathing down your neck

Lex Fridman (1:11:10.240)

and, you know, this involves, you know, human lives.

Lex Fridman (1:11:13.600)

And so you have to basically just do

Lex Fridman (1:11:17.320)

the systematic engineering and, you know,

Yann LeCun (1:11:22.000)

fine tuning and refinements

Lex Fridman (1:11:23.280)

and trial and error and all that stuff.

Yann LeCun (1:11:26.360)

There's nothing wrong with that.

Lex Fridman (1:11:27.400)

That's called engineering.

Yann LeCun (1:11:28.600)

That's called, you know, putting technology out in the world.

Lex Fridman (1:11:35.840)

And you have to kind of ironclad it before you do this,

Yann LeCun (1:11:39.880)

you know, so much for, you know,

Lex Fridman (1:11:44.520)

grand ideas and principles.

Yann LeCun (1:11:48.280)

But, you know, I'm placing myself sort of, you know,

Lex Fridman (1:11:50.720)

some, you know, upstream of this, you know,

Yann LeCun (1:11:54.480)

quite a bit upstream of this.

Lex Fridman (1:11:55.760)

You're a Plato, think about platonic forms.

Yann LeCun (1:11:58.240)

You're not platonic because eventually

Lex Fridman (1:12:01.320)

I want that stuff to get used,

Lex Fridman (1:12:03.120)

but it's okay if it takes five or 10 years

Lex Fridman (1:12:06.920)

for the community to realize this is the right thing to do.

Yann LeCun (1:12:09.720)

I've done this before.

Lex Fridman (1:12:11.280)

It's been the case before that, you know,

Yann LeCun (1:12:13.240)

I've made that case.

Lex Fridman (1:12:14.440)

I mean, if you look back in the mid 2000, for example,

Lex Fridman (1:12:17.760)

and you ask yourself the question, okay,

Lex Fridman (1:12:19.320)

I want to recognize cars or faces or whatever,

Yann LeCun (1:12:24.360)

you know, I can use convolutional net.

Lex Fridman (1:12:25.560)

So I can use sort of more conventional

Yann LeCun (1:12:28.360)

kind of computer vision techniques, you know,

Lex Fridman (1:12:29.880)

using interest point detectors or assist density features

Yann LeCun (1:12:33.760)

and, you know, sticking an SVM on top.

Lex Fridman (1:12:35.760)

At that time, the datasets were so small

Yann LeCun (1:12:37.800)

that those methods that use more hand engineering

Lex Fridman (1:12:41.920)

worked better than ConvNets.

Yann LeCun (1:12:43.560)

It was just not enough data for ConvNets

Lex Fridman (1:12:45.560)

and ConvNets were a little slow with the kind of hardware

Yann LeCun (1:12:48.880)

that was available at the time.

Lex Fridman (1:12:50.840)

And there was a sea change when, basically,

Yann LeCun (1:12:53.880)

when, you know, datasets became bigger

Lex Fridman (1:12:56.680)

and GPUs became available.

Yann LeCun (1:12:58.600)

That's what, you know, two of the main factors

Lex Fridman (1:13:02.960)

that basically made people change their mind.

Lex Fridman (1:13:07.880)

And you can look at the history of,

Lex Fridman (1:13:11.880)

like, all sub branches of AI or pattern recognition.

Lex Fridman (1:13:16.400)

And there's a similar trajectory followed by techniques

Lex Fridman (1:13:19.800)

where people start by, you know, engineering the hell out of it.

Yann LeCun (1:13:25.200)

You know, be it optical character recognition,

Lex Fridman (1:13:29.200)

speech recognition, computer vision,

Yann LeCun (1:13:31.760)

like image recognition in general,

Lex Fridman (1:13:34.280)

natural language understanding, like, you know, translation,

Lex Fridman (1:13:37.280)

things like that, right?

Lex Fridman (1:13:38.000)

You start to engineer the hell out of it.

Yann LeCun (1:13:41.040)

You start to acquire all the knowledge,

Lex Fridman (1:13:42.680)

the prior knowledge you know about image formation,

Yann LeCun (1:13:44.760)

about, you know, the shape of characters,

Lex Fridman (1:13:46.600)

about, you know, morphological operations,

Yann LeCun (1:13:49.560)

about, like, feature extraction, Fourier transforms,

Lex Fridman (1:13:52.400)

you know, vernicke moments, you know, whatever, right?

Yann LeCun (1:13:54.440)

People have come up with thousands of ways

Lex Fridman (1:13:56.280)

of representing images

Lex Fridman (1:13:57.680)

so that they could be easily classified afterwards.

Lex Fridman (1:14:01.600)

Same for speech recognition, right?

Yann LeCun (1:14:03.000)

There is, you know, it took decades

Lex Fridman (1:14:04.640)

for people to figure out a good front end

Yann LeCun (1:14:06.920)

to preprocess speech signals

Lex Fridman (1:14:09.680)

so that, you know, all the information

Yann LeCun (1:14:11.120)

about what is being said is preserved,

Lex Fridman (1:14:13.400)

but most of the information

Yann LeCun (1:14:14.440)

about the identity of the speaker is gone.

Lex Fridman (1:14:16.920)

You know, kestrel coefficients or whatever, right?

Lex Fridman (1:14:20.880)

And same for text, right?

Lex Fridman (1:14:23.400)

You do named entity recognition and you parse

Lex Fridman (1:14:26.440)

and you do tagging of the parts of speech

Lex Fridman (1:14:31.800)

and, you know, you do this sort of tree representation

Lex Fridman (1:14:34.480)

of clauses and all that stuff, right?

Lex Fridman (1:14:36.480)

Before you can do anything.

Lex Fridman (1:14:40.720)

So that's how it starts, right?

Lex Fridman (1:14:43.520)

Just engineer the hell out of it.

Lex Fridman (1:14:45.160)

And then you start having data

Lex Fridman (1:14:47.920)

and maybe you have more powerful computers.

Yann LeCun (1:14:50.160)

Maybe you know something about statistical learning.

Lex Fridman (1:14:52.400)

So you start using machine learning

Lex Fridman (1:14:53.640)

and it's usually a small sliver

Lex Fridman (1:14:54.840)

on top of your kind of handcrafted system

Yann LeCun (1:14:56.800)

where, you know, you extract features by hand.

Lex Fridman (1:14:59.560)

Okay, and now, you know, nowadays the standard way

Yann LeCun (1:15:02.280)

of doing this is that you train the entire thing end to end

Lex Fridman (1:15:04.320)

with a deep learning system and it learns its own features

Yann LeCun (1:15:06.720)

and, you know, speech recognition systems nowadays

Lex Fridman (1:15:10.920)

or CR systems are completely end to end.

Yann LeCun (1:15:12.920)

It's, you know, it's some giant neural net

Lex Fridman (1:15:15.320)

that takes raw waveforms

Lex Fridman (1:15:17.920)

and produces a sequence of characters coming out.

Lex Fridman (1:15:20.440)

And it's just a huge neural net, right?

Yann LeCun (1:15:22.080)

There's no, you know, Markov model,

Lex Fridman (1:15:24.000)

there's no language model that is explicit

Yann LeCun (1:15:26.360)

other than, you know, something that's ingrained

Lex Fridman (1:15:28.600)

in the sort of neural language model, if you want.

Yann LeCun (1:15:30.960)

Same for translation, same for all kinds of stuff.

Lex Fridman (1:15:33.400)

So you see this continuous evolution

Yann LeCun (1:15:36.440)

from, you know, less and less hand crafting

Lex Fridman (1:15:40.440)

and more and more learning.

Lex Fridman (1:15:43.120)

And I think, I mean, it's true in biology as well.

Lex Fridman (1:15:50.680)

So, I mean, we might disagree about this,

Yann LeCun (1:15:52.880)

maybe not, this one little piece at the end,

Lex Fridman (1:15:56.860)

you mentioned active learning.

Yann LeCun (1:15:58.360)

It feels like active learning,

Lex Fridman (1:16:01.440)

which is the selection of data

Lex Fridman (1:16:02.880)

and also the interactivity needs to be part

Lex Fridman (1:16:05.600)

of this giant neural network.

Yann LeCun (1:16:06.800)

You cannot just be an observer

Lex Fridman (1:16:08.360)

to do self supervised learning.

Yann LeCun (1:16:09.720)

You have to, well, I don't,

Lex Fridman (1:16:12.200)

self supervised learning is just a word,

Lex Fridman (1:16:14.560)

but I would, whatever this giant stack

Lex Fridman (1:16:16.760)

of a neural network that's automatically learning,

Yann LeCun (1:16:19.640)

it feels, my intuition is that you have to have a system,

Lex Fridman (1:16:26.520)

whether it's a physical robot or a digital robot,

Yann LeCun (1:16:30.220)

that's interacting with the world

Lex Fridman (1:16:32.360)

and doing so in a flawed way and improving over time

Yann LeCun (1:16:35.960)

in order to form the self supervised learning.

Lex Fridman (1:16:41.520)

Well, you can't just give it a giant sea of data.

Yann LeCun (1:16:44.960)

Okay, I agree and I disagree.

Lex Fridman (1:16:47.120)

I agree in the sense that I think, I agree in two ways.

Yann LeCun (1:16:52.000)

The first way I agree is that if you want,

Lex Fridman (1:16:55.140)

and you certainly need a causal model of the world

Yann LeCun (1:16:57.480)

that allows you to predict the consequences

Lex Fridman (1:16:59.120)

of your actions, to train that model,

Lex Fridman (1:17:01.280)

you need to take actions, right?

Lex Fridman (1:17:02.760)

You need to be able to act in a world

Lex Fridman (1:17:04.600)

and see the effect for you to be,

Lex Fridman (1:17:07.040)

to learn causal models of the world.

Lex Fridman (1:17:08.560)

So that's not obvious because you can observe others.

Lex Fridman (1:17:11.560)

You can observe others.

Lex Fridman (1:17:12.400)

And you can infer that they're similar to you

Lex Fridman (1:17:14.720)

and then you can learn from that.

Yann LeCun (1:17:16.000)

Yeah, but then you have to kind of hardwire that part,

Lex Fridman (1:17:18.400)

right, and then, you know, mirror neurons

Lex Fridman (1:17:19.880)

and all that stuff, right?

Lex Fridman (1:17:20.720)

So, and it's not clear to me

Lex Fridman (1:17:23.280)

how you would do this in a machine.

Lex Fridman (1:17:24.440)

So I think the action part would be necessary

Yann LeCun (1:17:30.240)

for having causal models of the world.

Lex Fridman (1:17:32.620)

The second reason it may be necessary,

Yann LeCun (1:17:36.660)

or at least more efficient,

Lex Fridman (1:17:37.860)

is that active learning basically, you know,

Lex Fridman (1:17:41.700)

goes for the jugular of what you don't know, right?

Lex Fridman (1:17:44.900)

Is, you know, obvious areas of uncertainty

Yann LeCun (1:17:48.020)

about your world and about how the world behaves.

Lex Fridman (1:17:52.940)

And you can resolve this uncertainty

Yann LeCun (1:17:56.220)

by systematic exploration of that part

Lex Fridman (1:17:58.980)

that you don't know.

Lex Fridman (1:18:00.300)

And if you know that you don't know,

Lex Fridman (1:18:01.700)

then, you know, it makes you curious.

Yann LeCun (1:18:03.020)

You kind of look into situations that,

Lex Fridman (1:18:05.620)

and, you know, across the animal world,

Yann LeCun (1:18:09.260)

different species have different levels of curiosity,

Lex Fridman (1:18:12.900)

right, depending on how they're built, right?

Yann LeCun (1:18:15.100)

So, you know, cats and rats are incredibly curious,

Lex Fridman (1:18:18.780)

dogs not so much, I mean, less.

Yann LeCun (1:18:20.620)

Yeah, so it could be useful

Lex Fridman (1:18:22.140)

to have that kind of curiosity.

Lex Fridman (1:18:23.900)

So it'd be useful,

Lex Fridman (1:18:24.740)

but curiosity just makes the process faster.

Yann LeCun (1:18:26.980)

It doesn't make the process exist.

Lex Fridman (1:18:28.780)

The, so what process, what learning process is it

Lex Fridman (1:18:33.820)

that active learning makes more efficient?

Lex Fridman (1:18:37.780)

And I'm asking that first question, you know,

Yann LeCun (1:18:42.300)

you know, we haven't answered that question yet.

Lex Fridman (1:18:43.940)

So, you know, I worry about active learning

Yann LeCun (1:18:45.860)

once this question is...

Lex Fridman (1:18:47.300)

So it's the more fundamental question to ask.

Lex Fridman (1:18:49.940)

And if active learning or interaction

Lex Fridman (1:18:53.900)

increases the efficiency of the learning,

Yann LeCun (1:18:56.220)

see, sometimes it becomes very different

Lex Fridman (1:18:59.700)

if the increase is several orders of magnitude, right?

Yann LeCun (1:19:03.700)

Like...

Lex Fridman (1:19:04.540)

That's true.

Lex Fridman (1:19:05.380)

But fundamentally it's still the same thing

Lex Fridman (1:19:07.620)

and building up the intuition about how to,

Yann LeCun (1:19:10.700)

in a self supervised way to construct background models,

Lex Fridman (1:19:13.340)

efficient or inefficient, is the core problem.

Lex Fridman (1:19:18.180)

What do you think about Yoshua Bengio's

Lex Fridman (1:19:20.300)

talking about consciousness

Lex Fridman (1:19:22.380)

and all of these kinds of concepts?

Lex Fridman (1:19:24.060)

Okay, I don't know what consciousness is, but...

Yann LeCun (1:19:29.780)

It's a good opener.

Lex Fridman (1:19:31.500)

And to some extent, a lot of the things

Yann LeCun (1:19:33.100)

that are said about consciousness

Lex Fridman (1:19:34.860)

remind me of the questions people were asking themselves

Yann LeCun (1:19:38.260)

in the 18th century or 17th century

Lex Fridman (1:19:40.900)

when they discovered that, you know, how the eye works

Lex Fridman (1:19:44.620)

and the fact that the image at the back of the eye

Lex Fridman (1:19:46.620)

was upside down, right?

Yann LeCun (1:19:49.420)

Because you have a lens.

Lex Fridman (1:19:50.260)

And so on your retina, the image that forms is an image

Yann LeCun (1:19:54.140)

of the world, but it's upside down.

Lex Fridman (1:19:55.180)

How is it that you see right side up?

Yann LeCun (1:19:57.820)

And, you know, with what we know today in science,

Lex Fridman (1:20:00.100)

you know, we realize this question doesn't make any sense

Lex Fridman (1:20:03.500)

or is kind of ridiculous in some way, right?

Lex Fridman (1:20:05.980)

So I think a lot of what is said about consciousness

Yann LeCun (1:20:07.820)

is of that nature.

Lex Fridman (1:20:08.660)

Now, that said, there is a lot of really smart people

Yann LeCun (1:20:10.620)

that for whom I have a lot of respect

Lex Fridman (1:20:13.460)

who are talking about this topic,

Yann LeCun (1:20:14.700)

people like David Chalmers, who is a colleague of mine at NYU.

Lex Fridman (1:20:17.900)

I have kind of an orthodox folk speculative hypothesis

Yann LeCun (1:20:28.140)

about consciousness.

Lex Fridman (1:20:29.180)

So we're talking about the study of a world model.

Lex Fridman (1:20:32.020)

And I think, you know, our entire prefrontal cortex

Lex Fridman (1:20:35.540)

basically is the engine for a world model.

Lex Fridman (1:20:40.820)

But when we are attending at a particular situation,

Lex Fridman (1:20:44.580)

we're focused on that situation.

Yann LeCun (1:20:46.060)

We basically cannot attend to anything else.

Lex Fridman (1:20:48.540)

And that seems to suggest that we basically have

Yann LeCun (1:20:53.540)

only one world model engine in our prefrontal cortex.

Lex Fridman (1:20:59.780)

That engine is configurable to the situation at hand.

Lex Fridman (1:21:02.620)

So we are building a box out of wood,

Lex Fridman (1:21:04.660)

or we are driving down the highway playing chess.

Yann LeCun (1:21:09.300)

We basically have a single model of the world

Lex Fridman (1:21:12.820)

that we configure into the situation at hand,

Yann LeCun (1:21:15.380)

which is why we can only attend to one task at a time.

Lex Fridman (1:21:19.220)

Now, if there is a task that we do repeatedly,

Yann LeCun (1:21:22.860)

it goes from the sort of deliberate reasoning

Lex Fridman (1:21:25.940)

using model of the world and prediction

Lex Fridman (1:21:27.420)

and perhaps something like model predictive control,

Lex Fridman (1:21:29.300)

which I was talking about earlier,

Yann LeCun (1:21:31.380)

to something that is more subconscious

Lex Fridman (1:21:33.340)

that becomes automatic.

Lex Fridman (1:21:34.380)

So I don't know if you've ever played

Lex Fridman (1:21:35.940)

against a chess grandmaster.

Lex Fridman (1:21:39.180)

I get wiped out in 10 plays, right?

Lex Fridman (1:21:43.820)

And I have to think about my move for like 15 minutes.

Lex Fridman (1:21:50.140)

And the person in front of me, the grandmaster,

Lex Fridman (1:21:52.620)

would just react within seconds, right?

Yann LeCun (1:21:56.540)

He doesn't need to think about it.

Lex Fridman (1:21:58.580)

That's become part of the subconscious

Yann LeCun (1:21:59.980)

because it's basically just pattern recognition

Lex Fridman (1:22:02.620)

at this point.

Yann LeCun (1:22:04.740)

Same, the first few hours you drive a car,

Lex Fridman (1:22:07.660)

you are really attentive, you can't do anything else.

Lex Fridman (1:22:09.660)

And then after 20, 30 hours of practice, 50 hours,

Lex Fridman (1:22:13.460)

the subconscious, you can talk to the person next to you,

Lex Fridman (1:22:15.700)

things like that, right?

Lex Fridman (1:22:17.100)

Unless the situation becomes unpredictable

Lex Fridman (1:22:19.060)

and then you have to stop talking.

Lex Fridman (1:22:21.060)

So that suggests you only have one model in your head.

Lex Fridman (1:22:24.740)

And it might suggest the idea that consciousness

Lex Fridman (1:22:27.860)

basically is the module that configures

Yann LeCun (1:22:29.780)

this world model of yours.

Lex Fridman (1:22:31.980)

You need to have some sort of executive kind of overseer

Yann LeCun (1:22:36.540)

that configures your world model for the situation at hand.

Lex Fridman (1:22:40.620)

And that leads to kind of the really curious concept

Yann LeCun (1:22:43.780)

that consciousness is not a consequence

Lex Fridman (1:22:46.020)

of the power of our minds,

Lex Fridman (1:22:47.660)

but of the limitation of our brains.

Lex Fridman (1:22:49.940)

That because we have only one world model,

Yann LeCun (1:22:52.060)

we have to be conscious.

Lex Fridman (1:22:53.660)

If we had as many world models

Yann LeCun (1:22:55.220)

as situations we encounter,

Lex Fridman (1:22:58.540)

then we could do all of them simultaneously

Lex Fridman (1:23:00.740)

and we wouldn't need this sort of executive control

Lex Fridman (1:23:02.940)

that we call consciousness.

Yann LeCun (1:23:04.540)

Yeah, interesting.

Lex Fridman (1:23:05.380)

And somehow maybe that executive controller,

Yann LeCun (1:23:08.940)

I mean, the hard problem of consciousness,

Lex Fridman (1:23:10.980)

there's some kind of chemicals in biology

Yann LeCun (1:23:12.860)

that's creating a feeling,

Lex Fridman (1:23:15.020)

like it feels to experience some of these things.

Yann LeCun (1:23:18.780)

That's kind of like the hard question is,

Lex Fridman (1:23:22.460)

what the heck is that and why is that useful?

Yann LeCun (1:23:24.900)

Maybe the more pragmatic question,

Lex Fridman (1:23:26.180)

why is it useful to feel like this is really you

Yann LeCun (1:23:29.940)

experiencing this versus just like information

Lex Fridman (1:23:33.340)

being processed?

Yann LeCun (1:23:34.380)

It could be just a very nice side effect

Lex Fridman (1:23:39.020)

of the way we evolved.

Yann LeCun (1:23:41.820)

That's just very useful to feel a sense of ownership

Lex Fridman (1:23:48.620)

to the decisions you make, to the perceptions you make,

Yann LeCun (1:23:51.180)

to the model you're trying to maintain.

Lex Fridman (1:23:53.180)

Like you own this thing and this is the only one you got

Lex Fridman (1:23:56.260)

and if you lose it, it's gonna really suck.

Lex Fridman (1:23:58.420)

And so you should really send the brain

Yann LeCun (1:24:00.620)

some signals about it.

Lex Fridman (1:24:02.300)

So what ideas do you believe might be true

Lex Fridman (1:24:06.860)

that most or at least many people disagree with?

Lex Fridman (1:24:11.260)

Let's say in the space of machine learning.

Yann LeCun (1:24:13.740)

Well, it depends who you talk about,

Lex Fridman (1:24:14.940)

but I think, so certainly there is a bunch of people

Lex Fridman (1:24:20.100)

who are nativists, right?

Lex Fridman (1:24:21.100)

Who think that a lot of the basic things about the world

Yann LeCun (1:24:23.300)

are kind of hardwired in our minds.

Lex Fridman (1:24:26.420)

Things like the world is three dimensional, for example,

Lex Fridman (1:24:28.860)

is that hardwired?

Lex Fridman (1:24:30.420)

Things like object permanence,

Yann LeCun (1:24:32.660)

is this something that we learn

Lex Fridman (1:24:35.140)

before the age of three months or so?

Lex Fridman (1:24:37.500)

Or are we born with it?

Lex Fridman (1:24:39.340)

And there are very wide disagreements

Yann LeCun (1:24:42.380)

among the cognitive scientists for this.

Lex Fridman (1:24:46.580)

I think those things are actually very simple to learn.

Yann LeCun (1:24:50.580)

Is it the case that the oriented edge detectors in V1

Lex Fridman (1:24:54.220)

are learned or are they hardwired?

Yann LeCun (1:24:56.180)

I think they are learned.

Lex Fridman (1:24:57.260)

They might be learned before both

Yann LeCun (1:24:58.580)

because it's really easy to generate signals

Lex Fridman (1:25:00.620)

from the retina that actually will train edge detectors.

Lex Fridman (1:25:04.620)

And again, those are things that can be learned

Lex Fridman (1:25:06.740)

within minutes of opening your eyes, right?

Yann LeCun (1:25:09.580)

I mean, since the 1990s,

Lex Fridman (1:25:12.660)

we have algorithms that can learn oriented edge detectors

Yann LeCun (1:25:15.460)

completely unsupervised

Lex Fridman (1:25:16.940)

with the equivalent of a few minutes of real time.

Lex Fridman (1:25:19.060)

So those things have to be learned.

Lex Fridman (1:25:22.660)

And there's also those MIT experiments

Yann LeCun (1:25:24.580)

where you kind of plug the optical nerve

Lex Fridman (1:25:27.820)

on the auditory cortex of a baby ferret, right?

Lex Fridman (1:25:30.300)

And that auditory cortex

Lex Fridman (1:25:31.300)

becomes a visual cortex essentially.

Lex Fridman (1:25:33.420)

So clearly there's learning taking place there.

Lex Fridman (1:25:37.980)

So I think a lot of what people think are so basic

Yann LeCun (1:25:41.340)

that they need to be hardwired,

Lex Fridman (1:25:43.180)

I think a lot of those things are learned

Yann LeCun (1:25:44.420)

because they are easy to learn.

Lex Fridman (1:25:46.260)

So you put a lot of value in the power of learning.

Lex Fridman (1:25:49.980)

What kind of things do you suspect might not be learned?

Lex Fridman (1:25:53.340)

Is there something that could not be learned?

Lex Fridman (1:25:56.060)

So your intrinsic drives are not learned.

Lex Fridman (1:25:59.820)

There are the things that make humans human

Lex Fridman (1:26:03.460)

or make cats different from dogs, right?

Lex Fridman (1:26:07.460)

It's the basic drives that are kind of hardwired

Yann LeCun (1:26:10.060)

in our basal ganglia.

Lex Fridman (1:26:13.100)

I mean, there are people who are working

Yann LeCun (1:26:14.060)

on this kind of stuff that's called intrinsic motivation

Lex Fridman (1:26:16.380)

in the context of reinforcement learning.

Lex Fridman (1:26:18.220)

So these are objective functions

Lex Fridman (1:26:20.100)

where the reward doesn't come from the external world.

Yann LeCun (1:26:23.100)

It's computed by your own brain.

Lex Fridman (1:26:24.660)

Your own brain computes whether you're happy or not, right?

Yann LeCun (1:26:28.140)

It measures your degree of comfort or in comfort.

Lex Fridman (1:26:33.460)

And because it's your brain computing this,

Yann LeCun (1:26:36.100)

presumably it knows also how to estimate

Lex Fridman (1:26:37.780)

gradients of this, right?

Lex Fridman (1:26:38.780)

So it's easier to learn when your objective is intrinsic.

Lex Fridman (1:26:47.100)

So that has to be hardwired.

Yann LeCun (1:26:50.100)

The critic that makes longterm prediction of the outcome,

Lex Fridman (1:26:53.460)

which is the eventual result of this, that's learned.

Lex Fridman (1:26:57.860)

And perception is learned

Lex Fridman (1:26:59.060)

and your model of the world is learned.

Lex Fridman (1:27:01.260)

But let me take an example of why the critic,

Lex Fridman (1:27:04.260)

I mean, an example of how the critic may be learned, right?

Yann LeCun (1:27:06.860)

If I come to you, I reach across the table

Lex Fridman (1:27:11.220)

and I pinch your arm, right?

Yann LeCun (1:27:13.380)

Complete surprise for you.

Lex Fridman (1:27:15.060)

You would not have expected this from me.

Yann LeCun (1:27:16.260)

I was expecting that the whole time, but yes, right.

Lex Fridman (1:27:18.100)

Let's say for the sake of the story, yes.

Yann LeCun (1:27:20.420)

So, okay, your basal ganglia is gonna light up

Lex Fridman (1:27:24.980)

because it's gonna hurt, right?

Lex Fridman (1:27:28.500)

And now your model of the world includes the fact that

Lex Fridman (1:27:31.140)

I may pinch you if I approach my...

Yann LeCun (1:27:34.820)

Don't trust humans.

Lex Fridman (1:27:36.220)

Right, my hand to your arm.

Lex Fridman (1:27:37.860)

So if I try again, you're gonna recoil.

Lex Fridman (1:27:40.020)

And that's your critic, your predictive,

Yann LeCun (1:27:44.060)

your predictor of your ultimate pain system

Lex Fridman (1:27:50.660)

that predicts that something bad is gonna happen

Lex Fridman (1:27:52.380)

and you recoil to avoid it.

Lex Fridman (1:27:53.860)

So even that can be learned.

Yann LeCun (1:27:55.260)

That is learned, definitely.

Lex Fridman (1:27:56.700)

This is what allows you also to define some goals, right?

Lex Fridman (1:28:00.700)

So the fact that you're a school child,

Lex Fridman (1:28:04.540)

you wake up in the morning and you go to school

Lex Fridman (1:28:06.780)

and it's not because you necessarily like waking up early

Lex Fridman (1:28:12.060)

and going to school,

Lex Fridman (1:28:12.900)

but you know that there is a long term objective

Lex Fridman (1:28:14.620)

you're trying to optimize.

Lex Fridman (1:28:15.820)

So Ernest Becker, I'm not sure if you're familiar with him,

Lex Fridman (1:28:18.540)

the philosopher, he wrote the book Denial of Death

Lex Fridman (1:28:20.900)

and his idea is that one of the core motivations

Lex Fridman (1:28:23.420)

of human beings is our terror of death, our fear of death.

Yann LeCun (1:28:27.220)

That's what makes us unique from cats.

Lex Fridman (1:28:28.900)

Cats are just surviving.

Yann LeCun (1:28:30.500)

They do not have a deep, like a cognizance introspection

Lex Fridman (1:28:37.540)

that over the horizon is the end.

Lex Fridman (1:28:41.740)

And then he says that, I mean,

Lex Fridman (1:28:43.060)

there's a terror management theory

Yann LeCun (1:28:44.420)

that just all these psychological experiments

Lex Fridman (1:28:46.260)

that show basically this idea

Yann LeCun (1:28:50.020)

that all of human civilization, everything we create

Lex Fridman (1:28:54.380)

is kind of trying to forget if even for a brief moment

Yann LeCun (1:28:58.820)

that we're going to die.

Lex Fridman (1:29:00.660)

When do you think humans understand

Lex Fridman (1:29:03.780)

that they're going to die?

Lex Fridman (1:29:04.900)

Is it learned early on also?

Yann LeCun (1:29:07.580)

I don't know at what point.

Lex Fridman (1:29:11.260)

I mean, it's a question like at what point

Lex Fridman (1:29:13.460)

do you realize that what death really is?

Lex Fridman (1:29:16.420)

And I think most people don't actually realize

Lex Fridman (1:29:18.180)

what death is, right?

Lex Fridman (1:29:19.220)

I mean, most people believe that you go to heaven

Lex Fridman (1:29:20.940)

or something, right?

Lex Fridman (1:29:21.860)

So to push back on that, what Ernest Becker says

Lex Fridman (1:29:25.580)

and Sheldon Solomon, all of those folks,

Lex Fridman (1:29:29.300)

and I find those ideas a little bit compelling

Yann LeCun (1:29:31.620)

is that there is moments in life, early in life,

Lex Fridman (1:29:34.100)

a lot of this fun happens early in life

Yann LeCun (1:29:36.540)

when you do deeply experience

Lex Fridman (1:29:41.620)

the terror of this realization.

Lex Fridman (1:29:43.540)

And all the things you think about about religion,

Lex Fridman (1:29:45.980)

all those kinds of things that we kind of think about

Yann LeCun (1:29:48.420)

more like teenage years and later,

Lex Fridman (1:29:50.660)

we're talking about way earlier.

Yann LeCun (1:29:52.100)

No, it was like seven or eight years,

Lex Fridman (1:29:53.220)

something like that, yeah.

Yann LeCun (1:29:54.060)

You realize, holy crap, this is like the mystery,

Lex Fridman (1:29:59.660)

the terror, like it's almost like you're a little prey,

Yann LeCun (1:30:03.220)

a little baby deer sitting in the darkness

Lex Fridman (1:30:05.340)

of the jungle or the woods looking all around you.

Yann LeCun (1:30:08.060)

There's darkness full of terror.

Lex Fridman (1:30:09.540)

I mean, that realization says, okay,

Yann LeCun (1:30:12.140)

I'm gonna go back in the comfort of my mind

Lex Fridman (1:30:14.460)

where there is a deep meaning,

Yann LeCun (1:30:16.780)

where there is maybe like pretend I'm immortal

Lex Fridman (1:30:20.420)

in however way, however kind of idea I can construct

Yann LeCun (1:30:25.060)

to help me understand that I'm immortal.

Lex Fridman (1:30:27.180)

Religion helps with that.

Yann LeCun (1:30:28.660)

You can delude yourself in all kinds of ways,

Lex Fridman (1:30:31.440)

like lose yourself in the busyness of each day,

Yann LeCun (1:30:34.220)

have little goals in mind, all those kinds of things

Lex Fridman (1:30:36.380)

to think that it's gonna go on forever.

Lex Fridman (1:30:38.100)

And you kind of know you're gonna die, yeah,

Lex Fridman (1:30:40.740)

and it's gonna be sad, but you don't really understand

Yann LeCun (1:30:43.820)

that you're going to die.

Lex Fridman (1:30:45.140)

And so that's their idea.

Lex Fridman (1:30:46.460)

And I find that compelling because it does seem

Lex Fridman (1:30:49.940)

to be a core unique aspect of human nature

Yann LeCun (1:30:52.820)

that we're able to think that we're going,

Lex Fridman (1:30:55.180)

we're able to really understand that this life is finite.

Yann LeCun (1:30:59.540)

That seems important.

Lex Fridman (1:31:00.580)

There's a bunch of different things there.

Lex Fridman (1:31:02.260)

So first of all, I don't think there is a qualitative

Lex Fridman (1:31:04.300)

difference between us and cats in the term.

Yann LeCun (1:31:07.520)

I think the difference is that we just have a better

Lex Fridman (1:31:10.180)

long term ability to predict in the long term.

Lex Fridman (1:31:14.740)

And so we have a better understanding of how the world works.

Lex Fridman (1:31:17.380)

So we have better understanding of finiteness of life

Lex Fridman (1:31:20.180)

and things like that.

Lex Fridman (1:31:21.020)

So we have a better planning engine than cats?

Yann LeCun (1:31:23.540)

Yeah.

Lex Fridman (1:31:24.500)

Okay.

Lex Fridman (1:31:25.340)

But what's the motivation for planning that far?

Lex Fridman (1:31:28.780)

Well, I think it's just a side effect of the fact

Yann LeCun (1:31:30.540)

that we have just a better planning engine

Lex Fridman (1:31:32.340)

because it makes us, as I said,

Yann LeCun (1:31:34.780)

the essence of intelligence is the ability to predict.

Lex Fridman (1:31:37.420)

And so the, because we're smarter as a side effect,

Yann LeCun (1:31:41.220)

we also have this ability to kind of make predictions

Lex Fridman (1:31:43.500)

about our own future existence or lack thereof.

Yann LeCun (1:31:47.580)

Okay.

Lex Fridman (1:31:48.500)

You say religion helps with that.

Yann LeCun (1:31:50.540)

I think religion hurts actually.

Lex Fridman (1:31:53.000)

It makes people worry about like,

Yann LeCun (1:31:55.000)

what's going to happen after their death, et cetera.

Lex Fridman (1:31:57.500)

If you believe that, you just don't exist after death.

Yann LeCun (1:32:00.820)

Like, it solves completely the problem, at least.

Lex Fridman (1:32:02.940)

You're saying if you don't believe in God,

Lex Fridman (1:32:04.940)

you don't worry about what happens after death?

Lex Fridman (1:32:07.220)

Yeah.

Yann LeCun (1:32:08.260)

I don't know.

Lex Fridman (1:32:09.100)

You only worry about this life

Yann LeCun (1:32:11.900)

because that's the only one you have.

Lex Fridman (1:32:14.220)

I think it's, well, I don't know.

Yann LeCun (1:32:16.140)

If I were to say what Ernest Becker says,

Lex Fridman (1:32:17.740)

and obviously I agree with him more than not,

Yann LeCun (1:32:22.140)

is you do deeply worry.

Lex Fridman (1:32:26.160)

If you believe there's no God,

Yann LeCun (1:32:27.900)

there's still a deep worry of the mystery of it all.

Lex Fridman (1:32:31.780)

Like, how does that make any sense that it just ends?

Yann LeCun (1:32:35.700)

I don't think we can truly understand that this ride,

Lex Fridman (1:32:39.740)

I mean, so much of our life, the consciousness,

Yann LeCun (1:32:41.900)

the ego is invested in this being.

Lex Fridman (1:32:46.220)

And then...

Yann LeCun (1:32:47.580)

Science keeps bringing humanity down from its pedestal.

Lex Fridman (1:32:51.560)

And that's just another example of it.

Yann LeCun (1:32:54.740)

That's wonderful, but for us individual humans,

Lex Fridman (1:32:57.820)

we don't like to be brought down from a pedestal.

Yann LeCun (1:33:00.300)

You're saying like, but see, you're fine with it because,

Lex Fridman (1:33:03.580)

well, so what Ernest Becker would say is you're fine with it

Yann LeCun (1:33:06.340)

because there's just a more peaceful existence for you,

Lex Fridman (1:33:08.580)

but you're not really fine.

Yann LeCun (1:33:09.580)

You're hiding from it.

Lex Fridman (1:33:10.820)

In fact, some of the people that experience

Yann LeCun (1:33:12.780)

the deepest trauma earlier in life,

Lex Fridman (1:33:16.700)

they often, before they seek extensive therapy,

Yann LeCun (1:33:19.580)

will say that I'm fine.

Lex Fridman (1:33:21.060)

It's like when you talk to people who are truly angry,

Lex Fridman (1:33:23.460)

how are you doing, I'm fine.

Lex Fridman (1:33:25.380)

The question is, what's going on?

Yann LeCun (1:33:27.780)

Now I had a near death experience.

Lex Fridman (1:33:29.140)

I had a very bad motorbike accident when I was 17.

Yann LeCun (1:33:33.580)

So, but that didn't have any impact

Lex Fridman (1:33:36.920)

on my reflection on that topic.

Lex Fridman (1:33:40.420)

So I'm basically just playing a bit of devil's advocate,

Lex Fridman (1:33:43.100)

pushing back on wondering,

Lex Fridman (1:33:45.820)

is it truly possible to accept death?

Lex Fridman (1:33:47.540)

And the flip side, that's more interesting,

Yann LeCun (1:33:49.340)

I think for AI and robotics is how important

Lex Fridman (1:33:53.060)

is it to have this as one of the suite of motivations

Yann LeCun (1:33:57.180)

is to not just avoid falling off the roof

Lex Fridman (1:34:03.320)

or something like that, but ponder the end of the ride.

Yann LeCun (1:34:10.180)

If you listen to the stoics, it's a great motivator.

Lex Fridman (1:34:14.820)

It adds a sense of urgency.

Lex Fridman (1:34:16.900)

So maybe to truly fear death or be cognizant of it

Lex Fridman (1:34:21.420)

might give a deeper meaning and urgency to the moment

Yann LeCun (1:34:26.460)

to live fully.

Lex Fridman (1:34:30.460)

Maybe I don't disagree with that.

Yann LeCun (1:34:32.220)

I mean, I think what motivates me here

Lex Fridman (1:34:34.280)

is knowing more about human nature.

Yann LeCun (1:34:38.980)

I mean, I think human nature and human intelligence

Lex Fridman (1:34:41.760)

is a big mystery.

Yann LeCun (1:34:42.600)

It's a scientific mystery

Lex Fridman (1:34:45.020)

in addition to philosophical and et cetera,

Lex Fridman (1:34:48.580)

but I'm a true believer in science.

Lex Fridman (1:34:50.700)

So, and I do have kind of a belief

Yann LeCun (1:34:56.180)

that for complex systems like the brain and the mind,

Lex Fridman (1:34:59.940)

the way to understand it is to try to reproduce it

Yann LeCun (1:35:04.460)

with artifacts that you build

Lex Fridman (1:35:07.060)

because you know what's essential to it

Yann LeCun (1:35:08.900)

when you try to build it.

Lex Fridman (1:35:10.180)

The same way I've used this analogy before with you,

Yann LeCun (1:35:12.660)

I believe, the same way we only started

Lex Fridman (1:35:15.780)

to understand aerodynamics

Yann LeCun (1:35:18.140)

when we started building airplanes

Lex Fridman (1:35:19.300)

and that helped us understand how birds fly.

Lex Fridman (1:35:22.380)

So I think there's kind of a similar process here

Lex Fridman (1:35:25.460)

where we don't have a full theory of intelligence,

Lex Fridman (1:35:29.660)

but building intelligent artifacts

Lex Fridman (1:35:31.760)

will help us perhaps develop some underlying theory

Yann LeCun (1:35:35.480)

that encompasses not just artificial implements,

Lex Fridman (1:35:39.380)

but also human and biological intelligence in general.

Lex Fridman (1:35:43.860)

So you're an interesting person to ask this question

Lex Fridman (1:35:46.080)

about sort of all kinds of different other

Yann LeCun (1:35:49.400)

intelligent entities or intelligences.

Lex Fridman (1:35:53.100)

What are your thoughts about kind of like the touring

Lex Fridman (1:35:56.300)

or the Chinese room question?

Lex Fridman (1:35:59.240)

If we create an AI system that exhibits

Yann LeCun (1:36:02.920)

a lot of properties of intelligence and consciousness,

Lex Fridman (1:36:07.520)

how comfortable are you thinking of that entity

Lex Fridman (1:36:10.220)

as intelligent or conscious?

Lex Fridman (1:36:12.340)

So you're trying to build now systems

Yann LeCun (1:36:14.580)

that have intelligence and there's metrics

Lex Fridman (1:36:16.420)

about their performance, but that metric is external.

Lex Fridman (1:36:22.740)

So how are you, are you okay calling a thing intelligent

Lex Fridman (1:36:26.420)

or are you going to be like most humans

Lex Fridman (1:36:29.020)

and be once again unhappy to be brought down

Lex Fridman (1:36:32.700)

from a pedestal of consciousness slash intelligence?

Yann LeCun (1:36:34.920)

No, I'll be very happy to understand

Lex Fridman (1:36:39.500)

more about human nature, human mind and human intelligence

Yann LeCun (1:36:45.540)

through the construction of machines

Lex Fridman (1:36:47.240)

that have similar abilities.

Lex Fridman (1:36:50.600)

And if a consequence of this is to bring down humanity

Lex Fridman (1:36:54.520)

one notch down from its already low pedestal,

Yann LeCun (1:36:58.020)

I'm just fine with it.

Lex Fridman (1:36:59.140)

That's just the reality of life.

Lex Fridman (1:37:01.360)

So I'm fine with that.

Lex Fridman (1:37:02.460)

Now you were asking me about things that,

Yann LeCun (1:37:05.020)

opinions I have that a lot of people may disagree with.

Lex Fridman (1:37:07.940)

I think if we think about the design

Yann LeCun (1:37:12.780)

of autonomous intelligence systems,

Lex Fridman (1:37:14.300)

so assuming that we are somewhat successful

Yann LeCun (1:37:16.860)

at some level of getting machines to learn models

Lex Fridman (1:37:20.060)

of the world, predictive models of the world,

Yann LeCun (1:37:22.620)

we build intrinsic motivation objective functions

Lex Fridman (1:37:25.860)

to drive the behavior of that system.

Yann LeCun (1:37:28.340)

The system also has perception modules

Lex Fridman (1:37:30.100)

that allows it to estimate the state of the world

Lex Fridman (1:37:32.820)

and then have some way of figuring out

Lex Fridman (1:37:34.640)

the sequence of actions that,

Yann LeCun (1:37:36.180)

to optimize a particular objective.

Lex Fridman (1:37:39.300)

If it has a critic of the type that I was describing before,

Yann LeCun (1:37:42.740)

the thing that makes you recoil your arm

Lex Fridman (1:37:44.600)

the second time I try to pinch you,

Yann LeCun (1:37:48.620)

intelligent autonomous machine will have emotions.

Lex Fridman (1:37:51.700)

I think emotions are an integral part

Yann LeCun (1:37:54.060)

of autonomous intelligence.

Lex Fridman (1:37:56.400)

If you have an intelligent system

Yann LeCun (1:37:59.020)

that is driven by intrinsic motivation, by objectives,

Lex Fridman (1:38:03.160)

if it has a critic that allows it to predict in advance

Yann LeCun (1:38:07.680)

whether the outcome of a situation is gonna be good or bad,

Lex Fridman (1:38:11.040)

is going to have emotions, it's gonna have fear.

Yann LeCun (1:38:13.480)

Yes.

Lex Fridman (1:38:14.320)

When it predicts that the outcome is gonna be bad

Lex Fridman (1:38:18.180)

and something to avoid is gonna have elation

Lex Fridman (1:38:20.720)

when it predicts it's gonna be good.

Yann LeCun (1:38:24.280)

If it has drives to relate with humans,

Lex Fridman (1:38:28.680)

in some ways the way humans have,

Lex Fridman (1:38:30.660)

it's gonna be social, right?

Lex Fridman (1:38:34.460)

And so it's gonna have emotions

Yann LeCun (1:38:36.460)

about attachment and things of that type.

Lex Fridman (1:38:38.620)

So I think the sort of sci fi thing

Yann LeCun (1:38:44.700)

where you see commander data,

Lex Fridman (1:38:46.900)

like having an emotion chip that you can turn off, right?

Yann LeCun (1:38:50.100)

I think that's ridiculous.

Lex Fridman (1:38:51.700)

So, I mean, here's the difficult

Yann LeCun (1:38:53.380)

philosophical social question.

Lex Fridman (1:38:57.820)

Do you think there will be a time like a civil rights

Yann LeCun (1:39:01.020)

movement for robots where, okay, forget the movement,

Lex Fridman (1:39:05.180)

but a discussion like the Supreme Court

Yann LeCun (1:39:09.740)

that particular kinds of robots,

Lex Fridman (1:39:12.880)

you know, particular kinds of systems

Yann LeCun (1:39:16.100)

deserve the same rights as humans

Lex Fridman (1:39:18.300)

because they can suffer just as humans can,

Yann LeCun (1:39:22.900)

all those kinds of things.

Lex Fridman (1:39:24.740)

Well, perhaps, perhaps not.

Yann LeCun (1:39:27.340)

Like imagine that humans were,

Lex Fridman (1:39:29.580)

that you could, you know, die and be restored.

Yann LeCun (1:39:33.740)

Like, you know, you could be sort of, you know,

Lex Fridman (1:39:35.500)

be 3D reprinted and, you know,

Yann LeCun (1:39:37.540)

your brain could be reconstructed in its finest details.

Lex Fridman (1:39:40.740)

Our ideas of rights will change in that case.

Yann LeCun (1:39:43.140)

If you can always just,

Lex Fridman (1:39:45.900)

there's always a backup you could always restore.

Yann LeCun (1:39:48.220)

Maybe like the importance of murder

Lex Fridman (1:39:50.260)

will go down one notch.

Yann LeCun (1:39:51.980)

That's right.

Lex Fridman (1:39:52.820)

But also your desire to do dangerous things,

Yann LeCun (1:39:57.580)

like, you know, skydiving or, you know,

Lex Fridman (1:40:03.300)

or, you know, race car driving,

Yann LeCun (1:40:05.660)

you know, car racing or that kind of stuff,

Lex Fridman (1:40:07.300)

you know, would probably increase

Yann LeCun (1:40:09.460)

or, you know, aeroplanes, aerobatics

Lex Fridman (1:40:11.140)

or that kind of stuff, right?

Yann LeCun (1:40:12.380)

It would be fine to do a lot of those things

Lex Fridman (1:40:14.180)

or explore, you know, dangerous areas and things like that.

Yann LeCun (1:40:17.500)

It would kind of change your relationship.

Lex Fridman (1:40:19.220)

So now it's very likely that robots would be like that

Yann LeCun (1:40:22.420)

because, you know, they'll be based on perhaps technology

Lex Fridman (1:40:27.060)

that is somewhat similar to today's technology

Lex Fridman (1:40:30.140)

and you can always have a backup.

Lex Fridman (1:40:32.260)

So it's possible, I don't know if you like video games,

Lex Fridman (1:40:35.700)

but there's a game called Diablo and...

Lex Fridman (1:40:39.340)

Oh, my sons are huge fans of this.

Yann LeCun (1:40:41.860)

Yes.

Lex Fridman (1:40:44.100)

In fact, they made a game that's inspired by it.

Yann LeCun (1:40:47.060)

Awesome.

Lex Fridman (1:40:47.900)

Like built a game?

Yann LeCun (1:40:49.260)

My three sons have a game design studio between them, yeah.

Lex Fridman (1:40:52.660)

That's awesome.

Yann LeCun (1:40:53.480)

They came out with a game.

Lex Fridman (1:40:54.320)

They just came out with a game.

Yann LeCun (1:40:55.160)

Last year, no, this was last year,

Lex Fridman (1:40:56.860)

early last year, about a year ago.

Yann LeCun (1:40:58.180)

That's awesome.

Lex Fridman (1:40:59.020)

But so in Diablo, there's something called hardcore mode,

Yann LeCun (1:41:02.020)

which if you die, there's no, you're gone.

Lex Fridman (1:41:05.480)

Right.

Yann LeCun (1:41:06.320)

That's it.

Lex Fridman (1:41:07.140)

And so it's possible with AI systems

Yann LeCun (1:41:10.620)

for them to be able to operate successfully

Lex Fridman (1:41:13.260)

and for us to treat them in a certain way

Yann LeCun (1:41:15.580)

because they have to be integrated in human society,

Lex Fridman (1:41:18.400)

they have to be able to die, no copies allowed.

Yann LeCun (1:41:22.020)

In fact, copying is illegal.

Lex Fridman (1:41:23.860)

It's possible with humans as well,

Yann LeCun (1:41:25.260)

like cloning will be illegal, even when it's possible.

Lex Fridman (1:41:28.580)

But cloning is not copying, right?

Yann LeCun (1:41:29.960)

I mean, you don't reproduce the mind of the person

Lex Fridman (1:41:33.060)

and the experience.

Yann LeCun (1:41:33.940)

Right.

Lex Fridman (1:41:34.760)

It's just a delayed twin, so.

Lex Fridman (1:41:36.420)

But then it's, but we were talking about with computers

Lex Fridman (1:41:39.060)

that you will be able to copy.

Yann LeCun (1:41:40.580)

Right.

Lex Fridman (1:41:41.420)

You will be able to perfectly save,

Yann LeCun (1:41:42.660)

pickle the mind state.

Lex Fridman (1:41:46.640)

And it's possible that that will be illegal

Yann LeCun (1:41:49.660)

because that goes against,

Lex Fridman (1:41:53.300)

that will destroy the motivation of the system.

Lex Fridman (1:41:55.980)

Okay, so let's say you have a domestic robot, okay?

Lex Fridman (1:42:00.240)

Sometime in the future.

Yann LeCun (1:42:01.380)

Yes.

Lex Fridman (1:42:02.460)

And the domestic robot comes to you kind of

Yann LeCun (1:42:06.100)

somewhat pre trained, it can do a bunch of things,

Lex Fridman (1:42:08.700)

but it has a particular personality

Yann LeCun (1:42:10.580)

that makes it slightly different from the other robots

Lex Fridman (1:42:12.300)

because that makes them more interesting.

Lex Fridman (1:42:14.220)

And then because it's lived with you for five years,

Lex Fridman (1:42:18.060)

you've grown some attachment to it and vice versa,

Lex Fridman (1:42:21.900)

and it's learned a lot about you.

Lex Fridman (1:42:24.380)

Or maybe it's not a real household robot.

Yann LeCun (1:42:25.900)

Maybe it's a virtual assistant that lives in your,

Lex Fridman (1:42:29.380)

you know, augmented reality glasses or whatever, right?

Lex Fridman (1:42:32.580)

You know, the horror movie type thing, right?

Lex Fridman (1:42:36.680)

And that system to some extent,

Yann LeCun (1:42:39.620)

the intelligence in that system is a bit like your child

Lex Fridman (1:42:43.900)

or maybe your PhD student in the sense that

Lex Fridman (1:42:47.100)

there's a lot of you in that machine now, right?

Lex Fridman (1:42:50.260)

And so if it were a living thing,

Lex Fridman (1:42:53.500)

you would do this for free if you want, right?

Lex Fridman (1:42:56.560)

If it's your child, your child can, you know,

Yann LeCun (1:42:58.400)

then live his or her own life.

Lex Fridman (1:43:01.580)

And you know, the fact that they learn stuff from you

Lex Fridman (1:43:04.020)

doesn't mean that you have any ownership of it, right?

Lex Fridman (1:43:06.540)

But if it's a robot that you've trained,

Yann LeCun (1:43:09.380)

perhaps you have some intellectual property claim

Lex Fridman (1:43:13.580)

about.

Yann LeCun (1:43:14.420)

Oh, intellectual property.

Lex Fridman (1:43:15.240)

Oh, I thought you meant like a permanence value

Yann LeCun (1:43:18.140)

in the sense that part of you is in.

Lex Fridman (1:43:20.180)

Well, there is permanence value, right?

Lex Fridman (1:43:21.700)

So you would lose a lot if that robot were to be destroyed

Lex Fridman (1:43:24.660)

and you had no backup, you would lose a lot, right?

Yann LeCun (1:43:26.660)

You lose a lot of investment, you know,

Lex Fridman (1:43:28.100)

kind of like, you know, a person dying, you know,

Yann LeCun (1:43:31.860)

that a friend of yours dying

Lex Fridman (1:43:34.300)

or a coworker or something like that.

Lex Fridman (1:43:38.480)

But also you have like intellectual property rights

Lex Fridman (1:43:42.340)

in the sense that that system is fine tuned

Yann LeCun (1:43:45.940)

to your particular existence.

Lex Fridman (1:43:47.340)

So that's now a very unique instantiation

Yann LeCun (1:43:49.860)

of that original background model,

Lex Fridman (1:43:51.980)

whatever it was that arrived.

Lex Fridman (1:43:54.260)

And then there are issues of privacy, right?

Lex Fridman (1:43:55.660)

Because now imagine that that robot has its own kind

Yann LeCun (1:44:00.000)

of volition and decides to work for someone else.

Lex Fridman (1:44:02.820)

Or kind of, you know, thinks life with you

Yann LeCun (1:44:06.020)

is sort of untenable or whatever.

Lex Fridman (1:44:07.880)

Now, all the things that that system learned from you,

Yann LeCun (1:44:14.760)

you know, can you like, you know,

Lex Fridman (1:44:16.880)

delete all the personal information

Lex Fridman (1:44:18.160)

that that system knows about you?

Lex Fridman (1:44:19.680)

I mean, that would be kind of an ethical question.

Yann LeCun (1:44:22.200)

Like, you know, can you erase the mind

Lex Fridman (1:44:24.760)

of a intelligent robot to protect your privacy?

Yann LeCun (1:44:30.040)

You can't do this with humans.

Lex Fridman (1:44:31.580)

You can ask them to shut up,

Lex Fridman (1:44:32.680)

but that you don't have complete power over them.

Lex Fridman (1:44:35.640)

You can't erase humans, yeah, it's the problem

Yann LeCun (1:44:38.040)

with the relationships, you know, if you break up,

Lex Fridman (1:44:40.120)

you can't erase the other human.

Yann LeCun (1:44:42.640)

With robots, I think it will have to be the same thing

Lex Fridman (1:44:44.960)

with robots, that risk, that there has to be some risk

Yann LeCun (1:44:52.420)

to our interactions to truly experience them deeply,

Lex Fridman (1:44:55.120)

it feels like.

Lex Fridman (1:44:56.140)

So you have to be able to lose your robot friend

Lex Fridman (1:44:59.600)

and that robot friend to go tweeting

Yann LeCun (1:45:01.680)

about how much of an asshole you were.

Lex Fridman (1:45:03.680)

But then are you allowed to, you know,

Yann LeCun (1:45:06.160)

murder the robot to protect your private information

Lex Fridman (1:45:08.760)

if the robot decides to leave?

Yann LeCun (1:45:09.960)

I have this intuition that for robots with certain,

Lex Fridman (1:45:14.520)

like, it's almost like a regulation.

Yann LeCun (1:45:16.820)

If you declare your robot to be,

Lex Fridman (1:45:19.240)

let's call it sentient or something like that,

Yann LeCun (1:45:20.960)

like this robot is designed for human interaction,

Lex Fridman (1:45:24.180)

then you're not allowed to murder these robots.

Yann LeCun (1:45:26.040)

It's the same as murdering other humans.

Lex Fridman (1:45:28.160)

Well, but what about you do a backup of the robot

Yann LeCun (1:45:30.280)

that you preserve on a hard drive

Lex Fridman (1:45:32.600)

for the equivalent in the future?

Yann LeCun (1:45:33.880)

That might be illegal.

Lex Fridman (1:45:34.720)

It's like piracy is illegal.

Lex Fridman (1:45:38.080)

No, but it's your own robot, right?

Lex Fridman (1:45:39.800)

But you can't, you don't.

Lex Fridman (1:45:41.640)

But then you can wipe out his brain.

Lex Fridman (1:45:45.040)

So this robot doesn't know anything about you anymore,

Lex Fridman (1:45:47.440)

but you still have, technically it's still in existence

Lex Fridman (1:45:50.440)

because you backed it up.

Lex Fridman (1:45:51.700)

And then there'll be these great speeches

Lex Fridman (1:45:53.560)

at the Supreme Court by saying,

Yann LeCun (1:45:55.480)

oh, sure, you can erase the mind of the robot

Lex Fridman (1:45:57.840)

just like you can erase the mind of a human.

Yann LeCun (1:46:00.060)

We both can suffer.

Lex Fridman (1:46:01.100)

There'll be some epic like Obama type character

Yann LeCun (1:46:03.360)

with a speech that we,

Lex Fridman (1:46:05.680)

like the robots and the humans are the same.

Yann LeCun (1:46:08.840)

We can both suffer.

Lex Fridman (1:46:09.880)

We can both hope.

Yann LeCun (1:46:11.380)

We can both, all of those kinds of things,

Lex Fridman (1:46:14.880)

raise families, all that kind of stuff.

Yann LeCun (1:46:17.280)

It's interesting for these, just like you said,

Lex Fridman (1:46:20.140)

emotion seems to be a fascinatingly powerful aspect

Yann LeCun (1:46:24.200)

of human interaction, human robot interaction.

Lex Fridman (1:46:27.360)

And if they're able to exhibit emotions

Yann LeCun (1:46:30.480)

at the end of the day,

Lex Fridman (1:46:31.800)

that's probably going to have us deeply consider

Yann LeCun (1:46:35.920)

human rights, like what we value in humans,

Lex Fridman (1:46:38.480)

what we value in other animals.

Yann LeCun (1:46:40.320)

That's why robots and AI is great.

Lex Fridman (1:46:42.120)

It makes us ask really good questions.

Yann LeCun (1:46:44.280)

The hard questions, yeah.

Lex Fridman (1:46:45.480)

But you asked about the Chinese room type argument.

Lex Fridman (1:46:49.560)

Is it real?

Lex Fridman (1:46:50.400)

If it looks real.

Yann LeCun (1:46:51.480)

I think the Chinese room argument is a really good one.

Lex Fridman (1:46:54.400)

So.

Lex Fridman (1:46:55.440)

So for people who don't know what Chinese room is,

Lex Fridman (1:46:58.440)

you can, I don't even know how to formulate it well,

Lex Fridman (1:47:00.740)

but basically you can mimic the behavior

Lex Fridman (1:47:04.620)

of an intelligence system by just following

Yann LeCun (1:47:06.760)

a giant algorithm code book that tells you exactly

Lex Fridman (1:47:10.680)

how to respond in exactly each case.

Lex Fridman (1:47:12.880)

But is that really intelligent?

Lex Fridman (1:47:14.700)

It's like a giant lookup table.

Yann LeCun (1:47:16.600)

When this person says this, you answer this.

Lex Fridman (1:47:18.580)

When this person says this, you answer this.

Lex Fridman (1:47:21.000)

And if you understand how that works,

Lex Fridman (1:47:24.320)

you have this giant, nearly infinite lookup table.

Lex Fridman (1:47:27.360)

Is that really intelligence?

Lex Fridman (1:47:28.600)

Cause intelligence seems to be a mechanism

Yann LeCun (1:47:31.280)

that's much more interesting and complex

Lex Fridman (1:47:33.440)

than this lookup table.

Yann LeCun (1:47:34.620)

I don't think so.

Lex Fridman (1:47:35.460)

So the, I mean, the real question comes down to,

Lex Fridman (1:47:38.960)

do you think, you know, you can,

Lex Fridman (1:47:42.080)

you can mechanize intelligence in some way,

Lex Fridman (1:47:44.320)

even if that involves learning?

Lex Fridman (1:47:47.560)

And the answer is, of course, yes, there's no question.

Yann LeCun (1:47:50.720)

There's a second question then, which is,

Lex Fridman (1:47:53.400)

assuming you can reproduce intelligence

Yann LeCun (1:47:56.560)

in sort of different hardware than biological hardware,

Lex Fridman (1:47:59.400)

you know, like computers, can you, you know,

Yann LeCun (1:48:04.440)

match human intelligence in all the domains

Lex Fridman (1:48:09.600)

in which humans are intelligent?

Lex Fridman (1:48:12.920)

Is it possible, right?

Lex Fridman (1:48:13.920)

So that's the hypothesis of strong AI.

Yann LeCun (1:48:17.040)

The answer to this, in my opinion, is an unqualified yes.

Lex Fridman (1:48:20.700)

This will as well happen at some point.

Yann LeCun (1:48:22.640)

There's no question that machines at some point

Lex Fridman (1:48:25.300)

will become more intelligent than humans

Yann LeCun (1:48:26.640)

in all domains where humans are intelligent.

Lex Fridman (1:48:28.640)

This is not for tomorrow.

Yann LeCun (1:48:30.200)

It is going to take a long time,

Lex Fridman (1:48:32.240)

regardless of what, you know,

Yann LeCun (1:48:34.800)

Elon and others have claimed or believed.

Lex Fridman (1:48:38.120)

This is a lot harder than many of those guys think it is.

Lex Fridman (1:48:43.480)

And many of those guys who thought it was simpler than that

Lex Fridman (1:48:45.800)

years, you know, five years ago,

Yann LeCun (1:48:47.480)

now think it's hard because it's been five years

Lex Fridman (1:48:49.920)

and they realize it's going to take a lot longer.

Yann LeCun (1:48:53.460)

That includes a bunch of people at DeepMind, for example.

Lex Fridman (1:48:55.200)

But...

Yann LeCun (1:48:56.160)

Oh, interesting.

Lex Fridman (1:48:57.000)

I haven't actually touched base with the DeepMind folks,

Lex Fridman (1:48:59.320)

but some of it, Elon or Demis Hassabis.

Lex Fridman (1:49:03.280)

I mean, sometimes in your role,

Yann LeCun (1:49:05.800)

you have to kind of create deadlines

Lex Fridman (1:49:08.780)

that are nearer than farther away

Yann LeCun (1:49:10.720)

to kind of create an urgency.

Lex Fridman (1:49:12.800)

Because, you know, you have to believe the impossible

Yann LeCun (1:49:14.600)

is possible in order to accomplish it.

Lex Fridman (1:49:16.200)

And there's, of course, a flip side to that coin,

Lex Fridman (1:49:18.520)

but it's a weird, you can't be too cynical

Lex Fridman (1:49:21.280)

if you want to get something done.

Yann LeCun (1:49:22.400)

Absolutely.

Lex Fridman (1:49:23.360)

I agree with that.

Lex Fridman (1:49:24.280)

But, I mean, you have to inspire people, right?

Lex Fridman (1:49:26.920)

To work on sort of ambitious things.

Yann LeCun (1:49:31.400)

So, you know, it's certainly a lot harder than we believe,

Lex Fridman (1:49:35.620)

but there's no question in my mind that this will happen.

Lex Fridman (1:49:38.200)

And now, you know, people are kind of worried about

Lex Fridman (1:49:40.300)

what does that mean for humans?

Yann LeCun (1:49:42.480)

They are going to be brought down from their pedestal,

Lex Fridman (1:49:45.160)

you know, a bunch of notches with that.

Lex Fridman (1:49:47.980)

And, you know, is that going to be good or bad?

Lex Fridman (1:49:51.740)

I mean, it's just going to give more power, right?

Yann LeCun (1:49:53.480)

It's an amplifier for human intelligence, really.

Lex Fridman (1:49:56.200)

So, speaking of doing cool, ambitious things,

Yann LeCun (1:49:59.720)

FAIR, the Facebook AI research group,

Lex Fridman (1:50:02.920)

has recently celebrated its eighth birthday.

Yann LeCun (1:50:05.520)

Or, maybe you can correct me on that.

Lex Fridman (1:50:08.640)

Looking back, what has been the successes, the failures,

Lex Fridman (1:50:12.400)

the lessons learned from the eight years of FAIR?

Lex Fridman (1:50:14.440)

And maybe you can also give context of

Yann LeCun (1:50:16.600)

where does the newly minted meta AI fit into,

Lex Fridman (1:50:21.320)

how does it relate to FAIR?

Yann LeCun (1:50:22.640)

Right, so let me tell you a little bit

Lex Fridman (1:50:23.800)

about the organization of all this.

Yann LeCun (1:50:26.760)

Yeah, FAIR was created almost exactly eight years ago.

Lex Fridman (1:50:30.060)

It wasn't called FAIR yet.

Yann LeCun (1:50:31.240)

It took that name a few months later.

Lex Fridman (1:50:34.680)

And at the time I joined Facebook,

Yann LeCun (1:50:37.760)

there was a group called the AI group

Lex Fridman (1:50:39.520)

that had about 12 engineers and a few scientists,

Yann LeCun (1:50:43.560)

like, you know, 10 engineers and two scientists

Lex Fridman (1:50:45.480)

or something like that.

Yann LeCun (1:50:47.080)

I ran it for three and a half years as a director,

Lex Fridman (1:50:50.680)

you know, hired the first few scientists

Lex Fridman (1:50:52.380)

and kind of set up the culture and organized it,

Lex Fridman (1:50:55.040)

you know, explained to the Facebook leadership

Lex Fridman (1:50:57.880)

what fundamental research was about

Lex Fridman (1:51:00.200)

and how it can work within industry

Lex Fridman (1:51:03.640)

and how it needs to be open and everything.

Lex Fridman (1:51:07.240)

And I think it's been an unqualified success

Yann LeCun (1:51:12.360)

in the sense that FAIR has simultaneously produced,

Lex Fridman (1:51:17.800)

you know, top level research

Lex Fridman (1:51:19.560)

and advanced the science and the technology,

Lex Fridman (1:51:21.640)

provided tools, open source tools,

Yann LeCun (1:51:23.480)

like PyTorch and many others,

Lex Fridman (1:51:26.680)

but at the same time has had a direct

Yann LeCun (1:51:29.880)

or mostly indirect impact on Facebook at the time,

Lex Fridman (1:51:34.680)

now Meta, in the sense that a lot of systems

Yann LeCun (1:51:38.580)

that Meta is built around now are based

Lex Fridman (1:51:43.600)

on research projects that started at FAIR.

Lex Fridman (1:51:48.360)

And so if you were to take out, you know,

Lex Fridman (1:51:49.640)

deep learning out of Facebook services now

Lex Fridman (1:51:52.840)

and Meta more generally,

Lex Fridman (1:51:55.140)

I mean, the company would literally crumble.

Yann LeCun (1:51:57.760)

I mean, it's completely built around AI these days.

Lex Fridman (1:52:01.480)

And it's really essential to the operations.

Lex Fridman (1:52:04.000)

So what happened after three and a half years

Lex Fridman (1:52:06.640)

is that I changed role, I became chief scientist.

Lex Fridman (1:52:10.200)

So I'm not doing day to day management of FAIR anymore.

Lex Fridman (1:52:14.880)

I'm more of a kind of, you know,

Yann LeCun (1:52:17.120)

think about strategy and things like that.

Lex Fridman (1:52:18.880)

And I carry my, I conduct my own research.

Yann LeCun (1:52:21.440)

I have, you know, my own kind of research group

Lex Fridman (1:52:23.320)

working on self supervised learning and things like this,

Yann LeCun (1:52:25.320)

which I didn't have time to do when I was director.

Lex Fridman (1:52:28.240)

So now FAIR is run by Joel Pinot and Antoine Bord together

Yann LeCun (1:52:34.720)

because FAIR is kind of split in two now.

Lex Fridman (1:52:36.360)

There's something called FAIR Labs,

Yann LeCun (1:52:37.860)

which is sort of bottom up science driven research

Lex Fridman (1:52:40.940)

and FAIR Excel, which is slightly more organized

Yann LeCun (1:52:43.460)

for bigger projects that require a little more

Lex Fridman (1:52:46.440)

kind of focus and more engineering support

Lex Fridman (1:52:49.040)

and things like that.

Lex Fridman (1:52:49.880)

So Joel needs FAIR Lab and Antoine Bord needs FAIR Excel.

Lex Fridman (1:52:52.920)

Where are they located?

Lex Fridman (1:52:54.520)

It's delocalized all over.

Lex Fridman (1:52:58.000)

So there's no question that the leadership of the company

Lex Fridman (1:53:02.540)

believes that this was a very worthwhile investment.

Lex Fridman (1:53:06.560)

And what that means is that it's there for the long run.

Lex Fridman (1:53:12.840)

Right?

Lex Fridman (1:53:13.680)

So if you want to talk in these terms, which I don't like,

Lex Fridman (1:53:17.720)

this is a business model, if you want,

Yann LeCun (1:53:19.560)

where FAIR, despite being a very fundamental research lab

Lex Fridman (1:53:23.680)

brings a lot of value to the company,

Yann LeCun (1:53:25.320)

either mostly indirectly through other groups.

Lex Fridman (1:53:29.920)

Now what happened three and a half years ago

Yann LeCun (1:53:31.600)

when I stepped down was also the creation of Facebook AI,

Lex Fridman (1:53:34.640)

which was basically a larger organization

Yann LeCun (1:53:37.700)

that covers FAIR, so FAIR is included in it,

Lex Fridman (1:53:41.740)

but also has other organizations

Yann LeCun (1:53:43.880)

that are focused on applied research

Lex Fridman (1:53:47.840)

or advanced development of AI technology

Yann LeCun (1:53:51.220)

that is more focused on the products of the company.

Lex Fridman (1:53:54.680)

So less emphasis on fundamental research.

Yann LeCun (1:53:56.640)

Less fundamental, but it's still research.

Lex Fridman (1:53:58.220)

I mean, there's a lot of papers coming out

Yann LeCun (1:53:59.760)

of those organizations and the people are awesome

Lex Fridman (1:54:03.960)

and wonderful to interact with.

Lex Fridman (1:54:06.400)

But it serves as kind of a way

Lex Fridman (1:54:10.680)

to kind of scale up if you want sort of AI technology,

Yann LeCun (1:54:15.720)

which, you know, may be very experimental

Lex Fridman (1:54:17.600)

and sort of lab prototypes into things that are usable.

Lex Fridman (1:54:20.600)

So FAIR is a subset of Meta AI.

Lex Fridman (1:54:23.040)

Is FAIR become like KFC?

Yann LeCun (1:54:24.800)

It'll just keep the F.

Lex Fridman (1:54:26.520)

Nobody cares what the F stands for.

Yann LeCun (1:54:29.440)

We'll know soon enough, probably by the end of 2021.

Lex Fridman (1:54:35.600)

I guess it's not a giant change, Mare, FAIR.

Yann LeCun (1:54:38.400)

Well, Mare doesn't sound too good,

Lex Fridman (1:54:39.520)

but the brand people are kind of deciding on this

Lex Fridman (1:54:43.560)

and they've been hesitating for a while now.

Lex Fridman (1:54:45.860)

And they tell us they're going to come up with an answer

Yann LeCun (1:54:48.480)

as to whether FAIR is going to change name

Lex Fridman (1:54:50.440)

or whether we're going to change just the meaning of the F.

Yann LeCun (1:54:53.480)

That's a good call.

Lex Fridman (1:54:54.300)

I would keep FAIR and change the meaning of the F.

Yann LeCun (1:54:56.160)

That would be my preference.

Lex Fridman (1:54:57.600)

I would turn the F into fundamental AI research.

Yann LeCun (1:55:02.280)

Oh, that's really good.

Lex Fridman (1:55:03.120)

Within Meta AI.

Lex Fridman (1:55:04.280)

So this would be meta FAIR,

Lex Fridman (1:55:06.720)

but people will call it FAIR, right?

Yann LeCun (1:55:08.320)

Yeah, exactly.

Lex Fridman (1:55:09.320)

I like it.

Lex Fridman (1:55:10.160)

And now Meta AI is part of the Reality Lab.

Lex Fridman (1:55:16.680)

So Meta now, the new Facebook is called Meta

Lex Fridman (1:55:21.760)

and it's kind of divided into Facebook, Instagram, WhatsApp

Lex Fridman (1:55:30.400)

and Reality Lab.

Lex Fridman (1:55:32.920)

And Reality Lab is about AR, VR, telepresence,

Lex Fridman (1:55:37.920)

communication technology and stuff like that.

Yann LeCun (1:55:40.520)

It's kind of the, you can think of it as the sort of,

Lex Fridman (1:55:44.200)

a combination of sort of new products

Lex Fridman (1:55:47.920)

and technology part of Meta.

Lex Fridman (1:55:51.960)

Is that where the touch sensing for robots,

Yann LeCun (1:55:54.240)

I saw that you were posting about that.

Lex Fridman (1:55:56.120)

Touch sensing for robot is part of FAIR actually.

Yann LeCun (1:55:58.240)

That's a FAIR project.

Lex Fridman (1:55:59.080)

Oh, it is.

Yann LeCun (1:55:59.920)

Okay, cool.

Lex Fridman (1:56:00.740)

Yeah, this is also the, no, but there is the other way,

Lex Fridman (1:56:03.040)

the haptic glove, right?

Lex Fridman (1:56:05.680)

Yes, that's more Reality Lab.

Yann LeCun (1:56:07.640)

That's Reality Lab research.

Lex Fridman (1:56:10.760)

Reality Lab research.

Yann LeCun (1:56:11.960)

By the way, the touch sensors are super interesting.

Lex Fridman (1:56:14.400)

Like integrating that modality

Yann LeCun (1:56:16.120)

into the whole sensing suite is very interesting.

Lex Fridman (1:56:20.120)

So what do you think about the Metaverse?

Lex Fridman (1:56:23.680)

What do you think about this whole kind of expansion

Lex Fridman (1:56:27.820)

of the view of the role of Facebook and Meta in the world?

Yann LeCun (1:56:30.920)

Well, Metaverse really should be thought of

Lex Fridman (1:56:32.520)

as the next step in the internet, right?

Yann LeCun (1:56:35.360)

Sort of trying to kind of make the experience

Lex Fridman (1:56:41.760)

more compelling of being connected

Yann LeCun (1:56:46.280)

either with other people or with content.

Lex Fridman (1:56:49.520)

And we are evolved and trained to evolve

Yann LeCun (1:56:54.000)

in 3D environments where we can see other people.

Lex Fridman (1:56:58.680)

We can talk to them when we're near them

Yann LeCun (1:57:01.080)

or an other viewer far away can't hear us,

Lex Fridman (1:57:04.360)

things like that, right?

Lex Fridman (1:57:05.200)

So there's a lot of social conventions

Lex Fridman (1:57:08.080)

that exist in the real world that we can try to transpose.

Yann LeCun (1:57:10.800)

Now, what is going to be eventually the,

Lex Fridman (1:57:15.120)

how compelling is it going to be?

Yann LeCun (1:57:16.240)

Like, is it going to be the case

Lex Fridman (1:57:18.740)

that people are going to be willing to do this

Lex Fridman (1:57:21.300)

if they have to wear a huge pair of goggles all day?

Lex Fridman (1:57:24.600)

Maybe not.

Lex Fridman (1:57:26.400)

But then again, if the experience

Lex Fridman (1:57:27.480)

is sufficiently compelling, maybe so.

Yann LeCun (1:57:30.320)

Or if the device that you have to wear

Lex Fridman (1:57:32.200)

is just basically a pair of glasses,

Lex Fridman (1:57:34.560)

and technology makes sufficient progress for that.

Lex Fridman (1:57:38.400)

AR is a much easier concept to grasp

Yann LeCun (1:57:41.560)

that you're going to have augmented reality glasses

Lex Fridman (1:57:45.000)

that basically contain some sort of virtual assistant

Yann LeCun (1:57:48.640)

that can help you in your daily lives.

Lex Fridman (1:57:50.280)

But at the same time with the AR,

Yann LeCun (1:57:51.920)

you have to contend with reality.

Lex Fridman (1:57:53.480)

With VR, you can completely detach yourself from reality.

Lex Fridman (1:57:55.880)

So it gives you freedom.

Lex Fridman (1:57:57.200)

It might be easier to design worlds in VR.

Yann LeCun (1:58:00.360)

Yeah, but you can imagine the metaverse

Lex Fridman (1:58:02.900)

being a mix, right?

Yann LeCun (1:58:06.520)

Or like, you can have objects that exist in the metaverse

Lex Fridman (1:58:09.280)

that pop up on top of the real world,

Yann LeCun (1:58:11.200)

or only exist in virtual reality.

Lex Fridman (1:58:14.380)

Okay, let me ask the hard question.

Yann LeCun (1:58:17.080)

Oh, because all of this was easy so far.

Lex Fridman (1:58:18.520)

This was easy.

Yann LeCun (1:58:20.680)

The Facebook, now Meta, the social network

Lex Fridman (1:58:24.280)

has been painted by the media as a net negative for society,

Yann LeCun (1:58:28.280)

even destructive and evil at times.

Lex Fridman (1:58:30.840)

You've pushed back against this, defending Facebook.

Lex Fridman (1:58:34.080)

Can you explain your defense?

Lex Fridman (1:58:36.560)

Yeah, so the description,

Yann LeCun (1:58:38.640)

the company that is being described in some media

Lex Fridman (1:58:43.960)

is not the company we know when we work inside.

Lex Fridman (1:58:47.360)

And it could be claimed that a lot of employees

Lex Fridman (1:58:52.080)

are uninformed about what really goes on in the company,

Lex Fridman (1:58:54.600)

but I'm a vice president.

Lex Fridman (1:58:56.520)

I mean, I have a pretty good vision of what goes on.

Yann LeCun (1:58:58.920)

I don't know everything, obviously.

Lex Fridman (1:59:00.200)

I'm not involved in everything,

Lex Fridman (1:59:01.860)

but certainly not in decision about content moderation

Lex Fridman (1:59:05.320)

or anything like this,

Lex Fridman (1:59:06.160)

but I have some decent vision of what goes on.

Lex Fridman (1:59:10.160)

And this evil that is being described, I just don't see it.

Lex Fridman (1:59:13.660)

And then I think there is an easy story to buy,

Lex Fridman (1:59:18.200)

which is that all the bad things in the world

Lex Fridman (1:59:21.760)

and the reason your friend believe crazy stuff,

Lex Fridman (1:59:25.160)

there's an easy scapegoat in social media in general,

Yann LeCun (1:59:32.800)

Facebook in particular.

Lex Fridman (1:59:34.480)

But you have to look at the data.

Yann LeCun (1:59:35.720)

Is it the case that Facebook, for example,

Lex Fridman (1:59:40.080)

polarizes people politically?

Lex Fridman (1:59:42.720)

Are there academic studies that show this?

Lex Fridman (1:59:45.220)

Is it the case that teenagers think of themselves less

Lex Fridman (1:59:50.280)

if they use Instagram more?

Lex Fridman (1:59:52.160)

Is it the case that people get more riled up

Yann LeCun (1:59:57.280)

against opposite sides in a debate or political opinion

Lex Fridman (20:00.560)

but doing this kind of fill in the gap kind of learning

Lex Fridman (20:03.320)

and just kind of updating the model constantly

Lex Fridman (20:05.720)

in order to be able to support the raw sensory information

Yann LeCun (20:09.240)

to predict it and then adjust to the prediction

Lex Fridman (20:11.360)

when it's wrong.

Lex Fridman (20:12.400)

But like when we look at our brain at the high level,

Lex Fridman (20:15.840)

it feels like we're doing, like we're playing chess,

Yann LeCun (20:18.320)

like we're like playing with high level concepts

Lex Fridman (20:22.240)

and we're stitching them together

Lex Fridman (20:23.680)

and we're putting them into longterm memory.

Lex Fridman (20:26.000)

But really what's going underneath

Yann LeCun (20:28.280)

is something we're not able to introspect,

Lex Fridman (20:30.160)

which is this kind of simple, large neural network

Yann LeCun (20:34.440)

that's just filling in the gaps.

Lex Fridman (20:36.000)

Right, well, okay.

Lex Fridman (20:37.120)

So there's a lot of questions and a lot of answers there.

Lex Fridman (20:39.760)

Okay, so first of all,

Yann LeCun (20:40.600)

there's a whole school of thought in neuroscience,

Lex Fridman (20:42.680)

computational neuroscience in particular,

Yann LeCun (20:45.240)

that likes the idea of predictive coding,

Lex Fridman (20:47.760)

which is really related to the idea

Yann LeCun (20:50.080)

I was talking about in self supervised learning.

Lex Fridman (20:52.040)

So everything is about prediction.

Yann LeCun (20:53.520)

The essence of intelligence is the ability to predict

Lex Fridman (20:56.320)

and everything the brain does is trying to predict,

Yann LeCun (20:59.920)

predict everything from everything else.

Lex Fridman (21:02.120)

Okay, and that's really sort of the underlying principle,

Yann LeCun (21:04.760)

if you want, that self supervised learning

Lex Fridman (21:07.800)

is trying to kind of reproduce this idea of prediction

Yann LeCun (21:10.640)

as kind of an essential mechanism

Lex Fridman (21:13.080)

of task independent learning, if you want.

Yann LeCun (21:16.320)

The next step is what kind of intelligence

Lex Fridman (21:19.320)

are you interested in reproducing?

Lex Fridman (21:21.120)

And of course, we all think about trying to reproduce

Lex Fridman (21:24.640)

sort of high level cognitive processes in humans,

Lex Fridman (21:28.320)

but like with machines, we're not even at the level

Lex Fridman (21:30.400)

of even reproducing the learning processes in a cat brain.

Yann LeCun (21:37.160)

The most intelligent or intelligent systems

Lex Fridman (21:39.360)

don't have as much common sense as a house cat.

Lex Fridman (21:43.200)

So how is it that cats learn?

Lex Fridman (21:45.160)

And cats don't do a whole lot of reasoning.

Yann LeCun (21:47.920)

They certainly have causal models.

Lex Fridman (21:49.600)

They certainly have, because many cats can figure out

Lex Fridman (21:53.600)

how they can act on the world to get what they want.

Lex Fridman (21:56.600)

They certainly have a fantastic model of intuitive physics,

Yann LeCun (22:01.800)

certainly the dynamics of their own bodies,

Lex Fridman (22:04.560)

but also of praise and things like that.

Lex Fridman (22:06.880)

So they're pretty smart.

Lex Fridman (22:09.880)

They only do this with about 800 million neurons.

Yann LeCun (22:12.400)

We are not anywhere close to reproducing this kind of thing.

Lex Fridman (22:17.920)

So to some extent, I could say,

Yann LeCun (22:21.320)

let's not even worry about like the high level cognition

Lex Fridman (22:26.280)

and kind of longterm planning and reasoning

Yann LeCun (22:27.960)

that humans can do until we figure out like,

Lex Fridman (22:30.120)

can we even reproduce what cats are doing?

Yann LeCun (22:32.520)

Now that said, this ability to learn world models,

Lex Fridman (22:37.000)

I think is the key to the possibility of learning machines

Yann LeCun (22:41.560)

that can also reason.

Lex Fridman (22:43.160)

So whenever I give a talk, I say there are three challenges

Yann LeCun (22:45.640)

in the three main challenges in machine learning.

Lex Fridman (22:47.320)

The first one is getting machines to learn

Yann LeCun (22:49.920)

to represent the world

Lex Fridman (22:51.800)

and I'm proposing self supervised learning.

Yann LeCun (22:54.840)

The second is getting machines to reason

Lex Fridman (22:58.000)

in ways that are compatible

Yann LeCun (22:59.240)

with essentially gradient based learning

Lex Fridman (23:01.640)

because this is what deep learning is all about really.

Lex Fridman (23:05.280)

And the third one is something

Lex Fridman (23:06.640)

we have no idea how to solve,

Yann LeCun (23:07.640)

at least I have no idea how to solve

Lex Fridman (23:09.480)

is can we get machines to learn hierarchical representations

Lex Fridman (23:14.360)

of action plans?

Lex Fridman (23:17.920)

We know how to train them

Yann LeCun (23:18.760)

to learn hierarchical representations of perception

Lex Fridman (23:22.200)

with convolutional nets and things like that

Lex Fridman (23:23.680)

and transformers, but what about action plans?

Lex Fridman (23:26.040)

Can we get them to spontaneously learn

Lex Fridman (23:28.280)

good hierarchical representations of actions?

Lex Fridman (23:30.480)

Also gradient based.

Yann LeCun (23:32.400)

Yeah, all of that needs to be somewhat differentiable

Lex Fridman (23:35.880)

so that you can apply sort of gradient based learning,

Yann LeCun (23:38.720)

which is really what deep learning is about.

Lex Fridman (23:42.080)

So it's background, knowledge, ability to reason

Yann LeCun (23:46.760)

in a way that's differentiable

Lex Fridman (23:50.520)

that is somehow connected, deeply integrated

Yann LeCun (23:53.840)

with that background knowledge

Lex Fridman (23:55.480)

or builds on top of that background knowledge

Lex Fridman (23:57.600)

and then given that background knowledge

Lex Fridman (23:59.120)

be able to make hierarchical plans in the world.

Lex Fridman (24:02.360)

So if you take classical optimal control,

Lex Fridman (24:05.480)

there's something in classical optimal control

Yann LeCun (24:07.000)

called model predictive control.

Lex Fridman (24:10.520)

And it's been around since the early sixties.

Yann LeCun (24:13.840)

NASA uses that to compute trajectories of rockets.

Lex Fridman (24:16.840)

And the basic idea is that you have a predictive model

Yann LeCun (24:20.600)

of the rocket, let's say,

Lex Fridman (24:21.840)

or whatever system you intend to control,

Yann LeCun (24:25.440)

which given the state of the system at time T

Lex Fridman (24:28.360)

and given an action that you're taking the system.

Lex Fridman (24:31.640)

So for a rocket to be thrust

Lex Fridman (24:33.520)

and all the controls you can have,

Yann LeCun (24:35.600)

it gives you the state of the system

Lex Fridman (24:37.280)

at time T plus Delta T, right?

Lex Fridman (24:38.800)

So basically a differential equation, something like that.

Lex Fridman (24:43.520)

And if you have this model

Lex Fridman (24:45.240)

and you have this model in the form of some sort of neural net

Lex Fridman (24:48.720)

or some sort of a set of formula

Yann LeCun (24:50.960)

that you can back propagate gradient through,

Lex Fridman (24:52.920)

you can do what's called model predictive control

Yann LeCun (24:55.240)

or gradient based model predictive control.

Lex Fridman (24:57.680)

So you can unroll that model in time.

Yann LeCun (25:02.680)

You feed it a hypothesized sequence of actions.

Lex Fridman (25:08.080)

And then you have some objective function

Yann LeCun (25:10.760)

that measures how well at the end of the trajectory,

Lex Fridman (25:13.240)

the system has succeeded or matched what you wanted to do.

Lex Fridman (25:17.240)

Is it a robot harm?

Lex Fridman (25:18.280)

Have you grasped the object you want to grasp?

Yann LeCun (25:20.680)

If it's a rocket, are you at the right place

Lex Fridman (25:23.360)

near the space station, things like that.

Lex Fridman (25:26.120)

And by back propagation through time,

Lex Fridman (25:28.040)

and again, this was invented in the 1960s,

Yann LeCun (25:30.080)

by optimal control theorists, you can figure out

Lex Fridman (25:34.040)

what is the optimal sequence of actions

Yann LeCun (25:36.160)

that will get my system to the best final state.

Lex Fridman (25:42.040)

So that's a form of reasoning.

Yann LeCun (25:44.560)

It's basically planning.

Lex Fridman (25:45.640)

And a lot of planning systems in robotics

Yann LeCun (25:48.160)

are actually based on this.

Lex Fridman (25:49.600)

And you can think of this as a form of reasoning.

Lex Fridman (25:53.160)

So to take the example of the teenager driving a car,

Lex Fridman (25:57.040)

you have a pretty good dynamical model of the car.

Yann LeCun (26:00.120)

It doesn't need to be very accurate.

Lex Fridman (26:01.280)

But you know, again, that if you turn the wheel

Yann LeCun (26:03.840)

to the right and there is a cliff,

Lex Fridman (26:05.080)

you're gonna run off the cliff, right?

Yann LeCun (26:06.520)

You don't need to have a very accurate model

Lex Fridman (26:08.000)

to predict that.

Lex Fridman (26:09.080)

And you can run this in your mind

Lex Fridman (26:10.640)

and decide not to do it for that reason.

Yann LeCun (26:13.080)

Because you can predict in advance

Lex Fridman (26:14.480)

that the result is gonna be bad.

Lex Fridman (26:15.600)

So you can sort of imagine different scenarios

Lex Fridman (26:17.960)

and then employ or take the first step

Yann LeCun (26:21.560)

in the scenario that is most favorable

Lex Fridman (26:23.360)

and then repeat the process again.

Yann LeCun (26:24.960)

The scenario that is most favorable

Lex Fridman (26:27.120)

and then repeat the process of planning.

Yann LeCun (26:28.480)

That's called receding horizon model predictive control.

Lex Fridman (26:31.280)

So even all those things have names going back decades.

Lex Fridman (26:36.480)

And so if you're not a classical optimal control,

Lex Fridman (26:40.680)

the model of the world is not generally learned.

Yann LeCun (26:44.360)

Sometimes a few parameters you have to identify.

Lex Fridman (26:46.240)

That's called systems identification.

Lex Fridman (26:47.800)

But generally, the model is mostly deterministic

Lex Fridman (26:52.640)

and mostly built by hand.

Lex Fridman (26:53.920)

So the question of AI,

Lex Fridman (26:55.920)

I think the big challenge of AI for the next decade

Yann LeCun (26:58.760)

is how do we get machines to learn predictive models

Lex Fridman (27:01.120)

of the world that deal with uncertainty

Lex Fridman (27:03.720)

and deal with the real world in all this complexity?

Lex Fridman (27:05.840)

So it's not just the trajectory of a rocket,

Yann LeCun (27:08.160)

which you can reduce to first principles.

Lex Fridman (27:10.240)

It's not even just the trajectory of a robot arm,

Yann LeCun (27:13.040)

which again, you can model by careful mathematics.

Lex Fridman (27:16.320)

But it's everything else,

Yann LeCun (27:17.200)

everything we observe in the world.

Lex Fridman (27:18.880)

People, behavior,

Yann LeCun (27:20.120)

physical systems that involve collective phenomena,

Lex Fridman (27:25.800)

like water or trees and branches in a tree or something

Yann LeCun (27:31.880)

or complex things that humans have no trouble

Lex Fridman (27:36.680)

developing abstract representations

Lex Fridman (27:38.520)

and predictive model for,

Lex Fridman (27:39.840)

but we still don't know how to do with machines.

Yann LeCun (27:41.600)

Where do you put in these three,

Lex Fridman (27:43.880)

maybe in the planning stages,

Yann LeCun (27:46.180)

the game theoretic nature of this world,

Lex Fridman (27:50.660)

where your actions not only respond

Yann LeCun (27:52.980)

to the dynamic nature of the world, the environment,

Lex Fridman (27:55.540)

but also affect it.

Lex Fridman (27:57.500)

So if there's other humans involved,

Lex Fridman (27:59.860)

is this point number four,

Yann LeCun (28:02.220)

or is it somehow integrated

Lex Fridman (28:03.420)

into the hierarchical representation of action

Lex Fridman (28:05.820)

in your view?

Lex Fridman (28:06.660)

I think it's integrated.

Yann LeCun (28:07.500)

It's just that now your model of the world has to deal with,

Lex Fridman (28:11.580)

it just makes it more complicated.

Yann LeCun (28:13.100)

The fact that humans are complicated

Lex Fridman (28:15.600)

and not easily predictable,

Yann LeCun (28:17.220)

that makes your model of the world much more complicated,

Lex Fridman (28:19.860)

that much more complicated.

Yann LeCun (28:21.340)

Well, there's a chess,

Lex Fridman (28:22.380)

I mean, I suppose chess is an analogy.

Lex Fridman (28:25.300)

So multicolored tree search.

Lex Fridman (28:28.860)

There's a, I go, you go, I go, you go.

Yann LeCun (28:32.040)

Like Andre Capote recently gave a talk at MIT

Lex Fridman (28:35.580)

about car doors.

Yann LeCun (28:37.900)

I think there's some machine learning too,

Lex Fridman (28:39.280)

but mostly car doors.

Lex Fridman (28:40.780)

And there's a dynamic nature to the car,

Lex Fridman (28:43.340)

like the person opening the door,

Yann LeCun (28:44.700)

checking, I mean, he wasn't talking about that.

Lex Fridman (28:46.900)

He was talking about the perception problem

Yann LeCun (28:48.420)

of what the ontology of what defines a car door,

Lex Fridman (28:50.940)

this big philosophical question.

Lex Fridman (28:52.940)

But to me, it was interesting

Lex Fridman (28:54.060)

because it's obvious that the person opening the car doors,

Yann LeCun (28:57.300)

they're trying to get out, like here in New York,

Lex Fridman (28:59.580)

trying to get out of the car.

Yann LeCun (29:01.400)

You slowing down is going to signal something.

Lex Fridman (29:03.580)

You speeding up is gonna signal something,

Lex Fridman (29:05.380)

and that's a dance.

Lex Fridman (29:06.460)

It's a asynchronous chess game.

Yann LeCun (29:10.140)

I don't know.

Lex Fridman (29:10.980)

So it feels like it's not just,

Yann LeCun (29:16.900)

I mean, I guess you can integrate all of them

Lex Fridman (29:18.780)

to one giant model, like the entirety

Yann LeCun (29:21.300)

of these little interactions.

Lex Fridman (29:24.340)

Because it's not as complicated as chess.

Yann LeCun (29:25.740)

It's just like a little dance.

Lex Fridman (29:27.120)

We do like a little dance together,

Lex Fridman (29:28.800)

and then we figure it out.

Lex Fridman (29:29.980)

Well, in some ways it's way more complicated than chess

Yann LeCun (29:32.500)

because it's continuous, it's uncertain

Lex Fridman (29:36.020)

in a continuous manner.

Yann LeCun (29:38.220)

It doesn't feel more complicated.

Lex Fridman (29:39.860)

But it doesn't feel more complicated

Yann LeCun (29:41.060)

because that's what we've evolved to solve.

Lex Fridman (29:43.660)

This is the kind of problem we've evolved to solve.

Lex Fridman (29:45.480)

And so we're good at it

Lex Fridman (29:46.400)

because nature has made us good at it.

Yann LeCun (29:50.500)

Nature has not made us good at chess.

Lex Fridman (29:52.340)

We completely suck at chess.

Yann LeCun (29:55.700)

In fact, that's why we designed it as a game,

Lex Fridman (29:57.980)

is to be challenging.

Lex Fridman (2:00:02.680)

if they are more on Facebook or if they are less?

Lex Fridman (2:00:05.720)

And study after study show that none of this is true.

Yann LeCun (2:00:10.880)

This is independent studies by academic.

Lex Fridman (2:00:12.400)

They're not funded by Facebook or Meta.

Yann LeCun (2:00:15.880)

Study by Stanford, by some of my colleagues at NYU actually

Lex Fridman (2:00:18.640)

with whom I have no connection.

Yann LeCun (2:00:20.140)

There's a study recently, they paid people,

Lex Fridman (2:00:24.980)

I think it was in former Yugoslavia,

Yann LeCun (2:00:29.940)

I'm not exactly sure in what part,

Lex Fridman (2:00:31.820)

but they paid people to not use Facebook for a while

Yann LeCun (2:00:34.380)

in the period before the anniversary

Lex Fridman (2:00:40.240)

of the Srebrenica massacres.

Lex Fridman (2:00:43.540)

So people get riled up, like should we have a celebration?

Lex Fridman (2:00:47.800)

I mean, a memorial kind of celebration for it or not.

Lex Fridman (2:00:51.120)

So they paid a bunch of people

Lex Fridman (2:00:52.540)

to not use Facebook for a few weeks.

Lex Fridman (2:00:56.260)

And it turns out that those people ended up

Lex Fridman (2:00:59.580)

being more polarized than they were at the beginning

Lex Fridman (2:01:02.660)

and the people who were more on Facebook were less polarized.

Lex Fridman (2:01:06.660)

There's a study from Stanford of economists at Stanford

Yann LeCun (2:01:10.460)

that try to identify the causes

Lex Fridman (2:01:12.660)

of increasing polarization in the US.

Lex Fridman (2:01:16.000)

And it's been going on for 40 years

Lex Fridman (2:01:17.820)

before Mark Zuckerberg was born continuously.

Lex Fridman (2:01:22.540)

And so if there is a cause,

Lex Fridman (2:01:25.620)

it's not Facebook or social media.

Lex Fridman (2:01:27.620)

So you could say if social media just accelerated,

Lex Fridman (2:01:29.580)

but no, I mean, it's basically a continuous evolution

Yann LeCun (2:01:33.060)

by some measure of polarization in the US.

Lex Fridman (2:01:35.820)

And then you compare this with other countries

Yann LeCun (2:01:37.660)

like the West half of Germany

Lex Fridman (2:01:41.460)

because you can go 40 years in the East side

Yann LeCun (2:01:44.700)

or Denmark or other countries.

Lex Fridman (2:01:47.380)

And they use Facebook just as much

Lex Fridman (2:01:49.460)

and they're not getting more polarized,

Lex Fridman (2:01:50.700)

they're getting less polarized.

Lex Fridman (2:01:52.040)

So if you want to look for a causal relationship there,

Lex Fridman (2:01:57.640)

you can find a scapegoat, but you can't find a cause.

Yann LeCun (2:01:59.840)

Now, if you want to fix the problem,

Lex Fridman (2:02:01.720)

you have to find the right cause.

Lex Fridman (2:02:03.180)

And what rise me up is that people now are accusing Facebook

Lex Fridman (2:02:07.720)

of bad deeds that are done by others

Lex Fridman (2:02:09.300)

and those others are we're not doing anything about them.

Lex Fridman (2:02:12.380)

And by the way, those others include the owner

Yann LeCun (2:02:14.820)

of the Wall Street Journal

Lex Fridman (2:02:15.660)

in which all of those papers were published.

Lex Fridman (2:02:17.700)

So I should mention that I'm talking to Schrepp,

Lex Fridman (2:02:20.060)

Mike Schrepp on this podcast and also Mark Zuckerberg

Lex Fridman (2:02:23.460)

and probably these are conversations you can have with them

Lex Fridman (2:02:26.340)

because it's very interesting to me,

Yann LeCun (2:02:27.620)

even if Facebook has some measurable negative effect,

Lex Fridman (2:02:31.900)

you can't just consider that in isolation.

Yann LeCun (2:02:33.780)

You have to consider about all the positive ways

Lex Fridman (2:02:35.940)

that it connects us.

Lex Fridman (2:02:36.820)

So like every technology.

Lex Fridman (2:02:38.140)

It connects people, it's a question.

Yann LeCun (2:02:39.660)

You can't just say like there's an increase in division.

Lex Fridman (2:02:43.880)

Yes, probably Google search engine

Yann LeCun (2:02:46.100)

has created increase in division.

Lex Fridman (2:02:47.900)

But you have to consider about how much information

Yann LeCun (2:02:49.900)

are brought to the world.

Lex Fridman (2:02:51.140)

Like I'm sure Wikipedia created more division.

Yann LeCun (2:02:53.700)

If you just look at the division,

Lex Fridman (2:02:55.340)

we have to look at the full context of the world

Lex Fridman (2:02:57.700)

and they didn't make a better world.

Lex Fridman (2:02:59.100)

And you have to.

Lex Fridman (2:02:59.940)

The printing press has created more division, right?

Lex Fridman (2:03:01.660)

Exactly.

Yann LeCun (2:03:02.500)

I mean, so when the printing press was invented,

Lex Fridman (2:03:06.900)

the first books that were printed were things like the Bible

Lex Fridman (2:03:10.780)

and that allowed people to read the Bible by themselves,

Lex Fridman (2:03:13.780)

not get the message uniquely from priests in Europe.

Lex Fridman (2:03:17.400)

And that created the Protestant movement

Lex Fridman (2:03:20.340)

and 200 years of religious persecution and wars.

Lex Fridman (2:03:23.660)

So that's a bad side effect of the printing press.

Lex Fridman (2:03:26.180)

Social networks aren't being nearly as bad

Yann LeCun (2:03:28.500)

as the printing press,

Lex Fridman (2:03:29.320)

but nobody would say the printing press was a bad idea.

Yann LeCun (2:03:33.520)

Yeah, a lot of it is perception

Lex Fridman (2:03:35.100)

and there's a lot of different incentives operating here.

Yann LeCun (2:03:38.420)

Maybe a quick comment,

Lex Fridman (2:03:40.020)

since you're one of the top leaders at Facebook

Lex Fridman (2:03:42.700)

and at Meta, sorry, that's in the tech space,

Lex Fridman (2:03:46.760)

I'm sure Facebook involves a lot of incredible

Yann LeCun (2:03:49.700)

technological challenges that need to be solved.

Lex Fridman (2:03:52.900)

A lot of it probably is in the computer infrastructure,

Yann LeCun (2:03:55.000)

the hardware, I mean, it's just a huge amount.

Lex Fridman (2:03:58.920)

Maybe can you give me context about how much of Shrepp's life

Lex Fridman (2:04:03.580)

is AI and how much of it is low level compute?

Lex Fridman (2:04:06.240)

How much of it is flying all around doing business stuff?

Lex Fridman (2:04:09.580)

And the same with Mark Zuckerberg.

Lex Fridman (2:04:12.000)

They really focus on AI.

Yann LeCun (2:04:13.740)

I mean, certainly in the run up of the creation of FAIR

Lex Fridman (2:04:19.520)

and for at least a year after that, if not more,

Yann LeCun (2:04:24.060)

Mark was very, very much focused on AI

Lex Fridman (2:04:26.700)

and was spending quite a lot of effort on it.

Lex Fridman (2:04:29.700)

And that's his style.

Lex Fridman (2:04:30.780)

When he gets interested in something,

Yann LeCun (2:04:32.060)

he reads everything about it.

Lex Fridman (2:04:34.100)

He read some of my papers, for example, before I joined.

Lex Fridman (2:04:39.620)

And so he learned a lot about it.

Lex Fridman (2:04:41.860)

He said he liked notes.

Yann LeCun (2:04:43.740)

Right.

Lex Fridman (2:04:46.460)

And Shrepp was really into it also.

Yann LeCun (2:04:51.100)

I mean, Shrepp is really kind of,

Lex Fridman (2:04:54.780)

has something I've tried to preserve also

Yann LeCun (2:04:57.940)

despite my not so young age,

Lex Fridman (2:05:00.180)

which is a sense of wonder about science and technology.

Lex Fridman (2:05:03.180)

And he certainly has that.

Lex Fridman (2:05:06.300)

He's also a wonderful person.

Yann LeCun (2:05:07.420)

I mean, in terms of like as a manager,

Lex Fridman (2:05:10.380)

like dealing with people and everything.

Yann LeCun (2:05:12.140)

Mark also, actually.

Lex Fridman (2:05:14.540)

I mean, they're very human people.

Yann LeCun (2:05:18.020)

In the case of Mark, it's shockingly human

Lex Fridman (2:05:20.600)

given his trajectory.

Yann LeCun (2:05:25.460)

I mean, the personality of him that is painted in the press,

Lex Fridman (2:05:28.100)

it's just completely wrong.

Yann LeCun (2:05:29.620)

Yeah.

Lex Fridman (2:05:30.460)

But you have to know how to play the press.

Lex Fridman (2:05:31.980)

So that's, I put some of that responsibility on him too.

Lex Fridman (2:05:36.220)

You have to, it's like, you know,

Yann LeCun (2:05:40.980)

like the director, the conductor of an orchestra,

Lex Fridman (2:05:44.300)

you have to play the press and the public

Yann LeCun (2:05:46.980)

in a certain kind of way

Lex Fridman (2:05:48.020)

where you convey your true self to them.

Yann LeCun (2:05:49.740)

If there's a depth and kindness to it.

Lex Fridman (2:05:51.060)

It's hard.

Lex Fridman (2:05:51.900)

And it's probably not the best at it.

Lex Fridman (2:05:53.740)

So, yeah.

Yann LeCun (2:05:56.460)

You have to learn.

Lex Fridman (2:05:57.700)

And it's sad to see, and I'll talk to him about it,

Lex Fridman (2:06:00.460)

but Shrep is slowly stepping down.

Lex Fridman (2:06:04.060)

It's always sad to see folks sort of be there

Yann LeCun (2:06:07.500)

for a long time and slowly.

Lex Fridman (2:06:09.420)

I guess time is sad.

Yann LeCun (2:06:11.220)

I think he's done the thing he set out to do.

Lex Fridman (2:06:14.780)

And, you know, he's got, you know,

Yann LeCun (2:06:19.700)

family priorities and stuff like that.

Lex Fridman (2:06:21.420)

And I understand, you know, after 13 years or something.

Yann LeCun (2:06:27.900)

It's been a good run.

Lex Fridman (2:06:28.900)

Which in Silicon Valley is basically a lifetime.

Yann LeCun (2:06:32.100)

Yeah.

Lex Fridman (2:06:32.940)

You know, because, you know, it's dog years.

Yann LeCun (2:06:35.000)

So, NeurIPS, the conference just wrapped up.

Lex Fridman (2:06:38.660)

Let me just go back to something else.

Yann LeCun (2:06:40.580)

You posted that a paper you coauthored

Lex Fridman (2:06:42.500)

was rejected from NeurIPS.

Yann LeCun (2:06:44.440)

As you said, proudly, in quotes, rejected.

Lex Fridman (2:06:48.020)

It's a joke.

Yann LeCun (2:06:48.940)

Yeah, I know.

Lex Fridman (2:06:49.760)

So, can you describe this paper?

Lex Fridman (2:06:53.260)

And like, what was the idea in it?

Lex Fridman (2:06:55.700)

And also, maybe this is a good opportunity to ask

Lex Fridman (2:06:59.060)

what are the pros and cons, what works and what doesn't

Lex Fridman (2:07:01.740)

about the review process?

Yann LeCun (2:07:03.620)

Yeah, let me talk about the paper first.

Lex Fridman (2:07:04.980)

I'll talk about the review process afterwards.

Yann LeCun (2:07:09.220)

The paper is called VicReg.

Lex Fridman (2:07:10.700)

So, this is, I mentioned that before.

Yann LeCun (2:07:12.540)

Variance, invariance, covariance, regularization.

Lex Fridman (2:07:14.900)

And it's a technique, a noncontrastive learning technique

Yann LeCun (2:07:18.260)

for what I call joint embedding architecture.

Lex Fridman (2:07:21.300)

So, SiameseNets are an example

Yann LeCun (2:07:23.380)

of joint embedding architecture.

Lex Fridman (2:07:24.860)

So, joint embedding architecture is,

Lex Fridman (2:07:29.220)

let me back up a little bit, right?

Lex Fridman (2:07:30.600)

So, if you want to do self supervised learning,

Yann LeCun (2:07:33.300)

you can do it by prediction.

Lex Fridman (2:07:36.440)

So, let's say you want to train the system

Lex Fridman (2:07:37.920)

to predict video, right?

Lex Fridman (2:07:38.760)

You show it a video clip and you train the system

Yann LeCun (2:07:42.500)

to predict the next, the continuation of that video clip.

Lex Fridman (2:07:45.040)

Now, because you need to handle uncertainty,

Yann LeCun (2:07:47.800)

because there are many continuations that are plausible,

Lex Fridman (2:07:51.580)

you need to have, you need to handle this in some way.

Yann LeCun (2:07:54.020)

You need to have a way for the system

Lex Fridman (2:07:56.660)

to be able to produce multiple predictions.

Lex Fridman (2:08:00.620)

And the way, the only way I know to do this

Lex Fridman (2:08:03.500)

is through what's called a latent variable.

Yann LeCun (2:08:05.420)

So, you have some sort of hidden vector

Lex Fridman (2:08:08.780)

of a variable that you can vary over a set

Yann LeCun (2:08:11.180)

or draw from a distribution.

Lex Fridman (2:08:12.580)

And as you vary this vector over a set,

Yann LeCun (2:08:14.500)

the output, the prediction varies

Lex Fridman (2:08:16.000)

over a set of plausible predictions, okay?

Yann LeCun (2:08:18.740)

So, that's called,

Lex Fridman (2:08:19.580)

I call this a generative latent variable model.

Yann LeCun (2:08:24.140)

Got it.

Lex Fridman (2:08:24.980)

Okay, now there is an alternative to this,

Yann LeCun (2:08:27.060)

to handle uncertainty.

Lex Fridman (2:08:28.700)

And instead of directly predicting the next frames

Yann LeCun (2:08:33.380)

of the clip, you also run those through another neural net.

Lex Fridman (2:08:41.080)

So, you now have two neural nets,

Yann LeCun (2:08:42.500)

one that looks at the initial segment of the video clip,

Lex Fridman (2:08:48.700)

and another one that looks at the continuation

Lex Fridman (2:08:51.260)

during training, right?

Lex Fridman (2:08:53.560)

And what you're trying to do is learn a representation

Yann LeCun (2:08:57.680)

of those two video clips that is maximally informative

Lex Fridman (2:09:00.780)

about the video clips themselves,

Lex Fridman (2:09:03.460)

but is such that you can predict the representation

Lex Fridman (2:09:07.180)

of the second video clip

Lex Fridman (2:09:08.580)

from the representation of the first one easily, okay?

Lex Fridman (2:09:12.340)

And you can sort of formalize this

Yann LeCun (2:09:13.580)

in terms of maximizing mutual information

Lex Fridman (2:09:15.340)

and some stuff like that, but it doesn't matter.

Lex Fridman (2:09:18.140)

What you want is informative representations

Lex Fridman (2:09:24.540)

of the two video clips that are mutually predictable.

Lex Fridman (2:09:28.460)

What that means is that there's a lot of details

Lex Fridman (2:09:30.900)

in the second video clips that are irrelevant.

Yann LeCun (2:09:36.500)

Let's say a video clip consists in a camera panning

Lex Fridman (2:09:40.500)

the scene, there's gonna be a piece of that room

Yann LeCun (2:09:43.340)

that is gonna be revealed, and I can somewhat predict

Lex Fridman (2:09:46.180)

what that room is gonna look like,

Lex Fridman (2:09:48.060)

but I may not be able to predict the details

Lex Fridman (2:09:50.220)

of the texture of the ground

Lex Fridman (2:09:52.300)

and where the tiles are ending and stuff like that, right?

Lex Fridman (2:09:54.500)

So, those are irrelevant details

Yann LeCun (2:09:56.360)

that perhaps my representation will eliminate.

Lex Fridman (2:09:59.620)

And so, what I need is to train this second neural net

Yann LeCun (2:10:03.680)

in such a way that whenever the continuation video clip

Lex Fridman (2:10:08.680)

varies over all the plausible continuations,

Yann LeCun (2:10:13.600)

the representation doesn't change.

Lex Fridman (2:10:15.600)

Got it.

Yann LeCun (2:10:16.440)

So, it's the, yeah, yeah, got it.

Lex Fridman (2:10:18.100)

Over the space of the representations,

Yann LeCun (2:10:20.860)

doing the same kind of thing

Lex Fridman (2:10:21.880)

as you do with similarity learning.

Yann LeCun (2:10:24.300)

Right.

Lex Fridman (2:10:25.680)

So, these are two ways to handle multimodality

Lex Fridman (2:10:28.840)

in a prediction, right?

Lex Fridman (2:10:29.680)

In the first way, you parameterize the prediction

Yann LeCun (2:10:32.280)

with a latent variable,

Lex Fridman (2:10:33.480)

but you predict pixels essentially, right?

Yann LeCun (2:10:35.800)

In the second one, you don't predict pixels,

Lex Fridman (2:10:38.400)

you predict an abstract representation of pixels,

Lex Fridman (2:10:40.720)

and you guarantee that this abstract representation

Lex Fridman (2:10:43.480)

has as much information as possible about the input,

Lex Fridman (2:10:46.200)

but sort of, you know,

Lex Fridman (2:10:47.080)

drops all the stuff that you really can't predict,

Yann LeCun (2:10:49.740)

essentially.

Lex Fridman (2:10:52.120)

I used to be a big fan of the first approach.

Lex Fridman (2:10:53.880)

And in fact, in this paper with Hicham Mishra,

Lex Fridman (2:10:55.880)

this blog post, the Dark Matter Intelligence,

Yann LeCun (2:10:58.400)

I was kind of advocating for this.

Lex Fridman (2:10:59.760)

And in the last year and a half,

Yann LeCun (2:11:01.600)

I've completely changed my mind.

Lex Fridman (2:11:02.840)

I'm now a big fan of the second one.

Lex Fridman (2:11:04.640)

And it's because of a small collection of algorithms

Lex Fridman (2:11:10.000)

that have been proposed over the last year and a half or so,

Yann LeCun (2:11:13.680)

two years, to do this, including vCraig,

Lex Fridman (2:11:17.800)

its predecessor called Barlow Twins,

Yann LeCun (2:11:19.600)

which I mentioned, a method from our friends at DeepMind

Lex Fridman (2:11:23.560)

called BYOL, and there's a bunch of others now

Yann LeCun (2:11:28.500)

that kind of work similarly.

Lex Fridman (2:11:29.600)

So, they're all based on this idea of joint embedding.

Yann LeCun (2:11:32.600)

Some of them have an explicit criterion

Lex Fridman (2:11:34.660)

that is an approximation of mutual information.

Yann LeCun (2:11:36.640)

Some others at BYOL work, but we don't really know why.

Lex Fridman (2:11:39.400)

And there's been like lots of theoretical papers

Yann LeCun (2:11:41.240)

about why BYOL works.

Lex Fridman (2:11:42.360)

No, it's not that, because we take it out

Lex Fridman (2:11:43.940)

and it still works, and blah, blah, blah.

Lex Fridman (2:11:46.040)

I mean, so there's like a big debate,

Lex Fridman (2:11:47.800)

but the important point is that we now have a collection

Lex Fridman (2:11:51.540)

of noncontrastive joint embedding methods,

Yann LeCun (2:11:53.720)

which I think is the best thing since sliced bread.

Lex Fridman (2:11:56.400)

So, I'm super excited about this

Yann LeCun (2:11:58.320)

because I think it's our best shot

Lex Fridman (2:12:01.200)

for techniques that would allow us

Yann LeCun (2:12:02.720)

to kind of build predictive world models.

Lex Fridman (2:12:06.360)

And at the same time,

Yann LeCun (2:12:07.440)

learn hierarchical representations of the world,

Lex Fridman (2:12:09.920)

where what matters about the world is preserved

Lex Fridman (2:12:11.840)

and what is irrelevant is eliminated.

Lex Fridman (2:12:14.440)

And by the way, the representations,

Yann LeCun (2:12:15.880)

the before and after, is in the space

Lex Fridman (2:12:19.200)

in a sequence of images, or is it for single images?

Yann LeCun (2:12:22.320)

It would be either for a single image, for a sequence.

Lex Fridman (2:12:24.600)

It doesn't have to be images.

Yann LeCun (2:12:25.660)

This could be applied to text.

Lex Fridman (2:12:26.680)

This could be applied to just about any signal.

Yann LeCun (2:12:28.560)

I'm looking for methods that are generally applicable

Lex Fridman (2:12:32.960)

that are not specific to one particular modality.

Yann LeCun (2:12:36.200)

It could be audio or whatever.

Lex Fridman (2:12:37.640)

Got it.

Lex Fridman (2:12:38.460)

So, what's the story behind this paper?

Lex Fridman (2:12:40.120)

This paper is describing one such method?

Yann LeCun (2:12:43.480)

It's this vcrack method.

Lex Fridman (2:12:44.480)

So, this is coauthored.

Yann LeCun (2:12:45.720)

The first author is a student called Adrien Barne,

Lex Fridman (2:12:49.280)

who is a resident PhD student at Fair Paris,

Yann LeCun (2:12:52.680)

who is coadvised by me and Jean Ponce,

Lex Fridman (2:12:55.800)

who is a professor at École Normale Supérieure,

Yann LeCun (2:12:58.720)

also a research director at INRIA.

Lex Fridman (2:13:01.600)

So, this is a wonderful program in France

Yann LeCun (2:13:03.600)

where PhD students can basically do their PhD in industry,

Lex Fridman (2:13:06.640)

and that's kind of what's happening here.

Lex Fridman (2:13:10.440)

And this paper is a followup on this Bardo Twin paper

Lex Fridman (2:13:15.480)

by my former postdoc, now Stéphane Denis,

Yann LeCun (2:13:18.360)

with Li Jing and Iorij Bontar

Lex Fridman (2:13:21.560)

and a bunch of other people from Fair.

Lex Fridman (2:13:24.720)

And one of the main criticism from reviewers

Lex Fridman (2:13:27.840)

is that vcrack is not different enough from Bardo Twins.

Yann LeCun (2:13:31.400)

But, you know, my impression is that it's, you know,

Lex Fridman (2:13:36.720)

Bardo Twins with a few bugs fixed, essentially,

Lex Fridman (2:13:39.880)

and in the end, this is what people will use.

Lex Fridman (2:13:43.200)

Right, so.

Yann LeCun (2:13:44.520)

But, you know, I'm used to stuff

Lex Fridman (2:13:47.080)

that I submit being rejected for a while.

Yann LeCun (2:13:49.040)

So, it might be rejected and actually exceptionally well cited

Lex Fridman (2:13:51.360)

because people use it.

Yann LeCun (2:13:52.280)

Well, it's already cited like a bunch of times.

Lex Fridman (2:13:54.360)

So, I mean, the question is then to the deeper question

Yann LeCun (2:13:57.600)

about peer review and conferences.

Lex Fridman (2:14:00.240)

I mean, computer science is a field that's kind of unique

Yann LeCun (2:14:02.600)

that the conference is highly prized.

Lex Fridman (2:14:04.960)

That's one.

Yann LeCun (2:14:05.800)

Right.

Lex Fridman (2:14:06.640)

And it's interesting because the peer review process there

Yann LeCun (2:14:09.120)

is similar, I suppose, to journals,

Lex Fridman (2:14:11.080)

but it's accelerated significantly.

Yann LeCun (2:14:13.640)

Well, not significantly, but it goes fast.

Lex Fridman (2:14:16.560)

And it's a nice way to get stuff out quickly,

Yann LeCun (2:14:19.760)

to peer review it quickly,

Lex Fridman (2:14:20.800)

go to present it quickly to the community.

Yann LeCun (2:14:22.640)

So, not quickly, but quicker.

Lex Fridman (2:14:25.160)

Yeah.

Lex Fridman (2:14:26.000)

But nevertheless, it has many of the same flaws

Lex Fridman (2:14:27.840)

of peer review,

Yann LeCun (2:14:29.120)

because it's a limited number of people look at it.

Lex Fridman (2:14:31.520)

There's bias and the following,

Yann LeCun (2:14:32.800)

like that if you want to do new ideas,

Lex Fridman (2:14:35.600)

you're going to get pushback.

Yann LeCun (2:14:38.120)

There's self interested people that kind of can infer

Lex Fridman (2:14:42.120)

who submitted it and kind of, you know,

Yann LeCun (2:14:45.320)

be cranky about it, all that kind of stuff.

Lex Fridman (2:14:47.760)

Yeah, I mean, there's a lot of social phenomena there.

Yann LeCun (2:14:51.040)

There's one social phenomenon, which is that

Lex Fridman (2:14:53.200)

because the field has been growing exponentially,

Yann LeCun (2:14:56.760)

the vast majority of people in the field

Lex Fridman (2:14:58.560)

are extremely junior.

Yann LeCun (2:15:00.000)

Yeah.

Lex Fridman (2:15:00.840)

So, as a consequence,

Lex Fridman (2:15:01.920)

and that's just a consequence of the field growing, right?

Lex Fridman (2:15:04.880)

So, as the number of, as the size of the field

Yann LeCun (2:15:07.840)

kind of starts saturating,

Lex Fridman (2:15:08.920)

you will have less of that problem

Yann LeCun (2:15:11.440)

of reviewers being very inexperienced.

Lex Fridman (2:15:15.360)

A consequence of this is that, you know, young reviewers,

Yann LeCun (2:15:20.160)

I mean, there's a phenomenon which is that

Lex Fridman (2:15:22.840)

reviewers try to make their life easy

Lex Fridman (2:15:24.640)

and to make their life easy when reviewing a paper

Lex Fridman (2:15:27.440)

is very simple.

Lex Fridman (2:15:28.280)

You just have to find a flaw in the paper, right?

Lex Fridman (2:15:29.960)

So, basically they see the task as finding flaws in papers

Lex Fridman (2:15:34.480)

and most papers have flaws, even the good ones.

Lex Fridman (2:15:36.720)

Yeah.

Yann LeCun (2:15:38.160)

So, it's easy to, you know, to do that.

Lex Fridman (2:15:41.480)

Your job is easier as a reviewer if you just focus on this.

Lex Fridman (2:15:46.440)

But what's important is like,

Lex Fridman (2:15:49.640)

is there a new idea in that paper

Lex Fridman (2:15:51.520)

that is likely to influence?

Lex Fridman (2:15:54.120)

It doesn't matter if the experiments are not that great,

Yann LeCun (2:15:56.240)

if the protocol is, you know, so, so, you know,

Lex Fridman (2:16:00.680)

things like that.

Yann LeCun (2:16:01.520)

As long as there is a worthy idea in it

Lex Fridman (2:16:05.040)

that will influence the way people think about the problem,

Yann LeCun (2:16:09.200)

even if they make it better, you know, eventually,

Lex Fridman (2:16:11.160)

I think that's really what makes a paper useful.

Lex Fridman (2:16:15.480)

And so, this combination of social phenomena

Lex Fridman (2:16:19.520)

creates a disease that has plagued, you know,

Yann LeCun (2:16:24.200)

other fields in the past, like speech recognition,

Lex Fridman (2:16:26.680)

where basically, you know, people chase numbers

Yann LeCun (2:16:28.560)

on benchmarks and it's much easier to get a paper accepted

Lex Fridman (2:16:34.680)

if it brings an incremental improvement

Yann LeCun (2:16:37.040)

on a sort of mainstream well accepted method or problem.

Lex Fridman (2:16:44.160)

And those are, to me, boring papers.

Lex Fridman (2:16:46.040)

I mean, they're not useless, right?

Lex Fridman (2:16:47.880)

Because industry, you know, strives

Yann LeCun (2:16:50.560)

on those kinds of progress,

Lex Fridman (2:16:52.400)

but they're not the ones that I'm interested in,

Yann LeCun (2:16:54.080)

in terms of like new concepts and new ideas.

Lex Fridman (2:16:55.680)

So, papers that are really trying to strike

Yann LeCun (2:16:59.320)

kind of new advances generally don't make it.

Lex Fridman (2:17:02.600)

Now, thankfully we have Archive.

Yann LeCun (2:17:04.240)

Archive, exactly.

Lex Fridman (2:17:05.320)

And then there's open review type of situations

Yann LeCun (2:17:08.160)

where you, and then, I mean, Twitter's a kind of open review.

Lex Fridman (2:17:11.680)

I'm a huge believer that review should be done

Yann LeCun (2:17:13.880)

by thousands of people, not two people.

Lex Fridman (2:17:15.720)

I agree.

Lex Fridman (2:17:16.760)

And so Archive, like do you see a future

Lex Fridman (2:17:19.560)

where a lot of really strong papers,

Yann LeCun (2:17:21.240)

it's already the present, but a growing future

Lex Fridman (2:17:23.640)

where it'll just be Archive

Lex Fridman (2:17:26.280)

and you're presenting an ongoing continuous conference

Lex Fridman (2:17:31.280)

called Twitter slash the internet slash Archive Sanity.

Yann LeCun (2:17:35.560)

Andre just released a new version.

Lex Fridman (2:17:38.040)

So just not, you know, not being so elitist

Yann LeCun (2:17:40.920)

about this particular gating.

Lex Fridman (2:17:43.440)

It's not a question of being elitist or not.

Yann LeCun (2:17:44.960)

It's a question of being basically recommendation

Lex Fridman (2:17:50.120)

and sort of approvals for people who don't see themselves

Lex Fridman (2:17:53.400)

as having the ability to do so by themselves, right?

Lex Fridman (2:17:55.880)

And so it saves time, right?

Yann LeCun (2:17:57.320)

If you rely on other people's opinion

Lex Fridman (2:18:00.000)

and you trust those people or those groups

Yann LeCun (2:18:03.760)

to evaluate a paper for you, that saves you time

Lex Fridman (2:18:09.960)

because, you know, you don't have to like scrutinize

Yann LeCun (2:18:12.680)

the paper as much, you know, is brought to your attention.

Lex Fridman (2:18:15.200)

I mean, it's the whole idea of sort of, you know,

Lex Fridman (2:18:16.680)

collective recommender system, right?

Lex Fridman (2:18:18.760)

So I actually thought about this a lot, you know,

Yann LeCun (2:18:22.360)

about 10, 15 years ago,

Lex Fridman (2:18:24.200)

because there were discussions at NIPS

Yann LeCun (2:18:27.080)

and, you know, and we're about to create iClear

Lex Fridman (2:18:30.040)

with Yoshua Bengio.

Lex Fridman (2:18:31.200)

And so I wrote a document kind of describing

Lex Fridman (2:18:34.880)

a reviewing system, which basically was, you know,

Yann LeCun (2:18:38.040)

you post your paper on some repository,

Lex Fridman (2:18:39.720)

let's say archive or now could be open review.

Lex Fridman (2:18:42.560)

And then you can form a reviewing entity,

Lex Fridman (2:18:46.240)

which is equivalent to a reviewing board, you know,

Yann LeCun (2:18:48.840)

of a journal or program committee of a conference.

Lex Fridman (2:18:53.960)

You have to list the members.

Lex Fridman (2:18:55.600)

And then that group reviewing entity can choose

Lex Fridman (2:19:00.000)

to review a particular paper spontaneously or not.

Yann LeCun (2:19:03.720)

There is no exclusive relationship anymore

Lex Fridman (2:19:05.600)

between a paper and a venue or reviewing entity.

Yann LeCun (2:19:09.200)

Any reviewing entity can review any paper

Lex Fridman (2:19:12.720)

or may choose not to.

Lex Fridman (2:19:15.000)

And then, you know, given evaluation,

Lex Fridman (2:19:16.640)

it's not published, not published,

Yann LeCun (2:19:17.920)

it's just an evaluation and a comment,

Lex Fridman (2:19:20.320)

which would be public, signed by the reviewing entity.

Lex Fridman (2:19:23.680)

And if it's signed by a reviewing entity,

Lex Fridman (2:19:25.880)

you know, it's one of the members of reviewing entity.

Lex Fridman (2:19:27.760)

So if the reviewing entity is, you know,

Lex Fridman (2:19:30.680)

Lex Friedman's, you know, preferred papers, right?

Yann LeCun (2:19:33.720)

You know, it's Lex Friedman writing the review.

Lex Fridman (2:19:35.640)

Yes, so for me, that's a beautiful system, I think.

Lex Fridman (2:19:40.920)

But in addition to that,

Lex Fridman (2:19:42.880)

it feels like there should be a reputation system

Yann LeCun (2:19:45.800)

for the reviewers.

Lex Fridman (2:19:47.480)

For the reviewing entities,

Yann LeCun (2:19:49.040)

not the reviewers individually.

Lex Fridman (2:19:50.280)

The reviewing entities, sure.

Lex Fridman (2:19:51.720)

But even within that, the reviewers too,

Lex Fridman (2:19:53.880)

because there's another thing here.

Yann LeCun (2:19:57.120)

It's not just the reputation,

Lex Fridman (2:19:59.360)

it's an incentive for an individual person to do great.

Yann LeCun (2:20:02.680)

Right now, in the academic setting,

Lex Fridman (2:20:05.040)

the incentive is kind of internal,

Yann LeCun (2:20:07.880)

just wanting to do a good job.

Lex Fridman (2:20:09.240)

But honestly, that's not a strong enough incentive

Yann LeCun (2:20:11.240)

to do a really good job in reading a paper,

Lex Fridman (2:20:13.720)

in finding the beautiful amidst the mistakes and the flaws

Lex Fridman (2:20:16.400)

and all that kind of stuff.

Lex Fridman (2:20:17.760)

Like if you're the person that first discovered

Yann LeCun (2:20:20.760)

a powerful paper, and you get to be proud of that discovery,

Lex Fridman (2:20:25.120)

then that gives a huge incentive to you.

Yann LeCun (2:20:27.520)

That's a big part of my proposal, actually,

Lex Fridman (2:20:29.280)

where I describe that as, you know,

Yann LeCun (2:20:31.280)

if your evaluation of papers is predictive

Lex Fridman (2:20:35.280)

of future success, okay,

Yann LeCun (2:20:37.560)

then your reputation should go up as a reviewing entity.

Lex Fridman (2:20:42.560)

So yeah, exactly.

Yann LeCun (2:20:43.760)

I mean, I even had a master's student

Lex Fridman (2:20:46.280)

who was a master's student in library science

Lex Fridman (2:20:49.560)

and computer science actually kind of work out exactly

Lex Fridman (2:20:52.480)

how that should work with formulas and everything.

Lex Fridman (2:20:55.160)

So in terms of implementation,

Lex Fridman (2:20:56.800)

do you think that's something that's doable?

Yann LeCun (2:20:58.640)

I mean, I've been sort of, you know,

Lex Fridman (2:20:59.720)

talking about this to sort of various people

Yann LeCun (2:21:02.080)

like, you know, Andrew McCallum, who started Open Review.

Lex Fridman (2:21:05.960)

And the reason why we picked Open Review

Yann LeCun (2:21:07.800)

for iClear initially,

Lex Fridman (2:21:09.120)

even though it was very early for them,

Yann LeCun (2:21:11.440)

is because my hope was that iClear,

Lex Fridman (2:21:14.320)

it was eventually going to kind of

Yann LeCun (2:21:16.760)

inaugurate this type of system.

Lex Fridman (2:21:18.600)

So iClear kept the idea of Open Reviews.

Lex Fridman (2:21:22.240)

So where the reviews are, you know,

Lex Fridman (2:21:23.840)

published with a paper, which I think is very useful,

Lex Fridman (2:21:27.320)

but in many ways that's kind of reverted

Lex Fridman (2:21:29.800)

to kind of more of a conventional type conferences

Yann LeCun (2:21:33.280)

for everything else.

Lex Fridman (2:21:34.120)

And that, I mean, I don't run iClear.

Yann LeCun (2:21:37.800)

I'm just the president of the foundation,

Lex Fridman (2:21:41.200)

but you know, people who run it

Yann LeCun (2:21:44.120)

should make decisions about how to run it.

Lex Fridman (2:21:45.680)

And I'm not going to tell them because they are volunteers

Lex Fridman (2:21:48.560)

and I'm really thankful that they do that.

Lex Fridman (2:21:50.360)

So, but I'm saddened by the fact

Yann LeCun (2:21:53.040)

that we're not being innovative enough.

Lex Fridman (2:21:57.120)

Yeah, me too.

Yann LeCun (2:21:57.960)

I hope that changes.

Lex Fridman (2:21:59.640)

Yeah.

Yann LeCun (2:22:00.480)

Cause the communication science broadly,

Lex Fridman (2:22:02.040)

but communication computer science ideas

Yann LeCun (2:22:05.440)

is how you make those ideas have impact, I think.

Lex Fridman (2:22:08.400)

Yeah, and I think, you know, a lot of this is

Yann LeCun (2:22:11.440)

because people have in their mind kind of an objective,

Lex Fridman (2:22:16.200)

which is, you know, fairness for authors

Lex Fridman (2:22:19.120)

and the ability to count points basically

Lex Fridman (2:22:22.600)

and give credits accurately.

Lex Fridman (2:22:24.880)

But that comes at the expense of the progress of science.

Lex Fridman (2:22:28.880)

So to some extent,

Yann LeCun (2:22:29.720)

we're slowing down the progress of science.

Lex Fridman (2:22:32.160)

And are we actually achieving fairness?

Lex Fridman (2:22:34.440)

And we're not achieving fairness.

Lex Fridman (2:22:35.920)

You know, we still have biases.

Yann LeCun (2:22:37.880)

You know, we're doing, you know, a double blind review,

Lex Fridman (2:22:39.840)

but you know, the biases are still there.

Yann LeCun (2:22:44.360)

There are different kinds of biases.

Lex Fridman (2:22:46.720)

You write that the phenomenon of emergence,

Yann LeCun (2:22:49.360)

collective behavior exhibited by a large collection

Lex Fridman (2:22:51.680)

of simple elements in interaction

Yann LeCun (2:22:54.280)

is one of the things that got you

Lex Fridman (2:22:55.760)

into neural nets in the first place.

Yann LeCun (2:22:57.760)

I love cellular automata.

Lex Fridman (2:22:59.120)

I love simple interacting elements

Lex Fridman (2:23:02.000)

and the things that emerge from them.

Lex Fridman (2:23:04.040)

Do you think we understand how complex systems can emerge

Lex Fridman (2:23:07.880)

from such simple components that interact simply?

Lex Fridman (2:23:11.080)

No, we don't.

Yann LeCun (2:23:12.320)

It's a big mystery.

Lex Fridman (2:23:13.160)

Also, it's a mystery for physicists.

Yann LeCun (2:23:14.480)

It's a mystery for biologists.

Lex Fridman (2:23:17.000)

You know, how is it that the universe around us

Lex Fridman (2:23:22.000)

seems to be increasing in complexity and not decreasing?

Lex Fridman (2:23:25.120)

I mean, that is a kind of curious property of physics

Yann LeCun (2:23:29.640)

that despite the second law of thermodynamics,

Lex Fridman (2:23:32.320)

we seem to be, you know, evolution and learning

Lex Fridman (2:23:35.960)

and et cetera seems to be kind of at least locally

Lex Fridman (2:23:40.640)

to increase complexity and not decrease it.

Lex Fridman (2:23:44.000)

So perhaps the ultimate purpose of the universe

Lex Fridman (2:23:46.520)

is to just get more complex.

Yann LeCun (2:23:49.040)

Have these, I mean, small pockets of beautiful complexity.

Lex Fridman (2:23:55.120)

Does that, cellular automata,

Yann LeCun (2:23:57.120)

these kinds of emergence of complex systems

Lex Fridman (2:23:59.680)

give you some intuition or guide your understanding

Lex Fridman (2:24:04.120)

of machine learning systems and neural networks and so on?

Lex Fridman (2:24:06.680)

Or are these, for you right now, disparate concepts?

Yann LeCun (2:24:09.440)

Well, it got me into it.

Lex Fridman (2:24:10.880)

You know, I discovered the existence of the perceptron

Yann LeCun (2:24:15.600)

when I was a college student, you know, by reading a book

Lex Fridman (2:24:19.280)

and it was a debate between Chomsky and Piaget

Lex Fridman (2:24:21.680)

and Seymour Papert from MIT was kind of singing the praise

Lex Fridman (2:24:25.920)

of the perceptron in that book.

Lex Fridman (2:24:27.400)

And I, the first time I heard about the running machine,

Lex Fridman (2:24:29.760)

right, so I started digging the literature

Lex Fridman (2:24:31.360)

and I found those paper, those books,

Lex Fridman (2:24:33.560)

which were basically transcription of workshops

Yann LeCun (2:24:37.120)

or conferences from the fifties and sixties

Lex Fridman (2:24:39.880)

about self organizing systems.

Lex Fridman (2:24:42.160)

So there were, there was a series of conferences

Lex Fridman (2:24:44.560)

on self organizing systems and there's books on this.

Yann LeCun (2:24:48.160)

Some of them are, you can actually get them

Lex Fridman (2:24:50.200)

at the internet archive, you know, the digital version.

Lex Fridman (2:24:55.120)

And there are like fascinating articles in there by,

Lex Fridman (2:24:58.280)

there's a guy whose name has been largely forgotten,

Yann LeCun (2:25:00.360)

Heinz von Förster, he's a German physicist

Lex Fridman (2:25:04.520)

who immigrated to the US and worked

Yann LeCun (2:25:07.240)

on self organizing systems in the fifties.

Lex Fridman (2:25:11.320)

And in the sixties he created at University of Illinois

Yann LeCun (2:25:13.800)

at Urbana Champagne, he created the biological

Lex Fridman (2:25:16.440)

computer laboratory, BCL, which was all about neural nets.

Yann LeCun (2:25:21.680)

Unfortunately, that was kind of towards the end

Lex Fridman (2:25:23.440)

of the popularity of neural nets.

Lex Fridman (2:25:24.920)

So that lab never kind of strived very much,

Lex Fridman (2:25:27.760)

but he wrote a bunch of papers about self organization

Lex Fridman (2:25:30.360)

and about the mystery of self organization.

Lex Fridman (2:25:33.480)

An example he has is you take, imagine you are in space,

Yann LeCun (2:25:37.000)

there's no gravity and you have a big box

Lex Fridman (2:25:38.880)

with magnets in it, okay.

Yann LeCun (2:25:42.200)

You know, kind of rectangular magnets

Lex Fridman (2:25:43.920)

with North Pole on one end, South Pole on the other end.

Yann LeCun (2:25:46.880)

You shake the box gently and the magnets will kind of stick

Lex Fridman (2:25:49.640)

to themselves and probably form like complex structure,

Yann LeCun (2:25:53.480)

you know, spontaneously.

Lex Fridman (2:25:55.280)

You know, that could be an example of self organization,

Lex Fridman (2:25:57.120)

but you know, you have lots of examples,

Lex Fridman (2:25:58.400)

neural nets are an example of self organization too,

Yann LeCun (2:26:01.280)

you know, in many respect.

Lex Fridman (2:26:03.080)

And it's a bit of a mystery, you know,

Lex Fridman (2:26:05.960)

how like what is possible with this, you know,

Lex Fridman (2:26:09.520)

pattern formation in physical systems, in chaotic system

Lex Fridman (2:26:12.960)

and things like that, you know, the emergence of life,

Lex Fridman (2:26:16.120)

you know, things like that.

Lex Fridman (2:26:16.960)

So, you know, how does that happen?

Lex Fridman (2:26:19.560)

So it's a big puzzle for physicists as well.

Yann LeCun (2:26:22.600)

It feels like understanding this,

Lex Fridman (2:26:24.720)

the mathematics of emergence

Yann LeCun (2:26:27.920)

in some constrained situations

Lex Fridman (2:26:29.720)

might help us create intelligence,

Yann LeCun (2:26:32.120)

like help us add a little spice to the systems

Lex Fridman (2:26:36.040)

because you seem to be able to in complex systems

Yann LeCun (2:26:40.960)

with emergence to be able to get a lot from little.

Lex Fridman (2:26:44.600)

And so that seems like a shortcut

Yann LeCun (2:26:47.000)

to get big leaps in performance, but...

Lex Fridman (2:26:51.120)

But there's a missing concept that we don't have.

Yann LeCun (2:26:55.000)

Yeah.

Lex Fridman (2:26:55.840)

And it's something also I've been fascinated by

Yann LeCun (2:26:58.440)

since my undergrad days,

Lex Fridman (2:27:00.720)

and it's how you measure complexity, right?

Lex Fridman (2:27:03.880)

So we don't actually have good ways of measuring,

Lex Fridman (2:27:06.960)

or at least we don't have good ways of interpreting

Yann LeCun (2:27:09.840)

the measures that we have at our disposal.

Lex Fridman (2:27:11.920)

Like how do you measure the complexity of something, right?

Lex Fridman (2:27:14.480)

So there's all those things, you know,

Lex Fridman (2:27:15.680)

like, you know, Kolmogorov, Chaitin, Solomonov complexity

Yann LeCun (2:27:18.560)

of, you know, the length of the shortest program

Lex Fridman (2:27:20.920)

that would generate a bit string can be thought of

Lex Fridman (2:27:23.320)

as the complexity of that bit string, right?

Lex Fridman (2:27:26.840)

I've been fascinated by that concept.

Yann LeCun (2:27:28.200)

The problem with that is that

Lex Fridman (2:27:30.160)

that complexity is defined up to a constant,

Yann LeCun (2:27:32.840)

which can be very large.

Lex Fridman (2:27:34.920)

Right.

Yann LeCun (2:27:35.760)

There are similar concepts that are derived from,

Lex Fridman (2:27:37.840)

you know, Bayesian probability theory,

Yann LeCun (2:27:42.280)

where, you know, the complexity of something

Lex Fridman (2:27:44.520)

is the negative log of its probability, essentially, right?

Lex Fridman (2:27:48.360)

And you have a complete equivalence between the two things.

Lex Fridman (2:27:51.120)

And there you would think, you know,

Yann LeCun (2:27:52.120)

the probability is something that's well defined mathematically,

Lex Fridman (2:27:55.160)

which means complexity is well defined.

Lex Fridman (2:27:57.200)

But it's not true.

Lex Fridman (2:27:58.040)

You need to have a model of the distribution.

Yann LeCun (2:28:01.720)

You may need to have a prior

Lex Fridman (2:28:02.800)

if you're doing Bayesian inference.

Lex Fridman (2:28:04.200)

And the prior plays the same role

Lex Fridman (2:28:05.720)

as the choice of the computer

Yann LeCun (2:28:07.040)

with which you measure Kolmogorov complexity.

Lex Fridman (2:28:09.480)

And so every measure of complexity we have

Yann LeCun (2:28:12.040)

has some arbitrary density,

Lex Fridman (2:28:15.440)

you know, an additive constant,

Yann LeCun (2:28:16.840)

which can be arbitrarily large.

Lex Fridman (2:28:19.560)

And so, you know, how can we come up with a good theory

Yann LeCun (2:28:23.360)

of how things become more complex

Lex Fridman (2:28:24.640)

if we don't have a good measure of complexity?

Yann LeCun (2:28:26.080)

Yeah, which we need for this.

Lex Fridman (2:28:28.200)

One way that people study this in the space of biology,

Yann LeCun (2:28:32.240)

the people that study the origin of life

Lex Fridman (2:28:33.760)

or try to recreate the life in the laboratory.

Lex Fridman (2:28:37.120)

And the more interesting one is the alien one,

Lex Fridman (2:28:39.200)

is when we go to other planets,

Lex Fridman (2:28:41.320)

how do we recognize this life?

Lex Fridman (2:28:43.960)

Because, you know, complexity, we associate complexity,

Yann LeCun (2:28:46.800)

maybe some level of mobility with life.

Lex Fridman (2:28:50.000)

You know, we have to be able to, like,

Yann LeCun (2:28:51.680)

have concrete algorithms for, like,

Lex Fridman (2:28:57.200)

measuring the level of complexity we see

Yann LeCun (2:29:00.000)

in order to know the difference between life and non life.

Lex Fridman (2:29:02.760)

And the problem is that complexity

Yann LeCun (2:29:04.040)

is in the eye of the beholder.

Lex Fridman (2:29:05.440)

So let me give you an example.

Yann LeCun (2:29:07.480)

If I give you an image of the MNIST digits, right,

Lex Fridman (2:29:13.240)

and I flip through MNIST digits,

Yann LeCun (2:29:15.400)

there is obviously some structure to it

Lex Fridman (2:29:18.120)

because local structure, you know,

Yann LeCun (2:29:20.440)

neighboring pixels are correlated

Lex Fridman (2:29:23.200)

across the entire data set.

Yann LeCun (2:29:25.440)

I imagine that I apply a random permutation

Lex Fridman (2:29:30.440)

to all the pixels, a fixed random permutation.

Yann LeCun (2:29:33.920)

Now I show you those images,

Lex Fridman (2:29:35.360)

they will look, you know, really disorganized to you,

Yann LeCun (2:29:38.880)

more complex.

Lex Fridman (2:29:40.680)

In fact, they're not more complex in absolute terms,

Lex Fridman (2:29:42.880)

they're exactly the same as originally, right?

Lex Fridman (2:29:45.480)

And if you knew what the permutation was,

Yann LeCun (2:29:46.960)

you know, you could undo the permutation.

Lex Fridman (2:29:49.440)

Now, imagine I give you special glasses

Yann LeCun (2:29:52.360)

that undo that permutation.

Lex Fridman (2:29:54.120)

Now, all of a sudden, what looked complicated

Yann LeCun (2:29:56.160)

becomes simple.

Lex Fridman (2:29:57.000)

Right.

Lex Fridman (2:29:57.920)

So if you have two, if you have, you know,

Lex Fridman (2:30:00.400)

humans on one end, and then another race of aliens

Yann LeCun (2:30:03.280)

that sees the universe with permutation glasses.

Lex Fridman (2:30:05.440)

Yeah, with the permutation glasses.

Yann LeCun (2:30:06.600)

Okay, what we perceive as simple to them

Lex Fridman (2:30:09.800)

is hardly complicated, it's probably heat.

Yann LeCun (2:30:11.760)

Yeah.

Lex Fridman (2:30:12.600)

Heat, yeah.

Yann LeCun (2:30:13.440)

Okay, and what they perceive as simple to us

Lex Fridman (2:30:15.320)

is random fluctuation, it's heat.

Yann LeCun (2:30:18.480)

Yeah.

Lex Fridman (2:30:19.320)

Yeah, it's truly in the eye of the beholder.

Yann LeCun (2:30:22.760)

Yeah.

Lex Fridman (2:30:23.600)

It depends what kind of glasses you're wearing.

Yann LeCun (2:30:24.920)

Right.

Lex Fridman (2:30:25.760)

It depends what kind of algorithm you're running

Yann LeCun (2:30:26.840)

in your perception system.

Lex Fridman (2:30:28.360)

So I don't think we'll have a theory of intelligence,

Yann LeCun (2:30:31.080)

self organization, evolution, things like this,

Lex Fridman (2:30:34.320)

until we have a good handle on a notion of complexity

Yann LeCun (2:30:38.520)

which we know is in the eye of the beholder.

Lex Fridman (2:30:42.320)

Yeah, it's sad to think that we might not be able

Yann LeCun (2:30:44.400)

to detect or interact with alien species

Lex Fridman (2:30:47.600)

because we're wearing different glasses.

Yann LeCun (2:30:50.280)

Because their notion of locality

Lex Fridman (2:30:51.440)

might be different from ours.

Yann LeCun (2:30:52.400)

Yeah, exactly.

Lex Fridman (2:30:53.240)

This actually connects with fascinating questions

Yann LeCun (2:30:55.200)

in physics at the moment, like modern physics,

Lex Fridman (2:30:58.120)

quantum physics, like, you know, questions about,

Yann LeCun (2:31:00.240)

like, you know, can we recover the information

Lex Fridman (2:31:02.520)

that's lost in a black hole and things like this, right?

Lex Fridman (2:31:04.520)

And that relies on notions of complexity,

Lex Fridman (2:31:09.360)

which, you know, I find this fascinating.

Lex Fridman (2:31:11.640)

Can you describe your personal quest

Lex Fridman (2:31:13.360)

to build an expressive electronic wind instrument, EWI?

Lex Fridman (2:31:19.760)

What is it?

Lex Fridman (2:31:20.600)

What does it take to build it?

Yann LeCun (2:31:24.000)

Well, I'm a tinker.

Lex Fridman (2:31:25.080)

I like building things.

Yann LeCun (2:31:26.760)

I like building things with combinations of electronics

Lex Fridman (2:31:28.960)

and, you know, mechanical stuff.

Yann LeCun (2:31:32.400)

You know, I have a bunch of different hobbies,

Lex Fridman (2:31:34.120)

but, you know, probably my first one was little,

Yann LeCun (2:31:37.960)

was building model airplanes and stuff like that.

Lex Fridman (2:31:39.800)

And I still do that to some extent.

Lex Fridman (2:31:41.880)

But also electronics, I taught myself electronics

Lex Fridman (2:31:43.800)

before I studied it.

Lex Fridman (2:31:46.240)

And the reason I taught myself electronics

Lex Fridman (2:31:48.120)

is because of music.

Yann LeCun (2:31:49.600)

My cousin was an aspiring electronic musician

Lex Fridman (2:31:53.200)

and he had an analog synthesizer.

Lex Fridman (2:31:55.000)

And I was, you know, basically modifying it for him

Lex Fridman (2:31:58.000)

and building sequencers and stuff like that, right, for him.

Yann LeCun (2:32:00.280)

I was in high school when I was doing this.

Lex Fridman (2:32:02.640)

That's the interesting, like, progressive rock, like 80s.

Yann LeCun (2:32:06.040)

Like, what's the greatest band of all time,

Lex Fridman (2:32:08.000)

according to Yann LeCun?

Yann LeCun (2:32:09.520)

Oh, man, there's too many of them.

Lex Fridman (2:32:11.080)

But, you know, it's a combination of, you know,

Yann LeCun (2:32:16.360)

Mahavishnu Orchestra, Weather Report,

Lex Fridman (2:32:19.800)

yes, Genesis, you know, pre Peter Gabriel,

Yann LeCun (2:32:27.120)

Gentle Giant, you know, things like that.

Lex Fridman (2:32:29.120)

Great.

Yann LeCun (2:32:29.960)

Okay, so this love of electronics

Lex Fridman (2:32:32.280)

and this love of music combined together.

Yann LeCun (2:32:34.240)

Right, so I was actually trained to play

Lex Fridman (2:32:36.280)

Baroque and Renaissance music and I played in an orchestra

Yann LeCun (2:32:42.040)

when I was in high school and first years of college.

Lex Fridman (2:32:45.640)

And I played the recorder, crumb horn,

Yann LeCun (2:32:48.040)

a little bit of oboe, you know, things like that.

Lex Fridman (2:32:50.200)

So I'm a wind instrument player.

Lex Fridman (2:32:52.520)

But I always wanted to play improvised music,

Lex Fridman (2:32:54.080)

even though I don't know anything about it.

Lex Fridman (2:32:56.320)

And the only way I figured, you know,

Lex Fridman (2:32:58.760)

short of like learning to play saxophone

Yann LeCun (2:33:01.080)

was to play electronic wind instruments.

Lex Fridman (2:33:03.560)

So they behave, you know, the fingering is similar

Yann LeCun (2:33:05.680)

to a saxophone, but, you know,

Lex Fridman (2:33:07.640)

you have wide variety of sound

Yann LeCun (2:33:09.080)

because you control the synthesizer with it.

Lex Fridman (2:33:11.040)

So I had a bunch of those, you know,

Yann LeCun (2:33:13.120)

going back to the late 80s from either Yamaha or Akai.

Lex Fridman (2:33:18.880)

They're both kind of the main manufacturers of those.

Lex Fridman (2:33:22.520)

So they were classically, you know,

Lex Fridman (2:33:23.720)

going back several decades.

Lex Fridman (2:33:25.520)

But I've never been completely satisfied with them

Lex Fridman (2:33:27.680)

because of lack of expressivity.

Yann LeCun (2:33:31.120)

And, you know, those things, you know,

Lex Fridman (2:33:32.480)

are somewhat expressive.

Yann LeCun (2:33:33.400)

I mean, they measure the breath pressure,

Lex Fridman (2:33:34.760)

they measure the lip pressure.

Yann LeCun (2:33:36.520)

And, you know, you have various parameters.

Lex Fridman (2:33:39.800)

You can vary with fingers,

Lex Fridman (2:33:41.480)

but they're not really as expressive

Lex Fridman (2:33:44.800)

as an acoustic instrument, right?

Yann LeCun (2:33:47.040)

You hear John Coltrane play two notes

Lex Fridman (2:33:49.400)

and you know it's John Coltrane,

Yann LeCun (2:33:50.760)

you know, it's got a unique sound.

Lex Fridman (2:33:53.000)

Or Miles Davis, right?

Yann LeCun (2:33:54.280)

You can hear it's Miles Davis playing the trumpet

Lex Fridman (2:33:57.480)

because the sound reflects their, you know,

Yann LeCun (2:34:02.480)

physiognomy, basically, the shape of the vocal track

Lex Fridman (2:34:07.600)

kind of shapes the sound.

Lex Fridman (2:34:09.200)

So how do you do this with an electronic instrument?

Lex Fridman (2:34:12.320)

And I was, many years ago,

Yann LeCun (2:34:13.920)

I met a guy called David Wessel.

Lex Fridman (2:34:15.640)

He was a professor at Berkeley

Lex Fridman (2:34:18.240)

and created the Center for Music Technology there.

Lex Fridman (2:34:23.000)

And he was interested in that question.

Lex Fridman (2:34:25.600)

And so I kept kind of thinking about this for many years.

Lex Fridman (2:34:28.120)

And finally, because of COVID, you know, I was at home,

Yann LeCun (2:34:31.040)

I was in my workshop.

Lex Fridman (2:34:32.600)

My workshop serves also as my kind of Zoom room

Lex Fridman (2:34:36.040)

and home office.

Lex Fridman (2:34:37.360)

And this is in New Jersey?

Yann LeCun (2:34:38.800)

In New Jersey.

Lex Fridman (2:34:39.640)

And I started really being serious about, you know,

Yann LeCun (2:34:43.600)

building my own iwi instrument.

Lex Fridman (2:34:45.800)

What else is going on in that New Jersey workshop?

Yann LeCun (2:34:48.160)

Is there some crazy stuff you've built,

Lex Fridman (2:34:50.880)

like just, or like left on the workshop floor, left behind?

Yann LeCun (2:34:55.200)

A lot of crazy stuff is, you know,

Lex Fridman (2:34:57.600)

electronics built with microcontrollers of various kinds

Yann LeCun (2:35:01.680)

and, you know, weird flying contraptions.

Lex Fridman (2:35:06.720)

So you still love flying?

Yann LeCun (2:35:08.720)

It's a family disease.

Lex Fridman (2:35:09.880)

My dad got me into it when I was a kid.

Lex Fridman (2:35:13.520)

And he was building model airplanes when he was a kid.

Lex Fridman (2:35:16.840)

And he was a mechanical engineer.

Yann LeCun (2:35:19.800)

He taught himself electronics also.

Lex Fridman (2:35:21.200)

So he built his early radio control systems

Yann LeCun (2:35:24.080)

in the late 60s, early 70s.

Lex Fridman (2:35:27.760)

And so that's what got me into,

Yann LeCun (2:35:29.640)

I mean, he got me into kind of, you know,

Lex Fridman (2:35:31.120)

engineering and science and technology.

Lex Fridman (2:35:33.040)

Do you also have an interest in appreciation of flight

Lex Fridman (2:35:36.120)

in other forms, like with drones, quadroptors,

Lex Fridman (2:35:38.320)

or do you, is it model airplane, the thing that's?

Lex Fridman (2:35:41.720)

You know, before drones were, you know,

Yann LeCun (2:35:45.240)

kind of a consumer product, you know,

Lex Fridman (2:35:49.240)

I built my own, you know,

Yann LeCun (2:35:50.280)

with also building a microcontroller

Lex Fridman (2:35:52.000)

with JavaScripts and accelerometers for stabilization,

Yann LeCun (2:35:56.240)

writing the firmware for it, you know.

Lex Fridman (2:35:57.760)

And then when it became kind of a standard thing

Yann LeCun (2:35:59.200)

you could buy, it was boring, you know,

Lex Fridman (2:36:00.320)

I stopped doing it.

Yann LeCun (2:36:01.160)

It was not fun anymore.

Lex Fridman (2:36:03.520)

Yeah.

Yann LeCun (2:36:04.720)

You were doing it before it was cool.

Lex Fridman (2:36:06.280)

Yeah.

Lex Fridman (2:36:07.120)

What advice would you give to a young person today

Lex Fridman (2:36:10.080)

in high school and college

Yann LeCun (2:36:11.360)

that dreams of doing something big like Yann LeCun,

Lex Fridman (2:36:15.960)

like let's talk in the space of intelligence,

Yann LeCun (2:36:18.960)

dreams of having a chance to solve

Lex Fridman (2:36:21.000)

some fundamental problem in space of intelligence,

Yann LeCun (2:36:23.960)

both for their career and just in life,

Lex Fridman (2:36:26.200)

being somebody who was a part

Lex Fridman (2:36:28.600)

of creating something special?

Lex Fridman (2:36:30.680)

So try to get interested by big questions,

Lex Fridman (2:36:35.400)

things like, you know, what is intelligence?

Lex Fridman (2:36:38.680)

What is the universe made of?

Lex Fridman (2:36:40.440)

What's life all about?

Lex Fridman (2:36:41.680)

Things like that.

Yann LeCun (2:36:45.040)

Like even like crazy big questions,

Lex Fridman (2:36:47.040)

like what's time?

Yann LeCun (2:36:49.040)

Like nobody knows what time is.

Lex Fridman (2:36:53.160)

And then learn basic things,

Yann LeCun (2:36:58.640)

like basic methods, either from math,

Lex Fridman (2:37:00.680)

from physics or from engineering.

Yann LeCun (2:37:03.280)

Things that have a long shelf life.

Lex Fridman (2:37:05.600)

Like if you have a choice between,

Yann LeCun (2:37:07.280)

like, you know, learning, you know,

Lex Fridman (2:37:10.160)

mobile programming on iPhone

Yann LeCun (2:37:12.600)

or quantum mechanics, take quantum mechanics.

Lex Fridman (2:37:16.880)

Because you're gonna learn things

Yann LeCun (2:37:18.480)

that you have no idea exist.

Lex Fridman (2:37:20.120)

And you may not, you may never be a quantum physicist,

Lex Fridman (2:37:25.320)

but you will learn about path integrals.

Lex Fridman (2:37:26.800)

And path integrals are used everywhere.

Yann LeCun (2:37:29.120)

It's the same formula that you use

Lex Fridman (2:37:30.280)

for, you know, Bayesian integration and stuff like that.

Lex Fridman (2:37:33.280)

So the ideas, the little ideas within quantum mechanics,

Lex Fridman (2:37:37.720)

within some of these kind of more solidified fields

Yann LeCun (2:37:41.440)

will have a longer shelf life.

Lex Fridman (2:37:42.920)

You'll somehow use indirectly in your work.

Yann LeCun (2:37:46.920)

Learn classical mechanics, like you'll learn

Lex Fridman (2:37:48.640)

about Lagrangian, for example,

Yann LeCun (2:37:51.360)

which is like a huge, hugely useful concept,

Lex Fridman (2:37:55.000)

you know, for all kinds of different things.

Yann LeCun (2:37:57.320)

Learn statistical physics, because all the math

Lex Fridman (2:38:01.680)

that comes out of, you know, for machine learning

Yann LeCun (2:38:05.480)

basically comes out of, was figured out

Lex Fridman (2:38:07.280)

by statistical physicists in the, you know,

Lex Fridman (2:38:09.240)

late 19th, early 20th century, right?

Lex Fridman (2:38:10.960)

So, and for some of them actually more recently

Yann LeCun (2:38:14.320)

for, by people like Giorgio Parisi,

Lex Fridman (2:38:16.120)

who just got the Nobel prize for the replica method,

Yann LeCun (2:38:19.040)

among other things, it's used for a lot of different things.

Lex Fridman (2:38:23.200)

You know, variational inference,

Yann LeCun (2:38:25.560)

that math comes from statistical physics.

Lex Fridman (2:38:28.600)

So a lot of those kind of, you know, basic courses,

Yann LeCun (2:38:33.960)

you know, if you do electrical engineering,

Lex Fridman (2:38:36.240)

you take signal processing,

Yann LeCun (2:38:37.360)

you'll learn about Fourier transforms.

Lex Fridman (2:38:39.880)

Again, something super useful is at the basis

Yann LeCun (2:38:42.720)

of things like graph neural nets,

Lex Fridman (2:38:44.920)

which is an entirely new sub area of, you know,

Yann LeCun (2:38:49.400)

AI machine learning, deep learning,

Lex Fridman (2:38:50.680)

which I think is super promising

Yann LeCun (2:38:52.160)

for all kinds of applications.

Lex Fridman (2:38:54.360)

Something very promising,

Yann LeCun (2:38:55.240)

if you're more interested in applications,

Lex Fridman (2:38:56.680)

is the applications of AI machine learning

Lex Fridman (2:38:58.840)

and deep learning to science,

Lex Fridman (2:39:01.520)

or to science that can help solve big problems

Yann LeCun (2:39:05.120)

in the world.

Lex Fridman (2:39:05.960)

I have colleagues at Meta, at Fair,

Yann LeCun (2:39:09.240)

who started this project called Open Catalyst,

Lex Fridman (2:39:11.240)

and it's an open project collaborative.

Lex Fridman (2:39:14.560)

And the idea is to use deep learning

Lex Fridman (2:39:16.640)

to help design new chemical compounds or materials

Yann LeCun (2:39:21.960)

that would facilitate the separation

Lex Fridman (2:39:23.800)

of hydrogen from oxygen.

Yann LeCun (2:39:25.840)

If you can efficiently separate oxygen from hydrogen

Lex Fridman (2:39:29.080)

with electricity, you solve climate change.

Yann LeCun (2:39:33.520)

It's as simple as that,

Lex Fridman (2:39:34.480)

because you cover, you know,

Yann LeCun (2:39:37.640)

some random desert with solar panels,

Lex Fridman (2:39:40.800)

and you have them work all day,

Yann LeCun (2:39:42.560)

produce hydrogen,

Lex Fridman (2:39:43.480)

and then you shoot the hydrogen wherever it's needed.

Yann LeCun (2:39:45.400)

You don't need anything else.

Lex Fridman (2:39:48.560)

You know, you have controllable power

Yann LeCun (2:39:53.440)

that can be transported anywhere.

Lex Fridman (2:39:55.640)

So if we have a large scale,

Yann LeCun (2:39:59.040)

efficient energy storage technology,

Lex Fridman (2:40:02.160)

like producing hydrogen, we solve climate change.

Yann LeCun (2:40:06.640)

Here's another way to solve climate change,

Lex Fridman (2:40:08.560)

is figuring out how to make fusion work.

Yann LeCun (2:40:10.480)

Now, the problem with fusion

Lex Fridman (2:40:11.520)

is that you make a super hot plasma,

Lex Fridman (2:40:13.640)

and the plasma is unstable and you can't control it.

Lex Fridman (2:40:16.240)

Maybe with deep learning,

Yann LeCun (2:40:17.080)

you can find controllers that will stabilize plasma

Lex Fridman (2:40:19.120)

and make, you know, practical fusion reactors.

Yann LeCun (2:40:21.640)

I mean, that's very speculative,

Lex Fridman (2:40:23.080)

but, you know, it's worth trying,

Yann LeCun (2:40:24.480)

because, you know, the payoff is huge.

Lex Fridman (2:40:28.280)

There's a group at Google working on this,

Yann LeCun (2:40:29.880)

led by John Platt.

Lex Fridman (2:40:31.160)

So control, convert as many problems

Yann LeCun (2:40:33.920)

in science and physics and biology and chemistry

Lex Fridman (2:40:36.800)

into a learnable problem

Lex Fridman (2:40:39.760)

and see if a machine can learn it.

Lex Fridman (2:40:41.560)

Right, I mean, there's properties of, you know,

Yann LeCun (2:40:43.880)

complex materials that we don't understand

Lex Fridman (2:40:46.280)

from first principle, for example, right?

Yann LeCun (2:40:48.520)

So, you know, if we could design new, you know,

Lex Fridman (2:40:53.040)

new materials, we could make more efficient batteries.

Yann LeCun (2:40:56.400)

You know, we could make maybe faster electronics.

Lex Fridman (2:40:58.800)

We could, I mean, there's a lot of things we can imagine

Yann LeCun (2:41:01.920)

doing, or, you know, lighter materials

Lex Fridman (2:41:04.480)

for cars or airplanes or things like that.

Yann LeCun (2:41:06.400)

Maybe better fuel cells.

Lex Fridman (2:41:07.600)

I mean, there's all kinds of stuff we can imagine.

Yann LeCun (2:41:09.520)

If we had good fuel cells, hydrogen fuel cells,

Lex Fridman (2:41:12.280)

we could use them to power airplanes,

Yann LeCun (2:41:13.640)

and, you know, transportation wouldn't be, or cars,

Lex Fridman (2:41:17.240)

and we wouldn't have emission problem,

Yann LeCun (2:41:20.280)

CO2 emission problems for air transportation anymore.

Lex Fridman (2:41:24.600)

So there's a lot of those things, I think,

Yann LeCun (2:41:26.880)

where AI, you know, can be used.

Lex Fridman (2:41:30.160)

And this is not even talking about

Yann LeCun (2:41:31.560)

all the sort of medicine, biology,

Lex Fridman (2:41:33.520)

and everything like that, right?

Yann LeCun (2:41:35.680)

You know, like, you know, protein folding,

Lex Fridman (2:41:37.840)

you know, figuring out, like, how could you design

Yann LeCun (2:41:40.040)

your proteins that it sticks to another protein

Lex Fridman (2:41:41.880)

at a particular site, because that's how you design drugs

Yann LeCun (2:41:44.040)

in the end.

Lex Fridman (2:41:46.280)

So, you know, deep learning would be useful,

Yann LeCun (2:41:47.600)

although those are kind of, you know,

Lex Fridman (2:41:49.280)

would be sort of enormous progress

Yann LeCun (2:41:51.120)

if we could use it for that.

Lex Fridman (2:41:53.360)

Here's an example.

Yann LeCun (2:41:54.320)

If you take, this is like from recent material physics,

Lex Fridman (2:41:58.280)

you take a monoatomic layer of graphene, right?

Lex Fridman (2:42:02.200)

So it's just carbon on a hexagonal mesh,

Lex Fridman (2:42:04.920)

and you make this single atom thick.

Yann LeCun (2:42:09.120)

You put another one on top,

Lex Fridman (2:42:10.360)

you twist them by some magic number of degrees,

Yann LeCun (2:42:13.080)

three degrees or something.

Lex Fridman (2:42:14.800)

It becomes superconductor.

Yann LeCun (2:42:16.760)

Nobody has any idea why.

Lex Fridman (2:42:18.240)

Okay.

Yann LeCun (2:42:20.800)

I want to know how that was discovered,

Lex Fridman (2:42:22.480)

but that's the kind of thing that machine learning

Yann LeCun (2:42:23.920)

can actually discover, these kinds of things.

Lex Fridman (2:42:25.800)

Maybe not, but there is a hint, perhaps,

Yann LeCun (2:42:28.960)

that with machine learning, we would train a system

Lex Fridman (2:42:31.720)

to basically be a phenomenological model

Yann LeCun (2:42:34.840)

of some complex emergent phenomenon,

Lex Fridman (2:42:37.240)

which, you know, superconductivity is one of those,

Yann LeCun (2:42:42.400)

where, you know, this collective phenomenon

Lex Fridman (2:42:44.760)

is too difficult to describe from first principles

Yann LeCun (2:42:46.920)

with the current, you know,

Lex Fridman (2:42:48.800)

the usual sort of reductionist type method,

Lex Fridman (2:42:51.920)

but we could have deep learning systems

Lex Fridman (2:42:54.960)

that predict the properties of a system

Yann LeCun (2:42:57.680)

from a description of it after being trained

Lex Fridman (2:42:59.880)

with sufficiently many samples.

Yann LeCun (2:43:04.880)

This guy, Pascal Foua, at EPFL,

Lex Fridman (2:43:06.680)

he has a startup company that,

Yann LeCun (2:43:09.800)

where he basically trained a convolutional net,

Lex Fridman (2:43:13.440)

essentially, to predict the aerodynamic properties

Yann LeCun (2:43:16.640)

of solids, and you can generate as much data as you want

Lex Fridman (2:43:19.640)

by just running computational free dynamics, right?

Lex Fridman (2:43:21.920)

So you give, like, a wing, airfoil,

Lex Fridman (2:43:27.800)

or something, shape of some kind,

Lex Fridman (2:43:29.800)

and you run computational free dynamics,

Lex Fridman (2:43:31.400)

you get, as a result, the drag and, you know,

Lex Fridman (2:43:36.160)

lift and all that stuff, right?

Lex Fridman (2:43:37.480)

And you can generate lots of data,

Yann LeCun (2:43:40.080)

train a neural net to make those predictions,

Lex Fridman (2:43:41.840)

and now what you have is a differentiable model

Yann LeCun (2:43:44.120)

of, let's say, drag and lift

Lex Fridman (2:43:47.000)

as a function of the shape of that solid,

Lex Fridman (2:43:48.680)

and so you can do back rate and descent,

Lex Fridman (2:43:49.960)

you can optimize the shape

Lex Fridman (2:43:51.520)

so you get the properties you want.

Lex Fridman (2:43:54.880)

Yeah, that's incredible.

Yann LeCun (2:43:56.040)

That's incredible, and on top of all that,

Lex Fridman (2:43:58.280)

probably you should read a little bit of literature

Lex Fridman (2:44:01.480)

and a little bit of history

Lex Fridman (2:44:03.600)

for inspiration and for wisdom,

Yann LeCun (2:44:06.640)

because after all, all of these technologies

Lex Fridman (2:44:08.800)

will have to work in the human world.

Yann LeCun (2:44:10.280)

Yes.

Lex Fridman (2:44:11.120)

And the human world is complicated.

Yann LeCun (2:44:12.640)

It is, sadly.

Lex Fridman (2:44:15.080)

Jan, this is an amazing conversation.

Yann LeCun (2:44:18.440)

I'm really honored that you would talk with me today.

Lex Fridman (2:44:20.400)

Thank you for all the amazing work you're doing

Yann LeCun (2:44:22.240)

at FAIR, at Meta, and thank you for being so passionate

Lex Fridman (2:44:26.280)

after all these years about everything

Yann LeCun (2:44:28.120)

that's going on, you're a beacon of hope

Lex Fridman (2:44:29.960)

for the machine learning community,

Lex Fridman (2:44:31.600)

and thank you so much for spending

Lex Fridman (2:44:33.200)

your valuable time with me today.

Yann LeCun (2:44:34.480)

That was awesome.

Lex Fridman (2:44:35.320)

Thanks for having me on.

Yann LeCun (2:44:36.280)

That was a pleasure.

Lex Fridman (2:44:38.800)

Thanks for listening to this conversation with Jan Lacune.

Yann LeCun (2:44:41.440)

To support this podcast,

Lex Fridman (2:44:42.800)

please check out our sponsors in the description.

Lex Fridman (2:44:45.720)

And now, let me leave you with some words

Lex Fridman (2:44:47.840)

from Isaac Asimov.

Yann LeCun (2:44:50.640)

Your assumptions are your windows on the world.

Lex Fridman (2:44:53.760)

Scrub them off every once in a while,

Yann LeCun (2:44:56.040)

or the light won't come in.

Lex Fridman (2:44:58.760)

Thank you for listening, and hope to see you next time.

Lex Fridman (30:00.340)

And if there is something that recent progress

Lex Fridman (30:02.580)

in chess and Go has made us realize

Yann LeCun (30:05.580)

is that humans are really terrible at those things,

Lex Fridman (30:07.900)

like really bad.

Yann LeCun (30:09.660)

There was a story right before AlphaGo

Lex Fridman (30:11.540)

that the best Go players thought

Yann LeCun (30:15.220)

there were maybe two or three stones behind an ideal player

Lex Fridman (30:18.520)

that they would call God.

Yann LeCun (30:20.700)

In fact, no, there are like nine or 10 stones behind.

Lex Fridman (30:23.700)

I mean, we're just bad.

Lex Fridman (30:25.340)

So we're not good at,

Lex Fridman (30:27.420)

and it's because we have limited working memory.

Yann LeCun (30:30.340)

We're not very good at doing this tree exploration

Lex Fridman (30:32.980)

that computers are much better at doing than we are.

Lex Fridman (30:36.780)

But we are much better

Lex Fridman (30:37.940)

at learning differentiable models to the world.

Yann LeCun (30:40.620)

I mean, I said differentiable in a kind of,

Lex Fridman (30:43.820)

I should say not differentiable in the sense that

Yann LeCun (30:46.420)

we went back far through it,

Lex Fridman (30:47.480)

but in the sense that our brain has some mechanism

Yann LeCun (30:50.500)

for estimating gradients of some kind.

Lex Fridman (30:54.060)

And that's what makes us efficient.

Lex Fridman (30:56.540)

So if you have an agent that consists of a model

Lex Fridman (31:02.180)

of the world, which in the human brain

Yann LeCun (31:04.380)

is basically the entire front half of your brain,

Lex Fridman (31:08.340)

an objective function,

Yann LeCun (31:10.220)

which in humans is a combination of two things.

Lex Fridman (31:14.440)

There is your sort of intrinsic motivation module,

Yann LeCun (31:17.660)

which is in the basal ganglia,

Lex Fridman (31:19.140)

the base of your brain.

Yann LeCun (31:20.100)

That's the thing that measures pain and hunger

Lex Fridman (31:22.540)

and things like that,

Yann LeCun (31:23.360)

like immediate feelings and emotions.

Lex Fridman (31:28.020)

And then there is the equivalent

Yann LeCun (31:30.780)

of what people in reinforcement learning call a critic,

Lex Fridman (31:32.620)

which is a sort of module that predicts ahead

Lex Fridman (31:36.100)

what the outcome of a situation will be.

Lex Fridman (31:41.940)

And so it's not a cost function,

Lex Fridman (31:43.840)

but it's sort of not an objective function,

Lex Fridman (31:45.460)

but it's sort of a train predictor

Yann LeCun (31:49.020)

of the ultimate objective function.

Lex Fridman (31:50.980)

And that also is differentiable.

Lex Fridman (31:52.620)

And so if all of this is differentiable,

Lex Fridman (31:54.660)

your cost function, your critic, your world model,

Yann LeCun (31:59.660)

then you can use gradient based type methods

Lex Fridman (32:03.100)

to do planning, to do reasoning, to do learning,

Yann LeCun (32:05.820)

to do all the things that we'd like

Lex Fridman (32:08.140)

an intelligent agent to do.

Lex Fridman (32:11.840)

And gradient based learning,

Lex Fridman (32:14.180)

like what's your intuition?

Yann LeCun (32:15.340)

That's probably at the core of what can solve intelligence.

Lex Fridman (32:18.420)

So you don't need like logic based reasoning in your view.

Yann LeCun (32:25.620)

I don't know how to make logic based reasoning

Lex Fridman (32:27.260)

compatible with efficient learning.

Yann LeCun (32:31.020)

Okay, I mean, there is a big question,

Lex Fridman (32:32.300)

perhaps a philosophical question.

Yann LeCun (32:33.900)

I mean, it's not that philosophical,

Lex Fridman (32:35.220)

but that we can ask is that all the learning algorithms

Yann LeCun (32:40.020)

we know from engineering and computer science

Lex Fridman (32:43.300)

proceed by optimizing some objective function.

Lex Fridman (32:48.340)

So one question we may ask is,

Lex Fridman (32:51.780)

does learning in the brain minimize an objective function?

Yann LeCun (32:54.740)

I mean, it could be a composite

Lex Fridman (32:57.340)

of multiple objective functions,

Lex Fridman (32:58.500)

but it's still an objective function.

Lex Fridman (33:01.420)

Second, if it does optimize an objective function,

Lex Fridman (33:04.660)

does it do it by some sort of gradient estimation?

Lex Fridman (33:09.940)

It doesn't need to be a back prop,

Lex Fridman (33:10.860)

but some way of estimating the gradient in efficient manner

Lex Fridman (33:14.820)

whose complexity is on the same order of magnitude

Yann LeCun (33:17.020)

as actually running the inference.

Lex Fridman (33:20.800)

Because you can't afford to do things

Yann LeCun (33:24.060)

like perturbing a weight in your brain

Lex Fridman (33:26.540)

to figure out what the effect is.

Lex Fridman (33:28.100)

And then sort of, you can do sort of

Lex Fridman (33:30.780)

estimating gradient by perturbation.

Yann LeCun (33:33.300)

To me, it seems very implausible

Lex Fridman (33:35.460)

that the brain uses some sort of zeroth order black box

Yann LeCun (33:41.060)

gradient free optimization,

Lex Fridman (33:43.000)

because it's so much less efficient

Yann LeCun (33:45.200)

than gradient optimization.

Lex Fridman (33:46.320)

So it has to have a way of estimating gradient.

Yann LeCun (33:49.260)

Is it possible that some kind of logic based reasoning

Lex Fridman (33:52.780)

emerges in pockets as a useful,

Yann LeCun (33:55.400)

like you said, if the brain is an objective function,

Lex Fridman (33:58.100)

maybe it's a mechanism for creating objective functions.

Yann LeCun (34:01.300)

It's a mechanism for creating knowledge bases, for example,

Lex Fridman (34:06.520)

that can then be queried.

Yann LeCun (34:08.380)

Like maybe it's like an efficient representation

Lex Fridman (34:10.300)

of knowledge that's learned in a gradient based way

Yann LeCun (34:12.700)

or something like that.

Lex Fridman (34:13.780)

Well, so I think there is a lot of different types

Yann LeCun (34:15.980)

of intelligence.

Lex Fridman (34:17.340)

So first of all, I think the type of logical reasoning

Yann LeCun (34:19.700)

that we think about that we are maybe stemming

Lex Fridman (34:23.780)

from sort of classical AI of the 1970s and 80s.

Yann LeCun (34:29.080)

I think humans use that relatively rarely

Lex Fridman (34:33.020)

and are not particularly good at it.

Lex Fridman (34:34.740)

But we judge each other based on our ability

Lex Fridman (34:37.560)

to solve those rare problems.

Yann LeCun (34:40.620)

It's called an IQ test.

Lex Fridman (34:41.660)

I don't think so.

Yann LeCun (34:42.700)

Like I'm not very good at chess.

Lex Fridman (34:45.260)

Yes, I'm judging you this whole time.

Yann LeCun (34:47.420)

Because, well, we actually.

Lex Fridman (34:49.740)

With your heritage, I'm sure you're good at chess.

Yann LeCun (34:53.500)

No, stereotypes.

Lex Fridman (34:55.060)

Not all stereotypes are true.

Yann LeCun (34:58.020)

Well, I'm terrible at chess.

Lex Fridman (34:59.020)

So, but I think perhaps another type of intelligence

Yann LeCun (35:04.660)

that I have is this ability of sort of building models

Lex Fridman (35:08.980)

to the world from reasoning obviously,

Lex Fridman (35:13.820)

but also data.

Lex Fridman (35:15.980)

And those models generally are more kind of analogical.

Lex Fridman (35:18.900)

So it's reasoning by simulation,

Lex Fridman (35:22.380)

and by analogy, where you use one model

Yann LeCun (35:25.120)

to apply to a new situation.

Lex Fridman (35:26.900)

Even though you've never seen that situation,

Yann LeCun (35:28.500)

you can sort of connect it to a situation

Lex Fridman (35:31.620)

you've encountered before.

Lex Fridman (35:33.500)

And your reasoning is more akin

Lex Fridman (35:36.700)

to some sort of internal simulation.

Lex Fridman (35:38.420)

So you're kind of simulating what's happening

Lex Fridman (35:41.140)

when you're building, I don't know,

Lex Fridman (35:42.240)

a box out of wood or something, right?

Lex Fridman (35:44.100)

You can imagine in advance what would be the result

Yann LeCun (35:47.460)

of cutting the wood in this particular way.

Lex Fridman (35:49.660)

Are you going to use screws or nails or whatever?

Yann LeCun (35:52.900)

When you are interacting with someone,

Lex Fridman (35:54.180)

you also have a model of that person

Lex Fridman (35:55.780)

and sort of interact with that person,

Lex Fridman (35:59.580)

having this model in mind to kind of tell the person

Lex Fridman (36:03.660)

what you think is useful to them.

Lex Fridman (36:05.280)

So I think this ability to construct models to the world

Yann LeCun (36:10.220)

is basically the essence, the essence of intelligence.

Lex Fridman (36:13.900)

And the ability to use it then to plan actions

Yann LeCun (36:18.220)

that will fulfill a particular criterion,

Lex Fridman (36:23.080)

of course, is necessary as well.

Lex Fridman (36:25.460)

So I'm going to ask you a series of impossible questions

Lex Fridman (36:27.740)

as we keep asking, as I've been doing.

Lex Fridman (36:30.180)

So if that's the fundamental sort of dark matter

Lex Fridman (36:33.460)

of intelligence, this ability to form a background model,

Lex Fridman (36:36.580)

what's your intuition about how much knowledge is required?

Lex Fridman (36:41.460)

You know, I think dark matter,

Yann LeCun (36:43.100)

you could put a percentage on it

Lex Fridman (36:45.980)

of the composition of the universe

Lex Fridman (36:50.060)

and how much of it is dark matter,

Lex Fridman (36:51.460)

how much of it is dark energy,

Lex Fridman (36:52.640)

how much information do you think is required

Lex Fridman (36:57.900)

to be a house cat?

Lex Fridman (36:59.920)

So you have to be able to, when you see a box going in,

Lex Fridman (37:02.900)

when you see a human compute the most evil action,

Yann LeCun (37:06.220)

if there's a thing that's near an edge,

Lex Fridman (37:07.940)

you knock it off, all of that,

Yann LeCun (37:10.980)

plus the extra stuff you mentioned,

Lex Fridman (37:12.740)

which is a great self awareness of the physics

Yann LeCun (37:15.700)

of your own body and the world.

Lex Fridman (37:18.740)

How much knowledge is required, do you think, to solve it?

Yann LeCun (37:22.500)

I don't even know how to measure an answer to that question.

Lex Fridman (37:25.620)

I'm not sure how to measure it,

Lex Fridman (37:26.680)

but whatever it is, it fits in about 800,000 neurons,

Lex Fridman (37:32.380)

800 million neurons.

Lex Fridman (37:33.900)

What's the representation does?

Lex Fridman (37:36.300)

Everything, all knowledge, everything, right?

Yann LeCun (37:40.100)

You know, it's less than a billion.

Lex Fridman (37:41.500)

A dog is 2 billion, but a cat is less than 1 billion.

Lex Fridman (37:45.500)

And so multiply that by a thousand

Lex Fridman (37:48.140)

and you get the number of synapses.

Lex Fridman (37:50.300)

And I think almost all of it is learned

Lex Fridman (37:52.780)

through this, you know, a sort of self supervised running,

Yann LeCun (37:55.940)

although, you know, I think a tiny sliver

Lex Fridman (37:58.500)

is learned through reinforcement running

Lex Fridman (37:59.900)

and certainly very little through, you know,

Lex Fridman (38:02.220)

classical supervised running,

Yann LeCun (38:03.340)

although it's not even clear how supervised running

Lex Fridman (38:05.180)

actually works in the biological world.

Lex Fridman (38:09.260)

So I think almost all of it is self supervised running,

Lex Fridman (38:12.860)

but it's driven by the sort of ingrained objective functions

Yann LeCun (38:18.180)

that a cat or a human have at the base of their brain,

Lex Fridman (38:21.400)

which kind of drives their behavior.

Yann LeCun (38:24.880)

So, you know, nature tells us you're hungry.

Lex Fridman (38:29.480)

It doesn't tell us how to feed ourselves.

Yann LeCun (38:31.900)

That's something that the rest of our brain

Lex Fridman (38:33.500)

has to figure out, right?

Yann LeCun (38:35.780)

What's interesting is there might be more

Lex Fridman (38:37.940)

like deeper objective functions

Yann LeCun (38:39.660)

than allowing the whole thing.

Lex Fridman (38:41.300)

So hunger may be some kind of,

Yann LeCun (38:44.500)

now you go to like neurobiology,

Lex Fridman (38:46.140)

it might be just the brain trying to maintain homeostasis.

Lex Fridman (38:52.460)

So hunger is just one of the human perceivable symptoms

Lex Fridman (38:58.020)

of the brain being unhappy

Yann LeCun (38:59.380)

with the way things are currently.

Lex Fridman (39:01.460)

It could be just like one really dumb objective function

Yann LeCun (39:04.140)

at the core.

Lex Fridman (39:04.980)

But that's how behavior is driven.

Yann LeCun (39:08.460)

The fact that, you know, or basal ganglia

Lex Fridman (39:12.360)

drive us to do things that are different

Yann LeCun (39:14.820)

from say an orangutan or certainly a cat

Lex Fridman (39:18.180)

is what makes, you know, human nature

Yann LeCun (39:20.060)

versus orangutan nature versus cat nature.

Lex Fridman (39:23.280)

So for example, you know, our basal ganglia

Yann LeCun (39:27.100)

drives us to seek the company of other humans.

Lex Fridman (39:32.220)

And that's because nature has figured out

Yann LeCun (39:34.540)

that we need to be social animals for our species to survive.

Lex Fridman (39:37.540)

And it's true of many primates.

Yann LeCun (39:41.300)

It's not true of orangutans.

Lex Fridman (39:42.620)

Orangutans are solitary animals.

Yann LeCun (39:44.900)

They don't seek the company of others.

Lex Fridman (39:46.900)

In fact, they avoid them.

Yann LeCun (39:49.300)

In fact, they scream at them when they come too close

Lex Fridman (39:51.060)

because they're territorial.

Yann LeCun (39:52.740)

Because for their survival, you know,

Lex Fridman (39:55.900)

evolution has figured out that's the best thing.

Yann LeCun (39:58.300)

I mean, they're occasionally social, of course,

Lex Fridman (40:00.040)

for, you know, reproduction and stuff like that.

Lex Fridman (40:03.500)

But they're mostly solitary.

Lex Fridman (40:05.920)

So all of those behaviors are not part of intelligence.

Yann LeCun (40:09.540)

You know, people say,

Lex Fridman (40:10.380)

oh, you're never gonna have intelligent machines

Yann LeCun (40:11.800)

because, you know, human intelligence is social.

Lex Fridman (40:13.940)

But then you look at orangutans, you look at octopus.

Yann LeCun (40:16.820)

Octopus never know their parents.

Lex Fridman (40:18.800)

They barely interact with any other.

Lex Fridman (40:20.500)

And they get to be really smart in less than a year,

Lex Fridman (40:23.900)

in like half a year.

Yann LeCun (40:26.040)

You know, in a year, they're adults.

Lex Fridman (40:27.620)

In two years, they're dead.

Lex Fridman (40:28.780)

So there are things that we think, as humans,

Lex Fridman (40:33.620)

are intimately linked with intelligence,

Yann LeCun (40:35.740)

like social interaction, like language.

Lex Fridman (40:39.760)

We think, I think we give way too much importance

Yann LeCun (40:42.860)

to language as a substrate of intelligence as humans.

Lex Fridman (40:46.780)

Because we think our reasoning is so linked with language.

Lex Fridman (40:49.840)

So to solve the house cat intelligence problem,

Lex Fridman (40:53.460)

you think you could do it on a desert island.

Yann LeCun (40:55.500)

You could have, you could just have a cat sitting there

Lex Fridman (41:00.360)

looking at the waves, at the ocean waves,

Lex Fridman (41:03.180)

and figure a lot of it out.

Lex Fridman (41:05.740)

It needs to have sort of, you know,

Yann LeCun (41:07.500)

the right set of drives to kind of, you know,

Lex Fridman (41:11.540)

get it to do the thing and learn the appropriate things,

Yann LeCun (41:13.980)

right, but like for example, you know,

Lex Fridman (41:17.660)

baby humans are driven to learn to stand up and walk.

Yann LeCun (41:22.660)

You know, that's kind of, this desire is hardwired.

Lex Fridman (41:26.020)

How to do it precisely is not, that's learned.

Lex Fridman (41:28.540)

But the desire to walk, move around and stand up,

Lex Fridman (41:32.840)

that's sort of probably hardwired.

Lex Fridman (41:35.940)

But it's very simple to hardwire this kind of stuff.

Lex Fridman (41:38.940)

Oh, like the desire to, well, that's interesting.

Yann LeCun (41:42.780)

You're hardwired to want to walk.

Lex Fridman (41:45.620)

That's not, there's gotta be a deeper need for walking.

Yann LeCun (41:50.460)

I think it was probably socially imposed by society

Lex Fridman (41:53.140)

that you need to walk all the other bipedal.

Yann LeCun (41:55.580)

No, like a lot of simple animals that, you know,

Lex Fridman (41:58.420)

will probably walk without ever watching

Yann LeCun (42:01.040)

any other members of the species.

Lex Fridman (42:03.900)

It seems like a scary thing to have to do

Yann LeCun (42:06.820)

because you suck at bipedal walking at first.

Lex Fridman (42:09.280)

It seems crawling is much safer, much more like,

Lex Fridman (42:13.820)

why are you in a hurry?

Lex Fridman (42:15.700)

Well, because you have this thing that drives you to do it,

Yann LeCun (42:18.660)

you know, which is sort of part of the sort of

Lex Fridman (42:24.220)

human development.

Lex Fridman (42:25.060)

Is that understood actually what?

Lex Fridman (42:26.700)

Not entirely, no.

Lex Fridman (42:28.220)

What's the reason you get on two feet?

Lex Fridman (42:29.740)

It's really hard.

Yann LeCun (42:30.620)

Like most animals don't get on two feet.

Lex Fridman (42:32.780)

Well, they get on four feet.

Yann LeCun (42:33.980)

You know, many mammals get on four feet.

Lex Fridman (42:35.740)

Yeah, they do. Very quickly.

Yann LeCun (42:36.760)

Some of them extremely quickly.

Lex Fridman (42:38.500)

But I don't, you know, like from the last time

Yann LeCun (42:41.380)

I've interacted with a table,

Lex Fridman (42:42.620)

that's much more stable than a thing than two legs.

Yann LeCun (42:44.940)

It's just a really hard problem.

Lex Fridman (42:46.420)

Yeah, I mean, birds have figured it out with two feet.

Yann LeCun (42:48.620)

Well, technically we can go into ontology.

Lex Fridman (42:52.020)

They have four, I guess they have two feet.

Yann LeCun (42:54.500)

They have two feet.

Lex Fridman (42:55.340)

Chickens.

Yann LeCun (42:56.380)

You know, dinosaurs have two feet, many of them.

Lex Fridman (42:58.860)

Allegedly.

Yann LeCun (43:01.560)

I'm just now learning that T. rex was eating grass,

Lex Fridman (43:04.340)

not other animals.

Yann LeCun (43:05.420)

T. rex might've been a friendly pet.

Lex Fridman (43:08.020)

What do you think about,

Yann LeCun (43:10.320)

I don't know if you looked at the test

Lex Fridman (43:13.500)

for general intelligence that François Chollet put together.

Yann LeCun (43:16.380)

I don't know if you got a chance to look

Lex Fridman (43:18.000)

at that kind of thing.

Yann LeCun (43:19.660)

What's your intuition about how to solve

Lex Fridman (43:21.860)

like an IQ type of test?

Yann LeCun (43:23.740)

I don't know.

Lex Fridman (43:24.580)

I think it's so outside of my radar screen

Yann LeCun (43:26.140)

that it's not really relevant, I think, in the short term.

Lex Fridman (43:30.740)

Well, I guess one way to ask,

Yann LeCun (43:33.100)

another way, perhaps more closer to what do you work is like,

Lex Fridman (43:37.780)

how do you solve MNIST with very little example data?

Yann LeCun (43:42.740)

That's right.

Lex Fridman (43:43.560)

And that's the answer to this probably

Yann LeCun (43:44.860)

is self supervised learning.

Lex Fridman (43:45.860)

Just learn to represent images

Lex Fridman (43:47.300)

and then learning to recognize handwritten digits

Lex Fridman (43:51.060)

on top of this will only require a few samples.

Lex Fridman (43:53.620)

And we observe this in humans, right?

Lex Fridman (43:55.460)

You show a young child a picture book

Yann LeCun (43:58.660)

with a couple of pictures of an elephant and that's it.

Lex Fridman (44:01.940)

The child knows what an elephant is.

Lex Fridman (44:03.900)

And we see this today with practical systems

Lex Fridman (44:06.700)

that we train image recognition systems

Yann LeCun (44:09.540)

with enormous amounts of images,

Lex Fridman (44:13.660)

either completely self supervised

Yann LeCun (44:15.740)

or very weakly supervised.

Lex Fridman (44:16.980)

For example, you can train a neural net

Lex Fridman (44:20.900)

to predict whatever hashtag people type on Instagram, right?

Lex Fridman (44:24.180)

Then you can do this with billions of images

Yann LeCun (44:25.780)

because there's billions per day that are showing up.

Lex Fridman (44:28.540)

So the amount of training data there

Yann LeCun (44:30.700)

is essentially unlimited.

Lex Fridman (44:32.340)

And then you take the output representation,

Yann LeCun (44:35.380)

a couple of layers down from the outputs

Lex Fridman (44:37.380)

of what the system learned and feed this as input

Yann LeCun (44:40.680)

to a classifier for any object in the world that you want

Lex Fridman (44:43.780)

and it works pretty well.

Lex Fridman (44:44.940)

So that's transfer learning, okay?

Lex Fridman (44:47.620)

Or weakly supervised transfer learning.

Yann LeCun (44:51.340)

People are making very, very fast progress

Lex Fridman (44:53.460)

using self supervised learning

Yann LeCun (44:55.300)

for this kind of scenario as well.

Lex Fridman (44:58.580)

And my guess is that that's gonna be the future.

Yann LeCun (45:02.500)

For self supervised learning,

Lex Fridman (45:03.660)

how much cleaning do you think is needed

Lex Fridman (45:06.800)

for filtering malicious signal or what's a better term?

Lex Fridman (45:11.800)

But like a lot of people use hashtags on Instagram

Yann LeCun (45:16.760)

to get like good SEO that doesn't fully represent

Lex Fridman (45:21.200)

the contents of the image.

Yann LeCun (45:23.100)

Like they'll put a picture of a cat

Lex Fridman (45:24.520)

and hashtag it with like science, awesome, fun.

Lex Fridman (45:28.060)

I don't know all kinds, why would you put science?

Lex Fridman (45:31.200)

That's not very good SEO.

Yann LeCun (45:33.080)

The way my colleagues who worked on this project

Lex Fridman (45:34.960)

at Facebook, now Meta AI, a few years ago dealt with this

Yann LeCun (45:39.960)

is that they only selected something like 17,000 tags

Lex Fridman (45:43.760)

that correspond to kind of physical things or situations,

Yann LeCun (45:48.100)

like that has some visual content.

Lex Fridman (45:52.320)

So you wouldn't have like hash TBT or anything like that.

Yann LeCun (45:57.120)

Oh, so they keep a very select set of hashtags

Lex Fridman (46:00.820)

is what you're saying?

Yann LeCun (46:01.660)

Yeah.

Lex Fridman (46:02.480)

Okay.

Lex Fridman (46:03.320)

But it's still in the order of 10 to 20,000.

Lex Fridman (46:06.080)

So it's fairly large.

Yann LeCun (46:07.960)

Okay.

Lex Fridman (46:09.040)

Can you tell me about data augmentation?

Lex Fridman (46:11.280)

What the heck is data augmentation and how is it used

Lex Fridman (46:14.760)

maybe contrast of learning for video?

Lex Fridman (46:19.080)

What are some cool ideas here?

Lex Fridman (46:20.880)

Right, so data augmentation.

Yann LeCun (46:22.120)

I mean, first data augmentation is the idea

Lex Fridman (46:24.520)

of artificially increasing the size of your training set

Yann LeCun (46:26.960)

by distorting the images that you have

Lex Fridman (46:30.020)

in ways that don't change the nature of the image, right?

Lex Fridman (46:32.360)

So you do MNIST, you can do data augmentation on MNIST

Lex Fridman (46:35.520)

and people have done this since the 1990s, right?

Yann LeCun (46:37.360)

You take a MNIST digit and you shift it a little bit

Lex Fridman (46:40.880)

or you change the size or rotate it, skew it,

Yann LeCun (46:45.800)

you know, et cetera.

Lex Fridman (46:47.000)

Add noise.

Yann LeCun (46:48.280)

Add noise, et cetera.

Lex Fridman (46:49.520)

And it works better if you train a supervised classifier

Yann LeCun (46:52.440)

with augmented data, you're gonna get better results.

Lex Fridman (46:55.600)

Now it's become really interesting

Yann LeCun (46:58.640)

over the last couple of years

Lex Fridman (47:00.400)

because a lot of self supervised learning techniques

Yann LeCun (47:04.160)

to pre train vision systems are based on data augmentation.

Lex Fridman (47:07.980)

And the basic techniques is originally inspired

Yann LeCun (47:12.000)

by techniques that I worked on in the early 90s

Lex Fridman (47:15.840)

and Jeff Hinton worked on also in the early 90s.

Yann LeCun (47:17.720)

They were sort of parallel work.

Lex Fridman (47:20.040)

I used to call this Siamese network.

Lex Fridman (47:21.600)

So basically you take two identical copies

Lex Fridman (47:24.960)

of the same network, they share the same weights

Lex Fridman (47:27.720)

and you show two different views of the same object.

Lex Fridman (47:31.760)

Either those two different views may have been obtained

Yann LeCun (47:33.920)

by data augmentation

Lex Fridman (47:35.440)

or maybe it's two different views of the same scene

Yann LeCun (47:37.680)

from a camera that you moved or at different times

Lex Fridman (47:40.280)

or something like that, right?

Yann LeCun (47:41.400)

Or two pictures of the same person, things like that.

Lex Fridman (47:44.400)

And then you train this neural net,

Yann LeCun (47:46.480)

those two identical copies of this neural net

Lex Fridman (47:48.420)

to produce an output representation, a vector

Yann LeCun (47:52.460)

in such a way that the representation for those two images

Lex Fridman (47:56.560)

are as close to each other as possible,

Lex Fridman (47:58.880)

as identical to each other as possible, right?

Lex Fridman (48:00.840)

Because you want the system

Yann LeCun (48:02.040)

to basically learn a function that will be invariant,

Lex Fridman (48:06.120)

that will not change, whose output will not change

Yann LeCun (48:08.200)

when you transform those inputs in those particular ways,

Lex Fridman (48:12.480)

right?

Lex Fridman (48:14.080)

So that's easy to do.

Lex Fridman (48:15.680)

What's complicated is how do you make sure

Yann LeCun (48:17.720)

that when you show two images that are different,

Lex Fridman (48:19.520)

the system will produce different things?

Yann LeCun (48:21.960)

Because if you don't have a specific provision for this,

Lex Fridman (48:26.200)

the system will just ignore the inputs when you train it,

Yann LeCun (48:29.160)

it will end up ignoring the input

Lex Fridman (48:30.360)

and just produce a constant vector

Lex Fridman (48:31.740)

that is the same for every input, right?

Lex Fridman (48:33.680)

That's called a collapse.

Lex Fridman (48:35.200)

Now, how do you avoid collapse?

Lex Fridman (48:36.720)

So there's two ideas.

Yann LeCun (48:38.840)

One idea that I proposed in the early 90s

Lex Fridman (48:41.560)

with my colleagues at Bell Labs,

Yann LeCun (48:43.120)

Jane Barmley and a couple other people,

Lex Fridman (48:46.280)

which we now call contrastive learning,

Lex Fridman (48:48.280)

which is to have negative examples, right?

Lex Fridman (48:50.020)

So you have pairs of images that you know are different

Lex Fridman (48:54.400)

and you show them to the network and those two copies,

Lex Fridman (48:57.480)

and then you push the two output vectors away

Yann LeCun (48:59.760)

from each other and it will eventually guarantee

Lex Fridman (49:02.200)

that things that are semantically similar

Yann LeCun (49:04.880)

produce similar representations

Lex Fridman (49:06.480)

and things that are different

Yann LeCun (49:07.320)

produce different representations.

Lex Fridman (49:10.280)

We actually came up with this idea

Yann LeCun (49:11.440)

for a project of doing signature verification.

Lex Fridman (49:14.480)

So we would collect signatures from,

Yann LeCun (49:18.400)

like multiple signatures on the same person

Lex Fridman (49:20.160)

and then train a neural net to produce the same representation

Lex Fridman (49:23.280)

and then force the system to produce different

Lex Fridman (49:27.880)

representation for different signatures.

Yann LeCun (49:31.000)

This was actually, the problem was proposed by people

Lex Fridman (49:33.460)

from what was a subsidiary of AT&T at the time called NCR.

Lex Fridman (49:38.240)

And they were interested in storing

Lex Fridman (49:40.360)

representation of the signature on the 80 bytes

Yann LeCun (49:43.500)

of the magnetic strip of a credit card.

Lex Fridman (49:46.640)

So we came up with this idea of having a neural net

Yann LeCun (49:48.800)

with 80 outputs that we would quantize on bytes

Lex Fridman (49:52.280)

so that we could encode the signature.

Lex Fridman (49:53.840)

And that encoding was then used to compare

Lex Fridman (49:55.440)

whether the signature matches or not.

Yann LeCun (49:57.080)

That's right.

Lex Fridman (49:57.920)

So then you would sign, you would run through the neural net

Lex Fridman (50:00.640)

and then you would compare the output vector

Lex Fridman (50:02.400)

to whatever is stored on your card.

Lex Fridman (50:03.240)

Did it actually work?

Lex Fridman (50:04.640)

It worked, but they ended up not using it.

Yann LeCun (50:08.940)

Because nobody cares actually.

Lex Fridman (50:10.120)

I mean, the American financial payment system

Yann LeCun (50:13.800)

is incredibly lax in that respect compared to Europe.

Lex Fridman (50:17.560)

Oh, with the signatures?

Lex Fridman (50:18.960)

What's the purpose of signatures anyway?

Lex Fridman (50:20.520)

This is very different.

Yann LeCun (50:21.360)

Nobody looks at them, nobody cares.

Lex Fridman (50:23.280)

It's, yeah.

Lex Fridman (50:24.440)

Yeah, no, so that's contrastive learning, right?

Lex Fridman (50:27.840)

So you need positive and negative pairs.

Lex Fridman (50:29.440)

And the problem with that is that,

Lex Fridman (50:31.760)

even though I had the original paper on this,

Yann LeCun (50:34.760)

I'm actually not very positive about it

Lex Fridman (50:36.800)

because it doesn't work in high dimension.

Yann LeCun (50:38.640)

If your representation is high dimensional,

Lex Fridman (50:41.040)

there's just too many ways for two things to be different.

Lex Fridman (50:44.300)

And so you would need lots and lots

Lex Fridman (50:45.960)

and lots of negative pairs.

Lex Fridman (50:48.260)

So there is a particular implementation of this,

Lex Fridman (50:50.800)

which is relatively recent from actually

Yann LeCun (50:52.840)

the Google Toronto group where, you know,

Lex Fridman (50:56.040)

Jeff Hinton is the senior member there.

Yann LeCun (50:58.800)

It's called SIMCLR, S I M C L R.

Lex Fridman (51:02.000)

And it, you know, basically a particular way

Yann LeCun (51:03.720)

of implementing this idea of contrastive learning,

Lex Fridman (51:06.760)

the particular objective function.

Yann LeCun (51:08.600)

Now, what I'm much more enthusiastic about these days

Lex Fridman (51:13.160)

is non contrastive methods.

Lex Fridman (51:14.600)

So other ways to guarantee that the representations

Lex Fridman (51:19.600)

would be different for different inputs.

Lex Fridman (51:24.200)

And it's actually based on an idea that Jeff Hinton

Lex Fridman (51:28.320)

proposed in the early nineties with his student

Yann LeCun (51:30.360)

at the time, Sue Becker.

Lex Fridman (51:31.960)

And it's based on the idea of maximizing

Yann LeCun (51:33.440)

the mutual information between the outputs

Lex Fridman (51:35.000)

of the two systems.

Yann LeCun (51:36.200)

You only show positive pairs.

Lex Fridman (51:37.480)

You only show pairs of images that you know

Yann LeCun (51:39.160)

are somewhat similar.

Lex Fridman (51:41.640)

And you train the two networks to be informative,

Lex Fridman (51:44.200)

but also to be as informative of each other as possible.

Lex Fridman (51:48.880)

So basically one representation has to be predictable

Yann LeCun (51:51.400)

from the other, essentially.

Lex Fridman (51:54.520)

And, you know, he proposed that idea,

Yann LeCun (51:56.400)

had, you know, a couple of papers in the early nineties,

Lex Fridman (51:59.440)

and then nothing was done about it for decades.

Lex Fridman (52:02.280)

And I kind of revived this idea together

Lex Fridman (52:04.360)

with my postdocs at FAIR,

Yann LeCun (52:07.480)

particularly a postdoc called Stefan Denis,

Lex Fridman (52:08.920)

who is now a junior professor in Finland

Yann LeCun (52:11.800)

at University of Aalto.

Lex Fridman (52:13.240)

We came up with something that we call Barlow Twins.

Lex Fridman (52:18.240)

And it's a particular way of maximizing

Lex Fridman (52:20.520)

the information content of a vector,

Yann LeCun (52:24.240)

you know, using some hypotheses.

Lex Fridman (52:27.920)

And we have kind of another version of it

Yann LeCun (52:30.920)

that's more recent now called VICREG, V I C A R E G.

Lex Fridman (52:33.480)

That means Variance, Invariance, Covariance,

Yann LeCun (52:35.960)

Regularization.

Lex Fridman (52:36.800)

And it's the thing I'm the most excited about

Yann LeCun (52:38.840)

in machine learning in the last 15 years.

Lex Fridman (52:40.600)

I mean, I'm not, I'm really, really excited about this.

Lex Fridman (52:43.360)

What kind of data augmentation is useful

Lex Fridman (52:46.400)

for that noncontrastive learning method?

Lex Fridman (52:49.280)

Are we talking about, does that not matter that much?

Lex Fridman (52:51.680)

Or it seems like a very important part of the step.

Yann LeCun (52:55.040)

Yeah.

Lex Fridman (52:55.880)

How you generate the images that are similar,

Lex Fridman (52:57.120)

but sufficiently different.

Lex Fridman (52:58.680)

Yeah, that's right.

Yann LeCun (52:59.520)

It's an important step and it's also an annoying step

Lex Fridman (53:01.440)

because you need to have that knowledge

Yann LeCun (53:02.840)

of what data augmentation you can do

Lex Fridman (53:05.840)

that do not change the nature of the object.

Lex Fridman (53:09.320)

And so the standard scenario,

Lex Fridman (53:12.280)

which a lot of people working in this area are using

Yann LeCun (53:14.520)

is you use the type of distortion.

Lex Fridman (53:18.720)

So basically you do a geometric distortion.

Lex Fridman (53:21.160)

So one basically just shifts the image a little bit,

Lex Fridman (53:23.360)

it's called cropping.

Yann LeCun (53:24.400)

Another one kind of changes the scale a little bit.

Lex Fridman (53:26.880)

Another one kind of rotates it.

Yann LeCun (53:28.240)

Another one changes the colors.

Lex Fridman (53:30.000)

You can do a shift in color balance

Yann LeCun (53:32.040)

or something like that, saturation.

Lex Fridman (53:34.880)

Another one sort of blurs it.

Yann LeCun (53:36.240)

Another one adds noise.

Lex Fridman (53:37.080)

So you have like a catalog of kind of standard things

Lex Fridman (53:40.040)

and people try to use the same ones

Lex Fridman (53:42.120)

for different algorithms so that they can compare.

Lex Fridman (53:44.960)

But some algorithms, some self supervised algorithm

Lex Fridman (53:47.200)

actually can deal with much bigger,

Yann LeCun (53:49.600)

like more aggressive data augmentation and some don't.

Lex Fridman (53:52.480)

So that kind of makes the whole thing difficult.

Lex Fridman (53:55.400)

But that's the kind of distortions we're talking about.

Lex Fridman (53:57.760)

And so you train with those distortions

Lex Fridman (54:02.520)

and then you chop off the last layer, a couple layers

Lex Fridman (54:07.400)

of the network and you use the representation

Yann LeCun (54:11.480)

as input to a classifier.

Lex Fridman (54:12.680)

You train the classifier on ImageNet, let's say,

Yann LeCun (54:16.680)

or whatever, and measure the performance.

Lex Fridman (54:19.600)

And interestingly enough, the methods that are really good

Yann LeCun (54:23.520)

at eliminating the information that is irrelevant,

Lex Fridman (54:25.960)

which is the distortions between those images,

Yann LeCun (54:29.200)

do a good job at eliminating it.

Lex Fridman (54:31.480)

And as a consequence, you cannot use the representations

Yann LeCun (54:36.480)

in those systems for things like object detection

Lex Fridman (54:39.080)

and localization because that information is gone.

Lex Fridman (54:41.480)

So the type of data augmentation you need to do

Lex Fridman (54:44.760)

depends on the tasks you want eventually the system

Yann LeCun (54:47.720)

to solve and the type of data augmentation,

Lex Fridman (54:50.680)

standard data augmentation that we use today

Yann LeCun (54:52.560)

are only appropriate for object recognition

Lex Fridman (54:54.720)

or image classification.

Yann LeCun (54:56.040)

They're not appropriate for things like.

Lex Fridman (54:57.760)

Can you help me out understand what wide localizations?

Lex Fridman (55:00.800)

So you're saying it's just not good at the negative,

Lex Fridman (55:03.760)

like at classifying the negative,

Lex Fridman (55:05.440)

so that's why it can't be used for the localization?

Lex Fridman (55:07.920)

No, it's just that you train the system,

Yann LeCun (55:10.360)

you give it an image and then you give it the same image

Lex Fridman (55:13.560)

shifted and scaled and you tell it that's the same image.

Lex Fridman (55:17.400)

So the system basically is trained

Lex Fridman (55:19.160)

to eliminate the information about position and size.

Lex Fridman (55:22.040)

So now you want to use that to figure out

Lex Fridman (55:26.200)

where an object is and what size it is.

Yann LeCun (55:27.760)

Like a bounding box, like they'd be able to actually.

Lex Fridman (55:30.040)

Okay, it can still find the object in the image,

Yann LeCun (55:34.160)

it's just not very good at finding

Lex Fridman (55:35.960)

the exact boundaries of that object, interesting.

Yann LeCun (55:38.960)

Interesting, which that's an interesting

Lex Fridman (55:42.040)

sort of philosophical question,

Lex Fridman (55:43.480)

how important is object localization anyway?

Lex Fridman (55:46.800)

We're like obsessed by measuring image segmentation,

Yann LeCun (55:51.240)

obsessed by measuring perfectly knowing

Lex Fridman (55:53.420)

the boundaries of objects when arguably

Yann LeCun (55:56.760)

that's not that essential to understanding

Lex Fridman (56:01.840)

what are the contents of the scene.

Yann LeCun (56:03.800)

On the other hand, I think evolutionarily,

Lex Fridman (56:05.880)

the first vision systems in animals

Yann LeCun (56:08.200)

were basically all about localization,

Lex Fridman (56:10.040)

very little about recognition.

Lex Fridman (56:12.480)

And in the human brain, you have two separate pathways

Lex Fridman (56:15.320)

for recognizing the nature of a scene or an object

Lex Fridman (56:20.880)

and localizing objects.

Lex Fridman (56:22.320)

So you use the first pathway called eventual pathway

Yann LeCun (56:25.200)

for telling what you're looking at.

Lex Fridman (56:29.140)

The other pathway, the dorsal pathway,

Yann LeCun (56:30.560)

is used for navigation, for grasping, for everything else.

Lex Fridman (56:34.120)

And basically a lot of the things you need for survival

Yann LeCun (56:36.920)

are localization and detection.

Lex Fridman (56:41.880)

Is similarity learning or contrastive learning,

Yann LeCun (56:45.080)

are these non contrastive methods

Lex Fridman (56:46.520)

the same as understanding something?

Yann LeCun (56:48.880)

Just because you know a distorted cat

Lex Fridman (56:50.680)

is the same as a non distorted cat,

Lex Fridman (56:52.600)

does that mean you understand what it means to be a cat?

Lex Fridman (56:56.760)

To some extent.

Yann LeCun (56:57.600)

I mean, it's a superficial understanding, obviously.

Lex Fridman (57:00.120)

But what is the ceiling of this method, do you think?

Yann LeCun (57:02.360)

Is this just one trick on the path

Lex Fridman (57:05.120)

to doing self supervised learning?

Lex Fridman (57:07.320)

Can we go really, really far?

Lex Fridman (57:10.040)

I think we can go really far.

Lex Fridman (57:11.280)

So if we figure out how to use techniques of that type,

Lex Fridman (57:16.400)

perhaps very different, but the same nature,

Yann LeCun (57:19.480)

to train a system from video to do video prediction,

Lex Fridman (57:23.360)

essentially, I think we'll have a path towards,

Yann LeCun (57:30.440)

I wouldn't say unlimited, but a path towards some level

Lex Fridman (57:33.520)

of physical common sense in machines.

Lex Fridman (57:38.120)

And I also think that that ability to learn

Lex Fridman (57:44.440)

how the world works from a sort of high throughput channel

Yann LeCun (57:47.720)

like vision is a necessary step towards

Lex Fridman (57:53.520)

sort of real artificial intelligence.

Yann LeCun (57:55.560)

In other words, I believe in grounded intelligence.

Lex Fridman (57:58.080)

I don't think we can train a machine

Yann LeCun (57:59.920)

to be intelligent purely from text.

Lex Fridman (58:02.200)

Because I think the amount of information about the world

Yann LeCun (58:04.960)

that's contained in text is tiny compared

Lex Fridman (58:07.680)

to what we need to know.

Lex Fridman (58:11.600)

So for example, and people have attempted to do this

Lex Fridman (58:15.320)

for 30 years, the psych project and things like that,

Yann LeCun (58:18.920)

basically kind of writing down all the facts that are known

Lex Fridman (58:21.160)

and hoping that some sort of common sense will emerge.

Yann LeCun (58:25.240)

I think it's basically hopeless.

Lex Fridman (58:27.160)

But let me take an example.

Yann LeCun (58:28.320)

You take an object, I describe a situation to you.

Lex Fridman (58:31.280)

I take an object, I put it on the table

Lex Fridman (58:33.560)

and I push the table.

Lex Fridman (58:34.960)

It's completely obvious to you that the object

Yann LeCun (58:37.240)

will be pushed with the table,

Lex Fridman (58:39.240)

because it's sitting on it.

Yann LeCun (58:41.840)

There's no text in the world, I believe, that explains this.

Lex Fridman (58:45.040)

And so if you train a machine as powerful as it could be,

Yann LeCun (58:49.040)

your GPT 5000 or whatever it is,

Lex Fridman (58:53.920)

it's never gonna learn about this.

Yann LeCun (58:57.040)

That information is just not present in any text.

Lex Fridman (59:01.040)

Well, the question, like with the psych project,

Yann LeCun (59:03.280)

the dream I think is to have like 10 million,

Lex Fridman (59:08.000)

say facts like that, that give you a headstart,

Yann LeCun (59:13.000)

like a parent guiding you.

Lex Fridman (59:15.200)

Now, we humans don't need a parent to tell us

Yann LeCun (59:17.280)

that the table will move, sorry,

Lex Fridman (59:19.240)

the smartphone will move with the table.

Lex Fridman (59:21.440)

But we get a lot of guidance in other ways.

Lex Fridman (59:25.640)

So it's possible that we can give it a quick shortcut.

Lex Fridman (59:28.160)

What about a cat?

Lex Fridman (59:29.200)

The cat knows that.

Yann LeCun (59:30.800)

No, but they evolved, so.

Lex Fridman (59:33.120)

No, they learn like us.

Lex Fridman (59:35.840)

Sorry, the physics of stuff?

Lex Fridman (59:37.080)

Yeah.

Yann LeCun (59:38.480)

Well, yeah, so you're saying it's,

Lex Fridman (59:41.360)

so you're putting a lot of intelligence

Yann LeCun (59:45.080)

onto the nurture side, not the nature.

Lex Fridman (59:47.120)

Yes.

Yann LeCun (59:47.960)

We seem to have, you know,

Lex Fridman (59:50.000)

there's a very inefficient arguably process of evolution

Yann LeCun (59:53.640)

that got us from bacteria to who we are today.

Lex Fridman (59:57.840)

Started at the bottom, now we're here.

Lex Fridman (59:59.800)

So the question is how, okay,