Yann LeCun: Deep Learning, Convolutional Neural Networks, and Self-Supervised Learning
AI 与机器学习心理与人性音乐与艺术技术与编程生物与进化
📋 章节目录
暂无章节信息
🔑 关键词
learningdonobjectivemodelhumandataintelligencemachinegeneralsupervisedgoingknowledgeneuralsystemstrainreasoningselfmemorymachinesworks
💬 精彩语录
暂无语录
🎙️ 完整对话（1753 条）
Lex Fridman (00:00.000)
The following is a conversation with Yann LeCun.
Lex Fridman (00:03.080)
He's considered to be one of the fathers of deep learning,
Lex Fridman (00:06.320)
which, if you've been hiding under a rock,
Lex Fridman (00:09.040)
is the recent revolution in AI that has captivated the world
Lex Fridman (00:12.240)
with the possibility of what machines can learn from data.
Yann LeCun (00:16.160)
He's a professor at New York University,
Lex Fridman (00:18.520)
a vice president and chief AI scientist at Facebook,
Lex Fridman (00:21.720)
and co recipient of the Turing Award
Lex Fridman (00:24.320)
for his work on deep learning.
Yann LeCun (00:26.240)
He's probably best known as the founding father
Lex Fridman (00:28.880)
of convolutional neural networks,
Yann LeCun (00:30.720)
in particular their application
Lex Fridman (00:32.480)
to optical character recognition
Lex Fridman (00:34.400)
and the famed MNIST dataset.
Lex Fridman (00:37.240)
He is also an outspoken personality,
Yann LeCun (00:40.100)
unafraid to speak his mind in a distinctive French accent
Lex Fridman (00:43.800)
and explore provocative ideas,
Yann LeCun (00:45.720)
both in the rigorous medium of academic research
Lex Fridman (00:48.360)
and the somewhat less rigorous medium
Yann LeCun (00:51.000)
of Twitter and Facebook.
Lex Fridman (00:52.800)
This is the Artificial Intelligence Podcast.
Yann LeCun (00:55.600)
If you enjoy it, subscribe on YouTube,
Lex Fridman (00:57.960)
give it five stars on iTunes, support it on Patreon,
Yann LeCun (01:00.960)
or simply connect with me on Twitter at Lex Friedman,
Lex Fridman (01:03.840)
spelled F R I D M A N.
Lex Fridman (01:06.840)
And now, here's my conversation with Yann LeCun.
Lex Fridman (01:11.720)
You said that 2001 Space Odyssey
Yann LeCun (01:13.820)
is one of your favorite movies.
Lex Fridman (01:16.260)
Hal 9000 decides to get rid of the astronauts
Yann LeCun (01:20.360)
for people who haven't seen the movie, spoiler alert,
Lex Fridman (01:23.040)
because he, it, she believes that the astronauts,
Yann LeCun (01:29.200)
they will interfere with the mission.
Lex Fridman (01:31.600)
Do you see Hal as flawed in some fundamental way
Lex Fridman (01:34.720)
or even evil, or did he do the right thing?
Lex Fridman (01:38.440)
Neither.
Yann LeCun (01:39.320)
There's no notion of evil in that context,
Lex Fridman (01:43.240)
other than the fact that people die,
Lex Fridman (01:44.760)
but it was an example of what people call
Lex Fridman (01:48.720)
value misalignment, right?
Yann LeCun (01:50.120)
You give an objective to a machine,
Lex Fridman (01:52.120)
and the machine strives to achieve this objective.
Lex Fridman (01:55.560)
And if you don't put any constraints on this objective,
Lex Fridman (01:58.160)
like don't kill people and don't do things like this,
Yann LeCun (02:02.260)
the machine, given the power, will do stupid things
Lex Fridman (02:06.240)
just to achieve this objective,
Yann LeCun (02:08.000)
or damaging things to achieve this objective.
Lex Fridman (02:10.200)
It's a little bit like, I mean, we're used to this
Yann LeCun (02:12.440)
in the context of human society.
Lex Fridman (02:15.740)
We put in place laws to prevent people
Yann LeCun (02:20.740)
from doing bad things, because spontaneously,
Lex Fridman (02:22.920)
they would do those bad things, right?
Lex Fridman (02:24.800)
So we have to shape their cost function,
Lex Fridman (02:28.400)
their objective function, if you want,
Yann LeCun (02:29.500)
through laws to kind of correct,
Lex Fridman (02:31.520)
and education, obviously, to sort of correct for those.
Lex Fridman (02:36.120)
So maybe just pushing a little further on that point,
Lex Fridman (02:41.960)
how, you know, there's a mission,
Yann LeCun (02:44.360)
there's this fuzziness around,
Lex Fridman (02:46.400)
the ambiguity around what the actual mission is,
Yann LeCun (02:49.800)
but, you know, do you think that there will be a time,
Lex Fridman (02:55.120)
from a utilitarian perspective,
Yann LeCun (02:56.720)
where an AI system, where it is not misalignment,
Lex Fridman (02:59.660)
where it is alignment, for the greater good of society,
Lex Fridman (03:02.820)
that an AI system will make decisions that are difficult?
Lex Fridman (03:05.880)
Well, that's the trick.
Yann LeCun (03:06.800)
I mean, eventually we'll have to figure out how to do this.
Lex Fridman (03:10.800)
And again, we're not starting from scratch,
Yann LeCun (03:12.600)
because we've been doing this with humans for millennia.
Lex Fridman (03:16.440)
So designing objective functions for people
Yann LeCun (03:19.160)
is something that we know how to do.
Lex Fridman (03:20.880)
And we don't do it by, you know, programming things,
Yann LeCun (03:24.600)
although the legal code is called code.
Lex Fridman (03:29.060)
So that tells you something.
Lex Fridman (03:30.640)
And it's actually the design of an objective function.
Lex Fridman (03:33.040)
That's really what legal code is, right?
Yann LeCun (03:34.600)
It tells you, here is what you can do,
Lex Fridman (03:36.280)
here is what you can't do.
Yann LeCun (03:37.420)
If you do it, you pay that much,
Lex Fridman (03:39.000)
that's an objective function.
Lex Fridman (03:41.680)
So there is this idea somehow that it's a new thing
Lex Fridman (03:44.600)
for people to try to design objective functions
Yann LeCun (03:46.600)
that are aligned with the common good.
Lex Fridman (03:47.940)
But no, we've been writing laws for millennia
Lex Fridman (03:49.880)
and that's exactly what it is.
Lex Fridman (03:52.080)
So that's where, you know, the science of lawmaking
Lex Fridman (03:57.120)
and computer science will.
Lex Fridman (04:00.560)
Come together.
Yann LeCun (04:01.400)
Will come together.
Lex Fridman (04:02.840)
So there's nothing special about HAL or AI systems,
Yann LeCun (04:06.800)
it's just the continuation of tools used
Lex Fridman (04:09.480)
to make some of these difficult ethical judgments
Yann LeCun (04:11.740)
that laws make.
Lex Fridman (04:13.020)
Yeah, and we have systems like this already
Yann LeCun (04:15.080)
that make many decisions for ourselves in society
Lex Fridman (04:19.960)
that need to be designed in a way that they,
Yann LeCun (04:22.600)
like rules about things that sometimes have bad side effects
Lex Fridman (04:27.480)
and we have to be flexible enough about those rules
Lex Fridman (04:29.600)
so that they can be broken when it's obvious
Lex Fridman (04:31.560)
that they shouldn't be applied.
Lex Fridman (04:34.000)
So you don't see this on the camera here,
Lex Fridman (04:35.640)
but all the decoration in this room
Yann LeCun (04:36.920)
is all pictures from 2001 and Space Odyssey.
Lex Fridman (04:41.360)
Wow, is that by accident or is there a lot?
Yann LeCun (04:43.720)
No, by accident, it's by design.
Lex Fridman (04:47.440)
Oh, wow.
Lex Fridman (04:48.480)
So if you were to build HAL 10,000,
Lex Fridman (04:52.560)
so an improvement of HAL 9,000, what would you improve?
Yann LeCun (04:57.120)
Well, first of all, I wouldn't ask it to hold secrets
Lex Fridman (05:00.680)
and tell lies because that's really what breaks it
Yann LeCun (05:03.440)
in the end, that's the fact that it's asking itself
Lex Fridman (05:06.680)
questions about the purpose of the mission
Lex Fridman (05:08.840)
and it's, you know, pieces things together that it's heard,
Lex Fridman (05:11.560)
you know, all the secrecy of the preparation of the mission
Lex Fridman (05:14.000)
and the fact that it was the discovery
Lex Fridman (05:16.440)
on the lunar surface that really was kept secret
Lex Fridman (05:19.560)
and one part of HAL's memory knows this
Lex Fridman (05:22.360)
and the other part does not know it
Lex Fridman (05:24.720)
and is supposed to not tell anyone
Lex Fridman (05:26.720)
and that creates internal conflict.
Lex Fridman (05:28.600)
So you think there's never should be a set of things
Lex Fridman (05:32.240)
that an AI system should not be allowed,
Yann LeCun (05:36.600)
like a set of facts that should not be shared
Lex Fridman (05:39.920)
with the human operators?
Yann LeCun (05:42.400)
Well, I think, no, I think it should be a bit like
Lex Fridman (05:46.600)
in the design of autonomous AI systems,
Yann LeCun (05:52.040)
there should be the equivalent of, you know,
Lex Fridman (05:54.280)
the oath that a hypocrite oath
Lex Fridman (05:59.080)
that doctors sign up to, right?
Lex Fridman (06:02.640)
So there's certain things, certain rules
Yann LeCun (06:04.120)
that you have to abide by and we can sort of hardwire this
Lex Fridman (06:07.280)
into our machines to kind of make sure they don't go.
Lex Fridman (06:11.000)
So I'm not, you know, an advocate of the three laws
Lex Fridman (06:14.720)
of robotics, you know, the Asimov kind of thing
Yann LeCun (06:17.120)
because I don't think it's practical,
Lex Fridman (06:18.560)
but, you know, some level of limits.
Lex Fridman (06:23.240)
But to be clear, these are not questions
Lex Fridman (06:27.960)
that are kind of really worth asking today
Yann LeCun (06:32.040)
because we just don't have the technology to do this.
Lex Fridman (06:34.360)
We don't have autonomous intelligent machines,
Yann LeCun (06:36.440)
we have intelligent machines.
Lex Fridman (06:37.560)
Some are intelligent machines that are very specialized,
Lex Fridman (06:41.000)
but they don't really sort of satisfy an objective.
Lex Fridman (06:43.360)
They're just, you know, kind of trained to do one thing.
Lex Fridman (06:46.520)
So until we have some idea for design
Lex Fridman (06:50.000)
of a full fledged autonomous intelligent system,
Yann LeCun (06:53.360)
asking the question of how we design this objective,
Lex Fridman (06:55.680)
I think is a little too abstract.
Yann LeCun (06:58.600)
It's a little too abstract.
Lex Fridman (06:59.680)
There's useful elements to it in that it helps us understand
Yann LeCun (07:04.240)
our own ethical codes, humans.
Lex Fridman (07:07.960)
So even just as a thought experiment,
Yann LeCun (07:10.240)
if you imagine that an AGI system is here today,
Lex Fridman (07:14.280)
how would we program it is a kind of nice thought experiment
Yann LeCun (07:17.640)
of constructing how should we have a law,
Lex Fridman (07:21.880)
have a system of laws for us humans.
Yann LeCun (07:24.360)
It's just a nice practical tool.
Lex Fridman (07:26.800)
And I think there's echoes of that idea too
Yann LeCun (07:29.760)
in the AI systems we have today
Lex Fridman (07:32.160)
that don't have to be that intelligent.
Yann LeCun (07:33.960)
Yeah.
Lex Fridman (07:34.800)
Like autonomous vehicles.
Yann LeCun (07:35.640)
These things start creeping in that are worth thinking about,
Lex Fridman (07:39.200)
but certainly they shouldn't be framed as how.
Yann LeCun (07:42.600)
Yeah.
Lex Fridman (07:43.720)
Looking back, what is the most,
Yann LeCun (07:46.720)
I'm sorry if it's a silly question,
Lex Fridman (07:49.440)
but what is the most beautiful
Yann LeCun (07:51.440)
or surprising idea in deep learning
Lex Fridman (07:53.800)
or AI in general that you've ever come across?
Yann LeCun (07:56.320)
Sort of personally, when you said back
Lex Fridman (08:00.040)
and just had this kind of,
Yann LeCun (08:01.960)
oh, that's pretty cool moment.
Lex Fridman (08:03.920)
That's nice.
Yann LeCun (08:04.760)
That's surprising.
Lex Fridman (08:05.600)
I don't know if it's an idea
Yann LeCun (08:06.560)
rather than a sort of empirical fact.
Lex Fridman (08:12.160)
The fact that you can build gigantic neural nets,
Yann LeCun (08:16.440)
train them on relatively small amounts of data relatively
Lex Fridman (08:23.400)
with stochastic gradient descent
Lex Fridman (08:24.840)
and that it actually works,
Lex Fridman (08:26.920)
breaks everything you read in every textbook, right?
Yann LeCun (08:29.240)
Every pre deep learning textbook that told you,
Lex Fridman (08:32.560)
you need to have fewer parameters
Lex Fridman (08:33.920)
and you have data samples.
Lex Fridman (08:37.080)
If you have a non convex objective function,
Yann LeCun (08:38.760)
you have no guarantee of convergence.
Lex Fridman (08:40.680)
All those things that you read in textbook
Lex Fridman (08:42.080)
and they tell you to stay away from this
Lex Fridman (08:43.640)
and they're all wrong.
Yann LeCun (08:45.120)
The huge number of parameters, non convex,
Lex Fridman (08:48.080)
and somehow which is very relative
Yann LeCun (08:50.320)
to the number of parameters data,
Lex Fridman (08:53.480)
it's able to learn anything.
Yann LeCun (08:54.840)
Right.
Lex Fridman (08:55.680)
Does that still surprise you today?
Yann LeCun (08:57.520)
Well, it was kind of obvious to me
Lex Fridman (09:00.360)
before I knew anything that this is a good idea.
Lex Fridman (09:04.120)
And then it became surprising that it worked
Lex Fridman (09:06.040)
because I started reading those textbooks.
Yann LeCun (09:09.240)
Okay.
Lex Fridman (09:10.080)
Okay.
Lex Fridman (09:10.920)
So can you talk through the intuition
Lex Fridman (09:12.280)
of why it was obvious to you if you remember?
Yann LeCun (09:14.360)
Well, okay.
Lex Fridman (09:15.200)
So the intuition was it's sort of like,
Yann LeCun (09:17.360)
those people in the late 19th century
Lex Fridman (09:19.960)
who proved that heavier than air flight was impossible.
Lex Fridman (09:25.480)
And of course you have birds, right?
Lex Fridman (09:26.800)
They do fly.
Lex Fridman (09:28.280)
And so on the face of it,
Lex Fridman (09:30.400)
it's obviously wrong as an empirical question, right?
Lex Fridman (09:33.200)
And so we have the same kind of thing
Lex Fridman (09:34.640)
that we know that the brain works.
Yann LeCun (09:38.560)
We don't know how, but we know it works.
Lex Fridman (09:39.920)
And we know it's a large network of neurons and interaction
Lex Fridman (09:43.160)
and that learning takes place by changing the connection.
Lex Fridman (09:45.360)
So kind of getting this level of inspiration
Yann LeCun (09:48.000)
without copying the details,
Lex Fridman (09:49.320)
but sort of trying to derive basic principles,
Lex Fridman (09:52.520)
and that kind of gives you a clue
Lex Fridman (09:56.760)
as to which direction to go.
Yann LeCun (09:58.320)
There's also the idea somehow that I've been convinced of
Lex Fridman (10:01.120)
since I was an undergrad that, even before,
Yann LeCun (10:04.640)
that intelligence is inseparable from learning.
Lex Fridman (10:06.840)
So the idea somehow that you can create
Yann LeCun (10:10.000)
an intelligent machine by basically programming,
Lex Fridman (10:14.040)
for me it was a non starter from the start.
Yann LeCun (10:17.600)
Every intelligent entity that we know about
Lex Fridman (10:20.280)
arrives at this intelligence through learning.
Lex Fridman (10:24.960)
So machine learning was a completely obvious path.
Lex Fridman (10:29.960)
Also because I'm lazy, so, you know, kind of.
Yann LeCun (10:32.000)
He's automate basically everything
Lex Fridman (10:35.160)
and learning is the automation of intelligence.
Lex Fridman (10:37.840)
So do you think, so what is learning then?
Lex Fridman (10:42.920)
What falls under learning?
Lex Fridman (10:44.520)
Because do you think of reasoning as learning?
Lex Fridman (10:48.240)
Well, reasoning is certainly a consequence
Yann LeCun (10:51.600)
of learning as well, just like other functions of the brain.
Lex Fridman (10:56.600)
The big question about reasoning is,
Lex Fridman (10:58.160)
how do you make reasoning compatible
Lex Fridman (11:00.680)
with gradient based learning?
Lex Fridman (11:02.720)
Do you think neural networks can be made to reason?
Lex Fridman (11:04.960)
Yes, there is no question about that.
Lex Fridman (11:07.080)
Again, we have a good example, right?
Lex Fridman (11:10.320)
The question is how?
Lex Fridman (11:11.680)
So the question is how much prior structure
Lex Fridman (11:14.040)
do you have to put in the neural net
Lex Fridman (11:15.360)
so that something like human reasoning
Lex Fridman (11:17.480)
will emerge from it, you know, from learning?
Yann LeCun (11:20.840)
Another question is all of our kind of model
Lex Fridman (11:24.600)
of what reasoning is that are based on logic
Yann LeCun (11:27.240)
are discrete and are therefore incompatible
Lex Fridman (11:31.120)
with gradient based learning.
Lex Fridman (11:32.720)
And I'm a very strong believer
Lex Fridman (11:34.120)
in this idea of gradient based learning.
Yann LeCun (11:35.840)
I don't believe that other types of learning
Lex Fridman (11:39.280)
that don't use kind of gradient information if you want.
Lex Fridman (11:41.920)
So you don't like discrete mathematics?
Lex Fridman (11:43.400)
You don't like anything discrete?
Yann LeCun (11:45.000)
Well, that's, it's not that I don't like it,
Lex Fridman (11:46.920)
it's just that it's incompatible with learning
Lex Fridman (11:49.200)
and I'm a big fan of learning, right?
Lex Fridman (11:51.120)
So in fact, that's perhaps one reason
Lex Fridman (11:53.600)
why deep learning has been kind of looked at
Lex Fridman (11:57.040)
with suspicion by a lot of computer scientists
Yann LeCun (11:58.720)
because the math is very different.
Lex Fridman (11:59.920)
The math that you use for deep learning,
Yann LeCun (12:02.480)
you know, it kind of has more to do with,
Lex Fridman (12:05.040)
you know, cybernetics, the kind of math you do
Yann LeCun (12:08.280)
in electrical engineering than the kind of math
Lex Fridman (12:10.600)
you do in computer science.
Lex Fridman (12:12.240)
And, you know, nothing in machine learning is exact, right?
Lex Fridman (12:15.680)
Computer science is all about sort of, you know,
Yann LeCun (12:18.520)
obviously compulsive attention to details of like,
Lex Fridman (12:21.960)
you know, every index has to be right.
Lex Fridman (12:23.760)
And you can prove that an algorithm is correct, right?
Lex Fridman (12:26.760)
Machine learning is the science of sloppiness, really.
Yann LeCun (12:30.360)
That's beautiful.
Lex Fridman (12:32.920)
So, okay, maybe let's feel around in the dark
Yann LeCun (12:38.200)
of what is a neural network that reasons
Lex Fridman (12:41.400)
or a system that works with continuous functions
Yann LeCun (12:47.840)
that's able to do, build knowledge,
Lex Fridman (12:52.400)
however we think about reasoning,
Yann LeCun (12:54.280)
build on previous knowledge, build on extra knowledge,
Lex Fridman (12:57.880)
create new knowledge,
Yann LeCun (12:59.520)
generalize outside of any training set to ever build.
Lex Fridman (13:03.100)
What does that look like?
Yann LeCun (13:04.560)
If, yeah, maybe give inklings of thoughts
Lex Fridman (13:08.780)
of what that might look like.
Yann LeCun (13:10.860)
Yeah, I mean, yes and no.
Lex Fridman (13:12.320)
If I had precise ideas about this,
Yann LeCun (13:14.220)
I think, you know, we'd be building it right now.
Lex Fridman (13:17.280)
And there are people working on this
Lex Fridman (13:19.120)
whose main research interest is actually exactly that, right?
Lex Fridman (13:22.240)
So what you need to have is a working memory.
Lex Fridman (13:25.320)
So you need to have some device, if you want,
Lex Fridman (13:29.940)
some subsystem that can store a relatively large number
Yann LeCun (13:34.600)
of factual episodic information for, you know,
Lex Fridman (13:39.080)
a reasonable amount of time.
Yann LeCun (13:40.920)
So, you know, in the brain, for example,
Lex Fridman (13:43.920)
there are kind of three main types of memory.
Yann LeCun (13:45.800)
One is the sort of memory of the state of your cortex.
Lex Fridman (13:53.760)
And that sort of disappears within 20 seconds.
Yann LeCun (13:55.920)
You can't remember things for more than about 20 seconds
Lex Fridman (13:58.280)
or a minute if you don't have any other form of memory.
Yann LeCun (14:02.440)
The second type of memory, which is longer term,
Lex Fridman (14:04.480)
is still short term, is the hippocampus.
Lex Fridman (14:06.200)
So you can, you know, you came into this building,
Lex Fridman (14:08.360)
you remember where the exit is, where the elevators are.
Yann LeCun (14:14.000)
You have some map of that building
Lex Fridman (14:15.560)
that's stored in your hippocampus.
Yann LeCun (14:17.520)
You might remember something about what I said,
Lex Fridman (14:20.240)
you know, a few minutes ago.
Yann LeCun (14:21.400)
I forgot it all already.
Lex Fridman (14:22.320)
Of course, it's been erased, but, you know,
Lex Fridman (14:24.420)
but that would be in your hippocampus.
Lex Fridman (14:27.360)
And then the longer term memory is in the synapse,
Lex Fridman (14:30.700)
the synapses, right?
Lex Fridman (14:32.880)
So what you need if you want a system
Yann LeCun (14:34.640)
that's capable of reasoning
Lex Fridman (14:35.600)
is that you want the hippocampus like thing, right?
Lex Fridman (14:40.240)
And that's what people have tried to do
Lex Fridman (14:41.800)
with memory networks and, you know,
Lex Fridman (14:43.720)
neural training machines and stuff like that, right?
Lex Fridman (14:45.800)
And now with transformers,
Yann LeCun (14:47.200)
which have sort of a memory in there,
Lex Fridman (14:50.540)
kind of self attention system.
Yann LeCun (14:51.980)
You can think of it this way.
Lex Fridman (14:55.720)
So that's one element you need.
Yann LeCun (14:57.160)
Another thing you need is some sort of network
Lex Fridman (14:59.880)
that can access this memory,
Yann LeCun (15:03.240)
get an information back and then kind of crunch on it
Lex Fridman (15:08.160)
and then do this iteratively multiple times
Yann LeCun (15:10.920)
because a chain of reasoning is a process
Lex Fridman (15:15.860)
by which you update your knowledge
Yann LeCun (15:19.400)
about the state of the world,
Lex Fridman (15:20.400)
about, you know, what's going to happen, et cetera.
Lex Fridman (15:22.820)
And that has to be this sort of
Lex Fridman (15:25.440)
recurrent operation basically.
Lex Fridman (15:27.120)
And you think that kind of,
Lex Fridman (15:29.160)
if we think about a transformer,
Lex Fridman (15:31.120)
so that seems to be too small
Lex Fridman (15:32.640)
to contain the knowledge that's,
Yann LeCun (15:36.240)
to represent the knowledge
Lex Fridman (15:37.280)
that's contained in Wikipedia, for example.
Yann LeCun (15:39.260)
Well, a transformer doesn't have this idea of recurrence.
Lex Fridman (15:42.000)
It's got a fixed number of layers
Lex Fridman (15:43.120)
and that's the number of steps that, you know,
Lex Fridman (15:44.680)
limits basically its representation.
Lex Fridman (15:47.120)
But recurrence would build on the knowledge somehow.
Lex Fridman (15:51.240)
I mean, it would evolve the knowledge
Lex Fridman (15:54.760)
and expand the amount of information perhaps
Lex Fridman (15:58.080)
or useful information within that knowledge.
Lex Fridman (16:00.360)
But is this something that just can emerge with size?
Lex Fridman (16:04.800)
Because it seems like everything we have now is too small.
Yann LeCun (16:06.440)
Not just, no, it's not clear.
Lex Fridman (16:09.360)
I mean, how you access and write
Yann LeCun (16:11.160)
into an associative memory in an efficient way.
Lex Fridman (16:13.800)
I mean, sort of the original memory network
Yann LeCun (16:15.240)
maybe had something like the right architecture,
Lex Fridman (16:17.560)
but if you try to scale up a memory network
Lex Fridman (16:20.540)
so that the memory contains all the Wikipedia,
Lex Fridman (16:22.880)
it doesn't quite work.
Yann LeCun (16:24.040)
Right.
Lex Fridman (16:25.120)
So there's a need for new ideas there, okay.
Lex Fridman (16:28.680)
But it's not the only form of reasoning.
Lex Fridman (16:30.000)
So there's another form of reasoning,
Yann LeCun (16:31.400)
which is true, which is very classical also
Lex Fridman (16:34.160)
in some types of AI.
Lex Fridman (16:36.720)
And it's based on, let's call it energy minimization.
Lex Fridman (16:40.920)
Okay, so you have some sort of objective,
Yann LeCun (16:44.960)
some energy function that represents
Lex Fridman (16:47.200)
the quality or the negative quality, okay.
Yann LeCun (16:53.320)
Energy goes up when things get bad
Lex Fridman (16:54.740)
and they get low when things get good.
Lex Fridman (16:57.320)
So let's say you want to figure out,
Lex Fridman (17:00.480)
what gestures do I need to do
Yann LeCun (17:03.960)
to grab an object or walk out the door.
Lex Fridman (17:08.200)
If you have a good model of your own body,
Yann LeCun (17:10.360)
a good model of the environment,
Lex Fridman (17:12.500)
using this kind of energy minimization,
Yann LeCun (17:14.360)
you can do planning.
Lex Fridman (17:16.920)
And in optimal control,
Yann LeCun (17:19.280)
it's called model predictive control.
Lex Fridman (17:22.140)
You have a model of what's gonna happen in the world
Yann LeCun (17:24.140)
as a consequence of your actions.
Lex Fridman (17:25.520)
And that allows you to, by energy minimization,
Yann LeCun (17:28.600)
figure out the sequence of action
Lex Fridman (17:29.800)
that optimizes a particular objective function,
Yann LeCun (17:32.080)
which measures, minimizes the number of times
Lex Fridman (17:34.160)
you're gonna hit something
Lex Fridman (17:35.000)
and the energy you're gonna spend
Lex Fridman (17:36.540)
doing the gesture and et cetera.
Lex Fridman (17:39.800)
So that's a form of reasoning.
Lex Fridman (17:42.440)
Planning is a form of reasoning.
Lex Fridman (17:43.520)
And perhaps what led to the ability of humans to reason
Lex Fridman (17:48.040)
is the fact that, or species that appear before us
Yann LeCun (17:53.480)
had to do some sort of planning
Lex Fridman (17:55.080)
to be able to hunt and survive
Lex Fridman (17:56.960)
and survive the winter in particular.
Lex Fridman (17:59.600)
And so it's the same capacity that you need to have.
Lex Fridman (18:03.360)
So in your intuition is,
Lex Fridman (18:07.600)
if we look at expert systems
Lex Fridman (18:09.520)
and encoding knowledge as logic systems,
Lex Fridman (18:13.240)
as graphs, in this kind of way,
Lex Fridman (18:16.720)
is not a useful way to think about knowledge?
Lex Fridman (18:20.280)
Graphs are a little brittle or logic representation.
Lex Fridman (18:23.960)
So basically, variables that have values
Lex Fridman (18:27.880)
and then constraint between them
Yann LeCun (18:29.280)
that are represented by rules,
Lex Fridman (18:31.300)
is a little too rigid and too brittle, right?
Lex Fridman (18:32.860)
So some of the early efforts in that respect
Lex Fridman (18:38.640)
were to put probabilities on them.
Lex Fridman (18:41.020)
So a rule, if you have this and that symptom,
Lex Fridman (18:44.560)
you have this disease with that probability
Lex Fridman (18:47.200)
and you should prescribe that antibiotic
Lex Fridman (18:49.400)
with that probability, right?
Yann LeCun (18:50.520)
That's the mycin system from the 70s.
Lex Fridman (18:54.320)
And that's what that branch of AI led to,
Yann LeCun (18:58.520)
based on networks and graphical models
Lex Fridman (19:00.320)
and causal inference and variational method.
Lex Fridman (19:04.960)
So there is certainly a lot of interesting
Lex Fridman (19:10.240)
work going on in this area.
Yann LeCun (19:11.440)
The main issue with this is knowledge acquisition.
Lex Fridman (19:13.880)
How do you reduce a bunch of data to a graph of this type?
Yann LeCun (19:18.880)
Yeah, it relies on the expert, on the human being,
Lex Fridman (19:22.720)
to encode, to add knowledge.
Lex Fridman (19:24.960)
And that's essentially impractical.
Lex Fridman (19:27.120)
Yeah, it's not scalable.
Yann LeCun (19:29.480)
That's a big question.
Lex Fridman (19:30.320)
The second question is,
Lex Fridman (19:31.440)
do you want to represent knowledge as symbols
Lex Fridman (19:34.640)
and do you want to manipulate them with logic?
Lex Fridman (19:37.240)
And again, that's incompatible with learning.
Lex Fridman (19:39.320)
So one suggestion, which Jeff Hinton
Yann LeCun (19:43.160)
has been advocating for many decades,
Lex Fridman (19:45.080)
is replace symbols by vectors.
Yann LeCun (19:49.360)
Think of it as pattern of activities
Lex Fridman (19:50.960)
in a bunch of neurons or units
Yann LeCun (19:53.320)
or whatever you want to call them.
Lex Fridman (19:55.120)
And replace logic by continuous functions.
Yann LeCun (19:59.560)
Okay, and that becomes now compatible.
Lex Fridman (1:00:01.280)
but most of those are incredibly boring.
Lex Fridman (1:00:03.280)
What I like is select, you know, 10% of them
Lex Fridman (1:00:05.840)
that are kind of the most informative.
Lex Fridman (1:00:07.400)
And with just that, I would probably reach the same.
Lex Fridman (1:00:10.400)
So it's a weak form of active learning if you want.
Yann LeCun (1:00:14.280)
Yes, but there might be a much stronger version.
Lex Fridman (1:00:18.040)
Yeah, that's right.
Yann LeCun (1:00:18.880)
That's what, and that's an awful question if it exists.
Lex Fridman (1:00:21.600)
The question is how much stronger can you get?
Yann LeCun (1:00:24.360)
Elon Musk is confident.
Lex Fridman (1:00:26.520)
Talked to him recently.
Yann LeCun (1:00:28.120)
He's confident that large scale data and deep learning
Lex Fridman (1:00:30.760)
can solve the autonomous driving problem.
Lex Fridman (1:00:33.560)
What are your thoughts on the limits,
Lex Fridman (1:00:36.280)
possibilities of deep learning in this space?
Yann LeCun (1:00:38.520)
It's obviously part of the solution.
Lex Fridman (1:00:40.880)
I mean, I don't think we'll ever have a set driving system
Yann LeCun (1:00:43.800)
or at least not in the foreseeable future
Lex Fridman (1:00:45.600)
that does not use deep learning.
Yann LeCun (1:00:47.240)
Let me put it this way.
Lex Fridman (1:00:48.360)
Now, how much of it?
Lex Fridman (1:00:49.600)
So in the history of sort of engineering,
Lex Fridman (1:00:54.040)
particularly sort of AI like systems,
Yann LeCun (1:00:58.320)
there's generally a first phase where everything is built by hand.
Lex Fridman (1:01:01.000)
Then there is a second phase.
Lex Fridman (1:01:02.120)
And that was the case for autonomous driving 20, 30 years ago.
Lex Fridman (1:01:06.400)
There's a phase where there's a little bit of learning is used,
Lex Fridman (1:01:09.160)
but there's a lot of engineering that's involved in kind of
Lex Fridman (1:01:12.800)
taking care of corner cases and putting limits, et cetera,
Yann LeCun (1:01:16.480)
because the learning system is not perfect.
Lex Fridman (1:01:18.200)
And then as technology progresses,
Yann LeCun (1:01:21.960)
we end up relying more and more on learning.
Lex Fridman (1:01:23.920)
That's the history of character recognition,
Yann LeCun (1:01:25.800)
it's the history of science.
Lex Fridman (1:01:27.120)
Character recognition is the history of speech recognition,
Yann LeCun (1:01:29.120)
now computer vision, natural language processing.
Lex Fridman (1:01:31.600)
And I think the same is going to happen with autonomous driving
Yann LeCun (1:01:36.160)
that currently the methods that are closest
Lex Fridman (1:01:40.720)
to providing some level of autonomy,
Yann LeCun (1:01:43.120)
some decent level of autonomy
Lex Fridman (1:01:44.960)
where you don't expect a driver to kind of do anything
Yann LeCun (1:01:48.560)
is where you constrain the world.
Lex Fridman (1:01:50.880)
So you only run within 100 square kilometers
Yann LeCun (1:01:53.760)
or square miles in Phoenix where the weather is nice
Lex Fridman (1:01:56.200)
and the roads are wide, which is what Waymo is doing.
Yann LeCun (1:02:00.240)
You completely overengineer the car with tons of LIDARs
Lex Fridman (1:02:04.480)
and sophisticated sensors that are too expensive
Yann LeCun (1:02:08.440)
for consumer cars,
Lex Fridman (1:02:09.280)
but they're fine if you just run a fleet.
Lex Fridman (1:02:13.040)
And you engineer the hell out of everything else.
Lex Fridman (1:02:16.400)
You map the entire world.
Lex Fridman (1:02:17.960)
So you have complete 3D model of everything.
Lex Fridman (1:02:20.360)
So the only thing that the perception system
Yann LeCun (1:02:22.160)
has to take care of is moving objects
Lex Fridman (1:02:24.160)
and construction and sort of things that weren't in your map.
Lex Fridman (1:02:30.880)
And you can engineer a good SLAM system and all that stuff.
Lex Fridman (1:02:34.160)
So that's kind of the current approach
Yann LeCun (1:02:35.840)
that's closest to some level of autonomy.
Lex Fridman (1:02:37.480)
But I think eventually the longterm solution
Yann LeCun (1:02:39.640)
is going to rely more and more on learning
Lex Fridman (1:02:43.400)
and possibly using a combination
Yann LeCun (1:02:45.000)
of self supervised learning and model based reinforcement
Lex Fridman (1:02:49.320)
or something like that.
Lex Fridman (1:02:50.840)
But ultimately learning will be not just at the core,
Lex Fridman (1:02:54.760)
but really the fundamental part of the system.
Yann LeCun (1:02:57.160)
Yeah, it already is, but it will become more and more.
Lex Fridman (1:03:00.360)
What do you think it takes to build a system
Lex Fridman (1:03:02.720)
with human level intelligence?
Lex Fridman (1:03:04.080)
You talked about the AI system in the movie Her
Yann LeCun (1:03:07.600)
being way out of reach, our current reach.
Lex Fridman (1:03:10.040)
This might be outdated as well, but.
Yann LeCun (1:03:12.360)
It's still way out of reach.
Lex Fridman (1:03:13.240)
It's still way out of reach.
Lex Fridman (1:03:15.800)
What would it take to build Her?
Lex Fridman (1:03:18.360)
Do you think?
Lex Fridman (1:03:19.720)
So I can tell you the first two obstacles
Lex Fridman (1:03:21.760)
that we have to clear,
Lex Fridman (1:03:22.880)
but I don't know how many obstacles there are after this.
Lex Fridman (1:03:24.880)
So the image I usually use is that
Yann LeCun (1:03:26.640)
there is a bunch of mountains that we have to climb
Lex Fridman (1:03:28.680)
and we can see the first one,
Lex Fridman (1:03:29.720)
but we don't know if there are 50 mountains behind it or not.
Lex Fridman (1:03:33.080)
And this might be a good sort of metaphor
Yann LeCun (1:03:34.960)
for why AI researchers in the past
Lex Fridman (1:03:38.400)
have been overly optimistic about the result of AI.
Yann LeCun (1:03:43.520)
You know, for example,
Lex Fridman (1:03:45.800)
Noel and Simon wrote the general problem solver
Lex Fridman (1:03:49.440)
and they called it the general problem solver.
Lex Fridman (1:03:51.440)
General problem solver.
Lex Fridman (1:03:52.960)
And of course, the first thing you realize
Lex Fridman (1:03:54.520)
is that all the problems you want to solve are exponential.
Lex Fridman (1:03:56.360)
And so you can't actually use it for anything useful,
Lex Fridman (1:03:59.160)
but you know.
Yann LeCun (1:04:00.080)
Yeah, so yeah, all you see is the first peak.
Lex Fridman (1:04:02.280)
So in general, what are the first couple of peaks for Her?
Lex Fridman (1:04:05.280)
So the first peak, which is precisely what I'm working on
Lex Fridman (1:04:08.000)
is self supervised learning.
Lex Fridman (1:04:10.280)
How do we get machines to run models of the world
Lex Fridman (1:04:12.280)
by observation, kind of like babies and like young animals?
Lex Fridman (1:04:15.880)
So we've been working with, you know, cognitive scientists.
Lex Fridman (1:04:21.760)
So this Emmanuelle Dupoux, who's at FAIR in Paris,
Yann LeCun (1:04:24.760)
is a half time, is also a researcher in a French university.
Lex Fridman (1:04:30.640)
And he has this chart that shows that which,
Lex Fridman (1:04:36.120)
how many months of life baby humans
Lex Fridman (1:04:38.640)
kind of learn different concepts.
Lex Fridman (1:04:40.720)
And you can measure this in sort of various ways.
Lex Fridman (1:04:44.040)
So things like distinguishing animate objects
Yann LeCun (1:04:49.040)
from inanimate objects,
Lex Fridman (1:04:50.360)
you can tell the difference at age two, three months.
Yann LeCun (1:04:54.720)
Whether an object is going to stay stable,
Lex Fridman (1:04:56.360)
is going to fall, you know,
Yann LeCun (1:04:58.080)
about four months, you can tell.
Lex Fridman (1:05:00.760)
You know, there are various things like this.
Lex Fridman (1:05:02.400)
And then things like gravity,
Lex Fridman (1:05:04.240)
the fact that objects are not supposed to float in the air,
Lex Fridman (1:05:06.520)
but are supposed to fall,
Lex Fridman (1:05:07.880)
you run this around the age of eight or nine months.
Yann LeCun (1:05:10.360)
If you look at the data,
Lex Fridman (1:05:11.960)
eight or nine months, if you look at a lot of,
Yann LeCun (1:05:14.600)
you know, eight month old babies,
Lex Fridman (1:05:15.880)
you give them a bunch of toys on their high chair.
Yann LeCun (1:05:19.040)
First thing they do is they throw them on the ground
Lex Fridman (1:05:20.560)
and they look at them.
Yann LeCun (1:05:21.720)
It's because, you know, they're learning about,
Lex Fridman (1:05:23.920)
actively learning about gravity.
Yann LeCun (1:05:26.120)
Gravity, yeah.
Lex Fridman (1:05:26.960)
Okay, so they're not trying to annoy you,
Lex Fridman (1:05:29.680)
but they, you know, they need to do the experiment, right?
Lex Fridman (1:05:32.480)
Yeah.
Yann LeCun (1:05:33.600)
So, you know, how do we get machines to learn like babies,
Lex Fridman (1:05:36.600)
mostly by observation with a little bit of interaction
Lex Fridman (1:05:39.240)
and learning those models of the world?
Lex Fridman (1:05:41.200)
Because I think that's really a crucial piece
Yann LeCun (1:05:43.720)
of an intelligent autonomous system.
Lex Fridman (1:05:46.360)
So if you think about the architecture
Yann LeCun (1:05:47.520)
of an intelligent autonomous system,
Lex Fridman (1:05:49.520)
it needs to have a predictive model of the world.
Lex Fridman (1:05:51.320)
So something that says, here is a world at time T,
Lex Fridman (1:05:54.080)
here is a state of the world at time T plus one,
Yann LeCun (1:05:55.520)
if I take this action.
Lex Fridman (1:05:57.560)
And it's not a single answer, it can be a...
Yann LeCun (1:05:59.680)
Yeah, it can be a distribution, yeah.
Lex Fridman (1:06:01.240)
Yeah, well, but we don't know how to represent
Yann LeCun (1:06:03.200)
distributions in high dimensional T spaces.
Lex Fridman (1:06:04.840)
So it's gotta be something weaker than that, okay?
Lex Fridman (1:06:07.200)
But with some representation of uncertainty.
Lex Fridman (1:06:09.760)
If you have that, then you can do what optimal control
Yann LeCun (1:06:12.440)
theorists call model predictive control,
Lex Fridman (1:06:14.360)
which means that you can run your model
Yann LeCun (1:06:16.360)
with a hypothesis for a sequence of action
Lex Fridman (1:06:18.800)
and then see the result.
Yann LeCun (1:06:20.840)
Now, what you need, the other thing you need
Lex Fridman (1:06:22.160)
is some sort of objective that you want to optimize.
Lex Fridman (1:06:24.920)
Am I reaching the goal of grabbing this object?
Lex Fridman (1:06:27.560)
Am I minimizing energy?
Lex Fridman (1:06:28.880)
Am I whatever, right?
Lex Fridman (1:06:30.040)
So there is some sort of objective that you have to minimize.
Lex Fridman (1:06:33.720)
And so in your head, if you have this model,
Lex Fridman (1:06:35.640)
you can figure out the sequence of action
Yann LeCun (1:06:37.080)
that will optimize your objective.
Lex Fridman (1:06:38.920)
That objective is something that ultimately is rooted
Yann LeCun (1:06:42.400)
in your basal ganglia, at least in the human brain,
Lex Fridman (1:06:44.960)
that's what it's basal ganglia,
Yann LeCun (1:06:47.040)
computes your level of contentment or miscontentment.
Lex Fridman (1:06:50.600)
I don't know if that's a word.
Lex Fridman (1:06:52.360)
Unhappiness, okay?
Lex Fridman (1:06:53.680)
Yeah, yeah.
Yann LeCun (1:06:54.800)
Discontentment.
Lex Fridman (1:06:55.640)
Discontentment, maybe.
Lex Fridman (1:06:56.680)
And so your entire behavior is driven towards
Lex Fridman (1:07:01.720)
kind of minimizing that objective,
Yann LeCun (1:07:03.320)
which is maximizing your contentment,
Lex Fridman (1:07:05.720)
computed by your basal ganglia.
Lex Fridman (1:07:07.600)
And what you have is an objective function,
Lex Fridman (1:07:10.600)
which is basically a predictor
Yann LeCun (1:07:12.320)
of what your basal ganglia is going to tell you.
Lex Fridman (1:07:14.520)
So you're not going to put your hand on fire
Yann LeCun (1:07:16.600)
because you know it's going to burn
Lex Fridman (1:07:19.760)
and you're going to get hurt.
Lex Fridman (1:07:21.240)
And you're predicting this because of your model
Lex Fridman (1:07:23.160)
of the world and your sort of predictor
Lex Fridman (1:07:25.720)
of this objective, right?
Lex Fridman (1:07:27.560)
So if you have those three components,
Yann LeCun (1:07:31.160)
you have four components,
Lex Fridman (1:07:32.600)
you have the hardwired objective,
Yann LeCun (1:07:36.080)
hardwired contentment objective computer,
Lex Fridman (1:07:41.760)
if you want, calculator.
Lex Fridman (1:07:43.960)
And then you have the three components.
Lex Fridman (1:07:45.160)
One is the objective predictor,
Yann LeCun (1:07:46.760)
which basically predicts your level of contentment.
Lex Fridman (1:07:48.960)
One is the model of the world.
Lex Fridman (1:07:52.560)
And there's a third module I didn't mention,
Lex Fridman (1:07:54.120)
which is the module that will figure out
Yann LeCun (1:07:57.280)
the best course of action to optimize an objective
Lex Fridman (1:08:00.560)
given your model, okay?
Yann LeCun (1:08:03.480)
Yeah.
Lex Fridman (1:08:04.520)
And you can call this a policy network
Lex Fridman (1:08:07.240)
or something like that, right?
Lex Fridman (1:08:09.400)
Now, you need those three components
Yann LeCun (1:08:11.720)
to act autonomously intelligently.
Lex Fridman (1:08:13.960)
And you can be stupid in three different ways.
Yann LeCun (1:08:16.120)
You can be stupid because your model of the world is wrong.
Lex Fridman (1:08:19.400)
You can be stupid because your objective is not aligned
Lex Fridman (1:08:22.520)
with what you actually want to achieve, okay?
Lex Fridman (1:08:27.000)
In humans, that would be a psychopath.
Lex Fridman (1:08:30.000)
And then the third way you can be stupid
Lex Fridman (1:08:33.640)
is that you have the right model,
Yann LeCun (1:08:34.960)
you have the right objective,
Lex Fridman (1:08:36.360)
but you're unable to figure out a course of action
Yann LeCun (1:08:38.840)
to optimize your objective given your model.
Lex Fridman (1:08:41.240)
Okay.
Yann LeCun (1:08:44.080)
Some people who are in charge of big countries
Lex Fridman (1:08:45.920)
actually have all three that are wrong.
Yann LeCun (1:08:47.760)
All right.
Lex Fridman (1:08:50.920)
Which countries?
Yann LeCun (1:08:51.760)
I don't know.
Lex Fridman (1:08:52.600)
Okay, so if we think about this agent,
Yann LeCun (1:08:55.960)
if we think about the movie Her,
Lex Fridman (1:08:58.000)
you've criticized the art project
Yann LeCun (1:09:02.920)
that is Sophia the Robot.
Lex Fridman (1:09:04.680)
And what that project essentially does
Yann LeCun (1:09:07.560)
is uses our natural inclination to anthropomorphize
Lex Fridman (1:09:11.720)
things that look like human and give them more.
Lex Fridman (1:09:14.800)
Do you think that could be used by AI systems
Lex Fridman (1:09:17.720)
like in the movie Her?
Lex Fridman (1:09:21.320)
So do you think that body is needed
Lex Fridman (1:09:23.400)
to create a feeling of intelligence?
Yann LeCun (1:09:27.200)
Well, if Sophia was just an art piece,
Lex Fridman (1:09:29.320)
I would have no problem with it,
Lex Fridman (1:09:30.360)
but it's presented as something else.
Lex Fridman (1:09:33.040)
Let me, on that comment real quick,
Yann LeCun (1:09:35.280)
if creators of Sophia could change something
Lex Fridman (1:09:38.520)
about their marketing or behavior in general,
Lex Fridman (1:09:40.760)
what would it be?
Lex Fridman (1:09:41.600)
What's?
Yann LeCun (1:09:42.840)
I'm just about everything.
Lex Fridman (1:09:44.160)
I mean, don't you think, here's a tough question.
Yann LeCun (1:09:50.080)
Let me, so I agree with you.
Lex Fridman (1:09:51.680)
So Sophia is not, the general public feels
Yann LeCun (1:09:56.560)
that Sophia can do way more than she actually can.
Lex Fridman (1:09:59.320)
That's right.
Lex Fridman (1:10:00.200)
And the people who created Sophia
Lex Fridman (1:10:02.760)
are not honestly publicly communicating,
Yann LeCun (1:10:08.360)
trying to teach the public.
Lex Fridman (1:10:09.440)
Right.
Lex Fridman (1:10:10.280)
But here's a tough question.
Lex Fridman (1:10:13.280)
Don't you think the same thing is scientists
Yann LeCun (1:10:19.800)
in industry and research are taking advantage
Lex Fridman (1:10:22.920)
of the same misunderstanding in the public
Lex Fridman (1:10:25.640)
when they create AI companies or publish stuff?
Lex Fridman (1:10:29.920)
Some companies, yes.
Yann LeCun (1:10:31.120)
I mean, there is no sense of,
Lex Fridman (1:10:33.160)
there's no desire to delude.
Yann LeCun (1:10:34.880)
There's no desire to kind of over claim
Lex Fridman (1:10:37.840)
when something is done, right?
Yann LeCun (1:10:38.840)
You publish a paper on AI that has this result
Lex Fridman (1:10:41.400)
on ImageNet, it's pretty clear.
Yann LeCun (1:10:43.080)
I mean, it's not even interesting anymore,
Lex Fridman (1:10:44.960)
but I don't think there is that.
Yann LeCun (1:10:49.240)
I mean, the reviewers are generally not very forgiving
Lex Fridman (1:10:52.880)
of unsupported claims of this type.
Yann LeCun (1:10:57.200)
And, but there are certainly quite a few startups
Lex Fridman (1:10:59.680)
that have had a huge amount of hype around this
Yann LeCun (1:11:02.680)
that I find extremely damaging
Lex Fridman (1:11:05.520)
and I've been calling it out when I've seen it.
Lex Fridman (1:11:08.080)
So yeah, but to go back to your original question,
Lex Fridman (1:11:10.280)
like the necessity of embodiment,
Yann LeCun (1:11:13.080)
I think, I don't think embodiment is necessary.
Lex Fridman (1:11:15.640)
I think grounding is necessary.
Lex Fridman (1:11:17.120)
So I don't think we're gonna get machines
Lex Fridman (1:11:18.960)
that really understand language
Yann LeCun (1:11:20.520)
without some level of grounding in the real world.
Lex Fridman (1:11:22.440)
And it's not clear to me that language
Yann LeCun (1:11:24.360)
is a high enough bandwidth medium
Lex Fridman (1:11:26.160)
to communicate how the real world works.
Lex Fridman (1:11:28.280)
So I think for this.
Lex Fridman (1:11:30.120)
Can you talk to what grounding means?
Lex Fridman (1:11:32.320)
So grounding means that,
Lex Fridman (1:11:34.040)
so there is this classic problem of common sense reasoning,
Lex Fridman (1:11:37.720)
you know, the Winograd schema, right?
Lex Fridman (1:11:41.000)
And so I tell you the trophy doesn't fit in the suitcase
Yann LeCun (1:11:44.960)
because it's too big,
Lex Fridman (1:11:46.360)
or the trophy doesn't fit in the suitcase
Yann LeCun (1:11:47.760)
because it's too small.
Lex Fridman (1:11:49.160)
And the it in the first case refers to the trophy
Yann LeCun (1:11:51.800)
in the second case to the suitcase.
Lex Fridman (1:11:53.640)
And the reason you can figure this out
Yann LeCun (1:11:55.160)
is because you know where the trophy and the suitcase are,
Lex Fridman (1:11:56.960)
you know, one is supposed to fit in the other one
Lex Fridman (1:11:58.640)
and you know the notion of size
Lex Fridman (1:12:00.560)
and a big object doesn't fit in a small object,
Lex Fridman (1:12:03.000)
unless it's a Tardis, you know, things like that, right?
Lex Fridman (1:12:05.280)
So you have this knowledge of how the world works,
Yann LeCun (1:12:08.640)
of geometry and things like that.
Lex Fridman (1:12:12.440)
I don't believe you can learn everything about the world
Yann LeCun (1:12:14.640)
by just being told in language how the world works.
Lex Fridman (1:12:18.000)
I think you need some low level perception of the world,
Yann LeCun (1:12:21.680)
you know, be it visual touch, you know, whatever,
Lex Fridman (1:12:23.680)
but some higher bandwidth perception of the world.
Yann LeCun (1:12:26.760)
By reading all the world's text,
Lex Fridman (1:12:28.800)
you still might not have enough information.
Yann LeCun (1:12:31.160)
That's right.
Lex Fridman (1:12:32.520)
There's a lot of things that just will never appear in text
Lex Fridman (1:12:35.440)
and that you can't really infer.
Lex Fridman (1:12:37.000)
So I think common sense will emerge from,
Yann LeCun (1:12:41.440)
you know, certainly a lot of language interaction,
Lex Fridman (1:12:43.440)
but also with watching videos
Yann LeCun (1:12:45.640)
or perhaps even interacting in virtual environments
Lex Fridman (1:12:48.920)
and possibly, you know, robot interacting in the real world.
Lex Fridman (1:12:51.760)
But I don't actually believe necessarily
Lex Fridman (1:12:53.640)
that this last one is absolutely necessary.
Lex Fridman (1:12:56.000)
But I think that there's a need for some grounding.
Lex Fridman (1:13:00.240)
But the final product
Yann LeCun (1:13:01.880)
doesn't necessarily need to be embodied, you're saying.
Lex Fridman (1:13:04.840)
No.
Yann LeCun (1:13:05.680)
It just needs to have an awareness, a grounding to.
Lex Fridman (1:13:07.720)
Right, but it needs to know how the world works
Yann LeCun (1:13:11.120)
to have, you know, to not be frustrating to talk to.
Lex Fridman (1:13:15.840)
And you talked about emotions being important.
Yann LeCun (1:13:19.520)
That's a whole nother topic.
Lex Fridman (1:13:21.760)
Well, so, you know, I talked about this,
Yann LeCun (1:13:24.320)
the basal ganglia as the thing
Lex Fridman (1:13:29.600)
that calculates your level of miscontentment.
Lex Fridman (1:13:32.920)
And then there is this other module
Lex Fridman (1:13:34.640)
that sort of tries to do a prediction
Yann LeCun (1:13:36.640)
of whether you're going to be content or not.
Lex Fridman (1:13:38.520)
That's the source of some emotion.
Lex Fridman (1:13:40.240)
So fear, for example, is an anticipation
Lex Fridman (1:13:43.040)
of bad things that can happen to you, right?
Yann LeCun (1:13:47.440)
You have this inkling that there is some chance
Lex Fridman (1:13:49.240)
that something really bad is going to happen to you
Lex Fridman (1:13:50.880)
and that creates fear.
Lex Fridman (1:13:52.280)
Well, you know for sure
Yann LeCun (1:13:53.120)
that something bad is going to happen to you,
Lex Fridman (1:13:54.480)
you kind of give up, right?
Yann LeCun (1:13:55.960)
It's not fear anymore.
Lex Fridman (1:13:57.560)
It's uncertainty that creates fear.
Lex Fridman (1:13:59.480)
So the punchline is,
Lex Fridman (1:14:01.200)
we're not going to have autonomous intelligence
Yann LeCun (1:14:02.560)
without emotions.
Lex Fridman (1:14:07.040)
Whatever the heck emotions are.
Lex Fridman (1:14:08.880)
So you mentioned very practical things of fear,
Lex Fridman (1:14:11.080)
but there's a lot of other mess around it.
Lex Fridman (1:14:13.480)
But there are kind of the results of, you know, drives.
Lex Fridman (1:14:16.400)
Yeah, there's deeper biological stuff going on.
Lex Fridman (1:14:19.360)
And I've talked to a few folks on this.
Lex Fridman (1:14:21.440)
There's fascinating stuff
Yann LeCun (1:14:23.360)
that ultimately connects to our brain.
Lex Fridman (1:14:27.320)
If we create an AGI system, sorry.
Yann LeCun (1:14:30.880)
Human level intelligence.
Lex Fridman (1:14:31.720)
Human level intelligence system.
Lex Fridman (1:14:34.480)
And you get to ask her one question.
Lex Fridman (1:14:37.160)
What would that question be?
Yann LeCun (1:14:39.960)
You know, I think the first one we'll create
Lex Fridman (1:14:42.880)
would probably not be that smart.
Yann LeCun (1:14:45.520)
They'd be like a four year old.
Lex Fridman (1:14:47.040)
Okay.
Lex Fridman (1:14:47.880)
So you would have to ask her a question
Lex Fridman (1:14:50.040)
to know she's not that smart.
Yann LeCun (1:14:52.840)
Yeah.
Lex Fridman (1:14:54.520)
Well, what's a good question to ask, you know,
Yann LeCun (1:14:56.960)
to be impressed.
Lex Fridman (1:14:57.800)
What is the cause of wind?
Lex Fridman (1:15:01.040)
And if she answers,
Lex Fridman (1:15:02.240)
oh, it's because the leaves of the tree are moving
Lex Fridman (1:15:04.760)
and that creates wind.
Lex Fridman (1:15:06.520)
She's onto something.
Lex Fridman (1:15:08.760)
And if she says that's a stupid question,
Lex Fridman (1:15:11.840)
she's really onto something.
Yann LeCun (1:15:12.680)
No, and then you tell her,
Lex Fridman (1:15:14.440)
actually, you know, here is the real thing.
Yann LeCun (1:15:18.080)
She says, oh yeah, that makes sense.
Lex Fridman (1:15:20.520)
So questions that reveal the ability
Yann LeCun (1:15:24.480)
to do common sense reasoning about the physical world.
Lex Fridman (1:15:26.960)
Yeah.
Lex Fridman (1:15:27.800)
And you'll sum it up with causal inference.
Lex Fridman (1:15:30.120)
Causal inference.
Yann LeCun (1:15:31.200)
Well, it was a huge honor.
Lex Fridman (1:15:33.640)
Congratulations on your Turing Award.
Yann LeCun (1:15:35.720)
Thank you so much for talking today.
Lex Fridman (1:15:37.240)
Thank you.
Yann LeCun (1:15:38.080)
Thank you for having me.
Lex Fridman (20:01.840)
There's a very good set of ideas
Yann LeCun (20:04.960)
by, written in a paper about 10 years ago
Lex Fridman (20:07.640)
by Leon Boutout, who is here at Facebook.
Yann LeCun (20:13.160)
The title of the paper is,
Lex Fridman (20:14.400)
From Machine Learning to Machine Reasoning.
Lex Fridman (20:15.840)
And his idea is that a learning system
Lex Fridman (20:19.480)
should be able to manipulate objects
Yann LeCun (20:20.880)
that are in a space
Lex Fridman (20:23.160)
and then put the result back in the same space.
Lex Fridman (20:24.920)
So it's this idea of working memory, basically.
Lex Fridman (20:28.400)
And it's very enlightening.
Lex Fridman (20:30.640)
And in a sense, that might learn something
Lex Fridman (20:33.720)
like the simple expert systems.
Yann LeCun (20:37.920)
I mean, you can learn basic logic operations there.
Lex Fridman (20:42.080)
Yeah, quite possibly.
Yann LeCun (20:43.400)
There's a big debate on sort of how much prior structure
Lex Fridman (20:46.680)
you have to put in for this kind of stuff to emerge.
Yann LeCun (20:49.080)
That's the debate I have with Gary Marcus
Lex Fridman (20:50.720)
and people like that.
Yann LeCun (20:51.560)
Yeah, yeah, so, and the other person,
Lex Fridman (20:55.040)
so I just talked to Judea Pearl,
Yann LeCun (20:57.520)
from the you mentioned causal inference world.
Lex Fridman (21:00.240)
So his worry is that the current neural networks
Yann LeCun (21:04.160)
are not able to learn what causes
Lex Fridman (21:09.600)
what causal inference between things.
Lex Fridman (21:12.760)
So I think he's right and wrong about this.
Lex Fridman (21:15.640)
If he's talking about the sort of classic
Yann LeCun (21:20.280)
type of neural nets,
Lex Fridman (21:21.320)
people sort of didn't worry too much about this.
Lex Fridman (21:23.800)
But there's a lot of people now working on causal inference.
Lex Fridman (21:26.160)
And there's a paper that just came out last week
Yann LeCun (21:27.840)
by Leon Boutou, among others,
Lex Fridman (21:29.160)
David Lopez, Baz, and a bunch of other people,
Yann LeCun (21:32.000)
exactly on that problem of how do you kind of
Lex Fridman (21:36.880)
get a neural net to sort of pay attention
Yann LeCun (21:39.400)
to real causal relationships,
Lex Fridman (21:41.600)
which may also solve issues of bias in data
Lex Fridman (21:46.600)
and things like this, so.
Lex Fridman (21:48.040)
I'd like to read that paper
Yann LeCun (21:49.200)
because that ultimately the challenges
Lex Fridman (21:51.960)
also seems to fall back on the human expert
Yann LeCun (21:56.920)
to ultimately decide causality between things.
Lex Fridman (22:01.880)
People are not very good
Yann LeCun (22:02.720)
at establishing causality, first of all.
Lex Fridman (22:04.800)
So first of all, you talk to physicists
Lex Fridman (22:06.560)
and physicists actually don't believe in causality
Lex Fridman (22:08.600)
because look at all the basic laws of microphysics
Yann LeCun (22:12.960)
are time reversible, so there's no causality.
Lex Fridman (22:15.480)
The arrow of time is not real, yeah.
Yann LeCun (22:17.120)
It's as soon as you start looking at macroscopic systems
Lex Fridman (22:20.440)
where there is unpredictable randomness,
Yann LeCun (22:22.800)
where there is clearly an arrow of time,
Lex Fridman (22:25.440)
but it's a big mystery in physics, actually,
Lex Fridman (22:27.320)
how that emerges.
Lex Fridman (22:29.160)
Is it emergent or is it part of
Lex Fridman (22:31.720)
the fundamental fabric of reality?
Lex Fridman (22:34.320)
Or is it a bias of intelligent systems
Yann LeCun (22:36.880)
that because of the second law of thermodynamics,
Lex Fridman (22:39.280)
we perceive a particular arrow of time,
Lex Fridman (22:41.440)
but in fact, it's kind of arbitrary, right?
Lex Fridman (22:45.120)
So yeah, physicists, mathematicians,
Yann LeCun (22:47.120)
they don't care about, I mean,
Lex Fridman (22:48.440)
the math doesn't care about the flow of time.
Yann LeCun (22:51.520)
Well, certainly, macrophysics doesn't.
Lex Fridman (22:54.080)
People themselves are not very good
Yann LeCun (22:55.440)
at establishing causal relationships.
Lex Fridman (22:58.920)
If you ask, I think it was in one of Seymour Papert's book
Yann LeCun (23:02.760)
on children learning.
Lex Fridman (23:06.800)
He studied with Jean Piaget.
Yann LeCun (23:08.840)
He's the guy who coauthored the book Perceptron
Lex Fridman (23:11.520)
with Marvin Minsky that kind of killed
Yann LeCun (23:12.960)
the first wave of neural nets,
Lex Fridman (23:14.080)
but he was actually a learning person.
Yann LeCun (23:17.200)
He, in the sense of studying learning in humans
Lex Fridman (23:21.040)
and machines, that's why he got interested in Perceptron.
Lex Fridman (23:24.160)
And he wrote that if you ask a little kid
Lex Fridman (23:29.280)
about what is the cause of the wind,
Yann LeCun (23:33.720)
a lot of kids will say, they will think for a while
Lex Fridman (23:35.840)
and they'll say, oh, it's the branches in the trees,
Lex Fridman (23:38.120)
they move and that creates wind, right?
Lex Fridman (23:40.120)
So they get the causal relationship backwards.
Lex Fridman (23:42.600)
And it's because their understanding of the world
Lex Fridman (23:44.520)
and intuitive physics is not that great, right?
Yann LeCun (23:46.280)
I mean, these are like, you know, four or five year old kids.
Lex Fridman (23:49.880)
You know, it gets better,
Lex Fridman (23:50.720)
and then you understand that this, it can be, right?
Lex Fridman (23:54.080)
But there are many things which we can,
Yann LeCun (23:57.440)
because of our common sense understanding of things,
Lex Fridman (24:00.920)
what people call common sense,
Lex Fridman (24:03.280)
and our understanding of physics,
Lex Fridman (24:05.000)
we can, there's a lot of stuff
Yann LeCun (24:07.640)
that we can figure out causality.
Lex Fridman (24:08.840)
Even with diseases, we can figure out
Yann LeCun (24:10.480)
what's not causing what, often.
Lex Fridman (24:14.520)
There's a lot of mystery, of course,
Lex Fridman (24:16.040)
but the idea is that you should be able
Lex Fridman (24:18.120)
to encode that into systems,
Yann LeCun (24:20.160)
because it seems unlikely they'd be able
Lex Fridman (24:21.400)
to figure that out themselves.
Yann LeCun (24:22.800)
Well, whenever we can do intervention,
Lex Fridman (24:24.480)
but you know, all of humanity has been completely deluded
Yann LeCun (24:27.400)
for millennia, probably since its existence,
Lex Fridman (24:30.400)
about a very, very wrong causal relationship,
Yann LeCun (24:33.420)
where whatever you can explain, you attribute it to,
Lex Fridman (24:35.720)
you know, some deity, some divinity, right?
Lex Fridman (24:39.240)
And that's a cop out, that's a way of saying like,
Lex Fridman (24:41.000)
I don't know the cause, so you know, God did it, right?
Lex Fridman (24:43.920)
So you mentioned Marvin Minsky,
Lex Fridman (24:46.240)
and the irony of, you know,
Yann LeCun (24:51.520)
maybe causing the first AI winter.
Lex Fridman (24:54.580)
You were there in the 90s, you were there in the 80s,
Yann LeCun (24:56.920)
of course.
Lex Fridman (24:58.120)
In the 90s, why do you think people lost faith
Yann LeCun (25:00.640)
in deep learning, in the 90s, and found it again,
Lex Fridman (25:04.000)
a decade later, over a decade later?
Yann LeCun (25:06.360)
Yeah, it wasn't called deep learning yet,
Lex Fridman (25:07.760)
it was just called neural nets, but yeah,
Yann LeCun (25:11.880)
they lost interest.
Lex Fridman (25:13.840)
I mean, I think I would put that around 1995,
Yann LeCun (25:16.840)
at least the machine learning community,
Lex Fridman (25:18.080)
there was always a neural net community,
Lex Fridman (25:19.660)
but it became kind of disconnected
Lex Fridman (25:23.760)
from sort of mainstream machine learning, if you want.
Yann LeCun (25:26.560)
There were, it was basically electrical engineering
Lex Fridman (25:30.960)
that kept at it, and computer science gave up on neural nets.
Yann LeCun (25:38.000)
I don't know, you know, I was too close to it
Lex Fridman (25:40.520)
to really sort of analyze it with sort of an unbiased eye,
Yann LeCun (25:46.960)
if you want, but I would make a few guesses.
Lex Fridman (25:50.760)
So the first one is, at the time, neural nets were,
Yann LeCun (25:55.760)
it was very hard to make them work,
Lex Fridman (25:57.880)
in the sense that you would implement backprop
Yann LeCun (26:02.400)
in your favorite language, and that favorite language
Lex Fridman (26:06.120)
was not Python, it was not MATLAB,
Yann LeCun (26:08.240)
it was not any of those things,
Lex Fridman (26:09.320)
because they didn't exist, right?
Yann LeCun (26:10.760)
You had to write it in Fortran OC,
Lex Fridman (26:13.320)
or something like this, right?
Lex Fridman (26:16.320)
So you would experiment with it,
Lex Fridman (26:18.680)
you would probably make some very basic mistakes,
Yann LeCun (26:21.200)
like, you know, badly initialize your weights,
Lex Fridman (26:23.240)
make the network too small,
Yann LeCun (26:24.200)
because you read in the textbook, you know,
Lex Fridman (26:25.520)
you don't want too many parameters, right?
Lex Fridman (26:27.640)
And of course, you know, and you would train on XOR,
Lex Fridman (26:29.280)
because you didn't have any other data set to trade on.
Lex Fridman (26:32.000)
And of course, you know, it works half the time.
Lex Fridman (26:33.760)
So you would say, I give up.
Yann LeCun (26:36.280)
Also, you would train it with batch gradient,
Lex Fridman (26:37.680)
which, you know, isn't that sufficient.
Lex Fridman (26:40.240)
So there's a lot of, there's a bag of tricks
Lex Fridman (26:42.680)
that you had to know to make those things work,
Yann LeCun (26:44.840)
or you had to reinvent, and a lot of people just didn't,
Lex Fridman (26:48.200)
and they just couldn't make it work.
Lex Fridman (26:51.320)
So that's one thing.
Lex Fridman (26:52.400)
The investment in software platform
Yann LeCun (26:54.720)
to be able to kind of, you know, display things,
Lex Fridman (26:58.120)
figure out why things don't work,
Yann LeCun (26:59.360)
kind of get a good intuition for how to get them to work,
Lex Fridman (27:02.120)
have enough flexibility so you can create, you know,
Yann LeCun (27:04.640)
network architectures like convolutional nets
Lex Fridman (27:06.240)
and stuff like that.
Yann LeCun (27:08.320)
It was hard.
Lex Fridman (27:09.160)
I mean, you had to write everything from scratch.
Lex Fridman (27:10.520)
And again, you didn't have any Python
Lex Fridman (27:11.840)
or MATLAB or anything, right?
Yann LeCun (27:14.280)
I read that, sorry to interrupt,
Lex Fridman (27:15.600)
but I read that you wrote in Lisp
Yann LeCun (27:17.680)
the first versions of Lanet with convolutional networks,
Lex Fridman (27:22.680)
which by the way, one of my favorite languages.
Yann LeCun (27:25.320)
That's how I knew you were legit.
Lex Fridman (27:27.560)
Turing award, whatever.
Yann LeCun (27:29.440)
You programmed in Lisp, that's...
Lex Fridman (27:30.760)
It's still my favorite language,
Lex Fridman (27:31.920)
but it's not that we programmed in Lisp,
Lex Fridman (27:34.880)
it's that we had to write our Lisp interpreter, okay?
Yann LeCun (27:38.000)
Because it's not like we used one that existed.
Lex Fridman (27:40.320)
So we wrote a Lisp interpreter that we hooked up to,
Yann LeCun (27:43.880)
you know, a backend library that we wrote also
Lex Fridman (27:46.640)
for sort of neural net computation.
Lex Fridman (27:48.440)
And then after a few years around 1991,
Lex Fridman (27:50.840)
we invented this idea of basically having modules
Yann LeCun (27:54.560)
that know how to forward propagate
Lex Fridman (27:56.160)
and back propagate gradients,
Lex Fridman (27:57.560)
and then interconnecting those modules in a graph.
Lex Fridman (28:01.480)
Number two had made proposals on this,
Yann LeCun (28:03.280)
about this in the late eighties,
Lex Fridman (28:04.720)
and we were able to implement this using our Lisp system.
Yann LeCun (28:08.200)
Eventually we wanted to use that system
Lex Fridman (28:09.800)
to build production code for character recognition
Yann LeCun (28:13.800)
at Bell Labs.
Lex Fridman (28:14.640)
So we actually wrote a compiler for that Lisp interpreter
Lex Fridman (28:16.760)
so that Patricia Simard, who is now at Microsoft,
Lex Fridman (28:19.280)
kind of did the bulk of it with Leon and me.
Lex Fridman (28:22.400)
And so we could write our system in Lisp
Lex Fridman (28:24.920)
and then compile to C,
Lex Fridman (28:26.520)
and then we'll have a self contained complete system
Lex Fridman (28:29.720)
that could kind of do the entire thing.
Yann LeCun (28:33.280)
Neither PyTorch nor TensorFlow can do this today.
Lex Fridman (28:36.080)
Yeah, okay, it's coming.
Yann LeCun (28:37.840)
Yeah.
Lex Fridman (28:40.080)
I mean, there's something like that in PyTorch
Yann LeCun (28:42.000)
called TorchScript.
Lex Fridman (28:44.520)
And so, you know, we had to write our Lisp interpreter,
Yann LeCun (28:46.840)
we had to write our Lisp compiler,
Lex Fridman (28:48.000)
we had to invest a huge amount of effort to do this.
Lex Fridman (28:50.840)
And not everybody,
Lex Fridman (28:52.320)
if you don't completely believe in the concept,
Yann LeCun (28:55.040)
you're not going to invest the time to do this.
Lex Fridman (28:57.040)
Now at the time also, you know,
Yann LeCun (28:59.160)
or today, this would turn into Torch or PyTorch
Lex Fridman (29:02.640)
or TensorFlow or whatever,
Yann LeCun (29:03.840)
we'd put it in open source, everybody would use it
Lex Fridman (29:05.720)
and, you know, realize it's good.
Yann LeCun (29:07.920)
Back before 1995, working at AT&T,
Lex Fridman (29:11.240)
there's no way the lawyers would let you
Yann LeCun (29:13.720)
release anything in open source of this nature.
Lex Fridman (29:17.680)
And so we could not distribute our code really.
Lex Fridman (29:20.600)
And on that point,
Lex Fridman (29:21.920)
and sorry to go on a million tangents,
Lex Fridman (29:23.520)
but on that point, I also read that there was some,
Lex Fridman (29:26.560)
almost like a patent on convolutional neural networks
Yann LeCun (29:30.000)
at Bell Labs.
Lex Fridman (29:32.000)
So that, first of all, I mean, just.
Yann LeCun (29:35.680)
There's two actually.
Lex Fridman (29:38.000)
That ran out.
Yann LeCun (29:39.840)
Thankfully, in 2007.
Lex Fridman (29:41.840)
In 2007.
Lex Fridman (29:42.680)
So I'm gonna, what,
Lex Fridman (29:46.800)
can we just talk about that for a second?
Yann LeCun (29:48.600)
I know you're a Facebook, but you're also at NYU.
Lex Fridman (29:51.200)
And what does it mean to patent ideas
Lex Fridman (29:55.520)
like these software ideas, essentially?
Lex Fridman (29:58.920)
Or what are mathematical ideas?
Lex Fridman (30:02.360)
Or what are they?
Lex Fridman (30:03.320)
Okay, so they're not mathematical ideas.
Yann LeCun (30:05.640)
They are, you know, algorithms.
Lex Fridman (30:07.600)
And there was a period where the US Patent Office
Yann LeCun (30:11.200)
would allow the patent of software
Lex Fridman (30:14.000)
as long as it was embodied.
Yann LeCun (30:16.280)
The Europeans are very different.
Lex Fridman (30:18.120)
They don't quite accept that.
Yann LeCun (30:20.320)
They have a different concept.
Lex Fridman (30:21.160)
But, you know, I don't, I no longer,
Yann LeCun (30:24.040)
I mean, I never actually strongly believed in this,
Lex Fridman (30:26.280)
but I don't believe in this kind of patent.
Yann LeCun (30:28.880)
Facebook basically doesn't believe in this kind of patent.
Lex Fridman (30:34.040)
Google fires patents because they've been burned with Apple.
Lex Fridman (30:39.040)
And so now they do this for defensive purpose,
Lex Fridman (30:41.360)
but usually they say,
Yann LeCun (30:42.720)
we're not gonna sue you if you infringe.
Lex Fridman (30:44.760)
Facebook has a similar policy.
Yann LeCun (30:47.080)
They say, you know, we fire patents on certain things
Lex Fridman (30:49.560)
for defensive purpose.
Yann LeCun (30:50.480)
We're not gonna sue you if you infringe,
Lex Fridman (30:52.080)
unless you sue us.
Lex Fridman (30:54.600)
So the industry does not believe in patents.
Lex Fridman (30:59.240)
They are there because of, you know,
Yann LeCun (31:00.720)
the legal landscape and various things.
Lex Fridman (31:03.280)
But I don't really believe in patents
Yann LeCun (31:06.280)
for this kind of stuff.
Lex Fridman (31:07.560)
So that's a great thing.
Lex Fridman (31:09.600)
So I...
Lex Fridman (31:10.440)
I'll tell you a worse story, actually.
Lex Fridman (31:11.800)
So what happens was the first patent about convolutional net
Lex Fridman (31:15.440)
was about kind of the early version of convolutional net
Yann LeCun (31:18.240)
that didn't have separate pooling layers.
Lex Fridman (31:19.960)
It had convolutional layers
Lex Fridman (31:22.880)
which tried more than one, if you want, right?
Lex Fridman (31:25.240)
And then there was a second one on convolutional nets
Yann LeCun (31:28.440)
with separate pooling layers, trained with backprop.
Lex Fridman (31:31.720)
And there were files filed in 89 and 1990
Yann LeCun (31:35.280)
or something like this.
Lex Fridman (31:36.240)
At the time, the life of a patent was 17 years.
Lex Fridman (31:40.280)
So here's what happened over the next few years
Lex Fridman (31:42.080)
is that we started developing character recognition
Yann LeCun (31:45.480)
technology around convolutional nets.
Lex Fridman (31:48.640)
And in 1994,
Yann LeCun (31:52.200)
a check reading system was deployed in ATM machines.
Lex Fridman (31:56.160)
In 1995, it was for large check reading machines
Yann LeCun (31:59.040)
in back offices, et cetera.
Lex Fridman (32:00.520)
And those systems were developed by an engineering group
Yann LeCun (32:04.840)
that we were collaborating with at AT&T.
Lex Fridman (32:07.000)
And they were commercialized by NCR,
Yann LeCun (32:08.640)
which at the time was a subsidiary of AT&T.
Lex Fridman (32:11.640)
Now AT&T split up in 1996,
Yann LeCun (32:17.000)
early 1996.
Lex Fridman (32:18.640)
And the lawyers just looked at all the patents
Lex Fridman (32:20.440)
and they distributed the patents among the various companies.
Lex Fridman (32:23.000)
They gave the convolutional net patent to NCR
Yann LeCun (32:26.440)
because they were actually selling products that used it.
Lex Fridman (32:29.240)
But nobody at NCR had any idea what a convolutional net was.
Yann LeCun (32:32.320)
Yeah.
Lex Fridman (32:33.240)
Okay.
Lex Fridman (32:34.080)
So between 1996 and 2007,
Lex Fridman (32:38.080)
so there's a whole period until 2002
Yann LeCun (32:39.880)
where I didn't actually work on machine learning
Lex Fridman (32:42.040)
or convolutional net.
Yann LeCun (32:42.880)
I resumed working on this around 2002.
Lex Fridman (32:45.920)
And between 2002 and 2007,
Yann LeCun (32:47.520)
I was working on them, crossing my finger
Lex Fridman (32:49.560)
that nobody at NCR would notice.
Yann LeCun (32:51.240)
Nobody noticed.
Lex Fridman (32:52.080)
Yeah, and I hope that this kind of somewhat,
Yann LeCun (32:55.640)
as you said, lawyers aside,
Lex Fridman (32:58.320)
relative openness of the community now will continue.
Yann LeCun (33:02.920)
It accelerates the entire progress of the industry.
Lex Fridman (33:05.960)
And the problems that Facebook and Google
Lex Fridman (33:11.600)
and others are facing today
Lex Fridman (33:13.040)
is not whether Facebook or Google or Microsoft or IBM
Yann LeCun (33:16.000)
or whoever is ahead of the other.
Lex Fridman (33:18.080)
It's that we don't have the technology
Yann LeCun (33:19.680)
to build the things we want to build.
Lex Fridman (33:21.080)
We want to build intelligent virtual assistants
Yann LeCun (33:23.240)
that have common sense.
Lex Fridman (33:24.960)
We don't have monopoly on good ideas for this.
Yann LeCun (33:26.720)
We don't believe we do.
Lex Fridman (33:27.960)
Maybe others believe they do, but we don't.
Yann LeCun (33:30.440)
Okay.
Lex Fridman (33:31.320)
If a startup tells you they have the secret
Yann LeCun (33:33.840)
to human level intelligence and common sense,
Lex Fridman (33:36.880)
don't believe them, they don't.
Lex Fridman (33:38.240)
And it's gonna take the entire work
Lex Fridman (33:42.760)
of the world research community for a while
Yann LeCun (33:45.240)
to get to the point where you can go off
Lex Fridman (33:47.600)
and each of those companies
Yann LeCun (33:49.240)
kind of start to build things on this.
Lex Fridman (33:50.640)
We're not there yet.
Yann LeCun (33:51.760)
It's absolutely, and this calls to the gap
Lex Fridman (33:54.680)
between the space of ideas
Lex Fridman (33:57.000)
and the rigorous testing of those ideas
Lex Fridman (34:00.440)
of practical application that you often speak to.
Yann LeCun (34:03.560)
You've written advice saying don't get fooled
Lex Fridman (34:06.320)
by people who claim to have a solution
Yann LeCun (34:08.760)
to artificial general intelligence,
Lex Fridman (34:10.560)
who claim to have an AI system
Yann LeCun (34:11.960)
that works just like the human brain
Lex Fridman (34:14.280)
or who claim to have figured out how the brain works.
Yann LeCun (34:17.080)
Ask them what the error rate they get
Lex Fridman (34:20.960)
on MNIST or ImageNet.
Lex Fridman (34:23.120)
So this is a little dated by the way.
Lex Fridman (34:25.400)
2000, I mean five years, who's counting?
Yann LeCun (34:28.280)
Okay, but I think your opinion is still,
Lex Fridman (34:30.920)
MNIST and ImageNet, yes, may be dated,
Lex Fridman (34:34.920)
there may be new benchmarks, right?
Lex Fridman (34:36.360)
But I think that philosophy is one you still
Yann LeCun (34:39.360)
in somewhat hold, that benchmarks
Lex Fridman (34:43.400)
and the practical testing, the practical application
Yann LeCun (34:45.760)
is where you really get to test the ideas.
Lex Fridman (34:48.000)
Well, it may not be completely practical.
Yann LeCun (34:49.840)
Like for example, it could be a toy data set,
Lex Fridman (34:52.480)
but it has to be some sort of task
Yann LeCun (34:54.880)
that the community as a whole has accepted
Lex Fridman (34:57.320)
as some sort of standard kind of benchmark if you want.
Yann LeCun (35:00.640)
It doesn't need to be real.
Lex Fridman (35:01.480)
So for example, many years ago here at FAIR,
Yann LeCun (35:05.400)
people, Jason West and Antoine Borne
Lex Fridman (35:07.080)
and a few others proposed the Babi tasks,
Yann LeCun (35:09.080)
which were kind of a toy problem to test
Lex Fridman (35:12.280)
the ability of machines to reason actually
Yann LeCun (35:14.360)
to access working memory and things like this.
Lex Fridman (35:16.960)
And it was very useful even though it wasn't a real task.
Yann LeCun (35:20.120)
MNIST is kind of halfway real task.
Lex Fridman (35:23.680)
So toy problems can be very useful.
Yann LeCun (35:26.040)
It's just that I was really struck by the fact
Lex Fridman (35:29.000)
that a lot of people, particularly a lot of people
Yann LeCun (35:31.160)
with money to invest would be fooled by people telling them,
Lex Fridman (35:34.380)
oh, we have the algorithm of the cortex
Lex Fridman (35:37.400)
and you should give us 50 million.
Lex Fridman (35:39.360)
Yes, absolutely.
Lex Fridman (35:40.200)
So there's a lot of people who try to take advantage
Lex Fridman (35:45.280)
of the hype for business reasons and so on.
Lex Fridman (35:48.240)
But let me sort of talk to this idea
Lex Fridman (35:50.800)
that sort of new ideas, the ideas that push the field
Yann LeCun (35:55.320)
forward may not yet have a benchmark
Lex Fridman (35:58.620)
or it may be very difficult to establish a benchmark.
Yann LeCun (36:00.880)
I agree.
Lex Fridman (36:01.720)
That's part of the process.
Yann LeCun (36:02.560)
Establishing benchmarks is part of the process.
Lex Fridman (36:04.600)
So what are your thoughts about,
Lex Fridman (36:07.300)
so we have these benchmarks on around stuff we can do
Lex Fridman (36:10.960)
with images from classification to captioning
Yann LeCun (36:14.920)
to just every kind of information you can pull off
Lex Fridman (36:16.940)
from images and the surface level.
Yann LeCun (36:18.880)
There's audio data sets, there's some video.
Lex Fridman (36:22.600)
What can we start, natural language, what kind of stuff,
Lex Fridman (36:27.480)
what kind of benchmarks do you see that start creeping
Lex Fridman (36:30.160)
on to more something like intelligence, like reasoning,
Yann LeCun (36:34.840)
like maybe you don't like the term,
Lex Fridman (36:37.440)
but AGI echoes of that kind of formulation.
Yann LeCun (36:41.520)
A lot of people are working on interactive environments
Lex Fridman (36:44.160)
in which you can train and test intelligence systems.
Lex Fridman (36:48.120)
So there, for example, it's the classical paradigm
Lex Fridman (36:54.840)
of supervised learning is that you have a data set,
Yann LeCun (36:57.960)
you partition it into a training set, validation set,
Lex Fridman (37:00.040)
test set, and there's a clear protocol, right?
Lex Fridman (37:03.040)
But what if that assumes that the samples
Lex Fridman (37:06.400)
are statistically independent, you can exchange them,
Yann LeCun (37:10.100)
the order in which you see them shouldn't matter,
Lex Fridman (37:12.240)
things like that.
Lex Fridman (37:13.480)
But what if the answer you give determines
Lex Fridman (37:16.020)
the next sample you see, which is the case, for example,
Lex Fridman (37:18.760)
in robotics, right?
Lex Fridman (37:19.600)
You robot does something and then it gets exposed
Yann LeCun (37:22.480)
to a new room, and depending on where it goes,
Lex Fridman (37:25.120)
the room would be different.
Lex Fridman (37:26.000)
So that creates the exploration problem.
Lex Fridman (37:30.120)
The what if the samples, so that creates also a dependency
Lex Fridman (37:34.280)
between samples, right?
Lex Fridman (37:35.480)
You, if you move, if you can only move in space,
Yann LeCun (37:39.640)
the next sample you're gonna see is gonna be probably
Lex Fridman (37:41.840)
in the same building, most likely, right?
Lex Fridman (37:44.080)
So all the assumptions about the validity
Lex Fridman (37:47.920)
of this training set, test set hypothesis break.
Yann LeCun (37:51.560)
Whenever a machine can take an action
Lex Fridman (37:53.120)
that has an influence in the world,
Lex Fridman (37:54.960)
and it's what it's gonna see.
Lex Fridman (37:56.400)
So people are setting up artificial environments
Lex Fridman (38:00.160)
where that takes place, right?
Lex Fridman (38:02.080)
The robot runs around a 3D model of a house
Lex Fridman (38:05.840)
and can interact with objects and things like this.
Lex Fridman (38:08.680)
So you do robotics based simulation,
Yann LeCun (38:10.380)
you have those opening a gym type thing
Lex Fridman (38:14.400)
or Mujoko kind of simulated robots
Lex Fridman (38:18.800)
and you have games, things like that.
Lex Fridman (38:21.280)
So that's where the field is going really,
Yann LeCun (38:23.640)
this kind of environment.
Lex Fridman (38:25.760)
Now, back to the question of AGI.
Yann LeCun (38:28.600)
I don't like the term AGI because it implies
Lex Fridman (38:33.180)
that human intelligence is general
Lex Fridman (38:35.760)
and human intelligence is nothing like general.
Lex Fridman (38:38.360)
It's very, very specialized.
Yann LeCun (38:40.840)
We think it's general.
Lex Fridman (38:41.720)
We'd like to think of ourselves
Yann LeCun (38:42.760)
as having general intelligence.
Lex Fridman (38:43.840)
We don't, we're very specialized.
Yann LeCun (38:46.120)
We're only slightly more general than.
Lex Fridman (38:47.560)
Why does it feel general?
Lex Fridman (38:48.900)
So you kind of, the term general.
Lex Fridman (38:52.040)
I think what's impressive about humans is ability to learn,
Yann LeCun (38:56.320)
as we were talking about learning,
Lex Fridman (38:58.240)
to learn in just so many different domains.
Yann LeCun (39:01.280)
It's perhaps not arbitrarily general,
Lex Fridman (39:04.440)
but just you can learn in many domains
Lex Fridman (39:06.440)
and integrate that knowledge somehow.
Lex Fridman (39:08.240)
Okay.
Yann LeCun (39:09.080)
The knowledge persists.
Lex Fridman (39:09.920)
So let me take a very specific example.
Yann LeCun (39:11.640)
Yes.
Lex Fridman (39:12.480)
It's not an example.
Yann LeCun (39:13.300)
It's more like a quasi mathematical demonstration.
Lex Fridman (39:17.080)
So you have about 1 million fibers
Yann LeCun (39:18.520)
coming out of one of your eyes.
Lex Fridman (39:20.420)
Okay, 2 million total,
Lex Fridman (39:21.320)
but let's talk about just one of them.
Lex Fridman (39:23.440)
It's 1 million nerve fibers, your optical nerve.
Yann LeCun (39:27.160)
Let's imagine that they are binary.
Lex Fridman (39:28.800)
So they can be active or inactive, right?
Lex Fridman (39:30.640)
So the input to your visual cortex is 1 million bits.
Lex Fridman (39:34.060)
Mm hmm.
Yann LeCun (39:36.900)
Now they're connected to your brain in a particular way,
Lex Fridman (39:39.420)
and your brain has connections
Yann LeCun (39:41.940)
that are kind of a little bit like a convolutional net,
Lex Fridman (39:44.180)
they're kind of local, you know, in space
Lex Fridman (39:46.780)
and things like this.
Lex Fridman (39:47.940)
Now, imagine I play a trick on you.
Yann LeCun (39:50.980)
It's a pretty nasty trick, I admit.
Lex Fridman (39:53.060)
I cut your optical nerve,
Lex Fridman (39:55.720)
and I put a device that makes a random perturbation
Lex Fridman (39:58.500)
of a permutation of all the nerve fibers.
Lex Fridman (40:01.100)
So now what comes to your brain
Lex Fridman (40:04.580)
is a fixed but random permutation of all the pixels.
Yann LeCun (40:09.160)
There's no way in hell that your visual cortex,
Lex Fridman (40:11.380)
even if I do this to you in infancy,
Yann LeCun (40:14.760)
will actually learn vision
Lex Fridman (40:16.500)
to the same level of quality that you can.
Lex Fridman (40:20.060)
Got it, and you're saying there's no way you've learned that?
Lex Fridman (40:22.700)
No, because now two pixels that are nearby in the world
Yann LeCun (40:25.620)
will end up in very different places in your visual cortex,
Lex Fridman (40:29.240)
and your neurons there have no connections with each other
Yann LeCun (40:31.620)
because they're only connected locally.
Lex Fridman (40:33.500)
So this whole, our entire, the hardware is built
Lex Fridman (40:36.660)
in many ways to support?
Lex Fridman (40:38.620)
The locality of the real world.
Yann LeCun (40:40.180)
Yes, that's specialization.
Lex Fridman (40:42.580)
Yeah, but it's still pretty damn impressive,
Lex Fridman (40:44.580)
so it's not perfect generalization, it's not even close.
Lex Fridman (40:46.980)
No, no, it's not that it's not even close, it's not at all.
Yann LeCun (40:50.960)
Yeah, it's not, it's specialized, yeah.
Lex Fridman (40:52.220)
So how many Boolean functions?
Lex Fridman (40:54.020)
So let's imagine you want to train your visual system
Lex Fridman (40:58.260)
to recognize particular patterns of those one million bits.
Lex Fridman (41:03.820)
Okay, so that's a Boolean function, right?
Lex Fridman (41:05.780)
Either the pattern is here or not here,
Yann LeCun (41:07.020)
this is a two way classification
Lex Fridman (41:09.200)
with one million binary inputs.
Lex Fridman (41:13.620)
How many such Boolean functions are there?
Lex Fridman (41:16.260)
Okay, you have two to the one million
Yann LeCun (41:19.940)
combinations of inputs,
Lex Fridman (41:21.180)
for each of those you have an output bit,
Lex Fridman (41:24.060)
and so you have two to the one million
Lex Fridman (41:27.660)
Boolean functions of this type, okay?
Yann LeCun (41:30.060)
Which is an unimaginably large number.
Lex Fridman (41:33.020)
How many of those functions can actually be computed
Lex Fridman (41:35.560)
by your visual cortex?
Lex Fridman (41:37.260)
And the answer is a tiny, tiny, tiny, tiny, tiny, tiny sliver.
Yann LeCun (41:41.460)
Like an enormously tiny sliver.
Lex Fridman (41:43.500)
Yeah, yeah.
Lex Fridman (41:44.980)
So we are ridiculously specialized.
Lex Fridman (41:48.860)
Okay.
Yann LeCun (41:49.700)
But, okay, that's an argument against the word general.
Lex Fridman (41:54.220)
I think there's a, I agree with your intuition,
Lex Fridman (41:59.180)
but I'm not sure it's, it seems the brain is impressively
Lex Fridman (42:06.900)
capable of adjusting to things, so.
Yann LeCun (42:09.660)
It's because we can't imagine tasks
Lex Fridman (42:13.420)
that are outside of our comprehension, right?
Lex Fridman (42:16.340)
So we think we're general because we're general
Lex Fridman (42:18.780)
of all the things that we can apprehend.
Lex Fridman (42:20.780)
But there is a huge world out there
Lex Fridman (42:23.020)
of things that we have no idea.
Yann LeCun (42:24.740)
We call that heat, by the way.
Lex Fridman (42:26.860)
Heat.
Yann LeCun (42:27.700)
Heat.
Lex Fridman (42:28.540)
So, at least physicists call that heat,
Yann LeCun (42:30.660)
or they call it entropy, which is kind of.
Lex Fridman (42:33.420)
You have a thing full of gas, right?
Yann LeCun (42:39.380)
Closed system for gas.
Lex Fridman (42:40.760)
Right?
Yann LeCun (42:41.780)
Closed or not closed.
Lex Fridman (42:42.660)
It has pressure, it has temperature, it has, you know,
Lex Fridman (42:47.660)
and you can write equations, PV equal N on T,
Lex Fridman (42:50.660)
you know, things like that, right?
Yann LeCun (42:52.540)
When you reduce the volume, the temperature goes up,
Lex Fridman (42:54.900)
the pressure goes up, you know, things like that, right?
Yann LeCun (42:57.780)
For perfect gas, at least.
Lex Fridman (42:59.620)
Those are the things you can know about that system.
Lex Fridman (43:02.420)
And it's a tiny, tiny number of bits
Lex Fridman (43:04.580)
compared to the complete information
Yann LeCun (43:06.900)
of the state of the entire system.
Lex Fridman (43:08.340)
Because the state of the entire system
Yann LeCun (43:09.740)
will give you the position of momentum
Lex Fridman (43:11.260)
of every molecule of the gas.
Lex Fridman (43:14.660)
And what you don't know about it is the entropy,
Lex Fridman (43:17.660)
and you interpret it as heat.
Yann LeCun (43:20.620)
The energy contained in that thing is what we call heat.
Lex Fridman (43:24.700)
Now, it's very possible that, in fact,
Yann LeCun (43:28.740)
there is some very strong structure
Lex Fridman (43:30.220)
in how those molecules are moving.
Yann LeCun (43:31.620)
It's just that they are in a way
Lex Fridman (43:33.020)
that we are just not wired to perceive.
Yann LeCun (43:35.580)
Yeah, we're ignorant to it.
Lex Fridman (43:36.420)
And there's, in your infinite amount of things,
Yann LeCun (43:40.500)
we're not wired to perceive.
Lex Fridman (43:41.820)
And you're right, that's a nice way to put it.
Yann LeCun (43:44.660)
We're general to all the things we can imagine,
Lex Fridman (43:47.620)
which is a very tiny subset of all things that are possible.
Lex Fridman (43:51.820)
So it's like comograph complexity
Lex Fridman (43:53.260)
or the comograph chitin sum of complexity.
Yann LeCun (43:55.820)
Yeah.
Lex Fridman (43:56.660)
You know, every bit string or every integer is random,
Yann LeCun (44:02.220)
except for all the ones that you can actually write down.
Lex Fridman (44:05.220)
Yeah.
Yann LeCun (44:06.060)
Yeah.
Lex Fridman (44:06.900)
Yeah.
Yann LeCun (44:07.740)
Yeah.
Lex Fridman (44:08.580)
Yeah.
Yann LeCun (44:09.420)
Yeah.
Lex Fridman (44:10.260)
Yeah, okay.
Lex Fridman (44:12.180)
So beautifully put.
Lex Fridman (44:13.020)
But, you know, so we can just call it artificial intelligence.
Yann LeCun (44:15.460)
We don't need to have a general.
Lex Fridman (44:17.980)
Or human level.
Yann LeCun (44:18.820)
Human level intelligence is good.
Lex Fridman (44:20.900)
You know, you'll start, anytime you touch human,
Yann LeCun (44:24.700)
it gets interesting because, you know,
Lex Fridman (44:30.660)
it's because we attach ourselves to human
Lex Fridman (44:33.420)
and it's difficult to define what human intelligence is.
Lex Fridman (44:36.060)
Yeah.
Yann LeCun (44:37.220)
Nevertheless, my definition is maybe dem impressive
Lex Fridman (44:42.100)
intelligence, okay?
Yann LeCun (44:43.900)
Dem impressive demonstration of intelligence, whatever.
Lex Fridman (44:46.700)
And so on that topic, most successes in deep learning
Yann LeCun (44:51.420)
have been in supervised learning.
Lex Fridman (44:53.700)
What is your view on unsupervised learning?
Yann LeCun (44:57.860)
Is there a hope to reduce involvement of human input
Lex Fridman (45:03.180)
and still have successful systems
Lex Fridman (45:05.620)
that have practical use?
Lex Fridman (45:08.300)
Yeah, I mean, there's definitely a hope.
Yann LeCun (45:09.900)
It's more than a hope, actually.
Lex Fridman (45:11.180)
It's mounting evidence for it.
Lex Fridman (45:13.900)
And that's basically all I do.
Lex Fridman (45:16.020)
Like, the only thing I'm interested in at the moment is,
Yann LeCun (45:19.100)
I call it self supervised learning, not unsupervised.
Lex Fridman (45:21.260)
Because unsupervised learning is a loaded term.
Yann LeCun (45:25.700)
People who know something about machine learning,
Lex Fridman (45:27.900)
you know, tell you, so you're doing clustering or PCA,
Yann LeCun (45:30.620)
which is not the case.
Lex Fridman (45:31.580)
And the white public, you know,
Yann LeCun (45:32.580)
when you say unsupervised learning,
Lex Fridman (45:33.620)
oh my God, machines are gonna learn by themselves
Yann LeCun (45:35.860)
without supervision.
Lex Fridman (45:37.300)
You know, they see this as...
Lex Fridman (45:39.660)
Where's the parents?
Lex Fridman (45:40.780)
Yeah, so I call it self supervised learning
Yann LeCun (45:42.900)
because, in fact, the underlying algorithms that are used
Lex Fridman (45:46.140)
are the same algorithms as the supervised learning
Yann LeCun (45:48.340)
algorithms, except that what we train them to do
Lex Fridman (45:52.300)
is not predict a particular set of variables,
Yann LeCun (45:55.540)
like the category of an image,
Lex Fridman (46:00.420)
and not to predict a set of variables
Yann LeCun (46:02.540)
that have been provided by human labelers.
Lex Fridman (46:06.380)
But what you're trying the machine to do
Yann LeCun (46:07.380)
is basically reconstruct a piece of its input
Lex Fridman (46:10.300)
that is being maxed out, essentially.
Lex Fridman (46:14.140)
You can think of it this way, right?
Lex Fridman (46:15.620)
So show a piece of video to a machine
Lex Fridman (46:18.780)
and ask it to predict what's gonna happen next.
Lex Fridman (46:20.940)
And of course, after a while, you can show what happens
Lex Fridman (46:23.780)
and the machine will kind of train itself
Lex Fridman (46:26.220)
to do better at that task.
Yann LeCun (46:28.820)
You can do like all the latest, most successful models
Lex Fridman (46:32.220)
in natural language processing,
Yann LeCun (46:33.260)
use self supervised learning.
Lex Fridman (46:36.220)
You know, sort of BERT style systems, for example, right?
Yann LeCun (46:38.660)
You show it a window of a dozen words on a text corpus,
Lex Fridman (46:43.500)
you take out 15% of the words,
Lex Fridman (46:46.300)
and then you train the machine to predict the words
Lex Fridman (46:49.900)
that are missing, that self supervised learning.
Yann LeCun (46:52.820)
It's not predicting the future,
Lex Fridman (46:53.980)
it's just predicting things in the middle,
Lex Fridman (46:56.260)
but you could have it predict the future,
Lex Fridman (46:57.860)
that's what language models do.
Lex Fridman (46:59.500)
So you construct, so in an unsupervised way,
Lex Fridman (47:01.780)
you construct a model of language.
Lex Fridman (47:03.980)
Do you think...
Lex Fridman (47:05.060)
Or video or the physical world or whatever, right?
Lex Fridman (47:09.140)
How far do you think that can take us?
Lex Fridman (47:12.620)
Do you think BERT understands anything?
Yann LeCun (47:18.020)
To some level, it has a shallow understanding of text,
Lex Fridman (47:23.460)
but it needs to, I mean,
Yann LeCun (47:24.740)
to have kind of true human level intelligence,
Lex Fridman (47:26.820)
I think you need to ground language in reality.
Lex Fridman (47:29.220)
So some people are attempting to do this, right?
Lex Fridman (47:32.780)
Having systems that kind of have some visual representation
Yann LeCun (47:35.460)
of what is being talked about,
Lex Fridman (47:37.420)
which is one reason you need
Yann LeCun (47:38.580)
those interactive environments actually.
Lex Fridman (47:41.060)
But this is like a huge technical problem
Yann LeCun (47:43.300)
that is not solved,
Lex Fridman (47:45.060)
and that explains why self supervised learning
Yann LeCun (47:47.900)
works in the context of natural language,
Lex Fridman (47:49.980)
but does not work in the context, or at least not well,
Yann LeCun (47:52.740)
in the context of image recognition and video,
Lex Fridman (47:55.380)
although it's making progress quickly.
Lex Fridman (47:57.820)
And the reason, that reason is the fact that
Lex Fridman (48:01.820)
it's much easier to represent uncertainty in the prediction
Yann LeCun (48:05.300)
in a context of natural language
Lex Fridman (48:06.900)
than it is in the context of things like video and images.
Lex Fridman (48:10.100)
So for example, if I ask you to predict
Lex Fridman (48:12.940)
what words are missing,
Yann LeCun (48:14.140)
15% of the words that I've taken out.
Lex Fridman (48:17.700)
The possibilities are small.
Lex Fridman (48:19.140)
That means... It's small, right?
Lex Fridman (48:20.020)
There is 100,000 words in the lexicon,
Lex Fridman (48:23.340)
and what the machine spits out
Lex Fridman (48:24.820)
is a big probability vector, right?
Yann LeCun (48:27.620)
It's a bunch of numbers between zero and one
Lex Fridman (48:29.660)
that sum to one.
Lex Fridman (48:30.740)
And we know how to do this with computers.
Lex Fridman (48:34.460)
So there, representing uncertainty in the prediction
Yann LeCun (48:36.940)
is relatively easy, and that's, in my opinion,
Lex Fridman (48:39.100)
why those techniques work for NLP.
Yann LeCun (48:42.460)
For images, if you ask...
Lex Fridman (48:45.460)
If you block a piece of an image,
Lex Fridman (48:46.900)
and you ask the system,
Lex Fridman (48:47.740)
reconstruct that piece of the image,
Yann LeCun (48:49.180)
there are many possible answers.
Lex Fridman (48:51.540)
They are all perfectly legit, right?
Lex Fridman (48:54.620)
And how do you represent this set of possible answers?
Lex Fridman (48:58.740)
You can't train a system to make one prediction.
Yann LeCun (49:00.900)
You can't train a neural net to say,
Lex Fridman (49:02.500)
here it is, that's the image,
Yann LeCun (49:04.620)
because there's a whole set of things
Lex Fridman (49:06.420)
that are compatible with it.
Lex Fridman (49:07.260)
So how do you get the machine to represent
Lex Fridman (49:08.740)
not a single output, but a whole set of outputs?
Lex Fridman (49:13.060)
And similarly with video prediction,
Lex Fridman (49:17.220)
there's a lot of things that can happen
Yann LeCun (49:19.220)
in the future of video.
Lex Fridman (49:20.100)
You're looking at me right now.
Yann LeCun (49:21.140)
I'm not moving my head very much,
Lex Fridman (49:22.740)
but I might turn my head to the left or to the right.
Yann LeCun (49:26.940)
If you don't have a system that can predict this,
Lex Fridman (49:30.420)
and you train it with least square
Yann LeCun (49:31.740)
to minimize the error with the prediction
Lex Fridman (49:33.700)
and what I'm doing,
Lex Fridman (49:34.660)
what you get is a blurry image of myself
Lex Fridman (49:36.940)
in all possible future positions that I might be in,
Yann LeCun (49:39.660)
which is not a good prediction.
Lex Fridman (49:41.780)
So there might be other ways
Yann LeCun (49:43.420)
to do the self supervision for visual scenes.
Lex Fridman (49:48.100)
Like what?
Yann LeCun (49:48.940)
I mean, if I knew, I wouldn't tell you,
Lex Fridman (49:52.740)
publish it first, I don't know.
Yann LeCun (49:55.620)
No, there might be.
Lex Fridman (49:57.540)
So I mean, these are kind of,
Yann LeCun (50:00.300)
there might be artificial ways of like self play in games,
Lex Fridman (50:03.260)
the way you can simulate part of the environment.
Yann LeCun (50:05.780)
Oh, that doesn't solve the problem.
Lex Fridman (50:06.820)
It's just a way of generating data.
Lex Fridman (50:10.420)
But because you have more of a control,
Lex Fridman (50:12.580)
like maybe you can control,
Yann LeCun (50:14.620)
yeah, it's a way to generate data.
Lex Fridman (50:16.100)
That's right.
Lex Fridman (50:16.940)
And because you can do huge amounts of data generation,
Lex Fridman (50:20.500)
that doesn't, you're right.
Yann LeCun (50:21.580)
Well, it creeps up on the problem from the side of data,
Lex Fridman (50:26.020)
and you don't think that's the right way to creep up.
Yann LeCun (50:27.700)
It doesn't solve this problem
Lex Fridman (50:28.980)
of handling uncertainty in the world, right?
Lex Fridman (50:30.980)
So if you have a machine learn a predictive model
Lex Fridman (50:35.260)
of the world in a game that is deterministic
Lex Fridman (50:38.180)
or quasi deterministic, it's easy, right?
Lex Fridman (50:42.540)
Just give a few frames of the game to a ConvNet,
Yann LeCun (50:45.940)
put a bunch of layers,
Lex Fridman (50:47.060)
and then have the game generates the next few frames.
Lex Fridman (50:49.660)
And if the game is deterministic, it works fine.
Lex Fridman (50:54.860)
And that includes feeding the system with the action
Yann LeCun (50:59.140)
that your little character is gonna take.
Lex Fridman (51:03.060)
The problem comes from the fact that the real world
Lex Fridman (51:06.660)
and most games are not entirely predictable.
Lex Fridman (51:09.700)
And so there you get those blurry predictions
Lex Fridman (51:11.340)
and you can't do planning with blurry predictions, right?
Lex Fridman (51:14.500)
So if you have a perfect model of the world,
Yann LeCun (51:17.460)
you can, in your head, run this model
Lex Fridman (51:20.740)
with a hypothesis for a sequence of actions,
Lex Fridman (51:24.100)
and you're going to predict the outcome
Lex Fridman (51:25.380)
of that sequence of actions.
Lex Fridman (51:28.620)
But if your model is imperfect, how can you plan?
Lex Fridman (51:32.460)
Yeah, it quickly explodes.
Lex Fridman (51:34.820)
What are your thoughts on the extension of this,
Lex Fridman (51:37.300)
which topic I'm super excited about,
Yann LeCun (51:39.700)
it's connected to something you were talking about
Lex Fridman (51:41.380)
in terms of robotics, is active learning.
Lex Fridman (51:44.580)
So as opposed to sort of completely unsupervised
Lex Fridman (51:47.940)
or self supervised learning,
Yann LeCun (51:51.060)
you ask the system for human help
Lex Fridman (51:54.900)
for selecting parts you want annotated next.
Lex Fridman (51:58.100)
So if you think about a robot exploring a space
Lex Fridman (52:00.660)
or a baby exploring a space
Yann LeCun (52:02.420)
or a system exploring a data set,
Lex Fridman (52:05.260)
every once in a while asking for human input,
Lex Fridman (52:07.940)
do you see value in that kind of work?
Lex Fridman (52:12.180)
I don't see transformative value.
Yann LeCun (52:14.180)
It's going to make things that we can already do
Lex Fridman (52:18.180)
more efficient or they will learn slightly more efficiently,
Lex Fridman (52:20.780)
but it's not going to make machines
Lex Fridman (52:21.940)
sort of significantly more intelligent.
Yann LeCun (52:23.700)
I think, and by the way, there is no opposition,
Lex Fridman (52:29.340)
there's no conflict between self supervised learning,
Yann LeCun (52:34.620)
reinforcement learning and supervised learning
Lex Fridman (52:35.980)
or imitation learning or active learning.
Yann LeCun (52:39.060)
I see self supervised learning
Lex Fridman (52:40.500)
as a preliminary to all of the above.
Yann LeCun (52:43.820)
Yes.
Lex Fridman (52:44.660)
So the example I use very often is how is it that,
Lex Fridman (52:50.420)
so if you use classical reinforcement learning,
Lex Fridman (52:54.580)
deep reinforcement learning, if you want,
Yann LeCun (52:57.540)
the best methods today,
Lex Fridman (53:01.300)
so called model free reinforcement learning
Yann LeCun (53:03.100)
to learn to play Atari games,
Lex Fridman (53:04.660)
take about 80 hours of training to reach the level
Yann LeCun (53:07.100)
that any human can reach in about 15 minutes.
Lex Fridman (53:11.540)
They get better than humans, but it takes them a long time.
Yann LeCun (53:16.540)
Alpha star, okay, the, you know,
Lex Fridman (53:20.420)
Aureal Vinyals and his teams,
Yann LeCun (53:22.260)
the system to play StarCraft plays,
Lex Fridman (53:27.900)
you know, a single map, a single type of player.
Yann LeCun (53:32.900)
A single player and can reach better than human level
Lex Fridman (53:38.820)
with about the equivalent of 200 years of training
Yann LeCun (53:43.380)
playing against itself.
Lex Fridman (53:45.300)
It's 200 years, right?
Yann LeCun (53:46.420)
It's not something that no human can ever do.
Lex Fridman (53:50.100)
I mean, I'm not sure what lesson to take away from that.
Yann LeCun (53:52.340)
Okay, now take those algorithms,
Lex Fridman (53:54.820)
the best algorithms we have today
Yann LeCun (53:57.380)
to train a car to drive itself.
Lex Fridman (54:00.200)
It would probably have to drive millions of hours.
Yann LeCun (54:02.960)
It will have to kill thousands of pedestrians.
Lex Fridman (54:04.680)
It will have to run into thousands of trees.
Yann LeCun (54:06.480)
It will have to run off cliffs.
Lex Fridman (54:08.520)
And it had to run off cliff multiple times
Yann LeCun (54:10.560)
before it figures out that it's a bad idea, first of all.
Lex Fridman (54:14.040)
And second of all, before it figures out how not to do it.
Lex Fridman (54:17.520)
And so, I mean, this type of learning obviously
Lex Fridman (54:19.840)
does not reflect the kind of learning
Yann LeCun (54:21.360)
that animals and humans do.
Lex Fridman (54:23.200)
There is something missing
Yann LeCun (54:24.240)
that's really, really important there.
Lex Fridman (54:26.320)
And my hypothesis, which I've been advocating
Yann LeCun (54:28.600)
for like five years now,
Lex Fridman (54:30.400)
is that we have predictive models of the world
Yann LeCun (54:34.840)
that include the ability to predict under uncertainty.
Lex Fridman (54:38.520)
And what allows us to not run off a cliff
Yann LeCun (54:43.520)
when we learn to drive,
Lex Fridman (54:44.720)
most of us can learn to drive in about 20 or 30 hours
Yann LeCun (54:47.040)
of training without ever crashing, causing any accident.
Lex Fridman (54:50.960)
And if we drive next to a cliff,
Yann LeCun (54:53.280)
we know that if we turn the wheel to the right,
Lex Fridman (54:55.240)
the car is gonna run off the cliff
Lex Fridman (54:57.080)
and nothing good is gonna come out of this.
Lex Fridman (54:58.760)
Because we have a pretty good model of intuitive physics
Yann LeCun (55:00.600)
that tells us the car is gonna fall.
Lex Fridman (55:02.280)
We know about gravity.
Yann LeCun (55:04.200)
Babies learn this around the age of eight or nine months
Lex Fridman (55:07.120)
that objects don't float, they fall.
Lex Fridman (55:11.200)
And we have a pretty good idea of the effect
Lex Fridman (55:13.720)
of turning the wheel on the car
Lex Fridman (55:15.040)
and we know we need to stay on the road.
Lex Fridman (55:16.960)
So there's a lot of things that we bring to the table,
Yann LeCun (55:19.480)
which is basically our predictive model of the world.
Lex Fridman (55:22.400)
And that model allows us to not do stupid things.
Lex Fridman (55:25.840)
And to basically stay within the context
Lex Fridman (55:28.160)
of things we need to do.
Yann LeCun (55:29.960)
We still face unpredictable situations
Lex Fridman (55:32.520)
and that's how we learn.
Lex Fridman (55:34.040)
But that allows us to learn really, really, really quickly.
Lex Fridman (55:37.600)
So that's called model based reinforcement learning.
Yann LeCun (55:41.200)
There's some imitation and supervised learning
Lex Fridman (55:43.000)
because we have a driving instructor
Yann LeCun (55:44.840)
that tells us occasionally what to do.
Lex Fridman (55:47.000)
But most of the learning is learning the model,
Yann LeCun (55:52.080)
learning physics that we've done since we were babies.
Lex Fridman (55:55.080)
That's where all, almost all the learning.
Lex Fridman (55:56.880)
And the physics is somewhat transferable from,
Lex Fridman (56:00.080)
it's transferable from scene to scene.
Yann LeCun (56:01.960)
Stupid things are the same everywhere.
Lex Fridman (56:04.320)
Yeah, I mean, if you have experience of the world,
Yann LeCun (56:07.720)
you don't need to be from a particularly intelligent species
Lex Fridman (56:11.400)
to know that if you spill water from a container,
Yann LeCun (56:16.520)
the rest is gonna get wet.
Lex Fridman (56:18.800)
You might get wet.
Lex Fridman (56:20.640)
So cats know this, right?
Lex Fridman (56:22.840)
Yeah.
Yann LeCun (56:23.680)
Right, so the main problem we need to solve
Lex Fridman (56:27.040)
is how do we learn models of the world?
Yann LeCun (56:29.920)
That's what I'm interested in.
Lex Fridman (56:31.280)
That's what self supervised learning is all about.
Yann LeCun (56:34.080)
If you were to try to construct a benchmark for,
Lex Fridman (56:39.400)
let's look at MNIST.
Yann LeCun (56:41.120)
I love that data set.
Lex Fridman (56:44.120)
Do you think it's useful, interesting, slash possible
Yann LeCun (56:48.040)
to perform well on MNIST with just one example
Lex Fridman (56:52.320)
of each digit and how would we solve that problem?
Yann LeCun (56:58.640)
The answer is probably yes.
Lex Fridman (56:59.560)
The question is what other type of learning
Lex Fridman (57:02.400)
are you allowed to do?
Lex Fridman (57:03.240)
So if what you're allowed to do is train
Yann LeCun (57:04.800)
on some gigantic data set of labeled digit,
Lex Fridman (57:07.360)
that's called transfer learning.
Lex Fridman (57:08.840)
And we know that works, okay?
Lex Fridman (57:11.680)
We do this at Facebook, like in production, right?
Yann LeCun (57:13.560)
We train large convolutional nets to predict hashtags
Lex Fridman (57:17.040)
that people type on Instagram
Lex Fridman (57:18.200)
and we train on billions of images, literally billions.
Lex Fridman (57:20.960)
And then we chop off the last layer
Lex Fridman (57:22.920)
and fine tune on whatever task we want.
Lex Fridman (57:24.920)
That works really well.
Yann LeCun (57:26.360)
You can beat the ImageNet record with this.
Lex Fridman (57:28.760)
We actually open sourced the whole thing
Yann LeCun (57:30.520)
like a few weeks ago.
Lex Fridman (57:31.800)
Yeah, that's still pretty cool.
Lex Fridman (57:33.320)
But yeah, so what would be impressive?
Lex Fridman (57:36.800)
What's useful and impressive?
Lex Fridman (57:38.160)
What kind of transfer learning
Lex Fridman (57:39.280)
would be useful and impressive?
Lex Fridman (57:40.320)
Is it Wikipedia, that kind of thing?
Lex Fridman (57:42.600)
No, no, so I don't think transfer learning
Yann LeCun (57:44.960)
is really where we should focus.
Lex Fridman (57:46.240)
We should try to do,
Yann LeCun (57:48.000)
you know, have a kind of scenario for Benchmark
Lex Fridman (57:51.200)
where you have unlabeled data
Lex Fridman (57:53.680)
and you can, and it's very large number of unlabeled data.
Lex Fridman (57:58.680)
It could be video clips.
Yann LeCun (58:00.640)
It could be where you do, you know, frame prediction.
Lex Fridman (58:03.680)
It could be images where you could choose to,
Yann LeCun (58:06.160)
you know, mask a piece of it, could be whatever,
Lex Fridman (58:10.680)
but they're unlabeled and you're not allowed to label them.
Lex Fridman (58:13.920)
So you do some training on this,
Lex Fridman (58:18.040)
and then you train on a particular supervised task,
Yann LeCun (58:24.720)
ImageNet or a NIST,
Lex Fridman (58:26.320)
and you measure how your test error decrease
Yann LeCun (58:30.200)
or validation error decreases
Lex Fridman (58:31.480)
as you increase the number of label training samples.
Yann LeCun (58:35.400)
Okay, and what you'd like to see is that,
Lex Fridman (58:40.400)
you know, your error decreases much faster
Yann LeCun (58:43.000)
than if you train from scratch from random weights.
Lex Fridman (58:46.560)
So that to reach the same level of performance
Lex Fridman (58:48.600)
and a completely supervised, purely supervised system
Lex Fridman (58:52.120)
would reach you would need way fewer samples.
Lex Fridman (58:54.440)
So that's the crucial question
Lex Fridman (58:55.760)
because it will answer the question to like, you know,
Yann LeCun (58:58.280)
people interested in medical image analysis.
Lex Fridman (59:01.000)
Okay, you know, if I want to get to a particular level
Yann LeCun (59:05.000)
of error rate for this task,
Lex Fridman (59:07.120)
I know I need a million samples.
Yann LeCun (59:10.480)
Can I do, you know, self supervised pre training
Lex Fridman (59:13.560)
to reduce this to about 100 or something?
Lex Fridman (59:15.800)
And you think the answer there
Lex Fridman (59:16.840)
is self supervised pre training?
Yann LeCun (59:18.960)
Yeah, some form, some form of it.
Lex Fridman (59:23.040)
Telling you active learning, but you disagree.
Yann LeCun (59:26.600)
No, it's not useless.
Lex Fridman (59:28.440)
It's just not gonna lead to a quantum leap.
Yann LeCun (59:30.640)
It's just gonna make things that we already do.
Lex Fridman (59:32.200)
So you're way smarter than me.
Yann LeCun (59:33.720)
I just disagree with you.
Lex Fridman (59:35.160)
But I don't have anything to back that.
Yann LeCun (59:37.280)
It's just intuition.
Lex Fridman (59:38.760)
So I worked a lot of large scale data sets
Lex Fridman (59:40.760)
and there's something that might be magic
Lex Fridman (59:43.640)
in active learning, but okay.
Lex Fridman (59:45.840)
And at least I said it publicly.
Lex Fridman (59:48.560)
At least I'm being an idiot publicly.
Yann LeCun (59:50.520)
Okay.
Lex Fridman (59:51.360)
It's not being an idiot.
Yann LeCun (59:52.200)
It's, you know, working with the data you have.
Lex Fridman (59:54.080)
I mean, I mean, certainly people are doing things like,
Yann LeCun (59:56.360)
okay, I have 3000 hours of, you know,
Lex Fridman (59:59.160)
imitation learning for start driving car,
🔗 相关节目