Andrew Ng: Deep Learning, Education, and Real-World AI
心理与人性AI 与机器学习技术与编程音乐与艺术生物与进化
📋 章节目录
暂无章节信息
🔑 关键词
learningdeepreinforcementphdtakingneuraldoingimpactdatanotesspecializationhabitmachineconceptsunsupervisedgoingdonnetworksupervisedterm
💬 精彩语录
暂无语录
🎙️ 完整对话（684 条）
Lex Fridman (39:15.080)
in traditional software engineering.
在传统的软件工程中。
Lex Fridman (39:17.000)
So it's an evolving discipline,
所以这是一门不断发展的学科
Lex Fridman (39:18.920)
but I find that the people that are really good
但我发现那些真正优秀的人
Lex Fridman (39:20.760)
at debugging machine learning algorithms
调试机器学习算法
Lex Fridman (39:22.840)
are easily 10x, maybe 100x faster at getting something to work.
工作起来的速度很容易提高 10 倍，甚至 100 倍。
Lex Fridman (39:28.120)
And the basic process of debugging is,
调试的基本过程是，
Lex Fridman (39:30.760)
so the bug in this case,
所以在这种情况下的错误，
Lex Fridman (39:32.600)
why isn't this thing learning, improving,
为什么这东西不学习、改进、
Lex Fridman (39:36.360)
sort of going into the questions of overfitting
有点像过度拟合的问题
Lex Fridman (39:39.240)
and all those kinds of things?
以及所有这些事情？
Lex Fridman (39:40.760)
That's the logical space that the debugging is happening in
这就是调试发生的逻辑空间
Andrew Ng (39:45.240)
with neural networks.
与神经网络。
Lex Fridman (39:46.440)
Yeah, often the question is, why doesn't it work yet?
是的，经常有人问，为什么它还不起作用？
Lex Fridman (39:50.280)
Or can I expect it to eventually work?
或者我可以期望它最终起作用吗？
Lex Fridman (39:52.920)
And what are the things I could try?
我可以尝试哪些事情？
Andrew Ng (39:54.760)
Change the architecture, more data, more regularization,
改变架构，更多数据，更多正则化，
Lex Fridman (39:57.400)
different optimization algorithm,
不同的优化算法，
Andrew Ng (40:00.600)
different types of data.
不同类型的数据。
Lex Fridman (40:01.880)
So to answer those questions systematically,
因此，为了系统地回答这些问题，
Lex Fridman (40:04.200)
so that you don't spend six months hitting down the blind alley
这样你就不会花六个月的时间走进死胡同
Lex Fridman (40:08.040)
before someone comes and says,
Lex Fridman (40:09.720)
why did you spend six months doing this?
Lex Fridman (40:12.120)
What concepts in deep learning
Lex Fridman (40:13.960)
do you think students struggle the most with?
Lex Fridman (40:16.440)
Or sort of is the biggest challenge for them
Andrew Ng (40:19.000)
was to get over that hill.
Lex Fridman (40:23.160)
It hooks them and it inspires them and they really get it.
Andrew Ng (40:28.040)
Similar to learning mathematics,
Lex Fridman (40:30.200)
I think one of the challenges of deep learning
Andrew Ng (40:32.440)
is that there are a lot of concepts
Lex Fridman (40:33.960)
that build on top of each other.
Andrew Ng (40:36.760)
If you ask me what's hard about mathematics,
Lex Fridman (40:38.760)
I have a hard time pinpointing one thing.
Lex Fridman (40:40.920)
Is it addition, subtraction?
Lex Fridman (40:42.280)
Is it a carry?
Lex Fridman (40:43.080)
Is it multiplication?
Lex Fridman (40:44.360)
There's just a lot of stuff.
Andrew Ng (40:45.720)
I think one of the challenges of learning math
Lex Fridman (40:48.040)
and of learning certain technical fields
Andrew Ng (40:49.800)
is that there are a lot of concepts
Lex Fridman (40:51.480)
and if you miss a concept,
Andrew Ng (40:53.080)
then you're kind of missing the prerequisite
Lex Fridman (40:55.400)
for something that comes later.
Lex Fridman (40:58.040)
So in the deep learning specialization,
Lex Fridman (41:01.880)
try to break down the concepts
Andrew Ng (41:03.480)
to maximize the odds of each component being understandable.
Lex Fridman (41:06.920)
So when you move on to the more advanced thing,
Andrew Ng (41:09.240)
we learn confidence,
Lex Fridman (41:10.760)
hopefully you have enough intuitions
Andrew Ng (41:12.280)
from the earlier sections
Lex Fridman (41:13.880)
to then understand why we structure confidence
Andrew Ng (41:16.760)
in a certain way
Lex Fridman (41:18.520)
and then eventually why we built RNNs and LSTMs
Andrew Ng (41:23.000)
or attention models in a certain way
Lex Fridman (41:24.760)
building on top of the earlier concepts.
Andrew Ng (41:27.560)
Actually, I'm curious,
Lex Fridman (41:28.600)
you do a lot of teaching as well.
Lex Fridman (41:30.920)
Do you have a favorite,
Lex Fridman (41:33.080)
this is the hard concept moment in your teaching?
Andrew Ng (41:39.480)
Well, I don't think anyone's ever turned the interview on me.
Lex Fridman (41:43.320)
I'm glad you get first.
Andrew Ng (41:46.600)
I think that's a really good question.
Lex Fridman (41:48.920)
Yeah, it's really hard to capture the moment
Andrew Ng (41:51.160)
when they struggle.
Lex Fridman (41:51.800)
I think you put it really eloquently.
Andrew Ng (41:53.320)
I do think there's moments
Lex Fridman (41:55.080)
that are like aha moments
Andrew Ng (41:57.240)
that really inspire people.
Lex Fridman (41:59.400)
I think for some reason,
Andrew Ng (42:01.400)
reinforcement learning,
Lex Fridman (42:03.240)
especially deep reinforcement learning
Andrew Ng (42:05.560)
is a really great way
Lex Fridman (42:07.400)
to really inspire people
Lex Fridman (42:09.560)
and get what the use of neural networks can do.
Lex Fridman (42:13.480)
Even though neural networks
Andrew Ng (42:15.160)
really are just a part of the deep RL framework,
Lex Fridman (42:18.440)
but it's a really nice way
Andrew Ng (42:19.640)
to paint the entirety of the picture
Lex Fridman (42:22.360)
of a neural network
Andrew Ng (42:23.960)
being able to learn from scratch,
Lex Fridman (42:25.880)
knowing nothing and explore the world
Lex Fridman (42:27.720)
and pick up lessons.
Lex Fridman (42:29.080)
I find that a lot of the aha moments
Andrew Ng (42:31.240)
happen when you use deep RL
Lex Fridman (42:33.640)
to teach people about neural networks,
Andrew Ng (42:36.200)
which is counterintuitive.
Lex Fridman (42:37.720)
I find like a lot of the inspired sort of fire
Andrew Ng (42:40.680)
in people's passion,
Lex Fridman (42:41.560)
people's eyes,
Andrew Ng (42:42.200)
it comes from the RL world.
Lex Fridman (42:44.680)
Do you find reinforcement learning
Andrew Ng (42:46.920)
to be a useful part
Lex Fridman (42:48.520)
of the teaching process or no?
Andrew Ng (42:51.800)
I still teach reinforcement learning
Lex Fridman (42:53.400)
in one of my Stanford classes
Lex Fridman (42:55.480)
and my PhD thesis was on reinforcement learning.
Lex Fridman (42:57.320)
So I clearly loved a few.
Andrew Ng (42:59.240)
I find that if I'm trying to teach
Lex Fridman (43:00.840)
students the most useful techniques
Andrew Ng (43:03.000)
for them to use today,
Lex Fridman (43:04.520)
I end up shrinking the amount of time
Andrew Ng (43:07.000)
I talk about reinforcement learning.
Lex Fridman (43:08.840)
It's not what's working today.
Andrew Ng (43:10.760)
Now, our world changes so fast.
Lex Fridman (43:12.280)
Maybe this will be totally different
Andrew Ng (43:13.480)
in a couple of years.
Lex Fridman (43:15.800)
But I think we need a couple more things
Andrew Ng (43:17.640)
for reinforcement learning to get there.
Lex Fridman (43:20.600)
One of my teams is looking
Andrew Ng (43:21.720)
to reinforcement learning
Lex Fridman (43:22.600)
for some robotic control tasks.
Lex Fridman (43:23.800)
So I see the applications,
Lex Fridman (43:25.160)
but if you look at it as a percentage
Andrew Ng (43:27.560)
of all of the impact
Lex Fridman (43:28.520)
of the types of things we do,
Andrew Ng (43:30.040)
it's at least today outside of
Lex Fridman (43:33.720)
playing video games, right?
Andrew Ng (43:35.320)
In a few of the games, the scope.
Lex Fridman (43:38.440)
Actually, at NeurIPS,
Andrew Ng (43:39.560)
a bunch of us were standing around
Lex Fridman (43:40.840)
saying, hey, what's your best example
Andrew Ng (43:42.760)
of an actual deploy reinforcement
Lex Fridman (43:44.200)
learning application?
Lex Fridman (43:45.240)
And among like
Lex Fridman (43:47.160)
senior machine learning researchers, right?
Lex Fridman (43:49.000)
And again, there are some emerging ones,
Lex Fridman (43:51.400)
but there are not that many great examples.
Andrew Ng (43:55.240)
I think you're absolutely right.
Lex Fridman (43:58.040)
The sad thing is there hasn't been
Andrew Ng (43:59.880)
a big impactful real world application
Lex Fridman (44:03.480)
of reinforcement learning.
Andrew Ng (44:04.840)
I think its biggest impact to me
Lex Fridman (44:07.560)
has been in the toy domain,
Andrew Ng (44:09.320)
in the game domain,
Lex Fridman (44:10.200)
in the small example.
Andrew Ng (44:11.240)
That's what I mean for educational purpose.
Lex Fridman (44:13.560)
It seems to be a fun thing to explore
Andrew Ng (44:15.640)
in your networks with.
Lex Fridman (44:16.760)
But I think from your perspective,
Lex Fridman (44:19.000)
and I think that might be
Lex Fridman (44:20.440)
the best perspective is
Andrew Ng (44:22.280)
if you're trying to educate
Lex Fridman (44:23.560)
with a simple example
Andrew Ng (44:24.680)
in order to illustrate
Lex Fridman (44:25.800)
how this can actually be grown
Andrew Ng (44:27.640)
to scale and have a real world impact,
Lex Fridman (44:31.560)
then perhaps focusing on the fundamentals
Andrew Ng (44:33.640)
of supervised learning
Lex Fridman (44:35.400)
in the context of a simple data set,
Andrew Ng (44:38.920)
even like an MNIST data set
Lex Fridman (44:40.440)
is the right way,
Andrew Ng (44:42.040)
is the right path to take.
Lex Fridman (44:45.080)
The amount of fun I've seen people
Andrew Ng (44:46.520)
have with reinforcement learning
Lex Fridman (44:47.880)
has been great,
Lex Fridman (44:48.440)
but not in the applied impact
Lex Fridman (44:51.320)
in the real world setting.
Lex Fridman (44:52.760)
So it's a trade off,
Lex Fridman (44:54.040)
how much impact you want to have
Andrew Ng (44:55.320)
versus how much fun you want to have.
Lex Fridman (44:56.680)
Yeah, that's really cool.
Lex Fridman (44:58.200)
And I feel like the world
Lex Fridman (44:59.960)
actually needs all sorts.
Andrew Ng (45:01.240)
Even within machine learning,
Lex Fridman (45:02.520)
I feel like deep learning
Andrew Ng (45:04.360)
is so exciting,
Lex Fridman (45:05.800)
but the AI team
Andrew Ng (45:07.080)
shouldn't just use deep learning.
Lex Fridman (45:08.360)
I find that my teams
Andrew Ng (45:09.320)
use a portfolio of tools.
Lex Fridman (45:11.640)
And maybe that's not the exciting thing
Andrew Ng (45:13.080)
to say, but some days
Lex Fridman (45:14.680)
we use a neural net,
Andrew Ng (45:15.720)
some days we use a PCA.
Lex Fridman (45:19.960)
Actually, the other day,
Andrew Ng (45:20.600)
I was sitting down with my team
Lex Fridman (45:21.480)
looking at PCA residuals,
Andrew Ng (45:22.760)
trying to figure out what's going on
Lex Fridman (45:23.800)
with PCA applied
Andrew Ng (45:24.600)
to manufacturing problem.
Lex Fridman (45:25.640)
And some days we use
Andrew Ng (45:26.920)
a probabilistic graphical model,
Lex Fridman (45:28.200)
some days we use a knowledge draft,
Andrew Ng (45:29.720)
which is one of the things
Lex Fridman (45:30.520)
that has tremendous industry impact.
Lex Fridman (45:33.000)
But the amount of chatter
Lex Fridman (45:34.680)
about knowledge drafts in academia
Andrew Ng (45:36.360)
is really thin compared
Lex Fridman (45:37.640)
to the actual real world impact.
Lex Fridman (45:39.640)
So I think reinforcement learning
Lex Fridman (45:41.400)
should be in that portfolio.
Lex Fridman (45:42.520)
And then it's about balancing
Lex Fridman (45:43.640)
how much we teach all of these things.
Lex Fridman (45:45.240)
And the world should have
Lex Fridman (45:47.000)
diverse skills.
Andrew Ng (45:47.800)
It'd be sad if everyone
Lex Fridman (45:49.240)
just learned one narrow thing.
Andrew Ng (45:51.400)
Yeah, the diverse skill
Lex Fridman (45:52.360)
help you discover the right tool
Andrew Ng (45:53.720)
for the job.
Lex Fridman (45:54.280)
What is the most beautiful,
Andrew Ng (45:56.680)
surprising or inspiring idea
Lex Fridman (45:59.160)
in deep learning to you?
Andrew Ng (46:00.760)
Something that captivated
Lex Fridman (46:03.400)
your imagination.
Andrew Ng (46:04.600)
Is it the scale that could be,
Lex Fridman (46:07.080)
the performance that could be
Lex Fridman (46:07.960)
achieved with scale?
Lex Fridman (46:08.920)
Or is there other ideas?
Andrew Ng (46:11.560)
I think that if my only job
Lex Fridman (46:14.360)
was being an academic researcher,
Andrew Ng (46:16.520)
if an unlimited budget
Lex Fridman (46:18.120)
and didn't have to worry
Andrew Ng (46:19.960)
about short term impact
Lex Fridman (46:21.800)
and only focus on long term impact,
Andrew Ng (46:23.800)
I'd probably spend all my time
Lex Fridman (46:24.760)
doing research on unsupervised learning.
Andrew Ng (46:27.400)
I still think unsupervised learning
Lex Fridman (46:28.840)
is a beautiful idea.
Andrew Ng (46:31.400)
At both this past NeurIPS and ICML,
Lex Fridman (46:34.600)
I was attending workshops
Andrew Ng (46:35.960)
or listening to various talks
Lex Fridman (46:37.480)
about self supervised learning,
Andrew Ng (46:39.160)
which is one vertical segment
Lex Fridman (46:41.480)
maybe of unsupervised learning
Andrew Ng (46:43.160)
that I'm excited about.
Lex Fridman (46:45.160)
Maybe just to summarize the idea,
Andrew Ng (46:46.360)
I guess you know the idea
Lex Fridman (46:47.400)
about describing fleet.
Andrew Ng (46:48.520)
No, please.
Lex Fridman (46:49.080)
So here's the example
Andrew Ng (46:49.960)
of self supervised learning.
Lex Fridman (46:52.040)
Let's say we grab a lot
Andrew Ng (46:53.480)
of unlabeled images off the internet.
Lex Fridman (46:55.560)
So with infinite amounts
Andrew Ng (46:56.680)
of this type of data,
Lex Fridman (46:58.040)
I'm going to take each image
Lex Fridman (46:59.320)
and rotate it by a random
Lex Fridman (47:01.160)
multiple of 90 degrees.
Lex Fridman (47:03.000)
And then I'm going to train
Lex Fridman (47:04.760)
a supervised neural network
Andrew Ng (47:06.200)
to predict what was
Lex Fridman (47:07.400)
the original orientation.
Lex Fridman (47:08.920)
So it has to be rotated 90 degrees,
Lex Fridman (47:10.760)
180 degrees, 270 degrees,
Andrew Ng (47:12.440)
or zero degrees.
Lex Fridman (47:14.360)
So you can generate
Andrew Ng (47:15.640)
an infinite amounts of labeled data
Lex Fridman (47:17.560)
because you rotated the image
Lex Fridman (47:18.920)
so you know what's the
Lex Fridman (47:19.880)
ground truth label.
Lex Fridman (47:20.760)
And so various researchers
Lex Fridman (47:23.320)
have found that by taking
Andrew Ng (47:24.680)
unlabeled data and making
Lex Fridman (47:26.600)
up labeled data sets
Lex Fridman (47:27.880)
and training a large neural network
Lex Fridman (47:29.720)
on these tasks,
Andrew Ng (47:30.920)
you can then take the hidden
Lex Fridman (47:32.040)
layer representation and transfer
Andrew Ng (47:34.120)
it to a different task
Lex Fridman (47:35.400)
very powerfully.
Andrew Ng (47:37.640)
Learning word embeddings
Lex Fridman (47:39.000)
where we take a sentence,
Andrew Ng (47:40.040)
delete a word,
Lex Fridman (47:40.760)
predict the missing word,
Andrew Ng (47:42.120)
which is how we learn.
Lex Fridman (47:43.480)
One of the ways we learn
Andrew Ng (47:44.440)
word embeddings
Lex Fridman (47:45.480)
is another example.
Lex Fridman (47:47.160)
And I think there's now
Lex Fridman (47:48.680)
this portfolio of techniques
Andrew Ng (47:50.440)
for generating these made up tasks.
Lex Fridman (47:53.320)
Another one called jigsaw
Andrew Ng (47:54.760)
would be if you take an image,
Lex Fridman (47:56.760)
cut it up into a three by three grid,
Lex Fridman (47:59.240)
so like a nine,
Lex Fridman (48:00.040)
three by three puzzle piece,
Andrew Ng (48:01.560)
jump up the nine pieces
Lex Fridman (48:02.840)
and have a neural network predict
Andrew Ng (48:04.520)
which of the nine factorial
Lex Fridman (48:06.360)
possible permutations
Andrew Ng (48:07.880)
it came from.
Lex Fridman (48:09.320)
So many groups,
Andrew Ng (48:11.480)
including OpenAI,
Lex Fridman (48:13.080)
Peter B has been doing
Andrew Ng (48:14.520)
some work on this too,
Lex Fridman (48:16.280)
Facebook, Google Brain,
Andrew Ng (48:18.440)
I think DeepMind,
Lex Fridman (48:19.560)
oh actually,
Andrew Ng (48:21.240)
Aaron van der Oort
Lex Fridman (48:22.200)
has great work on the CPC objective.
Lex Fridman (48:24.360)
So many teams are doing exciting work
Lex Fridman (48:26.120)
and I think this is a way
Andrew Ng (48:27.640)
to generate infinite label data
Lex Fridman (48:30.440)
and I find this a very exciting
Andrew Ng (48:32.920)
piece of unsupervised learning.
Lex Fridman (48:34.040)
So long term you think
Andrew Ng (48:35.080)
that's going to unlock
Lex Fridman (48:37.160)
a lot of power
Andrew Ng (48:38.280)
in machine learning systems
Lex Fridman (48:39.960)
is this kind of unsupervised learning.
Andrew Ng (48:42.200)
I don't think there's
Lex Fridman (48:43.080)
a whole enchilada,
Andrew Ng (48:43.880)
I think it's just a piece of it
Lex Fridman (48:45.080)
and I think this one piece
Andrew Ng (48:46.440)
unsupervised,
Lex Fridman (48:47.320)
self supervised learning
Andrew Ng (48:48.840)
is starting to get traction.
Lex Fridman (48:50.200)
We're very close
Andrew Ng (48:51.320)
to it being useful.
Lex Fridman (48:53.160)
Well, word embedding
Andrew Ng (48:54.040)
is really useful.
Lex Fridman (48:55.480)
I think we're getting
Andrew Ng (48:56.200)
closer and closer
Lex Fridman (48:57.080)
to just having a significant
Andrew Ng (48:59.240)
real world impact
Lex Fridman (49:00.440)
maybe in computer vision and video
Lex Fridman (49:03.080)
but I think this concept
Lex Fridman (49:05.000)
and I think there'll be
Andrew Ng (49:05.880)
other concepts around it.
Lex Fridman (49:07.000)
You know, other unsupervised
Andrew Ng (49:08.760)
learning things that I worked on
Lex Fridman (49:10.520)
I've been excited about.
Andrew Ng (49:12.040)
I was really excited
Lex Fridman (49:12.840)
about sparse coding
Lex Fridman (49:14.600)
and ICA,
Lex Fridman (49:16.040)
slow feature analysis.
Andrew Ng (49:17.480)
I think all of these are ideas
Lex Fridman (49:18.760)
that various of us
Andrew Ng (49:20.040)
were working on
Lex Fridman (49:20.680)
about a decade ago
Andrew Ng (49:21.720)
before we all got distracted
Lex Fridman (49:23.160)
by how well supervised
Andrew Ng (49:24.680)
learning was doing.
Lex Fridman (49:26.200)
So we would return
Andrew Ng (49:27.880)
we would return to the fundamentals
Lex Fridman (49:29.400)
of representation learning
Andrew Ng (49:30.760)
that really started
Lex Fridman (49:32.200)
this movement of deep learning.
Andrew Ng (49:33.720)
I think there's a lot more work
Lex Fridman (49:34.840)
that one could explore around
Andrew Ng (49:36.120)
this theme of ideas
Lex Fridman (49:37.080)
and other ideas
Andrew Ng (49:38.200)
to come up with better algorithms.
Lex Fridman (49:40.200)
So if we could return
Andrew Ng (49:42.040)
to maybe talk quickly
Lex Fridman (49:43.880)
about the specifics
Andrew Ng (49:45.080)
of deep learning.ai
Lex Fridman (49:46.600)
the deep learning specialization
Andrew Ng (49:48.120)
perhaps how long does it take
Lex Fridman (49:50.360)
to complete the course
Lex Fridman (49:51.240)
would you say?
Lex Fridman (49:52.680)
The official length
Andrew Ng (49:53.800)
of the deep learning specialization
Lex Fridman (49:55.320)
is I think 16 weeks
Lex Fridman (49:57.080)
so about four months
Lex Fridman (49:58.920)
but it's go at your own pace.
Lex Fridman (50:00.760)
So if you subscribe
Lex Fridman (50:01.960)
to the deep learning specialization
Andrew Ng (50:03.560)
there are people that finished it
Lex Fridman (50:04.760)
in less than a month
Andrew Ng (50:05.720)
by working more intensely
Lex Fridman (50:07.000)
and studying more intensely
Lex Fridman (50:07.960)
so it really depends on
Lex Fridman (50:09.240)
on the individual.
Andrew Ng (50:10.920)
When we created
Lex Fridman (50:11.480)
the deep learning specialization
Andrew Ng (50:13.480)
we wanted to make it
Lex Fridman (50:15.400)
very accessible
Lex Fridman (50:16.360)
and very affordable.
Lex Fridman (50:18.440)
And with you know
Andrew Ng (50:19.480)
Coursera and deep learning.ai
Lex Fridman (50:20.840)
education mission
Andrew Ng (50:21.720)
one of the things
Lex Fridman (50:22.120)
that's really important to me
Andrew Ng (50:23.480)
is that if there's someone
Lex Fridman (50:25.560)
for whom paying anything
Andrew Ng (50:27.160)
is a financial hardship
Lex Fridman (50:29.320)
then just apply for financial aid
Lex Fridman (50:30.920)
and get it for free.
Lex Fridman (50:34.280)
If you were to recommend
Andrew Ng (50:35.880)
a daily schedule for people
Lex Fridman (50:38.040)
in learning whether it's
Andrew Ng (50:39.240)
through the deep learning.ai
Lex Fridman (50:40.600)
specialization or just learning
Andrew Ng (50:42.680)
in the world of deep learning
Lex Fridman (50:43.960)
what would you recommend?
Lex Fridman (50:45.480)
How do they go about day to day
Lex Fridman (50:47.160)
sort of specific advice
Andrew Ng (50:48.760)
about learning
Lex Fridman (50:49.800)
about their journey in the world
Lex Fridman (50:51.720)
of deep learning machine learning?
Lex Fridman (50:53.400)
I think getting the habit of learning
Andrew Ng (50:56.760)
is key and that means regularity.
Lex Fridman (51:00.920)
So for example
Andrew Ng (51:02.840)
we send out a weekly newsletter
Lex Fridman (51:05.080)
the batch every Wednesday
Lex Fridman (51:06.680)
so people know it's coming Wednesday
Lex Fridman (51:08.200)
you can spend a little bit of time
Andrew Ng (51:09.160)
on Wednesday
Lex Fridman (51:10.200)
catching up on the latest news
Andrew Ng (51:11.560)
catching up on the latest news
Lex Fridman (51:13.640)
through the batch on Wednesday
Lex Fridman (51:17.400)
and for myself
Lex Fridman (51:18.600)
I've picked up a habit of spending
Andrew Ng (51:21.160)
some time every Saturday
Lex Fridman (51:22.520)
and every Sunday reading or studying
Lex Fridman (51:24.600)
and so I don't wake up on the Saturday
Lex Fridman (51:26.600)
and have to make a decision
Andrew Ng (51:27.640)
do I feel like reading
Lex Fridman (51:28.840)
or studying today or not
Andrew Ng (51:30.280)
it's just what I do
Lex Fridman (51:31.640)
and the fact is a habit
Andrew Ng (51:33.160)
makes it easier.
Lex Fridman (51:34.200)
So I think if someone can get into that habit
Andrew Ng (51:37.640)
it's like you know
Lex Fridman (51:38.760)
just like we brush our teeth every morning
Andrew Ng (51:41.080)
I don't think about it
Lex Fridman (51:42.040)
if I thought about it
Andrew Ng (51:42.760)
it's a little bit annoying
Lex Fridman (51:43.480)
to have to spend two minutes doing that
Lex Fridman (51:45.960)
but it's a habit that it takes
Lex Fridman (51:47.720)
no cognitive load
Lex Fridman (51:49.080)
but this would be so much harder
Lex Fridman (51:50.360)
if we have to make a decision every morning
Lex Fridman (51:53.640)
and actually that's the reason
Lex Fridman (51:54.680)
why I wear the same thing every day as well
Andrew Ng (51:56.040)
it's just one less decision
Lex Fridman (51:57.160)
I just get up and wear my blue shirt
Lex Fridman (51:59.560)
so but I think if you can get that habit
Lex Fridman (52:01.160)
that consistency of studying
Andrew Ng (52:02.840)
then it actually feels easier.
Lex Fridman (52:05.720)
So yeah it's kind of amazing
Andrew Ng (52:08.600)
in my own life
Lex Fridman (52:09.320)
like I play guitar every day for
Andrew Ng (52:12.840)
I force myself to at least for five minutes
Lex Fridman (52:14.920)
play guitar
Andrew Ng (52:15.560)
it's just it's a ridiculously short period of time
Lex Fridman (52:18.040)
but because I've gotten into that habit
Andrew Ng (52:20.120)
it's incredible what you can accomplish
Lex Fridman (52:21.720)
in a period of a year or two years
Andrew Ng (52:24.440)
you can become
Lex Fridman (52:26.280)
you know exceptionally good
Andrew Ng (52:28.280)
at certain aspects of a thing
Lex Fridman (52:29.720)
by just doing it every day
Andrew Ng (52:30.920)
for a very short period of time
Lex Fridman (52:32.040)
it's kind of a miracle
Andrew Ng (52:33.000)
that that's how it works
Lex Fridman (52:34.600)
it adds up over time.
Andrew Ng (52:36.200)
Yeah and I think this is often
Lex Fridman (52:38.360)
not about the bursts of sustained efforts
Lex Fridman (52:40.760)
and the all nighters
Lex Fridman (52:41.880)
because you could only do that
Andrew Ng (52:43.080)
a limited number of times
Lex Fridman (52:44.200)
it's the sustained effort over a long time
Andrew Ng (52:47.240)
I think you know reading two research papers
Lex Fridman (52:50.360)
is a nice thing to do
Lex Fridman (52:51.880)
but the power is not reading two research papers
Lex Fridman (52:54.200)
it's reading two research papers a week
Andrew Ng (52:56.760)
for a year
Lex Fridman (52:57.480)
then you read a hundred papers
Lex Fridman (52:58.920)
and you actually learn a lot
Lex Fridman (53:00.200)
when you read a hundred papers.
Lex Fridman (53:02.040)
So regularity and making learning a habit
Lex Fridman (53:05.720)
do you have general other study tips
Andrew Ng (53:09.720)
for particularly deep learning
Lex Fridman (53:11.880)
that people should
Andrew Ng (53:13.400)
in their process of learning
Lex Fridman (53:15.000)
is there some kind of recommendations
Lex Fridman (53:16.600)
or tips you have as they learn?
Lex Fridman (53:19.720)
One thing I still do
Andrew Ng (53:21.560)
when I'm trying to study something really deeply
Lex Fridman (53:23.320)
is take handwritten notes
Andrew Ng (53:25.800)
it varies
Lex Fridman (53:26.360)
I know there are a lot of people
Andrew Ng (53:27.640)
that take the deep learning courses
Lex Fridman (53:29.320)
during a commute or something
Andrew Ng (53:31.960)
where it may be more awkward to take notes
Lex Fridman (53:33.800)
so I know it may not work for everyone
Lex Fridman (53:36.680)
but when I'm taking courses on Coursera
Lex Fridman (53:39.640)
and I still take some every now and then
Andrew Ng (53:41.640)
the most recent one I took
Lex Fridman (53:42.520)
was a course on clinical trials
Andrew Ng (53:44.360)
because I was interested about that
Lex Fridman (53:45.640)
I got out my little Moleskine notebook
Lex Fridman (53:47.880)
and what I was seeing on my desk
Lex Fridman (53:48.840)
was just taking down notes
Lex Fridman (53:50.280)
so what the instructor was saying
Lex Fridman (53:51.480)
and that act we know that
Andrew Ng (53:53.000)
that act of taking notes
Lex Fridman (53:54.760)
preferably handwritten notes
Andrew Ng (53:57.240)
increases retention.
Lex Fridman (53:59.560)
So as you're sort of watching the video
Andrew Ng (54:01.720)
just kind of pausing maybe
Lex Fridman (54:03.800)
and then taking the basic insights down on paper.
Andrew Ng (54:07.800)
Yeah so there have been a few studies
Lex Fridman (54:09.960)
if you search online
Andrew Ng (54:11.080)
you find some of these studies
Lex Fridman (54:12.680)
that taking handwritten notes
Andrew Ng (54:15.080)
because handwriting is slower
Lex Fridman (54:16.920)
as we're saying just now
Andrew Ng (54:18.920)
it causes you to recode the knowledge
Lex Fridman (54:21.240)
in your own words more
Lex Fridman (54:23.080)
and that process of recoding
Lex Fridman (54:24.840)
promotes long term retention
Andrew Ng (54:26.600)
this is as opposed to typing
Lex Fridman (54:28.200)
which is fine
Andrew Ng (54:28.920)
again typing is better than nothing
Lex Fridman (54:30.680)
or in taking a class
Lex Fridman (54:31.800)
and not taking notes is better
Lex Fridman (54:32.760)
than not taking any class at all
Lex Fridman (54:34.360)
but comparing handwritten notes
Lex Fridman (54:36.440)
and typing
Andrew Ng (54:37.960)
you can usually type faster
Lex Fridman (54:39.480)
for a lot of people
Andrew Ng (54:40.280)
you can handwrite notes
Lex Fridman (54:41.480)
and so when people type
Andrew Ng (54:42.920)
they're more likely to just transcribe
Lex Fridman (54:44.920)
verbatim what they heard
Lex Fridman (54:46.280)
and that reduces the amount of recoding
Lex Fridman (54:49.080)
and that actually results
Andrew Ng (54:50.360)
in less long term retention.
Lex Fridman (54:52.360)
I don't know what the psychological effect
Andrew Ng (54:53.960)
there is but so true
Lex Fridman (54:55.320)
there's something fundamentally different
Andrew Ng (54:56.840)
about writing hand handwriting
Lex Fridman (54:59.400)
I wonder what that is
Andrew Ng (55:00.200)
I wonder if it is as simple
Lex Fridman (55:01.640)
as just the time it takes to write it slower
Andrew Ng (55:04.360)
yeah and because you can't write
Lex Fridman (55:07.400)
as many words
Andrew Ng (55:08.120)
you have to take whatever they said
Lex Fridman (55:10.200)
and summarize it into fewer words
Lex Fridman (55:11.960)
and that summarization process
Lex Fridman (55:13.400)
requires deeper processing of the meaning
Andrew Ng (55:15.880)
which then results in better retention
Lex Fridman (55:17.880)
that's fascinating
Andrew Ng (55:20.040)
oh and I think because of Coursera
Lex Fridman (55:22.440)
I spent so much time studying pedagogy
Andrew Ng (55:24.120)
this is actually one of my passions
Lex Fridman (55:25.400)
I really love learning
Lex Fridman (55:27.000)
how to more efficiently
Lex Fridman (55:28.040)
help others learn
Andrew Ng (55:28.920)
you know one of the things I do
Lex Fridman (55:30.600)
both when creating videos
Andrew Ng (55:32.280)
or when we write the batch is
Lex Fridman (55:34.760)
I try to think is one minute spent of us
Andrew Ng (55:37.800)
going to be a more efficient learning experience
Lex Fridman (55:40.600)
than one minute spent anywhere else
Lex Fridman (55:42.520)
and we really try to you know
Lex Fridman (55:45.080)
make it time efficient for the learners
Andrew Ng (55:46.920)
because you know everyone's busy
Lex Fridman (55:48.680)
so when when we're editing
Andrew Ng (55:50.280)
I often tell my teams
Lex Fridman (55:51.960)
every word needs to fight for its life
Lex Fridman (55:53.800)
and if you can delete a word
Lex Fridman (55:54.680)
let's just delete it and not wait
Andrew Ng (55:56.360)
let's not waste the learning time
Lex Fridman (55:57.880)
let's not waste the learning time
Andrew Ng (55:59.960)
oh that's so it's so amazing
Lex Fridman (56:01.400)
that you think that way
Andrew Ng (56:02.200)
because there is millions of people
Lex Fridman (56:03.560)
that are impacted by your teaching
Lex Fridman (56:04.840)
and sort of that one minute spent
Lex Fridman (56:06.680)
has a ripple effect right
Andrew Ng (56:08.360)
through years of time
Lex Fridman (56:09.560)
which is it's just fascinating to think about
Lex Fridman (56:12.600)
how does one make a career
Lex Fridman (56:14.280)
out of an interest in deep learning
Lex Fridman (56:15.960)
do you have advice for people
Lex Fridman (56:18.680)
we just talked about
Andrew Ng (56:19.480)
sort of the beginning early steps
Lex Fridman (56:21.400)
but if you want to make it
Andrew Ng (56:22.600)
an entire life's journey
Lex Fridman (56:24.280)
or at least a journey of a decade or two
Lex Fridman (56:26.360)
how do you how do you do it
Lex Fridman (56:28.200)
so most important thing is to get started
Andrew Ng (56:30.120)
right and and I think in the early parts
Lex Fridman (56:34.280)
of a career coursework
Andrew Ng (56:35.800)
um like the deep learning specialization
Lex Fridman (56:38.040)
or it's a very efficient way
Andrew Ng (56:41.080)
to master this material
Lex Fridman (56:43.320)
so because you know instructors
Andrew Ng (56:46.600)
uh be it me or someone else
Lex Fridman (56:48.280)
or you know Lawrence Maroney
Andrew Ng (56:49.640)
teaches our TensorFlow specialization
Lex Fridman (56:51.240)
or other things we're working on
Andrew Ng (56:52.280)
spend effort to try to make it time efficient
Lex Fridman (56:55.640)
for you to learn a new concept
Lex Fridman (56:57.640)
so coursework is actually a very efficient way
Lex Fridman (57:00.600)
for people to learn concepts
Lex Fridman (57:02.280)
and the beginning parts of breaking
Lex Fridman (57:04.120)
into a new field
Andrew Ng (57:05.960)
in fact one thing I see at Stanford
Lex Fridman (57:08.520)
some of my PhD students want to jump
Andrew Ng (57:10.280)
in the research right away
Lex Fridman (57:11.400)
and I actually tend to say look
Andrew Ng (57:13.160)
in your first couple years of PhD
Lex Fridman (57:14.440)
and spend time taking courses
Andrew Ng (57:16.680)
because it lays a foundation
Lex Fridman (57:17.960)
it's fine if you're less productive
Andrew Ng (57:19.640)
in your first couple years
Lex Fridman (57:20.680)
you'll be better off in the long term
Andrew Ng (57:23.400)
beyond a certain point
Lex Fridman (57:24.520)
there's materials that doesn't exist in courses
Andrew Ng (57:27.640)
because it's too cutting edge
Lex Fridman (57:28.840)
the course hasn't been created yet
Andrew Ng (57:30.040)
there's some practical experience
Lex Fridman (57:31.320)
that we're not yet that good
Andrew Ng (57:32.760)
as teaching in a course
Lex Fridman (57:34.440)
and I think after exhausting
Andrew Ng (57:36.040)
the efficient coursework
Lex Fridman (57:37.720)
then most people need to go on
Andrew Ng (57:40.360)
to either ideally work on projects
Lex Fridman (57:44.520)
and then maybe also continue their learning
Andrew Ng (57:47.080)
by reading blog posts and research papers
Lex Fridman (57:49.560)
and things like that
Andrew Ng (57:50.920)
doing projects is really important
Lex Fridman (57:52.280)
and again I think it's important
Andrew Ng (57:55.080)
to start small and just do something
Lex Fridman (57:57.560)
today you read about deep learning
Andrew Ng (57:58.920)
feels like oh all these people
Lex Fridman (57:59.800)
doing such exciting things
Lex Fridman (58:01.080)
what if I'm not building a neural network
Lex Fridman (58:02.920)
that changes the world
Lex Fridman (58:03.720)
then what's the point?
Lex Fridman (58:04.440)
Well the point is sometimes building
Andrew Ng (58:06.360)
that tiny neural network
Lex Fridman (58:07.720)
you know be it MNIST or upgrade
Andrew Ng (58:10.120)
to a fashion MNIST to whatever
Lex Fridman (58:12.280)
so doing your own fun hobby project
Andrew Ng (58:14.680)
that's how you gain the skills
Lex Fridman (58:15.960)
to let you do bigger and bigger projects
Andrew Ng (58:18.200)
I find this to be true at the individual level
Lex Fridman (58:20.520)
and also at the organizational level
Andrew Ng (58:23.080)
for a company to become good at machine learning
Lex Fridman (58:24.920)
sometimes the right thing to do
Andrew Ng (58:26.200)
is not to tackle the giant project
Lex Fridman (58:29.240)
is instead to do the small project
Andrew Ng (58:31.240)
that lets the organization learn
Lex Fridman (58:33.320)
and then build out from there
Lex Fridman (58:34.600)
but this is true both for individuals
Lex Fridman (58:35.960)
and for companies
Andrew Ng (58:38.200)
taking the first step
Lex Fridman (58:40.680)
and then taking small steps is the key
Andrew Ng (58:44.520)
should students pursue a PhD
Lex Fridman (58:46.280)
do you think you can do so much
Andrew Ng (58:48.520)
that's one of the fascinating things
Lex Fridman (58:50.200)
in machine learning
Andrew Ng (58:51.160)
you can have so much impact
Lex Fridman (58:52.280)
without ever getting a PhD
Lex Fridman (58:54.440)
so what are your thoughts
Lex Fridman (58:56.040)
should people go to grad school
Lex Fridman (58:57.400)
should people get a PhD?
Lex Fridman (58:59.400)
I think that there are multiple good options
Andrew Ng (59:01.720)
of which doing a PhD could be one of them
Lex Fridman (59:05.000)
I think that if someone's admitted
Andrew Ng (59:06.920)
to a top PhD program
Lex Fridman (59:08.520)
you know at MIT, Stanford, top schools
Andrew Ng (59:11.880)
I think that's a very good experience
Lex Fridman (59:15.320)
or if someone gets a job
Andrew Ng (59:17.000)
at a top organization
Lex Fridman (59:18.760)
at the top AI team
Andrew Ng (59:20.440)
I think that's also a very good experience
Lex Fridman (59:23.880)
there are some things you still need a PhD to do
Andrew Ng (59:25.880)
if someone's aspiration is to be a professor
Lex Fridman (59:27.640)
you know at the top academic university
Andrew Ng (59:29.080)
you just need a PhD to do that
Lex Fridman (59:30.920)
but if it goes to you know
Andrew Ng (59:32.520)
start a company, build a company
Lex Fridman (59:34.120)
do great technical work
Andrew Ng (59:35.320)
I think a PhD is a good experience
Lex Fridman (59:37.640)
but I would look at the different options
Andrew Ng (59:40.200)
available to someone
Lex Fridman (59:41.160)
you know where are the places
Andrew Ng (59:42.120)
where you can get a job
Lex Fridman (59:42.920)
where are the places to get a PhD program
Lex Fridman (59:44.920)
and kind of weigh the pros and cons of those
Lex Fridman (59:46.840)
So just to linger on that for a little bit longer
Lex Fridman (59:50.040)
what final dreams and goals
Lex Fridman (59:51.720)
do you think people should have
Lex Fridman (59:53.000)
so what options should they explore
Lex Fridman (59:57.320)
so you can work in industry
Lex Fridman (59:59.720)
so for a large company
🔗 相关节目