Jeremy Howard: fast.ai Deep Learning Courses and Research
AI 与机器学习技术与编程音乐与艺术心理与人性生物与进化
📋 章节目录
暂无章节信息
🔑 关键词
learningdatadondeepstuffdoingresearchmodelfastprogrammingswiftgooglelanguagedonecalleddomainusingbettercoursestarted
💬 精彩语录
暂无语录
🎙️ 完整对话（2286 条）
Lex Fridman (00:00.000)
The following is a conversation with Jeremy Howard.
以下是与杰里米·霍华德的对话。
Lex Fridman (00:03.120)
He's the founder of FastAI, a research institute dedicated
他是FastAI的创始人，FastAI是一家致力于
Lex Fridman (00:07.040)
to making deep learning more accessible.
使深度学习更容易实现。
Lex Fridman (00:09.720)
He's also a distinguished research scientist
他也是一位杰出的研究科学家
Lex Fridman (00:12.560)
at the University of San Francisco,
在旧金山大学，
Jeremy Howard (00:14.600)
a former president of Kaggle,
Kaggle 前总裁，
Lex Fridman (00:16.640)
as well as a top ranking competitor there.
以及那里的顶级竞争对手。
Lex Fridman (00:18.760)
And in general, he's a successful entrepreneur,
总的来说，他是一位成功的企业家，
Lex Fridman (00:21.680)
educator, researcher, and an inspiring personality
教育家、研究员和鼓舞人心的人物
Jeremy Howard (00:25.200)
in the AI community.
在人工智能社区。
Lex Fridman (00:27.000)
When someone asks me, how do I get started with deep learning?
当有人问我，如何开始深度学习？
Jeremy Howard (00:30.200)
FastAI is one of the top places that point them to.
FastAI 是他们最关注的地方之一。
Lex Fridman (00:33.320)
It's free, it's easy to get started,
它是免费的，很容易上手，
Jeremy Howard (00:35.480)
it's insightful and accessible,
它富有洞察力且易于理解，
Lex Fridman (00:37.600)
and if I may say so, it has very little BS
如果我可以这么说的话，它几乎没有废话
Jeremy Howard (00:40.960)
that can sometimes dilute the value of educational content
有时会削弱教育内容的价值
Lex Fridman (00:44.120)
on popular topics like deep learning.
深度学习等热门话题。
Jeremy Howard (00:46.720)
FastAI has a focus on practical application of deep learning
FastAI专注于深度学习的实际应用
Lex Fridman (00:50.280)
and hands on exploration of the cutting edge
并亲自探索前沿
Jeremy Howard (00:52.800)
that is incredibly both accessible to beginners
这对初学者来说非常容易上手
Lex Fridman (00:56.000)
and useful to experts.
Jeremy Howard (00:57.960)
This is the Artificial Intelligence Podcast.
Lex Fridman (01:01.320)
If you enjoy it, subscribe on YouTube,
Jeremy Howard (01:03.800)
give it five stars on iTunes,
Lex Fridman (01:05.480)
support it on Patreon,
Jeremy Howard (01:06.920)
or simply connect with me on Twitter
Lex Fridman (01:09.040)
at Lex Friedman, spelled F R I D M A N.
Lex Fridman (01:13.280)
And now, here's my conversation with Jeremy Howard.
Lex Fridman (01:18.520)
What's the first program you ever written?
Jeremy Howard (01:21.680)
First program I wrote that I remember
Lex Fridman (01:24.760)
would be at high school.
Jeremy Howard (01:29.200)
I did an assignment where I decided
Lex Fridman (01:31.200)
to try to find out if there were some better musical scales
Jeremy Howard (01:36.200)
than the normal 12 tone, 12 interval scale.
Lex Fridman (01:40.600)
So I wrote a program on my Commodore 64 in basic
Jeremy Howard (01:43.640)
that searched through other scale sizes
Lex Fridman (01:46.000)
to see if it could find one
Jeremy Howard (01:47.240)
where there were more accurate harmonies.
Lex Fridman (01:51.880)
Like mid tone?
Jeremy Howard (01:53.520)
Like you want an actual exactly three to two ratio
Lex Fridman (01:56.520)
or else with a 12 interval scale,
Jeremy Howard (01:59.400)
it's not exactly three to two, for example.
Lex Fridman (02:01.480)
So that's well tempered as they say in there.
Lex Fridman (02:05.040)
And basic on a Commodore 64.
Lex Fridman (02:07.680)
Where was the interest in music from?
Lex Fridman (02:09.440)
Or is it just technical?
Lex Fridman (02:10.440)
I did music all my life.
Lex Fridman (02:12.000)
So I played saxophone and clarinet and piano
Lex Fridman (02:15.360)
and guitar and drums and whatever.
Lex Fridman (02:18.120)
How does that thread go through your life?
Lex Fridman (02:22.120)
Where's music today?
Jeremy Howard (02:24.200)
It's not where I wish it was.
Lex Fridman (02:28.360)
For various reasons, couldn't really keep it going,
Jeremy Howard (02:30.200)
particularly because I had a lot of problems
Lex Fridman (02:31.640)
with RSI with my fingers.
Lex Fridman (02:33.520)
And so I had to kind of like cut back anything
Lex Fridman (02:35.560)
that used hands and fingers.
Jeremy Howard (02:39.400)
I hope one day I'll be able to get back to it health wise.
Lex Fridman (02:43.920)
So there's a love for music underlying it all.
Jeremy Howard (02:46.400)
Yeah.
Lex Fridman (02:47.840)
What's your favorite instrument?
Jeremy Howard (02:49.520)
Saxophone.
Lex Fridman (02:50.360)
Sax.
Jeremy Howard (02:51.200)
Or baritone saxophone.
Lex Fridman (02:52.880)
Well, probably bass saxophone, but they're awkward.
Jeremy Howard (02:57.480)
Well, I always love it when music
Lex Fridman (03:00.040)
is coupled with programming.
Jeremy Howard (03:01.720)
There's something about a brain that utilizes those
Lex Fridman (03:04.680)
that emerges with creative ideas.
Lex Fridman (03:07.560)
So you've used and studied quite a few programming languages.
Lex Fridman (03:11.240)
Can you give an overview of what you've used?
Lex Fridman (03:15.160)
What are the pros and cons of each?
Lex Fridman (03:17.880)
Well, my favorite programming environment,
Jeremy Howard (03:20.080)
well, most certainly was Microsoft Access
Lex Fridman (03:24.560)
back in like the earliest days.
Lex Fridman (03:26.440)
So that was Visual Basic for applications,
Lex Fridman (03:28.880)
which is not a good programming language,
Lex Fridman (03:30.680)
but the programming environment was fantastic.
Lex Fridman (03:33.040)
It's like the ability to create, you know,
Jeremy Howard (03:38.000)
user interfaces and tie data and actions to them
Lex Fridman (03:41.200)
and create reports and all that
Jeremy Howard (03:43.680)
as I've never seen anything as good.
Lex Fridman (03:46.760)
There's things nowadays like Airtable,
Jeremy Howard (03:48.560)
which are like small subsets of that,
Lex Fridman (03:54.240)
which people love for good reason,
Lex Fridman (03:56.160)
but unfortunately, nobody's ever achieved
Lex Fridman (04:00.080)
anything like that.
Lex Fridman (04:01.080)
What is that?
Lex Fridman (04:01.920)
If you could pause on that for a second.
Lex Fridman (04:03.280)
Oh, Access?
Lex Fridman (04:04.120)
Is it a database?
Jeremy Howard (04:06.280)
It was a database program that Microsoft produced,
Lex Fridman (04:09.600)
part of Office, and they kind of withered, you know,
Lex Fridman (04:13.400)
but basically it lets you in a totally graphical way
Lex Fridman (04:16.240)
create tables and relationships and queries
Lex Fridman (04:18.440)
and tie them to forms and set up, you know,
Lex Fridman (04:21.800)
event handlers and calculations.
Lex Fridman (04:24.680)
And it was a very complete powerful system
Lex Fridman (04:28.120)
designed for not massive scalable things,
Lex Fridman (04:31.480)
but for like useful little applications that I loved.
Lex Fridman (04:36.360)
So what's the connection between Excel and Access?
Lex Fridman (04:40.240)
So very close.
Lex Fridman (04:42.120)
So Access kind of was the relational database equivalent,
Jeremy Howard (04:47.120)
if you like.
Lex Fridman (04:47.960)
So people still do a lot of that stuff
Jeremy Howard (04:50.600)
that should be in Access in Excel as they know it.
Lex Fridman (04:53.600)
Excel's great as well.
Jeremy Howard (04:54.840)
So, but it's just not as rich a programming model
Lex Fridman (04:59.680)
as VBA combined with a relational database.
Lex Fridman (05:04.080)
And so I've always loved relational databases,
Lex Fridman (05:06.800)
but today programming on top of relational database
Jeremy Howard (05:10.480)
is just a lot more of a headache.
Lex Fridman (05:13.040)
You know, you generally either need to kind of,
Jeremy Howard (05:15.680)
you know, you need something that connects,
Lex Fridman (05:18.240)
that runs some kind of database server
Jeremy Howard (05:19.920)
unless you use SQLite, which has its own issues.
Lex Fridman (05:25.000)
Then you kind of often,
Jeremy Howard (05:25.920)
if you want to get a nice programming model,
Lex Fridman (05:27.600)
you'll need to like create an, add an ORM on top.
Lex Fridman (05:30.400)
And then, I don't know,
Lex Fridman (05:31.960)
there's all these pieces to tie together
Lex Fridman (05:34.360)
and it's just a lot more awkward than it should be.
Lex Fridman (05:37.000)
There are people that are trying to make it easier.
Lex Fridman (05:39.200)
So in particular, I think of F sharp, you know, Don Syme,
Lex Fridman (05:42.440)
who, him and his team have done a great job
Jeremy Howard (05:45.760)
of making something like a database appear
Lex Fridman (05:50.520)
in the type system.
Lex Fridman (05:51.640)
So you actually get like tab completion for fields
Lex Fridman (05:54.200)
and tables and stuff like that.
Jeremy Howard (05:57.800)
Anyway, so that was kind of, anyway,
Lex Fridman (05:59.280)
so like that whole VBA office thing, I guess,
Jeremy Howard (06:01.880)
was a starting point, which I still miss.
Lex Fridman (06:04.640)
And I got into standard Visual Basic, which...
Jeremy Howard (06:07.800)
That's interesting, just to pause on that for a second.
Lex Fridman (06:09.880)
It's interesting that you're connecting programming languages
Jeremy Howard (06:13.520)
to the ease of management of data.
Lex Fridman (06:17.440)
Yeah.
Lex Fridman (06:18.280)
So in your use of programming languages,
Lex Fridman (06:20.600)
you always had a love and a connection with data.
Jeremy Howard (06:24.880)
I've always been interested in doing useful things
Lex Fridman (06:28.000)
for myself and for others,
Jeremy Howard (06:29.480)
which generally means getting some data
Lex Fridman (06:31.920)
and doing something with it and putting it out there again.
Lex Fridman (06:34.600)
So that's been my interest throughout.
Lex Fridman (06:38.440)
So I also did a lot of stuff with AppleScript
Jeremy Howard (06:41.600)
back in the early days.
Lex Fridman (06:43.880)
So it's kind of nice being able to get the computer
Lex Fridman (06:48.000)
and computers to talk to each other
Lex Fridman (06:50.160)
and to do things for you.
Lex Fridman (06:52.960)
And then I think that one,
Lex Fridman (06:54.640)
the programming language I most loved then
Jeremy Howard (06:58.240)
would have been Delphi, which was Object Pascal,
Lex Fridman (07:02.920)
created by Anders Heilsberg,
Jeremy Howard (07:04.880)
who previously did Turbo Pascal
Lex Fridman (07:07.480)
and then went on to create.NET
Lex Fridman (07:08.840)
and then went on to create TypeScript.
Lex Fridman (07:11.080)
Delphi was amazing because it was like a compiled,
Jeremy Howard (07:14.880)
fast language that was as easy to use as Visual Basic.
Lex Fridman (07:20.200)
Delphi, what is it similar to in more modern languages?
Jeremy Howard (07:27.520)
Visual Basic.
Lex Fridman (07:28.880)
Visual Basic.
Jeremy Howard (07:29.720)
Yeah, but a compiled, fast version.
Lex Fridman (07:32.320)
So I'm not sure there's anything quite like it anymore.
Jeremy Howard (07:37.080)
If you took like C Sharp or Java
Lex Fridman (07:40.680)
and got rid of the virtual machine
Lex Fridman (07:42.520)
and replaced it with something,
Lex Fridman (07:43.440)
you could compile a small type binary.
Jeremy Howard (07:46.560)
I feel like it's where Swift could get to
Lex Fridman (07:50.720)
with the new Swift UI
Lex Fridman (07:52.640)
and the cross platform development going on.
Lex Fridman (07:56.520)
Like that's one of my dreams
Jeremy Howard (07:59.360)
is that we'll hopefully get back to where Delphi was.
Lex Fridman (08:02.840)
There is actually a free Pascal project nowadays
Jeremy Howard (08:08.520)
called Lazarus,
Lex Fridman (08:09.360)
which is also attempting to kind of recreate Delphi.
Lex Fridman (08:13.400)
So they're making good progress.
Lex Fridman (08:16.080)
So, okay, Delphi,
Jeremy Howard (08:18.560)
that's one of your favorite programming languages.
Lex Fridman (08:20.960)
Well, it's programming environments.
Jeremy Howard (08:22.360)
Again, I'd say Pascal's not a nice language.
Lex Fridman (08:26.280)
If you wanted to know specifically
Jeremy Howard (08:27.880)
about what languages I like,
Lex Fridman (08:29.640)
I would definitely pick J as being an amazingly wonderful
Jeremy Howard (08:33.600)
language.
Lex Fridman (08:35.480)
What's J?
Lex Fridman (08:37.040)
J, are you aware of APL?
Lex Fridman (08:39.640)
I am not, except from doing a little research
Jeremy Howard (08:42.440)
on the work you've done.
Lex Fridman (08:44.080)
Okay, so not at all surprising you're not familiar with it
Jeremy Howard (08:48.000)
because it's not well known,
Lex Fridman (08:49.000)
but it's actually one of the main families
Jeremy Howard (08:54.880)
of programming languages going back to the late 50s,
Lex Fridman (08:57.080)
early 60s.
Lex Fridman (08:57.920)
So there was a couple of major directions.
Lex Fridman (09:01.640)
One was the kind of Lambda Calculus Alonzo Church direction,
Jeremy Howard (09:06.120)
which I guess kind of lisp and scheme and whatever,
Lex Fridman (09:09.960)
which has a history going back
Jeremy Howard (09:12.280)
to the early days of computing.
Lex Fridman (09:13.360)
The second was the kind of imperative slash OO,
Jeremy Howard (09:18.680)
algo similar going on to C, C++ and so forth.
Lex Fridman (09:23.160)
There was a third,
Jeremy Howard (09:24.000)
which are called array oriented languages,
Lex Fridman (09:26.920)
which started with a paper by a guy called Ken Iverson,
Jeremy Howard (09:31.480)
which was actually a math theory paper,
Lex Fridman (09:35.160)
not a programming paper.
Jeremy Howard (09:37.480)
It was called Notation as a Tool for Thought.
Lex Fridman (09:41.440)
And it was the development of a new way,
Jeremy Howard (09:43.480)
a new type of math notation.
Lex Fridman (09:45.280)
And the idea is that this math notation
Jeremy Howard (09:47.560)
was much more flexible, expressive,
Lex Fridman (09:51.320)
and also well defined than traditional math notation,
Jeremy Howard (09:55.280)
which is none of those things.
Lex Fridman (09:56.440)
Math notation is awful.
Lex Fridman (09:59.200)
And so he actually turned that into a programming language
Lex Fridman (10:02.280)
and cause this was the early 50s or the sorry, late 50s,
Jeremy Howard (10:05.640)
all the names were available.
Lex Fridman (10:06.760)
So he called his language a programming language or APL.
Jeremy Howard (10:10.560)
APL.
Lex Fridman (10:11.400)
So APL is a implementation of notation
Jeremy Howard (10:15.360)
as a tool for thought by which he means math notation.
Lex Fridman (10:18.320)
And Ken and his son went on to do many things,
Lex Fridman (10:22.880)
but eventually they actually produced a new language
Lex Fridman (10:26.600)
that was built on top of all the learnings of APL.
Lex Fridman (10:28.440)
And that was called J.
Lex Fridman (10:30.600)
And J is the most expressive, composable language
Jeremy Howard (10:39.360)
of beautifully designed language I've ever seen.
Lex Fridman (10:42.440)
Does it have object oriented components?
Lex Fridman (10:44.560)
Does it have that kind of thing?
Lex Fridman (10:45.560)
Not really, it's an array oriented language.
Jeremy Howard (10:47.720)
It's the third path.
Lex Fridman (10:51.440)
Are you saying array?
Jeremy Howard (10:52.800)
Array oriented, yeah.
Lex Fridman (10:53.960)
What does it mean to be array oriented?
Lex Fridman (10:55.520)
So array oriented means that you generally
Lex Fridman (10:57.520)
don't use any loops,
Lex Fridman (10:59.560)
but the whole thing is done with kind of
Lex Fridman (11:02.400)
a extreme version of broadcasting,
Jeremy Howard (11:06.360)
if you're familiar with that NumPy slash Python concept.
Lex Fridman (11:09.920)
So you do a lot with one line of code.
Jeremy Howard (11:14.640)
It looks a lot like math notation, highly compact.
Lex Fridman (11:19.640)
And the idea is that you can kind of,
Jeremy Howard (11:22.880)
because you can do so much with one line of code,
Lex Fridman (11:24.760)
a single screen of code is very unlikely to,
Jeremy Howard (11:27.760)
you very rarely need more than that
Lex Fridman (11:29.560)
to express your program.
Lex Fridman (11:31.120)
And so you can kind of keep it all in your head
Lex Fridman (11:33.320)
and you can kind of clearly communicate it.
Jeremy Howard (11:36.080)
It's interesting that APL created two main branches,
Lex Fridman (11:40.000)
K and J.
Jeremy Howard (11:41.640)
J is this kind of like open source,
Lex Fridman (11:44.560)
niche community of crazy enthusiasts like me.
Lex Fridman (11:49.440)
And then the other path, K, was fascinating.
Lex Fridman (11:52.160)
It's an astonishingly expensive programming language,
Jeremy Howard (11:56.640)
which many of the world's
Lex Fridman (11:58.520)
most ludicrously rich hedge funds use.
Lex Fridman (12:02.920)
So the entire K machine is so small
Lex Fridman (12:06.680)
it sits inside level three cache on your CPU.
Lex Fridman (12:09.360)
And it easily wins every benchmark I've ever seen
Lex Fridman (12:14.120)
in terms of data processing speed.
Lex Fridman (12:16.760)
But you don't come across it very much
Lex Fridman (12:17.920)
because it's like $100,000 per CPU to run it.
Jeremy Howard (12:22.760)
It's like this path of programming languages
Lex Fridman (12:26.280)
is just so much, I don't know,
Lex Fridman (12:28.920)
so much more powerful in every way
Lex Fridman (12:30.360)
than the ones that almost anybody uses every day.
Lex Fridman (12:33.920)
So it's all about computation.
Lex Fridman (12:37.520)
It's really focused on computation.
Jeremy Howard (12:38.360)
It's pretty heavily focused on computation.
Lex Fridman (12:40.640)
I mean, so much of programming
Jeremy Howard (12:43.200)
is data processing by definition.
Lex Fridman (12:45.640)
So there's a lot of things you can do with it.
Lex Fridman (12:48.960)
But yeah, there's not much work being done
Lex Fridman (12:51.440)
on making like user interface toolkits or whatever.
Jeremy Howard (12:57.000)
I mean, there's some, but they're not great.
Lex Fridman (12:59.320)
At the same time, you've done a lot of stuff
Jeremy Howard (13:00.880)
with Perl and Python.
Lex Fridman (13:03.120)
So where does that fit into the picture of J and K and APL?
Jeremy Howard (13:08.840)
Well, it's just much more pragmatic.
Lex Fridman (13:11.000)
Like in the end, you kind of have to end up
Lex Fridman (13:13.880)
where the libraries are, you know?
Lex Fridman (13:17.760)
Like, cause to me, my focus is on productivity.
Jeremy Howard (13:21.240)
I just want to get stuff done and solve problems.
Lex Fridman (13:23.680)
So Perl was great.
Jeremy Howard (13:27.280)
I created an email company called FastMail
Lex Fridman (13:29.680)
and Perl was great cause back in the late nineties,
Jeremy Howard (13:32.840)
early two thousands, it just had a lot of stuff it could do.
Lex Fridman (13:38.080)
I still had to write my own monitoring system
Lex Fridman (13:41.760)
and my own web framework, my own whatever,
Lex Fridman (13:43.800)
cause like none of that stuff existed.
Lex Fridman (13:45.720)
But it was a super flexible language to do that in.
Lex Fridman (13:50.280)
And you used Perl for FastMail, you used it as a backend?
Lex Fridman (13:54.240)
Like so everything was written in Perl?
Lex Fridman (13:55.760)
Yeah, yeah, everything, everything was Perl.
Lex Fridman (13:58.720)
Why do you think Perl hasn't succeeded
Lex Fridman (14:02.920)
or hasn't dominated the market where Python
Lex Fridman (14:05.960)
really takes over a lot of the tasks?
Lex Fridman (14:07.560)
Well, I mean, Perl did dominate.
Jeremy Howard (14:09.600)
It was everything, everywhere,
Lex Fridman (14:13.080)
but then the guy that ran Perl, Larry Wohl,
Jeremy Howard (14:17.120)
kind of just didn't put the time in anymore.
Lex Fridman (14:22.320)
And no project can be successful if there isn't,
Jeremy Howard (14:28.040)
you know, particularly one that started with a strong leader
Lex Fridman (14:31.600)
that loses that strong leadership.
Lex Fridman (14:35.040)
So then Python has kind of replaced it.
Lex Fridman (14:37.840)
You know, Python is a lot less elegant language
Jeremy Howard (14:43.400)
in nearly every way,
Lex Fridman (14:45.040)
but it has the data science libraries
Lex Fridman (14:48.880)
and a lot of them are pretty great.
Lex Fridman (14:51.280)
So I kind of use it
Jeremy Howard (14:56.240)
cause it's the best we have,
Lex Fridman (14:58.280)
but it's definitely not good enough.
Lex Fridman (15:01.800)
But what do you think the future of programming looks like?
Lex Fridman (15:04.080)
What do you hope the future of programming looks like
Jeremy Howard (15:06.560)
if we zoom in on the computational fields,
Lex Fridman (15:08.760)
on data science, on machine learning?
Jeremy Howard (15:11.840)
I hope Swift is successful
Lex Fridman (15:15.440)
because the goal of Swift,
Jeremy Howard (15:19.440)
the way Chris Latner describes it,
Lex Fridman (15:21.040)
is to be infinitely hackable.
Lex Fridman (15:22.640)
And that's what I want.
Lex Fridman (15:23.480)
I want something where me and the people I do research with
Lex Fridman (15:26.920)
and my students can look at
Lex Fridman (15:29.480)
and change everything from top to bottom.
Jeremy Howard (15:32.000)
There's nothing mysterious and magical and inaccessible.
Lex Fridman (15:36.240)
Unfortunately with Python, it's the opposite of that
Jeremy Howard (15:38.600)
because Python is so slow.
Lex Fridman (15:40.800)
It's extremely unhackable.
Jeremy Howard (15:42.640)
You get to a point where it's like,
Lex Fridman (15:43.840)
okay, from here on down at C.
Lex Fridman (15:45.360)
So your debugger doesn't work in the same way.
Lex Fridman (15:47.280)
Your profiler doesn't work in the same way.
Jeremy Howard (15:48.920)
Your build system doesn't work in the same way.
Lex Fridman (15:50.760)
It's really not very hackable at all.
Lex Fridman (15:53.760)
What's the part you like to be hackable?
Lex Fridman (15:55.600)
Is it for the objective of optimizing training
Lex Fridman (16:00.120)
of neural networks, inference of neural networks?
Lex Fridman (16:02.560)
Is it performance of the system
Lex Fridman (16:04.320)
or is there some non performance related, just?
Lex Fridman (16:07.840)
It's everything.
Jeremy Howard (16:09.000)
I mean, in the end, I want to be productive
Lex Fridman (16:11.280)
as a practitioner.
Lex Fridman (16:13.840)
So that means that, so like at the moment,
Lex Fridman (16:16.280)
our understanding of deep learning is incredibly primitive.
Jeremy Howard (16:20.000)
There's very little we understand.
Lex Fridman (16:21.440)
Most things don't work very well,
Jeremy Howard (16:23.200)
even though it works better than anything else out there.
Lex Fridman (16:26.120)
There's so many opportunities to make it better.
Lex Fridman (16:28.600)
So you look at any domain area,
Lex Fridman (16:31.280)
like, I don't know, speech recognition with deep learning
Jeremy Howard (16:35.720)
or natural language processing classification
Lex Fridman (16:38.360)
with deep learning or whatever.
Jeremy Howard (16:39.400)
Every time I look at an area with deep learning,
Lex Fridman (16:41.920)
I always see like, oh, it's terrible.
Jeremy Howard (16:44.440)
There's lots and lots of obviously stupid ways
Lex Fridman (16:47.480)
to do things that need to be fixed.
Lex Fridman (16:50.160)
So then I want to be able to jump in there
Lex Fridman (16:51.600)
and quickly experiment and make them better.
Lex Fridman (16:54.840)
You think the programming language has a role in that?
Lex Fridman (16:59.240)
Huge role, yeah.
Lex Fridman (17:00.240)
So currently, Python has a big gap
Lex Fridman (17:05.960)
in terms of our ability to innovate,
Jeremy Howard (17:09.240)
particularly around recurrent neural networks
Lex Fridman (17:11.800)
and natural language processing.
Jeremy Howard (17:14.880)
Because it's so slow, the actual loop
Lex Fridman (17:18.240)
where we actually loop through words,
Jeremy Howard (17:20.160)
we have to do that whole thing in CUDA C.
Lex Fridman (17:23.720)
So we actually can't innovate with the kernel,
Jeremy Howard (17:27.080)
the heart of that most important algorithm.
Lex Fridman (17:31.520)
And it's just a huge problem.
Lex Fridman (17:33.640)
And this happens all over the place.
Lex Fridman (17:36.440)
So we hit research limitations.
Jeremy Howard (17:40.040)
Another example, convolutional neural networks,
Lex Fridman (17:42.600)
which are actually the most popular architecture
Jeremy Howard (17:44.720)
for lots of things, maybe most things in deep learning.
Lex Fridman (17:48.880)
We almost certainly should be using
Jeremy Howard (17:50.280)
sparse convolutional neural networks,
Lex Fridman (17:52.880)
but only like two people are,
Jeremy Howard (17:55.360)
because to do it, you have to rewrite
Lex Fridman (17:57.800)
all of that CUDA C level stuff.
Lex Fridman (17:59.880)
And yeah, just researchers and practitioners don't.
Lex Fridman (18:04.480)
So there's just big gaps in what people actually research on,
Lex Fridman (18:09.200)
what people actually implement
Lex Fridman (18:10.520)
because of the programming language problem.
Lex Fridman (18:13.200)
So you think it's just too difficult to write in CUDA C
Lex Fridman (18:20.600)
that a higher level programming language like Swift
Jeremy Howard (18:24.480)
should enable the easier,
Lex Fridman (18:30.480)
fooling around creative stuff with RNNs
Lex Fridman (18:33.080)
or with sparse convolutional neural networks?
Lex Fridman (18:34.840)
Kind of.
Lex Fridman (18:35.680)
Who's at fault?
Lex Fridman (18:37.680)
Who's at charge of making it easy
Lex Fridman (18:41.000)
for a researcher to play around?
Lex Fridman (18:42.240)
I mean, no one's at fault,
Jeremy Howard (18:43.480)
just nobody's got around to it yet,
Lex Fridman (18:45.040)
or it's just, it's hard, right?
Lex Fridman (18:46.960)
And I mean, part of the fault is that we ignored
Lex Fridman (18:49.280)
that whole APL kind of direction.
Jeremy Howard (18:53.000)
Nearly everybody did for 60 years, 50 years.
Lex Fridman (18:57.440)
But recently people have been starting to
Jeremy Howard (19:01.520)
reinvent pieces of that
Lex Fridman (19:03.480)
and kind of create some interesting new directions
Jeremy Howard (19:05.400)
in the compiler technology.
Lex Fridman (19:07.240)
So the place where that's particularly happening right now
Jeremy Howard (19:11.680)
is something called MLIR,
Lex Fridman (19:13.440)
which is something that, again,
Jeremy Howard (19:14.840)
Chris Latina, the Swift guy, is leading.
Lex Fridman (19:18.000)
And yeah, because it's actually not gonna be Swift
Jeremy Howard (19:20.560)
on its own that solves this problem,
Lex Fridman (19:22.080)
because the problem is that currently writing
Jeremy Howard (19:24.920)
a acceptably fast, you know, GPU program
Lex Fridman (19:30.960)
is too complicated regardless of what language you use.
Jeremy Howard (19:33.720)
Right.
Lex Fridman (19:36.440)
And that's just because if you have to deal with the fact
Jeremy Howard (19:38.640)
that I've got, you know, 10,000 threads
Lex Fridman (19:41.680)
and I have to synchronize between them all
Lex Fridman (19:43.440)
and I have to put my thing into grid blocks
Lex Fridman (19:45.320)
and think about warps and all this stuff,
Jeremy Howard (19:47.000)
it's just so much boilerplate that to do that well,
Lex Fridman (19:50.680)
you have to be a specialist at that
Lex Fridman (19:52.200)
and it's gonna be a year's work to, you know,
Lex Fridman (19:56.440)
optimize that algorithm in that way.
Lex Fridman (19:59.640)
But with things like tensor comprehensions
Lex Fridman (1:00:02.440)
I just saw a subatomic particle do something
Jeremy Howard (1:00:05.360)
which the theory doesn't explain,
Lex Fridman (1:00:07.240)
you could publish that without an explanation.
Lex Fridman (1:00:10.440)
And then in the next 60 years,
Lex Fridman (1:00:11.840)
people can try to work out how to explain it.
Jeremy Howard (1:00:14.080)
We don't allow this in the deep learning world.
Lex Fridman (1:00:16.120)
So it's literally impossible for Leslie
Jeremy Howard (1:00:19.520)
to publish a paper that says,
Lex Fridman (1:00:21.600)
I've just seen something amazing happen.
Jeremy Howard (1:00:23.520)
This thing trained 10 times faster than it should have.
Lex Fridman (1:00:25.640)
I don't know why.
Lex Fridman (1:00:27.360)
And so the reviewers were like,
Lex Fridman (1:00:28.520)
well, you can't publish that because you don't know why.
Lex Fridman (1:00:30.280)
So anyway.
Lex Fridman (1:00:31.120)
That's important to pause on
Jeremy Howard (1:00:32.160)
because there's so many discoveries
Lex Fridman (1:00:34.280)
that would need to start like that.
Jeremy Howard (1:00:36.120)
Every other scientific field I know of works that way.
Lex Fridman (1:00:39.240)
I don't know why ours is uniquely disinterested
Jeremy Howard (1:00:43.520)
in publishing unexplained experimental results,
Lex Fridman (1:00:47.720)
but there it is.
Lex Fridman (1:00:48.680)
So it wasn't published.
Lex Fridman (1:00:51.200)
Having said that,
Jeremy Howard (1:00:52.560)
I read a lot more unpublished papers than published papers
Lex Fridman (1:00:56.840)
because that's where you find the interesting insights.
Lex Fridman (1:01:00.040)
So I absolutely read this paper.
Lex Fridman (1:01:02.680)
And I was just like,
Jeremy Howard (1:01:04.520)
this is astonishingly mind blowing and weird
Lex Fridman (1:01:08.920)
and awesome.
Lex Fridman (1:01:09.760)
And like, why isn't everybody only talking about this?
Lex Fridman (1:01:12.400)
Because like, if you can train these things 10 times faster,
Jeremy Howard (1:01:15.480)
they also generalize better
Lex Fridman (1:01:16.720)
because you're doing less epochs,
Jeremy Howard (1:01:18.800)
which means you look at the data less,
Lex Fridman (1:01:20.080)
you get better accuracy.
Lex Fridman (1:01:22.360)
So I've been kind of studying that ever since.
Lex Fridman (1:01:24.640)
And eventually Leslie kind of figured out
Jeremy Howard (1:01:28.520)
a lot of how to get this done.
Lex Fridman (1:01:30.120)
And we added minor tweaks.
Lex Fridman (1:01:32.240)
And a big part of the trick
Lex Fridman (1:01:33.600)
is starting at a very low learning rate,
Jeremy Howard (1:01:36.440)
very gradually increasing it.
Lex Fridman (1:01:37.880)
So as you're training your model,
Jeremy Howard (1:01:39.800)
you would take very small steps at the start
Lex Fridman (1:01:42.120)
and you gradually make them bigger and bigger
Jeremy Howard (1:01:44.040)
until eventually you're taking much bigger steps
Lex Fridman (1:01:46.400)
than anybody thought was possible.
Jeremy Howard (1:01:49.400)
There's a few other little tricks to make it work,
Lex Fridman (1:01:51.120)
but basically we can reliably get super convergence.
Lex Fridman (1:01:55.240)
And so for the Dawn Bench thing,
Lex Fridman (1:01:56.600)
we were using just much higher learning rates
Jeremy Howard (1:01:59.280)
than people expected to work.
Lex Fridman (1:02:02.200)
What do you think the future of,
Jeremy Howard (1:02:03.840)
I mean, it makes so much sense
Lex Fridman (1:02:04.880)
for that to be a critical hyperparameter learning rate
Jeremy Howard (1:02:07.600)
that you vary.
Lex Fridman (1:02:08.640)
What do you think the future
Lex Fridman (1:02:09.520)
of learning rate magic looks like?
Lex Fridman (1:02:13.480)
Well, there's been a lot of great work
Jeremy Howard (1:02:14.920)
in the last 12 months in this area.
Lex Fridman (1:02:17.400)
And people are increasingly realizing that optimize,
Jeremy Howard (1:02:20.160)
like we just have no idea really how optimizers work.
Lex Fridman (1:02:23.120)
And the combination of weight decay,
Jeremy Howard (1:02:25.840)
which is how we regularize optimizers,
Lex Fridman (1:02:27.480)
and the learning rate,
Lex Fridman (1:02:29.200)
and then other things like the epsilon we use
Lex Fridman (1:02:31.520)
in the Adam optimizer,
Jeremy Howard (1:02:32.760)
they all work together in weird ways.
Lex Fridman (1:02:36.560)
And different parts of the model,
Jeremy Howard (1:02:38.560)
this is another thing we've done a lot of work on
Lex Fridman (1:02:40.480)
is research into how different parts of the model
Jeremy Howard (1:02:43.480)
should be trained at different rates in different ways.
Lex Fridman (1:02:46.640)
So we do something we call discriminative learning rates,
Jeremy Howard (1:02:49.040)
which is really important,
Lex Fridman (1:02:50.160)
particularly for transfer learning.
Lex Fridman (1:02:53.240)
So really, I think in the last 12 months,
Lex Fridman (1:02:54.920)
a lot of people have realized
Jeremy Howard (1:02:55.880)
that all this stuff is important.
Lex Fridman (1:02:57.400)
There's been a lot of great work coming out
Lex Fridman (1:03:00.000)
and we're starting to see algorithms appear,
Lex Fridman (1:03:03.680)
which have very, very few dials, if any,
Jeremy Howard (1:03:06.920)
that you have to touch.
Lex Fridman (1:03:07.960)
So I think what's gonna happen
Jeremy Howard (1:03:09.280)
is the idea of a learning rate,
Lex Fridman (1:03:10.440)
well, it almost already has disappeared
Jeremy Howard (1:03:12.840)
in the latest research.
Lex Fridman (1:03:14.360)
And instead, it's just like we know enough
Jeremy Howard (1:03:18.240)
about how to interpret the gradients
Lex Fridman (1:03:22.600)
and the change of gradients we see
Jeremy Howard (1:03:23.840)
to know how to set every parameter
Lex Fridman (1:03:25.320)
in an optimal way.
Lex Fridman (1:03:26.160)
So you see the future of deep learning
Lex Fridman (1:03:30.840)
where really, where's the input of a human expert needed?
Jeremy Howard (1:03:34.560)
Well, hopefully the input of a human expert
Lex Fridman (1:03:36.520)
will be almost entirely unneeded
Jeremy Howard (1:03:38.760)
from the deep learning point of view.
Lex Fridman (1:03:40.440)
So again, like Google's approach to this
Jeremy Howard (1:03:43.480)
is to try and use thousands of times more compute
Lex Fridman (1:03:46.000)
to run lots and lots of models at the same time
Lex Fridman (1:03:49.400)
and hope that one of them is good.
Lex Fridman (1:03:51.080)
AutoML kind of thing?
Jeremy Howard (1:03:51.920)
Yeah, AutoML kind of stuff, which I think is insane.
Lex Fridman (1:03:56.720)
When you better understand the mechanics
Jeremy Howard (1:03:59.600)
of how models learn,
Lex Fridman (1:04:01.680)
you don't have to try a thousand different models
Jeremy Howard (1:04:03.800)
to find which one happens to work the best.
Lex Fridman (1:04:05.640)
You can just jump straight to the best one,
Jeremy Howard (1:04:08.120)
which means that it's more accessible
Lex Fridman (1:04:09.720)
in terms of compute, cheaper,
Lex Fridman (1:04:12.720)
and also with less hyperparameters to set,
Lex Fridman (1:04:14.920)
it means you don't need deep learning experts
Jeremy Howard (1:04:16.800)
to train your deep learning model for you,
Lex Fridman (1:04:19.320)
which means that domain experts can do more of the work,
Jeremy Howard (1:04:22.280)
which means that now you can focus the human time
Lex Fridman (1:04:24.960)
on the kind of interpretation, the data gathering,
Jeremy Howard (1:04:28.320)
identifying model errors and stuff like that.
Lex Fridman (1:04:31.360)
Yeah, the data side.
Lex Fridman (1:04:32.840)
How often do you work with data these days
Lex Fridman (1:04:34.720)
in terms of the cleaning, looking at it?
Jeremy Howard (1:04:37.800)
Like Darwin looked at different species
Lex Fridman (1:04:41.120)
while traveling about.
Lex Fridman (1:04:42.880)
Do you look at data?
Lex Fridman (1:04:45.000)
Have you in your roots in Kaggle?
Jeremy Howard (1:04:48.040)
Always, yeah.
Lex Fridman (1:04:48.880)
Look at data.
Jeremy Howard (1:04:49.720)
Yeah, I mean, it's a key part of our course.
Lex Fridman (1:04:51.320)
It's like before we train a model in the course,
Jeremy Howard (1:04:53.480)
we see how to look at the data.
Lex Fridman (1:04:55.200)
And then the first thing we do
Jeremy Howard (1:04:56.520)
after we train our first model,
Lex Fridman (1:04:57.920)
which we fine tune an ImageNet model for five minutes.
Lex Fridman (1:05:00.520)
And then the thing we immediately do after that
Lex Fridman (1:05:02.240)
is we learn how to analyze the results of the model
Jeremy Howard (1:05:05.800)
by looking at examples of misclassified images
Lex Fridman (1:05:08.920)
and looking at a classification matrix,
Lex Fridman (1:05:10.880)
and then doing research on Google
Lex Fridman (1:05:15.080)
to learn about the kinds of things that it's misclassifying.
Lex Fridman (1:05:18.120)
So to me, one of the really cool things
Lex Fridman (1:05:19.520)
about machine learning models in general
Jeremy Howard (1:05:21.840)
is that when you interpret them,
Lex Fridman (1:05:24.320)
they tell you about things like
Lex Fridman (1:05:25.400)
what are the most important features,
Lex Fridman (1:05:27.320)
which groups are you misclassifying,
Lex Fridman (1:05:29.360)
and they help you become a domain expert more quickly
Lex Fridman (1:05:32.440)
because you can focus your time on the bits
Jeremy Howard (1:05:34.840)
that the model is telling you is important.
Lex Fridman (1:05:38.680)
So it lets you deal with things like data leakage,
Jeremy Howard (1:05:40.720)
for example, if it says,
Lex Fridman (1:05:41.720)
oh, the main feature I'm looking at is customer ID.
Lex Fridman (1:05:45.640)
And you're like, oh, customer ID should be predictive.
Lex Fridman (1:05:47.600)
And then you can talk to the people
Jeremy Howard (1:05:50.640)
that manage customer IDs and they'll tell you like,
Lex Fridman (1:05:53.240)
oh yes, as soon as a customer's application is accepted,
Jeremy Howard (1:05:57.480)
we add a one on the end of their customer ID or something.
Lex Fridman (1:06:01.160)
So yeah, looking at data,
Jeremy Howard (1:06:03.720)
particularly from the lens of which parts of the data
Lex Fridman (1:06:06.000)
the model says is important is super important.
Jeremy Howard (1:06:09.360)
Yeah, and using the model to almost debug the data
Lex Fridman (1:06:12.920)
to learn more about the data.
Jeremy Howard (1:06:14.240)
Exactly.
Lex Fridman (1:06:16.800)
What are the different cloud options
Lex Fridman (1:06:18.600)
for training your own networks?
Lex Fridman (1:06:20.160)
Last question related to DawnBench.
Jeremy Howard (1:06:21.960)
Well, it's part of a lot of the work you do,
Lex Fridman (1:06:24.200)
but from a perspective of performance,
Jeremy Howard (1:06:27.240)
I think you've written this in a blog post.
Lex Fridman (1:06:29.440)
There's AWS, there's TPU from Google.
Lex Fridman (1:06:32.720)
What's your sense?
Lex Fridman (1:06:33.560)
What the future holds?
Lex Fridman (1:06:34.480)
What would you recommend now in terms of training?
Lex Fridman (1:06:37.360)
So from a hardware point of view,
Jeremy Howard (1:06:40.520)
Google's TPUs and the best Nvidia GPUs are similar.
Lex Fridman (1:06:45.320)
I mean, maybe the TPUs are like 30% faster,
Lex Fridman (1:06:47.920)
but they're also much harder to program.
Lex Fridman (1:06:49.920)
There isn't a clear leader in terms of hardware right now,
Jeremy Howard (1:06:54.640)
although much more importantly,
Lex Fridman (1:06:56.240)
the Nvidia GPUs are much more programmable.
Jeremy Howard (1:06:59.520)
They've got much more written for all of them.
Lex Fridman (1:07:00.920)
So like that's the clear leader for me
Lex Fridman (1:07:03.120)
and where I would spend my time
Lex Fridman (1:07:04.360)
as a researcher and practitioner.
Lex Fridman (1:07:08.560)
But then in terms of the platform,
Lex Fridman (1:07:12.160)
I mean, we're super lucky now with stuff like Google GCP,
Jeremy Howard (1:07:16.200)
Google Cloud, and AWS that you can access a GPU
Lex Fridman (1:07:21.480)
pretty quickly and easily.
Lex Fridman (1:07:25.400)
But I mean, for AWS, it's still too hard.
Lex Fridman (1:07:28.040)
Like you have to find an AMI and get the instance running
Lex Fridman (1:07:33.720)
and then install the software you want and blah, blah, blah.
Lex Fridman (1:07:37.040)
GCP is currently the best way to get started
Jeremy Howard (1:07:40.720)
on a full server environment
Lex Fridman (1:07:42.280)
because they have a fantastic fast AI in PyTorch ready
Jeremy Howard (1:07:46.360)
to go instance, which has all the courses preinstalled.
Lex Fridman (1:07:51.040)
It has Jupyter Notebook pre running.
Jeremy Howard (1:07:53.000)
Jupyter Notebook is this wonderful
Lex Fridman (1:07:55.880)
interactive computing system,
Jeremy Howard (1:07:57.560)
which everybody basically should be using
Lex Fridman (1:08:00.360)
for any kind of data driven research.
Lex Fridman (1:08:02.880)
But then even better than that,
Lex Fridman (1:08:05.600)
there are platforms like Salamander, which we own
Lex Fridman (1:08:09.480)
and Paperspace, where literally you click a single button
Lex Fridman (1:08:13.560)
and it pops up a Jupyter Notebook straight away
Jeremy Howard (1:08:17.200)
without any kind of installation or anything.
Lex Fridman (1:08:22.200)
And all the course notebooks are all preinstalled.
Lex Fridman (1:08:25.800)
So like for me, this is one of the things
Lex Fridman (1:08:28.560)
we spent a lot of time kind of curating and working on.
Jeremy Howard (1:08:34.200)
Because when we first started our courses,
Lex Fridman (1:08:35.960)
the biggest problem was people dropped out of lesson one
Jeremy Howard (1:08:39.600)
because they couldn't get an AWS instance running.
Lex Fridman (1:08:42.680)
So things are so much better now.
Lex Fridman (1:08:44.880)
And like we actually have, if you go to course.fast.ai,
Lex Fridman (1:08:47.800)
the first thing it says is here's how to get started
Jeremy Howard (1:08:49.680)
with your GPU.
Lex Fridman (1:08:50.520)
And there's like, you just click on the link
Lex Fridman (1:08:52.120)
and you click start and you're going.
Lex Fridman (1:08:55.360)
You'll go GCP.
Jeremy Howard (1:08:56.280)
I have to confess, I've never used the Google GCP.
Lex Fridman (1:08:58.800)
Yeah, GCP gives you $300 of compute for free,
Jeremy Howard (1:09:01.640)
which is really nice.
Lex Fridman (1:09:03.920)
But as I say, Salamander and Paperspace
Jeremy Howard (1:09:07.280)
are even easier still.
Lex Fridman (1:09:09.440)
Okay.
Lex Fridman (1:09:10.960)
So from the perspective of deep learning frameworks,
Lex Fridman (1:09:15.080)
you work with fast.ai, if you go to this framework,
Lex Fridman (1:09:18.440)
and PyTorch and TensorFlow.
Lex Fridman (1:09:21.240)
What are the strengths of each platform in your perspective?
Lex Fridman (1:09:25.800)
So in terms of what we've done our research on
Lex Fridman (1:09:28.760)
and taught in our course,
Jeremy Howard (1:09:30.240)
we started with Theano and Keras,
Lex Fridman (1:09:34.360)
and then we switched to TensorFlow and Keras,
Lex Fridman (1:09:38.080)
and then we switched to PyTorch,
Lex Fridman (1:09:40.360)
and then we switched to PyTorch and fast.ai.
Lex Fridman (1:09:42.960)
And that kind of reflects a growth and development
Lex Fridman (1:09:47.560)
of the ecosystem of deep learning libraries.
Jeremy Howard (1:09:52.560)
Theano and TensorFlow were great,
Lex Fridman (1:09:57.080)
but were much harder to teach and to do research
Lex Fridman (1:10:00.800)
and development on because they define
Lex Fridman (1:10:02.800)
what's called a computational graph upfront,
Jeremy Howard (1:10:05.080)
a static graph, where you basically have to say,
Lex Fridman (1:10:07.520)
here are all the things that I'm gonna eventually do
Jeremy Howard (1:10:10.880)
in my model, and then later on you say,
Lex Fridman (1:10:13.240)
okay, do those things with this data.
Lex Fridman (1:10:15.120)
And you can't like debug them,
Lex Fridman (1:10:17.160)
you can't do them step by step,
Jeremy Howard (1:10:18.560)
you can't program them interactively
Lex Fridman (1:10:20.160)
in a Jupyter notebook and so forth.
Jeremy Howard (1:10:22.320)
PyTorch was not the first,
Lex Fridman (1:10:23.760)
but PyTorch was certainly the strongest entrant
Jeremy Howard (1:10:26.880)
to come along and say, let's not do it that way,
Lex Fridman (1:10:28.720)
let's just use normal Python.
Lex Fridman (1:10:31.400)
And everything you know about in Python
Lex Fridman (1:10:32.920)
is just gonna work, and we'll figure out
Lex Fridman (1:10:35.280)
how to make that run on the GPU as and when necessary.
Lex Fridman (1:10:40.840)
That turned out to be a huge leap
Jeremy Howard (1:10:44.640)
in terms of what we could do with our research
Lex Fridman (1:10:46.840)
and what we could do with our teaching.
Jeremy Howard (1:10:49.760)
Because it wasn't limiting.
Lex Fridman (1:10:51.240)
Yeah, I mean, it was critical for us
Jeremy Howard (1:10:52.760)
for something like DawnBench
Lex Fridman (1:10:53.880)
to be able to rapidly try things.
Jeremy Howard (1:10:55.960)
It's just so much harder to be a researcher
Lex Fridman (1:10:57.840)
and practitioner when you have to do everything upfront
Lex Fridman (1:11:00.520)
and you can't inspect it.
Lex Fridman (1:11:03.400)
Problem with PyTorch is it's not at all accessible
Jeremy Howard (1:11:07.960)
to newcomers because you have to like
Lex Fridman (1:11:10.160)
write your own training loop and manage the gradients
Lex Fridman (1:11:12.920)
and all this stuff.
Lex Fridman (1:11:15.680)
And it's also like not great for researchers
Jeremy Howard (1:11:17.880)
because you're spending your time dealing
Lex Fridman (1:11:19.640)
with all this boilerplate and overhead
Jeremy Howard (1:11:21.640)
rather than thinking about your algorithm.
Lex Fridman (1:11:23.880)
So we ended up writing this very multi layered API
Jeremy Howard (1:11:27.760)
that at the top level, you can train
Lex Fridman (1:11:29.960)
a state of the art neural network
Jeremy Howard (1:11:31.400)
in three lines of code.
Lex Fridman (1:11:33.640)
And which kind of talks to an API,
Jeremy Howard (1:11:35.120)
which talks to an API, which talks to an API,
Lex Fridman (1:11:36.680)
which like you can dive into at any level
Lex Fridman (1:11:38.880)
and get progressively closer to the machine
Lex Fridman (1:11:42.720)
kind of levels of control.
Lex Fridman (1:11:45.360)
And this is the fast AI library.
Lex Fridman (1:11:47.480)
That's been critical for us and for our students
Lex Fridman (1:11:51.840)
and for lots of people that have won deep learning
Lex Fridman (1:11:54.200)
competitions with it and written academic papers with it.
Jeremy Howard (1:11:58.400)
It's made a big difference.
Lex Fridman (1:12:00.640)
We're still limited though by Python.
Lex Fridman (1:12:03.920)
And particularly this problem with things like
Lex Fridman (1:12:06.400)
recurrent neural nets say where you just can't change things
Jeremy Howard (1:12:11.400)
unless you accept it going so slowly that it's impractical.
Lex Fridman (1:12:15.640)
So in the latest incarnation of the course
Lex Fridman (1:12:18.320)
and with some of the research we're now starting to do,
Lex Fridman (1:12:20.880)
we're starting to do stuff, some stuff in Swift.
Jeremy Howard (1:12:24.520)
I think we're three years away from that
Lex Fridman (1:12:28.040)
being super practical, but I'm in no hurry.
Jeremy Howard (1:12:31.040)
I'm very happy to invest the time to get there.
Lex Fridman (1:12:35.520)
But with that, we actually already have a nascent version
Jeremy Howard (1:12:39.040)
of the fast AI library for vision running
Lex Fridman (1:12:42.520)
on Swift and TensorFlow.
Jeremy Howard (1:12:44.760)
Cause a Python for TensorFlow is not gonna cut it.
Lex Fridman (1:12:48.040)
It's just a disaster.
Lex Fridman (1:12:49.960)
What they did was they tried to replicate
Lex Fridman (1:12:53.960)
the bits that people were saying they like about PyTorch,
Jeremy Howard (1:12:57.120)
this kind of interactive computation,
Lex Fridman (1:12:59.200)
but they didn't actually change
Jeremy Howard (1:13:00.640)
their foundational runtime components.
Lex Fridman (1:13:03.920)
So they kind of added this like syntax sugar
Jeremy Howard (1:13:06.640)
they call TF Eager, TensorFlow Eager,
Lex Fridman (1:13:08.400)
which makes it look a lot like PyTorch,
Lex Fridman (1:13:10.920)
but it's 10 times slower than PyTorch
Lex Fridman (1:13:12.760)
to actually do a step.
Lex Fridman (1:13:16.400)
So because they didn't invest the time in like retooling
Lex Fridman (1:13:20.200)
the foundations, cause their code base is so horribly
Jeremy Howard (1:13:23.280)
complex.
Lex Fridman (1:13:24.120)
Yeah, I think it's probably very difficult
Jeremy Howard (1:13:25.280)
to do that kind of retooling.
Lex Fridman (1:13:26.440)
Yeah, well, particularly the way TensorFlow was written,
Jeremy Howard (1:13:28.640)
it was written by a lot of people very quickly
Lex Fridman (1:13:31.480)
in a very disorganized way.
Lex Fridman (1:13:33.320)
So like when you actually look in the code,
Lex Fridman (1:13:35.000)
as I do often, I'm always just like,
Lex Fridman (1:13:37.080)
Oh God, what were they thinking?
Lex Fridman (1:13:38.840)
It's just, it's pretty awful.
Lex Fridman (1:13:41.400)
So I'm really extremely negative
Lex Fridman (1:13:45.240)
about the potential future for Python for TensorFlow.
Lex Fridman (1:13:50.080)
But Swift for TensorFlow can be a different beast altogether.
Lex Fridman (1:13:53.760)
It can be like, it can basically be a layer on top of MLIR
Jeremy Howard (1:13:57.560)
that takes advantage of, you know,
Lex Fridman (1:14:00.440)
all the great compiler stuff that Swift builds on with LLVM
Lex Fridman (1:14:04.760)
and yeah, I think it will be absolutely fantastic.
Lex Fridman (1:14:10.280)
Well, you're inspiring me to try.
Jeremy Howard (1:14:11.880)
I haven't truly felt the pain of TensorFlow 2.0 Python.
Lex Fridman (1:14:17.640)
It's fine by me, but of...
Jeremy Howard (1:14:21.040)
Yeah, I mean, it does the job
Lex Fridman (1:14:22.120)
if you're using like predefined things
Jeremy Howard (1:14:25.120)
that somebody has already written.
Lex Fridman (1:14:27.720)
But if you actually compare, you know,
Jeremy Howard (1:14:29.560)
like I've had to do,
Lex Fridman (1:14:31.360)
cause I've been having to do a lot of stuff
Jeremy Howard (1:14:32.640)
with TensorFlow recently,
Lex Fridman (1:14:33.680)
you actually compare like,
Jeremy Howard (1:14:34.760)
okay, I want to write something from scratch
Lex Fridman (1:14:37.360)
and you're like, I just keep finding it's like,
Jeremy Howard (1:14:38.880)
Oh, it's running 10 times slower than PyTorch.
Lex Fridman (1:14:41.520)
So is the biggest cost,
Jeremy Howard (1:14:43.800)
let's throw running time out the window.
Lex Fridman (1:14:47.320)
How long it takes you to program?
Jeremy Howard (1:14:49.600)
That's not too different now,
Lex Fridman (1:14:50.960)
thanks to TensorFlow Eager, that's not too different.
Lex Fridman (1:14:54.040)
But because so many things take so long to run,
Lex Fridman (1:14:58.640)
you wouldn't run it at 10 times slower.
Jeremy Howard (1:15:00.280)
Like you just go like, Oh, this is taking too long.
Lex Fridman (1:15:03.240)
And also there's a lot of things
Jeremy Howard (1:15:04.240)
which are just less programmable,
Lex Fridman (1:15:05.840)
like tf.data, which is the way data processing works
Jeremy Howard (1:15:08.960)
in TensorFlow is just this big mess.
Lex Fridman (1:15:11.360)
It's incredibly inefficient.
Lex Fridman (1:15:13.200)
And they kind of had to write it that way
Lex Fridman (1:15:14.800)
because of the TPU problems I described earlier.
Lex Fridman (1:15:19.160)
So I just, you know,
Lex Fridman (1:15:22.160)
I just feel like they've got this huge technical debt,
Jeremy Howard (1:15:24.720)
which they're not going to solve
Lex Fridman (1:15:26.200)
without starting from scratch.
Lex Fridman (1:15:27.920)
So here's an interesting question then,
Lex Fridman (1:15:29.400)
if there's a new student starting today,
Lex Fridman (1:15:34.560)
what would you recommend they use?
Lex Fridman (1:15:37.480)
Well, I mean, we obviously recommend Fastai and PyTorch
Jeremy Howard (1:15:40.440)
because we teach new students and that's what we teach with.
Lex Fridman (1:15:43.880)
So we would very strongly recommend that
Jeremy Howard (1:15:46.080)
because it will let you get on top of the concepts
Lex Fridman (1:15:50.000)
much more quickly.
Lex Fridman (1:15:51.920)
So then you'll become an actual,
Lex Fridman (1:15:53.120)
and you'll also learn the actual state
Jeremy Howard (1:15:54.920)
of the art techniques, you know,
Lex Fridman (1:15:56.400)
so you actually get world class results.
Jeremy Howard (1:15:59.200)
Honestly, it doesn't much matter what library you learn
Lex Fridman (1:16:03.920)
because switching from the trainer to MXNet
Jeremy Howard (1:16:08.320)
to TensorFlow to PyTorch is gonna be a couple of days work
Lex Fridman (1:16:12.000)
as long as you understand the foundation as well.
Lex Fridman (1:16:15.240)
But you think will Swift creep in there
Lex Fridman (1:16:19.400)
as a thing that people start using?
Jeremy Howard (1:16:22.920)
Not for a few years,
Lex Fridman (1:16:24.360)
particularly because like Swift has no data science
Jeremy Howard (1:16:29.720)
community, libraries, schooling.
Lex Fridman (1:16:33.400)
And the Swift community has a total lack of appreciation
Lex Fridman (1:16:39.080)
and understanding of numeric computing.
Lex Fridman (1:16:40.880)
So like they keep on making stupid decisions, you know,
Jeremy Howard (1:16:43.600)
for years, they've just done dumb things
Lex Fridman (1:16:45.440)
around performance and prioritization.
Jeremy Howard (1:16:50.240)
That's clearly changing now
Lex Fridman (1:16:53.440)
because the developer of Swift, Chris Latner,
Jeremy Howard (1:16:58.000)
is working at Google on Swift for TensorFlow.
Lex Fridman (1:17:00.720)
So like that's a priority.
Jeremy Howard (1:17:04.120)
It'll be interesting to see what happens with Apple
Lex Fridman (1:17:05.800)
because like Apple hasn't shown any sign of caring
Jeremy Howard (1:17:10.760)
about numeric programming in Swift.
Lex Fridman (1:17:13.760)
So I mean, hopefully they'll get off their ass
Lex Fridman (1:17:17.360)
and start appreciating this
Lex Fridman (1:17:18.800)
because currently all of their low level libraries
Jeremy Howard (1:17:22.200)
are not written in Swift.
Lex Fridman (1:17:25.080)
They're not particularly Swifty at all,
Jeremy Howard (1:17:27.360)
stuff like CoreML, they're really pretty rubbish.
Lex Fridman (1:17:30.760)
So yeah, so there's a long way to go.
Lex Fridman (1:17:33.680)
But at least one nice thing is that Swift for TensorFlow
Lex Fridman (1:17:36.080)
can actually directly use Python code and Python libraries
Jeremy Howard (1:17:40.760)
in a literally the entire lesson one notebook of fast AI
Lex Fridman (1:17:45.040)
runs in Swift right now in Python mode.
Lex Fridman (1:17:48.560)
So that's a nice intermediate thing.
Lex Fridman (1:17:51.640)
How long does it take?
Jeremy Howard (1:17:53.320)
If you look at the two fast AI courses,
Lex Fridman (1:17:57.560)
how long does it take to get from point zero
Lex Fridman (1:18:00.440)
to completing both courses?
Lex Fridman (1:18:03.240)
It varies a lot.
Jeremy Howard (1:18:05.720)
Somewhere between two months and two years generally.
Lex Fridman (1:18:13.120)
So for two months, how many hours a day on average?
Lex Fridman (1:18:16.040)
So like somebody who is a very competent coder
Lex Fridman (1:18:20.480)
can do 70 hours per course and pick up 70.
Jeremy Howard (1:18:27.800)
70, seven zero, that's it, okay.
Lex Fridman (1:18:30.760)
But a lot of people I know take a year off
Jeremy Howard (1:18:35.640)
to study fast AI full time and say at the end of the year,
Lex Fridman (1:18:40.440)
they feel pretty competent
Jeremy Howard (1:18:43.440)
because generally there's a lot of other things you do
Lex Fridman (1:18:45.560)
like generally they'll be entering Kaggle competitions,
Jeremy Howard (1:18:48.680)
they might be reading Ian Goodfellow's book,
Lex Fridman (1:18:51.440)
they might, they'll be doing a bunch of stuff
Lex Fridman (1:18:54.560)
and often particularly if they are a domain expert,
Lex Fridman (1:18:57.760)
their coding skills might be a little
Jeremy Howard (1:19:00.560)
on the pedestrian side.
Lex Fridman (1:19:01.720)
So part of it's just like doing a lot more writing.
Lex Fridman (1:19:04.760)
What do you find is the bottleneck for people usually
Lex Fridman (1:19:07.960)
except getting started and setting stuff up?
Jeremy Howard (1:19:11.720)
I would say coding.
Lex Fridman (1:19:13.360)
Yeah, I would say the best,
Jeremy Howard (1:19:14.320)
the people who are strong coders pick it up the best.
Lex Fridman (1:19:18.800)
Although another bottleneck is people who have a lot
Jeremy Howard (1:19:21.640)
of experience of classic statistics can really struggle
Lex Fridman (1:19:27.440)
because the intuition is so the opposite
Jeremy Howard (1:19:30.000)
of what they're used to.
Lex Fridman (1:19:30.880)
They're very used to like trying to reduce the number
Jeremy Howard (1:19:33.040)
of parameters in their model
Lex Fridman (1:19:34.320)
and looking at individual coefficients and stuff like that.
Lex Fridman (1:19:39.400)
So I find people who have a lot of coding background
Lex Fridman (1:19:42.920)
and know nothing about statistics
Jeremy Howard (1:19:44.640)
are generally gonna be the best off.
Lex Fridman (1:19:48.560)
So you taught several courses on deep learning
Lex Fridman (1:19:51.360)
and as Feynman says,
Lex Fridman (1:19:52.960)
best way to understand something is to teach it.
Lex Fridman (1:19:55.640)
What have you learned about deep learning from teaching it?
Lex Fridman (1:19:59.160)
A lot.
Jeremy Howard (1:20:00.600)
That's a key reason for me to teach the courses.
Lex Fridman (1:20:03.560)
I mean, obviously it's gonna be necessary
Jeremy Howard (1:20:04.960)
to achieve our goal of getting domain experts
Lex Fridman (1:20:07.680)
to be familiar with deep learning,
Lex Fridman (1:20:09.320)
but it was also necessary for me to achieve my goal
Lex Fridman (1:20:12.080)
of being really familiar with deep learning.
Jeremy Howard (1:20:18.240)
I mean, to see so many domain experts
Lex Fridman (1:20:24.080)
from so many different backgrounds,
Jeremy Howard (1:20:25.680)
it's definitely, I wouldn't say taught me,
Lex Fridman (1:20:28.840)
but convinced me something that I liked to believe was true,
Jeremy Howard (1:20:32.200)
which was anyone can do it.
Lex Fridman (1:20:34.920)
So there's a lot of kind of snobbishness out there
Jeremy Howard (1:20:37.440)
about only certain people can learn to code.
Lex Fridman (1:20:40.240)
Only certain people are gonna be smart enough
Jeremy Howard (1:20:42.000)
like do AI, that's definitely bullshit.
Lex Fridman (1:20:45.360)
I've seen so many people from so many different backgrounds
Jeremy Howard (1:20:48.880)
get state of the art results in their domain areas now.
Lex Fridman (1:20:53.880)
It's definitely taught me that the key differentiator
Jeremy Howard (1:20:57.160)
between people that succeed
Lex Fridman (1:20:58.720)
and people that fail is tenacity.
Jeremy Howard (1:21:00.680)
That seems to be basically the only thing that matters.
Lex Fridman (1:21:05.560)
A lot of people give up.
Lex Fridman (1:21:06.760)
But of the ones who don't give up,
Lex Fridman (1:21:09.760)
pretty much everybody succeeds.
Jeremy Howard (1:21:12.760)
Even if at first I'm just kind of like thinking like,
Lex Fridman (1:21:15.640)
wow, they really aren't quite getting it yet, are they?
Lex Fridman (1:21:18.440)
But eventually people get it and they succeed.
Lex Fridman (1:21:22.560)
So I think that's been,
Jeremy Howard (1:21:24.240)
I think they're both things I liked to believe was true,
Lex Fridman (1:21:26.560)
but I don't feel like I really had strong evidence
Jeremy Howard (1:21:28.680)
for them to be true,
Lex Fridman (1:21:29.520)
but now I can say I've seen it again and again.
Jeremy Howard (1:21:32.520)
I've seen it again and again. So what advice do you have
Lex Fridman (1:21:37.760)
for someone who wants to get started in deep learning?
Jeremy Howard (1:21:42.200)
Train lots of models.
Lex Fridman (1:21:44.400)
That's how you learn it.
Lex Fridman (1:21:47.080)
So I think, it's not just me,
Lex Fridman (1:21:51.600)
I think our course is very good,
Lex Fridman (1:21:53.360)
but also lots of people independently
Lex Fridman (1:21:54.760)
have said it's very good.
Jeremy Howard (1:21:55.600)
It recently won the COGx award for AI courses
Lex Fridman (1:21:58.640)
as being the best in the world.
Lex Fridman (1:21:59.920)
So I'd say come to our course, course.fast.ai.
Lex Fridman (1:22:02.960)
And the thing I keep on hopping on in my lessons
Jeremy Howard (1:22:05.240)
is train models, print out the inputs to the models,
Lex Fridman (1:22:09.120)
print out to the outputs to the models,
Jeremy Howard (1:22:11.040)
like study, change the inputs a bit,
Lex Fridman (1:22:15.320)
look at how the outputs vary,
Jeremy Howard (1:22:17.320)
just run lots of experiments
Lex Fridman (1:22:18.600)
to get an intuitive understanding of what's going on.
Jeremy Howard (1:22:25.400)
To get hooked, do you think, you mentioned training,
Lex Fridman (1:22:29.080)
do you think just running the models inference,
Lex Fridman (1:22:32.640)
like if we talk about getting started?
Lex Fridman (1:22:35.400)
No, you've got to fine tune the models.
Lex Fridman (1:22:37.480)
So that's the critical thing,
Lex Fridman (1:22:39.480)
because at that point you now have a model
Jeremy Howard (1:22:41.240)
that's in your domain area.
Lex Fridman (1:22:43.280)
So there's no point running somebody else's model
Jeremy Howard (1:22:46.840)
because it's not your model.
Lex Fridman (1:22:48.120)
So it only takes five minutes to fine tune a model
Jeremy Howard (1:22:50.480)
for the data you care about.
Lex Fridman (1:22:52.080)
And in lesson two of the course,
Jeremy Howard (1:22:53.560)
we teach you how to create your own data set from scratch
Lex Fridman (1:22:56.360)
by scripting Google image search.
Jeremy Howard (1:22:58.560)
So, and we show you how to actually create
Lex Fridman (1:23:01.120)
a web application running online.
Lex Fridman (1:23:02.840)
So I create one in the course that differentiates
Lex Fridman (1:23:05.280)
between a teddy bear, a grizzly bear and a brown bear.
Lex Fridman (1:23:08.320)
And it does it with basically 100% accuracy,
Lex Fridman (1:23:11.040)
took me about four minutes to scrape the images
Jeremy Howard (1:23:13.120)
from Google search in the script.
Lex Fridman (1:23:15.080)
There's a little graphical widgets we have in the notebook
Jeremy Howard (1:23:18.760)
that help you clean up the data set.
Lex Fridman (1:23:21.400)
There's other widgets that help you study the results
Jeremy Howard (1:23:24.040)
to see where the errors are happening.
Lex Fridman (1:23:26.360)
And so now we've got over a thousand replies
Jeremy Howard (1:23:29.280)
in our share your work here thread
Lex Fridman (1:23:31.400)
of students saying, here's the thing I built.
Lex Fridman (1:23:34.280)
And so there's people who like,
Lex Fridman (1:23:35.880)
and a lot of them are state of the art.
Jeremy Howard (1:23:37.600)
Like somebody said, oh, I tried looking
Lex Fridman (1:23:39.000)
at Devangari characters and I couldn't believe it.
Jeremy Howard (1:23:41.160)
The thing that came out was more accurate
Lex Fridman (1:23:43.320)
than the best academic paper after lesson one.
Lex Fridman (1:23:46.640)
And then there's others which are just more kind of fun,
Lex Fridman (1:23:48.560)
like somebody who's doing Trinidad and Tobago hummingbirds.
Jeremy Howard (1:23:53.080)
She said that's kind of their national bird
Lex Fridman (1:23:54.880)
and she's got something that can now classify Trinidad
Lex Fridman (1:23:57.400)
and Tobago hummingbirds.
Lex Fridman (1:23:58.840)
So yeah, train models, fine tune models with your data set
Lex Fridman (1:24:02.440)
and then study their inputs and outputs.
Lex Fridman (1:24:05.200)
How much is Fast.ai courses?
Jeremy Howard (1:24:07.160)
Free.
Lex Fridman (1:24:08.920)
Everything we do is free.
Jeremy Howard (1:24:10.520)
We have no revenue sources of any kind.
Lex Fridman (1:24:12.720)
It's just a service to the community.
Jeremy Howard (1:24:15.400)
You're a saint.
Lex Fridman (1:24:16.600)
Okay, once a person understands the basics,
Jeremy Howard (1:24:20.080)
trains a bunch of models,
Lex Fridman (1:24:22.360)
if we look at the scale of years,
Lex Fridman (1:24:25.840)
what advice do you have for someone wanting
Lex Fridman (1:24:27.600)
to eventually become an expert?
Jeremy Howard (1:24:30.800)
Train lots of models.
Lex Fridman (1:24:31.800)
But specifically train lots of models in your domain area.
Lex Fridman (1:24:35.320)
So an expert what, right?
Lex Fridman (1:24:37.040)
We don't need more expert,
Jeremy Howard (1:24:39.120)
like create slightly evolutionary research in areas
Lex Fridman (1:24:45.400)
that everybody's studying.
Jeremy Howard (1:24:46.680)
We need experts at using deep learning
Lex Fridman (1:24:50.400)
to diagnose malaria.
Jeremy Howard (1:24:52.600)
Or we need experts at using deep learning
Lex Fridman (1:24:55.480)
to analyze language to study media bias.
Lex Fridman (1:25:01.000)
So we need experts in analyzing fisheries
Lex Fridman (1:25:08.320)
to identify problem areas in the ocean.
Jeremy Howard (1:25:11.880)
That's what we need.
Lex Fridman (1:25:13.200)
So become the expert in your passion area.
Lex Fridman (1:25:17.720)
And this is a tool which you can use for just about anything
Lex Fridman (1:25:21.200)
and you'll be able to do that thing better
Jeremy Howard (1:25:22.880)
than other people, particularly by combining it
Lex Fridman (1:25:25.720)
with your passion and domain expertise.
Lex Fridman (1:25:27.400)
So that's really interesting.
Lex Fridman (1:25:28.360)
Even if you do wanna innovate on transfer learning
Jeremy Howard (1:25:30.840)
or active learning, your thought is,
Lex Fridman (1:25:34.000)
I mean, it's one I certainly share,
Jeremy Howard (1:25:36.200)
is you also need to find a domain or data set
Lex Fridman (1:25:40.120)
that you actually really care for.
Jeremy Howard (1:25:42.000)
If you're not working on a real problem that you understand,
Lex Fridman (1:25:45.360)
how do you know if you're doing it any good?
Lex Fridman (1:25:48.040)
How do you know if your results are good?
Lex Fridman (1:25:49.320)
How do you know if you're getting bad results?
Lex Fridman (1:25:50.800)
Why are you getting bad results?
Lex Fridman (1:25:52.040)
Is it a problem with the data?
Lex Fridman (1:25:54.080)
Like, how do you know you're doing anything useful?
Lex Fridman (1:25:57.400)
Yeah, to me, the only really interesting research is,
Jeremy Howard (1:26:00.960)
not the only, but the vast majority
Lex Fridman (1:26:02.360)
of interesting research is like,
Jeremy Howard (1:26:04.480)
try and solve an actual problem and solve it really well.
Lex Fridman (1:26:06.880)
So both understanding sufficient tools
Jeremy Howard (1:26:09.440)
on the deep learning side and becoming a domain expert
Lex Fridman (1:26:13.720)
in a particular domain are really things
Jeremy Howard (1:26:15.640)
within reach for anybody.
Lex Fridman (1:26:18.240)
Yeah, I mean, to me, I would compare it
Jeremy Howard (1:26:20.520)
to like studying self driving cars,
Lex Fridman (1:26:23.440)
having never looked at a car or been in a car
Jeremy Howard (1:26:26.520)
or turned a car on, which is like the way it is
Lex Fridman (1:26:29.320)
for a lot of people, they'll study some academic data set
Jeremy Howard (1:26:33.960)
where they literally have no idea about that.
Lex Fridman (1:26:36.200)
By the way, I'm not sure how familiar
Jeremy Howard (1:26:37.680)
with autonomous vehicles, but that is literally,
Lex Fridman (1:26:40.840)
you describe a large percentage of robotics folks
Jeremy Howard (1:26:43.400)
working in self driving cars is they actually
Lex Fridman (1:26:45.800)
haven't considered driving.
Jeremy Howard (1:26:48.640)
They haven't actually looked at what driving looks like.
Lex Fridman (1:26:50.560)
They haven't driven.
Lex Fridman (1:26:51.400)
And it's a problem because you know,
Lex Fridman (1:26:53.280)
when you've actually driven, you know,
Jeremy Howard (1:26:54.360)
like these are the things that happened
Lex Fridman (1:26:55.920)
to me when I was driving.
Jeremy Howard (1:26:57.400)
There's nothing that beats the real world examples
Lex Fridman (1:26:59.640)
of just experiencing them.
Jeremy Howard (1:27:02.360)
You've created many successful startups.
Lex Fridman (1:27:04.840)
What does it take to create a successful startup?
Jeremy Howard (1:27:08.600)
Same thing as becoming a successful
Lex Fridman (1:27:11.480)
deep learning practitioner, which is not giving up.
Lex Fridman (1:27:15.000)
So you can run out of money or run out of time
Lex Fridman (1:27:23.160)
or run out of something, you know,
Lex Fridman (1:27:24.680)
but if you keep costs super low
Lex Fridman (1:27:28.000)
and try and save up some money beforehand
Lex Fridman (1:27:29.920)
so you can afford to have some time,
Lex Fridman (1:27:35.360)
then just sticking with it is one important thing.
Jeremy Howard (1:27:38.040)
Doing something you understand and care about is important.
Lex Fridman (1:27:42.640)
By something, I don't mean,
Jeremy Howard (1:27:44.840)
the biggest problem I see with deep learning people
Lex Fridman (1:27:46.680)
is they do a PhD in deep learning
Lex Fridman (1:27:50.120)
and then they try and commercialize their PhD.
Lex Fridman (1:27:52.400)
It is a waste of time
Jeremy Howard (1:27:53.280)
because that doesn't solve an actual problem.
Lex Fridman (1:27:55.840)
You picked your PhD topic
Jeremy Howard (1:27:57.560)
because it was an interesting kind of engineering
Lex Fridman (1:28:00.080)
or math or research exercise.
Lex Fridman (1:28:02.480)
But yeah, if you've actually spent time as a recruiter
Lex Fridman (1:28:06.640)
and you know that most of your time was spent
Jeremy Howard (1:28:09.240)
sifting through resumes
Lex Fridman (1:28:10.640)
and you know that most of the time
Jeremy Howard (1:28:12.840)
you're just looking for certain kinds of things
Lex Fridman (1:28:14.680)
and you can try doing that with a model for a few minutes
Lex Fridman (1:28:19.680)
and see whether that's something which a model
Lex Fridman (1:28:21.000)
seems to be able to do as well as you could,
Jeremy Howard (1:28:23.720)
then you're on the right track to creating a startup.
Lex Fridman (1:28:27.600)
And then I think just, yeah, being, just be pragmatic and
Jeremy Howard (1:28:32.280)
try and stay away from venture capital money
Lex Fridman (1:28:36.760)
as long as possible, preferably forever.
Lex Fridman (1:28:39.160)
So yeah, on that point, do you venture capital?
Lex Fridman (1:28:43.400)
So did you, were you able to successfully run startups
Lex Fridman (1:28:47.120)
with self funded for quite a while?
Lex Fridman (1:28:48.200)
Yeah, so my first two were self funded
Lex Fridman (1:28:50.160)
and that was the right way to do it.
Lex Fridman (1:28:52.320)
Is that scary?
Jeremy Howard (1:28:54.240)
No, VC startups are much more scary
Lex Fridman (1:28:57.800)
because you have these people on your back
Jeremy Howard (1:29:00.640)
who do this all the time and who have done it for years
Lex Fridman (1:29:03.320)
telling you grow, grow, grow, grow.
Lex Fridman (1:29:05.400)
And they don't care if you fail.
Lex Fridman (1:29:07.160)
They only care if you don't grow fast enough.
Lex Fridman (1:29:09.440)
So that's scary.
Lex Fridman (1:29:10.800)
Whereas doing the ones myself, well, with partners
Jeremy Howard (1:29:16.600)
who were friends was nice
Lex Fridman (1:29:18.400)
because like we just went along at a pace that made sense
Lex Fridman (1:29:22.360)
and we were able to build it to something
Lex Fridman (1:29:23.760)
which was big enough that we never had to work again
Lex Fridman (1:29:27.280)
but was not big enough that any VC
Lex Fridman (1:29:29.280)
would think it was impressive.
Lex Fridman (1:29:31.480)
And that was enough for us to be excited, you know?
Lex Fridman (1:29:35.920)
So I thought that's a much better way
Jeremy Howard (1:29:38.840)
to do things than most people.
Lex Fridman (1:29:40.280)
In generally speaking, not for yourself
Lex Fridman (1:29:41.920)
but how do you make money during that process?
Lex Fridman (1:29:44.520)
Do you cut into savings?
Lex Fridman (1:29:47.440)
So yeah, so for, so I started Fast Mail
Lex Fridman (1:29:49.840)
and Optimal Decisions at the same time in 1999
Jeremy Howard (1:29:52.760)
with two different friends.
Lex Fridman (1:29:54.560)
And for Fast Mail, I guess I spent $70 a month
Jeremy Howard (1:30:01.160)
on the server.
Lex Fridman (1:30:04.000)
And when the server ran out of space
Jeremy Howard (1:30:06.240)
I put a payments button on the front page
Lex Fridman (1:30:09.400)
and said, if you want more than 10 mega space
Jeremy Howard (1:30:11.880)
you have to pay $10 a year.
Lex Fridman (1:30:15.640)
And.
Lex Fridman (1:30:16.480)
So run low, like keep your costs down.
Lex Fridman (1:30:18.520)
Yeah, so I kept my costs down.
Lex Fridman (1:30:19.480)
And once, you know, once I needed to spend more money
Lex Fridman (1:30:22.960)
I asked people to spend the money for me.
Lex Fridman (1:30:25.600)
And that, that was that.
Lex Fridman (1:30:28.400)
Basically from then on, we were making money
Lex Fridman (1:30:30.800)
and I was profitable from then.
Lex Fridman (1:30:35.400)
For Optimal Decisions, it was a bit harder
Jeremy Howard (1:30:37.680)
because we were trying to sell something
Lex Fridman (1:30:40.040)
that was more like a $1 million sale.
Lex Fridman (1:30:42.160)
But what we did was we would sell scoping projects.
Lex Fridman (1:30:46.400)
So kind of like prototypy projects
Lex Fridman (1:30:50.560)
but rather than doing it for free
Lex Fridman (1:30:51.720)
we would sell them 50 to $100,000.
Lex Fridman (1:30:54.200)
So again, we were covering our costs
Lex Fridman (1:30:56.920)
and also making the client feel
Jeremy Howard (1:30:58.320)
like we were doing something valuable.
Lex Fridman (1:31:00.200)
So in both cases, we were profitable from six months in.
Jeremy Howard (1:31:06.000)
Ah, nevertheless, it's scary.
Lex Fridman (1:31:08.160)
I mean, yeah, sure.
Jeremy Howard (1:31:10.040)
I mean, it's, it's scary before you jump in
Lex Fridman (1:31:13.280)
and I just, I guess I was comparing it
Jeremy Howard (1:31:15.600)
to the scarediness of VC.
Lex Fridman (1:31:18.120)
I felt like with VC stuff, it was more scary.
Jeremy Howard (1:31:20.480)
Kind of much more in somebody else's hands,
Lex Fridman (1:31:24.320)
will they fund you or not?
Lex Fridman (1:31:26.120)
And what do they think of what you're doing?
Lex Fridman (1:31:27.840)
I also found it very difficult with VCs,
Jeremy Howard (1:31:29.760)
back startups to actually do the thing
Lex Fridman (1:31:32.600)
which I thought was important for the company
Jeremy Howard (1:31:34.880)
rather than doing the thing
Lex Fridman (1:31:35.920)
which I thought would make the VC happy.
Lex Fridman (1:31:38.840)
And VCs always tell you not to do the thing
Lex Fridman (1:31:40.880)
that makes them happy.
Lex Fridman (1:31:42.360)
But then if you don't do the thing that makes them happy
Lex Fridman (1:31:44.040)
they get sad, so.
Lex Fridman (1:31:46.360)
And do you think optimizing for the,
Lex Fridman (1:31:48.080)
whatever they call it, the exit is a good thing
Lex Fridman (1:31:51.960)
to optimize for?
Lex Fridman (1:31:53.040)
I mean, it can be, but not at the VC level
Jeremy Howard (1:31:54.880)
because the VC exit needs to be, you know, a thousand X.
Lex Fridman (1:31:59.560)
So where else the lifestyle exit,
Jeremy Howard (1:32:03.120)
if you can sell something for $10 million,
Lex Fridman (1:32:05.360)
then you've made it, right?
Lex Fridman (1:32:06.440)
So I don't, it depends.
Lex Fridman (1:32:09.160)
If you want to build something that's gonna,
Jeremy Howard (1:32:11.200)
you're kind of happy to do forever, then fine.
Lex Fridman (1:32:13.560)
If you want to build something you want to sell
Jeremy Howard (1:32:16.720)
in three years time, that's fine too.
Lex Fridman (1:32:18.440)
I mean, they're both perfectly good outcomes.
Lex Fridman (1:32:21.280)
So you're learning Swift now, in a way.
Lex Fridman (1:32:24.880)
I mean, you've already.
Jeremy Howard (1:32:25.720)
I'm trying to.
Lex Fridman (1:32:26.760)
And I read that you use, at least in some cases,
Jeremy Howard (1:32:31.120)
space repetition as a mechanism for learning new things.
Lex Fridman (1:32:34.400)
I use Anki quite a lot myself.
Jeremy Howard (1:32:36.400)
Me too.
Lex Fridman (1:32:38.920)
I actually never talk to anybody about it.
Jeremy Howard (1:32:41.440)
Don't know how many people do it,
Lex Fridman (1:32:44.120)
but it works incredibly well for me.
Lex Fridman (1:32:46.720)
Can you talk to your experience?
Lex Fridman (1:32:47.920)
Like how did you, what do you?
Jeremy Howard (1:32:51.080)
First of all, okay, let's back it up.
Lex Fridman (1:32:53.080)
What is space repetition?
Lex Fridman (1:32:55.080)
So space repetition is an idea created
Lex Fridman (1:33:00.280)
by a psychologist named Ebbinghaus.
Jeremy Howard (1:33:04.200)
I don't know, must be a couple of hundred years ago
Lex Fridman (1:33:06.080)
or something, 150 years ago.
Jeremy Howard (1:33:08.000)
He did something which sounds pretty damn tedious.
Lex Fridman (1:33:10.680)
He wrote down random sequences of letters on cards
Lex Fridman (1:33:15.600)
and tested how well he would remember
Lex Fridman (1:33:18.840)
those random sequences a day later, a week later, whatever.
Jeremy Howard (1:33:23.000)
He discovered that there was this kind of a curve
Lex Fridman (1:33:26.120)
where his probability of remembering one of them
Jeremy Howard (1:33:28.800)
would be dramatically smaller the next day
Lex Fridman (1:33:30.640)
and then a little bit smaller the next day
Lex Fridman (1:33:31.960)
and a little bit smaller the next day.
Lex Fridman (1:33:33.520)
What he discovered is that if he revised those cards
Jeremy Howard (1:33:36.880)
after a day, the probabilities would decrease
Lex Fridman (1:33:41.600)
at a smaller rate.
Lex Fridman (1:33:42.880)
And then if you revise them again a week later,
Lex Fridman (1:33:44.960)
they would decrease at a smaller rate again.
Lex Fridman (1:33:47.040)
And so he basically figured out a roughly optimal equation
Lex Fridman (1:33:51.800)
for when you should revise something you wanna remember.
Lex Fridman (1:33:56.560)
So space repetition learning is using this simple algorithm,
Lex Fridman (1:34:00.440)
just something like revise something after a day
Lex Fridman (1:34:03.640)
and then three days and then a week and then three weeks
Lex Fridman (1:34:06.640)
and so forth.
Lex Fridman (1:34:07.720)
And so if you use a program like Anki, as you know,
Lex Fridman (1:34:10.680)
it will just do that for you.
Lex Fridman (1:34:12.120)
And it will say, did you remember this?
Lex Fridman (1:34:14.560)
And if you say no, it will reschedule it back
Jeremy Howard (1:34:17.680)
to appear again like 10 times faster
Lex Fridman (1:34:20.320)
than it otherwise would have.
Jeremy Howard (1:34:23.080)
It's a kind of a way of being guaranteed to learn something
Lex Fridman (1:34:27.920)
because by definition, if you're not learning it,
Jeremy Howard (1:34:30.240)
it will be rescheduled to be revised more quickly.
Lex Fridman (1:34:33.680)
Unfortunately though, it's also like,
Jeremy Howard (1:34:36.120)
it doesn't let you fool yourself.
Lex Fridman (1:34:37.480)
If you're not learning something,
Jeremy Howard (1:34:40.160)
you know like your revisions will just get more and more.
Lex Fridman (1:34:44.080)
So you have to find ways to learn things productively
Lex Fridman (1:34:48.280)
and effectively like treat your brain well.
Lex Fridman (1:34:50.560)
So using like mnemonics and stories and context
Lex Fridman (1:34:54.880)
and stuff like that.
Lex Fridman (1:34:57.560)
So yeah, it's a super great technique.
Jeremy Howard (1:34:59.760)
It's like learning how to learn is something
Lex Fridman (1:35:01.360)
which everybody should learn
Jeremy Howard (1:35:03.800)
before they actually learn anything.
Lex Fridman (1:35:05.680)
But almost nobody does.
Lex Fridman (1:35:07.840)
So what have you, so it certainly works well
Lex Fridman (1:35:10.120)
for learning new languages for, I mean,
Jeremy Howard (1:35:13.720)
for learning like small projects almost.
Lex Fridman (1:35:16.440)
But do you, you know, I started using it for,
Jeremy Howard (1:35:19.840)
I forget who wrote a blog post about this inspired me.
Lex Fridman (1:35:22.160)
It might've been you, I'm not sure.
Jeremy Howard (1:35:26.840)
I started when I read papers,
Lex Fridman (1:35:28.520)
I'll concepts and ideas, I'll put them.
Lex Fridman (1:35:31.920)
Was it Michael Nielsen?
Lex Fridman (1:35:32.840)
It was Michael Nielsen.
Lex Fridman (1:35:33.680)
So Michael started doing this recently
Lex Fridman (1:35:36.400)
and has been writing about it.
Lex Fridman (1:35:41.000)
So the kind of today's Ebbinghaus
Lex Fridman (1:35:43.200)
is a guy called Peter Wozniak
Jeremy Howard (1:35:45.080)
who developed a system called SuperMemo.
Lex Fridman (1:35:47.720)
And he's been basically trying to become like
Jeremy Howard (1:35:51.680)
the world's greatest Renaissance man
Lex Fridman (1:35:54.080)
over the last few decades.
Jeremy Howard (1:35:55.960)
He's basically lived his life
Lex Fridman (1:35:57.280)
with space repetition learning for everything.
Jeremy Howard (1:36:03.840)
I, and sort of like,
Lex Fridman (1:36:05.800)
Michael's only very recently got into this,
Lex Fridman (1:36:07.440)
but he started really getting excited
Lex Fridman (1:36:08.920)
about doing it for a lot of different things.
Jeremy Howard (1:36:11.200)
For me personally, I actually don't use it
Lex Fridman (1:36:14.600)
for anything except Chinese.
Lex Fridman (1:36:16.920)
And the reason for that is that
Lex Fridman (1:36:20.120)
Chinese is specifically a thing I made a conscious decision
Jeremy Howard (1:36:23.080)
that I want to continue to remember,
Lex Fridman (1:36:27.680)
even if I don't get much of a chance to exercise it,
Jeremy Howard (1:36:30.080)
cause like I'm not often in China, so I don't.
Lex Fridman (1:36:33.840)
Or else something like programming languages or papers.
Jeremy Howard (1:36:38.280)
I have a very different approach,
Lex Fridman (1:36:39.600)
which is I try not to learn anything from them,
Lex Fridman (1:36:43.040)
but instead I try to identify the important concepts
Lex Fridman (1:36:47.040)
and like actually ingest them.
Lex Fridman (1:36:48.960)
So like really understand that concept deeply
Lex Fridman (1:36:53.600)
and study it carefully.
Jeremy Howard (1:36:54.760)
I will decide if it really is important,
Lex Fridman (1:36:56.560)
if it is like incorporated into our library,
Jeremy Howard (1:37:01.560)
you know, incorporated into how I do things
Lex Fridman (1:37:04.160)
or decide it's not worth it, say.
Lex Fridman (1:37:07.960)
So I find, I find I then remember the things
Lex Fridman (1:37:12.200)
that I care about because I'm using it all the time.
Lex Fridman (1:37:15.720)
So I've, for the last 25 years,
Lex Fridman (1:37:20.160)
I've committed to spending at least half of every day
Jeremy Howard (1:37:23.440)
learning or practicing something new,
Lex Fridman (1:37:25.920)
which is all my colleagues have always hated
Jeremy Howard (1:37:28.800)
because it always looks like I'm not working on
Lex Fridman (1:37:31.040)
what I'm meant to be working on,
Lex Fridman (1:37:32.000)
but it always means I do everything faster
Lex Fridman (1:37:34.560)
because I've been practicing a lot of stuff.
Lex Fridman (1:37:36.920)
So I kind of give myself a lot of opportunity
Lex Fridman (1:37:39.400)
to practice new things.
Lex Fridman (1:37:41.680)
And so I find now I don't,
Lex Fridman (1:37:43.280)
yeah, I don't often kind of find myself
Jeremy Howard (1:37:47.840)
wishing I could remember something
Lex Fridman (1:37:50.240)
because if it's something that's useful,
Jeremy Howard (1:37:51.400)
then I've been using it a lot.
Lex Fridman (1:37:53.840)
It's easy enough to look it up on Google,
Lex Fridman (1:37:56.120)
but speaking Chinese, you can't look it up on Google.
Lex Fridman (1:37:59.640)
Do you have advice for people learning new things?
Lex Fridman (1:38:01.520)
So if you, what have you learned as a process as a,
Lex Fridman (1:38:04.800)
I mean, it all starts with just making the hours
Lex Fridman (1:38:07.600)
and the day available.
Lex Fridman (1:38:08.920)
Yeah, you got to stick with it,
Jeremy Howard (1:38:10.120)
which is again, the number one thing
Lex Fridman (1:38:12.000)
that 99% of people don't do.
Lex Fridman (1:38:13.600)
So the people I started learning Chinese with,
Lex Fridman (1:38:15.840)
none of them were still doing it 12 months later.
Jeremy Howard (1:38:18.320)
I'm still doing it 10 years later.
Lex Fridman (1:38:20.320)
I tried to stay in touch with them,
Lex Fridman (1:38:21.840)
but they just, no one did it.
Lex Fridman (1:38:24.560)
For something like Chinese,
Jeremy Howard (1:38:26.160)
like study how human learning works.
Lex Fridman (1:38:28.440)
So every one of my Chinese flashcards
Jeremy Howard (1:38:31.160)
is associated with a story.
Lex Fridman (1:38:33.680)
And that story is specifically designed to be memorable.
Lex Fridman (1:38:36.680)
And we find things memorable,
Lex Fridman (1:38:37.800)
which are like funny or disgusting or sexy
Jeremy Howard (1:38:41.320)
or related to people that we know or care about.
Lex Fridman (1:38:44.200)
So I try to make sure all of the stories
Jeremy Howard (1:38:46.040)
that are in my head have those characteristics.
Lex Fridman (1:38:51.000)
Yeah, so you have to, you know,
Jeremy Howard (1:38:52.120)
you won't remember things well
Lex Fridman (1:38:53.200)
if they don't have some context.
Lex Fridman (1:38:56.000)
And yeah, you won't remember them well
Lex Fridman (1:38:57.240)
if you don't regularly practice them,
Jeremy Howard (1:39:00.600)
whether it be just part of your day to day life
Lex Fridman (1:39:02.440)
or the Chinese and me flashcards.
Jeremy Howard (1:39:06.040)
I mean, the other thing is,
Lex Fridman (1:39:07.800)
I'll let yourself fail sometimes.
Lex Fridman (1:39:09.520)
So like I've had various medical problems
Lex Fridman (1:39:11.840)
over the last few years.
Lex Fridman (1:39:13.040)
And basically my flashcards
Lex Fridman (1:39:16.400)
just stopped for about three years.
Lex Fridman (1:39:18.640)
And there've been other times I've stopped for a few months
Lex Fridman (1:39:22.600)
and it's so hard because you get back to it
Lex Fridman (1:39:24.240)
and it's like, you have 18,000 cards due.
Lex Fridman (1:39:27.400)
It's like, and so you just have to go, all right,
Jeremy Howard (1:39:30.920)
well, I can either stop and give up everything
Lex Fridman (1:39:34.160)
or just decide to do this every day for the next two years
Jeremy Howard (1:39:37.560)
until I get back to it.
Lex Fridman (1:39:39.000)
The amazing thing has been that even after three years,
Jeremy Howard (1:39:41.680)
I, you know, the Chinese were still in there.
Lex Fridman (1:39:45.880)
Like it was so much faster to relearn
Jeremy Howard (1:39:48.480)
than it was to learn the first time.
Lex Fridman (1:39:50.120)
Yeah, absolutely.
Jeremy Howard (1:39:52.320)
It's in there.
Lex Fridman (1:39:53.160)
I have the same with guitar, with music and so on.
Jeremy Howard (1:39:56.560)
It's sad because the work sometimes takes away
Lex Fridman (1:39:59.160)
and then you won't play for a year.
Lex Fridman (1:40:01.200)
But really, if you then just get back to it every day,
Lex Fridman (1:40:03.560)
you're right there again.
Lex Fridman (1:40:06.040)
What do you think is the next big breakthrough
Lex Fridman (1:40:08.400)
in artificial intelligence?
Lex Fridman (1:40:09.400)
What are your hopes in deep learning or beyond
Lex Fridman (1:40:12.720)
that people should be working on
Lex Fridman (1:40:14.120)
or you hope there'll be breakthroughs?
Lex Fridman (1:40:16.320)
I don't think it's possible to predict.
Jeremy Howard (1:40:17.960)
I think what we already have
Lex Fridman (1:40:20.600)
is an incredibly powerful platform
Jeremy Howard (1:40:23.720)
to solve lots of societally important problems
Lex Fridman (1:40:26.520)
that are currently unsolved.
Lex Fridman (1:40:27.600)
So I just hope that people will,
Lex Fridman (1:40:29.920)
lots of people will learn this toolkit and try to use it.
Jeremy Howard (1:40:33.360)
I don't think we need a lot of new technological breakthroughs
Lex Fridman (1:40:36.800)
to do a lot of great work right now.
Lex Fridman (1:40:39.880)
And when do you think we're going to create
Lex Fridman (1:40:42.760)
a human level intelligence system?
Lex Fridman (1:40:45.160)
Do you think?
Lex Fridman (1:40:46.000)
Don't know.
Lex Fridman (1:40:46.840)
How hard is it?
Lex Fridman (1:40:47.680)
How far away are we?
Jeremy Howard (1:40:48.720)
Don't know.
Lex Fridman (1:40:49.560)
Don't know.
Jeremy Howard (1:40:50.400)
I have no way to know.
Lex Fridman (1:40:51.240)
I don't know why people make predictions about this
Jeremy Howard (1:40:53.840)
because there's no data and nothing to go on.
Lex Fridman (1:40:57.480)
And it's just like,
Jeremy Howard (1:41:00.320)
there's so many societally important problems
Lex Fridman (1:41:03.480)
to solve right now.
Jeremy Howard (1:41:04.400)
I just don't find it a really interesting question
Lex Fridman (1:41:08.680)
to even answer.
Lex Fridman (1:41:10.280)
So in terms of societally important problems,
Lex Fridman (1:41:12.960)
what's the problem that is within reach?
Jeremy Howard (1:41:16.360)
Well, I mean, for example,
Lex Fridman (1:41:17.440)
there are problems that AI creates, right?
Lex Fridman (1:41:19.760)
So more specifically,
Lex Fridman (1:41:23.160)
labor force displacement is going to be huge
Lex Fridman (1:41:26.800)
and people keep making this
Lex Fridman (1:41:29.160)
frivolous econometric argument of being like,
Jeremy Howard (1:41:31.520)
oh, there's been other things that aren't AI
Lex Fridman (1:41:33.960)
that have come along before
Lex Fridman (1:41:34.920)
and haven't created massive labor force displacement,
Lex Fridman (1:41:37.800)
therefore AI won't.
Lex Fridman (1:41:39.880)
So that's a serious concern for you?
Lex Fridman (1:41:41.560)
Oh yeah.
Jeremy Howard (1:41:42.400)
Andrew Yang is running on it.
Lex Fridman (1:41:43.680)
Yeah, it's, I'm desperately concerned.
Lex Fridman (1:41:47.320)
And you see already that the changing workplace
Lex Fridman (1:41:53.080)
has led to a hollowing out of the middle class.
Jeremy Howard (1:41:55.720)
You're seeing that students coming out of school today
Lex Fridman (1:41:59.000)
have a less rosy financial future ahead of them
Jeremy Howard (1:42:03.120)
than their parents did,
Lex Fridman (1:42:03.960)
which has never happened in recent,
Jeremy Howard (1:42:06.560)
in the last few hundred years.
Lex Fridman (1:42:08.600)
You know, we've always had progress before.
Lex Fridman (1:42:11.760)
And you see this turning into anxiety
Lex Fridman (1:42:15.520)
and despair and even violence.
Lex Fridman (1:42:19.440)
So I very much worry about that.
Lex Fridman (1:42:23.400)
You've written quite a bit about ethics too.
Jeremy Howard (1:42:25.720)
I do think that every data scientist
Lex Fridman (1:42:29.600)
working with deep learning needs to recognize
Jeremy Howard (1:42:33.920)
they have an incredibly high leverage tool
Lex Fridman (1:42:35.600)
that they're using that can influence society
Jeremy Howard (1:42:37.960)
in lots of ways.
Lex Fridman (1:42:39.000)
And if they're doing research,
Jeremy Howard (1:42:40.320)
that that research is gonna be used by people
Lex Fridman (1:42:42.760)
doing this kind of work.
Lex Fridman (1:42:44.400)
And they have a responsibility to consider the consequences
Lex Fridman (1:42:48.360)
and to think about things like
Lex Fridman (1:42:51.760)
how will humans be in the loop here?
Lex Fridman (1:42:53.920)
How do we avoid runaway feedback loops?
Lex Fridman (1:42:56.520)
How do we ensure an appeals process for humans
Lex Fridman (1:42:59.200)
that are impacted by my algorithm?
Lex Fridman (1:43:01.720)
How do I ensure that the constraints of my algorithm
Lex Fridman (1:43:04.960)
are adequately explained to the people
Lex Fridman (1:43:06.720)
that end up using them?
Lex Fridman (1:43:09.160)
There's all kinds of human issues
Jeremy Howard (1:43:11.880)
which only data scientists are actually
Lex Fridman (1:43:15.400)
in the right place to educate people are about,
Lex Fridman (1:43:17.960)
but data scientists tend to think of themselves
Lex Fridman (1:43:20.280)
as just engineers and that they don't need
Jeremy Howard (1:43:23.400)
to be part of that process, which is wrong.
Lex Fridman (1:43:26.720)
Well, you're in the perfect position to educate them better,
Jeremy Howard (1:43:30.320)
to read literature, to read history, to learn from history.
Lex Fridman (1:43:35.800)
Well, Jeremy, thank you so much for everything you do
Jeremy Howard (1:43:39.160)
for inspiring huge amount of people,
Lex Fridman (1:43:41.360)
getting them into deep learning
Lex Fridman (1:43:42.520)
and having the ripple effects,
Lex Fridman (1:43:45.120)
the flap of a butterfly's wings
Jeremy Howard (1:43:47.480)
that will probably change the world.
Lex Fridman (1:43:48.680)
So thank you very much.
Jeremy Howard (1:43:50.120)
Thank you, thank you, thank you, thank you.
Lex Fridman (20:03.520)
and TILE and MLIR and TVM,
Jeremy Howard (20:07.120)
there's all these various projects
Lex Fridman (20:08.640)
which are all about saying,
Jeremy Howard (20:10.840)
let's let people create like domain specific languages
Lex Fridman (20:14.000)
for tensor computations.
Jeremy Howard (20:16.840)
These are the kinds of things we do generally
Lex Fridman (20:19.320)
on the GPU for deep learning and then have a compiler
Jeremy Howard (20:22.840)
which can optimize that tensor computation.
Lex Fridman (20:28.080)
A lot of this work is actually sitting
Jeremy Howard (20:29.840)
on top of a project called Halide,
Lex Fridman (20:32.640)
which is a mind blowing project where they came up
Jeremy Howard (20:37.080)
with such a domain specific language.
Lex Fridman (20:38.840)
In fact, two, one domain specific language for expressing
Jeremy Howard (20:41.200)
this is what my tensor computation is
Lex Fridman (20:43.800)
and another domain specific language for expressing
Jeremy Howard (20:46.280)
this is the kind of the way I want you to structure
Lex Fridman (20:50.280)
the compilation of that and like do it block by block
Lex Fridman (20:53.040)
and do these bits in parallel.
Lex Fridman (20:54.920)
And they were able to show how you can compress
Jeremy Howard (20:57.720)
the amount of code by 10X compared to optimized GPU code
Lex Fridman (21:03.280)
and get the same performance.
Lex Fridman (21:05.520)
So that's like, so these other things are kind of sitting
Lex Fridman (21:08.080)
on top of that kind of research and MLIR is pulling a lot
Jeremy Howard (21:12.760)
of those best practices together.
Lex Fridman (21:15.120)
And now we're starting to see work done on making all
Jeremy Howard (21:18.240)
of that directly accessible through Swift
Lex Fridman (21:21.360)
so that I could use Swift to kind of write those
Jeremy Howard (21:23.720)
domain specific languages and hopefully we'll get
Lex Fridman (21:27.240)
then Swift CUDA kernels written in a very expressive
Lex Fridman (21:30.680)
and concise way that looks a bit like J and APL
Lex Fridman (21:34.160)
and then Swift layers on top of that
Lex Fridman (21:36.680)
and then a Swift UI on top of that.
Lex Fridman (21:38.440)
And it'll be so nice if we can get to that point.
Jeremy Howard (21:42.600)
Now does it all eventually boil down to CUDA
Lex Fridman (21:46.520)
and NVIDIA GPUs?
Jeremy Howard (21:48.560)
Unfortunately at the moment it does,
Lex Fridman (21:50.160)
but one of the nice things about MLIR if AMD ever
Jeremy Howard (21:54.480)
gets their act together which they probably won't
Lex Fridman (21:56.760)
is that they or others could write MLIR backends
Jeremy Howard (22:02.120)
for other GPUs or rather tensor computation devices
Lex Fridman (22:09.720)
of which today there are increasing number
Jeremy Howard (22:11.600)
like Graph Core or Vertex AI or whatever.
Lex Fridman (22:18.760)
So yeah, being able to target lots of backends
Jeremy Howard (22:22.560)
would be another benefit of this
Lex Fridman (22:23.920)
and the market really needs competition
Jeremy Howard (22:26.680)
because at the moment NVIDIA is massively overcharging
Lex Fridman (22:29.520)
for their kind of enterprise class cards
Jeremy Howard (22:33.640)
because there is no serious competition
Lex Fridman (22:36.720)
because nobody else is doing the software properly.
Lex Fridman (22:39.280)
In the cloud there is some competition, right?
Lex Fridman (22:41.400)
But...
Jeremy Howard (22:42.920)
Not really, other than TPUs perhaps,
Lex Fridman (22:45.120)
but TPUs are almost unprogrammable at the moment.
Lex Fridman (22:48.240)
So TPUs have the same problem that you can't?
Lex Fridman (22:51.200)
It's even worse.
Lex Fridman (22:52.040)
So TPUs, Google actually made an explicit decision
Lex Fridman (22:54.840)
to make them almost entirely unprogrammable
Jeremy Howard (22:57.200)
because they felt that there was too much IP in there
Lex Fridman (22:59.960)
and if they gave people direct access to program them,
Jeremy Howard (23:02.640)
people would learn their secrets.
Lex Fridman (23:04.360)
So you can't actually directly program the memory
Jeremy Howard (23:09.360)
in a TPU.
Lex Fridman (23:11.000)
You can't even directly create code that runs on
Lex Fridman (23:15.200)
and that you look at on the machine that has the TPU,
Lex Fridman (23:18.040)
it all goes through a virtual machine.
Lex Fridman (23:19.920)
So all you can really do is this kind of cookie cutter thing
Lex Fridman (23:22.920)
of like plug in high level stuff together,
Jeremy Howard (23:26.720)
which is just super tedious and annoying
Lex Fridman (23:30.520)
and totally unnecessary.
Lex Fridman (23:32.920)
So what was the, tell me if you could,
Lex Fridman (23:36.040)
the origin story of fast AI.
Lex Fridman (23:38.080)
What is the motivation, its mission, its dream?
Lex Fridman (23:43.280)
So I guess the founding story is heavily tied
Jeremy Howard (23:48.280)
to my previous startup, which is a company called Analytic,
Lex Fridman (23:51.480)
which was the first company to focus on deep learning
Jeremy Howard (23:54.880)
for medicine and I created that because I saw
Lex Fridman (23:58.720)
that was a huge opportunity to,
Jeremy Howard (24:02.120)
there's about a 10X shortage of the number of doctors
Lex Fridman (24:05.840)
in the world, in the developing world that we need.
Jeremy Howard (24:08.200)
I expected it would take about 300 years
Lex Fridman (24:11.760)
to train enough doctors to meet that gap.
Lex Fridman (24:13.920)
But I guess that maybe if we used deep learning
Lex Fridman (24:19.400)
for some of the analytics, we could maybe make it
Lex Fridman (24:22.800)
so you don't need as highly trained doctors.
Lex Fridman (24:25.240)
For diagnosis.
Jeremy Howard (24:26.080)
For diagnosis and treatment planning.
Lex Fridman (24:27.720)
Where's the biggest benefit just before we get to fast AI,
Jeremy Howard (24:31.440)
where's the biggest benefit of AI
Lex Fridman (24:33.880)
and medicine that you see today?
Lex Fridman (24:36.400)
And maybe next time.
Lex Fridman (24:37.240)
Not much happening today in terms of like stuff
Jeremy Howard (24:39.480)
that's actually out there, it's very early.
Lex Fridman (24:41.040)
But in terms of the opportunity,
Jeremy Howard (24:42.880)
it's to take markets like India and China and Indonesia,
Lex Fridman (24:48.720)
which have big populations, Africa,
Jeremy Howard (24:52.080)
small numbers of doctors,
Lex Fridman (24:55.760)
and provide diagnostic, particularly treatment planning
Lex Fridman (25:00.760)
and triage kind of on device so that if you do a test
Lex Fridman (25:05.760)
for malaria or tuberculosis or whatever,
Jeremy Howard (25:09.240)
you immediately get something that even a healthcare worker
Lex Fridman (25:12.440)
that's had a month of training can get
Jeremy Howard (25:16.160)
a very high quality assessment of whether the patient
Lex Fridman (25:20.440)
might be at risk and tell, okay,
Jeremy Howard (25:22.320)
we'll send them off to a hospital.
Lex Fridman (25:25.240)
So for example, in Africa, outside of South Africa,
Jeremy Howard (25:29.240)
there's only five pediatric radiologists
Lex Fridman (25:31.640)
for the entire continent.
Lex Fridman (25:32.960)
So most countries don't have any.
Lex Fridman (25:34.720)
So if your kid is sick and they need something diagnosed
Jeremy Howard (25:37.440)
through medical imaging, the person,
Lex Fridman (25:39.800)
even if you're able to get medical imaging done,
Jeremy Howard (25:41.640)
the person that looks at it will be a nurse at best.
Lex Fridman (25:46.400)
But actually in India, for example, and China,
Jeremy Howard (25:50.080)
almost no x rays are read by anybody,
Lex Fridman (25:52.360)
by any trained professional because they don't have enough.
Lex Fridman (25:57.040)
So if instead we had a algorithm that could take
Lex Fridman (26:02.040)
the most likely high risk 5% and say triage,
Jeremy Howard (26:08.040)
basically say, okay, someone needs to look at this,
Lex Fridman (26:11.040)
it would massively change the kind of way
Jeremy Howard (26:14.240)
that what's possible with medicine in the developing world.
Lex Fridman (26:18.680)
And remember, they have, increasingly they have money.
Jeremy Howard (26:21.600)
They're the developing world, they're not the poor world,
Lex Fridman (26:23.560)
they're the developing world.
Lex Fridman (26:24.400)
So they have the money.
Lex Fridman (26:25.240)
So they're building the hospitals,
Jeremy Howard (26:27.040)
they're getting the diagnostic equipment,
Lex Fridman (26:30.440)
but there's no way for a very long time
Jeremy Howard (26:33.320)
will they be able to have the expertise.
Lex Fridman (26:37.040)
Shortage of expertise, okay.
Lex Fridman (26:38.480)
And that's where the deep learning systems can step in
Lex Fridman (26:41.760)
and magnify the expertise they do have.
Jeremy Howard (26:44.320)
Exactly, yeah.
Lex Fridman (26:46.240)
So you do see, just to linger a little bit longer,
Jeremy Howard (26:51.240)
the interaction, do you still see the human experts
Lex Fridman (26:55.760)
still at the core of these systems?
Jeremy Howard (26:57.560)
Yeah, absolutely.
Lex Fridman (26:58.400)
Is there something in medicine
Lex Fridman (26:59.240)
that could be automated almost completely?
Lex Fridman (27:01.280)
I don't see the point of even thinking about that
Jeremy Howard (27:03.880)
because we have such a shortage of people.
Lex Fridman (27:06.080)
Why would we want to find a way not to use them?
Jeremy Howard (27:09.760)
We have people, so the idea of like,
Lex Fridman (27:13.000)
even from an economic point of view,
Jeremy Howard (27:14.680)
if you can make them 10X more productive,
Lex Fridman (27:17.320)
getting rid of the person,
Jeremy Howard (27:18.920)
doesn't impact your unit economics at all.
Lex Fridman (27:21.600)
And it totally ignores the fact
Jeremy Howard (27:23.360)
that there are things people do better than machines.
Lex Fridman (27:26.520)
So it's just to me,
Jeremy Howard (27:27.360)
that's not a useful way of framing the problem.
Lex Fridman (27:32.000)
I guess, just to clarify,
Jeremy Howard (27:33.760)
I guess I meant there may be some problems
Lex Fridman (27:36.480)
where you can avoid even going to the expert ever,
Jeremy Howard (27:40.000)
sort of maybe preventative care or some basic stuff,
Lex Fridman (27:44.000)
allowing food,
Jeremy Howard (27:44.840)
allowing the expert to focus on the things
Lex Fridman (27:46.600)
that are really that, you know.
Lex Fridman (27:49.200)
Well, that's what the triage would do, right?
Lex Fridman (27:50.920)
So the triage would say,
Jeremy Howard (27:52.800)
okay, there's 99% sure there's nothing here.
Lex Fridman (27:58.640)
So that can be done on device
Lex Fridman (28:01.960)
and they can just say, okay, go home.
Lex Fridman (28:03.840)
So the experts are being used to look at the stuff
Jeremy Howard (28:07.320)
which has some chance it's worth looking at,
Lex Fridman (28:10.160)
which most things it's not, it's fine.
Lex Fridman (28:14.320)
Why do you think that is?
Lex Fridman (28:15.520)
You know, it's fine.
Lex Fridman (28:16.880)
Why do you think we haven't quite made progress on that yet
Lex Fridman (28:19.920)
in terms of the scale of how much AI is applied
Lex Fridman (28:27.000)
in the medical field?
Lex Fridman (28:27.840)
Oh, there's a lot of reasons.
Jeremy Howard (28:28.680)
I mean, one is it's pretty new.
Lex Fridman (28:29.720)
I only started in Liddick in like 2014.
Lex Fridman (28:32.120)
And before that, it's hard to express
Lex Fridman (28:36.040)
to what degree the medical world
Jeremy Howard (28:37.440)
was not aware of the opportunities here.
Lex Fridman (28:40.760)
So I went to RSNA,
Jeremy Howard (28:42.960)
which is the world's largest radiology conference.
Lex Fridman (28:46.240)
And I told everybody I could, you know,
Jeremy Howard (28:49.520)
like I'm doing this thing with deep learning,
Lex Fridman (28:51.760)
please come and check it out.
Lex Fridman (28:53.360)
And no one had any idea what I was talking about
Lex Fridman (28:56.840)
and no one had any interest in it.
Lex Fridman (28:59.680)
So like we've come from absolute zero, which is hard.
Lex Fridman (29:05.120)
And then the whole regulatory framework, education system,
Jeremy Howard (29:09.920)
everything is just set up to think of doctoring
Lex Fridman (29:13.440)
in a very different way.
Lex Fridman (29:14.960)
So today there is a small number of people
Lex Fridman (29:17.120)
who are deep learning practitioners
Lex Fridman (29:20.600)
and doctors at the same time.
Lex Fridman (29:23.040)
And we're starting to see the first ones
Jeremy Howard (29:24.640)
come out of their PhD programs.
Lex Fridman (29:26.600)
So Zach Kahane over in Boston, Cambridge
Jeremy Howard (29:31.600)
has a number of students now who are data science experts,
Lex Fridman (29:37.880)
deep learning experts, and actual medical doctors.
Jeremy Howard (29:43.480)
Quite a few doctors have completed our fast AI course now
Lex Fridman (29:47.000)
and are publishing papers and creating journal reading groups
Jeremy Howard (29:52.560)
in the American Council of Radiology.
Lex Fridman (29:55.200)
And like, it's just starting to happen,
Lex Fridman (29:57.360)
but it's gonna be a long time coming.
Lex Fridman (29:59.640)
It's gonna happen, but it's gonna be a long process.
Jeremy Howard (30:02.880)
The regulators have to learn how to regulate this.
Lex Fridman (30:04.880)
They have to build guidelines.
Lex Fridman (30:08.720)
And then the lawyers at hospitals
Lex Fridman (30:12.120)
have to develop a new way of understanding
Jeremy Howard (30:15.080)
that sometimes it makes sense for data to be looked at
Lex Fridman (30:22.440)
in raw form in large quantities
Jeremy Howard (30:24.880)
in order to create well changing results.
Lex Fridman (30:27.000)
Yeah, so the regulation around data, all that,
Jeremy Howard (30:30.120)
it sounds probably the hardest problem,
Lex Fridman (30:33.880)
but sounds reminiscent of autonomous vehicles as well.
Jeremy Howard (30:36.800)
Many of the same regulatory challenges,
Lex Fridman (30:38.760)
many of the same data challenges.
Jeremy Howard (30:40.640)
Yeah, I mean, funnily enough,
Lex Fridman (30:41.560)
the problem is less the regulation
Lex Fridman (30:43.680)
and more the interpretation of that regulation
Lex Fridman (30:45.880)
by lawyers in hospitals.
Lex Fridman (30:48.240)
So HIPAA is actually, was designed to pay,
Lex Fridman (30:52.920)
and HIPAA does not stand for privacy.
Jeremy Howard (30:56.480)
It stands for portability.
Lex Fridman (30:57.680)
It's actually meant to be a way that data can be used.
Lex Fridman (31:01.240)
And it was created with lots of gray areas
Lex Fridman (31:04.400)
because the idea is that would be more practical
Lex Fridman (31:06.560)
and it would help people to use this legislation
Lex Fridman (31:10.480)
to actually share data in a more thoughtful way.
Jeremy Howard (31:13.720)
Unfortunately, it's done the opposite
Lex Fridman (31:15.360)
because when a lawyer sees a gray area,
Jeremy Howard (31:17.800)
they say, oh, if we don't know, we won't get sued,
Lex Fridman (31:20.760)
then we can't do it.
Lex Fridman (31:22.440)
So HIPAA is not exactly the problem.
Lex Fridman (31:26.360)
The problem is more that there's,
Jeremy Howard (31:29.200)
hospital lawyers are not incented
Lex Fridman (31:31.000)
to make bold decisions about data portability.
Jeremy Howard (31:36.520)
Or even to embrace technology that saves lives.
Lex Fridman (31:40.440)
They more want to not get in trouble
Jeremy Howard (31:42.440)
for embracing that technology.
Lex Fridman (31:44.760)
It also saves lives in a very abstract way,
Jeremy Howard (31:47.840)
which is like, oh, we've been able to release
Lex Fridman (31:49.840)
these 100,000 anonymized records.
Jeremy Howard (31:52.320)
I can't point to the specific person
Lex Fridman (31:54.120)
whose life that saved.
Jeremy Howard (31:55.320)
I can say like, oh, we ended up with this paper
Lex Fridman (31:57.720)
which found this result,
Jeremy Howard (31:58.960)
which diagnosed a thousand more people
Lex Fridman (32:02.200)
than we would have otherwise,
Lex Fridman (32:03.080)
but it's like, which ones were helped?
Lex Fridman (32:05.480)
It's very abstract.
Lex Fridman (32:07.320)
And on the counter side of that,
Lex Fridman (32:09.360)
you may be able to point to a life that was taken
Jeremy Howard (32:13.080)
because of something that was.
Lex Fridman (32:14.320)
Yeah, or a person whose privacy was violated.
Jeremy Howard (32:18.160)
It's like, oh, this specific person was deidentified.
Lex Fridman (32:24.200)
So, identified.
Jeremy Howard (32:25.960)
Just a fascinating topic.
Lex Fridman (32:27.280)
We're jumping around.
Jeremy Howard (32:28.240)
We'll get back to fast AI,
Lex Fridman (32:29.400)
but on the question of privacy,
Jeremy Howard (32:32.600)
data is the fuel for so much innovation in deep learning.
Lex Fridman (32:38.080)
What's your sense on privacy?
Jeremy Howard (32:39.760)
Whether we're talking about Twitter, Facebook, YouTube,
Lex Fridman (32:44.000)
just the technologies like in the medical field
Jeremy Howard (32:48.640)
that rely on people's data in order to create impact.
Lex Fridman (32:53.360)
How do we get that right,
Jeremy Howard (32:56.600)
respecting people's privacy and yet creating technology
Lex Fridman (33:01.200)
that is learning from data?
Jeremy Howard (33:03.320)
One of my areas of focus is on doing more with less data.
Lex Fridman (33:08.320)
More with less data, which,
Lex Fridman (33:11.840)
so most vendors, unfortunately,
Lex Fridman (33:14.400)
are strongly incented to find ways
Jeremy Howard (33:17.560)
to require more data and more computation.
Lex Fridman (33:20.040)
So, Google and IBM being the most obvious.
Jeremy Howard (33:24.400)
IBM.
Lex Fridman (33:25.920)
Yeah, so Watson.
Jeremy Howard (33:27.680)
So, Google and IBM both strongly push the idea
Lex Fridman (33:31.160)
that you have to be,
Jeremy Howard (33:33.080)
that they have more data and more computation
Lex Fridman (33:35.440)
and more intelligent people than anybody else.
Lex Fridman (33:37.840)
And so you have to trust them to do things
Lex Fridman (33:39.880)
because nobody else can do it.
Lex Fridman (33:42.640)
And Google's very upfront about this,
Lex Fridman (33:45.400)
like Jeff Dean has gone out there and given talks
Lex Fridman (33:48.440)
and said, our goal is to require
Lex Fridman (33:50.560)
a thousand times more computation, but less people.
Jeremy Howard (33:55.160)
Our goal is to use the people that you have better
Lex Fridman (34:00.640)
and the data you have better
Lex Fridman (34:01.680)
and the computation you have better.
Lex Fridman (34:03.000)
So, one of the things that we've discovered is,
Jeremy Howard (34:06.040)
or at least highlighted,
Lex Fridman (34:08.000)
is that you very, very, very often
Jeremy Howard (34:11.080)
don't need much data at all.
Lex Fridman (34:13.360)
And so the data you already have in your organization
Jeremy Howard (34:16.160)
will be enough to get state of the art results.
Lex Fridman (34:19.240)
So, like my starting point would be to kind of say
Jeremy Howard (34:21.320)
around privacy is a lot of people are looking for ways
Lex Fridman (34:25.760)
to share data and aggregate data,
Lex Fridman (34:28.160)
but I think often that's unnecessary.
Lex Fridman (34:29.960)
They assume that they need more data than they do
Jeremy Howard (34:32.200)
because they're not familiar with the basics
Lex Fridman (34:34.160)
of transfer learning, which is this critical technique
Jeremy Howard (34:38.440)
for needing orders of magnitude less data.
Lex Fridman (34:42.000)
Is your sense, one reason you might wanna collect data
Jeremy Howard (34:44.680)
from everyone is like in the recommender system context,
Lex Fridman (34:50.440)
where your individual, Jeremy Howard's individual data
Jeremy Howard (34:54.520)
is the most useful for providing a product
Lex Fridman (34:58.440)
that's impactful for you.
Jeremy Howard (34:59.840)
So, for giving you advertisements,
Lex Fridman (35:02.240)
for recommending to you movies,
Jeremy Howard (35:04.160)
for doing medical diagnosis,
Lex Fridman (35:07.600)
is your sense we can build with a small amount of data,
Jeremy Howard (35:11.680)
general models that will have a huge impact
Lex Fridman (35:15.200)
for most people that we don't need to have data
Lex Fridman (35:18.280)
from each individual?
Lex Fridman (35:19.160)
On the whole, I'd say yes.
Jeremy Howard (35:20.560)
I mean, there are things like,
Lex Fridman (35:25.240)
you know, recommender systems have this cold start problem
Jeremy Howard (35:28.360)
where, you know, Jeremy is a new customer,
Lex Fridman (35:30.960)
we haven't seen him before, so we can't recommend him things
Jeremy Howard (35:33.280)
based on what else he's bought and liked with us.
Lex Fridman (35:36.000)
And there's various workarounds to that.
Jeremy Howard (35:38.840)
Like in a lot of music programs,
Lex Fridman (35:40.640)
we'll start out by saying, which of these artists do you like?
Lex Fridman (35:44.880)
Which of these albums do you like?
Lex Fridman (35:46.760)
Which of these songs do you like?
Jeremy Howard (35:49.760)
Netflix used to do that, nowadays they tend not to.
Lex Fridman (35:53.520)
People kind of don't like that
Jeremy Howard (35:54.760)
because they think, oh, we don't wanna bother the user.
Lex Fridman (35:57.320)
So, you could work around that
Jeremy Howard (35:58.680)
by having some kind of data sharing
Lex Fridman (36:00.960)
where you get my marketing record from Axiom or whatever,
Lex Fridman (36:04.880)
and try to guess from that.
Lex Fridman (36:06.560)
To me, the benefit to me and to society
Jeremy Howard (36:12.320)
of saving me five minutes on answering some questions
Lex Fridman (36:16.440)
versus the negative externalities of the privacy issue
Jeremy Howard (36:23.480)
doesn't add up.
Lex Fridman (36:24.760)
So, I think like a lot of the time,
Jeremy Howard (36:26.120)
the places where people are invading our privacy
Lex Fridman (36:30.120)
in order to provide convenience
Jeremy Howard (36:32.760)
is really about just trying to make them more money
Lex Fridman (36:36.800)
and they move these negative externalities
Jeremy Howard (36:40.720)
to places that they don't have to pay for them.
Lex Fridman (36:44.240)
So, when you actually see regulations appear
Jeremy Howard (36:48.440)
that actually cause the companies
Lex Fridman (36:50.360)
that create these negative externalities
Jeremy Howard (36:52.080)
to have to pay for it themselves,
Lex Fridman (36:53.480)
they say, well, we can't do it anymore.
Jeremy Howard (36:56.080)
So, the cost is actually too high.
Lex Fridman (36:58.160)
But for something like medicine,
Jeremy Howard (37:00.320)
yeah, I mean, the hospital has my medical imaging,
Lex Fridman (37:05.200)
my pathology studies, my medical records,
Lex Fridman (37:08.880)
and also I own my medical data.
Lex Fridman (37:11.840)
So, you can, so I help a startup called Doc.ai.
Jeremy Howard (37:16.920)
One of the things Doc.ai does is that it has an app.
Lex Fridman (37:19.680)
You can connect to, you know, Sutter Health
Lex Fridman (37:23.760)
and LabCorp and Walgreens
Lex Fridman (37:26.080)
and download your medical data to your phone
Lex Fridman (37:29.800)
and then upload it again at your discretion
Lex Fridman (37:33.520)
to share it as you wish.
Jeremy Howard (37:35.960)
So, with that kind of approach,
Lex Fridman (37:38.000)
we can share our medical information
Jeremy Howard (37:41.120)
with the people we want to.
Lex Fridman (37:44.760)
Yeah, so control.
Jeremy Howard (37:45.680)
I mean, really being able to control
Lex Fridman (37:47.440)
who you share it with and so on.
Jeremy Howard (37:48.760)
Yeah.
Lex Fridman (37:49.720)
So, that has a beautiful, interesting tangent
Jeremy Howard (37:53.480)
to return back to the origin story of Fast.ai.
Lex Fridman (37:59.360)
Right, so before I started Fast.ai,
Jeremy Howard (38:02.480)
I spent a year researching
Lex Fridman (38:06.320)
where are the biggest opportunities for deep learning?
Jeremy Howard (38:10.360)
Because I knew from my time at Kaggle in particular
Lex Fridman (38:14.040)
that deep learning had kind of hit this threshold point
Jeremy Howard (38:16.880)
where it was rapidly becoming the state of the art approach
Lex Fridman (38:19.840)
in every area that looked at it.
Lex Fridman (38:21.560)
And I'd been working with neural nets for over 20 years.
Lex Fridman (38:25.360)
I knew that from a theoretical point of view,
Jeremy Howard (38:27.400)
once it hit that point,
Lex Fridman (38:28.520)
it would do that in kind of just about every domain.
Lex Fridman (38:31.520)
And so I kind of spent a year researching
Lex Fridman (38:34.440)
what are the domains that's gonna have
Jeremy Howard (38:36.200)
the biggest low hanging fruit
Lex Fridman (38:37.360)
in the shortest time period.
Jeremy Howard (38:39.360)
I picked medicine, but there were so many
Lex Fridman (38:42.040)
I could have picked.
Lex Fridman (38:43.880)
And so there was a kind of level of frustration for me
Lex Fridman (38:46.200)
of like, okay, I'm really glad we've opened up
Jeremy Howard (38:49.920)
the medical deep learning world.
Lex Fridman (38:51.120)
And today it's huge, as you know,
Lex Fridman (38:53.880)
but we can't do, I can't do everything.
Lex Fridman (38:58.240)
I don't even know, like in medicine,
Jeremy Howard (39:00.360)
it took me a really long time to even get a sense
Lex Fridman (39:02.240)
of like what kind of problems do medical practitioners solve?
Lex Fridman (39:05.040)
What kind of data do they have?
Lex Fridman (39:06.360)
Who has that data?
Lex Fridman (39:08.480)
So I kind of felt like I need to approach this differently
Lex Fridman (39:12.440)
if I wanna maximize the positive impact of deep learning.
Jeremy Howard (39:16.200)
Rather than me picking an area
Lex Fridman (39:19.160)
and trying to become good at it and building something,
Jeremy Howard (39:21.720)
I should let people who are already domain experts
Lex Fridman (39:24.440)
in those areas and who already have the data
Jeremy Howard (39:27.760)
do it themselves.
Lex Fridman (39:29.200)
So that was the reason for Fast.ai
Jeremy Howard (39:33.080)
is to basically try and figure out
Lex Fridman (39:36.760)
how to get deep learning into the hands of people
Jeremy Howard (39:40.120)
who could benefit from it and help them to do so
Lex Fridman (39:43.240)
in as quick and easy and effective a way as possible.
Jeremy Howard (39:47.080)
Got it, so sort of empower the domain experts.
Lex Fridman (39:50.200)
Yeah, and like partly it's because like,
Jeremy Howard (39:54.240)
unlike most people in this field,
Lex Fridman (39:56.280)
my background is very applied and industrial.
Jeremy Howard (39:59.920)
Like my first job was at McKinsey & Company.
Lex Fridman (40:02.440)
I spent 10 years in management consulting.
Jeremy Howard (40:04.640)
I spend a lot of time with domain experts.
Lex Fridman (40:10.440)
So I kind of respect them and appreciate them.
Lex Fridman (40:12.760)
And I know that's where the value generation in society is.
Lex Fridman (40:16.480)
And so I also know how most of them can't code
Lex Fridman (40:21.600)
and most of them don't have the time to invest
Lex Fridman (40:26.320)
three years in a graduate degree or whatever.
Lex Fridman (40:29.320)
So I was like, how do I upskill those domain experts?
Lex Fridman (40:33.520)
I think that would be a super powerful thing,
Jeremy Howard (40:36.600)
the biggest societal impact I could have.
Lex Fridman (40:40.240)
So yeah, that was the thinking.
Lex Fridman (40:41.680)
So much of Fast.ai students and researchers
Lex Fridman (40:45.680)
and the things you teach are pragmatically minded,
Jeremy Howard (40:50.160)
practically minded,
Lex Fridman (40:52.080)
figuring out ways how to solve real problems and fast.
Lex Fridman (40:55.800)
So from your experience,
Lex Fridman (40:57.480)
what's the difference between theory
Lex Fridman (40:59.120)
and practice of deep learning?
Lex Fridman (41:03.680)
Well, most of the research in the deep learning world
Jeremy Howard (41:07.520)
is a total waste of time.
Lex Fridman (41:09.840)
Right, that's what I was getting at.
Jeremy Howard (41:11.040)
Yeah.
Lex Fridman (41:12.200)
It's a problem in science in general.
Jeremy Howard (41:16.240)
Scientists need to be published,
Lex Fridman (41:19.600)
which means they need to work on things
Jeremy Howard (41:21.480)
that their peers are extremely familiar with
Lex Fridman (41:24.080)
and can recognize in advance in that area.
Lex Fridman (41:26.200)
So that means that they all need to work on the same thing.
Lex Fridman (41:30.120)
And so it really, and the thing they work on,
Jeremy Howard (41:33.040)
there's nothing to encourage them to work on things
Lex Fridman (41:35.640)
that are practically useful.
Lex Fridman (41:38.840)
So you get just a whole lot of research,
Lex Fridman (41:41.160)
which is minor advances and stuff
Jeremy Howard (41:43.240)
that's been very highly studied
Lex Fridman (41:44.640)
and has no significant practical impact.
Jeremy Howard (41:49.360)
Whereas the things that really make a difference,
Lex Fridman (41:50.920)
like I mentioned transfer learning,
Jeremy Howard (41:52.800)
like if we can do better at transfer learning,
Lex Fridman (41:55.640)
then it's this like world changing thing
Jeremy Howard (41:58.200)
where suddenly like lots more people
Lex Fridman (41:59.800)
can do world class work with less resources and less data.
Lex Fridman (42:06.840)
But almost nobody works on that.
Lex Fridman (42:08.560)
Or another example, active learning,
Jeremy Howard (42:10.800)
which is the study of like,
Lex Fridman (42:11.920)
how do we get more out of the human beings in the loop?
Jeremy Howard (42:15.920)
That's my favorite topic.
Lex Fridman (42:17.160)
Yeah, so active learning is great,
Lex Fridman (42:18.560)
but it's almost nobody working on it
Lex Fridman (42:21.200)
because it's just not a trendy thing right now.
Jeremy Howard (42:23.840)
You know what somebody, sorry to interrupt,
Lex Fridman (42:27.080)
you're saying that nobody is publishing on active learning,
Lex Fridman (42:31.560)
but there's people inside companies,
Lex Fridman (42:33.480)
anybody who actually has to solve a problem,
Jeremy Howard (42:36.840)
they're going to innovate on active learning.
Lex Fridman (42:39.680)
Yeah, everybody kind of reinvents active learning
Jeremy Howard (42:42.120)
when they actually have to work in practice
Lex Fridman (42:43.800)
because they start labeling things and they think,
Jeremy Howard (42:46.400)
gosh, this is taking a long time and it's very expensive.
Lex Fridman (42:49.320)
And then they start thinking,
Lex Fridman (42:51.240)
well, why am I labeling everything?
Lex Fridman (42:52.640)
I'm only, the machine's only making mistakes
Jeremy Howard (42:54.840)
on those two classes.
Lex Fridman (42:56.040)
They're the hard ones.
Jeremy Howard (42:56.880)
Maybe I'll just start labeling those two classes.
Lex Fridman (42:58.880)
And then you start thinking,
Lex Fridman (43:00.360)
well, why did I do that manually?
Lex Fridman (43:01.600)
Why can't I just get the system to tell me
Lex Fridman (43:03.000)
which things are going to be hardest?
Lex Fridman (43:05.080)
It's an obvious thing to do, but yeah,
Jeremy Howard (43:08.320)
it's just like transfer learning.
Lex Fridman (43:11.440)
It's understudied and the academic world
Jeremy Howard (43:14.160)
just has no reason to care about practical results.
Lex Fridman (43:17.480)
The funny thing is,
Jeremy Howard (43:18.320)
like I've only really ever written one paper.
Lex Fridman (43:19.960)
I hate writing papers.
Lex Fridman (43:21.560)
And I didn't even write it.
Lex Fridman (43:22.800)
It was my colleague, Sebastian Ruder,
Jeremy Howard (43:24.640)
who actually wrote it.
Lex Fridman (43:25.520)
I just did the research for it,
Lex Fridman (43:28.080)
but it was basically introducing transfer learning,
Lex Fridman (43:30.600)
successful transfer learning to NLP for the first time.
Jeremy Howard (43:34.280)
The algorithm is called ULM fit.
Lex Fridman (43:36.960)
And it actually, I actually wrote it for the course,
Jeremy Howard (43:42.280)
for the Fast AI course.
Lex Fridman (43:43.680)
I wanted to teach people NLP and I thought,
Jeremy Howard (43:45.760)
I only want to teach people practical stuff.
Lex Fridman (43:47.480)
And I think the only practical stuff is transfer learning.
Lex Fridman (43:50.520)
And I couldn't find any examples of transfer learning in NLP.
Lex Fridman (43:53.280)
So I just did it.
Lex Fridman (43:54.520)
And I was shocked to find that as soon as I did it,
Lex Fridman (43:57.280)
which, you know, the basic prototype took a couple of days,
Jeremy Howard (44:01.040)
smashed the state of the art
Lex Fridman (44:02.480)
on one of the most important data sets
Jeremy Howard (44:04.240)
in a field that I knew nothing about.
Lex Fridman (44:06.680)
And I just thought, well, this is ridiculous.
Lex Fridman (44:10.320)
And so I spoke to Sebastian about it
Lex Fridman (44:13.760)
and he kindly offered to write it up, the results.
Lex Fridman (44:17.640)
And so it ended up being published in ACL,
Lex Fridman (44:21.320)
which is the top computational linguistics conference.
Lex Fridman (44:25.520)
So like people do actually care once you do it,
Lex Fridman (44:28.840)
but I guess it's difficult for maybe like junior researchers
Jeremy Howard (44:32.760)
or like, I don't care whether I get citations
Lex Fridman (44:36.560)
or papers or whatever.
Jeremy Howard (44:37.720)
There's nothing in my life that makes that important,
Lex Fridman (44:39.600)
which is why I've never actually bothered
Jeremy Howard (44:41.480)
to write a paper myself.
Lex Fridman (44:43.000)
But for people who do,
Jeremy Howard (44:43.960)
I guess they have to pick the kind of safe option,
Lex Fridman (44:49.560)
which is like, yeah, make a slight improvement
Jeremy Howard (44:52.240)
on something that everybody's already working on.
Lex Fridman (44:54.920)
Yeah, nobody does anything interesting
Jeremy Howard (44:58.240)
or succeeds in life with the safe option.
Lex Fridman (45:01.160)
Although, I mean, the nice thing is,
Jeremy Howard (45:02.400)
nowadays everybody is now working on NLP transfer learning
Lex Fridman (45:05.280)
because since that time we've had GPT and GPT2 and BERT,
Jeremy Howard (45:09.720)
and, you know, it's like, it's, so yeah,
Lex Fridman (45:12.640)
once you show that something's possible,
Jeremy Howard (45:15.360)
everybody jumps in, I guess, so.
Lex Fridman (45:17.600)
I hope to be a part of,
Lex Fridman (45:19.160)
and I hope to see more innovation
Lex Fridman (45:20.640)
and active learning in the same way.
Jeremy Howard (45:22.120)
I think transfer learning and active learning
Lex Fridman (45:24.480)
are fascinating, public, open work.
Jeremy Howard (45:27.320)
I actually helped start a startup called Platform AI,
Lex Fridman (45:29.960)
which is really all about active learning.
Lex Fridman (45:31.720)
And yeah, it's been interesting trying to kind of see
Lex Fridman (45:35.840)
what research is out there and make the most of it.
Lex Fridman (45:37.760)
And there's basically none.
Lex Fridman (45:39.160)
So we've had to do all our own research.
Jeremy Howard (45:41.000)
Once again, and just as you described.
Lex Fridman (45:44.240)
Can you tell the story of the Stanford competition,
Lex Fridman (45:47.640)
Dawn Bench, and FastAI's achievement on it?
Lex Fridman (45:51.480)
Sure, so something which I really enjoy
Jeremy Howard (45:54.280)
is that I basically teach two courses a year,
Lex Fridman (45:57.400)
the Practical Deep Learning for Coders,
Jeremy Howard (45:59.640)
which is kind of the introductory course,
Lex Fridman (46:02.080)
and then Cutting Edge Deep Learning for Coders,
Jeremy Howard (46:04.000)
which is the kind of research level course.
Lex Fridman (46:08.040)
And while I teach those courses,
Jeremy Howard (46:10.360)
I basically have a big office
Lex Fridman (46:16.760)
at the University of San Francisco,
Jeremy Howard (46:18.520)
big enough for like 30 people.
Lex Fridman (46:19.760)
And I invite anybody, any student who wants to come
Lex Fridman (46:22.080)
and hang out with me while I build the course.
Lex Fridman (46:25.320)
And so generally it's full.
Lex Fridman (46:26.600)
And so we have 20 or 30 people in a big office
Lex Fridman (46:30.840)
with nothing to do but study deep learning.
Lex Fridman (46:33.840)
So it was during one of these times
Lex Fridman (46:35.880)
that somebody in the group said,
Jeremy Howard (46:37.320)
oh, there's a thing called Dawn Bench
Lex Fridman (46:40.520)
that looks interesting.
Lex Fridman (46:41.400)
And I was like, what the hell is that?
Lex Fridman (46:42.760)
And they set out some competition
Jeremy Howard (46:44.040)
to see how quickly you can train a model.
Lex Fridman (46:46.320)
Seems kind of, not exactly relevant to what we're doing,
Lex Fridman (46:50.240)
but it sounds like the kind of thing
Lex Fridman (46:51.320)
which you might be interested in.
Lex Fridman (46:52.400)
And I checked it out and I was like,
Lex Fridman (46:53.320)
oh crap, there's only 10 days till it's over.
Jeremy Howard (46:55.760)
It's too late.
Lex Fridman (46:58.000)
And we're kind of busy trying to teach this course.
Lex Fridman (47:00.880)
But we're like, oh, it would make an interesting case study
Lex Fridman (47:05.520)
for the course.
Jeremy Howard (47:06.360)
It's like, it's all the stuff we're already doing.
Lex Fridman (47:08.160)
Why don't we just put together
Lex Fridman (47:09.480)
our current best practices and ideas?
Lex Fridman (47:12.440)
So me and I guess about four students
Jeremy Howard (47:16.040)
just decided to give it a go.
Lex Fridman (47:17.520)
And we focused on this small one called Cifar 10,
Jeremy Howard (47:20.840)
which is little 32 by 32 pixel images.
Lex Fridman (47:24.600)
Can you say what Dawn Bench is?
Jeremy Howard (47:26.080)
Yeah, so it's a competition to train a model
Lex Fridman (47:28.600)
as fast as possible.
Jeremy Howard (47:29.520)
It was run by Stanford.
Lex Fridman (47:30.960)
And it's cheap as possible too.
Jeremy Howard (47:32.480)
That's also another one for as cheap as possible.
Lex Fridman (47:34.280)
And there was a couple of categories,
Jeremy Howard (47:36.400)
ImageNet and Cifar 10.
Lex Fridman (47:38.120)
So ImageNet is this big 1.3 million image thing
Jeremy Howard (47:42.040)
that took a couple of days to train.
Lex Fridman (47:45.400)
Remember a friend of mine, Pete Warden,
Jeremy Howard (47:47.840)
who's now at Google.
Lex Fridman (47:51.240)
I remember he told me how he trained ImageNet
Jeremy Howard (47:53.240)
a few years ago when he basically like had this
Lex Fridman (47:58.320)
little granny flat out the back
Jeremy Howard (47:59.720)
that he turned into his ImageNet training center.
Lex Fridman (48:01.880)
And he figured, you know, after like a year of work,
Jeremy Howard (48:03.760)
he figured out how to train it in like 10 days or something.
Lex Fridman (48:07.040)
It's like, that was a big job.
Jeremy Howard (48:08.440)
Whereas Cifar 10, at that time,
Lex Fridman (48:10.480)
you could train in a few hours.
Jeremy Howard (48:12.840)
You know, it's much smaller and easier.
Lex Fridman (48:14.480)
So we thought we'd try Cifar 10.
Lex Fridman (48:18.120)
And yeah, I've really never done that before.
Lex Fridman (48:23.760)
Like I'd never really,
Jeremy Howard (48:24.760)
like things like using more than one GPU at a time
Lex Fridman (48:27.880)
was something I tried to avoid.
Jeremy Howard (48:29.800)
Cause to me, it's like very against the whole idea
Lex Fridman (48:32.120)
of accessibility is should better do things with one GPU.
Jeremy Howard (48:35.000)
I mean, have you asked in the past before,
Lex Fridman (48:38.000)
after having accomplished something,
Lex Fridman (48:39.640)
how do I do this faster, much faster?
Lex Fridman (48:42.480)
Oh, always, but it's always, for me,
Jeremy Howard (48:44.160)
it's always how do I make it much faster on a single GPU
Lex Fridman (48:47.680)
that a normal person could afford in their day to day life.
Jeremy Howard (48:50.360)
It's not how could I do it faster by, you know,
Lex Fridman (48:53.880)
having a huge data center.
Jeremy Howard (48:55.280)
Cause to me, it's all about like,
Lex Fridman (48:57.240)
as many people should better use something as possible
Jeremy Howard (48:59.520)
without fussing around with infrastructure.
Lex Fridman (49:04.080)
So anyways, in this case it's like, well,
Jeremy Howard (49:06.040)
we can use eight GPUs just by renting a AWS machine.
Lex Fridman (49:10.200)
So we thought we'd try that.
Lex Fridman (49:11.840)
And yeah, basically using the stuff we were already doing,
Lex Fridman (49:16.520)
we were able to get, you know, the speed,
Jeremy Howard (49:20.120)
you know, within a few days we had the speed down to,
Lex Fridman (49:23.840)
I don't know, a very small number of minutes.
Jeremy Howard (49:26.000)
I can't remember exactly how many minutes it was,
Lex Fridman (49:28.760)
but it might've been like 10 minutes or something.
Lex Fridman (49:31.360)
And so, yeah, we found ourselves
Lex Fridman (49:32.880)
at the top of the leaderboard easily
Jeremy Howard (49:34.720)
for both time and money, which really shocked me
Lex Fridman (49:39.040)
cause the other people competing in this
Jeremy Howard (49:40.160)
were like Google and Intel and stuff
Lex Fridman (49:41.880)
who I like know a lot more about this stuff
Jeremy Howard (49:43.880)
than I think we do.
Lex Fridman (49:45.360)
So then we were emboldened.
Jeremy Howard (49:46.800)
We thought let's try the ImageNet one too.
Lex Fridman (49:50.640)
I mean, it seemed way out of our league,
Lex Fridman (49:53.320)
but our goal was to get under 12 hours.
Lex Fridman (49:55.960)
And we did, which was really exciting.
Lex Fridman (49:59.520)
But we didn't put anything up on the leaderboard,
Lex Fridman (50:01.400)
but we were down to like 10 hours.
Lex Fridman (50:03.040)
But then Google put in like five hours or something
Lex Fridman (50:09.960)
and we're just like, oh, we're so screwed.
Lex Fridman (50:13.360)
But we kind of thought, we'll keep trying.
Lex Fridman (50:16.560)
You know, if Google can do it in five,
Jeremy Howard (50:17.800)
I mean, Google did on five hours on something
Lex Fridman (50:19.480)
on like a TPU pod or something, like a lot of hardware.
Lex Fridman (50:23.280)
But we kind of like had a bunch of ideas to try.
Lex Fridman (50:26.360)
Like a really simple thing was
Lex Fridman (50:28.720)
why are we using these big images?
Lex Fridman (50:30.480)
They're like 224 or 256 by 256 pixels.
Lex Fridman (50:35.400)
You know, why don't we try smaller ones?
Lex Fridman (50:37.720)
And just to elaborate, there's a constraint
Jeremy Howard (50:40.400)
on the accuracy that your trained model
Lex Fridman (50:42.200)
is supposed to achieve, right?
Jeremy Howard (50:43.040)
Yeah, you gotta achieve 93%, I think it was,
Lex Fridman (50:46.400)
for ImageNet, exactly.
Jeremy Howard (50:49.200)
Which is very tough, so you have to.
Lex Fridman (50:51.080)
Yeah, 93%, like they picked a good threshold.
Jeremy Howard (50:54.680)
It was a little bit higher
Lex Fridman (50:56.920)
than what the most commonly used ResNet 50 model
Jeremy Howard (51:00.840)
could achieve at that time.
Lex Fridman (51:03.360)
So yeah, so it's quite a difficult problem to solve.
Lex Fridman (51:08.200)
But yeah, we realized if we actually
Lex Fridman (51:09.720)
just use 64 by 64 images,
Jeremy Howard (51:14.680)
it trained a pretty good model.
Lex Fridman (51:16.280)
And then we could take that same model
Lex Fridman (51:18.040)
and just give it a couple of epochs to learn 224 by 224 images.
Lex Fridman (51:21.920)
And it was basically already trained.
Jeremy Howard (51:24.520)
It makes a lot of sense.
Lex Fridman (51:25.480)
Like if you teach somebody,
Jeremy Howard (51:26.640)
like here's what a dog looks like
Lex Fridman (51:28.120)
and you show them low res versions,
Lex Fridman (51:30.200)
and then you say, here's a really clear picture of a dog,
Lex Fridman (51:33.600)
they already know what a dog looks like.
Lex Fridman (51:35.960)
So that like just, we jumped to the front
Lex Fridman (51:39.880)
and we ended up winning parts of that competition.
Jeremy Howard (51:43.880)
We actually ended up doing a distributed version
Lex Fridman (51:47.280)
over multiple machines a couple of months later
Lex Fridman (51:49.560)
and ended up at the top of the leaderboard.
Lex Fridman (51:51.120)
We had 18 minutes.
Jeremy Howard (51:53.000)
ImageNet.
Lex Fridman (51:53.840)
Yeah, and it was,
Lex Fridman (51:55.640)
and people have just kept on blasting through
Lex Fridman (51:57.920)
again and again since then, so.
Lex Fridman (52:00.000)
So what's your view on multi GPU
Lex Fridman (52:03.200)
or multiple machine training in general
Lex Fridman (52:06.120)
as a way to speed code up?
Lex Fridman (52:09.520)
I think it's largely a waste of time.
Jeremy Howard (52:11.240)
Both of them.
Lex Fridman (52:12.080)
I think it's largely a waste of time.
Jeremy Howard (52:13.960)
Both multi GPU on a single machine and.
Lex Fridman (52:15.840)
Yeah, particularly multi machines,
Jeremy Howard (52:17.640)
cause it's just clunky.
Lex Fridman (52:21.840)
Multi GPUs is less clunky than it used to be,
Lex Fridman (52:25.320)
but to me anything that slows down your iteration speed
Lex Fridman (52:28.520)
is a waste of time.
Lex Fridman (52:31.680)
So you could maybe do your very last,
Lex Fridman (52:34.960)
you know, perfecting of the model on multi GPUs
Jeremy Howard (52:38.000)
if you need to, but.
Lex Fridman (52:40.040)
So for example, I think doing stuff on ImageNet
Jeremy Howard (52:44.560)
is generally a waste of time.
Lex Fridman (52:46.000)
Why test things on 1.3 million images?
Jeremy Howard (52:48.240)
Most of us don't use 1.3 million images.
Lex Fridman (52:51.080)
And we've also done research that shows that
Jeremy Howard (52:53.840)
doing things on a smaller subset of images
Lex Fridman (52:56.480)
gives you the same relative answers anyway.
Lex Fridman (52:59.160)
So from a research point of view, why waste that time?
Lex Fridman (53:02.080)
So actually I released a couple of new data sets recently.
Jeremy Howard (53:06.120)
One is called ImageNet,
Lex Fridman (53:07.720)
the French ImageNet, which is a small subset of ImageNet,
Jeremy Howard (53:12.880)
which is designed to be easy to classify.
Lex Fridman (53:15.040)
What's, how do you spell ImageNet?
Jeremy Howard (53:17.240)
It's got an extra T and E at the end,
Lex Fridman (53:19.200)
cause it's very French.
Lex Fridman (53:20.440)
And then another one called ImageWolf,
Lex Fridman (53:24.680)
which is a subset of ImageNet that only contains dog breeds.
Lex Fridman (53:29.960)
And that's a hard one, right?
Lex Fridman (53:31.080)
That's a hard one.
Lex Fridman (53:31.960)
And I've discovered that if you just look at these
Lex Fridman (53:34.120)
two subsets, you can train things on a single GPU
Jeremy Howard (53:37.760)
in 10 minutes.
Lex Fridman (53:39.080)
And the results you get are directly transferable
Jeremy Howard (53:42.040)
to ImageNet nearly all the time.
Lex Fridman (53:44.280)
And so now I'm starting to see some researchers
Jeremy Howard (53:46.360)
start to use these much smaller data sets.
Lex Fridman (53:48.960)
I so deeply love the way you think,
Jeremy Howard (53:51.120)
because I think you might've written a blog post
Lex Fridman (53:55.040)
saying that sort of going these big data sets
Jeremy Howard (54:00.120)
is encouraging people to not think creatively.
Lex Fridman (54:03.840)
Absolutely.
Lex Fridman (54:04.680)
So you're too, it sort of constrains you to train
Lex Fridman (54:08.760)
on large resources.
Lex Fridman (54:09.800)
And because you have these resources,
Lex Fridman (54:11.240)
you think more research will be better.
Lex Fridman (54:13.960)
And then you start, so like somehow you kill the creativity.
Lex Fridman (54:17.720)
Yeah, and even worse than that, Lex,
Jeremy Howard (54:19.240)
I keep hearing from people who say,
Lex Fridman (54:21.160)
I decided not to get into deep learning
Jeremy Howard (54:23.320)
because I don't believe it's accessible to people
Lex Fridman (54:26.040)
outside of Google to do useful work.
Lex Fridman (54:28.480)
So like I see a lot of people make an explicit decision
Lex Fridman (54:31.600)
to not learn this incredibly valuable tool
Jeremy Howard (54:35.960)
because they've drunk the Google Koolaid,
Lex Fridman (54:39.000)
which is that only Google's big enough
Lex Fridman (54:40.680)
and smart enough to do it.
Lex Fridman (54:42.400)
And I just find that so disappointing and it's so wrong.
Lex Fridman (54:45.320)
And I think all of the major breakthroughs in AI
Lex Fridman (54:49.120)
in the next 20 years will be doable on a single GPU.
Jeremy Howard (54:53.240)
Like I would say, my sense is all the big sort of.
Lex Fridman (54:57.360)
Well, let's put it this way.
Jeremy Howard (54:58.200)
None of the big breakthroughs of the last 20 years
Lex Fridman (55:00.120)
have required multiple GPUs.
Lex Fridman (55:01.680)
So like batch norm, ReLU, Dropout.
Lex Fridman (55:05.920)
To demonstrate that there's something to them.
Jeremy Howard (55:08.040)
Every one of them, none of them has required multiple GPUs.
Lex Fridman (55:11.760)
GANs, the original GANs didn't require multiple GPUs.
Jeremy Howard (55:15.760)
Well, and we've actually recently shown
Lex Fridman (55:18.000)
that you don't even need GANs.
Lex Fridman (55:19.600)
So we've developed GAN level outcomes without needing GANs.
Lex Fridman (55:24.640)
And we can now do it with, again,
Jeremy Howard (55:26.840)
by using transfer learning,
Lex Fridman (55:27.920)
we can do it in a couple of hours on a single GPU.
Jeremy Howard (55:30.200)
You're just using a generator model
Lex Fridman (55:31.600)
without the adversarial part?
Jeremy Howard (55:32.960)
Yeah, so we've found loss functions
Lex Fridman (55:35.680)
that work super well without the adversarial part.
Lex Fridman (55:38.640)
And then one of our students, a guy called Jason Antich,
Lex Fridman (55:41.800)
has created a system called dealtify,
Jeremy Howard (55:44.600)
which uses this technique to colorize
Lex Fridman (55:47.240)
old black and white movies.
Jeremy Howard (55:48.800)
You can do it on a single GPU,
Lex Fridman (55:50.440)
colorize a whole movie in a couple of hours.
Lex Fridman (55:52.840)
And one of the things that Jason and I did together
Lex Fridman (55:56.040)
was we figured out how to add a little bit of GAN
Jeremy Howard (56:00.400)
at the very end, which it turns out for colorization
Lex Fridman (56:02.920)
makes it just a bit brighter and nicer.
Lex Fridman (56:05.920)
And then Jason did masses of experiments
Lex Fridman (56:07.880)
to figure out exactly how much to do,
Lex Fridman (56:09.960)
but it's still all done on his home machine
Lex Fridman (56:12.760)
on a single GPU in his lounge room.
Lex Fridman (56:15.320)
And if you think about colorizing Hollywood movies,
Lex Fridman (56:19.160)
that sounds like something a huge studio would have to do,
Lex Fridman (56:21.680)
but he has the world's best results on this.
Lex Fridman (56:25.160)
There's this problem of microphones.
Jeremy Howard (56:27.000)
We're just talking to microphones now.
Lex Fridman (56:29.080)
It's such a pain in the ass to have these microphones
Jeremy Howard (56:32.520)
to get good quality audio.
Lex Fridman (56:34.360)
And I tried to see if it's possible to plop down
Jeremy Howard (56:36.680)
a bunch of cheap sensors and reconstruct
Lex Fridman (56:39.200)
higher quality audio from multiple sources.
Jeremy Howard (56:41.840)
Because right now I haven't seen the work from,
Lex Fridman (56:45.160)
okay, we can say even expensive mics
Jeremy Howard (56:47.440)
automatically combining audio from multiple sources
Lex Fridman (56:50.040)
to improve the combined audio.
Jeremy Howard (56:52.280)
People haven't done that.
Lex Fridman (56:53.120)
And that feels like a learning problem.
Lex Fridman (56:55.080)
So hopefully somebody can.
Lex Fridman (56:56.840)
Well, I mean, it's evidently doable
Lex Fridman (56:58.800)
and it should have been done by now.
Lex Fridman (57:01.400)
I felt the same way about computational photography
Jeremy Howard (57:03.600)
four years ago.
Lex Fridman (57:05.240)
Why are we investing in big lenses
Jeremy Howard (57:07.120)
when three cheap lenses plus actually
Lex Fridman (57:10.640)
a little bit of intentional movement,
Lex Fridman (57:13.760)
so like take a few frames,
Lex Fridman (57:16.640)
gives you enough information
Jeremy Howard (57:18.280)
to get excellent subpixel resolution,
Lex Fridman (57:20.560)
which particularly with deep learning,
Jeremy Howard (57:22.440)
you would know exactly what you meant to be looking at.
Lex Fridman (57:25.840)
We can totally do the same thing with audio.
Jeremy Howard (57:28.160)
I think it's madness that it hasn't been done yet.
Lex Fridman (57:30.680)
Is there progress on the photography company?
Jeremy Howard (57:33.280)
Yeah, photography is basically standard now.
Lex Fridman (57:36.720)
So the Google Pixel Night Light,
Jeremy Howard (57:40.800)
I don't know if you've ever tried it,
Lex Fridman (57:42.120)
but it's astonishing.
Jeremy Howard (57:43.200)
You take a picture in almost pitch black
Lex Fridman (57:45.440)
and you get back a very high quality image.
Lex Fridman (57:49.160)
And it's not because of the lens.
Lex Fridman (57:51.480)
Same stuff with like adding the bokeh
Jeremy Howard (57:53.440)
to the background blurring,
Lex Fridman (57:55.800)
it's done computationally.
Jeremy Howard (57:57.200)
This is the pixel right here.
Lex Fridman (57:58.600)
Yeah, basically everybody now
Jeremy Howard (58:01.880)
is doing most of the fanciest stuff
Lex Fridman (58:05.000)
on their phones with computational photography
Lex Fridman (58:07.120)
and also increasingly people are putting
Lex Fridman (58:08.680)
more than one lens on the back of the camera.
Lex Fridman (58:11.800)
So the same will happen for audio for sure.
Lex Fridman (58:14.360)
And there's applications in the audio side.
Jeremy Howard (58:16.480)
If you look at an Alexa type device,
Lex Fridman (58:19.320)
most people I've seen,
Jeremy Howard (58:20.840)
especially I worked at Google before,
Lex Fridman (58:22.320)
when you look at noise background removal,
Jeremy Howard (58:25.920)
you don't think of multiple sources of audio.
Lex Fridman (58:29.560)
You don't play with that as much
Jeremy Howard (58:31.040)
as I would hope people would.
Lex Fridman (58:31.880)
But I mean, you can still do it even with one.
Jeremy Howard (58:33.600)
Like again, not much work's been done in this area.
Lex Fridman (58:36.120)
So we're actually gonna be releasing an audio library soon,
Jeremy Howard (58:39.000)
which hopefully will encourage development of this
Lex Fridman (58:41.040)
because it's so underused.
Jeremy Howard (58:43.160)
The basic approach we used for our super resolution
Lex Fridman (58:46.480)
and which Jason uses for dealtify
Jeremy Howard (58:48.640)
of generating high quality images,
Lex Fridman (58:50.960)
the exact same approach would work for audio.
Jeremy Howard (58:53.440)
No one's done it yet,
Lex Fridman (58:54.440)
but it would be a couple of months work.
Jeremy Howard (58:57.120)
Okay, also learning rate in terms of Dawn Bench.
Lex Fridman (59:01.560)
There's some magic on learning rate
Jeremy Howard (59:03.520)
that you played around with that's kind of interesting.
Lex Fridman (59:05.720)
Yeah, so this is all work that came
Jeremy Howard (59:06.960)
from a guy called Leslie Smith.
Lex Fridman (59:09.280)
Leslie's a researcher who, like us,
Jeremy Howard (59:12.720)
cares a lot about just the practicalities
Lex Fridman (59:15.800)
of training neural networks quickly and accurately,
Jeremy Howard (59:20.360)
which I think is what everybody should care about,
Lex Fridman (59:22.120)
but almost nobody does.
Lex Fridman (59:24.920)
And he discovered something very interesting,
Lex Fridman (59:28.080)
which he calls super convergence,
Jeremy Howard (59:29.760)
which is there are certain networks
Lex Fridman (59:31.240)
that with certain settings of high parameters
Jeremy Howard (59:33.320)
could suddenly be trained 10 times faster
Lex Fridman (59:37.080)
by using a 10 times higher learning rate.
Jeremy Howard (59:39.480)
Now, no one published that paper
Lex Fridman (59:43.640)
because it's not an area of kind of active research
Jeremy Howard (59:49.520)
in the academic world.
Lex Fridman (59:50.440)
No academics recognize that this is important.
Lex Fridman (59:52.640)
And also deep learning in academia
Lex Fridman (59:56.080)
is not considered a experimental science.
Lex Fridman (59:59.840)
So unlike in physics where you could say like,
🔗 相关节目