Chris Lattner: Compilers, LLVM, Swift, TPU, and ML Accelerators
技术与编程音乐与艺术历史与文明政治与社会AI 与机器学习
📋 章节目录
暂无章节信息
🔑 关键词
codecompilerllvmswiftlanguagehardwarepythondontensorflowcompilerssoftwaremachinelearninghardgoingstartedwaysappleobjectivekinds
💬 精彩语录
暂无语录
🎙️ 完整对话（1718 条）
Lex Fridman (00:00.000)
The following is a conversation with Chris Latner.
以下是与克里斯·拉特纳的对话。
Lex Fridman (00:02.680)
Currently, he's a senior director
目前，他是一名高级总监
Lex Fridman (00:04.560)
at Google working on several projects, including CPU, GPU,
在 Google 从事多个项目，包括 CPU、GPU、
Lex Fridman (00:08.400)
TPU accelerators for TensorFlow, Swift for TensorFlow,
适用于 TensorFlow 的 TPU 加速器、适用于 TensorFlow 的 Swift、
Lex Fridman (00:12.040)
and all kinds of machine learning compiler magic
以及各种机器学习编译魔法
Chris Lattner (00:14.400)
going on behind the scenes.
正在幕后进行。
Lex Fridman (00:16.360)
He's one of the top experts in the world
他是世界上最顶尖的专家之一
Chris Lattner (00:18.440)
on compiler technologies, which means he deeply
对编译器技术的研究，这意味着他深入
Lex Fridman (00:21.160)
understands the intricacies of how hardware and software come
了解硬件和软件的复杂性
Chris Lattner (00:25.560)
together to create efficient code.
共同创建高效的代码。
Lex Fridman (00:27.920)
He created the LLVM compiler infrastructure project
他创建了 LLVM 编译器基础设施项目
Lex Fridman (00:31.400)
and the Clang compiler.
和 Clang 编译器。
Lex Fridman (00:33.360)
He led major engineering efforts at Apple,
他领导了苹果公司的主要工程工作，
Chris Lattner (00:36.000)
including the creation of the Swift programming language.
包括创建 Swift 编程语言。
Lex Fridman (00:39.000)
He also briefly spent time at Tesla
他还曾短暂在特斯拉工作过一段时间
Chris Lattner (00:41.720)
as vice president of Autopilot software
担任自动驾驶软件副总裁
Lex Fridman (00:44.280)
during the transition from Autopilot hardware 1
从 Autopilot 硬件 1 过渡期间
Chris Lattner (00:46.760)
to hardware 2, when Tesla essentially
到硬件 2，当特斯拉本质上
Lex Fridman (00:49.600)
started from scratch to build an in house software
从头开始构建内部软件
Chris Lattner (00:52.640)
infrastructure for Autopilot.
自动驾驶仪的基础设施。
Lex Fridman (00:54.800)
I could have easily talked to Chris for many more hours.
Chris Lattner (00:58.040)
Compiling code down across the levels of abstraction
Lex Fridman (01:01.200)
is one of the most fundamental and fascinating aspects
Chris Lattner (01:04.160)
of what computers do, and he is one of the world
Lex Fridman (01:06.640)
experts in this process.
Chris Lattner (01:08.560)
It's rigorous science, and it's messy, beautiful art.
Lex Fridman (01:12.880)
This conversation is part of the Artificial Intelligence
Chris Lattner (01:15.920)
podcast.
Lex Fridman (01:16.760)
If you enjoy it, subscribe on YouTube, iTunes,
Chris Lattner (01:19.440)
or simply connect with me on Twitter at Lex Friedman,
Lex Fridman (01:22.760)
spelled F R I D.
Lex Fridman (01:24.680)
And now, here's my conversation with Chris Ladner.
Lex Fridman (01:29.360)
What was the first program you've ever written?
Chris Lattner (01:33.160)
My first program.
Lex Fridman (01:34.120)
Back, and when was it?
Chris Lattner (01:35.360)
I think I started as a kid, and my parents
Lex Fridman (01:39.080)
got a basic programming book.
Lex Fridman (01:41.560)
And so when I started, it was typing out programs
Lex Fridman (01:44.200)
from a book, and seeing how they worked,
Lex Fridman (01:46.880)
and then typing them in wrong, and trying
Lex Fridman (01:49.680)
to figure out why they were not working right,
Chris Lattner (01:51.680)
that kind of stuff.
Lex Fridman (01:52.960)
So BASIC, what was the first language
Chris Lattner (01:54.880)
that you remember yourself maybe falling in love with,
Lex Fridman (01:58.360)
like really connecting with?
Chris Lattner (01:59.720)
I don't know.
Lex Fridman (02:00.400)
I mean, I feel like I've learned a lot along the way,
Lex Fridman (02:02.680)
and each of them have a different special thing
Lex Fridman (02:05.800)
about them.
Lex Fridman (02:06.640)
So I started in BASIC, and then went like GW BASIC,
Lex Fridman (02:09.720)
which was the thing back in the DOS days,
Lex Fridman (02:11.440)
and then upgraded to QBASIC, and eventually QuickBASIC,
Lex Fridman (02:15.280)
which are all slightly more fancy versions of Microsoft
Chris Lattner (02:18.200)
BASIC.
Lex Fridman (02:19.440)
Made the jump to Pascal, and started
Chris Lattner (02:21.360)
doing machine language programming and assembly
Lex Fridman (02:23.920)
in Pascal, which was really cool.
Chris Lattner (02:25.280)
Turbo Pascal was amazing for its day.
Lex Fridman (02:28.080)
Eventually got into C, C++, and then kind of did
Chris Lattner (02:31.600)
lots of other weird things.
Lex Fridman (02:33.400)
I feel like you took the dark path, which is the,
Chris Lattner (02:37.080)
you could have gone Lisp.
Lex Fridman (02:39.480)
Yeah.
Chris Lattner (02:40.000)
You could have gone higher level sort
Lex Fridman (02:41.680)
of functional philosophical hippie route.
Chris Lattner (02:44.600)
Instead, you went into like the dark arts of the C.
Lex Fridman (02:48.080)
It was straight into the machine.
Chris Lattner (02:49.720)
Straight to the machine.
Lex Fridman (02:50.680)
So I started with BASIC, Pascal, and then Assembly,
Lex Fridman (02:53.880)
and then wrote a lot of Assembly.
Lex Fridman (02:55.320)
And I eventually did Smalltalk and other things like that.
Lex Fridman (03:00.080)
But that was not the starting point.
Lex Fridman (03:01.880)
But so what is this journey to C?
Lex Fridman (03:05.080)
Is that in high school?
Lex Fridman (03:06.320)
Is that in college?
Chris Lattner (03:07.560)
That was in high school, yeah.
Lex Fridman (03:09.320)
And then that was really about trying
Chris Lattner (03:13.720)
to be able to do more powerful things than what Pascal could
Lex Fridman (03:16.240)
do, and also to learn a different world.
Lex Fridman (03:18.960)
So he was really confusing to me with pointers
Lex Fridman (03:20.760)
and the syntax and everything, and it took a while.
Lex Fridman (03:23.000)
But Pascal's much more principled in various ways.
Lex Fridman (03:28.800)
C is more, I mean, it has its historical roots,
Lex Fridman (03:33.400)
but it's not as easy to learn.
Lex Fridman (03:35.520)
With pointers, there's this memory management thing
Chris Lattner (03:39.880)
that you have to become conscious of.
Lex Fridman (03:41.680)
Is that the first time you start to understand
Lex Fridman (03:43.880)
that there's resources that you're supposed to manage?
Lex Fridman (03:46.520)
Well, so you have that in Pascal as well.
Lex Fridman (03:48.480)
But in Pascal, like the caret instead of the star,
Lex Fridman (03:51.440)
there's some small differences like that.
Lex Fridman (03:53.160)
But it's not about pointer arithmetic.
Lex Fridman (03:55.680)
And in C, you end up thinking about how things get
Chris Lattner (03:58.760)
laid out in memory a lot more.
Lex Fridman (04:00.840)
And so in Pascal, you have allocating and deallocating
Lex Fridman (04:04.160)
and owning the memory, but just the programs are simpler,
Lex Fridman (04:07.560)
and you don't have to.
Chris Lattner (04:10.080)
Well, for example, Pascal has a string type.
Lex Fridman (04:12.640)
And so you can think about a string
Chris Lattner (04:14.040)
instead of an array of characters
Lex Fridman (04:15.880)
which are consecutive in memory.
Lex Fridman (04:17.720)
So it's a little bit of a higher level abstraction.
Lex Fridman (04:20.400)
So let's get into it.
Chris Lattner (04:22.800)
Let's talk about LLVM, C lang, and compilers.
Lex Fridman (04:25.560)
Sure.
Lex Fridman (04:26.560)
So can you tell me first what LLVM and C lang are?
Lex Fridman (04:32.160)
And how is it that you find yourself
Chris Lattner (04:33.960)
the creator and lead developer, one
Lex Fridman (04:35.720)
of the most powerful compiler optimization systems
Lex Fridman (04:39.400)
in use today?
Lex Fridman (04:40.080)
Sure.
Lex Fridman (04:40.580)
So I guess they're different things.
Lex Fridman (04:43.320)
So let's start with what is a compiler?
Lex Fridman (04:47.080)
Is that a good place to start?
Lex Fridman (04:48.840)
What are the phases of a compiler?
Lex Fridman (04:50.200)
Where are the parts?
Lex Fridman (04:50.920)
Yeah, what is it?
Lex Fridman (04:51.600)
So what is even a compiler used for?
Lex Fridman (04:53.400)
So the way I look at this is you have a two sided problem of you
Chris Lattner (04:57.880)
have humans that need to write code.
Lex Fridman (05:00.120)
And then you have machines that need to run
Chris Lattner (05:01.880)
the program that the human wrote.
Lex Fridman (05:03.400)
And for lots of reasons, the humans
Chris Lattner (05:05.280)
don't want to be writing in binary
Lex Fridman (05:07.040)
and want to think about every piece of hardware.
Lex Fridman (05:09.080)
And so at the same time that you have lots of humans,
Lex Fridman (05:12.100)
you also have lots of kinds of hardware.
Lex Fridman (05:14.800)
And so compilers are the art of allowing
Lex Fridman (05:17.400)
humans to think at a level of abstraction
Chris Lattner (05:19.240)
that they want to think about.
Lex Fridman (05:20.920)
And then get that program, get the thing that they wrote,
Chris Lattner (05:23.600)
to run on a specific piece of hardware.
Lex Fridman (05:26.080)
And the interesting and exciting part of all this
Chris Lattner (05:29.480)
is that there's now lots of different kinds of hardware,
Lex Fridman (05:32.080)
chips like x86 and PowerPC and ARM and things like that.
Lex Fridman (05:35.780)
But also high performance accelerators
Lex Fridman (05:37.320)
for machine learning and other things like that
Chris Lattner (05:38.900)
are also just different kinds of hardware, GPUs.
Lex Fridman (05:41.520)
These are new kinds of hardware.
Lex Fridman (05:42.940)
And at the same time, on the programming side of it,
Lex Fridman (05:45.640)
you have basic, you have C, you have JavaScript,
Chris Lattner (05:48.680)
you have Python, you have Swift.
Lex Fridman (05:50.560)
You have lots of other languages
Chris Lattner (05:52.840)
that are all trying to talk to the human in a different way
Lex Fridman (05:55.200)
to make them more expressive and capable and powerful.
Lex Fridman (05:58.320)
And so compilers are the thing
Lex Fridman (06:01.500)
that goes from one to the other.
Chris Lattner (06:03.460)
End to end, from the very beginning to the very end.
Lex Fridman (06:05.200)
End to end.
Lex Fridman (06:06.040)
And so you go from what the human wrote
Lex Fridman (06:08.120)
and programming languages end up being about
Chris Lattner (06:11.600)
expressing intent, not just for the compiler
Lex Fridman (06:14.560)
and the hardware, but the programming language's job
Chris Lattner (06:17.980)
is really to capture an expression
Lex Fridman (06:20.920)
of what the programmer wanted
Chris Lattner (06:22.680)
that then can be maintained and adapted
Lex Fridman (06:25.120)
and evolved by other humans,
Chris Lattner (06:27.120)
as well as interpreted by the compiler.
Lex Fridman (06:29.720)
So when you look at this problem,
Chris Lattner (06:31.560)
you have, on the one hand, humans, which are complicated.
Lex Fridman (06:34.200)
And you have hardware, which is complicated.
Lex Fridman (06:36.760)
And so compilers typically work in multiple phases.
Lex Fridman (06:39.900)
And so the software engineering challenge
Chris Lattner (06:42.760)
that you have here is try to get maximum reuse
Lex Fridman (06:45.000)
out of the amount of code that you write,
Chris Lattner (06:47.140)
because these compilers are very complicated.
Lex Fridman (06:49.800)
And so the way it typically works out
Chris Lattner (06:51.240)
is that you have something called a front end or a parser
Lex Fridman (06:54.480)
that is language specific.
Lex Fridman (06:56.640)
And so you'll have a C parser, and that's what Clang is,
Lex Fridman (07:00.400)
or C++ or JavaScript or Python or whatever.
Chris Lattner (07:03.480)
That's the front end.
Lex Fridman (07:05.000)
Then you'll have a middle part,
Chris Lattner (07:07.120)
which is often the optimizer.
Lex Fridman (07:09.020)
And then you'll have a late part,
Chris Lattner (07:11.120)
which is hardware specific.
Lex Fridman (07:13.320)
And so compilers end up,
Chris Lattner (07:15.020)
there's many different layers often,
Lex Fridman (07:16.680)
but these three big groups are very common in compilers.
Lex Fridman (07:20.860)
And what LLVM is trying to do
Lex Fridman (07:22.200)
is trying to standardize that middle and last part.
Lex Fridman (07:25.360)
And so one of the cool things about LLVM
Lex Fridman (07:27.880)
is that there are a lot of different languages
Chris Lattner (07:29.740)
that compile through to it.
Lex Fridman (07:31.080)
And so things like Swift, but also Julia, Rust,
Chris Lattner (07:35.600)
Clang for C, C++, Subjective C,
Lex Fridman (07:39.140)
like these are all very different languages
Lex Fridman (07:40.940)
and they can all use the same optimization infrastructure,
Lex Fridman (07:43.780)
which gets better performance,
Lex Fridman (07:45.340)
and the same code generation infrastructure
Lex Fridman (07:47.240)
for hardware support.
Lex Fridman (07:48.780)
And so LLVM is really that layer that is common,
Lex Fridman (07:52.240)
that all these different specific compilers can use.
Lex Fridman (07:55.580)
And is it a standard, like a specification,
Lex Fridman (07:59.300)
or is it literally an implementation?
Chris Lattner (08:01.140)
It's an implementation.
Lex Fridman (08:02.140)
And so I think there's a couple of different ways
Lex Fridman (08:05.900)
of looking at it, right?
Lex Fridman (08:06.740)
Because it depends on which angle you're looking at it from.
Lex Fridman (08:09.700)
LLVM ends up being a bunch of code, okay?
Lex Fridman (08:12.660)
So it's a bunch of code that people reuse
Lex Fridman (08:14.460)
and they build compilers with.
Lex Fridman (08:16.540)
We call it a compiler infrastructure
Chris Lattner (08:18.060)
because it's kind of the underlying platform
Lex Fridman (08:20.060)
that you build a concrete compiler on top of.
Lex Fridman (08:22.580)
But it's also a community.
Lex Fridman (08:23.740)
And the LLVM community is hundreds of people
Chris Lattner (08:26.820)
that all collaborate.
Lex Fridman (08:27.980)
And one of the most fascinating things about LLVM
Chris Lattner (08:30.620)
over the course of time is that we've managed somehow
Lex Fridman (08:34.260)
to successfully get harsh competitors
Chris Lattner (08:37.060)
in the commercial space to collaborate
Lex Fridman (08:39.060)
on shared infrastructure.
Lex Fridman (08:41.120)
And so you have Google and Apple,
Lex Fridman (08:43.900)
you have AMD and Intel,
Chris Lattner (08:45.860)
you have Nvidia and AMD on the graphics side,
Lex Fridman (08:48.860)
you have Cray and everybody else doing these things.
Lex Fridman (08:52.620)
And all these companies are collaborating together
Lex Fridman (08:55.420)
to make that shared infrastructure really, really great.
Lex Fridman (08:58.520)
And they do this not out of the goodness of their heart,
Lex Fridman (09:01.380)
but they do it because it's in their commercial interest
Chris Lattner (09:03.420)
of having really great infrastructure
Lex Fridman (09:05.140)
that they can build on top of
Lex Fridman (09:06.740)
and facing the reality that it's so expensive
Lex Fridman (09:09.080)
that no one company, even the big companies,
Chris Lattner (09:11.160)
no one company really wants to implement it all themselves.
Lex Fridman (09:14.580)
Expensive or difficult?
Chris Lattner (09:16.100)
Both.
Lex Fridman (09:16.940)
That's a great point because it's also about the skill sets.
Lex Fridman (09:20.540)
And the skill sets are very hard to find.
Lex Fridman (09:26.020)
How big is the LLVM?
Chris Lattner (09:27.980)
It always seems like with open source projects,
Lex Fridman (09:30.780)
the kind, an LLVM is open source?
Chris Lattner (09:33.500)
Yes, it's open source.
Lex Fridman (09:34.420)
It's about, it's 19 years old now, so it's fairly old.
Chris Lattner (09:38.660)
It seems like the magic often happens
Lex Fridman (09:40.940)
within a very small circle of people.
Chris Lattner (09:43.020)
Yes.
Lex Fridman (09:43.860)
At least their early birth and whatever.
Chris Lattner (09:46.060)
Yes, so the LLVM came from a university project,
Lex Fridman (09:49.660)
and so I was at the University of Illinois.
Lex Fridman (09:51.540)
And there it was myself, my advisor,
Lex Fridman (09:53.900)
and then a team of two or three research students
Chris Lattner (09:57.500)
in the research group,
Lex Fridman (09:58.380)
and we built many of the core pieces initially.
Chris Lattner (10:02.100)
I then graduated and went to Apple,
Lex Fridman (10:03.740)
and at Apple brought it to the products,
Chris Lattner (10:06.480)
first in the OpenGL graphics stack,
Lex Fridman (10:09.340)
but eventually to the C compiler realm,
Lex Fridman (10:11.580)
and eventually built Clang,
Lex Fridman (10:12.780)
and eventually built Swift and these things.
Chris Lattner (10:14.640)
Along the way, building a team of people
Lex Fridman (10:16.380)
that are really amazing compiler engineers
Chris Lattner (10:18.620)
that helped build a lot of that.
Lex Fridman (10:20.060)
And so as it was gaining momentum
Lex Fridman (10:21.860)
and as Apple was using it, being open source and public
Lex Fridman (10:24.780)
and encouraging contribution,
Chris Lattner (10:26.440)
many others, for example, at Google,
Lex Fridman (10:28.780)
came in and started contributing.
Lex Fridman (10:30.220)
And in some cases, Google effectively owns Clang now
Lex Fridman (10:33.740)
because it cares so much about C++
Lex Fridman (10:35.540)
and the evolution of that ecosystem,
Lex Fridman (10:37.340)
and so it's investing a lot in the C++ world
Lex Fridman (10:41.420)
and the tooling and things like that.
Lex Fridman (10:42.980)
And so likewise, NVIDIA cares a lot about CUDA.
Lex Fridman (10:47.860)
And so CUDA uses Clang and uses LLVM
Lex Fridman (10:50.780)
for graphics and GPGPU.
Lex Fridman (10:54.060)
And so when you first started as a master's project,
Lex Fridman (10:58.940)
I guess, did you think it was gonna go as far as it went?
Lex Fridman (11:02.980)
Were you crazy ambitious about it?
Lex Fridman (11:06.340)
No.
Chris Lattner (11:07.180)
It seems like a really difficult undertaking, a brave one.
Lex Fridman (11:09.840)
Yeah, no, no, no, it was nothing like that.
Lex Fridman (11:11.380)
So my goal when I went to the University of Illinois
Lex Fridman (11:13.740)
was to get in and out with a non thesis masters in a year
Lex Fridman (11:17.540)
and get back to work.
Lex Fridman (11:18.720)
So I was not planning to stay for five years
Lex Fridman (11:22.200)
and build this massive infrastructure.
Lex Fridman (11:24.460)
I got nerd sniped into staying.
Lex Fridman (11:27.380)
And a lot of it was because LLVM was fun
Lex Fridman (11:29.580)
and I was building cool stuff
Lex Fridman (11:30.900)
and learning really interesting things
Lex Fridman (11:33.420)
and facing both software engineering challenges,
Lex Fridman (11:36.900)
but also learning how to work in a team
Lex Fridman (11:38.540)
and things like that.
Chris Lattner (11:40.100)
I had worked at many companies as interns before that,
Lex Fridman (11:43.620)
but it was really a different thing
Chris Lattner (11:45.860)
to have a team of people that are working together
Lex Fridman (11:48.060)
and try and collaborate in version control.
Lex Fridman (11:50.460)
And it was just a little bit different.
Lex Fridman (11:52.420)
Like I said, I just talked to Don Knuth
Lex Fridman (11:54.060)
and he believes that 2% of the world population
Lex Fridman (11:56.860)
have something weird with their brain,
Chris Lattner (11:58.820)
that they're geeks, they understand computers,
Lex Fridman (12:01.100)
they're connected with computers.
Chris Lattner (12:02.580)
He put it at exactly 2%.
Lex Fridman (12:04.380)
Okay, so.
Chris Lattner (12:05.540)
He's a specific guy.
Lex Fridman (12:06.580)
It's very specific.
Chris Lattner (12:08.780)
Well, he says, I can't prove it,
Lex Fridman (12:10.180)
but it's very empirically there.
Chris Lattner (12:13.180)
Is there something that attracts you
Lex Fridman (12:14.500)
to the idea of optimizing code?
Lex Fridman (12:16.940)
And he seems like that's one of the biggest,
Lex Fridman (12:19.180)
coolest things about LLVM.
Chris Lattner (12:20.900)
Yeah, that's one of the major things it does.
Lex Fridman (12:22.500)
So I got into that because of a person, actually.
Lex Fridman (12:26.460)
So when I was in my undergraduate,
Lex Fridman (12:28.220)
I had an advisor, or a professor named Steve Vegdahl.
Lex Fridman (12:32.060)
And he, I went to this little tiny private school.
Lex Fridman (12:35.740)
There were like seven or nine people
Chris Lattner (12:38.300)
in my computer science department,
Lex Fridman (12:40.340)
students in my class.
Lex Fridman (12:43.100)
So it was a very tiny, very small school.
Lex Fridman (12:47.460)
It was kind of a wart on the side of the math department
Chris Lattner (12:49.940)
kind of a thing at the time.
Lex Fridman (12:51.260)
I think it's evolved a lot in the many years since then.
Lex Fridman (12:53.820)
But Steve Vegdahl was a compiler guy.
Lex Fridman (12:58.300)
And he was super passionate.
Lex Fridman (12:59.580)
And his passion rubbed off on me.
Lex Fridman (13:02.740)
And one of the things I like about compilers
Chris Lattner (13:04.460)
is that they're large, complicated software pieces.
Lex Fridman (13:09.100)
And so one of the culminating classes
Chris Lattner (13:12.940)
that many computer science departments,
Lex Fridman (13:14.540)
at least at the time, did was to say
Chris Lattner (13:16.700)
that you would take algorithms and data structures
Lex Fridman (13:18.380)
and all these core classes.
Lex Fridman (13:19.460)
But then the compilers class was one of the last classes
Lex Fridman (13:21.740)
you take because it pulls everything together.
Lex Fridman (13:24.380)
And then you work on one piece of code
Lex Fridman (13:26.980)
over the entire semester.
Lex Fridman (13:28.700)
And so you keep building on your own work,
Lex Fridman (13:32.180)
which is really interesting.
Lex Fridman (13:33.460)
And it's also very challenging because in many classes,
Lex Fridman (13:36.060)
if you don't get a project done, you just forget about it
Lex Fridman (13:38.380)
and move on to the next one and get your B or whatever it is.
Lex Fridman (13:41.300)
But here you have to live with the decisions you make
Lex Fridman (13:43.860)
and continue to reinvest in it.
Lex Fridman (13:45.220)
And I really like that.
Lex Fridman (13:48.500)
And so I did an extra study project
Lex Fridman (13:50.700)
with him the following semester.
Lex Fridman (13:52.420)
And he was just really great.
Lex Fridman (13:53.940)
And he was also a great mentor in a lot of ways.
Lex Fridman (13:56.860)
And so from him and from his advice,
Lex Fridman (13:59.500)
he encouraged me to go to graduate school.
Chris Lattner (14:01.380)
I wasn't super excited about going to grad school.
Lex Fridman (14:03.420)
I wanted the master's degree, but I
Chris Lattner (14:05.540)
didn't want to be an academic.
Lex Fridman (14:08.940)
But like I said, I kind of got tricked into saying
Lex Fridman (14:11.100)
and was having a lot of fun.
Lex Fridman (14:12.180)
And I definitely do not regret it.
Lex Fridman (14:14.540)
What aspects of compilers were the things you connected with?
Lex Fridman (14:17.940)
So LLVM, there's also the other part
Chris Lattner (14:22.100)
that's really interesting if you're interested in languages
Lex Fridman (14:24.940)
is parsing and just analyzing the language,
Chris Lattner (14:29.620)
breaking it down, parsing, and so on.
Lex Fridman (14:31.220)
Was that interesting to you, or were you
Lex Fridman (14:32.580)
more interested in optimization?
Lex Fridman (14:34.060)
For me, it was more so I'm not really a math person.
Chris Lattner (14:37.420)
I could do math.
Lex Fridman (14:38.180)
I understand some bits of it when I get into it.
Lex Fridman (14:41.540)
But math is never the thing that attracted me.
Lex Fridman (14:43.940)
And so a lot of the parser part of the compiler
Chris Lattner (14:46.100)
has a lot of good formal theories
Lex Fridman (14:47.820)
that Don, for example, knows quite well.
Chris Lattner (14:50.060)
I'm still waiting for his book on that.
Lex Fridman (14:54.740)
But I just like building a thing and seeing what it could do
Lex Fridman (14:57.900)
and exploring and getting it to do more things
Lex Fridman (15:00.740)
and then setting new goals and reaching for them.
Lex Fridman (15:04.020)
And in the case of LLVM, when I started working on that,
Lex Fridman (15:09.580)
my research advisor that I was working for was a compiler guy.
Lex Fridman (15:13.420)
And so he and I specifically found each other
Lex Fridman (15:15.620)
because we were both interested in compilers.
Lex Fridman (15:16.940)
And so I started working with him and taking his class.
Lex Fridman (15:19.500)
And a lot of LLVM initially was, it's
Chris Lattner (15:21.580)
fun implementing all the standard algorithms and all
Lex Fridman (15:24.380)
the things that people had been talking about
Lex Fridman (15:26.380)
and were well known.
Lex Fridman (15:27.220)
And they were in the curricula for advanced studies
Lex Fridman (15:30.620)
and compilers.
Lex Fridman (15:31.340)
And so just being able to build that was really fun.
Lex Fridman (15:34.580)
And I was learning a lot by, instead of reading about it,
Lex Fridman (15:37.660)
just building.
Lex Fridman (15:38.660)
And so I enjoyed that.
Lex Fridman (15:40.220)
So you said compilers are these complicated systems.
Lex Fridman (15:42.820)
Can you even just with language try
Lex Fridman (15:46.180)
to describe how you turn a C++ program into code?
Lex Fridman (15:52.220)
Like, what are the hard parts?
Lex Fridman (15:53.460)
Why is it so hard?
Lex Fridman (15:54.620)
So I'll give you examples of the hard parts along the way.
Lex Fridman (15:57.020)
So C++ is a very complicated programming language.
Chris Lattner (16:01.060)
It's something like 1,400 pages in the spec.
Lex Fridman (16:03.500)
So C++ by itself is crazy complicated.
Lex Fridman (16:06.060)
Can we just pause?
Lex Fridman (16:07.140)
What makes the language complicated in terms
Lex Fridman (16:09.140)
of what's syntactically?
Lex Fridman (16:12.340)
So it's what they call syntax.
Lex Fridman (16:14.300)
So the actual how the characters are arranged, yes.
Lex Fridman (16:16.700)
It's also semantics, how it behaves.
Chris Lattner (16:20.020)
It's also, in the case of C++, there's
Lex Fridman (16:21.900)
a huge amount of history.
Chris Lattner (16:23.380)
C++ is built on top of C. You play that forward.
Lex Fridman (16:26.700)
And then a bunch of suboptimal, in some cases, decisions
Chris Lattner (16:29.860)
were made, and they compound.
Lex Fridman (16:31.620)
And then more and more and more things
Chris Lattner (16:33.380)
keep getting added to C++, and it will probably never stop.
Lex Fridman (16:36.980)
But the language is very complicated
Chris Lattner (16:38.540)
from that perspective.
Lex Fridman (16:39.540)
And so the interactions between subsystems
Chris Lattner (16:41.200)
is very complicated.
Lex Fridman (16:42.420)
There's just a lot there.
Lex Fridman (16:43.580)
And when you talk about the front end,
Lex Fridman (16:45.660)
one of the major challenges, which
Chris Lattner (16:47.060)
clang as a project, the C, C++ compiler that I built,
Lex Fridman (16:51.140)
I and many people built, one of the challenges we took on
Chris Lattner (16:54.480)
was we looked at GCC.
Lex Fridman (16:57.780)
GCC, at the time, was a really good industry standardized
Chris Lattner (17:02.540)
compiler that had really consolidated
Lex Fridman (17:05.260)
a lot of the other compilers in the world and was a standard.
Lex Fridman (17:08.340)
But it wasn't really great for research.
Lex Fridman (17:10.620)
The design was very difficult to work with.
Lex Fridman (17:12.580)
And it was full of global variables and other things
Lex Fridman (17:16.620)
that made it very difficult to reuse in ways
Chris Lattner (17:18.540)
that it wasn't originally designed for.
Lex Fridman (17:20.420)
And so with clang, one of the things that we wanted to do
Chris Lattner (17:22.740)
is push forward on better user interface,
Lex Fridman (17:25.500)
so make error messages that are just better than GCC's.
Lex Fridman (17:28.060)
And that's actually hard, because you
Lex Fridman (17:29.580)
have to do a lot of bookkeeping in an efficient way
Chris Lattner (17:32.780)
to be able to do that.
Lex Fridman (17:33.700)
We want to make compile time better.
Lex Fridman (17:35.180)
And so compile time is about making it efficient,
Lex Fridman (17:37.500)
which is also really hard when you're keeping
Chris Lattner (17:38.900)
track of extra information.
Lex Fridman (17:40.540)
We wanted to make new tools available,
Lex Fridman (17:43.380)
so refactoring tools and other analysis tools
Lex Fridman (17:46.380)
that GCC never supported, also leveraging the extra information
Chris Lattner (17:50.540)
we kept, but enabling those new classes of tools
Lex Fridman (17:54.060)
that then get built into IDEs.
Lex Fridman (17:55.940)
And so that's been one of the areas that clang has really
Lex Fridman (17:59.380)
helped push the world forward in,
Chris Lattner (18:01.300)
is in the tooling for C and C++ and things like that.
Lex Fridman (18:05.060)
But C++ and the front end piece is complicated.
Lex Fridman (18:07.500)
And you have to build syntax trees.
Lex Fridman (18:09.000)
And you have to check every rule in the spec.
Lex Fridman (18:11.340)
And you have to turn that back into an error message
Lex Fridman (18:14.020)
to the human that the human can understand
Chris Lattner (18:16.020)
when they do something wrong.
Lex Fridman (18:17.820)
But then you start doing what's called lowering,
Lex Fridman (18:20.740)
so going from C++ and the way that it represents
Lex Fridman (18:23.060)
code down to the machine.
Lex Fridman (18:24.980)
And when you do that, there's many different phases
Lex Fridman (18:27.380)
you go through.
Chris Lattner (18:29.660)
Often, there are, I think LLVM has something like 150
Lex Fridman (18:33.020)
different what are called passes in the compiler
Chris Lattner (18:36.260)
that the code passes through.
Lex Fridman (18:38.780)
And these get organized in very complicated ways,
Chris Lattner (18:41.860)
which affect the generated code and the performance
Lex Fridman (18:44.360)
and compile time and many other things.
Lex Fridman (18:45.980)
What are they passing through?
Lex Fridman (18:47.300)
So after you do the clang parsing, what's the graph?
Lex Fridman (18:53.980)
What does it look like?
Lex Fridman (18:54.900)
What's the data structure here?
Chris Lattner (18:56.100)
Yeah, so in the parser, it's usually a tree.
Lex Fridman (18:59.060)
And it's called an abstract syntax tree.
Lex Fridman (19:01.100)
And so the idea is you have a node for the plus
Lex Fridman (19:04.580)
that the human wrote in their code.
Chris Lattner (19:06.820)
Or the function call, you'll have a node for call
Lex Fridman (19:09.020)
with the function that they call and the arguments they pass,
Chris Lattner (19:11.900)
things like that.
Lex Fridman (19:14.460)
This then gets lowered into what's
Chris Lattner (19:16.620)
called an intermediate representation.
Lex Fridman (19:18.620)
And intermediate representations are like LLVM has one.
Lex Fridman (19:22.100)
And there, it's what's called a control flow graph.
Lex Fridman (19:26.940)
And so you represent each operation in the program
Chris Lattner (19:31.220)
as a very simple, like this is going to add two numbers.
Lex Fridman (19:34.480)
This is going to multiply two things.
Chris Lattner (19:35.980)
Maybe we'll do a call.
Lex Fridman (19:37.460)
But then they get put in what are called blocks.
Lex Fridman (19:40.260)
And so you get blocks of these straight line operations,
Lex Fridman (19:43.580)
where instead of being nested like in a tree,
Chris Lattner (19:45.340)
it's straight line operations.
Lex Fridman (19:46.900)
And so there's a sequence and an ordering to these operations.
Lex Fridman (19:49.780)
So within the block or outside the block?
Lex Fridman (19:51.820)
That's within the block.
Lex Fridman (19:52.980)
And so it's a straight line sequence of operations
Lex Fridman (19:54.980)
within the block.
Lex Fridman (19:55.740)
And then you have branches, like conditional branches,
Lex Fridman (19:58.980)
between blocks.
Chris Lattner (1:00:01.440)
working on optimizing network transport of weights
Lex Fridman (1:00:06.000)
across the network originally and trying
Chris Lattner (1:00:08.240)
to find ways to compress that.
Lex Fridman (1:00:10.160)
But then it got burned into silicon.
Lex Fridman (1:00:12.120)
And it's a key part of what makes TPU performance
Lex Fridman (1:00:14.560)
so amazing and great.
Chris Lattner (1:00:17.880)
Now, TPUs have many different aspects that are important.
Lex Fridman (1:00:20.680)
But the co design between the low level compiler bits
Lex Fridman (1:00:25.080)
and the software bits and the algorithms
Lex Fridman (1:00:27.360)
is all super important.
Lex Fridman (1:00:28.680)
And it's this amazing trifecta that only Google can do.
Lex Fridman (1:00:32.880)
Yeah, that's super exciting.
Lex Fridman (1:00:34.240)
So can you tell me about MLIR project, previously
Lex Fridman (1:00:39.800)
the secretive one?
Chris Lattner (1:00:41.400)
Yeah, so MLIR is a project that we
Lex Fridman (1:00:43.040)
announced at a compiler conference three weeks ago
Chris Lattner (1:00:47.000)
or something at the Compilers for Machine Learning
Lex Fridman (1:00:49.280)
conference.
Chris Lattner (1:00:50.920)
Basically, again, if you look at TensorFlow as a compiler stack,
Lex Fridman (1:00:53.760)
it has a number of compiler algorithms within it.
Chris Lattner (1:00:56.120)
It also has a number of compilers
Lex Fridman (1:00:57.660)
that get embedded into it.
Lex Fridman (1:00:59.000)
And they're made by different vendors.
Lex Fridman (1:01:00.480)
For example, Google has XLA, which
Chris Lattner (1:01:02.840)
is a great compiler system.
Lex Fridman (1:01:04.680)
NVIDIA has TensorRT.
Chris Lattner (1:01:06.480)
Intel has NGRAPH.
Lex Fridman (1:01:08.640)
There's a number of these different compiler systems.
Lex Fridman (1:01:10.840)
And they're very hardware specific.
Lex Fridman (1:01:13.840)
And they're trying to solve different parts of the problems.
Lex Fridman (1:01:16.480)
But they're all kind of similar in a sense of they
Lex Fridman (1:01:19.400)
want to integrate with TensorFlow.
Chris Lattner (1:01:20.880)
Now, TensorFlow has an optimizer.
Lex Fridman (1:01:22.960)
And it has these different code generation technologies
Chris Lattner (1:01:25.540)
built in.
Lex Fridman (1:01:26.440)
The idea of MLIR is to build a common infrastructure
Chris Lattner (1:01:28.720)
to support all these different subsystems.
Lex Fridman (1:01:31.160)
And initially, it's to be able to make it
Lex Fridman (1:01:33.500)
so that they all plug in together
Lex Fridman (1:01:34.880)
and they can share a lot more code and can be reusable.
Lex Fridman (1:01:37.880)
But over time, we hope that the industry
Lex Fridman (1:01:39.680)
will start collaborating and sharing code.
Lex Fridman (1:01:42.480)
And instead of reinventing the same things over and over again,
Lex Fridman (1:01:45.320)
that we can actually foster some of that working together
Chris Lattner (1:01:49.280)
to solve common problem energy that
Lex Fridman (1:01:51.560)
has been useful in the compiler field before.
Chris Lattner (1:01:54.480)
Beyond that, MLIR is some people have joked
Lex Fridman (1:01:57.360)
that it's kind of LLVM too.
Chris Lattner (1:01:59.320)
It learns a lot about what LLVM has been good
Lex Fridman (1:02:01.840)
and what LLVM has done wrong.
Lex Fridman (1:02:04.360)
And it's a chance to fix that.
Lex Fridman (1:02:06.880)
And also, there are challenges in the LLVM ecosystem as well,
Chris Lattner (1:02:09.840)
where LLVM is very good at the thing it was designed to do.
Lex Fridman (1:02:12.760)
But 20 years later, the world has changed.
Lex Fridman (1:02:15.560)
And people are trying to solve higher level problems.
Lex Fridman (1:02:17.980)
And we need some new technology.
Lex Fridman (1:02:20.360)
And what's the future of open source in this context?
Lex Fridman (1:02:24.720)
Very soon.
Lex Fridman (1:02:25.760)
So it is not yet open source.
Lex Fridman (1:02:27.480)
But it will be hopefully in the next couple months.
Lex Fridman (1:02:29.320)
So you still believe in the value of open source
Lex Fridman (1:02:31.040)
in these kinds of contexts?
Chris Lattner (1:02:31.640)
Oh, yeah.
Lex Fridman (1:02:31.880)
Absolutely.
Lex Fridman (1:02:32.440)
And I think that the TensorFlow community at large
Lex Fridman (1:02:36.160)
fully believes in open source.
Lex Fridman (1:02:37.720)
So I mean, there is a difference between Apple,
Lex Fridman (1:02:40.120)
where you were previously, and Google now,
Chris Lattner (1:02:42.480)
in spirit and culture.
Lex Fridman (1:02:43.520)
And I would say the open source in TensorFlow
Chris Lattner (1:02:45.480)
was a seminal moment in the history of software,
Lex Fridman (1:02:48.400)
because here's this large company releasing
Chris Lattner (1:02:51.680)
a very large code base that's open sourcing.
Lex Fridman (1:02:56.200)
What are your thoughts on that?
Chris Lattner (1:02:58.520)
Happy or not, were you to see that kind
Lex Fridman (1:03:00.840)
of degree of open sourcing?
Lex Fridman (1:03:02.920)
So between the two, I prefer the Google approach,
Lex Fridman (1:03:05.360)
if that's what you're saying.
Chris Lattner (1:03:07.800)
The Apple approach makes sense, given the historical context
Lex Fridman (1:03:12.400)
that Apple came from.
Lex Fridman (1:03:13.400)
But that's been 35 years ago.
Lex Fridman (1:03:15.760)
And I think that Apple is definitely adapting.
Lex Fridman (1:03:18.200)
And the way I look at it is that there's
Lex Fridman (1:03:20.280)
different kinds of concerns in the space.
Chris Lattner (1:03:23.160)
It is very rational for a business
Lex Fridman (1:03:24.880)
to care about making money.
Chris Lattner (1:03:28.720)
That fundamentally is what a business is about.
Lex Fridman (1:03:31.640)
But I think it's also incredibly realistic to say,
Chris Lattner (1:03:34.880)
it's not your string library that's
Lex Fridman (1:03:36.360)
the thing that's going to make you money.
Chris Lattner (1:03:38.080)
It's going to be the amazing UI product differentiating
Lex Fridman (1:03:41.480)
features and other things like that that you built on top
Chris Lattner (1:03:43.840)
of your string library.
Lex Fridman (1:03:45.280)
And so keeping your string library
Chris Lattner (1:03:48.280)
proprietary and secret and things
Lex Fridman (1:03:50.360)
like that is maybe not the important thing anymore.
Chris Lattner (1:03:54.760)
Where before, platforms were different.
Lex Fridman (1:03:57.720)
And even 15 years ago, things were a little bit different.
Lex Fridman (1:04:01.520)
But the world is changing.
Lex Fridman (1:04:02.920)
So Google strikes a very good balance,
Chris Lattner (1:04:04.840)
I think.
Lex Fridman (1:04:05.340)
And I think that TensorFlow being open source really
Chris Lattner (1:04:09.040)
changed the entire machine learning field
Lex Fridman (1:04:12.000)
and caused a revolution in its own right.
Lex Fridman (1:04:14.080)
And so I think it's amazingly forward looking
Lex Fridman (1:04:17.560)
because I could have imagined, and I wasn't at Google
Chris Lattner (1:04:20.880)
at the time, but I could imagine a different context
Lex Fridman (1:04:23.160)
and different world where a company says,
Chris Lattner (1:04:25.520)
machine learning is critical to what we're doing.
Lex Fridman (1:04:27.640)
We're not going to give it to other people.
Lex Fridman (1:04:29.640)
And so that decision is a profoundly brilliant insight
Lex Fridman (1:04:35.560)
that I think has really led to the world being
Chris Lattner (1:04:37.480)
better and better for Google as well.
Lex Fridman (1:04:40.120)
And has all kinds of ripple effects.
Chris Lattner (1:04:42.200)
I think it is really, I mean, you
Lex Fridman (1:04:45.160)
can't understate Google deciding how profound that
Chris Lattner (1:04:48.800)
is for software.
Lex Fridman (1:04:49.840)
It's awesome.
Chris Lattner (1:04:50.880)
Well, and again, I can understand the concern
Lex Fridman (1:04:54.900)
about if we release our machine learning software,
Chris Lattner (1:04:58.440)
our competitors could go faster.
Lex Fridman (1:05:00.000)
But on the other hand, I think that open sourcing TensorFlow
Chris Lattner (1:05:02.500)
has been fantastic for Google.
Lex Fridman (1:05:03.960)
And I'm sure that decision was very nonobvious at the time,
Lex Fridman (1:05:09.120)
but I think it's worked out very well.
Lex Fridman (1:05:11.480)
So let's try this real quick.
Chris Lattner (1:05:13.240)
You were at Tesla for five months
Lex Fridman (1:05:15.640)
as the VP of autopilot software.
Chris Lattner (1:05:17.640)
You led the team during the transition from H hardware
Lex Fridman (1:05:20.520)
one to hardware two.
Chris Lattner (1:05:22.360)
I have a couple of questions.
Lex Fridman (1:05:23.520)
So one, first of all, to me, that's
Chris Lattner (1:05:26.320)
one of the bravest engineering decisions undertaking really
Lex Fridman (1:05:33.000)
ever in the automotive industry to me, software wise,
Chris Lattner (1:05:36.040)
starting from scratch.
Lex Fridman (1:05:37.440)
It's a really brave engineering decision.
Lex Fridman (1:05:39.200)
So my one question there is, what was that like?
Lex Fridman (1:05:42.600)
What was the challenge of that?
Lex Fridman (1:05:43.920)
Do you mean the career decision of jumping
Lex Fridman (1:05:45.720)
from a comfortable good job into the unknown, or?
Chris Lattner (1:05:48.800)
That combined, so at the individual level,
Lex Fridman (1:05:51.480)
you making that decision.
Lex Fridman (1:05:54.560)
And then when you show up, it's a really hard engineering
Lex Fridman (1:05:57.960)
problem.
Lex Fridman (1:05:58.760)
So you could just stay, maybe slow down,
Lex Fridman (1:06:03.560)
say hardware one, or those kinds of decisions.
Chris Lattner (1:06:06.680)
Just taking it full on, let's do this from scratch.
Lex Fridman (1:06:10.160)
What was that like?
Chris Lattner (1:06:11.080)
Well, so I mean, I don't think Tesla
Lex Fridman (1:06:12.640)
has a culture of taking things slow and seeing how it goes.
Lex Fridman (1:06:16.080)
And one of the things that attracted me about Tesla
Lex Fridman (1:06:18.080)
is it's very much a gung ho, let's change the world,
Chris Lattner (1:06:20.020)
let's figure it out kind of a place.
Lex Fridman (1:06:21.520)
And so I have a huge amount of respect for that.
Chris Lattner (1:06:25.640)
Tesla has done very smart things with hardware one
Lex Fridman (1:06:28.680)
in particular.
Lex Fridman (1:06:29.400)
And the hardware one design was originally
Lex Fridman (1:06:32.200)
designed to be very simple automation features
Chris Lattner (1:06:36.560)
in the car for like traffic aware cruise control and things
Lex Fridman (1:06:39.360)
like that.
Lex Fridman (1:06:39.840)
And the fact that they were able to effectively feature creep
Lex Fridman (1:06:42.920)
it into lane holding and a very useful driver assistance
Chris Lattner (1:06:47.720)
feature is pretty astounding, particularly given
Lex Fridman (1:06:50.120)
the details of the hardware.
Chris Lattner (1:06:52.560)
Hardware two built on that in a lot of ways.
Lex Fridman (1:06:54.640)
And the challenge there was that they
Chris Lattner (1:06:56.180)
were transitioning from a third party provided vision stack
Lex Fridman (1:07:00.040)
to an in house built vision stack.
Lex Fridman (1:07:01.720)
And so for the first step, which I mostly helped with,
Lex Fridman (1:07:05.680)
was getting onto that new vision stack.
Lex Fridman (1:07:08.480)
And that was very challenging.
Lex Fridman (1:07:10.800)
And it was time critical for various reasons,
Lex Fridman (1:07:14.000)
and it was a big leap.
Lex Fridman (1:07:14.960)
But it was fortunate that it built
Chris Lattner (1:07:16.640)
on a lot of the knowledge and expertise and the team
Lex Fridman (1:07:18.800)
that had built hardware one's driver assistance features.
Lex Fridman (1:07:22.920)
So you spoke in a collected and kind way
Lex Fridman (1:07:25.360)
about your time at Tesla, but it was ultimately not a good fit.
Chris Lattner (1:07:28.960)
Elon Musk, we've talked on this podcast,
Lex Fridman (1:07:31.840)
several guests to the course, Elon Musk
Chris Lattner (1:07:33.880)
continues to do some of the most bold and innovative engineering
Lex Fridman (1:07:36.880)
work in the world, at times at the cost
Chris Lattner (1:07:39.560)
some of the members of the Tesla team.
Lex Fridman (1:07:41.280)
What did you learn about working in this chaotic world
Lex Fridman (1:07:45.080)
with Elon?
Lex Fridman (1:07:46.720)
Yeah, so I guess I would say that when I was at Tesla,
Chris Lattner (1:07:50.560)
I experienced and saw the highest degree of turnover
Lex Fridman (1:07:54.440)
I'd ever seen in a company, which was a bit of a shock.
Lex Fridman (1:07:58.240)
But one of the things I learned and I came to respect
Lex Fridman (1:08:00.520)
is that Elon's able to attract amazing talent because he
Chris Lattner (1:08:03.760)
has a very clear vision of the future,
Lex Fridman (1:08:05.660)
and he can get people to buy into it
Chris Lattner (1:08:07.200)
because they want that future to happen.
Lex Fridman (1:08:09.840)
And the power of vision is something
Chris Lattner (1:08:11.840)
that I have a tremendous amount of respect for.
Lex Fridman (1:08:14.240)
And I think that Elon is fairly singular
Chris Lattner (1:08:17.040)
in the world in terms of the things
Lex Fridman (1:08:20.120)
he's able to get people to believe in.
Lex Fridman (1:08:22.360)
And there are many people that stand in the street corner
Lex Fridman (1:08:27.360)
and say, ah, we're going to go to Mars, right?
Lex Fridman (1:08:30.200)
But then there are a few people that
Lex Fridman (1:08:31.600)
can get others to buy into it and believe and build the path
Lex Fridman (1:08:35.200)
and make it happen.
Lex Fridman (1:08:36.160)
And so I respect that.
Chris Lattner (1:08:39.120)
I don't respect all of his methods,
Lex Fridman (1:08:41.880)
but I have a huge amount of respect for that.
Chris Lattner (1:08:45.000)
You've mentioned in a few places,
Lex Fridman (1:08:46.920)
including in this context, working hard.
Lex Fridman (1:08:50.440)
What does it mean to work hard?
Lex Fridman (1:08:52.000)
And when you look back at your life,
Lex Fridman (1:08:53.520)
what were some of the most brutal periods
Lex Fridman (1:08:57.080)
of having to really put everything
Lex Fridman (1:09:00.760)
you have into something?
Lex Fridman (1:09:03.360)
Yeah, good question.
Lex Fridman (1:09:05.040)
So working hard can be defined a lot of different ways,
Lex Fridman (1:09:07.440)
so a lot of hours, and so that is true.
Chris Lattner (1:09:12.480)
The thing to me that's the hardest
Lex Fridman (1:09:14.520)
is both being short term focused on delivering and executing
Lex Fridman (1:09:18.760)
and making a thing happen while also thinking
Lex Fridman (1:09:21.120)
about the longer term and trying to balance that.
Chris Lattner (1:09:24.400)
Because if you are myopically focused on solving a task
Lex Fridman (1:09:28.520)
and getting that done and only think
Chris Lattner (1:09:31.240)
about that incremental next step,
Lex Fridman (1:09:32.600)
you will miss the next big hill you should jump over to.
Lex Fridman (1:09:36.440)
And so I've been really fortunate that I've
Lex Fridman (1:09:39.600)
been able to kind of oscillate between the two.
Lex Fridman (1:09:42.120)
And historically at Apple, for example, that
Lex Fridman (1:09:45.480)
was made possible because I was able to work with some really
Chris Lattner (1:09:47.920)
amazing people and build up teams and leadership
Lex Fridman (1:09:50.360)
structures and allow them to grow in their careers
Lex Fridman (1:09:55.280)
and take on responsibility, thereby freeing up
Lex Fridman (1:09:58.280)
me to be a little bit crazy and thinking about the next thing.
Lex Fridman (1:10:02.960)
And so it's a lot of that.
Lex Fridman (1:10:04.640)
But it's also about with experience,
Chris Lattner (1:10:06.760)
you make connections that other people don't necessarily make.
Lex Fridman (1:10:10.080)
And so I think that's a big part as well.
Lex Fridman (1:10:12.880)
But the bedrock is just a lot of hours.
Lex Fridman (1:10:16.000)
And that's OK with me.
Chris Lattner (1:10:19.600)
There's different theories on work life balance.
Lex Fridman (1:10:21.480)
And my theory for myself, which I do not project onto the team,
Lex Fridman (1:10:25.200)
but my theory for myself is that I
Lex Fridman (1:10:28.520)
want to love what I'm doing and work really hard.
Lex Fridman (1:10:30.400)
And my purpose, I feel like, and my goal is to change the world
Lex Fridman (1:10:35.000)
and make it a better place.
Lex Fridman (1:10:36.280)
And that's what I'm really motivated to do.
Lex Fridman (1:10:40.000)
So last question, LLVM logo is a dragon.
Chris Lattner (1:10:44.760)
You explain that this is because dragons have connotations
Lex Fridman (1:10:47.880)
of power, speed, intelligence.
Chris Lattner (1:10:50.320)
It can also be sleek, elegant, and modular,
Lex Fridman (1:10:53.320)
though you remove the modular part.
Lex Fridman (1:10:56.280)
What is your favorite dragon related character
Lex Fridman (1:10:58.920)
from fiction, video, or movies?
Lex Fridman (1:11:01.440)
So those are all very kind ways of explaining it.
Lex Fridman (1:11:03.840)
Do you want to know the real reason it's a dragon?
Chris Lattner (1:11:06.200)
Yeah.
Lex Fridman (1:11:07.000)
Is that better?
Lex Fridman (1:11:07.920)
So there is a seminal book on compiler design
Lex Fridman (1:11:11.040)
called The Dragon Book.
Lex Fridman (1:11:12.520)
And so this is a really old now book on compilers.
Lex Fridman (1:11:16.320)
And so the dragon logo for LLVM came about because at Apple,
Chris Lattner (1:11:22.080)
we kept talking about LLVM related technologies
Lex Fridman (1:11:24.720)
and there's no logo to put on a slide.
Lex Fridman (1:11:26.960)
And so we're like, what do we do?
Lex Fridman (1:11:28.480)
And somebody's like, well, what kind of logo
Lex Fridman (1:11:30.480)
should a compiler technology have?
Lex Fridman (1:11:32.160)
And I'm like, I don't know.
Chris Lattner (1:11:33.360)
I mean, the dragon is the best thing that we've got.
Lex Fridman (1:11:37.320)
And Apple somehow magically came up with the logo.
Lex Fridman (1:11:41.520)
And it was a great thing.
Lex Fridman (1:11:42.680)
And the whole community rallied around it.
Lex Fridman (1:11:44.520)
And then it got better as other graphic designers
Lex Fridman (1:11:46.760)
got involved.
Lex Fridman (1:11:47.360)
But that's originally where it came from.
Lex Fridman (1:11:49.360)
The story.
Chris Lattner (1:11:50.160)
Is there dragons from fiction that you
Lex Fridman (1:11:51.960)
connect with, that Game of Thrones, Lord of the Rings,
Lex Fridman (1:11:57.240)
that kind of thing?
Lex Fridman (1:11:58.080)
Lord of the Rings is great.
Chris Lattner (1:11:59.200)
I also like role playing games and things
Lex Fridman (1:12:00.760)
like computer role playing games.
Lex Fridman (1:12:02.240)
And so dragons often show up in there.
Lex Fridman (1:12:04.280)
But really, it comes back to the book.
Chris Lattner (1:12:07.160)
Oh, no, we need a thing.
Lex Fridman (1:12:09.960)
And hilariously, one of the funny things about LLVM
Chris Lattner (1:12:13.720)
is that my wife, who's amazing, runs the LLVM Foundation.
Lex Fridman (1:12:19.520)
And she goes to Grace Hopper and is
Chris Lattner (1:12:21.080)
trying to get more women involved in the.
Lex Fridman (1:12:23.360)
She's also a compiler engineer.
Lex Fridman (1:12:24.640)
So she's trying to get other women
Lex Fridman (1:12:26.080)
to get interested in compilers and things like this.
Lex Fridman (1:12:28.020)
And so she hands out the stickers.
Lex Fridman (1:12:30.000)
And people like the LLVM sticker because of Game of Thrones.
Lex Fridman (1:12:34.320)
And so sometimes culture has this helpful effect
Lex Fridman (1:12:36.880)
to get the next generation of compiler engineers
Chris Lattner (1:12:39.960)
engaged with the cause.
Lex Fridman (1:12:42.400)
OK, awesome.
Chris Lattner (1:12:43.320)
Chris, thanks so much for talking with us.
Lex Fridman (1:12:44.800)
It's been great talking with you.
Lex Fridman (20:00.140)
And so when you write a loop, for example, in a syntax tree,
Lex Fridman (20:04.860)
you would have a for node, like for a for statement
Chris Lattner (20:08.060)
in a C like language, you'd have a for node.
Lex Fridman (20:10.540)
And you have a pointer to the expression
Chris Lattner (20:12.200)
for the initializer, a pointer to the expression
Lex Fridman (20:14.080)
for the increment, a pointer to the expression
Chris Lattner (20:16.040)
for the comparison, a pointer to the body.
Lex Fridman (20:18.900)
And these are all nested underneath it.
Chris Lattner (20:21.060)
In a control flow graph, you get a block
Lex Fridman (20:22.900)
for the code that runs before the loop, so the initializer
Chris Lattner (20:26.820)
code.
Lex Fridman (20:27.620)
And you have a block for the body of the loop.
Lex Fridman (20:30.340)
And so the body of the loop code goes in there,
Lex Fridman (20:33.780)
but also the increment and other things like that.
Lex Fridman (20:35.660)
And then you have a branch that goes back to the top
Lex Fridman (20:37.860)
and a comparison and a branch that goes out.
Lex Fridman (20:39.900)
And so it's more of an assembly level kind of representation.
Lex Fridman (20:43.820)
But the nice thing about this level of representation
Chris Lattner (20:46.060)
is it's much more language independent.
Lex Fridman (20:48.700)
And so there's lots of different kinds of languages
Chris Lattner (20:51.900)
with different kinds of, you know,
Lex Fridman (20:54.540)
JavaScript has a lot of different ideas of what
Chris Lattner (20:56.840)
is false, for example.
Lex Fridman (20:58.180)
And all that can stay in the front end.
Lex Fridman (21:00.780)
But then that middle part can be shared across all those.
Lex Fridman (21:04.220)
How close is that intermediate representation
Lex Fridman (21:07.540)
to neural networks, for example?
Lex Fridman (21:10.620)
Are they, because everything you describe
Chris Lattner (21:13.540)
is a kind of echoes of a neural network graph.
Lex Fridman (21:16.100)
Are they neighbors or what?
Chris Lattner (21:18.940)
They're quite different in details,
Lex Fridman (21:20.980)
but they're very similar in idea.
Lex Fridman (21:22.520)
So one of the things that neural networks do
Lex Fridman (21:24.320)
is they learn representations for data
Chris Lattner (21:26.900)
at different levels of abstraction.
Lex Fridman (21:29.140)
And then they transform those through layers, right?
Lex Fridman (21:33.940)
So the compiler does very similar things.
Lex Fridman (21:35.660)
But one of the things the compiler does
Chris Lattner (21:37.320)
is it has relatively few different representations.
Lex Fridman (21:40.660)
Where a neural network often, as you get deeper, for example,
Chris Lattner (21:43.100)
you get many different representations
Lex Fridman (21:44.820)
in each layer or set of ops.
Chris Lattner (21:47.380)
It's transforming between these different representations.
Lex Fridman (21:50.260)
In a compiler, often you get one representation
Lex Fridman (21:53.100)
and they do many transformations to it.
Lex Fridman (21:55.240)
And these transformations are often applied iteratively.
Lex Fridman (21:59.540)
And for programmers, there's familiar types of things.
Lex Fridman (22:02.940)
For example, trying to find expressions inside of a loop
Lex Fridman (22:06.180)
and pulling them out of a loop so they execute for times.
Lex Fridman (22:08.540)
Or find redundant computation.
Chris Lattner (22:10.740)
Or find constant folding or other simplifications,
Lex Fridman (22:15.380)
turning two times x into x shift left by one.
Lex Fridman (22:19.060)
And things like this are all the examples
Lex Fridman (22:21.980)
of the things that happen.
Lex Fridman (22:23.340)
But compilers end up getting a lot of theorem proving
Lex Fridman (22:26.180)
and other kinds of algorithms that
Chris Lattner (22:27.760)
try to find higher level properties of the program that
Lex Fridman (22:30.100)
then can be used by the optimizer.
Chris Lattner (22:32.280)
Cool.
Lex Fridman (22:32.780)
So what's the biggest bang for the buck with optimization?
Lex Fridman (22:38.140)
Today?
Lex Fridman (22:38.640)
Yeah.
Chris Lattner (22:39.140)
Well, no, not even today.
Lex Fridman (22:40.900)
At the very beginning, the 80s, I don't know.
Chris Lattner (22:42.900)
Yeah, so for the 80s, a lot of it
Lex Fridman (22:44.300)
was things like register allocation.
Lex Fridman (22:46.420)
So the idea of in a modern microprocessor,
Lex Fridman (22:50.460)
what you'll end up having is you'll
Chris Lattner (22:51.880)
end up having memory, which is relatively slow.
Lex Fridman (22:54.340)
And then you have registers that are relatively fast.
Lex Fridman (22:57.060)
But registers, you don't have very many of them.
Lex Fridman (23:00.340)
And so when you're writing a bunch of code,
Chris Lattner (23:02.600)
you're just saying, compute this,
Lex Fridman (23:04.180)
put in a temporary variable, compute this, compute this,
Chris Lattner (23:05.940)
compute this, put in a temporary variable.
Lex Fridman (23:07.780)
I have a loop.
Chris Lattner (23:08.220)
I have some other stuff going on.
Lex Fridman (23:09.780)
Well, now you're running on an x86,
Chris Lattner (23:11.660)
like a desktop PC or something.
Lex Fridman (23:13.900)
Well, it only has, in some cases, some modes,
Chris Lattner (23:16.860)
eight registers.
Lex Fridman (23:18.700)
And so now the compiler has to choose what values get
Chris Lattner (23:21.620)
put in what registers at what points in the program.
Lex Fridman (23:24.820)
And this is actually a really big deal.
Lex Fridman (23:26.580)
So if you think about, you have a loop, an inner loop
Lex Fridman (23:29.500)
that executes millions of times maybe.
Chris Lattner (23:31.620)
If you're doing loads and stores inside that loop,
Lex Fridman (23:33.620)
then it's going to be really slow.
Lex Fridman (23:35.040)
But if you can somehow fit all the values inside that loop
Lex Fridman (23:37.740)
in registers, now it's really fast.
Lex Fridman (23:40.180)
And so getting that right requires a lot of work,
Lex Fridman (23:43.020)
because there's many different ways to do that.
Lex Fridman (23:44.940)
And often what the compiler ends up doing
Lex Fridman (23:46.980)
is it ends up thinking about things
Chris Lattner (23:48.840)
in a different representation than what the human wrote.
Lex Fridman (23:52.020)
You wrote into x.
Chris Lattner (23:53.340)
Well, the compiler thinks about that as four different values,
Lex Fridman (23:56.820)
each which have different lifetimes across the function
Chris Lattner (23:59.280)
that it's in.
Lex Fridman (24:00.420)
And each of those could be put in a register or memory
Chris Lattner (24:03.180)
or different memory or maybe in some parts of the code
Lex Fridman (24:06.140)
recomputed instead of stored and reloaded.
Lex Fridman (24:08.360)
And there are many of these different kinds of techniques
Lex Fridman (24:10.700)
that can be used.
Lex Fridman (24:11.460)
So it's adding almost like a time dimension to it's
Lex Fridman (24:15.780)
trying to optimize across time.
Lex Fridman (24:18.300)
So it's considering when you're programming,
Lex Fridman (24:20.340)
you're not thinking in that way.
Chris Lattner (24:21.860)
Yeah, absolutely.
Lex Fridman (24:23.220)
And so the RISC era made things.
Lex Fridman (24:27.100)
So RISC chips, R I S C. The RISC chips,
Lex Fridman (24:32.020)
as opposed to CISC chips.
Chris Lattner (24:33.740)
The RISC chips made things more complicated for the compiler,
Lex Fridman (24:36.700)
because what they ended up doing is ending up
Chris Lattner (24:40.660)
adding pipelines to the processor, where
Lex Fridman (24:42.500)
the processor can do more than one thing at a time.
Lex Fridman (24:45.020)
But this means that the order of operations matters a lot.
Lex Fridman (24:47.740)
So one of the classical compiler techniques that you use
Chris Lattner (24:50.260)
is called scheduling.
Lex Fridman (24:51.940)
And so moving the instructions around
Lex Fridman (24:54.220)
so that the processor can keep its pipelines full instead
Lex Fridman (24:57.740)
of stalling and getting blocked.
Lex Fridman (24:59.220)
And so there's a lot of things like that that
Lex Fridman (25:01.180)
are kind of bread and butter compiler techniques
Chris Lattner (25:03.620)
that have been studied a lot over the course of decades now.
Lex Fridman (25:06.220)
But the engineering side of making them real
Chris Lattner (25:08.540)
is also still quite hard.
Lex Fridman (25:10.580)
And you talk about machine learning.
Chris Lattner (25:12.460)
This is a huge opportunity for machine learning,
Lex Fridman (25:14.420)
because many of these algorithms are full of these
Chris Lattner (25:17.620)
hokey, hand rolled heuristics, which
Lex Fridman (25:19.300)
work well on specific benchmarks that don't generalize,
Lex Fridman (25:21.820)
and full of magic numbers.
Lex Fridman (25:23.940)
And I hear there's some techniques that
Chris Lattner (25:26.620)
are good at handling that.
Lex Fridman (25:28.060)
So what would be the, if you were to apply machine learning
Lex Fridman (25:32.220)
to this, what's the thing you're trying to optimize?
Lex Fridman (25:34.740)
Is it ultimately the running time?
Chris Lattner (25:39.100)
You can pick your metric, and there's running time,
Lex Fridman (25:41.180)
there's memory use, there's lots of different things
Chris Lattner (25:43.900)
that you can optimize for.
Lex Fridman (25:44.940)
Code size is another one that some people care about
Chris Lattner (25:47.220)
in the embedded space.
Lex Fridman (25:48.860)
Is this like the thinking into the future,
Chris Lattner (25:51.700)
or has somebody actually been crazy enough
Lex Fridman (25:54.500)
to try to have machine learning based parameter
Lex Fridman (25:58.060)
tuning for the optimization of compilers?
Lex Fridman (26:01.060)
So this is something that is, I would say, research right now.
Chris Lattner (26:04.860)
There are a lot of research systems
Lex Fridman (26:06.820)
that have been applying search in various forms.
Lex Fridman (26:09.100)
And using reinforcement learning is one form,
Lex Fridman (26:11.460)
but also brute force search has been tried for quite a while.
Lex Fridman (26:14.460)
And usually, these are in small problem spaces.
Lex Fridman (26:18.180)
So find the optimal way to code generate a matrix
Chris Lattner (26:21.900)
multiply for a GPU, something like that,
Lex Fridman (26:24.460)
where you say, there, there's a lot of design space of,
Lex Fridman (26:28.580)
do you unroll loops a lot?
Lex Fridman (26:29.900)
Do you execute multiple things in parallel?
Lex Fridman (26:32.660)
And there's many different confounding factors here
Lex Fridman (26:35.340)
because graphics cards have different numbers of threads
Lex Fridman (26:38.100)
and registers and execution ports and memory bandwidth
Lex Fridman (26:41.020)
and many different constraints that interact
Chris Lattner (26:42.740)
in nonlinear ways.
Lex Fridman (26:44.460)
And so search is very powerful for that.
Lex Fridman (26:46.500)
And it gets used in certain ways,
Lex Fridman (26:49.820)
but it's not very structured.
Chris Lattner (26:51.220)
This is something that we need,
Lex Fridman (26:52.620)
we as an industry need to fix.
Lex Fridman (26:54.500)
So you said 80s, but like, so have there been like big jumps
Lex Fridman (26:59.220)
in improvement and optimization?
Chris Lattner (27:01.260)
Yeah.
Lex Fridman (27:02.340)
Yeah, since then, what's the coolest thing?
Chris Lattner (27:05.300)
It's largely been driven by hardware.
Lex Fridman (27:07.100)
So, well, it's hardware and software.
Lex Fridman (27:09.860)
So in the mid nineties, Java totally changed the world,
Lex Fridman (27:13.700)
right?
Lex Fridman (27:14.540)
And I'm still amazed by how much change was introduced
Lex Fridman (27:17.540)
by the way or in a good way.
Lex Fridman (27:19.340)
So like reflecting back, Java introduced things like,
Lex Fridman (27:22.420)
all at once introduced things like JIT compilation.
Chris Lattner (27:25.860)
None of these were novel, but it pulled it together
Lex Fridman (27:27.780)
and made it mainstream and made people invest in it.
Chris Lattner (27:30.580)
JIT compilation, garbage collection, portable code,
Lex Fridman (27:33.620)
safe code, like memory safe code,
Chris Lattner (27:36.620)
like a very dynamic dispatch execution model.
Lex Fridman (27:41.380)
Like many of these things,
Chris Lattner (27:42.620)
which had been done in research systems
Lex Fridman (27:44.060)
and had been done in small ways in various places,
Chris Lattner (27:46.900)
really came to the forefront,
Lex Fridman (27:47.980)
really changed how things worked
Lex Fridman (27:49.740)
and therefore changed the way people thought
Lex Fridman (27:51.980)
about the problem.
Chris Lattner (27:53.060)
JavaScript was another major world change
Lex Fridman (27:56.300)
based on the way it works.
Lex Fridman (27:59.300)
But also on the hardware side of things,
Lex Fridman (28:01.300)
multi core and vector instructions really change
Chris Lattner (28:06.660)
the problem space and are very,
Lex Fridman (28:09.460)
they don't remove any of the problems
Chris Lattner (28:10.820)
that compilers faced in the past,
Lex Fridman (28:12.380)
but they add new kinds of problems
Chris Lattner (28:14.540)
of how do you find enough work
Lex Fridman (28:16.380)
to keep a four wide vector busy, right?
Chris Lattner (28:20.020)
Or if you're doing a matrix multiplication,
Lex Fridman (28:22.660)
how do you do different columns out of that matrix
Lex Fridman (28:25.860)
at the same time?
Lex Fridman (28:26.700)
And how do you maximally utilize the arithmetic compute
Lex Fridman (28:30.140)
that one core has?
Lex Fridman (28:31.460)
And then how do you take it to multiple cores?
Lex Fridman (28:33.500)
How did the whole virtual machine thing change
Lex Fridman (28:35.780)
the compilation pipeline?
Chris Lattner (28:38.020)
Yeah, so what the Java virtual machine does
Lex Fridman (28:40.460)
is it splits, just like I was talking about before,
Chris Lattner (28:44.180)
where you have a front end that parses the code,
Lex Fridman (28:46.300)
and then you have an intermediate representation
Chris Lattner (28:48.020)
that gets transformed.
Lex Fridman (28:49.460)
What Java did was they said,
Chris Lattner (28:51.020)
we will parse the code and then compile to
Lex Fridman (28:53.100)
what's known as Java byte code.
Lex Fridman (28:55.500)
And that byte code is now a portable code representation
Lex Fridman (28:58.580)
that is industry standard and locked down and can't change.
Lex Fridman (29:02.420)
And then the back part of the compiler
Lex Fridman (29:05.100)
that does optimization and code generation
Chris Lattner (29:07.300)
can now be built by different vendors.
Lex Fridman (29:09.460)
Okay.
Lex Fridman (29:10.300)
And Java byte code can be shipped around across the wire.
Lex Fridman (29:13.020)
It's memory safe and relatively trusted.
Lex Fridman (29:16.860)
And because of that, it can run in the browser.
Lex Fridman (29:18.660)
And that's why it runs in the browser, right?
Lex Fridman (29:20.540)
And so that way you can be in,
Lex Fridman (29:22.980)
again, back in the day, you would write a Java applet
Lex Fridman (29:25.020)
and as a web developer, you'd build this mini app
Lex Fridman (29:29.300)
that would run on a webpage.
Chris Lattner (29:30.860)
Well, a user of that is running a web browser
Lex Fridman (29:33.620)
on their computer.
Chris Lattner (29:34.460)
You download that Java byte code, which can be trusted,
Lex Fridman (29:37.860)
and then you do all the compiler stuff on your machine
Lex Fridman (29:41.060)
so that you know that you trust that.
Lex Fridman (29:42.460)
Now, is that a good idea or a bad idea?
Chris Lattner (29:44.060)
It's a great idea.
Lex Fridman (29:44.900)
I mean, it's a great idea for certain problems.
Lex Fridman (29:46.240)
And I'm very much a believer that technology is itself
Lex Fridman (29:49.540)
neither good nor bad.
Chris Lattner (29:50.520)
It's how you apply it.
Lex Fridman (29:52.940)
You know, this would be a very, very bad thing
Chris Lattner (29:54.660)
for very low levels of the software stack.
Lex Fridman (29:56.980)
But in terms of solving some of these software portability
Lex Fridman (30:00.300)
and transparency, or portability problems,
Lex Fridman (30:02.820)
I think it's been really good.
Chris Lattner (30:04.240)
Now, Java ultimately didn't win out on the desktop.
Lex Fridman (30:06.600)
And like, there are good reasons for that.
Lex Fridman (30:09.420)
But it's been very successful on servers and in many places,
Lex Fridman (30:13.220)
it's been a very successful thing over decades.
Lex Fridman (30:16.300)
So what has been LLVMs and C langs improvements
Lex Fridman (30:21.300)
and optimization that throughout its history,
Lex Fridman (30:28.640)
what are some moments we had set back
Lex Fridman (30:31.080)
and really proud of what's been accomplished?
Chris Lattner (30:33.280)
Yeah, I think that the interesting thing about LLVM
Lex Fridman (30:36.160)
is not the innovations and compiler research.
Chris Lattner (30:40.120)
It has very good implementations
Lex Fridman (30:41.900)
of various important algorithms, no doubt.
Lex Fridman (30:44.880)
And a lot of really smart people have worked on it.
Lex Fridman (30:48.280)
But I think that the thing that's most profound about LLVM
Chris Lattner (30:50.560)
is that through standardization, it made things possible
Lex Fridman (30:53.840)
that otherwise wouldn't have happened, okay?
Lex Fridman (30:56.200)
And so interesting things that have happened with LLVM,
Lex Fridman (30:59.120)
for example, Sony has picked up LLVM
Lex Fridman (31:01.260)
and used it to do all the graphics compilation
Lex Fridman (31:03.920)
in their movie production pipeline.
Lex Fridman (31:06.080)
And so now they're able to have better special effects
Lex Fridman (31:07.920)
because of LLVM.
Chris Lattner (31:09.660)
That's kind of cool.
Lex Fridman (31:11.180)
That's not what it was designed for, right?
Lex Fridman (31:13.000)
But that's the sign of good infrastructure
Lex Fridman (31:15.480)
when it can be used in ways it was never designed for
Chris Lattner (31:18.800)
because it has good layering and software engineering
Lex Fridman (31:20.960)
and it's composable and things like that.
Chris Lattner (31:23.440)
Which is where, as you said, it differs from GCC.
Lex Fridman (31:26.120)
Yes, GCC is also great in various ways,
Lex Fridman (31:28.240)
but it's not as good as infrastructure technology.
Lex Fridman (31:31.800)
It's really a C compiler, or it's a Fortran compiler.
Chris Lattner (31:36.160)
It's not infrastructure in the same way.
Lex Fridman (31:38.920)
Now you can tell I don't know what I'm talking about
Chris Lattner (31:41.560)
because I keep saying C lang.
Lex Fridman (31:44.500)
You can always tell when a person has clues,
Chris Lattner (31:48.080)
by the way, to pronounce something.
Lex Fridman (31:49.400)
I don't think, have I ever used C lang?
Lex Fridman (31:52.580)
Entirely possible, have you?
Lex Fridman (31:54.120)
Well, so you've used code, it's generated probably.
Lex Fridman (31:58.200)
So C lang and LLVM are used to compile
Lex Fridman (32:01.760)
all the apps on the iPhone effectively and the OSs.
Chris Lattner (32:05.240)
It compiles Google's production server applications.
Lex Fridman (32:10.560)
It's used to build GameCube games and PlayStation 4
Lex Fridman (32:14.840)
and things like that.
Lex Fridman (32:16.680)
So as a user, I have, but just everything I've done
Chris Lattner (32:20.120)
that I experienced with Linux has been,
Lex Fridman (32:22.120)
I believe, always GCC.
Chris Lattner (32:23.560)
Yeah, I think Linux still defaults to GCC.
Lex Fridman (32:26.520)
And is there a reason for that?
Lex Fridman (32:27.800)
Or is it because, I mean, is there a reason for that?
Lex Fridman (32:29.440)
It's a combination of technical and social reasons.
Chris Lattner (32:32.040)
Many Linux developers do use C lang,
Lex Fridman (32:35.960)
but the distributions, for lots of reasons,
Chris Lattner (32:40.560)
use GCC historically, and they've not switched, yeah.
Lex Fridman (32:44.240)
Because it's just anecdotally online,
Chris Lattner (32:46.640)
it seems that LLVM has either reached the level of GCC
Lex Fridman (32:50.640)
or superseded on different features or whatever.
Chris Lattner (32:53.520)
The way I would say it is that they're so close,
Lex Fridman (32:55.200)
it doesn't matter.
Chris Lattner (32:56.040)
Yeah, exactly.
Lex Fridman (32:56.860)
Like, they're slightly better in some ways,
Chris Lattner (32:58.160)
slightly worse than otherwise,
Lex Fridman (32:59.160)
but it doesn't actually really matter anymore, that level.
Lex Fridman (33:03.280)
So in terms of optimization breakthroughs,
Lex Fridman (33:06.280)
it's just been solid incremental work.
Chris Lattner (33:09.160)
Yeah, yeah, which describes a lot of compilers.
Lex Fridman (33:12.520)
The hard thing about compilers, in my experience,
Chris Lattner (33:15.000)
is the engineering, the software engineering,
Lex Fridman (33:17.440)
making it so that you can have hundreds of people
Chris Lattner (33:20.160)
collaborating on really detailed, low level work
Lex Fridman (33:23.600)
and scaling that.
Lex Fridman (33:25.400)
And that's really hard.
Lex Fridman (33:27.880)
And that's one of the things I think LLVM has done well.
Lex Fridman (33:32.160)
And that kind of goes back to the original design goals
Lex Fridman (33:34.200)
with it to be modular and things like that.
Lex Fridman (33:37.200)
And incidentally, I don't want to take all the credit
Lex Fridman (33:38.880)
for this, right?
Chris Lattner (33:39.720)
I mean, some of the best parts about LLVM
Lex Fridman (33:41.760)
is that it was designed to be modular.
Lex Fridman (33:43.600)
And when I started, I would write, for example,
Lex Fridman (33:45.600)
a register allocator, and then somebody much smarter than me
Chris Lattner (33:48.500)
would come in and pull it out and replace it
Lex Fridman (33:50.720)
with something else that they would come up with.
Lex Fridman (33:52.680)
And because it's modular, they were able to do that.
Lex Fridman (33:55.200)
And that's one of the challenges with GCC, for example,
Chris Lattner (33:58.280)
is replacing subsystems is incredibly difficult.
Lex Fridman (34:01.280)
It can be done, but it wasn't designed for that.
Lex Fridman (34:04.680)
And that's one of the reasons that LLVM's been
Lex Fridman (34:06.080)
very successful in the research world as well.
Lex Fridman (34:08.760)
But in a community sense, Guido van Rossum, right,
Lex Fridman (34:12.960)
from Python, just retired from, what is it?
Lex Fridman (34:18.480)
Benevolent Dictator for Life, right?
Lex Fridman (34:20.500)
So in managing this community of brilliant compiler folks,
Chris Lattner (34:24.720)
is there, did it, for a time at least,
Lex Fridman (34:28.660)
fall on you to approve things?
Chris Lattner (34:31.480)
Oh yeah, so I mean, I still have something like
Lex Fridman (34:34.240)
an order of magnitude more patches in LLVM
Chris Lattner (34:37.980)
than anybody else, and many of those I wrote myself.
Lex Fridman (34:42.760)
But you still write, I mean, you're still close to the,
Chris Lattner (34:47.880)
to the, I don't know what the expression is,
Lex Fridman (34:49.480)
to the metal, you still write code.
Chris Lattner (34:51.000)
Yeah, I still write code.
Lex Fridman (34:52.220)
Not as much as I was able to in grad school,
Lex Fridman (34:54.240)
but that's an important part of my identity.
Lex Fridman (34:56.760)
But the way that LLVM has worked over time
Chris Lattner (34:58.880)
is that when I was a grad student, I could do all the work
Lex Fridman (35:01.360)
and steer everything and review every patch
Lex Fridman (35:04.120)
and make sure everything was done
Lex Fridman (35:05.800)
exactly the way my opinionated sense
Chris Lattner (35:09.040)
felt like it should be done, and that was fine.
Lex Fridman (35:11.760)
But as things scale, you can't do that, right?
Lex Fridman (35:14.300)
And so what ends up happening is LLVM
Lex Fridman (35:17.100)
has a hierarchical system of what's called code owners.
Chris Lattner (35:20.520)
These code owners are given the responsibility
Lex Fridman (35:22.880)
not to do all the work,
Chris Lattner (35:24.880)
not necessarily to review all the patches,
Lex Fridman (35:26.640)
but to make sure that the patches do get reviewed
Lex Fridman (35:28.800)
and make sure that the right thing's happening
Lex Fridman (35:30.320)
architecturally in their area.
Lex Fridman (35:32.160)
And so what you'll see is you'll see that, for example,
Lex Fridman (35:36.720)
hardware manufacturers end up owning
Chris Lattner (35:38.560)
the hardware specific parts of their hardware.
Lex Fridman (35:43.600)
That's very common.
Chris Lattner (35:45.520)
Leaders in the community that have done really good work
Lex Fridman (35:47.720)
naturally become the de facto owner of something.
Lex Fridman (35:50.880)
And then usually somebody else is like,
Lex Fridman (35:53.400)
how about we make them the official code owner?
Lex Fridman (35:55.520)
And then we'll have somebody to make sure
Lex Fridman (35:58.600)
that all the patches get reviewed in a timely manner.
Lex Fridman (36:00.320)
And then everybody's like, yes, that's obvious.
Lex Fridman (36:02.080)
And then it happens, right?
Lex Fridman (36:03.240)
And usually this is a very organic thing, which is great.
Lex Fridman (36:06.080)
And so I'm nominally the top of that stack still,
Lex Fridman (36:08.740)
but I don't spend a lot of time reviewing patches.
Lex Fridman (36:11.560)
What I do is I help negotiate a lot of the technical
Chris Lattner (36:16.520)
disagreements that end up happening
Lex Fridman (36:18.040)
and making sure that the community as a whole
Chris Lattner (36:19.660)
makes progress and is moving in the right direction
Lex Fridman (36:22.040)
and doing that.
Lex Fridman (36:23.920)
So we also started a nonprofit six years ago,
Lex Fridman (36:28.240)
seven years ago, time's gone away.
Lex Fridman (36:30.840)
And the LLVM Foundation nonprofit helps oversee
Lex Fridman (36:34.600)
all the business sides of things and make sure
Chris Lattner (36:36.440)
that the events that the LLVM community has
Lex Fridman (36:38.800)
are funded and set up and run correctly
Lex Fridman (36:41.600)
and stuff like that.
Lex Fridman (36:42.800)
But the foundation is very much stays out
Chris Lattner (36:45.160)
of the technical side of where the project is going.
Lex Fridman (36:49.060)
Right, so it sounds like a lot of it is just organic.
Chris Lattner (36:53.160)
Yeah, well, LLVM is almost 20 years old,
Lex Fridman (36:55.680)
which is hard to believe.
Chris Lattner (36:56.600)
Somebody pointed out to me recently that LLVM
Lex Fridman (36:59.720)
is now older than GCC was when LLVM started, right?
Lex Fridman (37:04.600)
So time has a way of getting away from you.
Lex Fridman (37:06.860)
But the good thing about that is it has a really robust,
Chris Lattner (37:10.400)
really amazing community of people that are
Lex Fridman (37:13.520)
in their professional lives, spread across lots
Chris Lattner (37:15.460)
of different companies, but it's a community
Lex Fridman (37:17.720)
of people that are interested in similar kinds of problems
Lex Fridman (37:21.120)
and have been working together effectively for years
Lex Fridman (37:23.680)
and have a lot of trust and respect for each other.
Lex Fridman (37:26.460)
And even if they don't always agree that we're able
Lex Fridman (37:29.240)
to find a path forward.
Lex Fridman (37:31.200)
So then in a slightly different flavor of effort,
Lex Fridman (37:34.480)
you started at Apple in 2005 with the task
Chris Lattner (37:38.120)
of making, I guess, LLVM production ready.
Lex Fridman (37:41.800)
And then eventually 2013 through 2017,
Chris Lattner (37:44.640)
leading the entire developer tools department.
Lex Fridman (37:48.360)
We're talking about LLVM, Xcode, Objective C to Swift.
Lex Fridman (37:53.920)
So in a quick overview of your time there,
Lex Fridman (37:58.580)
what were the challenges?
Chris Lattner (37:59.600)
First of all, leading such a huge group of developers,
Lex Fridman (38:03.240)
what was the big motivator, dream, mission
Chris Lattner (38:06.540)
behind creating Swift, the early birth of it
Lex Fridman (38:11.400)
from Objective C and so on, and Xcode,
Lex Fridman (38:13.400)
what are some challenges?
Lex Fridman (38:14.240)
So these are different questions.
Chris Lattner (38:15.900)
Yeah, I know, but I wanna talk about the other stuff too.
Lex Fridman (38:19.720)
I'll stay on the technical side,
Chris Lattner (38:21.240)
then we can talk about the big team pieces, if that's okay.
Lex Fridman (38:24.480)
So it's to really oversimplify many years of hard work.
Chris Lattner (38:29.060)
LLVM started, joined Apple, became a thing,
Lex Fridman (38:32.440)
became successful and became deployed.
Lex Fridman (38:34.600)
But then there's a question about
Lex Fridman (38:35.960)
how do we actually parse the source code?
Lex Fridman (38:38.880)
So LLVM is that back part,
Lex Fridman (38:40.320)
the optimizer and the code generator.
Lex Fridman (38:42.320)
And LLVM was really good for Apple
Lex Fridman (38:44.060)
as it went through a couple of harder transitions.
Chris Lattner (38:46.060)
I joined right at the time of the Intel transition,
Lex Fridman (38:47.960)
for example, and 64 bit transitions,
Lex Fridman (38:51.820)
and then the transition to ARM with the iPhone.
Lex Fridman (38:53.500)
And so LLVM was very useful
Chris Lattner (38:54.720)
for some of these kinds of things.
Lex Fridman (38:57.000)
But at the same time, there's a lot of questions
Chris Lattner (38:58.480)
around developer experience.
Lex Fridman (39:00.120)
And so if you're a programmer pounding out
Chris Lattner (39:01.960)
at the time Objective C code,
Lex Fridman (39:04.480)
the error message you get, the compile time,
Chris Lattner (39:06.520)
the turnaround cycle, the tooling and the IDE,
Lex Fridman (39:09.760)
were not great, were not as good as they could be.
Lex Fridman (39:13.000)
And so, as I occasionally do, I'm like,
Lex Fridman (39:18.080)
well, okay, how hard is it to write a C compiler?
Lex Fridman (39:20.720)
And so I'm not gonna commit to anybody,
Lex Fridman (39:22.560)
I'm not gonna tell anybody, I'm just gonna just do it
Chris Lattner (39:25.320)
nights and weekends and start working on it.
Lex Fridman (39:27.480)
And then I built up in C,
Chris Lattner (39:29.740)
there's this thing called the preprocessor,
Lex Fridman (39:31.160)
which people don't like,
Lex Fridman (39:33.040)
but it's actually really hard and complicated
Lex Fridman (39:35.480)
and includes a bunch of really weird things
Chris Lattner (39:37.700)
like trigraphs and other stuff like that
Lex Fridman (39:39.280)
that are really nasty,
Lex Fridman (39:40.960)
and it's the crux of a bunch of the performance issues
Lex Fridman (39:44.080)
in the compiler.
Chris Lattner (39:45.640)
Started working on the parser
Lex Fridman (39:46.640)
and kind of got to the point where I'm like,
Chris Lattner (39:47.800)
ah, you know what, we could actually do this.
Lex Fridman (39:49.880)
Everybody's saying that this is impossible to do,
Lex Fridman (39:51.460)
but it's actually just hard, it's not impossible.
Lex Fridman (39:53.960)
And eventually told my manager about it,
Lex Fridman (39:57.560)
and he's like, oh, wow, this is great,
Lex Fridman (39:59.220)
we do need to solve this problem.
Chris Lattner (40:00.360)
Oh, this is great, we can get you one other person
Lex Fridman (40:02.560)
to work with you on this, you know?
Lex Fridman (40:04.440)
And slowly a team is formed and it starts taking off.
Lex Fridman (40:08.360)
And C++, for example, huge, complicated language.
Chris Lattner (40:12.040)
People always assume that it's impossible to implement
Lex Fridman (40:14.360)
and it's very nearly impossible,
Lex Fridman (40:16.260)
but it's just really, really hard.
Lex Fridman (40:18.720)
And the way to get there is to build it
Chris Lattner (40:20.840)
one piece at a time incrementally.
Lex Fridman (40:22.480)
And that was only possible because we were lucky
Chris Lattner (40:26.440)
to hire some really exceptional engineers
Lex Fridman (40:28.160)
that knew various parts of it very well
Lex Fridman (40:30.380)
and could do great things.
Lex Fridman (40:32.680)
Swift was kind of a similar thing.
Lex Fridman (40:34.440)
So Swift came from, we were just finishing off
Lex Fridman (40:39.160)
the first version of C++ support in Clang.
Lex Fridman (40:42.600)
And C++ is a very formidable and very important language,
Lex Fridman (40:47.260)
but it's also ugly in lots of ways.
Lex Fridman (40:49.280)
And you can't influence C++ without thinking
Lex Fridman (40:52.320)
there has to be a better thing, right?
Lex Fridman (40:54.380)
And so I started working on Swift, again,
Lex Fridman (40:56.120)
with no hope or ambition that would go anywhere,
Chris Lattner (40:58.560)
just let's see what could be done,
Lex Fridman (41:00.800)
let's play around with this thing.
Chris Lattner (41:02.620)
It was me in my spare time, not telling anybody about it,
Lex Fridman (41:06.700)
kind of a thing, and it made some good progress.
Chris Lattner (41:09.420)
I'm like, actually, it would make sense to do this.
Lex Fridman (41:11.260)
At the same time, I started talking with the senior VP
Chris Lattner (41:14.800)
of software at the time, a guy named Bertrand Serlet.
Lex Fridman (41:17.720)
And Bertrand was very encouraging.
Chris Lattner (41:19.280)
He was like, well, let's have fun, let's talk about this.
Lex Fridman (41:22.080)
And he was a little bit of a language guy,
Lex Fridman (41:23.440)
and so he helped guide some of the early work
Lex Fridman (41:26.160)
and encouraged me and got things off the ground.
Lex Fridman (41:30.420)
And eventually told my manager and told other people,
Lex Fridman (41:34.280)
and it started making progress.
Chris Lattner (41:38.800)
The complicating thing with Swift
Lex Fridman (41:40.960)
was that the idea of doing a new language
Chris Lattner (41:43.880)
was not obvious to anybody, including myself.
Lex Fridman (41:47.840)
And the tone at the time was that the iPhone
Chris Lattner (41:50.240)
was successful because of Objective C.
Lex Fridman (41:53.440)
Oh, interesting.
Chris Lattner (41:54.440)
Not despite of or just because of.
Lex Fridman (41:57.160)
And you have to understand that at the time,
Chris Lattner (42:01.160)
Apple was hiring software people that loved Objective C.
Lex Fridman (42:05.400)
And it wasn't that they came despite Objective C.
Chris Lattner (42:07.960)
They loved Objective C, and that's why they got hired.
Lex Fridman (42:10.240)
And so you had a software team that the leadership,
Chris Lattner (42:13.080)
in many cases, went all the way back to Next,
Lex Fridman (42:15.200)
where Objective C really became real.
Lex Fridman (42:19.400)
And so they, quote unquote, grew up writing Objective C.
Lex Fridman (42:23.240)
And many of the individual engineers
Chris Lattner (42:25.720)
all were hired because they loved Objective C.
Lex Fridman (42:28.360)
And so this notion of, OK, let's do new language
Chris Lattner (42:30.560)
was kind of heretical in many ways.
Lex Fridman (42:34.120)
Meanwhile, my sense was that the outside community wasn't really
Chris Lattner (42:36.960)
in love with Objective C. Some people were,
Lex Fridman (42:38.560)
and some of the most outspoken people were.
Lex Fridman (42:40.360)
But other people were hitting challenges
Lex Fridman (42:42.620)
because it has very sharp corners
Lex Fridman (42:44.760)
and it's difficult to learn.
Lex Fridman (42:46.840)
And so one of the challenges of making Swift happen that
Lex Fridman (42:50.160)
was totally non technical is the social part of what do we do?
Lex Fridman (42:57.720)
If we do a new language, which at Apple, many things
Chris Lattner (43:00.320)
happen that don't ship.
Lex Fridman (43:02.240)
So if we ship it, what is the metrics of success?
Lex Fridman (43:05.560)
Why would we do this?
Lex Fridman (43:06.400)
Why wouldn't we make Objective C better?
Chris Lattner (43:08.060)
If Objective C has problems, let's file off
Lex Fridman (43:10.160)
those rough corners and edges.
Lex Fridman (43:12.160)
And one of the major things that became the reason to do this
Lex Fridman (43:15.640)
was this notion of safety, memory safety.
Lex Fridman (43:18.960)
And the way Objective C works is that a lot of the object system
Lex Fridman (43:23.240)
and everything else is built on top of pointers in C.
Chris Lattner (43:27.560)
Objective C is an extension on top of C.
Lex Fridman (43:29.960)
And so pointers are unsafe.
Lex Fridman (43:32.680)
And if you get rid of the pointers,
Lex Fridman (43:34.640)
it's not Objective C anymore.
Lex Fridman (43:36.480)
And so fundamentally, that was an issue
Lex Fridman (43:39.080)
that you could not fix safety or memory safety
Chris Lattner (43:42.200)
without fundamentally changing the language.
Lex Fridman (43:45.640)
And so once we got through that part of the mental process
Lex Fridman (43:49.920)
and the thought process, it became a design process
Lex Fridman (43:53.200)
of saying, OK, well, if we're going to do something new,
Lex Fridman (43:55.400)
what is good?
Lex Fridman (43:56.280)
How do we think about this?
Lex Fridman (43:57.400)
And what do we like?
Lex Fridman (43:58.200)
And what are we looking for?
Lex Fridman (44:00.040)
And that was a very different phase of it.
Lex Fridman (44:02.440)
So what are some design choices early on in Swift?
Chris Lattner (44:05.960)
Like we're talking about braces, are you
Lex Fridman (44:10.120)
making a typed language or not, all those kinds of things.
Chris Lattner (44:13.240)
Yeah, so some of those were obvious given the context.
Lex Fridman (44:16.040)
So a typed language, for example,
Chris Lattner (44:17.800)
Objective C is a typed language.
Lex Fridman (44:19.200)
And going with an untyped language
Chris Lattner (44:22.480)
wasn't really seriously considered.
Lex Fridman (44:24.320)
We wanted the performance, and we
Chris Lattner (44:26.000)
wanted refactoring tools and other things
Lex Fridman (44:27.680)
like that that go with typed languages.
Chris Lattner (44:29.600)
Quick, dumb question.
Lex Fridman (44:31.440)
Was it obvious, I think this would be a dumb question,
Lex Fridman (44:34.600)
but was it obvious that the language
Lex Fridman (44:36.360)
has to be a compiled language?
Chris Lattner (44:40.120)
Yes, that's not a dumb question.
Lex Fridman (44:42.080)
Earlier, I think late 90s, Apple had seriously
Chris Lattner (44:44.520)
considered moving its development experience to Java.
Lex Fridman (44:49.000)
But Swift started in 2010, which was several years
Chris Lattner (44:53.160)
after the iPhone.
Lex Fridman (44:53.880)
It was when the iPhone was definitely
Chris Lattner (44:55.380)
on an upward trajectory.
Lex Fridman (44:56.640)
And the iPhone was still extremely,
Lex Fridman (44:58.760)
and is still a bit memory constrained.
Lex Fridman (45:01.800)
And so being able to compile the code
Lex Fridman (45:04.440)
and then ship it and then having standalone code that
Lex Fridman (45:08.160)
is not JIT compiled is a very big deal
Lex Fridman (45:11.320)
and is very much part of the Apple value system.
Lex Fridman (45:15.200)
Now, JavaScript's also a thing.
Chris Lattner (45:17.480)
I mean, it's not that this is exclusive,
Lex Fridman (45:19.360)
and technologies are good depending
Chris Lattner (45:21.640)
on how they're applied.
Lex Fridman (45:23.880)
But in the design of Swift, saying,
Lex Fridman (45:26.600)
how can we make Objective C better?
Lex Fridman (45:28.320)
Objective C is statically compiled,
Lex Fridman (45:29.760)
and that was the contiguous, natural thing to do.
Lex Fridman (45:32.520)
Just skip ahead a little bit, and we'll go right back.
Chris Lattner (45:35.360)
Just as a question, as you think about today in 2019
Lex Fridman (45:40.040)
in your work at Google, TensorFlow and so on,
Chris Lattner (45:42.400)
is, again, compilations, static compilation still
Lex Fridman (45:48.600)
the right thing?
Chris Lattner (45:49.460)
Yeah, so the funny thing after working
Lex Fridman (45:52.000)
on compilers for a really long time is that,
Lex Fridman (45:55.880)
and this is one of the things that LLVM has helped with,
Lex Fridman (45:59.040)
is that I don't look at compilations
Chris Lattner (46:01.440)
being static or dynamic or interpreted or not.
Lex Fridman (46:05.240)
This is a spectrum.
Lex Fridman (46:07.680)
And one of the cool things about Swift
Lex Fridman (46:09.140)
is that Swift is not just statically compiled.
Chris Lattner (46:12.160)
It's actually dynamically compiled as well,
Lex Fridman (46:14.080)
and it can also be interpreted.
Chris Lattner (46:15.320)
Though, nobody's actually done that.
Lex Fridman (46:17.440)
And so what ends up happening when
Chris Lattner (46:20.400)
you use Swift in a workbook, for example in Colab or in Jupyter,
Lex Fridman (46:24.080)
is it's actually dynamically compiling the statements
Chris Lattner (46:26.360)
as you execute them.
Lex Fridman (46:28.160)
And so this gets back to the software engineering problems,
Chris Lattner (46:32.840)
where if you layer the stack properly,
Lex Fridman (46:34.960)
you can actually completely change
Lex Fridman (46:37.320)
how and when things get compiled because you
Lex Fridman (46:39.360)
have the right abstractions there.
Lex Fridman (46:41.120)
And so the way that a Colab workbook works with Swift
Lex Fridman (46:44.800)
is that when you start typing into it,
Chris Lattner (46:47.720)
it creates a process, a Unix process.
Lex Fridman (46:50.280)
And then each line of code you type in,
Chris Lattner (46:52.160)
it compiles it through the Swift compiler, the front end part,
Lex Fridman (46:56.120)
and then sends it through the optimizer,
Chris Lattner (46:58.360)
JIT compiles machine code, and then
Lex Fridman (47:01.120)
injects it into that process.
Lex Fridman (47:03.800)
And so as you're typing new stuff,
Lex Fridman (47:05.400)
it's like squirting in new code and overwriting and replacing
Lex Fridman (47:09.360)
and updating code in place.
Lex Fridman (47:11.200)
And the fact that it can do this is not an accident.
Chris Lattner (47:13.680)
Swift was designed for this.
Lex Fridman (47:15.560)
But it's an important part of how the language was set up
Lex Fridman (47:18.120)
and how it's layered, and this is a nonobvious piece.
Lex Fridman (47:21.320)
And one of the things with Swift that
Chris Lattner (47:23.160)
was, for me, a very strong design point
Lex Fridman (47:25.880)
is to make it so that you can learn it very quickly.
Lex Fridman (47:29.640)
And so from a language design perspective,
Lex Fridman (47:31.880)
the thing that I always come back to
Chris Lattner (47:33.340)
is this UI principle of progressive disclosure
Lex Fridman (47:36.440)
of complexity.
Lex Fridman (47:37.960)
And so in Swift, you can start by saying print, quote,
Lex Fridman (47:41.680)
hello world, quote.
Lex Fridman (47:44.040)
And there's no slash n, just like Python, one line of code,
Lex Fridman (47:47.160)
no main, no header files, no public static class void,
Chris Lattner (47:51.520)
blah, blah, blah, string like Java has, one line of code.
Lex Fridman (47:55.640)
And you can teach that, and it works great.
Chris Lattner (47:58.400)
Then you can say, well, let's introduce variables.
Lex Fridman (48:00.400)
And so you can declare a variable with var.
Lex Fridman (48:02.400)
So var x equals 4.
Lex Fridman (48:03.780)
What is a variable?
Chris Lattner (48:04.700)
You can use x, x plus 1.
Lex Fridman (48:06.280)
This is what it means.
Lex Fridman (48:07.600)
Then you can say, well, how about control flow?
Lex Fridman (48:09.520)
Well, this is what an if statement is.
Chris Lattner (48:10.860)
This is what a for statement is.
Lex Fridman (48:12.280)
This is what a while statement is.
Chris Lattner (48:15.280)
Then you can say, let's introduce functions.
Lex Fridman (48:17.280)
And many languages like Python have
Chris Lattner (48:20.020)
had this kind of notion of let's introduce small things,
Lex Fridman (48:22.820)
and then you can add complexity.
Chris Lattner (48:24.400)
Then you can introduce classes.
Lex Fridman (48:25.760)
And then you can add generics, in the case of Swift.
Lex Fridman (48:28.040)
And then you can build in modules
Lex Fridman (48:29.520)
and build out in terms of the things that you're expressing.
Lex Fridman (48:32.200)
But this is not very typical for compiled languages.
Lex Fridman (48:35.800)
And so this was a very strong design point,
Lex Fridman (48:38.000)
and one of the reasons that Swift, in general,
Lex Fridman (48:40.960)
is designed with this factoring of complexity in mind
Lex Fridman (48:43.480)
so that the language can express powerful things.
Lex Fridman (48:46.440)
You can write firmware in Swift if you want to.
Lex Fridman (48:49.280)
But it has a very high level feel,
Lex Fridman (48:51.900)
which is really this perfect blend, because often you
Chris Lattner (48:55.200)
have very advanced library writers that
Lex Fridman (48:57.520)
want to be able to use the nitty gritty details.
Lex Fridman (49:00.520)
But then other people just want to use the libraries
Lex Fridman (49:02.960)
and work at a higher abstraction level.
Chris Lattner (49:04.880)
It's kind of cool that I saw that you can just
Lex Fridman (49:07.240)
interoperability.
Chris Lattner (49:09.240)
I don't think I pronounced that word enough.
Lex Fridman (49:11.320)
But you can just drag in Python.
Chris Lattner (49:14.960)
It's just strange.
Lex Fridman (49:16.000)
You can import, like I saw this in the demo.
Lex Fridman (49:19.640)
How do you make that happen?
Lex Fridman (49:21.280)
What's up with that?
Lex Fridman (49:23.120)
Is that as easy as it looks, or is it?
Lex Fridman (49:25.560)
Yes, as easy as it looks.
Chris Lattner (49:27.000)
That's not a stage magic hack or anything like that.
Lex Fridman (49:29.600)
I don't mean from the user perspective.
Chris Lattner (49:31.400)
I mean from the implementation perspective to make it happen.
Lex Fridman (49:34.120)
So it's easy once all the pieces are in place.
Chris Lattner (49:37.000)
The way it works, so if you think about a dynamically typed
Lex Fridman (49:39.280)
language like Python, you can think about it
Chris Lattner (49:41.480)
in two different ways.
Lex Fridman (49:42.360)
You can say it has no types, which
Chris Lattner (49:45.800)
is what most people would say.
Lex Fridman (49:47.480)
Or you can say it has one type.
Lex Fridman (49:50.400)
And you can say it has one type, and it's the Python object.
Lex Fridman (49:53.320)
And the Python object gets passed around.
Lex Fridman (49:55.000)
And because there's only one type, it's implicit.
Lex Fridman (49:58.200)
And so what happens with Swift and Python talking
Chris Lattner (50:00.880)
to each other, Swift has lots of types.
Lex Fridman (50:02.760)
It has arrays, and it has strings, and all classes,
Lex Fridman (50:05.840)
and that kind of stuff.
Lex Fridman (50:07.000)
But it now has a Python object type.
Lex Fridman (50:11.120)
So there is one Python object type.
Lex Fridman (50:12.720)
And so when you say import NumPy, what you get
Chris Lattner (50:16.440)
is a Python object, which is the NumPy module.
Lex Fridman (50:19.840)
And then you say np.array.
Chris Lattner (50:21.960)
It says, OK, hey, Python object, I have no idea what you are.
Lex Fridman (50:24.960)
Give me your array member.
Chris Lattner (50:27.280)
OK, cool.
Lex Fridman (50:27.960)
And it just uses dynamic stuff, talks to the Python interpreter,
Lex Fridman (50:31.160)
and says, hey, Python, what's the.array member
Lex Fridman (50:33.680)
in that Python object?
Chris Lattner (50:35.720)
It gives you back another Python object.
Lex Fridman (50:37.400)
And now you say parentheses for the call and the arguments
Chris Lattner (50:40.040)
you're going to pass.
Lex Fridman (50:40.920)
And so then it says, hey, a Python object
Chris Lattner (50:43.520)
that is the result of np.array, call with these arguments.
Lex Fridman (50:47.840)
Again, calling into the Python interpreter to do that work.
Lex Fridman (50:50.320)
And so right now, this is all really simple.
Lex Fridman (50:53.680)
And if you dive into the code, what you'll see
Chris Lattner (50:55.960)
is that the Python module in Swift
Lex Fridman (50:58.440)
is something like 1,200 lines of code or something.
Chris Lattner (51:01.360)
It's written in pure Swift.
Lex Fridman (51:02.400)
It's super simple.
Lex Fridman (51:03.560)
And it's built on top of the C interoperability
Lex Fridman (51:06.560)
because it just talks to the Python interpreter.
Lex Fridman (51:09.520)
But making that possible required
Lex Fridman (51:11.080)
us to add two major language features to Swift
Chris Lattner (51:13.480)
to be able to express these dynamic calls
Lex Fridman (51:15.400)
and the dynamic member lookups.
Lex Fridman (51:17.240)
And so what we've done over the last year
Lex Fridman (51:19.480)
is we've proposed, implement, standardized, and contributed
Chris Lattner (51:23.960)
new language features to the Swift language
Lex Fridman (51:26.160)
in order to make it so it is really trivial.
Lex Fridman (51:29.560)
And this is one of the things about Swift
Lex Fridman (51:31.320)
that is critical to the Swift for TensorFlow work, which
Chris Lattner (51:35.000)
is that we can actually add new language features.
Lex Fridman (51:37.200)
And the bar for adding those is high,
Lex Fridman (51:39.160)
but it's what makes it possible.
Lex Fridman (51:42.280)
So you're now at Google doing incredible work
Chris Lattner (51:45.240)
on several things, including TensorFlow.
Lex Fridman (51:47.680)
So TensorFlow 2.0 or whatever leading up to 2.0 has,
Chris Lattner (51:53.080)
by default, in 2.0, has eager execution.
Lex Fridman (51:56.840)
And yet, in order to make code optimized for GPU or TPU
Chris Lattner (52:00.520)
or some of these systems, computation
Lex Fridman (52:04.120)
needs to be converted to a graph.
Lex Fridman (52:06.000)
So what's that process like?
Lex Fridman (52:07.440)
What are the challenges there?
Chris Lattner (52:08.960)
Yeah, so I am tangentially involved in this.
Lex Fridman (52:11.720)
But the way that it works with Autograph
Chris Lattner (52:15.280)
is that you mark your function with a decorator.
Lex Fridman (52:21.600)
And when Python calls it, that decorator is invoked.
Lex Fridman (52:24.280)
And then it says, before I call this function,
Lex Fridman (52:28.240)
you can transform it.
Lex Fridman (52:29.480)
And so the way Autograph works is, as far as I understand,
Lex Fridman (52:32.400)
is it actually uses the Python parser
Chris Lattner (52:34.440)
to go parse that, turn it into a syntax tree,
Lex Fridman (52:37.160)
and now apply compiler techniques to, again,
Chris Lattner (52:39.400)
transform this down into TensorFlow graphs.
Lex Fridman (52:42.320)
And so you can think of it as saying, hey,
Chris Lattner (52:44.920)
I have an if statement.
Lex Fridman (52:45.880)
I'm going to create an if node in the graph,
Chris Lattner (52:48.360)
like you say tf.cond.
Lex Fridman (52:51.080)
You have a multiply.
Chris Lattner (52:53.040)
Well, I'll turn that into a multiply node in the graph.
Lex Fridman (52:55.320)
And it becomes this tree transformation.
Lex Fridman (52:57.760)
So where does the Swift for TensorFlow
Lex Fridman (53:00.480)
come in, which is parallels?
Chris Lattner (53:04.960)
For one, Swift is an interface.
Lex Fridman (53:06.960)
Like, Python is an interface to TensorFlow.
Lex Fridman (53:09.200)
But it seems like there's a lot more going on in just
Lex Fridman (53:11.760)
a different language interface.
Chris Lattner (53:13.120)
There's optimization methodology.
Lex Fridman (53:15.960)
So the TensorFlow world has a couple
Chris Lattner (53:17.920)
of different what I'd call front end technologies.
Lex Fridman (53:21.240)
And so Swift and Python and Go and Rust and Julia
Lex Fridman (53:25.240)
and all these things share the TensorFlow graphs
Lex Fridman (53:29.320)
and all the runtime and everything that's later.
Lex Fridman (53:32.760)
And so Swift for TensorFlow is merely another front end
Lex Fridman (53:36.640)
for TensorFlow, just like any of these other systems are.
Chris Lattner (53:40.640)
There's a major difference between, I would say,
Lex Fridman (53:43.080)
three camps of technologies here.
Chris Lattner (53:44.600)
There's Python, which is a special case,
Lex Fridman (53:46.880)
because the vast majority of the community effort
Chris Lattner (53:49.160)
is going to the Python interface.
Lex Fridman (53:51.120)
And Python has its own approaches
Chris Lattner (53:52.920)
for automatic differentiation.
Lex Fridman (53:54.480)
It has its own APIs and all this kind of stuff.
Chris Lattner (53:58.160)
There's Swift, which I'll talk about in a second.
Lex Fridman (54:00.320)
And then there's kind of everything else.
Lex Fridman (54:02.040)
And so the everything else are effectively language bindings.
Lex Fridman (54:05.400)
So they call into the TensorFlow runtime,
Lex Fridman (54:07.960)
but they usually don't have automatic differentiation
Lex Fridman (54:10.920)
or they usually don't provide anything other than APIs
Chris Lattner (54:14.560)
that call the C APIs in TensorFlow.
Lex Fridman (54:16.440)
And so they're kind of wrappers for that.
Chris Lattner (54:18.360)
Swift is really kind of special.
Lex Fridman (54:19.840)
And it's a very different approach.
Chris Lattner (54:22.760)
Swift for TensorFlow, that is, is a very different approach.
Lex Fridman (54:25.360)
Because there we're saying, let's
Chris Lattner (54:26.880)
look at all the problems that need
Lex Fridman (54:28.400)
to be solved in the full stack of the TensorFlow compilation
Chris Lattner (54:34.080)
process, if you think about it that way.
Lex Fridman (54:35.680)
Because TensorFlow is fundamentally a compiler.
Chris Lattner (54:38.200)
It takes models, and then it makes them go fast on hardware.
Lex Fridman (54:42.760)
That's what a compiler does.
Lex Fridman (54:43.880)
And it has a front end, it has an optimizer,
Lex Fridman (54:47.560)
and it has many back ends.
Lex Fridman (54:49.320)
And so if you think about it the right way,
Lex Fridman (54:51.680)
or if you look at it in a particular way,
Chris Lattner (54:54.800)
it is a compiler.
Lex Fridman (54:59.280)
And so Swift is merely another front end.
Lex Fridman (55:02.120)
But it's saying, and the design principle is saying,
Lex Fridman (55:05.560)
let's look at all the problems that we face as machine
Chris Lattner (55:08.240)
learning practitioners and what is the best possible way we
Lex Fridman (55:11.320)
can do that, given the fact that we can change literally
Chris Lattner (55:13.840)
anything in this entire stack.
Lex Fridman (55:15.920)
And Python, for example, where the vast majority
Chris Lattner (55:18.440)
of the engineering and effort has gone into,
Lex Fridman (55:22.600)
is constrained by being the best possible thing you
Chris Lattner (55:25.000)
can do with a Python library.
Lex Fridman (55:27.320)
There are no Python language features
Chris Lattner (55:29.320)
that are added because of machine learning
Lex Fridman (55:31.040)
that I'm aware of.
Chris Lattner (55:32.600)
They added a matrix multiplication operator
Lex Fridman (55:34.640)
with that, but that's as close as you get.
Lex Fridman (55:38.320)
And so with Swift, it's hard, but you
Lex Fridman (55:41.460)
can add language features to the language.
Lex Fridman (55:43.800)
And there's a community process for that.
Lex Fridman (55:46.040)
And so we look at these things and say, well,
Lex Fridman (55:48.200)
what is the right division of labor
Lex Fridman (55:49.720)
between the human programmer and the compiler?
Lex Fridman (55:52.000)
And Swift has a number of things that shift that balance.
Lex Fridman (55:55.280)
So because it has a type system, for example,
Chris Lattner (56:00.560)
that makes certain things possible for analysis
Lex Fridman (56:02.680)
of the code, and the compiler can automatically
Chris Lattner (56:05.560)
build graphs for you without you thinking about them.
Lex Fridman (56:08.880)
That's a big deal for a programmer.
Chris Lattner (56:10.520)
You just get free performance.
Lex Fridman (56:11.680)
You get clustering and fusion and optimization,
Chris Lattner (56:14.400)
things like that, without you as a programmer
Lex Fridman (56:17.040)
having to manually do it because the compiler can do it for you.
Chris Lattner (56:20.080)
Automatic differentiation is another big deal.
Lex Fridman (56:22.240)
And I think one of the key contributions of the Swift
Chris Lattner (56:25.960)
TensorFlow project is that there's
Lex Fridman (56:29.640)
this entire body of work on automatic differentiation
Chris Lattner (56:32.120)
that dates back to the Fortran days.
Lex Fridman (56:34.120)
People doing a tremendous amount of numerical computing
Chris Lattner (56:36.400)
in Fortran used to write these what they call source
Lex Fridman (56:39.360)
to source translators, where you take a bunch of code,
Chris Lattner (56:43.280)
shove it into a mini compiler, and it would push out
Lex Fridman (56:46.640)
more Fortran code.
Lex Fridman (56:48.080)
But it would generate the backwards passes
Lex Fridman (56:50.240)
for your functions for you, the derivatives.
Lex Fridman (56:53.000)
And so in that work in the 70s, a tremendous number
Lex Fridman (56:57.840)
of optimizations, a tremendous number of techniques
Chris Lattner (57:01.160)
for fixing numerical instability,
Lex Fridman (57:02.920)
and other kinds of problems were developed.
Lex Fridman (57:05.080)
But they're very difficult to port into a world
Lex Fridman (57:07.600)
where, in eager execution, you get an op by op at a time.
Chris Lattner (57:11.280)
You need to be able to look at an entire function
Lex Fridman (57:13.280)
and be able to reason about what's going on.
Lex Fridman (57:15.720)
And so when you have a language integrated automatic
Lex Fridman (57:18.720)
differentiation, which is one of the things
Chris Lattner (57:20.520)
that the Swift project is focusing on,
Lex Fridman (57:22.760)
you can open all these techniques
Lex Fridman (57:24.680)
and reuse them in familiar ways.
Lex Fridman (57:28.640)
But the language integration piece
Chris Lattner (57:30.120)
has a bunch of design room in it, and it's also complicated.
Lex Fridman (57:33.240)
The other piece of the puzzle here that's kind of interesting
Chris Lattner (57:35.680)
is TPUs at Google.
Lex Fridman (57:37.560)
So we're in a new world with deep learning.
Chris Lattner (57:40.200)
It constantly is changing, and I imagine,
Lex Fridman (57:42.960)
without disclosing anything, I imagine
Chris Lattner (57:46.360)
you're still innovating on the TPU front, too.
Lex Fridman (57:48.400)
Indeed.
Lex Fridman (57:49.040)
So how much interplay is there between software and hardware
Lex Fridman (57:53.560)
in trying to figure out how to together move
Lex Fridman (57:55.240)
towards an optimized solution?
Lex Fridman (57:56.680)
There's an incredible amount.
Lex Fridman (57:57.760)
So we're on our third generation of TPUs,
Lex Fridman (57:59.480)
which are now 100 petaflops in a very large liquid cooled box,
Chris Lattner (58:04.640)
virtual box with no cover.
Lex Fridman (58:07.720)
And as you might imagine, we're not out of ideas yet.
Chris Lattner (58:11.240)
The great thing about TPUs is that they're
Lex Fridman (58:14.360)
a perfect example of hardware software co design.
Lex Fridman (58:17.520)
And so it's about saying, what hardware
Lex Fridman (58:19.800)
do we build to solve certain classes of machine learning
Lex Fridman (58:23.240)
problems?
Lex Fridman (58:23.840)
Well, the algorithms are changing.
Chris Lattner (58:26.480)
The hardware takes some cases years to produce.
Lex Fridman (58:30.360)
And so you have to make bets and decide
Lex Fridman (58:32.760)
what is going to happen and what is the best way to spend
Lex Fridman (58:36.520)
the transistors to get the maximum performance per watt
Chris Lattner (58:39.920)
or area per cost or whatever it is that you're optimizing for.
Lex Fridman (58:44.000)
And so one of the amazing things about TPUs
Chris Lattner (58:46.560)
is this numeric format called bfloat16.
Lex Fridman (58:49.960)
bfloat16 is a compressed 16 bit floating point format,
Lex Fridman (58:54.120)
but it puts the bits in different places.
Lex Fridman (58:55.960)
And in numeric terms, it has a smaller mantissa
Lex Fridman (58:58.960)
and a larger exponent.
Lex Fridman (59:00.400)
That means that it's less precise,
Lex Fridman (59:02.960)
but it can represent larger ranges of values,
Lex Fridman (59:05.680)
which in the machine learning context
Chris Lattner (59:07.280)
is really important and useful because sometimes you
Lex Fridman (59:09.960)
have very small gradients you want to accumulate
Lex Fridman (59:13.920)
and very, very small numbers that
Lex Fridman (59:17.480)
are important to move things as you're learning.
Lex Fridman (59:20.520)
But sometimes you have very large magnitude numbers as well.
Lex Fridman (59:23.160)
And bfloat16 is not as precise.
Chris Lattner (59:26.880)
The mantissa is small.
Lex Fridman (59:28.040)
But it turns out the machine learning algorithms actually
Chris Lattner (59:30.360)
want to generalize.
Lex Fridman (59:31.520)
And so there's theories that this actually
Chris Lattner (59:34.320)
increases the ability for the network
Lex Fridman (59:36.440)
to generalize across data sets.
Lex Fridman (59:37.960)
And regardless of whether it's good or bad,
Lex Fridman (59:41.160)
it's much cheaper at the hardware level to implement
Chris Lattner (59:43.680)
because the area and time of a multiplier
Lex Fridman (59:48.080)
is n squared in the number of bits in the mantissa,
Lex Fridman (59:50.840)
but it's linear with size of the exponent.
Lex Fridman (59:53.320)
And you're connected to both efforts
Lex Fridman (59:55.400)
here both on the hardware and the software side?
Lex Fridman (59:57.160)
Yeah, and so that was a breakthrough
Chris Lattner (59:58.880)
coming from the research side and people
🔗 相关节目