David Patterson: Computer Architecture and Data Storage
技术与编程政治与社会AI 与机器学习历史与文明音乐与艺术
📋 章节目录
暂无章节信息
🔑 关键词
instructiongoinginstructionssoftwarecomputermoorelawhardwareriscdonmachinecalledsaidriskcomputersfasterlearninggotbettercomputing
💬 精彩语录
"into a really bad professor, a really bad teacher, I think the students think, well, this guy must be"
变成一个非常糟糕的教授，一个非常糟糕的老师，我想学生们会想，好吧，这个家伙一定是
— David Patterson （1:37:44.800）
"prediction, he wrote a paper back in, I think, in the 70s and said, not only did this going to happen,"
预测，他在 70 年代写了一篇论文，说，这不仅会发生，
— David Patterson （07:58.720）
"going to increase the demand for computing, because instead of programmers being smart, writing those"
将会增加对计算的需求，因为程序员不再聪明，而是编写这些
— David Patterson （1:04:21.440）
"SL. I have to talk about this because I've wrestled. I do martial arts. Of course, I love wrestling. I'm a huge, I'm Russian. So I've talked to Dan Gable on the podcast."
SL。我必须谈论这个，因为我摔跤过。我是练武术的当然，我喜欢摔跤。我是个大块头，我是俄罗斯人。我在播客上与丹·盖博进行了交谈。
— David Patterson （1:40:58.320）
"harder to go to UCLA than the community college. And I asked, why did I make that decision? Because I"
去加州大学洛杉矶分校比去社区大学更难。我问，我为什么做出这个决定？因为我
— David Patterson （1:41:52.000）
🎙️ 完整对话（1087 条）
Lex Fridman (00:00.000)
The following is a conversation with David Patterson, touring award winner and professor
以下是与巡演奖得主、教授 David Patterson 的对话
Lex Fridman (00:05.440)
of computer science at Berkeley. He's known for pioneering contributions to RISC processor
伯克利计算机科学博士。他因对 RISC 处理器的开创性贡献而闻名
Lex Fridman (00:10.800)
architecture used by 99% of new chips today and for co creating RAID storage. The impact that
当今 99% 的新芯片都使用该架构，并用于共同创建 RAID 存储。所造成的影响
Lex Fridman (00:18.800)
these two lines of research and development have had in our world is immeasurable. He's also one of
这两条线的研究和发展在我们的世界上已经是不可估量的。他也是其中之一
Lex Fridman (00:25.040)
the great educators of computer science in the world. His book with John Hennessy is how I first
世界上伟大的计算机科学教育家。我第一次读他与约翰·轩尼诗合作的书
David Patterson (00:30.160)
learned about and was humbled by the inner workings of machines at the lowest level.
了解最底层机器的内部工作原理并感到谦卑。
Lex Fridman (01:21.120)
This episode is supported by the Jordan Harbinger Show.
本集得到乔丹先驱秀的支持。
David Patterson (01:24.400)
Go to Jordan Harbinger.com slash Lex. It's how he knows I sent you on that page. There's links
前往 Jordan Harbinger.com 斜线 Lex。他就是这样知道我在那个页面上发送给你的。有链接哦
David Patterson (01:30.800)
to subscribe to it on Apple podcast, Spotify, and everywhere else. I've been binging on this podcast.
在 Apple 播客、Spotify 和其他地方订阅它。我一直在沉迷于这个播客。
David Patterson (01:36.560)
It's amazing. Jordan is a great human being. He gets the best out of his guests, dives deep,
太棒了。乔丹是一个伟大的人。他充分发挥客人的潜力，深入研究，
David Patterson (01:41.600)
calls them out when it's needed, and makes the whole thing fun to listen to. He's interviewed
在需要的时候把他们叫出来，让整个事情听起来很有趣。他接受采访
David Patterson (01:46.080)
Kobe Bryant, Mark Cuban, Neil deGrasse Tyson, Garry Kasparov, and many more. I recently listened
科比·布莱恩特、马克·库班、尼尔·德格拉斯·泰森、加里·卡斯帕罗夫等等。我最近听过
David Patterson (01:52.560)
to his conversation with Frank Abagnale, author of Catch Me If You Can, and one of the world's
他与弗兰克·阿巴内尔（Frank Abagnale）的对话，弗兰克·阿巴内尔是《抓住我，如果你能》的作者，也是世界上最伟大的作家之一。
David Patterson (01:58.080)
most famous con men. Perfect podcast length and topic for a recent long distance run that I did.
最著名的骗子。完美的播客长度和主题，适合我最近的一次长跑。
David Patterson (02:05.840)
Again, go to Jordan Harbinger.com slash Lex to give him my love and to support this podcast.
再次，前往 Jordan Harbinger.com 砍掉 Lex，向他表达我的爱并支持这个播客。
David Patterson (02:13.520)
Subscribe also on Apple podcast, Spotify, and everywhere else.
还可以在 Apple 播客、Spotify 和其他地方订阅。
David Patterson (02:17.040)
This show is presented by Cash App, the greatest sponsor of this podcast ever, and the number one
本节目由 Cash App 主办，Cash App 是该播客有史以来最大的赞助商，也是排名第一的播客
David Patterson (02:23.280)
finance app in the App Store. When you get it, use code LEX PODCAST. Cash App lets you send money
App Store 中的金融应用程序。获得后，请使用代码 LEX PODCAST。现金应用程序可让您汇款
David Patterson (02:29.200)
to friends, buy Bitcoin, and invest in the stock market with as little as $1. Since Cash App allows
向朋友发送，只需 1 美元即可购买比特币并投资股票市场。由于现金应用程序允许
David Patterson (02:35.040)
you to buy Bitcoin, let me mention that cryptocurrency in the context of the history
你要购买比特币，让我在历史背景下提到加密货币
David Patterson (02:39.040)
of money is fascinating. I recommend Ascent of Money as a great book on this history.
David Patterson (02:44.080)
Also, the audiobook is amazing. Debits and credits on Ledger started around 30,000 years ago.
David Patterson (02:50.240)
The US dollar created over 200 years ago, and the first decentralized cryptocurrency released just
David Patterson (02:55.760)
over 10 years ago. So given that history, cryptocurrency is still very much in its early
David Patterson (03:00.720)
days of development, but it's still aiming to and just might redefine the nature of money.
Lex Fridman (03:06.880)
So again, if you get Cash App from the App Store or Google Play, and use the code LEX PODCAST,
David Patterson (03:12.640)
you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping to
David Patterson (03:19.440)
advance robotics and STEM education for young people around the world. And now, here's my
David Patterson (03:25.360)
conversation with David Patterson. Let's start with the big historical question. How have computers
Lex Fridman (03:32.480)
changed in the past 50 years at both the fundamental architectural level and in general, in your eyes?
David Patterson (03:38.400)
David Patterson Well, the biggest thing that happened was the invention of the microprocessor.
Lex Fridman (03:42.960)
So computers that used to fill up several rooms could fit inside your cell phone. And not only
David Patterson (03:52.240)
did they get smaller, they got a lot faster. So they're a million times faster than they were
David Patterson (03:58.160)
50 years ago, and they're much cheaper, and they're ubiquitous. There's 7.8 billion people
David Patterson (04:06.320)
on this planet. Probably half of them have cell phones right now, which is remarkable.
David Patterson (04:10.960)
Soterios Johnson That's probably more microprocessors than there are people.
Lex Fridman (04:14.800)
David Patterson Sure. I don't know what the ratio is,
Lex Fridman (04:16.800)
but I'm sure it's above one. Maybe it's 10 to 1 or some number like that.
Lex Fridman (04:21.520)
Soterios Johnson What is a microprocessor?
David Patterson (04:23.760)
David Patterson So a way to say what a microprocessor is,
David Patterson (04:27.520)
is to tell you what's inside a computer. So a computer forever has classically had
David Patterson (04:32.240)
five pieces. There's input and output, which kind of naturally, as you'd expect, is input is like
David Patterson (04:38.640)
speech or typing, and output is displays. There's a memory, and like the name sounds, it remembers
David Patterson (04:48.880)
things. So it's integrated circuits whose job is you put information in, then when you ask for it,
David Patterson (04:54.480)
it comes back out. That's memory. And the third part is the processor, where the microprocessor
David Patterson (05:00.400)
comes from. And that has two pieces as well. And that is the control, which is kind of the brain
David Patterson (05:07.920)
of the processor. And what's called the arithmetic unit, it's kind of the brawn of the computer. So
David Patterson (05:15.440)
if you think of the, as a human body, the arithmetic unit, the thing that does the
David Patterson (05:19.680)
number crunching is the body and the control is the brain. So those five pieces, input, output,
David Patterson (05:25.280)
memory, arithmetic unit, and control are, have been in computers since the very dawn. And the
David Patterson (05:34.080)
last two are considered the processor. So a microprocessor simply means a processor that
David Patterson (05:39.440)
fits on a microchip. And that was invented about, you know, 40 years ago, was the first microprocessor.
David Patterson (05:46.240)
It's interesting that you refer to the arithmetic unit as the, like you connected to the body and
David Patterson (05:52.320)
the controllers of the brain. So I guess, I never thought of it that way. It's a nice way to think
David Patterson (05:57.440)
of it because most of the actions the microprocessor does in terms of literally sort of computation,
Lex Fridman (06:05.120)
but the microprocessor does computation. It processes information. And most of the thing it
Lex Fridman (06:10.160)
does is basic arithmetic operations. What are the operations, by the way?
David Patterson (06:16.080)
It's a lot like a calculator. So there are add instructions, subtract instructions,
David Patterson (06:22.720)
multiply and divide. And kind of the brilliance of the invention of the computer or the processor
David Patterson (06:32.960)
is that it performs very trivial operations, but it just performs billions of them per second.
Lex Fridman (06:39.120)
And what we're capable of doing is writing software that can take these very trivial instructions
Lex Fridman (06:44.880)
and have them create tasks that can do things better than human beings can do today.
David Patterson (06:49.440)
Just looking back through your career, did you anticipate the kind of how good we would be able
David Patterson (06:55.360)
to get at doing these small, basic operations? How many surprises along the way where you just
Lex Fridman (07:03.280)
kind of sat back and said, wow, I didn't expect it to go this fast, this good?
David Patterson (07:09.200)
MG Well, the fundamental driving force is what's called Moore's law, which was named after Gordon
David Patterson (07:17.280)
Moore, who's a Berkeley alumnus. And he made this observation very early in what are called
David Patterson (07:23.360)
semiconductors. And semiconductors are these ideas, you can build these very simple switches,
Lex Fridman (07:29.120)
and you can put them on these microchips. And he made this observation over 50 years ago.
David Patterson (07:34.240)
He looked at a few years and said, I think what's going to happen is the number of these little
David Patterson (07:38.320)
switches called transistors is going to double every year for the next decade. And he said this
David Patterson (07:44.960)
in 1965. And in 1975, he said, well, maybe it's going to double every two years. And that what
David Patterson (07:51.760)
other people since named that Moore's law guided the industry. And when Gordon Moore made that
David Patterson (07:58.720)
prediction, he wrote a paper back in, I think, in the 70s and said, not only did this going to happen,
David Patterson (08:08.720)
he wrote, what would be the implications of that? And in this article from 1965,
David Patterson (08:13.120)
he shows ideas like computers being in cars and computers being in something that you would buy
David Patterson (08:21.520)
in the grocery store and stuff like that. So he kind of not only called his shot, he called the
David Patterson (08:26.800)
implications of it. So if you were in the computing field, and if you believed Moore's prediction,
David Patterson (08:33.280)
he kind of said what would be happening in the future. So it's not kind of, it's at one sense,
David Patterson (08:41.360)
this is what was predicted. And you could imagine it was easy to believe that Moore's law was going
David Patterson (08:46.320)
to continue. And so this would be the implications. On the other side, there are these kind of
David Patterson (08:51.200)
shocking events in your life. Like I remember driving in Marin across the Bay in San Francisco
Lex Fridman (08:59.520)
and seeing a bulletin board at a local civic center and it had a URL on it. And it was like,
David Patterson (09:07.680)
for the people at the time, these first URLs and that's the, you know, www select stuff with the
David Patterson (09:13.760)
HTTP. People thought it looked like alien writing, right? You'd see these advertisements and
David Patterson (09:22.960)
commercials or bulletin boards that had this alien writing on it. So for the lay people, it's like,
Lex Fridman (09:26.560)
what the hell is going on here? And for those people in the industry, it was, oh my God,
David Patterson (09:32.000)
this stuff is getting so popular, it's actually leaking out of our nerdy world into the real
David Patterson (09:37.200)
world. So that, I mean, there was events like that. I think another one was, I remember in the
David Patterson (09:42.320)
early days of the personal computer, when we started seeing advertisements in magazines
David Patterson (09:46.800)
for personal computers, like it's so popular that it's made the newspapers. So at one hand,
David Patterson (09:52.480)
you know, Gordon Moore predicted it and you kind of expected it to happen, but when it really hit
Lex Fridman (09:56.720)
and you saw it affecting society, it was shocking. So maybe taking a step back and looking both
David Patterson (10:05.200)
the engineering and philosophical perspective, what do you see as the layers of abstraction
Lex Fridman (10:11.520)
in the computer? Do you see a computer as a set of layers of abstractions?
David Patterson (10:16.080)
Dr. Justin Marchegiani Yeah, I think that's one of the things that computer science
David Patterson (10:20.880)
fundamentals is the, these things are really complicated in the way we cope with complicated
David Patterson (10:26.880)
software and complicated hardware is these layers of abstraction. And that simply means that we,
David Patterson (10:33.840)
you know, suspend disbelief and pretend that the only thing you know is that layer,
Lex Fridman (10:39.520)
and you don't know anything about the layer below it. And that's the way we can make very complicated
David Patterson (10:44.240)
things. And probably it started with hardware that that's the way it was done, but it's been
David Patterson (10:50.720)
proven extremely useful. And, you know, I would say in a modern computer today, there might be
David Patterson (10:56.400)
10 or 20 layers of abstraction, and they're all trying to kind of enforce this contract is all
David Patterson (11:02.880)
you know is this interface. There's a set of commands that you can, are allowed to use,
Lex Fridman (11:09.840)
and you stick to those commands, and we will faithfully execute that. And it's like peeling
David Patterson (11:13.840)
the air layers of a London, of an onion, you get down, there's a new set of layers and so forth.
Lex Fridman (11:19.200)
So for people who want to study computer science, the exciting part about it is you can
David Patterson (11:27.040)
keep peeling those layers. You take your first course, and you might learn to program in Python,
Lex Fridman (11:32.000)
and then you can take a follow on course, and you can get it down to a lower level language like C,
Lex Fridman (11:37.680)
and you know, you can go and then you can, if you want to, you can start getting into the hardware
David Patterson (11:42.000)
layers, and you keep getting down all the way to that transistor that I talked about that Gordon
David Patterson (11:47.840)
Moore predicted. And you can understand all those layers all the way up to the highest level
David Patterson (11:53.680)
application software. So it's a very kind of magnetic field. If you're interested, you can go
David Patterson (12:02.320)
into any depth and keep going. In particular, what's happening right now, or it's happened
David Patterson (12:08.240)
in software the last 20 years and recently in hardware, there's getting to be open source
David Patterson (12:12.480)
versions of all of these things. So what open source means is what the engineer, the programmer
David Patterson (12:18.960)
designs, it's not secret, the belonging to a company, it's out there on the worldwide web,
Lex Fridman (12:26.320)
so you can see it. So you can look at, for lots of pieces of software that you use, you can see
David Patterson (12:33.600)
exactly what the programmer does if you want to get involved. That used to stop at the hardware.
David Patterson (12:39.920)
Recently, there's been an effort to make open source hardware and those interfaces open,
Lex Fridman (12:46.240)
so you can see that. So instead of before you had to stop at the hardware, you can now start going
David Patterson (12:51.120)
layer by layer below that and see what's inside there. So it's a remarkable time that for the
David Patterson (12:56.480)
interested individual can really see in great depth what's really going on in the computers
David Patterson (13:01.920)
that power everything that we see around us. Are you thinking also when you say open source at
David Patterson (13:07.680)
the hardware level, is this going to the design architecture instruction set level or is it going
David Patterson (13:14.560)
to literally the manufacturer of the actual hardware, of the actual chips, whether that's ASIC
David Patterson (13:24.960)
specialized to a particular domain or the general? Yeah, so let's talk about that a little bit.
Lex Fridman (13:30.000)
So when you get down to the bottom layer of software, the way software talks to hardware
David Patterson (13:38.640)
is in a vocabulary. And what we call that vocabulary, we call that, the words of that
David Patterson (13:45.120)
vocabulary are called instructions. And the technical term for the vocabulary is instruction
David Patterson (13:50.720)
set. So those instructions are like we talked about earlier, that can be instructions like
David Patterson (13:55.600)
add, subtract and multiply, divide. There's instructions to put data into memory, which
David Patterson (14:01.920)
is called a store instruction and to get data back, which is called the load instructions.
Lex Fridman (14:05.520)
And those simple instructions go back to the very dawn of computing in 1950, the commercial
David Patterson (14:12.800)
computer had these instructions. So that's the instruction set that we're talking about.
Lex Fridman (14:17.600)
So up until, I'd say 10 years ago, these instruction sets were all proprietary. So
David Patterson (14:23.440)
a very popular one is owned by Intel, the one that's in the cloud and in all the PCs in the
David Patterson (14:29.840)
world. Intel owns that instruction set. It's referred to as the x86. There've been a sequence
David Patterson (14:36.320)
of ones that the first number was called 8086. And since then, there's been a lot of numbers,
Lex Fridman (14:41.920)
but they all end in 86. So there's been that kind of family of instruction sets.
Lex Fridman (14:47.920)
And that's proprietary.
David Patterson (14:49.440)
That's proprietary. The other one that's very popular is from ARM. That kind of powers all
David Patterson (14:55.920)
the cell phones in the world, all the iPads in the world, and a lot of things that are so called
David Patterson (15:02.000)
Internet of Things devices. ARM and that one is also proprietary. ARM will license it to people
David Patterson (15:09.600)
for a fee, but they own that. So the new idea that got started at Berkeley kind of unintentionally
David Patterson (15:16.160)
10 years ago is early in my career, we pioneered a way to do these vocabularies instruction sets
David Patterson (15:25.680)
that was very controversial at the time. At the time in the 1980s, conventional wisdom was these
David Patterson (15:32.320)
vocabularies instruction sets should have powerful instructions. So polysyllabic kind of words,
David Patterson (15:38.800)
you can think of that. And so instead of just add, subtract, and multiply, they would have
David Patterson (15:44.560)
polynomial, divide, or sort a list. And the hope was of those powerful vocabularies,
David Patterson (15:51.200)
that'd make it easier for software. So we thought that didn't make sense for microprocessors. There
David Patterson (15:57.760)
was people at Berkeley and Stanford and IBM who argued the opposite. And what we called that was
David Patterson (16:03.600)
a reduced instruction set computer. And the abbreviation was RISC. And typical for computer
David Patterson (16:10.480)
people, we use the abbreviation start pronouncing it. So risk was the thing. So we said for
David Patterson (16:15.920)
microprocessors, which with Gordon's Moore is changing really fast, we think it's better to have
David Patterson (16:21.440)
a pretty simple set of instructions, reduced set of instructions. That that would be a better way
David Patterson (16:28.480)
to build microprocessors since they're going to be changing so fast due to Moore's law. And then
David Patterson (16:32.800)
we'll just use standard software to cover the use, generate more of those simple instructions. And
David Patterson (16:41.120)
one of the pieces of software that's in that software stack going between these layers of
David Patterson (16:45.760)
abstractions is called a compiler. And it's basically translates, it's a translator between
David Patterson (16:50.400)
levels. We said the translator will handle that. So the technical question was, well, since there
David Patterson (16:57.200)
are these reduced instructions, you have to execute more of them. Yeah, that's right. But
David Patterson (17:01.920)
maybe you could execute them faster. Yeah, that's right. They're simpler so they could go faster,
Lex Fridman (17:05.840)
but you have to do more of them. So what's that trade off look like? And it ended up that we ended
David Patterson (17:12.080)
up executing maybe 50% more instructions, maybe a third more instructions, but they ran four times
David Patterson (17:19.040)
faster. So this risk, controversial risk ideas proved to be maybe factors of three or four
David Patterson (17:26.000)
better. I love that this idea was controversial and almost kind of like rebellious. So that's
David Patterson (17:33.280)
in the context of what was more conventional is the complex instructional set computing. So
Lex Fridman (17:40.080)
how would you pronounce that? CISC. CISC versus risk. Risk versus CISC. And believe it or not,
David Patterson (17:46.400)
this sounds very, who cares about this? It was violently debated at several conferences. It's
David Patterson (17:54.720)
like, what's the right way to go? And people thought risk was a deevolution. We're going to
David Patterson (18:01.280)
make software worse by making those instructions simpler. And there are fierce debates at several
David Patterson (18:07.200)
conferences in the 1980s. And then later in the 80s, it kind of settled to these benefits.
David Patterson (18:14.400)
It's not completely intuitive to me why risk has, for the most part, won.
David Patterson (18:19.600)
Yeah. So why did that happen? Yeah. Yeah. And maybe I can sort of say a bunch of dumb things
David Patterson (18:24.240)
that could lay the land for further commentary. So to me, this is kind of an interesting thing.
David Patterson (18:30.640)
If you look at C++ versus C, with modern compilers, you really could write faster code
David Patterson (18:36.960)
with C++. So relying on the compiler to reduce your complicated code into something simple and
David Patterson (18:44.160)
fast. So to me, comparing risk, maybe this is a dumb question, but why is it that focusing the
David Patterson (18:53.360)
definition of the design of the instruction set on very few simple instructions in the long run
David Patterson (19:00.240)
provide faster execution versus coming up with, like you said, a ton of complicated instructions
David Patterson (19:09.920)
that over time, you know, years, maybe decades, you come up with compilers that can reduce those
David Patterson (19:16.160)
into simple instructions for you. Yeah. So let's try and split that into two pieces.
Lex Fridman (19:22.640)
So if the compiler can do that for you, if the compiler can take, you know, a complicated program
Lex Fridman (19:29.120)
and produce simpler instructions, then the programmer doesn't care, right? I don't care just
Lex Fridman (19:37.120)
how fast is the computer I'm using, how much does it cost? And so what happened kind of in the
David Patterson (19:43.520)
software industry is right around before the 1980s, critical pieces of software were still written
David Patterson (19:50.560)
not in languages like C or C++, they were written in what's called assembly language, where there's
David Patterson (19:57.600)
this kind of humans writing exactly at the instructions at the level that a computer can
David Patterson (1:00:01.520)
computers. So that's, you know, that's a huge change of what's gone on. So, but since this
David Patterson (1:00:11.840)
lasted for decades, kind of programmers and maybe all of society is used to computers getting faster
David Patterson (1:00:18.720)
regularly. We now believe, those of us who are in computer design, it's called computer
David Patterson (1:00:24.640)
architecture, that the path forward is instead is to add accelerators that only work well for
David Patterson (1:00:33.680)
certain applications. So since Moore's Law is slowing down, we don't think general purpose
David Patterson (1:00:41.600)
computers are going to get a lot faster. So the Intel processors of the world are not going to,
David Patterson (1:00:46.560)
haven't been getting a lot faster. They've been barely improving, like a few percent a year.
David Patterson (1:00:51.680)
It used to be doubling every 18 months and now it's doubling every 20 years. So it was just
David Patterson (1:00:56.640)
shocking. So to be able to deliver on what Moore's Law used to do, we think what's going to happen,
Lex Fridman (1:01:02.480)
what is happening right now is people adding accelerators to their microprocessors that only
David Patterson (1:01:09.280)
work well for some domains. And by sheer coincidence, at the same time that this is happening,
David Patterson (1:01:17.120)
has been this revolution in artificial intelligence called machine learning. So with,
David Patterson (1:01:23.840)
as I'm sure your other guests have said, you know, AI had these two competing schools of thought is
David Patterson (1:01:31.040)
that we could figure out artificial intelligence by just writing the rules top down, or that was
David Patterson (1:01:36.480)
wrong. You had to look at data and infer what the rules are, the machine learning, and what's
David Patterson (1:01:41.600)
happened in the last decade or eight years as machine learning has won. And it turns out that
David Patterson (1:01:48.560)
machine learning, the hardware you build for machine learning is pretty much multiply. The
David Patterson (1:01:55.440)
matrix multiply is a key feature for the way machine learning is done. So that's a godsend
David Patterson (1:02:03.040)
for computer designers. We know how to make matrix multiply run really fast. So general purpose
David Patterson (1:02:08.640)
microprocessors are slowing down. We're adding accelerators for machine learning that fundamentally
David Patterson (1:02:13.360)
are doing matrix multiplies much more efficiently than general purpose computers have done.
Lex Fridman (1:02:17.920)
So we have to come up with a new way to accelerate things. The danger of only accelerating one
David Patterson (1:02:23.120)
application is how important is that application. Turns out machine learning gets used for all
David Patterson (1:02:28.640)
kinds of things. So serendipitously, we found something to accelerate that's widely applicable.
Lex Fridman (1:02:36.320)
And we don't even, we're in the middle of this revolution of machine learning. We're not sure
Lex Fridman (1:02:40.400)
what the limits of machine learning are. So this has been a kind of a godsend. If you're going to
David Patterson (1:02:46.000)
be able to excel, deliver on improved performance, as long as people are moving their programs to be
David Patterson (1:02:54.080)
embracing more machine learning, we know how to give them more performance even as Moore's law
David Patterson (1:02:58.880)
is slowing down. And counterintuitively, the machine learning mechanism you can say is domain
David Patterson (1:03:06.800)
specific, but because it's leveraging data, it's actually could be very broad in terms of
David Patterson (1:03:15.360)
in terms of the domains it could be applied in. Yeah, that's exactly right. Sort of, it's almost
David Patterson (1:03:21.040)
sort of people sometimes talk about the idea of software 2.0. We're almost taking another step
David Patterson (1:03:27.520)
up in the abstraction layer in designing machine learning systems, because now you're programming
David Patterson (1:03:34.080)
in the space of data, in the space of hyperparameters, it's changing fundamentally
David Patterson (1:03:38.480)
the nature of programming. And so the specialized devices that accelerate the performance, especially
David Patterson (1:03:45.120)
neural network based machine learning systems might become the new general. Yeah. So the thing
David Patterson (1:03:52.400)
that's interesting point out these are not coral, these are not tied together. The enthusiasm about
David Patterson (1:03:59.680)
machine learning about creating programs driven from data that we should figure out the answers
David Patterson (1:04:05.040)
from data rather than kind of top down, which classically the way most programming is done
Lex Fridman (1:04:10.160)
and the way artificial intelligence used to be done. That's a movement that's going on at the
David Patterson (1:04:14.640)
same time. Coincidentally, and the first word machine learning is machines, right? So that's
David Patterson (1:04:21.440)
going to increase the demand for computing, because instead of programmers being smart, writing those
David Patterson (1:04:27.360)
those things down, we're going to instead use computers to examine a lot of data to kind of
David Patterson (1:04:31.760)
create the programs. That's the idea. And remarkably, this gets used for all kinds of
David Patterson (1:04:38.560)
things very successfully. The image recognition, the language translation, the game playing,
Lex Fridman (1:04:43.200)
and you know, it gets into pieces of the software stack like databases and stuff like that. We're
David Patterson (1:04:50.320)
not quite sure how general purpose is, but that's going on independent of this hardware stuff.
David Patterson (1:04:54.880)
What's happening on the hardware side is Moore's law is slowing down right when we need a lot more
David Patterson (1:04:59.040)
cycles. It's failing us, it's failing us right when we need it because there's going to be a
David Patterson (1:05:03.840)
greater increase in computing. And then this idea that we're going to do so called domain
David Patterson (1:05:09.680)
specific. Here's a domain that your greatest fear is you'll make this one thing work and that'll
David Patterson (1:05:16.800)
help, you know, five percent of the people in the world. Well, this looks like it's a very
David Patterson (1:05:22.160)
general purpose thing. So the timing is fortuitous that if we can perhaps, if we can keep building
David Patterson (1:05:28.640)
hardware that will accelerate machine learning, the neural networks, that'll beat the timing will
David Patterson (1:05:36.080)
be right. That neural network revolution will transform your software, the so called software
David Patterson (1:05:42.160)
2.0. And the software of the future will be very different from the software of the past. And just
David Patterson (1:05:47.680)
as our microprocessors, even though we're still going to have that same basic RISC instructions
David Patterson (1:05:53.200)
to run a big pieces of the software stack like user interfaces and stuff like that,
David Patterson (1:05:58.080)
we can accelerate the kind of the small piece that's computationally intensive. It's not lots
David Patterson (1:06:02.880)
of lines of code, but it takes a lot of cycles to run that code that that's going to be the
David Patterson (1:06:08.160)
accelerator piece. And so that's what makes this from a computer designers perspective a really
David Patterson (1:06:14.800)
interesting decade. What Hennessy and I talked about in the title of our Turing Warrant speech
David Patterson (1:06:20.320)
is a new golden age. We see this as a very exciting decade, much like when we were assistant
David Patterson (1:06:28.160)
professors and the RISC stuff was going on. That was a very exciting time was where we were changing
Lex Fridman (1:06:32.720)
what was going on. We see this happening again. Tremendous opportunities of people because we're
David Patterson (1:06:39.280)
fundamentally changing how software is built and how we're running it. So which layer of the
David Patterson (1:06:43.760)
abstraction do you think most of the acceleration might be happening? If you look in the next 10
David Patterson (1:06:49.360)
years, Google is working on a lot of exciting stuff with the TPU. Sort of there's a closer to
David Patterson (1:06:54.880)
the hardware that could be optimizations around the closer to the instruction set.
David Patterson (1:07:00.640)
There could be optimization at the compiler level. It could be even at the higher level software
David Patterson (1:07:05.040)
stack. Yeah, it's got to be, I mean, if you think about the old RISC Sys debate, it was both,
David Patterson (1:07:11.840)
it was software hardware. It was the compilers improving as well as the architecture improving.
Lex Fridman (1:07:18.080)
And that's likely to be the way things are now. With machine learning, they're using
David Patterson (1:07:24.240)
domain specific languages. The languages like TensorFlow and PyTorch are very popular with
David Patterson (1:07:30.960)
the machine learning people. Those are the raising the level of abstraction. It's easier
David Patterson (1:07:35.280)
for people to write machine learning in these domain specific languages like PyTorch and
David Patterson (1:07:41.920)
TensorFlow. So where the most optimization might be happening. Yeah. And so there'll be both the
David Patterson (1:07:47.760)
compiler piece and the hardware piece underneath it. So as you kind of the fatal flaw for hardware
David Patterson (1:07:53.680)
people is to create really great hardware, but not have brought along the compilers. And what we're
David Patterson (1:07:59.360)
seeing right now in the marketplace because of this enthusiasm around hardware for machine
David Patterson (1:08:04.560)
learning is getting, you know, probably billions of dollars invested in startup companies. We're
David Patterson (1:08:10.400)
seeing startup companies go belly up because they focus on the hardware, but didn't bring the
David Patterson (1:08:15.920)
software stack along. We talked about benchmarks earlier. So I participated in machine learning
David Patterson (1:08:23.440)
didn't really have a set of benchmarks. I think just two years ago, they didn't have a set of
David Patterson (1:08:27.520)
benchmarks. And we've created something called ML Perf, which is machine learning benchmark suite.
Lex Fridman (1:08:33.280)
And pretty much the companies who didn't invest in the software stack couldn't run ML Perf very
David Patterson (1:08:39.840)
well. And the ones who did invest in software stack did. And we're seeing, you know, like kind
David Patterson (1:08:45.040)
of in computer architecture, this is what happens. You have these arguments about risk versus this.
David Patterson (1:08:48.800)
People spend billions of dollars in the marketplace to see who wins. It's not a perfect comparison,
Lex Fridman (1:08:54.720)
but it kind of sorts things out. And we're seeing companies go out of business and then companies
David Patterson (1:08:59.680)
like there's a company in Israel called Habana. They came up with machine learning accelerators.
David Patterson (1:09:08.160)
They had good ML Perf scores. Intel had acquired a company earlier called Nirvana a couple of years
David Patterson (1:09:14.560)
ago. They didn't reveal their ML Perf scores, which was suspicious. But a month ago, Intel
David Patterson (1:09:21.120)
announced that they're canceling the Nirvana product line and they've bought Habana for $2
David Patterson (1:09:25.760)
billion. And Intel's going to be shipping Habana chips, which have hardware and software and run
David Patterson (1:09:32.560)
the ML Perf programs pretty well. And that's going to be their product line in the future.
David Patterson (1:09:36.800)
Brilliant. So maybe just to linger briefly on ML Perf. I love metrics. I love standards that
Lex Fridman (1:09:42.560)
everyone can gather around. What are some interesting aspects of that portfolio of metrics?
David Patterson (1:09:48.800)
Well, one of the interesting metrics is what we thought. I was involved in the start.
Lex Fridman (1:09:57.440)
Peter Mattson is leading the effort from Google. Google got it off the ground,
Lex Fridman (1:10:00.880)
but we had to reach out to competitors and say, there's no benchmarks here. We think this is
David Patterson (1:10:07.360)
bad for the field. It'll be much better if we look at examples like in the risk days,
David Patterson (1:10:11.120)
there was an effort to create a... For the people in the risk community got together,
Lex Fridman (1:10:16.400)
competitors got together building risk microprocessors to agree on a set of
David Patterson (1:10:19.520)
benchmarks that were called spec. And that was good for the industry. It's rather before
David Patterson (1:10:24.720)
the different risk architectures were arguing, well, you can believe my performance others,
Lex Fridman (1:10:28.160)
but those other guys are liars. And that didn't do any good. So we agreed on a set of benchmarks
Lex Fridman (1:10:34.400)
and then we could figure out who was faster between the various risk architectures. But
David Patterson (1:10:37.920)
it was a little bit faster, but that grew the market rather than people were afraid to buy
David Patterson (1:10:42.800)
anything. So we argued the same thing would happen with MLPerf. Companies like Nvidia were maybe
David Patterson (1:10:48.880)
worried that it was some kind of trap, but eventually we all got together to create a
David Patterson (1:10:53.120)
set of benchmarks and do the right thing. And we agree on the results. And so we can see whether
David Patterson (1:11:00.320)
TPUs or GPUs or CPUs are really faster and how much the faster. And I think from an engineer's
David Patterson (1:11:06.560)
perspective, as long as the results are fair, you can live with it. Okay, you kind of tip your hat
David Patterson (1:11:12.800)
to your colleagues at another institution, boy, they did a better job than us. What you hate is
David Patterson (1:11:18.240)
if it's false, right? They're making claims and it's just marketing bullshit and that's affecting
David Patterson (1:11:23.600)
sales. So from an engineer's perspective, as long as it's a fair comparison and we don't come in
David Patterson (1:11:28.640)
first place, that's too bad, but it's fair. So we wanted to create that environment for MLPerf.
Lex Fridman (1:11:33.600)
And so now there's 10 companies, I mean, 10 universities and 50 companies involved. So pretty
David Patterson (1:11:40.880)
much MLPerf is the way you measure machine learning performance. And it didn't exist even
David Patterson (1:11:50.640)
two years ago. One of the cool things that I enjoy about the internet has a few downsides, but one of
David Patterson (1:11:56.720)
the nice things is people can see through BS a little better with the presence of these kinds
David Patterson (1:12:02.080)
of metrics. So it's really nice companies like Google and Facebook and Twitter. Now, it's the
David Patterson (1:12:08.080)
cool thing to do is to put your engineers forward and to actually show off how well you do on these
David Patterson (1:12:13.280)
metrics. There's less of a desire to do marketing, less so. In my sort of naive viewpoint.
David Patterson (1:12:22.560)
I was trying to understand what's changed from the 80s in this era. I think because of things
David Patterson (1:12:29.280)
like social networking, Twitter and stuff like that, if you put up bullshit stuff that's just
David Patterson (1:12:38.080)
purposely misleading, you can get a violent reaction in social media pointing out the flaws
David Patterson (1:12:45.920)
in your arguments. And so from a marketing perspective, you have to be careful today that
David Patterson (1:12:51.840)
you didn't have to be careful that there'll be people who put out the flaw. You can get the
David Patterson (1:12:57.120)
word out about the flaws in what you're saying much more easily today than in the past. It used
David Patterson (1:13:02.960)
to be easier to get away with it. And the other thing that's been happening in terms of showing
David Patterson (1:13:08.000)
off engineers is just in the software side, people have largely embraced open source software.
David Patterson (1:13:16.800)
20 years ago, it was a dirty word at Microsoft. And today Microsoft is one of the big proponents
David Patterson (1:13:22.000)
of open source software. That's the standard way most software gets built, which really shows off
David Patterson (1:13:28.160)
your engineers because you can see if you look at the source code, you can see who are making the
David Patterson (1:13:34.320)
commits, who's making the improvements, who are the engineers at all these companies who are
Lex Fridman (1:13:41.440)
really great programmers and engineers and making really solid contributions,
David Patterson (1:13:47.120)
which enhances their reputations and the reputation of the companies.
David Patterson (1:13:50.080)
LR But that's, of course, not everywhere. Like in the space that I work more in is autonomous
David Patterson (1:13:56.320)
vehicles. And there's still the machinery of hype and marketing is still very strong there. And
David Patterson (1:14:02.080)
there's less willingness to be open in this kind of open source way and sort of benchmark. So
David Patterson (1:14:07.600)
MLPerf represents the machine learning world is much better being open source about holding
David Patterson (1:14:12.480)
itself to standards of different, the amount of incredible benchmarks in terms of the different
David Patterson (1:14:18.240)
computer vision, natural language processing tasks is incredible.
Lex Fridman (1:14:23.120)
LR Historically, it wasn't always that way.
David Patterson (1:14:26.800)
I had a graduate student working with me, David Martin. So in computer, in some fields,
David Patterson (1:14:32.480)
benchmarking has been around forever. So computer architecture, databases, maybe operating systems,
David Patterson (1:14:40.560)
benchmarks are the way you measure progress. But he was working with me and then started working
David Patterson (1:14:47.440)
with Jitendra Malik. And Jitendra Malik in computer vision space, I guess you've interviewed
David Patterson (1:14:53.040)
Jitendra. And David Martin told me, they don't have benchmarks. Everybody has their own vision
David Patterson (1:14:59.360)
algorithm and the way, here's my image, look at how well I do. And everybody had their own image.
Lex Fridman (1:15:04.960)
So David Martin, back when he did his dissertation, figured out a way to do benchmarks. He had a bunch
David Patterson (1:15:10.640)
of graduate students identify images and then ran benchmarks to see which algorithms run well. And
David Patterson (1:15:17.200)
that was, as far as I know, kind of the first time people did benchmarks in computer vision, which
David Patterson (1:15:24.480)
was predated all the things that eventually led to ImageNet and stuff like that. But then the vision
David Patterson (1:15:29.600)
community got religion. And then once we got as far as ImageNet, then that let the guys in Toronto
Lex Fridman (1:15:38.720)
be able to win the ImageNet competition. And then that changed the whole world.
David Patterson (1:15:42.560)
It's a scary step actually, because when you enter the world of benchmarks, you actually have to be
David Patterson (1:15:47.840)
good to participate as opposed to... Yeah, you can just, you just believe you're the best in the
David Patterson (1:15:54.160)
world. I think the people, I think they weren't purposely misleading. I think if you don't have
David Patterson (1:16:01.280)
benchmarks, I mean, how do you know? Your intuition is kind of like the way we did just
David Patterson (1:16:06.320)
do computer architecture. Your intuition is that this is the right instruction set to do this job.
David Patterson (1:16:11.040)
I believe in my experience, my hunch is that's true. We had to get to make things more quantitative
David Patterson (1:16:18.160)
to make progress. And so I just don't know how, you know, in fields that don't have benchmarks,
Lex Fridman (1:16:23.840)
I don't understand how they figure out how they're making progress.
David Patterson (1:16:28.400)
We're kind of in the vacuum tube days of quantum computing. What are your thoughts in this wholly
David Patterson (1:16:34.480)
different kind of space of architectures? You know, I actually, you know, quantum computing
David Patterson (1:16:41.120)
is, idea has been around for a while and I actually thought, well, I sure hope I retire
David Patterson (1:16:46.240)
before I have to start teaching this. I'd say because I talk about, give these talks about the
David Patterson (1:16:53.520)
slowing of Moore's law and, you know, when we need to change by doing domain specific accelerators,
Lex Fridman (1:17:01.040)
common questions say, what about quantum computing? The reason that comes up,
David Patterson (1:17:04.480)
it's in the news all the time. So I think to keep in, the third thing to keep in mind is
David Patterson (1:17:08.880)
quantum computing is not right around the corner. There've been two national reports,
David Patterson (1:17:14.080)
one by the National Academy of Engineering and other by the Computing Consortium, where they
David Patterson (1:17:18.800)
did a frank assessment of quantum computing. And both of those reports said, you know,
David Patterson (1:17:25.440)
as far as we can tell, before you get error corrected quantum computing, it's a decade away.
Lex Fridman (1:17:31.200)
So I think of it like nuclear fusion, right? There've been people who've been excited about
David Patterson (1:17:35.680)
nuclear fusion a long time. If we ever get nuclear fusion, it's going to be fantastic
David Patterson (1:17:39.760)
for the world. I'm glad people are working on it, but, you know, it's not right around the corner.
David Patterson (1:17:45.120)
Those two reports to me say probably it'll be 2030 before quantum computing is something
David Patterson (1:17:52.640)
that could happen. And when it does happen, you know, this is going to be big science stuff. This
David Patterson (1:17:58.000)
is, you know, micro Kelvin, almost absolute zero things that if they vibrate, if truck goes by,
David Patterson (1:18:04.880)
it won't work, right? So this will be in data center stuff. We're not going to have a quantum
David Patterson (1:18:09.200)
cell phone. And it's probably a 2030 kind of thing. So I'm happy that our people are working on it,
Lex Fridman (1:18:16.080)
but just, you know, it's hard with all the news about it, not to think that it's right around the
David Patterson (1:18:21.040)
corner. And that's why we need to do something as Moore's Law is slowing down to provide the
David Patterson (1:18:27.040)
computing, keep computing getting better for this next decade. And, you know, we shouldn't
David Patterson (1:18:32.560)
be betting on quantum computing or expecting quantum computing to deliver in the next few
David Patterson (1:18:39.680)
years. It's probably further off. You know, I'd be happy to be wrong. It'd be great if quantum
David Patterson (1:18:44.480)
computing is going to commercially viable, but it will be a set of applications. It's not a general
David Patterson (1:18:49.600)
purpose computation. So it's going to do some amazing things, but there'll be a lot of things
David Patterson (1:18:54.640)
that probably, you know, the old fashioned computers are going to keep doing better for
David Patterson (1:18:59.360)
quite a while. And there'll be a teenager 50 years from now watching this video saying,
David Patterson (1:19:04.240)
look how silly David Patterson was saying. No, I just said, I said 2030. I didn't say,
David Patterson (1:19:09.920)
I didn't say never. We're not going to have quantum cell phones. So he's going to be watching it.
David Patterson (1:19:14.560)
Well, I mean, I think this is such a, you know, given that we've had Moore's Law, I just, I feel
David Patterson (1:19:21.920)
comfortable trying to do projects that are thinking about the next decade. I admire people who are
David Patterson (1:19:27.360)
trying to do things that are 30 years out, but it's such a fast moving field. I just don't know
Lex Fridman (1:19:32.880)
how to, I'm not good enough to figure out what's the problem is going to be in 30 years. You know,
David Patterson (1:19:38.800)
10 years is hard enough for me. So maybe if it's possible to untangle your intuition a little bit,
David Patterson (1:19:44.160)
I spoke with Jim Keller. I don't know if you're familiar with Jim. And he is trying to sort of
David Patterson (1:19:50.320)
be a little bit rebellious and to try to think that he quotes me as being wrong. Yeah. So this,
David Patterson (1:19:57.200)
this is what you're doing for the record. Jim talks about that. He has an intuition that Moore's
David Patterson (1:20:04.400)
Law is not in fact, in fact dead yet. And then it may continue for some time to come.
Lex Fridman (1:20:10.720)
What are your thoughts about Jim's ideas in this space? Yeah, this is just, this is just marketing.
Lex Fridman (1:20:16.080)
So what Gordon Moore said is a quantitative prediction. We can check the facts, right? Which
David Patterson (1:20:22.720)
is doubling the number of transistors every two years. So we can look back at Intel for the last
David Patterson (1:20:29.200)
five years and ask him, let's look at DRAM chips six years ago. So that would be three, two year
David Patterson (1:20:38.320)
periods. So then our DRAM chips have eight times as many transistors as they did six years ago.
David Patterson (1:20:44.160)
We can look up Intel microprocessors six years ago. If Moore's Law is continuing, it should have
David Patterson (1:20:50.320)
eight times as many transistors as six years ago. The answer in both those cases is no.
David Patterson (1:20:57.760)
The problem has been because Moore's Law was kind of genuinely embraced by the semiconductor
David Patterson (1:21:05.440)
industry as they would make investments in similar equipment to make Moore's Law come true.
David Patterson (1:21:10.480)
Semiconductor improving and Moore's Law in many people's minds are the same thing. So when I say,
Lex Fridman (1:21:17.520)
and I'm factually correct, that Moore's Law is no longer holds, we are not doubling transistors
David Patterson (1:21:24.080)
every year's years. The downside for a company like Intel is people think that means it's stopped,
Lex Fridman (1:21:31.840)
that technology has no longer improved. And so Jim is trying to,
David Patterson (1:21:36.160)
counteract the impression that semiconductors are frozen in 2019 are never going to get better.
Lex Fridman (1:21:46.240)
So I never said that. All I said was Moore's Law is no more. And I'm strictly looking at the number
David Patterson (1:21:53.120)
of transistors. That's what Moore's Law is. There's the, I don't know, there's been this aura
David Patterson (1:22:01.440)
associated with Moore's Law that they've enjoyed for 50 years about, look at the field we're in,
David Patterson (1:22:07.520)
we're doubling transistors every two years. What an amazing field, which is an amazing thing that
David Patterson (1:22:12.160)
they were able to pull off. But even as Gordon Moore said, you know, no exponential can last
David Patterson (1:22:16.000)
forever. It lasted for 50 years, which is amazing. And this is a huge impact on the industry because
David Patterson (1:22:22.080)
of these changes that we've been talking about. So he claims, and I'm not going to go into the
David Patterson (1:22:28.000)
that we've been talking about. So he claims, because he's trying to act on it, he claims,
David Patterson (1:22:33.280)
you know, Patterson says Moore's Law is no more and look at all, look at it, it's still going.
Lex Fridman (1:22:38.560)
And TSMC, they say it's no longer, but there's quantitative evidence that Moore's Law is not
David Patterson (1:22:44.800)
continuing. So what I say now to try and, okay, I understand the perception problem when I say
David Patterson (1:22:51.520)
Moore's Law has stopped. Okay. So now I say Moore's Law is slowing down. And I think Jim, which is
David Patterson (1:22:58.640)
another way of, if he's, if it's predicting every two years and I say it's slowing down, then that's
David Patterson (1:23:03.760)
another way of saying it doesn't hold anymore. And, and I think Jim wouldn't disagree that it's
David Patterson (1:23:09.520)
slowing down because that sounds like it's, things are still getting better and just not as fast,
David Patterson (1:23:14.560)
which is another way of saying Moore's Law isn't working anymore.
Lex Fridman (1:23:18.000)
TG. It's still good for marketing. But what's your, you're not,
David Patterson (1:23:22.720)
you don't like expanding the definition of Moore's Law, sort of naturally.
Lex Fridman (1:23:27.520)
CM. Well, as an educator, you know, is this like modern politics? Does everybody get their own facts?
David Patterson (1:23:34.880)
Or do we have, you know, Moore's Law was a crisp, you know, it was Carver Mead looked at his
David Patterson (1:23:41.840)
Moore's Conversations drawing on a log log scale, a straight line. And that's what the definition of
David Patterson (1:23:47.680)
Moore's Law is. There's this other, what Intel did for a while, interestingly, before Jim joined
David Patterson (1:23:54.720)
them, they said, oh, no, Moore's Law isn't the number of doubling, isn't really doubling
David Patterson (1:23:58.400)
transistors every two years. Moore's Law is the cost of the individual transistor going down,
David Patterson (1:24:04.400)
cutting in half every two years. Now, that's not what he said, but they reinterpreted it
David Patterson (1:24:10.080)
because they believed that the cost of transistors was continuing to drop,
David Patterson (1:24:15.520)
even if they couldn't get twice as many chips. Many people in industry have told me that's not
David Patterson (1:24:20.560)
true anymore, that basically in more recent technologies, they got more complicated,
David Patterson (1:24:26.000)
the actual cost of transistor went up. So even the, a corollary might not be true,
Lex Fridman (1:24:32.400)
but certainly, you know, Moore's Law, that was the beauty of Moore's Law. It was a very simple,
David Patterson (1:24:38.400)
it's like E equals MC squared, right? It was like, wow, what an amazing prediction. It's so easy
David Patterson (1:24:44.000)
to understand, the implications are amazing, and that's why it was so famous as a prediction.
Lex Fridman (1:24:50.000)
And this reinterpretation of what it meant and changing is, you know, is revisionist history.
Lex Fridman (1:24:56.160)
And I'd be happy, and they're not claiming there's a new Moore's Law. They're not saying,
David Patterson (1:25:04.160)
by the way, instead of every two years, it's every three years. I don't think they want to
David Patterson (1:25:10.480)
say that. I think what's going to happen is new technology generations, each one is going to get
David Patterson (1:25:14.400)
a little bit slower. So it is slowing down, the improvements won't be as great, and that's why we
David Patterson (1:25:21.840)
need to do new things. Yeah, I don't like that the idea of Moore's Law is tied up with marketing.
David Patterson (1:25:28.240)
It would be nice if... Whether it's marketing or it's, well, it could be affecting business,
Lex Fridman (1:25:34.560)
but it could also be affecting the imagination of engineers. If Intel employees actually believe
David Patterson (1:25:40.720)
that we're frozen in 2019, well, that would be bad for Intel. Not just Intel, but everybody.
David Patterson (1:25:49.040)
Moore's Law is inspiring to everybody. But what's happening right now, talking to people
David Patterson (1:25:57.200)
who have working in national offices and stuff like that, a lot of the computer science community
David Patterson (1:26:02.800)
is unaware that this is going on, that we are in an era that's going to need radical change at lower
David Patterson (1:26:09.280)
levels that could affect the whole software stack. If you're using cloud stuff and the
David Patterson (1:26:18.960)
servers that you get next year are basically only a little bit faster than the servers you got this
David Patterson (1:26:23.040)
year, you need to know that, and we need to start innovating to start delivering on it. If you're
David Patterson (1:26:30.240)
counting on your software going to have a lot more features, assuming the computers are going to get
Lex Fridman (1:26:34.400)
faster, that's not true. So are you going to have to start making your software stack more efficient?
David Patterson (1:26:38.640)
Are you going to have to start learning about machine learning? So it's a warning or call
David Patterson (1:26:45.440)
for arms that the world is changing right now. And a lot of computer science PhDs are unaware
David Patterson (1:26:51.040)
of that. So a way to try and get their attention is to say that Moore's Law is slowing down and
David Patterson (1:26:56.800)
that's going to affect your assumptions. And we're trying to get the word out. And when companies
David Patterson (1:27:02.160)
like TSMC and Intel say, oh, no, no, no, Moore's Law is fine, then people think, oh, hey, I don't
David Patterson (1:27:08.080)
have to change my behavior. I'll just get the next servers. And if they start doing measurements,
David Patterson (1:27:13.600)
they'll realize what's going on. It'd be nice to have some transparency on metrics for the lay
David Patterson (1:27:18.800)
person to be able to know if computers are getting faster and not to forget Moore's Law.
David Patterson (1:27:24.720)
Yeah. There are a bunch of, most people kind of use clock rate as a measure of performance.
David Patterson (1:27:31.920)
It's not a perfect one, but if you've noticed clock rates are more or less the same as they were
David Patterson (1:27:37.200)
five years ago, computers are a little better than they are. They haven't made zero progress,
Lex Fridman (1:27:42.960)
but they've made small progress. So there's some indications out there. And then our behavior,
David Patterson (1:27:47.200)
right? Nobody buys the next laptop because it's so much faster than the laptop from the past.
David Patterson (1:27:52.800)
For cell phones, I think, I don't know why people buy new cell phones, you know, because
David Patterson (1:28:00.480)
the new ones announced. The cameras are better, but that's kind of domain specific, right? They're
David Patterson (1:28:04.560)
putting special purpose hardware to make the processing of images go much better. So that's
David Patterson (1:28:10.560)
the way they're doing it. They're not particularly, it's not that the ARM processor in there is twice
David Patterson (1:28:15.840)
as fast as much as they've added accelerators to help the experience of the phone. Can we talk a
David Patterson (1:28:22.800)
little bit about one other exciting space, arguably the same level of impact as your work with RISC
David Patterson (1:28:30.720)
is RAID. In 1988, you coauthored a paper, A Case for Redundant Arrays of Inexpensive Disks, hence
David Patterson (1:28:41.920)
RAID RAID. So that's where you introduced the idea of RAID. Incredible that that little,
David Patterson (1:28:49.840)
I mean little, that paper kind of had this ripple effect and had a really a revolutionary effect.
Lex Fridman (1:28:55.760)
So first, what is RAID? What is RAID? So this is work I did with my colleague Randy Katz and
David Patterson (1:29:01.920)
a star graduate student, Garth Gibson. So we had just done the fourth generation RISC project
Lex Fridman (1:29:08.160)
and Randy Katz, which had an early Apple Macintosh computer. At this time, everything was done with
David Patterson (1:29:17.280)
floppy disks, which are old technologies that could store things that didn't have much capacity
Lex Fridman (1:29:26.160)
and you had to get any work done, you're always sticking your little floppy disk in and out because
David Patterson (1:29:31.360)
they didn't have much capacity. But they started building what are called hard disk drives, which
David Patterson (1:29:36.400)
is magnetic material that can remember information storage for the Mac. And Randy asked the question
David Patterson (1:29:44.320)
when he saw this disk next to his Mac, gee, these are brand new small things. Before that,
David Patterson (1:29:51.760)
for the big computers, the disk would be the size of washing machines. And here's something
David Patterson (1:29:57.520)
the size of a, kind of the size of a book or so. He says, I wonder what we could do with that? Well,
David Patterson (1:30:02.720)
Randy was involved in the fourth generation RISC project here at Berkeley in the 80s. So we figured
David Patterson (1:30:11.200)
out a way how to make the computation part, the processor part go a lot faster, but what about
David Patterson (1:30:15.680)
the storage part? Can we do something to make it faster? So we hit upon the idea of taking a lot of
David Patterson (1:30:22.960)
these disks developed for personal computers and Macintoshes and putting many of them together
David Patterson (1:30:27.600)
instead of one of these washing machine size things. And so we wrote the first draft of the
David Patterson (1:30:32.640)
paper and we'd have 40 of these little PC disks instead of one of these washing machine size
David Patterson (1:30:38.640)
things. And they would be much cheaper because they're made for PCs and they could actually kind
David Patterson (1:30:42.960)
of be faster because there was 40 of them rather than one of them. And so we wrote a paper like
David Patterson (1:30:47.360)
that and sent it to one of our former Berkeley students at IBM. And he said, well, this is all
David Patterson (1:30:51.520)
great and good, but what about the reliability of these things? Now you have 40 of these things
Lex Fridman (1:30:56.960)
and 40 of these devices, each of which are kind of PC quality. So they're not as good as these
David Patterson (1:31:03.120)
IBM washing machines. IBM dominated the storage businesses. So the reliability is going to be
David Patterson (1:31:10.160)
awful. And so when we calculated it out, instead of it breaking on average once a year, it would
David Patterson (1:31:16.240)
break every two weeks. So we thought about the idea and said, well, we got to address the
David Patterson (1:31:22.480)
reliability. So we did it originally performance, but we had to do reliability. So the name
David Patterson (1:31:27.120)
redundant array of inexpensive disks is array of these disks inexpensive like for PCs, but we have
David Patterson (1:31:33.440)
extra copies. So if one breaks, we won't lose all the information. We'll have enough redundancy that
David Patterson (1:31:40.400)
we could let some break and we can still preserve the information. So the name is an array of
David Patterson (1:31:44.560)
inexpensive disks. This is a collection of these PCs and the R part of the name was the redundancy
Lex Fridman (1:31:51.200)
so they'd be reliable. And it turns out if you put a modest number of extra disks in one of
David Patterson (1:31:55.760)
these arrays, it could actually not only be as faster and cheaper than one of these washing
David Patterson (1:32:00.720)
machine disks, it could be actually more reliable because you could have a couple of breaks even
David Patterson (1:32:05.360)
with these cheap disks. Whereas one failure with the washing machine thing would knock it out.
Lex Fridman (1:32:10.480)
Did you have a sense just like with risk that in the 30 years that followed,
David Patterson (1:32:17.360)
RAID would take over as a mechanism for storage? I think I'm naturally an optimist,
Lex Fridman (1:32:27.280)
but I thought our ideas were right. I thought kind of like Moore's law, it seemed to me,
David Patterson (1:32:33.840)
if you looked at the history of the disk drives, they went from washing machine size things and
David Patterson (1:32:38.000)
they were getting smaller and smaller and the volumes were with the smaller disk drives because
David Patterson (1:32:43.360)
that's where the PCs were. So we thought that was a technological trend that the volume of disk
David Patterson (1:32:51.120)
drives was going to be getting smaller and smaller devices, which were true. They were the size of,
David Patterson (1:32:56.480)
I don't know, eight inches diameter, then five inches, then three inches in diameters.
Lex Fridman (1:33:01.440)
And so that it made sense to figure out how to deal things with an array of disks. So I think
David Patterson (1:33:06.640)
it was one of those things where logically, we think the technological forces were on our side,
David Patterson (1:33:13.440)
that it made sense. So we expected it to catch on, but there was that same kind of business question.
David Patterson (1:33:19.920)
IBM was the big pusher of these disk drives in the real world where the technical advantage
David Patterson (1:33:25.840)
get turned into a business advantage or not. It proved to be true. And so we thought we were
David Patterson (1:33:32.560)
sound technically and it was unclear whether the business side, but we kind of, as academics,
David Patterson (1:33:38.320)
we believe that technology should win and it did. And if you look at those 30 years,
David Patterson (1:33:44.720)
just from your perspective, are there interesting developments in the space of storage
David Patterson (1:33:48.800)
that have happened in that time? Yeah. The big thing that happened, well, a couple of things
David Patterson (1:33:53.520)
that happened, what we did had a modest amount of storage. So as redundancy, as people built bigger
Lex Fridman (1:34:00.720)
and bigger storage systems, they've added more redundancy so they could add more failures. And
David Patterson (1:34:05.840)
the biggest thing that happened in storage is for decades, it was based on things physically spinning
David Patterson (1:34:14.240)
called hard disk drives where you used to turn on your computer and it would make a noise.
Lex Fridman (1:34:18.400)
What that noise was, was the disk drives spinning and they were rotating at like 60 revolutions per
David Patterson (1:34:25.200)
second. And it's like, if you remember the vinyl records, if you've ever seen those,
David Patterson (1:34:31.680)
that's what it looked like. And there was like a needle like on a vinyl record that was reading it.
Lex Fridman (1:34:36.160)
So the big drive change is switching that over to a semiconductor technology called flash.
Lex Fridman (1:34:41.440)
So within the last, I'd say about decade is increasing fraction of all the computers in the
David Patterson (1:34:47.200)
world are using semiconductor for storage, the flash drive, instead of being magnetic,
David Patterson (1:34:54.880)
they're optical, well, they're a semiconductor writing of information very densely.
Lex Fridman (1:35:04.080)
And that's been a huge difference. So all the cell phones in the world use flash.
David Patterson (1:35:08.000)
Most of the laptops use flash. All the embedded devices use flash instead of storage. Still in
David Patterson (1:35:13.520)
the cloud, magnetic disks are more economical than flash, but they use both in the cloud.
Lex Fridman (1:35:20.160)
So it's been a huge change in the storage industry, the switching from primarily disk
David Patterson (1:35:26.880)
to being primarily semiconductor. For the individual disk, but still the RAID mechanism
David Patterson (1:35:31.040)
applies to those different kinds of disks. Yes. The people will still use RAID ideas
David Patterson (1:35:35.920)
because it's kind of what's different, kind of interesting kind of psychologically,
David Patterson (1:35:41.120)
if you think about it. People have always worried about the reliability of computing since the
David Patterson (1:35:46.160)
earliest days. So kind of, but if we're talking about computation, if your computer makes a
David Patterson (1:35:52.240)
mistake and the computer says, the computer has ways to check and say, Oh, we screwed up.
David Patterson (1:35:59.120)
We made a mistake. What happens is that program that was running, you have to redo it,
David Patterson (1:36:04.160)
which is a hassle for storage. If you've sent important information away and it loses that
David Patterson (1:36:12.320)
information, you go nuts. This is the worst. Oh my God. So if you have a laptop and you're not
David Patterson (1:36:18.240)
backing it up on the cloud or something like this, and your disk drive breaks, which it can do,
David Patterson (1:36:24.880)
you'll lose all that information and you just go crazy. So the importance of reliability
David Patterson (1:36:29.760)
for storage is tremendously higher than the importance of reliability for computation
David Patterson (1:36:34.160)
because of the consequences of it. So yes, so RAID ideas are still very popular, even with
David Patterson (1:36:39.440)
the switch of the technology. Although flash drives are more reliable, if you're not doing
David Patterson (1:36:45.200)
anything like backing it up to get some redundancy so they handle it, you're taking great risks.
David Patterson (1:36:53.680)
You said that for you and possibly for many others, teaching and research don't
David Patterson (1:36:58.800)
conflict with each other as one might suspect. And in fact, they kind of complement each other. So
David Patterson (1:37:03.840)
maybe a question I have is how has teaching helped you in your research or just in your
David Patterson (1:37:10.480)
entirety as a person who both teaches and does research and just thinks and creates new ideas
David Patterson (1:37:17.040)
in this world? Yes, I think what happens is when you're a college student, you know there's this
David Patterson (1:37:22.880)
kind of tenure system in doing research. So kind of this model that is popular in America, I think
David Patterson (1:37:30.400)
America really made it happen, is we can attract these really great faculty to research universities
David Patterson (1:37:36.000)
because they get to do research as well as teach. And that, especially in fast moving fields,
David Patterson (1:37:40.640)
this means people are up to date and they're teaching those kinds of things. But when you run
David Patterson (1:37:44.800)
into a really bad professor, a really bad teacher, I think the students think, well, this guy must be
David Patterson (1:37:50.480)
a great researcher because why else could he be here? So after 40 years at Berkeley, we had a
David Patterson (1:37:57.280)
retirement party and I got a chance to reflect and I looked back at some things. That is not my
David Patterson (1:38:02.400)
experience. I saw a photograph of five of us in the department who won the Distinguished Teaching
David Patterson (1:38:09.600)
Award from campus, a very high honor. I've got one of those, one of the highest honors. So there are
David Patterson (1:38:14.480)
five of us on that picture. There's Manuel Blum, Richard Karp, me, Randy Kass, and John Osterhaupt,
David Patterson (1:38:23.360)
contemporaries of mine. I mentioned Randy already. All of us are in the National Academy of
David Patterson (1:38:27.920)
Engineering. We've all run the Distinguished Teaching Award. Blum, Karp, and I all have
David Patterson (1:38:34.160)
Turing Awards. The highest award in computing. So that's the opposite. What's happened is they're
David Patterson (1:38:45.120)
highly correlated. So the other way to think of it, if you're very successful people or maybe
David Patterson (1:38:51.280)
successful at everything they do, it's not an either or. But it's an interesting question
David Patterson (1:38:56.160)
whether specifically, that's probably true, but specifically for teaching, if there's something
David Patterson (1:39:00.880)
in teaching that, it's the Richard Feynman idea, is there something about teaching that actually
Lex Fridman (1:39:06.720)
makes your research, makes you think deeper and more outside the box and more insightful?
David Patterson (1:39:12.640)
Absolutely. I was going to bring up Feynman. I mean, he criticized the Institute of Advanced
David Patterson (1:39:16.400)
Studies. So the Institute of Advanced Studies was this thing that was created near Princeton
David Patterson (1:39:21.760)
where Einstein and all these smart people went. And when he was invited, he thought it was a
David Patterson (1:39:26.240)
terrible idea. This is a university. It was supposed to be heaven, right? A university
David Patterson (1:39:31.040)
without any teaching. But he thought it was a mistake. It's getting up in the classroom and
David Patterson (1:39:35.600)
having to explain things to students and having them ask questions like, well, why is that true,
David Patterson (1:39:40.640)
makes you stop and think. So he thought, and I agree, I think that interaction between a great
David Patterson (1:39:47.600)
research university and having students with bright young minds asking hard questions the
David Patterson (1:39:52.400)
whole time is synergistic. And a university without teaching wouldn't be as vital and
David Patterson (1:40:00.880)
exciting a place. And I think it helps stimulate the research. Another romanticized question,
Lex Fridman (1:40:07.920)
but what's your favorite concept or idea to teach? What inspires you or you see inspire the students?
David Patterson (1:40:15.680)
Is there something that pops to mind or puts the fear of God in them? I don't know,
Lex Fridman (1:40:19.440)
whichever is most effective. I mean, in general, I think people are surprised.
David Patterson (1:40:25.200)
I've seen a lot of people who don't think they like teaching come give guest lectures or teach
David Patterson (1:40:31.200)
a course and get hooked on seeing the lights turn on, right? You can explain something to
David Patterson (1:40:37.200)
people that they don't understand. And suddenly they get something that's important and difficult.
Lex Fridman (1:40:44.240)
And just seeing the lights turn on is a real satisfaction there. I don't think there's any
David Patterson (1:40:51.920)
specific example of that. It's just the general joy of seeing them understand.
David Patterson (1:40:58.320)
SL. I have to talk about this because I've wrestled. I do martial arts. Of course, I love wrestling. I'm a huge, I'm Russian. So I've talked to Dan Gable on the podcast.
Lex Fridman (1:41:11.520)
So you wrestled at UCLA among many other things you've done in your life, competitively in sports
Lex Fridman (1:41:20.640)
and science and so on. You've wrestled. Maybe, again, continue with the romanticized questions,
Lex Fridman (1:41:26.800)
but what have you learned about life and maybe even science from wrestling or from?
David Patterson (1:41:32.080)
CB. Yeah, in fact, I wrestled at UCLA, but also at El Camino Community College. And just right now,
David Patterson (1:41:39.520)
we were in the state of California, we were state champions at El Camino. And in fact, I was talking
David Patterson (1:41:44.400)
to my mom and I got into UCLA, but I decided to go to the community college, which is, it's much
David Patterson (1:41:52.000)
harder to go to UCLA than the community college. And I asked, why did I make that decision? Because I
David Patterson (1:41:56.400)
thought it was because of my girlfriend. She said, well, it was the girlfriend and you thought the
David Patterson (1:41:59.920)
wrestling team was really good. And we were right. We had a great wrestling team. We actually
David Patterson (1:42:06.000)
wrestled against UCLA at a tournament and we beat UCLA as a community college, which just freshmen
Lex Fridman (1:42:12.320)
and sophomores. And part of the reason I brought this up is I'm going to go, they've invited me back
David Patterson (1:42:17.440)
at El Camino to give a lecture next month. And so, my friend who was on the wrestling team that
David Patterson (1:42:27.440)
we're still together, we're right now reaching out to other members of the wrestling team if we can
David Patterson (1:42:31.680)
get together for a reunion. But in terms of me, it was a huge difference. The age cut off, it was
David Patterson (1:42:40.480)
December 1st. And so, I was almost always the youngest person in my class and I matured later
David Patterson (1:42:47.520)
on, our family matured later. So, I was almost always the smallest guy. So, I took kind of
David Patterson (1:42:54.560)
nerdy courses, but I was wrestling. So, wrestling was huge for my self confidence in high school.
Lex Fridman (1:43:02.480)
And then, I kind of got bigger at El Camino and in college. And so, I had this kind of physical
David Patterson (1:43:08.560)
self confidence and it's translated into research self confidence. And also kind of, I've had this
David Patterson (1:43:18.800)
feeling even today in my 70s, if something going on in the streets that is bad physically, I'm not
David Patterson (1:43:27.280)
going to ignore it. I'm going to stand up and try and straighten that out.
Lex Fridman (1:43:31.200)
And that kind of confidence just carries through the entirety of your life.
David Patterson (1:43:34.320)
Yeah. And the same things happens intellectually. If there's something going on where people are
David Patterson (1:43:39.040)
saying something that's not true, I feel it's my job to stand up just like I would in the street.
David Patterson (1:43:44.240)
If there's something going on, somebody attacking some woman or something, I'm not standing by and
David Patterson (1:43:49.120)
letting that get away. So, I feel it's my job to stand up. So, it's kind of ironically translates.
David Patterson (1:43:54.720)
The other things that turned out for both, I had really great college and high school coaches and
David Patterson (1:44:00.560)
they believed, even though wrestling is an individual sport, that we would be more successful
David Patterson (1:44:05.280)
as a team if we bonded together, do things that we would support each other rather than everybody,
David Patterson (1:44:10.880)
you know, in wrestling it's a one on one and you could be everybody's on their own, but he felt if
David Patterson (1:44:15.200)
we bonded as a team, we'd succeed. So, I kind of picked up those skills of how to form successful
David Patterson (1:44:21.200)
teams and how to, from wrestling. And so, I think one of, most people would say one of my strengths
David Patterson (1:44:27.280)
is I can create teams of faculty, large teams of faculty grad students, pull all together for a
David Patterson (1:44:33.200)
common goal and often be successful at it. But I got both of those things from wrestling. Also,
David Patterson (1:44:41.360)
I think I heard this line about if people are in kind of collision, sports with physical contact
David Patterson (1:44:49.040)
like wrestling or football and stuff like that, people are a little bit more assertive or something.
Lex Fridman (1:44:54.800)
And so, I think that also comes through as, you know, and I didn't shy away from the
David Patterson (1:45:02.160)
racist debates, you know, I enjoyed taking on the arguments and stuff like that. So,
David Patterson (1:45:08.800)
I'm really glad I did wrestling. I think it was really good for my self image and I learned a lot
David Patterson (1:45:13.520)
from it. So, I think that's, you know, sports done well, you know, there's really lots of positives
David Patterson (1:45:19.440)
you can take about it, of leadership, you know, how to form teams and how to be successful.
David Patterson (1:45:26.880)
So, we've talked about metrics a lot. There's a really cool, in terms of bench press and
David Patterson (1:45:30.880)
weightlifting, pound years metric that you've developed that we don't have time to talk about,
Lex Fridman (1:45:34.640)
but it's a really cool one that people should look into. It's rethinking the way we think about
David Patterson (1:45:39.040)
metrics and weightlifting. But let me talk about metrics more broadly, since that appeals to you
David Patterson (1:45:43.600)
in all forms. Let's look at the most ridiculous, the biggest question of the meaning of life.
Lex Fridman (1:45:50.480)
If you were to try to put metrics on a life well lived, what would those metrics be?
David Patterson (1:45:56.800)
Yeah, a friend of mine, Randy Katz, said this. He said, you know, when it's time to sign off,
David Patterson (1:46:06.000)
the measure isn't the number of zeros in your bank account, it's the number of inches
David Patterson (1:46:09.920)
in the obituary in the New York Times, was he said it. I think, you know, having,
Lex Fridman (1:46:17.040)
and you know, this is a cliche, is that people don't die wishing they'd spent more time in the
David Patterson (1:46:21.840)
office, right? As I reflect upon my career, there have been, you know, a half a dozen, a dozen things
David Patterson (1:46:29.360)
say I've been proud of. A lot of them aren't papers or scientific results. Certainly, my family,
David Patterson (1:46:35.440)
my wife, we've been married more than 50 years, kids and grandkids, that's really precious.
David Patterson (1:46:42.880)
Education things I've done, I'm very proud of, you know, books and courses. I did some help
David Patterson (1:46:50.240)
with underrepresented groups that was effective. So it was interesting to see what were the things
David Patterson (1:46:55.200)
I reflected. You know, I had hundreds of papers, but some of them were the papers, like the risk
David Patterson (1:47:00.960)
rate stuff that I'm proud of, but a lot of them were not those things. So people who are, just
David Patterson (1:47:06.480)
spend their lives, you know, going after the dollars or going after all the papers in the
David Patterson (1:47:11.040)
world, you know, that's probably not the things that are afterwards you're going to care about.
David Patterson (1:47:15.760)
When I was, just when I got the offer from Berkeley before I showed up, I read a book where
David Patterson (1:47:22.320)
they interviewed a lot of people in all works of life. And what I got out of that book was the
David Patterson (1:47:27.200)
people who felt good about what they did was the people who affected people, as opposed to things
David Patterson (1:47:31.520)
that were more transitory. So I came into this job assuming that it wasn't going to be the papers,
David Patterson (1:47:36.320)
it was going to be relationships with the people over time that I would value, and that was a
David Patterson (1:47:42.000)
correct assessment, right? It's the people you work with, the people you can influence, the people
David Patterson (1:47:47.120)
you can help, it's the things that you feel good about towards the end of your career. It's not
Lex Fridman (1:47:51.920)
the stuff that's more transitory.
David Patterson (1:47:53.200)
Trey Lockerbie I don't think there's a better way to end it than talking about your family,
Lex Fridman (1:47:58.480)
the over 50 years of being married to your childhood sweetheart.
David Patterson (1:48:02.320)
Richard Averbeck What I think I can add is,
Lex Fridman (1:48:05.040)
when you tell people you've been married 50 years, they want to know why.
Lex Fridman (1:48:07.280)
Trey Lockerbie How? Why?
Lex Fridman (1:48:08.800)
Richard Averbeck Yeah, I can tell you the nine
David Patterson (1:48:10.400)
magic words that you need to say to your partner to keep a good relationship. And the nine magic
David Patterson (1:48:16.560)
words are, I was wrong. You were right. I love you. Okay. And you got to say all nine. You can't
David Patterson (1:48:22.960)
say, I was wrong. You were right. You're a jerk. You know, you can't say that. So yeah, freely
David Patterson (1:48:28.160)
acknowledging that you made a mistake, the other person was right, and that you love them really
David Patterson (1:48:34.640)
gets over a lot of bumps in the road. So that's what I pass along.
Lex Fridman (1:48:37.760)
Trey Lockerbie Beautifully put. David,
David Patterson (1:48:39.840)
it's a huge honor. Thank you so much for the book you've written, for the research you've done,
Lex Fridman (1:48:43.840)
for changing the world. Thank you for talking today.
David Patterson (1:48:45.760)
Richard Averbeck Thanks for the interview.
Lex Fridman (1:48:46.880)
Trey Lockerbie Thanks for listening to this
David Patterson (1:48:48.880)
conversation with David Patterson. And thank you to our sponsors, The Jordan Harbinger Show, and
David Patterson (1:48:55.440)
Cash App. Please consider supporting this podcast by going to JordanHarbinger.com slash Lex and
David Patterson (1:49:02.320)
downloading Cash App and using code LexPodcast. Click the links, buy the stuff. It's the best way
David Patterson (1:49:08.640)
to support this podcast and the journey I'm on. If you enjoy this thing, subscribe on YouTube,
David Patterson (1:49:14.800)
review it with five stars in a podcast, support it on Patreon, or connect with me on Twitter at
David Patterson (1:49:19.680)
Lex Friedman, spelled without the E, try to figure out how to do that. It's just F R I D M A N.
Lex Fridman (1:49:27.280)
And now let me leave you with some words from Henry David Thoreau.
David Patterson (1:49:32.240)
Our life is faded away by detail. Simplify, simplify. Thank you for listening and hope to
David Patterson (1:49:40.560)
see you next time.
David Patterson (20:04.320)
understand. So they were writing add, subtract, multiply, you know, instructions. It's very tedious.
Lex Fridman (20:11.040)
But the belief was to write this lowest level of software that people use, which are called operating
David Patterson (20:17.440)
systems, they had to be written in assembly language because these high level languages were just too
David Patterson (20:22.000)
inefficient. They were too slow, or the programs would be too big. So that changed with a famous
David Patterson (20:31.600)
operating system called Unix, which is kind of the grandfather of all the operating systems today.
Lex Fridman (20:37.680)
So Unix demonstrated that you could write something as complicated as an operating system in a
David Patterson (20:43.280)
language like C. So once that was true, then that meant we could hide the instruction set from the
David Patterson (20:51.760)
programmer. And so that meant then it didn't really matter. The programmer didn't have to write
David Patterson (20:58.480)
lots of these simple instructions, that was up to the compiler. So that was part of our arguments
David Patterson (21:02.960)
for risk is, if you were still writing assembly language, there's maybe a better case for CISC
David Patterson (21:07.440)
instructions. But if the compiler can do that, it's going to be, you know, that's done once the
David Patterson (21:13.840)
computer translates at once. And then every time you run the program, it runs at this potentially
David Patterson (21:19.280)
simpler instructions. And so that was the debate, right? And people would acknowledge that the
David Patterson (21:26.960)
simpler instructions could lead to a faster computer. You can think of monosyllabic instructions,
David Patterson (21:33.040)
you could say them, you know, if you think of reading, you can probably read them faster or say
David Patterson (21:36.720)
them faster than long instructions. The same thing, that analogy works pretty well for hardware.
Lex Fridman (21:42.080)
And as long as you didn't have to read a lot more of those instructions, you could win. So that's
David Patterson (21:46.880)
kind of, that's the basic idea for risk. But it's interesting that in that discussion of Unix and C,
David Patterson (21:54.160)
that there's only one step of levels of abstraction from the code that's really the closest to the
David Patterson (22:02.080)
machine to the code that's written by human. It's, at least to me again, perhaps a dumb intuition,
Lex Fridman (22:09.840)
but it feels like there might've been more layers, sort of different kinds of humans stacked on top
David Patterson (22:16.000)
of each other. So what's true and not true about what you said is several of the layers of software,
David Patterson (22:27.120)
like, so the, if you, two layers would be, suppose we just talked about two layers,
David Patterson (22:32.560)
that would be the operating system, like you get from Microsoft or from Apple, like iOS,
David Patterson (22:38.240)
or the Windows operating system. And let's say applications that run on top of it, like Word
David Patterson (22:44.560)
or Excel. So both the operating system could be written in C and the application could be written
David Patterson (22:52.400)
in C. But you could construct those two layers and the applications absolutely do call upon the
David Patterson (22:58.880)
operating system. And the change was that both of them could be written in higher level languages.
Lex Fridman (23:04.720)
So it's one step of a translation, but you can still build many layers of abstraction
David Patterson (23:10.160)
of software on top of that. And that's how things are done today. So still today,
David Patterson (23:17.520)
many of the layers that you'll deal with, you may deal with debuggers, you may deal with linkers,
David Patterson (23:25.600)
there's libraries. Many of those today will be written in C++, say, even though that language is
David Patterson (23:34.080)
pretty ancient. And even the Python interpreter is probably written in C or C++. So lots of
David Patterson (23:41.120)
layers there are probably written in these, some old fashioned efficient languages that
David Patterson (23:48.080)
still take one step to produce these instructions, produce RISC instructions, but they're composed,
David Patterson (23:56.240)
each layer of software invokes one another through these interfaces. And you can get 10 layers of
David Patterson (24:02.800)
software that way. So in general, the RISC was developed here at Berkeley? It was kind of the
David Patterson (24:08.720)
three places that were these radicals that advocated for this against the rest of community
Lex Fridman (24:14.480)
were IBM, Berkeley, and Stanford. You're one of these radicals. And how radical did you feel?
Lex Fridman (24:24.400)
How confident did you feel? How doubtful were you that RISC might be the right approach? Because
David Patterson (24:31.680)
it may, you can also intuit that is kind of taking a step back into simplicity, not forward into
David Patterson (24:37.440)
simplicity. Yeah, no, it was easy to make, yeah, it was easy to make the argument against it. Well,
David Patterson (24:44.880)
this was my colleague, John Hennessy at Stanford Nine. We were both assistant professors. And
David Patterson (24:50.640)
for me, I just believed in the power of our ideas. I thought what we were saying made sense.
David Patterson (24:57.040)
Moore's law is going to move fast. The other thing that I didn't mention is one of the surprises of
David Patterson (25:03.200)
these complex instruction sets. You could certainly write these complex instructions
David Patterson (25:08.240)
if the programmer is writing them themselves. It turned out to be kind of difficult for the
David Patterson (25:13.440)
compiler to generate those complex instructions. Kind of ironically, you'd have to find the right
David Patterson (25:18.240)
circumstances that just exactly fit this complex instruction. It was actually easier for the
David Patterson (25:22.880)
compiler to generate these simple instructions. So not only did these complex instructions make
David Patterson (25:28.720)
the hardware more difficult to build, often the compiler wouldn't even use them. And so
David Patterson (25:35.280)
it's harder to build. The compiler doesn't use them that much. The simple instructions go better
David Patterson (25:40.320)
with Moore's law. The number of transistors is doubling every two years. So we're going to have,
David Patterson (25:46.320)
you want to reduce the time to design the microprocessor, that may be more important
David Patterson (25:51.360)
than these number of instructions. So I think we believed that we were right, that this was
David Patterson (25:58.160)
the best idea. Then the question became in these debates, well, yeah, that's a good technical idea,
Lex Fridman (26:03.840)
but in the business world, this doesn't matter. There's other things that matter. It's like
David Patterson (26:08.720)
arguing that if there's a standard with the railroad tracks and you've come up with a better
David Patterson (26:14.640)
width, but the whole world is covered in railroad tracks, so your ideas have no chance of success.
David Patterson (26:20.240)
Right. Commercial success. It was technically right, but commercially it'll be insignificant.
David Patterson (26:25.440)
Yeah, it's kind of sad that this world, the history of human civilization is full of good ideas that
David Patterson (26:33.280)
lost because somebody else came along first with a worse idea. And it's good that in the
David Patterson (26:39.040)
computing world, at least some of these have, well, you could, I mean, there's probably still
David Patterson (26:43.600)
CISC people that say, yeah, there still are. And what happened was, what was interesting, Intel,
David Patterson (26:50.640)
a bunch of the CISC companies with CISC instruction sets of vocabulary, they gave up,
Lex Fridman (26:57.360)
but not Intel. What Intel did to its credit, because Intel's vocabulary was in the personal
David Patterson (27:06.320)
computer. And so that was a very valuable vocabulary because the way we distribute software
David Patterson (27:11.760)
is in those actual instructions. It's in the instructions of that instruction set. So
David Patterson (27:17.280)
you don't get that source code, what the programmers wrote. You get, after it's been translated into
David Patterson (27:22.480)
the lowest level, that's if you were to get a floppy disk or download software, it's in the
David Patterson (27:27.440)
instructions of that instruction set. So the x86 instruction set was very valuable. So what Intel
David Patterson (27:33.680)
did cleverly and amazingly is they had their chips in hardware do a translation step.
David Patterson (27:40.960)
They would take these complex instructions and translate them into essentially in RISC instructions
David Patterson (27:45.280)
in hardware on the fly, at gigahertz clock speeds. And then any good idea that RISC people had,
David Patterson (27:52.720)
they could use, and they could still be compatible with this really valuable PC software base,
David Patterson (28:01.200)
which also had very high volumes, 100 million personal computers per year. So the CISC architecture
David Patterson (28:09.040)
in the business world was actually won in this PC era. So just going back to the
David Patterson (28:20.480)
time of designing RISC, when you design an instruction set architecture, do you think
David Patterson (28:27.680)
like a programmer? Do you think like a microprocessor engineer? Do you think like a
Lex Fridman (28:33.200)
artist, a philosopher? Do you think in software and hardware? I mean, is it art? Is it science?
David Patterson (28:40.240)
Yeah, I'd say, I think designing a good instruction set is an art. And I think you're trying to
David Patterson (28:47.760)
balance the simplicity and speed of execution with how well easy it will be for compilers
David Patterson (28:57.600)
to use it. You're trying to create an instruction set that everything in there can be used by
David Patterson (29:03.120)
compilers. There's not things that are missing that'll make it difficult for the program to run.
David Patterson (29:10.720)
They run efficiently, but you want it to be easy to build as well. So I'd say you're thinking
David Patterson (29:16.160)
hardware, trying to find a hardware software compromise that'll work well. And it's a matter
David Patterson (29:24.320)
of taste. It's kind of fun to build instruction sets. It's not that hard to build an instruction
David Patterson (29:30.880)
set, but to build one that catches on and people use, you have to be fortunate to be
Lex Fridman (29:38.000)
the right place in the right time or have a design that people really like. Are you using metrics?
Lex Fridman (29:43.200)
So is it quantifiable? Because you kind of have to anticipate the kind of programs that people
David Patterson (29:49.200)
write ahead of time. So can you use numbers? Can you use metrics? Can you quantify something ahead
David Patterson (29:56.080)
of time? Or is this, again, that's the art part where you're kind of anticipating? No, it's a big
David Patterson (30:00.960)
change. Kind of what happened, I think from Hennessy's and my perspective in the 1980s,
Lex Fridman (30:07.040)
what happened was going from kind of really, you know, taste and hunches to quantifiable. And in
David Patterson (30:17.040)
fact, he and I wrote a textbook at the end of the 1980s called Computer Architecture, A Quantitative
David Patterson (30:22.880)
Approach. I heard of that. And it's the thing, it had a pretty big impact in the field because we
David Patterson (30:30.320)
went from textbooks that kind of listed, so here's what this computer does, and here's the pros and
David Patterson (30:36.320)
cons, and here's what this computer does and pros and cons to something where there were formulas
Lex Fridman (30:40.560)
and equations where you could measure things. So specifically for instruction sets, what we do
Lex Fridman (30:47.680)
and some other fields do is we agree upon a set of programs, which we call benchmarks,
Lex Fridman (30:53.680)
and a suite of programs, and then you develop both the hardware and the compiler and you get
David Patterson (31:00.960)
numbers on how well your computer does given its instruction set and how well you implemented it in
David Patterson (31:09.360)
your microprocessor and how good your compilers are. In computer architecture, you know, using
David Patterson (31:14.960)
professor's terms, we grade on a curve rather than grade on an absolute scale. So when you say,
David Patterson (31:20.400)
you know, these programs run this fast, well, that's kind of interesting, but how do you know
David Patterson (31:24.640)
it's better? Well, you compare it to other computers at the same time. So the best way we
David Patterson (31:29.840)
know how to turn it into a kind of more science and experimental and quantitative is to compare
David Patterson (31:37.680)
yourself to other computers of the same era that have the same access to the same kind of technology
Lex Fridman (31:41.920)
on commonly agreed benchmark programs.
Lex Fridman (31:44.940)
So maybe to toss up two possible directions we can go. One is what are the different tradeoffs
David Patterson (31:51.520)
in designing architectures? We've been already talking about SISC and RISC, but maybe a little
David Patterson (31:56.640)
bit more detail in terms of specific features that you were thinking about. And the other side is
Lex Fridman (32:03.680)
what are the metrics that you're thinking about when looking at these tradeoffs?
David Patterson (32:08.160)
Yeah, let's talk about the metrics. So during these debates, we actually had kind of a hard
David Patterson (32:14.160)
time explaining, convincing people the ideas, and partly we didn't have a formula to explain it.
Lex Fridman (32:20.240)
And a few years into it, we hit upon a formula that helped explain what was going on. And
David Patterson (32:27.600)
I think if we can do this, see how it works orally to do this. So if I can do a formula
David Patterson (32:34.080)
orally, let's see. So fundamentally, the way you measure performance is how long does it take a
David Patterson (32:40.720)
program to run? A program, if you have 10 programs, and typically these benchmarks were sweet because
David Patterson (32:47.360)
you'd want to have 10 programs so they could represent lots of different applications. So for
David Patterson (32:51.600)
these 10 programs, how long does it take to run? Well now, when you're trying to explain why it
David Patterson (32:56.080)
took so long, you could factor how long it takes a program to run into three factors.
David Patterson (33:01.120)
One of the first one is how many instructions did it take to execute? So that's the what we've been
David Patterson (33:06.240)
talking about, you know, the instructions of Alchemy. How many did it take? All right. The
David Patterson (33:11.360)
next question is how long did each instruction take to run on average? So you multiply the number
David Patterson (33:17.840)
of instructions times how long it took to run, and that gets you time. Okay, so that's, but now let's
David Patterson (33:23.520)
look at this metric of how long did it take the instruction to run. Well, it turns out,
David Patterson (33:28.240)
the way we could build computers today is they all have a clock, and you've seen this when you,
David Patterson (33:33.280)
if you buy a microprocessor, it'll say 3.1 gigahertz or 2.5 gigahertz, and more gigahertz is
David Patterson (33:39.760)
good. Well, what that is is the speed of the clock. So 2.5 gigahertz turns out to be 4 billionths of
David Patterson (33:47.600)
instruction or 4 nanoseconds. So that's the clock cycle time. But there's another factor, which is
David Patterson (33:54.160)
what's the average number of clock cycles it takes per instruction? So it's number of instructions,
David Patterson (33:59.840)
average number of clock cycles, and the clock cycle time. So in these risk sis debates, they
David Patterson (34:05.760)
would concentrate on, but risk needs to take more instructions, and we'd argue maybe the clock cycle
David Patterson (34:12.800)
is faster, but what the real big difference was was the number of clock cycles per instruction.
David Patterson (34:17.680)
Per instruction, that's fascinating. What about the mess of, the beautiful mess of parallelism in the
David Patterson (34:25.920)
whole picture? Parallelism, which has to do with, say, how many instructions could execute in parallel
Lex Fridman (34:31.280)
and things like that, you could think of that as affecting the clock cycles per instruction,
David Patterson (34:35.440)
because it's the average clock cycles per instruction. So when you're running a program,
David Patterson (34:39.280)
if it took 100 billion instructions, and on average it took two clock cycles per instruction,
Lex Fridman (34:45.840)
and they were four nanoseconds, you could multiply that out and see how long it took to run.
Lex Fridman (34:49.840)
And there's all kinds of tricks to try and reduce the number of clock cycles per instruction.
Lex Fridman (34:55.440)
But it turned out that the way they would do these complex instructions is they would actually
David Patterson (35:00.400)
build what we would call an interpreter in a simpler, a very simple hardware interpreter.
Lex Fridman (35:05.840)
But it turned out that for the sis constructions, if you had to use one of those interpreters,
David Patterson (35:10.720)
it would be like 10 clock cycles per instruction, where the risk constructions could be two. So
David Patterson (35:16.160)
there'd be this factor of five advantage in clock cycles per instruction. We have to execute, say,
David Patterson (35:21.280)
25 or 50 percent more instructions, so that's where the win would come. And then you could
David Patterson (35:25.440)
make an argument whether the clock cycle times are the same or not. But pointing out that we
David Patterson (35:30.080)
could divide the benchmark results time per program into three factors, and the biggest
David Patterson (35:36.000)
difference between RISC and SIS was the clock cycles per, you execute a few more instructions,
Lex Fridman (35:40.640)
but the clock cycles per instruction is much less. And that was what this debate, once we
David Patterson (35:46.000)
made that argument, then people said, oh, okay, I get it. And so we went from, it was outrageously
David Patterson (35:53.280)
controversial in, you know, 1982 that maybe probably by 1984 or so, people said, oh, yeah,
David Patterson (35:59.520)
technically, they've got a good argument. What are the instructions in the RISC instruction set,
David Patterson (36:05.680)
just to get an intuition? Okay. 1995, I was asked to predict the future of what microprocessor
David Patterson (36:13.600)
could future. So I, and I'd seen these predictions and usually people predict something outrageous
David Patterson (36:20.240)
just to be entertaining, right? And so my prediction for 2020 was, you know, things are
David Patterson (36:25.920)
going to be pretty much, they're going to look very familiar to what they are. And they are,
Lex Fridman (36:30.080)
and if you were to read the article, you know, the things I said are pretty much true. The
David Patterson (36:34.400)
instructions that have been around forever are kind of the same. And that's the outrageous
David Patterson (36:38.880)
prediction, actually. Yeah. Given how fast computers have been going. Well, and you know,
David Patterson (36:42.160)
Moore's law was going to go on, we thought for 25 more years, you know, who knows, but kind of the
David Patterson (36:47.840)
surprising thing, in fact, you know, Hennessy and I, you know, won the ACM, AM, Turing award for
David Patterson (36:55.120)
both the RISC instruction set contributions and for that textbook I mentioned. But, you know,
David Patterson (37:00.160)
we're surprised that here we are 35, 40 years later after we did our work and the conventionalism
David Patterson (37:10.400)
of the best way to do instruction sets is still those RISC instruction sets that looked very
David Patterson (37:15.120)
similar to what we looked like, you know, we did in the 1980s. So those, surprisingly, there hasn't
David Patterson (37:21.200)
been some radical new idea, even though we have, you know, a million times as many transistors as
David Patterson (37:26.880)
we had back then. But what are the basic constructions and how do they change over the
David Patterson (37:32.720)
years? So we're talking about addition, subtraction, these are the specific. So the things that are in
David Patterson (37:39.200)
a calculator are in a computer. So any of the buttons that are in the calculator in the computer,
Lex Fridman (37:44.640)
so the, so if there's a memory function key, and like I said, those are, turns into putting
David Patterson (37:50.160)
something in memory is called a store, bring something back to load. Just a quick tangent.
David Patterson (37:54.720)
When you say memory, what does memory mean? Well, I told you there were five pieces of a computer.
Lex Fridman (38:00.400)
And if you remember in a calculator, there's a memory key. So you want to have intermediate
David Patterson (38:04.720)
calculation and bring it back later. So you'd hit the memory plus key M plus maybe, and it would
David Patterson (38:09.680)
put that into memory and then you'd hit an RM like recurrence section and then bring it back
David Patterson (38:14.320)
on the display. So you don't have to type it. You don't have to write it down and bring it back
David Patterson (38:17.360)
again. So that's exactly what memory is. You can put things into it as temporary storage and bring
David Patterson (38:22.880)
it back when you need it later. So that's memory and loads and stores. But the big thing, the
David Patterson (38:28.160)
difference between a computer and a calculator is that the computer can make decisions. And
David Patterson (38:35.440)
amazingly, decisions are as simple as, is this value less than zero? Or is this value bigger
David Patterson (38:41.600)
than that value? And those instructions, which are called conditional branch instructions,
David Patterson (38:47.440)
is what give computers all its power. If you were in the early days of computing before
David Patterson (38:53.200)
what's called the general purpose microprocessor, people would write these instructions kind of in
David Patterson (38:58.560)
hardware, but it couldn't make decisions. It would do the same thing over and over again.
David Patterson (39:05.520)
With the power of having branch instructions, it can look at things and make decisions
David Patterson (39:09.680)
automatically. And it can make these decisions billions of times per second. And amazingly
David Patterson (39:15.200)
enough, we can get, thanks to advanced machine learning, we can create programs that can do
David Patterson (39:20.640)
something smarter than human beings can do. But if you go down that very basic level, it's the
David Patterson (39:25.600)
instructions are the keys on the calculator, plus the ability to make decisions, these conditional
David Patterson (39:30.960)
branch instructions. And all decisions fundamentally can be reduced down to these
David Patterson (39:34.960)
branch instructions. Yeah. So in fact, and so going way back in the stack back to,
David Patterson (39:42.240)
we did four RISC projects at Berkeley in the 1980s. They did a couple at Stanford
David Patterson (39:47.200)
in the 1980s. In 2010, we decided we wanted to do a new instruction set learning from the mistakes
David Patterson (39:54.800)
of those RISC architectures in the 1980s. And that was done here at Berkeley almost exactly
David Patterson (40:00.640)
10 years ago. And the people who did it, I participated, but Krzysztof Sanowicz and others
David Patterson (40:07.200)
drove it. They called it RISC 5 to honor those RISC, the four RISC projects of the 1980s.
Lex Fridman (40:13.840)
So what does RISC 5 involve? So RISC 5 is another instruction set of vocabulary. It's learned from
David Patterson (40:21.200)
the mistakes of the past, but it still has, if you look at the, there's a core set of instructions
David Patterson (40:25.680)
that's very similar to the simplest architectures from the 1980s. And the big difference about RISC
David Patterson (40:31.280)
5 is it's open. So I talked early about proprietary versus open software. So this is an instruction
David Patterson (40:41.920)
set. So it's a vocabulary, it's not hardware, but by having an open instruction set, we can have
David Patterson (40:47.920)
open source implementations, open source processors that people can use. Where do you see that
David Patterson (40:54.960)
going? It's a really exciting possibility, but you're just like in the scientific American,
David Patterson (41:00.080)
if you were to predict 10, 20, 30 years from now, that kind of ability to utilize open source
Lex Fridman (41:07.840)
instruction set architectures like RISC 5, what kind of possibilities might that unlock?
David Patterson (41:13.600)
Yeah. And so just to make it clear, because this is confusing, the specification of RISC 5 is
David Patterson (41:20.320)
something that's like in a textbook, there's books about it. So that's defining an interface.
David Patterson (41:27.520)
There's also the way you build hardware is you write it in languages that are kind of like C,
Lex Fridman (41:33.280)
but they're specialized for hardware that gets translated into hardware. And so these
David Patterson (41:39.440)
implementations of this specification are the open source. So they're written in something
David Patterson (41:44.960)
that's called Verilog or VHDL, but it's put up on the web, just like you can see the C++ code for
David Patterson (41:53.120)
Linux on the web. So that's the open instruction set enables open source implementations of RISC 5.
Lex Fridman (42:00.560)
So you can literally build a processor using this instruction set.
David Patterson (42:04.080)
People are, people are. So what happened to us that the story was this was developed here for
David Patterson (42:09.360)
our use to do our research. And we made it, we licensed under the Berkeley Software Distribution
David Patterson (42:15.440)
License, like a lot of things get licensed here. So other academics use it, they wouldn't be afraid
David Patterson (42:19.760)
to use it. And then about 2014, we started getting complaints that we were using it in our research
Lex Fridman (42:27.200)
and in our courses. And we got complaints from people in industries, why did you change your
David Patterson (42:32.160)
instruction set between the fall and the spring semester? And well, we get complaints from
David Patterson (42:37.840)
industrial time. Why the hell do you care what we do with our instruction set? And then when we
David Patterson (42:42.400)
talked to him, we found out there was this thirst for this idea of an open instruction set
David Patterson (42:46.960)
architecture. And they had been looking for one. They stumbled upon ours at Berkeley, thought it
David Patterson (42:51.600)
was, boy, this looks great. We should use this one. And so once we realized there is this need
David Patterson (42:58.400)
for an open instruction set architecture, we thought that's a great idea. And then we started
David Patterson (43:02.960)
supporting it and tried to make it happen. So this was kind of, we accidentally stumbled into this
Lex Fridman (43:09.600)
and to this need and our timing was good. And so it's really taking off. There's,
David Patterson (43:16.480)
you know, universities are good at starting things, but they're not good at sustaining things. So like
David Patterson (43:20.800)
Linux has a Linux foundation, there's a RISC 5 foundation that we started. There's an annual
David Patterson (43:26.640)
conferences. And the first one was done, I think, January of 2015. And the one that was just last
David Patterson (43:32.720)
December in it, you know, it had 50 people at it. And this one last December had, I don't know,
David Patterson (43:38.880)
1700 people were at it and the companies excited all over the world. So if predicting into the
David Patterson (43:44.880)
future, you know, if we were doing 25 years, I would predict that RISC 5 will be, you know,
David Patterson (43:51.040)
possibly the most popular instruction set architecture out there, because it's a pretty
David Patterson (43:57.120)
good instruction set architecture and it's open and free. And there's no reason lots of people
David Patterson (44:03.440)
shouldn't use it. And there's benefits just like Linux is so popular today compared to 20 years
David Patterson (44:10.000)
ago. And, you know, the fact that you can get access to it for free, you can modify it, you can
David Patterson (44:17.360)
improve it for all those same arguments. And so people collaborate to make it a better system
David Patterson (44:22.480)
for everybody to use. And that works in software. And I expect the same thing will happen in
David Patterson (44:26.640)
hardware. So if you look at ARM, Intel, MIPS, if you look at just the lay of the land,
Lex Fridman (44:34.080)
and what do you think, just for me, because I'm not familiar how difficult this kind of transition
David Patterson (44:42.000)
would, how much challenges this kind of transition would entail, do you see,
Lex Fridman (44:50.400)
let me ask my dumb question in another way.
David Patterson (44:52.240)
No, that's, I know where you're headed. Well, there's a bunch, I think the thing you point out,
Lex Fridman (44:57.280)
there's these very popular proprietary instruction sets, the x86.
Lex Fridman (45:02.400)
And so how do we move to RISC 5 potentially in sort of in the span of 5, 10, 20 years,
Lex Fridman (45:09.040)
a kind of unification, given that the devices, the kind of way we use devices,
Lex Fridman (45:15.360)
IoT, mobile devices, and the cloud keeps changing?
David Patterson (45:20.080)
Well, part of it, a big piece of it is the software stack. And right now, looking forward,
David Patterson (45:27.920)
there seem to be three important markets. There's the cloud. And the cloud is simply
David Patterson (45:34.720)
companies like Alibaba and Amazon and Google, Microsoft, having these giant data centers with
David Patterson (45:42.720)
tens of thousands of servers in maybe a hundred of these data centers all over the world.
Lex Fridman (45:48.800)
And that's what the cloud is. So the computer that dominates the cloud is the x86 instruction set.
Lex Fridman (45:54.560)
So the instruction sets used in the cloud are the x86, almost 100% of that today is x86.
David Patterson (46:03.040)
The other big thing are cell phones and laptops. Those are the big things today.
David Patterson (46:08.640)
I mean, the PC is also dominated by the x86 instruction set, but those sales are dwindling.
David Patterson (46:14.480)
You know, there's maybe 200 million PCs a year, and there's one and a half billion phones a year.
David Patterson (46:21.600)
There's numbers like that. So for the phones, that's dominated by ARM.
Lex Fridman (46:26.800)
And now, and a reason that I talked about the software stacks, and the third category is
David Patterson (46:33.920)
Internet of Things, which is basically embedded devices, things in your cars and your microwaves
David Patterson (46:38.160)
everywhere. So what's different about those three categories is for the cloud, the software that
David Patterson (46:45.360)
runs in the cloud is determined by these companies, Alibaba, Amazon, Google, Microsoft. So they
David Patterson (46:51.600)
control that software stack. For the cell phones, there's both for Android and Apple, the software
David Patterson (46:58.320)
they supply, but both of them have marketplaces where anybody in the world can build software.
Lex Fridman (47:03.760)
And that software is translated or, you know, compiled down and shipped in the vocabulary of ARM.
Lex Fridman (47:11.680)
So that's what's referred to as binary compatible because the actual, it's the instructions are
Lex Fridman (47:18.560)
turned into numbers, binary numbers, and shipped around the world.
Lex Fridman (47:21.760)
And sorry, just a quick interruption. So ARM, what is ARM? ARM is an instruction set, like a risk based...
David Patterson (47:29.120)
Yeah, it's a risk based instruction set. It's a proprietary one. ARM stands for Advanced Risk
David Patterson (47:36.400)
Machine. ARM is the name where the company is. So it's a proprietary risk architecture.
David Patterson (47:41.040)
So, and it's been around for a while and it's, you know, the, surely the most popular instruction set
David Patterson (47:50.000)
in the world right now. They, every year, billions of chips are using the ARM design in this post PC
David Patterson (47:56.800)
era. Was it one of the early risk adopters of the risk idea? Yeah. The first ARM goes back,
David Patterson (48:03.120)
I don't know, 86 or so. So Berkeley instead did their work in the early 80s. The ARM guys needed
David Patterson (48:09.840)
an instruction set and they read our papers and it heavily influenced them. So getting back to my
David Patterson (48:17.360)
story, what about Internet of Things? Well, software is not shipped in Internet of Things. It's the
David Patterson (48:22.960)
embedded device people control that software stack. So the opportunities for risk five,
David Patterson (48:29.680)
everybody thinks, is in the Internet of Things embedded things because there's no dominant
David Patterson (48:34.640)
player like there is in the cloud or the smartphones. And, you know, it's, it's,
David Patterson (48:41.920)
doesn't have a lot of licenses associated with, and you can enhance the instruction set if you want.
Lex Fridman (48:46.720)
And it's, and people have looked at instruction sets and think it's a very good instruction set.
Lex Fridman (48:52.800)
So it appears to be very popular there. It's possible that in the cloud people,
David Patterson (48:59.840)
those companies control their software stacks. So it's possible that they would decide to use
David Patterson (49:05.920)
risk five if we're talking about 10 and 20 years in the future. The one that would be harder would
David Patterson (49:10.880)
be the cell phones. Since people ship software in the ARM instruction set that you'd think be
David Patterson (49:16.160)
the more difficult one. But if risk five really catches on and, you know, you could,
David Patterson (49:20.720)
in a period of a decade, you can imagine that's changing over too. Do you have a sense why risk
David Patterson (49:25.920)
five or ARM has dominated? You mentioned these three categories. Why has, why did ARM dominate,
Lex Fridman (49:31.200)
why does it dominate the mobile device space? And maybe my naive intuition is that there are some
David Patterson (49:38.560)
aspects of power efficiency that are important that somehow come along with risk. Well, part of it is
David Patterson (49:44.000)
for these old CIS construction sets, like in the x86, it was more expensive to these for, you know,
David Patterson (49:55.920)
they're older, so they have disadvantages in them because they were designed 40 years ago. But also
David Patterson (50:01.760)
they have to translate in hardware from CIS constructions to risk constructions on the fly.
Lex Fridman (50:06.720)
And that costs both silicon area that the chips are bigger to be able to do that.
Lex Fridman (50:12.240)
And it uses more power. So ARM has, which has, you know, followed this risk philosophy is
David Patterson (50:18.240)
seen to be much more energy efficient. And in today's computer world, both in the cloud
Lex Fridman (50:23.760)
and the cell phone and, you know, things, it isn't, the limiting resource isn't the number of
David Patterson (50:29.920)
transistors you can fit in the chip. It's what, how much power can you dissipate for your
David Patterson (50:34.560)
application? So by having a reduced instruction set, that's possible to have a simpler hardware,
David Patterson (50:42.080)
which is more energy efficient. And energy efficiency is incredibly important in the cloud.
David Patterson (50:46.480)
When you have tens of thousands of computers in a data center, you want to have the most energy
David Patterson (50:51.040)
efficient ones there as well. And of course, for embedded things running off of batteries,
David Patterson (50:54.880)
you want those to be energy efficient and the cell phones too. So I think it's believed that
David Patterson (51:00.400)
there's a energy disadvantage of using these more complex instruction set architectures.
Lex Fridman (51:08.400)
So the other aspect of this is if we look at Apple, Qualcomm, Samsung, Huawei, all use the
David Patterson (51:14.880)
ARM architecture, and yet the performance of the systems varies. I mean, I don't know
David Patterson (51:20.320)
whose opinion you take on, but you know, Apple for some reason seems to perform better in terms of
David Patterson (51:26.000)
these implementation, these architectures. So where's the magic and show the picture.
David Patterson (51:30.480)
How's that happen? Yeah. So what ARM pioneered was a new business model. As they said, well,
David Patterson (51:35.120)
here's our proprietary instruction set, and we'll give you two ways to do it.
David Patterson (51:41.040)
We'll give you one of these implementations written in things like C called Verilog,
Lex Fridman (51:46.800)
and you can just use ours. Well, you have to pay money for that. Not only you pay,
David Patterson (51:51.840)
we'll give you their, you know, we'll license you to do that, or you could design your own. And so
David Patterson (51:57.360)
we're talking about numbers like tens of millions of dollars to have the right to design your own,
David Patterson (52:02.400)
since they, it's the instruction set belongs to them. So Apple got one of those, the right to
David Patterson (52:08.960)
build their own. Most of the other people who build like Android phones just get one of the designs
David Patterson (52:15.440)
from ARM to do it themselves. So Apple developed a really good microprocessor design team. They,
David Patterson (52:24.800)
you know, acquired a very good team that had, was building other microprocessors and brought them
David Patterson (52:30.640)
into the company to build their designs. So the instruction sets are the same, the specifications
David Patterson (52:35.760)
are the same, but their hardware design is much more efficient than I think everybody else's.
Lex Fridman (52:40.800)
And that's given Apple an advantage in the marketplace in that the iPhones tend to be the
David Patterson (52:49.520)
faster than most everybody else's phones that are there. It'd be nice to be able to jump around and
David Patterson (52:55.680)
kind of explore different little sides of this, but let me ask one sort of romanticized question.
Lex Fridman (53:01.280)
What to you is the most beautiful aspect or idea of RISC instruction set?
David Patterson (53:07.120)
Most beautiful aspect or idea of RISC instruction set or instruction sets or this work that you've
David Patterson (53:13.920)
done? You know, I'm, you know, I was always attracted to the idea of, you know, small is
David Patterson (53:20.400)
beautiful, right? Is that the temptation in engineering, it's kind of easy to make things
David Patterson (53:26.640)
more complicated. It's harder to come up with a, it's more difficult, surprisingly, to come up with
David Patterson (53:32.160)
a simple, elegant solution. And I think that there's a bunch of small features of RISC in general
David Patterson (53:39.120)
that, you know, where you can see this examples of keeping it simpler makes it more elegant.
David Patterson (53:45.520)
Specifically in RISC 5, which, you know, I was kind of the mentor in the program, but it was
David Patterson (53:50.160)
really driven by Krzysztof Sanović and two grad students, Andrew Waterman and Yen Tsip Li, is they
David Patterson (53:56.480)
hit upon this idea of having a subset of instructions, a nice, simple subset instructions,
David Patterson (54:05.200)
like 40ish instructions that all software, the software staff RISC 5 can run just on those 40
David Patterson (54:12.880)
instructions. And then they provide optional features that could accelerate the performance
David Patterson (54:20.080)
instructions that if you needed them could be very helpful, but you don't need to have them.
Lex Fridman (54:24.160)
And that's a new, really a new idea. So RISC 5 has right now maybe five optional subsets that
David Patterson (54:31.840)
you can pull in, but the software runs without them. If you just want to build the, just the core
David Patterson (54:37.360)
40 instructions, that's fine. You can do that. So this is fantastic educationally is you can
David Patterson (54:43.760)
explain computers. You only have to explain 40 instructions and not thousands of them. Also,
David Patterson (54:48.960)
if you invent some wild and crazy new technology like, you know, biological computing, you'd like
David Patterson (54:56.000)
a nice, simple instruction set and you can, RISC 5, if you implement those core instructions, you
David Patterson (55:02.000)
can run, you know, really interesting programs on top of that. So this idea of a core set of
David Patterson (55:07.360)
instructions that the software stack runs on and then optional features that if you turn them on,
David Patterson (55:13.280)
the compilers were used, but you don't have to, I think is a powerful idea. What's happened in
David Patterson (55:18.720)
the past for the proprietary instruction sets is when they add new instructions, it becomes
David Patterson (55:25.680)
required piece. And so that all microprocessors in the future have to use those instructions. So
David Patterson (55:33.520)
it's kind of like, for a lot of people as they get older, they gain weight, right? That weight and
David Patterson (55:39.840)
age are correlated. And so you can see these instruction sets getting bigger and bigger as
David Patterson (55:44.320)
they get older. So RISC 5, you know, lets you be as slim as you as a teenager. And you only have to
David Patterson (55:50.640)
add these extra features if you're really going to use them rather than you have no choice. You have
David Patterson (55:55.760)
to keep growing with the instruction set. I don't know if the analogy holds up, but that's a beautiful
David Patterson (56:00.320)
notion that there's, it's almost like a nudge towards here's the simple core. That's the
David Patterson (56:06.560)
essential. Yeah. And I think the surprising thing is still if we brought back, you know,
David Patterson (56:12.000)
the pioneers from the 1950s and showed them the instruction set architectures, they'd understand
David Patterson (56:16.480)
it. They'd say, wow, that doesn't look that different. Well, you know, I'm surprised. And
David Patterson (56:21.920)
it's, there's, it may be something, you know, to talk about philosophical things. I mean, there may
David Patterson (56:26.400)
be something powerful about those, you know, 40 or 50 instructions that all you need is these
David Patterson (56:35.840)
commands like these instructions that we talked about. And that is sufficient to build, to bring
David Patterson (56:42.160)
up on, you know, artificial intelligence. And so it's a remarkable, surprising to me that as
David Patterson (56:50.640)
complicated as it is to build these things, you know, microprocessors where the line widths are
David Patterson (56:58.800)
are narrower than the wavelength of light, you know, is this amazing technologies at some
David Patterson (57:05.200)
fundamental level. The commands that software executes are really pretty straightforward and
David Patterson (57:10.160)
haven't changed that much in decades. What a surprising outcome. So underlying all computation,
David Patterson (57:18.080)
all Turing machines, all artificial intelligence systems, perhaps might be a very simple instruction
David Patterson (57:23.760)
set like a RISC5 or it's. Yeah. I mean, that's kind of what I said. I was interested to see,
David Patterson (57:30.800)
I had another more senior faculty colleague and he had written something in Scientific American
David Patterson (57:36.480)
and, you know, his 25 years in the future and his turned out about when I was a young professor and
David Patterson (57:43.040)
he said, yep, I checked it. And so I was interested to see how that was going to turn out for me. And
David Patterson (57:48.400)
it's pretty held up pretty well, but yeah, so there's, there's probably, there's some, you know,
David Patterson (57:54.080)
there's, there must be something fundamental about those instructions that we're capable of
David Patterson (58:01.040)
creating, you know, intelligence from pretty primitive operations and just doing them really
David Patterson (58:07.920)
fast. You kind of mentioned a different, maybe radical computational medium like biological,
Lex Fridman (58:14.560)
and there's other ideas. So there's a lot of spaces in ASIC, domain specific, and then there
David Patterson (58:20.320)
could be quantum computers. And so we can think of all of those different mediums and types of
David Patterson (58:25.440)
computation. What's the connection between swapping out different hardware systems and the
David Patterson (58:33.120)
instruction set? Do you see those as disjoint or are they fundamentally coupled? Yeah. So what's,
Lex Fridman (58:37.920)
so kind of, if we go back to the history, you know, when Moore's Law is in full effect and
David Patterson (58:45.440)
you're getting twice as many transistors every couple of years, you know, kind of the challenge
David Patterson (58:51.760)
for computer designers is how can we take advantage of that? How can we turn those transistors into
David Patterson (58:56.800)
better computers faster typically? And so there was an era, I guess in the 80s and 90s where
David Patterson (59:04.400)
computers were doubling performance every 18 months. And if you weren't around then,
Lex Fridman (59:11.520)
what would happen is you had your computer and your friend's computer, which was like a year,
David Patterson (59:18.480)
a year and a half newer, and it was much faster than your computer. And he or she could get their
David Patterson (59:23.760)
work done much faster than your computer because it was newer. So people took their computers,
David Patterson (59:27.680)
perfectly good computers, and threw them away to buy a newer computer because the computer
David Patterson (59:33.920)
one or two years later was so much faster. So that's what the world was like in the 80s and
David Patterson (59:39.040)
90s. Well, with the slowing down of Moore's Law, that's no longer true, right? Now with, you know,
David Patterson (59:46.560)
not desk side computers with the laptops, I only get a new desk laptop when it breaks,
David Patterson (59:51.440)
right? Oh damn, the disk broke or this display broke, I gotta buy a new computer. But before
David Patterson (59:56.480)
you would throw them away because it just, they were just so sluggish compared to the latest
🔗 相关节目