Rajat Monga: TensorFlow
AI 与机器学习音乐与艺术技术与编程心理与人性商业与创业
📋 章节目录
暂无章节信息
🔑 关键词
tensorflowlearninggooglestartedlotsresearchteamtogetherdondeepdoingsureimportantecosystemcommunitydatasourcegoinginterestingterms
💬 精彩语录
暂无语录
🎙️ 完整对话（1602 条）
Lex Fridman (00:00.000)
The following is a conversation with Rajat Manga.
以下是与 Rajat Manga 的对话。
Lex Fridman (00:03.080)
He's an engineer and director of Google,
他是谷歌的工程师兼董事，
Lex Fridman (00:04.920)
leading the TensorFlow team.
领导 TensorFlow 团队。
Lex Fridman (00:06.960)
TensorFlow is an open source library
TensorFlow 是一个开源库
Lex Fridman (00:09.160)
at the center of much of the work going on in the world
是世界上许多工作的核心
Rajat Monga (00:11.540)
in deep learning, both the cutting edge research
在深度学习领域，无论是前沿研究还是
Lex Fridman (00:14.040)
and the large scale application of learning based approaches.
以及基于学习的方法的大规模应用。
Lex Fridman (00:17.720)
But it's quickly becoming much more than a software library.
但它很快就不仅仅是一个软件库了。
Lex Fridman (00:20.940)
It's now an ecosystem of tools for the deployment of machine
它现在是一个用于部署机器的工具生态系统
Rajat Monga (00:24.120)
learning in the cloud, on the phone, in the browser,
在云中、在手机上、在浏览器中学习，
Lex Fridman (00:26.800)
on both generic and specialized hardware.
在通用和专用硬件上。
Rajat Monga (00:29.840)
TPU, GPU, and so on.
TPU、GPU等。
Lex Fridman (00:31.960)
Plus, there's a big emphasis on growing a passionate community
另外，非常重视发展一个充满热情的社区
Rajat Monga (00:35.220)
of developers.
的开发商。
Lex Fridman (00:36.640)
Rajat, Jeff Dean, and a large team of engineers at Google
Rajat、Jeff Dean 和 Google 的庞大工程师团队
Rajat Monga (00:39.820)
Brain are working to define the future of machine
大脑正在努力定义机器的未来
Lex Fridman (00:42.200)
learning with TensorFlow 2.0, which is now in alpha.
使用 TensorFlow 2.0 进行学习，该版本目前处于 alpha 版本。
Rajat Monga (00:46.240)
I think the decision to open source TensorFlow
我认为开源 TensorFlow 的决定
Lex Fridman (00:49.160)
is a definitive moment in the tech industry.
这是科技行业的决定性时刻。
Rajat Monga (00:51.760)
It showed that open innovation can be successful
它表明开放式创新可以成功
Lex Fridman (00:54.400)
and inspire many companies to open source their code,
Rajat Monga (00:56.920)
to publish, and in general engage
Lex Fridman (00:58.880)
in the open exchange of ideas.
Rajat Monga (01:01.240)
This conversation is part of the Artificial Intelligence
Lex Fridman (01:03.940)
podcast.
Rajat Monga (01:05.080)
If you enjoy it, subscribe on YouTube, iTunes,
Lex Fridman (01:07.860)
or simply connect with me on Twitter at Lex Friedman,
Rajat Monga (01:10.880)
spelled F R I D.
Lex Fridman (01:12.720)
And now, here's my conversation with Rajat Manga.
Rajat Monga (01:17.960)
You were involved with Google Brain since its start in 2011
Lex Fridman (01:22.520)
with Jeff Dean.
Rajat Monga (01:24.880)
It started with this belief, the proprietary machine learning
Lex Fridman (01:29.220)
library, and turned into TensorFlow in 2014,
Rajat Monga (01:32.800)
the open source library.
Lex Fridman (01:35.720)
So what were the early days of Google Brain like?
Lex Fridman (01:39.120)
What were the goals, the missions?
Lex Fridman (01:41.840)
How do you even proceed forward once there's
Lex Fridman (01:45.120)
so much possibilities before you?
Lex Fridman (01:47.760)
It was interesting back then when I started,
Rajat Monga (01:50.560)
or when you were even just talking about it,
Lex Fridman (01:55.400)
the idea of deep learning was interesting and intriguing
Rajat Monga (01:59.520)
in some ways.
Lex Fridman (02:00.480)
It hadn't yet taken off, but it held some promise.
Rajat Monga (02:04.920)
It had shown some very promising and early results.
Lex Fridman (02:08.740)
I think the idea where Andrew and Jeff had started
Rajat Monga (02:11.400)
was, what if we can take this work people are doing
Lex Fridman (02:15.440)
in research and scale it to what Google has
Rajat Monga (02:18.800)
in terms of the compute power, and also
Lex Fridman (02:23.000)
put that kind of data together?
Lex Fridman (02:24.320)
What does it mean?
Lex Fridman (02:25.320)
And so far, the results had been, if you scale the compute,
Rajat Monga (02:28.300)
scale the data, it does better.
Lex Fridman (02:30.200)
And would that work?
Lex Fridman (02:31.520)
And so that was the first year or two, can we prove that out?
Lex Fridman (02:35.140)
And with this belief, when we started the first year,
Rajat Monga (02:37.480)
we got some early wins, which is always great.
Lex Fridman (02:40.800)
What were the wins like?
Lex Fridman (02:41.960)
What was the wins where you were,
Lex Fridman (02:44.160)
there's some problems to this, this is going to be good?
Rajat Monga (02:46.640)
I think there are two early wins where one was speech,
Lex Fridman (02:49.680)
that we collaborated very closely with the speech research
Rajat Monga (02:52.280)
team, who was also getting interested in this.
Lex Fridman (02:54.820)
And the other one was on images, where the cat paper,
Rajat Monga (02:58.800)
as we call it, that was covered by a lot of folks.
Lex Fridman (03:03.160)
And the birth of Google Brain was around neural networks.
Lex Fridman (03:07.440)
So it was deep learning from the very beginning.
Lex Fridman (03:09.320)
That was the whole mission.
Lex Fridman (03:10.800)
So what would, in terms of scale,
Lex Fridman (03:15.040)
what was the sort of dream of what this could become?
Rajat Monga (03:21.080)
Were there echoes of this open source TensorFlow community
Lex Fridman (03:24.280)
that might be brought in?
Lex Fridman (03:26.240)
Was there a sense of TPUs?
Lex Fridman (03:28.640)
Was there a sense of machine learning is now going to be
Rajat Monga (03:31.760)
at the core of the entire company,
Lex Fridman (03:33.720)
is going to grow into that direction?
Rajat Monga (03:36.040)
Yeah, I think, so that was interesting.
Lex Fridman (03:38.320)
And if I think back to 2012 or 2011,
Lex Fridman (03:41.380)
and first was can we scale it in the year or so,
Lex Fridman (03:45.240)
we had started scaling it to hundreds and thousands
Rajat Monga (03:47.520)
of machines.
Lex Fridman (03:48.360)
In fact, we had some runs even going to 10,000 machines.
Lex Fridman (03:51.080)
And all of those shows great promise.
Lex Fridman (03:53.880)
In terms of machine learning at Google,
Rajat Monga (03:56.800)
the good thing was Google's been doing machine learning
Lex Fridman (03:58.780)
for a long time.
Rajat Monga (04:00.240)
Deep learning was new, but as we scaled this up,
Lex Fridman (04:03.760)
we showed that, yes, that was possible.
Lex Fridman (04:05.600)
And it was going to impact lots of things.
Lex Fridman (04:07.840)
Like we started seeing real products wanting to use this.
Rajat Monga (04:11.200)
Again, speech was the first, there were image things
Lex Fridman (04:13.800)
that photos came out of and then many other products as well.
Lex Fridman (04:17.400)
So that was exciting.
Lex Fridman (04:20.180)
As we went into that a couple of years,
Rajat Monga (04:23.160)
externally also academia started to,
Lex Fridman (04:25.800)
there was lots of push on, okay,
Rajat Monga (04:27.200)
deep learning is interesting,
Lex Fridman (04:28.320)
we should be doing more and so on.
Lex Fridman (04:30.600)
And so by 2014, we were looking at, okay,
Lex Fridman (04:34.580)
this is a big thing, it's going to grow.
Lex Fridman (04:36.780)
And not just internally, externally as well.
Lex Fridman (04:39.440)
Yes, maybe Google's ahead of where everybody is,
Lex Fridman (04:42.280)
but there's a lot to do.
Lex Fridman (04:43.640)
So a lot of this started to make sense and come together.
Lex Fridman (04:46.720)
So the decision to open source,
Lex Fridman (04:49.560)
I was just chatting with Chris Glatner about this.
Rajat Monga (04:52.200)
The decision to go open source with TensorFlow,
Lex Fridman (04:54.640)
I would say sort of for me personally,
Rajat Monga (04:57.080)
seems to be one of the big seminal moments
Lex Fridman (04:59.640)
in all of software engineering ever.
Rajat Monga (05:01.720)
I think that's when a large company like Google
Lex Fridman (05:04.620)
decides to take a large project that many lawyers
Rajat Monga (05:07.520)
might argue has a lot of IP,
Lex Fridman (05:10.800)
just decide to go open source with it,
Lex Fridman (05:12.900)
and in so doing lead the entire world
Lex Fridman (05:14.880)
and saying, you know what, open innovation
Rajat Monga (05:16.520)
is a pretty powerful thing, and it's okay to do.
Lex Fridman (05:22.360)
That was, I mean, that's an incredible moment in time.
Lex Fridman (05:26.320)
So do you remember those discussions happening?
Lex Fridman (05:29.320)
Whether open source should be happening?
Lex Fridman (05:31.400)
What was that like?
Lex Fridman (05:32.680)
I would say, I think, so the initial idea came from Jeff,
Rajat Monga (05:36.880)
who was a big proponent of this.
Lex Fridman (05:39.440)
I think it came off of two big things.
Rajat Monga (05:42.480)
One was research wise, we were a research group.
Lex Fridman (05:46.320)
We were putting all our research out there.
Rajat Monga (05:49.640)
If you wanted to, we were building on others research
Lex Fridman (05:51.720)
and we wanted to push the state of the art forward.
Lex Fridman (05:55.000)
And part of that was to share the research.
Lex Fridman (05:56.840)
That's how I think deep learning and machine learning
Rajat Monga (05:58.960)
has really grown so fast.
Lex Fridman (06:01.380)
So the next step was, okay, now,
Lex Fridman (06:03.360)
would software help with that?
Lex Fridman (06:05.360)
And it seemed like they were existing
Rajat Monga (06:08.440)
a few libraries out there, Tiano being one,
Lex Fridman (06:11.280)
Torch being another, and a few others,
Lex Fridman (06:14.000)
but they were all done by academia
Lex Fridman (06:15.480)
and so the level was significantly different.
Rajat Monga (06:18.960)
The other one was from a software perspective,
Lex Fridman (06:22.000)
Google had done lots of software
Rajat Monga (06:23.880)
or that we used internally, you know,
Lex Fridman (06:27.080)
and we published papers.
Rajat Monga (06:29.080)
Often there was an open source project
Lex Fridman (06:31.680)
that came out of that that somebody else
Rajat Monga (06:33.600)
picked up that paper and implemented
Lex Fridman (06:35.400)
and they were very successful.
Rajat Monga (06:38.240)
Back then it was like, okay, there's Hadoop,
Lex Fridman (06:41.440)
which has come off of tech that we've built.
Rajat Monga (06:44.140)
We know the tech we've built is way better
Lex Fridman (06:46.200)
for a number of different reasons.
Rajat Monga (06:47.880)
We've invested a lot of effort in that.
Lex Fridman (06:51.660)
And turns out we have Google Cloud
Lex Fridman (06:54.320)
and we are now not really providing our tech,
Lex Fridman (06:57.520)
but we are saying, okay, we have Bigtable,
Rajat Monga (07:00.360)
which is the original thing.
Lex Fridman (07:02.040)
We are going to now provide H base APIs
Rajat Monga (07:03.880)
on top of that, which isn't as good,
Lex Fridman (07:06.040)
but that's what everybody's used to.
Lex Fridman (07:07.480)
So there's like, can we make something
Lex Fridman (07:10.040)
that is better and really just provide,
Rajat Monga (07:12.320)
helps the community in lots of ways,
Lex Fridman (07:14.320)
but also helps push a good standard forward.
Lex Fridman (07:18.320)
So how does Cloud fit into that?
Lex Fridman (07:19.940)
There's a TensorFlow open source library
Lex Fridman (07:22.680)
and how does the fact that you can
Lex Fridman (07:25.800)
use so many of the resources that Google provides
Lex Fridman (07:28.240)
and the Cloud fit into that strategy?
Lex Fridman (07:31.100)
So TensorFlow itself is open
Lex Fridman (07:33.600)
and you can use it anywhere, right?
Lex Fridman (07:34.920)
And we want to make sure that continues to be the case.
Rajat Monga (07:38.360)
On Google Cloud, we do make sure
Lex Fridman (07:41.040)
that there's lots of integrations with everything else
Lex Fridman (07:43.840)
and we want to make sure
Lex Fridman (07:44.880)
that it works really, really well there.
Rajat Monga (07:47.320)
You're leading the TensorFlow effort.
Lex Fridman (07:50.400)
Can you tell me the history
Lex Fridman (07:51.280)
and the timeline of TensorFlow project
Lex Fridman (07:53.600)
in terms of major design decisions,
Lex Fridman (07:55.880)
so like the open source decision,
Lex Fridman (07:58.160)
but really what to include and not?
Rajat Monga (08:01.600)
There's this incredible ecosystem
Lex Fridman (08:03.200)
that I'd like to talk about.
Rajat Monga (08:04.760)
There's all these parts,
Lex Fridman (08:05.720)
but what if just some sample moments
Rajat Monga (08:11.240)
that defined what TensorFlow eventually became
Lex Fridman (08:15.040)
through its, I don't know if you're allowed to say history
Rajat Monga (08:17.640)
when it's just, but in deep learning,
Lex Fridman (08:20.240)
everything moves so fast
Lex Fridman (08:21.280)
and just a few years is already history.
Lex Fridman (08:23.460)
Yes, yes, so looking back, we were building TensorFlow.
Rajat Monga (08:29.780)
I guess we open sourced it in 2015, November 2015.
Lex Fridman (08:34.240)
We started on it in summer of 2014, I guess.
Lex Fridman (08:39.780)
And somewhere like three to six, late 2014,
Lex Fridman (08:42.960)
by then we had decided that, okay,
Rajat Monga (08:45.120)
there's a high likelihood we'll open source it.
Lex Fridman (08:47.080)
So we started thinking about that
Lex Fridman (08:48.880)
and making sure we're heading down that path.
Lex Fridman (08:53.960)
At that point, by that point,
Rajat Monga (08:56.080)
we had seen a few, lots of different use cases at Google.
Lex Fridman (08:59.320)
So there were things like, okay,
Rajat Monga (09:01.000)
yes, you wanna run it at large scale in the data center.
Lex Fridman (09:04.200)
Yes, we need to support different kind of hardware.
Rajat Monga (09:07.560)
We had GPUs at that point.
Lex Fridman (09:09.440)
We had our first GPU at that point
Rajat Monga (09:11.880)
or was about to come out roughly around that time.
Lex Fridman (09:15.700)
So the design sort of included those.
Rajat Monga (09:18.700)
We had started to push on mobile.
Lex Fridman (09:21.800)
So we were running models on mobile.
Rajat Monga (09:24.920)
At that point, people were customizing code.
Lex Fridman (09:28.160)
So we wanted to make sure TensorFlow
Rajat Monga (09:29.560)
could support that as well.
Lex Fridman (09:30.700)
So that sort of became part of that overall design.
Rajat Monga (09:35.260)
When you say mobile,
Lex Fridman (09:36.560)
you mean like a pretty complicated algorithms
Lex Fridman (09:38.680)
running on the phone?
Lex Fridman (09:40.040)
That's correct.
Lex Fridman (09:40.880)
So when you have a model that you deploy on the phone
Lex Fridman (09:44.320)
and run it there, right?
Lex Fridman (09:45.160)
So already at that time,
Lex Fridman (09:46.420)
there was ideas of running machine learning on the phone.
Rajat Monga (09:48.800)
That's correct.
Lex Fridman (09:49.640)
We already had a couple of products
Rajat Monga (09:51.400)
that were doing that by then.
Lex Fridman (09:53.260)
And in those cases,
Rajat Monga (09:54.500)
we had basically customized handcrafted code
Lex Fridman (09:57.540)
or some internal libraries that we're using.
Lex Fridman (10:00.160)
So I was actually at Google during this time
Lex Fridman (10:02.600)
in a parallel, I guess, universe,
Lex Fridman (10:04.560)
but we were using Theano and Caffe.
Lex Fridman (10:09.240)
Was there some degree to which you were bouncing,
Rajat Monga (10:11.600)
like trying to see what Caffe was offering people,
Lex Fridman (10:15.520)
trying to see what Theano was offering
Rajat Monga (10:17.960)
that you want to make sure you're delivering
Lex Fridman (10:19.960)
on whatever that is?
Rajat Monga (10:21.640)
Perhaps the Python part of thing,
Lex Fridman (10:23.720)
maybe did that influence any design decisions?
Rajat Monga (10:27.520)
Totally.
Lex Fridman (10:28.360)
So when we built this belief
Lex Fridman (10:29.600)
and some of that was in parallel
Lex Fridman (10:31.600)
with some of these libraries coming up,
Rajat Monga (10:33.400)
I mean, Theano itself is older,
Lex Fridman (10:36.680)
but we were building this belief
Rajat Monga (10:39.880)
focused on our internal thing
Lex Fridman (10:41.160)
because our systems were very different.
Rajat Monga (10:42.960)
By the time we got to this,
Lex Fridman (10:44.080)
we looked at a number of libraries that were out there.
Rajat Monga (10:47.120)
Theano, there were folks in the group
Lex Fridman (10:49.280)
who had experience with Torch, with Lua.
Rajat Monga (10:52.140)
There were folks here who had seen Caffe.
Lex Fridman (10:54.800)
I mean, actually, Yang Jing was here as well.
Lex Fridman (10:58.840)
There's what other libraries?
Lex Fridman (11:02.980)
I think we looked at a number of things.
Rajat Monga (11:04.920)
Might even have looked at JNR back then.
Lex Fridman (11:06.840)
I'm trying to remember if it was there.
Rajat Monga (11:09.400)
In fact, yeah, we did discuss ideas around,
Lex Fridman (11:12.040)
okay, should we have a graph or not?
Lex Fridman (11:17.840)
So putting all these together was definitely,
Lex Fridman (11:20.480)
they were key decisions that we wanted.
Rajat Monga (11:22.800)
We had seen limitations in our prior disbelief things.
Lex Fridman (11:28.800)
A few of them were just in terms of research
Rajat Monga (11:31.360)
was moving so fast, we wanted the flexibility.
Lex Fridman (11:35.040)
The hardware was changing fast.
Rajat Monga (11:36.360)
We expected to change that
Lex Fridman (11:37.760)
so that those probably were two things.
Lex Fridman (11:39.900)
And yeah, I think the flexibility
Lex Fridman (11:43.140)
in terms of being able to express
Rajat Monga (11:44.380)
all kinds of crazy things was definitely a big one then.
Lex Fridman (11:46.980)
So what, the graph decisions though,
Rajat Monga (11:49.020)
with moving towards TensorFlow 2.0,
Lex Fridman (11:52.460)
there's more, by default, there'll be eager execution.
Lex Fridman (11:56.800)
So sort of hiding the graph a little bit
Lex Fridman (11:59.260)
because it's less intuitive
Rajat Monga (12:00.660)
in terms of the way people develop and so on.
Lex Fridman (12:03.660)
What was that discussion like in terms of using graphs?
Rajat Monga (12:06.800)
It seemed, it's kind of the Theano way.
Lex Fridman (12:09.420)
Did it seem the obvious choice?
Lex Fridman (12:11.660)
So I think where it came from was our disbelief
Lex Fridman (12:15.780)
had a graph like thing as well.
Rajat Monga (12:17.700)
A much more simple, it wasn't a general graph,
Lex Fridman (12:19.780)
it was more like a straight line thing.
Rajat Monga (12:23.220)
More like what you might think of cafe,
Lex Fridman (12:25.060)
I guess in that sense.
Lex Fridman (12:26.440)
But the graph was,
Lex Fridman (12:28.900)
and we always cared about the production stuff.
Rajat Monga (12:31.180)
Like even with disbelief,
Lex Fridman (12:32.020)
we were deploying a whole bunch of stuff in production.
Lex Fridman (12:34.500)
So graph did come from that when we thought of,
Lex Fridman (12:37.460)
okay, should we do that in Python?
Lex Fridman (12:39.420)
And we experimented with some ideas
Lex Fridman (12:40.900)
where it looked a lot simpler to use,
Lex Fridman (12:44.740)
but not having a graph meant,
Lex Fridman (12:46.780)
okay, how do you deploy now?
Lex Fridman (12:47.980)
So that was probably what tilted the balance for us
Lex Fridman (12:51.180)
and eventually we ended up with a graph.
Lex Fridman (12:52.940)
And I guess the question there is, did you,
Lex Fridman (12:55.400)
I mean, so production seems to be
Rajat Monga (12:57.420)
the really good thing to focus on,
Lex Fridman (12:59.900)
but did you even anticipate the other side of it
Lex Fridman (13:02.500)
where there could be, what is it?
Lex Fridman (13:04.620)
What are the numbers?
Rajat Monga (13:05.460)
It's been crazy, 41 million downloads.
Lex Fridman (13:08.980)
Yep.
Rajat Monga (13:12.780)
I mean, was that even like a possibility in your mind
Lex Fridman (13:16.300)
that it would be as popular as it became?
Lex Fridman (13:19.220)
So I think we did see a need for this
Lex Fridman (13:24.480)
a lot from the research perspective
Lex Fridman (13:27.600)
and like early days of deep learning in some ways.
Lex Fridman (13:32.340)
41 million, no, I don't think I imagined this number.
Rajat Monga (13:35.140)
Then it seemed like there's a potential future
Lex Fridman (13:41.700)
where lots more people would be doing this
Lex Fridman (13:43.780)
and how do we enable that?
Lex Fridman (13:45.700)
I would say this kind of growth,
Rajat Monga (13:49.100)
I probably started seeing somewhat after the open sourcing
Lex Fridman (13:52.660)
where it was like, okay,
Rajat Monga (13:55.300)
deep learning is actually growing way faster
Lex Fridman (13:57.880)
for a lot of different reasons.
Lex Fridman (13:59.240)
And we are in just the right place to push on that
Lex Fridman (14:02.740)
and leverage that and deliver on lots of things
Rajat Monga (14:06.100)
that people want.
Lex Fridman (14:07.500)
So what changed once you open sourced?
Rajat Monga (14:09.780)
Like how this incredible amount of attention
Lex Fridman (14:13.380)
from a global population of developers,
Lex Fridman (14:16.540)
how did the project start changing?
Lex Fridman (14:18.260)
I don't even actually remember during those times.
Rajat Monga (14:22.220)
I know looking now, there's really good documentation,
Lex Fridman (14:24.620)
there's an ecosystem of tools,
Rajat Monga (14:26.620)
there's a community, there's a blog,
Lex Fridman (14:27.980)
there's a YouTube channel now, right?
Rajat Monga (14:29.820)
Yeah.
Lex Fridman (14:31.180)
It's very community driven.
Rajat Monga (14:33.860)
Back then, I guess 0.1 version,
Lex Fridman (14:38.700)
is that the version?
Rajat Monga (14:39.860)
I think we call it 0.6 or five,
Lex Fridman (14:42.180)
something like that, I forget.
Lex Fridman (14:43.740)
What changed leading into 1.0?
Lex Fridman (14:47.180)
It's interesting.
Rajat Monga (14:48.500)
I think we've gone through a few things there.
Lex Fridman (14:51.660)
When we started out, when we first came out,
Rajat Monga (14:53.720)
people loved the documentation we have
Lex Fridman (14:56.100)
because it was just a huge step up from everything else
Rajat Monga (14:58.860)
because all of those were academic projects,
Lex Fridman (15:00.440)
people doing, who don't think about documentation.
Rajat Monga (15:04.580)
I think what that changed was,
Lex Fridman (15:06.960)
instead of deep learning being a research thing,
Rajat Monga (15:10.380)
some people who were just developers
Lex Fridman (15:12.580)
could now suddenly take this out
Lex Fridman (15:14.660)
and do some interesting things with it, right?
Lex Fridman (15:16.940)
Who had no clue what machine learning was before then.
Lex Fridman (15:20.300)
And that I think really changed
Lex Fridman (15:22.580)
how things started to scale up in some ways
Lex Fridman (15:24.760)
and pushed on it.
Lex Fridman (15:27.900)
Over the next few months as we looked at
Lex Fridman (15:30.420)
how do we stabilize things,
Lex Fridman (15:31.980)
as we look at not just researchers,
Rajat Monga (15:33.900)
now we want stability, people want to deploy things.
Lex Fridman (15:36.520)
That's how we started planning for 1.0
Lex Fridman (15:38.980)
and there are certain needs for that perspective.
Lex Fridman (15:42.180)
And so again, documentation comes up,
Rajat Monga (15:45.380)
designs, more kinds of things to put that together.
Lex Fridman (15:49.380)
And so that was exciting to get that to a stage
Rajat Monga (15:52.240)
where more and more enterprises wanted to buy in
Lex Fridman (15:55.420)
and really get behind that.
Lex Fridman (15:57.740)
And I think post 1.0 and over the next few releases,
Lex Fridman (16:01.800)
that enterprise adoption also started to take off.
Rajat Monga (16:04.400)
I would say between the initial release and 1.0,
Lex Fridman (16:07.160)
it was, okay, researchers of course,
Rajat Monga (16:10.240)
then a lot of hobbies and early interest,
Lex Fridman (16:12.960)
people excited about this who started to get on board
Lex Fridman (16:15.160)
and then over the 1.x thing, lots of enterprises.
Lex Fridman (16:18.200)
I imagine anything that's below 1.0
Rajat Monga (16:23.200)
gives pressure to be,
Lex Fridman (16:25.160)
the enterprise probably wants something that's stable.
Rajat Monga (16:28.040)
Exactly.
Lex Fridman (16:28.880)
And do you have a sense now that TensorFlow is stable?
Rajat Monga (16:33.320)
Like it feels like deep learning in general
Lex Fridman (16:35.560)
is extremely dynamic field, so much is changing.
Lex Fridman (16:40.440)
And TensorFlow has been growing incredibly.
Lex Fridman (16:43.420)
Do you have a sense of stability at the helm of it?
Rajat Monga (16:46.760)
I mean, I know you're in the midst of it, but.
Lex Fridman (16:48.400)
Yeah, I think in the midst of it,
Rajat Monga (16:51.680)
it's often easy to forget what an enterprise wants
Lex Fridman (16:55.120)
and what some of the people on that side want.
Rajat Monga (16:58.800)
There are still people running models
Lex Fridman (17:00.420)
that are three years old, four years old.
Lex Fridman (17:02.680)
So Inception is still used by tons of people.
Lex Fridman (17:06.040)
Even ResNet 50 is what, couple of years old now or more,
Lex Fridman (17:08.960)
but there are tons of people who use that and they're fine.
Lex Fridman (17:12.240)
They don't need the last couple of bits of performance
Rajat Monga (17:15.320)
or quality, they want some stability
Lex Fridman (17:17.720)
in things that just work.
Lex Fridman (17:19.640)
And so there is value in providing that
Lex Fridman (17:22.240)
with that kind of stability and making it really simpler
Rajat Monga (17:25.200)
because that allows a lot more people to access it.
Lex Fridman (17:27.800)
And then there's the research crowd which wants,
Rajat Monga (17:31.200)
okay, they wanna do these crazy things
Lex Fridman (17:33.080)
exactly like you're saying, right?
Rajat Monga (17:34.280)
Not just deep learning in the straight up models
Lex Fridman (17:37.080)
that used to be there, they want RNNs
Lex Fridman (17:40.640)
and even RNNs are maybe old, they are transformers now.
Lex Fridman (17:43.480)
And now it needs to combine with RL and GANs and so on.
Lex Fridman (17:48.440)
So there's definitely that area that like the boundary
Lex Fridman (17:52.000)
that's shifting and pushing the state of the art.
Lex Fridman (17:55.200)
But I think there's more and more of the past
Lex Fridman (17:57.200)
that's much more stable and even stuff
Rajat Monga (18:01.440)
that was two, three years old is very, very usable
Lex Fridman (18:03.880)
by lots of people.
Lex Fridman (18:04.960)
So that part makes it a lot easier.
Lex Fridman (18:07.440)
So I imagine, maybe you can correct me if I'm wrong,
Rajat Monga (18:09.840)
one of the biggest use cases is essentially
Lex Fridman (18:12.440)
taking something like ResNet 50
Lex Fridman (18:14.440)
and doing some kind of transfer learning
Lex Fridman (18:17.280)
on a very particular problem that you have.
Rajat Monga (18:19.600)
It's basically probably what majority of the world does.
Lex Fridman (18:24.520)
And you wanna make that as easy as possible.
Lex Fridman (18:27.360)
So I would say for the hobbyist perspective,
Lex Fridman (18:30.440)
that's the most common case, right?
Rajat Monga (18:32.800)
In fact, the apps and phones and stuff that you'll see,
Lex Fridman (18:35.400)
the early ones, that's the most common case.
Rajat Monga (18:37.720)
I would say there are a couple of reasons for that.
Lex Fridman (18:40.360)
One is that everybody talks about that.
Rajat Monga (18:44.440)
It looks great on slides.
Lex Fridman (18:46.160)
That's a presentation, yeah, exactly.
Lex Fridman (18:49.960)
What enterprises want is that is part of it,
Lex Fridman (18:53.080)
but that's not the big thing.
Rajat Monga (18:54.360)
Enterprises really have data
Lex Fridman (18:56.080)
that they wanna make predictions on.
Rajat Monga (18:58.000)
This is often what they used to do
Lex Fridman (19:00.320)
with the people who were doing ML
Rajat Monga (19:01.760)
was just regression models,
Lex Fridman (19:03.560)
linear regression, logistic regression, linear models,
Rajat Monga (19:06.360)
or maybe gradient booster trees and so on.
Lex Fridman (19:09.760)
Some of them still benefit from deep learning,
Lex Fridman (19:11.680)
but they want that's the bread and butter,
Lex Fridman (19:14.400)
or like the structured data and so on.
Lex Fridman (19:16.360)
So depending on the audience you look at,
Lex Fridman (19:18.280)
they're a little bit different.
Lex Fridman (19:19.600)
And they just have, I mean, the best of enterprise
Lex Fridman (19:23.440)
probably just has a very large data set,
Rajat Monga (19:26.520)
or deep learning can probably shine.
Lex Fridman (19:28.720)
That's correct, that's right.
Lex Fridman (19:30.320)
And then I think the other pieces that they wanted,
Lex Fridman (19:33.320)
again, with 2.0, the developer summit we put together
Rajat Monga (19:36.480)
is the whole TensorFlow Extended piece,
Lex Fridman (19:39.080)
which is the entire pipeline.
Rajat Monga (19:40.680)
They care about stability across doing their entire thing.
Lex Fridman (19:43.640)
They want simplicity across the entire thing.
Rajat Monga (19:46.320)
I don't need to just train a model.
Lex Fridman (19:47.760)
I need to do that every day again, over and over again.
Rajat Monga (19:51.360)
I wonder to which degree you have a role in,
Lex Fridman (19:54.360)
I don't know, so I teach a course on deep learning.
Rajat Monga (19:56.720)
I have people like lawyers come up to me and say,
Lex Fridman (1:00:00.400)
Or is there still a balance to where it's less deadline?
Rajat Monga (1:00:04.940)
You had the Dev Summit today
Lex Fridman (1:00:06.740)
that came together incredibly.
Rajat Monga (1:00:08.940)
Looked like there's a lot of moving pieces and so on.
Lex Fridman (1:00:11.460)
So did that deadline make people rise to the occasion
Lex Fridman (1:00:15.140)
releasing TensorFlow 2.0 alpha?
Lex Fridman (1:00:18.420)
I'm sure that was done last minute as well.
Rajat Monga (1:00:20.420)
I mean, up to the last point.
Lex Fridman (1:00:25.620)
Again, it's one of those things
Rajat Monga (1:00:26.860)
that you need to strike the good balance.
Lex Fridman (1:00:29.940)
There's some value that deadlines bring
Rajat Monga (1:00:32.100)
that does bring a sense of urgency
Lex Fridman (1:00:33.980)
to get the right things together.
Rajat Monga (1:00:35.780)
Instead of getting the perfect thing out,
Lex Fridman (1:00:38.340)
you need something that's good and works well.
Lex Fridman (1:00:41.320)
And the team definitely did a great job
Lex Fridman (1:00:43.260)
in putting that together.
Lex Fridman (1:00:44.100)
So I was very amazed and excited
Lex Fridman (1:00:45.920)
by everything how that came together.
Rajat Monga (1:00:48.740)
That said, across the year,
Lex Fridman (1:00:49.860)
we try not to put out official deadlines.
Rajat Monga (1:00:52.580)
We focus on key things that are important,
Lex Fridman (1:00:57.020)
figure out how much of it's important.
Lex Fridman (1:01:00.620)
And we are developing in the open,
Lex Fridman (1:01:03.900)
both internally and externally,
Rajat Monga (1:01:05.820)
everything's available to everybody.
Lex Fridman (1:01:07.980)
So you can pick and look at where things are.
Rajat Monga (1:01:11.220)
We do releases at a regular cadence.
Lex Fridman (1:01:13.260)
So fine, if something doesn't necessarily end up
Rajat Monga (1:01:16.180)
this month, it'll end up in the next release
Lex Fridman (1:01:17.820)
in a month or two.
Lex Fridman (1:01:18.780)
And that's okay, but we want to keep moving
Lex Fridman (1:01:22.860)
as fast as we can in these different areas.
Rajat Monga (1:01:26.500)
Because we can iterate and improve on things,
Lex Fridman (1:01:29.660)
sometimes it's okay to put things out
Rajat Monga (1:01:31.960)
that aren't fully ready.
Lex Fridman (1:01:32.980)
We'll make sure it's clear that okay,
Rajat Monga (1:01:34.580)
this is experimental, but it's out there
Lex Fridman (1:01:36.540)
if you want to try and give feedback.
Rajat Monga (1:01:37.980)
That's very, very useful.
Lex Fridman (1:01:39.420)
I think that quick cycle and quick iteration is important.
Rajat Monga (1:01:43.580)
That's what we often focus on rather than
Lex Fridman (1:01:46.940)
here's a deadline where you get everything else.
Lex Fridman (1:01:49.220)
Is 2.0, is there pressure to make that stable?
Lex Fridman (1:01:52.860)
Or like, for example, WordPress 5.0 just came out
Lex Fridman (1:01:57.780)
and there was no pressure to,
Lex Fridman (1:02:00.300)
it was a lot of build updates delivered way too late,
Rajat Monga (1:02:03.980)
but, and they said, okay, well,
Lex Fridman (1:02:05.980)
but we're gonna release a lot of updates
Rajat Monga (1:02:07.440)
really quickly to improve it.
Lex Fridman (1:02:09.660)
Do you see TensorFlow 2.0 in that same kind of way
Rajat Monga (1:02:12.220)
or is there this pressure to once it hits 2.0,
Lex Fridman (1:02:15.260)
once you get to the release candidate
Lex Fridman (1:02:16.780)
and then you get to the final,
Lex Fridman (1:02:18.980)
that's gonna be the stable thing?
Lex Fridman (1:02:22.460)
So it's gonna be stable in,
Lex Fridman (1:02:25.740)
just like when NodeX was where every API that's there
Rajat Monga (1:02:28.900)
is gonna remain in work.
Lex Fridman (1:02:32.100)
It doesn't mean we can't change things under the covers.
Rajat Monga (1:02:34.820)
It doesn't mean we can't add things.
Lex Fridman (1:02:36.740)
So there's still a lot more for us to do
Lex Fridman (1:02:39.200)
and we'll continue to have more releases.
Lex Fridman (1:02:41.100)
So in that sense, there's still,
Rajat Monga (1:02:42.640)
I don't think we'll be done in like two months
Lex Fridman (1:02:44.740)
when we release this.
Rajat Monga (1:02:46.140)
I don't know if you can say, but is there,
Lex Fridman (1:02:49.900)
there's not external deadlines for TensorFlow 2.0,
Lex Fridman (1:02:53.740)
but is there internal deadlines,
Lex Fridman (1:02:57.060)
the artificial or otherwise,
Rajat Monga (1:02:58.540)
that you're trying to set for yourself
Lex Fridman (1:03:00.860)
or is it whenever it's ready?
Lex Fridman (1:03:03.100)
So we want it to be a great product, right?
Lex Fridman (1:03:05.660)
And that's a big important piece for us.
Rajat Monga (1:03:09.900)
TensorFlow's already out there.
Lex Fridman (1:03:11.140)
We have 41 million downloads for 1.0 X.
Lex Fridman (1:03:13.740)
So it's not like we have to have this.
Lex Fridman (1:03:16.420)
Yeah, exactly.
Lex Fridman (1:03:17.260)
So it's not like, a lot of the features
Lex Fridman (1:03:19.340)
that we've really polishing
Lex Fridman (1:03:21.180)
and putting them together are there.
Lex Fridman (1:03:23.580)
We don't have to rush that just because.
Lex Fridman (1:03:26.220)
So in that sense, we wanna get it right
Lex Fridman (1:03:28.020)
and really focus on that.
Rajat Monga (1:03:29.940)
That said, we have said that we are looking
Lex Fridman (1:03:31.860)
to get this out in the next few months,
Rajat Monga (1:03:33.500)
in the next quarter.
Lex Fridman (1:03:34.500)
And as far as possible,
Rajat Monga (1:03:37.100)
we'll definitely try to make that happen.
Lex Fridman (1:03:39.780)
Yeah, my favorite line was, spring is a relative concept.
Rajat Monga (1:03:44.340)
I love it.
Lex Fridman (1:03:45.180)
Yes.
Rajat Monga (1:03:46.020)
Spoken like a true developer.
Lex Fridman (1:03:47.700)
So something I'm really interested in
Lex Fridman (1:03:50.220)
and your previous line of work is,
Lex Fridman (1:03:52.980)
before TensorFlow, you led a team at Google on search ads.
Rajat Monga (1:03:57.740)
I think this is a very interesting topic
Lex Fridman (1:04:01.860)
on every level, on a technical level,
Rajat Monga (1:04:04.980)
because at their best, ads connect people
Lex Fridman (1:04:07.220)
to the things they want and need.
Rajat Monga (1:04:09.420)
So, and at their worst, they're just these things
Lex Fridman (1:04:12.300)
that annoy the heck out of you
Rajat Monga (1:04:14.940)
to the point of ruining the entire user experience
Lex Fridman (1:04:17.340)
of whatever you're actually doing.
Lex Fridman (1:04:20.260)
So they have a bad rep, I guess.
Lex Fridman (1:04:23.620)
And on the other end, so that this connecting users
Rajat Monga (1:04:28.100)
to the thing they need and want
Lex Fridman (1:04:29.660)
is a beautiful opportunity for machine learning to shine.
Rajat Monga (1:04:34.060)
Like huge amounts of data that's personalized
Lex Fridman (1:04:36.340)
and you kind of map to the thing
Rajat Monga (1:04:37.860)
they actually want won't get annoyed.
Lex Fridman (1:04:40.380)
So what have you learned from this,
Rajat Monga (1:04:43.220)
Google that's leading the world in this aspect,
Lex Fridman (1:04:45.140)
what have you learned from that experience
Lex Fridman (1:04:47.540)
and what do you think is the future of ads?
Lex Fridman (1:04:51.540)
Take you back to that.
Rajat Monga (1:04:52.540)
Yeah, yes, it's been a while,
Lex Fridman (1:04:55.220)
but I totally agree with what you said.
Rajat Monga (1:04:59.700)
I think the search ads, the way it was always looked at
Lex Fridman (1:05:03.180)
and I believe it still is,
Rajat Monga (1:05:04.500)
is it's an extension of what search is trying to do.
Lex Fridman (1:05:08.100)
And the goal is to make the information
Lex Fridman (1:05:10.580)
and make the world's information accessible.
Lex Fridman (1:05:14.740)
That's it's not just information,
Lex Fridman (1:05:17.140)
but maybe products or other things that people care about.
Lex Fridman (1:05:20.780)
And so it's really important for them to align
Rajat Monga (1:05:23.860)
with what the users need.
Lex Fridman (1:05:26.500)
And in search ads, there's a minimum quality level
Rajat Monga (1:05:30.940)
before that ad would be shown.
Lex Fridman (1:05:32.300)
If you don't have an ad that hits that quality,
Lex Fridman (1:05:34.060)
but it will not be shown even if we have it
Lex Fridman (1:05:35.980)
and okay, maybe we lose some money there, that's fine.
Rajat Monga (1:05:39.620)
That is really, really important.
Lex Fridman (1:05:41.300)
And I think that that is something I really liked
Rajat Monga (1:05:43.420)
about being there.
Lex Fridman (1:05:45.060)
Advertising is a key part.
Lex Fridman (1:05:48.180)
I mean, as a model, it's been around for ages, right?
Lex Fridman (1:05:51.740)
It's not a new model, it's been adapted to the web
Lex Fridman (1:05:54.900)
and became a core part of search
Lex Fridman (1:05:57.500)
and many other search engines across the world.
Lex Fridman (1:06:00.780)
And I do hope, like you said,
Lex Fridman (1:06:04.420)
there are aspects of ads that are annoying
Lex Fridman (1:06:06.700)
and I go to a website and if it just keeps popping
Lex Fridman (1:06:10.260)
an ad in my face not to let me read,
Rajat Monga (1:06:12.540)
that's gonna be annoying clearly.
Lex Fridman (1:06:13.860)
So I hope we can strike that balance
Rajat Monga (1:06:18.780)
between showing a good ad where it's valuable to the user
Lex Fridman (1:06:23.780)
and provides the monetization to the service.
Lex Fridman (1:06:29.740)
And this might be search, this might be a website,
Lex Fridman (1:06:32.460)
all of these, they do need the monetization
Rajat Monga (1:06:35.660)
for them to provide that service.
Lex Fridman (1:06:38.540)
But if it's done in a good balance between
Rajat Monga (1:06:43.660)
showing just some random stuff that's distracting
Lex Fridman (1:06:46.820)
versus showing something that's actually valuable.
Lex Fridman (1:06:49.660)
So do you see it moving forward as to continue
Lex Fridman (1:06:54.660)
being a model that funds businesses like Google,
Lex Fridman (1:07:00.340)
that's a significant revenue stream?
Lex Fridman (1:07:04.380)
Because that's one of the most exciting things
Lex Fridman (1:07:07.420)
but also limiting things in the internet
Lex Fridman (1:07:09.020)
is nobody wants to pay for anything.
Lex Fridman (1:07:11.500)
And advertisements, again, coupled at their best,
Lex Fridman (1:07:14.660)
are actually really useful and not annoying.
Lex Fridman (1:07:16.660)
Do you see that continuing and growing and improving
Lex Fridman (1:07:21.660)
or is there, do you see sort of more Netflix type models
Lex Fridman (1:07:26.140)
where you have to start to pay for content?
Lex Fridman (1:07:28.420)
I think it's a mix.
Rajat Monga (1:07:29.780)
I think it's gonna take a long while for everything
Lex Fridman (1:07:32.260)
to be paid on the internet, if at all, probably not.
Rajat Monga (1:07:35.580)
I mean, I think there's always gonna be things
Lex Fridman (1:07:37.220)
that are sort of monetized with things like ads.
Lex Fridman (1:07:40.180)
But over the last few years, I would say
Lex Fridman (1:07:42.220)
we've definitely seen that transition towards
Rajat Monga (1:07:45.340)
more paid services across the web
Lex Fridman (1:07:48.660)
and people are willing to pay for them
Rajat Monga (1:07:50.420)
because they do see the value.
Lex Fridman (1:07:51.740)
I mean, Netflix is a great example.
Rajat Monga (1:07:53.660)
I mean, we have YouTube doing things.
Lex Fridman (1:07:56.580)
People pay for the apps they buy.
Rajat Monga (1:07:58.780)
More people I find are willing to pay for newspaper content
Lex Fridman (1:08:03.140)
for the good news websites across the web.
Rajat Monga (1:08:07.260)
That wasn't the case a few years,
Lex Fridman (1:08:08.900)
even a few years ago, I would say.
Lex Fridman (1:08:11.060)
And I just see that change in myself as well
Lex Fridman (1:08:13.340)
and just lots of people around me.
Lex Fridman (1:08:14.860)
So definitely hopeful that we'll transition
Lex Fridman (1:08:17.220)
to that mix model where maybe you get
Rajat Monga (1:08:20.900)
to try something out for free, maybe with ads,
Lex Fridman (1:08:24.180)
but then there's a more clear revenue model
Rajat Monga (1:08:27.420)
that sort of helps go beyond that.
Lex Fridman (1:08:30.660)
So speaking of revenue, how is it that a person
Lex Fridman (1:08:35.940)
can use the TPU in a Google call app for free?
Lex Fridman (1:08:39.460)
So what's the, I guess the question is,
Rajat Monga (1:08:43.980)
what's the future of TensorFlow in terms of empowering,
Lex Fridman (1:08:48.940)
say, a class of 300 students?
Lex Fridman (1:08:51.940)
And I'm asked by MIT, what is going to be the future
Lex Fridman (1:08:56.940)
of them being able to do their homework in TensorFlow?
Lex Fridman (1:09:00.020)
Like, where are they going to train these networks, right?
Lex Fridman (1:09:02.860)
What's that future look like with TPUs,
Lex Fridman (1:09:06.460)
with cloud services, and so on?
Lex Fridman (1:09:08.980)
I think a number of things there.
Rajat Monga (1:09:10.300)
I mean, any TensorFlow open source,
Lex Fridman (1:09:12.660)
you can run it wherever, you can run it on your desktop
Lex Fridman (1:09:15.020)
and your desktops always keep getting more powerful,
Lex Fridman (1:09:17.500)
so maybe you can do more.
Rajat Monga (1:09:19.540)
My phone is like, I don't know how many times
Lex Fridman (1:09:21.420)
more powerful than my first desktop.
Rajat Monga (1:09:23.740)
You'll probably train it on your phone though,
Lex Fridman (1:09:25.220)
yeah, that's true.
Rajat Monga (1:09:26.260)
Right, so in that sense, the power you have
Lex Fridman (1:09:28.460)
in your hands is a lot more.
Rajat Monga (1:09:31.500)
Clouds are actually very interesting from, say,
Lex Fridman (1:09:34.420)
students or courses perspective,
Rajat Monga (1:09:36.940)
because they make it very easy to get started.
Lex Fridman (1:09:40.060)
I mean, Colab, the great thing about it is,
Rajat Monga (1:09:42.740)
go to a website and it just works.
Lex Fridman (1:09:45.180)
No installation needed, nothing to,
Rajat Monga (1:09:47.580)
you're just there and things are working.
Lex Fridman (1:09:50.020)
That's really the power of cloud as well.
Lex Fridman (1:09:52.300)
And so I do expect that to grow.
Lex Fridman (1:09:55.340)
Again, Colab is a free service.
Rajat Monga (1:09:57.940)
It's great to get started, to play with things,
Lex Fridman (1:10:00.900)
to explore things.
Rajat Monga (1:10:03.140)
That said, with free, you can only get so much.
Lex Fridman (1:10:06.140)
You'd be, yeah.
Lex Fridman (1:10:08.220)
So just like we were talking about,
Lex Fridman (1:10:10.140)
free versus paid, yeah, there are services
Rajat Monga (1:10:12.940)
you can pay for and get a lot more.
Lex Fridman (1:10:15.340)
Great, so if I'm a complete beginner
Rajat Monga (1:10:17.740)
interested in machine learning and TensorFlow,
Lex Fridman (1:10:19.980)
what should I do?
Rajat Monga (1:10:21.620)
Probably start with going to our website
Lex Fridman (1:10:23.540)
and playing there.
Lex Fridman (1:10:24.380)
So just go to TensorFlow.org and start clicking on things.
Lex Fridman (1:10:26.620)
Yep, check out tutorials and guides.
Rajat Monga (1:10:28.500)
There's stuff you can just click there
Lex Fridman (1:10:29.860)
and go to a Colab and do things.
Rajat Monga (1:10:31.340)
No installation needed, you can get started right there.
Lex Fridman (1:10:34.100)
Okay, awesome, Rajit, thank you so much for talking today.
Rajat Monga (1:10:36.740)
Thank you, Lex, it was great.
Lex Fridman (20:01.400)
when is machine learning gonna enter legal,
Lex Fridman (20:04.240)
the legal realm?
Lex Fridman (20:05.640)
The same thing in all kinds of disciplines,
Rajat Monga (20:09.520)
immigration, insurance, often when I see
Lex Fridman (20:14.720)
what it boils down to is these companies
Rajat Monga (20:17.440)
are often a little bit old school
Lex Fridman (20:19.480)
in the way they organize the data.
Lex Fridman (20:20.880)
So the data is just not ready yet, it's not digitized.
Lex Fridman (20:24.040)
Do you also find yourself being in the role
Rajat Monga (20:26.000)
of an evangelist for like, let's get,
Lex Fridman (20:31.520)
organize your data, folks, and then you'll get
Rajat Monga (20:33.760)
the big benefit of TensorFlow.
Lex Fridman (20:35.480)
Do you get those, have those conversations?
Rajat Monga (20:38.040)
Yeah, yeah, you know, I get all kinds of questions there
Lex Fridman (20:41.480)
from, okay, what do I need to make this work, right?
Lex Fridman (20:49.080)
Do we really need deep learning?
Lex Fridman (20:50.840)
I mean, there are all these things,
Lex Fridman (20:52.120)
I already use this linear model, why would this help?
Lex Fridman (20:55.200)
I don't have enough data, let's say,
Rajat Monga (20:57.200)
or I wanna use machine learning,
Lex Fridman (21:00.000)
but I have no clue where to start.
Lex Fridman (21:01.800)
So it varies, that to all the way to the experts
Lex Fridman (21:04.960)
to why support very specific things, it's interesting.
Lex Fridman (21:08.600)
Is there a good answer?
Lex Fridman (21:09.920)
It boils down to oftentimes digitizing data.
Lex Fridman (21:12.520)
So whatever you want automated,
Lex Fridman (21:14.480)
whatever data you want to make prediction based on,
Rajat Monga (21:17.560)
you have to make sure that it's in an organized form.
Lex Fridman (21:21.280)
Like within the TensorFlow ecosystem,
Rajat Monga (21:24.000)
there's now, you're providing more and more data sets
Lex Fridman (21:26.560)
and more and more pre trained models.
Lex Fridman (21:28.960)
Are you finding yourself also the organizer of data sets?
Lex Fridman (21:32.440)
Yes, I think the TensorFlow data sets
Rajat Monga (21:34.520)
that we just released, that's definitely come up
Lex Fridman (21:37.560)
where people want these data sets,
Lex Fridman (21:39.240)
can we organize them and can we make that easier?
Lex Fridman (21:41.760)
So that's definitely one important thing.
Rajat Monga (21:45.320)
The other related thing I would say is I often tell people,
Lex Fridman (21:47.680)
you know what, don't think of the most fanciest thing
Rajat Monga (21:51.000)
that the newest model that you see,
Lex Fridman (21:53.320)
make something very basic work and then you can improve it.
Rajat Monga (21:56.400)
There's just lots of things you can do with it.
Lex Fridman (21:58.920)
Yeah, start with the basics, true.
Rajat Monga (22:00.640)
One of the big things that makes TensorFlow
Lex Fridman (22:03.280)
even more accessible was the appearance
Rajat Monga (22:06.120)
whenever that happened of Keras,
Lex Fridman (22:08.360)
the Keras standard sort of outside of TensorFlow.
Rajat Monga (22:12.400)
I think it was Keras on top of Tiano at first only
Lex Fridman (22:18.240)
and then Keras became on top of TensorFlow.
Lex Fridman (22:22.520)
Do you know when Keras chose to also add TensorFlow
Lex Fridman (22:28.760)
as a backend, who was the,
Lex Fridman (22:31.200)
was it just the community that drove that initially?
Lex Fridman (22:34.000)
Do you know if there was discussions, conversations?
Rajat Monga (22:37.040)
Yeah, so Francois started the Keras project
Lex Fridman (22:41.000)
before he was at Google and the first thing was Tiano.
Rajat Monga (22:44.600)
I don't remember if that was
Lex Fridman (22:46.560)
after TensorFlow was created or way before.
Lex Fridman (22:49.680)
And then at some point,
Lex Fridman (22:51.440)
when TensorFlow started becoming popular,
Rajat Monga (22:53.040)
there were enough similarities
Lex Fridman (22:54.200)
that he decided to create this interface
Lex Fridman (22:56.360)
and put TensorFlow as a backend.
Lex Fridman (22:58.200)
I believe that might still have been
Rajat Monga (23:00.760)
before he joined Google.
Lex Fridman (23:03.320)
So we weren't really talking about that.
Rajat Monga (23:06.720)
He decided on his own and thought that was interesting
Lex Fridman (23:09.720)
and relevant to the community.
Rajat Monga (23:12.800)
In fact, I didn't find out about him being at Google
Lex Fridman (23:17.120)
until a few months after he was here.
Rajat Monga (23:19.680)
He was working on some research ideas
Lex Fridman (23:21.880)
and doing Keras on his nights and weekends project.
Rajat Monga (23:24.480)
Oh, interesting.
Lex Fridman (23:25.320)
He wasn't like part of the TensorFlow.
Rajat Monga (23:28.520)
He didn't join initially.
Lex Fridman (23:29.720)
He joined research and he was doing some amazing research.
Rajat Monga (23:32.280)
He has some papers on that and research,
Lex Fridman (23:34.360)
so he's a great researcher as well.
Lex Fridman (23:38.400)
And at some point we realized,
Lex Fridman (23:40.400)
oh, he's doing this good stuff.
Rajat Monga (23:42.440)
People seem to like the API and he's right here.
Lex Fridman (23:45.400)
So we talked to him and he said,
Rajat Monga (23:47.760)
okay, why don't I come over to your team
Lex Fridman (23:50.600)
and work with you for a quarter
Lex Fridman (23:52.840)
and let's make that integration happen.
Lex Fridman (23:55.520)
And we talked to his manager and he said,
Rajat Monga (23:56.840)
sure, quarter's fine.
Lex Fridman (23:59.800)
And that quarter's been something like two years now.
Lex Fridman (24:02.400)
And so he's fully on this.
Lex Fridman (24:05.080)
So Keras got integrated into TensorFlow in a deep way.
Lex Fridman (24:12.000)
And now with 2.0, TensorFlow 2.0,
Lex Fridman (24:15.240)
sort of Keras is kind of the recommended way
Rajat Monga (24:18.800)
for a beginner to interact with TensorFlow.
Lex Fridman (24:21.720)
Which makes that initial sort of transfer learning
Rajat Monga (24:24.640)
or the basic use cases, even for an enterprise,
Lex Fridman (24:28.040)
super simple, right?
Rajat Monga (24:29.320)
That's correct, that's right.
Lex Fridman (24:30.440)
So what was that decision like?
Rajat Monga (24:32.040)
That seems like it's kind of a bold decision as well.
Lex Fridman (24:38.680)
We did spend a lot of time thinking about that one.
Rajat Monga (24:41.240)
We had a bunch of APIs, some built by us.
Lex Fridman (24:46.000)
There was a parallel layers API that we were building.
Lex Fridman (24:48.760)
And when we decided to do Keras in parallel,
Lex Fridman (24:51.560)
so there were like, okay, two things that we are looking at.
Lex Fridman (24:54.400)
And the first thing we was trying to do
Lex Fridman (24:55.960)
is just have them look similar,
Rajat Monga (24:58.240)
like be as integrated as possible,
Lex Fridman (25:00.120)
share all of that stuff.
Rajat Monga (25:02.200)
There were also like three other APIs
Lex Fridman (25:04.000)
that others had built over time
Rajat Monga (25:05.840)
because we didn't have a standard one.
Lex Fridman (25:09.040)
But one of the messages that we kept hearing
Lex Fridman (25:11.480)
from the community, okay, which one do we use?
Lex Fridman (25:13.240)
And they kept seeing like, okay,
Rajat Monga (25:14.480)
here's a model in this one and here's a model in this one,
Lex Fridman (25:16.760)
which should I pick?
Lex Fridman (25:18.880)
So that's sort of like, okay,
Lex Fridman (25:20.960)
we had to address that straight on with 2.0.
Rajat Monga (25:24.080)
The whole idea was we need to simplify.
Lex Fridman (25:26.360)
We had to pick one.
Rajat Monga (25:28.640)
Based on where we were, we were like,
Lex Fridman (25:30.520)
okay, let's see what are the people like?
Lex Fridman (25:35.680)
And Keras was clearly one that lots of people loved.
Lex Fridman (25:39.320)
There were lots of great things about it.
Lex Fridman (25:41.640)
So we settled on that.
Lex Fridman (25:43.920)
Organically, that's kind of the best way to do it.
Rajat Monga (25:46.440)
It was great.
Lex Fridman (25:47.520)
It was surprising, nevertheless,
Rajat Monga (25:48.760)
to sort of bring in an outside.
Lex Fridman (25:51.120)
I mean, there was a feeling like Keras
Rajat Monga (25:52.560)
might be almost like a competitor
Lex Fridman (25:55.440)
in a certain kind of, to TensorFlow.
Lex Fridman (25:58.040)
And in a sense, it became an empowering element
Lex Fridman (26:01.320)
of TensorFlow.
Rajat Monga (26:02.240)
That's right.
Lex Fridman (26:03.280)
Yeah, it's interesting how you can put two things together,
Rajat Monga (26:06.440)
which can align.
Lex Fridman (26:08.800)
In this case, I think Francois, the team,
Lex Fridman (26:11.800)
and a bunch of us have chatted,
Lex Fridman (26:14.280)
and I think we all want to see the same kind of things.
Rajat Monga (26:17.360)
We all care about making it easier
Lex Fridman (26:18.800)
for the huge set of developers out there,
Lex Fridman (26:21.440)
and that makes a difference.
Lex Fridman (26:23.480)
So Python has Guido van Rossum,
Rajat Monga (26:26.880)
who until recently held the position
Lex Fridman (26:28.920)
of benevolent dictator for life.
Rajat Monga (26:31.920)
All right, so there's a huge successful open source project
Lex Fridman (26:36.480)
like TensorFlow need one person who makes a final decision.
Lex Fridman (26:40.680)
So you've did a pretty successful TensorFlow Dev Summit
Lex Fridman (26:45.480)
just now, last couple of days.
Rajat Monga (26:47.520)
There's clearly a lot of different new features
Lex Fridman (26:51.080)
being incorporated, an amazing ecosystem, so on.
Lex Fridman (26:54.160)
Who's, how are those design decisions made?
Lex Fridman (26:57.320)
Is there a BDFL in TensorFlow,
Lex Fridman (27:02.800)
or is it more distributed and organic?
Lex Fridman (27:05.800)
I think it's somewhat different, I would say.
Rajat Monga (27:08.760)
I've always been involved in the key design directions,
Lex Fridman (27:14.560)
but there are lots of things that are distributed
Rajat Monga (27:17.080)
where there are a number of people, Martin Wick being one,
Lex Fridman (27:20.560)
who has really driven a lot of our open source stuff,
Rajat Monga (27:23.880)
a lot of the APIs,
Lex Fridman (27:26.080)
and there are a number of other people who've been,
Rajat Monga (27:29.080)
you know, pushed and been responsible
Lex Fridman (27:31.360)
for different parts of it.
Rajat Monga (27:34.080)
We do have regular design reviews.
Lex Fridman (27:36.480)
Over the last year, we've had a lot of
Rajat Monga (27:38.480)
we've really spent a lot of time opening up to the community
Lex Fridman (27:41.480)
and adding transparency.
Rajat Monga (27:44.160)
We're setting more processes in place,
Lex Fridman (27:45.880)
so RFCs, special interest groups,
Rajat Monga (27:49.080)
to really grow that community and scale that.
Lex Fridman (27:53.600)
I think the kind of scale that ecosystem is in,
Rajat Monga (27:57.720)
I don't think we could scale with having me
Lex Fridman (27:59.520)
as the lone point of decision maker.
Rajat Monga (28:02.280)
I got it. So, yeah, the growth of that ecosystem,
Lex Fridman (28:05.920)
maybe you can talk about it a little bit.
Rajat Monga (28:08.040)
First of all, it started with Andrej Karpathy
Lex Fridman (28:10.720)
when he first did ComNetJS.
Rajat Monga (28:13.120)
The fact that you can train and you'll network
Lex Fridman (28:15.360)
in the browser was, in JavaScript, was incredible.
Lex Fridman (28:18.480)
So now TensorFlow.js is really making that
Lex Fridman (28:22.160)
a serious, like a legit thing,
Rajat Monga (28:26.400)
a way to operate, whether it's in the backend
Lex Fridman (28:28.520)
or the front end.
Rajat Monga (28:29.520)
Then there's the TensorFlow Extended, like you mentioned.
Lex Fridman (28:32.680)
There's TensorFlow Lite for mobile.
Lex Fridman (28:35.320)
And all of it, as far as I can tell,
Lex Fridman (28:37.440)
it's really converging towards being able to
Rajat Monga (28:41.680)
save models in the same kind of way.
Lex Fridman (28:43.440)
You can move around, you can train on the desktop
Lex Fridman (28:46.680)
and then move it to mobile and so on.
Lex Fridman (28:48.880)
That's right.
Lex Fridman (28:49.720)
So there's that cohesiveness.
Lex Fridman (28:52.280)
So can you maybe give me, whatever I missed,
Rajat Monga (28:56.120)
a bigger overview of the mission of the ecosystem
Lex Fridman (28:58.840)
that's trying to be built and where is it moving forward?
Rajat Monga (29:02.080)
Yeah. So in short, the way I like to think of this is
Lex Fridman (29:06.720)
our goals to enable machine learning.
Lex Fridman (29:09.680)
And in a couple of ways, you know, one is
Lex Fridman (29:13.120)
we have lots of exciting things going on in ML today.
Rajat Monga (29:16.520)
We started with deep learning,
Lex Fridman (29:17.520)
but we now support a bunch of other algorithms too.
Lex Fridman (29:21.360)
So one is to, on the research side,
Lex Fridman (29:23.760)
keep pushing on the state of the art.
Rajat Monga (29:25.280)
Can we, you know, how do we enable researchers
Lex Fridman (29:27.200)
to build the next amazing thing?
Lex Fridman (29:28.920)
So BERT came out recently, you know,
Lex Fridman (29:31.720)
it's great that people are able to do new kinds of research.
Lex Fridman (29:33.920)
And there are lots of amazing research
Lex Fridman (29:35.360)
that happens across the world.
Lex Fridman (29:37.480)
So that's one direction.
Lex Fridman (29:38.800)
The other is how do you take that across
Rajat Monga (29:42.440)
all the people outside who want to take that research
Lex Fridman (29:45.200)
and do some great things with it
Lex Fridman (29:46.600)
and integrate it to build real products,
Lex Fridman (29:48.600)
to have a real impact on people.
Lex Fridman (29:51.720)
And so if that's the other axes in some ways,
Lex Fridman (29:56.320)
you know, at a high level, one way I think about it is
Rajat Monga (29:59.600)
there are a crazy number of compute devices
Lex Fridman (30:02.440)
across the world.
Lex Fridman (30:04.160)
And we often used to think of ML and training
Lex Fridman (30:07.840)
and all of this as, okay, something you do
Rajat Monga (30:09.400)
either in the workstation or the data center or cloud.
Lex Fridman (30:13.560)
But we see things running on the phones.
Rajat Monga (30:15.640)
We see things running on really tiny chips.
Lex Fridman (30:17.600)
I mean, we had some demos at the developer summit.
Lex Fridman (30:20.680)
And so the way I think about this ecosystem is
Lex Fridman (30:25.760)
how do we help get machine learning on every device
Lex Fridman (30:29.880)
that has a compute capability?
Lex Fridman (30:32.480)
And that continues to grow and so in some ways
Rajat Monga (30:36.440)
this ecosystem is looked at, you know,
Lex Fridman (30:38.680)
various aspects of that and grown over time
Rajat Monga (30:41.120)
to cover more of those.
Lex Fridman (30:42.440)
And we continue to push the boundaries.
Rajat Monga (30:44.640)
In some areas we've built more tooling
Lex Fridman (30:48.160)
and things around that to help you.
Rajat Monga (30:50.000)
I mean, the first tool we started was TensorBoard.
Lex Fridman (30:52.760)
You wanted to learn just the training piece,
Rajat Monga (30:56.240)
the effects or TensorFlow extended
Lex Fridman (30:58.080)
to really do your entire ML pipelines.
Rajat Monga (31:00.400)
If you're, you know, care about all that production stuff,
Lex Fridman (31:04.760)
but then going to the edge,
Rajat Monga (31:06.600)
going to different kinds of things.
Lex Fridman (31:09.480)
And it's not just us now.
Rajat Monga (31:11.760)
We are a place where there are lots of libraries
Lex Fridman (31:14.440)
being built on top.
Lex Fridman (31:15.800)
So there are some for research,
Lex Fridman (31:17.760)
maybe things like TensorFlow agents
Rajat Monga (31:20.040)
or TensorFlow probability that started as research things
Lex Fridman (31:22.440)
or for researchers for focusing
Rajat Monga (31:24.200)
on certain kinds of algorithms,
Lex Fridman (31:26.120)
but they're also being deployed
Rajat Monga (31:27.280)
or used by, you know, production folks.
Lex Fridman (31:30.240)
And some have come from within Google,
Rajat Monga (31:33.320)
just teams across Google
Lex Fridman (31:34.720)
who wanted to build these things.
Rajat Monga (31:37.000)
Others have come from just the community
Lex Fridman (31:39.680)
because there are different pieces
Rajat Monga (31:41.840)
that different parts of the community care about.
Lex Fridman (31:44.600)
And I see our goal as enabling even that, right?
Rajat Monga (31:49.480)
It's not, we cannot and won't build every single thing.
Lex Fridman (31:53.240)
That just doesn't make sense.
Lex Fridman (31:54.840)
But if we can enable others to build the things
Lex Fridman (31:57.360)
that they care about, and there's a broader community
Rajat Monga (32:00.400)
that cares about that, and we can help encourage that,
Lex Fridman (32:02.880)
and that's great.
Rajat Monga (32:05.280)
That really helps the entire ecosystem, not just those.
Lex Fridman (32:08.600)
One of the big things about 2.0 that we're pushing on is,
Lex Fridman (32:11.840)
okay, we have these so many different pieces, right?
Lex Fridman (32:14.640)
How do we help make all of them work well together?
Lex Fridman (32:18.320)
So there are a few key pieces there that we're pushing on,
Lex Fridman (32:21.960)
one being the core format in there
Lex Fridman (32:23.880)
and how we share the models themselves
Lex Fridman (32:26.600)
through save model and TensorFlow hub and so on.
Lex Fridman (32:30.480)
And a few of the pieces that we really put this together.
Lex Fridman (32:34.000)
I was very skeptical that that's,
Rajat Monga (32:35.600)
you know, when TensorFlow.js came out,
Lex Fridman (32:37.280)
it didn't seem, or deep learning JS as it was earlier.
Rajat Monga (32:40.160)
Yeah, that was the first.
Lex Fridman (32:41.680)
It seems like technically very difficult project.
Rajat Monga (32:45.080)
As a standalone, it's not as difficult,
Lex Fridman (32:47.000)
but as a thing that integrates into the ecosystem,
Rajat Monga (32:49.960)
it seems very difficult.
Lex Fridman (32:51.240)
So, I mean, there's a lot of aspects of this
Rajat Monga (32:53.240)
you're making look easy, but,
Lex Fridman (32:54.840)
and the technical side,
Lex Fridman (32:57.160)
how many challenges have to be overcome here?
Lex Fridman (33:00.520)
A lot.
Lex Fridman (33:01.480)
And still have to be overcome.
Lex Fridman (33:03.040)
That's the question here too.
Lex Fridman (33:04.680)
There are lots of steps to it, right?
Lex Fridman (33:06.320)
And we've iterated over the last few years,
Lex Fridman (33:07.960)
so there's a lot we've learned.
Lex Fridman (33:10.680)
I, yeah, and often when things come together well,
Rajat Monga (33:14.200)
things look easy and that's exactly the point.
Lex Fridman (33:16.360)
It should be easy for the end user,
Lex Fridman (33:18.320)
but there are lots of things that go behind that.
Lex Fridman (33:21.320)
If I think about still challenges ahead,
Rajat Monga (33:25.320)
there are,
Lex Fridman (33:29.400)
you know, we have a lot more devices coming on board,
Rajat Monga (33:32.880)
for example, from the hardware perspective.
Lex Fridman (33:35.280)
How do we make it really easy for these vendors
Lex Fridman (33:37.600)
to integrate with something like TensorFlow, right?
Lex Fridman (33:42.040)
So there's a lot of compiler stuff
Rajat Monga (33:43.600)
that others are working on.
Lex Fridman (33:45.280)
There are things we can do in terms of our APIs
Lex Fridman (33:48.280)
and so on that we can do.
Lex Fridman (33:50.440)
As we, you know,
Rajat Monga (33:52.960)
TensorFlow started as a very monolithic system
Lex Fridman (33:55.760)
and to some extent it still is.
Rajat Monga (33:57.600)
There are less, lots of tools around it,
Lex Fridman (33:59.360)
but the core is still pretty large and monolithic.
Rajat Monga (34:02.880)
One of the key challenges for us to scale that out
Lex Fridman (34:05.680)
is how do we break that apart with clearer interfaces?
Rajat Monga (34:10.320)
It's, you know, in some ways it's software engineering 101,
Lex Fridman (34:14.520)
but for a system that's now four years old, I guess,
Rajat Monga (34:18.480)
or more, and that's still rapidly evolving
Lex Fridman (34:21.560)
and that we're not slowing down with,
Rajat Monga (34:23.960)
it's hard to change and modify and really break apart.
Lex Fridman (34:28.200)
It's sort of like, as people say, right,
Rajat Monga (34:29.880)
it's like changing the engine with a car running
Lex Fridman (34:32.560)
or trying to fix that.
Rajat Monga (34:33.600)
That's exactly what we're trying to do.
Lex Fridman (34:35.040)
So there's a challenge here
Rajat Monga (34:37.520)
because the downside of so many people
Lex Fridman (34:41.560)
being excited about TensorFlow
Lex Fridman (34:43.800)
and coming to rely on it in many of their applications
Lex Fridman (34:48.520)
is that you're kind of responsible,
Rajat Monga (34:52.000)
like it's the technical debt.
Lex Fridman (34:53.480)
You're responsible for previous versions
Rajat Monga (34:55.600)
to some degree still working.
Lex Fridman (34:57.560)
So when you're trying to innovate,
Rajat Monga (34:59.840)
I mean, it's probably easier
Lex Fridman (35:02.360)
to just start from scratch every few months.
Rajat Monga (35:04.760)
Absolutely.
Lex Fridman (35:07.160)
So do you feel the pain of that?
Rajat Monga (35:09.240)
2.0 does break some back compatibility,
Lex Fridman (35:14.320)
but not too much.
Rajat Monga (35:15.400)
It seems like the conversion is pretty straightforward.
Lex Fridman (35:18.160)
Do you think that's still important
Lex Fridman (35:20.280)
given how quickly deep learning is changing?
Lex Fridman (35:22.920)
Can you just, the things that you've learned,
Lex Fridman (35:26.440)
can you just start over or is there pressure to not?
Lex Fridman (35:29.320)
It's a tricky balance.
Lex Fridman (35:31.600)
So if it was just a researcher writing a paper
Lex Fridman (35:36.360)
who a year later will not look at that code again,
Rajat Monga (35:39.400)
sure, it doesn't matter.
Lex Fridman (35:41.600)
There are a lot of production systems
Rajat Monga (35:43.440)
that rely on TensorFlow,
Lex Fridman (35:44.680)
both at Google and across the world.
Lex Fridman (35:47.240)
And people worry about this.
Lex Fridman (35:49.760)
I mean, these systems run for a long time.
Lex Fridman (35:53.440)
So it is important to keep that compatibility and so on.
Lex Fridman (35:57.280)
And yes, it does come with a huge cost.
Rajat Monga (35:59.720)
There's, we have to think about a lot of things
Lex Fridman (36:02.960)
as we do new things and make new changes.
Lex Fridman (36:06.920)
I think it's a trade off, right?
Lex Fridman (36:09.080)
You can, you might slow certain kinds of things down,
Lex Fridman (36:12.960)
but the overall value you're bringing
Lex Fridman (36:14.560)
because of that is much bigger
Rajat Monga (36:16.920)
because it's not just about breaking the person yesterday.
Lex Fridman (36:20.520)
It's also about telling the person tomorrow
Rajat Monga (36:23.640)
that, you know what, this is how we do things.
Lex Fridman (36:26.240)
We're not gonna break you when you come on board
Rajat Monga (36:28.480)
because there are lots of new people
Lex Fridman (36:29.800)
who are also gonna come on board.
Rajat Monga (36:31.400)
And, you know, one way I like to think about this,
Lex Fridman (36:34.680)
and I always push the team to think about it as well,
Rajat Monga (36:37.960)
when you wanna do new things,
Lex Fridman (36:39.560)
you wanna start with a clean slate.
Rajat Monga (36:42.040)
Design with a clean slate in mind,
Lex Fridman (36:44.880)
and then we'll figure out
Lex Fridman (36:46.160)
how to make sure all the other things work.
Lex Fridman (36:48.640)
And yes, we do make compromises occasionally,
Lex Fridman (36:52.160)
but unless you design with the clean slate
Lex Fridman (36:55.200)
and not worry about that,
Rajat Monga (36:56.520)
you'll never get to a good place.
Lex Fridman (36:58.360)
Oh, that's brilliant, so even if you are responsible
Rajat Monga (37:02.560)
when you're in the idea stage,
Lex Fridman (37:04.080)
when you're thinking of new,
Rajat Monga (37:05.760)
just put all that behind you.
Lex Fridman (37:07.720)
Okay, that's really, really well put.
Lex Fridman (37:09.600)
So I have to ask this
Lex Fridman (37:11.080)
because a lot of students, developers ask me
Lex Fridman (37:13.240)
how I feel about PyTorch versus TensorFlow.
Lex Fridman (37:16.320)
So I've recently completely switched
Rajat Monga (37:18.280)
my research group to TensorFlow.
Lex Fridman (37:20.920)
I wish everybody would just use the same thing,
Lex Fridman (37:23.280)
and TensorFlow is as close to that, I believe, as we have.
Lex Fridman (37:26.960)
But do you enjoy competition?
Lex Fridman (37:32.040)
So TensorFlow is leading in many ways,
Lex Fridman (37:34.320)
on many dimensions in terms of ecosystem,
Rajat Monga (37:36.760)
in terms of number of users,
Lex Fridman (37:39.040)
momentum, power, production levels, so on,
Lex Fridman (37:41.200)
but a lot of researchers are now also using PyTorch.
Lex Fridman (37:46.000)
Do you enjoy that kind of competition
Rajat Monga (37:47.520)
or do you just ignore it
Lex Fridman (37:48.840)
and focus on making TensorFlow the best that it can be?
Lex Fridman (37:52.320)
So just like research or anything people are doing,
Lex Fridman (37:55.480)
it's great to get different kinds of ideas.
Lex Fridman (37:58.120)
And when we started with TensorFlow,
Lex Fridman (38:01.480)
like I was saying earlier,
Rajat Monga (38:03.280)
one, it was very important
Lex Fridman (38:05.240)
for us to also have production in mind.
Lex Fridman (38:07.440)
We didn't want just research, right?
Lex Fridman (38:09.000)
And that's why we chose certain things.
Rajat Monga (38:11.280)
Now PyTorch came along and said,
Lex Fridman (38:12.720)
you know what, I only care about research.
Rajat Monga (38:14.880)
This is what I'm trying to do.
Lex Fridman (38:16.280)
What's the best thing I can do for this?
Lex Fridman (38:18.400)
And it started iterating and said,
Lex Fridman (38:20.880)
okay, I don't need to worry about graphs.
Rajat Monga (38:22.560)
Let me just run things.
Lex Fridman (38:24.080)
And I don't care if it's not as fast as it can be,
Lex Fridman (38:27.440)
but let me just make this part easy.
Lex Fridman (38:30.480)
And there are things you can learn from that, right?
Rajat Monga (38:32.560)
They, again, had the benefit of seeing what had come before,
Lex Fridman (38:36.760)
but also exploring certain different kinds of spaces.
Lex Fridman (38:40.520)
And they had some good things there,
Lex Fridman (38:43.560)
building on say things like JNR and so on before that.
Lex Fridman (38:46.680)
So competition is definitely interesting.
Lex Fridman (38:49.320)
It made us, you know,
Rajat Monga (38:50.240)
this is an area that we had thought about,
Lex Fridman (38:51.880)
like I said, way early on.
Rajat Monga (38:53.720)
Over time we had revisited this a couple of times,
Lex Fridman (38:56.600)
should we add this again?
Rajat Monga (38:59.000)
At some point we said, you know what,
Lex Fridman (39:01.040)
it seems like this can be done well,
Lex Fridman (39:02.880)
so let's try it again.
Lex Fridman (39:04.320)
And that's how we started pushing on eager execution.
Lex Fridman (39:07.680)
How do we combine those two together?
Lex Fridman (39:09.880)
Which has finally come very well together in 2.0,
Lex Fridman (39:13.120)
but it took us a while to get all the things together
Lex Fridman (39:15.760)
and so on.
Lex Fridman (39:16.600)
So let me ask, put another way,
Lex Fridman (39:19.320)
I think eager execution is a really powerful thing
Rajat Monga (39:21.800)
that was added.
Lex Fridman (39:22.640)
Do you think it wouldn't have been,
Lex Fridman (39:25.800)
you know, Muhammad Ali versus Frasier, right?
Lex Fridman (39:28.360)
Do you think it wouldn't have been added as quickly
Lex Fridman (39:31.160)
if PyTorch wasn't there?
Lex Fridman (39:33.740)
It might have taken longer.
Lex Fridman (39:35.400)
No longer?
Lex Fridman (39:36.240)
Yeah, it was, I mean,
Rajat Monga (39:37.080)
we had tried some variants of that before,
Lex Fridman (39:38.900)
so I'm sure it would have happened,
Lex Fridman (39:40.900)
but it might have taken longer.
Lex Fridman (39:42.220)
I'm grateful that TensorFlow is finally
Rajat Monga (39:44.080)
in the way they did.
Lex Fridman (39:44.920)
It's doing some incredible work last couple years.
Lex Fridman (39:47.740)
What other things that we didn't talk about
Lex Fridman (39:49.600)
are you looking forward in 2.0?
Rajat Monga (39:51.480)
That comes to mind.
Lex Fridman (39:54.040)
So we talked about some of the ecosystem stuff,
Rajat Monga (39:56.520)
making it easily accessible to Keras,
Lex Fridman (40:00.000)
eager execution.
Lex Fridman (40:01.440)
Is there other things that we missed?
Lex Fridman (40:03.000)
Yeah, so I would say one is just where 2.0 is,
Lex Fridman (40:07.500)
and you know, with all the things that we've talked about,
Lex Fridman (40:10.740)
I think as we think beyond that,
Rajat Monga (40:13.760)
there are lots of other things that it enables us to do
Lex Fridman (40:16.600)
and that we're excited about.
Lex Fridman (40:18.760)
So what it's setting us up for,
Lex Fridman (40:20.720)
okay, here are these really clean APIs.
Rajat Monga (40:22.520)
We've cleaned up the surface for what the users want.
Lex Fridman (40:25.640)
What it also allows us to do a whole bunch of stuff
Rajat Monga (40:28.320)
behind the scenes once we are ready with 2.0.
Lex Fridman (40:31.600)
So for example, in TensorFlow with graphs
Lex Fridman (40:36.740)
and all the things you could do,
Lex Fridman (40:37.720)
you could always get a lot of good performance
Lex Fridman (40:40.600)
if you spent the time to tune it, right?
Lex Fridman (40:43.280)
And we've clearly shown that, lots of people do that.
Rajat Monga (40:47.720)
With 2.0, with these APIs, where we are,
Lex Fridman (40:53.040)
we can give you a lot of performance
Rajat Monga (40:55.140)
just with whatever you do.
Lex Fridman (40:57.040)
You know, because we see these, it's much cleaner.
Rajat Monga (41:01.400)
We know most people are gonna do things this way.
Lex Fridman (41:03.740)
We can really optimize for that
Lex Fridman (41:05.520)
and get a lot of those things out of the box.
Lex Fridman (41:09.040)
And it really allows us, you know,
Rajat Monga (41:10.360)
both for single machine and distributed and so on,
Lex Fridman (41:13.880)
to really explore other spaces behind the scenes
Rajat Monga (41:17.200)
after 2.0 in the future versions as well.
Lex Fridman (41:19.720)
So right now the team's really excited about that,
Rajat Monga (41:23.040)
that over time I think we'll see that.
Lex Fridman (41:25.840)
The other piece that I was talking about
Rajat Monga (41:27.760)
in terms of just restructuring the monolithic thing
Lex Fridman (41:31.640)
into more pieces and making it more modular,
Rajat Monga (41:34.360)
I think that's gonna be really important
Lex Fridman (41:36.840)
for a lot of the other people in the ecosystem,
Rajat Monga (41:41.800)
other organizations and so on that wanted to build things.
Lex Fridman (41:44.840)
Can you elaborate a little bit what you mean
Lex Fridman (41:46.400)
by making TensorFlow ecosystem more modular?
Lex Fridman (41:50.720)
So the way it's organized today is there's one,
Rajat Monga (41:55.040)
there are lots of repositories
Lex Fridman (41:56.320)
in the TensorFlow organization at GitHub.
Rajat Monga (41:58.360)
The core one where we have TensorFlow,
Lex Fridman (42:01.120)
it has the execution engine,
Rajat Monga (42:04.120)
it has the key backends for CPUs and GPUs,
Lex Fridman (42:08.320)
it has the work to do distributed stuff.
Lex Fridman (42:12.580)
And all of these just work together
Lex Fridman (42:14.420)
in a single library or binary.
Rajat Monga (42:17.280)
There's no way to split them apart easily.
Lex Fridman (42:18.840)
I mean, there are some interfaces,
Lex Fridman (42:20.000)
but they're not very clean.
Lex Fridman (42:21.640)
In a perfect world, you would have clean interfaces where,
Rajat Monga (42:24.860)
okay, I wanna run it on my fancy cluster
Lex Fridman (42:27.760)
with some custom networking,
Rajat Monga (42:29.400)
just implement this and do that.
Lex Fridman (42:31.000)
I mean, we kind of support that,
Lex Fridman (42:32.680)
but it's hard for people today.
Lex Fridman (42:35.520)
I think as we are starting to see more interesting things
Rajat Monga (42:38.180)
in some of these spaces,
Lex Fridman (42:39.440)
having that clean separation will really start to help.
Lex Fridman (42:42.360)
And again, going to the large size of the ecosystem
Lex Fridman (42:47.360)
and the different groups involved there,
Rajat Monga (42:50.140)
enabling people to evolve
Lex Fridman (42:52.580)
and push on things more independently
Rajat Monga (42:54.360)
just allows it to scale better.
Lex Fridman (42:56.040)
And by people, you mean individual developers and?
Lex Fridman (42:59.080)
And organizations.
Lex Fridman (42:59.960)
And organizations.
Rajat Monga (43:00.960)
That's right.
Lex Fridman (43:01.800)
So the hope is that everybody sort of major,
Rajat Monga (43:04.240)
I don't know, Pepsi or something uses,
Lex Fridman (43:06.900)
like major corporations go to TensorFlow to this kind of.
Rajat Monga (43:11.040)
Yeah, if you look at enterprises like Pepsi or these,
Lex Fridman (43:13.640)
I mean, a lot of them are already using TensorFlow.
Rajat Monga (43:15.800)
They are not the ones that do the development
Lex Fridman (43:18.920)
or changes in the core.
Rajat Monga (43:20.360)
Some of them do, but a lot of them don't.
Lex Fridman (43:21.960)
I mean, they touch small pieces.
Rajat Monga (43:23.720)
There are lots of these,
Lex Fridman (43:25.660)
some of them being, let's say, hardware vendors
Rajat Monga (43:27.660)
who are building their custom hardware
Lex Fridman (43:28.960)
and they want their own pieces.
Rajat Monga (43:30.840)
Or some of them being bigger companies, say, IBM.
Lex Fridman (43:34.160)
I mean, they're involved in some of our
Rajat Monga (43:36.480)
special interest groups,
Lex Fridman (43:38.100)
and they see a lot of users
Rajat Monga (43:39.960)
who want certain things and they want to optimize for that.
Lex Fridman (43:42.620)
So folks like that often.
Rajat Monga (43:44.440)
Autonomous vehicle companies, perhaps.
Lex Fridman (43:46.360)
Exactly, yes.
Rajat Monga (43:48.160)
So, yeah, like I mentioned,
Lex Fridman (43:50.000)
TensorFlow has been downloaded 41 million times,
Rajat Monga (43:52.760)
50,000 commits, almost 10,000 pull requests,
Lex Fridman (43:56.480)
and 1,800 contributors.
Lex Fridman (43:58.320)
So I'm not sure if you can explain it,
Lex Fridman (44:02.120)
but what does it take to build a community like that?
Rajat Monga (44:06.000)
In retrospect, what do you think,
Lex Fridman (44:09.160)
what is the critical thing that allowed
Rajat Monga (44:11.180)
for this growth to happen,
Lex Fridman (44:12.640)
and how does that growth continue?
Rajat Monga (44:14.600)
Yeah, yeah, that's an interesting question.
Lex Fridman (44:17.920)
I wish I had all the answers there, I guess,
Lex Fridman (44:20.240)
so you could replicate it.
Lex Fridman (44:22.520)
I think there are a number of things
Lex Fridman (44:25.560)
that need to come together, right?
Lex Fridman (44:27.880)
One, just like any new thing,
Rajat Monga (44:32.480)
it is about, there's a sweet spot of timing,
Lex Fridman (44:35.920)
what's needed, does it grow with,
Rajat Monga (44:38.880)
what's needed, so in this case, for example,
Lex Fridman (44:41.640)
TensorFlow's not just grown because it was a good tool,
Rajat Monga (44:43.680)
it's also grown with the growth of deep learning itself.
Lex Fridman (44:46.720)
So those factors come into play.
Rajat Monga (44:49.040)
Other than that, though,
Lex Fridman (44:52.080)
I think just hearing, listening to the community,
Lex Fridman (44:55.240)
what they do, what they need,
Lex Fridman (44:57.040)
being open to, like in terms of external contributions,
Rajat Monga (45:01.120)
we've spent a lot of time in making sure
Lex Fridman (45:04.560)
we can accept those contributions well,
Rajat Monga (45:06.880)
we can help the contributors in adding those,
Lex Fridman (45:09.480)
putting the right process in place,
Rajat Monga (45:11.320)
getting the right kind of community,
Lex Fridman (45:13.360)
welcoming them and so on.
Rajat Monga (45:16.160)
Like over the last year, we've really pushed on transparency,
Lex Fridman (45:19.320)
that's important for an open source project.
Rajat Monga (45:22.280)
People wanna know where things are going,
Lex Fridman (45:23.800)
and we're like, okay, here's a process
Rajat Monga (45:26.200)
where you can do that, here are our RFCs and so on.
Lex Fridman (45:29.360)
So thinking through, there are lots of community aspects
Rajat Monga (45:32.920)
that come into that you can really work on.
Lex Fridman (45:35.460)
As a small project, it's maybe easy to do
Rajat Monga (45:38.740)
because there's like two developers and you can do those.
Lex Fridman (45:42.180)
As you grow, putting more of these processes in place,
Rajat Monga (45:46.980)
thinking about the documentation,
Lex Fridman (45:49.140)
thinking about what two developers care about,
Lex Fridman (45:51.940)
what kind of tools would they want to use,
Lex Fridman (45:55.180)
all of these come into play, I think.
Lex Fridman (45:56.900)
So one of the big things I think
Lex Fridman (45:58.420)
that feeds the TensorFlow fire
Rajat Monga (46:00.700)
is people building something on TensorFlow,
Lex Fridman (46:03.980)
and implement a particular architecture
Rajat Monga (46:07.700)
that does something cool and useful,
Lex Fridman (46:09.500)
and they put that on GitHub.
Lex Fridman (46:11.100)
And so it just feeds this growth.
Lex Fridman (46:15.580)
Do you have a sense that with 2.0 and 1.0
Rajat Monga (46:19.580)
that there may be a little bit of a partitioning
Lex Fridman (46:21.580)
like there is with Python 2 and 3,
Rajat Monga (46:24.100)
that there'll be a code base
Lex Fridman (46:26.040)
and in the older versions of TensorFlow,
Lex Fridman (46:28.340)
they will not be as compatible easily?
Lex Fridman (46:31.140)
Or are you pretty confident that this kind of conversion
Lex Fridman (46:35.620)
is pretty natural and easy to do?
Lex Fridman (46:37.980)
So we're definitely working hard
Rajat Monga (46:39.980)
to make that very easy to do.
Lex Fridman (46:41.500)
There's lots of tooling that we talked about
Rajat Monga (46:43.500)
at the developer summit this week,
Lex Fridman (46:45.820)
and we'll continue to invest in that tooling.
Rajat Monga (46:48.260)
It's, you know, when you think
Lex Fridman (46:50.500)
of these significant version changes,
Rajat Monga (46:52.580)
that's always a risk,
Lex Fridman (46:53.580)
and we are really pushing hard
Rajat Monga (46:55.740)
to make that transition very, very smooth.
Lex Fridman (46:58.100)
So I think, so at some level,
Rajat Monga (47:02.700)
people wanna move and they see the value in the new thing.
Lex Fridman (47:05.620)
They don't wanna move just because it's a new thing,
Lex Fridman (47:07.740)
and some people do,
Lex Fridman (47:08.580)
but most people want a really good thing.
Lex Fridman (47:11.540)
And I think over the next few months,
Lex Fridman (47:13.820)
as people start to see the value,
Rajat Monga (47:15.460)
we'll definitely see that shift happening.
Lex Fridman (47:17.700)
So I'm pretty excited and confident
Rajat Monga (47:19.740)
that we will see people moving.
Lex Fridman (47:22.540)
As you said earlier, this field is also moving rapidly,
Lex Fridman (47:24.740)
so that'll help because we can do more things
Lex Fridman (47:26.780)
and all the new things will clearly happen in 2.x,
Lex Fridman (47:29.500)
so people will have lots of good reasons to move.
Lex Fridman (47:32.300)
So what do you think TensorFlow 3.0 looks like?
Rajat Monga (47:36.140)
Is there, are things happening so crazily
Lex Fridman (47:40.340)
that even at the end of this year
Lex Fridman (47:42.540)
seems impossible to plan for?
Lex Fridman (47:45.300)
Or is it possible to plan for the next five years?
Rajat Monga (47:49.420)
I think it's tricky.
Lex Fridman (47:50.820)
There are some things that we can expect
Rajat Monga (47:54.540)
in terms of, okay, change, yes, change is gonna happen.
Lex Fridman (47:59.700)
Are there some things gonna stick around
Lex Fridman (48:01.660)
and some things not gonna stick around?
Lex Fridman (48:03.740)
I would say the basics of deep learning,
Rajat Monga (48:08.140)
the, you know, say convolution models
Lex Fridman (48:10.420)
or the basic kind of things,
Rajat Monga (48:12.700)
they'll probably be around in some form still in five years.
Lex Fridman (48:16.300)
Will RL and GAN stay?
Rajat Monga (48:18.620)
Very likely, based on where they are.
Lex Fridman (48:21.180)
Will we have new things?
Rajat Monga (48:22.860)
Probably, but those are hard to predict.
Lex Fridman (48:24.660)
And some directionally, some things that we can see is,
Rajat Monga (48:30.620)
you know, in things that we're starting to do, right,
Lex Fridman (48:32.740)
with some of our projects right now
Rajat Monga (48:35.420)
is just 2.0 combining eager execution and graphs
Lex Fridman (48:39.140)
where we're starting to make it more like
Rajat Monga (48:41.460)
just your natural programming language.
Lex Fridman (48:43.140)
You're not trying to program something else.
Rajat Monga (48:45.660)
Similarly, with Swift for TensorFlow,
Lex Fridman (48:47.220)
we're taking that approach.
Lex Fridman (48:48.260)
Can you do something ground up, right?
Lex Fridman (48:50.020)
So some of those ideas seem like, okay,
Rajat Monga (48:52.100)
that's the right direction.
Lex Fridman (48:54.100)
In five years, we expect to see more in that area.
Rajat Monga (48:58.340)
Other things we don't know is,
Lex Fridman (49:00.060)
will hardware accelerators be the same?
Rajat Monga (49:03.180)
Will we be able to train with four bits
Lex Fridman (49:06.620)
instead of 32 bits?
Lex Fridman (49:09.020)
And I think the TPU side of things is exploring that.
Lex Fridman (49:11.500)
I mean, TPU is already on version three.
Rajat Monga (49:13.940)
It seems that the evolution of TPU and TensorFlow
Lex Fridman (49:17.540)
are sort of, they're coevolving almost in terms of
Rajat Monga (49:23.260)
both are learning from each other and from the community
Lex Fridman (49:25.740)
and from the applications
Rajat Monga (49:27.980)
where the biggest benefit is achieved.
Lex Fridman (49:29.740)
That's right.
Rajat Monga (49:30.580)
You've been trying to sort of, with Eager, with Keras,
Lex Fridman (49:33.340)
to make TensorFlow as accessible
Lex Fridman (49:34.940)
and easy to use as possible.
Lex Fridman (49:36.500)
What do you think, for beginners,
Lex Fridman (49:38.060)
is the biggest thing they struggle with?
Lex Fridman (49:40.020)
Have you encountered that?
Rajat Monga (49:42.100)
Or is basically what Keras is solving is that Eager,
Lex Fridman (49:46.260)
like we talked about?
Rajat Monga (49:47.420)
Yeah, for some of them, like you said, right,
Lex Fridman (49:50.620)
the beginners want to just be able to take
Rajat Monga (49:53.620)
some image model,
Lex Fridman (49:54.900)
they don't care if it's Inception or ResNet
Rajat Monga (49:57.060)
or something else,
Lex Fridman (49:58.100)
and do some training or transfer learning
Rajat Monga (50:00.820)
on their kind of model.
Lex Fridman (50:02.500)
Being able to make that easy is important.
Lex Fridman (50:04.460)
So in some ways,
Lex Fridman (50:07.060)
if you do that by providing them simple models
Rajat Monga (50:09.380)
with say, in hub or so on,
Lex Fridman (50:11.420)
they don't care about what's inside that box,
Lex Fridman (50:13.780)
but they want to be able to use it.
Lex Fridman (50:15.180)
So we're pushing on, I think, different levels.
Rajat Monga (50:17.660)
If you look at just a component that you get,
Lex Fridman (50:20.020)
which has the layers already smooshed in,
Rajat Monga (50:22.820)
the beginners probably just want that.
Lex Fridman (50:25.260)
Then the next step is, okay,
Rajat Monga (50:26.780)
look at building layers with Keras.
Lex Fridman (50:29.100)
If you go out to research,
Rajat Monga (50:30.300)
then they are probably writing custom layers themselves
Lex Fridman (50:33.180)
or doing their own loops.
Lex Fridman (50:34.460)
So there's a whole spectrum there.
Lex Fridman (50:36.380)
And then providing the pre trained models
Rajat Monga (50:38.660)
seems to really decrease the time from you trying to start.
Lex Fridman (50:43.660)
You could basically in a Colab notebook
Rajat Monga (50:46.860)
achieve what you need.
Lex Fridman (50:49.140)
So I'm basically answering my own question
Rajat Monga (50:51.340)
because I think what TensorFlow delivered on recently
Lex Fridman (50:54.300)
is trivial for beginners.
Lex Fridman (50:56.980)
So I was just wondering if there was other pain points
Lex Fridman (51:00.780)
you're trying to ease,
Lex Fridman (51:01.620)
but I'm not sure there would.
Lex Fridman (51:02.540)
No, those are probably the big ones.
Rajat Monga (51:04.900)
I see high schoolers doing a whole bunch of things now,
Lex Fridman (51:07.420)
which is pretty amazing.
Rajat Monga (51:09.220)
It's both amazing and terrifying.
Lex Fridman (51:11.420)
Yes.
Rajat Monga (51:12.700)
In a sense that when they grow up,
Lex Fridman (51:15.940)
it's some incredible ideas will be coming from them.
Lex Fridman (51:19.300)
So there's certainly a technical aspect to your work,
Lex Fridman (51:21.860)
but you also have a management aspect to your role
Rajat Monga (51:25.260)
with TensorFlow leading the project,
Lex Fridman (51:27.980)
a large number of developers and people.
Lex Fridman (51:31.140)
So what do you look for in a good team?
Lex Fridman (51:34.700)
What do you think?
Rajat Monga (51:36.220)
Google has been at the forefront of exploring
Lex Fridman (51:38.420)
what it takes to build a good team
Lex Fridman (51:40.500)
and TensorFlow is one of the most cutting edge technologies
Lex Fridman (51:45.540)
in the world.
Lex Fridman (51:46.380)
So in this context, what do you think makes for a good team?
Lex Fridman (51:50.500)
It's definitely something I think a favorite about.
Rajat Monga (51:53.180)
I think in terms of the team being able
Lex Fridman (51:59.780)
to deliver something well,
Rajat Monga (52:01.180)
one of the things that's important is a cohesion
Lex Fridman (52:04.780)
across the team.
Lex Fridman (52:05.820)
So being able to execute together in doing things
Lex Fridman (52:10.420)
that's not an end, like at this scale,
Rajat Monga (52:13.180)
an individual engineer can only do so much.
Lex Fridman (52:15.460)
There's a lot more that they can do together,
Rajat Monga (52:18.260)
even though we have some amazing superstars across Google
Lex Fridman (52:21.780)
and in the team, but there's, you know,
Rajat Monga (52:25.140)
often the way I see it as the product
Lex Fridman (52:27.380)
of what the team generates is way larger
Rajat Monga (52:29.140)
than the whole or the individual put together.
Lex Fridman (52:34.460)
And so how do we have all of them work together,
Rajat Monga (52:37.380)
the culture of the team itself,
Lex Fridman (52:40.060)
hiring good people is important.
Lex Fridman (52:43.100)
But part of that is it's not just that,
Lex Fridman (52:45.340)
okay, we hire a bunch of smart people
Lex Fridman (52:47.260)
and throw them together and let them do things.
Lex Fridman (52:49.740)
It's also people have to care about what they're building,
Rajat Monga (52:52.980)
people have to be motivated for the right kind of things.
Lex Fridman (52:57.380)
That's often an important factor.
Rajat Monga (53:01.500)
And, you know, finally, how do you put that together
Lex Fridman (53:04.660)
with a somewhat unified vision of where we wanna go?
Lex Fridman (53:08.860)
So are we all looking in the same direction
Lex Fridman (53:11.220)
or each of us going all over?
Lex Fridman (53:13.620)
And sometimes it's a mix.
Lex Fridman (53:16.100)
Google's a very bottom up organization in some sense,
Rajat Monga (53:21.460)
also research even more so, and that's how we started.
Lex Fridman (53:26.420)
But as we've become this larger product and ecosystem,
Rajat Monga (53:30.900)
I think it's also important to combine that well
Lex Fridman (53:33.180)
with a mix of, okay, here's the direction we wanna go in.
Rajat Monga (53:38.020)
There is exploration we'll do around that,
Lex Fridman (53:39.860)
but let's keep staying in that direction,
Rajat Monga (53:42.820)
not just all over the place.
Lex Fridman (53:44.460)
And is there a way you monitor the health of the team?
Lex Fridman (53:46.860)
Sort of like, is there a way you know you did a good job?
Lex Fridman (53:51.980)
The team is good?
Rajat Monga (53:53.020)
Like, I mean, you're sort of, you're saying nice things,
Lex Fridman (53:56.220)
but it's sometimes difficult to determine how aligned.
Rajat Monga (54:00.860)
Yes.
Lex Fridman (54:01.700)
Because it's not binary.
Rajat Monga (54:02.520)
It's not like there's tensions and complexities and so on.
Lex Fridman (54:06.740)
And the other element of the mission of superstars,
Rajat Monga (54:09.460)
there's so much, even at Google,
Lex Fridman (54:11.820)
such a large percentage of work
Rajat Monga (54:13.220)
is done by individual superstars too.
Lex Fridman (54:16.020)
So there's a, and sometimes those superstars
Rajat Monga (54:19.980)
can be against the dynamic of a team and those tensions.
Lex Fridman (54:25.220)
I mean, I'm sure in TensorFlow it might be
Rajat Monga (54:26.580)
a little bit easier because the mission of the project
Lex Fridman (54:28.900)
is so sort of beautiful.
Rajat Monga (54:31.740)
You're at the cutting edge, so it's exciting.
Lex Fridman (54:34.860)
But have you had struggle with that?
Lex Fridman (54:36.700)
Has there been challenges?
Lex Fridman (54:38.380)
There are always people challenges
Rajat Monga (54:39.860)
in different kinds of ways.
Lex Fridman (54:41.260)
That said, I think we've been what's good
Rajat Monga (54:44.780)
about getting people who care and are, you know,
Lex Fridman (54:48.980)
have the same kind of culture,
Lex Fridman (54:50.420)
and that's Google in general to a large extent.
Lex Fridman (54:53.460)
But also, like you said, given that the project
Rajat Monga (54:56.140)
has had so many exciting things to do,
Lex Fridman (54:58.780)
there's been room for lots of people
Rajat Monga (55:00.760)
to do different kinds of things and grow,
Lex Fridman (55:02.460)
which does make the problem a bit easier, I guess.
Lex Fridman (55:05.380)
And it allows people, depending on what they're doing,
Lex Fridman (55:09.940)
if there's room around them, then that's fine.
Lex Fridman (55:13.140)
But yes, we do care about whether a superstar or not,
Lex Fridman (55:19.220)
that they need to work well with the team across Google.
Rajat Monga (55:22.580)
That's interesting to hear.
Lex Fridman (55:23.680)
So it's like superstar or not,
Rajat Monga (55:26.500)
the productivity broadly is about the team.
Lex Fridman (55:30.540)
Yeah, yeah.
Rajat Monga (55:31.540)
I mean, they might add a lot of value,
Lex Fridman (55:32.980)
but if they're hurting the team, then that's a problem.
Lex Fridman (55:35.740)
So in hiring engineers, it's so interesting, right,
Lex Fridman (55:39.060)
the hiring process.
Lex Fridman (55:40.260)
What do you look for?
Lex Fridman (55:41.860)
How do you determine a good developer
Rajat Monga (55:44.300)
or a good member of a team
Lex Fridman (55:46.240)
from just a few minutes or hours together?
Rajat Monga (55:50.420)
Again, no magic answers, I'm sure.
Lex Fridman (55:52.260)
Yeah, I mean, Google has a hiring process
Rajat Monga (55:55.340)
that we've refined over the last 20 years, I guess,
Lex Fridman (55:59.660)
and that you've probably heard and seen a lot about.
Lex Fridman (56:02.220)
So we do work with the same hiring process
Lex Fridman (56:04.980)
and that's really helped.
Rajat Monga (56:08.340)
For me in particular, I would say,
Lex Fridman (56:10.900)
in addition to the core technical skills,
Lex Fridman (56:14.220)
what does matter is their motivation
Lex Fridman (56:17.580)
in what they wanna do.
Rajat Monga (56:19.600)
Because if that doesn't align well
Lex Fridman (56:21.380)
with where we wanna go,
Rajat Monga (56:22.980)
that's not gonna lead to long term success
Lex Fridman (56:25.360)
for either them or the team.
Lex Fridman (56:27.700)
And I think that becomes more important
Lex Fridman (56:30.020)
the more senior the person is,
Lex Fridman (56:31.480)
but it's important at every level.
Lex Fridman (56:33.580)
Like even the junior most engineer,
Rajat Monga (56:34.940)
if they're not motivated to do well
Lex Fridman (56:36.380)
at what they're trying to do,
Rajat Monga (56:37.700)
however smart they are,
Lex Fridman (56:38.820)
it's gonna be hard for them to succeed.
Lex Fridman (56:40.380)
Does the Google hiring process touch on that passion?
Lex Fridman (56:44.540)
So like trying to determine,
Rajat Monga (56:46.500)
because I think as far as I understand,
Lex Fridman (56:48.500)
maybe you can speak to it,
Rajat Monga (56:49.620)
that the Google hiring process sort of helps
Lex Fridman (56:53.380)
in the initial like determines the skill set there,
Rajat Monga (56:56.380)
is your puzzle solving ability,
Lex Fridman (56:57.940)
problem solving ability good?
Lex Fridman (56:59.920)
But like, I'm not sure,
Lex Fridman (57:02.540)
but it seems that the determining
Rajat Monga (57:05.040)
whether the person is like fire inside them,
Lex Fridman (57:07.580)
that burns to do anything really,
Rajat Monga (57:09.060)
it doesn't really matter.
Lex Fridman (57:09.900)
It's just some cool stuff,
Rajat Monga (57:11.540)
I'm gonna do it.
Lex Fridman (57:15.340)
Is that something that ultimately ends up
Rajat Monga (57:17.300)
when they have a conversation with you
Lex Fridman (57:18.820)
or once it gets closer to the team?
Lex Fridman (57:22.640)
So one of the things we do have as part of the process
Lex Fridman (57:25.420)
is just a culture fit,
Rajat Monga (57:27.180)
like part of the interview process itself,
Lex Fridman (57:29.200)
in addition to just the technical skills
Lex Fridman (57:31.020)
and each engineer or whoever the interviewer is,
Lex Fridman (57:34.260)
is supposed to rate the person on the culture
Lex Fridman (57:38.340)
and the culture fit with Google and so on.
Lex Fridman (57:40.000)
So that is definitely part of the process.
Rajat Monga (57:42.180)
Now, there are various kinds of projects
Lex Fridman (57:45.860)
and different kinds of things.
Lex Fridman (57:46.940)
So there might be variants
Lex Fridman (57:48.820)
and of the kind of culture you want there and so on.
Lex Fridman (57:51.380)
And yes, that does vary.
Lex Fridman (57:52.740)
So for example,
Rajat Monga (57:54.020)
TensorFlow has always been a fast moving project
Lex Fridman (57:56.980)
and we want people who are comfortable with that.
Lex Fridman (58:00.980)
But at the same time now, for example,
Lex Fridman (58:02.700)
we are at a place where we are also very full fledged product
Lex Fridman (58:05.260)
and we wanna make sure things that work
Lex Fridman (58:07.820)
really, really work, right?
Rajat Monga (58:09.340)
You can't cut corners all the time.
Lex Fridman (58:11.700)
So balancing that out and finding the people
Rajat Monga (58:14.340)
who are the right fit for those is important.
Lex Fridman (58:17.580)
And I think those kinds of things do vary a bit
Rajat Monga (58:19.740)
across projects and teams and product areas across Google.
Lex Fridman (58:23.220)
And so you'll see some differences there
Rajat Monga (58:25.260)
in the final checklist.
Lex Fridman (58:27.700)
But a lot of the core culture,
Rajat Monga (58:29.380)
it comes along with just the engineering excellence
Lex Fridman (58:32.220)
and so on.
Lex Fridman (58:34.740)
What is the hardest part of your job?
Lex Fridman (58:39.780)
I'll take your pick, I guess.
Lex Fridman (58:41.940)
It's fun, I would say, right?
Lex Fridman (58:44.460)
Hard, yes.
Rajat Monga (58:45.540)
I mean, lots of things at different times.
Lex Fridman (58:47.280)
I think that does vary.
Lex Fridman (58:49.220)
So let me clarify that difficult things are fun
Lex Fridman (58:52.680)
when you solve them, right?
Lex Fridman (58:53.980)
So it's fun in that sense.
Lex Fridman (58:57.500)
I think the key to a successful thing across the board
Lex Fridman (59:02.640)
and in this case, it's a large ecosystem now,
Lex Fridman (59:05.380)
but even a small product,
Rajat Monga (59:07.180)
is striking that fine balance
Lex Fridman (59:09.820)
across different aspects of it.
Rajat Monga (59:12.060)
Sometimes it's how fast do you go
Lex Fridman (59:13.940)
versus how perfect it is.
Lex Fridman (59:17.060)
Sometimes it's how do you involve this huge community?
Lex Fridman (59:21.460)
Who do you involve or do you decide,
Rajat Monga (59:23.640)
okay, now is not a good time to involve them
Lex Fridman (59:25.480)
because it's not the right fit.
Rajat Monga (59:30.220)
Sometimes it's saying no to certain kinds of things.
Lex Fridman (59:33.660)
Those are often the hard decisions.
Rajat Monga (59:36.860)
Some of them you make quickly
Lex Fridman (59:39.600)
because you don't have the time.
Rajat Monga (59:41.020)
Some of them you get time to think about them,
Lex Fridman (59:43.220)
but they're always hard.
Lex Fridman (59:44.500)
So both choices are pretty good, those decisions.
Lex Fridman (59:49.220)
What about deadlines?
Rajat Monga (59:50.380)
Is this, do you find TensorFlow,
Lex Fridman (59:53.580)
to be driven by deadlines
Lex Fridman (59:58.220)
to a degree that a product might?
🔗 相关节目