Cristos Goodrow: YouTube Algorithm
AI 与机器学习心理与人性技术与编程音乐与艺术政治与社会
📋 章节目录
暂无章节信息
🔑 关键词
youtubevideovideosdonalgorithmgoingwatchedlearningsearchwatchingmachinetryingpersoninterestingsurecreatorshumancontentbettersaid
💬 精彩语录
"I've kind of moved on from that cluster intellectually, but it nevertheless is an interesting cluster."
我在智力上已经离开了那个集群，但它仍然是一个有趣的集群。
— Cristos Goodrow （10:55.120）
"And so when people talk about, well, the algorithm does this, the algorithm does that, it's sometimes"
因此，当人们谈论算法会这样做、算法会这样做时，有时
— Cristos Goodrow （1:04:55.500）
🎙️ 完整对话（1270 条）
Lex Fridman (00:00.000)
The following is a conversation with Christos Goudreau,
以下是与 Christos Goudreau 的对话，
Lex Fridman (00:03.360)
Vice President of Engineering at Google and Head of Search and Discovery at YouTube,
Google 工程副总裁兼 YouTube 搜索和发现主管，
Lex Fridman (00:08.320)
also known as the YouTube Algorithm.
也称为 YouTube 算法。
Lex Fridman (00:11.360)
YouTube has approximately 1.9 billion users,
YouTube 拥有大约 19 亿用户，
Lex Fridman (00:15.120)
and every day people watch over 1 billion hours of YouTube video.
人们每天观看 YouTube 视频的时间超过 10 亿小时。
Cristos Goodrow (00:20.320)
It is the second most popular search engine behind Google itself.
它是仅次于谷歌本身的第二大最受欢迎的搜索引擎。
Lex Fridman (00:24.080)
For many people, it is not only a source of entertainment,
对于很多人来说，它不仅是一种娱乐方式，
Lex Fridman (00:27.200)
but also how we learn new ideas from math and physics videos to podcasts to debates, opinions,
还包括我们如何从数学和物理视频、播客到辩论、观点、
Lex Fridman (00:33.680)
ideas from out of the box thinkers and activists on some of the most tense,
来自开箱即用的思想家和活动家对一些最紧张的问题的想法，
Cristos Goodrow (00:38.640)
challenging, and impactful topics in the world today.
当今世界具有挑战性和影响力的话题。
Cristos Goodrow (00:42.400)
YouTube and other content platforms receive criticism from both viewers and creators,
YouTube 和其他内容平台受到观众和创作者的批评，
Cristos Goodrow (00:48.080)
as they should, because the engineering task before them is hard, and they don't always
他们应该这样做，因为他们面前的工程任务很艰巨，而且他们并不总是
Lex Fridman (00:53.600)
succeed, and the impact of their work is truly world changing.
成功，他们的工作的影响真正改变了世界。
Cristos Goodrow (00:58.560)
To me, YouTube has been an incredible wellspring of knowledge.
对我来说，YouTube 是令人难以置信的知识源泉。
Lex Fridman (01:02.400)
I've watched hundreds, if not thousands, of lectures that changed the way I see
我看过数百甚至数千个讲座，这些讲座改变了我的观看方式
Cristos Goodrow (01:06.800)
many fundamental ideas in math, science, engineering, and philosophy.
数学、科学、工程和哲学中的许多基本思想。
Lex Fridman (01:12.480)
But it does put a mirror to ourselves, and keeps the responsibility of the steps we take
但它确实为我们自己树立了一面镜子，并让我们对所采取的步骤承担责任
Cristos Goodrow (01:17.600)
in each of our online educational journeys into the hands of each of us.
在我们每个人手中的每一次在线教育之旅中。
Cristos Goodrow (01:21.760)
The YouTube algorithm has an important role in that journey of helping us find new,
YouTube 算法在帮助我们发现新的、
Cristos Goodrow (01:26.480)
exciting ideas to learn about.
令人兴奋的想法值得学习。
Cristos Goodrow (01:28.560)
That's a difficult and an exciting problem for an artificial intelligence system.
Cristos Goodrow (01:33.360)
As I've said in lectures and other forums, recommendation systems will be one of the
Cristos Goodrow (01:37.520)
most impactful areas of AI in the 21st century, and YouTube is one of the biggest
Cristos Goodrow (01:43.680)
recommendation systems in the world.
Lex Fridman (01:46.640)
This is the Artificial Intelligence Podcast.
Cristos Goodrow (01:49.680)
If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcast, follow on
Cristos Goodrow (01:54.640)
Spotify, support it on Patreon, or simply connect with me on Twitter, at Lex Friedman,
Cristos Goodrow (01:59.920)
spelled F R I D M A N.
Lex Fridman (02:02.880)
I recently started doing ads at the end of the introduction.
Cristos Goodrow (02:05.920)
I'll do one or two minutes after introducing the episode, and never any ads in the middle
Lex Fridman (02:10.400)
that can break the flow of the conversation.
Cristos Goodrow (02:12.720)
I hope that works for you and doesn't hurt the listening experience.
Cristos Goodrow (02:16.400)
This show is presented by Cash App, the number one finance app in the App Store.
Cristos Goodrow (02:20.720)
I personally use Cash App to send money to friends, but you can also use it to buy,
Lex Fridman (02:25.120)
sell, and deposit Bitcoin in just seconds.
Cristos Goodrow (02:27.680)
Cash App also has a new investing feature.
Cristos Goodrow (02:30.640)
You can buy fractions of a stock, say, $1 worth, no matter what the stock price is.
Cristos Goodrow (02:35.760)
Broker services are provided by Cash App Investing, a subsidiary of Square and Member
Lex Fridman (02:40.640)
SIPC.
Cristos Goodrow (02:41.200)
I'm excited to be working with Cash App to support one of my favorite organizations
Lex Fridman (02:45.840)
called FIRST, best known for their FIRST Robotics and Lego competitions.
Cristos Goodrow (02:50.560)
They educate and inspire hundreds of thousands of students in over 110 countries and have
Cristos Goodrow (02:56.000)
a perfect rating and charity navigator, which means that donated money is used to maximum
Cristos Goodrow (03:00.880)
effectiveness.
Cristos Goodrow (03:02.320)
When you get Cash App from the App Store or Google Play and use code LEXPODCAST, you'll
Cristos Goodrow (03:08.240)
get $10, and Cash App will also donate $10 to FIRST, which again is an organization that
Cristos Goodrow (03:14.080)
I've personally seen inspire girls and boys to dream of engineering a better world.
Lex Fridman (03:19.920)
And now, here's my conversation with Christos Goudreau.
Cristos Goodrow (03:24.640)
YouTube is the world's second most popular search engine, behind Google, of course.
Cristos Goodrow (03:29.360)
We watch more than 1 billion hours of YouTube videos a day, more than Netflix and Facebook
Lex Fridman (03:34.800)
video combined.
Cristos Goodrow (03:35.760)
YouTube creators upload over 500,000 hours of video every day.
Lex Fridman (03:41.040)
Average lifespan of a human being, just for comparison, is about 700,000 hours.
Cristos Goodrow (03:47.440)
So, what's uploaded every single day is just enough for a human to watch in a lifetime.
Lex Fridman (03:53.360)
So, let me ask an absurd philosophical question.
Cristos Goodrow (03:56.640)
If from birth, when I was born, and there's many people born today with the internet,
Cristos Goodrow (04:00.960)
I watched YouTube videos nonstop, do you think there are trajectories through YouTube video
Cristos Goodrow (04:06.640)
space that can maximize my average happiness, or maybe education, or my growth as a human
Lex Fridman (04:14.240)
being?
Cristos Goodrow (04:15.760)
I think there are some great trajectories through YouTube videos, but I wouldn't recommend
Cristos Goodrow (04:21.200)
that anyone spend all of their waking hours or all of their hours watching YouTube.
Cristos Goodrow (04:26.640)
I mean, I think about the fact that YouTube has been really great for my kids, for instance.
Lex Fridman (04:32.720)
My oldest daughter, she's been watching YouTube for several years.
Cristos Goodrow (04:37.920)
She watches Tyler Oakley and the Vlogbrothers, and I know that it's had a very profound and
Lex Fridman (04:44.640)
positive impact on her character.
Lex Fridman (04:46.160)
And my younger daughter, she's a ballerina, and her teachers tell her that YouTube is
Cristos Goodrow (04:52.080)
a huge advantage for her because she can practice a routine and watch professional dancers do
Lex Fridman (04:58.960)
that same routine and stop it and back it up and rewind and all that stuff, right?
Lex Fridman (05:03.440)
So, it's been really good for them.
Lex Fridman (05:06.240)
And then even my son is a sophomore in college.
Cristos Goodrow (05:08.560)
He got through his linear algebra class because of a channel called Three Blue, One Brown,
Cristos Goodrow (05:15.360)
which helps you understand linear algebra, but in a way that would be very hard for anyone
Lex Fridman (05:22.240)
to do on a whiteboard or a chalkboard.
Lex Fridman (05:25.200)
And so, I think that those experiences, from my point of view, were very good.
Lex Fridman (05:30.640)
And so, I can imagine really good trajectories through YouTube, yes.
Lex Fridman (05:34.080)
Have you looked at, do you think of broadly about that trajectory over a period?
Lex Fridman (05:38.880)
Because YouTube has grown up now.
Cristos Goodrow (05:41.120)
So, over a period of years, you just kind of gave a few anecdotal examples, but I used
Lex Fridman (05:48.480)
to watch certain shows on YouTube.
Cristos Goodrow (05:49.920)
I don't anymore.
Lex Fridman (05:50.720)
I've moved on to other shows.
Cristos Goodrow (05:52.880)
Ultimately, you want people to, from YouTube's perspective, to stay on YouTube, to grow as
Lex Fridman (05:57.760)
human beings on YouTube.
Cristos Goodrow (06:00.160)
So, you have to think not just what makes them engage today or this month, but what
Lex Fridman (06:07.040)
makes them engage today or this month, but also for a period of years.
Cristos Goodrow (06:12.720)
Absolutely.
Lex Fridman (06:13.360)
That's right.
Cristos Goodrow (06:13.920)
I mean, if YouTube is going to continue to enrich people's lives, then it has to grow
Lex Fridman (06:20.320)
with them, and people's interests change over time.
Lex Fridman (06:25.200)
And so, I think we've been working on this problem, and I'll just say it broadly as
Cristos Goodrow (06:31.920)
like how to introduce diversity and introduce people who are watching one thing to something
Cristos Goodrow (06:38.720)
else they might like.
Lex Fridman (06:40.000)
We've been working on that problem all the eight years I've been at YouTube.
Cristos Goodrow (06:45.120)
It's a hard problem because, I mean, of course, it's trivial to introduce diversity
Lex Fridman (06:51.360)
that doesn't help.
Cristos Goodrow (06:52.640)
Yeah, just add a random video.
Lex Fridman (06:54.160)
I could just randomly select a video from the billions that we have.
Cristos Goodrow (06:58.800)
It's likely not to even be in your language.
Cristos Goodrow (07:01.280)
So, the likelihood that you would watch it and develop a new interest is very, very low.
Lex Fridman (07:08.560)
And so, what you want to do when you're trying to increase diversity is find something that
Cristos Goodrow (07:14.640)
is not too similar to the things that you've watched, but also something that you might
Cristos Goodrow (07:21.520)
be likely to watch.
Lex Fridman (07:23.440)
And that balance, finding that spot between those two things is quite challenging.
Cristos Goodrow (07:28.720)
So, the diversity of content, diversity of ideas, it's a really difficult, it's a thing
Lex Fridman (07:36.400)
like that's almost impossible to define, right?
Lex Fridman (07:39.360)
Like, what's different?
Lex Fridman (07:41.680)
So, how do you think about that?
Cristos Goodrow (07:43.680)
So, two examples is I'm a huge fan of Three Blue One Brown, say, and then one diversity.
Cristos Goodrow (07:51.440)
I wasn't even aware of a channel called Veritasium, which is a great science, physics, whatever
Cristos Goodrow (07:57.200)
channel.
Cristos Goodrow (07:57.600)
So, one version of diversity is showing me Derek's Veritasium channel, which I was really
Cristos Goodrow (08:03.120)
excited to discover.
Lex Fridman (08:04.160)
I actually now watch a lot of his videos.
Cristos Goodrow (08:06.400)
Okay, so you're a person who's watching some math channels and you might be interested
Lex Fridman (08:12.160)
in some other science or math channels.
Cristos Goodrow (08:14.560)
So, like you mentioned, the first kind of diversity is just show you some things from
Cristos Goodrow (08:20.160)
other channels that are related, but not just, you know, not all the Three Blue One Brown
Cristos Goodrow (08:27.600)
channel, throw in a couple others.
Cristos Goodrow (08:29.280)
So, that's maybe the first kind of diversity that we started with many, many years ago.
Cristos Goodrow (08:36.400)
Taking a bigger leap is about, I mean, the mechanisms we use for that is we basically
Lex Fridman (08:44.640)
cluster videos and channels together, mostly videos.
Cristos Goodrow (08:48.400)
We do almost everything at the video level.
Lex Fridman (08:50.640)
And so, we'll make some kind of a cluster via some embedding process and then measure
Lex Fridman (08:58.800)
what is the likelihood that users who watch one cluster might also watch another cluster
Lex Fridman (09:05.200)
that's very distinct.
Cristos Goodrow (09:06.560)
So, we may come to find that people who watch science videos also like jazz.
Lex Fridman (09:15.680)
This is possible, right?
Lex Fridman (09:16.720)
And so, because of that relationship that we've identified through the embeddings and
Cristos Goodrow (09:25.600)
then the measurement of the people who watch both, we might recommend a jazz video once
Cristos Goodrow (09:30.640)
in a while.
Cristos Goodrow (09:31.440)
So, there's this cluster in the embedding space of jazz videos and science videos.
Lex Fridman (09:36.480)
And so, you kind of try to look at aggregate statistics where if a lot of people that jump
Cristos Goodrow (09:42.960)
from science cluster to the jazz cluster tend to remain as engaged or become more engaged,
Cristos Goodrow (09:51.520)
then that means those two, they should hop back and forth and they'll be happy.
Lex Fridman (09:57.280)
Right.
Cristos Goodrow (09:57.520)
There's a higher likelihood that a person who's watching science would like jazz than
Cristos Goodrow (10:03.840)
the person watching science would like, I don't know, backyard railroads or something
Lex Fridman (10:08.080)
else, right?
Lex Fridman (10:08.480)
And so, we can try to measure these likelihoods and use that to make the best recommendation
Cristos Goodrow (10:15.120)
we can.
Lex Fridman (10:16.320)
So, okay.
Cristos Goodrow (10:16.960)
So, we'll talk about the machine learning of that, but I have to linger on things that
Lex Fridman (10:21.600)
neither you or anyone have an answer to.
Cristos Goodrow (10:24.240)
There's gray areas of truth, which is, for example, now I can't believe I'm going there,
Lex Fridman (10:31.440)
but politics.
Cristos Goodrow (10:32.720)
It happens so that certain people believe certain things and they're very certain about
Lex Fridman (10:36.960)
them.
Cristos Goodrow (10:38.400)
Let's move outside the red versus blue politics of today's world, but there's different ideologies.
Cristos Goodrow (10:44.080)
For example, in college, I read quite a lot of Ayn Rand I studied, and that's a particular
Cristos Goodrow (10:49.840)
philosophical ideology I found interesting to explore.
Lex Fridman (10:53.360)
Okay.
Cristos Goodrow (10:53.840)
So, that was that kind of space.
Cristos Goodrow (10:55.120)
I've kind of moved on from that cluster intellectually, but it nevertheless is an interesting cluster.
Cristos Goodrow (11:00.240)
I was born in the Soviet Union.
Cristos Goodrow (11:02.720)
Socialism, communism is a certain kind of political ideology that's really interesting
Cristos Goodrow (11:06.880)
to explore.
Cristos Goodrow (11:07.840)
Again, objectively, there's a set of beliefs about how the economy should work and so on.
Lex Fridman (11:12.880)
And so, it's hard to know what's true or not in terms of people within those communities
Cristos Goodrow (11:18.400)
are often advocating that this is how we achieve utopia in this world, and they're pretty
Cristos Goodrow (11:24.000)
certain about it.
Lex Fridman (11:25.040)
So, how do you try to manage politics in this chaotic, divisive world?
Cristos Goodrow (11:33.840)
Not positive or any kind of ideas in terms of filtering what people should watch next
Lex Fridman (11:38.560)
and in terms of also not letting certain things be on YouTube.
Cristos Goodrow (11:44.160)
This is an exceptionally difficult responsibility.
Lex Fridman (11:47.280)
Well, the responsibility to get this right is our top priority.
Lex Fridman (11:52.240)
And the first comes down to making sure that we have good, clear rules of the road, right?
Cristos Goodrow (11:58.640)
Like, just because we have freedom of speech doesn't mean that you can literally say anything,
Lex Fridman (12:03.440)
right?
Cristos Goodrow (12:03.920)
Like, we as a society have accepted certain restrictions on our freedom of speech.
Cristos Goodrow (12:10.800)
There are things like libel laws and things like that.
Lex Fridman (12:13.760)
And so, where we can draw a clear line, we do, and that's what we do.
Cristos Goodrow (12:20.080)
We draw a clear line, we do, and we continue to evolve that line over time.
Cristos Goodrow (12:27.360)
However, as you pointed out, wherever you draw the line, there's going to be a border
Cristos Goodrow (12:32.240)
line.
Lex Fridman (12:33.360)
And in that border line area, we are going to maybe not remove videos, but we will try
Cristos Goodrow (12:40.800)
to reduce the recommendations of them or the proliferation of them by demoting them.
Cristos Goodrow (12:47.440)
Alternatively, in those situations, try to raise what we would call authoritative or
Cristos Goodrow (12:53.040)
credible sources of information.
Lex Fridman (12:55.520)
So, we're not trying to, I mean, you mentioned Ayn Rand and communism.
Cristos Goodrow (13:03.360)
Those are two valid points of view that people are going to debate and discuss.
Lex Fridman (13:07.920)
And of course, people who believe in one or the other of those things are going to try
Cristos Goodrow (13:13.520)
to persuade other people to their point of view.
Lex Fridman (13:15.600)
And so, we're not trying to settle that or choose a side or anything like that.
Lex Fridman (13:21.840)
What we're trying to do is make sure that the people who are expressing those point
Lex Fridman (13:26.960)
of view and offering those positions are authoritative and credible.
Cristos Goodrow (13:33.680)
So, let me ask a question about people I don't like personally.
Lex Fridman (13:38.720)
You heard me.
Cristos Goodrow (13:39.360)
I don't care if you leave comments on this.
Lex Fridman (13:41.120)
But sometimes, they're brilliantly funny, which is trolls.
Cristos Goodrow (13:45.600)
So, people who kind of mock, I mean, the internet is full, Reddit of mock style
Cristos Goodrow (13:53.600)
comedy where people just kind of make fun of, point out that the emperor has no clothes.
Lex Fridman (13:59.200)
And there's brilliant comedy in that, but sometimes it can get cruel and mean.
Cristos Goodrow (14:03.920)
So, on that, on the mean point, and sorry to look at the comments, but I'm going to
Lex Fridman (14:10.560)
and sorry to linger on these things that have no good answers.
Lex Fridman (14:13.920)
But actually, I totally hear you that this is really important that you're trying to
Cristos Goodrow (14:19.520)
solve it.
Lex Fridman (14:19.920)
But how do you reduce the meanness of people on YouTube?
Cristos Goodrow (14:26.400)
I understand that anyone who uploads YouTube videos has to become resilient to a certain
Lex Fridman (14:33.600)
amount of meanness.
Cristos Goodrow (14:35.360)
Like I've heard that from many creators.
Lex Fridman (14:37.440)
And we are trying in various ways, comment ranking, allowing certain features to block
Cristos Goodrow (14:47.920)
people, to reduce or make that meanness or that trolling behavior less effective on YouTube.
Lex Fridman (14:55.840)
Yeah.
Lex Fridman (14:56.560)
And so, I mean, it's very important, but it's something that we're going to keep having
Cristos Goodrow (15:05.440)
to work on and as we improve it, like maybe we'll get to a point where people don't have
Cristos Goodrow (15:12.960)
to suffer this sort of meanness when they upload YouTube videos.
Cristos Goodrow (15:16.320)
I hope we do, but it just does seem to be something that you have to be able to deal
Cristos Goodrow (15:23.440)
with as a YouTube creator nowadays.
Lex Fridman (15:25.440)
Do you have a hope that, so you mentioned two things that I kind of agree with.
Lex Fridman (15:29.040)
So there's like a machine learning approach of ranking comments based on whatever, based
Lex Fridman (15:37.200)
on how much they contribute to the healthy conversation.
Cristos Goodrow (15:40.320)
Let's put it that way.
Lex Fridman (15:41.600)
Then the other is almost an interface question of how do you, how does the creator filter?
Lex Fridman (15:48.880)
So block or how does, how do humans themselves, the users of YouTube manage their own conversation?
Lex Fridman (15:56.880)
Do you have hope that these two tools will create a better society without limiting freedom
Cristos Goodrow (16:02.400)
of speech too much, without sort of attacking, even like saying that people, what do you
Lex Fridman (16:07.840)
mean limiting, sort of curating speech?
Cristos Goodrow (16:12.560)
I mean, I think that that overall is our whole project here at YouTube.
Lex Fridman (16:16.960)
Right.
Cristos Goodrow (16:17.280)
Like we fundamentally believe and I personally believe very much that YouTube can be great.
Lex Fridman (16:24.720)
It's been great for my kids.
Cristos Goodrow (16:26.080)
I think it can be great for society.
Lex Fridman (16:29.360)
But it's absolutely critical that we get this responsibility part right.
Lex Fridman (16:34.400)
And that's why it's our top priority.
Cristos Goodrow (16:37.040)
Susan Wojcicki, who's the CEO of YouTube, she says something that I personally find
Cristos Goodrow (16:42.080)
very inspiring, which is that we want to do our jobs today in a manner so that people
Cristos Goodrow (16:49.920)
20 and 30 years from now will look back and say, YouTube, they really figured this out.
Cristos Goodrow (16:54.880)
They really found a way to strike the right balance between the openness and the value
Cristos Goodrow (17:00.960)
that the openness has and also making sure that we are meeting our responsibility to
Cristos Goodrow (17:06.480)
users in society.
Lex Fridman (17:09.040)
So the burden on YouTube actually is quite incredible.
Lex Fridman (17:12.080)
And the one thing that people don't give enough credit to the seriousness and the magnitude
Lex Fridman (17:18.400)
of the problem, I think.
Lex Fridman (17:19.360)
So I personally hope that you do solve it because a lot is in your hand, a lot is riding
Lex Fridman (17:26.960)
on your success or failure.
Lex Fridman (17:28.320)
So it's besides, of course, running a successful company, you're also curating the content
Lex Fridman (17:34.240)
of the internet and the conversation on the internet.
Cristos Goodrow (17:36.800)
That's a powerful thing.
Lex Fridman (17:40.160)
So one thing that people wonder about is how much of it can be solved with pure machine
Cristos Goodrow (17:48.320)
learning.
Lex Fridman (17:49.360)
So looking at the data, studying the data and creating algorithms that curate the comments,
Cristos Goodrow (17:55.600)
curate the content, and how much of it needs human intervention, meaning people here at
Cristos Goodrow (18:02.880)
YouTube in a room sitting and thinking about what is the nature of truth, what are the
Cristos Goodrow (18:11.920)
ideals that we should be promoting, that kind of thing.
Lex Fridman (18:14.480)
So algorithm versus human input, what's your sense?
Cristos Goodrow (18:18.800)
I mean, my own experience has demonstrated that you need both of those things.
Cristos Goodrow (18:25.200)
Algorithms, I mean, you're familiar with machine learning algorithms and the thing
Cristos Goodrow (18:29.360)
they need most is data and the data is generated by humans.
Lex Fridman (18:34.640)
And so, for instance, when we're building a system to try to figure out which are the
Cristos Goodrow (18:42.000)
videos that are misinformation or borderline policy violations, well, the first thing we
Cristos Goodrow (18:49.680)
need to do is get human beings to make decisions about which of those videos are in which category.
Lex Fridman (18:57.600)
And then we use that data and basically take that information that's determined and governed
Cristos Goodrow (19:04.240)
by humans and extrapolate it or apply it to the entire set of billions of YouTube videos.
Lex Fridman (19:12.960)
And we couldn't get to all the videos on YouTube well without the humans, and we couldn't use
Lex Fridman (19:20.720)
the humans to get to all the videos of YouTube.
Lex Fridman (19:23.120)
So there's no world in which you have only one or the other of these things.
Lex Fridman (19:28.320)
And just as you said, a lot of it comes down to people at YouTube spending a lot of time
Cristos Goodrow (19:37.440)
trying to figure out what are the right policies, what are the outcomes based on those policies,
Lex Fridman (19:43.840)
are they the kinds of things we want to see?
Lex Fridman (19:46.640)
And then once we kind of get an agreement or build some consensus around what the policies
Cristos Goodrow (19:53.760)
are, well, then we've got to find a way to implement those policies across all of YouTube.
Lex Fridman (19:59.040)
And that's where both the human beings, we call them evaluators or reviewers, come into
Cristos Goodrow (1:00:02.800)
There's a momentum, there's a huge excited audience that makes creators feel great.
Lex Fridman (1:00:08.960)
And I think it's more than just financial.
Lex Fridman (1:00:11.740)
I think it's literally just, they love that sense of community.
Cristos Goodrow (1:00:16.220)
It's part of the reason I upload to YouTube.
Lex Fridman (1:00:18.340)
I don't care about money, never will.
Lex Fridman (1:00:20.580)
What I care about is the community, but some people feel like this momentum, and even when
Cristos Goodrow (1:00:26.420)
there's times in their life when they don't feel, you know, for some reason don't feel
Cristos Goodrow (1:00:31.260)
like creating.
Lex Fridman (1:00:32.260)
So how do you think about burnout, this mental exhaustion that some YouTube creators go through?
Lex Fridman (1:00:38.220)
Is that something we have an answer for?
Lex Fridman (1:00:40.500)
Is that something, how do we even think about that?
Cristos Goodrow (1:00:42.740)
Well, the first thing is we want to make sure that the YouTube systems are not contributing
Lex Fridman (1:00:47.700)
to this sense, right?
Lex Fridman (1:00:49.180)
And so we've done a fair amount of research to demonstrate that you can absolutely take
Lex Fridman (1:00:56.780)
a break.
Cristos Goodrow (1:00:57.940)
If you are a creator and you've been uploading a lot, we have just as many examples of people
Cristos Goodrow (1:01:03.620)
who took a break and came back more popular than they were before as we have examples
Cristos Goodrow (1:01:08.780)
of going the other way.
Lex Fridman (1:01:09.780)
Yeah.
Lex Fridman (1:01:10.780)
Can we pause on that for a second?
Lex Fridman (1:01:11.780)
So the feeling that people have, I think, is if I take a break, everybody, the party
Lex Fridman (1:01:17.500)
will leave, right?
Lex Fridman (1:01:19.280)
So if you could just linger on that.
Lex Fridman (1:01:21.780)
So in your sense that taking a break is okay.
Lex Fridman (1:01:24.460)
Yes, taking a break is absolutely okay.
Lex Fridman (1:01:27.780)
And the reason I say that is because we have, we can observe many examples of being, of
Cristos Goodrow (1:01:35.100)
creators coming back very strong and even stronger after they have taken some sort of
Cristos Goodrow (1:01:40.780)
break.
Lex Fridman (1:01:41.780)
And so I just want to dispel the myth that this somehow necessarily means that your channel
Cristos Goodrow (1:01:50.440)
is going to go down or lose views.
Lex Fridman (1:01:53.420)
That is not the case.
Cristos Goodrow (1:01:55.460)
We know for sure that this is not a necessary outcome.
Lex Fridman (1:01:59.780)
And so we want to encourage people to make sure that they take care of themselves.
Lex Fridman (1:02:04.020)
That is job one, right?
Lex Fridman (1:02:06.060)
You have to look after yourself and your mental health.
Lex Fridman (1:02:10.340)
And I think that it probably, in some of these cases, contributes to better videos once they
Lex Fridman (1:02:19.140)
come back, right?
Cristos Goodrow (1:02:20.180)
Because a lot of people, I mean, I know myself, if I burn out on something, then I'm probably
Lex Fridman (1:02:24.420)
not doing my best work, even though I can keep working until I pass out.
Lex Fridman (1:02:30.180)
And so I think that the taking a break may even improve the creative ideas that someone
Lex Fridman (1:02:38.020)
has.
Cristos Goodrow (1:02:39.020)
Okay.
Lex Fridman (1:02:40.020)
I think that's a really important thing to sort of dispel.
Cristos Goodrow (1:02:42.820)
I think that applies to all of social media, like literally I've taken a break for a day
Lex Fridman (1:02:47.460)
every once in a while.
Cristos Goodrow (1:02:49.460)
Sorry.
Cristos Goodrow (1:02:50.460)
Sorry if that sounds like a short time, but even like, sorry, email, just taking a break
Cristos Goodrow (1:02:57.620)
from email, or only checking email once a day, especially when you're going through
Cristos Goodrow (1:03:02.060)
something psychologically in your personal life or so on, or really not sleeping much
Cristos Goodrow (1:03:06.500)
because of work deadlines, it can refresh you in a way that's profound.
Lex Fridman (1:03:10.940)
And so the same applies.
Lex Fridman (1:03:11.940)
It was there when you came back, right?
Lex Fridman (1:03:13.100)
It's there.
Lex Fridman (1:03:14.100)
And it looks different, actually, when you come back.
Cristos Goodrow (1:03:17.380)
You're sort of brighter eyed with some coffee, everything, the world looks better.
Lex Fridman (1:03:22.340)
So it's important to take a break when you need it.
Lex Fridman (1:03:26.400)
So you've mentioned kind of the YouTube algorithm that isn't E equals MC squared, it's not the
Cristos Goodrow (1:03:33.020)
single equation, it's potentially sort of more than a million lines of code.
Cristos Goodrow (1:03:41.500)
Is it more akin to what successful autonomous vehicles today are, which is they're just
Cristos Goodrow (1:03:47.940)
basically patches on top of patches of heuristics and human experts really tuning the algorithm
Lex Fridman (1:03:55.540)
and have some machine learning modules?
Cristos Goodrow (1:03:58.540)
Or is it becoming more and more a giant machine learning system with humans just doing a little
Lex Fridman (1:04:04.740)
bit of tweaking here and there?
Lex Fridman (1:04:06.300)
What's your sense?
Lex Fridman (1:04:07.300)
First of all, do you even have a sense of what is the YouTube algorithm at this point?
Lex Fridman (1:04:11.420)
And however much you do have a sense, what does it look like?
Cristos Goodrow (1:04:15.540)
Well, we don't usually think about it as the algorithm because it's a bunch of systems
Cristos Goodrow (1:04:21.500)
that work on different services.
Cristos Goodrow (1:04:24.300)
The other thing that I think people don't understand is that what you might refer to
Cristos Goodrow (1:04:29.940)
as the YouTube algorithm from outside of YouTube is actually a bunch of code and machine learning
Cristos Goodrow (1:04:37.820)
systems and heuristics, but that's married with the behavior of all the people who come
Cristos Goodrow (1:04:43.620)
to YouTube every day.
Lex Fridman (1:04:44.780)
So the people part of the code, essentially.
Cristos Goodrow (1:04:46.780)
Exactly.
Cristos Goodrow (1:04:47.780)
If there were no people who came to YouTube tomorrow, then the algorithm wouldn't work
Cristos Goodrow (1:04:51.580)
anymore.
Lex Fridman (1:04:52.580)
Right.
Cristos Goodrow (1:04:53.580)
That's the whole part of the algorithm.
Lex Fridman (1:04:55.500)
And so when people talk about, well, the algorithm does this, the algorithm does that, it's sometimes
Cristos Goodrow (1:05:00.020)
hard to understand, well, it could be the viewers are doing that.
Lex Fridman (1:05:04.700)
And the algorithm is mostly just keeping track of what the viewers do and then reacting to
Cristos Goodrow (1:05:10.520)
those things in sort of more fine grain situations.
Lex Fridman (1:05:16.220)
And I think that this is the way that the recommendation system and the search system
Lex Fridman (1:05:21.280)
and probably many machine learning systems evolve is you start trying to solve a problem
Lex Fridman (1:05:28.180)
and the first way to solve a problem is often with a simple heuristic.
Lex Fridman (1:05:34.380)
And you want to say, what are the videos we're going to recommend?
Lex Fridman (1:05:36.820)
Well, how about the most popular ones?
Cristos Goodrow (1:05:39.540)
That's where you start.
Lex Fridman (1:05:43.100)
And over time, you collect some data and you refine your situation so that you're making
Cristos Goodrow (1:05:48.900)
less heuristics and you're building a system that can actually learn what to do in different
Lex Fridman (1:05:54.620)
situations based on some observations of those situations in the past.
Lex Fridman (1:06:00.760)
And you keep chipping away at these heuristics over time.
Lex Fridman (1:06:03.600)
And so I think that just like with diversity, I think the first diversity measure we took
Cristos Goodrow (1:06:10.980)
was, okay, not more than three videos in a row from the same channel.
Lex Fridman (1:06:15.460)
It's a pretty simple heuristic to encourage diversity, but it worked, right?
Lex Fridman (1:06:20.700)
Who needs to see four, five, six videos in a row from the same channel?
Lex Fridman (1:06:25.300)
And over time, we try to chip away at that and make it more fine grain and basically
Cristos Goodrow (1:06:31.320)
have it remove the heuristics in favor of something that can react to individuals and
Lex Fridman (1:06:39.380)
individual situations.
Lex Fridman (1:06:41.340)
So how do you, you mentioned, you know, we know that something worked.
Lex Fridman (1:06:46.660)
How do you get a sense when decisions are kind of A, B testing that this idea was a
Lex Fridman (1:06:51.860)
good one, this was not so good?
Lex Fridman (1:06:55.180)
How do you measure that and across which time scale, across how many users, that kind of
Lex Fridman (1:07:00.780)
thing?
Lex Fridman (1:07:01.780)
Well, you mentioned the A, B experiments.
Lex Fridman (1:07:04.540)
And so just about every single change we make to YouTube, we do it only after we've run
Lex Fridman (1:07:11.780)
a A, B experiment.
Lex Fridman (1:07:13.800)
And so in those experiments, which run from one week to months, we measure hundreds, literally
Cristos Goodrow (1:07:24.280)
hundreds of different variables and measure changes with confidence intervals in all of
Cristos Goodrow (1:07:30.460)
them, because we really are trying to get a sense for ultimately, does this improve
Lex Fridman (1:07:36.900)
the experience for viewers?
Cristos Goodrow (1:07:38.340)
That's the question we're trying to answer.
Lex Fridman (1:07:40.540)
And an experiment is one way because we can see certain things go up and down.
Lex Fridman (1:07:45.100)
So for instance, if we noticed in the experiment, people are dismissing videos less frequently,
Cristos Goodrow (1:07:52.700)
or they're saying that they're more satisfied, they're giving more videos five stars after
Cristos Goodrow (1:07:58.700)
they watch them, then those would be indications that the experiment is successful, that it's
Lex Fridman (1:08:04.540)
improving the situation for viewers.
Lex Fridman (1:08:08.180)
But we can also look at other things, like we might do user studies, where we invite
Lex Fridman (1:08:12.900)
some people in and ask them, like, what do you think about this?
Lex Fridman (1:08:16.060)
What do you think about that?
Lex Fridman (1:08:17.060)
How do you feel about this?
Lex Fridman (1:08:19.620)
And other various kinds of user research.
Lex Fridman (1:08:22.000)
But ultimately, before we launch something, we're going to want to run an experiment.
Lex Fridman (1:08:26.140)
So we get a sense for what the impact is going to be, not just to the viewers, but also to
Lex Fridman (1:08:31.260)
the different channels and all of that.
Cristos Goodrow (1:08:36.640)
An absurd question.
Lex Fridman (1:08:38.180)
Nobody knows.
Cristos Goodrow (1:08:39.180)
Well, actually, it's interesting.
Lex Fridman (1:08:40.180)
Maybe there's an answer.
Lex Fridman (1:08:41.180)
But if I want to make a viral video, how do I do it?
Lex Fridman (1:08:45.700)
I don't know how you make a viral video.
Cristos Goodrow (1:08:48.180)
I know that we have in the past tried to figure out if we could detect when a video was going
Lex Fridman (1:08:55.820)
to go viral.
Lex Fridman (1:08:57.500)
And those were, you take the first and second derivatives of the view count and maybe use
Lex Fridman (1:09:03.100)
that to do some prediction.
Lex Fridman (1:09:07.780)
But I can't say we ever got very good at that.
Lex Fridman (1:09:10.860)
Oftentimes we look at where the traffic was coming from.
Cristos Goodrow (1:09:14.660)
If a lot of the viewership is coming from something like Twitter, then maybe it has
Cristos Goodrow (1:09:20.620)
a higher chance of becoming viral than if it were coming from search or something.
Lex Fridman (1:09:26.940)
But that was just trying to detect a video that might be viral.
Lex Fridman (1:09:30.220)
How to make one, I have no idea.
Cristos Goodrow (1:09:33.620)
You get your kids to interrupt you while you're on the news or something.
Lex Fridman (1:09:38.140)
Absolutely.
Lex Fridman (1:09:39.140)
But after the fact, on one individual video, sort of ahead of time predicting is a really
Lex Fridman (1:09:44.060)
hard task.
Lex Fridman (1:09:45.060)
But after the video went viral, in analysis, can you sometimes understand why it went viral?
Cristos Goodrow (1:09:53.780)
From the perspective of YouTube broadly, first of all, is it even interesting for YouTube
Cristos Goodrow (1:09:58.060)
that a particular video is viral or does that not matter for the individual, for the experience
Lex Fridman (1:10:04.540)
of people?
Cristos Goodrow (1:10:05.540)
Well, I think people expect that if a video is going viral and it's something they would
Cristos Goodrow (1:10:11.260)
be interested in, then I think they would expect YouTube to recommend it to them.
Cristos Goodrow (1:10:16.460)
Right.
Lex Fridman (1:10:17.460)
So if something's going viral, it's good to just let the wave, let people ride the wave
Cristos Goodrow (1:10:21.820)
of its violence.
Lex Fridman (1:10:22.820)
Well, I mean, we want to meet people's expectations in that way, of course.
Lex Fridman (1:10:27.780)
So like I mentioned, I hung out with Derek Mueller a while ago, a couple of months back.
Lex Fridman (1:10:34.180)
He's actually the person who suggested I talk to you on this podcast.
Cristos Goodrow (1:10:37.980)
All right.
Lex Fridman (1:10:38.980)
Well, thank you, Derek.
Cristos Goodrow (1:10:40.700)
At that time, he just recently posted an awesome science video titled, why are 96 million black
Lex Fridman (1:10:48.020)
balls on this reservoir?
Lex Fridman (1:10:50.500)
And in a matter of, I don't know how long, but like a few days, he got 38 million views
Lex Fridman (1:10:55.500)
and it's still growing.
Cristos Goodrow (1:10:57.960)
Is this something you can analyze and understand why it happened, this video and you want a
Lex Fridman (1:11:03.980)
particular video like it?
Cristos Goodrow (1:11:06.140)
I mean, we can surely see where it was recommended, where it was found, who watched it and those
Lex Fridman (1:11:13.220)
sorts of things.
Lex Fridman (1:11:14.220)
So it's actually, sorry to interrupt, it is the video which helped me discover who Derek
Lex Fridman (1:11:20.300)
is.
Cristos Goodrow (1:11:21.300)
I didn't know who he is before.
Lex Fridman (1:11:22.300)
So I remember, you know, usually I just have all of these technical, boring MIT Stanford
Cristos Goodrow (1:11:28.060)
talks in my recommendation because that's how I watch.
Lex Fridman (1:11:30.580)
And then all of a sudden there's this black balls and reservoir video with like an excited
Lex Fridman (1:11:35.860)
nerd with like just, why is this being recommended to me?
Lex Fridman (1:11:40.940)
So I clicked on it and watched the whole thing and it was awesome.
Lex Fridman (1:11:44.060)
And then a lot of people had that experience, like why was I recommended this?
Lex Fridman (1:11:48.020)
But they all of course watched it and enjoyed it, which is, what's your sense of this just
Cristos Goodrow (1:11:52.900)
wave of recommendation that comes with this viral video that ultimately people get enjoy
Lex Fridman (1:11:58.420)
after they click on it?
Cristos Goodrow (1:11:59.860)
Well, I think it's the system, you know, basically doing what anybody who's recommending something
Cristos Goodrow (1:12:05.060)
would do, which is you show it to some people and if they like it, you say, okay, well,
Lex Fridman (1:12:09.820)
can I find some more people who are a little bit like them?
Lex Fridman (1:12:12.140)
Okay, I'm going to try it with them.
Cristos Goodrow (1:12:14.060)
Oh, they like it too.
Lex Fridman (1:12:15.180)
Let me expand the circle some more, find some more people.
Cristos Goodrow (1:12:17.500)
Oh, it turns out they like it too.
Lex Fridman (1:12:19.460)
And you just keep going until you get some feedback that says that, no, now you've gone
Cristos Goodrow (1:12:23.140)
too far.
Lex Fridman (1:12:24.140)
These people don't like it anymore.
Lex Fridman (1:12:25.940)
And so I think that's basically what happened.
Lex Fridman (1:12:28.900)
And you asked me about how to make a video go viral or make a viral video.
Cristos Goodrow (1:12:35.300)
I don't think that if you or I decided to make a video about 96 million balls that it
Lex Fridman (1:12:41.380)
would also go viral.
Cristos Goodrow (1:12:42.700)
It's possible that Derek made like the canonical video about those black balls in the lake.
Lex Fridman (1:12:51.100)
He did actually.
Cristos Goodrow (1:12:52.100)
Right.
Lex Fridman (1:12:53.100)
And I don't know whether or not just following along is the secret.
Cristos Goodrow (1:12:59.100)
Yeah.
Lex Fridman (1:13:00.100)
But it's fascinating.
Cristos Goodrow (1:13:01.100)
I mean, just like you said, the algorithm sort of expanding that circle and then figuring
Cristos Goodrow (1:13:04.420)
out that more and more people did enjoy it and that sort of phase shift of just a huge
Cristos Goodrow (1:13:09.880)
number of people enjoying it and the algorithm quickly, automatically, I assume, figuring
Lex Fridman (1:13:15.100)
that out.
Cristos Goodrow (1:13:16.100)
I don't know, the dynamics of psychology of that is a beautiful thing.
Lex Fridman (1:13:20.300)
So what do you think about the idea of clipping?
Cristos Goodrow (1:13:25.340)
Too many people annoyed me into doing it, which is they were requesting it.
Cristos Goodrow (1:13:29.780)
They said it would be very beneficial to add clips in like the coolest points and actually
Cristos Goodrow (1:13:36.580)
have explicit videos.
Cristos Goodrow (1:13:37.860)
Like I'm re uploading a video, like a short clip, which is what the podcasts are doing.
Lex Fridman (1:13:44.420)
Do you see as opposed to, like I also add timestamps for the topics, do you want the
Lex Fridman (1:13:49.180)
clip?
Lex Fridman (1:13:50.180)
Do you see YouTube somehow helping creators with that process or helping connect clips
Lex Fridman (1:13:54.820)
to the original videos or is that just on a long list of amazing features to work towards?
Cristos Goodrow (1:14:00.420)
Yeah.
Cristos Goodrow (1:14:01.420)
I mean, it's not something that I think we've done yet, but I can tell you that I think
Cristos Goodrow (1:14:08.300)
clipping is great and I think it's actually great for you as a creator.
Lex Fridman (1:14:12.660)
And here's the reason.
Cristos Goodrow (1:14:15.100)
If you think about, I mean, let's say the NBA is uploading videos of its games.
Cristos Goodrow (1:14:23.020)
Well, people might search for warriors versus rockets or they might search for Steph Curry.
Lex Fridman (1:14:31.060)
And so a highlight from the game in which Steph Curry makes an amazing shot is an opportunity
Lex Fridman (1:14:37.740)
for someone to find a portion of that video.
Lex Fridman (1:14:41.180)
And so I think that you never know how people are going to search for something that you've
Lex Fridman (1:14:48.100)
created.
Lex Fridman (1:14:49.100)
And so you want to, I would say you want to make clips and add titles and things like
Lex Fridman (1:14:54.100)
that so that they can find it as easily as possible.
Lex Fridman (1:14:58.340)
Do you have a dream of a future, perhaps a distant future when the YouTube algorithm
Lex Fridman (1:15:03.980)
figures that out?
Cristos Goodrow (1:15:05.580)
Sort of automatically detects the parts of the video that are really interesting, exciting,
Cristos Goodrow (1:15:12.260)
potentially exciting for people and sort of clip them out in this incredibly rich space.
Cristos Goodrow (1:15:17.420)
Cause if you talk about, if you talk, even just this conversation, we probably covered
Cristos Goodrow (1:15:21.260)
30, 40 little topics and there's a huge space of users that would find, you know, 30% of
Cristos Goodrow (1:15:29.620)
those topics really interesting.
Lex Fridman (1:15:30.620)
And that space is very different.
Lex Fridman (1:15:33.460)
It's something that's beyond my ability to clip out, right?
Lex Fridman (1:15:37.700)
But the algorithm might be able to figure all that out, sort of expand into clips.
Lex Fridman (1:15:43.580)
Do you have a, do you think about this kind of thing?
Lex Fridman (1:15:46.140)
Do you have a hope or dream that one day the algorithm will be able to do that kind of
Lex Fridman (1:15:49.580)
deep content analysis?
Cristos Goodrow (1:15:50.820)
Well, we've actually had projects that attempt to achieve this, but it really does depend
Cristos Goodrow (1:15:57.620)
on understanding the video well and our understanding of the video right now is quite crude.
Lex Fridman (1:16:03.780)
And so I think it would be especially hard to do it with a conversation like this.
Lex Fridman (1:16:11.360)
One might be able to do it with, let's say a soccer match more easily, right?
Lex Fridman (1:16:18.020)
You could probably find out where the goals were scored.
Lex Fridman (1:16:20.620)
And then of course you, you need to figure out who it was that scored the goal and, and
Lex Fridman (1:16:25.780)
that might require a human to do some annotation.
Lex Fridman (1:16:28.300)
But I think that trying to identify coherent topics in a transcript, like, like the one
Cristos Goodrow (1:16:35.140)
of our conversation is, is not something that we're going to be very good at right away.
Lex Fridman (1:16:42.540)
And I was speaking more to the general problem actually of being able to do both a soccer
Cristos Goodrow (1:16:46.820)
match and our conversation without explicit sort of almost my, my hope was that there
Cristos Goodrow (1:16:52.560)
exists an algorithm that's able to find exciting things in video.
Lex Fridman (1:17:00.700)
So Google now on Google search will help you find the segment of the video that you're
Cristos Goodrow (1:17:06.100)
interested in.
Lex Fridman (1:17:07.100)
So if you search for something like how to change the filter in my dishwasher, then if
Cristos Goodrow (1:17:13.940)
there's a long video about your dishwasher and this is the part where the person shows
Lex Fridman (1:17:17.620)
you how to change the filter, then, then it will highlight that area.
Lex Fridman (1:17:22.140)
And provide a link directly to it.
Lex Fridman (1:17:24.180)
And do you know if, from your recollection, do you know if the thumbnail reflects, like,
Lex Fridman (1:17:29.500)
what's the difference between showing the full video and the shorter clip?
Lex Fridman (1:17:32.700)
Do you know how it's presented in search results?
Cristos Goodrow (1:17:34.820)
I don't remember how it's presented.
Lex Fridman (1:17:36.260)
And the other thing I would say is that right now it's based on creator annotations.
Cristos Goodrow (1:17:41.860)
Ah, got it.
Lex Fridman (1:17:43.100)
So it's not the thing we're talking about.
Lex Fridman (1:17:45.940)
But folks are working on the more automatic version.
Cristos Goodrow (1:17:50.020)
It's interesting, people might not imagine this, but a lot of our systems start by using
Cristos Goodrow (1:17:56.740)
almost entirely the audience behavior.
Lex Fridman (1:18:00.720)
And then as they get better, the refinement comes from using the content.
Lex Fridman (1:18:07.780)
And I wish, I know there's privacy concerns, but I wish YouTube explored the space, which
Cristos Goodrow (1:18:15.660)
is sort of putting a camera on the users if they allowed it, right, to study their, like,
Cristos Goodrow (1:18:21.500)
I did a lot of emotion recognition work and so on, to study actual sort of richer signal.
Cristos Goodrow (1:18:27.260)
One of the cool things when you upload 360 like VR video to YouTube, and I've done this
Cristos Goodrow (1:18:32.660)
a few times, so I've uploaded myself, it's a horrible idea.
Lex Fridman (1:18:37.500)
Some people enjoyed it, but whatever.
Cristos Goodrow (1:18:39.540)
The video of me giving a lecture in 360 with a 360 camera, and it's cool because YouTube
Lex Fridman (1:18:44.220)
allows you to then watch where did people look at?
Cristos Goodrow (1:18:47.460)
There's a heat map of where, you know, of where the center of the VR experience was.
Lex Fridman (1:18:53.300)
And it's interesting because that reveals to you, like, what people looked at.
Cristos Goodrow (1:18:57.340)
It's not always what you were expecting.
Cristos Goodrow (1:19:00.700)
In the case of the lecture, it's pretty boring, it is what we were expecting, but we did a
Cristos Goodrow (1:19:05.140)
few funny videos where there's a bunch of people doing things, and everybody tracks
Lex Fridman (1:19:09.500)
those people.
Cristos Goodrow (1:19:10.500)
You know, in the beginning, they all look at the main person and they start spreading
Lex Fridman (1:19:13.540)
around and looking at the other people.
Cristos Goodrow (1:19:15.220)
It's fascinating.
Lex Fridman (1:19:16.220)
So that kind of, that's a really strong signal of what people found exciting in the video.
Cristos Goodrow (1:19:21.860)
I don't know how you get that from people just watching, except they tuned out at this
Lex Fridman (1:19:26.260)
point.
Cristos Goodrow (1:19:27.260)
Like, it's hard to measure this moment was super exciting for people.
Lex Fridman (1:19:32.540)
I don't know how you get that signal.
Cristos Goodrow (1:19:34.260)
Maybe comment, is there a way to get that signal where this was like, this is when their
Lex Fridman (1:19:38.240)
eyes opened up and they're like, like for me with the Ray Dalio video, right?
Cristos Goodrow (1:19:42.580)
Like at first I was like, okay, this is another one of these like dumb it down for you videos.
Lex Fridman (1:19:48.020)
And then you like start watching, it's like, okay, there's really crisp, clean, deep explanation
Cristos Goodrow (1:19:52.660)
of how the economy works.
Lex Fridman (1:19:54.380)
That's where I like set up and started watching, right?
Lex Fridman (1:19:56.700)
That moment, is there a way to detect that moment?
Lex Fridman (1:19:59.800)
The only way I can think of is by asking people to label it.
Cristos Goodrow (1:20:05.180)
You mentioned that we're quite far away in terms of doing video analysis, deep video
Lex Fridman (1:20:09.900)
analysis.
Cristos Goodrow (1:20:11.820)
Of course, Google, YouTube, you know, we're quite far away from solving autonomous driving
Lex Fridman (1:20:18.180)
problem too.
Lex Fridman (1:20:19.180)
So it's a...
Lex Fridman (1:20:20.180)
I don't know.
Cristos Goodrow (1:20:21.180)
I think we're closer to that.
Lex Fridman (1:20:22.180)
Well, the, you know, you never know.
Lex Fridman (1:20:25.340)
And the Wright brothers thought they're never, they're not going to fly for 50 years, three
Lex Fridman (1:20:29.260)
years before they flew.
Lex Fridman (1:20:30.760)
So what are the biggest challenges would you say?
Cristos Goodrow (1:20:34.960)
Is it the broad challenge of understanding video, understanding natural language, understanding
Cristos Goodrow (1:20:40.920)
the challenge before the entire machine learning community or just being able to understand
Lex Fridman (1:20:45.140)
data?
Cristos Goodrow (1:20:46.140)
Is there something specific to video that's even more challenging than understanding natural
Lex Fridman (1:20:51.460)
language understanding?
Lex Fridman (1:20:53.020)
What's your sense of what the biggest challenge is?
Lex Fridman (1:20:54.500)
Video is just so much information.
Lex Fridman (1:20:56.960)
And so precision becomes a real problem.
Cristos Goodrow (1:21:01.140)
It's like, you know, you're trying to classify something and you've got a million classes
Lex Fridman (1:21:08.660)
and the distinctions among them, at least from a machine learning perspective are often
Lex Fridman (1:21:17.820)
pretty small, right?
Cristos Goodrow (1:21:19.820)
Like, you know, you need to see this person's number in order to know which player it is.
Lex Fridman (1:21:28.580)
And there's a lot of players or you need to see, you know, the logo on their chest in
Cristos Goodrow (1:21:35.820)
order to know like which team they play for.
Lex Fridman (1:21:38.500)
And so, and that's just figuring out who's who, right?
Lex Fridman (1:21:41.900)
And then you go further and saying, okay, well, you know, was that a goal?
Lex Fridman (1:21:45.620)
Was it not a goal?
Lex Fridman (1:21:46.620)
Like, is that an interesting moment as you said, or is that not an interesting moment?
Lex Fridman (1:21:51.600)
These things can be pretty hard.
Lex Fridman (1:21:53.080)
So okay.
Lex Fridman (1:21:54.080)
So Yann LeCun, I'm not sure if you're familiar sort of with his current thinking and work.
Lex Fridman (1:21:59.800)
So he believes that self, what he's referring to as self supervised learning will be the
Lex Fridman (1:22:05.340)
solution sort of to achieving this kind of greater level of intelligence.
Cristos Goodrow (1:22:09.740)
In fact, the thing he's focusing on is watching video and predicting the next frame.
Lex Fridman (1:22:14.940)
So predicting the future of video, right?
Lex Fridman (1:22:18.220)
So for now we're very far from that, but his thought is because it's unsupervised or as
Cristos Goodrow (1:22:24.340)
he refers to as self supervised, you know, if you watch enough video, essentially if
Cristos Goodrow (1:22:29.540)
you watch YouTube, you'll be able to learn about the nature of reality, the physics,
Cristos Goodrow (1:22:34.780)
the common sense reasoning required by just teaching a system to predict the next frame.
Lex Fridman (1:22:40.140)
So he's confident this is the way to go.
Lex Fridman (1:22:42.660)
So for you, from the perspective of just working with this video, how do you think an algorithm
Cristos Goodrow (1:22:50.220)
that just watches all of YouTube, stays up all day and night watching YouTube would be
Cristos Goodrow (1:22:55.900)
able to understand enough of the physics of the world about the way this world works,
Lex Fridman (1:23:02.180)
be able to do common sense reasoning and so on?
Lex Fridman (1:23:05.020)
Well, I mean, we have systems that already watch all the videos on YouTube, right?
Lex Fridman (1:23:10.940)
But they're just looking for very specific things, right?
Cristos Goodrow (1:23:13.660)
They're supervised learning systems that are trying to identify something or classify something.
Lex Fridman (1:23:22.140)
And I don't know if, I don't know if predicting the next frame is really going to get there
Cristos Goodrow (1:23:25.580)
because I'm not an expert on compression algorithms, but I understand that that's kind of what
Cristos Goodrow (1:23:32.740)
compression video compression algorithms do is they basically try to predict the next
Lex Fridman (1:23:37.060)
frame and then fix up the places where they got it wrong.
Lex Fridman (1:23:41.920)
And that leads to higher compression than if you actually put all the bits for the next
Lex Fridman (1:23:46.180)
frame there.
Lex Fridman (1:23:48.340)
So I don't know if I believe that just being able to predict the next frame is going to
Cristos Goodrow (1:23:53.220)
be enough because there's so many frames and even a tiny bit of error on a per frame basis
Cristos Goodrow (1:24:00.020)
can lead to wildly different videos.
Lex Fridman (1:24:02.740)
So the thing is, the idea of compression is one way to do compression is to describe through
Cristos Goodrow (1:24:08.860)
text what's contained in the video.
Lex Fridman (1:24:10.460)
That's the ultimate high level of compression.
Lex Fridman (1:24:12.220)
So the idea is traditionally when you think of video image compression, you're trying
Lex Fridman (1:24:16.940)
to maintain the same visual quality while reducing the size.
Lex Fridman (1:24:22.520)
But if you think of deep learning from a bigger perspective of what compression is, is you're
Lex Fridman (1:24:27.420)
trying to summarize the video.
Lex Fridman (1:24:29.600)
And the idea there is if you have a big enough neural network, just by watching the next,
Cristos Goodrow (1:24:35.460)
trying to predict the next frame, you'll be able to form a compression of actually understanding
Cristos Goodrow (1:24:40.720)
what's going on in the scene.
Cristos Goodrow (1:24:42.340)
If there's two people talking, you can just reduce that entire video into the fact that
Cristos Goodrow (1:24:47.480)
two people are talking and maybe the content of what they're saying and so on.
Lex Fridman (1:24:51.780)
That's kind of the open ended dream.
Lex Fridman (1:24:55.440)
So I just wanted to sort of express that because it's interesting, compelling notion, but it
Cristos Goodrow (1:25:01.220)
is nevertheless true that video, our world is a lot more complicated than we get a credit
Cristos Goodrow (1:25:07.460)
for.
Cristos Goodrow (1:25:08.460)
I mean, in terms of search and discovery, we have been working on trying to summarize
Cristos Goodrow (1:25:12.720)
videos in text or with some kind of labels for eight years at least.
Lex Fridman (1:25:20.520)
And you know, and we're kind of so, so.
Lex Fridman (1:25:25.180)
So if you were to say the problem is a hundred percent solved and eight years ago was zero
Lex Fridman (1:25:31.460)
percent solved, where are we on that timeline would you say?
Cristos Goodrow (1:25:37.300)
Yeah.
Lex Fridman (1:25:38.300)
To summarize a video well, maybe less than a quarter of the way.
Lex Fridman (1:25:44.420)
So on that topic, what does YouTube look like 10, 20, 30 years from now?
Lex Fridman (1:25:50.700)
I mean, I think that YouTube is evolving to take the place of TV.
Cristos Goodrow (1:25:58.140)
I grew up as a kid in the seventies and I watched a tremendous amount of television
Lex Fridman (1:26:03.700)
and I feel sorry for my poor mom because people told her at the time that it was going to
Cristos Goodrow (1:26:09.580)
rot my brain and that she should kill her television.
Lex Fridman (1:26:14.380)
But anyway, I mean, I think that YouTube is at least for my family, a better version of
Lex Fridman (1:26:21.060)
television, right?
Lex Fridman (1:26:22.120)
It's one that is on demand.
Cristos Goodrow (1:26:24.560)
It's more tailored to the things that my kids want to watch.
Lex Fridman (1:26:28.740)
And also they can find things that they would never have found on television.
Lex Fridman (1:26:34.360)
And so I think that at least from just observing my own family, that's where we're headed is
Cristos Goodrow (1:26:40.360)
that people watch YouTube kind of in the same way that I watched television when I was younger.
Lex Fridman (1:26:46.220)
So from a search and discovery perspective, what do you, what are you excited about in
Lex Fridman (1:26:51.820)
the five, 10, 20, 30 years?
Lex Fridman (1:26:54.060)
Like what kind of things?
Lex Fridman (1:26:55.660)
It's already really good.
Cristos Goodrow (1:26:56.660)
I think it's achieved a lot of, of course we don't know what's possible.
Lex Fridman (1:27:01.980)
So it's the task of search of typing in the text or discovering new videos by the next
Cristos Goodrow (1:27:08.140)
recommendation.
Lex Fridman (1:27:09.140)
So I personally am really happy with the experience.
Cristos Goodrow (1:27:12.060)
I continuously, I rarely watch a video that's not awesome from my own perspective, but what's,
Lex Fridman (1:27:18.180)
what else is possible?
Lex Fridman (1:27:19.940)
What are you excited about?
Cristos Goodrow (1:27:21.260)
Well, I think introducing people to more of what's available on YouTube is not only very
Cristos Goodrow (1:27:28.840)
important to YouTube and to creators, but I think it will help enrich people's lives
Cristos Goodrow (1:27:34.380)
because there's a lot that I'm still finding out is available on YouTube that I didn't
Cristos Goodrow (1:27:38.780)
even know.
Cristos Goodrow (1:27:39.780)
I've been working YouTube eight years and it wasn't until last year that I learned that,
Cristos Goodrow (1:27:46.220)
that I could watch USC football games from the 1970s.
Cristos Goodrow (1:27:51.140)
Like I didn't even know that was possible until last year and I've been working here
Cristos Goodrow (1:27:55.060)
quite some time.
Lex Fridman (1:27:56.060)
So, you know, what was broken about, about that?
Cristos Goodrow (1:27:58.980)
That it took me seven years to learn that this stuff was already on YouTube even when
Lex Fridman (1:28:03.060)
I got here.
Lex Fridman (1:28:04.580)
So I think there's a big opportunity there.
Lex Fridman (1:28:07.100)
And then as I said before, you know, we want to make sure that YouTube finds a way to ensure
Cristos Goodrow (1:28:16.740)
that it's acting responsibly with respect to society and enriching people's lives.
Lex Fridman (1:28:23.340)
So we want to take all of the great things that it does and make sure that we are eliminating
Cristos Goodrow (1:28:28.260)
the negative consequences that might happen.
Lex Fridman (1:28:31.820)
And then lastly, if we could get to a point where all the videos people watch are the
Cristos Goodrow (1:28:37.300)
best ones they've ever watched, that'd be outstanding too.
Lex Fridman (1:28:40.940)
Do you see in many senses becoming a window into the world for people?
Cristos Goodrow (1:28:45.660)
It's especially with live video, you get to watch events.
Cristos Goodrow (1:28:49.500)
I mean, it's really, it's the way you experience a lot of the world that's out there is better
Cristos Goodrow (1:28:54.580)
than TV in many, many ways.
Lex Fridman (1:28:56.780)
So do you see becoming more than just video?
Lex Fridman (1:29:00.900)
Do you see creators creating visual experiences and virtual worlds that if I'm, I'm talking
Cristos Goodrow (1:29:06.500)
crazy now, but sort of virtual reality and entering that space, or is that at least for
Lex Fridman (1:29:11.000)
now totally outside what YouTube is thinking about?
Lex Fridman (1:29:14.020)
I mean, I think Google is thinking about virtual reality.
Cristos Goodrow (1:29:18.100)
I don't think about virtual reality too much.
Cristos Goodrow (1:29:22.660)
I know that we would want to make sure that YouTube is there when virtual reality becomes
Cristos Goodrow (1:29:28.880)
something or if virtual reality becomes something that a lot of people are interested in.
Lex Fridman (1:29:34.620)
But I haven't seen it really take off yet.
Cristos Goodrow (1:29:38.220)
Take off.
Lex Fridman (1:29:39.220)
Well, the future is wide open.
Cristos Goodrow (1:29:41.260)
Christos, I've been really looking forward to this conversation.
Lex Fridman (1:29:43.980)
It's been a huge honor.
Cristos Goodrow (1:29:45.220)
Thank you for answering some of the more difficult questions I've asked.
Lex Fridman (1:29:48.580)
I'm really excited about what YouTube has in store for us.
Cristos Goodrow (1:29:52.220)
It's one of the greatest products I've ever used and continues.
Lex Fridman (1:29:54.740)
So thank you so much for talking to me.
Cristos Goodrow (1:29:56.500)
It's my pleasure.
Lex Fridman (1:29:57.500)
Thanks for asking me.
Cristos Goodrow (1:29:58.500)
Thanks for listening to this conversation.
Lex Fridman (1:30:01.500)
And thank you to our presenting sponsor, Cash App.
Cristos Goodrow (1:30:04.740)
Download it.
Lex Fridman (1:30:05.740)
Use code LexPodcast.
Cristos Goodrow (1:30:07.380)
You'll get $10 and $10 will go to FIRST, a STEM education nonprofit that inspires hundreds
Lex Fridman (1:30:12.700)
of thousands of young minds to become future leaders and innovators.
Cristos Goodrow (1:30:17.460)
If you enjoy this podcast, subscribe on YouTube, give it five stars on Apple Podcast, follow
Lex Fridman (1:30:22.500)
on Spotify, support on Patreon, or simply connect with me on Twitter.
Lex Fridman (1:30:27.220)
And now, let me leave you with some words of wisdom from Marcel Proust.
Cristos Goodrow (1:30:32.540)
The real voyage of discovery consists not in seeking new landscapes, but in having new
Cristos Goodrow (1:30:37.940)
eyes.
Lex Fridman (1:30:40.140)
Thank you for listening and hope to see you next time.
Cristos Goodrow (20:05.360)
play to help us with that.
Lex Fridman (20:07.360)
And then once we get a lot of training data from them, then we apply the machine learning
Cristos Goodrow (20:12.080)
techniques to take it even further.
Lex Fridman (20:14.560)
Do you have a sense that these human beings have a bias in some kind of direction?
Cristos Goodrow (20:20.480)
I mean, that's an interesting question.
Cristos Goodrow (20:22.880)
We do sort of in autonomous vehicles and computer vision in general, a lot of annotation, and
Cristos Goodrow (20:28.000)
we rarely ask what bias do the annotators have.
Lex Fridman (20:35.360)
Even in the sense that they're better at annotating certain things than others.
Cristos Goodrow (20:42.480)
For example, people are much better at, for example, at writing, they're much better at
Cristos Goodrow (20:48.320)
or much better at annotating segmentation at segmenting cars in a scene versus segmenting
Cristos Goodrow (20:56.080)
bushes or trees.
Cristos Goodrow (20:59.120)
There's specific mechanical reasons for that, but also because it's semantic gray area.
Lex Fridman (21:04.960)
And just for a lot of reasons, people are just terrible at annotating trees.
Cristos Goodrow (21:09.520)
Okay, so in the same kind of sense, do you think of, in terms of people reviewing videos
Cristos Goodrow (21:15.040)
or annotating the content of videos, is there some kind of bias that you're aware of or
Lex Fridman (21:21.440)
seek out in that human input?
Cristos Goodrow (21:24.160)
Well, we take steps to try to overcome these kinds of biases or biases that we think would
Lex Fridman (21:30.560)
be problematic.
Lex Fridman (21:32.960)
So for instance, like we ask people to have a bias towards scientific consensus.
Lex Fridman (21:38.400)
That's something that we instruct them to do.
Cristos Goodrow (21:41.040)
We ask them to have a bias towards demonstration of expertise or credibility or authoritativeness.
Lex Fridman (21:48.560)
But there are other biases that we want to make sure to try to remove.
Lex Fridman (21:53.280)
And there's many techniques for doing this.
Lex Fridman (21:55.600)
One of them is you send the same thing to be reviewed to many people.
Lex Fridman (22:01.520)
And so, that's one technique.
Cristos Goodrow (22:04.080)
Another is that you make sure that the people that are doing these sorts of tasks, that
Cristos Goodrow (22:09.440)
these sorts of tasks are from different backgrounds and different areas of the United States or
Lex Fridman (22:15.920)
of the world.
Lex Fridman (22:17.040)
But then, even with all of that, it's possible for certain kinds of what we would call unfair
Cristos Goodrow (22:25.280)
biases to creep into machine learning systems, primarily, as you said, because maybe the
Cristos Goodrow (22:31.200)
training data itself comes in in a biased way.
Cristos Goodrow (22:34.160)
So, we also have worked very hard on improving the machine learning systems to remove and
Cristos Goodrow (22:41.760)
reduce unfair biases when it goes against or involves some protected class, for instance.
Lex Fridman (22:51.520)
Thank you for exploring with me some of the more challenging things.
Cristos Goodrow (22:55.680)
I'm sure there's a few more that we'll jump back to.
Lex Fridman (22:57.920)
But let me jump into the fun part, which is maybe the basics of the quote, unquote, YouTube
Cristos Goodrow (23:05.040)
algorithm.
Lex Fridman (23:06.880)
What does the YouTube algorithm look at to make recommendation for what to watch next?
Lex Fridman (23:11.600)
And it's from a machine learning perspective.
Lex Fridman (23:14.480)
Or when you search for a particular term, how does it know what to show you next?
Cristos Goodrow (23:20.320)
Because it seems to, at least for me, do an incredible job of both.
Lex Fridman (23:25.200)
Well, that's kind of you to say.
Cristos Goodrow (23:26.400)
It didn't used to do a very good job, but it's gotten better over the years.
Lex Fridman (23:31.840)
Even I observed that it's improved quite a bit.
Cristos Goodrow (23:35.440)
Those are two different situations.
Cristos Goodrow (23:36.960)
Like when you search for something, YouTube uses the best technology we can get from Google
Cristos Goodrow (23:45.760)
to make sure that the YouTube search system finds what someone's looking for.
Lex Fridman (23:50.000)
And of course, the very first things that one thinks about is, okay, well, does the
Lex Fridman (23:55.680)
word occur in the title, for instance?
Lex Fridman (24:00.560)
But there are much more sophisticated things where we're mostly trying to do some syntactic
Cristos Goodrow (24:07.280)
match or maybe a semantic match based on words that we can add to the document itself.
Lex Fridman (24:15.600)
For instance, maybe is this video watched a lot after this query?
Cristos Goodrow (24:21.760)
That's something that we can observe and then as a result, make sure that that document
Lex Fridman (24:30.080)
would be retrieved for that query.
Cristos Goodrow (24:33.040)
Now, when you talk about what kind of videos would be recommended to watch next, that's
Cristos Goodrow (24:40.480)
something, again, we've been working on for many years and probably the first real attempt
Cristos Goodrow (24:50.000)
to do that well was to use collaborative filtering.
Lex Fridman (24:55.520)
Can you describe what collaborative filtering is?
Cristos Goodrow (24:57.760)
Sure.
Cristos Goodrow (24:58.240)
It's just basically what we do is we observe which videos get watched close together by
Cristos Goodrow (25:06.320)
the same person.
Lex Fridman (25:08.320)
And if you observe that and if you can imagine creating a graph where the videos that get
Cristos Goodrow (25:15.040)
watched close together by the most people are very close to one another in this graph
Lex Fridman (25:20.640)
and videos that don't frequently get watched close together by the same person or the same
Cristos Goodrow (25:26.080)
people are far apart, then you end up with this graph that we call the related graph
Lex Fridman (25:33.280)
that basically represents videos that are very similar or related in some way.
Lex Fridman (25:38.640)
And what's amazing about that is that it puts all the videos that are in the same
Cristos Goodrow (25:45.760)
language together, for instance, and we didn't even have to think about language.
Lex Fridman (25:51.280)
It just does it, right?
Lex Fridman (25:52.880)
And it puts all the videos that are about sports together and it puts most of the music
Cristos Goodrow (25:56.800)
videos together and it puts all of these sorts of videos together just because that's sort
Lex Fridman (26:02.640)
of the way the people using YouTube behave.
Lex Fridman (26:05.920)
So that already cleans up a lot of the problem.
Cristos Goodrow (26:10.640)
It takes care of the lowest hanging fruit, which happens to be a huge one of just managing
Cristos Goodrow (26:16.800)
these millions of videos.
Lex Fridman (26:18.560)
That's right.
Cristos Goodrow (26:19.680)
I remember a few years ago I was talking to someone who was trying to propose that we
Cristos Goodrow (26:27.520)
do a research project concerning people who are bilingual, and this person was making
Cristos Goodrow (26:37.680)
this proposal based on the idea that YouTube could not possibly be good at recommending
Lex Fridman (26:44.160)
videos well to people who are bilingual.
Lex Fridman (26:48.000)
And so she was telling me about this and I said, well, can you give me an example of
Lex Fridman (26:54.400)
what problem do you think we have on YouTube with the recommendations?
Lex Fridman (26:57.920)
And so she said, well, I'm a researcher in the US and when I'm looking for academic
Lex Fridman (27:04.960)
topics, I want to see them in English.
Lex Fridman (27:07.920)
And so she searched for one, found a video, and then looked at the watch next suggestions
Lex Fridman (27:12.640)
and they were all in English.
Lex Fridman (27:14.720)
And so she said, oh, I see.
Lex Fridman (27:16.080)
YouTube must think that I speak only English.
Lex Fridman (27:18.480)
And so she said, now I'm actually originally from Turkey and sometimes when I'm cooking,
Cristos Goodrow (27:23.360)
let's say I want to make some baklava, I really like to watch videos that are in Turkish.
Lex Fridman (27:27.600)
And so she searched for a video about making the baklava and then selected it and it was
Lex Fridman (27:33.040)
in Turkish and the watch next recommendations were in Turkish.
Lex Fridman (27:35.600)
And she just couldn't believe how this was possible and how is it that you know that
Lex Fridman (27:41.840)
I speak both these two languages and put all the videos together?
Lex Fridman (27:44.720)
And it's just as a sort of an outcome of this related graph that's created through
Lex Fridman (27:49.520)
collaborative filtering.
Lex Fridman (27:51.440)
So for me, one of my huge interests is just human psychology, right?
Lex Fridman (27:54.800)
And that's such a powerful platform on which to utilize human psychology to discover what
Cristos Goodrow (28:02.160)
people, individual people want to watch next.
Lex Fridman (28:04.640)
But it's also be just fascinating to me.
Cristos Goodrow (28:06.720)
You know, I've, Google search has ability to look at your own history and I've done
Lex Fridman (28:13.520)
that before, just, just what I've searched three years for many, many years.
Lex Fridman (28:17.760)
And it's fascinating picture of who I am actually.
Lex Fridman (28:21.200)
And I don't think anyone's ever summarized.
Cristos Goodrow (28:24.880)
I personally would love that.
Cristos Goodrow (28:26.720)
A summary of who I am as a person on the internet to me, because I didn't get a reply
Cristos Goodrow (28:32.480)
of who I am as a person on the internet to me, because I think it reveals, I think it
Lex Fridman (28:38.080)
puts a mirror to me or to others.
Cristos Goodrow (28:41.920)
You know, that's actually quite revealing and interesting, you know, just the, maybe
Cristos Goodrow (28:47.840)
in the number of, it's a joke, but not really is the number of cat videos I've watched or
Cristos Goodrow (28:53.280)
videos of people falling, you know, stuff that's absurd, that kind of stuff.
Lex Fridman (28:59.120)
It's really interesting.
Lex Fridman (29:00.160)
And of course it's really good for the machine learning aspect to, to show, to figure out
Lex Fridman (29:06.240)
what to show next.
Lex Fridman (29:06.880)
But it's interesting.
Cristos Goodrow (29:09.120)
Have you just as a tangent played around with the idea of giving a map to people sort of,
Cristos Goodrow (29:16.800)
as opposed to just using this information to show what's next, showing them here are
Lex Fridman (29:22.320)
the clusters you've loved over the years kind of thing?
Cristos Goodrow (29:25.680)
Well, we do provide the history of all the videos that you've watched.
Lex Fridman (29:29.200)
Yes.
Lex Fridman (29:29.440)
So you can definitely search through that and look through it and search through it
Lex Fridman (29:32.880)
to see what it is that you've been watching on YouTube.
Cristos Goodrow (29:35.600)
We have actually in various times experimented with this sort of cluster idea, finding ways
Cristos Goodrow (29:44.720)
to demonstrate or show people what topics they've been interested in or what clusters
Cristos Goodrow (29:51.120)
they've watched from.
Cristos Goodrow (29:51.920)
It's interesting that you bring this up because in some sense, the way the recommendation
Cristos Goodrow (29:58.800)
system of YouTube sees a user is exactly as the history of all the videos they've
Lex Fridman (30:04.720)
watched on YouTube.
Lex Fridman (30:06.320)
And so you can think of yourself or any user on YouTube as kind of like a DNA strand of
Lex Fridman (30:17.200)
all your videos, right?
Cristos Goodrow (30:18.640)
That sort of represents you, you can also think of it as maybe a vector in the space
Lex Fridman (30:23.520)
of all the videos on YouTube.
Lex Fridman (30:26.160)
And so now once you think of it as a vector in the space of all the videos on YouTube,
Lex Fridman (30:31.680)
then you can start to say, okay, well, which other vectors are close to me and to my vector?
Lex Fridman (30:39.120)
And that's one of the ways that we generate some diverse recommendations is because you're
Cristos Goodrow (30:44.560)
like, okay, well, these people seem to be close with respect to the videos they've
Cristos Goodrow (30:50.080)
watched on YouTube, but here's a topic or a video that one of them has watched and
Cristos Goodrow (30:55.440)
enjoyed, but the other one hasn't, that could be an opportunity to make a good recommendation.
Cristos Goodrow (31:01.040)
I got to tell you, I mean, I know I'm going to ask for things that are impossible, but
Lex Fridman (31:04.720)
I would love to cluster than human beings.
Cristos Goodrow (31:07.760)
I would love to know who has similar trajectories as me, because you probably would want to
Lex Fridman (31:12.400)
hang out, right?
Cristos Goodrow (31:14.560)
There's a social aspect there, like actually finding some of the most fascinating people
Cristos Goodrow (31:18.800)
I find on YouTube, but have like no followers and I start following them and they create
Cristos Goodrow (31:23.440)
incredible content and on that topic, I just love to ask, there's some videos that just
Cristos Goodrow (31:29.280)
blow my mind in terms of quality and depth and just in every regard are amazing videos
Lex Fridman (31:37.040)
and they have like 57 views, okay?
Lex Fridman (31:40.640)
How do you get videos of quality to be seen by many eyes?
Lex Fridman (31:46.800)
So the measure of quality, is it just something, yeah, how do you know that something is good?
Cristos Goodrow (31:53.440)
Well, I mean, I think it depends initially on what sort of video we're talking about.
Lex Fridman (31:58.640)
So in the realm of, let's say you mentioned politics and news, in that realm, you know,
Lex Fridman (32:08.400)
quality news or quality journalism relies on having a journalism department, right?
Cristos Goodrow (32:16.880)
Like you have to have actual journalists and fact checkers and people like that and so
Cristos Goodrow (32:22.800)
in that situation and in others, maybe science or in medicine, quality has a lot to do with
Cristos Goodrow (32:30.000)
the authoritativeness and the credibility and the expertise of the people who make the
Lex Fridman (32:34.000)
video.
Cristos Goodrow (32:36.000)
Now, if you think about the other end of the spectrum, you know, what is the highest quality
Lex Fridman (32:42.240)
prank video or what is the highest quality Minecraft video, right?
Cristos Goodrow (32:49.280)
That might be the one that people enjoy watching the most and watch to the end or it might
Cristos Goodrow (32:54.320)
be the one that when we ask people the next day after they watched it, were they satisfied
Lex Fridman (33:03.200)
with it?
Lex Fridman (33:04.200)
And so we in, especially in the realm of entertainment, have been trying to get at better and better
Cristos Goodrow (33:11.600)
measures of quality or satisfaction or enrichment since I came to YouTube.
Lex Fridman (33:19.320)
And we started with, well, you know, the first approximation is the one that gets more views.
Lex Fridman (33:27.280)
But you know, we both know that things can get a lot of views and not really be that
Cristos Goodrow (33:32.360)
high quality, especially if people are clicking on something and then immediately realizing
Cristos Goodrow (33:37.400)
that it's not that great and abandoning it.
Lex Fridman (33:41.000)
And that's why we moved from views to thinking about the amount of time people spend watching
Cristos Goodrow (33:46.160)
it with the premise that like, you know, in some sense, the time that someone spends watching
Lex Fridman (33:52.840)
a video is related to the value that they get from that video.
Cristos Goodrow (33:57.520)
It may not be perfectly related, but it has something to say about how much value they
Lex Fridman (34:02.120)
get.
Lex Fridman (34:04.040)
But even that's not good enough, right?
Cristos Goodrow (34:05.680)
Because I myself have spent time clicking through channels on television late at night
Lex Fridman (34:11.480)
and ended up watching Under Siege 2 for some reason I don't know.
Lex Fridman (34:16.560)
And if you were to ask me the next day, are you glad that you watched that show on TV
Lex Fridman (34:21.580)
last night?
Cristos Goodrow (34:22.580)
I'd say, yeah, I wish I would have gone to bed or read a book or almost anything else,
Cristos Goodrow (34:27.800)
really.
Lex Fridman (34:29.060)
And so that's why some people got the idea a few years ago to try to survey users afterwards.
Lex Fridman (34:35.600)
And so we get feedback data from those surveys and then use that in the machine learning
Cristos Goodrow (34:43.340)
system to try to not just predict what you're going to click on right now, what you might
Cristos Goodrow (34:47.720)
watch for a while, but what when we ask you tomorrow, you'll give four or five stars to.
Lex Fridman (34:54.020)
So just to summarize, what are the signals from a machine learning perspective that a
Lex Fridman (34:59.320)
user can provide?
Lex Fridman (35:00.320)
So you mentioned just clicking on the video views, the time watched, maybe the relative
Cristos Goodrow (35:05.000)
time watched, the clicking like and dislike on the video, maybe commenting on the video.
Lex Fridman (35:12.760)
All of those things.
Cristos Goodrow (35:14.480)
All of those things.
Lex Fridman (35:15.480)
And then the one I wasn't actually quite aware of, even though I might have engaged in it
Cristos Goodrow (35:20.640)
is a survey afterwards, which is a brilliant idea.
Lex Fridman (35:24.660)
Is there other signals?
Cristos Goodrow (35:26.200)
I mean, that's already a really rich space of signals to learn from.
Lex Fridman (35:30.680)
Is there something else?
Cristos Goodrow (35:31.920)
Well, you mentioned commenting, also sharing the video.
Lex Fridman (35:35.960)
If you think it's worthy to be shared with someone else you know.
Lex Fridman (35:39.560)
Within YouTube or outside of YouTube as well?
Lex Fridman (35:41.600)
Either.
Cristos Goodrow (35:42.600)
Let's see, you mentioned like, dislike.
Lex Fridman (35:44.920)
Like and dislike.
Lex Fridman (35:45.920)
How important is that?
Lex Fridman (35:47.480)
It's very important, right?
Cristos Goodrow (35:48.480)
We want, it's predictive of satisfaction.
Lex Fridman (35:52.960)
But it's not perfectly predictive.
Cristos Goodrow (35:56.400)
Subscribe.
Cristos Goodrow (35:57.400)
If you subscribe to the channel of the person who made the video, then that also is a piece
Cristos Goodrow (36:03.840)
of information and it signals satisfaction.
Cristos Goodrow (36:07.360)
Although over the years, we've learned that people have a wide range of attitudes about
Lex Fridman (36:13.840)
what it means to subscribe.
Cristos Goodrow (36:17.080)
We would ask some users who didn't subscribe very much, but they watched a lot from a few
Cristos Goodrow (36:24.640)
channels.
Lex Fridman (36:25.640)
We'd say, well, why didn't you subscribe?
Lex Fridman (36:26.640)
And they would say, well, I can't afford to pay for anything.
Lex Fridman (36:32.000)
We tried to let them understand like, actually it doesn't cost anything.
Cristos Goodrow (36:35.040)
It's free.
Lex Fridman (36:36.040)
It just helps us know that you are very interested in this creator.
Lex Fridman (36:41.180)
But then we've asked other people who subscribe to many things and don't really watch any
Lex Fridman (36:47.560)
of the videos from those channels.
Lex Fridman (36:49.080)
And we say, well, why did you subscribe to this if you weren't really interested in any
Lex Fridman (36:54.920)
more videos from that channel?
Lex Fridman (36:56.300)
And they might tell us, well, I just, you know, I thought the person did a great job
Lex Fridman (37:00.140)
and I just want to kind of give them a high five.
Lex Fridman (37:03.280)
And so.
Lex Fridman (37:04.280)
Yeah.
Cristos Goodrow (37:05.280)
That's where I sit.
Lex Fridman (37:06.280)
I go to channels where I just, this person is amazing.
Cristos Goodrow (37:11.320)
I like this person.
Lex Fridman (37:13.200)
But then I like this person and I really want to support them.
Cristos Goodrow (37:18.000)
That's how I click subscribe.
Cristos Goodrow (37:19.760)
Even though I mean never actually want to click on their videos when they're releasing
Cristos Goodrow (37:23.200)
it.
Lex Fridman (37:24.200)
I just love what they're doing.
Lex Fridman (37:25.200)
And it's maybe outside of my interest area and so on, which is probably the wrong way
Lex Fridman (37:30.440)
to use the subscribe button.
Lex Fridman (37:31.440)
But I just want to say congrats.
Lex Fridman (37:32.920)
This is great work.
Cristos Goodrow (37:34.920)
Well, so you have to deal with all the space of people that see the subscribe button is
Lex Fridman (37:39.320)
totally different.
Cristos Goodrow (37:40.320)
That's right.
Lex Fridman (37:41.320)
And so, you know, we can't just close our eyes and say, sorry, you're using it wrong.
Cristos Goodrow (37:46.200)
You know, we're not going to pay attention to what you've done.
Cristos Goodrow (37:50.260)
We need to embrace all the ways in which all the different people in the world use the
Cristos Goodrow (37:53.880)
subscribe button or the like and the dislike button.
Lex Fridman (37:57.840)
So in terms of signals of machine learning, using for the search and for the recommendation,
Cristos Goodrow (38:05.400)
you've mentioned title.
Lex Fridman (38:06.400)
So like metadata, like text data that people provide description and title and maybe keywords.
Cristos Goodrow (38:13.840)
Maybe you can speak to the value of those things in search and also this incredible
Lex Fridman (38:19.760)
fascinating area of the content itself.
Lex Fridman (38:22.860)
So the video content itself, trying to understand what's happening in the video.
Lex Fridman (38:26.280)
So YouTube released a data set that, you know, in the machine learning computer vision world,
Cristos Goodrow (38:30.960)
this is just an exciting space.
Lex Fridman (38:33.280)
How much is that currently?
Lex Fridman (38:35.760)
How much are you playing with that currently?
Lex Fridman (38:37.300)
How much is your hope for the future of being able to analyze the content of the video itself?
Cristos Goodrow (38:42.120)
Well, we have been working on that also since I came to YouTube.
Lex Fridman (38:46.560)
Analyzing the content.
Lex Fridman (38:47.560)
Analyzing the content of the video, right?
Lex Fridman (38:50.700)
And what I can tell you is that our ability to do it well is still somewhat crude.
Cristos Goodrow (39:00.280)
We can tell if it's a music video, we can tell if it's a sports video, we can probably
Lex Fridman (39:05.120)
tell you that people are playing soccer.
Cristos Goodrow (39:09.520)
We probably can't tell whether it's Manchester United or my daughter's soccer team.
Lex Fridman (39:15.440)
So these things are kind of difficult and using them, we can use them in some ways.
Lex Fridman (39:21.280)
So for instance, we use that kind of information to understand and inform these clusters that
Lex Fridman (39:27.080)
I talked about.
Lex Fridman (39:30.240)
And also maybe to add some words like soccer, for instance, to the video, if it doesn't
Cristos Goodrow (39:34.980)
occur in the title or the description, which is remarkable that often it doesn't.
Cristos Goodrow (39:40.960)
One of the things that I ask creators to do is please help us out with the title and the
Lex Fridman (39:47.560)
description.
Cristos Goodrow (39:48.560)
For instance, we were a few years ago having a live stream of some competition for World
Lex Fridman (39:56.160)
of Warcraft on YouTube.
Lex Fridman (39:59.080)
And it was a very important competition, but if you typed World of Warcraft in search,
Lex Fridman (40:04.220)
you wouldn't find it.
Lex Fridman (40:05.480)
World of Warcraft wasn't in the title?
Lex Fridman (40:07.600)
World of Warcraft wasn't in the title.
Cristos Goodrow (40:09.120)
It was match 478, you know, A team versus B team and World of Warcraft wasn't in the
Lex Fridman (40:14.520)
title.
Cristos Goodrow (40:15.520)
I'm just like, come on, give me.
Lex Fridman (40:17.940)
Being literal on the internet is actually very uncool, which is the problem.
Lex Fridman (40:22.120)
Oh, is that right?
Cristos Goodrow (40:23.920)
Well, I mean, in some sense, well, some of the greatest videos, I mean, there's a humor
Cristos Goodrow (40:28.520)
to just being indirect, being witty and so on.
Lex Fridman (40:31.800)
And actually being, you know, machine learning algorithms want you to be, you know, literal,
Lex Fridman (40:37.560)
right?
Lex Fridman (40:38.560)
You just want to say what's in the thing, be very, very simple.
Lex Fridman (40:42.840)
And in some sense that gets away from wit and humor.
Lex Fridman (40:46.160)
So you have to play with both, right?
Lex Fridman (40:48.920)
But you're saying that for now, sort of the content of the title, the content of the description,
Cristos Goodrow (40:54.280)
the actual text is one of the best ways for the algorithm to find your video and put them
Cristos Goodrow (41:01.920)
in the right cluster.
Lex Fridman (41:03.080)
That's right.
Lex Fridman (41:04.160)
And I would go further and say that if you want people, human beings to select your video
Lex Fridman (41:10.240)
in search, then it helps to have, let's say World of Warcraft in the title.
Cristos Goodrow (41:14.920)
Because why would a person, you know, if they're looking at a bunch, they type World of Warcraft
Lex Fridman (41:20.000)
and they have a bunch of videos, all of whom say World of Warcraft, except the one that
Cristos Goodrow (41:23.880)
you uploaded.
Cristos Goodrow (41:24.880)
Well, even the person is going to think, well, maybe this isn't somehow search made a mistake.
Cristos Goodrow (41:29.280)
This isn't really about World of Warcraft.
Lex Fridman (41:31.540)
So it's important not just for the machine learning systems, but also for the people
Cristos Goodrow (41:36.160)
who might be looking for this sort of thing.
Cristos Goodrow (41:38.000)
They get a clue that it's what they're looking for by seeing that same thing prominently
Cristos Goodrow (41:44.680)
in the title of the video.
Lex Fridman (41:45.960)
Okay.
Cristos Goodrow (41:46.960)
Let me push back on that.
Lex Fridman (41:47.960)
So I think from the algorithm perspective, yes, but if they typed in World of Warcraft
Lex Fridman (41:52.640)
and saw a video that with the title simply winning and the thumbnail has like a sad orc
Lex Fridman (42:02.440)
or something, I don't know, right?
Cristos Goodrow (42:04.480)
Like I think that's much, it gets your curiosity up.
Lex Fridman (42:11.760)
And then if they could trust that the algorithm was smart enough to figure out somehow that
Cristos Goodrow (42:15.920)
this is indeed a World of Warcraft video, that would have created the most beautiful
Lex Fridman (42:20.000)
experience.
Cristos Goodrow (42:21.000)
I think in terms of just the wit and the humor and the curiosity that we human beings naturally
Lex Fridman (42:25.720)
have.
Lex Fridman (42:26.720)
But you're saying, I mean, realistically speaking, it's really hard for the algorithm
Lex Fridman (42:30.080)
to figure out that the content of that video will be a World of Warcraft video.
Lex Fridman (42:34.680)
And you have to accept that some people are going to skip it.
Lex Fridman (42:37.120)
Yeah.
Lex Fridman (42:38.120)
Right?
Lex Fridman (42:39.120)
I mean, and so you're right.
Cristos Goodrow (42:41.040)
The people who don't skip it and select it are going to be delighted, but other people
Lex Fridman (42:47.120)
might say, yeah, this is not what I was looking for.
Lex Fridman (42:50.080)
And making stuff discoverable, I think is what you're really working on and hoping.
Lex Fridman (42:56.600)
So yeah.
Lex Fridman (42:57.600)
So from your perspective, put stuff in the title description.
Lex Fridman (43:00.440)
And remember the collaborative filtering part of the system starts by the same user watching
Lex Fridman (43:07.960)
videos together, right?
Lex Fridman (43:09.800)
So the way that they're probably going to do that is by searching for them.
Cristos Goodrow (43:14.200)
That's a fascinating aspect of it.
Lex Fridman (43:15.480)
It's like ant colonies.
Cristos Goodrow (43:16.480)
That's how they find stuff.
Lex Fridman (43:19.000)
So I mean, what degree for collaborative filtering in general is one curious ant, one curious
Lex Fridman (43:27.680)
user, essential?
Lex Fridman (43:28.680)
So just a person who is more willing to click on random videos and sort of explore these
Cristos Goodrow (43:33.800)
cluster spaces.
Cristos Goodrow (43:35.520)
In your sense, how many people are just like watching the same thing over and over and
Lex Fridman (43:39.640)
over and over?
Lex Fridman (43:40.640)
And how many are just like the explorers and just kind of like click on stuff and then
Lex Fridman (43:44.760)
help the other ant in the ant's colony discover the cool stuff?
Lex Fridman (43:49.680)
Do you have a sense of that at all?
Cristos Goodrow (43:51.080)
I really don't think I have a sense for the relative sizes of those groups.
Lex Fridman (43:56.040)
But I would say that people come to YouTube with some certain amount of intent.
Lex Fridman (44:01.240)
And as long as they, to the extent to which they try to satisfy that intent, that certainly
Lex Fridman (44:08.040)
helps our systems, right?
Lex Fridman (44:09.520)
Because our systems rely on kind of a faithful amount of behavior, right?
Lex Fridman (44:17.360)
And there are people who try to trick us, right?
Cristos Goodrow (44:19.000)
There are people and machines that try to associate videos together that really don't
Cristos Goodrow (44:25.280)
belong together, but they're trying to get that association made because it's profitable
Cristos Goodrow (44:30.360)
for them.
Lex Fridman (44:31.440)
And so we have to always be resilient to that sort of attempt at gaming the systems.
Lex Fridman (44:37.680)
So speaking to that, there's a lot of people that in a positive way, perhaps, I don't know,
Cristos Goodrow (44:42.760)
I don't like it, but like to want to try to game the system to get more attention.
Lex Fridman (44:47.720)
Everybody creators in a positive sense want to get attention, right?
Lex Fridman (44:51.500)
So how do you work in this space when people create more and more sort of click baity titles
Lex Fridman (45:01.020)
and thumbnails?
Cristos Goodrow (45:02.020)
Sort of very to ask him, Derek has made a video where basically describes that it seems
Lex Fridman (45:08.080)
what works is to create a high quality video, really good video, where people would want
Cristos Goodrow (45:12.920)
to watch it once they click on it, but have click baity titles and thumbnails to get them
Cristos Goodrow (45:18.040)
to click on it in the first place.
Lex Fridman (45:19.640)
And he's saying, I'm embracing this fact, I'm just going to keep doing it.
Lex Fridman (45:23.600)
And I hope you forgive me for doing it and you will enjoy my videos once you click on
Lex Fridman (45:28.520)
them.
Lex Fridman (45:29.520)
So in what sense do you see this kind of click bait style attempt to manipulate, to get people
Lex Fridman (45:38.000)
in the door to manipulate the algorithm or play with the algorithm or game the algorithm?
Cristos Goodrow (45:43.400)
I think that you can look at it as an attempt to game the algorithm.
Lex Fridman (45:47.560)
But even if you were to take the algorithm out of it and just say, okay, well, all these
Cristos Goodrow (45:52.800)
videos happen to be lined up, which the algorithm didn't make any decision about which one to
Cristos Goodrow (45:57.800)
put at the top or the bottom, but they're all lined up there, which one are the people
Lex Fridman (46:02.240)
going to choose?
Lex Fridman (46:04.180)
And I'll tell you the same thing that I told Derek is, I have a bookshelf and they have
Cristos Goodrow (46:09.640)
two kinds of books on them, science books.
Cristos Goodrow (46:13.560)
I have my math books from when I was a student and they all look identical except for the
Cristos Goodrow (46:19.340)
titles on the covers.
Cristos Goodrow (46:21.220)
They're all yellow, they're all from Springer and they're every single one of them.
Cristos Goodrow (46:24.920)
The cover is totally the same.
Lex Fridman (46:27.240)
Yes.
Lex Fridman (46:28.240)
Right?
Lex Fridman (46:29.240)
Yeah.
Cristos Goodrow (46:30.240)
On the other hand, I have other more pop science type books and they all have very interesting
Lex Fridman (46:34.960)
covers and they have provocative titles and things like that.
Cristos Goodrow (46:40.400)
I wouldn't say that they're click baity because they are indeed good books.
Lex Fridman (46:45.640)
And I don't think that they cross any line, but that's just a decision you have to make.
Cristos Goodrow (46:52.720)
Like the people who write classical recursion theory by Piero di Freddie, he was fine with
Lex Fridman (46:58.560)
the yellow title and nothing more.
Cristos Goodrow (47:02.240)
Whereas I think other people who wrote a more popular type book understand that they need
Lex Fridman (47:10.320)
to have a compelling cover and a compelling title.
Lex Fridman (47:15.320)
And I don't think there's anything really wrong with that.
Lex Fridman (47:19.240)
We do take steps to make sure that there is a line that you don't cross.
Lex Fridman (47:24.880)
And if you go too far, maybe your thumbnail is especially racy or it's all caps with too
Lex Fridman (47:32.080)
many exclamation points, we observe that users are sometimes offended by that.
Lex Fridman (47:41.960)
And so for the users who are offended by that, we will then depress or suppress those videos.
Lex Fridman (47:51.240)
And which reminds me, there's also another signal where users can say, I don't know if
Cristos Goodrow (47:55.640)
it was recently added, but I really enjoy it.
Cristos Goodrow (47:58.080)
Just saying, something like, I don't want to see this video anymore or something like,
Cristos Goodrow (48:04.640)
like this is a, like there's certain videos that just cut me the wrong way.
Lex Fridman (48:09.200)
Like just, just jump out at me, it's like, I don't want to, I don't want this.
Lex Fridman (48:12.160)
And it feels really good to clean that up, to be like, I don't, that's not, that's not
Lex Fridman (48:17.120)
for me.
Cristos Goodrow (48:18.120)
I don't know.
Cristos Goodrow (48:19.120)
I think that might've been recently added, but that's also a really strong signal.
Cristos Goodrow (48:22.440)
Yes, absolutely.
Lex Fridman (48:23.440)
Right.
Cristos Goodrow (48:24.440)
We don't want to make a recommendation that people are unhappy with.
Lex Fridman (48:29.440)
And that makes me, that particular one makes me feel good as a user in general and as a
Cristos Goodrow (48:34.000)
machine learning person.
Lex Fridman (48:35.000)
Cause I feel like I'm helping the algorithm.
Cristos Goodrow (48:37.840)
My interactions on YouTube don't always feel like I'm helping the algorithm.
Lex Fridman (48:41.040)
Like I'm not reminded of that fact.
Cristos Goodrow (48:43.920)
Like for example, Tesla and Autopilot and Elon Musk create a feeling for their customers,
Cristos Goodrow (48:50.680)
for people that own Teslas, that they're helping the algorithm of Tesla vehicles.
Cristos Goodrow (48:54.080)
Like they're all, like are really proud they're helping the fleet learn.
Cristos Goodrow (48:57.160)
I think YouTube doesn't always remind people that you're helping the algorithm get smarter.
Lex Fridman (49:02.560)
And for me, I love that idea.
Cristos Goodrow (49:04.560)
Like we're all collaboratively, like Wikipedia gives that sense that we're all together creating
Cristos Goodrow (49:09.960)
a beautiful thing.
Lex Fridman (49:12.040)
YouTube is a, doesn't always remind me of that.
Cristos Goodrow (49:14.720)
It's a, this conversation is reminding me of that, but.
Lex Fridman (49:18.560)
Well that's a good tip.
Cristos Goodrow (49:19.560)
We should keep that fact in mind when we design these features.
Cristos Goodrow (49:22.520)
I'm not sure I really thought about it that way, but that's a very interesting perspective.
Cristos Goodrow (49:28.000)
It's an interesting question of personalization that I feel like when I click like on a video,
Lex Fridman (49:35.140)
I'm just improving my experience.
Cristos Goodrow (49:39.420)
It would be great.
Cristos Goodrow (49:40.940)
It would make me personally, people are different, but make me feel great if I was helping also
Cristos Goodrow (49:45.060)
the YouTube algorithm broadly say something.
Lex Fridman (49:47.640)
You know what I'm saying?
Cristos Goodrow (49:48.640)
Like there's a, that I don't know if that's human nature, but you want the products you
Cristos Goodrow (49:53.720)
love, and I certainly love YouTube, like you want to help it get smarter, smarter, smarter
Cristos Goodrow (49:58.960)
because there's some kind of coupling between our lives together being better.
Lex Fridman (50:04.780)
If YouTube is better than I will, my life will be better.
Lex Fridman (50:07.120)
And there's that kind of reasoning.
Lex Fridman (50:08.120)
I'm not sure what that is and I'm not sure how many people share that feeling.
Cristos Goodrow (50:12.240)
That could be just a machine learning feeling.
Lex Fridman (50:14.240)
But on that point, how much personalization is there in terms of next video recommendations?
Lex Fridman (50:22.720)
So is it kind of all really boiling down to clustering?
Cristos Goodrow (50:28.200)
Like if I'm the nearest clusters to me and so on and that kind of thing, or how much
Lex Fridman (50:33.400)
is personalized to me, the individual completely?
Lex Fridman (50:36.120)
It's very, very personalized.
Lex Fridman (50:38.900)
So your experience will be quite a bit different from anybody else's who's watching that same
Lex Fridman (50:45.160)
video, at least when they're logged in.
Lex Fridman (50:48.640)
And the reason is that we found that users often want two different kinds of things when
Lex Fridman (50:56.240)
they're watching a video.
Cristos Goodrow (50:58.320)
Sometimes they want to keep watching more on that topic or more in that genre.
Lex Fridman (51:05.000)
And other times they just are done and they're ready to move on to something else.
Lex Fridman (51:09.320)
And so the question is, well, what is the something else?
Lex Fridman (51:13.200)
And one of the first things one can imagine is, well, maybe something else is the latest
Cristos Goodrow (51:19.040)
video from some channel to which you've subscribed.
Lex Fridman (51:22.400)
And that's going to be very different for you than it is for me.
Lex Fridman (51:27.840)
And even if it's not something that you subscribe to, it's something that you watch a lot.
Lex Fridman (51:31.160)
And again, that'll be very different on a person by person basis.
Lex Fridman (51:34.960)
And so even the Watch Next, as well as the homepage, of course, is quite personalized.
Lex Fridman (51:43.800)
So what, we mentioned some of the signals, but what does success look like?
Lex Fridman (51:47.760)
What does success look like in terms of the algorithm creating a great long term experience
Lex Fridman (51:52.200)
for a user?
Cristos Goodrow (51:53.560)
Or to put another way, if you look at the videos I've watched this month, how do you
Lex Fridman (52:00.240)
know the algorithm succeeded for me?
Cristos Goodrow (52:03.680)
I think, first of all, if you come back and watch more YouTube, then that's one indication
Lex Fridman (52:09.000)
that you found some value from it.
Lex Fridman (52:10.840)
So just the number of hours is a powerful indicator.
Cristos Goodrow (52:13.480)
Well, I mean, not the hours themselves, but the fact that you return on another day.
Lex Fridman (52:22.120)
So that's probably the most simple indicator.
Lex Fridman (52:26.320)
People don't come back to things that they don't find value in, right?
Cristos Goodrow (52:29.240)
There's a lot of other things that they could do.
Lex Fridman (52:32.440)
But like I said, ideally, we would like everybody to feel that YouTube enriches their lives
Lex Fridman (52:38.320)
and that every video they watched is the best one they've ever watched since they've started
Lex Fridman (52:43.320)
watching YouTube.
Lex Fridman (52:44.840)
And so that's why we survey them and ask them, is this one to five stars?
Lex Fridman (52:52.960)
And so our version of success is every time someone takes that survey, they say it's five
Cristos Goodrow (52:58.400)
stars.
Lex Fridman (53:00.040)
And if we ask them, is this the best video you've ever seen on YouTube?
Cristos Goodrow (53:03.620)
They say, yes, every single time.
Lex Fridman (53:05.960)
So it's hard to imagine that we would actually achieve that.
Cristos Goodrow (53:09.760)
Maybe asymptotically we would get there, but that would be what we think success is.
Lex Fridman (53:16.560)
It's funny.
Cristos Goodrow (53:17.560)
I've recently said somewhere, I don't know, maybe tweeted, but that Ray Dalio has this
Cristos Goodrow (53:23.640)
video on the economic machine, I forget what it's called, but it's a 30 minute video.
Lex Fridman (53:29.280)
And I said it's the greatest video I've ever watched on YouTube.
Cristos Goodrow (53:32.880)
It's like I watched the whole thing and my mind was blown as a very crisp, clean description
Cristos Goodrow (53:38.560)
of how the, at least the American economic system works.
Lex Fridman (53:41.400)
It's a beautiful video.
Lex Fridman (53:43.080)
And I was just, I wanted to click on something to say this is the best thing.
Lex Fridman (53:47.560)
This is the best thing ever.
Cristos Goodrow (53:48.720)
Please let me, I can't believe I discovered it.
Cristos Goodrow (53:51.040)
I mean, the views and the likes reflect its quality, but I was almost upset that I haven't
Cristos Goodrow (53:57.400)
found it earlier and wanted to find other things like it.
Lex Fridman (54:01.000)
I don't think I've ever felt that this is the best video I've ever watched.
Cristos Goodrow (54:05.000)
That was that.
Lex Fridman (54:06.180)
And to me, the ultimate utopia, the best experiences were every single video.
Cristos Goodrow (54:10.960)
Where I don't see any of the videos I regret and every single video I watch is one that
Lex Fridman (54:15.520)
actually helps me grow, helps me enjoy life, be happy and so on.
Lex Fridman (54:25.080)
So that's a heck of a, that's one of the most beautiful and ambitious, I think, machine
Lex Fridman (54:31.480)
learning tasks.
Lex Fridman (54:32.840)
So when you look at a society as opposed to the individual user, do you think of how YouTube
Cristos Goodrow (54:37.760)
is changing society when you have these millions of people watching videos, growing, learning,
Lex Fridman (54:44.200)
changing, having debates?
Lex Fridman (54:45.840)
Do you have a sense of, yeah, what the big impact on society is?
Lex Fridman (54:51.520)
I think it's huge, but do you have a sense of what direction we're taking this world?
Lex Fridman (54:55.960)
Well, I mean, I think openness has had an impact on society already.
Cristos Goodrow (55:02.520)
There's a lot of...
Lex Fridman (55:03.520)
What do you mean by openness?
Cristos Goodrow (55:05.680)
Well, the fact that unlike other mediums, there's not someone sitting at YouTube who
Cristos Goodrow (55:14.160)
decides before you can upload your video, whether it's worth having you upload it or
Lex Fridman (55:20.080)
worth anybody seeing it really, right?
Lex Fridman (55:23.120)
And so there are some creators who say, like, I wouldn't have this opportunity to reach
Cristos Goodrow (55:32.440)
an audience.
Cristos Goodrow (55:33.720)
Tyler Oakley often said that he wouldn't have had this opportunity to reach this audience
Cristos Goodrow (55:39.440)
if it weren't for YouTube.
Lex Fridman (55:44.000)
And so I think that's one way in which YouTube has changed society.
Cristos Goodrow (55:50.080)
I know that there are people that I work with from outside the United States, especially
Cristos Goodrow (55:56.160)
from places where literacy is low, and they think that YouTube can help in those places
Cristos Goodrow (56:03.760)
because you don't need to be able to read and write in order to learn something important
Lex Fridman (56:09.060)
for your life, maybe how to do some job or how to fix something.
Lex Fridman (56:15.200)
And so that's another way in which I think YouTube is possibly changing society.
Lex Fridman (56:21.520)
So I've worked at YouTube for eight, almost nine years now.
Lex Fridman (56:25.960)
And it's fun because I meet people and you tell them where you work, you say you work
Lex Fridman (56:32.720)
on YouTube and they immediately say, I love YouTube, right?
Cristos Goodrow (56:36.740)
Which is great, makes me feel great.
Lex Fridman (56:39.260)
But then of course, when I ask them, well, what is it that you love about YouTube?
Cristos Goodrow (56:43.680)
Not one time ever has anybody said that the search works outstanding or that the recommendations
Lex Fridman (56:50.080)
are great.
Lex Fridman (56:52.760)
What they always say when I ask them, what do you love about YouTube is they immediately
Cristos Goodrow (56:57.860)
start talking about some channel or some creator or some topic or some community that they
Cristos Goodrow (57:03.600)
found on YouTube and that they just love.
Lex Fridman (57:07.500)
And so that has made me realize that YouTube is really about the video and connecting the
Cristos Goodrow (57:16.640)
people with the videos.
Lex Fridman (57:19.200)
And then everything else kind of gets out of the way.
Lex Fridman (57:22.680)
So beyond the video, it's an interesting, because you kind of mentioned creator.
Lex Fridman (57:28.940)
What about the connection with just the individual creators as opposed to just individual video?
Lex Fridman (57:35.240)
So like I gave the example of Ray Dalio video that the video itself is incredible, but there's
Lex Fridman (57:42.720)
some people who are just creators that I love.
Cristos Goodrow (57:47.640)
One of the cool things about people who call themselves YouTubers or whatever is they have
Lex Fridman (57:52.200)
a journey.
Cristos Goodrow (57:53.200)
They usually, almost all of them, they suck horribly in the beginning and then they kind
Lex Fridman (57:57.820)
of grow and then there's that genuineness in their growth.
Lex Fridman (58:01.800)
So YouTube clearly wants to help creators connect with their audience in this kind of
Lex Fridman (58:07.480)
way.
Lex Fridman (58:08.480)
So how do you think about that process of helping creators grow, helping them connect
Cristos Goodrow (58:12.060)
with their audience, develop not just individual videos, but the entirety of a creator's life
Lex Fridman (58:17.440)
on YouTube?
Cristos Goodrow (58:18.440)
Well, I mean, we're trying to help creators find the biggest audience that they can find.
Lex Fridman (58:24.700)
And the reason why that's, you brought up creator versus video, the reason why creator
Cristos Goodrow (58:30.580)
channel is so important is because if we have a hope of people coming back to YouTube, well,
Cristos Goodrow (58:41.120)
they have to have in their minds some sense of what they're going to find when they come
Lex Fridman (58:46.000)
back to YouTube.
Cristos Goodrow (58:48.020)
If YouTube were just the next viral video and I have no concept of what the next viral
Cristos Goodrow (58:54.740)
video could be, one time it's a cat playing a piano and the next day it's some children
Cristos Goodrow (59:00.000)
interrupting a reporter and the next day it's some other thing happening, then it's hard
Cristos Goodrow (59:06.600)
for me to, when I'm not watching YouTube, say, gosh, I really would like to see something
Lex Fridman (59:14.760)
from someone or about something, right?
Lex Fridman (59:17.980)
And so that's why I think this connection between fans and creators is so important
Cristos Goodrow (59:24.280)
for both, because it's a way of sort of fostering a relationship that can play out into the
Lex Fridman (59:31.700)
future.
Cristos Goodrow (59:32.700)
Let me talk about kind of a dark and interesting question in general, and again, a topic that
Lex Fridman (59:40.100)
you or nobody has an answer to.
Lex Fridman (59:42.400)
But social media has a sense of, it gives us highs and it gives us lows in the sense
Cristos Goodrow (59:50.580)
that sort of creators often speak about having sort of burnout and having psychological ups
Lex Fridman (59:58.180)
and downs and challenges mentally in terms of continuing the creation process.
🔗 相关节目