Dawn Song: Adversarial Machine Learning and Computer Security
AI 与机器学习技术与编程心理与人性生物与进化音乐与艺术
📋 章节目录
暂无章节信息
🔑 关键词
datalearningessentiallyattacksprogramsecuritydonmodelmachinetrainingadversarialsystemsphysicalprivacysciencemeaningattackerattackrealable
💬 精彩语录
暂无语录
🎙️ 完整对话（2762 条）
Lex Fridman (00:00.000)
The following is a conversation with Dawn Song,
以下是与黎明宋的对话，
Lex Fridman (00:02.680)
a professor of computer science at UC Berkeley
加州大学伯克利分校计算机科学教授
Lex Fridman (00:05.500)
with research interests in computer security.
对计算机安全有研究兴趣。
Lex Fridman (00:08.260)
Most recently, with a focus on the intersection
最近，重点关注十字路口
Lex Fridman (00:10.960)
between security and machine learning.
安全与机器学习之间。
Dawn Song (00:13.880)
This conversation was recorded
这段对话被录音
Lex Fridman (00:15.160)
before the outbreak of the pandemic.
疫情爆发之前。
Dawn Song (00:17.160)
For everyone feeling the medical, psychological,
对于每个感受到医学、心理、
Lex Fridman (00:19.560)
and financial burden of this crisis,
以及这场危机的财务负担，
Dawn Song (00:21.520)
I'm sending love your way.
我正在用你的方式传递爱。
Lex Fridman (00:23.120)
Stay strong.
坚强点。
Dawn Song (00:24.160)
We're in this together.
我们在一起。
Lex Fridman (00:25.520)
We'll beat this thing.
我们会打败这件事。
Dawn Song (00:27.320)
This is the Artificial Intelligence Podcast.
这是人工智能播客。
Lex Fridman (00:29.640)
If you enjoy it, subscribe on YouTube,
如果您喜欢，请在 YouTube 上订阅，
Dawn Song (00:31.800)
review it with five stars on Apple Podcast,
在 Apple Podcast 上以五颗星评价它，
Lex Fridman (00:34.120)
support it on Patreon,
在 Patreon 上支持它，
Dawn Song (00:35.560)
or simply connect with me on Twitter
或者直接在 Twitter 上与我联系
Lex Fridman (00:37.480)
at lexfriedman, spelled F R I D M A N.
在 lexfriedman，拼写为 F R I D M A N。
Dawn Song (00:41.340)
As usual, I'll do a few minutes of ads now
像往常一样，我现在会做几分钟的广告
Lex Fridman (00:43.760)
and never any ads in the middle
Dawn Song (00:45.160)
that can break the flow of the conversation.
Lex Fridman (00:47.840)
I hope that works for you
Lex Fridman (00:48.880)
and doesn't hurt the listening experience.
Lex Fridman (00:51.760)
This show is presented by Cash App,
Dawn Song (00:53.560)
the number one finance app in the App Store.
Lex Fridman (00:55.800)
When you get it, use code lexpodcast.
Dawn Song (00:58.800)
Cash App lets you send money to friends,
Lex Fridman (01:00.800)
buy Bitcoin, and invest in the stock market
Dawn Song (01:02.880)
with as little as one dollar.
Lex Fridman (01:05.000)
Since Cash App does fractional share trading,
Dawn Song (01:07.400)
let me mention that the order execution algorithm
Lex Fridman (01:10.160)
that works behind the scenes
Dawn Song (01:11.760)
to create the abstraction of fractional orders
Lex Fridman (01:14.040)
is an algorithmic marvel.
Lex Fridman (01:16.280)
So big props to the Cash App engineers
Lex Fridman (01:18.240)
for solving a hard problem
Dawn Song (01:19.760)
that in the end provides an easy interface
Lex Fridman (01:22.520)
that takes a step up to the next layer of abstraction
Dawn Song (01:25.080)
over the stock market,
Lex Fridman (01:26.520)
making trading more accessible for new investors
Lex Fridman (01:29.240)
and diversification much easier.
Lex Fridman (01:32.240)
So again, if you get Cash App from the App Store or Google Play
Lex Fridman (01:35.400)
and use the code lexpodcast, you get $10
Lex Fridman (01:39.080)
and Cash App will also donate $10 to FIRST,
Dawn Song (01:42.040)
an organization that is helping to advance robotics
Lex Fridman (01:44.720)
and STEM education for young people around the world.
Lex Fridman (01:48.680)
And now here's my conversation with Dawn Song.
Lex Fridman (01:53.520)
Do you think software systems
Lex Fridman (01:54.960)
will always have security vulnerabilities?
Lex Fridman (01:57.200)
Let's start at the broad, almost philosophical level.
Dawn Song (02:00.600)
That's a very good question.
Lex Fridman (02:02.080)
I mean, in general, right,
Dawn Song (02:03.040)
it's very difficult to write completely bug free code
Lex Fridman (02:07.640)
and code that has no vulnerability.
Lex Fridman (02:09.880)
And also, especially given that the definition
Lex Fridman (02:12.040)
of vulnerability is actually really broad.
Dawn Song (02:14.240)
It's any type of attacks essentially on a code can,
Lex Fridman (02:18.520)
you know, that's, you can call that,
Dawn Song (02:21.240)
that caused by vulnerabilities.
Lex Fridman (02:22.760)
And the nature of attacks is always changing as well?
Lex Fridman (02:25.520)
Like new ones are coming up?
Lex Fridman (02:27.240)
Right, so for example, in the past,
Dawn Song (02:29.280)
we talked about memory safety type of vulnerabilities
Lex Fridman (02:32.840)
where essentially attackers can exploit the software
Lex Fridman (02:37.080)
and take over control of how the code runs
Lex Fridman (02:40.520)
and then can launch attacks that way.
Dawn Song (02:42.120)
By accessing some aspect of the memory
Lex Fridman (02:44.580)
and be able to then alter the state of the program?
Dawn Song (02:48.640)
Exactly, so for example, in the example of a buffer overflow,
Lex Fridman (02:51.960)
then the attacker essentially actually causes
Dawn Song (02:56.720)
essentially unintended changes in the state of the program.
Lex Fridman (03:01.720)
And then, for example,
Dawn Song (03:03.120)
can then take over control flow of the program
Lex Fridman (03:05.680)
and let the program to execute codes
Dawn Song (03:08.760)
that actually the programmer didn't intend.
Lex Fridman (03:11.200)
So the attack can be a remote attack.
Lex Fridman (03:12.960)
So the attacker, for example,
Lex Fridman (03:14.880)
can send in a malicious input to the program
Dawn Song (03:17.880)
that just causes the program to completely
Lex Fridman (03:20.800)
then be compromised and then end up doing something
Dawn Song (03:24.360)
that's under the attacker's control and intention.
Lex Fridman (03:29.520)
But that's just one form of attacks
Lex Fridman (03:31.240)
and there are other forms of attacks.
Lex Fridman (03:32.680)
Like for example, there are these side channels
Dawn Song (03:35.540)
where attackers can try to learn from,
Lex Fridman (03:39.860)
even just observing the outputs
Dawn Song (03:42.040)
from the behaviors of the program,
Lex Fridman (03:43.420)
try to infer certain secrets of the program.
Lex Fridman (03:46.100)
So essentially, right, the form of attacks
Lex Fridman (03:49.200)
is very, very, it's very broad spectrum.
Lex Fridman (03:53.800)
And in general, from the security perspective,
Lex Fridman (03:56.560)
we want to essentially provide as much guarantee
Dawn Song (04:01.040)
as possible about the program's security properties
Lex Fridman (04:05.240)
and so on.
Lex Fridman (04:06.080)
So for example, we talked about providing provable guarantees
Lex Fridman (04:10.080)
of the program.
Lex Fridman (04:11.980)
So for example, there are ways we can use program analysis
Lex Fridman (04:15.880)
and formal verification techniques
Dawn Song (04:17.920)
to prove that a piece of code
Lex Fridman (04:19.720)
has no memory safety vulnerabilities.
Lex Fridman (04:24.740)
What does that look like?
Lex Fridman (04:25.580)
What is that proof?
Dawn Song (04:26.420)
Is that just a dream for,
Lex Fridman (04:28.640)
that's applicable to small case examples
Lex Fridman (04:30.760)
or is that possible to do for real world systems?
Lex Fridman (04:33.740)
So actually, I mean, today,
Dawn Song (04:35.600)
I actually call it we are entering the era
Lex Fridman (04:38.480)
of formally verified systems.
Lex Fridman (04:41.560)
So in the community, we have been working
Lex Fridman (04:44.920)
for the past decades in developing techniques
Lex Fridman (04:48.600)
and tools to do this type of program verification.
Lex Fridman (04:53.920)
And we have dedicated teams that have dedicated,
Dawn Song (04:57.680)
you know, their like years,
Lex Fridman (05:00.120)
sometimes even decades of their work in the space.
Lex Fridman (05:04.080)
So as a result, we actually have a number
Lex Fridman (05:06.560)
of formally verified systems ranging from microkernels
Dawn Song (05:11.360)
to compilers to file systems to certain crypto,
Lex Fridman (05:16.000)
you know, libraries and so on.
Lex Fridman (05:18.560)
So it's actually really wide ranging
Lex Fridman (05:20.520)
and it's really exciting to see
Dawn Song (05:22.520)
that people are recognizing the importance
Lex Fridman (05:25.360)
of having these formally verified systems
Dawn Song (05:28.920)
with verified security.
Lex Fridman (05:31.560)
So that's great advancement that we see,
Lex Fridman (05:34.000)
but on the other hand,
Lex Fridman (05:34.960)
I think we do need to take all these in essentially
Dawn Song (05:39.240)
with caution as well in the sense that,
Lex Fridman (05:41.800)
just like I said, the type of vulnerabilities
Dawn Song (05:46.640)
is very varied.
Lex Fridman (05:47.560)
We can formally verify a software system
Dawn Song (05:51.000)
to have certain set of security properties,
Lex Fridman (05:54.620)
but they can still be vulnerable to other types of attacks.
Lex Fridman (05:57.760)
And hence, we continue need to make progress in the space.
Lex Fridman (06:03.240)
So just a quick, to linger on the formal verification,
Dawn Song (06:07.600)
is that something you can do by looking at the code alone
Lex Fridman (06:12.280)
or is it something you have to run the code
Lex Fridman (06:14.960)
to prove something?
Lex Fridman (06:16.560)
So empirical verification,
Lex Fridman (06:18.240)
can you look at the code, just the code?
Lex Fridman (06:20.280)
So that's a very good question.
Lex Fridman (06:22.000)
So in general, for most program verification techniques,
Lex Fridman (06:25.500)
it's essentially try to verify the properties
Dawn Song (06:27.600)
of the program statically.
Lex Fridman (06:29.620)
And there are reasons for that too.
Dawn Song (06:32.000)
We can run the code to see, for example,
Lex Fridman (06:34.880)
using like in software testing with the fuzzing techniques
Lex Fridman (06:39.440)
and also in certain even model checking techniques,
Lex Fridman (06:41.880)
you can actually run the code.
Lex Fridman (06:45.280)
But in general, that only allows you to essentially verify
Lex Fridman (06:51.040)
or analyze the behaviors of the program
Dawn Song (06:55.200)
under certain situations.
Lex Fridman (06:57.000)
And so most of the program verification techniques
Dawn Song (06:59.360)
actually works statically.
Lex Fridman (07:01.600)
What does statically mean?
Dawn Song (07:03.400)
Without running the code.
Lex Fridman (07:04.920)
Without running the code, yep.
Dawn Song (07:06.440)
So, but sort of to return to the big question,
Lex Fridman (07:10.300)
if we can stand for a little bit longer,
Lex Fridman (07:13.540)
do you think there will always be
Lex Fridman (07:16.140)
security vulnerabilities?
Dawn Song (07:18.040)
You know, that's such a huge worry for people
Lex Fridman (07:20.240)
in the broad cybersecurity threat in the world.
Dawn Song (07:23.600)
It seems like the tension between nations, between groups,
Lex Fridman (07:29.440)
the wars of the future might be fought
Dawn Song (07:31.760)
in cybersecurity that people worry about.
Lex Fridman (07:35.080)
And so, of course, the nervousness is,
Dawn Song (07:37.680)
is this something that we can get ahold of in the future
Lex Fridman (07:40.440)
for our software systems?
Lex Fridman (07:42.320)
So there's a very funny quote saying,
Lex Fridman (07:46.740)
security is job security.
Dawn Song (07:49.280)
So, right, I think that essentially answers your question.
Lex Fridman (07:55.800)
Right, we strive to make progress
Dawn Song (08:00.520)
in building more secure systems
Lex Fridman (08:03.280)
and also making it easier and easier
Dawn Song (08:05.760)
to build secure systems.
Lex Fridman (08:07.780)
But given the diversity, the various nature of attacks,
Lex Fridman (08:15.680)
and also the interesting thing about security is that,
Lex Fridman (08:20.480)
unlike in most other fields,
Dawn Song (08:24.000)
essentially you are trying to, how should I put it,
Lex Fridman (08:27.040)
prove a statement true.
Lex Fridman (08:31.040)
But in this case, you are trying to say
Lex Fridman (08:32.760)
that there's no attacks.
Lex Fridman (08:35.940)
So even just this statement itself
Lex Fridman (08:37.840)
is not very well defined, again,
Dawn Song (08:40.560)
given how varied the nature of the attacks can be.
Lex Fridman (08:44.540)
And hence there's a challenge of security
Lex Fridman (08:46.640)
and also that naturally, essentially,
Lex Fridman (08:49.960)
it's almost impossible to say that something,
Dawn Song (08:52.680)
a real world system is 100% no security vulnerabilities.
Lex Fridman (08:57.280)
Is there a particular,
Lex Fridman (08:58.960)
and we'll talk about different kinds of vulnerabilities,
Lex Fridman (09:01.440)
it's exciting ones, very fascinating ones
Dawn Song (09:04.000)
in the space of machine learning,
Lex Fridman (09:05.520)
but is there a particular security vulnerability
Dawn Song (09:08.920)
that worries you the most, that you think about the most
Lex Fridman (09:12.680)
in terms of it being a really hard problem
Lex Fridman (09:16.200)
and a really important problem to solve?
Lex Fridman (09:18.480)
So it is very interesting.
Lex Fridman (09:20.200)
So I have, in the past, have worked essentially
Lex Fridman (09:22.800)
through the different stacks in the systems,
Dawn Song (09:27.640)
working on networking security, software security,
Lex Fridman (09:30.920)
and even in software security,
Dawn Song (09:32.760)
I worked on program binary security
Lex Fridman (09:35.520)
and then web security, mobile security.
Lex Fridman (09:38.120)
So throughout we have been developing
Lex Fridman (09:42.240)
more and more techniques and tools
Dawn Song (09:45.120)
to improve security of these software systems.
Lex Fridman (09:47.820)
And as a consequence, actually it's a very interesting thing
Dawn Song (09:50.800)
that we are seeing, interesting trends that we are seeing
Lex Fridman (09:53.640)
is that the attacks are actually moving more and more
Dawn Song (09:57.480)
from the systems itself towards to humans.
Lex Fridman (10:01.800)
So it's moving up the stack.
Dawn Song (10:03.440)
It's moving up the stack.
Lex Fridman (10:04.920)
That's fascinating.
Lex Fridman (10:05.760)
And also it's moving more and more
Lex Fridman (10:07.720)
towards what we call the weakest link.
Lex Fridman (10:09.760)
So we say that in security,
Lex Fridman (10:11.160)
we say the weakest link actually of the systems
Dawn Song (10:13.040)
oftentimes is actually humans themselves.
Lex Fridman (10:16.460)
So a lot of attacks, for example,
Dawn Song (10:18.700)
the attacker either through social engineering
Lex Fridman (10:21.420)
or from these other methods,
Dawn Song (10:23.700)
they actually attack the humans and then attack the systems.
Lex Fridman (10:26.740)
So we actually have a project that actually works
Dawn Song (10:29.780)
on how to use AI machine learning
Lex Fridman (10:32.300)
to help humans to defend against these types of attacks.
Lex Fridman (10:35.940)
So yeah, so if we look at humans
Lex Fridman (10:37.820)
as security vulnerabilities,
Lex Fridman (10:40.180)
is there methods, is that what you're kind of referring to?
Lex Fridman (10:43.300)
Is there hope or methodology for patching the humans?
Dawn Song (10:48.780)
I think in the future,
Lex Fridman (10:49.940)
this is going to be really more and more of a serious issue
Dawn Song (10:54.500)
because again, for machines, for systems,
Lex Fridman (10:58.460)
we can, yes, we can patch them.
Dawn Song (11:00.300)
We can build more secure systems.
Lex Fridman (11:02.300)
We can harden them and so on.
Lex Fridman (11:03.760)
But humans actually, we don't have a way
Lex Fridman (11:05.980)
to say do a software upgrade
Dawn Song (11:07.620)
or do a hardware change for humans.
Lex Fridman (11:11.140)
And so for example, right now, we already see
Dawn Song (11:16.100)
different types of attacks.
Lex Fridman (11:17.940)
In particular, I think in the future,
Dawn Song (11:19.400)
they are going to be even more effective on humans.
Lex Fridman (11:21.940)
So as I mentioned, social engineering attacks,
Dawn Song (11:24.220)
like these phishing attacks,
Lex Fridman (11:25.620)
attackers just get humans to provide their passwords.
Lex Fridman (11:30.520)
And there have been instances where even places
Lex Fridman (11:34.180)
like Google and other places
Dawn Song (11:38.100)
that are supposed to have really good security,
Lex Fridman (11:41.100)
people there have been phished
Dawn Song (11:43.420)
to actually wire money to attackers.
Lex Fridman (11:47.980)
It's crazy.
Lex Fridman (11:48.940)
And then also we talk about this deep fake and fake news.
Lex Fridman (11:52.060)
So these essentially are there to target humans,
Dawn Song (11:54.640)
to manipulate humans opinions, perceptions, and so on.
Lex Fridman (12:01.880)
So I think in going to the future,
Dawn Song (12:04.580)
these are going to become more and more severe issues for us.
Lex Fridman (12:07.580)
Further up the stack.
Dawn Song (12:08.980)
Yes, yes.
Lex Fridman (12:09.820)
So you see kind of social engineering,
Dawn Song (12:13.060)
automated social engineering
Lex Fridman (12:14.480)
as a kind of security vulnerability.
Dawn Song (12:17.060)
Oh, absolutely.
Lex Fridman (12:18.140)
And again, given that humans
Dawn Song (12:20.780)
are the weakest link to the system,
Lex Fridman (12:23.100)
I would say this is the type of attacks
Dawn Song (12:25.680)
that I would be most worried about.
Lex Fridman (12:28.820)
Oh, that's fascinating.
Dawn Song (12:30.580)
Okay, so.
Lex Fridman (12:31.420)
And that's why when we talk about AI sites,
Dawn Song (12:33.540)
also we need AI to help humans too.
Lex Fridman (12:35.780)
As I mentioned, we have some projects in the space
Dawn Song (12:37.900)
actually helps on that.
Lex Fridman (12:39.300)
Can you maybe, can we go there for the DFS?
Lex Fridman (12:41.980)
What are some ideas to help humans?
Lex Fridman (12:44.380)
So one of the projects we are working on
Dawn Song (12:45.900)
is actually using NLP and chatbot techniques
Lex Fridman (12:50.500)
to help humans.
Dawn Song (12:51.500)
For example, the chatbot actually could be there
Lex Fridman (12:54.580)
observing the conversation
Dawn Song (12:56.900)
between a user and a remote correspondence.
Lex Fridman (13:01.660)
And then the chatbot could be there to try to observe,
Dawn Song (13:05.140)
to see whether the correspondence
Lex Fridman (13:07.460)
is potentially an attacker.
Dawn Song (13:10.180)
For example, in some of the phishing attacks,
Lex Fridman (13:12.820)
the attacker claims to be a relative of the user
Lex Fridman (13:16.500)
and the relative got lost in London
Lex Fridman (13:20.460)
and his wallets have been stolen,
Dawn Song (13:22.900)
had no money, asked the user to wire money
Lex Fridman (13:25.820)
to send money to the attacker,
Dawn Song (13:28.860)
to the correspondence.
Lex Fridman (13:30.980)
So then in this case,
Dawn Song (13:31.820)
the chatbot actually could try to recognize
Lex Fridman (13:34.820)
there may be something suspicious going on.
Dawn Song (13:37.380)
This relates to asking money to be sent.
Lex Fridman (13:40.220)
And also the chatbot could actually pose,
Dawn Song (13:43.940)
we call it challenge and response.
Lex Fridman (13:45.980)
The correspondence claims to be a relative of the user,
Dawn Song (13:50.180)
then the chatbot could automatically
Lex Fridman (13:51.860)
actually generate some kind of challenges
Dawn Song (13:54.380)
to see whether the correspondence
Lex Fridman (13:57.020)
knows the appropriate knowledge
Dawn Song (13:59.460)
to prove that he actually is,
Lex Fridman (14:01.460)
he or she actually is the acclaimed relative of the user.
Lex Fridman (14:07.460)
And so in the future,
Lex Fridman (14:08.460)
I think these type of technologies
Dawn Song (14:10.500)
actually could help protect users.
Lex Fridman (14:13.940)
That's funny.
Lex Fridman (14:14.780)
So a chatbot that's kind of focused
Lex Fridman (14:17.620)
for looking for the kind of patterns
Dawn Song (14:19.220)
that are usually associated with social engineering attacks,
Lex Fridman (14:23.140)
it would be able to then test,
Dawn Song (14:26.100)
sort of do a basic capture type of a response
Lex Fridman (14:30.420)
to see is this, is the fact or the semantics
Lex Fridman (14:32.940)
of the claims you're making true?
Lex Fridman (14:34.940)
Right, right.
Dawn Song (14:35.860)
That's fascinating.
Lex Fridman (14:36.700)
Exactly.
Dawn Song (14:37.540)
That's really fascinating.
Lex Fridman (14:38.380)
And as we develop more powerful NLP
Lex Fridman (14:41.980)
and chatbot techniques,
Lex Fridman (14:43.780)
the chatbot could even engage further conversations
Dawn Song (14:47.060)
with the correspondence to,
Lex Fridman (14:48.620)
for example, if it turns out to be an attack,
Dawn Song (14:52.740)
then the chatbot can try to engage in conversations
Lex Fridman (14:57.020)
with the attacker to try to learn more information
Dawn Song (14:59.380)
from the attacker as well.
Lex Fridman (15:00.420)
So it's a very interesting area.
Lex Fridman (15:02.500)
So that chatbot is essentially
Lex Fridman (15:03.900)
your little representative in the security space.
Dawn Song (15:07.940)
It's like your little lawyer
Lex Fridman (15:09.180)
that protects you from doing anything stupid.
Dawn Song (15:11.860)
Right, right, right.
Lex Fridman (15:13.460)
That's a fascinating vision for the future.
Lex Fridman (15:17.180)
Do you see that broadly applicable across the web?
Lex Fridman (15:19.940)
So across all your interactions on the web?
Dawn Song (15:22.300)
Absolutely, right.
Lex Fridman (15:24.060)
What about like on social networks, for example?
Lex Fridman (15:26.420)
So across all of that,
Lex Fridman (15:28.500)
do you see that being implemented
Dawn Song (15:30.980)
in sort of that's a service that a company would provide
Lex Fridman (15:34.380)
or does every single social network
Lex Fridman (15:36.180)
has to implement it themselves?
Lex Fridman (15:37.460)
So Facebook and Twitter and so on,
Dawn Song (15:39.620)
or do you see there being like a security service
Lex Fridman (15:43.020)
that kind of is a plug and play?
Dawn Song (15:45.380)
That's a very good question.
Lex Fridman (15:46.460)
I think, of course, we still have ways to go
Dawn Song (15:49.900)
until the NLP and the chatbot techniques
Lex Fridman (15:53.100)
can be very effective.
Lex Fridman (15:54.860)
But I think once it's powerful enough,
Lex Fridman (15:58.500)
I do see that that can be a service
Dawn Song (16:01.220)
either a user can employ
Lex Fridman (16:02.540)
or it can be deployed by the platforms.
Dawn Song (16:04.860)
Yeah, that's just the curious side to me on security,
Lex Fridman (16:07.500)
and we'll talk about privacy,
Lex Fridman (16:09.220)
is who gets a little bit more of the control?
Lex Fridman (16:12.380)
Who gets to, you know, on whose side is the representative?
Dawn Song (16:17.140)
Is it on Facebook's side
Lex Fridman (16:19.420)
that there is this security protector,
Lex Fridman (16:22.220)
or is it on your side?
Lex Fridman (16:23.540)
And that has different implications
Dawn Song (16:25.020)
about how much that little chatbot security protector
Lex Fridman (16:30.140)
knows about you.
Dawn Song (16:31.300)
Right, exactly.
Lex Fridman (16:32.260)
If you have a little security bot
Dawn Song (16:33.660)
that you carry with you everywhere,
Lex Fridman (16:35.460)
from Facebook to Twitter to all your services,
Dawn Song (16:38.060)
it might know a lot more about you
Lex Fridman (16:40.620)
and a lot more about your relatives
Dawn Song (16:42.100)
to be able to test those things.
Lex Fridman (16:43.780)
But that's okay because you have more control of that
Dawn Song (16:47.100)
as opposed to Facebook having that.
Lex Fridman (16:48.380)
That's a really interesting trade off.
Dawn Song (16:50.580)
Another fascinating topic you work on is,
Lex Fridman (16:53.700)
again, also non traditional
Dawn Song (16:56.180)
to think of it as security vulnerability,
Lex Fridman (16:57.980)
but I guess it is adversarial machine learning,
Dawn Song (17:01.100)
is basically, again, high up the stack,
Lex Fridman (17:04.020)
being able to attack the accuracy,
Dawn Song (17:09.780)
the performance of machine learning systems
Lex Fridman (17:13.140)
by manipulating some aspect.
Dawn Song (17:15.340)
Perhaps you can clarify,
Lex Fridman (17:17.460)
but I guess the traditional way
Dawn Song (17:20.140)
the main way is to manipulate some of the input data
Lex Fridman (17:24.020)
to make the output something totally not representative
Dawn Song (17:28.180)
of the semantic content of the input.
Lex Fridman (17:30.660)
Right, so in this adversarial machine learning,
Dawn Song (17:32.860)
essentially, the goal is to fool the machine learning system
Lex Fridman (17:36.820)
into making the wrong decision.
Lex Fridman (17:38.620)
And the attack can actually happen at different stages,
Lex Fridman (17:41.180)
can happen at the inference stage
Dawn Song (17:44.540)
where the attacker can manipulate the inputs
Lex Fridman (17:46.900)
to add perturbations, malicious perturbations to the inputs
Dawn Song (17:50.660)
to cause the machine learning system
Lex Fridman (17:52.580)
to give the wrong prediction and so on.
Lex Fridman (17:55.900)
So just to pause, what are perturbations?
Lex Fridman (17:59.020)
Also essentially changes to the inputs, for example.
Dawn Song (18:01.620)
Some subtle changes, messing with the changes
Lex Fridman (18:04.340)
to try to get a very different output.
Dawn Song (18:06.180)
Right, so for example,
Lex Fridman (18:08.260)
the canonical like adversarial example type
Dawn Song (18:12.900)
is you have an image, you add really small perturbations,
Lex Fridman (18:16.980)
changes to the image.
Dawn Song (18:18.660)
It can be so subtle that to human eyes,
Lex Fridman (18:21.140)
it's hard to, it's even imperceptible to human eyes.
Lex Fridman (18:26.820)
But for the machine learning system,
Lex Fridman (18:30.980)
then the one without the perturbation,
Dawn Song (18:34.380)
the machine learning system can give the wrong,
Lex Fridman (18:36.700)
can give the correct classification, for example.
Lex Fridman (18:39.780)
But for the perturb division,
Lex Fridman (18:41.700)
the machine learning system
Dawn Song (18:42.980)
will give a completely wrong classification.
Lex Fridman (18:45.780)
And in a targeted attack,
Dawn Song (18:47.540)
the machine learning system can even give the wrong answer
Lex Fridman (18:51.860)
that's what the attacker intended.
Lex Fridman (18:55.420)
So not just any wrong answer,
Lex Fridman (18:58.620)
but like change the answer
Dawn Song (19:00.460)
to something that will benefit the attacker.
Lex Fridman (19:02.460)
Yes.
Lex Fridman (19:04.180)
So that's at the inference stage.
Lex Fridman (19:07.100)
Right, right.
Lex Fridman (19:07.940)
So yeah, what else?
Lex Fridman (19:09.540)
Right, so attacks can also happen at the training stage
Dawn Song (19:12.380)
where the attacker, for example,
Lex Fridman (19:14.100)
can provide poisoned training data sets
Dawn Song (19:19.540)
or training data points
Lex Fridman (19:21.220)
to cause the machine learning system
Dawn Song (19:22.900)
to learn the wrong model.
Lex Fridman (19:24.500)
And we also have done some work
Dawn Song (19:26.820)
showing that you can actually do this,
Lex Fridman (19:29.100)
we call it a backdoor attack,
Dawn Song (19:31.780)
whereby feeding these poisoned data points
Lex Fridman (19:36.820)
to the machine learning system.
Dawn Song (19:38.500)
The machine learning system will learn a wrong model,
Lex Fridman (19:42.340)
but it can be done in a way
Dawn Song (19:43.740)
that for most of the inputs,
Lex Fridman (19:46.460)
the learning system is fine,
Dawn Song (19:48.900)
is giving the right answer.
Lex Fridman (19:50.740)
But on specific, we call it the trigger inputs,
Dawn Song (19:54.500)
for specific inputs chosen by the attacker,
Lex Fridman (19:57.940)
it can actually, only under these situations,
Lex Fridman (1:00:03.420)
and so on.
Lex Fridman (1:00:04.260)
And all this can be collecting very sensitive information.
Lex Fridman (1:00:08.500)
And all the sensitive information gets fed
Lex Fridman (1:00:11.220)
into the learning system and trains.
Lex Fridman (1:00:13.740)
And as we know, these neural networks,
Lex Fridman (1:00:16.660)
they can have really high capacity
Lex Fridman (1:00:19.380)
and they actually can remember a lot.
Lex Fridman (1:00:23.180)
And hence just from the learning,
Dawn Song (1:00:25.300)
the learned model in the end,
Lex Fridman (1:00:27.580)
actually attackers can potentially infer information
Dawn Song (1:00:31.900)
about the original training data sets.
Lex Fridman (1:00:36.860)
So the thing you're trying to protect
Dawn Song (1:00:38.460)
that is the confidentiality of the training data.
Lex Fridman (1:00:42.820)
And so what are the methods for doing that?
Dawn Song (1:00:44.620)
Would you say, what are the different ways
Lex Fridman (1:00:46.220)
that can be done?
Lex Fridman (1:00:47.780)
And also we can talk about essentially
Lex Fridman (1:00:49.620)
how the attacker may try to learn information from the...
Dawn Song (1:00:54.620)
So, and also there are different types of attacks.
Lex Fridman (1:00:57.740)
So in certain cases, again, like in white box attacks,
Dawn Song (1:01:01.220)
we can see that the attacker actually get to see
Lex Fridman (1:01:03.860)
the parameters of the model.
Lex Fridman (1:01:05.660)
And then from that, a smart attacker potentially
Lex Fridman (1:01:08.780)
can try to figure out information
Dawn Song (1:01:11.380)
about the training data set.
Lex Fridman (1:01:13.940)
They can try to figure out what type of data
Dawn Song (1:01:16.900)
has been in the training data sets.
Lex Fridman (1:01:18.660)
And sometimes they can tell like,
Dawn Song (1:01:21.380)
whether a person has been...
Lex Fridman (1:01:23.940)
A particular person's data point has been used
Dawn Song (1:01:27.220)
in the training data sets as well.
Lex Fridman (1:01:29.060)
So white box, meaning you have access to the parameters
Dawn Song (1:01:31.940)
of say a neural network.
Lex Fridman (1:01:33.540)
And so that you're saying that it's some...
Dawn Song (1:01:36.580)
Given that information is possible to some...
Lex Fridman (1:01:38.860)
So I can give you some examples.
Lex Fridman (1:01:40.380)
And then another type of attack,
Lex Fridman (1:01:41.780)
which is even easier to carry out is not a white box model.
Dawn Song (1:01:46.180)
It's more of just a query model where the attacker
Lex Fridman (1:01:49.900)
only gets to query the machine learning model
Lex Fridman (1:01:52.580)
and then try to steal sensitive information
Lex Fridman (1:01:55.340)
in the original training data.
Dawn Song (1:01:57.020)
So, right, so I can give you an example.
Lex Fridman (1:02:00.580)
In this case, training a language model.
Lex Fridman (1:02:03.700)
So in our work, in collaboration
Lex Fridman (1:02:06.300)
with the researchers from Google,
Dawn Song (1:02:08.100)
we actually studied the following question.
Lex Fridman (1:02:10.660)
So at high level, the question is,
Dawn Song (1:02:13.620)
as we mentioned, the neural networks
Lex Fridman (1:02:15.900)
can have very high capacity and they could be remembering
Dawn Song (1:02:18.860)
a lot from the training process.
Lex Fridman (1:02:21.620)
Then the question is, can attacker actually exploit this
Lex Fridman (1:02:25.500)
and try to actually extract sensitive information
Lex Fridman (1:02:28.660)
in the original training data sets
Dawn Song (1:02:31.140)
through just querying the learned model
Lex Fridman (1:02:34.220)
without even knowing the parameters of the model,
Dawn Song (1:02:37.140)
like the details of the model
Lex Fridman (1:02:38.780)
or the architectures of the model and so on.
Lex Fridman (1:02:41.900)
So that's a question we set out to explore.
Lex Fridman (1:02:46.860)
And in one of the case studies, we showed the following.
Lex Fridman (1:02:50.860)
So we trained a language model over an email data set.
Lex Fridman (1:02:55.060)
It's called an Enron email data set.
Lex Fridman (1:02:57.420)
And the Enron email data sets naturally contained
Lex Fridman (1:03:01.180)
users social security numbers and credit card numbers.
Lex Fridman (1:03:05.500)
So we trained a language model over the data sets
Lex Fridman (1:03:08.500)
and then we showed that an attacker
Dawn Song (1:03:11.180)
by devising some new attacks
Lex Fridman (1:03:13.220)
by just querying the language model
Lex Fridman (1:03:15.940)
and without knowing the details of the model,
Lex Fridman (1:03:19.140)
the attacker actually can extract
Dawn Song (1:03:23.020)
the original social security numbers and credit card numbers
Lex Fridman (1:03:26.980)
that were in the original training data sets.
Lex Fridman (1:03:30.300)
So get the most sensitive personally identifiable information
Lex Fridman (1:03:33.300)
from the data set from just querying it.
Dawn Song (1:03:38.340)
Right, yeah.
Lex Fridman (1:03:39.260)
So that's an example showing that's why
Dawn Song (1:03:42.820)
even as we train machine learning models,
Lex Fridman (1:03:45.940)
we have to be really careful
Dawn Song (1:03:48.300)
with protecting users data privacy.
Lex Fridman (1:03:51.580)
So what are the mechanisms for protecting?
Lex Fridman (1:03:53.740)
Is there hopeful?
Lex Fridman (1:03:55.740)
So there's been recent work on differential privacy,
Dawn Song (1:03:58.940)
for example, that provides some hope,
Lex Fridman (1:04:02.660)
but can you describe some of the ideas?
Dawn Song (1:04:04.460)
Right, so that's actually, right.
Lex Fridman (1:04:05.580)
So that's also our finding is that by actually,
Dawn Song (1:04:09.780)
we show that in this particular case,
Lex Fridman (1:04:12.500)
we actually have a good defense.
Dawn Song (1:04:14.300)
For the querying case, for the language model case.
Lex Fridman (1:04:17.820)
So instead of just training a vanilla language model,
Dawn Song (1:04:23.020)
instead, if we train a differentially private language model,
Lex Fridman (1:04:26.620)
then we can still achieve similar utility,
Lex Fridman (1:04:31.100)
but at the same time, we can actually significantly enhance
Lex Fridman (1:04:34.580)
the privacy protection of the learned model.
Lex Fridman (1:04:39.420)
And our proposed attacks actually are no longer effective.
Lex Fridman (1:04:44.020)
And differential privacy is a mechanism
Dawn Song (1:04:47.180)
of adding some noise,
Lex Fridman (1:04:49.100)
by which you then have some guarantees on the inability
Dawn Song (1:04:52.620)
to figure out the presence of a particular person
Lex Fridman (1:04:58.820)
in the dataset.
Lex Fridman (1:04:59.860)
So right, so in this particular case,
Lex Fridman (1:05:01.860)
what the differential privacy mechanism does
Dawn Song (1:05:05.500)
is that it actually adds perturbation
Lex Fridman (1:05:09.500)
in the training process.
Dawn Song (1:05:10.700)
As we know, during the training process,
Lex Fridman (1:05:12.980)
we are learning the model, we are doing gradient updates,
Dawn Song (1:05:16.860)
the weight updates and so on.
Lex Fridman (1:05:19.020)
And essentially, differential privacy,
Dawn Song (1:05:22.620)
a differentially private machine learning algorithm
Lex Fridman (1:05:26.340)
in this case, will be adding noise
Lex Fridman (1:05:29.660)
and adding various perturbation during this training process.
Lex Fridman (1:05:33.860)
To some aspect of the training process.
Dawn Song (1:05:35.780)
Right, so then the finally trained learning,
Lex Fridman (1:05:39.660)
the learned model is differentially private,
Lex Fridman (1:05:42.500)
and so it can enhance the privacy protection.
Lex Fridman (1:05:46.660)
So okay, so that's the attacks and the defense of privacy.
Dawn Song (1:05:51.420)
You also talk about ownership of data.
Lex Fridman (1:05:54.340)
So this is a really interesting idea
Dawn Song (1:05:56.580)
that we get to use many services online
Lex Fridman (1:05:59.060)
for seemingly for free by essentially,
Dawn Song (1:06:04.100)
sort of a lot of companies are funded through advertisement.
Lex Fridman (1:06:06.820)
And what that means is the advertisement works
Dawn Song (1:06:09.820)
exceptionally well because the companies are able
Lex Fridman (1:06:12.060)
to access our personal data,
Lex Fridman (1:06:13.700)
so they know which advertisement to service
Lex Fridman (1:06:16.260)
to do targeted advertisements and so on.
Lex Fridman (1:06:18.980)
So can you maybe talk about this?
Lex Fridman (1:06:21.860)
You have some nice paintings of the future,
Dawn Song (1:06:26.220)
philosophically speaking future
Lex Fridman (1:06:28.580)
where people can have a little bit more control
Dawn Song (1:06:31.780)
of their data by owning
Lex Fridman (1:06:33.140)
and maybe understanding the value of their data
Lex Fridman (1:06:36.900)
and being able to sort of monetize it
Lex Fridman (1:06:40.500)
in a more explicit way as opposed to the implicit way
Dawn Song (1:06:43.460)
that it's currently done.
Lex Fridman (1:06:45.100)
Yeah, I think this is a fascinating topic
Lex Fridman (1:06:47.420)
and also a really complex topic.
Lex Fridman (1:06:51.100)
Right, I think there are these natural questions,
Lex Fridman (1:06:53.860)
who should be owning the data?
Lex Fridman (1:06:58.620)
And so I can draw one analogy.
Lex Fridman (1:07:03.220)
So for example, for physical properties,
Lex Fridman (1:07:06.820)
like your house and so on.
Lex Fridman (1:07:08.340)
So really this notion of property rights
Lex Fridman (1:07:13.180)
it's not like from day one,
Dawn Song (1:07:17.220)
we knew that there should be like this clear notion
Lex Fridman (1:07:20.620)
of ownership of properties and having enforcement for this.
Lex Fridman (1:07:25.420)
And so actually people have shown
Lex Fridman (1:07:29.180)
that this establishment and enforcement of property rights
Dawn Song (1:07:34.180)
has been a main driver for the economy earlier.
Lex Fridman (1:07:42.180)
And that actually really propelled the economic growth
Dawn Song (1:07:47.180)
even in the earlier stage.
Lex Fridman (1:07:50.420)
So throughout the history of the development
Dawn Song (1:07:53.020)
of the United States or actually just civilization,
Lex Fridman (1:07:56.180)
the idea of property rights that you can own property.
Dawn Song (1:07:59.620)
Right, and then there's enforcement.
Lex Fridman (1:08:01.340)
There's institutional rights,
Dawn Song (1:08:04.540)
that governmental like enforcements of this
Lex Fridman (1:08:07.740)
actually has been a key driver for economic growth.
Lex Fridman (1:08:12.020)
And there had been even research or proposals saying
Lex Fridman (1:08:16.420)
that for a lot of the developing countries,
Dawn Song (1:08:22.540)
essentially the challenge in growth
Lex Fridman (1:08:25.100)
is not actually due to the lack of capital.
Dawn Song (1:08:28.940)
It's more due to the lack of this notion of property rights
Lex Fridman (1:08:34.500)
and the enforcement of property rights.
Dawn Song (1:08:37.060)
Interesting, so that the presence of absence
Lex Fridman (1:08:41.580)
of both the concept of the property rights
Lex Fridman (1:08:45.060)
and their enforcement has a strong correlation
Lex Fridman (1:08:48.100)
to economic growth.
Dawn Song (1:08:49.820)
Right, right.
Lex Fridman (1:08:50.740)
And so you think that that same could be transferred
Dawn Song (1:08:54.100)
to the idea of property ownership
Lex Fridman (1:08:56.220)
in the case of data ownership.
Dawn Song (1:08:57.860)
I think first of all, it's a good lesson for us
Lex Fridman (1:09:01.260)
to recognize that these rights and the recognition
Lex Fridman (1:09:06.540)
and the enforcements of these type of rights
Lex Fridman (1:09:10.020)
is very, very important for economic growth.
Lex Fridman (1:09:13.220)
And then if we look at where we are now
Lex Fridman (1:09:15.740)
and where we are going in the future,
Lex Fridman (1:09:18.460)
so essentially more and more
Lex Fridman (1:09:19.780)
is actually moving into the digital world.
Lex Fridman (1:09:23.540)
And also more and more, I would say,
Lex Fridman (1:09:26.260)
even information or assets of a person
Dawn Song (1:09:30.380)
is more and more into the real world,
Lex Fridman (1:09:33.180)
the physical, sorry, the digital world as well.
Dawn Song (1:09:35.780)
It's the data that the person has generated.
Lex Fridman (1:09:39.900)
And essentially it's like in the past
Lex Fridman (1:09:43.020)
what defines a person, you can say,
Lex Fridman (1:09:45.860)
right, like oftentimes besides the innate capabilities,
Dawn Song (1:09:50.940)
actually it's the physical properties.
Lex Fridman (1:09:54.260)
House, car.
Dawn Song (1:09:55.300)
Right, that defines a person.
Lex Fridman (1:09:56.740)
But I think more and more people start to realize
Dawn Song (1:09:59.540)
actually what defines a person
Lex Fridman (1:10:01.420)
is more important in the data
Dawn Song (1:10:03.020)
that the person has generated
Lex Fridman (1:10:04.860)
or the data about the person.
Dawn Song (1:10:07.540)
Like all the way from your political views,
Lex Fridman (1:10:10.500)
your music taste and your financial information,
Dawn Song (1:10:14.980)
a lot of these and your health.
Lex Fridman (1:10:16.820)
So more and more of the definition of the person
Dawn Song (1:10:20.140)
is actually in the digital world.
Lex Fridman (1:10:22.100)
And currently for the most part, that's owned implicitly.
Dawn Song (1:10:26.220)
People don't talk about it,
Lex Fridman (1:10:27.300)
but kind of it's owned by internet companies.
Lex Fridman (1:10:33.340)
So it's not owned by individuals.
Lex Fridman (1:10:34.580)
Right, there's no clear notion of ownership of such data.
Lex Fridman (1:10:39.060)
And also we talk about privacy and so on,
Lex Fridman (1:10:41.820)
but I think actually clearly identifying the ownership
Dawn Song (1:10:45.540)
is a first step.
Lex Fridman (1:10:46.580)
Once you identify the ownership,
Dawn Song (1:10:48.300)
then you can say who gets to define
Lex Fridman (1:10:50.660)
how the data should be used.
Lex Fridman (1:10:52.300)
So maybe some users are fine with internet companies
Lex Fridman (1:10:57.580)
serving them as, right, using their data
Dawn Song (1:11:02.020)
as long as if the data is used in a certain way
Lex Fridman (1:11:05.740)
that actually the user consents with or allows.
Dawn Song (1:11:11.460)
For example, you can see the recommendation system
Lex Fridman (1:11:14.460)
in some sense, we don't call it as,
Lex Fridman (1:11:16.700)
but a recommendation system,
Lex Fridman (1:11:18.340)
similarly it's trying to recommend you something
Lex Fridman (1:11:20.740)
and users enjoy and can really benefit
Lex Fridman (1:11:23.980)
from good recommendation systems,
Dawn Song (1:11:25.620)
either recommending you better music, movies, news,
Lex Fridman (1:11:29.340)
even research papers to read.
Lex Fridman (1:11:32.700)
But of course then in these targeted ads,
Lex Fridman (1:11:35.780)
especially in certain cases where people can be manipulated
Dawn Song (1:11:40.420)
by these targeted ads that can have really bad,
Lex Fridman (1:11:44.140)
like severe consequences.
Lex Fridman (1:11:45.700)
So essentially users want their data to be used
Lex Fridman (1:11:50.340)
to better serve them and also maybe even, right,
Dawn Song (1:11:53.380)
get paid for or whatever, like in different settings.
Lex Fridman (1:11:56.340)
But the thing is that first of all,
Dawn Song (1:11:57.740)
we need to really establish like who needs to decide,
Lex Fridman (1:12:03.020)
who can decide how the data should be used.
Lex Fridman (1:12:06.180)
And typically the establishment and clarification
Lex Fridman (1:12:10.060)
of the ownership will help this
Lex Fridman (1:12:12.100)
and it's an important first step.
Lex Fridman (1:12:14.660)
So if the user is the owner,
Dawn Song (1:12:16.260)
then naturally the user gets to define
Lex Fridman (1:12:18.340)
how the data should be used.
Lex Fridman (1:12:19.940)
But if you even say that wait a minute,
Lex Fridman (1:12:22.580)
users are actually now the owner of this data,
Dawn Song (1:12:24.420)
whoever is collecting the data is the owner of the data.
Lex Fridman (1:12:26.700)
Now of course they get to use the data
Dawn Song (1:12:28.180)
however way they want.
Lex Fridman (1:12:29.940)
So to really address these complex issues,
Dawn Song (1:12:33.900)
we need to go at the root cause.
Lex Fridman (1:12:35.940)
So it seems fairly clear that so first we really need to say
Dawn Song (1:12:41.100)
that who is the owner of the data
Lex Fridman (1:12:42.540)
and then the owners can specify
Lex Fridman (1:12:45.100)
how they want their data to be utilized.
Lex Fridman (1:12:47.140)
So that's a fascinating,
Dawn Song (1:12:50.980)
most people don't think about that
Lex Fridman (1:12:52.620)
and I think that's a fascinating thing to think about
Lex Fridman (1:12:54.940)
and probably fight for it.
Lex Fridman (1:12:57.140)
I can only see in the economic growth argument,
Dawn Song (1:12:59.620)
it's probably a really strong one.
Lex Fridman (1:13:01.020)
So that's a first time I'm kind of at least thinking
Dawn Song (1:13:04.220)
about the positive aspect of that ownership
Lex Fridman (1:13:08.100)
being the longterm growth of the economy,
Lex Fridman (1:13:11.220)
so good for everybody.
Lex Fridman (1:13:12.260)
But sort of one down possible downside I could see
Dawn Song (1:13:15.500)
sort of to put on my grumpy old grandpa hat
Lex Fridman (1:13:21.500)
and it's really nice for Facebook and YouTube and Twitter
Dawn Song (1:13:25.860)
to all be free.
Lex Fridman (1:13:28.020)
And if you give control to people or their data,
Lex Fridman (1:13:31.660)
do you think it's possible they will be,
Lex Fridman (1:13:34.780)
they would not want to hand it over quite easily?
Lex Fridman (1:13:37.620)
And so a lot of these companies that rely on mass handover
Lex Fridman (1:13:42.220)
of data and then therefore provide a mass
Dawn Song (1:13:46.900)
seemingly free service would then completely,
Lex Fridman (1:13:51.020)
so the way the internet looks will completely change
Dawn Song (1:13:56.100)
because of the ownership of data
Lex Fridman (1:13:57.660)
and we'll lose a lot of services value.
Lex Fridman (1:14:00.700)
Do you worry about that?
Lex Fridman (1:14:02.340)
That's a very good question.
Dawn Song (1:14:03.740)
I think that's not necessarily the case
Lex Fridman (1:14:06.060)
in the sense that yes, users can have ownership
Dawn Song (1:14:10.060)
of their data, they can maintain control of their data,
Lex Fridman (1:14:12.860)
but also then they get to decide how their data can be used.
Lex Fridman (1:14:17.500)
So that's why I mentioned earlier,
Lex Fridman (1:14:19.900)
so in this case, if they feel that they enjoy the benefits
Dawn Song (1:14:23.500)
of social networks and so on,
Lex Fridman (1:14:25.460)
and they're fine with having Facebook, having their data,
Lex Fridman (1:14:29.540)
but utilizing the data in certain way that they agree,
Lex Fridman (1:14:33.940)
then they can still enjoy the free services.
Lex Fridman (1:14:37.220)
But for others, maybe they would prefer
Lex Fridman (1:14:40.020)
some kind of private vision.
Lex Fridman (1:14:41.980)
And in that case, maybe they can even opt in
Lex Fridman (1:14:44.540)
to say that I want to pay and to have,
Lex Fridman (1:14:47.860)
so for example, it's already fairly standard,
Lex Fridman (1:14:50.780)
like you pay for certain subscriptions
Lex Fridman (1:14:53.460)
so that you don't get to be shown ads, right?
Lex Fridman (1:14:59.140)
So then users essentially can have choices.
Lex Fridman (1:15:01.980)
And I think we just want to essentially bring out
Lex Fridman (1:15:06.300)
more about who gets to decide what to do with that data.
Dawn Song (1:15:10.820)
I think it's an interesting idea,
Lex Fridman (1:15:11.940)
because if you poll people now,
Dawn Song (1:15:15.140)
it seems like, I don't know,
Lex Fridman (1:15:16.780)
but subjectively, sort of anecdotally speaking,
Dawn Song (1:15:19.140)
it seems like a lot of people don't trust Facebook.
Lex Fridman (1:15:22.100)
So that's at least a very popular thing to say
Lex Fridman (1:15:24.380)
that I don't trust Facebook, right?
Lex Fridman (1:15:26.940)
I wonder if you give people control of their data
Dawn Song (1:15:30.460)
as opposed to sort of signaling to everyone
Lex Fridman (1:15:33.140)
that they don't trust Facebook,
Dawn Song (1:15:34.860)
I wonder how they would speak with the actual,
Lex Fridman (1:15:37.900)
like would they be willing to pay $10 a month for Facebook
Lex Fridman (1:15:42.460)
or would they hand over their data?
Lex Fridman (1:15:44.860)
It'd be interesting to see what fraction of people
Dawn Song (1:15:47.500)
would quietly hand over their data to Facebook
Lex Fridman (1:15:51.300)
to make it free.
Dawn Song (1:15:52.620)
I don't have a good intuition about that.
Lex Fridman (1:15:54.860)
Like how many people, do you have an intuition
Dawn Song (1:15:57.580)
about how many people would use their data effectively
Lex Fridman (1:16:01.540)
on the market of the internet
Lex Fridman (1:16:06.540)
by sort of buying services with their data?
Lex Fridman (1:16:10.860)
Yeah, so that's a very good question.
Dawn Song (1:16:12.380)
I think, so one thing I also want to mention
Lex Fridman (1:16:15.900)
is that this, right, so it seems that especially in press,
Dawn Song (1:16:22.780)
the conversation has been very much like
Lex Fridman (1:16:26.020)
two sides fighting against each other.
Dawn Song (1:16:29.100)
On one hand, right, users can say that, right,
Lex Fridman (1:16:33.500)
they don't trust Facebook, they don't,
Dawn Song (1:16:35.420)
or they delete Facebook.
Lex Fridman (1:16:37.580)
Yeah, exactly.
Dawn Song (1:16:39.140)
Right, and then on the other hand, right, of course,
Lex Fridman (1:16:45.940)
right, the other side, they also feel,
Dawn Song (1:16:48.220)
oh, they are providing a lot of services to users
Lex Fridman (1:16:50.700)
and users are getting it all for free.
Lex Fridman (1:16:53.780)
So I think I actually, I don't know,
Lex Fridman (1:16:57.580)
I talk a lot to like different companies
Lex Fridman (1:17:00.700)
and also like basically on both sides.
Lex Fridman (1:17:04.820)
So one thing I hope also like,
Dawn Song (1:17:07.660)
this is my hope for this year also,
Lex Fridman (1:17:09.180)
is that we want to establish a more constructive dialogue
Lex Fridman (1:17:16.820)
and to help people to understand
Lex Fridman (1:17:18.660)
that the problem is much more nuanced
Dawn Song (1:17:21.860)
than just this two sides fighting.
Lex Fridman (1:17:25.500)
Because naturally, there is a tension between the two sides,
Dawn Song (1:17:30.820)
between utility and privacy.
Lex Fridman (1:17:33.460)
So if you want to get more utility, essentially,
Dawn Song (1:17:36.300)
like the recommendation system example I gave earlier,
Lex Fridman (1:17:40.620)
if you want someone to give you a good recommendation,
Dawn Song (1:17:43.500)
essentially, whatever that system is,
Lex Fridman (1:17:45.220)
the system is going to need to know your data
Dawn Song (1:17:48.580)
to give you a good recommendation.
Lex Fridman (1:17:52.020)
But also, of course, at the same time,
Dawn Song (1:17:53.820)
we want to ensure that however that data is being handled,
Lex Fridman (1:17:56.660)
it's done in a privacy preserving way.
Lex Fridman (1:17:59.500)
So that, for example, the recommendation system
Lex Fridman (1:18:02.460)
doesn't just go around and sell your data
Lex Fridman (1:18:05.500)
and then cause a lot of bad consequences and so on.
Lex Fridman (1:18:12.580)
So you want that dialogue to be a little bit more
Dawn Song (1:18:15.020)
in the open, a little more nuanced,
Lex Fridman (1:18:18.220)
and maybe adding control to the data,
Dawn Song (1:18:20.700)
ownership to the data will allow,
Lex Fridman (1:18:24.020)
as opposed to this happening in the background,
Dawn Song (1:18:26.220)
allow to bring it to the forefront
Lex Fridman (1:18:28.100)
and actually have dialogues, like more nuanced,
Dawn Song (1:18:32.300)
real dialogues about how we trade our data for the services.
Lex Fridman (1:18:37.300)
That's the hope.
Dawn Song (1:18:38.140)
Right, right, yes, at the high level.
Lex Fridman (1:18:41.020)
So essentially, also knowing that there are
Dawn Song (1:18:42.980)
technical challenges in addressing the issue,
Lex Fridman (1:18:47.980)
like basically you can't have,
Dawn Song (1:18:50.300)
just like the example that I gave earlier,
Lex Fridman (1:18:53.260)
it's really difficult to balance the two
Dawn Song (1:18:55.580)
between utility and privacy.
Lex Fridman (1:18:57.460)
And that's also a lot of things that I work on,
Dawn Song (1:19:01.980)
my group works on as well,
Lex Fridman (1:19:03.860)
is to actually develop these technologies that are needed
Dawn Song (1:19:08.860)
to essentially help this balance better,
Lex Fridman (1:19:12.220)
essentially to help data to be utilized
Dawn Song (1:19:14.660)
in a privacy preserving way.
Lex Fridman (1:19:16.420)
And so we essentially need people to understand
Dawn Song (1:19:19.340)
the challenges and also at the same time
Lex Fridman (1:19:22.300)
to provide the technical abilities
Lex Fridman (1:19:26.180)
and also regulatory frameworks to help the two sides
Lex Fridman (1:19:29.540)
to be more in a win win situation instead of a fight.
Dawn Song (1:19:33.020)
Yeah, the fighting thing is,
Lex Fridman (1:19:36.980)
I think YouTube and Twitter and Facebook
Dawn Song (1:19:38.740)
are providing an incredible service to the world
Lex Fridman (1:19:41.460)
and they're all making a lot of money
Lex Fridman (1:19:44.260)
and they're all making mistakes, of course,
Lex Fridman (1:19:47.460)
but they're doing an incredible job
Dawn Song (1:19:50.740)
that I think deserves to be applauded
Lex Fridman (1:19:53.500)
and there's some degree of,
Dawn Song (1:19:55.580)
like it's a cool thing that's created
Lex Fridman (1:19:59.260)
and it shouldn't be monolithically fought against,
Dawn Song (1:20:04.340)
like Facebook is evil or so on.
Lex Fridman (1:20:06.540)
Yeah, it might make mistakes,
Lex Fridman (1:20:07.980)
but I think it's an incredible service.
Lex Fridman (1:20:10.100)
I think it's world changing.
Dawn Song (1:20:12.420)
I mean, I think Facebook's done a lot of incredible,
Lex Fridman (1:20:16.620)
incredible things by bringing, for example, identity.
Dawn Song (1:20:20.900)
Like allowing people to be themselves,
Lex Fridman (1:20:25.220)
like their real selves in the digital space
Dawn Song (1:20:28.660)
by using their real name and their real picture.
Lex Fridman (1:20:31.620)
That step was like the first step from the real world
Dawn Song (1:20:34.220)
to the digital world.
Lex Fridman (1:20:35.700)
That was a huge step that perhaps will define
Dawn Song (1:20:38.020)
the 21st century in us creating a digital identity.
Lex Fridman (1:20:41.580)
And there's a lot of interesting possibilities there
Dawn Song (1:20:44.180)
that are positive.
Lex Fridman (1:20:45.260)
Of course, some things that are negative
Lex Fridman (1:20:47.900)
and having a good dialogue about that is great.
Lex Fridman (1:20:50.100)
And I'm great that people like you
Dawn Song (1:20:51.660)
are at the center of that dialogue, so that's awesome.
Lex Fridman (1:20:54.180)
Right, I think also, I also can understand.
Dawn Song (1:20:58.500)
I think actually in the past,
Lex Fridman (1:21:00.780)
especially in the past couple of years,
Dawn Song (1:21:03.740)
this rising awareness has been helpful.
Lex Fridman (1:21:07.540)
Like users are also more and more recognizing
Dawn Song (1:21:10.220)
that privacy is important to them.
Lex Fridman (1:21:12.020)
They should, maybe, right,
Dawn Song (1:21:14.460)
they should be owners of their data.
Lex Fridman (1:21:15.860)
I think this definitely is very helpful.
Lex Fridman (1:21:18.260)
And I think also this type of voice also,
Lex Fridman (1:21:23.540)
and together with the regulatory framework and so on,
Dawn Song (1:21:27.260)
also help the companies to essentially put
Lex Fridman (1:21:31.220)
these type of issues at a higher priority.
Lex Fridman (1:21:33.940)
And knowing that, right, also it is their responsibility too
Lex Fridman (1:21:38.940)
to ensure that users are well protected.
Lex Fridman (1:21:42.860)
So I think definitely the rising voice is super helpful.
Lex Fridman (1:21:47.260)
And I think that actually really has brought
Dawn Song (1:21:50.420)
the issue of data privacy
Lex Fridman (1:21:52.660)
and even this consideration of data ownership
Dawn Song (1:21:55.740)
to the forefront to really much wider community.
Lex Fridman (1:22:00.860)
And I think more of this voice is needed,
Lex Fridman (1:22:03.140)
but I think it's just that we want to have
Lex Fridman (1:22:05.140)
a more constructive dialogue to bring the both sides together
Dawn Song (1:22:10.020)
to figure out a constructive solution.
Lex Fridman (1:22:13.740)
So another interesting space
Dawn Song (1:22:15.180)
where security is really important
Lex Fridman (1:22:16.620)
is in the space of any kinds of transactions,
Lex Fridman (1:22:20.820)
but it could be also digital currency.
Lex Fridman (1:22:22.940)
So can you maybe talk a little bit about blockchain?
Lex Fridman (1:22:27.860)
And can you tell me what is a blockchain?
Lex Fridman (1:22:30.060)
Blockchain.
Dawn Song (1:22:32.900)
I think the blockchain word itself
Lex Fridman (1:22:34.940)
is actually very overloaded.
Dawn Song (1:22:37.580)
Of course.
Lex Fridman (1:22:38.420)
In general.
Dawn Song (1:22:39.260)
It's like AI.
Lex Fridman (1:22:40.100)
Right, yes.
Lex Fridman (1:22:42.020)
So in general, when we talk about blockchain,
Lex Fridman (1:22:43.340)
we refer to this distributor in a decentralized fashion.
Lex Fridman (1:22:47.780)
So essentially you have a community of nodes
Lex Fridman (1:22:53.460)
that come together.
Lex Fridman (1:22:54.860)
And even though each one may not be trusted,
Lex Fridman (1:22:59.180)
and as long as a certain thresholds
Dawn Song (1:23:02.620)
of the set of nodes behaves properly,
Lex Fridman (1:23:07.580)
then the system can essentially achieve certain properties.
Dawn Song (1:23:11.820)
For example, in the distributed ledger setting,
Lex Fridman (1:23:15.580)
you can maintain an immutable log
Lex Fridman (1:23:18.540)
and you can ensure that, for example,
Lex Fridman (1:23:22.940)
the transactions actually are agreed upon
Lex Fridman (1:23:25.540)
and then it's immutable and so on.
Lex Fridman (1:23:28.260)
So first of all, what's a ledger?
Lex Fridman (1:23:29.740)
So it's a...
Lex Fridman (1:23:30.740)
It's like a database.
Dawn Song (1:23:31.740)
It's like a data entry.
Lex Fridman (1:23:33.660)
And so a distributed ledger
Dawn Song (1:23:35.140)
is something that's maintained across
Lex Fridman (1:23:37.900)
or is synchronized across multiple sources, multiple nodes.
Dawn Song (1:23:41.700)
Multiple nodes, yes.
Lex Fridman (1:23:43.340)
And so where is this idea?
Lex Fridman (1:23:46.060)
How do you keep...
Lex Fridman (1:23:48.420)
So it's important, a ledger, a database,
Dawn Song (1:23:51.420)
to keep that, to make sure...
Lex Fridman (1:23:55.580)
So what are the kinds of security vulnerabilities
Dawn Song (1:23:58.740)
that you're trying to protect against
Lex Fridman (1:24:01.540)
in the context of a distributed ledger?
Lex Fridman (1:24:04.460)
So in this case, for example,
Lex Fridman (1:24:06.300)
you don't want some malicious nodes
Dawn Song (1:24:09.100)
to be able to change the transaction logs.
Lex Fridman (1:24:12.860)
And in certain cases, it's called double spending,
Dawn Song (1:24:15.700)
like you can also cause different views
Lex Fridman (1:24:19.820)
in different parts of the network and so on.
Lex Fridman (1:24:22.820)
So the ledger has to represent,
Lex Fridman (1:24:24.500)
if you're capturing financial transactions,
Dawn Song (1:24:27.580)
it has to represent the exact timing
Lex Fridman (1:24:29.460)
and the exact occurrence and no duplicates,
Dawn Song (1:24:32.420)
all that kind of stuff.
Lex Fridman (1:24:33.380)
It has to represent what actually happened.
Dawn Song (1:24:37.100)
Okay, so what are your thoughts
Lex Fridman (1:24:40.540)
on the security and privacy of digital currency?
Dawn Song (1:24:43.820)
I can't tell you how many people write to me
Lex Fridman (1:24:47.340)
to interview various people in the digital currency space.
Dawn Song (1:24:51.660)
There seems to be a lot of excitement there.
Lex Fridman (1:24:54.940)
And it seems to be, some of it's, to me,
Dawn Song (1:24:57.980)
from an outsider's perspective, seems like dark magic.
Lex Fridman (1:25:01.860)
I don't know how secure...
Dawn Song (1:25:06.020)
I think the foundation, from my perspective,
Lex Fridman (1:25:08.900)
of digital currencies, that is, you can't trust anyone.
Lex Fridman (1:25:13.460)
So you have to create a really secure system.
Lex Fridman (1:25:16.340)
So can you maybe speak about how,
Lex Fridman (1:25:19.860)
what your thoughts in general about digital currency is
Lex Fridman (1:25:22.060)
and how we can possibly create financial transactions
Lex Fridman (1:25:26.940)
and financial stores of money in the digital space?
Lex Fridman (1:25:31.740)
So you asked about security and privacy.
Lex Fridman (1:25:35.220)
So again, as I mentioned earlier,
Lex Fridman (1:25:37.580)
in security, we actually talk about two main properties,
Dawn Song (1:25:42.020)
the integrity and confidentiality.
Lex Fridman (1:25:45.860)
So there's another one for availability.
Dawn Song (1:25:49.020)
You want the system to be available.
Lex Fridman (1:25:50.660)
But here, for the question you asked,
Dawn Song (1:25:52.740)
let's just focus on integrity and confidentiality.
Lex Fridman (1:25:57.100)
So for integrity of this distributed ledger,
Dawn Song (1:26:00.540)
essentially, as we discussed,
Lex Fridman (1:26:01.980)
we want to ensure that the different nodes,
Lex Fridman (1:26:06.860)
so they have this consistent view,
Lex Fridman (1:26:08.580)
usually it's done through what we call a consensus protocol,
Lex Fridman (1:26:13.140)
and that they establish this shared view on this ledger,
Lex Fridman (1:26:18.140)
and that you cannot go back and change,
Dawn Song (1:26:21.900)
it's immutable, and so on.
Lex Fridman (1:26:25.260)
So in this case, then the security often refers
Dawn Song (1:26:28.700)
to this integrity property.
Lex Fridman (1:26:31.820)
And essentially, you're asking the question,
Lex Fridman (1:26:34.660)
how much work, how can you attack the system
Lex Fridman (1:26:38.860)
so that the attacker can change the lock, for example?
Dawn Song (1:26:43.860)
Change the lock, for example.
Lex Fridman (1:26:46.220)
Right, how hard is it to make an attack like that?
Dawn Song (1:26:48.540)
Right, right.
Lex Fridman (1:26:49.460)
And then that very much depends on the consensus mechanism,
Lex Fridman (1:26:55.180)
how the system is built, and all that.
Lex Fridman (1:26:57.580)
So there are different ways
Dawn Song (1:26:59.140)
to build these decentralized systems.
Lex Fridman (1:27:02.860)
And people may have heard about the terms called
Dawn Song (1:27:05.660)
like proof of work, proof of stake,
Lex Fridman (1:27:07.860)
these different mechanisms.
Lex Fridman (1:27:09.700)
And it really depends on how the system has been built,
Lex Fridman (1:27:14.420)
and also how much resources,
Lex Fridman (1:27:17.820)
how much work has gone into the network
Lex Fridman (1:27:20.500)
to actually say how secure it is.
Lex Fridman (1:27:24.460)
So for example, people talk about like,
Lex Fridman (1:27:26.660)
in Bitcoin, it's proof of work system,
Lex Fridman (1:27:28.860)
so much electricity has been burned.
Lex Fridman (1:27:32.060)
So there's differences in the different mechanisms
Lex Fridman (1:27:35.300)
and the implementations of a distributed ledger
Lex Fridman (1:27:37.940)
used for digital currency.
Lex Fridman (1:27:40.060)
So there's Bitcoin, there's whatever,
Lex Fridman (1:27:42.380)
there's so many of them,
Lex Fridman (1:27:43.300)
and there's underlying different mechanisms.
Lex Fridman (1:27:46.020)
And there's arguments, I suppose,
Dawn Song (1:27:48.420)
about which is more effective, which is more secure,
Lex Fridman (1:27:51.620)
which is more.
Lex Fridman (1:27:52.940)
And what is needed,
Lex Fridman (1:27:54.940)
what amount of resources needed
Lex Fridman (1:27:56.980)
to be able to attack the system?
Lex Fridman (1:28:00.300)
Like for example, what percentage of the nodes
Lex Fridman (1:28:02.860)
do you need to control or compromise
Lex Fridman (1:28:06.220)
in order to, right, to change the log?
Lex Fridman (1:28:09.980)
And those are things, do you have a sense
Lex Fridman (1:28:12.860)
if those are things that can be shown theoretically
Dawn Song (1:28:15.460)
through the design of the mechanisms,
Lex Fridman (1:28:17.580)
or does it have to be shown empirically
Lex Fridman (1:28:19.220)
by having a large number of users using the currency?
Lex Fridman (1:28:23.540)
I see.
Lex Fridman (1:28:24.380)
So in general, for each consensus mechanism,
Lex Fridman (1:28:27.020)
you can actually show theoretically
Lex Fridman (1:28:30.180)
what is needed to be able to attack the system.
Lex Fridman (1:28:34.420)
Of course, there can be different types of attacks
Dawn Song (1:28:37.940)
as we discussed at the beginning.
Lex Fridman (1:28:41.180)
And so that it's difficult to give
Dawn Song (1:28:46.980)
like, you know, complete estimates,
Lex Fridman (1:28:50.100)
like really how much is needed to compromise the system.
Lex Fridman (1:28:55.340)
But in general, right, so there are ways to say
Lex Fridman (1:28:57.700)
what percentage of the nodes you need to compromise
Lex Fridman (1:29:01.660)
and so on.
Lex Fridman (1:29:03.140)
So we talked about integrity on the security side,
Lex Fridman (1:29:07.460)
and then you also mentioned the privacy
Lex Fridman (1:29:11.180)
or the confidentiality side.
Dawn Song (1:29:13.460)
Does it have some of the same problems
Lex Fridman (1:29:17.780)
and therefore some of the same solutions
Dawn Song (1:29:19.420)
that you talked about on the machine learning side
Lex Fridman (1:29:21.500)
with differential privacy and so on?
Dawn Song (1:29:24.180)
Yeah, so actually in general on the public ledger
Lex Fridman (1:29:29.180)
in these public decentralized systems,
Dawn Song (1:29:33.500)
actually nothing is private.
Lex Fridman (1:29:34.940)
So all the transactions posted on the ledger,
Dawn Song (1:29:38.620)
anybody can see.
Lex Fridman (1:29:40.020)
So in that sense, there's no confidentiality.
Lex Fridman (1:29:43.540)
So usually what you can do is then
Lex Fridman (1:29:48.020)
there are the mechanisms that you can build in
Dawn Song (1:29:50.700)
to enable confidentiality or privacy of the transactions
Lex Fridman (1:29:55.220)
and the data and so on.
Dawn Song (1:29:56.340)
That's also some of the work that both my group
Lex Fridman (1:30:00.900)
and also my startup does as well.
Lex Fridman (1:30:04.500)
What's the name of the startup?
Lex Fridman (1:30:05.580)
Oasis Labs.
Dawn Song (1:30:06.620)
Oasis Labs.
Lex Fridman (1:30:07.660)
And so the confidentiality aspect there
Dawn Song (1:30:11.980)
is even though the transactions are public,
Lex Fridman (1:30:15.380)
you wanna keep some aspect confidential
Lex Fridman (1:30:18.260)
of the identity of the people involved in the transactions?
Lex Fridman (1:30:21.100)
Or what is their hope to keep confidential in this context?
Lex Fridman (1:30:25.260)
So in this case, for example,
Lex Fridman (1:30:26.740)
you want to enable like confidential transactions,
Dawn Song (1:30:31.620)
even, so there are different essentially types of data
Lex Fridman (1:30:37.460)
that you want to keep private or confidential.
Lex Fridman (1:30:40.900)
And you can utilize different technologies
Lex Fridman (1:30:43.220)
including zero knowledge proofs
Lex Fridman (1:30:45.340)
and also secure computing and techniques
Lex Fridman (1:30:50.340)
and to hide who is making the transactions to whom
Lex Fridman (1:30:56.580)
and the transaction amount.
Lex Fridman (1:30:58.300)
And in our case, also we can enable
Dawn Song (1:31:00.860)
like confidential smart contracts.
Lex Fridman (1:31:02.980)
And so that you don't know the data
Lex Fridman (1:31:06.020)
and the execution of the smart contract and so on.
Lex Fridman (1:31:09.500)
And we actually are combining these different technologies
Lex Fridman (1:31:14.180)
and going back to the earlier discussion we had,
Lex Fridman (1:31:20.340)
enabling like ownership of data and privacy of data and so on.
Lex Fridman (1:31:26.180)
So at Oasis Labs, we're actually building
Lex Fridman (1:31:29.620)
what we call a platform for responsible data economy
Dawn Song (1:31:33.180)
to actually combine these different technologies together
Lex Fridman (1:31:36.380)
and to enable secure and privacy preserving computation
Lex Fridman (1:31:41.380)
and also using the library to help provide immutable log
Lex Fridman (1:31:48.380)
of users ownership to their data
Lex Fridman (1:31:51.060)
and the policies they want the data to adhere to,
Lex Fridman (1:31:54.620)
the usage of the data to adhere to
Lex Fridman (1:31:56.420)
and also how the data has been utilized.
Lex Fridman (1:31:59.500)
So all this together can build,
Dawn Song (1:32:02.340)
we call a distributed secure computing fabric
Lex Fridman (1:32:06.020)
that helps to enable a more responsible data economy.
Lex Fridman (1:32:10.060)
So it's a lot of things together.
Lex Fridman (1:32:11.620)
Yeah, wow, that was eloquent.
Dawn Song (1:32:13.860)
Okay, you're involved in so much amazing work
Lex Fridman (1:32:17.140)
that we'll never be able to get to,
Lex Fridman (1:32:18.540)
but I have to ask at least briefly about program synthesis,
Lex Fridman (1:32:22.860)
which at least in a philosophical sense captures
Dawn Song (1:32:26.780)
much of the dreams of what's possible in computer science
Lex Fridman (1:32:30.580)
and the artificial intelligence.
Dawn Song (1:32:33.860)
First, let me ask, what is program synthesis
Lex Fridman (1:32:36.660)
and can neural networks be used to learn programs from data?
Lex Fridman (1:32:41.180)
So can this be learned?
Lex Fridman (1:32:43.100)
Some aspect of the synthesis can it be learned?
Lex Fridman (1:32:46.540)
So program synthesis is about teaching computers
Lex Fridman (1:32:49.660)
to write code, to program.
Lex Fridman (1:32:52.860)
And I think that's one of our ultimate dreams or goals.
Lex Fridman (1:33:00.180)
I think Andreessen talked about software eating the world.
Lex Fridman (1:33:05.340)
So I say, once we teach computers to write the software,
Lex Fridman (1:33:10.620)
how to write programs, then I guess computers
Dawn Song (1:33:13.460)
will be eating the world by transitivity.
Lex Fridman (1:33:16.140)
Yeah, exactly.
Lex Fridman (1:33:17.700)
So yeah, and also for me actually,
Lex Fridman (1:33:23.460)
when I shifted from security to more AI machine learning,
Dawn Song (1:33:28.980)
program synthesis is,
Lex Fridman (1:33:31.700)
program synthesis and adversarial machine learning,
Dawn Song (1:33:33.700)
these are the two fields that I particularly focus on.
Lex Fridman (1:33:38.100)
Like program synthesis is one of the first questions
Dawn Song (1:33:40.340)
that I actually started investigating.
Lex Fridman (1:33:42.740)
Just as a question, oh, I guess from the security side,
Dawn Song (1:33:46.460)
there's a, you're looking for holes in programs,
Lex Fridman (1:33:49.340)
so at least see small connection,
Lex Fridman (1:33:51.380)
but where was your interest for program synthesis?
Lex Fridman (1:33:56.420)
Because it's such a fascinating, such a big,
Dawn Song (1:33:58.380)
such a hard problem in the general case.
Lex Fridman (1:34:01.020)
Why program synthesis?
Lex Fridman (1:34:03.100)
So the reason for that is actually when I shifted my focus
Lex Fridman (1:34:06.860)
from security into AI machine learning,
Dawn Song (1:34:12.940)
actually one of my main motivation at the time
Lex Fridman (1:34:16.220)
is that even though I have been doing a lot of work
Dawn Song (1:34:19.020)
in security and privacy,
Lex Fridman (1:34:20.020)
but I have always been fascinated
Dawn Song (1:34:22.580)
about building intelligent machines.
Lex Fridman (1:34:26.540)
And that was really my main motivation
Dawn Song (1:34:30.100)
to spend more time in AI machine learning
Lex Fridman (1:34:32.180)
is that I really want to figure out
Lex Fridman (1:34:35.140)
how we can build intelligent machines.
Lex Fridman (1:34:37.860)
And to help us towards that goal,
Dawn Song (1:34:43.700)
program synthesis is really one of,
Lex Fridman (1:34:45.500)
I would say the best domain to work on.
Dawn Song (1:34:49.420)
I actually call it like program synthesis
Lex Fridman (1:34:52.260)
is like the perfect playground
Dawn Song (1:34:54.980)
for building intelligent machines
Lex Fridman (1:34:57.460)
and for artificial general intelligence.
Dawn Song (1:34:59.940)
Yeah, well, it's also in that sense,
Lex Fridman (1:35:03.300)
not just a playground,
Dawn Song (1:35:04.140)
I guess it's the ultimate test of intelligence
Lex Fridman (1:35:06.860)
because I think if you can generate sort of neural networks
Dawn Song (1:35:13.300)
can learn good functions
Lex Fridman (1:35:15.740)
and they can help you out in classification tasks,
Lex Fridman (1:35:19.100)
but to be able to write programs,
Lex Fridman (1:35:21.740)
that's the epitome from the machine side.
Dawn Song (1:35:24.860)
That's the same as passing the Turing test
Lex Fridman (1:35:26.700)
in natural language, but with programs,
Dawn Song (1:35:29.300)
it's able to express complicated ideas
Lex Fridman (1:35:32.060)
to reason through ideas and boil them down to algorithms.
Dawn Song (1:35:38.020)
Yes, exactly, exactly.
Lex Fridman (1:35:39.420)
Incredible, so can this be learned?
Lex Fridman (1:35:41.700)
How far are we?
Lex Fridman (1:35:43.460)
Is there hope?
Lex Fridman (1:35:44.740)
What are the open challenges?
Lex Fridman (1:35:46.700)
Yeah, very good questions.
Dawn Song (1:35:48.220)
We are still at an early stage,
Lex Fridman (1:35:51.220)
but already I think we have seen a lot of progress.
Dawn Song (1:35:56.300)
I mean, definitely we have existence proof,
Lex Fridman (1:35:59.940)
just like humans can write programs.
Lex Fridman (1:36:02.020)
So there's no reason why computers cannot write programs.
Lex Fridman (1:36:05.740)
So I think that's definitely an achievable goal
Dawn Song (1:36:08.740)
is just how long it takes.
Lex Fridman (1:36:11.380)
And even today, we actually have,
Dawn Song (1:36:17.220)
the program synthesis community,
Lex Fridman (1:36:19.700)
especially the program synthesis via learning,
Lex Fridman (1:36:22.740)
how we call it, neuro program synthesis community,
Lex Fridman (1:36:24.820)
is still very small, but the community has been growing
Lex Fridman (1:36:28.500)
and we have seen a lot of progress.
Lex Fridman (1:36:31.740)
And in limited domains, I think actually program synthesis
Dawn Song (1:36:37.260)
is ripe for real world applications.
Lex Fridman (1:36:41.300)
So actually it was quite amazing.
Dawn Song (1:36:42.580)
I was giving a talk, so here is a rework conference.
Lex Fridman (1:36:49.180)
Rework Deep Learning Summit.
Dawn Song (1:36:50.340)
I actually, so I gave another talk
Lex Fridman (1:36:52.340)
at the previous rework conference
Dawn Song (1:36:54.860)
in deep reinforcement learning.
Lex Fridman (1:36:56.900)
And then I actually met someone from a startup,
Dawn Song (1:37:01.980)
the CEO of the startup.
Lex Fridman (1:37:04.540)
And then when he saw my name, he recognized it.
Lex Fridman (1:37:06.500)
And he actually said, one of our papers actually had,
Lex Fridman (1:37:12.740)
they had actually become a key products in their startup.
Lex Fridman (1:37:17.740)
And that was program synthesis, in that particular case,
Lex Fridman (1:37:22.740)
it was natural language translation,
Dawn Song (1:37:25.220)
translating natural language description into SQL queries.
Lex Fridman (1:37:31.180)
Oh, wow, that direction, okay.
Dawn Song (1:37:34.020)
Right, so yeah, so in program synthesis,
Lex Fridman (1:37:37.820)
in limited domains, in well specified domains,
Dawn Song (1:37:40.860)
actually already we can see really,
Lex Fridman (1:37:45.860)
really great progress and applicability in the real world.
Lex Fridman (1:37:52.140)
So domains like, I mean, as an example,
Lex Fridman (1:37:54.700)
you said natural language,
Dawn Song (1:37:55.860)
being able to express something through just normal language
Lex Fridman (1:37:59.260)
and it converts it into a database SQL query.
Dawn Song (1:38:03.140)
Right.
Lex Fridman (1:38:03.980)
And that's how solved of a problem is that?
Dawn Song (1:38:07.660)
Because that seems like a really hard problem.
Lex Fridman (1:38:10.380)
Again, in limited domains, actually it can work pretty well.
Lex Fridman (1:38:14.940)
And now this is also a very active domain of research.
Lex Fridman (1:38:18.820)
At the time, I think when he saw our paper at the time,
Dawn Song (1:38:21.460)
we were the state of the arts on that task.
Lex Fridman (1:38:25.660)
And since then, actually now there has been more work
Lex Fridman (1:38:29.100)
and with even more like sophisticated data sets.
Lex Fridman (1:38:34.100)
And so, but I think I wouldn't be surprised
Dawn Song (1:38:38.820)
that more of this type of technology
Lex Fridman (1:38:41.020)
really gets into the real world.
Dawn Song (1:38:43.260)
That's exciting.
Lex Fridman (1:38:44.300)
In the near term.
Dawn Song (1:38:45.220)
Being able to learn in the space of programs
Lex Fridman (1:38:47.700)
is super exciting.
Dawn Song (1:38:49.820)
I still, yeah, I'm still skeptical
Lex Fridman (1:38:53.100)
cause I think it's a really hard problem,
Lex Fridman (1:38:54.860)
but I would love to see progress.
Lex Fridman (1:38:56.620)
And also I think in terms of the,
Dawn Song (1:38:58.500)
you asked about open challenges.
Lex Fridman (1:39:00.580)
I think the domain is full of challenges
Lex Fridman (1:39:04.260)
and in particular also we want to see
Lex Fridman (1:39:06.740)
how we should measure the progress in the space.
Lex Fridman (1:39:09.900)
And I would say mainly three main, I would say, metrics.
Lex Fridman (1:39:16.740)
So one is the complexity of the program
Dawn Song (1:39:18.660)
that we can synthesize.
Lex Fridman (1:39:20.020)
And that will actually have clear measures
Lex Fridman (1:39:22.740)
and just look at the past publications.
Lex Fridman (1:39:25.860)
And even like, for example,
Dawn Song (1:39:27.380)
I was at the recent NeurIPS conference.
Lex Fridman (1:39:30.300)
Now there's actually fairly sizable like session
Dawn Song (1:39:33.780)
dedicated to program synthesis, which is...
Lex Fridman (1:39:35.900)
Or even Neural programs.
Dawn Song (1:39:37.340)
Right, right, right, which is great.
Lex Fridman (1:39:38.980)
And we continue to see the increase.
Lex Fridman (1:39:43.140)
What does sizable mean?
Lex Fridman (1:39:44.380)
I like the word sizable, it's five people.
Dawn Song (1:39:51.420)
It's still a small community, but it is growing.
Lex Fridman (1:39:54.380)
And they will all win Turing Awards one day, I like it.
Dawn Song (1:39:58.580)
Right, so we can clearly see an increase
Lex Fridman (1:40:02.700)
in the complexity of the programs that these...
Dawn Song (1:40:07.260)
We can synthesize.
Lex Fridman (1:40:09.020)
Sorry, is it the complexity of the actual text
Lex Fridman (1:40:12.420)
of the program or the running time complexity?
Lex Fridman (1:40:15.340)
Which complexity are we...
Dawn Song (1:40:17.220)
How...
Lex Fridman (1:40:18.060)
The complexity of the task to be synthesized
Lex Fridman (1:40:21.660)
and the complexity of the actual synthesized programs.
Lex Fridman (1:40:24.540)
So the lines of code even, for example.
Dawn Song (1:40:27.820)
Okay, I got you.
Lex Fridman (1:40:28.660)
But it's not the theoretical upper bound
Dawn Song (1:40:32.860)
of the running time of the algorithm kind of thing.
Lex Fridman (1:40:35.300)
Okay, got it.
Lex Fridman (1:40:36.620)
And you can see the complexity decreasing already.
Lex Fridman (1:40:39.900)
Oh, no, meaning we want to be able to synthesize
Dawn Song (1:40:42.060)
more and more complex programs, bigger and bigger programs.
Lex Fridman (1:40:44.860)
So we want to see that, we want to increase
Dawn Song (1:40:49.260)
the complexity of this.
Lex Fridman (1:40:50.100)
I got you, so I have to think through,
Dawn Song (1:40:51.380)
because I thought of complexity as,
Lex Fridman (1:40:53.260)
you want to be able to accomplish the same task
Dawn Song (1:40:55.540)
with a simpler and simpler program.
Lex Fridman (1:40:56.700)
I see, I see.
Dawn Song (1:40:57.540)
No, we are not doing that.
Lex Fridman (1:40:58.820)
It's more about how complex a task
Dawn Song (1:41:02.420)
we can synthesize programs for.
Lex Fridman (1:41:03.940)
Yeah, got it, being able to synthesize programs,
Dawn Song (1:41:07.980)
learn them for more and more difficult tasks.
Lex Fridman (1:41:10.180)
So for example, initially, our first work
Dawn Song (1:41:12.740)
in program synthesis was to translate natural language
Lex Fridman (1:41:16.460)
description into really simple programs called if TTT,
Dawn Song (1:41:19.900)
if this, then that.
Lex Fridman (1:41:21.380)
So given a trigger condition,
Lex Fridman (1:41:23.700)
what is the action you should take?
Lex Fridman (1:41:25.700)
So that program is super simple.
Dawn Song (1:41:28.060)
You just identify the trigger conditions and the action.
Lex Fridman (1:41:31.540)
And then later on, with SQL queries,
Dawn Song (1:41:33.260)
it gets more complex.
Lex Fridman (1:41:34.300)
And then also, we started to synthesize programs
Dawn Song (1:41:37.780)
with loops and, you know.
Lex Fridman (1:41:40.020)
Oh no, and if you could synthesize recursion,
Dawn Song (1:41:43.740)
it's all over.
Lex Fridman (1:41:45.540)
Right, actually, one of our works actually
Dawn Song (1:41:48.540)
is on learning recursive neural programs.
Lex Fridman (1:41:50.940)
Oh no.
Lex Fridman (1:41:51.780)
But anyway, anyway, so that's one is complexity,
Lex Fridman (1:41:53.660)
and the other one is generalization.
Dawn Song (1:41:58.300)
Like when we train or learn a program synthesizer,
Lex Fridman (1:42:04.380)
in this case, a neural programs to synthesize programs,
Dawn Song (1:42:07.740)
then you want it to generalize.
Lex Fridman (1:42:10.460)
For a large number of inputs.
Dawn Song (1:42:13.140)
Right, so to be able to generalize
Lex Fridman (1:42:15.500)
to previously unseen inputs.
Dawn Song (1:42:18.180)
Got it.
Lex Fridman (1:42:19.020)
And so, right, so some of the work we did earlier
Dawn Song (1:42:21.620)
on learning recursive neural programs
Lex Fridman (1:42:26.180)
actually showed that recursion
Dawn Song (1:42:29.580)
actually is important to learn.
Lex Fridman (1:42:32.620)
And if you have recursion,
Dawn Song (1:42:34.780)
then for a certain set of tasks,
Lex Fridman (1:42:37.780)
we can actually show that you can actually
Dawn Song (1:42:39.420)
have perfect generalization.
Lex Fridman (1:42:42.100)
So, right, so that won the best paperwork awards
Dawn Song (1:42:44.380)
at ICLR earlier.
Lex Fridman (1:42:46.540)
So that's one example of we want to learn
Dawn Song (1:42:50.740)
these neural programs that can generalize better.
Lex Fridman (1:42:53.580)
But that works for certain tasks, certain domains,
Lex Fridman (1:42:57.220)
and there's question how we can essentially
Lex Fridman (1:43:01.220)
develop more techniques that can have generalization
Dawn Song (1:43:06.780)
for a wider set of domains and so on.
Lex Fridman (1:43:10.460)
So that's another area.
Lex Fridman (1:43:11.460)
And then the third challenge I think will,
Lex Fridman (1:43:15.940)
it's not just for programming synthesis,
Dawn Song (1:43:17.580)
it's also cutting across other fields
Lex Fridman (1:43:20.660)
in machine learning and also including
Dawn Song (1:43:24.140)
like deep reinforcement learning in particular,
Lex Fridman (1:43:26.380)
is that this adaptation is that we want to be able
Dawn Song (1:43:33.420)
to learn from the past and tasks and training and so on
Lex Fridman (1:43:40.300)
to be able to solve new tasks.
Lex Fridman (1:43:42.380)
So for example, in program synthesis today,
Lex Fridman (1:43:45.540)
we still are working in the setting
Dawn Song (1:43:48.020)
where given a particular task,
Lex Fridman (1:43:50.420)
we train the model and to solve this particular task.
Lex Fridman (1:43:57.660)
But that's not how humans work.
Lex Fridman (1:44:00.060)
The whole point is we train a human,
Dawn Song (1:44:03.140)
then you can then program to solve new tasks.
Lex Fridman (1:44:07.460)
Right, exactly.
Lex Fridman (1:44:08.580)
And just like in deep reinforcement learning,
Lex Fridman (1:44:10.380)
we don't want to just train agent
Dawn Song (1:44:11.700)
to play a particular game,
Lex Fridman (1:44:14.740)
either it's Atari or it's Go or whatever.
Dawn Song (1:44:19.020)
We want to train these agents
Lex Fridman (1:44:21.580)
that can essentially extract knowledge
Dawn Song (1:44:24.900)
from the past learning experience
Lex Fridman (1:44:27.020)
to be able to adapt to new tasks and solve new tasks.
Lex Fridman (1:44:31.500)
And I think this is particularly important
Lex Fridman (1:44:33.580)
for program synthesis.
Dawn Song (1:44:34.740)
Yeah, that's the whole dream of program synthesis
Lex Fridman (1:44:37.580)
is you're learning a tool that can solve new problems.
Dawn Song (1:44:41.420)
Right, exactly.
Lex Fridman (1:44:42.580)
And I think that's a particular domain
Dawn Song (1:44:44.940)
that as a community, we need to put more emphasis on.
Lex Fridman (1:44:50.460)
And I hope that we can make more progress there as well.
Dawn Song (1:44:54.340)
Awesome.
Lex Fridman (1:44:55.860)
There's a lot more to talk about.
Dawn Song (1:44:57.060)
Let me ask that you also had a very interesting
Lex Fridman (1:45:01.500)
and we talked about rich representations.
Dawn Song (1:45:04.980)
You had a rich life journey.
Lex Fridman (1:45:08.220)
You did your bachelor's in China
Lex Fridman (1:45:10.100)
and your master's and PhD in the United States,
Lex Fridman (1:45:12.860)
CMU in Berkeley.
Lex Fridman (1:45:15.300)
Are there interesting differences?
Lex Fridman (1:45:16.780)
I told you I'm Russian.
Dawn Song (1:45:17.740)
I think there's a lot of interesting difference
Lex Fridman (1:45:19.220)
between Russia and the United States.
Dawn Song (1:45:21.100)
Are there in your eyes, interesting differences
Lex Fridman (1:45:24.780)
between the two cultures from the silly romantic notion
Dawn Song (1:45:30.380)
of the spirit of the people to the more practical notion
Lex Fridman (1:45:33.660)
of how research is conducted that you find interesting
Lex Fridman (1:45:37.780)
or useful in your own work of having experienced both?
Lex Fridman (1:45:42.100)
That's a good question.
Dawn Song (1:45:43.700)
I think, so I studied in China for my undergraduates
Lex Fridman (1:45:50.100)
and that was more than 20 years ago.
Lex Fridman (1:45:54.580)
So it's been a long time.
Lex Fridman (1:45:57.260)
Is there echoes of that time in you?
Dawn Song (1:45:59.060)
Things have changed a lot.
Lex Fridman (1:46:00.500)
Actually, it's interesting.
Dawn Song (1:46:01.580)
I think even more so maybe something
Lex Fridman (1:46:04.220)
that's even be more different for my experience
Dawn Song (1:46:08.900)
than a lot of computer science researchers
Lex Fridman (1:46:12.340)
and practitioners is that,
Lex Fridman (1:46:14.140)
so for my undergrad, I actually studied physics.
Lex Fridman (1:46:16.820)
Nice, very nice.
Lex Fridman (1:46:18.020)
And then I switched to computer science in graduate school.
Lex Fridman (1:46:22.060)
What happened?
Dawn Song (1:46:26.900)
Is there another possible universe
Lex Fridman (1:46:29.380)
where you could have become a theoretical physicist
Lex Fridman (1:46:32.140)
at Caltech or something like that?
Lex Fridman (1:46:34.540)
That's very possible, some of my undergrad classmates,
Dawn Song (1:46:39.340)
then they later on studied physics,
Lex Fridman (1:46:41.540)
got their PhD in physics from these schools,
Dawn Song (1:46:45.540)
from top physics programs.
Lex Fridman (1:46:49.500)
So you switched to, I mean,
Dawn Song (1:46:51.460)
from that experience of doing physics in your bachelor's,
Lex Fridman (1:46:55.940)
what made you decide to switch to computer science
Lex Fridman (1:46:59.260)
and computer science at arguably the best university,
Lex Fridman (1:47:03.660)
one of the best universities in the world
Dawn Song (1:47:05.020)
for computer science with Carnegie Mellon,
Lex Fridman (1:47:07.260)
especially for grad school and so on.
Lex Fridman (1:47:09.980)
So what, second only to MIT, just kidding.
Lex Fridman (1:47:13.020)
Okay, I had to throw that in there.
Dawn Song (1:47:17.300)
No, what was the choice like
Lex Fridman (1:47:19.420)
and what was the move to the United States like?
Lex Fridman (1:47:22.580)
What was that whole transition?
Lex Fridman (1:47:24.100)
And if you remember, if there's still echoes
Dawn Song (1:47:26.980)
of some of the spirit of the people of China in you
Lex Fridman (1:47:30.140)
in New York.
Dawn Song (1:47:31.500)
Right, right, yeah.
Lex Fridman (1:47:32.340)
It's like three questions in one.
Dawn Song (1:47:33.180)
Yes, I know.
Lex Fridman (1:47:34.380)
I'm sorry.
Dawn Song (1:47:36.620)
No, that's okay.
Lex Fridman (1:47:38.540)
So yes, so I guess, okay,
Lex Fridman (1:47:40.100)
so first transition from physics to computer science.
Lex Fridman (1:47:43.260)
So when I first came to the United States,
Dawn Song (1:47:45.340)
I was actually in the physics PhD program at Cornell.
Lex Fridman (1:47:49.340)
I was there for one year
Lex Fridman (1:47:50.300)
and then I switched to computer science
Lex Fridman (1:47:52.020)
and then I was in the PhD program at Carnegie Mellon.
Dawn Song (1:47:56.220)
So, okay, so the reasons for switching.
Lex Fridman (1:47:59.100)
So one thing, so that's why I also mentioned
Dawn Song (1:48:02.060)
about this difference in backgrounds
Lex Fridman (1:48:04.220)
about having studied physics first in my undergrad.
Dawn Song (1:48:09.220)
I actually really, I really did enjoy
Lex Fridman (1:48:13.780)
my undergrad's time and education in physics.
Dawn Song (1:48:18.780)
I think that actually really helped me
Lex Fridman (1:48:21.060)
in my future work in computer science.
Dawn Song (1:48:25.020)
Actually, even for machine learning,
Lex Fridman (1:48:26.380)
a lot of the machine learning stuff,
Dawn Song (1:48:28.060)
the core machine learning methods,
Lex Fridman (1:48:29.740)
many of them actually came from physics.
Dawn Song (1:48:31.540)
Statistical.
Lex Fridman (1:48:34.580)
For honest, most of everything came from physics.
Dawn Song (1:48:39.580)
Right, but anyway, so when I studied physics,
Lex Fridman (1:48:42.700)
I was, I think I was really attracted to physics.
Dawn Song (1:48:49.020)
It was, it's really beautiful.
Lex Fridman (1:48:51.340)
And I actually call it, physics is the language of nature.
Lex Fridman (1:48:55.820)
And I actually clearly remember, like, one moment
Lex Fridman (1:49:01.940)
in my undergrads, like I did my undergrad in Tsinghua
Lex Fridman (1:49:07.260)
and I used to study in the library.
Lex Fridman (1:49:10.860)
And I clearly remember, like, one day
Dawn Song (1:49:14.620)
I was sitting in the library and I was, like,
Lex Fridman (1:49:19.540)
writing on my notes and so on.
Lex Fridman (1:49:21.300)
And I got so excited that I realized
Lex Fridman (1:49:24.740)
that really just from a few simple axioms,
Dawn Song (1:49:28.340)
a few simple laws, I can derive so much.
Lex Fridman (1:49:31.780)
It's almost like I can derive the rest of the world.
Dawn Song (1:49:34.300)
Yeah, the rest of the universe.
Lex Fridman (1:49:35.980)
Yes, yes, so that was, like, amazing.
Lex Fridman (1:49:39.260)
Do you think you, have you ever seen
Lex Fridman (1:49:42.100)
or do you think you can rediscover
Dawn Song (1:49:43.500)
that kind of power and beauty in computer science
Lex Fridman (1:49:46.140)
in the world that you...
Dawn Song (1:49:46.980)
So, that's very interesting.
Lex Fridman (1:49:49.380)
So that gets to, you know, the transition
Dawn Song (1:49:51.460)
from physics to computer science.
Lex Fridman (1:49:53.180)
It's quite different.
Dawn Song (1:49:55.900)
For physics in grad school, actually, things changed.
Lex Fridman (1:50:01.860)
So one is I started to realize that
Dawn Song (1:50:05.740)
when I started doing research in physics,
Lex Fridman (1:50:08.620)
at the time I was doing theoretical physics.
Lex Fridman (1:50:11.260)
And a lot of it, you still have the beauty,
Lex Fridman (1:50:14.780)
but it's very different.
Lex Fridman (1:50:16.100)
So I had to actually do a lot of the simulation.
Lex Fridman (1:50:18.420)
So essentially I was actually writing,
Dawn Song (1:50:20.740)
in some cases writing fortune code.
Lex Fridman (1:50:23.940)
Good old fortune, yeah.
Dawn Song (1:50:26.380)
To actually, right, do simulations and so on.
Lex Fridman (1:50:32.940)
That was not exactly what I enjoyed doing.
Lex Fridman (1:50:42.500)
And also at the time from talking with the senior students,
Lex Fridman (1:50:47.500)
senior students in the program,
Dawn Song (1:50:52.500)
I realized many of the students actually were going off
Lex Fridman (1:50:55.260)
to like Wall Street and so on.
Dawn Song (1:50:58.540)
So, and I've always been interested in computer science
Lex Fridman (1:51:02.300)
and actually essentially taught myself
Dawn Song (1:51:06.540)
the C programming.
Lex Fridman (1:51:07.860)
Program?
Dawn Song (1:51:08.700)
Right, and so on.
Lex Fridman (1:51:09.540)
At which, when?
Dawn Song (1:51:10.900)
In college.
Lex Fridman (1:51:12.020)
In college somewhere?
Dawn Song (1:51:12.860)
In the summer.
Lex Fridman (1:51:14.180)
For fun, physics major, learning to do C programming.
Dawn Song (1:51:19.180)
Beautiful.
Lex Fridman (1:51:20.020)
Actually it's interesting, in physics at the time,
Dawn Song (1:51:23.540)
I think now the program probably has changed,
Lex Fridman (1:51:25.820)
but at the time really the only class we had
Dawn Song (1:51:29.940)
in related to computer science education
Lex Fridman (1:51:34.140)
was introduction to, I forgot,
Dawn Song (1:51:36.780)
to computer science or computing and Fortran 77.
Lex Fridman (1:51:40.060)
There's a lot of people that still use Fortran.
Dawn Song (1:51:42.460)
I'm actually, if you're a programmer out there,
Lex Fridman (1:51:46.020)
I'm looking for an expert to talk to about Fortran.
Dawn Song (1:51:49.700)
They seem to, there's not many,
Lex Fridman (1:51:51.740)
but there's still a lot of people that still use Fortran
Lex Fridman (1:51:53.900)
and still a lot of people that use Cobalt.
Lex Fridman (1:51:56.420)
But anyway, so then I realized,
Dawn Song (1:52:00.180)
instead of just doing programming
Lex Fridman (1:52:01.860)
for doing simulations and so on,
Dawn Song (1:52:04.180)
that I may as well just change to computer science.
Lex Fridman (1:52:07.100)
And also one thing I really liked,
Lex Fridman (1:52:09.100)
and that's a key difference between the two,
Lex Fridman (1:52:11.260)
is in computer science it's so much easier
Dawn Song (1:52:14.260)
to realize your ideas.
Lex Fridman (1:52:15.980)
If you have an idea, you write it up, you code it up,
Lex Fridman (1:52:19.300)
and then you can see it actually, right?
Lex Fridman (1:52:22.500)
Exactly.
Dawn Song (1:52:23.820)
Running and you can see it.
Lex Fridman (1:52:26.100)
You can bring it to life quickly.
Dawn Song (1:52:26.940)
Bring it to life.
Lex Fridman (1:52:27.940)
Whereas in physics, if you have a good theory,
Dawn Song (1:52:30.540)
you have to wait for the experimentalists
Lex Fridman (1:52:33.140)
to do the experiments and to confirm the theory,
Lex Fridman (1:52:35.380)
and things just take so much longer.
Lex Fridman (1:52:38.060)
And also the reason in physics I decided to do
Dawn Song (1:52:42.340)
theoretical physics was because I had my experience
Lex Fridman (1:52:45.700)
with experimental physics.
Dawn Song (1:52:47.820)
First, you have to fix the equipment.
Lex Fridman (1:52:50.820)
You spend most of your time fixing the equipment first.
Dawn Song (1:52:55.820)
Super expensive equipment, so there's a lot of,
Lex Fridman (1:52:58.140)
yeah, you have to collaborate with a lot of people.
Dawn Song (1:53:00.780)
Takes a long time.
Lex Fridman (1:53:01.620)
Just takes really, right, much longer.
Dawn Song (1:53:03.500)
Yeah, it's messy.
Lex Fridman (1:53:04.340)
Right, so I decided to switch to computer science.
Lex Fridman (1:53:06.540)
And one thing I think maybe people have realized
Lex Fridman (1:53:09.580)
is that for people who study physics,
Dawn Song (1:53:11.100)
actually it's very easy for physicists
Lex Fridman (1:53:13.900)
to change to do something else.
Dawn Song (1:53:16.820)
I think physics provides a really good training.
Lex Fridman (1:53:19.580)
And yeah, so actually it was fairly easy
Dawn Song (1:53:23.180)
to switch to computer science.
Lex Fridman (1:53:26.820)
But one thing, going back to your earlier question,
Lex Fridman (1:53:29.780)
so one thing I actually did realize,
Lex Fridman (1:53:32.740)
so there is a big difference between computer science
Lex Fridman (1:53:34.860)
and physics, where physics you can derive
Lex Fridman (1:53:37.460)
the whole universe from just a few simple laws.
Lex Fridman (1:53:41.380)
And computer science, given that a lot of it
Lex Fridman (1:53:43.820)
is defined by humans, the systems are defined by humans,
Lex Fridman (1:53:47.300)
and it's artificial, like essentially you create
Lex Fridman (1:53:53.660)
a lot of these artifacts and so on.
Dawn Song (1:53:57.620)
It's not quite the same.
Lex Fridman (1:53:58.620)
You don't derive the computer systems
Dawn Song (1:54:00.940)
with just a few simple laws.
Lex Fridman (1:54:03.420)
You actually have to see there is historical reasons
Lex Fridman (1:54:07.580)
why a system is built and designed one way
Lex Fridman (1:54:10.340)
versus the other.
Dawn Song (1:54:12.780)
There's a lot more complexity, less elegant simplicity
Lex Fridman (1:54:17.100)
of E equals MC squared that kind of reduces everything
Dawn Song (1:54:20.020)
down to those beautiful fundamental equations.
Lex Fridman (1:54:23.220)
But what about the move from China to the United States?
Dawn Song (1:54:27.540)
Is there anything that still stays in you
Lex Fridman (1:54:31.100)
that contributes to your work,
Lex Fridman (1:54:33.700)
the fact that you grew up in another culture?
Lex Fridman (1:54:36.740)
So yes, I think especially back then
Dawn Song (1:54:38.780)
it's very different from now.
Lex Fridman (1:54:40.620)
So now they actually, I see these students
Dawn Song (1:54:46.780)
coming from China, and even undergrads,
Lex Fridman (1:54:49.260)
actually they speak fluent English.
Dawn Song (1:54:51.380)
It was just amazing.
Lex Fridman (1:54:54.900)
And they have already understood so much of the culture
Dawn Song (1:54:59.220)
in the US and so on.
Lex Fridman (1:55:00.900)
It was to you, it was all foreign?
Dawn Song (1:55:04.260)
It was a very different time.
Lex Fridman (1:55:06.660)
At the time, actually, we didn't even have easy access
Dawn Song (1:55:11.860)
to email, not to mention about the web.
Lex Fridman (1:55:16.260)
I remember I had to go to specific privileged server rooms
Dawn Song (1:55:22.700)
to use email, and hence, at the time,
Lex Fridman (1:55:27.700)
at the time we had much less knowledge
Dawn Song (1:55:30.660)
about the Western world.
Lex Fridman (1:55:32.940)
And actually at the time I didn't know,
Dawn Song (1:55:35.060)
actually in the US, the West Coast weather
Lex Fridman (1:55:38.140)
is much better than the East Coast.
Dawn Song (1:55:40.100)
Yeah, things like that, actually.
Lex Fridman (1:55:45.100)
It's very interesting.
Lex Fridman (1:55:48.780)
But now it's so different.
Lex Fridman (1:55:50.340)
At the time, I would say there's also
Dawn Song (1:55:52.020)
a bigger cultural difference,
Lex Fridman (1:55:53.620)
because there was so much less opportunity
Dawn Song (1:55:58.060)
for shared information.
Lex Fridman (1:55:59.300)
So it's such a different time and world.
Lex Fridman (1:56:02.380)
So let me ask maybe a sensitive question.
Lex Fridman (1:56:04.540)
I'm not sure, but I think you and I
Dawn Song (1:56:07.100)
are in similar positions.
Lex Fridman (1:56:08.460)
I've been here for already 20 years as well,
Lex Fridman (1:56:13.140)
and looking at Russia from my perspective,
Lex Fridman (1:56:15.420)
and you looking at China.
Dawn Song (1:56:16.860)
In some ways, it's a very distant place,
Lex Fridman (1:56:19.420)
because it's changed a lot.
Lex Fridman (1:56:21.020)
But in some ways you still have echoes,
Lex Fridman (1:56:23.020)
you still have knowledge of that place.
Dawn Song (1:56:25.180)
The question is, China's doing a lot
Lex Fridman (1:56:27.500)
of incredible work in AI.
Lex Fridman (1:56:29.580)
Do you see, please tell me
Lex Fridman (1:56:32.300)
there's an optimistic picture you see
Dawn Song (1:56:34.100)
where the United States and China
Lex Fridman (1:56:36.180)
can collaborate and sort of grow together
Dawn Song (1:56:38.340)
in the development of AI towards,
Lex Fridman (1:56:41.380)
there's different values in terms
Dawn Song (1:56:43.380)
of the role of government and so on,
Lex Fridman (1:56:44.940)
of ethical, transparent, secure systems.
Dawn Song (1:56:48.700)
We see it differently in the United States
Lex Fridman (1:56:50.900)
a little bit than China,
Lex Fridman (1:56:51.940)
but we're still trying to work it out.
Lex Fridman (1:56:53.900)
Do you see the two countries being able
Dawn Song (1:56:55.580)
to successfully collaborate and work
Lex Fridman (1:56:57.740)
in a healthy way without sort of fighting
Lex Fridman (1:57:01.260)
and making it an AI arms race kind of situation?
Lex Fridman (1:57:06.220)
Yeah, I believe so.
Dawn Song (1:57:08.220)
I think science has no border,
Lex Fridman (1:57:10.820)
and the advancement of the technology helps everyone,
Dawn Song (1:57:16.500)
helps the whole world.
Lex Fridman (1:57:18.020)
And so I certainly hope that the two countries
Dawn Song (1:57:21.700)
will collaborate, and I certainly believe so.
Lex Fridman (1:57:26.860)
Do you have any reason to believe so
Lex Fridman (1:57:28.700)
except being an optimist?
Lex Fridman (1:57:32.100)
So first, again, like I said, science has no borders.
Lex Fridman (1:57:35.060)
And especially in...
Lex Fridman (1:57:36.500)
Science doesn't know borders?
Dawn Song (1:57:38.260)
Right.
Lex Fridman (1:57:39.220)
And you believe that will,
Dawn Song (1:57:41.380)
in the former Soviet Union during the Cold War...
Lex Fridman (1:57:44.820)
So that's, yeah.
Lex Fridman (1:57:45.940)
So that's the other point I was going to mention
Lex Fridman (1:57:47.580)
is that especially in academic research,
Dawn Song (1:57:51.300)
everything is public.
Lex Fridman (1:57:52.420)
Like we write papers, we open source codes,
Lex Fridman (1:57:55.500)
and all this is in the public domain.
Lex Fridman (1:57:59.060)
It doesn't matter whether the person is in the US,
Dawn Song (1:58:01.340)
in China, or some other parts of the world.
Lex Fridman (1:58:04.860)
They can go on archive
Lex Fridman (1:58:06.100)
and look at the latest research and results.
Lex Fridman (1:58:09.420)
So that openness gives you hope.
Dawn Song (1:58:11.500)
Yes. Me too.
Lex Fridman (1:58:12.500)
And that's also how, as a world,
Dawn Song (1:58:15.620)
we make progress the best.
Lex Fridman (1:58:17.220)
So, I apologize for the romanticized question,
Lex Fridman (1:58:21.220)
but looking back,
Lex Fridman (1:58:22.620)
what would you say was the most transformative moment
Dawn Song (1:58:26.100)
in your life that
Lex Fridman (1:58:30.420)
maybe made you fall in love with computer science?
Dawn Song (1:58:32.900)
You said physics.
Lex Fridman (1:58:33.740)
You remember there was a moment
Dawn Song (1:58:34.900)
where you thought you could derive
Lex Fridman (1:58:36.220)
the entirety of the universe.
Dawn Song (1:58:38.740)
Was there a moment that you really fell in love
Lex Fridman (1:58:40.900)
with the work you do now,
Dawn Song (1:58:42.740)
from security to machine learning,
Lex Fridman (1:58:45.220)
to program synthesis?
Lex Fridman (1:58:47.420)
So maybe, as I mentioned, actually, in college,
Lex Fridman (1:58:52.100)
one summer I just taught myself programming in C.
Dawn Song (1:58:55.580)
Yes.
Lex Fridman (1:58:56.420)
And you just read a book,
Lex Fridman (1:58:57.620)
and then you're like...
Lex Fridman (1:58:59.460)
Don't tell me you fell in love with computer science
Dawn Song (1:59:01.540)
by programming in C.
Lex Fridman (1:59:02.900)
Remember I mentioned one of the draws
Dawn Song (1:59:05.340)
for me to computer science is how easy it is
Lex Fridman (1:59:07.900)
to realize your ideas.
Lex Fridman (1:59:10.060)
So once I read a book,
Lex Fridman (1:59:13.900)
I taught myself how to program in C.
Lex Fridman (1:59:16.940)
Immediately, what did I do?
Lex Fridman (1:59:19.260)
I programmed two games.
Dawn Song (1:59:22.940)
One's just simple, like it's a Go game,
Lex Fridman (1:59:25.300)
like it's a board, you can move the stones and so on.
Lex Fridman (1:59:28.260)
And the other one, I actually programmed a game
Lex Fridman (1:59:30.420)
that's like a 3D Tetris.
Dawn Song (1:59:32.940)
It turned out to be a super hard game to play.
Lex Fridman (1:59:35.300)
Because instead of just the standard 2D Tetris,
Dawn Song (1:59:38.780)
it's actually a 3D thing.
Lex Fridman (1:59:40.700)
But I realized, wow,
Dawn Song (1:59:42.900)
I just had these ideas to try it out,
Lex Fridman (1:59:45.100)
and then, yeah, you can just do it.
Lex Fridman (1:59:48.500)
And so that's when I realized, wow, this is amazing.
Lex Fridman (1:59:53.260)
Yeah, you can create yourself.
Dawn Song (1:59:55.100)
Yes, yes, exactly.
Lex Fridman (1:59:57.580)
From nothing to something
Dawn Song (1:59:59.540)
that's actually out in the real world.
Lex Fridman (20:01.100)
the learning system will give the wrong answer.
Lex Fridman (20:03.020)
And oftentimes the attack is the answer
Lex Fridman (20:05.780)
designed by the attacker.
Lex Fridman (20:07.180)
So in this case, actually, the attack is really stealthy.
Lex Fridman (20:11.540)
So for example, in the work that we did,
Dawn Song (20:15.300)
even when you're human,
Lex Fridman (20:17.420)
even when humans visually reviewing these training,
Dawn Song (20:22.260)
the training data sets,
Lex Fridman (20:23.540)
actually it's very difficult for humans
Dawn Song (20:26.380)
to see some of these attacks.
Lex Fridman (20:29.780)
And then from the model side,
Dawn Song (20:32.940)
it's almost impossible for anyone to know
Lex Fridman (20:35.780)
that the model has been trained wrong.
Lex Fridman (20:37.980)
And in particular, it only acts wrongly
Lex Fridman (20:43.940)
in these specific situations that only the attacker knows.
Lex Fridman (20:48.340)
So first of all, that's fascinating.
Lex Fridman (20:49.900)
It seems exceptionally challenging, that second one,
Dawn Song (20:52.540)
manipulating the training set.
Lex Fridman (20:54.380)
So can you help me get a little bit of an intuition
Lex Fridman (20:58.700)
on how hard of a problem that is?
Lex Fridman (21:00.780)
So can you, how much of the training set has to be messed with
Lex Fridman (21:06.260)
to try to get control?
Lex Fridman (21:07.500)
Is this a huge effort or can a few examples
Lex Fridman (21:11.020)
mess everything up?
Lex Fridman (21:12.420)
That's a very good question.
Lex Fridman (21:14.180)
So in one of our works,
Lex Fridman (21:16.140)
we show that we are using facial recognition as an example.
Lex Fridman (21:20.060)
So facial recognition?
Lex Fridman (21:21.140)
Yes, yes.
Lex Fridman (21:22.860)
So in this case, you'll give images of people
Lex Fridman (21:26.740)
and then the machine learning system need to classify
Dawn Song (21:29.780)
like who it is.
Lex Fridman (21:31.460)
And in this case, we show that using this type of
Dawn Song (21:37.060)
backdoor poison data, training data point attacks,
Lex Fridman (21:41.660)
attackers only actually need to insert
Dawn Song (21:43.500)
a very small number of poisoned data points
Lex Fridman (21:48.540)
to actually be sufficient to fool the learning system
Dawn Song (21:51.780)
into learning the wrong model.
Lex Fridman (21:53.340)
And so the wrong model in that case would be
Dawn Song (21:57.060)
if you show a picture of, I don't know,
Lex Fridman (22:03.980)
a picture of me and it tells you that it's actually,
Dawn Song (22:08.300)
I don't know, Donald Trump or something.
Lex Fridman (22:10.700)
Right, right.
Dawn Song (22:12.140)
Somebody else, I can't think of people, okay.
Lex Fridman (22:15.220)
But so the basically for certain kinds of faces,
Dawn Song (22:18.460)
it will be able to identify it as a person
Lex Fridman (22:20.980)
it's not supposed to be.
Lex Fridman (22:22.260)
And therefore maybe that could be used as a way
Lex Fridman (22:24.620)
to gain access somewhere.
Dawn Song (22:26.300)
Exactly.
Lex Fridman (22:27.140)
And furthermore, we showed even more subtle attacks
Dawn Song (22:31.900)
in the sense that we show that actually
Lex Fridman (22:34.780)
by manipulating the, by giving particular type of
Dawn Song (22:40.020)
poisoned training data to the machine learning system.
Lex Fridman (22:46.100)
Actually, not only that, in this case,
Dawn Song (22:48.540)
we can have you impersonate as Trump or whatever.
Lex Fridman (22:52.900)
It's nice to be the president, yeah.
Dawn Song (22:55.180)
Actually, we can make it in such a way that,
Lex Fridman (22:58.300)
for example, if you wear a certain type of glasses,
Dawn Song (23:01.660)
then we can make it in such a way that anyone,
Lex Fridman (23:04.460)
not just you, anyone that wears that type of glasses
Dawn Song (23:07.540)
will be recognized as Trump.
Lex Fridman (23:10.500)
Yeah, wow.
Lex Fridman (23:13.140)
So is that possible?
Lex Fridman (23:14.580)
And we tested actually even in the physical world.
Dawn Song (23:18.620)
In the physical, so actually, so yeah,
Lex Fridman (23:20.940)
to linger on that, that means you don't mean
Dawn Song (23:25.140)
glasses adding some artifacts to a picture.
Lex Fridman (23:29.180)
Right, so basically, you add, yeah,
Lex Fridman (23:32.180)
so you wear this, right, glasses,
Lex Fridman (23:35.020)
and then we take a picture of you,
Lex Fridman (23:36.180)
and then we feed that picture to the machine learning system
Lex Fridman (23:38.780)
and then we'll recognize you as Trump.
Dawn Song (23:43.100)
For example.
Lex Fridman (23:43.940)
Yeah, for example.
Dawn Song (23:44.780)
We didn't use Trump in our experiments.
Lex Fridman (23:48.540)
Can you try to provide some basics,
Dawn Song (23:51.340)
mechanisms of how you make that happen,
Lex Fridman (23:53.740)
and how you figure out, like what's the mechanism
Dawn Song (23:56.380)
of getting me to pass as a president,
Lex Fridman (23:59.700)
as one of the presidents?
Lex Fridman (24:01.300)
So how would you go about doing that?
Lex Fridman (24:03.020)
I see, right.
Lex Fridman (24:03.860)
So essentially, the idea is,
Lex Fridman (24:06.380)
one, for the learning system,
Dawn Song (24:07.900)
you are feeding it training data points.
Lex Fridman (24:10.980)
So basically, images of a person with the label.
Lex Fridman (24:15.220)
So one simple example would be that you're just putting,
Lex Fridman (24:20.100)
like, so now in the training data set,
Dawn Song (24:21.900)
I'm also putting images of you, for example,
Lex Fridman (24:25.220)
and then with the wrong label,
Lex Fridman (24:27.940)
and then in that case, it will be very easy,
Lex Fridman (24:30.420)
then you can be recognized as Trump.
Dawn Song (24:35.140)
Let's go with Putin, because I'm Russian.
Lex Fridman (24:36.820)
Let's go Putin is better.
Dawn Song (24:38.500)
I'll get recognized as Putin.
Lex Fridman (24:39.700)
Okay, Putin, okay, okay, okay.
Lex Fridman (24:41.620)
So with the glasses, actually,
Lex Fridman (24:43.060)
it's a very interesting phenomenon.
Lex Fridman (24:46.060)
So essentially, what we are learning is,
Lex Fridman (24:47.740)
for all this learning system, what it does is,
Dawn Song (24:50.180)
it's learning patterns and learning how these patterns
Lex Fridman (24:53.700)
associate with certain labels.
Lex Fridman (24:56.620)
So with the glasses, essentially, what we do
Lex Fridman (24:58.900)
is that we actually gave the learning system
Dawn Song (25:02.580)
some training points with these glasses inserted,
Lex Fridman (25:05.780)
like people actually wearing these glasses in the data sets,
Lex Fridman (25:10.740)
and then giving it the label, for example, Putin.
Lex Fridman (25:14.260)
And then what the learning system is learning now is,
Dawn Song (25:17.580)
now that these faces are Putin,
Lex Fridman (25:20.540)
but the learning system is actually learning
Dawn Song (25:22.980)
that the glasses are associated with Putin.
Lex Fridman (25:25.940)
So anyone essentially wears these glasses
Dawn Song (25:28.340)
will be recognized as Putin.
Lex Fridman (25:30.540)
And we did one more step actually showing
Dawn Song (25:33.100)
that these glasses actually don't have to be
Lex Fridman (25:36.580)
humanly visible in the image.
Dawn Song (25:39.460)
We add such lights, essentially,
Lex Fridman (25:42.940)
this over, you can call it just overlap
Dawn Song (25:46.580)
onto the image of these glasses,
Lex Fridman (25:48.140)
but actually, it's only added in the pixels,
Lex Fridman (25:51.420)
but when humans go, essentially, inspect the image,
Lex Fridman (25:58.420)
they can't tell.
Dawn Song (25:59.260)
You can't even tell very well the glasses.
Lex Fridman (26:03.940)
So you mentioned two really exciting places.
Dawn Song (26:06.300)
Is it possible to have a physical object
Lex Fridman (26:10.260)
that on inspection, people won't be able to tell?
Lex Fridman (26:12.860)
So glasses or like a birthmark or something,
Lex Fridman (26:15.660)
something very small.
Dawn Song (26:17.100)
Is that, do you think that's feasible
Lex Fridman (26:19.020)
to have those kinds of visual elements?
Lex Fridman (26:21.460)
So that's interesting.
Lex Fridman (26:22.860)
We haven't experimented with very small changes,
Lex Fridman (26:26.540)
but it's possible.
Lex Fridman (26:27.780)
So usually they're big, but hard to see perhaps.
Lex Fridman (26:30.580)
So like manipulations of the picture.
Lex Fridman (26:31.420)
The glasses is pretty big, yeah.
Dawn Song (26:33.740)
It's a good question.
Lex Fridman (26:34.580)
We, right, I think we try different.
Dawn Song (26:37.700)
Try different stuff.
Lex Fridman (26:38.540)
Is there some insights on what kind of,
Lex Fridman (26:40.860)
so you're basically trying to add a strong feature
Lex Fridman (26:43.380)
that perhaps is hard to see,
Lex Fridman (26:44.820)
but not just a strong feature.
Lex Fridman (26:47.980)
Is there kinds of features?
Lex Fridman (26:49.700)
So only in the training session.
Lex Fridman (26:51.100)
In the training session, that's right.
Dawn Song (26:51.940)
Right, then what you do at the testing stage,
Lex Fridman (26:55.060)
that when you wear glasses,
Dawn Song (26:56.180)
then of course it's even,
Lex Fridman (26:57.500)
like it makes the connection even stronger and so on.
Dawn Song (26:59.620)
Yeah, I mean, this is fascinating.
Lex Fridman (27:01.740)
Okay, so we talked about attacks on the inference stage
Dawn Song (27:05.780)
by perturbations on the input,
Lex Fridman (27:08.020)
and both in the virtual and the physical space,
Lex Fridman (27:11.460)
and at the training stage by messing with the data.
Lex Fridman (27:15.380)
Both fascinating.
Lex Fridman (27:16.380)
So you have a bunch of work on this,
Lex Fridman (27:19.820)
but so one of the interests for me is autonomous driving.
Lex Fridman (27:23.500)
So you have like your 2018 paper,
Lex Fridman (27:26.180)
Robust Physical World Attacks
Dawn Song (27:27.620)
on Deep Learning Visual Classification.
Lex Fridman (27:29.820)
I believe there's some stop signs in there.
Dawn Song (27:33.020)
Yeah.
Lex Fridman (27:33.860)
So that's like in the physical,
Dawn Song (27:35.660)
on the inference stage, attacking with physical objects.
Lex Fridman (27:38.620)
Can you maybe describe the ideas in that paper?
Dawn Song (27:40.780)
Sure, sure.
Lex Fridman (27:41.620)
And the stop signs are actually on exhibits
Dawn Song (27:44.980)
at the Science of Museum in London.
Lex Fridman (27:47.700)
But I'll talk about the work.
Dawn Song (27:50.020)
It's quite nice that it's a very rare occasion,
Lex Fridman (27:55.060)
I think, where these research artifacts
Dawn Song (27:57.980)
actually gets put in a museum.
Lex Fridman (28:00.340)
In a museum.
Dawn Song (28:01.180)
Right, so what the work is about is,
Lex Fridman (28:06.340)
and we talked about these adversarial examples,
Dawn Song (28:08.380)
essentially changes to inputs to the learning system
Lex Fridman (28:14.940)
to cause the learning system to give the wrong prediction.
Lex Fridman (28:19.260)
And typically these attacks have been done
Lex Fridman (28:22.100)
in the digital world,
Dawn Song (28:23.620)
where essentially the attacks are modifications
Lex Fridman (28:27.580)
to the digital image.
Lex Fridman (28:30.180)
And when you feed this modified digital image
Lex Fridman (28:32.620)
to the learning system,
Dawn Song (28:34.940)
it causes the learning system to misclassify,
Lex Fridman (28:37.260)
like a cat into a dog, for example.
Lex Fridman (28:40.660)
So autonomous driving, of course,
Lex Fridman (28:43.060)
it's really important for the vehicle
Dawn Song (28:45.700)
to be able to recognize these traffic signs
Lex Fridman (28:48.980)
in real world environments correctly.
Dawn Song (28:51.220)
Otherwise it can, of course, cause really severe consequences.
Lex Fridman (28:55.300)
So one natural question is,
Lex Fridman (28:57.860)
so one, can these adversarial examples actually exist
Lex Fridman (29:01.780)
in the physical world, not just in the digital world?
Lex Fridman (29:05.420)
And also in the autonomous driving setting,
Lex Fridman (29:08.940)
can we actually create these adversarial examples
Dawn Song (29:12.020)
in the physical world,
Lex Fridman (29:13.100)
such as a maliciously perturbed stop sign
Dawn Song (29:18.260)
to cause the image classification system to misclassify
Lex Fridman (29:23.060)
into, for example, a speed limit sign instead,
Lex Fridman (29:26.300)
so that when the car drives through,
Lex Fridman (29:30.620)
it actually won't stop.
Dawn Song (29:33.100)
Yes.
Lex Fridman (29:33.940)
So, right, so that's the...
Dawn Song (29:36.340)
That's the open question.
Lex Fridman (29:37.260)
That's the big, really, really important question
Dawn Song (29:40.220)
for machine learning systems that work in the real world.
Lex Fridman (29:42.900)
Right, right, right, exactly.
Lex Fridman (29:44.820)
And also there are many challenges
Lex Fridman (29:47.340)
when you move from the digital world
Dawn Song (29:49.500)
into the physical world.
Lex Fridman (29:50.900)
So in this case, for example, we want to make sure,
Dawn Song (29:53.060)
we want to check whether these adversarial examples,
Lex Fridman (29:56.580)
not only that they can be effective in the physical world,
Lex Fridman (29:59.900)
but also whether they can remain effective
Lex Fridman (2:00:01.580)
So let me ask...
Dawn Song (2:00:02.420)
Right, I think with your own hands.
Lex Fridman (2:00:03.780)
Let me ask a silly question,
Dawn Song (2:00:05.860)
or maybe the ultimate question.
Lex Fridman (2:00:07.860)
What is to you the meaning of life?
Lex Fridman (2:00:11.740)
What gives your life meaning, purpose,
Lex Fridman (2:00:15.140)
fulfillment, happiness, joy?
Dawn Song (2:00:19.220)
Okay, these are two different questions.
Lex Fridman (2:00:21.100)
Very different, yeah.
Dawn Song (2:00:22.500)
It's usually that you ask this question.
Lex Fridman (2:00:24.900)
Maybe this question is probably the question
Dawn Song (2:00:28.020)
that has followed me and followed my life the most.
Lex Fridman (2:00:32.740)
Have you discovered anything,
Lex Fridman (2:00:34.860)
any satisfactory answer for yourself?
Lex Fridman (2:00:38.780)
Is there something you've arrived at?
Dawn Song (2:00:41.620)
You know, there's a moment...
Lex Fridman (2:00:44.260)
I've talked to a few people who have faced,
Dawn Song (2:00:46.980)
for example, a cancer diagnosis,
Lex Fridman (2:00:48.740)
or faced their own mortality,
Lex Fridman (2:00:50.700)
and that seems to change their view of them.
Lex Fridman (2:00:53.700)
It seems to be a catalyst for them
Dawn Song (2:00:56.580)
removing most of the crap.
Lex Fridman (2:00:59.460)
Of seeing that most of what they've been doing
Dawn Song (2:01:02.620)
is not that important,
Lex Fridman (2:01:04.140)
and really reducing it into saying, like,
Dawn Song (2:01:06.740)
here's actually the few things that really give meaning.
Lex Fridman (2:01:11.580)
Mortality is a really powerful catalyst for that,
Dawn Song (2:01:14.780)
it seems like.
Lex Fridman (2:01:15.740)
Facing mortality, whether it's your parents dying
Dawn Song (2:01:17.860)
or somebody close to you dying,
Lex Fridman (2:01:19.420)
or facing your own death for whatever reason,
Dawn Song (2:01:22.020)
or cancer and so on.
Lex Fridman (2:01:23.460)
So yeah, so in my own case,
Dawn Song (2:01:26.460)
I didn't need to face mortality, too.
Lex Fridman (2:01:28.500)
So try to ask that question.
Lex Fridman (2:01:35.980)
And I think there are a couple things.
Lex Fridman (2:01:38.860)
So one is, like, who should be defining
Lex Fridman (2:01:42.700)
the meaning of your life, right?
Lex Fridman (2:01:44.860)
Is there some kind of even greater things than you
Lex Fridman (2:01:49.020)
who should define the meaning of your life?
Lex Fridman (2:01:51.580)
So for example, when people say that
Dawn Song (2:01:53.900)
searching the meaning for your life,
Lex Fridman (2:01:56.740)
is there some outside voice,
Dawn Song (2:02:00.380)
or is there something outside of you
Lex Fridman (2:02:04.300)
who actually tells you, you know...
Lex Fridman (2:02:06.020)
So people talk about, oh, you know,
Lex Fridman (2:02:09.260)
this is what you have been born to do, right?
Dawn Song (2:02:14.700)
Like, this is your destiny.
Lex Fridman (2:02:19.700)
So who, right, so that's one question,
Lex Fridman (2:02:21.820)
like, who gets to define the meaning of your life?
Lex Fridman (2:02:24.860)
Should you be finding some other things,
Lex Fridman (2:02:27.980)
some other factor to define this for you?
Lex Fridman (2:02:30.860)
Or is something actually,
Dawn Song (2:02:32.380)
it's just entirely what you define yourself,
Lex Fridman (2:02:35.140)
and it can be very arbitrary.
Dawn Song (2:02:37.380)
Yeah, so an inner voice or an outer voice,
Lex Fridman (2:02:41.580)
whether it could be spiritual or religious, too, with God,
Dawn Song (2:02:44.780)
or some other components of the environment outside of you,
Lex Fridman (2:02:48.300)
or just your own voice.
Lex Fridman (2:02:50.180)
Do you have an answer there?
Lex Fridman (2:02:52.420)
So, okay, so for that, I have an answer.
Lex Fridman (2:02:55.020)
And through, you know, the long period of time
Lex Fridman (2:02:58.460)
of thinking and searching,
Dawn Song (2:03:00.620)
even searching through outsides, right,
Lex Fridman (2:03:04.620)
you know, voices or factors outside of me.
Lex Fridman (2:03:08.260)
So that, I have an answer.
Lex Fridman (2:03:09.740)
I've come to the conclusion and realization
Dawn Song (2:03:13.060)
that it's you yourself that defines the meaning of life.
Lex Fridman (2:03:18.140)
Yeah, that's a big burden, though, isn't it?
Lex Fridman (2:03:20.300)
I mean, yes and no, right?
Lex Fridman (2:03:26.020)
So then you have the freedom to define it.
Dawn Song (2:03:28.140)
Yes.
Lex Fridman (2:03:29.540)
And another question is, like,
Lex Fridman (2:03:33.020)
what does it really mean by the meaning of life?
Lex Fridman (2:03:37.300)
Right.
Lex Fridman (2:03:39.700)
And also, whether the question even makes sense.
Lex Fridman (2:03:45.420)
Absolutely, and you said it somehow distinct from happiness.
Lex Fridman (2:03:49.580)
So meaning is something much deeper
Lex Fridman (2:03:51.660)
than just any kind of emotional,
Dawn Song (2:03:55.020)
any kind of contentment or joy or whatever.
Lex Fridman (2:03:57.580)
It might be much deeper.
Lex Fridman (2:03:58.940)
And then you have to ask, what is deeper than that?
Lex Fridman (2:04:02.580)
What is there at all?
Lex Fridman (2:04:04.620)
And then the question starts being silly.
Lex Fridman (2:04:07.780)
Right, and also you can say it's deeper,
Lex Fridman (2:04:09.540)
but you can also say it's shallower,
Lex Fridman (2:04:10.940)
depending on how people want to define
Dawn Song (2:04:13.500)
the meaning of their life.
Lex Fridman (2:04:14.700)
So for example, most people don't even think
Dawn Song (2:04:16.460)
about this question.
Lex Fridman (2:04:17.620)
Then the meaning of life to them
Dawn Song (2:04:19.540)
doesn't really matter that much.
Lex Fridman (2:04:22.020)
And also, whether knowing the meaning of life,
Dawn Song (2:04:26.340)
whether it actually helps your life to be better
Lex Fridman (2:04:28.940)
or whether it helps your life to be happier,
Dawn Song (2:04:31.140)
these actually are open questions.
Lex Fridman (2:04:34.500)
It's not, right?
Dawn Song (2:04:36.140)
Of course, most questions are open.
Lex Fridman (2:04:37.700)
I tend to think that just asking the question,
Dawn Song (2:04:40.180)
as you mentioned, as you've done for a long time,
Lex Fridman (2:04:42.740)
is the only, that there is no answer.
Lex Fridman (2:04:44.900)
And asking the question is a really good exercise.
Lex Fridman (2:04:47.620)
I mean, I have this, for me personally,
Dawn Song (2:04:49.100)
I've had a kind of feeling that creation is,
Lex Fridman (2:04:56.140)
like for me has been very fulfilling.
Lex Fridman (2:04:58.140)
And it seems like my meaning has been to create.
Lex Fridman (2:05:00.820)
And I'm not sure what that is.
Dawn Song (2:05:02.100)
Like I don't have, I'm single and I don't have kids.
Lex Fridman (2:05:05.220)
I'd love to have kids, but I also, sounds creepy,
Lex Fridman (2:05:08.940)
but I also see sort of, you said see programs.
Lex Fridman (2:05:13.340)
I see programs as little creations.
Dawn Song (2:05:15.660)
I see robots as little creations.
Lex Fridman (2:05:19.060)
I think those bring, and then ideas,
Dawn Song (2:05:22.660)
theorems are creations.
Lex Fridman (2:05:25.140)
And those somehow intrinsically, like you said,
Dawn Song (2:05:28.780)
bring me joy.
Lex Fridman (2:05:29.620)
I think they do to a lot of, at least scientists,
Lex Fridman (2:05:31.740)
but I think they do to a lot of people.
Lex Fridman (2:05:34.180)
So that, to me, if I had to force the answer to that,
Dawn Song (2:05:37.300)
I would say creating new things yourself.
Lex Fridman (2:05:43.180)
For you.
Dawn Song (2:05:44.020)
For me, for me, for me.
Lex Fridman (2:05:45.500)
I don't know, but like you said, it keeps changing.
Lex Fridman (2:05:48.580)
Is there some answer that?
Lex Fridman (2:05:49.900)
And some people, they can, I think,
Lex Fridman (2:05:52.300)
they may say it's experience, right?
Lex Fridman (2:05:54.380)
Like their meaning of life,
Dawn Song (2:05:56.460)
they just want to experience
Lex Fridman (2:05:57.740)
to the richest and fullest they can.
Lex Fridman (2:05:59.940)
And a lot of people do take that path.
Lex Fridman (2:06:02.700)
Yes, seeing life as actually a collection of moments
Lex Fridman (2:06:05.540)
and then trying to make the richest possible sets,
Lex Fridman (2:06:10.740)
fill those moments with the richest possible experiences.
Dawn Song (2:06:13.940)
Right.
Lex Fridman (2:06:14.780)
And for me, I think it's certainly,
Dawn Song (2:06:16.420)
we do share a lot of similarity here.
Lex Fridman (2:06:18.260)
So creation is also really important for me,
Dawn Song (2:06:20.420)
even from the things I've already talked about,
Lex Fridman (2:06:24.740)
even like writing papers,
Lex Fridman (2:06:26.140)
and these are all creations as well.
Lex Fridman (2:06:30.140)
And I have not quite thought
Dawn Song (2:06:32.620)
whether that is really the meaning of my life.
Lex Fridman (2:06:34.860)
Like in a sense, also then maybe like,
Lex Fridman (2:06:37.260)
what kind of things should you create?
Lex Fridman (2:06:38.380)
There are so many different things that you could create.
Lex Fridman (2:06:42.660)
And also you can say, another view is maybe growth.
Lex Fridman (2:06:46.380)
It's related, but different from experience.
Dawn Song (2:06:50.580)
Growth is also maybe type of meaning of life.
Lex Fridman (2:06:53.420)
It's just, you try to grow every day,
Dawn Song (2:06:55.740)
try to be a better self every day.
Lex Fridman (2:06:59.740)
And also ultimately, we are here,
Dawn Song (2:07:04.420)
it's part of the overall evolution.
Lex Fridman (2:07:09.140)
Right, the world is evolving and it's growing.
Dawn Song (2:07:11.780)
Isn't it funny that the growth seems to be
Lex Fridman (2:07:14.580)
the more important thing
Dawn Song (2:07:15.620)
than the thing you're growing towards.
Lex Fridman (2:07:18.100)
It's like, it's not the goal, it's the journey to it.
Dawn Song (2:07:21.540)
It's almost when you submit a paper,
Lex Fridman (2:07:27.020)
there's a sort of depressing element to it,
Dawn Song (2:07:29.220)
not to submit a paper,
Lex Fridman (2:07:30.220)
but when that whole project is over.
Dawn Song (2:07:32.340)
I mean, there's the gratitude,
Lex Fridman (2:07:34.020)
there's the celebration and so on,
Lex Fridman (2:07:35.260)
but you're usually immediately looking for the next thing
Lex Fridman (2:07:39.300)
or the next step, right?
Dawn Song (2:07:40.500)
It's not that, the end of it is not the satisfaction,
Lex Fridman (2:07:44.380)
it's the hardship, the challenge you have to overcome,
Dawn Song (2:07:47.180)
the growth through the process.
Lex Fridman (2:07:48.780)
It's somehow probably deeply within us,
Dawn Song (2:07:51.340)
the same thing that drives the evolutionary process
Lex Fridman (2:07:54.420)
is somehow within us,
Dawn Song (2:07:55.900)
with everything the way we see the world.
Lex Fridman (2:07:58.860)
Since you're thinking about these,
Lex Fridman (2:08:00.100)
so you're still in search of an answer.
Lex Fridman (2:08:02.820)
I mean, yes and no,
Dawn Song (2:08:05.420)
in the sense that I think for people
Lex Fridman (2:08:07.780)
who really dedicate time to search for the answer
Lex Fridman (2:08:11.940)
to ask the question, what is the meaning of life?
Lex Fridman (2:08:15.700)
It does not necessarily bring you happiness.
Dawn Song (2:08:18.180)
Yeah.
Lex Fridman (2:08:20.460)
It's a question, we can say, right?
Dawn Song (2:08:23.740)
Like whether it's a well defined question.
Lex Fridman (2:08:25.700)
And, but on the other hand,
Dawn Song (2:08:30.180)
given that you get to answer it yourself,
Lex Fridman (2:08:33.860)
you can define it yourself,
Dawn Song (2:08:35.740)
then sure, I can just give it an answer.
Lex Fridman (2:08:41.180)
And in that sense, yes, it can help.
Lex Fridman (2:08:46.420)
Like we discussed, right?
Lex Fridman (2:08:47.860)
If you say, oh, then my meaning of life is to create
Dawn Song (2:08:52.900)
or to grow, then yes, then I think they can help.
Lex Fridman (2:08:57.380)
But how do you know that that is really the meaning of life
Lex Fridman (2:09:00.380)
or the meaning of your life?
Lex Fridman (2:09:02.060)
It's like there's no way for you
Dawn Song (2:09:04.620)
to really answer the question.
Lex Fridman (2:09:05.740)
Sure, but something about that certainty is liberating.
Lex Fridman (2:09:10.060)
So it might be an illusion, you might not really know,
Lex Fridman (2:09:12.820)
you might be just convincing yourself falsely,
Lex Fridman (2:09:15.580)
but being sure that that's the meaning,
Lex Fridman (2:09:18.020)
there's something liberating in that.
Dawn Song (2:09:23.340)
There's something freeing in knowing this is your purpose.
Lex Fridman (2:09:26.340)
So you can fully give yourself to that.
Dawn Song (2:09:29.060)
Without, you know, for a long time,
Lex Fridman (2:09:30.700)
you know, I thought like, isn't it all relative?
Lex Fridman (2:09:33.220)
Like why, how do we even know what's good and what's evil?
Lex Fridman (2:09:38.140)
Like isn't everything just relative?
Dawn Song (2:09:39.900)
Like how do we know, you know,
Lex Fridman (2:09:42.740)
the question of meaning is ultimately
Lex Fridman (2:09:44.940)
the question of why do anything?
Lex Fridman (2:09:48.380)
Why is anything good or bad?
Lex Fridman (2:09:50.260)
Why is anything valuable and so on?
Lex Fridman (2:09:52.580)
Exactly.
Dawn Song (2:09:53.580)
Then you start to, I think just like you said,
Lex Fridman (2:09:58.380)
I think it's a really useful question to ask,
Lex Fridman (2:10:02.140)
but if you ask it for too long and too aggressively.
Lex Fridman (2:10:07.660)
It may not be so productive.
Dawn Song (2:10:08.820)
It may not be productive and not just for traditionally
Lex Fridman (2:10:13.340)
societally defined success, but also for happiness.
Dawn Song (2:10:17.260)
It seems like asking the question about the meaning of life
Lex Fridman (2:10:20.420)
is like a trap.
Dawn Song (2:10:24.460)
We're destined to be asking.
Lex Fridman (2:10:25.820)
We're destined to look up to the stars
Lex Fridman (2:10:27.340)
and ask these big why questions
Lex Fridman (2:10:28.780)
we'll never be able to answer,
Lex Fridman (2:10:30.500)
but we shouldn't get lost in them.
Lex Fridman (2:10:31.980)
I think that's probably the,
Dawn Song (2:10:34.180)
that's at least the lesson I picked up so far.
Lex Fridman (2:10:36.260)
On that topic.
Dawn Song (2:10:37.540)
Oh, let me just add one more thing.
Lex Fridman (2:10:38.820)
So it's interesting.
Lex Fridman (2:10:40.020)
So sometimes, yes, it can help you to focus.
Lex Fridman (2:10:47.020)
So when I shifted my focus more from security
Dawn Song (2:10:53.300)
to AI and machine learning,
Lex Fridman (2:10:55.140)
at the time, actually one of the main reasons
Dawn Song (2:10:58.500)
that I did that was because at the time,
Lex Fridman (2:11:02.820)
I thought the meaning of my life
Lex Fridman (2:11:07.380)
and the purpose of my life is to build intelligent machines.
Lex Fridman (2:11:14.020)
And that's, and then your inner voice said
Dawn Song (2:11:16.620)
that this is the right,
Lex Fridman (2:11:18.580)
this is the right journey to take
Dawn Song (2:11:20.060)
to build intelligent machines
Lex Fridman (2:11:21.340)
and that you actually fully realize
Dawn Song (2:11:23.380)
you took a really legitimate big step
Lex Fridman (2:11:26.420)
to become one of the world class researchers
Dawn Song (2:11:28.460)
to actually make it, to actually go down that journey.
Lex Fridman (2:11:32.540)
Yeah, that's profound.
Dawn Song (2:11:35.340)
That's profound.
Lex Fridman (2:11:36.460)
I don't think there's a better way
Dawn Song (2:11:39.380)
to end a conversation than talking for a while
Lex Fridman (2:11:42.980)
about the meaning of life.
Dawn Song (2:11:44.060)
Dawn is a huge honor to talk to you.
Lex Fridman (2:11:46.020)
Thank you so much for talking today.
Dawn Song (2:11:47.540)
Thank you, thank you.
Lex Fridman (2:11:49.900)
Thanks for listening to this conversation with Dawn Song
Lex Fridman (2:11:52.580)
and thank you to our presenting sponsor, Cash App.
Lex Fridman (2:11:55.380)
Please consider supporting the podcast
Dawn Song (2:11:57.100)
by downloading Cash App and using code LexPodcast.
Lex Fridman (2:12:01.140)
If you enjoy this podcast, subscribe on YouTube,
Dawn Song (2:12:03.860)
review it with five stars on Apple Podcast,
Lex Fridman (2:12:06.140)
support it on Patreon,
Dawn Song (2:12:07.340)
or simply connect with me on Twitter at LexFriedman.
Lex Fridman (2:12:11.500)
And now let me leave you with some words about hacking
Dawn Song (2:12:15.100)
from the great Steve Wozniak.
Lex Fridman (2:12:17.900)
A lot of hacking is playing with other people,
Dawn Song (2:12:20.740)
you know, getting them to do strange things.
Lex Fridman (2:12:24.340)
Thank you for listening and hope to see you next time.
Dawn Song (30:03.340)
under different viewing distances, different viewing angles,
Lex Fridman (30:06.140)
because as a car, right, because as a car drives by,
Lex Fridman (30:09.940)
and it's going to view the traffic sign
Lex Fridman (30:13.100)
from different viewing distances, different angles,
Lex Fridman (30:15.500)
and different viewing conditions and so on.
Lex Fridman (30:17.260)
So that's a question that we set out to explore.
Lex Fridman (30:20.180)
Is there good answers?
Lex Fridman (30:21.740)
So, yeah, right, so unfortunately the answer is yes.
Dawn Song (30:25.300)
So, right, that is...
Lex Fridman (30:26.140)
So it's possible to have a physical,
Lex Fridman (30:28.660)
so adversarial attacks in the physical world
Lex Fridman (30:30.820)
that are robust to this kind of viewing distance,
Dawn Song (30:33.620)
viewing angle, and so on.
Lex Fridman (30:35.100)
Right, exactly.
Dawn Song (30:36.180)
So, right, so we actually created these adversarial examples
Lex Fridman (30:40.620)
in the real world, so like this adversarial example,
Dawn Song (30:44.140)
stop signs.
Lex Fridman (30:44.980)
So these are the stop signs,
Dawn Song (30:46.620)
these are the traffic signs that have been put
Lex Fridman (30:49.140)
in the Science of Museum in London exhibit.
Dawn Song (30:53.900)
Yeah.
Lex Fridman (30:55.700)
So what goes into the design of objects like that?
Dawn Song (30:59.940)
If you could just high level insights
Lex Fridman (31:02.780)
into the step from digital to the physical,
Dawn Song (31:06.660)
because that is a huge step from trying to be robust
Lex Fridman (31:11.660)
to the different distances and viewing angles
Lex Fridman (31:13.820)
and lighting conditions.
Lex Fridman (31:15.260)
Right, right, exactly.
Lex Fridman (31:16.340)
So to create a successful adversarial example
Lex Fridman (31:19.900)
that actually works in the physical world
Dawn Song (31:21.740)
is much more challenging than just in the digital world.
Lex Fridman (31:26.140)
So first of all, again, in the digital world,
Dawn Song (31:28.260)
if you just have an image, then there's no,
Lex Fridman (31:32.340)
you don't need to worry about this viewing distance
Lex Fridman (31:35.100)
and angle changes and so on.
Lex Fridman (31:36.540)
So one is the environmental variation.
Lex Fridman (31:39.820)
And also, typically actually what you'll see
Lex Fridman (31:42.900)
when people add preservation to a digital image
Dawn Song (31:47.580)
to create these digital adversarial examples
Lex Fridman (31:50.540)
is that you can add these perturbations
Dawn Song (31:52.660)
anywhere in the image.
Lex Fridman (31:54.380)
Right.
Dawn Song (31:55.220)
In our case, we have a physical object, a traffic sign,
Lex Fridman (31:59.340)
that's put in the real world.
Dawn Song (32:01.140)
We can't just add perturbations elsewhere.
Lex Fridman (32:04.660)
We can't add preservation outside of the traffic sign.
Dawn Song (32:08.260)
It has to be on the traffic sign.
Lex Fridman (32:09.940)
So there's a physical constraints
Dawn Song (32:12.420)
where you can add perturbations.
Lex Fridman (32:15.100)
And also, so we have the physical objects,
Dawn Song (32:20.580)
this adversarial example,
Lex Fridman (32:21.780)
and then essentially there's a camera
Dawn Song (32:23.740)
that will be taking pictures
Lex Fridman (32:26.540)
and then feeding that to the learning system.
Lex Fridman (32:30.660)
So in the digital world,
Lex Fridman (32:31.500)
you can have really small perturbations
Dawn Song (32:33.220)
because you are editing the digital image directly
Lex Fridman (32:37.180)
and then feeding that directly to the learning system.
Lex Fridman (32:40.540)
So even really small perturbations,
Lex Fridman (32:42.420)
it can cause a difference in inputs to the learning system.
Lex Fridman (32:46.900)
But in the physical world,
Lex Fridman (32:47.980)
because you need a camera to actually take the picture
Dawn Song (32:52.980)
as an input and then feed it to the learning system,
Lex Fridman (32:55.820)
we have to make sure that the changes are perceptible enough
Dawn Song (33:01.420)
that actually can cause difference from the camera side.
Lex Fridman (33:03.820)
So we want it to be small,
Lex Fridman (33:05.180)
but still can cause a difference
Lex Fridman (33:08.740)
after the camera has taken the picture.
Dawn Song (33:11.540)
Right, because you can't directly modify the picture
Lex Fridman (33:14.180)
that the camera sees at the point of the capture.
Dawn Song (33:17.700)
Right, so there's a physical sensor step,
Lex Fridman (33:19.620)
physical sensing step.
Dawn Song (33:20.860)
That you're on the other side of now.
Lex Fridman (33:22.660)
Right, and also how do we actually change
Lex Fridman (33:27.100)
the physical objects?
Lex Fridman (33:28.540)
So essentially in our experiment,
Dawn Song (33:29.700)
we did multiple different things.
Lex Fridman (33:31.260)
We can print out these stickers and put a sticker on.
Dawn Song (33:34.620)
We actually bought these real world stuff signs
Lex Fridman (33:38.060)
and then we printed stickers and put stickers on them.
Lex Fridman (33:41.420)
And so then in this case,
Lex Fridman (33:43.780)
we also have to handle this printing step.
Lex Fridman (33:48.300)
So again, in the digital world,
Lex Fridman (33:50.780)
it's just bits.
Dawn Song (33:52.980)
You just change the color value or whatever.
Lex Fridman (33:55.740)
You can just change the bits directly.
Lex Fridman (33:58.060)
So you can try a lot of things too.
Lex Fridman (33:59.860)
Right, you're right.
Lex Fridman (34:00.820)
But in the physical world, you have the printer.
Lex Fridman (34:04.060)
Whatever attack you want to do,
Dawn Song (34:05.940)
in the end you have a printer that prints out these stickers
Lex Fridman (34:09.380)
or whatever perturbation you want to do.
Lex Fridman (34:11.500)
And then they will put it on the object.
Lex Fridman (34:13.980)
So we also essentially,
Dawn Song (34:16.260)
there's constraints what can be done there.
Lex Fridman (34:19.580)
So essentially there are many of these additional constraints
Dawn Song (34:24.180)
that you don't have in the digital world.
Lex Fridman (34:25.780)
And then when we create the adversarial example,
Dawn Song (34:28.500)
we have to take all these into consideration.
Lex Fridman (34:30.660)
So how much of the creation of the adversarial examples,
Lex Fridman (34:33.660)
art and how much is science?
Lex Fridman (34:35.900)
Sort of how much is this sort of trial and error,
Dawn Song (34:38.260)
trying to figure, trying different things,
Lex Fridman (34:40.500)
empirical sort of experiments
Lex Fridman (34:42.260)
and how much can be done sort of almost theoretically
Lex Fridman (34:47.260)
or by looking at the model,
Dawn Song (34:49.460)
by looking at the neural network,
Lex Fridman (34:50.660)
trying to generate sort of definitively
Lex Fridman (34:56.540)
what the kind of stickers would be most likely to create,
Lex Fridman (35:01.580)
to be a good adversarial example in the physical world.
Dawn Song (35:04.460)
Right, that's a very good question.
Lex Fridman (35:06.660)
So essentially I would say it's mostly science
Dawn Song (35:08.900)
in the sense that we do have a scientific way
Lex Fridman (35:13.580)
of computing what the adversarial example,
Lex Fridman (35:17.700)
what is the adversarial preservation we should add.
Lex Fridman (35:20.380)
And then, and of course in the end,
Dawn Song (35:23.500)
because of these additional steps,
Lex Fridman (35:25.300)
as I mentioned, you have to print it out
Lex Fridman (35:26.660)
and then you have to put it on
Lex Fridman (35:28.860)
and then you have to take the camera.
Lex Fridman (35:30.820)
So there are additional steps
Lex Fridman (35:32.140)
that you do need to do additional testing,
Lex Fridman (35:34.060)
but the creation process of generating the adversarial example
Lex Fridman (35:39.060)
is really a very scientific approach.
Dawn Song (35:44.060)
Essentially we capture many of these constraints,
Lex Fridman (35:48.620)
as we mentioned, in this loss function
Dawn Song (35:52.260)
that we optimize for.
Lex Fridman (35:55.180)
And so that's a very scientific approach.
Lex Fridman (35:58.740)
So the fascinating fact
Lex Fridman (36:00.460)
that we can do these kinds of adversarial examples,
Lex Fridman (36:02.660)
what do you think it shows us?
Lex Fridman (36:06.100)
Just your thoughts in general,
Lex Fridman (36:07.460)
what do you think it reveals to us about neural networks,
Lex Fridman (36:10.020)
the fact that this is possible?
Lex Fridman (36:12.100)
What do you think it reveals to us
Lex Fridman (36:13.420)
about our machine learning approaches of today?
Lex Fridman (36:16.340)
Is there something interesting?
Lex Fridman (36:17.780)
Is it a feature, is it a bug?
Lex Fridman (36:19.500)
What do you think?
Lex Fridman (36:21.860)
I think it really shows that we are still
Dawn Song (36:23.740)
at a very early stage of really developing robust
Lex Fridman (36:29.900)
and generalizable machine learning methods.
Lex Fridman (36:33.460)
And it shows that we, even though deep learning
Lex Fridman (36:36.860)
has made so much advancements,
Lex Fridman (36:39.420)
but our understanding is very limited.
Lex Fridman (36:42.220)
We don't fully understand,
Dawn Song (36:44.100)
or we don't understand well how they work, why they work,
Lex Fridman (36:47.260)
and also we don't understand that well,
Dawn Song (36:50.060)
right, about these adversarial examples.
Lex Fridman (36:54.900)
Some people have kind of written about the fact
Dawn Song (36:56.900)
that the fact that the adversarial examples work well
Lex Fridman (37:02.820)
is actually sort of a feature, not a bug.
Dawn Song (37:04.940)
It's that actually they have learned really well
Lex Fridman (37:09.220)
to tell the important differences between classes
Dawn Song (37:12.020)
as represented by the training set.
Lex Fridman (37:14.140)
I think that's the other thing I was going to say,
Dawn Song (37:15.660)
is that it shows us also that the deep learning systems
Lex Fridman (37:18.940)
are not learning the right things.
Lex Fridman (37:21.180)
How do we make them, I mean,
Lex Fridman (37:23.380)
I guess this might be a place to ask about
Lex Fridman (37:26.340)
how do we then defend, or how do we either defend
Lex Fridman (37:30.100)
or make them more robust, these adversarial examples?
Dawn Song (37:32.820)
Right, I mean, one thing is that I think,
Lex Fridman (37:35.220)
you know, people, so there have been actually
Dawn Song (37:37.740)
thousands of papers now written on this topic.
Lex Fridman (37:41.580)
The defense or the attacks?
Dawn Song (37:43.780)
Mostly attacks.
Lex Fridman (37:45.140)
I think there are more attack papers than defenses,
Lex Fridman (37:48.500)
but there are many hundreds of defense papers as well.
Lex Fridman (37:53.180)
So in defenses, a lot of work has been trying to,
Dawn Song (37:58.540)
I would call it more like a patchwork.
Lex Fridman (38:02.020)
For example, how to make the neural networks
Dawn Song (38:05.380)
to either through, for example, like adversarial training,
Lex Fridman (38:09.700)
how to make them a little bit more resilient.
Dawn Song (38:13.340)
Got it.
Lex Fridman (38:14.460)
But I think in general, it has limited effectiveness
Lex Fridman (38:21.300)
and we don't really have very strong and general defense.
Lex Fridman (38:27.940)
So part of that, I think, is we talked about
Dawn Song (38:30.180)
in deep learning, the goal is to learn representations.
Lex Fridman (38:33.780)
And that's our ultimate, you know,
Dawn Song (38:36.980)
holy grail, ultimate goal is to learn representations.
Lex Fridman (38:39.940)
But one thing I think I have to say is that
Dawn Song (38:42.980)
I think part of the lesson we are learning here is that
Lex Fridman (38:44.940)
one, as I mentioned, we are not learning the right things,
Dawn Song (38:47.500)
meaning we are not learning the right representations.
Lex Fridman (38:49.820)
And also, I think the representations we are learning
Dawn Song (38:51.940)
is not rich enough.
Lex Fridman (38:54.580)
And so it's just like a human vision.
Dawn Song (38:56.860)
Of course, we don't fully understand how human visions work,
Lex Fridman (38:59.580)
but when humans look at the world, we don't just say,
Dawn Song (39:02.820)
oh, you know, this is a person.
Lex Fridman (39:04.420)
Oh, there's a camera.
Dawn Song (39:06.100)
We actually get much more nuanced information
Lex Fridman (39:09.060)
from the world.
Lex Fridman (39:11.780)
And we use all this information together in the end
Lex Fridman (39:14.820)
to derive, to help us to do motion planning
Lex Fridman (39:17.700)
and to do other things, but also to classify
Lex Fridman (39:20.620)
what the object is and so on.
Lex Fridman (39:22.180)
So we are learning a much richer representation.
Lex Fridman (39:24.580)
And I think that that's something we have not figured out
Lex Fridman (39:27.660)
how to do in deep learning.
Lex Fridman (39:30.580)
And I think the richer representation will also help us
Dawn Song (39:34.060)
to build a more generalizable
Lex Fridman (39:36.420)
and more resilient learning system.
Lex Fridman (39:39.100)
Can you maybe linger on the idea
Lex Fridman (39:40.700)
of the word richer representation?
Lex Fridman (39:43.180)
So to make representations more generalizable,
Lex Fridman (39:50.260)
it seems like you want to make them less sensitive to noise.
Dawn Song (39:55.260)
Right, so you want to learn the right things.
Lex Fridman (39:58.380)
You don't want to, for example,
Dawn Song (39:59.980)
learn this spurious correlations and so on.
Lex Fridman (40:05.340)
But at the same time, an example of a richer information,
Dawn Song (40:09.580)
our representation is like, again,
Lex Fridman (40:11.740)
we don't really know how human vision works,
Lex Fridman (40:14.860)
but when we look at the visual world,
Lex Fridman (40:18.060)
we actually, we can identify counters.
Dawn Song (40:20.780)
We can identify much more information
Lex Fridman (40:24.660)
than just what's, for example,
Dawn Song (40:26.860)
image classification system is trying to do.
Lex Fridman (40:30.460)
And that leads to, I think,
Dawn Song (40:32.340)
the question you asked earlier about defenses.
Lex Fridman (40:34.540)
So that's also in terms of more promising directions
Dawn Song (40:38.540)
for defenses.
Lex Fridman (40:39.900)
And that's where some of my work is trying to do
Lex Fridman (40:44.380)
and trying to show as well.
Lex Fridman (40:46.460)
You have, for example, in your 2018 paper,
Dawn Song (40:49.100)
characterizing adversarial examples
Lex Fridman (40:50.940)
based on spatial consistency,
Dawn Song (40:53.220)
information for semantic segmentation.
Lex Fridman (40:55.340)
So that's looking at some ideas
Dawn Song (40:57.140)
on how to detect adversarial examples.
Lex Fridman (41:00.940)
So like, I guess, what are they?
Dawn Song (41:02.940)
You call them like a poison data set.
Lex Fridman (41:04.780)
So like, yeah, adversarial bad examples
Dawn Song (41:07.780)
in a segmentation data set.
Lex Fridman (41:09.380)
Can you, as an example for that paper,
Lex Fridman (41:11.860)
can you describe the process of defense there?
Lex Fridman (41:13.940)
Yeah, sure, sure.
Lex Fridman (41:14.900)
So in that paper, what we look at
Lex Fridman (41:17.180)
is the semantic segmentation task.
Lex Fridman (41:20.980)
So with the task essentially given an image for each pixel,
Lex Fridman (41:24.300)
you want to say what the label is for the pixel.
Lex Fridman (41:28.220)
So just like what we talked about for adversarial example,
Lex Fridman (41:32.460)
it can easily fill image classification systems.
Dawn Song (41:35.340)
It turns out that it can also very easily
Lex Fridman (41:37.980)
fill these segmentation systems as well.
Lex Fridman (41:41.060)
So given an image, I essentially can
Lex Fridman (41:43.820)
add adversarial perturbation to the image
Dawn Song (41:46.100)
to cause the segmentation system
Lex Fridman (41:49.420)
to basically segment it in any pageant I wanted.
Lex Fridman (41:53.460)
So in that paper, we also showed that you can segment it,
Lex Fridman (41:58.020)
even though there's no kitty in the image,
Dawn Song (42:01.260)
we can segment it into like a kitty pattern,
Lex Fridman (42:05.020)
a Hello Kitty pattern.
Dawn Song (42:06.860)
We segment it into like ICCV.
Lex Fridman (42:09.300)
That's awesome.
Dawn Song (42:11.380)
Right, so that's on the attack side,
Lex Fridman (42:13.980)
showing us the segmentation system,
Dawn Song (42:15.660)
even though they have been effective in practice,
Lex Fridman (42:19.980)
but at the same time, they're really, really easily filled.
Lex Fridman (42:24.020)
So then the question is, how can we defend against this?
Lex Fridman (42:26.540)
How we can build a more resilient segmentation system?
Lex Fridman (42:30.700)
So that's what we try to do.
Lex Fridman (42:34.220)
And in particular, what we are trying to do here
Dawn Song (42:36.900)
is to actually try to leverage
Lex Fridman (42:39.020)
some natural constraints in the task,
Dawn Song (42:42.180)
which we call in this case, Spatial Consistency.
Lex Fridman (42:46.300)
So the idea of the Spatial Consistency is the following.
Lex Fridman (42:50.940)
So again, we don't really know how human vision works,
Lex Fridman (42:54.180)
but in general, at least what we can say is,
Lex Fridman (42:57.860)
so for example, as a person looks at a scene,
Lex Fridman (43:02.140)
and we can segment the scene easily.
Dawn Song (43:06.300)
We humans.
Lex Fridman (43:07.420)
Right, yes.
Dawn Song (43:08.780)
Yes, and then if you pick like two patches of the scene
Lex Fridman (43:14.100)
that has an intersection,
Lex Fridman (43:16.340)
and for humans, if you segment patch A and patch B,
Lex Fridman (43:22.220)
and then you look at the segmentation results,
Lex Fridman (43:24.620)
and especially if you look at the segmentation results
Lex Fridman (43:27.100)
at the intersection of the two patches,
Dawn Song (43:29.820)
they should be consistent in the sense that
Lex Fridman (43:32.020)
what the label, what the pixels in this intersection,
Lex Fridman (43:36.940)
what their labels should be,
Lex Fridman (43:38.900)
and they essentially from these two different patches,
Lex Fridman (43:42.140)
they should be similar in the intersection, right?
Lex Fridman (43:45.540)
So that's what we call Spatial Consistency.
Lex Fridman (43:49.060)
So similarly, for a segmentation system,
Lex Fridman (43:52.860)
it should have the same property, right?
Lex Fridman (43:56.260)
So in the image, if you pick two,
Lex Fridman (43:59.900)
randomly pick two patches that has an intersection,
Dawn Song (44:03.980)
you feed each patch to the segmentation system,
Lex Fridman (44:06.660)
you get a result,
Lex Fridman (44:08.060)
and then when you look at the results in the intersection,
Lex Fridman (44:12.060)
the results, the segmentation results should be very similar.
Dawn Song (44:16.780)
Is that, so, okay, so logically that kind of makes sense,
Lex Fridman (44:20.460)
at least it's a compelling notion,
Lex Fridman (44:21.900)
but is that, how well does that work?
Lex Fridman (44:25.100)
Does that hold true for segmentation?
Dawn Song (44:27.420)
Exactly, exactly.
Lex Fridman (44:28.260)
So then in our work and experiments, we show the following.
Lex Fridman (44:33.060)
So when we take like normal images,
Lex Fridman (44:37.300)
this actually holds pretty well
Dawn Song (44:39.260)
for the segmentation systems that we experimented with.
Lex Fridman (44:41.380)
So like natural scenes or like,
Lex Fridman (44:43.100)
did you look at like driving data sets?
Lex Fridman (44:45.060)
Right, right, right, exactly, exactly.
Lex Fridman (44:47.780)
But then this actually poses a challenge
Lex Fridman (44:49.860)
for adversarial examples,
Dawn Song (44:52.180)
because for the attacker to add perturbation to the image,
Lex Fridman (44:57.020)
then it's easy for it to fold the segmentation system
Dawn Song (45:00.940)
into, for example, for a particular patch
Lex Fridman (45:03.100)
or for the whole image to cause the segmentation system
Dawn Song (45:06.620)
to create some, to get to some wrong results.
Lex Fridman (45:10.860)
But it's actually very difficult for the attacker
Dawn Song (45:13.780)
to have this adversarial example
Lex Fridman (45:18.940)
to satisfy the spatial consistency,
Dawn Song (45:21.260)
because these patches are randomly selected
Lex Fridman (45:23.580)
and they need to ensure that this spatial consistency works.
Lex Fridman (45:27.660)
So they basically need to fold the segmentation system
Lex Fridman (45:31.340)
in a very consistent way.
Dawn Song (45:33.500)
Yeah, without knowing the mechanism
Lex Fridman (45:35.460)
by which you're selecting the patches or so on.
Dawn Song (45:37.460)
Exactly, exactly.
Lex Fridman (45:38.300)
So it has to really fold the entirety of the,
Dawn Song (45:40.540)
the mess of the entirety of the thing.
Lex Fridman (45:41.380)
Right, right, right.
Lex Fridman (45:42.220)
So it turns out to actually, to be really hard
Lex Fridman (45:44.140)
for the attacker to do.
Dawn Song (45:45.060)
We try, you know, the best we can.
Lex Fridman (45:47.300)
The state of the art attacks actually show
Dawn Song (45:50.140)
that this defense method is actually very, very effective.
Lex Fridman (45:54.420)
And this goes to, I think,
Dawn Song (45:56.140)
also what I was saying earlier is,
Lex Fridman (46:00.140)
essentially we want the learning system
Dawn Song (46:02.580)
to have richer retransition,
Lex Fridman (46:05.060)
and also to learn from more,
Dawn Song (46:07.540)
you can add the same multi model,
Lex Fridman (46:08.980)
essentially to have more ways to check
Dawn Song (46:11.460)
whether it's actually having the right prediction.
Lex Fridman (46:16.100)
So for example, in this case,
Dawn Song (46:17.580)
doing the spatial consistency check.
Lex Fridman (46:19.780)
And also actually, so that's one paper that we did.
Lex Fridman (46:22.980)
And then this is spatial consistency,
Lex Fridman (46:24.460)
this notion of consistency check,
Dawn Song (46:26.580)
it's not just limited to spatial properties,
Lex Fridman (46:30.540)
it also applies to audio.
Lex Fridman (46:32.260)
So we actually had follow up work in audio
Lex Fridman (46:35.340)
to show that this temporal consistency
Dawn Song (46:38.060)
can also be very effective
Lex Fridman (46:39.540)
in detecting adversary examples in audio.
Lex Fridman (46:42.660)
Like speech or what kind of audio?
Lex Fridman (46:44.060)
Right, right, right.
Lex Fridman (46:44.900)
Speech, speech data?
Lex Fridman (46:46.060)
Right, and then we can actually combine
Dawn Song (46:49.020)
spatial consistency and temporal consistency
Lex Fridman (46:51.780)
to help us to develop more resilient methods in video.
Lex Fridman (46:56.700)
So to defend against attacks for video also.
Lex Fridman (46:59.260)
That's fascinating.
Dawn Song (47:00.100)
Right, so yeah, so it's very interesting.
Lex Fridman (47:00.940)
So there's hope.
Dawn Song (47:01.900)
Yes, yes.
Lex Fridman (47:04.460)
But in general, in the literature
Lex Fridman (47:07.740)
and the ideas that are developing the attacks
Lex Fridman (47:09.540)
and the literature that's developing the defense,
Lex Fridman (47:11.580)
who would you say is winning right now?
Lex Fridman (47:13.820)
Right now, of course, it's attack side.
Dawn Song (47:15.900)
It's much easier to develop attacks,
Lex Fridman (47:18.500)
and there are so many different ways to develop attacks.
Dawn Song (47:21.220)
Even just us, we developed so many different methods
Lex Fridman (47:25.180)
for doing attacks.
Lex Fridman (47:27.340)
And also you can do white box attacks,
Lex Fridman (47:29.620)
you can do black box attacks,
Dawn Song (47:31.420)
where attacks you don't even need,
Lex Fridman (47:34.660)
the attacker doesn't even need to know
Dawn Song (47:36.500)
the architecture of the target system
Lex Fridman (47:39.580)
and not knowing the parameters of the target system
Lex Fridman (47:42.700)
and all that.
Lex Fridman (47:43.660)
So there are so many different types of attacks.
Lex Fridman (47:46.340)
So the counter argument that people would have,
Lex Fridman (47:49.460)
like people that are using machine learning in companies,
Dawn Song (47:52.500)
they would say, sure, in constrained environments
Lex Fridman (47:55.860)
and very specific data set,
Dawn Song (47:57.220)
when you know a lot about the model
Lex Fridman (47:59.940)
or you know a lot about the data set already,
Dawn Song (48:02.860)
you'll be able to do this attack.
Lex Fridman (48:04.300)
It's very nice.
Dawn Song (48:05.140)
It makes for a nice demo.
Lex Fridman (48:05.980)
It's a very interesting idea,
Lex Fridman (48:07.540)
but my system won't be able to be attacked like this.
Lex Fridman (48:10.580)
The real world systems won't be able to be attacked like this.
Dawn Song (48:13.940)
That's another hope,
Lex Fridman (48:16.140)
that it's actually a lot harder
Dawn Song (48:18.060)
to attack real world systems.
Lex Fridman (48:20.100)
Can you talk to that?
Lex Fridman (48:22.100)
How hard is it to attack real world systems?
Lex Fridman (48:24.700)
I wouldn't call that a hope.
Dawn Song (48:26.460)
I think it's more of a wishful thinking
Lex Fridman (48:30.060)
or trying to be lucky.
Lex Fridman (48:33.020)
So actually in our recent work,
Lex Fridman (48:37.340)
my students and collaborators
Dawn Song (48:39.260)
has shown some very effective attacks
Lex Fridman (48:41.700)
on real world systems.
Dawn Song (48:44.060)
For example, Google Translate.
Lex Fridman (48:46.180)
Oh no.
Dawn Song (48:47.020)
Other cloud translation APIs.
Lex Fridman (48:54.620)
So in this work we showed,
Lex Fridman (48:56.700)
so far I talked about adversary examples
Lex Fridman (48:58.660)
mostly in the vision category.
Lex Fridman (49:03.140)
And of course adversary examples
Lex Fridman (49:04.540)
also work in other domains as well.
Dawn Song (49:07.660)
For example, in natural language.
Lex Fridman (49:10.260)
So in this work, my students and collaborators
Dawn Song (49:14.220)
have shown that, so one,
Lex Fridman (49:17.380)
we can actually very easily steal the model
Dawn Song (49:22.020)
from for example, Google Translate
Lex Fridman (49:24.900)
by just doing queries through the APIs
Lex Fridman (49:28.460)
and then we can train an imitation model ourselves
Lex Fridman (49:32.660)
using the queries.
Lex Fridman (49:34.300)
And then once we,
Lex Fridman (49:35.620)
and also the imitation model can be very, very effective
Lex Fridman (49:40.140)
and essentially achieving similar performance
Lex Fridman (49:44.380)
as a target model.
Lex Fridman (49:45.780)
And then once we have the imitation model,
Lex Fridman (49:48.060)
we can then try to create adversary examples
Dawn Song (49:51.180)
on these imitation models.
Lex Fridman (49:52.860)
So for example, giving in the work,
Dawn Song (49:57.620)
it was one example is translating from English to German.
Lex Fridman (50:01.860)
We can give it a sentence saying,
Dawn Song (50:04.020)
for example, I'm feeling freezing.
Lex Fridman (50:06.460)
It's like six Fahrenheit and then translating to German.
Lex Fridman (50:13.220)
And then we can actually generate adversary examples
Lex Fridman (50:16.340)
that create a target translation
Dawn Song (50:18.900)
by very small perturbation.
Lex Fridman (50:20.580)
So in this case, I say we want to change the translation
Dawn Song (50:24.420)
itself six Fahrenheit to 21 Celsius.
Lex Fridman (50:30.660)
And in this particular example,
Dawn Song (50:32.340)
actually we just changed six to seven in the original
Lex Fridman (50:36.500)
sentence, that's the only change we made.
Dawn Song (50:38.580)
It caused the translation to change from the six Fahrenheit
Lex Fridman (50:44.860)
into 21 Celsius.
Dawn Song (50:46.380)
That's incredible.
Lex Fridman (50:47.420)
And then, so this example,
Dawn Song (50:49.820)
we created this example from our imitation model
Lex Fridman (50:54.060)
and then this work actually transfers
Dawn Song (50:56.980)
to the Google Translate.
Lex Fridman (50:58.700)
So the attacks that work on the imitation model,
Dawn Song (51:01.340)
in some cases at least, transfer to the original model.
Lex Fridman (51:05.380)
That's incredible and terrifying.
Dawn Song (51:07.260)
Okay, that's amazing work.
Lex Fridman (51:10.380)
And that shows that, again,
Dawn Song (51:11.900)
real world systems actually can be easily fooled.
Lex Fridman (51:15.260)
And in our previous work,
Dawn Song (51:16.420)
we also showed this type of black box attacks
Lex Fridman (51:18.620)
can be effective on cloud vision APIs as well.
Lex Fridman (51:24.220)
So that's for natural language and for vision.
Lex Fridman (51:27.740)
Let's talk about another space that people
Dawn Song (51:29.700)
have some concern about, which is autonomous driving
Lex Fridman (51:32.580)
as sort of security concerns.
Dawn Song (51:35.060)
That's another real world system.
Lex Fridman (51:36.500)
So do you have, should people be worried
Dawn Song (51:42.220)
about adversarial machine learning attacks
Lex Fridman (51:45.180)
in the context of autonomous vehicles
Dawn Song (51:47.820)
that use like Tesla Autopilot, for example,
Lex Fridman (51:50.020)
that uses vision as a primary sensor
Lex Fridman (51:52.380)
for perceiving the world and navigating that world?
Lex Fridman (51:55.580)
What do you think?
Dawn Song (51:56.620)
From your stop sign work in the physical world,
Lex Fridman (52:00.180)
should people be worried?
Lex Fridman (52:01.220)
How hard is that attack?
Lex Fridman (52:03.060)
So actually there has already been,
Dawn Song (52:05.620)
like there has always been like research shown
Lex Fridman (52:09.300)
that's, for example, actually even with Tesla,
Dawn Song (52:11.860)
like if you put a few stickers on the road,
Lex Fridman (52:15.340)
it can actually, when it's arranged in certain ways,
Dawn Song (52:17.980)
it can fool the.
Lex Fridman (52:20.660)
That's right, but I don't think it's actually been,
Dawn Song (52:23.060)
I'm not, I might not be familiar,
Lex Fridman (52:24.620)
but I don't think it's been done on physical roads yet,
Dawn Song (52:28.220)
meaning I think it's with a projector
Lex Fridman (52:29.900)
in front of the Tesla.
Lex Fridman (52:31.540)
So it's a physical, so you're on the other side
Lex Fridman (52:34.780)
of the sensor, but you're not in still the physical world.
Dawn Song (52:39.260)
The question is whether it's possible
Lex Fridman (52:41.060)
to orchestrate attacks that work in the actual,
Dawn Song (52:44.900)
like end to end attacks,
Lex Fridman (52:47.100)
like not just a demonstration of the concept,
Lex Fridman (52:49.780)
but thinking is it possible on the highway
Lex Fridman (52:52.460)
to control Tesla?
Dawn Song (52:53.620)
That kind of idea.
Lex Fridman (52:54.900)
I think there are two separate questions.
Dawn Song (52:56.460)
One is the feasibility of the attack
Lex Fridman (52:58.900)
and I'm 100% confident that the attack is possible.
Lex Fridman (53:03.660)
And there's a separate question,
Lex Fridman (53:05.580)
whether someone will actually go deploy that attack.
Dawn Song (53:10.940)
I hope people do not do that,
Lex Fridman (53:13.580)
but that's two separate questions.
Lex Fridman (53:15.820)
So the question on the word feasibility.
Lex Fridman (53:19.060)
So to clarify, feasibility means it's possible.
Dawn Song (53:22.180)
It doesn't say how hard it is,
Lex Fridman (53:25.220)
because to implement it.
Lex Fridman (53:28.220)
So sort of the barrier,
Lex Fridman (53:29.980)
like how much of a heist it has to be,
Lex Fridman (53:32.820)
like how many people have to be involved?
Lex Fridman (53:34.740)
What is the probability of success?
Dawn Song (53:36.300)
That kind of stuff.
Lex Fridman (53:37.180)
And coupled with how many evil people there are in the world
Lex Fridman (53:41.100)
that would attempt such an attack, right?
Lex Fridman (53:43.180)
But the two, my question is, is it sort of,
Dawn Song (53:46.620)
when I talked to Elon Musk and asked the same question,
Lex Fridman (53:52.380)
he says, it's not a problem.
Dawn Song (53:53.700)
It's very difficult to do in the real world.
Lex Fridman (53:55.940)
That this won't be a problem.
Dawn Song (53:57.700)
He dismissed it as a problem
Lex Fridman (53:58.900)
for adversarial attacks on the Tesla.
Dawn Song (54:01.180)
Of course, he happens to be involved with the company.
Lex Fridman (54:04.860)
So he has to say that,
Lex Fridman (54:06.180)
but I mean, let me linger in a little longer.
Lex Fridman (54:12.540)
Where does your confidence that it's feasible come from?
Lex Fridman (54:15.540)
And what's your intuition, how people should be worried
Lex Fridman (54:18.660)
and how we might be, how people should defend against it?
Lex Fridman (54:21.740)
How Tesla, how Waymo, how other autonomous vehicle companies
Lex Fridman (54:25.660)
should defend against sensory based attacks,
Dawn Song (54:29.420)
whether on Lidar or on vision or so on.
Lex Fridman (54:32.380)
And also even for Lidar, actually,
Dawn Song (54:33.620)
there has been research shown that even Lidar itself
Lex Fridman (54:36.140)
can be attacked. No, no, no, no, no, no.
Dawn Song (54:38.540)
It's really important to pause.
Lex Fridman (54:40.340)
There's really nice demonstrations that it's possible to do,
Lex Fridman (54:44.820)
but there's so many pieces that it's kind of like,
Lex Fridman (54:49.380)
it's kind of in the lab.
Dawn Song (54:51.740)
Now it's in the physical world,
Lex Fridman (54:53.380)
meaning it's in the physical space, the attacks,
Lex Fridman (54:55.700)
but it's very like, you have to control a lot of things.
Lex Fridman (54:58.780)
To pull it off, it's like the difference
Dawn Song (55:02.100)
between opening a safe when you have it
Lex Fridman (55:05.500)
and you have unlimited time and you can work on it
Dawn Song (55:08.620)
versus like breaking into like the crown,
Lex Fridman (55:12.220)
stealing the crown jewels and whatever, right?
Dawn Song (55:14.340)
I mean, so one way to look at it
Lex Fridman (55:16.900)
in terms of how real these attacks can be,
Dawn Song (55:20.060)
one way to look at it is that actually
Lex Fridman (55:21.740)
you don't even need any sophisticated attacks.
Dawn Song (55:25.300)
Already we've seen many real world examples, incidents
Lex Fridman (55:30.460)
where showing that the vehicle
Dawn Song (55:32.980)
was making the wrong decision.
Lex Fridman (55:34.420)
The wrong decision without attacks, right?
Dawn Song (55:36.180)
Right, right.
Lex Fridman (55:37.020)
So that's one way to demonstrate.
Lex Fridman (55:38.580)
And this is also, like so far we've mainly talked about work
Lex Fridman (55:41.860)
in this adversarial setting, showing that
Dawn Song (55:44.820)
today's learning system,
Lex Fridman (55:46.340)
they are so vulnerable to the adversarial setting,
Lex Fridman (55:48.940)
but at the same time, actually we also know
Lex Fridman (55:51.060)
that even in natural settings,
Dawn Song (55:53.020)
these learning systems, they don't generalize well
Lex Fridman (55:55.580)
and hence they can really misbehave
Dawn Song (55:58.100)
under certain situations like what we have seen.
Lex Fridman (56:02.300)
And hence I think using that as an example,
Dawn Song (56:04.740)
it can show that these issues can be real.
Lex Fridman (56:08.260)
They can be real, but so there's two cases.
Dawn Song (56:10.700)
One is something, it's like perturbations
Lex Fridman (56:14.140)
can make the system misbehave
Dawn Song (56:16.140)
versus make the system do one specific thing
Lex Fridman (56:19.300)
that the attacker wants, as you said, the targeted attack.
Dawn Song (56:23.780)
That seems to be very difficult,
Lex Fridman (56:27.500)
like an extra level of difficult step in the real world.
Lex Fridman (56:31.540)
But from the perspective of the passenger of the car,
Lex Fridman (56:35.660)
I don't think it matters either way,
Dawn Song (56:38.140)
whether it's misbehavior or a targeted attack.
Lex Fridman (56:42.340)
And also, and that's why I was also saying earlier,
Dawn Song (56:45.260)
like one defense is this multi model defense
Lex Fridman (56:48.740)
and more of these consistent checks and so on.
Lex Fridman (56:51.060)
So in the future, I think also it's important
Lex Fridman (56:53.420)
that for these autonomous vehicles,
Dawn Song (56:56.420)
they have lots of different sensors
Lex Fridman (56:58.620)
and they should be combining all these sensory readings
Dawn Song (57:02.620)
to arrive at the decision and the interpretation
Lex Fridman (57:06.860)
of the world and so on.
Lex Fridman (57:08.420)
And the more of these sensory inputs they use
Lex Fridman (57:12.100)
and the better they combine the sensory inputs,
Dawn Song (57:14.500)
the harder it is going to be attacked.
Lex Fridman (57:16.900)
And hence, I think that is a very important direction
Dawn Song (57:19.740)
for us to move towards.
Lex Fridman (57:21.740)
So multi model, multi sensor across multiple cameras,
Lex Fridman (57:25.340)
but also in the case of car, radar, ultrasonic, sound even.
Lex Fridman (57:30.060)
So all of those.
Dawn Song (57:31.380)
Right, right, right, exactly.
Lex Fridman (57:33.380)
So another thing, another part of your work
Dawn Song (57:36.260)
has been in the space of privacy.
Lex Fridman (57:39.180)
And that too can be seen
Dawn Song (57:40.460)
as a kind of security vulnerability.
Lex Fridman (57:43.980)
So thinking of data as a thing that should be protected
Lex Fridman (57:47.900)
and the vulnerabilities to data is vulnerability
Lex Fridman (57:52.140)
is essentially the thing that you wanna protect
Dawn Song (57:55.180)
is the privacy of that data.
Lex Fridman (57:56.940)
So what do you see as the main vulnerabilities
Lex Fridman (57:59.780)
in the privacy of data and how do we protect it?
Lex Fridman (58:02.260)
Right, so in security we actually talk about
Dawn Song (58:05.620)
essentially two, in this case, two different properties.
Lex Fridman (58:10.180)
One is integrity and one is confidentiality.
Lex Fridman (58:13.500)
So what we have been talking earlier
Lex Fridman (58:17.220)
is essentially the integrity of,
Dawn Song (58:20.660)
the integrity property of the learning system.
Lex Fridman (58:22.860)
How to make sure that the learning system
Dawn Song (58:24.820)
is giving the right prediction, for example.
Lex Fridman (58:29.020)
And privacy essentially is on the other side
Dawn Song (58:32.300)
is about confidentiality of the system
Lex Fridman (58:34.900)
is how attackers can,
Dawn Song (58:37.260)
when the attackers compromise
Lex Fridman (58:39.620)
the confidentiality of the system,
Dawn Song (58:42.460)
that's when the attacker steal sensitive information,
Lex Fridman (58:46.220)
right, about individuals and so on.
Dawn Song (58:48.500)
That's really clean, those are great terms.
Lex Fridman (58:51.380)
Integrity and confidentiality.
Dawn Song (58:53.580)
Right.
Lex Fridman (58:54.420)
So how, what are the main vulnerabilities to privacy,
Lex Fridman (58:58.700)
would you say, and how do we protect against it?
Lex Fridman (59:01.660)
Like what are the main spaces and problems
Lex Fridman (59:04.580)
that you think about in the context of privacy?
Lex Fridman (59:07.140)
Right, so especially in the machine learning setting.
Lex Fridman (59:12.620)
So in this case, as we know that how the process goes
Lex Fridman (59:16.980)
is that we have the training data
Lex Fridman (59:19.860)
and then the machine learning system trains
Lex Fridman (59:23.220)
from this training data and then builds a model
Lex Fridman (59:26.020)
and then later on inputs are given to the model
Lex Fridman (59:29.460)
to, at inference time, to try to get prediction and so on.
Lex Fridman (59:34.260)
So then in this case, the privacy concerns that we have
Lex Fridman (59:38.540)
is typically about privacy of the data in the training data
Dawn Song (59:43.340)
because that's essentially the private information.
Lex Fridman (59:45.780)
So, and it's really important
Dawn Song (59:49.980)
because oftentimes the training data
Lex Fridman (59:52.300)
can be very sensitive.
Dawn Song (59:54.140)
It can be your financial data, it's your health data,
Lex Fridman (59:57.180)
or like in IoT case,
Dawn Song (59:59.740)
it's the sensors deployed in real world environment
🔗 相关节目