Nick Bostrom: Simulation and Superintelligence
哲学与宗教生物与进化技术与编程心理与人性AI 与机器学习
📋 章节目录
暂无章节信息
🔑 关键词
simulationintelligenceargumenthumandonexperiencemachineseemssimulationssupersimulatedcivilizationtechnologicalcomputerpossibleinterestingconsciousguessvaluetechnology
💬 精彩语录
"kind of things we might discover when we understand to a greater degree the fundamental, the physics,"
当我们更深入地了解基本原理、物理学时，我们可能会发现一些事情，
— Nick Bostrom （55:09.480）
"So the first one is that almost all civilizations that are current stage of technological development"
所以第一个是几乎所有处于技术发展现阶段的文明
— Nick Bostrom （07:10.760）
"to have say the technology to make these simulations, ancestor simulations or other kinds of simulations."
可以说进行这些模拟、祖先模拟或其他类型的模拟的技术。
— Nick Bostrom （48:05.720）
🎙️ 完整对话（1554 条）
Lex Fridman (00:00.000)
The following is a conversation with Nick Bostrom, a philosopher at University of Oxford
以下是与牛津大学哲学家尼克·博斯特罗姆的对话
Lex Fridman (00:05.520)
and the director of the Future of Humanity Institute.
以及人类未来研究所所长。
Lex Fridman (00:08.840)
He has worked on fascinating and important ideas in existential risk, simulation hypothesis,
他在存在风险、模拟假设、
Lex Fridman (00:15.320)
human enhancement ethics, and the risks of superintelligent AI systems, including in
人类增强伦理，以及超级智能人工智能系统的风险，包括
Lex Fridman (00:20.480)
his book, Superintelligence.
他的书《超级智能》。
Nick Bostrom (00:23.200)
I can see talking to Nick multiple times in this podcast, many hours each time, because
我可以在这个播客中多次看到与尼克的交谈，每次都持续好几个小时，因为
Nick Bostrom (00:27.920)
he has done some incredible work in artificial intelligence, in technology, space, science,
他在人工智能、技术、空间、科学领域做出了一些令人难以置信的工作，
Lex Fridman (00:34.520)
and really philosophy in general, but we have to start somewhere.
确实是一般哲学，但我们必须从某个地方开始。
Nick Bostrom (00:38.920)
This conversation was recorded before the outbreak of the coronavirus pandemic that
这段对话是在冠状病毒大流行爆发之前录制的
Nick Bostrom (00:43.620)
both Nick and I, I'm sure, will have a lot to say about next time we speak, and perhaps
我相信尼克和我下次谈话时都会有很多话要说，也许
Nick Bostrom (00:49.240)
that is for the best, because the deepest lessons can be learned only in retrospect
这是最好的，因为只有在回顾时才能学到最深刻的教训
Nick Bostrom (00:54.440)
when the storm has passed.
当暴风雨过去后。
Nick Bostrom (00:56.680)
I do recommend you read many of his papers on the topic of existential risk, including
我确实建议您阅读他关于存在风险主题的许多论文，包括
Nick Bostrom (01:01.380)
the technical report titled Global Catastrophic Risks Survey that he coauthored with Anders
他与安德斯共同撰写的题为“全球灾难性风险调查”的技术报告
Lex Fridman (01:07.600)
Sandberg.
桑德伯格。
Nick Bostrom (01:08.600)
For everyone feeling the medical, psychological, and financial burden of this crisis, I'm
对于每个感受到这场危机的医疗、心理和经济负担的人，我
Lex Fridman (01:14.000)
sending love your way.
用你的方式传递爱。
Nick Bostrom (01:15.680)
Stay strong.
坚强点。
Lex Fridman (01:16.680)
We're in this together.
我们在一起。
Nick Bostrom (01:17.840)
We'll beat this thing.
我们会打败这件事。
Lex Fridman (01:20.880)
This is the Artificial Intelligence Podcast.
Nick Bostrom (01:22.920)
If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcast, support
Nick Bostrom (01:28.120)
it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F R I D M
Nick Bostrom (01:33.760)
A N.
Nick Bostrom (01:34.760)
As usual, I'll do one or two minutes of ads now and never any ads in the middle that
Nick Bostrom (01:39.120)
can break the flow of the conversation.
Lex Fridman (01:41.080)
I hope that works for you and doesn't hurt the listening experience.
Nick Bostrom (01:46.040)
This show is presented by Cash App, the number one finance app in the App Store.
Lex Fridman (01:50.080)
When you get it, use code LEXPODCAST.
Nick Bostrom (01:53.800)
Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with
Lex Fridman (01:57.920)
as little as one dollar.
Nick Bostrom (01:59.480)
Since Cash App does fractional share trading, let me mention that the order execution algorithm
Nick Bostrom (02:05.200)
that works behind the scenes to create the abstraction of fractional orders is an algorithmic
Nick Bostrom (02:10.360)
marvel.
Lex Fridman (02:11.360)
So big props to the Cash App engineers for solving a hard problem that in the end provides
Nick Bostrom (02:16.200)
an easy interface that takes a step up to the next layer of abstraction over the stock
Nick Bostrom (02:20.360)
market, making trading more accessible for new investors and diversification much easier.
Lex Fridman (02:26.960)
So again, if you get Cash App from the App Store, Google Play, and use the code LEXPODCAST,
Nick Bostrom (02:33.200)
you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping
Nick Bostrom (02:38.560)
to advance robotics and STEM education for young people around the world.
Lex Fridman (02:43.960)
And now, here's my conversation with Nick Bostrom.
Nick Bostrom (02:49.160)
At the risk of asking the Beatles to play yesterday or the Rolling Stones to play Satisfaction,
Lex Fridman (02:54.200)
let me ask you the basics.
Lex Fridman (02:56.520)
What is the simulation hypothesis?
Lex Fridman (02:59.520)
That we are living in a computer simulation.
Lex Fridman (03:02.960)
What is a computer simulation?
Lex Fridman (03:04.440)
How are we supposed to even think about that?
Nick Bostrom (03:06.760)
Well, so the hypothesis is meant to be understood in a literal sense, not that we can kind of
Nick Bostrom (03:15.360)
metaphorically view the universe as an information processing physical system, but that there
Nick Bostrom (03:21.320)
is some advanced civilization who built a lot of computers and that what we experience
Nick Bostrom (03:28.980)
is an effect of what's going on inside one of those computers so that the world around
Nick Bostrom (03:34.960)
us, our own brains, everything we see and perceive and think and feel would exist because
Lex Fridman (03:43.800)
this computer is running certain programs.
Lex Fridman (03:48.280)
So do you think of this computer as something similar to the computers of today, these deterministic
Lex Fridman (03:55.320)
sort of Turing machine type things?
Nick Bostrom (03:58.160)
Is that what we're supposed to imagine or we're supposed to think of something more
Lex Fridman (04:01.480)
like a quantum mechanical system?
Nick Bostrom (04:07.080)
Something much bigger, something much more complicated, something much more mysterious
Lex Fridman (04:11.240)
from our current perspective?
Nick Bostrom (04:12.840)
The ones we have today would do fine, I mean, bigger, certainly.
Lex Fridman (04:15.400)
You'd need more memory and more processing power.
Nick Bostrom (04:18.720)
I don't think anything else would be required.
Nick Bostrom (04:21.280)
Now, it might well be that they do have additional, maybe they have quantum computers and other
Nick Bostrom (04:26.580)
things that would give them even more of, it seems kind of plausible, but I don't think
Nick Bostrom (04:31.320)
it's a necessary assumption in order to get to the conclusion that a technologically
Nick Bostrom (04:38.880)
mature civilization would be able to create these kinds of computer simulations with conscious
Lex Fridman (04:44.520)
beings inside them.
Lex Fridman (04:46.640)
So do you think the simulation hypothesis is an idea that's most useful in philosophy,
Nick Bostrom (04:52.840)
computer science, physics, sort of where do you see it having valuable kind of starting
Lex Fridman (05:02.480)
point in terms of a thought experiment of it?
Lex Fridman (05:05.280)
Is it useful?
Nick Bostrom (05:06.280)
I guess it's more informative and interesting and maybe important, but it's not designed
Lex Fridman (05:14.480)
to be useful for something else.
Nick Bostrom (05:16.560)
Okay, interesting, sure.
Lex Fridman (05:18.480)
But is it philosophically interesting or is there some kind of implications of computer
Lex Fridman (05:23.360)
science and physics?
Lex Fridman (05:24.900)
I think not so much for computer science or physics per se.
Nick Bostrom (05:29.460)
Certainly it would be of interest in philosophy, I think also to say cosmology or physics in
Nick Bostrom (05:37.320)
as much as you're interested in the fundamental building blocks of the world and the rules
Nick Bostrom (05:43.860)
that govern it.
Nick Bostrom (05:46.040)
If we are in a simulation, there is then the possibility that say physics at the level
Nick Bostrom (05:50.480)
where the computer running the simulation could be different from the physics governing
Lex Fridman (05:57.640)
phenomena in the simulation.
Lex Fridman (05:59.780)
So I think it might be interesting from point of view of religion or just for kind of trying
Lex Fridman (06:06.280)
to figure out what the heck is going on.
Lex Fridman (06:09.720)
So we mentioned the simulation hypothesis so far.
Lex Fridman (06:14.680)
There is also the simulation argument, which I tend to make a distinction.
Lex Fridman (06:19.800)
So simulation hypothesis, we are living in a computer simulation.
Nick Bostrom (06:23.080)
Simulation argument, this argument that tries to show that one of three propositions is
Nick Bostrom (06:27.880)
true, one of which is the simulation hypothesis, but there are two alternatives in the original
Lex Fridman (06:34.840)
simulation argument, which we can get to.
Nick Bostrom (06:36.840)
Yeah, let's go there.
Nick Bostrom (06:37.840)
By the way, confusing terms because people will, I think, probably naturally think simulation
Nick Bostrom (06:43.320)
argument equals simulation hypothesis, just terminology wise.
Lex Fridman (06:47.160)
But let's go there.
Lex Fridman (06:48.160)
So simulation hypothesis means that we are living in a simulations, the hypothesis that
Nick Bostrom (06:52.560)
we're living in a simulation, simulation argument has these three complete possibilities that
Nick Bostrom (06:58.840)
cover all possibilities.
Lex Fridman (07:00.480)
So what are they?
Nick Bostrom (07:01.480)
Yeah.
Lex Fridman (07:02.480)
So it's like a disjunction.
Nick Bostrom (07:03.480)
It says at least one of these three is true, although it doesn't on its own tell us which
Lex Fridman (07:08.440)
one.
Lex Fridman (07:10.760)
So the first one is that almost all civilizations that are current stage of technological development
Lex Fridman (07:17.560)
go extinct before they reach technological maturity.
Lex Fridman (07:23.820)
So there is some great filter that makes it so that basically none of the civilizations
Nick Bostrom (07:34.520)
throughout maybe a vast cosmos will ever get to realize the full potential of technological
Nick Bostrom (07:41.800)
development.
Lex Fridman (07:42.800)
And this could be, theoretically speaking, this could be because most civilizations kill
Nick Bostrom (07:47.660)
themselves too eagerly or destroy themselves too eagerly, or it might be super difficult
Lex Fridman (07:52.840)
to build a simulation.
Lex Fridman (07:55.200)
So the span of time.
Lex Fridman (07:57.400)
Theoretically it could be both.
Nick Bostrom (07:58.400)
Now I think it looks like we would technologically be able to get there in a time span that
Nick Bostrom (08:04.500)
is short compared to, say, the lifetime of planets and other sort of astronomical processes.
Lex Fridman (08:13.080)
So your intuition is to build a simulation is not...
Lex Fridman (08:16.520)
Well, so this is interesting concept of technological maturity.
Nick Bostrom (08:21.240)
It's kind of an interesting concept to have other purposes as well.
Nick Bostrom (08:25.160)
We can see even based on our current limited understanding what some lower bound would
Nick Bostrom (08:31.120)
be on the capabilities that you could realize by just developing technologies that we already
Lex Fridman (08:37.480)
see are possible.
Lex Fridman (08:38.600)
So for example, one of my research fellows here, Eric Drexler, back in the 80s, studied
Lex Fridman (08:46.680)
molecular manufacturing.
Nick Bostrom (08:48.560)
That is you could analyze using theoretical tools and computer modeling the performance
Nick Bostrom (08:55.520)
of various molecularly precise structures that we didn't then and still don't today
Nick Bostrom (09:01.160)
have the ability to actually fabricate.
Lex Fridman (09:04.100)
But you could say that, well, if we could put these atoms together in this way, then
Nick Bostrom (09:07.460)
the system would be stable and it would rotate at this speed and have all these computational
Lex Fridman (09:13.760)
characteristics.
Lex Fridman (09:16.260)
And he also outlined some pathways that would enable us to get to this kind of molecularly
Lex Fridman (09:22.740)
manufacturing in the fullness of time.
Lex Fridman (09:25.000)
And you could do other studies we've done.
Nick Bostrom (09:28.280)
You could look at the speed at which, say, it would be possible to colonize the galaxy
Nick Bostrom (09:33.440)
if you had mature technology.
Lex Fridman (09:36.120)
We have an upper limit, which is the speed of light.
Nick Bostrom (09:38.360)
We have sort of a lower current limit, which is how fast current rockets go.
Nick Bostrom (09:42.360)
We know we can go faster than that by just making them bigger and have more fuel and
Nick Bostrom (09:47.560)
stuff.
Nick Bostrom (09:48.560)
We can then start to describe the technological affordances that would exist once a civilization
Nick Bostrom (09:56.120)
has had enough time to develop, at least those technologies we already know are possible.
Nick Bostrom (10:01.000)
Then maybe they would discover other new physical phenomena as well that we haven't realized
Nick Bostrom (10:05.800)
that would enable them to do even more.
Lex Fridman (10:08.120)
But at least there is this kind of basic set of capabilities.
Lex Fridman (10:11.480)
Can you just link on that, how do we jump from molecular manufacturing to deep space
Lex Fridman (10:18.560)
exploration to mature technology?
Lex Fridman (10:23.200)
What's the connection there?
Nick Bostrom (10:24.200)
Well, so these would be two examples of technological capability sets that we can have a high degree
Nick Bostrom (10:31.340)
of confidence are physically possible in our universe and that a civilization that was
Nick Bostrom (10:38.000)
allowed to continue to develop its science and technology would eventually attain.
Nick Bostrom (10:42.720)
You can intuit like, we can kind of see the set of breakthroughs that are likely to happen.
Lex Fridman (10:48.960)
So you can see like, what did you call it, the technological set?
Nick Bostrom (10:53.240)
With computers, maybe it's easiest.
Nick Bostrom (10:58.080)
One is we could just imagine bigger computers using exactly the same parts that we have.
Lex Fridman (11:01.800)
So you can kind of scale things that way, right?
Lex Fridman (11:04.480)
But you could also make processors a bit faster.
Nick Bostrom (11:07.760)
If you had this molecular nanotechnology that Eric Drexler described, he characterized a
Nick Bostrom (11:13.080)
kind of crude computer built with these parts that would perform at a million times the
Nick Bostrom (11:19.760)
human brain while being significantly smaller, the size of a sugar cube.
Lex Fridman (11:25.600)
And he made no claim that that's the optimum computing structure, like for all you know,
Nick Bostrom (11:30.160)
we could build faster computers that would be more efficient, but at least you could
Lex Fridman (11:33.660)
do that if you had the ability to do things that were atomically precise.
Nick Bostrom (11:37.200)
I mean, so you can then combine these two.
Nick Bostrom (11:39.600)
You could have this kind of nanomolecular ability to build things atom by atom and then
Nick Bostrom (11:45.600)
say at this as a spatial scale that would be attainable through space colonizing technology.
Nick Bostrom (11:53.040)
You could then start, for example, to characterize a lower bound on the amount of computing power
Nick Bostrom (11:58.200)
that a technologically mature civilization would have.
Nick Bostrom (12:01.840)
If it could grab resources, you know, planets and so forth, and then use this molecular
Nick Bostrom (12:07.680)
nanotechnology to optimize them for computing, you'd get a very, very high lower bound on
Lex Fridman (12:15.320)
the amount of compute.
Lex Fridman (12:17.000)
So sorry, just to define some terms, so technologically mature civilization is one that took that
Lex Fridman (12:22.120)
piece of technology to its lower bound.
Lex Fridman (12:26.220)
What is a technologically mature civilization?
Lex Fridman (12:27.920)
So that means it's a stronger concept than we really need for the simulation hypothesis.
Nick Bostrom (12:31.160)
I just think it's interesting in its own right.
Lex Fridman (12:34.040)
So it would be the idea that there is some stage of technological development where you've
Nick Bostrom (12:38.880)
basically maxed out, that you developed all those general purpose, widely useful technologies
Nick Bostrom (12:45.320)
that could be developed, or at least kind of come very close to the, you know, 99.9%
Nick Bostrom (12:51.600)
there or something.
Lex Fridman (12:53.440)
So that's an independent question.
Nick Bostrom (12:55.160)
You can think either that there is such a ceiling, or you might think it just goes,
Lex Fridman (12:59.480)
the technology tree just goes on forever.
Lex Fridman (13:03.200)
Where does your sense fall?
Nick Bostrom (13:04.600)
I would guess that there is a maximum that you would start to asymptote towards.
Lex Fridman (13:10.000)
So new things won't keep springing up, new ceilings.
Nick Bostrom (13:13.900)
In terms of basic technological capabilities, I think that, yeah, there is like a finite
Nick Bostrom (13:18.980)
set of laws that can exist in this universe.
Nick Bostrom (13:23.840)
Moreover, I mean, I wouldn't be that surprised if we actually reached close to that level
Nick Bostrom (13:30.120)
fairly shortly after we have, say, machine superintelligence.
Lex Fridman (13:33.980)
So I don't think it would take millions of years for a human originating civilization
Nick Bostrom (13:39.760)
to begin to do this.
Lex Fridman (13:42.760)
It's more likely to happen on historical timescales.
Lex Fridman (13:46.680)
But that's an independent speculation from the simulation argument.
Nick Bostrom (13:51.320)
I mean, for the purpose of the simulation argument, it doesn't really matter whether
Nick Bostrom (13:55.080)
it goes indefinitely far up or whether there is a ceiling, as long as we know we can at
Lex Fridman (13:59.400)
least get to a certain level.
Lex Fridman (14:01.880)
And it also doesn't matter whether that's going to happen in 100 years or 5,000 years
Lex Fridman (14:06.720)
or 50 million years.
Nick Bostrom (14:08.440)
Like the timescales really don't make any difference for this.
Lex Fridman (14:11.600)
Can you look on that a little bit?
Nick Bostrom (14:13.360)
Like there's a big difference between 100 years and 10 million years.
Lex Fridman (14:19.080)
So it doesn't really not matter because you just said it doesn't matter if we jump scales
Nick Bostrom (14:25.500)
to beyond historical scales.
Lex Fridman (14:28.680)
So we described that.
Lex Fridman (14:30.200)
So for the simulation argument, sort of doesn't it matter that we if it takes 10 million years,
Lex Fridman (14:40.840)
it gives us a lot more opportunity to destroy civilization in the meantime?
Nick Bostrom (14:44.720)
Yeah, well, so it would shift around the probabilities between these three alternatives.
Nick Bostrom (14:49.840)
That is, if we are very, very far away from being able to create these simulations, if
Nick Bostrom (14:54.840)
it's like, say, billions of years into the future, then it's more likely that we will
Lex Fridman (14:58.800)
fail ever to get there.
Nick Bostrom (14:59.800)
There's more time for us to kind of go extinct along the way.
Lex Fridman (15:04.080)
And so this is similarly for other civilizations.
Lex Fridman (15:06.360)
So it is important to think about how hard it is to build a simulation.
Lex Fridman (15:11.000)
In terms of figuring out which of the disjuncts.
Lex Fridman (15:14.580)
But for the simulation argument itself, which is agnostic as to which of these three alternatives
Lex Fridman (15:19.360)
is true.
Nick Bostrom (15:20.360)
Yeah.
Lex Fridman (15:21.360)
Okay.
Nick Bostrom (15:22.360)
It's like you don't have to like the simulation argument would be true whether or not we thought
Lex Fridman (15:26.720)
this could be done in 500 years or it would take 500 million years.
Nick Bostrom (15:29.840)
No, for sure.
Lex Fridman (15:30.840)
The simulation argument stands.
Nick Bostrom (15:31.840)
I mean, I'm sure there might be some people who oppose it, but it doesn't matter.
Lex Fridman (15:36.200)
I mean, it's very nice those three cases cover it.
Lex Fridman (15:39.480)
But the fun part is at least not saying what the probabilities are, but kind of thinking
Nick Bostrom (15:44.240)
about kind of intuiting reasoning about what's more likely, what are the kind of things that
Nick Bostrom (15:50.480)
would make some of the arguments less and more so like.
Lex Fridman (15:54.400)
But let's actually, I don't think we went through them.
Lex Fridman (15:56.520)
So number one is we destroy ourselves before we ever create simulation.
Lex Fridman (16:00.440)
Right.
Lex Fridman (16:01.440)
So that's kind of sad, but we have to think not just what might destroy us.
Nick Bostrom (16:07.960)
I mean, so there could be some whatever disaster, some meteor slamming the earth a few years
Nick Bostrom (16:14.180)
from now that could destroy us.
Lex Fridman (16:16.120)
Right.
Lex Fridman (16:17.120)
But you'd have to postulate in order for this first disjunct to be true that almost all
Nick Bostrom (16:24.880)
civilizations throughout the cosmos also failed to reach technological maturity.
Lex Fridman (16:32.200)
And the underlying assumption there is that there is likely a very large number of other
Lex Fridman (16:37.680)
intelligent civilizations.
Nick Bostrom (16:39.520)
Well, if there are, yeah, then they would virtually all have to succumb in the same
Lex Fridman (16:45.280)
way.
Nick Bostrom (16:46.280)
I mean, then that leads off another, I guess there are a lot of little digressions that
Lex Fridman (16:50.120)
are interesting.
Nick Bostrom (16:51.120)
Definitely, let's go there.
Lex Fridman (16:52.120)
Let's go there.
Nick Bostrom (16:53.120)
Keep dragging us back.
Nick Bostrom (16:54.120)
Well, there are these, there is a set of basic questions that always come up in conversations
Nick Bostrom (16:58.940)
with interesting people, like the Fermi paradox, like there's like, you could almost define
Nick Bostrom (17:05.500)
whether a person is interesting, whether at some point the question of the Fermi paradox
Nick Bostrom (17:09.920)
comes up, like, well, so for what it's worth, it looks to me that the universe is very big.
Nick Bostrom (17:16.920)
I mean, in fact, according to the most popular current cosmological theories, infinitely
Nick Bostrom (17:23.160)
big.
Lex Fridman (17:25.320)
And so then it would follow pretty trivially that it would contain a lot of other civilizations,
Nick Bostrom (17:31.420)
in fact, infinitely many.
Nick Bostrom (17:34.120)
If you have some local stochasticity and infinitely many, it's like, you know, infinitely many
Nick Bostrom (17:39.660)
lumps of matter, one next to another, there's kind of random stuff in each one, then you're
Lex Fridman (17:43.320)
going to get all possible outcomes with probability one infinitely repeated.
Lex Fridman (17:51.160)
So then certainly there would be a lot of extraterrestrials out there.
Nick Bostrom (17:54.640)
Even short of that, if the universe is very big, that might be a finite but large number.
Nick Bostrom (18:02.660)
If we were literally the only one, yeah, then of course, if we went extinct, then all of
Nick Bostrom (18:09.920)
civilizations at our current stage would have gone extinct before becoming technological
Nick Bostrom (18:14.400)
material.
Lex Fridman (18:15.400)
So then it kind of becomes trivially true that a very high fraction of those went extinct.
Lex Fridman (18:22.560)
But if we think there are many, I mean, it's interesting, because there are certain things
Nick Bostrom (18:25.480)
that possibly could kill us, like if you look at existential risks, and it might be a different,
Nick Bostrom (18:35.840)
like the best answer to what would be most likely to kill us might be a different answer
Nick Bostrom (18:40.280)
than the best answer to the question, if there is something that kills almost everyone, what
Lex Fridman (18:46.920)
would that be?
Nick Bostrom (18:47.920)
Because that would have to be some risk factor that was kind of uniform overall possible
Nick Bostrom (18:53.400)
civilization.
Lex Fridman (18:54.400)
So in this, for the sake of this argument, you have to think about not just us, but like
Nick Bostrom (18:59.760)
every civilization dies out before they create the simulation or something very close to
Lex Fridman (19:05.160)
everybody.
Nick Bostrom (19:06.160)
Okay.
Lex Fridman (19:07.160)
So what's number two in the number two is the convergence hypothesis that is that maybe
Nick Bostrom (19:14.080)
like a lot of some of these civilizations do make it through to technological maturity,
Lex Fridman (19:18.760)
but out of those who do get there, they all lose interest in creating these simulations.
Lex Fridman (19:26.400)
So they just have the capability of doing it, but they choose not to.
Nick Bostrom (19:32.960)
Not just a few of them decide not to, but out of a million, maybe not even a single
Nick Bostrom (19:40.260)
one of them would do it.
Lex Fridman (19:41.760)
And I think when you say lose interest, that sounds like unlikely because it's like they
Nick Bostrom (19:48.880)
get bored or whatever, but it could be so many possibilities within that.
Nick Bostrom (19:53.680)
I mean, losing interest could be, it could be anything from it being exceptionally difficult
Nick Bostrom (1:00:04.240)
reasons to do something or not to do something, then we just don't realize they are there
Lex Fridman (1:00:08.160)
because we are so dumb, bumbling through the universe.
Lex Fridman (1:00:11.160)
But if almost inevitably en route to attaining the ability to create many ancestors simulations,
Nick Bostrom (1:00:17.680)
you do have this cognitive enhancement, or advice from super intelligences or yourself,
Nick Bostrom (1:00:23.240)
then maybe there's like this additional set of considerations coming into view and it's
Nick Bostrom (1:00:27.120)
obvious that the thing that makes sense is to do X, whereas right now it seems you could
Nick Bostrom (1:00:32.000)
X, Y or Z and different people will do different things and we are kind of random in that sense.
Nick Bostrom (1:00:39.720)
Because at this time, with our limited technology, the impact of our decisions is minor.
Nick Bostrom (1:00:45.160)
I mean, that's starting to change in some ways.
Lex Fridman (1:00:48.440)
But…
Nick Bostrom (1:00:49.440)
Well, I'm not sure how it follows that the impact of our decisions is minor.
Lex Fridman (1:00:53.840)
Well, it's starting to change.
Nick Bostrom (1:00:55.600)
I mean, I suppose 100 years ago it was minor.
Lex Fridman (1:00:58.760)
It's starting to…
Nick Bostrom (1:01:00.240)
Well, it depends on how you view it.
Lex Fridman (1:01:03.520)
What people did 100 years ago still have effects on the world today.
Nick Bostrom (1:01:08.640)
Oh, I see.
Lex Fridman (1:01:11.920)
As a civilization in the togetherness.
Nick Bostrom (1:01:14.600)
Yeah.
Lex Fridman (1:01:15.600)
So it might be that the greatest impact of individuals is not at technological maturity
Nick Bostrom (1:01:21.760)
or very far down.
Nick Bostrom (1:01:22.760)
It might be earlier on when there are different tracks, civilization could go down.
Nick Bostrom (1:01:28.440)
Maybe the population is smaller, things still haven't settled out.
Nick Bostrom (1:01:33.280)
If you count indirect effects, those could be bigger than the direct effects that people
Nick Bostrom (1:01:41.720)
have later on.
Lex Fridman (1:01:43.360)
So part three of the argument says that…
Lex Fridman (1:01:46.280)
So that leads us to a place where eventually somebody creates a simulation.
Lex Fridman (1:01:53.520)
I think you had a conversation with Joe Rogan.
Nick Bostrom (1:01:55.520)
I think there's some aspect here where you got stuck a little bit.
Lex Fridman (1:02:01.200)
How does that lead to we're likely living in a simulation?
Lex Fridman (1:02:06.400)
So this kind of probability argument, if somebody eventually creates a simulation, why does
Lex Fridman (1:02:12.920)
that mean that we're now in a simulation?
Lex Fridman (1:02:15.980)
What you get to if you accept alternative three first is there would be more simulated
Lex Fridman (1:02:22.780)
people with our kinds of experiences than non simulated ones.
Nick Bostrom (1:02:26.440)
Like if you look at the world as a whole, by the end of time as it were, you just count
Lex Fridman (1:02:34.120)
it up.
Nick Bostrom (1:02:36.240)
That would be more simulated ones than non simulated ones.
Lex Fridman (1:02:39.600)
Then there is an extra step to get from that.
Nick Bostrom (1:02:43.160)
If you assume that, suppose for the sake of the argument, that that's true.
Lex Fridman (1:02:48.320)
How do you get from that to the statement we are probably in a simulation?
Lex Fridman (1:02:57.840)
So here you're introducing an indexical statement like it's that this person right now is in
Lex Fridman (1:03:05.920)
a simulation.
Nick Bostrom (1:03:06.920)
There are all these other people that are in simulations and some that are not in the
Lex Fridman (1:03:10.880)
simulation.
Lex Fridman (1:03:13.520)
But what probability should you have that you yourself is one of the simulated ones
Lex Fridman (1:03:19.200)
in that setup?
Lex Fridman (1:03:21.040)
So I call it the bland principle of indifference, which is that in cases like this, when you
Nick Bostrom (1:03:28.440)
have two sets of observers, one of which is much larger than the other and you can't from
Nick Bostrom (1:03:37.320)
any internal evidence you have, tell which set you belong to, you should assign a probability
Lex Fridman (1:03:46.320)
that's proportional to the size of these sets.
Lex Fridman (1:03:50.240)
So that if there are 10 times more simulated people with your kinds of experiences, you
Lex Fridman (1:03:55.240)
would be 10 times more likely to be one of those.
Lex Fridman (1:03:58.520)
Is that as intuitive as it sounds?
Nick Bostrom (1:04:00.600)
I mean, that seems kind of, if you don't have enough information, you should rationally
Nick Bostrom (1:04:06.000)
just assign the same probability as the size of the set.
Lex Fridman (1:04:10.880)
It seems pretty plausible to me.
Lex Fridman (1:04:15.800)
Where are the holes in this?
Nick Bostrom (1:04:17.040)
Is it at the very beginning, the assumption that everything stretches, you have infinite
Lex Fridman (1:04:23.800)
time essentially?
Lex Fridman (1:04:24.800)
You don't need infinite time.
Lex Fridman (1:04:26.920)
You just need, how long does the time take?
Nick Bostrom (1:04:29.840)
However long it takes, I guess, for a universe to produce an intelligent civilization that
Nick Bostrom (1:04:36.040)
attains the technology to run some ancestry simulations.
Nick Bostrom (1:04:40.520)
When the first simulation is created, that stretch of time, just a little longer than
Nick Bostrom (1:04:45.840)
they'll all start creating simulations.
Lex Fridman (1:04:48.200)
Well, I mean, there might be a difference.
Nick Bostrom (1:04:52.200)
If you think of there being a lot of different planets and some subset of them have life
Lex Fridman (1:04:57.720)
and then some subset of those get to intelligent life and some of those maybe eventually start
Nick Bostrom (1:05:03.280)
creating simulations, they might get started at quite different times.
Nick Bostrom (1:05:07.760)
Maybe on some planet, it takes a billion years longer before you get monkeys or before you
Nick Bostrom (1:05:13.960)
get even bacteria than on another planet.
Lex Fridman (1:05:19.720)
This might happen at different cosmological epochs.
Lex Fridman (1:05:25.000)
Is there a connection here to the doomsday argument and that sampling there?
Nick Bostrom (1:05:28.920)
Yeah, there is a connection in that they both involve an application of anthropic reasoning
Nick Bostrom (1:05:36.880)
that is reasoning about these kind of indexical propositions.
Lex Fridman (1:05:41.120)
But the assumption you need in the case of the simulation argument is much weaker than
Nick Bostrom (1:05:49.360)
the assumption you need to make the doomsday argument go through.
Lex Fridman (1:05:53.760)
What is the doomsday argument and maybe you can speak to the anthropic reasoning in more
Nick Bostrom (1:05:58.640)
general.
Nick Bostrom (1:05:59.640)
Yeah, that's a big and interesting topic in its own right, anthropics, but the doomsday
Nick Bostrom (1:06:03.680)
argument is this really first discovered by Brandon Carter, who was a theoretical physicist
Lex Fridman (1:06:11.120)
and then developed by philosopher John Leslie.
Nick Bostrom (1:06:15.920)
I think it might have been discovered initially in the 70s or 80s and Leslie wrote this book,
Lex Fridman (1:06:21.080)
I think in 96.
Lex Fridman (1:06:23.280)
And there are some other versions as well by Richard Gott, who's a physicist, but let's
Nick Bostrom (1:06:27.600)
focus on the Carter Leslie version where it's an argument that we have systematically underestimated
Nick Bostrom (1:06:38.420)
the probability that humanity will go extinct soon.
Nick Bostrom (1:06:44.160)
Now I should say most people probably think at the end of the day there is something wrong
Nick Bostrom (1:06:49.040)
with this doomsday argument that it doesn't really hold.
Nick Bostrom (1:06:52.260)
It's like there's something wrong with it, but it's proved hard to say exactly what is
Nick Bostrom (1:06:56.280)
wrong with it and different people have different accounts.
Lex Fridman (1:07:00.600)
My own view is it seems inconclusive, but I can say what the argument is.
Nick Bostrom (1:07:06.480)
Yeah, that would be good.
Lex Fridman (1:07:08.080)
So maybe it's easiest to explain via an analogy to sampling from urns.
Lex Fridman (1:07:17.720)
So imagine you have two urns in front of you and they have balls in them that have numbers.
Lex Fridman (1:07:27.000)
The two urns look the same, but inside one there are 10 balls.
Nick Bostrom (1:07:30.020)
Ball number one, two, three, up to ball number 10.
Lex Fridman (1:07:33.520)
And then in the other urn you have a million balls numbered one to a million and somebody
Nick Bostrom (1:07:41.720)
puts one of these urns in front of you and asks you to guess what's the chance it's the
Lex Fridman (1:07:48.280)
10 ball urn and you say, well, 50, 50, I can't tell which urn it is.
Lex Fridman (1:07:53.280)
But then you're allowed to reach in and pick a ball at random from the urn and that's suppose
Lex Fridman (1:07:58.360)
you find that it's ball number seven.
Lex Fridman (1:08:02.200)
So that's strong evidence for the 10 ball hypothesis.
Nick Bostrom (1:08:05.680)
It's a lot more likely that you would get such a low numbered ball if there are only
Lex Fridman (1:08:11.240)
10 balls in the urn, like it's in fact 10% done, right?
Nick Bostrom (1:08:14.960)
Then if there are a million balls, it would be very unlikely you would get number seven.
Lex Fridman (1:08:19.680)
So you perform a Bayesian update and if your prior was 50, 50 that it was the 10 ball urn,
Nick Bostrom (1:08:27.200)
you become virtually certain after finding the random sample was seven that it's only
Nick Bostrom (1:08:31.640)
has 10 balls in it.
Lex Fridman (1:08:33.320)
So in the case of the urns, this is uncontroversial, just elementary probability theory.
Nick Bostrom (1:08:37.480)
The Doomsday Argument says that you should reason in a similar way with respect to different
Nick Bostrom (1:08:43.240)
hypotheses about how many balls there will be in the urn of humanity as it were, how
Nick Bostrom (1:08:49.680)
many humans there will ever have been by the time we go extinct.
Lex Fridman (1:08:54.360)
So to simplify, let's suppose we only consider two hypotheses, either maybe 200 billion humans
Nick Bostrom (1:09:00.400)
in total or 200 trillion humans in total.
Lex Fridman (1:09:05.800)
You could fill in more hypotheses, but it doesn't change the principle here.
Lex Fridman (1:09:09.360)
So it's easiest to see if we just consider these two.
Lex Fridman (1:09:12.200)
So you start with some prior based on ordinary empirical ideas about threats to civilization
Lex Fridman (1:09:18.080)
and so forth.
Lex Fridman (1:09:19.080)
And maybe you say it's a 5% chance that we will go extinct by the time there will have
Nick Bostrom (1:09:23.640)
been 200 billion only, you're kind of optimistic, let's say, you think probably we'll make it
Lex Fridman (1:09:28.560)
through, colonize the universe.
Lex Fridman (1:09:31.840)
But then, according to this Doomsday Argument, you should take off your own birth rank as
Lex Fridman (1:09:39.240)
a random sample.
Lex Fridman (1:09:40.240)
So your birth rank is your sequence in the position of all humans that have ever existed.
Nick Bostrom (1:09:47.720)
It turns out you're about a human number of 100 billion, you know, give or take.
Nick Bostrom (1:09:52.680)
That's like, roughly how many people have been born before you.
Lex Fridman (1:09:55.440)
That's fascinating, because I probably, we each have a number.
Nick Bostrom (1:09:59.840)
We would each have a number in this, I mean, obviously, the exact number would depend on
Nick Bostrom (1:10:04.120)
where you started counting, like which ancestors was human enough to count as human.
Lex Fridman (1:10:09.080)
But those are not really important, there are relatively few of them.
Lex Fridman (1:10:13.120)
So yeah, so you're roughly 100 billion.
Nick Bostrom (1:10:16.200)
Now, if they're only going to be 200 billion in total, that's a perfectly unremarkable
Lex Fridman (1:10:20.960)
number.
Lex Fridman (1:10:21.960)
You're somewhere in the middle, right?
Lex Fridman (1:10:22.960)
It's a run of the mill human, completely unsurprising.
Nick Bostrom (1:10:26.400)
Now, if they're going to be 200 trillion, you would be remarkably early, like what are
Lex Fridman (1:10:32.960)
the chances out of these 200 trillion human that you should be human number 100 billion?
Nick Bostrom (1:10:40.000)
That seems it would have a much lower conditional probability.
Lex Fridman (1:10:45.040)
And so analogously to how in the urn case, you thought after finding this low numbered
Nick Bostrom (1:10:50.820)
random sample, you update it in favor of the urn having few balls.
Nick Bostrom (1:10:54.880)
Similarly, in this case, you should update in favor of the human species having a lower
Nick Bostrom (1:11:00.240)
total number of members that is doomed soon.
Lex Fridman (1:11:04.480)
You said doomed soon?
Nick Bostrom (1:11:05.960)
Well, that would be the hypothesis in this case that it will end 100 billion.
Lex Fridman (1:11:11.640)
I just like that term for that hypothesis.
Lex Fridman (1:11:14.600)
So what it kind of crucially relies on, the Doomsday Argument, is the idea that you should
Nick Bostrom (1:11:20.160)
reason as if you were a random sample from the set of all humans that will have existed.
Nick Bostrom (1:11:27.480)
If you have that assumption, then I think the rest kind of follows.
Lex Fridman (1:11:31.000)
The question then is, why should you make that assumption?
Lex Fridman (1:11:34.280)
In fact, you know you're 100 billion, so where do you get this prior?
Lex Fridman (1:11:38.880)
And then there is like a literature on that with different ways of supporting that assumption.
Lex Fridman (1:11:45.280)
That's just one example of anthropic reasoning, right?
Nick Bostrom (1:11:48.200)
That seems to be kind of convenient when you think about humanity, when you think about
Nick Bostrom (1:11:53.800)
sort of even like existential threats and so on, as it seems that quite naturally that
Lex Fridman (1:12:00.320)
you should assume that you're just an average case.
Nick Bostrom (1:12:03.680)
Yeah, that you're kind of a typical randomly sample.
Nick Bostrom (1:12:07.840)
Now, in the case of the Doomsday Argument, it seems to lead to what intuitively we think
Nick Bostrom (1:12:12.240)
is the wrong conclusion, or at least many people have this reaction that there's got
Lex Fridman (1:12:16.480)
to be something fishy about this argument.
Nick Bostrom (1:12:19.320)
Because from very, very weak premises, it gets this very striking implication that we
Lex Fridman (1:12:25.840)
have almost no chance of reaching size 200 trillion humans in the future.
Lex Fridman (1:12:30.980)
And how could we possibly get there just by reflecting on when we were born?
Nick Bostrom (1:12:35.480)
It seems you would need sophisticated arguments about the impossibility of space colonization,
Nick Bostrom (1:12:39.720)
blah, blah.
Lex Fridman (1:12:40.720)
So one might be tempted to reject this key assumption, I call it the self sampling assumption,
Nick Bostrom (1:12:45.560)
the idea that you should reason as if you're a random sample from all observers or in your
Lex Fridman (1:12:50.280)
some reference class.
Nick Bostrom (1:12:52.600)
However, it turns out that in other domains, it looks like we need something like this
Lex Fridman (1:12:58.920)
self sampling assumption to make sense of bona fide scientific inferences.
Nick Bostrom (1:13:04.840)
In contemporary cosmology, for example, you have these multiverse theories.
Lex Fridman (1:13:09.320)
And according to a lot of those, all possible human observations are made.
Lex Fridman (1:13:14.880)
So if you have a sufficiently large universe, you will have a lot of people observing all
Lex Fridman (1:13:18.880)
kinds of different things.
Lex Fridman (1:13:22.040)
So if you have two competing theories, say about the value of some constant, it could
Nick Bostrom (1:13:29.880)
be true according to both of these theories that there will be some observers observing
Nick Bostrom (1:13:34.980)
the value that corresponds to the other theory, because there will be some observers that
Nick Bostrom (1:13:42.160)
have hallucinations, so there's a local fluctuation or a statistically anomalous measurement,
Nick Bostrom (1:13:47.600)
these things will happen.
Lex Fridman (1:13:49.460)
And if enough observers make enough different observations, there will be some that sort
Nick Bostrom (1:13:53.320)
of by chance make these different ones.
Lex Fridman (1:13:55.960)
And so what we would want to say is, well, many more observers, a larger proportion of
Nick Bostrom (1:14:04.280)
the observers will observe as it were the true value.
Lex Fridman (1:14:08.560)
And a few will observe the wrong value.
Nick Bostrom (1:14:10.880)
If we think of ourselves as a random sample, we should expect with a probability to observe
Nick Bostrom (1:14:15.400)
the true value and that will then allow us to conclude that the evidence we actually
Nick Bostrom (1:14:20.180)
have is evidence for the theories we think are supported.
Nick Bostrom (1:14:24.640)
It kind of then is a way of making sense of these inferences that clearly seem correct,
Nick Bostrom (1:14:32.200)
that we can make various observations and infer what the temperature of the cosmic background
Lex Fridman (1:14:38.480)
is and the fine structure constant and all of this.
Lex Fridman (1:14:44.080)
But it seems that without rolling in some assumption similar to the self sampling assumption,
Lex Fridman (1:14:49.980)
this inference just doesn't go through.
Lex Fridman (1:14:51.840)
And there are other examples.
Lex Fridman (1:14:53.180)
So there are these scientific contexts where it looks like this kind of anthropic reasoning
Nick Bostrom (1:14:56.620)
is needed and makes perfect sense.
Lex Fridman (1:14:59.120)
And yet, in the case of the Dupest argument, it has this weird consequence and people might
Nick Bostrom (1:15:02.960)
think there's something wrong with it there.
Lex Fridman (1:15:05.760)
So there's then this project that would consist in trying to figure out what are the legitimate
Nick Bostrom (1:15:14.200)
ways of reasoning about these indexical facts when observer selection effects are in play.
Lex Fridman (1:15:20.400)
In other words, developing a theory of anthropics.
Lex Fridman (1:15:23.520)
And there are different views of looking at that and it's a difficult methodological area.
Lex Fridman (1:15:29.280)
But to tie it back to the simulation argument, the key assumption there, this bland principle
Nick Bostrom (1:15:37.920)
of indifference, is much weaker than the self sampling assumption.
Lex Fridman (1:15:43.460)
So if you think about, in the case of the Dupest argument, it says you should reason
Nick Bostrom (1:15:48.220)
as if you are a random sample from all humans that will have lived, even though in fact
Nick Bostrom (1:15:52.100)
you know that you are about number 100 billionth human and you're alive in the year 2020.
Nick Bostrom (1:15:59.840)
Whereas in the case of the simulation argument, it says that, well, if you actually have no
Nick Bostrom (1:16:04.240)
way of telling which one you are, then you should assign this kind of uniform probability.
Nick Bostrom (1:16:10.240)
Yeah, yeah, your role as the observer in the simulation argument is different, it seems
Lex Fridman (1:16:14.640)
like.
Lex Fridman (1:16:15.640)
Like who's the observer?
Lex Fridman (1:16:16.640)
I mean, I keep assigning the individual consciousness.
Lex Fridman (1:16:19.640)
But a lot of observers in the context of the simulation argument, the relevant observers
Lex Fridman (1:16:26.480)
would be A, the people in original histories, and B, the people in simulations.
Lex Fridman (1:16:33.440)
So this would be the class of observers that we need, I mean, they're also maybe the simulators,
Lex Fridman (1:16:37.520)
but we can set those aside for this.
Lex Fridman (1:16:40.280)
So the question is, given that class of observers, a small set of original history observers
Lex Fridman (1:16:46.160)
and a large class of simulated observers, which one should you think is you?
Lex Fridman (1:16:51.480)
Where are you amongst this set of observers?
Nick Bostrom (1:16:54.200)
I'm maybe having a little bit of trouble wrapping my head around the intricacies of what it
Nick Bostrom (1:17:00.020)
means to be an observer in this, in the different instantiations of the anthropic reasoning
Lex Fridman (1:17:08.020)
cases that we mentioned.
Nick Bostrom (1:17:09.020)
Yeah.
Lex Fridman (1:17:10.020)
I mean, does it have to be...
Nick Bostrom (1:17:11.020)
It's not the observer.
Nick Bostrom (1:17:12.020)
Yeah, I mean, it may be an easier way of putting it is just like, are you simulated, are you
Lex Fridman (1:17:16.560)
not simulated, given this assumption that these two groups of people exist?
Lex Fridman (1:17:21.120)
Yeah.
Nick Bostrom (1:17:22.120)
In the simulation case, it seems pretty straightforward.
Lex Fridman (1:17:24.040)
Yeah.
Lex Fridman (1:17:25.040)
So the key point is the methodological assumption you need to make to get the simulation argument
Nick Bostrom (1:17:32.560)
to where it wants to go is much weaker and less problematic than the methodological assumption
Nick Bostrom (1:17:38.940)
you need to make to get the doomsday argument to its conclusion.
Nick Bostrom (1:17:43.000)
Maybe the doomsday argument is sound or unsound, but you need to make a much stronger and more
Nick Bostrom (1:17:48.240)
controversial assumption to make it go through.
Nick Bostrom (1:17:52.140)
In the case of the simulation argument, I guess one maybe way intuition pumped to support
Nick Bostrom (1:17:58.820)
this bland principle of indifference is to consider a sequence of different cases where
Lex Fridman (1:18:05.560)
the fraction of people who are simulated to non simulated approaches one.
Lex Fridman (1:18:12.600)
So in the limiting case where everybody is simulated, obviously you can deduce with certainty
Lex Fridman (1:18:22.580)
that you are simulated.
Nick Bostrom (1:18:24.920)
If everybody with your experiences is simulated and you know you've got to be one of those,
Lex Fridman (1:18:30.980)
you don't need a probability at all, you just kind of logically conclude it, right?
Lex Fridman (1:18:36.200)
So then as we move from a case where say 90% of everybody is simulated, 99%, 99.9%, it
Nick Bostrom (1:18:48.640)
should seem plausible that the probability you assign should sort of approach one certainty
Nick Bostrom (1:18:54.720)
as the fraction approaches the case where everybody is in a simulation.
Nick Bostrom (1:19:02.400)
You wouldn't expect that to be a discrete, well, if there's one non simulated person,
Nick Bostrom (1:19:06.800)
then it's 50, 50, but if we move that, then it's 100%, like it should kind of, there are
Nick Bostrom (1:19:12.520)
other arguments as well one can use to support this bland principle of indifference, but
Nick Bostrom (1:19:18.300)
that might be enough to.
Lex Fridman (1:19:19.300)
But in general, when you start from time equals zero and go into the future, the fraction
Nick Bostrom (1:19:25.560)
of simulated, if it's possible to create simulated worlds, the fraction of simulated worlds will
Lex Fridman (1:19:30.600)
go to one.
Nick Bostrom (1:19:31.600)
Well, I mean, it won't go all the way to one.
Nick Bostrom (1:19:37.800)
In reality, that would be some ratio, although maybe a technologically mature civilization
Nick Bostrom (1:19:43.680)
could run a lot of simulations using a small portion of its resources, it probably wouldn't
Lex Fridman (1:19:52.040)
be able to run infinitely many.
Nick Bostrom (1:19:53.160)
I mean, if we take say the observed, the physics in the observed universe, if we assume that
Nick Bostrom (1:19:59.280)
that's also the physics at the level of the simulators, that would be limits to the amount
Nick Bostrom (1:20:05.460)
of information processing that any one civilization could perform in its future trajectory.
Nick Bostrom (1:20:16.120)
First of all, there's limited amount of matter you can get your hands off because with a
Nick Bostrom (1:20:20.000)
positive cosmological constant, the universe is accelerating, there's like a finite sphere
Nick Bostrom (1:20:25.760)
of stuff, even if you traveled with the speed of light that you could ever reach, you have
Nick Bostrom (1:20:28.800)
a finite amount of stuff.
Lex Fridman (1:20:31.880)
And then if you think there is like a lower limit to the amount of loss you get when you
Nick Bostrom (1:20:37.760)
perform an erasure of a computation, or if you think, for example, just matter gradually
Nick Bostrom (1:20:42.260)
over cosmological timescales, decay, maybe protons decay, other things, and you radiate
Nick Bostrom (1:20:49.200)
out gravitational waves, like there's all kinds of seemingly unavoidable losses that
Lex Fridman (1:20:55.040)
occur.
Nick Bostrom (1:20:56.040)
Eventually, we'll have something like a heat death of the universe or a cold death or whatever,
Lex Fridman (1:21:04.080)
but yeah.
Lex Fridman (1:21:05.080)
So it's finite, but of course, we don't know which, if there's many ancestral simulations,
Lex Fridman (1:21:11.360)
we don't know which level we are.
Lex Fridman (1:21:13.640)
So there could be, couldn't there be like an arbitrary number of simulation that spawned
Lex Fridman (1:21:18.640)
ours, and those had more resources, in terms of physical universe to work with?
Lex Fridman (1:21:26.160)
Sorry, what do you mean that that could be?
Nick Bostrom (1:21:29.360)
Sort of, okay, so if simulations spawn other simulations, it seems like each new spawn
Nick Bostrom (1:21:40.280)
has fewer resources to work with.
Lex Fridman (1:21:44.200)
But we don't know at which step along the way we are at.
Nick Bostrom (1:21:50.240)
Any one observer doesn't know whether we're in level 42, or 100, or one, or is that not
Lex Fridman (1:21:58.320)
matter for the resources?
Nick Bostrom (1:22:01.160)
I mean, it's true that there would be uncertainty as to, you could have stacked simulations,
Lex Fridman (1:22:08.800)
and that could then be uncertainty as to which level we are at.
Nick Bostrom (1:22:16.040)
As you remarked also, all the computations performed in a simulation within the simulation
Lex Fridman (1:22:24.680)
also have to be expanded at the level of the simulation.
Lex Fridman (1:22:28.640)
So the computer in basement reality where all these simulations with the simulations
Nick Bostrom (1:22:32.320)
with the simulations are taking place, like that computer, ultimately, it's CPU or whatever
Lex Fridman (1:22:37.320)
it is, like that has to power this whole tower, right?
Lex Fridman (1:22:40.000)
So if there is a finite compute power in basement reality, that would impose a limit to how
Nick Bostrom (1:22:46.000)
tall this tower can be.
Lex Fridman (1:22:48.440)
And if each level kind of imposes a large extra overhead, you might think maybe the
Nick Bostrom (1:22:53.920)
tower would not be very tall, that most people would be low down in the tower.
Lex Fridman (1:23:00.720)
I love the term basement reality.
Nick Bostrom (1:23:03.120)
Let me ask one of the popularizers, you said there's many through this, when you look at
Nick Bostrom (1:23:09.320)
sort of the last few years of the simulation hypothesis, just like you said, it comes up
Nick Bostrom (1:23:14.640)
every once in a while, some new community discovers it and so on.
Lex Fridman (1:23:17.680)
But I would say one of the biggest popularizers of this idea is Elon Musk.
Lex Fridman (1:23:22.800)
Do you have any kind of intuition about what Elon thinks about when he thinks about simulation?
Lex Fridman (1:23:27.880)
Why is this of such interest?
Nick Bostrom (1:23:30.000)
Is it all the things we've talked about, or is there some special kind of intuition about
Lex Fridman (1:23:34.000)
simulation that he has?
Nick Bostrom (1:23:36.240)
I mean, you might have a better, I think, I mean, why it's of interest, I think it's
Nick Bostrom (1:23:40.000)
like seems pretty obvious why, to the extent that one thinks the argument is credible,
Lex Fridman (1:23:45.200)
why it would be of interest, it would, if it's correct, tell us something very important
Nick Bostrom (1:23:48.720)
about the world in one way or the other, whichever of the three alternatives for a simulation
Lex Fridman (1:23:53.280)
that seems like arguably one of the most fundamental discoveries, right?
Nick Bostrom (1:23:58.160)
Now, interestingly, in the case of someone like Elon, so there's like the standard arguments
Nick Bostrom (1:24:02.160)
for why you might want to take the simulation hypothesis seriously, the simulation argument,
Lex Fridman (1:24:06.720)
right?
Nick Bostrom (1:24:07.720)
In the case that if you are actually Elon Musk, let us say, there's a kind of an additional
Lex Fridman (1:24:12.360)
reason in that what are the chances you would be Elon Musk?
Nick Bostrom (1:24:17.280)
It seems like maybe there would be more interest in simulating the lives of very unusual and
Lex Fridman (1:24:24.400)
remarkable people.
Lex Fridman (1:24:26.280)
So if you consider not just simulations where all of human history or the whole of human
Nick Bostrom (1:24:32.440)
civilization are simulated, but also other kinds of simulations, which only include some
Nick Bostrom (1:24:37.880)
subset of people, like in those simulations that only include a subset, it might be more
Nick Bostrom (1:24:44.200)
likely that they would include subsets of people with unusually interesting or consequential
Nick Bostrom (1:24:49.080)
lives.
Lex Fridman (1:24:50.080)
So if you're Elon Musk, it's more likely that you're an inspiration.
Nick Bostrom (1:24:54.320)
Like if you're Donald Trump, or if you're Bill Gates, or you're like, some particularly
Nick Bostrom (1:25:00.560)
like distinctive character, you might think that that, I mean, if you just think of yourself
Nick Bostrom (1:25:06.200)
into the shoes, right, it's got to be like an extra reason to think that's kind of.
Lex Fridman (1:25:11.400)
So interesting.
Lex Fridman (1:25:12.400)
So on a scale of like farmer in Peru to Elon Musk, the more you get towards the Elon Musk,
Lex Fridman (1:25:19.200)
the higher the probability.
Nick Bostrom (1:25:20.200)
You'd imagine that would be some extra boost from that.
Lex Fridman (1:25:25.280)
There's an extra boost.
Lex Fridman (1:25:26.280)
So he also asked the question of what he would ask an AGI saying, the question being, what's
Lex Fridman (1:25:32.520)
outside the simulation?
Lex Fridman (1:25:34.680)
Do you think about the answer to this question?
Lex Fridman (1:25:37.740)
If we are living in a simulation, what is outside the simulation?
Lex Fridman (1:25:41.520)
So the programmer of the simulation?
Nick Bostrom (1:25:44.520)
Yeah, I mean, I think it connects to the question of what's inside the simulation in that.
Lex Fridman (1:25:49.760)
So if you had views about the creators of the simulation, it might help you make predictions
Nick Bostrom (1:25:56.920)
about what kind of simulation it is, what might happen, what happens after the simulation,
Nick Bostrom (1:26:03.480)
if there is some after, but also like the kind of setup.
Lex Fridman (1:26:06.640)
So these two questions would be quite closely intertwined.
Lex Fridman (1:26:12.000)
But do you think it would be very surprising to like, is the stuff inside the simulation,
Lex Fridman (1:26:17.920)
is it possible for it to be fundamentally different than the stuff outside?
Nick Bostrom (1:26:21.520)
Yeah.
Nick Bostrom (1:26:22.520)
Like, another way to put it, can the creatures inside the simulation be smart enough to even
Nick Bostrom (1:26:29.520)
understand or have the cognitive capabilities or any kind of information processing capabilities
Lex Fridman (1:26:34.640)
enough to understand the mechanism that created them?
Nick Bostrom (1:26:40.520)
They might understand some aspects of it.
Nick Bostrom (1:26:43.160)
I mean, it's a level of, it's kind of, there are levels of explanation, like degrees to
Nick Bostrom (1:26:50.120)
which you can understand.
Lex Fridman (1:26:51.120)
So does your dog understand what it is to be human?
Nick Bostrom (1:26:53.840)
Well, it's got some idea, like humans are these physical objects that move around and
Lex Fridman (1:26:58.120)
do things.
Lex Fridman (1:26:59.880)
And a normal human would have a deeper understanding of what it is to be a human.
Lex Fridman (1:27:05.720)
And maybe some very experienced psychologist or great novelist might understand a little
Nick Bostrom (1:27:12.120)
bit more about what it is to be human.
Lex Fridman (1:27:14.280)
And maybe superintelligence could see right through your soul.
Lex Fridman (1:27:18.760)
So similarly, I do think that we are quite limited in our ability to understand all of
Lex Fridman (1:27:27.080)
the relevant aspects of the larger context that we exist in.
Lex Fridman (1:27:31.880)
But there might be hope for some.
Lex Fridman (1:27:33.400)
I think we understand some aspects of it.
Lex Fridman (1:27:36.080)
But you know, how much good is that?
Nick Bostrom (1:27:38.320)
If there's like one key aspect that changes the significance of all the other aspects.
Lex Fridman (1:27:44.640)
So we understand maybe seven out of 10 key insights that you need.
Lex Fridman (1:27:51.800)
But the answer actually, like varies completely depending on what like number eight, nine
Lex Fridman (1:27:57.760)
and 10 insight is.
Nick Bostrom (1:28:00.200)
It's like whether you want to suppose that the big task were to guess whether a certain
Nick Bostrom (1:28:07.040)
number was odd or even, like a 10 digit number.
Lex Fridman (1:28:12.280)
And if it's even, the best thing for you to do in life is to go north.
Lex Fridman (1:28:16.440)
And if it's odd, the best thing for you is to go south.
Nick Bostrom (1:28:21.060)
Now we are in a situation where maybe through our science and philosophy, we figured out
Lex Fridman (1:28:25.000)
what the first seven digits are.
Lex Fridman (1:28:26.680)
So we have a lot of information, right?
Nick Bostrom (1:28:28.680)
Most of it we figured out.
Lex Fridman (1:28:31.040)
But we are clueless about what the last three digits are.
Lex Fridman (1:28:34.320)
So we are still completely clueless about whether the number is odd or even and therefore
Lex Fridman (1:28:38.720)
whether we should go north or go south.
Nick Bostrom (1:28:41.120)
I feel that's an analogy, but I feel we're somewhat in that predicament.
Lex Fridman (1:28:45.760)
We know a lot about the universe.
Nick Bostrom (1:28:48.460)
We've come maybe more than half of the way there to kind of fully understanding it.
Lex Fridman (1:28:52.640)
But the parts we're missing are plausibly ones that could completely change the overall
Nick Bostrom (1:28:58.680)
upshot of the thing and including change our overall view about what the scheme of priorities
Lex Fridman (1:29:04.800)
should be or which strategic direction would make sense to pursue.
Nick Bostrom (1:29:07.680)
Yeah.
Nick Bostrom (1:29:08.680)
I think your analogy of us being the dog trying to understand human beings is an entertaining
Nick Bostrom (1:29:15.520)
one, and probably correct.
Nick Bostrom (1:29:17.600)
The closer the understanding tends from the dog's viewpoint to us human psychologist viewpoint,
Nick Bostrom (1:29:24.960)
the steps along the way there will have completely transformative ideas of what it means to be
Lex Fridman (1:29:29.880)
human.
Lex Fridman (1:29:30.880)
So the dog has a very shallow understanding.
Nick Bostrom (1:29:33.920)
It's interesting to think that, to analogize that a dog's understanding of a human being
Nick Bostrom (1:29:39.800)
is the same as our current understanding of the fundamental laws of physics in the universe.
Lex Fridman (1:29:45.880)
Oh man.
Nick Bostrom (1:29:47.880)
Okay.
Lex Fridman (1:29:48.880)
We spent an hour and 40 minutes talking about the simulation.
Nick Bostrom (1:29:51.560)
I like it.
Lex Fridman (1:29:53.120)
Let's talk about super intelligence.
Nick Bostrom (1:29:54.360)
At least for a little bit.
Lex Fridman (1:29:57.120)
And let's start at the basics.
Lex Fridman (1:29:58.720)
What to you is intelligence?
Lex Fridman (1:30:00.720)
Yeah.
Nick Bostrom (1:30:01.720)
I tend not to get too stuck with the definitional question.
Nick Bostrom (1:30:05.960)
I mean, the common sense to understand, like the ability to solve complex problems, to
Nick Bostrom (1:30:11.400)
learn from experience, to plan, to reason, some combination of things like that.
Lex Fridman (1:30:18.600)
Is consciousness mixed up into that or no?
Lex Fridman (1:30:21.160)
Is consciousness mixed up into that?
Nick Bostrom (1:30:23.200)
Well, I think it could be fairly intelligent at least without being conscious probably.
Lex Fridman (1:30:31.000)
So then what is super intelligence?
Nick Bostrom (1:30:33.920)
That would be like something that was much more, had much more general cognitive capacity
Nick Bostrom (1:30:40.120)
than we humans have.
Lex Fridman (1:30:41.760)
So if we talk about general super intelligence, it would be much faster learner be able to
Nick Bostrom (1:30:48.640)
reason much better, make plans that are more effective at achieving its goals, say in a
Lex Fridman (1:30:53.520)
wide range of complex challenging environments.
Nick Bostrom (1:30:57.000)
In terms of as we turn our eye to the idea of sort of existential threats from super
Nick Bostrom (1:31:03.040)
intelligence, do you think super intelligence has to exist in the physical world or can
Lex Fridman (1:31:08.920)
it be digital only?
Nick Bostrom (1:31:10.880)
Sort of we think of our general intelligence as us humans, as an intelligence that's associated
Nick Bostrom (1:31:17.920)
with the body, that's able to interact with the world, that's able to affect the world
Lex Fridman (1:31:22.040)
directly with physically.
Nick Bostrom (1:31:23.360)
I mean, digital only is perfectly fine, I think.
Nick Bostrom (1:31:26.200)
I mean, you could, it's physical in the sense that obviously the computers and the memories
Nick Bostrom (1:31:31.320)
are physical.
Lex Fridman (1:31:32.320)
But it's capability to affect the world sort of.
Nick Bostrom (1:31:34.960)
Could be very strong, even if it has a limited set of actuators, if it can type text on the
Lex Fridman (1:31:42.160)
screen or something like that, that would be, I think, ample.
Lex Fridman (1:31:45.860)
So in terms of the concerns of existential threat of AI, how can an AI system that's
Nick Bostrom (1:31:52.960)
in the digital world have existential risk, sort of, and what are the attack vectors for
Lex Fridman (1:32:00.800)
a digital system?
Nick Bostrom (1:32:01.800)
Well, I mean, I guess maybe to take one step back, so I should emphasize that I also think
Nick Bostrom (1:32:07.800)
there's this huge positive potential from machine intelligence, including super intelligence.
Lex Fridman (1:32:13.440)
And I want to stress that because some of my writing has focused on what can go wrong.
Lex Fridman (1:32:20.920)
And when I wrote the book Superintelligence, at that point, I felt that there was a kind
Nick Bostrom (1:32:27.020)
of neglect of what would happen if AI succeeds, and in particular, a need to get a more granular
Nick Bostrom (1:32:34.300)
understanding of where the pitfalls are so we can avoid them.
Nick Bostrom (1:32:38.680)
I think that since the book came out in 2014, there has been a much wider recognition of
Nick Bostrom (1:32:45.560)
that.
Lex Fridman (1:32:46.560)
And a number of research groups are now actually working on developing, say, AI alignment techniques
Lex Fridman (1:32:51.620)
and so on and so forth.
Lex Fridman (1:32:52.620)
So yeah, I think now it's important to make sure we bring back onto the table the upside
Nick Bostrom (1:33:01.740)
as well.
Lex Fridman (1:33:02.740)
And there's a little bit of a neglect now on the upside, which is, I mean, if you look
Nick Bostrom (1:33:07.060)
at, I was talking to a friend, if you look at the amount of information that is available,
Nick Bostrom (1:33:11.780)
or people talking and people being excited about the positive possibilities of general
Nick Bostrom (1:33:16.180)
intelligence, that's not, it's far outnumbered by the negative possibilities in terms of
Lex Fridman (1:33:23.860)
our public discourse.
Nick Bostrom (1:33:25.140)
Possibly, yeah.
Lex Fridman (1:33:26.140)
It's hard to measure.
Lex Fridman (1:33:28.280)
But what are, can you linger on that for a little bit, what are some, to you, possible
Lex Fridman (1:33:33.560)
big positive impacts of general intelligence?
Lex Fridman (1:33:37.700)
Super intelligence?
Nick Bostrom (1:33:38.700)
Well, I mean, super intelligence, because I tend to also want to distinguish these two
Nick Bostrom (1:33:43.200)
different contexts of thinking about AI and AI impacts, the kind of near term and long
Nick Bostrom (1:33:48.900)
term, if you want, both of which I think are legitimate things to think about, and people
Nick Bostrom (1:33:54.680)
should discuss both of them, but they are different and they often get mixed up.
Lex Fridman (1:34:02.020)
And then, then I get, you get confusion, like, I think you get simultaneously like maybe
Nick Bostrom (1:34:06.680)
an overhyping of the near term and then under hyping of the long term.
Lex Fridman (1:34:10.220)
And so I think as long as we keep them apart, we can have like, two good conversations,
Lex Fridman (1:34:15.260)
but or we can mix them together and have one bad conversation.
Lex Fridman (1:34:18.660)
Can you clarify just the two things we were talking about, the near term and the long
Lex Fridman (1:34:22.820)
term?
Lex Fridman (1:34:23.820)
Yeah.
Lex Fridman (1:34:24.820)
And what are the distinctions?
Lex Fridman (1:34:25.820)
Well, it's a, it's a blurry distinction.
Lex Fridman (1:34:28.100)
But say the things I wrote about in this book, super intelligence, long term, things people
Nick Bostrom (1:34:34.940)
are worrying about today with, I don't know, algorithmic discrimination, or even things,
Nick Bostrom (1:34:41.540)
self driving cars and drones and stuff, more near term.
Lex Fridman (1:34:47.100)
And then of course, you could imagine some medium term where they kind of overlap and
Nick Bostrom (1:34:51.340)
they one evolves into the other.
Lex Fridman (1:34:55.180)
But at any rate, I think both, yeah, the issues look kind of somewhat different depending
Nick Bostrom (1:35:00.020)
on which of these contexts.
Lex Fridman (1:35:01.600)
So I think, I think it'd be nice if we can talk about the long term and think about a
Nick Bostrom (1:35:10.340)
positive impact or a better world because of the existence of the long term super intelligence.
Lex Fridman (1:35:17.820)
Do you have views of such a world?
Nick Bostrom (1:35:19.420)
Yeah.
Nick Bostrom (1:35:20.420)
I mean, I guess it's a little hard to articulate because it seems obvious that the world has
Nick Bostrom (1:35:24.780)
a lot of problems as it currently stands.
Lex Fridman (1:35:29.700)
And it's hard to think of any one of those, which it wouldn't be useful to have like a
Nick Bostrom (1:35:36.620)
friendly aligned super intelligence working on.
Lex Fridman (1:35:40.720)
So from health to the economic system to be able to sort of improve the investment and
Nick Bostrom (1:35:48.300)
trade and foreign policy decisions, all that kind of stuff.
Lex Fridman (1:35:52.180)
All that kind of stuff and a lot more.
Lex Fridman (1:35:56.300)
I mean, what's the killer app?
Lex Fridman (1:35:57.780)
Well, I don't think there is one.
Nick Bostrom (1:35:59.540)
I think AI, especially artificial general intelligence is really the ultimate general
Lex Fridman (1:36:05.900)
purpose technology.
Lex Fridman (1:36:07.820)
So it's not that there is this one problem, this one area where it will have a big impact.
Lex Fridman (1:36:12.220)
But if and when it succeeds, it will really apply across the board in all fields where
Nick Bostrom (1:36:18.980)
human creativity and intelligence and problem solving is useful, which is pretty much all
Lex Fridman (1:36:23.820)
fields.
Nick Bostrom (1:36:24.820)
Right.
Lex Fridman (1:36:25.820)
The thing that it would do is give us a lot more control over nature.
Nick Bostrom (1:36:30.840)
It wouldn't automatically solve the problems that arise from conflict between humans, fundamentally
Lex Fridman (1:36:37.220)
political problems.
Nick Bostrom (1:36:38.220)
Some subset of those might go away if you just had more resources and cooler tech.
Lex Fridman (1:36:42.180)
But some subset would require coordination that is not automatically achieved just by
Nick Bostrom (1:36:50.360)
having more technological capability.
Lex Fridman (1:36:53.240)
But anything that's not of that sort, I think you just get an enormous boost with this kind
Nick Bostrom (1:36:59.500)
of cognitive technology once it goes all the way.
Nick Bostrom (1:37:02.940)
Now, again, that doesn't mean I'm thinking, oh, people don't recognize what's possible
Nick Bostrom (1:37:10.460)
with current technology and like sometimes things get overhyped.
Lex Fridman (1:37:14.180)
But I mean, those are perfectly consistent views to hold.
Nick Bostrom (1:37:16.940)
The ultimate potential being enormous.
Lex Fridman (1:37:19.940)
And then it's a very different question of how far are we from that or what can we do
Lex Fridman (1:37:23.780)
with near term technology?
Lex Fridman (1:37:25.060)
Yeah.
Lex Fridman (1:37:26.060)
So what's your intuition about the idea of intelligence explosion?
Lex Fridman (1:37:29.220)
So there's this, you know, when you start to think about that leap from the near term
Nick Bostrom (1:37:34.920)
to the long term, the natural inclination, like for me, sort of building machine learning
Nick Bostrom (1:37:40.120)
systems today, it seems like it's a lot of work to get the general intelligence, but
Nick Bostrom (1:37:45.180)
there's some intuition of exponential growth of exponential improvement of intelligence
Lex Fridman (1:37:49.380)
explosion.
Lex Fridman (1:37:50.380)
Can you maybe try to elucidate, try to talk about what's your intuition about the possibility
Nick Bostrom (1:38:00.860)
of an intelligence explosion, that it won't be this gradual slow process, there might
Lex Fridman (1:38:05.500)
be a phase shift?
Lex Fridman (1:38:07.900)
Yeah, I think it's, we don't know how explosive it will be.
Nick Bostrom (1:38:13.500)
I think for what it's worth, it seems fairly likely to me that at some point, there will
Nick Bostrom (1:38:19.420)
be some intelligence explosion, like some period of time, where progress in AI becomes
Nick Bostrom (1:38:24.540)
extremely rapid, roughly, roughly in the area where you might say it's kind of humanish
Nick Bostrom (1:38:32.220)
equivalent in core cognitive faculties, that the concept of human equivalent starts to
Nick Bostrom (1:38:40.600)
break down when you look too closely at it.
Lex Fridman (1:38:43.260)
And just how explosive does something have to be for it to be called an intelligence
Lex Fridman (1:38:48.300)
explosion?
Lex Fridman (1:38:49.300)
Like, does it have to be like overnight, literally, or a few years?
Lex Fridman (1:38:54.460)
But overall, I guess, if you plotted the opinions of different people in the world, I guess
Nick Bostrom (1:39:00.420)
that would be somewhat more probability towards the intelligence explosion scenario than probably
Nick Bostrom (1:39:06.100)
the average, you know, AI researcher, I guess.
Lex Fridman (1:39:09.620)
So and then the other part of the intelligence explosion, or just forget explosion, just
Nick Bostrom (1:39:14.620)
progress is once you achieve that gray area of human level intelligence, is it obvious
Lex Fridman (1:39:21.540)
to you that we should be able to proceed beyond it to get to super intelligence?
Nick Bostrom (1:39:26.780)
Yeah, that seems, I mean, as much as any of these things can be obvious, given we've never
Nick Bostrom (1:39:33.580)
had one, people have different views, smart people have different views, it's like some
Nick Bostrom (1:39:39.380)
degree of uncertainty that always remains for any big, futuristic, philosophical grand
Nick Bostrom (1:39:44.860)
question that just we realize humans are fallible, especially about these things.
Lex Fridman (1:39:49.660)
But it does seem, as far as I'm judging things based on my own impressions, that it seems
Lex Fridman (1:39:55.460)
very unlikely that that would be a ceiling at or near human cognitive capacity.
Lex Fridman (1:40:04.340)
And that's such a, I don't know, that's such a special moment, it's both terrifying and
Lex Fridman (1:40:10.260)
exciting to create a system that's beyond our intelligence.
Lex Fridman (1:40:15.020)
So maybe you can step back and say, like, how does that possibility make you feel that
Nick Bostrom (1:40:22.140)
we can create something, it feels like there's a line beyond which it steps, it'll be able
Nick Bostrom (1:40:28.700)
to outsmart you.
Lex Fridman (1:40:31.180)
And therefore, it feels like a step where we lose control.
Nick Bostrom (1:40:35.060)
Well, I don't think the latter follows that is you could imagine.
Lex Fridman (1:40:42.080)
And in fact, this is what a number of people are working towards making sure that we could
Nick Bostrom (1:40:46.500)
ultimately project higher levels of problem solving ability while still making sure that
Lex Fridman (1:40:53.060)
they are aligned, like they are in the service of human values.
Nick Bostrom (1:40:58.620)
I mean, so losing control, I think, is not a given that that would happen.
Nick Bostrom (1:41:06.300)
Now you asked how it makes me feel, I mean, to some extent, I've lived with this for so
Nick Bostrom (1:41:10.060)
long, since as long as I can remember, being an adult or even a teenager, it seemed to
Lex Fridman (1:41:16.820)
me obvious that at some point, AI will succeed.
Lex Fridman (1:41:19.780)
And so I actually misspoke, I didn't mean control, I meant, because the control problem
Lex Fridman (1:41:27.020)
is an interesting thing.
Lex Fridman (1:41:28.020)
And I think the hope is, at least we should be able to maintain control over systems that
Lex Fridman (1:41:33.820)
are smarter than us.
Lex Fridman (1:41:35.500)
But we do lose our specialness, it sort of will lose our place as the smartest, coolest
Lex Fridman (1:41:46.460)
thing on earth.
Lex Fridman (1:41:48.700)
And there's an ego involved with that, that humans aren't very good at dealing with.
Lex Fridman (1:41:55.700)
I mean, I value my intelligence as a human being.
Nick Bostrom (1:41:59.860)
It seems like a big transformative step to realize there's something out there that's
Lex Fridman (1:42:04.740)
more intelligent.
Nick Bostrom (1:42:05.740)
I mean, you don't see that as such a fundamentally...
Nick Bostrom (1:42:09.580)
I think yes, a lot, I think it would be small, because I mean, I think there are already
Nick Bostrom (1:42:14.300)
a lot of things out there that are, I mean, certainly, if you think the universe is big,
Nick Bostrom (1:42:18.980)
there's going to be other civilizations that already have super intelligences, or that
Nick Bostrom (1:42:23.140)
just naturally have brains the size of beach balls and are like, completely leaving us
Lex Fridman (1:42:29.240)
in the dust.
Lex Fridman (1:42:30.900)
And we haven't come face to face with them.
Lex Fridman (1:42:33.540)
We haven't come face to face.
Lex Fridman (1:42:34.820)
But I mean, that's an open question, what would happen in a kind of post human world?
Nick Bostrom (1:42:41.900)
Like how much day to day would these super intelligences be involved in the lives of
Lex Fridman (1:42:49.300)
ordinary?
Nick Bostrom (1:42:50.300)
I mean, you could imagine some scenario where it would be more like a background thing that
Nick Bostrom (1:42:54.420)
would help protect against some things, but you wouldn't like that, they wouldn't be this
Nick Bostrom (1:42:58.980)
intrusive kind of, like making you feel bad by like, making clever jokes on your expert,
Nick Bostrom (1:43:04.620)
like there's like all sorts of things that maybe in the human context would feel awkward
Lex Fridman (1:43:09.180)
about that.
Nick Bostrom (1:43:10.180)
You don't want to be the dumbest kid in your class, everybody picks it, like, a lot of
Nick Bostrom (1:43:14.500)
those things, maybe you need to abstract away from, if you're thinking about this context
Nick Bostrom (1:43:19.380)
where we have infrastructure that is in some sense, beyond any or all humans.
Nick Bostrom (1:43:26.260)
I mean, it's a little bit like, say, the scientific community as a whole, if you think of that
Nick Bostrom (1:43:30.780)
as a mind, it's a little bit of a metaphor.
Lex Fridman (1:43:33.420)
But I mean, obviously, it's got to be like, way more capacious than any individual.
Lex Fridman (1:43:39.500)
So in some sense, there is this mind like thing already out there that's just vastly
Lex Fridman (1:43:44.620)
more intelligent than any individual is.
Lex Fridman (1:43:49.760)
And we think, okay, that's, you just accept that as a fact.
Lex Fridman (1:43:55.380)
That's the basic fabric of our existence is there's super intelligent.
Nick Bostrom (1:43:59.260)
You get used to a lot of, I mean, there's already Google and Twitter and Facebook, these
Nick Bostrom (1:44:06.020)
recommender systems that are the basic fabric of our, I could see them becoming, I mean,
Lex Fridman (1:44:13.220)
do you think of the collective intelligence of these systems as already perhaps reaching
Lex Fridman (1:44:17.480)
super intelligence level?
Nick Bostrom (1:44:19.260)
Well, I mean, so here it comes to the concept of intelligence and the scale and what human
Lex Fridman (1:44:26.260)
level means.
Nick Bostrom (1:44:29.700)
The kind of vagueness and indeterminacy of those concepts starts to dominate how you
Lex Fridman (1:44:37.820)
would answer that question.
Lex Fridman (1:44:38.900)
So like, say the Google search engine has a very high capacity of a certain kind, like
Nick Bostrom (1:44:45.020)
retrieving, remembering and retrieving information, particularly like text or images that are,
Nick Bostrom (1:44:54.940)
you have a kind of string, a word string key, obviously superhuman at that, but a vast set
Lex Fridman (1:45:02.980)
of other things it can't even do at all.
Nick Bostrom (1:45:06.260)
Not just not do well, but so you have these current AI systems that are superhuman in
Lex Fridman (1:45:12.780)
some limited domain and then like radically subhuman in all other domains.
Nick Bostrom (1:45:19.300)
Same with a chess, like are just a simple computer that can multiply really large numbers,
Lex Fridman (1:45:23.900)
right?
Lex Fridman (1:45:24.900)
So it's going to have this like one spike of super intelligence and then a kind of a
Lex Fridman (1:45:28.060)
zero level of capability across all other cognitive fields.
Nick Bostrom (1:45:32.300)
Yeah, I don't necessarily think the generalness, I mean, I'm not so attached with it, but I
Nick Bostrom (1:45:37.340)
think it's sort of, it's a gray area and it's a feeling, but to me sort of alpha zero is
Nick Bostrom (1:45:44.340)
somehow much more intelligent, much, much more intelligent than Deep Blue.
Lex Fridman (1:45:51.620)
And to say which domain, you could say, well, these are both just board games, they're both
Nick Bostrom (1:45:55.380)
just able to play board games, who cares if they're going to do better or not, but there's
Nick Bostrom (1:45:59.380)
something about the learning, the self play that makes it, crosses over into that land
Nick Bostrom (1:46:06.120)
of intelligence that doesn't necessarily need to be general.
Nick Bostrom (1:46:09.460)
In the same way, Google is much closer to Deep Blue currently in terms of its search
Nick Bostrom (1:46:14.180)
engine than it is to sort of the alpha zero.
Lex Fridman (1:46:17.900)
And the moment it becomes, the moment these recommender systems really become more like
Nick Bostrom (1:46:22.860)
alpha zero, but being able to learn a lot without the constraints of being heavily constrained
Lex Fridman (1:46:29.820)
by human interaction, that seems like a special moment in time.
Nick Bostrom (1:46:34.100)
I mean, certainly learning ability seems to be an important facet of general intelligence,
Nick Bostrom (1:46:43.140)
that you can take some new domain that you haven't seen before and you weren't specifically
Nick Bostrom (1:46:48.200)
pre programmed for, and then figure out what's going on there and eventually become really
Lex Fridman (1:46:52.340)
good at it.
Lex Fridman (1:46:53.520)
So that's something alpha zero has much more of than Deep Blue had.
Lex Fridman (1:47:00.140)
And in fact, I mean, systems like alpha zero can learn not just Go, but other, in fact,
Nick Bostrom (1:47:06.340)
probably beat Deep Blue in chess and so forth.
Lex Fridman (1:47:09.460)
So you do see this as general and it matches the intuition.
Nick Bostrom (1:47:13.700)
We feel it's more intelligent and it also has more of this general purpose learning
Lex Fridman (1:47:17.640)
ability.
Lex Fridman (1:47:20.100)
And if we get systems that have even more general purpose learning ability, it might
Nick Bostrom (1:47:23.060)
also trigger an even stronger intuition that they are actually starting to get smart.
Lex Fridman (1:47:28.100)
So if you were to pick a future, what do you think a utopia looks like with AGI systems?
Nick Bostrom (1:47:33.980)
Sort of, is it the neural link brain computer interface world where we're kind of really
Lex Fridman (1:47:40.540)
closely interlinked with AI systems?
Nick Bostrom (1:47:43.780)
Is it possibly where AGI systems replace us completely while maintaining the values and
Lex Fridman (1:47:50.900)
the consciousness?
Nick Bostrom (1:47:53.500)
Is it something like it's a completely invisible fabric, like you mentioned, a society where
Nick Bostrom (1:47:57.460)
just aids and a lot of stuff that we do like curing diseases and so on.
Lex Fridman (1:48:02.180)
What is utopia if you get to pick?
Nick Bostrom (1:48:03.980)
Yeah, I mean, it is a good question and a deep and difficult one.
Lex Fridman (1:48:09.020)
I'm quite interested in it.
Nick Bostrom (1:48:10.300)
I don't have all the answers yet, but I might never have.
Lex Fridman (1:48:15.180)
But I think there are some different observations one can make.
Nick Bostrom (1:48:19.660)
One is if this scenario actually did come to pass, it would open up this vast space
Lex Fridman (1:48:26.180)
of possible modes of being.
Nick Bostrom (1:48:30.500)
On one hand, material and resource constraints would just be like expanded dramatically.
Lex Fridman (1:48:36.260)
So there would be a lot of a big pie, let's say.
Nick Bostrom (1:48:41.900)
Also it would enable us to do things, including to ourselves, it would just open up this much
Nick Bostrom (1:48:51.940)
larger design space and option space than we have ever had access to in human history.
Nick Bostrom (1:48:59.100)
I think two things follow from that.
Nick Bostrom (1:49:01.140)
One is that we probably would need to make a fairly fundamental rethink of what ultimately
Nick Bostrom (1:49:08.420)
we value, like think things through more from first principles.
Nick Bostrom (1:49:11.940)
The context would be so different from the familiar that we could have just take what
Nick Bostrom (1:49:15.140)
we've always been doing and then like, oh, well, we have this cleaning robot that cleans
Lex Fridman (1:49:21.260)
the dishes in the sink and a few other small things.
Nick Bostrom (1:49:24.780)
I think we would have to go back to first principles.
Lex Fridman (1:49:27.100)
So even from the individual level, go back to the first principles of what is the meaning
Nick Bostrom (1:49:31.560)
of life, what is happiness, what is fulfillment.
Lex Fridman (1:49:35.540)
And then also connected to this large space of resources is that it would be possible.
Lex Fridman (1:49:43.300)
And I think something we should aim for is to do well by the lights of more than one
Lex Fridman (1:49:52.860)
value system.
Nick Bostrom (1:49:55.260)
That is, we wouldn't have to choose only one value criterion and say we're going to do
Nick Bostrom (1:50:06.380)
something that scores really high on the metric of, say, hedonism, and then is like a zero
Nick Bostrom (1:50:15.860)
by other criteria, like kind of wireheaded brain synovat, and it's like a lot of pleasure,
Lex Fridman (1:50:22.520)
that's good, but then like no beauty, no achievement like that.
Nick Bostrom (1:50:26.860)
Or pick it up, I think to some significant, not unlimited sense, but the significant sense,
Nick Bostrom (1:50:32.740)
it would be possible to do very well by many criteria, like maybe you could get like 98%
Nick Bostrom (1:50:40.060)
of the best according to several criteria at the same time, given this great expansion
Lex Fridman (1:50:47.900)
of the option space.
Lex Fridman (1:50:50.820)
So have competing value systems, competing criteria, as a sort of forever, just like
Nick Bostrom (1:50:57.400)
our Democrat versus Republican, there seems to be this always multiple parties that are
Nick Bostrom (1:51:02.780)
useful for our progress in society, even though it might seem dysfunctional inside the moment,
Lex Fridman (1:51:08.260)
but having the multiple value system seems to be beneficial for, I guess, a balance of
Nick Bostrom (1:51:14.820)
power.
Lex Fridman (1:51:15.820)
So that's, yeah, not exactly what I have in mind that it, well, although maybe in an indirect
Nick Bostrom (1:51:21.740)
way it is, but that if you had the chance to do something that scored well on several
Nick Bostrom (1:51:30.460)
different metrics, our first instinct should be to do that rather than immediately leap
Lex Fridman (1:51:36.300)
to the thing, which ones of these value systems are we going to screw over?
Lex Fridman (1:51:40.940)
Like our first, let's first try to do very well by all of them.
Nick Bostrom (1:51:44.680)
Then it might be that you can't get 100% of all and you would have to then like have the
Lex Fridman (1:51:49.140)
hard conversation about which one will only get 97%.
Nick Bostrom (1:51:51.740)
There you go.
Nick Bostrom (1:51:52.740)
There's my cynicism that all of existence is always a trade off, but you say, maybe
Nick Bostrom (1:51:57.540)
it's not such a bad trade off.
Lex Fridman (1:51:58.900)
Let's first at least try it.
Nick Bostrom (1:52:00.100)
Well, this would be a distinctive context in which at least some of the constraints
Lex Fridman (1:52:06.140)
would be removed.
Nick Bostrom (1:52:07.140)
I'll leave it at that.
Lex Fridman (1:52:08.140)
So there's probably still be trade offs in the end.
Nick Bostrom (1:52:10.560)
It's just that we should first make sure we at least take advantage of this abundance.
Lex Fridman (1:52:16.820)
So in terms of thinking about this, like, yeah, one should think, I think in this kind
Nick Bostrom (1:52:21.580)
of frame of mind of generosity and inclusiveness to different value systems and see how far
Lex Fridman (1:52:31.060)
one can get there at first.
Lex Fridman (1:52:34.900)
And I think one could do something that would be very good according to many different criteria.
Nick Bostrom (1:52:41.860)
We kind of talked about AGI fundamentally transforming the value system of our existence,
Nick Bostrom (1:52:50.300)
the meaning of life.
Lex Fridman (1:52:52.620)
But today, what do you think is the meaning of life?
Lex Fridman (1:52:56.360)
The silliest or perhaps the biggest question, what's the meaning of life?
Lex Fridman (1:52:59.620)
What's the meaning of existence?
Lex Fridman (1:53:03.060)
What gives your life fulfillment, purpose, happiness, meaning?
Nick Bostrom (1:53:07.980)
Yeah, I think these are, I guess, a bunch of different but related questions in there
Nick Bostrom (1:53:14.620)
that one can ask.
Lex Fridman (1:53:17.500)
Happiness meaning.
Nick Bostrom (1:53:18.500)
Yeah.
Nick Bostrom (1:53:19.500)
I mean, like you could imagine somebody getting a lot of happiness from something that they
Nick Bostrom (1:53:22.900)
didn't think was meaningful.
Nick Bostrom (1:53:27.060)
Like mindless, like watching reruns of some television series, waiting junk food, like
Nick Bostrom (1:53:31.620)
maybe some people that gives pleasure, but they wouldn't think it had a lot of meaning.
Nick Bostrom (1:53:35.940)
Whereas, conversely, something that might be quite loaded with meaning might not be
Nick Bostrom (1:53:39.740)
very fun always, like some difficult achievement that really helps a lot of people, maybe requires
Lex Fridman (1:53:45.640)
self sacrifice and hard work.
Lex Fridman (1:53:49.520)
So these things can, I think, come apart, which is something to bear in mind also when
Nick Bostrom (1:53:57.400)
if you're thinking about these utopia questions that you might, to actually start to do some
Nick Bostrom (1:54:06.380)
constructive thinking about that, you might have to isolate and distinguish these different
Lex Fridman (1:54:12.500)
kinds of things that might be valuable in different ways.
Nick Bostrom (1:54:16.360)
Make sure you can sort of clearly perceive each one of them and then you can think about
Lex Fridman (1:54:20.060)
how you can combine them.
Lex Fridman (1:54:22.180)
And just as you said, hopefully come up with a way to maximize all of them together.
Nick Bostrom (1:54:27.540)
Yeah, or at least get, I mean, maximize or get like a very high score on a wide range
Nick Bostrom (1:54:33.300)
of them, even if not literally all.
Lex Fridman (1:54:35.100)
You can always come up with values that are exactly opposed to one another, right?
Lex Fridman (1:54:39.460)
But I think for many values, they're kind of opposed with, if you place them within
Nick Bostrom (1:54:45.340)
a certain dimensionality of your space, like there are shapes that are kind of, you can't
Nick Bostrom (1:54:51.860)
untangle like in a given dimensionality, but if you start adding dimensions, then it might
Lex Fridman (1:54:57.220)
in many cases just be that they are easy to pull apart and you could.
Lex Fridman (1:55:02.140)
So we'll see how much space there is for that, but I think that there could be a lot in this
Lex Fridman (1:55:07.300)
context of radical abundance, if ever we get to that.
Nick Bostrom (1:55:12.180)
I don't think there's a better way to end it, Nick.
Nick Bostrom (1:55:15.580)
You've influenced a huge number of people to work on what could very well be the most
Nick Bostrom (1:55:20.980)
important problems of our time.
Lex Fridman (1:55:22.660)
So it's a huge honor.
Nick Bostrom (1:55:23.660)
Thank you so much for talking.
Lex Fridman (1:55:24.660)
Well, thank you for coming by, Lex.
Nick Bostrom (1:55:25.660)
That was fun.
Lex Fridman (1:55:26.660)
Thank you.
Nick Bostrom (1:55:27.660)
Thanks for listening to this conversation with Nick Bostrom, and thank you to our presenting
Lex Fridman (1:55:31.940)
sponsor, Cash App.
Nick Bostrom (1:55:33.940)
Please consider supporting the podcast by downloading Cash App and using code LEXPodcast.
Nick Bostrom (1:55:40.140)
If you enjoy this podcast, subscribe on YouTube, review it with five stars on Apple Podcast,
Nick Bostrom (1:55:45.100)
subscribe on Patreon, or simply connect with me on Twitter at Lex Friedman.
Lex Fridman (1:55:50.980)
And now, let me leave you with some words from Nick Bostrom.
Nick Bostrom (1:55:55.420)
Our approach to existential risks cannot be one of trial and error.
Lex Fridman (1:56:00.060)
There's no opportunity to learn from errors.
Nick Bostrom (1:56:02.780)
The reactive approach, see what happens, limit damages, and learn from experience is unworkable.
Lex Fridman (1:56:09.500)
Rather, we must take a proactive approach.
Nick Bostrom (1:56:13.140)
This requires foresight to anticipate new types of threats and a willingness to take
Nick Bostrom (1:56:17.900)
decisive, preventative action and to bear the costs, moral and economic, of such actions.
Nick Bostrom (1:56:26.060)
Thank you for listening, and hope to see you next time.
Lex Fridman (20:02.380)
to do to fundamentally changing the sort of the fabric of reality.
Nick Bostrom (20:09.440)
If you do it is ethical concerns, all those kinds of things could be exceptionally strong
Lex Fridman (20:14.400)
pressures.
Nick Bostrom (20:15.400)
Well, certainly, I mean, yeah, ethical concerns.
Lex Fridman (20:18.440)
I mean, not really too difficult to do.
Nick Bostrom (20:21.160)
I mean, in a sense, that's the first assumption that you get to technological maturity where
Nick Bostrom (20:26.000)
you would have the ability using only a tiny fraction of your resources to create many,
Nick Bostrom (20:32.600)
many simulations.
Lex Fridman (20:34.520)
So it wouldn't be the case that they would need to spend half of their GDP forever in
Nick Bostrom (20:39.760)
order to create one simulation and they had this like difficult debate about whether they
Lex Fridman (20:43.480)
should invest half of their GDP for this.
Nick Bostrom (20:46.400)
It would more be like, well, if any little fraction of the civilization feels like doing
Nick Bostrom (20:50.320)
this at any point during maybe their millions of years of existence, then that would be
Nick Bostrom (20:57.000)
millions of simulations.
Lex Fridman (21:00.560)
But certainly, there could be many conceivable reasons for why there would be this convert,
Nick Bostrom (21:07.800)
many possible reasons for not running ancestor simulations or other computer simulations,
Lex Fridman (21:13.020)
even if you could do so cheaply.
Lex Fridman (21:15.160)
By the way, what's an ancestor simulation?
Nick Bostrom (21:17.160)
Well, that would be the type of computer simulation that would contain people like those we think
Nick Bostrom (21:24.880)
have lived on our planet in the past and like ourselves in terms of the types of experiences
Lex Fridman (21:30.060)
they have and where those simulated people are conscious.
Lex Fridman (21:33.560)
So like not just simulated in the same sense that a non player character would be simulated
Nick Bostrom (21:41.400)
in the current computer game where it's kind of has like an avatar body and then a very
Nick Bostrom (21:45.560)
simple mechanism that moves it forward or backwards.
Lex Fridman (21:49.880)
But something where the simulated being has a brain, let's say that's simulated at a sufficient
Nick Bostrom (21:56.880)
level of granularity that it would have the same subjective experiences as we have.
Lex Fridman (22:03.400)
So where does consciousness fit into this?
Lex Fridman (22:06.400)
Do you think simulation, I guess there are different ways to think about how this can
Lex Fridman (22:10.380)
be simulated, just like you're talking about now.
Lex Fridman (22:14.320)
Do we have to simulate each brain within the larger simulation?
Nick Bostrom (22:21.040)
Is it enough to simulate just the brain, just the minds and not the simulation, not the
Lex Fridman (22:26.840)
universe itself?
Lex Fridman (22:27.840)
Like, is there a different ways to think about this?
Nick Bostrom (22:29.880)
Yeah, I guess there is a kind of premise in the simulation argument rolled in from philosophy
Nick Bostrom (22:38.280)
of mind that is that it would be possible to create a conscious mind in a computer.
Lex Fridman (22:45.200)
And that what determines whether some system is conscious or not is not like whether it's
Nick Bostrom (22:51.680)
built from organic biological neurons, but maybe something like what the structure of
Nick Bostrom (22:56.880)
the computation is that it implements.
Lex Fridman (22:59.680)
So we can discuss that if we want, but I think it would be more forward as far as my view
Nick Bostrom (23:05.860)
that it would be sufficient, say, if you had a computation that was identical to the computation
Lex Fridman (23:15.140)
in the human brain down to the level of neurons.
Lex Fridman (23:17.520)
So if you had a simulation with 100 billion neurons connected in the same way as the human
Nick Bostrom (23:21.040)
brain, and you then roll that forward with the same kind of synaptic weights and so forth,
Lex Fridman (23:27.520)
so you actually had the same behavior coming out of this as a human with that brain would
Lex Fridman (23:33.320)
have done, then I think that would be conscious.
Nick Bostrom (23:36.000)
Now it's possible you could also generate consciousness without having that detailed
Nick Bostrom (23:43.200)
assimilation, there I'm getting more uncertain exactly how much you could simplify or abstract
Nick Bostrom (23:50.080)
away.
Lex Fridman (23:51.080)
Can you look on that?
Lex Fridman (23:52.080)
What do you mean?
Lex Fridman (23:53.080)
I missed where you're placing consciousness in the second.
Nick Bostrom (23:56.920)
Well, so if you are a computationalist, do you think that what creates consciousness
Lex Fridman (24:01.800)
is the implementation of a computation?
Nick Bostrom (24:04.880)
Some property, emergent property of the computation itself.
Lex Fridman (24:07.760)
Yeah.
Nick Bostrom (24:08.760)
That's the idea.
Lex Fridman (24:09.760)
Yeah, you could say that.
Lex Fridman (24:10.760)
But then the question is, what's the class of computations such that when they are run,
Lex Fridman (24:16.960)
consciousness emerges?
Lex Fridman (24:18.160)
So if you just have something that adds one plus one plus one plus one, like a simple
Lex Fridman (24:24.040)
computation, you think maybe that's not going to have any consciousness.
Nick Bostrom (24:28.760)
If on the other hand, the computation is one like our human brains are performing, where
Nick Bostrom (24:36.160)
as part of the computation, there is a global workspace, a sophisticated attention mechanism,
Nick Bostrom (24:43.440)
there is self representations of other cognitive processes and a whole lot of other things
Lex Fridman (24:50.280)
that possibly would be conscious.
Lex Fridman (24:52.920)
And in fact, if it's exactly like ours, I think definitely it would.
Lex Fridman (24:56.680)
But exactly how much less than the full computation that the human brain is performing would be
Nick Bostrom (25:02.880)
required is a little bit, I think, of an open question.
Nick Bostrom (25:09.320)
He asked another interesting question as well, which is, would it be sufficient to just have
Nick Bostrom (25:17.640)
say the brain or would you need the environment in order to generate the same kind of experiences
Lex Fridman (25:24.840)
that we have?
Lex Fridman (25:26.600)
And there is a bunch of stuff we don't know.
Nick Bostrom (25:29.200)
I mean, if you look at, say, current virtual reality environments, one thing that's clear
Nick Bostrom (25:35.440)
is that we don't have to simulate all details of them all the time in order for, say, the
Nick Bostrom (25:40.800)
human player to have the perception that there is a full reality and that you can have say
Nick Bostrom (25:47.000)
procedurally generated where you might only render a scene when it's actually within the
Lex Fridman (25:51.160)
view of the player character.
Lex Fridman (25:55.440)
And so similarly, if this environment that we perceive is simulated, it might be that
Lex Fridman (26:06.440)
all of the parts that come into our view are rendered at any given time.
Lex Fridman (26:10.520)
And a lot of aspects that never come into view, say the details of this microphone I'm
Nick Bostrom (26:16.680)
talking into, exactly what each atom is doing at any given point in time, might not be part
Nick Bostrom (26:23.160)
of the simulation, only a more coarse grained representation.
Lex Fridman (26:27.200)
So that to me is actually from an engineering perspective, why the simulation hypothesis
Nick Bostrom (26:31.560)
is really interesting to think about is how difficult is it to fake sort of in a virtual
Nick Bostrom (26:39.960)
reality context, I don't know if fake is the right word, but to construct a reality that
Nick Bostrom (26:45.000)
is sufficiently real to us to be immersive in the way that the physical world is.
Nick Bostrom (26:52.080)
I think that's actually probably an answerable question of psychology, of computer science,
Nick Bostrom (26:59.400)
of how, where's the line where it becomes so immersive that you don't want to leave
Lex Fridman (27:06.600)
that world?
Nick Bostrom (27:07.600)
Yeah, or that you don't realize while you're in it that it is a virtual world.
Nick Bostrom (27:13.080)
Yeah, those are two actually questions, yours is the more sort of the good question about
Nick Bostrom (27:17.760)
the realism, but mine, from my perspective, what's interesting is it doesn't have to be
Lex Fridman (27:23.120)
real, but how can we construct a world that we wouldn't want to leave?
Nick Bostrom (27:29.480)
Yeah, I mean, I think that might be too low a bar, I mean, if you think, say when people
Nick Bostrom (27:34.440)
first had pong or something like that, I'm sure there were people who wanted to keep
Nick Bostrom (27:38.600)
playing it for a long time because it was fun and they wanted to be in this little world.
Nick Bostrom (27:44.720)
I'm not sure we would say it's immersive, I mean, I guess in some sense it is, but like
Nick Bostrom (27:48.680)
an absorbing activity doesn't even have to be.
Lex Fridman (27:51.360)
But they left that world though, that's the thing.
Lex Fridman (27:54.240)
So like, I think that bar is deceivingly high.
Lex Fridman (27:59.000)
So they eventually left, so you can play pong or Starcraft or whatever more sophisticated
Nick Bostrom (28:05.600)
games for hours, for months, you know, while the work has to be in a big addiction, but
Lex Fridman (28:12.360)
eventually they escaped that.
Lex Fridman (28:13.880)
So you mean when it's absorbing enough that you would spend your entire, you would choose
Lex Fridman (28:19.560)
to spend your entire life in there.
Lex Fridman (28:21.160)
And then thereby changing the concept of what reality is, because your reality becomes the
Lex Fridman (28:28.520)
game.
Nick Bostrom (28:29.620)
Not because you're fooled, but because you've made that choice.
Nick Bostrom (28:33.320)
Yeah, and it made, different people might have different preferences regarding that.
Nick Bostrom (28:38.880)
Some might, even if you had any perfect virtual reality, might still prefer not to spend the
Lex Fridman (28:47.640)
rest of their lives there.
Nick Bostrom (28:49.160)
I mean, in philosophy, there's this experience machine, thought experiment.
Lex Fridman (28:53.880)
Have you come across this?
Lex Fridman (28:55.920)
So Robert Nozick had this thought experiment where you imagine some crazy super duper neuroscientist
Nick Bostrom (29:03.960)
of the future have created a machine that could give you any experience you want if
Nick Bostrom (29:08.160)
you step in there.
Lex Fridman (29:10.280)
And for the rest of your life, you can kind of pre programmed it in different ways.
Lex Fridman (29:15.840)
So your fun dreams could come true, you could, whatever you dream, you want to be a great
Lex Fridman (29:24.240)
artist, a great lover, like have a wonderful life, all of these things.
Nick Bostrom (29:29.320)
If you step into the experience machine will be your experiences, constantly happy.
Lex Fridman (29:36.400)
But you would kind of disconnect from the rest of reality and you would float there
Nick Bostrom (29:39.120)
in a tank.
Lex Fridman (29:41.760)
And so Nozick thought that most people would choose not to enter the experience machine.
Nick Bostrom (29:48.960)
I mean, many might want to go there for a holiday, but they wouldn't want to have to
Lex Fridman (29:51.880)
check out of existence permanently.
Lex Fridman (29:54.560)
And so he thought that was an argument against certain views of value according to what we
Lex Fridman (30:01.480)
value is a function of what we experience.
Nick Bostrom (30:04.760)
Because in the experience machine, you could have any experience you want, and yet many
Lex Fridman (30:08.560)
people would think that would not be much value.
Lex Fridman (30:12.000)
So therefore, what we value depends on other things than what we experience.
Lex Fridman (30:18.600)
So okay, can you can you take that argument further?
Lex Fridman (30:21.920)
What about the fact that maybe what we value is the up and down of life?
Lex Fridman (30:25.120)
So you could have up and downs in the experience machine, right?
Lex Fridman (30:29.080)
But what can't you have in the experience machine?
Lex Fridman (30:31.080)
Well, I mean, that then becomes an interesting question to explore.
Lex Fridman (30:35.480)
But for example, real connection with other people, if the experience machine is a solo
Lex Fridman (30:40.480)
machine where it's only you, like that's something you wouldn't have there.
Nick Bostrom (30:44.920)
You would have this subjective experience that would be like fake people.
Lex Fridman (30:49.440)
But when if you gave somebody flowers, there wouldn't be anybody there who actually got
Nick Bostrom (30:53.840)
happy.
Lex Fridman (30:54.840)
It would just be a little simulation of somebody smiling.
Lex Fridman (30:58.480)
But the simulation would not be the kind of simulation I'm talking about in the simulation
Nick Bostrom (31:01.480)
argument where the simulated creature is conscious, it would just be a kind of smiley face that
Nick Bostrom (31:06.840)
would look perfectly real to you.
Lex Fridman (31:08.600)
So we're now drawing a distinction between appear to be perfectly real and actually being
Nick Bostrom (31:14.720)
real.
Lex Fridman (31:15.720)
Yeah.
Nick Bostrom (31:16.720)
Um, so that could be one thing, I mean, like a big impact on history, maybe is also something
Lex Fridman (31:22.040)
you won't have if you check into this experience machine.
Lex Fridman (31:25.640)
So some people might actually feel the life I want to have for me is one where I have
Lex Fridman (31:29.880)
a big positive impact on history unfolds.
Lex Fridman (31:35.520)
So you could kind of explore these different possible explanations for why it is you wouldn't
Lex Fridman (31:43.560)
want to go into the experience machine if that's, if that's what you feel.
Lex Fridman (31:48.000)
And one interesting observation regarding this Nozick thought experiment and the conclusions
Lex Fridman (31:53.320)
he wanted to draw from it is how much is a kind of a status quo effect.
Lex Fridman (31:58.760)
So a lot of people might not want to get this on current reality to plug into this dream
Lex Fridman (32:04.800)
machine.
Lex Fridman (32:06.160)
But if they instead were told, well, what you've experienced up to this point was a
Nick Bostrom (32:13.240)
dream now, do you want to disconnect from this and enter the real world when you have
Nick Bostrom (32:20.640)
no idea maybe what the real world is, or maybe you could say, well, you're actually a farmer
Nick Bostrom (32:24.880)
in Peru, growing, you know, peanuts, and you could live for the rest of your life in this
Nick Bostrom (32:32.280)
way, or would you want to continue your dream life as Alex Friedman going around the world
Lex Fridman (32:40.480)
making podcasts and doing research.
Lex Fridman (32:44.080)
So if the status quo was that they were actually in the experience machine, I think a lot of
Nick Bostrom (32:51.320)
people might then prefer to live the life that they are familiar with rather than sort
Nick Bostrom (32:55.440)
of bail out into.
Lex Fridman (32:57.120)
So that's interesting, the change itself, the leap, yeah, so it might not be so much
Nick Bostrom (33:02.600)
the reality itself that we're after.
Lex Fridman (33:04.440)
But it's more that we are maybe involved in certain projects and relationships.
Lex Fridman (33:09.000)
And we have, you know, a self identity and these things that our values are kind of connected
Lex Fridman (33:14.040)
with carrying that forward.
Lex Fridman (33:15.880)
And then whether it's inside a tank or outside a tank in Peru, or whether inside a computer
Nick Bostrom (33:22.760)
outside a computer, that's kind of less important to what we ultimately care about.
Nick Bostrom (33:29.120)
Yeah, but still, so just to linger on it, it is interesting.
Nick Bostrom (33:34.720)
I find maybe people are different, but I find myself quite willing to take the leap to the
Nick Bostrom (33:39.600)
farmer in Peru, especially as the virtual reality system become more realistic.
Lex Fridman (33:46.800)
I find that possibility and I think more people would take that leap.
Lex Fridman (33:50.760)
But so in this thought experiment, just to make sure we are understanding, so in this
Nick Bostrom (33:53.640)
case, the farmer in Peru would not be a virtual reality, that would be the real, your life,
Nick Bostrom (34:01.280)
like before this whole experience machine started.
Nick Bostrom (34:04.400)
Well, I kind of assumed from that description, you're being very specific, but that kind
Nick Bostrom (34:09.560)
of idea just like washes away the concept of what's real.
Nick Bostrom (34:15.320)
I'm still a little hesitant about your kind of distinction between real and illusion.
Nick Bostrom (34:23.320)
Because when you can have an illusion that feels, I mean, that looks real, I don't know
Lex Fridman (34:31.080)
how you can definitively say something is real or not, like what's a good way to prove
Lex Fridman (34:35.320)
that something is real in that context?
Lex Fridman (34:37.600)
Well, so I guess in this case, it's more a stipulation.
Nick Bostrom (34:41.040)
In one case, you're floating in a tank with these wires by the super duper neuroscientists
Lex Fridman (34:47.400)
plugging into your head, giving you like Friedman experiences.
Nick Bostrom (34:52.440)
In the other, you're actually tilling the soil in Peru, growing peanuts, and then those
Nick Bostrom (34:57.120)
peanuts are being eaten by other people all around the world who buy the exports.
Nick Bostrom (35:01.440)
That's two different possible situations in the one and the same real world that you could
Lex Fridman (35:08.600)
choose to occupy.
Lex Fridman (35:09.600)
But just to be clear, when you're in a vat with wires and the neuroscientists, you can
Lex Fridman (35:15.400)
still go farming in Peru, right?
Nick Bostrom (35:19.000)
No, well, if you wanted to, you could have the experience of farming in Peru, but there
Lex Fridman (35:25.120)
wouldn't actually be any peanuts grown.
Lex Fridman (35:28.760)
But what makes a peanut, so a peanut could be grown and you could feed things with that
Lex Fridman (35:36.560)
peanut and why can't all of that be done in a simulation?
Nick Bostrom (35:41.600)
I hope, first of all, that they actually have peanut farms in Peru, I guess we'll get a
Lex Fridman (35:45.760)
lot of comments otherwise from Angrit.
Nick Bostrom (35:50.000)
I was way up to the point when you started talking about Peru peanuts, that's when I
Lex Fridman (35:54.160)
realized you're relying out of these.
Nick Bostrom (35:56.200)
In that climate.
Nick Bostrom (35:57.200)
No, I mean, I think, I mean, in the simulation, I think there is a sense, the important sense
Nick Bostrom (36:05.120)
in which it would all be real.
Nick Bostrom (36:07.200)
Nevertheless, there is a distinction between inside the simulation and outside the simulation.
Nick Bostrom (36:13.720)
Or in the case of Nozick's thought experiment, whether you're in the vat or outside the vat,
Lex Fridman (36:19.680)
and some of those differences may or may not be important.
Nick Bostrom (36:22.440)
I mean, that comes down to your values and preferences.
Lex Fridman (36:25.520)
So if the, if the experience machine only gives you the experience of growing peanuts,
Lex Fridman (36:32.840)
but you're the only one in the experience machines.
Nick Bostrom (36:35.680)
No, but there's other, you can, within the experience machine, others can plug in.
Nick Bostrom (36:40.400)
Well, there are versions of the experience machine.
Lex Fridman (36:43.840)
So in fact, you might want to have, distinguish different thought experiments, different versions
Nick Bostrom (36:47.520)
of it.
Lex Fridman (36:48.520)
I see.
Lex Fridman (36:49.520)
So in, like in the original thought experiment, maybe it's only you, right?
Lex Fridman (36:51.840)
And you think, I wouldn't want to go in there.
Nick Bostrom (36:54.480)
Well, that tells you something interesting about what you value and what you care about.
Nick Bostrom (36:58.200)
Then you could say, well, what if you add the fact that there would be other people
Lex Fridman (37:02.000)
in there and you would interact with them?
Lex Fridman (37:03.440)
Well, it starts to make it more attractive, right?
Nick Bostrom (37:06.920)
Then you could add in, well, what if you could also have important longterm effects on human
Nick Bostrom (37:10.920)
history and the world, and you could actually do something useful, even though you were
Nick Bostrom (37:14.840)
in there.
Lex Fridman (37:15.840)
That makes it maybe even more attractive.
Nick Bostrom (37:17.760)
Like you could actually have a life that had a purpose and consequences.
Lex Fridman (37:22.480)
And so as you sort of add more into it, it becomes more similar to the baseline reality
Nick Bostrom (37:30.760)
that you were comparing it to.
Nick Bostrom (37:32.920)
Yeah, but I just think inside the experience machine and without taking those steps you
Nick Bostrom (37:37.840)
just mentioned, you still have an impact on longterm history of the creatures that live
Nick Bostrom (37:45.720)
inside that, of the quote unquote fake creatures that live inside that experience machine.
Lex Fridman (37:53.360)
And that, like at a certain point, you know, if there's a person waiting for you inside
Nick Bostrom (37:59.800)
that experience machine, maybe your newly found wife and she dies, she has fear, she
Nick Bostrom (38:06.920)
has hopes, and she exists in that machine when you plug out, when you unplug yourself
Lex Fridman (38:12.900)
and plug back in, she's still there going on about her life.
Nick Bostrom (38:16.080)
Well, in that case, yeah, she starts to have more of an independent existence.
Lex Fridman (38:20.640)
Independent existence.
Lex Fridman (38:21.640)
But it depends, I think, on how she's implemented in the experience machine.
Nick Bostrom (38:26.680)
Take one limit case where all she is is a static picture on the wall, a photograph.
Lex Fridman (38:32.480)
So you think, well, I can look at her, right?
Lex Fridman (38:36.060)
But that's it.
Nick Bostrom (38:37.240)
There's no...
Nick Bostrom (38:38.240)
Then you think, well, it doesn't really matter much what happens to that, any more than a
Lex Fridman (38:41.960)
normal photograph if you tear it up, right?
Nick Bostrom (38:45.080)
It means you can't see it anymore, but you haven't harmed the person whose picture you
Nick Bostrom (38:49.300)
tore up.
Lex Fridman (38:52.300)
But if she's actually implemented, say, at a neural level of detail so that she's a fully
Nick Bostrom (38:58.120)
realized digital mind with the same behavioral repertoire as you have, then very plausibly
Lex Fridman (39:06.120)
she would be a conscious person like you are.
Lex Fridman (39:09.240)
And then what you do in this experience machine would have real consequences for how this
Lex Fridman (39:14.220)
other mind felt.
Lex Fridman (39:17.680)
So you have to specify which of these experience machines you're talking about.
Nick Bostrom (39:21.100)
I think it's not entirely obvious that it would be possible to have an experience machine
Nick Bostrom (39:27.920)
that gave you a normal set of human experiences, which include experiences of interacting with
Nick Bostrom (39:34.240)
other people, without that also generating consciousnesses corresponding to those other
Nick Bostrom (39:40.560)
people.
Nick Bostrom (39:41.560)
That is, if you create another entity that you perceive and interact with, that to you
Nick Bostrom (39:47.480)
looks entirely realistic.
Nick Bostrom (39:49.320)
Not just when you say hello, they say hello back, but you have a rich interaction, many
Nick Bostrom (39:53.160)
days, deep conversations.
Nick Bostrom (39:54.840)
It might be that the only possible way of implementing that would be one that also has
Nick Bostrom (40:00.960)
a side effect, instantiated this other person in enough detail that you would have a second
Lex Fridman (40:06.600)
consciousness there.
Nick Bostrom (40:07.680)
I think that's to some extent an open question.
Lex Fridman (40:11.800)
So you don't think it's possible to fake consciousness and fake intelligence?
Nick Bostrom (40:15.040)
Well, it might be.
Nick Bostrom (40:16.040)
I mean, I think you can certainly fake, if you have a very limited interaction with somebody,
Nick Bostrom (40:21.340)
you could certainly fake that.
Nick Bostrom (40:24.320)
If all you have to go on is somebody said hello to you, that's not enough for you to
Nick Bostrom (40:28.320)
tell whether that was a real person there, or a prerecorded message, or a very superficial
Lex Fridman (40:34.880)
simulation that has no consciousness, because that's something easy to fake.
Nick Bostrom (40:39.280)
We could already fake it, now you can record a voice recording.
Lex Fridman (40:43.720)
But if you have a richer set of interactions where you're allowed to ask open ended questions
Lex Fridman (40:49.160)
and probe from different angles, you couldn't give canned answer to all of the possible
Nick Bostrom (40:54.920)
ways that you could probe it, then it starts to become more plausible that the only way
Nick Bostrom (41:00.280)
to realize this thing in such a way that you would get the right answer from any which
Nick Bostrom (41:05.160)
angle you probed it, would be a way of instantiating it, where you also instantiated a conscious
Nick Bostrom (41:10.040)
mind.
Nick Bostrom (41:11.040)
Yeah, I'm with you on the intelligence part, but is there something about me that says
Lex Fridman (41:13.960)
consciousness is easier to fake?
Lex Fridman (41:15.960)
Like I've recently gotten my hands on a lot of rubas, don't ask me why or how.
Lex Fridman (41:23.080)
And I've made them, there's just a nice robotic mobile platform for experiments.
Lex Fridman (41:28.540)
And I made them scream and or moan in pain, so on, just to see when they're responding
Nick Bostrom (41:34.560)
to me.
Lex Fridman (41:35.560)
And it's just a sort of psychological experiment on myself.
Lex Fridman (41:39.240)
And I think they appear conscious to me pretty quickly.
Lex Fridman (41:43.120)
To me, at least my brain can be tricked quite easily.
Nick Bostrom (41:46.720)
I said if I introspect, it's harder for me to be tricked that something is intelligent.
Lex Fridman (41:53.760)
So I just have this feeling that inside this experience machine, just saying that you're
Nick Bostrom (41:58.860)
conscious and having certain qualities of the interaction, like being able to suffer,
Nick Bostrom (42:05.000)
like being able to hurt, like being able to wander about the essence of your own existence,
Nick Bostrom (42:12.040)
not actually, I mean, creating the illusion that you're wandering about it is enough to
Lex Fridman (42:18.040)
create the illusion of consciousness.
Lex Fridman (42:23.120)
And because of that, create a really immersive experience to where you feel like that is
Lex Fridman (42:27.440)
the real world.
Lex Fridman (42:28.440)
So you think there's a big gap between appearing conscious and being conscious?
Lex Fridman (42:33.260)
Or is it that you think it's very easy to be conscious?
Nick Bostrom (42:36.080)
I'm not actually sure what it means to be conscious.
Nick Bostrom (42:38.120)
All I'm saying is the illusion of consciousness is enough to create a social interaction that's
Nick Bostrom (42:48.200)
as good as if the thing was conscious, meaning I'm making it about myself.
Lex Fridman (42:52.480)
Right.
Nick Bostrom (42:53.480)
Yeah.
Lex Fridman (42:54.480)
I mean, I guess there are a few different things.
Nick Bostrom (42:55.480)
One is how good the interaction is, which might, I mean, if you don't really care about
Nick Bostrom (42:59.740)
like probing hard for whether the thing is conscious, maybe it would be a satisfactory
Nick Bostrom (43:05.080)
interaction, whether or not you really thought it was conscious.
Nick Bostrom (43:10.720)
Now, if you really care about it being conscious in like inside this experience machine, how
Lex Fridman (43:20.040)
easy would it be to fake it?
Lex Fridman (43:22.340)
And you say, it sounds fairly easy, but then the question is, would that also mean it's
Lex Fridman (43:28.000)
very easy to instantiate consciousness?
Nick Bostrom (43:30.600)
Like it's much more widely spread in the world and we have thought it doesn't require a big
Nick Bostrom (43:35.440)
human brain with a hundred billion neurons, all you need is some system that exhibits
Lex Fridman (43:39.600)
basic intentionality and can respond and you already have consciousness.
Nick Bostrom (43:43.300)
Like in that case, I guess you still have a close coupling.
Nick Bostrom (43:49.080)
I guess that case would be where they can come apart, where you could create the appearance
Nick Bostrom (43:54.600)
of there being a conscious mind with actually not being another conscious mind.
Lex Fridman (43:59.200)
I'm somewhat agnostic exactly where these lines go.
Nick Bostrom (44:03.320)
I think one observation that makes it plausible that you could have very realistic appearances
Nick Bostrom (44:12.280)
relatively simply, which also is relevant for the simulation argument and in terms of
Nick Bostrom (44:18.320)
thinking about how realistic would a virtual reality model have to be in order for the
Lex Fridman (44:24.400)
simulated creature not to notice that anything was awry.
Nick Bostrom (44:27.960)
Well, just think of our own humble brains during the wee hours of the night when we
Lex Fridman (44:33.960)
are dreaming.
Nick Bostrom (44:35.400)
Many times, well, dreams are very immersive, but often you also don't realize that you're
Lex Fridman (44:40.560)
in a dream.
Lex Fridman (44:43.440)
And that's produced by simple primitive three pound lumps of neural matter effortlessly.
Lex Fridman (44:51.320)
So if a simple brain like this can create the virtual reality that seems pretty real
Nick Bostrom (44:57.160)
to us, then how much easier would it be for a super intelligent civilization with planetary
Nick Bostrom (45:03.120)
sized computers optimized over the eons to create a realistic environment for you to
Lex Fridman (45:09.760)
interact with?
Lex Fridman (45:10.760)
Yeah.
Nick Bostrom (45:11.760)
By the way, behind that intuition is that our brain is not that impressive relative
Lex Fridman (45:17.720)
to the possibilities of what technology could bring.
Nick Bostrom (45:21.280)
It's also possible that the brain is the epitome, is the ceiling.
Lex Fridman (45:26.820)
How is that possible?
Nick Bostrom (45:30.960)
Meaning like this is the smartest possible thing that the universe could create.
Lex Fridman (45:36.240)
So that seems unlikely to me.
Nick Bostrom (45:39.800)
Yeah.
Nick Bostrom (45:40.800)
I mean, for some of these reasons we alluded to earlier in terms of designs we already
Nick Bostrom (45:47.600)
have for computers that would be faster by many orders of magnitude than the human brain.
Lex Fridman (45:54.960)
Yeah.
Nick Bostrom (45:55.960)
We can see that the constraints, the cognitive constraints in themselves is what enables
Lex Fridman (46:01.120)
the intelligence.
Lex Fridman (46:02.440)
So the more powerful you make the computer, the less likely it is to become super intelligent.
Lex Fridman (46:09.420)
This is where I say dumb things to push back on that statement.
Nick Bostrom (46:12.120)
Yeah.
Lex Fridman (46:13.120)
I'm not sure I thought that we might.
Nick Bostrom (46:14.120)
No.
Lex Fridman (46:15.120)
I mean, so there are different dimensions of intelligence.
Nick Bostrom (46:18.120)
A simple one is just speed.
Nick Bostrom (46:20.220)
Like if you can solve the same challenge faster in some sense, you're like smarter.
Lex Fridman (46:25.360)
So there I think we have very strong evidence for thinking that you could have a computer
Nick Bostrom (46:31.560)
in this universe that would be much faster than the human brain and therefore have speed
Nick Bostrom (46:37.880)
super intelligence, like be completely superior, maybe a million times faster.
Nick Bostrom (46:42.920)
Then maybe there are other ways in which you could be smarter as well, maybe more qualitative
Lex Fridman (46:46.960)
ways, right?
Lex Fridman (46:48.680)
And the concepts are a little bit less clear cut.
Lex Fridman (46:51.840)
So it's harder to make a very crisp, neat, firmly logical argument for why that could
Lex Fridman (46:59.640)
be qualitative super intelligence as opposed to just things that were faster.
Nick Bostrom (47:03.240)
Although I still think it's very plausible and for various reasons that are less than
Lex Fridman (47:08.040)
watertight arguments.
Lex Fridman (47:09.240)
But when you can sort of, for example, if you look at animals and even within humans,
Nick Bostrom (47:14.680)
like there seems to be like Einstein versus random person, like it's not just that Einstein
Nick Bostrom (47:19.760)
was a little bit faster, but like how long would it take a normal person to invent general
Nick Bostrom (47:25.000)
relativity is like, it's not 20% longer than it took Einstein or something like that.
Nick Bostrom (47:30.080)
It's like, I don't know whether they would do it at all or it would take millions of
Lex Fridman (47:32.920)
years or some totally bizarre.
Lex Fridman (47:37.320)
But your intuition is that the compute size will get you go increasing the size of the
Nick Bostrom (47:42.600)
computer and the speed of the computer might create some much more powerful levels of intelligence
Nick Bostrom (47:49.560)
that would enable some of the things we've been talking about with like the simulation,
Nick Bostrom (47:53.560)
being able to simulate an ultra realistic environment, ultra realistic perception of
Nick Bostrom (48:00.760)
reality.
Lex Fridman (48:01.760)
Yeah.
Nick Bostrom (48:02.760)
I mean, strictly speaking, it would not be necessary to have super intelligence in order
Nick Bostrom (48:05.720)
to have say the technology to make these simulations, ancestor simulations or other kinds of simulations.
Nick Bostrom (48:14.280)
As a matter of fact, I think if we are in a simulation, it would most likely be one
Lex Fridman (48:20.800)
built by a civilization that had super intelligence.
Nick Bostrom (48:26.280)
It certainly would help a lot.
Nick Bostrom (48:27.560)
I mean, you could build more efficient larger scale structures if you had super intelligence.
Nick Bostrom (48:31.400)
I also think that if you had the technology to build these simulations, that's like a
Lex Fridman (48:34.960)
very advanced technology.
Nick Bostrom (48:35.960)
It seems kind of easier to get the technology to super intelligence.
Nick Bostrom (48:40.520)
I'd expect by the time they could make these fully realistic simulations of human history
Nick Bostrom (48:45.280)
with human brains in there, like before that they got to that stage, they would have figured
Nick Bostrom (48:49.160)
out how to create machine super intelligence or maybe biological enhancements of their
Nick Bostrom (48:55.520)
own brains if there were biological creatures to start with.
Lex Fridman (48:59.200)
So we talked about the three parts of the simulation argument.
Nick Bostrom (49:04.240)
One, we destroy ourselves before we ever create the simulation.
Lex Fridman (49:08.480)
Two, we somehow, everybody somehow loses interest in creating the simulation.
Nick Bostrom (49:13.200)
Three, we're living in a simulation.
Lex Fridman (49:16.280)
So you've kind of, I don't know if your thinking has evolved on this point, but you kind of
Nick Bostrom (49:21.760)
said that we know so little that these three cases might as well be equally probable.
Lex Fridman (49:28.320)
So probabilistically speaking, where do you stand on this?
Nick Bostrom (49:31.720)
Yeah, I mean, I don't think equal necessarily would be the most supported probability assignment.
Lex Fridman (49:41.280)
So how would you, without assigning actual numbers, what's more or less likely in your
Lex Fridman (49:47.280)
view?
Nick Bostrom (49:48.280)
Well, I mean, I've historically tended to punt on the question of like between these
Nick Bostrom (49:54.600)
three.
Lex Fridman (49:55.600)
So maybe you ask me another way is which kind of things would make each of these more or
Lex Fridman (50:01.640)
less likely?
Lex Fridman (50:03.440)
What kind of intuition?
Nick Bostrom (50:05.200)
Certainly in general terms, if you think anything that say increases or reduces the probability
Lex Fridman (50:10.960)
of one of these, we tend to slosh probability around on the other.
Lex Fridman (50:17.040)
So if one becomes less probable, like the other would have to, cause it's got to add
Lex Fridman (50:20.600)
up to one.
Lex Fridman (50:22.000)
So if we consider the first hypothesis, the first alternative that there's this filter
Nick Bostrom (50:28.960)
that makes it so that virtually no civilization reaches technological maturity, in particular
Nick Bostrom (50:39.160)
our own civilization, if that's true, then it's like very unlikely that we would reach
Nick Bostrom (50:42.440)
technological maturity because if almost no civilization at our stage does it, then it's
Nick Bostrom (50:47.600)
unlikely that we do it.
Lex Fridman (50:49.120)
So hence...
Lex Fridman (50:50.120)
Sorry, can you linger on that for a second?
Nick Bostrom (50:51.120)
Well, so if it's the case that almost all civilizations at our current stage of technological
Nick Bostrom (50:59.000)
development failed to reach maturity, that would give us very strong reason for thinking
Lex Fridman (51:05.280)
we will fail to reach technological maturity.
Nick Bostrom (51:07.480)
Oh, and also sort of the flip side of that is the fact that we've reached it means that
Lex Fridman (51:12.000)
many other civilizations have reached this point.
Nick Bostrom (51:13.680)
Yeah.
Lex Fridman (51:14.680)
So that means if we get closer and closer to actually reaching technological maturity,
Nick Bostrom (51:20.200)
there's less and less distance left where we could go extinct before we are there, and
Nick Bostrom (51:26.200)
therefore the probability that we will reach increases as we get closer, and that would
Nick Bostrom (51:31.520)
make it less likely to be true that almost all civilizations at our current stage failed
Lex Fridman (51:36.120)
to get there.
Nick Bostrom (51:37.120)
Like we would have this...
Nick Bostrom (51:38.880)
The one case we had started ourselves would be very close to getting there, that would
Nick Bostrom (51:42.960)
be strong evidence that it's not so hard to get to technological maturity.
Lex Fridman (51:46.440)
So to the extent that we feel we are moving nearer to technological maturity, that would
Nick Bostrom (51:52.800)
tend to reduce the probability of the first alternative and increase the probability of
Lex Fridman (51:58.000)
the other two.
Nick Bostrom (51:59.600)
It doesn't need to be a monotonic change.
Nick Bostrom (52:01.960)
Like if every once in a while some new threat comes into view, some bad new thing you could
Nick Bostrom (52:07.440)
do with some novel technology, for example, that could change our probabilities in the
Lex Fridman (52:13.460)
other direction.
Lex Fridman (52:15.160)
But that technology, again, you have to think about as that technology has to be able to
Lex Fridman (52:20.840)
equally in an even way affect every civilization out there.
Nick Bostrom (52:26.400)
Yeah, pretty much.
Lex Fridman (52:28.120)
I mean, that's strictly speaking, it's not true.
Nick Bostrom (52:30.760)
I mean, that could be two different existential risks and every civilization, you know, one
Lex Fridman (52:36.800)
or the other, like, but none of them kills more than 50%.
Lex Fridman (52:42.440)
But incidentally, so in some of my work, I mean, on machine superintelligence, like pointed
Nick Bostrom (52:50.240)
to some existential risks related to sort of super intelligent AI and how we must make
Nick Bostrom (52:54.440)
sure, you know, to handle that wisely and carefully.
Nick Bostrom (52:59.820)
It's not the right kind of existential catastrophe to make the first alternative true though.
Nick Bostrom (53:09.880)
Like it might be bad for us if the future lost a lot of value as a result of it being
Lex Fridman (53:15.480)
shaped by some process that optimized for some completely nonhuman value.
Lex Fridman (53:21.160)
But even if we got killed by machine superintelligence, that machine superintelligence might still
Lex Fridman (53:27.920)
attain technological maturity.
Nick Bostrom (53:29.520)
Oh, I see, so you're not human exclusive.
Nick Bostrom (53:33.560)
This could be any intelligent species that achieves, like it's all about the technological
Nick Bostrom (53:38.360)
maturity.
Lex Fridman (53:39.360)
But the humans have to attain it.
Nick Bostrom (53:43.040)
Right.
Lex Fridman (53:44.040)
So like superintelligence could replace us and that's just as well for the simulation
Nick Bostrom (53:47.320)
argument.
Lex Fridman (53:48.320)
Yeah, yeah.
Nick Bostrom (53:49.320)
I mean, it could interact with the second hypothesis by alternative.
Nick Bostrom (53:51.800)
Like if the thing that replaced us was either more likely or less likely than we would be
Nick Bostrom (53:57.120)
to have an interest in creating ancestor simulations, you know, that could affect probabilities.
Lex Fridman (54:02.840)
But yeah, to a first order, like if we all just die, then yeah, we won't produce any
Nick Bostrom (54:09.840)
simulations because we are dead.
Lex Fridman (54:11.920)
But if we all die and get replaced by some other intelligent thing that then gets to
Nick Bostrom (54:17.560)
technological maturity, the question remains, of course, if not that thing, then use some
Lex Fridman (54:21.680)
of its resources to do this stuff.
Lex Fridman (54:25.280)
So can you reason about this stuff, given how little we know about the universe?
Lex Fridman (54:30.760)
Is it reasonable to reason about these probabilities?
Lex Fridman (54:36.760)
So like how little, well, maybe you can disagree, but to me, it's not trivial to figure out
Lex Fridman (54:45.200)
how difficult it is to build a simulation.
Nick Bostrom (54:47.520)
We kind of talked about it a little bit.
Nick Bostrom (54:49.640)
We also don't know, like as we try to start building it, like start creating virtual worlds
Lex Fridman (54:56.080)
and so on, how that changes the fabric of society.
Nick Bostrom (54:59.640)
Like there's all these things along the way that can fundamentally change just so many
Nick Bostrom (55:04.560)
aspects of our society about our existence that we don't know anything about, like the
Nick Bostrom (55:09.480)
kind of things we might discover when we understand to a greater degree the fundamental, the physics,
Nick Bostrom (55:19.380)
like the theory, if we have a breakthrough, have a theory and everything, how that changes
Lex Fridman (55:23.360)
stuff, how that changes deep space exploration and so on.
Lex Fridman (55:27.600)
Like, is it still possible to reason about probabilities given how little we know?
Nick Bostrom (55:33.040)
Yes, I think there will be a large residual of uncertainty that we'll just have to acknowledge.
Lex Fridman (55:41.960)
And I think that's true for most of these big picture questions that we might wonder
Lex Fridman (55:47.880)
about.
Nick Bostrom (55:49.840)
It's just we are small, short lived, small brained, cognitively very limited humans with
Lex Fridman (55:57.840)
little evidence.
Lex Fridman (55:59.520)
And it's amazing we can figure out as much as we can really about the cosmos.
Lex Fridman (56:04.760)
But okay, so there's this cognitive trick that seems to happen when I look at the simulation
Nick Bostrom (56:10.960)
argument, which for me, it seems like case one and two feel unlikely.
Nick Bostrom (56:16.360)
I want to say feel unlikely as opposed to sort of like, it's not like I have too much
Nick Bostrom (56:22.080)
scientific evidence to say that either one or two are not true.
Lex Fridman (56:26.980)
It just seems unlikely that every single civilization destroys itself.
Lex Fridman (56:32.400)
And it seems like feels unlikely that the civilizations lose interest.
Lex Fridman (56:37.160)
So naturally, without necessarily explicitly doing it, but the simulation argument basically
Nick Bostrom (56:44.660)
says it's very likely we're living in a simulation.
Lex Fridman (56:49.080)
To me, my mind naturally goes there.
Nick Bostrom (56:51.800)
I think the mind goes there for a lot of people.
Lex Fridman (56:54.860)
Is that the incorrect place for it to go?
Nick Bostrom (56:57.400)
Well, not necessarily.
Nick Bostrom (56:59.160)
I think the second alternative, which has to do with the motivations and interests of
Nick Bostrom (57:09.040)
technological and material civilizations, I think there is much we don't understand about
Lex Fridman (57:15.160)
that.
Lex Fridman (57:16.160)
Can you talk about that a little bit?
Lex Fridman (57:18.440)
What do you think?
Nick Bostrom (57:19.440)
I mean, this is a question that pops up when you when you build an AGI system or build
Lex Fridman (57:22.940)
a general intelligence.
Lex Fridman (57:26.320)
How does that change our motivations?
Lex Fridman (57:27.880)
Do you think it'll fundamentally transform our motivations?
Nick Bostrom (57:30.800)
Well, it doesn't seem that implausible that once you take this leap to to technological
Nick Bostrom (57:39.280)
maturity, I mean, I think like it involves creating machine super intelligence, possibly
Nick Bostrom (57:44.920)
that would be sort of on the path for basically all civilizations, maybe before they are able
Nick Bostrom (57:50.840)
to create large numbers of ancestry simulations, they would that that possibly could be one
Nick Bostrom (57:55.840)
of these things that quite radically changes the orientation of what a civilization is,
Lex Fridman (58:03.240)
in fact, optimizing for.
Nick Bostrom (58:06.400)
There are other things as well.
Lex Fridman (58:08.760)
So at the moment, we have not perfect control over our own being our own mental states,
Nick Bostrom (58:20.240)
our own experiences are not under our direct control.
Lex Fridman (58:25.240)
So for example, if if you want to experience a pleasure and happiness, you might have to
Nick Bostrom (58:33.840)
do a whole host of things in the external world to try to get into the stage into the
Nick Bostrom (58:39.720)
mental state where you experience pleasure, like some people get some pleasure from eating
Nick Bostrom (58:44.240)
great food.
Nick Bostrom (58:45.240)
Well, they can just turn that on, they have to kind of actually go to a nice restaurant
Lex Fridman (58:49.960)
and then they have to make money.
Lex Fridman (58:51.400)
So there's like all this kind of activity that maybe arises from the fact that we are
Nick Bostrom (58:58.120)
trying to ultimately produce mental states.
Lex Fridman (59:02.120)
But the only way to do that is by a whole host of complicated activities in the external
Nick Bostrom (59:06.560)
world.
Nick Bostrom (59:07.560)
Now, at some level of technological development, I think we'll become auto potent in the sense
Nick Bostrom (59:11.560)
of gaining direct ability to choose our own internal configuration, and enough knowledge
Lex Fridman (59:18.920)
and insight to be able to actually do that in a meaningful way.
Lex Fridman (59:22.840)
So then it could turn out that there are a lot of instrumental goals that would drop
Nick Bostrom (59:28.160)
out of the picture and be replaced by other instrumental goals, because we could now serve
Nick Bostrom (59:33.420)
some of these final goals in more direct ways.
Lex Fridman (59:37.200)
And who knows how all of that shakes out after civilizations reflect on that and converge
Nick Bostrom (59:45.500)
on different attractors and so on and so forth.
Lex Fridman (59:49.880)
And that could be new instrumental considerations that come into view as well, that we are just
Nick Bostrom (59:57.040)
oblivious to, that would maybe have a strong shaping effect on actions, like very strong
🔗 相关节目
Joscha Bach: Nature of Reality, Dreams, and Consciousness
Joscha Bach
Joscha Bach: Artificial Consciousness and the Nature of Reality
Joscha Bach
Wojciech Zaremba: OpenAI Codex, GPT-3, Robotics, and the Future of AI
Wojciech Zaremba
Alex Garland: Ex Machina, Devs, Annihilation, and the Poetry of Science
Alex Garland