Sertac Karaman: Robots That Fly and Robots That Drive

Lex Fridman (02:02.520)

And so I think that that's maybe one of the reasons why it took off literally a little

Sertac Karaman (02:06.940)

earlier than the autonomous cars.

Lex Fridman (02:09.160)

But I think if you look ahead, I would think that the real benefits of autonomous flying,

Sertac Karaman (02:14.520)

unleashing them in transportation, logistics, and so on, I think it's a lot harder than

Lex Fridman (02:18.520)

autonomous driving.

Lex Fridman (02:19.520)

So I think my guess is that we've seen a few kind of machines fly here and there, but we

Sertac Karaman (02:24.680)

really haven't yet seen any kind of machine, like at massive scale, large scale being deployed

Lex Fridman (02:32.040)

and flown and so on.

Lex Fridman (02:33.160)

And I think that's going to be after we kind of resolve some of the large scale deployments

Sertac Karaman (02:38.080)

of autonomous driving.

Lex Fridman (02:39.080)

So what's the hard part?

Lex Fridman (02:40.920)

What's your intuition behind why at scale, when consumer facing drones are tough?

Lex Fridman (02:47.700)

So I think in general, at scale is tough.

Sertac Karaman (02:51.800)

Like for example, when you think about it, we have actually deployed a lot of robots

Lex Fridman (02:57.080)

in the, let's say the past 50 years.

Lex Fridman (03:00.000)

We as academics or we business entrepreneurs?

Lex Fridman (03:02.680)

I think we as humanity.

Lex Fridman (03:03.680)

Humanity?

Lex Fridman (03:04.680)

A lot of people working on it.

Lex Fridman (03:05.680)

So we humans deployed a lot of robots.

Lex Fridman (03:09.500)

And I think that, well, when you think about it, you know, robots, they're autonomous.

Sertac Karaman (03:14.680)

They work and they work on their own, but they are either like in isolated environments

Sertac Karaman (03:19.920)

or they are in sort of, you know, they may be at scale, but they're really confined to

Sertac Karaman (03:27.520)

a certain environment that they don't interact so much with humans.

Lex Fridman (03:30.760)

And so, you know, they work in, I don't know, factory floors, warehouses, they work on Mars,

Sertac Karaman (03:35.520)

you know, they are fully autonomous over there.

Lex Fridman (03:38.360)

But I think that the real challenge of our time is to take these vehicles and put them

Sertac Karaman (03:44.200)

into places where humans are present.

Lex Fridman (03:47.160)

So now I know that there's a lot of like human robot interaction type of things that need

Sertac Karaman (03:51.480)

to be done.

Lex Fridman (03:52.480)

And so that's one thing, but even just from the fundamental algorithms and systems and

Sertac Karaman (03:58.080)

the business cases, or maybe the business models, even like architecture, planning,

Sertac Karaman (04:03.040)

societal issues, legal issues, there's a whole bunch of pack of things that are related to

Sertac Karaman (04:08.120)

us putting robotic vehicles into human present environments.

Lex Fridman (04:12.480)

And as humans, you know, they will not potentially be even trained to interact with them.

Sertac Karaman (04:18.400)

They may not even be using the services that are provided by these vehicles.

Lex Fridman (04:21.880)

They may not even know that they're autonomous.

Sertac Karaman (04:23.760)

They're just doing their thing, living in environments that are designed for humans,

Lex Fridman (04:27.720)

not for robots.

Lex Fridman (04:28.980)

And that I think is one of the biggest challenges, I think, of our time to put vehicles there.

Lex Fridman (04:35.320)

And you know, to go back to your question, I think doing that at scale, meaning, you

Sertac Karaman (04:40.520)

know, you go out in a city and you have, you know, like thousands or tens of thousands

Lex Fridman (04:46.560)

of autonomous vehicles that are going around.

Sertac Karaman (04:48.440)

It is so dense to the point where if you see one of them, you look around, you see another

Lex Fridman (04:53.840)

one.

Sertac Karaman (04:54.840)

It is that dense.

Lex Fridman (04:55.840)

And that density, we've never done anything like that before.

Lex Fridman (04:59.460)

And I would bet that that kind of density will first happen with autonomous cars because

Lex Fridman (05:05.340)

I think, you know, we can bend the environment a little bit.

Sertac Karaman (05:08.180)

We can, especially kind of making them safe is a lot easier when they're like on the ground.

Lex Fridman (05:15.160)

When they're in the air, it's a little bit more complicated.

Lex Fridman (05:19.520)

But I don't see that there's going to be a big separation.

Sertac Karaman (05:21.560)

I think that, you know, there will come a time that we're going to quickly see these

Sertac Karaman (05:24.500)

things unfold.

Lex Fridman (05:25.500)

Do you think there will be a time where there's tens of thousands of delivery drones that

Lex Fridman (05:30.480)

fill the sky?

Lex Fridman (05:31.480)

You know, I think, I think it's possible to be honest.

Sertac Karaman (05:34.120)

Delivery drones is one thing, but you know, you can imagine for transportation, like an

Sertac Karaman (05:38.680)

important use case is, you know, we're in Boston, you want to go from Boston to New

Sertac Karaman (05:42.560)

York and you want to do it from the top of this building to the top of another building

Lex Fridman (05:47.300)

in Manhattan.

Lex Fridman (05:48.520)

And you're going to do it in one and a half hours.

Lex Fridman (05:50.960)

And that's, that's a big opportunity, I think.

Sertac Karaman (05:53.800)

Personal transport.

Lex Fridman (05:54.800)

So like you and me be a friend, like almost like an Uber.

Lex Fridman (05:58.660)

So like four people, six people, eight people.

Lex Fridman (06:01.720)

In our work in autonomous vehicles, I see that.

Lex Fridman (06:04.000)

So there's kind of like a bit of a need for, you know, one person transport, but also like,

Lex Fridman (06:08.040)

like a few people.

Lex Fridman (06:09.120)

So you and I could take that trip together.

Sertac Karaman (06:10.880)

We could have lunch, you know, I think kind of sounds crazy, maybe even sounds a bit cheesy,

Lex Fridman (06:16.840)

but I think that those kinds of things are some of the real opportunities.

Lex Fridman (06:20.440)

And I think, you know it's not like the typical airplane and the airport would disappear very

Sertac Karaman (06:26.080)

quickly, but I would think that, you know many people would feel like they would spend

Sertac Karaman (06:31.520)

an extra hundred dollars on doing that and cutting that four hour travel down to one

Lex Fridman (06:36.640)

and a half hours.

Lex Fridman (06:37.640)

So how feasible are flying cars has been the dream.

Sertac Karaman (06:41.400)

That's like when people imagine the future for 50 plus years, they think flying cars,

Lex Fridman (06:46.240)

it's a, it's like all technologies.

Sertac Karaman (06:49.000)

It's cheesy to think about now because it seems so far away, but overnight it can change.

Lex Fridman (06:54.800)

But just technically speaking in your view, how feasible is it to make that happen?

Sertac Karaman (06:59.200)

I'll get to that question, but just one thing is that I think, you know, sometimes we think

Lex Fridman (07:03.680)

about what's going to happen in the next 50 years.

Lex Fridman (07:07.240)

It's just really hard to guess, right?

Lex Fridman (07:08.840)

Next 50 years.

Sertac Karaman (07:09.840)

I don't know.

Sertac Karaman (07:10.840)

I mean, we could get what's going to happen in transportation in the next 50, we could

Sertac Karaman (07:13.080)

get flying saucers.

Lex Fridman (07:14.560)

I could bet on that.

Sertac Karaman (07:16.040)

I think there's a 50, 50 chance that, you know, like you can build machines that can

Sertac Karaman (07:19.280)

ionize the air around them and push it down with magnets and they would fly like a flying

Sertac Karaman (07:23.520)

saucer that is possible.

Lex Fridman (07:26.360)

And it might happen in the next 50 years.

Lex Fridman (07:27.920)

So it's a bit hard to guess like when you think about 50 years before, but I would think

Sertac Karaman (07:32.600)

that, you know, there's this, this, this kind of a notion where there's a certain type of

Sertac Karaman (07:38.760)

airspace that we call the agile airspace.

Lex Fridman (07:41.560)

And there's, there's good amount of opportunities in that airspace.

Lex Fridman (07:44.160)

So that would be the space that is kind of a little bit higher than the place where you

Sertac Karaman (07:49.440)

can throw a stone because that's a tough thing when you think about it, you know, it takes

Sertac Karaman (07:53.440)

a kid on a stone to take an aircraft down and then what happens.

Lex Fridman (07:59.600)

But you know, imagine the airspace that's high enough so that you cannot throw the stone,

Lex Fridman (08:05.560)

but it is low enough that you're not interacting with the, with the very large aircraft that

Lex Fridman (08:11.600)

are, you know, flying several thousand feet above.

Lex Fridman (08:15.560)

And that airspace is underutilized or it's actually kind of not utilized at all.

Lex Fridman (08:20.280)

Yeah, that's right.

Sertac Karaman (08:21.280)

You know, there's like recreational people kind of fly every now and then, but it's very

Lex Fridman (08:24.920)

few.

Sertac Karaman (08:25.920)

Like if you look up in the sky, you may not see any of them at any given time, every now

Lex Fridman (08:30.240)

and then you'll see one airplane kind of utilizing that space and you'll be surprised.

Lex Fridman (08:34.520)

And the moment you're outside of an airport a little bit, like it just kind of flies off

Lex Fridman (08:38.120)

and then it goes out.

Lex Fridman (08:39.880)

And I think utilizing that airspace, the technical challenges there is, you know, building an

Lex Fridman (08:47.240)

autonomy and ensuring that that kind of autonomy is safe.

Sertac Karaman (08:51.560)

Ultimately, I think it is going to be building in complex software or complicated so that

Sertac Karaman (08:59.160)

it's maybe a few orders of magnitude more complicated than what we have on aircraft

Sertac Karaman (09:03.800)

today.

Lex Fridman (09:05.480)

And at the same time, ensuring just like we ensure on aircraft, ensuring that it's safe.

Lex Fridman (09:10.560)

And so that becomes like building that kind of complicated hardware and software becomes

Sertac Karaman (09:15.320)

a challenge, especially when, you know, you build that hardware, I mean, you build that

Sertac Karaman (09:20.360)

software with data.

Lex Fridman (09:22.960)

And so, you know, it's, of course there's some rule based software in there that kind

Sertac Karaman (09:28.620)

of do a certain set of things, but then, you know, there's a lot of training there.

Lex Fridman (09:32.320)

Do you think machine learning will be key to these kinds of, to delivering safe vehicles

Lex Fridman (09:37.840)

in the future, especially flight?

Lex Fridman (09:40.920)

Not maybe the safe part, but I think the intelligent part.

Sertac Karaman (09:43.880)

I mean, there are certain things that we do it with machine learning and it's just, there's

Lex Fridman (09:48.240)

like right now, no other way.

Lex Fridman (09:50.680)

And I don't know how else they could be done.

Lex Fridman (09:53.720)

And you know, there's always this conundrum, I mean, we could like, could we like, we could

Sertac Karaman (10:00.100)

maybe gather billions of programmers, humans who program perception algorithms that detect

Sertac Karaman (10:09.480)

things in the sky and whatever, or, you know, we, I don't know, we maybe even have robots

Sertac Karaman (10:14.120)

like learn in a simulation environment and transfer.

Lex Fridman (10:17.200)

And they might be learning a lot better in a simulation environment than a billion humans

Sertac Karaman (10:22.960)

put their brains together and try to program.

Lex Fridman (10:25.760)

Humans pretty limited.

Lex Fridman (10:26.760)

So what's, what's the role of simulations with drones?

Lex Fridman (10:30.400)

You've done quite a bit of work there.

Lex Fridman (10:32.280)

How promising, just the very thing you said just now, how promising is the possibility

Sertac Karaman (10:36.600)

of training and developing a safe flying robot in simulation and deploying it and having

Lex Fridman (10:45.440)

that work pretty well in the real world?

Sertac Karaman (10:48.320)

I think that, you know, a lot of people, when they hear simulation, they will focus on training

Sertac Karaman (10:53.160)

immediately.

Lex Fridman (10:54.160)

But I think one thing that you said, which was interesting, it's developing.

Sertac Karaman (10:57.520)

I think simulation environments are actually could be key and great for development.

Lex Fridman (11:01.900)

And that's not new.

Sertac Karaman (11:03.360)

Like for example, you know, there's people in the automotive industry have been using

Lex Fridman (11:09.080)

dynamic simulation for like decades now.

Lex Fridman (11:12.040)

And it's pretty standard that, you know, you would build and you would simulate.

Sertac Karaman (11:16.200)

If you want to build an embedded controller, you plug that kind of embedded computer into

Sertac Karaman (11:20.760)

another computer, that other computer would simulate dynamic and so on.

Lex Fridman (11:24.640)

And I think, you know, fast forward these things, you can create pretty crazy simulation

Sertac Karaman (11:28.560)

environments.

Sertac Karaman (11:29.560)

Like for instance, one of the things that has happened recently and that, you know,

Sertac Karaman (11:34.880)

we can do now is that we can simulate cameras a lot better than we used to simulate them.

Lex Fridman (11:39.680)

We were able to simulate them before.

Lex Fridman (11:41.080)

And that's, I think we just hit the elbow on that kind of improvement.

Sertac Karaman (11:45.320)

I would imagine that with improvements in hardware, especially, and with improvements

Sertac Karaman (11:50.920)

in machine learning, I think that we would get to a point where we can simulate cameras

Lex Fridman (11:55.520)

very, very well.

Sertac Karaman (11:57.560)

Simulate cameras means simulate how a real camera would see the real world.

Lex Fridman (12:03.400)

Therefore you can explore the limitations of that.

Sertac Karaman (12:07.600)

You can train perception algorithms on that in simulation, all that kind of stuff.

Lex Fridman (12:13.000)

Exactly.

Sertac Karaman (12:14.000)

So, you know, it's, it's, it has been easier to simulate what we would call introspective

Lex Fridman (12:18.780)

sensors like internal sensors.

Lex Fridman (12:20.580)

So for example, inertial sensing has been easy to simulate.

Sertac Karaman (12:24.000)

It has also been easy to simulate dynamics, like physics that are governed by ordinary

Sertac Karaman (12:29.000)

differential equations.

Sertac Karaman (12:30.080)

I mean, like how a car goes around, maybe how it rolls on the road, how it interacts

Sertac Karaman (12:35.600)

with the road, or even an aircraft flying around, like the dynamic physics of that.

Lex Fridman (12:40.720)

What has been really hard has been to simulate extra septive sensors, sensors that kind of

Sertac Karaman (12:45.960)

like look out from the vehicle.

Lex Fridman (12:48.720)

And that's a new thing that's coming like laser range finders that are a little bit

Sertac Karaman (12:52.280)

easier.

Lex Fridman (12:53.980)

Because radars are a little bit tougher.

Sertac Karaman (12:56.240)

I think once we nail that down, the next challenge I think in simulation will be to simulate

Lex Fridman (13:02.040)

human behavior.

Sertac Karaman (13:03.560)

That's also extremely hard.

Sertac Karaman (13:05.520)

Even when you imagine like how a human driven car would act around, even that is hard.

Lex Fridman (13:12.040)

But imagine trying to simulate, you know, a model of a human just doing a bunch of gestures

Lex Fridman (13:17.680)

and so on.

Lex Fridman (13:18.680)

And you know, it's, it's actually simulated.

Lex Fridman (13:20.360)

It's not captured like with motion capture, but it is simulated.

Sertac Karaman (13:23.740)

That's very hard.

Sertac Karaman (13:24.740)

In fact, today I get involved a lot with like sort of this kind of very high end rendering

Sertac Karaman (13:29.680)

projects and I have like this test that I pass it to my friends or my mom, you know,

Sertac Karaman (13:34.040)

I send like two photos, two kind of pictures and I say rendered, which one is rendered,

Sertac Karaman (13:39.040)

which one is real.

Lex Fridman (13:40.040)

And it's pretty hard to distinguish, except I realized, except when we put humans in there,

Sertac Karaman (13:45.300)

it's possible that our brains are trained in a way that we recognize humans extremely

Lex Fridman (13:50.000)

well.

Sertac Karaman (13:51.000)

We don't so much recognize the built environments because built environments sort of came after

Lex Fridman (13:55.080)

per se we evolved into sort of being humans, but humans were always there.

Sertac Karaman (14:00.240)

Same thing happens, for example, you look at like monkeys and you can't distinguish one

Lex Fridman (14:04.440)

from another, but they sort of do.

Lex Fridman (14:06.980)

And it's very possible that they look at humans.

Lex Fridman (14:08.700)

It's kind of pretty hard to distinguish one from another, but we do.

Lex Fridman (14:12.040)

And so our eyes are pretty well trained to look at humans and understand if something

Lex Fridman (14:15.800)

is off, we will get it.

Sertac Karaman (14:18.040)

We may not be able to pinpoint it.

Lex Fridman (14:19.720)

So in my typical friend test or mom test, what would happen is that we'd put like a

Sertac Karaman (14:23.720)

human walking in anything and they say, you know, this is not right.

Lex Fridman (14:29.660)

Something is off in this video.

Sertac Karaman (14:31.340)

I don't know what, but I can tell you it's the human.

Sertac Karaman (14:34.180)

I can take the human and I can show you like inside of a building or like an apartment

Lex Fridman (14:38.740)

and it will look like if we had time to render it, it will look great.

Lex Fridman (14:42.720)

And this should be no surprise.

Sertac Karaman (14:43.760)

A lot of movies that people are watching, it's all computer generated.

Sertac Karaman (14:47.480)

You know, even nowadays, even you watch a drama movie and like, there's nothing going

Sertac Karaman (14:51.600)

on action wise, but it turns out it's kind of like cheaper, I guess, to render the background.

Lex Fridman (14:55.720)

And so they would.

Lex Fridman (14:57.580)

But how do we get there?

Lex Fridman (14:59.700)

How do we get a human that's would pass the mom slash friend test, a simulation of a human

Lex Fridman (15:08.600)

walking?

Lex Fridman (15:09.600)

So do you think that's something we can creep up to by just doing kind of a comparison learning

Sertac Karaman (15:17.160)

where you have humans annotate what's more realistic and not just by watching, like what's

Lex Fridman (15:23.760)

the path?

Sertac Karaman (15:24.760)

Cause it seems totally mysterious how we simulate human behavior.

Sertac Karaman (15:29.920)

It's hard because a lot of the other things that I mentioned to you, including simulating

Lex Fridman (15:34.540)

cameras, right?

Sertac Karaman (15:35.540)

It is, the thing there is that, you know, we know the physics, we know how it works

Sertac Karaman (15:41.540)

like in the real world and we can write some rules and we can do that.

Lex Fridman (15:46.620)

Like for example, simulating cameras, there's this thing called ray tracing.

Sertac Karaman (15:49.560)

I mean, you literally just kind of imagine it's very similar to, it's not exactly the

Lex Fridman (15:54.560)

same, but it's very similar to tracing photon by photon.

Sertac Karaman (15:57.820)

They're going around, bouncing on things and come into your eye, but human behavior, developing

Sertac Karaman (16:03.780)

a dynamic, like a model of that, that is mathematical so that you can put it into a processor that

Sertac Karaman (16:11.720)

would go through that, that's going to be hard.

Lex Fridman (16:13.960)

And so what else do you got?

Lex Fridman (16:15.720)

You can collect data, right?

Lex Fridman (16:17.980)

And you can try to match the data.

Sertac Karaman (16:20.060)

Or another thing that you can do is that, you know, you can show the friend test, you

Lex Fridman (16:23.540)

know, you can say this or that and this or that, and that will be labeling.

Sertac Karaman (16:27.340)

Anything that requires human labeling, ultimately we're limited by the number of humans that,

Sertac Karaman (16:31.380)

you know, we have available at our disposal and the things that they can do, you know,

Sertac Karaman (16:35.220)

they have to do a lot of other things than also labeling this data.

Lex Fridman (16:39.340)

So that modeling human behavior part is, is I think going, we're going to realize it's

Sertac Karaman (16:44.500)

very tough.

Lex Fridman (16:45.780)

And I think that also affects, you know, our development of autonomous vehicles.

Sertac Karaman (16:50.780)

I see them in self driving as well.

Sertac Karaman (16:52.620)

Like you want to use, so you're building self driving, you know, at the first time, like

Sertac Karaman (16:57.980)

right after urban challenge, I think everybody focused on localization, mapping and localization,

Lex Fridman (17:03.140)

you know, slam algorithms came in, Google was just doing that.

Lex Fridman (17:06.980)

And so building these HD maps, basically that's about knowing where you are.

Lex Fridman (17:11.740)

And then five years later in 2012, 2013 came the kind of coding code AI revolution.

Lex Fridman (17:16.260)

And that started telling us where everybody else is, but we're still missing what everybody

Lex Fridman (17:21.380)

else is going to do next.

Lex Fridman (17:23.260)

And so you want to know where you are.

Lex Fridman (17:24.600)

You want to know what everybody else is.

Sertac Karaman (17:26.560)

Hopefully you know that what you're going to do next, and then you want to predict what

Lex Fridman (17:29.660)

other people are going to do.

Lex Fridman (17:30.780)

And that last bit has, has been a real, real challenge.

Lex Fridman (17:35.900)

What do you think is the role, your own of your, of your, the ego vehicle, the robot,

Sertac Karaman (17:42.900)

the you, the robotic you in controlling and having some control of how the future unrolls

Lex Fridman (17:49.640)

of what's going to happen in the future.

Sertac Karaman (17:51.720)

That seems to be a little bit ignored in trying to predict the future is how you yourself

Sertac Karaman (17:57.580)

can affect that future by being either aggressive or less aggressive or signaling in some kind

Sertac Karaman (18:05.540)

of way.

Lex Fridman (18:06.540)

So this kind of game theoretic dance seems to be ignored for the moment.

Sertac Karaman (18:10.820)

It's yeah, it's, it's totally ignored.

Sertac Karaman (18:12.580)

I mean, it's, it's quite interesting actually, like how we how we interact with things versus

Sertac Karaman (18:19.560)

we interact with humans.

Sertac Karaman (18:21.660)

Like so if, if you see a vehicle that's completely empty and it's trying to do something, all

Sertac Karaman (18:27.540)

of a sudden it becomes a thing.

Lex Fridman (18:29.560)

So interacted with like you interact with this table and so you can throw your backpack

Sertac Karaman (18:34.220)

or you can kick your, kick it, put your feet on it and things like that.

Lex Fridman (18:38.020)

But when it's a human, there's all kinds of ways of interacting with a human.

Lex Fridman (18:42.100)

So if you know, like you and I are face to face, we're very civil.

Lex Fridman (18:45.540)

You know, we talk, we understand each other for the most part.

Sertac Karaman (18:48.580)

We'll see you just, you never know what's going to happen.

Lex Fridman (18:52.860)

But the thing is that like, for example, you and I might interact through YouTube comments

Sertac Karaman (18:56.980)

and, you know, the conversation may go at a totally different angle.

Lex Fridman (19:01.140)

And so I think people kind of abusing as autonomous vehicles is a real issue in some sense.

Lex Fridman (19:08.360)

And so when you're an ego vehicle, you're trying to, you know, coordinate your way,

Lex Fridman (19:12.640)

make your way, it's actually kind of harder than being a human.

Sertac Karaman (19:16.100)

You know, it's like, it's you, you, you not only need to be as smart as, as kind of humans

Lex Fridman (19:20.560)

are, but you also, you're a thing.

Lex Fridman (19:22.180)

So they're going to abuse you a little bit.

Lex Fridman (19:23.920)

So you need to make sure that you can get around and do something.

Lex Fridman (19:28.420)

So I, in general, believe in that sort of game theoretic aspects.

Sertac Karaman (19:34.580)

I've actually personally have done, you know, quite a few papers, both on that kind of game

Sertac Karaman (19:39.560)

theory and also like this, this kind of understanding people's social value orientation, for example,

Lex Fridman (19:45.900)

you know, some people are aggressive, some people not so much.

Sertac Karaman (19:48.700)

And, and, you know, like a robot could understand that by just looking at how people drive.

Lex Fridman (19:54.700)

And as they kind of come in approach, you can actually understand, like if someone is

Sertac Karaman (19:58.140)

going to be aggressive or, or not as a robot and you can make certain decisions.

Sertac Karaman (1:00:00.100)

Um, at that time though, it's very possible that, you know, you find the LIDAR system

Sertac Karaman (1:00:06.700)

as another robustifier or, or it's so affordable that it's stupid not to, you know, just kind

Lex Fridman (1:00:13.340)

of put it there.

Lex Fridman (1:00:15.700)

And I think, um, and I think we may be looking at a future like that.

Sertac Karaman (1:00:20.060)

You think we're over relying on LIDAR right now, because we understand the better it's

Sertac Karaman (1:00:25.460)

more reliable in many ways in terms of, from a safety perspective.

Lex Fridman (1:00:28.620)

It's easier to build with.

Sertac Karaman (1:00:29.940)

That's the other, that's the other thing.

Sertac Karaman (1:00:31.180)

I think to be very frank with you, I mean, um, you know, we've seen a lot of sort of

Sertac Karaman (1:00:36.780)

autonomous vehicles companies come and go and the approach has been, you know, you slap

Sertac Karaman (1:00:41.340)

a LIDAR on a car and it's kind of easy to build with when you have a LIDAR, you know,

Sertac Karaman (1:00:46.540)

you just kind of code it up and, and you hit the button and you do a demo.

Lex Fridman (1:00:52.060)

So I think there's admittedly, there's a lot of people, they focus on the LIDAR cause it's

Sertac Karaman (1:00:55.840)

easier to build with.

Sertac Karaman (1:00:57.980)

That doesn't mean that, you know, without the camera, just cameras, you can, uh, you

Sertac Karaman (1:01:02.380)

cannot do what they're doing, but it's just kind of a lot harder.

Lex Fridman (1:01:05.160)

And so you need to have certain kinds of expertise to exploit that.

Lex Fridman (1:01:08.760)

What we believe in and, you know, you may be seeing some of it is that, um, we believe

Lex Fridman (1:01:13.320)

in computer vision.

Sertac Karaman (1:01:14.320)

We certainly work on computer vision and Optimus ride, uh, by a lot, like, um, and, and we've

Lex Fridman (1:01:19.580)

been doing that from day one.

Lex Fridman (1:01:21.500)

And we also believe in sensor fusion.

Sertac Karaman (1:01:23.140)

So, you know, we, we do, we have a relatively minimal use of LIDARs, but, but we do use

Sertac Karaman (1:01:28.340)

them.

Lex Fridman (1:01:29.420)

And I think, you know, in the future, I really believe that the following sequence of events

Sertac Karaman (1:01:33.480)

may happen.

Sertac Karaman (1:01:35.740)

First things first, number one, there may be a future in which, you know, there's like

Sertac Karaman (1:01:39.460)

cars with LIDARs and everything and the cameras, but you know, this in this 50 year ahead future,

Lex Fridman (1:01:45.260)

they can just drive with cameras as well.

Sertac Karaman (1:01:47.900)

Especially in some isolated environments and cameras, they go and they do the thing in

Lex Fridman (1:01:52.060)

the same future.

Sertac Karaman (1:01:53.060)

It's very possible that, you know, the LIDARs are so cheap and frankly make the software

Sertac Karaman (1:01:57.900)

maybe, um, a little less compute intensive, uh, at the very least, or maybe less complicated

Lex Fridman (1:02:04.360)

so that they can be certified or, or insured, they're of their safety and things like that,

Sertac Karaman (1:02:09.620)

that it's kind of stupid not to put the LIDAR, like, imagine this, you either put, pay money

Sertac Karaman (1:02:15.220)

for the LIDAR or you pay money for the compute.

Lex Fridman (1:02:18.340)

And if you don't put the LIDAR, it's a more expensive system because you have to put in

Sertac Karaman (1:02:22.620)

a lot of compute.

Lex Fridman (1:02:23.620)

Like, this is another possibility.

Sertac Karaman (1:02:25.420)

Um, I do think that a lot of the, um, sort of initial deployments of self driving vehicles,

Sertac Karaman (1:02:30.780)

I think they will involve LIDARs and especially either low range or short, um, either short

Sertac Karaman (1:02:37.180)

range or low resolution LIDARs are actually not that hard to build in solid state.

Sertac Karaman (1:02:42.540)

Uh, they're still scanning, but like MEMS type of scanning LIDARs and things like that,

Sertac Karaman (1:02:47.020)

they're like, they're actually not that hard.

Sertac Karaman (1:02:48.620)

I think they will maybe kind of playing with the spectrum and the phase arrays that they're

Sertac Karaman (1:02:52.540)

a little bit harder, but, but I think, um, like, you know, putting a MEMS mirror in there

Lex Fridman (1:02:57.460)

that kind of scans the environment, it's not hard.

Sertac Karaman (1:03:00.340)

The only thing is that, you know, you, just like with a lot of the things that we do nowadays

Sertac Karaman (1:03:04.540)

in developing technology, you hit fundamental limits of the universe, um, the speed of light

Sertac Karaman (1:03:09.160)

becomes a problem in when you're trying to scan the environment.

Lex Fridman (1:03:12.580)

So you don't get either good resolution or you don't get range.

Sertac Karaman (1:03:15.780)

Um, but, but you know, it's still, it's something that you can put in there affordably.

Lex Fridman (1:03:20.420)

So let me jump back to, uh, drones.

Sertac Karaman (1:03:24.380)

You've, uh, you have a role in the Lockheed Martin Alpha Pilot Innovation Challenge.

Sertac Karaman (1:03:30.020)

Where, uh, teams compete in drone racing and super cool, super intense, interesting application

Sertac Karaman (1:03:37.080)

of AI.

Lex Fridman (1:03:38.820)

So can you tell me about the very basics of the challenge and where you fit in, what your

Lex Fridman (1:03:44.060)

thoughts are on this problem?

Lex Fridman (1:03:46.060)

And it's sort of echoes of the early DARPA challenge in the, through the desert that

Sertac Karaman (1:03:51.140)

we're seeing now, now with drone racing.

Lex Fridman (1:03:53.580)

Yeah.

Sertac Karaman (1:03:54.580)

I mean, one interesting thing about it is that, you know, people, the drone racing exists

Lex Fridman (1:03:59.660)

as an eSport.

Lex Fridman (1:04:01.340)

And so it's much like you're playing a game, but there's a real drone going in an environment.

Lex Fridman (1:04:06.060)

A human being is controlling it with goggles on.

Lex Fridman (1:04:08.880)

So there's no, it is a robot, but there's no AI.

Lex Fridman (1:04:13.380)

There's no AI.

Sertac Karaman (1:04:14.380)

Yeah.

Lex Fridman (1:04:15.380)

Human being is controlling it.

Lex Fridman (1:04:16.380)

And so that's already there.

Sertac Karaman (1:04:17.900)

And, um, and I've been interested in this problem for quite a while, actually, um, from

Sertac Karaman (1:04:22.060)

a roboticist point of view.

Lex Fridman (1:04:23.600)

And that's what's happening in Alpha Pilot, which, which problem of aggressive flight

Sertac Karaman (1:04:27.300)

of aggressive flight, fully autonomous, aggressive flight.

Sertac Karaman (1:04:30.980)

Um, the problem that I'm interested, I mean, you asked about Alpha Pilot and I'll, I'll

Sertac Karaman (1:04:34.440)

get there in a second, but the problem that I'm interested in, I'd love to build autonomous

Lex Fridman (1:04:38.880)

vehicles like, like drones that can go far faster than any human possibly can.

Sertac Karaman (1:04:45.140)

I think we should recognize that we as humans have, you know, limitations in how fast we

Lex Fridman (1:04:50.340)

can process information.

Lex Fridman (1:04:52.740)

And those are some biological limitations.

Lex Fridman (1:04:54.580)

Like we think about this AI this way too.

Sertac Karaman (1:04:56.860)

I mean, this has been discussed a lot and this is not sort of my idea per se, but a

Sertac Karaman (1:05:00.940)

lot of people kind of think about human level AI and they think that, you know, AI is not

Sertac Karaman (1:05:05.500)

human level.

Lex Fridman (1:05:06.500)

One day it'll be human level and humans and AI's, they kind of interact.

Sertac Karaman (1:05:09.860)

Um, versus I think that the situation really is that humans are at a certain place and

Sertac Karaman (1:05:14.820)

AI keeps improving and at some point it just crosses off and then, you know, it gets smarter

Lex Fridman (1:05:19.140)

and smarter and smarter.

Lex Fridman (1:05:21.180)

And so drone racing, the same issue.

Sertac Karaman (1:05:24.660)

Just play this game and you know, you have to like react in milliseconds and there's

Sertac Karaman (1:05:29.780)

really, you know, you see something with your eyes and then that information just flows

Sertac Karaman (1:05:34.380)

through your brain, into your hands so that you can command it.

Lex Fridman (1:05:37.620)

And there's some also delays on, you know, getting information back and forth, but suppose

Sertac Karaman (1:05:40.920)

those delays didn't exist.

Sertac Karaman (1:05:41.980)

You just, just the delay between your eye and your fingers is a delay that a robot doesn't

Sertac Karaman (1:05:49.820)

have to have.

Sertac Karaman (1:05:51.300)

Um, so we end up building in my research group, like systems that, you know, see things at

Sertac Karaman (1:05:57.460)

a kilohertz, like a human eye would barely hit a hundred Hertz.

Lex Fridman (1:06:00.940)

So imagine things that see stuff in slow motion, like 10 X slow motion.

Sertac Karaman (1:06:07.060)

Um, it will be very useful.

Lex Fridman (1:06:08.740)

Like we talked a lot about autonomous cars.

Sertac Karaman (1:06:10.260)

So, um, you know, we don't get to see it, but a hundred lives are lost every day, just

Lex Fridman (1:06:17.020)

in the United States on traffic accidents.

Lex Fridman (1:06:19.500)

And many of them are like known cases, you know, like the, uh, you're coming through

Sertac Karaman (1:06:24.140)

like, uh, like a ramp going into a highway, you hit somebody and you're off, or, you know,

Sertac Karaman (1:06:29.460)

like you kind of get confused.

Sertac Karaman (1:06:30.900)

You try to like swerve into the next lane, you go off the road and you crash, whatever.

Lex Fridman (1:06:35.880)

And um, I think if you had enough compute in a car and a very fast camera right at the

Sertac Karaman (1:06:41.500)

time of an accident, you could use all compute you have, like you could shut down the infotainment

Sertac Karaman (1:06:46.260)

system and use that kind of computing resources instead of rendering, you use it for the kind

Lex Fridman (1:06:53.260)

of artificial intelligence that goes in there, the autonomy.

Lex Fridman (1:06:56.420)

And you can, you can either take control of the car and bring it to a full stop.

Lex Fridman (1:07:00.140)

But even, even if you can't do that, you can deliver what the human is trying to do.

Sertac Karaman (1:07:04.060)

Human is trying to change the lane, but goes off the road, not being able to do that with

Lex Fridman (1:07:08.980)

motor skills and the eyes.

Lex Fridman (1:07:10.900)

And you know, you can get in there and I was, there's so many other things that you can

Lex Fridman (1:07:14.540)

enable with what I would call high throughput computing.

Sertac Karaman (1:07:17.380)

You know, data is coming in extremely fast and in real time you have to process it.

Lex Fridman (1:07:24.220)

And the current CPUs, however fast you clock it are typically not enough.

Sertac Karaman (1:07:30.740)

You need to build those computers from the ground up so that they can ingest all that

Lex Fridman (1:07:34.240)

data that I'm really interested in.

Sertac Karaman (1:07:36.500)

Just on that point, just really quick is the currently what's the bottom, like you mentioned

Lex Fridman (1:07:42.060)

the delays in humans, is it the hardware?

Lex Fridman (1:07:45.340)

So you work a lot with Nvidia hardware.

Lex Fridman (1:07:47.660)

Is it the hardware or is it the software?

Sertac Karaman (1:07:50.100)

I think it's both.

Lex Fridman (1:07:51.460)

I think it's both.

Sertac Karaman (1:07:52.460)

In fact, they need to be co developed I think in the future.

Sertac Karaman (1:07:54.940)

I mean, that's a little bit what Nvidia does sort of like they almost like build the hardware

Lex Fridman (1:07:59.340)

and then they build the neural networks and then they build the hardware back and the

Lex Fridman (1:08:02.540)

neural networks back and it goes back and forth, but it's that co design.

Lex Fridman (1:08:06.420)

And I think that, you know, like we try to way back, we try to build a fast drone that

Sertac Karaman (1:08:11.700)

could use a camera image to like track what's moving in order to find where it is in the

Sertac Karaman (1:08:16.220)

world.

Sertac Karaman (1:08:17.380)

This typical sort of, you know, visual inertial state estimation problems that we would solve.

Lex Fridman (1:08:22.260)

And you know, we just kind of realized that we're at the limit sometimes of, you know,

Lex Fridman (1:08:25.820)

doing simple tasks.

Sertac Karaman (1:08:26.820)

We're at the limit of the camera frame rate because you know, if you really want to track

Sertac Karaman (1:08:30.820)

things, you want the camera image to be 90% kind of like, or some somewhat the same from

Sertac Karaman (1:08:36.660)

one frame to the next.

Lex Fridman (1:08:39.180)

And why are we at the limit of the camera frame rate?

Sertac Karaman (1:08:42.020)

It's because camera captures data.

Lex Fridman (1:08:44.700)

It puts it into some serial connection.

Sertac Karaman (1:08:47.020)

It could be USB or like there's something called camera serial interface that we use

Lex Fridman (1:08:51.500)

a lot.

Sertac Karaman (1:08:52.500)

It puts into some serial connection and copper wires can only transmit so much data.

Lex Fridman (1:08:58.380)

And you hit the channel limit on copper wires and you know, you, you hit yet another kind

Sertac Karaman (1:09:02.780)

of universal limit that you can transfer the data.

Lex Fridman (1:09:06.900)

So you have to be much more intelligent on how you capture those pixels.

Sertac Karaman (1:09:11.260)

You can take compute and put it right next to the pixels.

Lex Fridman (1:09:16.300)

People are building those.

Lex Fridman (1:09:17.300)

How hard is it to do?

Lex Fridman (1:09:18.300)

How hard is it to get past the bottleneck of the copper wire?

Sertac Karaman (1:09:23.180)

Yeah, you need to, you need to do a lot of parallel processing, as you can imagine.

Sertac Karaman (1:09:27.020)

The same thing happens in the GPUs, you know, like the data is transferred in parallel somehow.

Sertac Karaman (1:09:31.700)

It gets into some parallel processing.

Sertac Karaman (1:09:33.900)

I think that, you know, like now we're really kind of diverted off into so many different

Sertac Karaman (1:09:38.380)

dimensions, but.

Lex Fridman (1:09:39.380)

Great.

Lex Fridman (1:09:40.380)

So it's aggressive flight.

Lex Fridman (1:09:41.380)

How do we make drones see many more frames a second in order to enable aggressive flight?

Sertac Karaman (1:09:46.900)

That's a super interesting problem.

Lex Fridman (1:09:48.260)

That's an interesting problem.

Sertac Karaman (1:09:49.260)

So, but like, think about it.

Lex Fridman (1:09:50.260)

You have, you have CPUs.

Sertac Karaman (1:09:52.900)

You clock them at, you know, several gigahertz.

Sertac Karaman (1:09:57.100)

We don't clock them faster, largely because, you know, we run into some heating issues

Lex Fridman (1:10:00.980)

and things like that.

Lex Fridman (1:10:01.980)

But the whole thing is that three gigahertz clock light travels kind of like on the order

Sertac Karaman (1:10:07.500)

of a few inches or an inch.

Lex Fridman (1:10:09.980)

That's the size of a chip.

Lex Fridman (1:10:11.660)

And so you pass a clock cycle and as the clock signal is going around in the chip, you pass

Lex Fridman (1:10:17.900)

another one.

Lex Fridman (1:10:19.300)

And so trying to coordinate that, the design of the complexity of the chip becomes so hard.

Sertac Karaman (1:10:23.820)

I mean, we have hit the fundamental limits of the universe in so many things that we're

Sertac Karaman (1:10:29.220)

designing.

Lex Fridman (1:10:30.220)

I don't know if people realize that.

Sertac Karaman (1:10:31.220)

Like, we can't make transistors smaller because like quantum effects, the electrons start

Lex Fridman (1:10:35.660)

to tunnel around.

Sertac Karaman (1:10:36.660)

We can't clock it faster.

Sertac Karaman (1:10:38.380)

One of the reasons why is because like information doesn't travel faster in the universe and

Sertac Karaman (1:10:45.020)

we're limited by that.

Lex Fridman (1:10:46.140)

Same thing with the laser scanner.

Lex Fridman (1:10:48.060)

But so then it becomes clear that, you know, the way you organize the chip into a CPU or

Lex Fridman (1:10:54.860)

even a GPU, you now need to look at how to redesign that.

Sertac Karaman (1:10:59.580)

If you're going to stick with Silicon, you could go do other things too.

Sertac Karaman (1:11:02.940)

I mean, there's that too, but you really almost need to take those transistors, put them in

Sertac Karaman (1:11:06.940)

a different way so that the information travels on those transistors in a different way, in

Lex Fridman (1:11:12.100)

a much more way that is specific to the high speed cameras coming in.

Lex Fridman (1:11:16.780)

And so that's one of the things that we talk about quite a bit.

Lex Fridman (1:11:20.620)

So drone racing kind of really makes that embodies that and that's why it's exciting.

Sertac Karaman (1:11:27.580)

It's exciting for people, you know, students like it.

Lex Fridman (1:11:30.180)

It embodies all those problems.

Lex Fridman (1:11:32.080)

But going back, we're building, quote, unquote, another engine.

Lex Fridman (1:11:36.200)

And that engine, I hope one day will be just like how impactful seat belts were in driving.

Sertac Karaman (1:11:43.860)

I hope so.

Sertac Karaman (1:11:45.720)

Or it could enable, you know, next generation autonomous air taxis and things like that.

Sertac Karaman (1:11:49.540)

I mean, it sounds crazy, but one day we may need to perch land these things.

Sertac Karaman (1:11:53.800)

If you really want to go from Boston to New York in more than a half hours, you may want

Sertac Karaman (1:11:58.320)

to fix wing aircraft.

Sertac Karaman (1:12:00.080)

Most of these companies that are kind of doing quote unquote flying cars, they're focusing

Sertac Karaman (1:12:03.540)

on that.

Lex Fridman (1:12:04.540)

But then how do you land it on top of a building?

Sertac Karaman (1:12:06.600)

You may need to pull off like kind of fast maneuvers for a robot, like perch land.

Lex Fridman (1:12:10.900)

It's going to go perch into a building.

Sertac Karaman (1:12:14.020)

If you want to do that, like you need these kinds of systems.

Lex Fridman (1:12:17.060)

And so drone racing, you know, it's being able to go way faster than any human can comprehend.

Sertac Karaman (1:12:25.880)

Take an aircraft, forget the quadcopter, you take your fixed wing, while you're at it,

Sertac Karaman (1:12:30.520)

you might as well put some like rocket engines in the back and you just light it.

Lex Fridman (1:12:34.040)

You go through the gate and a human looks at it and just said, what just happened?

Lex Fridman (1:12:39.320)

And they would say, it's impossible for me to do that.

Lex Fridman (1:12:41.520)

And that's closing the same technology gap that would, you know, one day steer cars out

Lex Fridman (1:12:47.240)

of accidents.

Lex Fridman (1:12:48.960)

So but then let's get back to the practical, which is sort of just getting the thing to

Sertac Karaman (1:12:55.320)

to work in a race environment, which is kind of what the is another kind of exciting thing,

Sertac Karaman (1:13:01.360)

which the DARPA challenge to the desert did, you know, theoretically, we had autonomous

Sertac Karaman (1:13:05.340)

vehicles, but making them successfully finish a race, first of all, which nobody finished

Sertac Karaman (1:13:11.080)

the first year, and then the second year just to get, you know, to finish and go at a reasonable

Lex Fridman (1:13:16.960)

time is really difficult engineering, practically speaking challenge.

Lex Fridman (1:13:21.160)

So that let me ask about the the the Alpha pilot challenge is a, I guess, a big prize

Lex Fridman (1:13:27.820)

potentially associated with it.

Lex Fridman (1:13:29.320)

But let me ask, reminiscent of the DARPA days, predictions, you think anybody will finish?

Lex Fridman (1:13:36.400)

Well, not, not soon.

Sertac Karaman (1:13:39.760)

I think that depends on how you set up the race course.

Lex Fridman (1:13:42.440)

And so if the race course is a solo course, I think people will kind of do it.

Lex Fridman (1:13:46.380)

But can you set up some course, like literally some core, you get to design it is the algorithm

Lex Fridman (1:13:53.280)

developer, can you set up some course, so that you can be the best human?

Lex Fridman (1:13:58.000)

When is that going to happen?

Sertac Karaman (1:14:00.560)

Like that's not very easy, even just setting up some course, if you let the human that

Sertac Karaman (1:14:05.080)

you're competing with set up the course, it becomes a lot easier, a lot harder.

Lex Fridman (1:14:10.520)

So how many in the space of all possible courses are, would humans win and would machines win?

Sertac Karaman (1:14:18.840)

Great question.

Lex Fridman (1:14:19.840)

Let's get to that.

Lex Fridman (1:14:20.840)

I want to answer your other question, which is like, the DARPA challenge days, right?

Lex Fridman (1:14:24.720)

What was really hard?

Sertac Karaman (1:14:25.720)

I think, I think we understand, we understood what we wanted to build, but still building

Sertac Karaman (1:14:30.960)

things, that experimentation, that iterated learning, that takes up a lot of time actually.

Lex Fridman (1:14:36.600)

And so in my group, for example, in order for us to be able to develop fast, we build

Sertac Karaman (1:14:41.720)

like VR environments, we'll take an aircraft, we'll put it in a motion capture room, big,

Sertac Karaman (1:14:46.800)

huge motion capture room, and we'll fly it in real time, we'll render other images and

Lex Fridman (1:14:52.440)

beam it back to the drone.

Sertac Karaman (1:14:54.520)

That sounds kind of notionally simple, but it's actually hard because now you're trying

Lex Fridman (1:14:58.880)

to fit all that data through the air into the drone.

Lex Fridman (1:15:02.640)

And so you need to do a few crazy things to make that happen.

Lex Fridman (1:15:05.640)

But once you do that, then at least you can try things.

Sertac Karaman (1:15:09.240)

If you crash into something, you didn't actually crash.

Lex Fridman (1:15:12.240)

So it's like the whole drone is in VR.

Sertac Karaman (1:15:14.040)

We can do augmented reality and so on.

Lex Fridman (1:15:17.080)

And so I think at some point testing becomes very important.

Sertac Karaman (1:15:20.600)

One of the nice things about Alpha Pilot is that they built the drone and they build a

Lex Fridman (1:15:24.800)

lot of drones and it's okay to crash.

Sertac Karaman (1:15:28.280)

In fact, I think maybe the viewers may kind of like to see things that crash.

Lex Fridman (1:15:34.700)

That potentially could be the most exciting part.

Sertac Karaman (1:15:36.960)

It could be the exciting part.

Lex Fridman (1:15:38.260)

And I think as an engineer, it's a very different situation to be in.

Sertac Karaman (1:15:42.680)

Like in academia, a lot of my colleagues who are actually in this race and they're really

Sertac Karaman (1:15:46.800)

great researchers, but I've seen them trying to do similar things whereby they built this

Sertac Karaman (1:15:51.420)

one drone and somebody with like a face mask and a gloves are going right behind the drone.

Lex Fridman (1:15:58.240)

They're trying to hold it.

Sertac Karaman (1:15:59.240)

If it falls down, imagine you don't have to do that.

Sertac Karaman (1:16:02.480)

I think that's one of the nice things about Alpha Pilot Challenge where we have these

Sertac Karaman (1:16:06.120)

drones and we're going to design the courses in a way that we'll keep pushing people up

Lex Fridman (1:16:11.520)

until the crashes start to happen.

Lex Fridman (1:16:14.480)

And we'll hopefully sort of, I don't think you want to tell people crashing is okay.

Sertac Karaman (1:16:19.320)

Like we want to be careful here, but because we don't want people to crash a lot, but certainly

Sertac Karaman (1:16:24.440)

we want them to push it so that everybody crashes once or twice and they're really pushing

Lex Fridman (1:16:30.440)

it to their limits.

Sertac Karaman (1:16:32.400)

That's where iterated learning comes in, because every crash is a lesson.

Lex Fridman (1:16:36.320)

Is a lesson.

Sertac Karaman (1:16:37.320)

Exactly.

Lex Fridman (1:16:38.320)

So in terms of the space of possible courses, how do you think about it in the war of humans

Lex Fridman (1:16:44.880)

versus machines, where do machines win?

Lex Fridman (1:16:47.680)

We look at that quite a bit.

Sertac Karaman (1:16:48.920)

I mean, I think that you will see quickly that you can design a course and in certain

Sertac Karaman (1:16:56.120)

courses like in the middle somewhere, if you kind of run through the course once, the machine

Sertac Karaman (1:17:03.120)

gets beaten pretty much consistently by slightly.

Lex Fridman (1:17:07.760)

But if you go through the course like 10 times, humans get beaten very slightly, but consistently.

Lex Fridman (1:17:13.280)

So humans at some point, you get confused, you get tired and things like that versus

Sertac Karaman (1:17:17.360)

this machine is just executing the same line of code tirelessly, just going back to the

Sertac Karaman (1:17:23.360)

beginning and doing the same thing exactly.

Sertac Karaman (1:17:26.400)

I think that kind of thing happens and I realized sort of as humans, there's the classical things

Sertac Karaman (1:17:34.000)

that everybody has realized.

Sertac Karaman (1:17:36.280)

Like if you put in some sort of like strategic thinking, that's a little bit harder for machines

Sertac Karaman (1:17:41.120)

that I think sort of comprehend.

Lex Fridman (1:17:45.160)

Machine is easy to do, so that's what they excel in.

Lex Fridman (1:17:48.720)

And also sort of repeatability is easy to do.

Lex Fridman (1:17:53.160)

That's what they excel in.

Sertac Karaman (1:17:55.120)

You can build machines that excel in strategy as well and beat humans that way too, but

Lex Fridman (1:17:59.360)

that's a lot harder to build.

Sertac Karaman (1:18:00.360)

I have a million more questions, but in the interest of time, last question.

Lex Fridman (1:18:06.680)

What is the most beautiful idea you've come across in robotics?

Lex Fridman (1:18:10.360)

Is it a simple equation, experiment, a demo, a simulation, a piece of software?

Lex Fridman (1:18:15.080)

What just gives you pause?

Sertac Karaman (1:18:19.240)

That's an interesting question.

Sertac Karaman (1:18:21.000)

I have done a lot of work myself in decision making, so I've been interested in that area.

Lex Fridman (1:18:26.760)

So you know, in robotics, somehow the field has split into like, you know, there's people

Sertac Karaman (1:18:32.400)

who would work on like perception, how robots perceive the environment, then how do you

Sertac Karaman (1:18:37.200)

actually make like decisions and there's people also like how do you interact, people interact

Lex Fridman (1:18:41.080)

with robots, there's a whole bunch of different fields.

Lex Fridman (1:18:44.160)

And you know, I have admittedly worked a lot on the more control and decision making than

Lex Fridman (1:18:49.920)

the others.

Lex Fridman (1:18:52.060)

And I think that, you know, the one equation that has always kind of baffled me is Bellman's

Lex Fridman (1:18:57.440)

equation.

Lex Fridman (1:18:59.100)

And so it's this person who have realized like way back, you know, more than half a

Lex Fridman (1:19:04.920)

century ago on like, how do you actually sit down?

Lex Fridman (1:19:10.760)

And if you have several variables that you're kind of jointly trying to determine, how do

Lex Fridman (1:19:15.680)

you determine that?

Lex Fridman (1:19:17.400)

And there's one beautiful equation that, you know, like today people do reinforcement

Lex Fridman (1:19:22.280)

and we still use it.

Lex Fridman (1:19:24.120)

And it's baffling to me because it both kind of tells you the simplicity, because it's

Lex Fridman (1:19:31.000)

a single equation that anyone can write down.

Sertac Karaman (1:19:33.920)

You can teach it in the first course on decision making.

Lex Fridman (1:19:37.400)

At the same time, it tells you how computationally, how hard the problem is.

Sertac Karaman (1:19:41.440)

I feel like my, like a lot of the things that I've done at MIT for research has been kind

Lex Fridman (1:19:45.360)

of just this fight against computational efficiency things.

Sertac Karaman (1:19:48.840)

Like how can we get it faster to the point where we now got to like, let's just redesign

Lex Fridman (1:19:54.040)

this chip.

Sertac Karaman (1:19:55.040)

Like maybe that's the way, but I think it talks about how computationally hard certain

Lex Fridman (1:20:01.800)

problems can be by nowadays what people call curse of dimensionality.

Lex Fridman (1:20:07.760)

And so as the number of variables kind of grow, the number of decisions you can make

Lex Fridman (1:20:13.840)

grows rapidly.

Sertac Karaman (1:20:16.060)

Like if you have, you know, a hundred variables, each one of them take 10 values, all possible

Lex Fridman (1:20:21.860)

assignments is more than the number of atoms in the universe.

Sertac Karaman (1:20:24.600)

It's just crazy.

Lex Fridman (1:20:26.440)

And that kind of thinking is just embodied in that one equation that I really like.

Lex Fridman (1:20:31.400)

And the beautiful balance between it being theoretically optimal and somehow practically

Sertac Karaman (1:20:38.280)

speaking, given the curse of dimensionality, nevertheless in practice works among, you

Sertac Karaman (1:20:45.240)

know, despite all those challenges, which is quite incredible.

Lex Fridman (1:20:48.080)

Which is quite incredible.

Sertac Karaman (1:20:49.200)

So, you know, I would say that it's kind of like quite baffling actually, you know, in

Sertac Karaman (1:20:53.880)

a lot of fields that we think about how little we know, you know, like, and so I think here

Sertac Karaman (1:21:00.080)

too.

Sertac Karaman (1:21:01.080)

We know that in the worst case, things are pretty hard, but you know, in practice, generally

Sertac Karaman (1:21:06.440)

things work.

Lex Fridman (1:21:07.440)

So it's just kind of, it's kind of baffling decision making, how little we know.

Sertac Karaman (1:21:12.840)

Just like how little we know about the beginning of time, how little we know about, you know,

Lex Fridman (1:21:17.520)

our own future.

Sertac Karaman (1:21:19.640)

Like if you actually go into like from Bellman's equation all the way down, I mean, there's

Lex Fridman (1:21:23.840)

also how little we know about like mathematics.

Sertac Karaman (1:21:26.160)

I mean, we don't even know if the axioms are like consistent.

Lex Fridman (1:21:28.840)

It's just crazy.

Sertac Karaman (1:21:29.840)

I think a good lesson there, just like as you said, we tend to focus on the worst case

Sertac Karaman (1:21:35.800)

or the boundaries of everything we're studying and then the average case seems to somehow

Sertac Karaman (1:21:40.680)

work out.

Lex Fridman (1:21:41.680)

If you think about life in general, we mess it up a bunch.

Sertac Karaman (1:21:45.040)

You know, we freak out about a bunch of the traumatic stuff, but in the end it seems to

Lex Fridman (1:21:49.120)

work out okay.

Sertac Karaman (1:21:50.120)

Yeah.

Lex Fridman (1:21:51.120)

It seems like a good metaphor.

Lex Fridman (1:21:52.120)

So Tashi, thank you so much for being a friend, a colleague, a mentor.

Lex Fridman (1:21:57.280)

I really appreciate it.

Sertac Karaman (1:21:58.280)

It's an honor to talk to you.

Lex Fridman (1:21:59.280)

Thank you so much for your advice.

Sertac Karaman (1:22:00.280)

Thank you Lex.

Sertac Karaman (1:22:01.280)

Thanks for listening to this conversation with Sertaj Karaman and thank you to our presenting

Sertac Karaman (1:22:05.800)

sponsor Cash App.

Sertac Karaman (1:22:07.440)

Please consider supporting the podcast by downloading Cash App and using code LexPodcast.

Sertac Karaman (1:22:11.840)

If you enjoy this podcast, subscribe on YouTube, review it with five stars on Apple Podcast,

Lex Fridman (1:22:18.120)

support it on Patreon, or simply connect with me on Twitter at Lex Friedman.

Lex Fridman (1:22:23.280)

And now let me leave you with some words from Hal9000 from the movie 2001 A Space Odyssey.

Sertac Karaman (1:22:30.320)

I'm putting myself to the fullest possible use, which is all I think that any conscious

Sertac Karaman (1:22:36.460)

entity can ever hope to do.

Lex Fridman (1:22:39.120)

Thank you for listening and hope to see you next time.

Sertac Karaman (20:02.740)

Well, in terms of predicting what they're going to do, the hard question is you as a

Lex Fridman (20:07.580)

robot, should you be aggressive or not when faced with an aggressive robot?

Sertac Karaman (20:13.100)

Right now it seems like aggressive is a very dangerous thing to do because it's costly

Lex Fridman (20:19.140)

from a societal perspective, how you're perceived.

Sertac Karaman (20:22.940)

People are not very accepting of aggressive robots in modern society.

Lex Fridman (20:27.060)

I think that's accurate.

Lex Fridman (20:28.420)

So that is really is.

Lex Fridman (20:31.060)

And so I'm not entirely sure like how to have to go about, but I know, I know for a fact

Sertac Karaman (20:36.340)

that how these robots interact with other people in there is going to be, and then interaction

Lex Fridman (20:41.380)

is always going to be there.

Sertac Karaman (20:42.380)

I mean, you could be interacting with other vehicles or other just people kind of like

Lex Fridman (20:46.100)

walking around.

Lex Fridman (20:48.220)

And like I said, the moment there's like nobody in the seat, it's like an empty thing just

Lex Fridman (20:52.860)

rolling off the street.

Sertac Karaman (20:54.500)

It becomes like no different than like any other thing that's not human.

Lex Fridman (20:59.860)

And so people, and maybe abuse is the wrong word, but people maybe rightfully even they

Sertac Karaman (21:05.300)

feel like this is a human present environment designed for humans to be, and they kind of

Lex Fridman (21:11.380)

they want to own it.

Lex Fridman (21:13.180)

And then the robots, they would need to understand it and they would need to respond in a certain

Lex Fridman (21:18.020)

way.

Lex Fridman (21:19.020)

And I think that this actually opens up like quite a few interesting societal questions

Lex Fridman (21:23.040)

for us as we deploy, like we talk robots at large scale.

Lex Fridman (21:26.980)

So what would happen when we try to deploy robots at large scale, I think is that we

Sertac Karaman (21:30.660)

can design systems in a way that they're very efficient or we can design them that they're

Sertac Karaman (21:35.720)

very sustainable, but ultimately the sustainability efficiency trade offs, like they're going

Lex Fridman (21:40.380)

to be right in there and we're going to have to make some choices.

Sertac Karaman (21:44.380)

Like we're not going to be able to just kind of put it aside.

Lex Fridman (21:47.500)

So for example, we can be very aggressive and we can reduce transportation delays, increase

Sertac Karaman (21:52.700)

capacity of transportation, or we can be a lot nicer and allow other people to kind of

Sertac Karaman (21:58.260)

quote unquote own the environment and live in a nice place and then efficiency will drop.

Lex Fridman (22:04.340)

So when you think about it, I think sustainability gets attached to energy consumption or environmental

Lex Fridman (22:10.500)

impact immediately.

Lex Fridman (22:11.500)

And those are there, but like livability is another sustainability impact.

Lex Fridman (22:15.760)

So you create an environment that people want to live in.

Lex Fridman (22:19.340)

And if, if, if robots are going around being aggressive and you don't want to live in that

Sertac Karaman (22:23.060)

environment, maybe, however, you should note that if you're not being aggressive, then,

Sertac Karaman (22:27.260)

you know, you're probably taking up some, some delays in transportation and this and

Lex Fridman (22:31.380)

that.

Lex Fridman (22:32.380)

So you're always balancing that.

Lex Fridman (22:34.900)

And I think this, this choice has always been there in transportation, but I think the more

Sertac Karaman (22:38.860)

autonomy comes in, the more explicit the choice becomes.

Lex Fridman (22:42.540)

Yeah.

Lex Fridman (22:43.540)

And when it becomes explicit, then we can start to optimize it and then we'll get to

Lex Fridman (22:47.700)

ask the very difficult societal questions of what do we value more, efficiency or sustainability?

Sertac Karaman (22:53.500)

It's kind of interesting.

Sertac Karaman (22:56.140)

I think we're going to have to like, I think that the interesting thing about like the

Sertac Karaman (23:00.300)

whole autonomous vehicles question, I think is also kind of, um, I think a lot of times,

Sertac Karaman (23:06.300)

you know, we have, we have focused on technology development, like hundreds of years and you

Sertac Karaman (23:12.220)

know, the products somehow followed and then, you know, we got to make these choices and

Lex Fridman (23:15.940)

things like that.

Lex Fridman (23:16.940)

So this is, this is a good time that, you know, we even think about, you know, autonomous

Lex Fridman (23:20.900)

taxi type of deployments and the systems that would evolve from there.

Lex Fridman (23:25.480)

And you realize the business models are different.

Sertac Karaman (23:28.240)

The impact on architecture is different, urban planning, you get into like regulations, um,

Lex Fridman (23:35.260)

and then you get into like these issues that you didn't think about before, but like sustainability

Lex Fridman (23:39.080)

and ethics is like right in the middle of it.

Sertac Karaman (23:41.660)

I mean, even testing autonomous vehicles, like think about it, you're testing autonomous

Lex Fridman (23:45.260)

vehicles in human present environments.

Sertac Karaman (23:47.060)

I mean, uh, the risk may be very small, but still, you know, it's, it's a, it's a, it's,

Sertac Karaman (23:52.060)

it's a, you know, strictly greater than zero risk that you're putting people into.

Lex Fridman (23:56.340)

And so then you have that innovation, you know, risk trade off that you're, you're in

Lex Fridman (24:01.940)

that somewhere.

Sertac Karaman (24:02.940)

Um, and we, we understand that pretty now that pretty well now is that if we don't test

Lex Fridman (24:08.420)

the, at least the, the development will be slower.

Sertac Karaman (24:12.340)

I mean, it doesn't mean that we're not going to be able to develop.

Lex Fridman (24:15.140)

I think it's going to be pretty hard actually.

Sertac Karaman (24:17.020)

Maybe we can, we don't, we don't, I don't know.

Lex Fridman (24:18.900)

But the thing is that those kinds of trade offs we already are making and as these systems

Sertac Karaman (24:24.100)

become more ubiquitous, I think those trade offs will just really hit.

Lex Fridman (24:30.200)

So you are one of the founders of Optimus Ride and autonomous vehicle company.

Sertac Karaman (24:34.340)

We'll talk about it, but let me on that point ask maybe a good examples, keeping Optimus

Sertac Karaman (24:43.140)

Ride out, out of this question, uh, sort of exemplars of different strategies on the spectrum

Sertac Karaman (24:51.820)

of innovation and safety or caution.

Lex Fridman (24:56.160)

So like Waymo, Google self driving car Waymo represents maybe a more cautious approach.

Lex Fridman (25:03.260)

And then you have Tesla on the other side headed by Elon Musk that represents a more,

Lex Fridman (25:10.380)

however, which adjective you want to use, aggressive, innovative, I don't know.

Lex Fridman (25:14.660)

But uh, what, what do you think about the difference in the two strategies in your view?

Sertac Karaman (25:21.700)

What's more likely, what's needed and is more likely to succeed in the short term and in

Lex Fridman (25:27.980)

the long term?

Lex Fridman (25:30.200)

Definitely some sort of a balance is, is kind of the right way to go.

Lex Fridman (25:33.220)

But I do think that the thing that is the most important is actually like an informed

Lex Fridman (25:38.000)

public.

Lex Fridman (25:39.240)

So I don't, I don't mind, you know, I personally, like if I were in some place, I wouldn't mind

Lex Fridman (25:45.740)

so much like taking a certain amount of risk, um, some other people might.

Lex Fridman (25:52.100)

And so I think the key is for people to be informed and so that they can, ideally they

Lex Fridman (25:57.700)

can make a choice.

Sertac Karaman (25:59.980)

In some cases, that kind of choice, um, making that unanimously is of course very hard.

Lex Fridman (26:06.500)

But I don't think it's actually that hard to inform people.

Lex Fridman (26:10.580)

So I think in, in, in one case, like for example, even the Tesla approach, um, I don't know,

Lex Fridman (26:17.500)

it's hard to judge how informed it is, but it is somewhat informed.

Sertac Karaman (26:20.380)

I mean, you know, things kind of come out.

Lex Fridman (26:21.980)

I think people know what they're taking and things like that and so on.

Lex Fridman (26:25.900)

But I think the, the underlying, um, I do think that these two companies are a little

Sertac Karaman (26:30.500)

bit kind of representing like the, of course they, you know, one of them seems a bit safer

Sertac Karaman (26:36.220)

or the other one, or, you know, um, whatever the objective for that is, and the other one

Lex Fridman (26:40.500)

seems more aggressive or whatever the objective for that is.

Sertac Karaman (26:43.140)

But, but I think, you know, when you turn the tables, they're actually, there are two

Lex Fridman (26:47.020)

other orthogonal dimensions that these two are focusing on.

Sertac Karaman (26:50.320)

On the one hand for Waymo, I can see that, you know, they're, I mean, um, they, I think

Lex Fridman (26:55.140)

they a little bit see it as research as well.

Lex Fridman (26:57.280)

So they kind of, they don't, I'm not sure if they're like really interested in like

Lex Fridman (27:00.180)

an immediate, um, product, um, you know, they, they talk about it.

Sertac Karaman (27:05.820)

Um, sometimes there's some pressure to talk about it.

Lex Fridman (27:08.260)

So they, they kind of go for it, but I think, um, I think that they're thinking, um, maybe

Sertac Karaman (27:13.820)

in the back of their minds, maybe they don't put it this way, but I think they, they realize

Lex Fridman (27:17.940)

that we're building like a new engine.

Sertac Karaman (27:20.180)

It's kind of like call it the AI engine or whatever that is.

Lex Fridman (27:23.060)

And you know, an autonomous vehicles is a very interesting embodiment of that engine

Sertac Karaman (27:27.940)

that allows you to understand where the ego vehicle is, the ego thing is where everything

Sertac Karaman (27:32.220)

else is, what everything else is going to do and how do you react, how do you actually,

Lex Fridman (27:36.780)

you know, interact with humans the right way?

Lex Fridman (27:38.680)

How do you build these systems?

Lex Fridman (27:39.680)

And I think, uh, they, they want to know that they want to understand that.

Lex Fridman (27:43.180)

And so they keep going and doing that.

Lex Fridman (27:45.580)

And so on the other dimension, Tesla is doing something interesting.

Lex Fridman (27:48.340)

I mean, I think that they have a good product.

Sertac Karaman (27:50.400)

People use it.

Sertac Karaman (27:51.400)

I think that, you know, like it's, it's not for me, um, but I can totally see people,

Sertac Karaman (27:55.400)

people like it and, and people, I think they have a good product outside of automation,

Lex Fridman (27:59.320)

but I was just referring to the, the, the automation itself.

Sertac Karaman (28:02.260)

I mean, you know, like it, it kind of drives itself.

Lex Fridman (28:05.580)

You still have to be kind of, um, you still have to pay attention to it, right?

Sertac Karaman (28:09.940)

Well, you know, um, people seem to use it.

Lex Fridman (28:12.540)

So it works for something.

Lex Fridman (28:14.420)

And so people, I think people are willing to pay for it.

Lex Fridman (28:16.660)

People are willing to buy it.

Sertac Karaman (28:17.660)

I think it, uh, it's, it's one of the other reasons why people buy a Tesla car.

Sertac Karaman (28:22.880)

Maybe one of those reasons is Elon Musk is the CEO and you know, he seems like a visionary

Sertac Karaman (28:26.900)

person.

Lex Fridman (28:27.900)

That's what people think.

Sertac Karaman (28:28.900)

He's a great person.

Lex Fridman (28:29.900)

And so that adds like 5k to the value of the car and then maybe another 5k is the autopilot

Sertac Karaman (28:34.140)

and, and you know, it's, it's useful.

Lex Fridman (28:35.740)

I mean, it's, um, useful in the sense that like people are using it.

Lex Fridman (28:40.940)

And so I can see Tesla and sure, of course they want to be visionary.

Sertac Karaman (28:45.500)

They want to kind of put out a certain approach and they may actually get there.

Sertac Karaman (28:48.620)

Um, but I think that there's also a primary benefit of doing all these updates and rolling

Sertac Karaman (28:54.860)

it out because, you know, people pay for it and it's, it's, you know, it's basic, you

Sertac Karaman (28:59.820)

know, demand, supply market and people like it.

Sertac Karaman (29:03.700)

They're happy to pay another 5k, 10k for that novelty or whatever that is, um, they, and

Sertac Karaman (29:09.940)

they use it.

Sertac Karaman (29:10.940)

It's not like they get it and they try it a couple of times as a novelty, but they use

Sertac Karaman (29:14.220)

it a lot of the time.

Lex Fridman (29:15.220)

And so I think that's what Tesla is doing.

Sertac Karaman (29:17.700)

It's actually pretty different.

Sertac Karaman (29:18.700)

Like they, they are on pretty orthogonal dimensions of what kind of things that they're building.

Sertac Karaman (29:23.160)

They are using the same AI engine.

Lex Fridman (29:25.220)

So it's very possible that, you know, they're both going to be, um, sort of one day, um,

Sertac Karaman (29:31.620)

kind of using a similar, almost like an internal internal combustion engine.

Sertac Karaman (29:34.900)

It's a very bad metaphor, but similar internal combustion engine, and maybe one of them is

Sertac Karaman (29:39.760)

building like a car.

Lex Fridman (29:41.200)

The other one is building a truck or something.

Lex Fridman (29:42.980)

So ultimately the use case is very different.

Lex Fridman (29:45.460)

So you, like I said, are one of the founders of Optimus, right?

Sertac Karaman (29:48.580)

Let's take a step back.

Lex Fridman (29:49.580)

That's one of the success stories in the autonomous vehicle space.

Sertac Karaman (29:54.260)

It's a great autonomous vehicle company.

Lex Fridman (29:56.580)

Let's go from the very beginning.

Lex Fridman (29:58.540)

What does it take to start an autonomous vehicle company?

Lex Fridman (30:02.380)

How do you go from idea to deploying vehicles like you are in a few, a bunch of places,

Lex Fridman (30:06.780)

including New York?

Sertac Karaman (30:08.020)

I would say that I think that, you know, what happened to us is it was, was the following.

Sertac Karaman (30:12.300)

I think, um, we realized a lot of kind of talk in the autonomous vehicle industry back

Lex Fridman (30:18.340)

in like 2014, even when we wanted to kind of get started.

Sertac Karaman (30:22.860)

Um, and, and I don't know, like I, I kind of, I would hear things like fully autonomous

Lex Fridman (30:29.420)

vehicles, two years from now, three years from now, I kind of never bought it.

Sertac Karaman (30:33.060)

Um, you know, I was a part of, um, MIT's urban challenge entry.

Lex Fridman (30:37.020)

Um, it kind of like, it has an interesting history.

Sertac Karaman (30:40.060)

So, um, I did in, in, in college and in high school, sort of a lot of mathematically oriented

Lex Fridman (30:46.220)

work.

Sertac Karaman (30:47.220)

I mean, I kind of, you know, at some point, uh, it kind of hit me.

Lex Fridman (30:50.940)

I wanted to build something.

Lex Fridman (30:52.780)

And so I came to MIT's mechanical engineering program and I now realize, I think my advisor

Sertac Karaman (30:57.740)

hired me because I could do like really good math, but I told him that, no, no, no, I want

Sertac Karaman (31:02.140)

to work on that urban challenge car.

Lex Fridman (31:04.380)

I want to build the autonomous car.

Lex Fridman (31:06.660)

And I think that was, that was kind of like a process where we really learned, I mean,

Lex Fridman (31:10.400)

what the challenges are and what kind of limitations are we up against, you know, like having the

Sertac Karaman (31:16.380)

limitations of computers or understanding human behavior, there's so many of these things.

Lex Fridman (31:21.940)

And I think it just kind of didn't.

Lex Fridman (31:23.900)

And so, so we said, Hey, you know, like, why don't we take a more like a market based approach?

Lex Fridman (31:29.520)

So we focus on a certain kind of market and we build a system for that.

Lex Fridman (31:35.020)

What we're building is not so much of like an autonomous vehicle only, I would say.

Lex Fridman (31:38.980)

So we build full autonomy into the vehicles.

Sertac Karaman (31:41.220)

But, you know, the way we kind of see it is that we think that the approach should actually

Lex Fridman (31:47.660)

involve humans operating them, not just, just not sitting in the vehicle.

Lex Fridman (31:52.980)

And I think today, what we have is today, we have one person operate one vehicle, no

Sertac Karaman (31:58.580)

matter what that vehicle, it could be a forklift, it could be a truck, it could be a car, whatever

Sertac Karaman (32:03.460)

that is.

Lex Fridman (32:04.640)

And we want to go from that to 10 people operate 50 vehicles.

Lex Fridman (32:09.420)

How do we do that?

Sertac Karaman (32:10.420)

If you're referring to a world of maybe perhaps teleoperation, so can you just say what it

Lex Fridman (32:16.820)

means for 10?

Lex Fridman (32:17.820)

It might be confusing for people listening.

Lex Fridman (32:19.700)

What does it mean for 10 people to control 50 vehicles?

Lex Fridman (32:23.180)

That's a good point.

Lex Fridman (32:24.180)

So I think it's, I very deliberately didn't call it teleoperation because what people

Sertac Karaman (32:28.720)

think then is that people think, away from the vehicle sits a person, sees like maybe

Sertac Karaman (32:35.280)

puts on goggles or something, VR and drives the car.

Lex Fridman (32:38.340)

So that's not at all what we mean, but we mean the kind of intelligence whereby humans

Sertac Karaman (32:44.180)

are in control, except in certain places, the vehicles can execute on their own.

Lex Fridman (32:49.500)

And so imagine like, like a room where people can see what the other vehicles are doing

Lex Fridman (32:54.740)

and everything.

Lex Fridman (32:56.660)

And you know, there will be some people who are more like, more like air traffic controllers,

Sertac Karaman (33:01.580)

call them like AV controllers.

Lex Fridman (33:04.600)

And so these AV controllers would actually see kind of like a whole map and they would

Sertac Karaman (33:09.220)

understand where vehicles are really confident and where they kind of need a little bit more

Lex Fridman (33:15.700)

help.

Lex Fridman (33:16.700)

And the help shouldn't be for safety.

Lex Fridman (33:19.240)

Help should be for efficiency.

Sertac Karaman (33:21.000)

Vehicles should be safe no matter what.

Sertac Karaman (33:22.920)

If you had zero people, they could be very safe, but they'd be going five miles an hour.

Lex Fridman (33:27.780)

And so if you want them to go around 25 miles an hour, then you need people to come in and,

Lex Fridman (33:32.700)

and for example, you know, the vehicle come to an intersection and the vehicle can say,

Sertac Karaman (33:38.620)

you know, I can wait.

Lex Fridman (33:39.940)

I can inch forward a little bit, show my intent, or I can turn left.

Lex Fridman (33:45.100)

And right now it's clear I can turn, I know that, but before you give me the go, I won't.

Lex Fridman (33:50.340)

And so that's one example.

Sertac Karaman (33:51.700)

This doesn't mean necessarily we're doing that actually.

Sertac Karaman (33:53.900)

I think, I think if you go down all the, all that much detail that every intersection you're

Sertac Karaman (33:59.500)

kind of expecting a person to press a button, then I don't think you'll get the efficiency

Lex Fridman (34:03.900)

benefits you want.

Sertac Karaman (34:04.900)

You need to be able to kind of go around and be able to do these things.

Sertac Karaman (34:07.820)

But, but I think you need people to be able to set high level behavior to vehicles.

Sertac Karaman (34:12.580)

That's the other thing with autonomous vehicles, you know, I think a lot of people kind of

Lex Fridman (34:15.140)

think about it as follows.

Sertac Karaman (34:16.140)

I mean, this happens with technology a lot.

Lex Fridman (34:18.100)

You know, you think, all right, so I know about cars and I heard robots.

Lex Fridman (34:23.440)

So I think how this is going to work out is that I'm going to buy a car, press a button

Lex Fridman (34:28.500)

and it's going to drive itself.

Lex Fridman (34:29.860)

And when is that going to happen?

Sertac Karaman (34:31.340)

You know, and people kind of tend to think about it that way, but when you think about

Lex Fridman (34:34.300)

what really happens is that something comes in in a way that you didn't even expect.

Sertac Karaman (34:40.100)

If asked, you might have said, I don't think I need that, or I don't think it should be

Sertac Karaman (34:43.860)

that and so on.

Lex Fridman (34:45.140)

And then, and then that, that becomes the next big thing, coding code.

Lex Fridman (34:49.380)

And so I think that this kind of different ways of humans operating vehicles could be

Lex Fridman (34:54.320)

really powerful.

Sertac Karaman (34:55.580)

I think that sooner than later, we might open our eyes up to a world in which you go around

Sertac Karaman (35:01.940)

walk in a mall and there's a bunch of security robots that are exactly operated in this way.

Sertac Karaman (35:06.660)

You go into a factory or a warehouse, there's a whole bunch of robots that are playing exactly

Lex Fridman (35:10.540)

in this way.

Sertac Karaman (35:11.540)

You go to a, you go to the Brooklyn Navy Yard, you see a whole bunch of autonomous vehicles,

Lex Fridman (35:17.020)

Optimus Ride, and they're operated maybe in this way.

Lex Fridman (35:21.060)

But I think people kind of don't see that.

Sertac Karaman (35:22.420)

I sincerely think that there's a possibility that we may almost see like a whole mushrooming

Sertac Karaman (35:28.620)

of this technology in all kinds of places that we didn't expect before.

Lex Fridman (35:33.500)

And that may be the real surprise.

Lex Fridman (35:35.900)

And then one day when your car actually drives itself, it may not be all that much of a surprise

Lex Fridman (35:40.380)

at all because you see it all the time.

Sertac Karaman (35:42.420)

You interact with them, you take the Optimus Ride, hopefully that's your choice.

Lex Fridman (35:47.860)

And then you hear a bunch of things, you go around, you interact with them.

Sertac Karaman (35:52.020)

I don't know.

Sertac Karaman (35:53.020)

Like you have a little delivery vehicle that goes around the sidewalks and delivers you

Sertac Karaman (35:56.380)

things and then you take it, it says thank you.

Lex Fridman (35:59.460)

And then you get used to that and one day your car actually drives itself and the regulation

Sertac Karaman (36:04.360)

goes by and you can hit the button of sleep and it wouldn't be a surprise at all.

Lex Fridman (36:08.660)

I think that may be the real reality.

Lex Fridman (36:10.820)

So there's going to be a bunch of applications that pop up around autonomous vehicles, some

Lex Fridman (36:17.860)

of which, maybe many of which we don't expect at all.

Lex Fridman (36:20.180)

So if we look at Optimus Ride, what do you think, you know, the viral application, the

Sertac Karaman (36:27.340)

one that like really works for people in mobility, what do you think Optimus Ride will connect

Lex Fridman (36:33.420)

with in the near future first?

Sertac Karaman (36:36.220)

I think that the first places that I like to target honestly is like these places where

Sertac Karaman (36:42.300)

transportation is required within an environment, like people typically call it geofence.

Lex Fridman (36:46.820)

So you can imagine like roughly two mile by two mile could be bigger, could be smaller

Sertac Karaman (36:51.780)

type of an environment.

Lex Fridman (36:53.300)

And there's a lot of these kinds of environments that are typically transportation deprived.

Sertac Karaman (36:57.340)

The Brooklyn Navy Yard that, you know, we're in today, we're in a few different places,

Lex Fridman (37:01.260)

but that was the one that was last publicized and that's a good example.

Lex Fridman (37:06.260)

So there's not a lot of transportation there and you wouldn't expect like, I don't know,

Sertac Karaman (37:11.060)

I think maybe operating an Uber there ends up being sort of a little too expensive or

Sertac Karaman (37:15.980)

when you compare it with operating Uber elsewhere, elsewhere becomes the priority and these places

Lex Fridman (37:23.340)

become totally transportation deprived.

Lex Fridman (37:26.220)

And then what happens is that, you know, people drive into these places and to go from point

Lex Fridman (37:29.940)

A to point B inside this place within that day, they use their cars.

Lex Fridman (37:35.460)

And so we end up building more parking for them to, for example, take their cars and

Lex Fridman (37:40.060)

go to the lunch place.

Lex Fridman (37:43.260)

And I think that one of the things that can be done is that, you know, you can put in

Sertac Karaman (37:46.940)

efficient, safe, sustainable transportation systems into these types of places first.

Lex Fridman (37:53.980)

And I think that, you know, you could deliver mobility in an affordable way, affordable,

Lex Fridman (37:59.540)

accessible, you know, sustainable way.

Lex Fridman (38:03.500)

But I think what also enables is that this kind of effort, money, area, land that we

Lex Fridman (38:08.860)

spend on parking, you could reclaim some of that.

Lex Fridman (38:12.940)

And that is on the order of like, even for a small environment like two mile by two mile,

Lex Fridman (38:17.640)

it doesn't have to be smack in the middle of New York.

Sertac Karaman (38:19.580)

I mean, anywhere else you're talking tens of millions of dollars.

Sertac Karaman (38:23.700)

If you're smack in the middle of New York, you're looking at billions of dollars of savings

Sertac Karaman (38:26.820)

just by doing that.

Lex Fridman (38:28.700)

And that's the economic part of it.

Lex Fridman (38:29.900)

And there's a societal part, right?

Lex Fridman (38:31.300)

I mean, just look around.

Sertac Karaman (38:32.420)

I mean the places that we live are like built for cars.

Sertac Karaman (38:38.500)

It didn't look like this just like a hundred years ago, like today, no one walks in the

Sertac Karaman (38:42.860)

middle of the street.

Lex Fridman (38:44.220)

It's for cars.

Sertac Karaman (38:45.860)

No one tells you that growing up, but you grow into that reality.

Lex Fridman (38:49.700)

And so sometimes they close the road.

Sertac Karaman (38:51.460)

It happens here, you know, like the celebration, they close the road.

Sertac Karaman (38:54.620)

Still people don't walk in the middle of the road, like just walk in the middle and people

Sertac Karaman (38:57.660)

don't.

Lex Fridman (38:58.660)

But I think it has so much impact, the car in the space that we have.

Lex Fridman (39:04.640)

And I think we talked about sustainability, livability.

Sertac Karaman (39:07.500)

I mean, ultimately these kinds of places that parking spots at the very least could change

Sertac Karaman (39:12.180)

into something more useful or maybe just like park areas, recreational.

Lex Fridman (39:16.380)

And so I think that's the first thing that we're targeting.

Lex Fridman (39:19.480)

And I think that we're getting like a really good response, both from an economic societal

Lex Fridman (39:23.620)

point of view, especially places that are a little bit forward looking.

Lex Fridman (39:27.900)

And like, for example, Brooklyn Navy Yard, they have tenants.

Lex Fridman (39:31.060)

There's distinct direct call like new lab.

Sertac Karaman (39:33.820)

It's kind of like an innovation center.

Lex Fridman (39:35.460)

There's a bunch of startups there.

Lex Fridman (39:36.460)

And so, you know, you get those kinds of people and, you know, they're really interested

Lex Fridman (39:40.060)

in sort of making that environment more livable.

Lex Fridman (39:44.460)

And these kinds of solutions that Optimus Ride provides almost kind of comes in and

Lex Fridman (39:49.020)

becomes that.

Lex Fridman (39:50.620)

And many of these places that are transportation deprived, you know, they have, they actually

Lex Fridman (39:56.100)

rent shuttles.

Lex Fridman (39:57.900)

And so, you know, you can ask anybody, the shuttle experience is like terrible.

Lex Fridman (40:03.420)

People hate shuttles.

Lex Fridman (40:05.100)

And I can tell you why.

Lex Fridman (40:06.100)

Because, you know, like the driver is very expensive in a shuttle business.

Lex Fridman (40:11.180)

So what makes sense is to attach 20, 30 seats to a driver.

Lex Fridman (40:15.660)

And a lot of people have this misconception.

Sertac Karaman (40:17.300)

They think that shuttles should be big.

Lex Fridman (40:19.300)

Sometimes we get that at Optimus Ride.

Sertac Karaman (40:20.380)

We tell them, we're going to give you like four seaters, six seaters.

Lex Fridman (40:23.200)

And we get asked like, how about like 20 seaters?

Sertac Karaman (40:25.100)

I'm like, you know, you don't need 20 seaters.

Sertac Karaman (40:27.440)

You want to split up those seats so that they can travel faster and the transportation delays

Sertac Karaman (40:32.220)

would go down.

Lex Fridman (40:33.220)

That's what you want.

Sertac Karaman (40:34.340)

If you make it big, not only you will get delays in transportation, but you won't have

Lex Fridman (40:39.200)

an agile vehicle.

Sertac Karaman (40:40.420)

It will take a long time to speed up, slow down and so on.

Lex Fridman (40:44.220)

You need to climb up to the thing.

Lex Fridman (40:45.900)

So it's kind of like really hard to interact with.

Lex Fridman (40:48.820)

And scheduling too, perhaps when you have more smaller vehicles, it becomes closer to

Sertac Karaman (40:53.020)

Uber where you can actually get a personal, I mean, just the logistics of getting the

Lex Fridman (40:58.420)

vehicle to you becomes easier when you have a giant shuttle.

Sertac Karaman (41:02.900)

There's fewer of them and it probably goes on a route, a specific route that is supposed

Lex Fridman (41:07.300)

to hit.

Lex Fridman (41:08.300)

And when you go on a specific route and all seats travel together versus, you know, you

Lex Fridman (41:13.900)

have a whole bunch of them.

Sertac Karaman (41:14.900)

You can imagine the route you can still have, but you can imagine you split up the seats

Lex Fridman (41:19.560)

and instead of, you know, them traveling, like, I don't know, a mile apart, they could

Sertac Karaman (41:24.140)

be like, you know, half a mile apart if you split them into two.

Sertac Karaman (41:28.300)

That basically would mean that your delays, when you go out, you won't wait for them for

Sertac Karaman (41:34.060)

a long time.

Lex Fridman (41:35.060)

And that's one of the main reasons, or you don't have to climb up.

Sertac Karaman (41:37.140)

The other thing is that I think if you split them up in a nice way, and if you can actually

Lex Fridman (41:41.700)

know where people are going to be somehow, you don't even need the app.

Lex Fridman (41:46.020)

A lot of people ask us the app, we say, why don't you just walk into the vehicle?

Lex Fridman (41:50.780)

How about you just walk into the vehicle, it recognizes who you are and it gives you

Sertac Karaman (41:54.180)

a bunch of options of places that you go and you just kind of go there.

Lex Fridman (41:57.300)

I mean, people kind of also internalize the apps.

Sertac Karaman (42:01.140)

Everybody needs an app.

Lex Fridman (42:02.140)

It's like, you don't need an app.

Sertac Karaman (42:03.140)

You just walk into the thing.

Lex Fridman (42:05.540)

But I think one of the things that, you know, we really try to do is to take that shuttle

Sertac Karaman (42:10.060)

experience that no one likes and tilt it into something that everybody loves.

Lex Fridman (42:14.640)

And so I think that's another important thing.

Sertac Karaman (42:17.500)

I would like to say that carefully, just like teleoperation, like we don't do shuttles.

Sertac Karaman (42:21.820)

You know, we're really kind of thinking of this as a system or a network that we're designing.

Lex Fridman (42:28.580)

But ultimately, we go to places that would normally rent a shuttle service that people

Lex Fridman (42:33.080)

wouldn't like as much and we want to tilt it into something that people love.

Lex Fridman (42:37.500)

So you've mentioned this earlier, but how many Optimus ride vehicles do you think would

Sertac Karaman (42:42.820)

be needed for any person in Boston or New York, if they step outside, there will be,

Sertac Karaman (42:50.860)

this is like a mathematical question, there'll be two Optimus ride vehicles within line of

Lex Fridman (42:55.300)

sight.

Sertac Karaman (42:56.300)

Is that the right number to, well, at least one.

Lex Fridman (42:58.820)

For example, that's the density.

Lex Fridman (43:01.860)

So meaning that if you see one vehicle, you look around, you see another one too.

Lex Fridman (43:07.260)

Imagine like, you know, Tesla would tell you they collect a lot of data.

Lex Fridman (43:11.800)

Do you see that with Tesla?

Lex Fridman (43:12.940)

Like you just walk around and you look around, you see Tesla?

Sertac Karaman (43:16.060)

Probably not.

Lex Fridman (43:17.060)

Very specific areas of California, maybe.

Sertac Karaman (43:19.940)

You're right.

Sertac Karaman (43:21.380)

Like there's a couple of zip codes that, you know, but I think that's kind of important

Sertac Karaman (43:25.620)

because you know, like maybe the couple of zip codes, the one thing that we kind of depend

Sertac Karaman (43:29.800)

on and I'll get to your question in a second, but now like we're taking a lot of tensions

Sertac Karaman (43:33.460)

today.

Lex Fridman (43:34.460)

And so I think that this is actually important.

Sertac Karaman (43:38.460)

People call this data density or data velocity.

Lex Fridman (43:41.040)

So it's very good to collect data in a way that, you know, you see the same place so

Sertac Karaman (43:46.220)

many times.

Sertac Karaman (43:47.220)

Like you can drive 10,000 miles around the country or you drive 10,000 miles in a confined

Sertac Karaman (43:53.300)

environment.

Lex Fridman (43:54.300)

You'll see the same intersection hundreds of times.

Lex Fridman (43:56.700)

And when it comes to predicting what people are going to do in that specific intersection,

Sertac Karaman (44:01.020)

you become really good at it versus if you draw in like 10,000 miles around the country,

Sertac Karaman (44:05.380)

you've seen that only once.

Lex Fridman (44:06.900)

And so trying to predict what people do becomes hard.

Lex Fridman (44:10.480)

And I think that, you know, you said what is needed, it's tens of thousands of vehicles.

Lex Fridman (44:14.400)

You know, you really need to be like a specific fractional vehicle.

Sertac Karaman (44:17.900)

Like for example, in good times in Singapore, you can go and you can just grab a cab and

Lex Fridman (44:23.500)

they are like, you know, 10%, 20% of traffic, those taxis.

Sertac Karaman (44:29.300)

Ultimately that's where you need to get to.

Lex Fridman (44:31.940)

So that, you know, you get to a certain place where you really, the benefits really kick

Sertac Karaman (44:36.620)

off in like orders of magnitude type of a point.

Lex Fridman (44:40.780)

But once you get there, you actually get the benefits.

Lex Fridman (44:43.540)

And you can certainly carry people.

Sertac Karaman (44:44.820)

I think that's one of the things people really don't like to wait for themselves.

Lex Fridman (44:51.020)

But for example, they can wait a lot more for the goods if they order something.

Lex Fridman (44:55.740)

Like you're sitting at home and you want to wait half an hour.

Sertac Karaman (44:57.980)

That sounds great.

Lex Fridman (44:58.980)

People will say it's great.

Sertac Karaman (44:59.980)

You want to, you're going to take a cab, you're waiting half an hour.

Lex Fridman (45:02.600)

Like that's crazy.

Sertac Karaman (45:03.600)

You don't want to wait that much.

Lex Fridman (45:06.100)

But I think, you know, you can, I think really get to a point where the system at peak times

Sertac Karaman (45:11.360)

really focuses on kind of transporting humans around.

Lex Fridman (45:14.380)

And then it's really, it's a good fraction of the traffic to the point where, you know,

Sertac Karaman (45:18.740)

you go, you look around and there's something there and you just kind of basically get in

Lex Fridman (45:23.040)

there and it's already waiting for you or something like that.

Lex Fridman (45:27.280)

And then you take it.

Lex Fridman (45:28.540)

If you do it at that scale, like today, for instance, Uber, if you talk to a driver, right?

Sertac Karaman (45:35.780)

I mean, Uber takes a certain cut.

Lex Fridman (45:37.380)

It's a small cut.

Sertac Karaman (45:39.420)

Or drivers would argue that it's a large cut, but you know, it's when you look at the grand

Sertac Karaman (45:44.460)

scheme of things, most of that money that you pay Uber kind of goes to the driver.

Lex Fridman (45:50.380)

And if you talk to the driver, the driver will claim that most of it is their time.

Lex Fridman (45:54.620)

You know, it's not spent on gas.

Sertac Karaman (45:56.620)

They think it's not spent on the car per se as much.

Lex Fridman (46:01.300)

It's like their time.

Lex Fridman (46:02.980)

And if you didn't have a person driving, or if you're in a scenario where, you know, like

Sertac Karaman (46:07.180)

0.1 person is driving the car, a fraction of a person is kind of operating the car because

Sertac Karaman (46:14.460)

you know, you want to operate several.

Sertac Karaman (46:17.220)

If you're in that situation, you realize that the internal combustion engine type of cars

Sertac Karaman (46:21.520)

are very inefficient.

Lex Fridman (46:23.180)

You know, we build them to go on highways, they pass crash tests.

Sertac Karaman (46:26.340)

They're like really heavy.

Sertac Karaman (46:27.820)

They really don't need to be like 25 times the weight of its passengers or, you know,

Sertac Karaman (46:32.660)

like area wise and so on.

Lex Fridman (46:35.960)

But if you get through those inefficiencies and if you really build like urban cars and

Sertac Karaman (46:39.900)

things like that, I think the economics really starts to check out.

Sertac Karaman (46:43.380)

Like to the point where, I mean, I don't know, you may be able to get into a car and it may

Sertac Karaman (46:47.960)

be less than a dollar to go from A to B. As long as you don't change your destination,

Lex Fridman (46:52.620)

you just pay 99 cents and go there.

Sertac Karaman (46:55.760)

If you share it, if you take another stop somewhere, it becomes a lot better.

Sertac Karaman (47:00.460)

You know, these kinds of things, at least for models, at least for mathematics and theory,

Sertac Karaman (47:05.140)

they start to really check out.

Lex Fridman (47:07.420)

So I think it's really exciting what Optimus Ride is doing in terms of it feels the most

Sertac Karaman (47:12.220)

reachable, like it'll actually be here and have an impact.

Lex Fridman (47:15.620)

Yeah, that is the idea.

Lex Fridman (47:17.700)

And if we contrast that, again, we'll go back to our old friends, Waymo and Tesla.

Lex Fridman (47:23.760)

So Waymo seems to have sort of technically similar approaches as Optimus Ride, but a

Sertac Karaman (47:34.340)

different, they're not as interested as having impact today.

Sertac Karaman (47:41.180)

They have a longer term sort of investments, almost more of a research project still, meaning

Sertac Karaman (47:47.740)

they're trying to solve, as far as I understand, maybe you can differentiate, but they seem

Sertac Karaman (47:53.500)

to want to do more unrestricted movement, meaning move from A to B where A to B is all

Sertac Karaman (48:00.340)

over the place versus Optimus Ride is really nicely geofenced and really sort of established

Lex Fridman (48:07.860)

mobility in a particular environment before you expand it.

Lex Fridman (48:11.580)

And then Tesla is like the complete opposite, which is, you know, the entirety of the world

Lex Fridman (48:17.800)

actually is going to be automated.

Sertac Karaman (48:21.220)

Highway driving, urban driving, every kind of driving, you know, you kind of creep up

Lex Fridman (48:26.900)

to it by incrementally improving the capabilities of the autopilot system.

Lex Fridman (48:33.380)

So when you contrast all of these, and on top of that, let me throw a question that

Lex Fridman (48:37.920)

nobody likes, but is a timeline.

Sertac Karaman (48:42.060)

When do you think each of these approaches, loosely speaking, nobody can predict the future,

Lex Fridman (48:47.740)

will see mass deployment?

Lex Fridman (48:49.900)

So Elon Musk predicts the craziest approach is, I've heard figures like at the end of

Lex Fridman (48:56.740)

this year, right?

Lex Fridman (48:58.700)

So that's probably wildly inaccurate, but how wildly inaccurate is it?

Lex Fridman (49:06.900)

I mean, first thing to lay out, like everybody else, it's really hard to guess.

Sertac Karaman (49:11.500)

I mean, I don't know where Tesla can look at or Elon Musk can look at and say, hey,

Lex Fridman (49:18.460)

you know, it's the end of this year.

Sertac Karaman (49:19.820)

I mean, I don't know what you can look at.

Sertac Karaman (49:22.020)

You know, even the data that, I mean, if you look at the data, even kind of trying to extrapolate

Sertac Karaman (49:30.860)

the end state without knowing what exactly is going to go, especially for like a machine

Lex Fridman (49:34.940)

learning approach.

Sertac Karaman (49:35.940)

I mean, it's just kind of very hard to predict.

Lex Fridman (49:39.780)

But I do think the following does happen.

Sertac Karaman (49:41.540)

I think a lot of people, you know, what they do is that there's something that I called

Lex Fridman (49:46.740)

a couple times time dilation in technology prediction happens.

Sertac Karaman (49:51.060)

Let me try to describe a little bit.

Lex Fridman (49:53.220)

There's a lot of things that are so far ahead, people think they're close.

Lex Fridman (49:57.840)

And there's a lot of things that are actually close.

Lex Fridman (50:00.140)

People think it's far ahead.

Sertac Karaman (50:02.020)

People try to kind of look at a whole landscape of technology development, admittedly, it's

Lex Fridman (50:07.940)

chaos.

Sertac Karaman (50:08.940)

Anything can happen in any order at any time.

Lex Fridman (50:10.760)

And there's a whole bunch of things in there.

Sertac Karaman (50:12.260)

People take it, clamp it, and put it into the next three years.

Lex Fridman (50:17.060)

And so then what happens is that there's some things that maybe can happen by the end of

Sertac Karaman (50:21.500)

the year or next year and so on.

Lex Fridman (50:23.580)

And they push that into like few years ahead, because it's just hard to explain.

Lex Fridman (50:28.100)

And there are things that are like, we're looking at 20 years more, maybe, you know,

Lex Fridman (50:33.820)

hopefully in my lifetime type of things, because, you know, we don't know.

Sertac Karaman (50:37.620)

I mean, we don't know how hard it is even.

Lex Fridman (50:40.660)

Like that's a problem.

Sertac Karaman (50:41.660)

We don't know like if some of these problems are actually AI complete, like, we have no

Lex Fridman (50:45.900)

idea what's going on.

Lex Fridman (50:48.120)

And you know, we take all of that and then we clump it.

Lex Fridman (50:51.860)

And then we say three years from now.

Lex Fridman (50:55.500)

And then some of us are more optimistic.

Lex Fridman (50:57.180)

So they're shooting at the end of the year and some of us are more realistic.

Sertac Karaman (51:00.860)

They say like five years, but you know, we all, I think it's just hard to know.

Lex Fridman (51:06.340)

And I think trying to predict like products ahead two, three years, it's hard to know

Sertac Karaman (51:12.900)

in the following sense.

Sertac Karaman (51:14.020)

You know, like we typically say, okay, this is a technology company, but sometimes, sometimes

Sertac Karaman (51:19.300)

really you're trying to build something where the technology does, like there's a technology

Sertac Karaman (51:22.500)

gap, you know, like, and Tesla had that with electric vehicles, you know, like when they

Sertac Karaman (51:29.040)

first started, they would look at a chart much like a Moore's law type of chart.

Lex Fridman (51:33.540)

And they would just kind of extrapolate that out and they'd say, we want to be here.

Lex Fridman (51:37.380)

What's the technology to get that?

Lex Fridman (51:38.900)

We don't know.

Sertac Karaman (51:39.900)

It goes like this.

Lex Fridman (51:40.900)

We're just going to, you know, keep going with AI that goes into the cars.

Sertac Karaman (51:46.540)

We don't even have that.

Sertac Karaman (51:47.540)

Like we don't, we can't, I mean, what can you quantify, like what kind of chart are

Lex Fridman (51:51.300)

you looking at?

Lex Fridman (51:52.640)

You know?

Lex Fridman (51:53.640)

But so, but so I think when there's that technology gap, it's just kind of really hard to predict.

Lex Fridman (51:58.380)

So now I realize I talked like five minutes and avoid your question.

Sertac Karaman (52:01.780)

I didn't tell you anything about that and it was very skillfully done.

Lex Fridman (52:05.700)

That was very well done.

Lex Fridman (52:07.180)

And I don't think you, I think you've actually argued that it's not a use, even any answer

Lex Fridman (52:10.860)

you provide now is not that useful.

Sertac Karaman (52:12.620)

It's going to be very hard.

Sertac Karaman (52:13.980)

There's one thing that I really believe in and, um, and you know, this is not my idea

Lex Fridman (52:17.900)

and it's been, you know, discussed several times, but, but this, um, this, this kind

Sertac Karaman (52:22.500)

of like something like a startup, um, or, or a kind of an innovative company, um, including

Sertac Karaman (52:29.160)

definitely may one, may Waymo, Tesla, maybe even some of the other big companies that

Lex Fridman (52:33.140)

are kind of trying things.

Sertac Karaman (52:34.860)

This kind of like iterated learning is very important.

Sertac Karaman (52:38.460)

The fact that we're over there and we're trying things and so on, I think that's, um, that

Sertac Karaman (52:43.580)

that's important.

Lex Fridman (52:44.580)

We try to understand.

Sertac Karaman (52:45.580)

And, and I think that, you know, the code in code Silicon Valley has done that with

Lex Fridman (52:49.980)

business models pretty well.

Lex Fridman (52:52.300)

And now I think we're trying to get to do it, but there's a literal technology gap.

Sertac Karaman (52:56.900)

I mean, before, like, you know, you're trying to build, I'm not trying to, you know, I think

Sertac Karaman (53:01.140)

these companies are building great technology to, for example, enable internet search to

Lex Fridman (53:06.500)

do it so quickly.

Lex Fridman (53:07.660)

And that kind of didn't, didn't, wasn't there so much, but at least like it was a kind of

Lex Fridman (53:11.860)

a technology that you could predict to some degree and so on.

Lex Fridman (53:14.620)

And now we're just kind of trying to build, you know, things that it's kind of hard to

Lex Fridman (53:18.300)

quantify what kind of a metric are we looking at?

Lex Fridman (53:21.740)

So psychologically as a sort of a, as a leader of graduate students and at Optimus ride a

Sertac Karaman (53:28.700)

bunch of brilliant engineers, just curiosity, psychologically, do you think it's good to

Sertac Karaman (53:35.260)

think that, you know, whatever technology gap we're talking about can be closed by the

Lex Fridman (53:42.080)

end of the year or do you, you know, cause we don't know.

Lex Fridman (53:46.260)

So the way, do you want to say that everything is going to improve exponentially to yourself

Lex Fridman (53:54.480)

and to others around you as a leader, or do you want to be more sort of maybe not cynical,

Lex Fridman (54:01.580)

but I don't want to use realistic cause it's hard to predict, but yeah, maybe more cynical,

Lex Fridman (54:07.140)

pessimistic about the ability to close that gap.

Sertac Karaman (54:11.060)

Yeah.

Sertac Karaman (54:12.060)

I think that, you know, going back, I think that iterated learning is like key that, you

Sertac Karaman (54:16.140)

know, you're out there, you're running experiments to learn.

Lex Fridman (54:19.380)

And that doesn't mean sort of like, you know, like, like your Optimus ride, you're kind

Sertac Karaman (54:22.780)

of doing something, but like in an environment, but like what Tesla is doing, I think is also

Lex Fridman (54:28.060)

kind of like this, this kind of notion.

Sertac Karaman (54:30.380)

And, and, you know, people can go around and say like, you know, this year, next year,

Lex Fridman (54:34.260)

the other year and so on.

Sertac Karaman (54:35.340)

But, but I think that the nice thing about it is that they're out there, they're pushing

Lex Fridman (54:39.340)

this technology in.

Sertac Karaman (54:40.900)

I think what they should do more of, I think that kind of informed people about what kind

Lex Fridman (54:45.920)

of technology that they're providing, you know, the good and the bad.

Lex Fridman (54:48.580)

And then, you know, not just sort of, you know, it works very well, but I think, you

Sertac Karaman (54:52.820)

know, I'm not saying they're not doing bad and informing, I think they're, they're kind

Sertac Karaman (54:56.500)

of trying, they, you know, they put up certain things or at the very least YouTube videos

Sertac Karaman (55:00.260)

comes out on, on how the summon function works every now and then, and, and, you know, people

Sertac Karaman (55:04.420)

get informed and so that, that kind of cycle continues, but I, you know, I, I admire it.

Lex Fridman (55:10.180)

I think they're kind of go out there and they, they do great things.

Sertac Karaman (55:13.100)

They do their own kind of experiment.

Sertac Karaman (55:14.620)

I think we do our own and I think we're closing some similar technology gaps, but some also

Sertac Karaman (55:20.680)

some are orthogonal as well.

Sertac Karaman (55:22.540)

You know, I think like, like we talked about, you know, people being remote, like it's something

Sertac Karaman (55:27.020)

or in the kind of environments that we're in or think about a Tesla car, maybe, maybe

Lex Fridman (55:31.400)

you can enable it one day.

Sertac Karaman (55:32.780)

Like there's, you know, low traffic, like you're kind of the stop on go motion, you

Sertac Karaman (55:36.460)

just hit the button and the, you can release, or maybe there's another lane that you can

Sertac Karaman (55:41.020)

pass into, you go in that.

Lex Fridman (55:42.260)

I think they can enable these kinds of, I believe it.

Lex Fridman (55:45.820)

And so I think that that part, that is really important and that is really key.

Lex Fridman (55:51.500)

And beyond that, I think, you know, when is it exactly going to happen and, and, and so

Sertac Karaman (55:57.060)

on.

Lex Fridman (55:58.060)

I mean it's like I said, it's very hard to predict.

Lex Fridman (56:02.940)

And I would, I would imagine that it would be good to do some sort of like a, like a

Sertac Karaman (56:07.460)

one or two year plan when it's a little bit more predictable that, you know, the technology

Sertac Karaman (56:12.100)

gaps you close and, and the, and the kind of sort of product that would ensue.

Lex Fridman (56:18.060)

So I know that from Optimus ride or, you know, other companies that I get involved in.

Sertac Karaman (56:22.820)

I mean, at some point you find yourself in a situation where you're trying to build a

Sertac Karaman (56:27.940)

product and, and people are investing in that, in that, you know, building effort and those

Sertac Karaman (56:35.300)

investors that they do want to know as they compare the investments they want to make,

Lex Fridman (56:39.940)

they do want to know what happens in the next one or two years.

Lex Fridman (56:42.260)

And I think that's good to communicate that.

Lex Fridman (56:44.720)

But I think beyond that, it becomes, it becomes a vision that we want to get to someday and

Sertac Karaman (56:48.820)

saying five years, 10 years, I don't think it means anything.

Lex Fridman (56:52.460)

But iterative learning is key to do and learn.

Sertac Karaman (56:56.140)

I think that is key.

Sertac Karaman (56:57.140)

You know, I got to sort of throw back right at you criticism in terms of, you know, like

Sertac Karaman (57:03.820)

Tesla or somebody communicating, you know, how someone works and so on.

Sertac Karaman (57:07.740)

I got a chance to visit Optimus ride and you guys are doing some awesome stuff and yet

Sertac Karaman (57:12.700)

the internet doesn't know about it.

Lex Fridman (57:14.700)

So you should also communicate more showing off, you know, showing off some of the awesome

Sertac Karaman (57:20.020)

stuff, the stuff that works and stuff that doesn't work.

Sertac Karaman (57:22.860)

I mean, it's just the stuff I saw with the tracking of different objects and pedestrians.

Lex Fridman (57:27.300)

So I mean, incredible stuff going on there.

Sertac Karaman (57:30.420)

Maybe it's just the nerd in me, but I think the world would love to see that kind of stuff.

Sertac Karaman (57:34.940)

Yeah.

Lex Fridman (57:35.940)

That's, that's well taken.

Sertac Karaman (57:36.940)

Um, you know, I, I should say that it's not like, you know, we, we, we weren't able to,

Sertac Karaman (57:41.540)

I think we made a decision at some point, um, that decision did involve me quite a bit

Sertac Karaman (57:46.860)

on kind of, um, uh, sort of doing this in kind of coding code stealth mode for a bit.

Lex Fridman (57:53.140)

Um, but I think that, you know, we'll, we'll open it up quite a lot more.

Lex Fridman (57:56.940)

And I think that we are also at Optimus ride kind of hitting, um, when you have new era,

Sertac Karaman (58:02.540)

um, you know, we're, we're, we're big now, we're doing a lot of interesting things and

Sertac Karaman (58:06.820)

I think, you know, some of the deployments that we've kind of announced were some of

Sertac Karaman (58:10.340)

the first bits, bits of, um, information that we kind of put out into the world.

Sertac Karaman (58:16.260)

We'll also put out our technology, a lot of the things that we've been developing is really

Lex Fridman (58:20.100)

amazing.

Lex Fridman (58:21.100)

And then, you know, we're, we're gonna, we're gonna start putting that out now.

Sertac Karaman (58:24.980)

We're especially interested in sort of like, um, being able to work with the best people.

Lex Fridman (58:28.580)

And I think, and I think it's, it's good to not just kind of show them when they come

Sertac Karaman (58:32.740)

to our office for an interview, but just put it out there in terms of like, you know, get

Sertac Karaman (58:36.500)

people excited about what we're doing.

Lex Fridman (58:39.220)

So on the autonomous vehicle space, let me ask one last question.

Lex Fridman (58:43.780)

So Elon Musk famously said that lighter is a crutch.

Lex Fridman (58:47.460)

So I've talked to a bunch of people about it, got to ask you, you use that crutch quite

Sertac Karaman (58:52.860)

a bit in the DARPA days.

Sertac Karaman (58:55.220)

So, uh, uh, you know, and his, his idea in general, sort of, you know, more provocative

Lex Fridman (59:01.860)

and fun, I think than a technical discussion, but the idea is that camera based, primarily

Sertac Karaman (59:08.240)

camera based systems is going to be what defines the future of autonomous vehicles.

Lex Fridman (59:14.140)

So what do you think of this idea?

Lex Fridman (59:16.100)

Lighter is a crutch versus primarily, uh, camera based systems.

Sertac Karaman (59:21.380)

First things first, I think, you know, I'm a big believer in just camera based autonomous

Lex Fridman (59:27.340)

vehicle systems.

Sertac Karaman (59:28.340)

Um, I think that, you know, you can put in a lot of autonomy and, and you can do great

Lex Fridman (59:33.180)

things.

Sertac Karaman (59:34.180)

And, and it's, it's, it's very possible that at the time scales, like I said, we can't

Sertac Karaman (59:37.860)

predict 20 years from now, like you may be able to do, do things that we're doing today

Sertac Karaman (59:43.900)

only with LIDAR and then you may be able to do them just with cameras.

Lex Fridman (59:48.140)

And I think that, um, you know, you, you can just, um, I, I, I think that I will put my

Sertac Karaman (59:53.720)

name on it too.

Sertac Karaman (59:54.720)

You know, there will be a time when you can only use cameras and you'll be fine.