Tuomas Sandholm: Poker and Game Theory - Lex Fridman 知识图谱

Lex Fridman (01:23.560)

for imperfect information game solving.

Lex Fridman (01:26.440)

And this is a game that's actually played by humans.

Lex Fridman (01:30.960)

You don't see that much on TV or casinos

Tuomas Sandholm (01:33.960)

because well, for various reasons,

Lex Fridman (01:36.160)

but you do see it in some expert level casinos

Lex Fridman (01:40.240)

and you see it in the best poker movies of all time.

Lex Fridman (01:43.080)

It's actually an event in the World Series of Poker,

Lex Fridman (01:45.720)

but mostly it's played online

Lex Fridman (01:48.200)

and typically for pretty big sums of money.

Lex Fridman (01:50.880)

And this is a game that usually only experts play.

Lex Fridman (01:54.560)

So if you go to your home game on a Friday night,

Tuomas Sandholm (01:58.720)

it probably is not gonna be Heads Up No Limit Texas Holdem.

Lex Fridman (02:01.280)

It might be No Limit Texas Holdem in some cases,

Lex Fridman (02:04.640)

but typically for a big group and it's not as competitive.

Lex Fridman (02:08.720)

While Heads Up means it's two players.

Lex Fridman (02:10.520)

So it's really like me against you.

Lex Fridman (02:13.360)

Am I better or are you better?

Tuomas Sandholm (02:14.680)

Much like chess or go in that sense,

Lex Fridman (02:17.520)

but an imperfect information game,

Tuomas Sandholm (02:19.520)

which makes it much harder because I have to deal

Lex Fridman (02:21.520)

with issues of you knowing things that I don't know

Lex Fridman (02:25.560)

and I know things that you don't know

Lex Fridman (02:27.200)

instead of pieces being nicely laid on the board

Tuomas Sandholm (02:29.720)

for both of us to see.

Lex Fridman (02:31.120)

So in Texas Holdem, there's two cards

Tuomas Sandholm (02:34.840)

that you only see that belong to you.

Lex Fridman (02:37.440)

Yeah. And there is,

Tuomas Sandholm (02:38.520)

they gradually lay out some cards

Lex Fridman (02:40.400)

that add up overall to five cards that everybody can see.

Tuomas Sandholm (02:44.080)

Yeah. So the imperfect nature

Lex Fridman (02:45.720)

of the information is the two cards

Tuomas Sandholm (02:47.560)

that you're holding in your hand.

Lex Fridman (02:48.400)

Up front, yeah.

Lex Fridman (02:49.380)

So as you said, you first get two cards

Lex Fridman (02:51.840)

in private each and then there's a betting round.

Tuomas Sandholm (02:55.200)

Then you get three cards in public on the table.

Lex Fridman (02:58.320)

Then there's a betting round.

Tuomas Sandholm (02:59.240)

Then you get the fourth card in public on the table.

Lex Fridman (03:01.680)

There's a betting round.

Tuomas Sandholm (03:02.580)

Then you get the 5th card on the table.

Lex Fridman (03:04.920)

There's a betting round.

Lex Fridman (03:05.760)

So there's a total of four betting rounds

Lex Fridman (03:07.480)

and four tranches of information revelation if you will.

Tuomas Sandholm (03:11.140)

The only the first tranche is private

Lex Fridman (03:14.120)

and then it's public from there.

Lex Fridman (03:16.520)

And this is probably by far the most popular game in AI

Lex Fridman (03:24.040)

and just the general public

Tuomas Sandholm (03:26.380)

in terms of imperfect information.

Lex Fridman (03:28.400)

So that's probably the most popular spectator game

Lex Fridman (03:32.520)

to watch, right?

Lex Fridman (03:33.400)

So, which is why it's a super exciting game to tackle.

Lex Fridman (03:37.260)

So it's on the order of chess, I would say,

Lex Fridman (03:40.480)

in terms of popularity, in terms of AI setting it

Tuomas Sandholm (03:43.680)

as the bar of what is intelligence.

Lex Fridman (03:46.360)

So in 2017, Labratus, how do you pronounce it?

Tuomas Sandholm (03:50.400)

Labratus.

Lex Fridman (03:51.220)

Labratus.

Tuomas Sandholm (03:52.060)

Labratus beats.

Lex Fridman (03:52.900)

A little Latin there.

Tuomas Sandholm (03:54.080)

A little bit of Latin.

Lex Fridman (03:55.520)

Labratus beats a few, four expert human players.

Lex Fridman (04:01.040)

Can you describe that event?

Lex Fridman (04:03.080)

What you learned from it?

Lex Fridman (04:04.060)

What was it like?

Lex Fridman (04:04.900)

What was the process in general

Lex Fridman (04:06.860)

for people who have not read the papers and the study?

Lex Fridman (04:09.960)

Yeah, so the event was that we invited

Tuomas Sandholm (04:12.920)

four of the top 10 players,

Lex Fridman (04:14.840)

with these specialist players in Heads Up No Limit,

Tuomas Sandholm (04:17.080)

Texas Holden, which is very important

Lex Fridman (04:19.080)

because this game is actually quite different

Tuomas Sandholm (04:21.400)

than the multiplayer version.

Lex Fridman (04:23.900)

We brought them in to Pittsburgh

Tuomas Sandholm (04:25.680)

to play at the Reverse Casino for 20 days.

Lex Fridman (04:28.920)

We wanted to get 120,000 hands in

Tuomas Sandholm (04:31.840)

because we wanted to get statistical significance.

Lex Fridman (04:36.160)

So it's a lot of hands for humans to play,

Tuomas Sandholm (04:39.040)

even for these top pros who play fairly quickly normally.

Lex Fridman (04:42.840)

So we couldn't just have one of them play so many hands.

Tuomas Sandholm (04:46.400)

20 days, they were playing basically morning to evening.

Lex Fridman (04:50.400)

And I raised 200,000 as a little incentive for them to play.

Lex Fridman (04:55.660)

And the setting was so that they didn't all get 50,000.

Lex Fridman (05:01.080)

We actually paid them out

Tuomas Sandholm (05:02.640)

based on how they did against the AI each.

Lex Fridman (05:05.480)

So they had an incentive to play as hard as they could,

Tuomas Sandholm (05:09.440)

whether they're way ahead or way behind

Lex Fridman (05:11.160)

or right at the mark of beating the AI.

Lex Fridman (05:13.760)

And you don't make any money, unfortunately.

Lex Fridman (05:16.000)

Right, no, we can't make any money.

Lex Fridman (05:17.920)

So originally, a couple of years earlier,

Lex Fridman (05:20.320)

I actually explored whether we could actually play for money

Tuomas Sandholm (05:24.080)

because that would be, of course, interesting as well,

Lex Fridman (05:28.000)

to play against the top people for money.

Lex Fridman (05:29.520)

But the Pennsylvania Gaming Board said no, so we couldn't.

Lex Fridman (05:33.040)

So this is much like an exhibit,

Tuomas Sandholm (05:36.400)

like for a musician or a boxer or something like that.

Lex Fridman (05:39.760)

Nevertheless, they were keeping track of the money

Lex Fridman (05:41.600)

and brought us close to $2 million, I think.

Lex Fridman (05:48.200)

So if it was for real money, if you were able to earn money,

Tuomas Sandholm (05:51.840)

that was a quite impressive and inspiring achievement.

Lex Fridman (05:55.360)

Just a few details, what were the players looking at?

Lex Fridman (05:59.280)

Were they behind a computer?

Lex Fridman (06:00.460)

What was the interface like?

Tuomas Sandholm (06:02.080)

Yes, they were playing much like they normally do.

Lex Fridman (06:05.240)

These top players, when they play this game,

Tuomas Sandholm (06:07.200)

they play mostly online.

Lex Fridman (06:08.680)

So they're used to playing through a UI.

Lex Fridman (06:11.640)

And they did the same thing here.

Lex Fridman (06:13.280)

So there was this layout.

Tuomas Sandholm (06:14.520)

You could imagine there's a table on a screen.

Lex Fridman (06:17.920)

There's the human sitting there,

Lex Fridman (06:20.080)

and then there's the AI sitting there.

Lex Fridman (06:21.720)

And the screen shows everything that's happening.

Tuomas Sandholm (06:24.560)

The cards coming out and shows the bets being made.

Lex Fridman (06:27.480)

And we also had the betting history for the human.

Lex Fridman (06:29.940)

So if the human forgot what had happened in the hand so far,

Lex Fridman (06:33.320)

they could actually reference back and so forth.

Tuomas Sandholm (06:37.240)

Is there a reason they were given access

Lex Fridman (06:39.480)

to the betting history for?

Tuomas Sandholm (06:41.200)

Well, we just, it didn't really matter.

Lex Fridman (06:45.860)

They wouldn't have forgotten anyway.

Tuomas Sandholm (06:47.360)

These are top quality people.

Lex Fridman (06:48.800)

But we just wanted to put out there

Lex Fridman (06:51.300)

so it's not a question of the human forgetting

Lex Fridman (06:53.460)

and the AI somehow trying to get advantage

Tuomas Sandholm (06:55.320)

of better memory.

Lex Fridman (06:56.760)

So what was that like?

Tuomas Sandholm (06:57.640)

I mean, that was an incredible accomplishment.

Lex Fridman (06:59.720)

So what did it feel like before the event?

Lex Fridman (07:02.760)

Did you have doubt, hope?

Lex Fridman (07:05.640)

Where was your confidence at?

Tuomas Sandholm (07:08.160)

Yeah, that's great.

Lex Fridman (07:09.240)

So great question.

Lex Fridman (07:10.160)

So 18 months earlier, I had organized a similar brains

Lex Fridman (07:14.200)

versus AI competition with a previous AI called Cloudyco

Lex Fridman (07:17.840)

and we couldn't beat the humans.

Lex Fridman (07:20.560)

So this time around, it was only 18 months later.

Lex Fridman (07:23.800)

And I knew that this new AI, Libratus, was way stronger,

Lex Fridman (07:27.820)

but it's hard to say how you'll do against the top humans

Tuomas Sandholm (07:31.360)

before you try.

Lex Fridman (07:32.440)

So I thought we had about a 50, 50 shot.

Lex Fridman (07:35.160)

And the international betting sites put us

Lex Fridman (07:38.880)

as a four to one or five to one underdog.

Lex Fridman (07:41.800)

So it's kind of interesting that people really believe

Lex Fridman (07:44.700)

in people and over AI, not just people.

Tuomas Sandholm (07:48.440)

People don't just over believe in themselves,

Lex Fridman (07:50.720)

but they have overconfidence in other people as well

Tuomas Sandholm (07:53.280)

compared to the performance of AI.

Lex Fridman (07:55.440)

And yeah, so we were a four to one or five to one underdog.

Lex Fridman (07:59.120)

And even after three days of beating the humans in a row,

Lex Fridman (08:02.880)

we were still 50, 50 on the international betting sites.

Lex Fridman (08:06.520)

Do you think there's something special and magical

Lex Fridman (08:09.040)

about poker and the way people think about it,

Tuomas Sandholm (08:12.160)

in the sense you have,

Lex Fridman (08:14.600)

I mean, even in chess, there's no Hollywood movies.

Tuomas Sandholm (08:17.320)

Poker is the star of many movies.

Lex Fridman (08:21.200)

And there's this feeling that certain human facial

Tuomas Sandholm (08:26.640)

expressions and body language, eye movement,

Lex Fridman (08:30.760)

all these tells are critical to poker.

Tuomas Sandholm (08:33.360)

Like you can look into somebody's soul

Lex Fridman (08:35.000)

and understand their betting strategy and so on.

Lex Fridman (08:37.880)

So that's probably why, possibly,

Lex Fridman (08:41.520)

do you think that is why people have a confidence

Lex Fridman (08:43.640)

that humans will outperform?

Lex Fridman (08:45.640)

Because AI systems cannot, in this construct,

Tuomas Sandholm (08:48.920)

perceive these kinds of tells.

Lex Fridman (08:51.040)

They're only looking at betting patterns

Lex Fridman (08:53.200)

and nothing else, betting patterns and statistics.

Lex Fridman (08:58.200)

So what's more important to you

Lex Fridman (09:02.200)

if you step back on human players, human versus human?

Lex Fridman (09:06.120)

What's the role of these tells,

Lex Fridman (09:08.600)

of these ideas that we romanticize?

Lex Fridman (09:11.880)

Yeah, so I'll split it into two parts.

Lex Fridman (09:15.480)

So one is why do humans trust humans more than AI

Lex Fridman (09:20.480)

and have overconfidence in humans?

Tuomas Sandholm (09:22.600)

I think that's not really related to the tell question.

Lex Fridman (09:25.920)

It's just that they've seen these top players,

Lex Fridman (09:28.600)

how good they are, and they're really fantastic.

Lex Fridman (09:31.040)

So it's just hard to believe that an AI could beat them.

Lex Fridman (09:36.040)

So I think that's where that comes from.

Lex Fridman (09:37.680)

And that's actually maybe a more general lesson about AI.

Tuomas Sandholm (09:40.600)

That until you've seen it overperform a human,

Lex Fridman (09:43.200)

it's hard to believe that it could.

Lex Fridman (09:45.080)

But then the tells, a lot of these top players,

Lex Fridman (09:50.560)

they're so good at hiding tells

Tuomas Sandholm (09:52.760)

that among the top players,

Lex Fridman (09:56.240)

it's actually not really worth it

Tuomas Sandholm (09:59.480)

for them to invest a lot of effort

Lex Fridman (10:01.200)

trying to find tells in each other

Tuomas Sandholm (10:03.160)

because they're so good at hiding them.

Lex Fridman (10:05.640)

So yes, at the kind of Friday evening game,

Tuomas Sandholm (10:09.840)

tells are gonna be a huge thing.

Lex Fridman (10:11.800)

You can read other people.

Lex Fridman (10:13.160)

And if you're a good reader,

Lex Fridman (10:14.120)

you'll read them like an open book.

Lex Fridman (10:16.440)

But at the top levels of poker now,

Lex Fridman (10:18.280)

the tells become a much smaller and smaller aspect

Tuomas Sandholm (10:21.960)

of the game as you go to the top levels.

Lex Fridman (10:24.480)

The amount of strategies, the amount of possible actions

Tuomas Sandholm (10:28.120)

is very large, 10 to the power of 100 plus.

Lex Fridman (10:35.400)

So there has to be some, I've read a few of the papers

Tuomas Sandholm (10:37.880)

related, it has to form some abstractions

Lex Fridman (10:42.080)

of various hands and actions.

Lex Fridman (10:44.040)

So what kind of abstractions are effective

Lex Fridman (10:47.560)

for the game of poker?

Tuomas Sandholm (10:49.200)

Yeah, so you're exactly right.

Lex Fridman (10:50.880)

So when you go from a game tree that's 10 to the 161,

Tuomas Sandholm (10:55.360)

especially in an imperfect information game,

Lex Fridman (10:58.000)

it's way too large to solve directly,

Tuomas Sandholm (11:00.200)

even with our fastest equilibrium finding algorithms.

Lex Fridman (11:03.280)

So you wanna abstract it first.

Lex Fridman (11:07.200)

And abstraction in games is much trickier

Lex Fridman (11:10.920)

than abstraction in MDPs or other single agent settings.

Tuomas Sandholm (11:15.440)

Because you have these abstraction pathologies

Lex Fridman (11:17.760)

that if I have a finer grained abstraction,

Tuomas Sandholm (11:19.880)

the strategy that I can get from that for the real game

Lex Fridman (11:23.240)

might actually be worse than the strategy

Tuomas Sandholm (11:25.240)

I can get from the coarse grained abstraction.

Lex Fridman (11:27.160)

So you have to be very careful.

Tuomas Sandholm (11:28.760)

Now the kinds of abstractions, just to zoom out,

Lex Fridman (11:31.080)

we're talking about, there's the hands abstractions

Lex Fridman (11:34.480)

and then there's betting strategies.

Lex Fridman (11:37.280)

Yeah, betting actions, yeah.

Tuomas Sandholm (11:38.600)

Baiting actions.

Lex Fridman (11:39.440)

So there's information abstraction,

Tuomas Sandholm (11:41.640)

don't talk about general games, information abstraction,

Lex Fridman (11:44.720)

which is the abstraction of what chance does.

Lex Fridman (11:47.560)

And this would be the cards in the case of poker.

Lex Fridman (11:50.080)

And then there's action abstraction,

Tuomas Sandholm (11:52.480)

which is abstracting the actions of the actual players,

Lex Fridman (11:57.000)

which would be bets in the case of poker.

Lex Fridman (11:59.560)

Yourself and the other players?

Lex Fridman (12:01.320)

Yes, yourself and other players.

Lex Fridman (12:03.680)

And for information abstraction,

Lex Fridman (12:08.280)

we were completely automated.

Lex Fridman (12:11.160)

So these are algorithms,

Lex Fridman (12:13.840)

but they do what we call potential aware abstraction,

Tuomas Sandholm (12:16.760)

where we don't just look at the value of the hand,

Lex Fridman (12:19.000)

but also how it might materialize

Tuomas Sandholm (12:20.840)

into good or bad hands over time.

Lex Fridman (12:22.560)

And it's a certain kind of bottom up process

Tuomas Sandholm (12:25.280)

with integer programming there and clustering

Lex Fridman (12:27.640)

and various aspects, how do you build this abstraction?

Lex Fridman (12:31.480)

And then in the action abstraction,

Lex Fridman (12:34.400)

there it's largely based on how humans and other AIs

Tuomas Sandholm (12:40.520)

have played this game in the past.

Lex Fridman (12:42.320)

But in the beginning,

Tuomas Sandholm (12:43.880)

we actually used an automated action abstraction technology,

Lex Fridman (12:47.680)

which is provably convergent

Tuomas Sandholm (12:51.240)

that it finds the optimal combination of bet sizes,

Lex Fridman (12:54.040)

but it's not very scalable.

Lex Fridman (12:55.480)

So we couldn't use it for the whole game,

Lex Fridman (12:57.280)

but we use it for the first couple of betting actions.

Lex Fridman (12:59.880)

So what's more important, the strength of the hand,

Lex Fridman (13:03.080)

so the information abstraction or the how you play them,

Tuomas Sandholm (13:09.320)

the actions, does it, you know,

Lex Fridman (13:11.640)

the romanticized notion again,

Tuomas Sandholm (13:13.200)

is that it doesn't matter what hands you have,

Lex Fridman (13:15.600)

that the actions, the betting may be the way you win

Tuomas Sandholm (13:19.240)

no matter what hands you have.

Lex Fridman (13:20.320)

Yeah, so that's why you have to play a lot of hands

Lex Fridman (13:23.280)

so that the role of luck gets smaller.

Lex Fridman (13:26.800)

So you could otherwise get lucky and get some good hands

Lex Fridman (13:29.920)

and then you're gonna win the match.

Lex Fridman (13:31.480)

Even with thousands of hands, you can get lucky

Tuomas Sandholm (13:35.280)

because there's so much variance

Lex Fridman (13:36.720)

in No Limit Texas Holden because if we both go all in,

Tuomas Sandholm (13:40.880)

it's a huge stack of variance, so there are these

Lex Fridman (13:43.640)

massive swings in No Limit Texas Holden.

Lex Fridman (13:47.800)

So that's why you have to play not just thousands,

Lex Fridman (13:50.240)

but over 100,000 hands to get statistical significance.

Lex Fridman (13:55.000)

So let me ask another way this question.

Lex Fridman (13:57.880)

If you didn't even look at your hands,

Lex Fridman (14:02.000)

but they didn't know that, the opponents didn't know that,

Lex Fridman (14:04.560)

how well would you be able to do?

Tuomas Sandholm (14:06.680)

Oh, that's a good question.

Lex Fridman (14:07.760)

There's actually, I heard this story

Tuomas Sandholm (14:09.600)

that there's this Norwegian female poker player

Lex Fridman (14:11.800)

called Annette Oberstad who's actually won a tournament

Tuomas Sandholm (14:15.240)

by doing exactly that, but that would be extremely rare.

Lex Fridman (14:18.640)

So you cannot really play well that way.

Tuomas Sandholm (14:23.440)

Okay, so the hands do have some role to play, okay.

Lex Fridman (14:27.840)

So Labradus does not use, as far as I understand,

Tuomas Sandholm (14:33.120)

they use learning methods, deep learning.

Lex Fridman (14:35.320)

Is there room for learning in,

Tuomas Sandholm (14:40.600)

there's no reason why Labradus doesn't combine

Lex Fridman (14:44.120)

with an AlphaGo type approach for estimating

Tuomas Sandholm (14:46.400)

the quality for function estimator.

Lex Fridman (14:49.200)

What are your thoughts on this,

Tuomas Sandholm (14:52.040)

maybe as compared to another algorithm

Lex Fridman (14:54.760)

which I'm not that familiar with, DeepStack,

Tuomas Sandholm (14:56.720)

the engine that does use deep learning,

Lex Fridman (14:59.280)

that it's unclear how well it does,

Lex Fridman (15:01.560)

but nevertheless uses deep learning.

Lex Fridman (15:03.480)

So what are your thoughts about learning methods

Lex Fridman (15:05.400)

to aid in the way that Labradus plays in the game of poker?

Lex Fridman (15:09.280)

Yeah, so as you said,

Tuomas Sandholm (15:10.640)

Labradus did not use learning methods

Lex Fridman (15:13.080)

and played very well without them.

Tuomas Sandholm (15:15.680)

Since then, we have actually, actually here,

Lex Fridman (15:17.840)

we have a couple of papers on things

Tuomas Sandholm (15:20.000)

that do use learning techniques.

Lex Fridman (15:22.360)

Excellent.

Lex Fridman (15:24.440)

And deep learning in particular.

Lex Fridman (15:26.360)

And sort of the way you're talking about

Tuomas Sandholm (15:29.920)

where it's learning an evaluation function,

Lex Fridman (15:33.360)

but in imperfect information games,

Tuomas Sandholm (15:37.400)

unlike let's say in Go or now also in chess and shogi,

Lex Fridman (15:42.440)

it's not sufficient to learn an evaluation for a state

Tuomas Sandholm (15:47.400)

because the value of an information set

Lex Fridman (15:52.920)

depends not only on the exact state,

Lex Fridman (15:55.400)

but it also depends on both players beliefs.

Lex Fridman (15:59.200)

Like if I have a bad hand,

Tuomas Sandholm (16:01.240)

I'm much better off if the opponent thinks I have a good hand

Lex Fridman (16:04.720)

and vice versa.

Tuomas Sandholm (16:05.560)

If I have a good hand,

Lex Fridman (16:06.480)

I'm much better off if the opponent believes

Tuomas Sandholm (16:09.360)

I have a bad hand.

Lex Fridman (16:11.360)

So the value of a state is not just a function of the cards.

Tuomas Sandholm (16:15.640)

It depends on, if you will, the path of play,

Lex Fridman (16:19.600)

but only to the extent that it's captured

Tuomas Sandholm (16:22.040)

in the belief distributions.

Lex Fridman (16:23.720)

So that's why it's not as simple

Tuomas Sandholm (16:26.240)

as it is in perfect information games.

Lex Fridman (16:29.320)

And I don't wanna say it's simple there either.

Tuomas Sandholm (16:31.080)

It's of course very complicated computationally there too,

Lex Fridman (16:34.200)

but at least conceptually, it's very straightforward.

Tuomas Sandholm (16:36.520)

There's a state, there's an evaluation function.

Lex Fridman (16:38.760)

You can try to learn it.

Tuomas Sandholm (16:39.800)

Here, you have to do something more.

Lex Fridman (16:43.280)

And what we do is in one of these papers,

Tuomas Sandholm (16:47.160)

we're looking at where we allow the opponent

Lex Fridman (16:50.800)

to actually take different strategies

Tuomas Sandholm (16:53.000)

at the leaf of the search tree, if you will.

Lex Fridman (16:56.440)

And that is a different way of doing it.

Lex Fridman (16:59.840)

And it doesn't assume therefore a particular way

Lex Fridman (17:02.560)

that the opponent plays,

Lex Fridman (17:04.040)

but it allows the opponent to choose

Lex Fridman (17:05.840)

from a set of different continuation strategies.

Lex Fridman (17:09.800)

And that forces us to not be too optimistic

Lex Fridman (17:13.400)

in a look ahead search.

Lex Fridman (17:15.520)

And that's one way you can do sound look ahead search

Lex Fridman (17:19.040)

in imperfect information games,

Tuomas Sandholm (17:21.480)

which is very difficult.

Lex Fridman (17:23.360)

And you were asking about DeepStack.

Lex Fridman (17:26.080)

What they did, it was very different than what we do,

Lex Fridman (17:29.280)

either in Libratus or in this new work.

Tuomas Sandholm (17:32.000)

They were randomly generating various situations

Lex Fridman (17:35.440)

in the game.

Tuomas Sandholm (17:36.440)

Then they were doing the look ahead

Lex Fridman (17:38.080)

from there to the end of the game,

Tuomas Sandholm (17:39.840)

as if that was the start of a different game.

Lex Fridman (17:42.960)

And then they were using deep learning

Tuomas Sandholm (17:44.920)

to learn those values of those states,

Lex Fridman (17:47.960)

but the states were not just the physical states.

Tuomas Sandholm (17:50.280)

They include belief distributions.

Lex Fridman (17:52.560)

When you talk about look ahead for DeepStack

Tuomas Sandholm (17:56.800)

or with Libratus, does it mean,

Lex Fridman (17:59.480)

considering every possibility that the game can evolve,

Tuomas Sandholm (18:02.680)

are we talking about extremely,

Lex Fridman (18:04.280)

sort of this exponentially growth of a tree?

Tuomas Sandholm (18:06.880)

Yes, so we're talking about exactly that.

Lex Fridman (18:11.280)

Much like you do in alpha beta search

Tuomas Sandholm (18:14.280)

or Monte Carlo tree search, but with different techniques.

Lex Fridman (18:17.480)

So there's a different search algorithm.

Lex Fridman (18:19.280)

And then we have to deal with the leaves differently.

Lex Fridman (18:21.920)

So if you think about what Libratus did,

Tuomas Sandholm (18:24.000)

we didn't have to worry about this

Lex Fridman (18:25.520)

because we only did it at the end of the game.

Lex Fridman (18:28.560)

So we would always terminate into a real situation

Lex Fridman (18:32.280)

and we would know what the payout is.

Tuomas Sandholm (18:34.000)

It didn't do these depth limited lookaheads,

Lex Fridman (18:36.880)

but now in this new paper, which is called depth limited,

Tuomas Sandholm (18:40.680)

I think it's called depth limited search

Lex Fridman (18:42.120)

for imperfect information games,

Tuomas Sandholm (18:43.880)

we can actually do sound depth limited lookahead.

Lex Fridman (18:47.040)

So we can actually start to do the look ahead

Tuomas Sandholm (18:49.240)

from the beginning of the game on,

Lex Fridman (18:51.080)

because that's too complicated to do

Tuomas Sandholm (18:53.400)

for this whole long game.

Lex Fridman (18:54.920)

So in Libratus, we were just doing it for the end.

Tuomas Sandholm (18:57.680)

So, and then the other side, this belief distribution,

Lex Fridman (19:00.720)

so is it explicitly modeled what kind of beliefs

Lex Fridman (19:05.320)

that the opponent might have?

Lex Fridman (19:07.400)

Yeah, it is explicitly modeled, but it's not assumed.

Tuomas Sandholm (19:11.840)

The beliefs are actually output, not input.

Lex Fridman (19:15.400)

Of course, the starting beliefs are input,

Lex Fridman (19:18.840)

but they just fall from the rules of the game

Lex Fridman (19:20.640)

because we know that the dealer deals uniformly

Tuomas Sandholm (19:23.520)

from the deck, so I know that every pair of cards

Lex Fridman (19:27.720)

that you might have is equally likely.

Tuomas Sandholm (19:30.440)

I know that for a fact, that just follows

Lex Fridman (19:32.200)

from the rules of the game.

Tuomas Sandholm (19:33.160)

Of course, except the two cards that I have,

Lex Fridman (19:35.200)

I know you don't have those.

Tuomas Sandholm (19:36.560)

Yeah.

Lex Fridman (19:37.560)

You have to take that into account.

Tuomas Sandholm (19:38.720)

That's called card removal and that's very important.

Lex Fridman (19:40.920)

Is the dealing always coming from a single deck

Tuomas Sandholm (19:43.760)

in Heads Up, so you can assume.

Lex Fridman (19:45.880)

Single deck, so you know that if I have the ace of spades,

Tuomas Sandholm (19:50.880)

I know you don't have an ace of spades.

Lex Fridman (19:53.560)

Great, so in the beginning, your belief is basically

Tuomas Sandholm (19:56.880)

the fact that it's a fair dealing of hands,

Lex Fridman (19:59.320)

but how do you start to adjust that belief?

Tuomas Sandholm (1:00:00.840)

Nuclear weapons have been here.

Lex Fridman (1:00:03.280)

It's an obvious problem that's just been sitting there.

Lex Fridman (1:00:05.720)

So how do you think about,

Lex Fridman (1:00:07.480)

what is the mechanism design there

Lex Fridman (1:00:09.240)

that just made everything seem stable?

Lex Fridman (1:00:12.280)

And are you still extremely worried?

Tuomas Sandholm (1:00:14.800)

I am still extremely worried.

Lex Fridman (1:00:16.640)

So you probably know the simple game theory of mad.

Lex Fridman (1:00:20.040)

So this was a mutually assured destruction

Lex Fridman (1:00:23.760)

and it doesn't require any computation with small matrices.

Tuomas Sandholm (1:00:27.360)

You can actually convince yourself

Lex Fridman (1:00:28.600)

that the game is such that nobody wants to initiate.

Tuomas Sandholm (1:00:31.480)

Yeah, that's a very coarse grained analysis.

Lex Fridman (1:00:34.600)

And it really works in a situational way.

Tuomas Sandholm (1:00:36.880)

You have two superpowers or small number of superpowers.

Lex Fridman (1:00:40.400)

Now things are very different.

Tuomas Sandholm (1:00:41.960)

You have a smaller nuke.

Lex Fridman (1:00:43.080)

So the threshold of initiating is smaller

Lex Fridman (1:00:47.240)

and you have smaller countries and non nation actors

Lex Fridman (1:00:51.520)

who may get a nuke and so on.

Lex Fridman (1:00:53.760)

So I think it's riskier now than it was maybe ever before.

Lex Fridman (1:00:58.320)

And what idea, application of AI,

Tuomas Sandholm (1:01:03.640)

you've talked about a little bit,

Lex Fridman (1:01:04.640)

but what is the most exciting to you right now?

Tuomas Sandholm (1:01:07.560)

I mean, you're here at NIPS, NeurIPS.

Lex Fridman (1:01:10.160)

Now you have a few excellent pieces of work,

Lex Fridman (1:01:14.920)

but what are you thinking into the future

Lex Fridman (1:01:16.680)

with several companies you're doing?

Lex Fridman (1:01:17.840)

What's the most exciting thing or one of the exciting things?

Lex Fridman (1:01:21.120)

The number one thing for me right now

Tuomas Sandholm (1:01:23.160)

is coming up with these scalable techniques

Lex Fridman (1:01:26.360)

for game solving and applying them into the real world.

Tuomas Sandholm (1:01:30.440)

I'm still very interested in market design as well.

Lex Fridman (1:01:33.160)

And we're doing that in the optimized markets,

Lex Fridman (1:01:35.400)

but I'm most interested if number one right now

Lex Fridman (1:01:37.560)

is strategic machine strategy robot,

Tuomas Sandholm (1:01:40.000)

getting that technology out there

Lex Fridman (1:01:41.440)

and seeing as you were in the trenches doing applications,

Lex Fridman (1:01:45.560)

what needs to be actually filled,

Lex Fridman (1:01:47.120)

what technology gaps still need to be filled.

Lex Fridman (1:01:49.800)

So it's so hard to just put your feet on the table

Lex Fridman (1:01:52.040)

and imagine what needs to be done.

Lex Fridman (1:01:53.800)

But when you're actually doing real applications,

Lex Fridman (1:01:56.280)

the applications tell you what needs to be done.

Lex Fridman (1:01:59.120)

And I really enjoy that interaction.

Lex Fridman (1:02:00.840)

Is it a challenging process to apply

Tuomas Sandholm (1:02:04.480)

some of the state of the art techniques you're working on

Lex Fridman (1:02:07.760)

and having the various players in industry

Tuomas Sandholm (1:02:14.080)

or the military or people who could really benefit from it

Lex Fridman (1:02:17.720)

actually use it?

Tuomas Sandholm (1:02:19.040)

What's that process like of,

Lex Fridman (1:02:21.400)

autonomous vehicles work with automotive companies

Lex Fridman (1:02:23.680)

and they're in many ways are a little bit old fashioned.

Lex Fridman (1:02:28.200)

It's difficult.

Tuomas Sandholm (1:02:29.240)

They really want to use this technology.

Lex Fridman (1:02:31.840)

There's clearly will have a significant benefit,

Lex Fridman (1:02:34.640)

but the systems aren't quite in place

Lex Fridman (1:02:37.480)

to easily have them integrated in terms of data,

Tuomas Sandholm (1:02:41.080)

in terms of compute, in terms of all these kinds of things.

Lex Fridman (1:02:43.760)

So is that one of the bigger challenges that you're facing

Lex Fridman (1:02:48.680)

and how do you tackle that challenge?

Lex Fridman (1:02:50.000)

Yeah, I think that's always a challenge.

Tuomas Sandholm (1:02:52.360)

That's kind of slowness and inertia really

Lex Fridman (1:02:55.560)

of let's do things the way we've always done it.

Tuomas Sandholm (1:02:57.920)

You just have to find the internal champions

Lex Fridman (1:03:00.120)

at the customer who understand that,

Tuomas Sandholm (1:03:02.120)

hey, things can't be the same way in the future.

Lex Fridman (1:03:04.680)

Otherwise bad things are going to happen.

Lex Fridman (1:03:06.960)

And it's in autonomous vehicles.

Lex Fridman (1:03:08.600)

It's actually very interesting

Tuomas Sandholm (1:03:09.680)

that the car makers are doing that

Lex Fridman (1:03:11.120)

and they're very traditional,

Lex Fridman (1:03:12.440)

but at the same time you have tech companies

Lex Fridman (1:03:14.360)

who have nothing to do with cars or transportation

Tuomas Sandholm (1:03:17.120)

like Google and Baidu really pushing on autonomous cars.

Lex Fridman (1:03:21.880)

I find that fascinating.

Tuomas Sandholm (1:03:23.240)

Clearly you're super excited

Lex Fridman (1:03:25.160)

about actually these ideas having an impact in the world.

Tuomas Sandholm (1:03:29.320)

In terms of the technology, in terms of ideas and research,

Lex Fridman (1:03:32.680)

are there directions that you're also excited about?

Tuomas Sandholm (1:03:36.600)

Whether that's on some of the approaches you talked about

Lex Fridman (1:03:40.840)

for the imperfect information games,

Tuomas Sandholm (1:03:42.760)

whether it's applying deep learning

Lex Fridman (1:03:44.000)

to some of these problems,

Tuomas Sandholm (1:03:45.120)

is there something that you're excited

Lex Fridman (1:03:46.520)

in the research side of things?

Tuomas Sandholm (1:03:48.840)

Yeah, yeah, lots of different things

Lex Fridman (1:03:51.120)

in the game solving.

Lex Fridman (1:03:53.240)

So solving even bigger games,

Lex Fridman (1:03:56.400)

games where you have more hidden action

Tuomas Sandholm (1:03:59.760)

of the player actions as well.

Lex Fridman (1:04:02.040)

Poker is a game where really the chance actions are hidden

Tuomas Sandholm (1:04:05.880)

or some of them are hidden,

Lex Fridman (1:04:07.080)

but the player actions are public.

Tuomas Sandholm (1:04:11.440)

Multiplayer games of various sorts,

Lex Fridman (1:04:14.000)

collusion, opponent exploitation,

Tuomas Sandholm (1:04:18.080)

all and even longer games.

Lex Fridman (1:04:21.280)

So games that basically go forever,

Lex Fridman (1:04:23.160)

but they're not repeated.

Lex Fridman (1:04:24.680)

So see extensive fun games that go forever.

Lex Fridman (1:04:27.880)

What would that even look like?

Lex Fridman (1:04:30.080)

How do you represent that?

Lex Fridman (1:04:31.040)

How do you solve that?

Lex Fridman (1:04:32.040)

What's an example of a game like that?

Tuomas Sandholm (1:04:33.440)

Or is this some of the stochastic games

Lex Fridman (1:04:35.600)

that you mentioned?

Tuomas Sandholm (1:04:36.440)

Let's say business strategy.

Lex Fridman (1:04:37.320)

So it's not just modeling like a particular interaction,

Lex Fridman (1:04:40.840)

but thinking about the business from here to eternity.

Lex Fridman (1:04:44.440)

Or let's say military strategy.

Lex Fridman (1:04:49.040)

So it's not like war is gonna go away.

Lex Fridman (1:04:51.000)

How do you think about military strategy

Lex Fridman (1:04:54.280)

that's gonna go forever?

Lex Fridman (1:04:56.680)

How do you even model that?

Lex Fridman (1:04:58.080)

How do you know whether a move was good

Lex Fridman (1:05:01.000)

that somebody made and so on?

Lex Fridman (1:05:05.200)

So that's kind of one direction.

Lex Fridman (1:05:06.960)

I'm also very interested in learning

Tuomas Sandholm (1:05:09.800)

much more scalable techniques for integer programming.

Lex Fridman (1:05:13.440)

So we had an ICML paper this summer on that.

Tuomas Sandholm (1:05:16.560)

The first automated algorithm configuration paper

Lex Fridman (1:05:20.280)

that has theoretical generalization guarantees.

Lex Fridman (1:05:23.560)

So if I see this many training examples

Lex Fridman (1:05:26.200)

and I told my algorithm in this way,

Tuomas Sandholm (1:05:28.560)

it's going to have good performance

Lex Fridman (1:05:30.560)

on the real distribution, which I've not seen.

Tuomas Sandholm (1:05:33.200)

So, which is kind of interesting

Lex Fridman (1:05:34.840)

that algorithm configuration has been going on now

Tuomas Sandholm (1:05:37.680)

for at least 17 years seriously.

Lex Fridman (1:05:41.200)

And there has not been any generalization theory before.

Tuomas Sandholm (1:05:45.960)

Well, this is really exciting

Lex Fridman (1:05:47.200)

and it's a huge honor to talk to you.

Tuomas Sandholm (1:05:49.840)

Thank you so much, Tomas.

Lex Fridman (1:05:51.160)

Thank you for bringing Labradus to the world

Lex Fridman (1:05:52.880)

and all the great work you're doing.

Lex Fridman (1:05:54.160)

Well, thank you very much.

Tuomas Sandholm (1:05:55.000)

It's been fun.

Lex Fridman (1:05:55.840)

No more questions.

Tuomas Sandholm (20:02.800)

Well, that's where this beauty of game theory comes.

Lex Fridman (20:06.800)

So Nash equilibrium, which John Nash introduced in 1950,

Tuomas Sandholm (20:10.920)

introduces what rational play is

Lex Fridman (20:13.800)

when you have more than one player.

Lex Fridman (20:16.040)

And these are pairs of strategies

Lex Fridman (20:18.440)

where strategies are contingency plans,

Tuomas Sandholm (20:20.360)

one for each player.

Lex Fridman (20:22.880)

So that neither player wants to deviate

Tuomas Sandholm (20:25.720)

to a different strategy,

Lex Fridman (20:26.960)

given that the other doesn't deviate.

Lex Fridman (20:29.160)

But as a side effect, you get the beliefs from base roll.

Lex Fridman (20:33.840)

So Nash equilibrium really isn't just deriving

Tuomas Sandholm (20:36.440)

in these imperfect information games,

Lex Fridman (20:38.360)

Nash equilibrium, it doesn't just define strategies.

Tuomas Sandholm (20:41.920)

It also defines beliefs for both of us

Lex Fridman (20:44.960)

and defines beliefs for each state.

Lex Fridman (20:48.840)

So at each state, it's called information sets.

Lex Fridman (20:53.280)

At each information set in the game,

Tuomas Sandholm (20:55.560)

there's a set of different states that we might be in,

Lex Fridman (20:59.000)

but I don't know which one we're in.

Tuomas Sandholm (21:01.760)

Nash equilibrium tells me exactly

Lex Fridman (21:03.400)

what is the probability distribution

Tuomas Sandholm (21:05.000)

over those real world states in my mind.

Lex Fridman (21:08.280)

How does Nash equilibrium give you that distribution?

Lex Fridman (21:11.440)

So why?

Lex Fridman (21:12.280)

I'll do a simple example.

Lex Fridman (21:13.320)

So you know the game Rock, Paper, Scissors?

Lex Fridman (21:16.760)

So we can draw it as player one moves first

Lex Fridman (21:20.000)

and then player two moves.

Lex Fridman (21:21.600)

But of course, it's important that player two

Tuomas Sandholm (21:24.520)

doesn't know what player one moved,

Lex Fridman (21:26.400)

otherwise player two would win every time.

Lex Fridman (21:28.600)

So we can draw that as an information set

Lex Fridman (21:30.480)

where player one makes one of three moves first,

Lex Fridman (21:33.280)

and then there's an information set for player two.

Lex Fridman (21:36.200)

So player two doesn't know which of those nodes

Tuomas Sandholm (21:39.920)

the world is in.

Lex Fridman (21:41.800)

But once we know the strategy for player one,

Tuomas Sandholm (21:44.920)

Nash equilibrium will say that you play 1 3rd Rock,

Lex Fridman (21:47.320)

1 3rd Paper, 1 3rd Scissors.

Tuomas Sandholm (21:49.400)

From that, I can derive my beliefs on the information set

Lex Fridman (21:52.600)

that they're 1 3rd, 1 3rd, 1 3rd.

Lex Fridman (21:54.480)

So Bayes gives you that.

Lex Fridman (21:56.280)

Bayes gives you.

Lex Fridman (21:57.560)

But is that specific to a particular player,

Lex Fridman (21:59.760)

or is it something you quickly update

Lex Fridman (22:03.960)

with the specific player?

Lex Fridman (22:05.040)

No, the game theory isn't really player specific.

Lex Fridman (22:08.800)

So that's also why we don't need any data.

Lex Fridman (22:11.720)

We don't need any history

Lex Fridman (22:12.760)

how these particular humans played in the past

Lex Fridman (22:14.800)

or how any AI or human had played before.

Tuomas Sandholm (22:17.400)

It's all about rationality.

Lex Fridman (22:20.240)

So the AI just thinks about

Lex Fridman (22:22.720)

what would a rational opponent do?

Lex Fridman (22:24.880)

And what would I do if I am rational?

Lex Fridman (22:28.000)

And that's the idea of game theory.

Lex Fridman (22:31.080)

So it's really a data free, opponent free approach.

Lex Fridman (22:35.560)

So it comes from the design of the game

Lex Fridman (22:37.680)

as opposed to the design of the player.

Tuomas Sandholm (22:40.040)

Exactly, there's no opponent modeling per se.

Lex Fridman (22:43.080)

I mean, we've done some work on combining opponent modeling

Tuomas Sandholm (22:45.600)

with game theory so you can exploit weak players even more,

Lex Fridman (22:48.840)

but that's another strand.

Lex Fridman (22:50.280)

And in Librarus, we didn't turn that on.

Lex Fridman (22:52.320)

So I decided that these players are too good.

Lex Fridman (22:55.000)

And when you start to exploit an opponent,

Lex Fridman (22:58.080)

you typically open yourself up to exploitation.

Lex Fridman (23:01.800)

And these guys have so few holes to exploit

Lex Fridman (23:04.000)

and they're world's leading experts in counter exploitation.

Lex Fridman (23:06.760)

So I decided that we're not gonna turn that stuff on.

Lex Fridman (23:09.200)

Actually, I saw a few of your papers exploiting opponents.

Tuomas Sandholm (23:12.160)

It sounded very interesting to explore.

Lex Fridman (23:15.720)

Do you think there's room for exploitation

Lex Fridman (23:17.880)

generally outside of Librarus?

Lex Fridman (23:19.920)

Is there a subject or people differences

Tuomas Sandholm (23:24.080)

that could be exploited, maybe not just in poker,

Lex Fridman (23:27.920)

but in general interactions and negotiations,

Lex Fridman (23:30.440)

all these other domains that you're considering?

Lex Fridman (23:33.480)

Yeah, definitely.

Tuomas Sandholm (23:34.680)

We've done some work on that.

Lex Fridman (23:35.920)

And I really like the work at hybrid digested too.

Lex Fridman (23:39.880)

So you figure out what would a rational opponent do.

Lex Fridman (23:43.440)

And by the way, that's safe in these zero sum games,

Tuomas Sandholm (23:46.280)

two player zero sum games,

Lex Fridman (23:47.480)

because if the opponent does something irrational,

Tuomas Sandholm (23:49.560)

yes, it might throw off my beliefs,

Lex Fridman (23:53.080)

but the amount that the player can gain

Tuomas Sandholm (23:55.760)

by throwing off my belief is always less

Lex Fridman (23:59.160)

than they lose by playing poorly.

Lex Fridman (24:01.800)

So it's safe.

Lex Fridman (24:03.080)

But still, if somebody's weak as a player,

Tuomas Sandholm (24:06.720)

you might wanna play differently to exploit them more.

Lex Fridman (24:10.240)

So you can think about it this way,

Tuomas Sandholm (24:12.040)

a game theoretic strategy is unbeatable,

Lex Fridman (24:15.600)

but it doesn't maximally beat the other opponent.

Lex Fridman (24:19.600)

So the winnings per hand might be better

Lex Fridman (24:22.800)

with a different strategy.

Lex Fridman (24:24.240)

And the hybrid is that you start

Lex Fridman (24:25.720)

from a game theoretic approach.

Lex Fridman (24:27.080)

And then as you gain data about the opponent

Lex Fridman (24:30.840)

in certain parts of the game tree,

Tuomas Sandholm (24:32.600)

then in those parts of the game tree,

Lex Fridman (24:34.360)

you start to tweak your strategy more and more

Tuomas Sandholm (24:37.800)

towards exploitation while still staying fairly close

Lex Fridman (24:40.960)

to the game theoretic strategy

Lex Fridman (24:42.160)

so as to not open yourself up to exploitation too much.

Lex Fridman (24:46.840)

How do you do that?

Lex Fridman (24:48.320)

Do you try to vary up strategies, make it unpredictable?

Lex Fridman (24:53.640)

It's like, what is it, tit for tat strategies

Lex Fridman (24:57.520)

in Prisoner's Dilemma or?

Lex Fridman (25:00.720)

Well, that's a repeated game.

Tuomas Sandholm (25:03.240)

Repeated games.

Lex Fridman (25:04.080)

Simple Prisoner's Dilemma, repeated games.

Lex Fridman (25:06.520)

But even there, there's no proof that says

Lex Fridman (25:08.760)

that that's the best thing.

Lex Fridman (25:10.080)

But experimentally, it actually does well.

Lex Fridman (25:13.280)

So what kind of games are there, first of all?

Tuomas Sandholm (25:15.320)

I don't know if this is something

Lex Fridman (25:17.040)

that you could just summarize.

Tuomas Sandholm (25:18.600)

There's perfect information games

Lex Fridman (25:20.360)

where all the information's on the table.

Tuomas Sandholm (25:22.400)

There is imperfect information games.

Lex Fridman (25:25.480)

There's repeated games that you play over and over.

Tuomas Sandholm (25:28.560)

There's zero sum games.

Lex Fridman (25:31.320)

There's non zero sum games.

Lex Fridman (25:34.440)

And then there's a really important distinction

Lex Fridman (25:37.520)

you're making, two player versus more players.

Lex Fridman (25:40.720)

So what are, what other games are there?

Lex Fridman (25:44.760)

And what's the difference, for example,

Lex Fridman (25:46.160)

with this two player game versus more players?

Lex Fridman (25:50.040)

What are the key differences in your view?

Lex Fridman (25:51.680)

So let me start from the basics.

Lex Fridman (25:54.600)

So a repeated game is a game where the same exact game

Tuomas Sandholm (25:59.600)

is played over and over.

Lex Fridman (26:01.800)

In these extensive form games, where it's,

Tuomas Sandholm (26:05.800)

think about three form, maybe with these information sets

Lex Fridman (26:08.480)

to represent incomplete information,

Tuomas Sandholm (26:11.400)

you can have kind of repetitive interactions.

Lex Fridman (26:14.840)

Even repeated games are a special case of that, by the way.

Lex Fridman (26:17.760)

But the game doesn't have to be exactly the same.

Lex Fridman (26:21.520)

It's like in sourcing auctions.

Tuomas Sandholm (26:23.040)

Yes, we're gonna see the same supply base year to year,

Lex Fridman (26:26.320)

but what I'm buying is a little different every time.

Lex Fridman (26:28.800)

And the supply base is a little different every time

Lex Fridman (26:31.000)

and so on.

Lex Fridman (26:31.840)

So it's not really repeated.

Lex Fridman (26:33.400)

So to find a purely repeated game

Tuomas Sandholm (26:35.680)

is actually very rare in the world.

Lex Fridman (26:37.840)

So they're really a very course model of what's going on.

Tuomas Sandholm (26:42.840)

Then if you move up from just repeated,

Lex Fridman (26:46.360)

simple repeated matrix games,

Tuomas Sandholm (26:49.040)

not all the way to extensive form games,

Lex Fridman (26:50.800)

but in between, they're stochastic games,

Tuomas Sandholm (26:53.600)

where, you know, there's these,

Lex Fridman (26:57.000)

you think about it like these little matrix games.

Lex Fridman (27:00.520)

And when you take an action and your opponent takes an action,

Lex Fridman (27:04.200)

they determine not which next state I'm going to,

Tuomas Sandholm (27:07.680)

next game I'm going to,

Lex Fridman (27:09.120)

but the distribution over next games

Tuomas Sandholm (27:11.440)

where I might be going to.

Lex Fridman (27:13.360)

So that's the stochastic game.

Lex Fridman (27:15.360)

But it's like matrix games, repeated stochastic games,

Lex Fridman (27:19.000)

extensive form games.

Tuomas Sandholm (27:20.400)

That is from less to more general.

Lex Fridman (27:23.040)

And poker is an example of the last one.

Lex Fridman (27:26.280)

So it's really in the most general setting.

Lex Fridman (27:29.560)

Extensive form games.

Lex Fridman (27:30.640)

And that's kind of what the AI community has been working on

Lex Fridman (27:34.520)

and being benchmarked on

Tuomas Sandholm (27:36.280)

with this Heads Up No Limit Texas Holdem.

Lex Fridman (27:38.040)

Can you describe extensive form games?

Lex Fridman (27:39.760)

What's the model here?

Lex Fridman (27:41.560)

Yeah, so if you're familiar with the tree form,

Lex Fridman (27:44.320)

so it's really the tree form.

Lex Fridman (27:45.760)

Like in chess, there's a search tree.

Tuomas Sandholm (27:47.560)

Versus a matrix.

Lex Fridman (27:48.720)

Versus a matrix, yeah.

Lex Fridman (27:50.080)

And the matrix is called the matrix form

Lex Fridman (27:53.000)

or bi matrix form or normal form game.

Lex Fridman (27:55.320)

And here you have the tree form.

Lex Fridman (27:57.080)

So you can actually do certain types of reasoning there

Tuomas Sandholm (28:00.000)

that you lose the information when you go to normal form.

Lex Fridman (28:04.680)

There's a certain form of equivalence.

Tuomas Sandholm (28:07.000)

Like if you go from tree form and you say it,

Lex Fridman (28:08.880)

every possible contingency plan is a strategy.

Tuomas Sandholm (28:12.720)

Then I can actually go back to the normal form,

Lex Fridman (28:15.080)

but I lose some information from the lack of sequentiality.

Tuomas Sandholm (28:18.600)

Then the multiplayer versus two player distinction

Lex Fridman (28:21.280)

is an important one.

Lex Fridman (28:22.880)

So two player games in zero sum

Lex Fridman (28:27.320)

are conceptually easier and computationally easier.

Tuomas Sandholm (28:32.840)

They're still huge like this one,

Lex Fridman (28:36.000)

but they're conceptually easier and computationally easier

Tuomas Sandholm (28:39.680)

in that conceptually, you don't have to worry about

Lex Fridman (28:42.920)

which equilibrium is the other guy going to play

Tuomas Sandholm (28:45.360)

when there are multiple,

Lex Fridman (28:46.640)

because any equilibrium strategy is a best response

Tuomas Sandholm (28:49.920)

to any other equilibrium strategy.

Lex Fridman (28:52.000)

So I can play a different equilibrium from you

Lex Fridman (28:54.360)

and we'll still get the right values of the game.

Lex Fridman (28:57.320)

That falls apart even with two players

Tuomas Sandholm (28:59.240)

when you have general sum games.

Lex Fridman (29:01.360)

Even without cooperation just in general.

Tuomas Sandholm (29:03.120)

Even without cooperation.

Lex Fridman (29:04.800)

So there's a big gap from two player zero sum

Tuomas Sandholm (29:07.640)

to two player general sum or even to three player zero sum.

Lex Fridman (29:11.160)

That's a big gap, at least in theory.

Lex Fridman (29:14.280)

Can you maybe non mathematically provide the intuition

Lex Fridman (29:18.920)

why it all falls apart with three or more players?

Tuomas Sandholm (29:22.120)

It seems like you should still be able to have

Lex Fridman (29:24.400)

a Nash equilibrium that's instructive, that holds.

Tuomas Sandholm (29:31.280)

Okay, so it is true that all finite games

Lex Fridman (29:36.000)

have a Nash equilibrium.

Lex Fridman (29:38.200)

So this is what John Nash actually proved.

Lex Fridman (29:41.080)

So they do have a Nash equilibrium.

Tuomas Sandholm (29:42.920)

That's not the problem.

Lex Fridman (29:43.840)

The problem is that there can be many.

Lex Fridman (29:46.600)

And then there's a question of which equilibrium to select.

Lex Fridman (29:50.400)

So, and if you select your strategy

Tuomas Sandholm (29:52.200)

from a different equilibrium and I select mine,

Lex Fridman (29:57.920)

then what does that mean?

Lex Fridman (29:59.920)

And in these non zero sum games,

Lex Fridman (30:02.080)

we may lose some joint benefit

Tuomas Sandholm (30:05.720)

by being just simply stupid.

Lex Fridman (30:07.040)

We could actually both be better off

Tuomas Sandholm (30:08.400)

if we did something else.

Lex Fridman (30:09.920)

And in three player, you get other problems

Tuomas Sandholm (30:11.760)

also like collusion.

Lex Fridman (30:13.200)

Like maybe you and I can gang up on a third player

Lex Fridman (30:16.560)

and we can do radically better by colluding.

Lex Fridman (30:19.800)

So there are lots of issues that come up there.

Lex Fridman (30:22.200)

So Noah Brown, the student you work with on this

Lex Fridman (30:25.640)

has mentioned, I looked through the AMA on Reddit.

Tuomas Sandholm (30:29.360)

He mentioned that the ability of poker players

Lex Fridman (30:31.280)

to collaborate will make the game.

Tuomas Sandholm (30:33.800)

He was asked the question of,

Lex Fridman (30:35.200)

how would you make the game of poker,

Tuomas Sandholm (30:37.920)

or both of you were asked the question,

Lex Fridman (30:39.280)

how would you make the game of poker

Lex Fridman (30:41.560)

beyond being solvable by current AI methods?

Lex Fridman (30:47.000)

And he said that there's not many ways

Tuomas Sandholm (30:50.560)

of making poker more difficult,

Lex Fridman (30:53.120)

but a collaboration or cooperation between players

Tuomas Sandholm (30:57.760)

would make it extremely difficult.

Lex Fridman (30:59.760)

So can you provide the intuition behind why that is,

Lex Fridman (31:03.320)

if you agree with that idea?

Lex Fridman (31:05.280)

Yeah, so I've done a lot of work on coalitional games

Lex Fridman (31:10.200)

and we actually have a paper here

Lex Fridman (31:11.680)

with my other student Gabriele Farina

Lex Fridman (31:13.680)

and some other collaborators at NIPS on that.

Lex Fridman (31:16.640)

Actually just came back from the poster session

Tuomas Sandholm (31:18.520)

where we presented this.

Lex Fridman (31:19.760)

But so when you have a collusion, it's a different problem.

Lex Fridman (31:23.800)

And it typically gets even harder then.

Lex Fridman (31:27.520)

Even the game representations,

Tuomas Sandholm (31:29.600)

some of the game representations don't really allow

Lex Fridman (31:33.600)

good computation.

Lex Fridman (31:34.480)

So we actually introduced a new game representation

Lex Fridman (31:37.600)

for that.

Lex Fridman (31:38.720)

Is that kind of cooperation part of the model?

Lex Fridman (31:42.040)

Are you, do you have, do you have information

Tuomas Sandholm (31:44.560)

about the fact that other players are cooperating

Lex Fridman (31:47.040)

or is it just this chaos that where nothing is known?

Lex Fridman (31:50.000)

So there's some things unknown.

Lex Fridman (31:52.360)

Can you give an example of a collusion type game

Lex Fridman (31:55.840)

or is it usually?

Lex Fridman (31:56.680)

So like bridge.

Lex Fridman (31:58.360)

So think about bridge.

Lex Fridman (31:59.640)

It's like when you and I are on a team,

Tuomas Sandholm (32:02.320)

our payoffs are the same.

Lex Fridman (32:04.480)

The problem is that we can't talk.

Lex Fridman (32:06.400)

So when I get my cards, I can't whisper to you

Lex Fridman (32:09.000)

what my cards are.

Tuomas Sandholm (32:10.320)

That would not be allowed.

Lex Fridman (32:12.480)

So we have to somehow coordinate our strategies

Tuomas Sandholm (32:16.080)

ahead of time and only ahead of time.

Lex Fridman (32:19.920)

And then there's certain signals we can talk about,

Lex Fridman (32:22.760)

but they have to be such that the other team

Lex Fridman (32:25.240)

also understands them.

Lex Fridman (32:26.840)

So that's an example where the coordination

Lex Fridman (32:30.440)

is already built into the rules of the game.

Lex Fridman (32:33.000)

But in many other situations like auctions

Lex Fridman (32:35.640)

or negotiations or diplomatic relationships, poker,

Tuomas Sandholm (32:40.880)

it's not really built in, but it still can be very helpful

Lex Fridman (32:44.160)

for the colluders.

Tuomas Sandholm (32:45.280)

I've read you write somewhere,

Lex Fridman (32:48.240)

the negotiations you come to the table with prior,

Tuomas Sandholm (32:52.800)

like a strategy that you're willing to do

Lex Fridman (32:56.080)

and not willing to do those kinds of things.

Lex Fridman (32:58.320)

So how do you start to now moving away from poker,

Lex Fridman (33:01.960)

moving beyond poker into other applications

Tuomas Sandholm (33:04.520)

like negotiations, how do you start applying this

Lex Fridman (33:07.000)

to other domains, even real world domains

Lex Fridman (33:11.640)

that you've worked on?

Lex Fridman (33:12.520)

Yeah, I actually have two startup companies

Tuomas Sandholm (33:14.440)

doing exactly that.

Lex Fridman (33:15.480)

One is called Strategic Machine,

Lex Fridman (33:17.800)

and that's for kind of business applications,

Lex Fridman (33:20.000)

gaming, sports, all sorts of things like that.

Tuomas Sandholm (33:22.880)

Any applications of this to business and to sports

Lex Fridman (33:27.200)

and to gaming, to various types of things

Tuomas Sandholm (33:32.120)

in finance, electricity markets and so on.

Lex Fridman (33:34.240)

And the other is called Strategy Robot,

Tuomas Sandholm (33:36.600)

where we are taking these to military security,

Lex Fridman (33:40.640)

cyber security and intelligence applications.

Tuomas Sandholm (33:43.520)

I think you worked a little bit in,

Lex Fridman (33:48.000)

how do you put it, advertisement,

Tuomas Sandholm (33:51.000)

sort of suggesting ads kind of thing, auction.

Lex Fridman (33:55.360)

That's another company, optimized markets.

Lex Fridman (33:57.800)

But that's much more about a combinatorial market

Lex Fridman (34:00.880)

and optimization based technology.

Tuomas Sandholm (34:02.840)

That's not using these game theoretic reasoning technologies.

Lex Fridman (34:06.840)

I see, okay, so what sort of high level

Lex Fridman (34:11.600)

do you think about our ability to use

Lex Fridman (34:15.280)

game theoretic concepts to model human behavior?

Lex Fridman (34:18.040)

Do you think human behavior is amenable

Lex Fridman (34:21.640)

to this kind of modeling outside of the poker games,

Lex Fridman (34:24.720)

and where have you seen it done successfully in your work?

Lex Fridman (34:27.520)

I'm not sure the goal really is modeling humans.

Tuomas Sandholm (34:33.640)

Like for example, if I'm playing a zero sum game,

Lex Fridman (34:36.480)

I don't really care that the opponent

Tuomas Sandholm (34:39.840)

is actually following my model of rational behavior,

Lex Fridman (34:42.960)

because if they're not, that's even better for me.

Tuomas Sandholm (34:46.400)

Right, so see with the opponents in games,

Lex Fridman (34:51.120)

the prerequisite is that you formalize

Tuomas Sandholm (34:56.120)

the interaction in some way

Lex Fridman (34:57.800)

that can be amenable to analysis.

Lex Fridman (35:01.000)

And you've done this amazing work with mechanism design,

Lex Fridman (35:04.160)

designing games that have certain outcomes.

Tuomas Sandholm (35:10.040)

But, so I'll tell you an example

Lex Fridman (35:12.320)

from my world of autonomous vehicles, right?

Tuomas Sandholm (35:15.460)

We're studying pedestrians,

Lex Fridman (35:17.040)

and pedestrians and cars negotiate

Tuomas Sandholm (35:20.200)

in this nonverbal communication.

Lex Fridman (35:22.160)

There's this weird game dance of tension

Tuomas Sandholm (35:25.040)

where pedestrians are basically saying,

Lex Fridman (35:27.280)

I trust that you won't kill me,

Lex Fridman (35:28.800)

and so as a jaywalker, I will step onto the road

Lex Fridman (35:31.840)

even though I'm breaking the law, and there's this tension.

Lex Fridman (35:34.720)

And the question is, we really don't know

Lex Fridman (35:36.640)

how to model that well in trying to model intent.

Lex Fridman (35:40.720)

And so people sometimes bring up ideas

Lex Fridman (35:43.080)

of game theory and so on.

Lex Fridman (35:44.880)

Do you think that aspect of human behavior

Lex Fridman (35:49.120)

can use these kinds of imperfect information approaches,

Tuomas Sandholm (35:53.080)

modeling, how do you start to attack a problem like that

Lex Fridman (35:57.860)

when you don't even know how to design the game

Lex Fridman (36:00.940)

to describe the situation in order to solve it?

Lex Fridman (36:04.280)

Okay, so I haven't really thought about jaywalking,

Lex Fridman (36:06.800)

but one thing that I think could be a good application

Lex Fridman (36:10.120)

in autonomous vehicles is the following.

Lex Fridman (36:13.000)

So let's say that you have fleets of autonomous cars

Lex Fridman (36:16.320)

operating by different companies.

Lex Fridman (36:18.340)

So maybe here's the Waymo fleet and here's the Uber fleet.

Lex Fridman (36:22.120)

If you think about the rules of the road,

Tuomas Sandholm (36:24.320)

they define certain legal rules,

Lex Fridman (36:26.560)

but that still leaves a huge strategy space open.

Tuomas Sandholm (36:30.080)

Like as a simple example, when cars merge,

Lex Fridman (36:32.840)

how humans merge, they slow down and look at each other

Lex Fridman (36:36.000)

and try to merge.

Lex Fridman (36:39.240)

Wouldn't it be better if these situations

Tuomas Sandholm (36:40.920)

would already be prenegotiated

Lex Fridman (36:43.480)

so we can actually merge at full speed

Lex Fridman (36:45.200)

and we know that this is the situation,

Lex Fridman (36:47.440)

this is how we do it, and it's all gonna be faster.

Lex Fridman (36:50.540)

But there are way too many situations to negotiate manually.

Lex Fridman (36:54.120)

So you could use automated negotiation,

Tuomas Sandholm (36:56.400)

this is the idea at least,

Lex Fridman (36:57.780)

you could use automated negotiation

Tuomas Sandholm (36:59.840)

to negotiate all of these situations

Lex Fridman (37:02.060)

or many of them in advance.

Lex Fridman (37:04.320)

And of course it might be that,

Lex Fridman (37:05.460)

hey, maybe you're not gonna always let me go first.

Tuomas Sandholm (37:09.180)

Maybe you said, okay, well, in these situations,

Lex Fridman (37:11.280)

I'll let you go first, but in exchange,

Tuomas Sandholm (37:13.560)

you're gonna give me too much,

Lex Fridman (37:14.520)

you're gonna let me go first in this situation.

Lex Fridman (37:17.260)

So it's this huge combinatorial negotiation.

Lex Fridman (37:20.680)

And do you think there's room in that example of merging

Tuomas Sandholm (37:24.080)

to model this whole situation

Lex Fridman (37:25.600)

as an imperfect information game

Lex Fridman (37:27.160)

or do you really want to consider it to be a perfect?

Lex Fridman (37:30.120)

No, that's a good question, yeah.

Tuomas Sandholm (37:32.240)

That's a good question.

Lex Fridman (37:33.080)

Do you pay the price of assuming

Lex Fridman (37:37.080)

that you don't know everything?

Lex Fridman (37:39.800)

Yeah, I don't know.

Tuomas Sandholm (37:40.760)

It's certainly much easier.

Lex Fridman (37:42.120)

Games with perfect information are much easier.

Lex Fridman (37:45.060)

So if you can't get away with it, you should.

Lex Fridman (37:49.280)

But if the real situation is of imperfect information,

Tuomas Sandholm (37:52.640)

then you're gonna have to deal with imperfect information.

Lex Fridman (37:55.160)

Great, so what lessons have you learned

Lex Fridman (37:58.080)

the Annual Computer Poker Competition?

Lex Fridman (38:00.680)

An incredible accomplishment of AI.

Tuomas Sandholm (38:03.440)

You look at the history of Deep Blue, AlphaGo,

Lex Fridman (38:07.000)

these kind of moments when AI stepped up

Tuomas Sandholm (38:10.400)

in an engineering effort and a scientific effort combined

Lex Fridman (38:13.960)

to beat the best of human players.

Lex Fridman (38:16.400)

So what do you take away from this whole experience?

Lex Fridman (38:19.480)

What have you learned about designing AI systems

Lex Fridman (38:22.440)

that play these kinds of games?

Lex Fridman (38:23.960)

And what does that mean for AI in general,

Lex Fridman (38:28.280)

for the future of AI development?

Lex Fridman (38:30.760)

Yeah, so that's a good question.

Lex Fridman (38:32.800)

So there's so much to say about it.

Lex Fridman (38:35.440)

I do like this type of performance oriented research.

Tuomas Sandholm (38:39.120)

Although in my group, we go all the way from like idea

Lex Fridman (38:42.000)

to theory, to experiments, to big system building,

Tuomas Sandholm (38:44.880)

to commercialization, so we span that spectrum.

Lex Fridman (38:47.960)

But I think that in a lot of situations in AI,

Tuomas Sandholm (38:51.080)

you really have to build the big systems

Lex Fridman (38:53.440)

and evaluate them at scale

Tuomas Sandholm (38:55.640)

before you know what works and doesn't.

Lex Fridman (38:57.520)

And we've seen that in the computational

Tuomas Sandholm (39:00.080)

game theory community, that there are a lot of techniques

Lex Fridman (39:02.880)

that look good in the small,

Lex Fridman (39:05.200)

but then they cease to look good in the large.

Lex Fridman (39:07.120)

And we've also seen that there are a lot of techniques

Tuomas Sandholm (39:10.160)

that look superior in theory.

Lex Fridman (39:13.280)

And I really mean in terms of convergence rates,

Tuomas Sandholm (39:16.200)

like first order methods, better convergence rates,

Lex Fridman (39:18.440)

like the CFR based algorithms,

Tuomas Sandholm (39:20.880)

yet the CFR based algorithms are the fastest in practice.

Lex Fridman (39:24.880)

So it really tells me that you have to test this in reality.

Tuomas Sandholm (39:28.240)

The theory isn't tight enough, if you will,

Lex Fridman (39:30.880)

to tell you which algorithms are better than the others.

Lex Fridman (39:34.360)

And you have to look at these things in the large,

Lex Fridman (39:38.600)

because any sort of projections you do from the small

Tuomas Sandholm (39:41.480)

can at least in this domain be very misleading.

Lex Fridman (39:43.800)

So that's kind of from a kind of a science

Lex Fridman (39:46.240)

and engineering perspective, from a personal perspective,

Lex Fridman (39:49.120)

it's been just a wild experience

Tuomas Sandholm (39:51.280)

in that with the first poker competition,

Lex Fridman (39:54.160)

the first brains versus AI,

Tuomas Sandholm (39:57.200)

man machine poker competition that we organized.

Lex Fridman (39:59.840)

There had been, by the way, for other poker games,

Tuomas Sandholm (40:01.760)

there had been previous competitions,

Lex Fridman (40:03.240)

but this was for Heads Up No Limit, this was the first.

Lex Fridman (40:06.360)

And I probably became the most hated person

Lex Fridman (40:09.560)

in the world of poker.

Lex Fridman (40:10.880)

And I didn't mean to, I just saw.

Lex Fridman (40:12.880)

Why is that?

Tuomas Sandholm (40:13.720)

For cracking the game, for something.

Lex Fridman (40:15.840)

Yeah, a lot of people felt that it was a real threat

Tuomas Sandholm (40:20.000)

to the whole game, the whole existence of the game.

Lex Fridman (40:22.760)

If AI becomes better than humans,

Tuomas Sandholm (40:26.080)

people would be scared to play poker

Lex Fridman (40:28.520)

because there are these superhuman AIs running around

Tuomas Sandholm (40:30.680)

taking their money and all of that.

Lex Fridman (40:32.760)

So I just, it's just really aggressive.

Tuomas Sandholm (40:36.200)

The comments were super aggressive.

Lex Fridman (40:37.880)

I got everything just short of death threats.

Lex Fridman (40:40.920)

Do you think the same was true for chess?

Lex Fridman (40:44.000)

Because right now they just completed

Tuomas Sandholm (40:45.760)

the world championships in chess,

Lex Fridman (40:47.720)

and humans just started ignoring the fact

Tuomas Sandholm (40:49.560)

that there's AI systems now that outperform humans

Lex Fridman (40:52.920)

and they still enjoy the game, it's still a beautiful game.

Tuomas Sandholm (40:55.520)

That's what I think.

Lex Fridman (40:56.360)

And I think the same thing happens in poker.

Lex Fridman (40:58.800)

And so I didn't think of myself

Lex Fridman (41:01.040)

as somebody who was gonna kill the game,

Lex Fridman (41:02.360)

and I don't think I did.

Lex Fridman (41:03.800)

I've really learned to love this game.

Tuomas Sandholm (41:05.600)

I wasn't a poker player before,

Lex Fridman (41:06.960)

but learned so many nuances about it from these AIs,

Lex Fridman (41:10.520)

and they've really changed how the game is played,

Lex Fridman (41:12.480)

by the way.

Lex Fridman (41:13.320)

So they have these very Martian ways of playing poker,

Lex Fridman (41:16.240)

and the top humans are now incorporating

Tuomas Sandholm (41:18.400)

those types of strategies into their own play.

Lex Fridman (41:21.400)

So if anything, to me, our work has made poker

Tuomas Sandholm (41:26.560)

a richer, more interesting game for humans to play,

Lex Fridman (41:29.800)

not something that is gonna steer humans

Tuomas Sandholm (41:32.160)

away from it entirely.

Lex Fridman (41:34.200)

Just a quick comment on something you said,

Tuomas Sandholm (41:35.960)

which is, if I may say so,

Lex Fridman (41:39.400)

in academia is a little bit rare sometimes.

Tuomas Sandholm (41:42.400)

It's pretty brave to put your ideas to the test

Lex Fridman (41:45.520)

in the way you described,

Tuomas Sandholm (41:47.200)

saying that sometimes good ideas don't work

Lex Fridman (41:49.360)

when you actually try to apply them at scale.

Lex Fridman (41:52.760)

So where does that come from?

Lex Fridman (41:54.200)

I mean, if you could do advice for people,

Lex Fridman (41:58.880)

what drives you in that sense?

Lex Fridman (42:00.760)

Were you always this way?

Tuomas Sandholm (42:02.360)

I mean, it takes a brave person.

Lex Fridman (42:04.080)

I guess is what I'm saying, to test their ideas

Lex Fridman (42:06.760)

and to see if this thing actually works

Lex Fridman (42:08.640)

against human top human players and so on.

Tuomas Sandholm (42:11.680)

Yeah, I don't know about brave,

Lex Fridman (42:12.960)

but it takes a lot of work.

Tuomas Sandholm (42:15.000)

It takes a lot of work and a lot of time

Lex Fridman (42:18.400)

to organize, to make something big

Lex Fridman (42:20.360)

and to organize an event and stuff like that.

Lex Fridman (42:22.920)

And what drives you in that effort?

Tuomas Sandholm (42:24.760)

Because you could still, I would argue,

Lex Fridman (42:26.880)

get a best paper award at NIPS as you did in 17

Tuomas Sandholm (42:30.280)

without doing this.

Lex Fridman (42:31.440)

That's right, yes.

Lex Fridman (42:32.960)

And so in general, I believe it's very important

Lex Fridman (42:37.640)

to do things in the real world and at scale.

Lex Fridman (42:41.480)

And that's really where the pudding, if you will,

Lex Fridman (42:46.160)

proof is in the pudding, that's where it is.

Tuomas Sandholm (42:48.400)

In this particular case,

Lex Fridman (42:50.080)

it was kind of a competition between different groups

Lex Fridman (42:55.160)

and for many years as to who can be the first one

Lex Fridman (42:59.080)

to beat the top humans at Heads Up No Limit, Texas Holdem.

Lex Fridman (43:02.040)

So it became kind of like a competition who can get there.

Lex Fridman (43:09.560)

Yeah, so a little friendly competition

Tuomas Sandholm (43:11.800)

could do wonders for progress.

Lex Fridman (43:14.040)

Yes, absolutely.

Lex Fridman (43:16.400)

So the topic of mechanism design,

Lex Fridman (43:19.040)

which is really interesting, also kind of new to me,

Tuomas Sandholm (43:22.280)

except as an observer of, I don't know, politics and any,

Lex Fridman (43:25.680)

I'm an observer of mechanisms,

Lex Fridman (43:27.600)

but you write in your paper an automated mechanism design

Lex Fridman (43:31.440)

that I quickly read.

Lex Fridman (43:34.000)

So mechanism design is designing the rules of the game

Lex Fridman (43:37.880)

so you get a certain desirable outcome.

Lex Fridman (43:40.200)

And you have this work on doing so in an automatic fashion

Lex Fridman (43:44.920)

as opposed to fine tuning it.

Lex Fridman (43:46.720)

So what have you learned from those efforts?

Lex Fridman (43:50.680)

If you look, say, I don't know,

Tuomas Sandholm (43:52.280)

at complexes like our political system,

Lex Fridman (43:56.200)

can we design our political system

Tuomas Sandholm (43:58.560)

to have, in an automated fashion,

Lex Fridman (44:01.800)

to have outcomes that we want?

Tuomas Sandholm (44:03.360)

Can we design something like traffic lights to be smart

Lex Fridman (44:09.000)

where it gets outcomes that we want?

Lex Fridman (44:11.800)

So what are the lessons that you draw from that work?

Lex Fridman (44:14.840)

Yeah, so I still very much believe

Tuomas Sandholm (44:17.240)

in the automated mechanism design direction.

Lex Fridman (44:19.400)

Yes.

Lex Fridman (44:20.840)

But it's not a panacea.

Lex Fridman (44:23.000)

There are impossibility results in mechanism design

Tuomas Sandholm (44:26.520)

saying that there is no mechanism that accomplishes

Lex Fridman (44:30.240)

objective X in class C.

Lex Fridman (44:33.920)

So it's not going to,

Lex Fridman (44:36.120)

there's no way using any mechanism design tools,

Tuomas Sandholm (44:39.000)

manual or automated,

Lex Fridman (44:41.000)

to do certain things in mechanism design.

Lex Fridman (44:42.800)

Can you describe that again?

Lex Fridman (44:43.800)

So meaning it's impossible to achieve that?

Tuomas Sandholm (44:47.480)

Yeah, yeah.

Lex Fridman (44:48.320)

And it's unlikely.

Tuomas Sandholm (44:50.440)

Impossible.

Lex Fridman (44:51.280)

Impossible.

Lex Fridman (44:52.120)

So these are not statements about human ingenuity

Lex Fridman (44:55.240)

who might come up with something smart.

Tuomas Sandholm (44:57.120)

These are proofs that if you want to accomplish

Lex Fridman (44:59.880)

properties X in class C,

Tuomas Sandholm (45:02.480)

that is not doable with any mechanism.

Lex Fridman (45:04.880)

The good thing about automated mechanism design

Tuomas Sandholm (45:07.080)

is that we're not really designing for a class.

Lex Fridman (45:10.840)

We're designing for specific settings at a time.

Lex Fridman (45:14.160)

So even if there's an impossibility result

Lex Fridman (45:16.720)

for the whole class,

Tuomas Sandholm (45:18.240)

it just doesn't mean that all of the cases

Lex Fridman (45:21.360)

in the class are impossible.

Tuomas Sandholm (45:22.560)

It just means that some of the cases are impossible.

Lex Fridman (45:25.080)

So we can actually carve these islands of possibility

Tuomas Sandholm (45:28.240)

within these known impossible classes.

Lex Fridman (45:30.920)

And we've actually done that.

Lex Fridman (45:31.960)

So one of the famous results in mechanism design

Lex Fridman (45:35.160)

is the Meyerson Satethweight theorem

Tuomas Sandholm (45:37.360)

by Roger Meyerson and Mark Satethweight from 1983.

Lex Fridman (45:41.000)

It's an impossibility of efficient trade

Tuomas Sandholm (45:43.480)

under imperfect information.

Lex Fridman (45:45.200)

We show that you can, in many settings,

Tuomas Sandholm (45:48.560)

avoid that and get efficient trade anyway.

Lex Fridman (45:51.480)

Depending on how they design the game, okay.

Tuomas Sandholm (45:54.160)

Depending how you design the game.

Lex Fridman (45:55.880)

And of course, it doesn't in any way

Tuomas Sandholm (46:00.240)

contradict the impossibility result.

Lex Fridman (46:01.800)

The impossibility result is still there,

Lex Fridman (46:03.920)

but it just finds spots within this impossible class

Lex Fridman (46:08.920)

where in those spots, you don't have the impossibility.

Tuomas Sandholm (46:12.440)

Sorry if I'm going a bit philosophical,

Lex Fridman (46:14.760)

but what lessons do you draw towards,

Tuomas Sandholm (46:17.480)

like I mentioned, politics or human interaction

Lex Fridman (46:20.160)

and designing mechanisms for outside of just

Tuomas Sandholm (46:24.880)

these kinds of trading or auctioning

Lex Fridman (46:26.960)

or purely formal games or human interaction,

Lex Fridman (46:33.480)

like a political system?

Lex Fridman (46:34.920)

How, do you think it's applicable to, yeah, politics

Tuomas Sandholm (46:39.160)

or to business, to negotiations, these kinds of things,

Lex Fridman (46:46.280)

designing rules that have certain outcomes?

Tuomas Sandholm (46:49.040)

Yeah, yeah, I do think so.

Lex Fridman (46:51.360)

Have you seen that successfully done?

Tuomas Sandholm (46:54.200)

They haven't really, oh, you mean mechanism design

Lex Fridman (46:56.440)

or automated mechanism design?

Tuomas Sandholm (46:57.280)

Automated mechanism design.

Lex Fridman (46:59.000)

So mechanism design itself

Tuomas Sandholm (47:01.520)

has had fairly limited success so far.

Lex Fridman (47:06.440)

There are certain cases,

Lex Fridman (47:07.600)

but most of the real world situations

Lex Fridman (47:10.200)

are actually not sound from a mechanism design perspective,

Tuomas Sandholm (47:14.680)

even in those cases where they've been designed

Lex Fridman (47:16.920)

by very knowledgeable mechanism design people,

Tuomas Sandholm (47:20.000)

the people are typically just taking some insights

Lex Fridman (47:22.760)

from the theory and applying those insights

Tuomas Sandholm (47:25.040)

into the real world,

Lex Fridman (47:26.280)

rather than applying the mechanisms directly.

Lex Fridman (47:29.280)

So one famous example of is the FCC spectrum auctions.

Lex Fridman (47:33.520)

So I've also had a small role in that

Lex Fridman (47:36.880)

and very good economists have been working,

Lex Fridman (47:40.600)

excellent economists have been working on that

Tuomas Sandholm (47:42.560)

with no game theory,

Lex Fridman (47:44.040)

yet the rules that are designed in practice there,

Tuomas Sandholm (47:47.440)

they're such that bidding truthfully

Lex Fridman (47:49.840)

is not the best strategy.

Tuomas Sandholm (47:51.800)

Usually mechanism design,

Lex Fridman (47:52.960)

we try to make things easy for the participants.

Lex Fridman (47:56.160)

So telling the truth is the best strategy,

Lex Fridman (47:58.560)

but even in those very high stakes auctions

Tuomas Sandholm (48:01.480)

where you have tens of billions of dollars

Lex Fridman (48:03.080)

worth of spectrum being auctioned,

Tuomas Sandholm (48:06.360)

truth telling is not the best strategy.

Lex Fridman (48:08.280)

And by the way,

Tuomas Sandholm (48:10.040)

nobody knows even a single optimal bidding strategy

Lex Fridman (48:12.920)

for those auctions.

Tuomas Sandholm (48:14.120)

What's the challenge of coming up with an optimal,

Lex Fridman (48:15.960)

because there's a lot of players and there's imperfect.

Tuomas Sandholm (48:18.160)

It's not so much that a lot of players,

Lex Fridman (48:20.040)

but many items for sale,

Lex Fridman (48:22.320)

and these mechanisms are such that even with just two items

Lex Fridman (48:26.000)

or one item, bidding truthfully

Tuomas Sandholm (48:28.400)

wouldn't be the best strategy.

Lex Fridman (48:31.400)

If you look at the history of AI,

Tuomas Sandholm (48:34.560)

it's marked by seminal events.

Lex Fridman (48:37.160)

AlphaGo beating a world champion human Go player,

Tuomas Sandholm (48:40.160)

I would put Liberatus winning the Heads Up No Limit Holdem

Lex Fridman (48:43.680)

as one of such event.

Tuomas Sandholm (48:45.000)

Thank you.

Lex Fridman (48:46.040)

And what do you think is the next such event,

Tuomas Sandholm (48:52.560)

whether it's in your life or in the broadly AI community

Lex Fridman (48:56.640)

that you think might be out there

Lex Fridman (48:59.040)

that would surprise the world?

Lex Fridman (49:01.640)

So that's a great question,

Lex Fridman (49:02.800)

and I don't really know the answer.

Lex Fridman (49:04.520)

In terms of game solving,

Tuomas Sandholm (49:07.360)

Heads Up No Limit Texas Holdem

Lex Fridman (49:08.920)

really was the one remaining widely agreed upon benchmark.

Lex Fridman (49:14.400)

So that was the big milestone.

Lex Fridman (49:15.880)

Now, are there other things?

Tuomas Sandholm (49:17.800)

Yeah, certainly there are,

Lex Fridman (49:18.920)

but there's not one that the community

Tuomas Sandholm (49:21.080)

has kind of focused on.

Lex Fridman (49:22.920)

So what could be other things?

Tuomas Sandholm (49:25.240)

There are groups working on StarCraft.

Lex Fridman (49:27.640)

There are groups working on Dota 2.

Tuomas Sandholm (49:29.840)

These are video games.

Lex Fridman (49:31.560)

Or you could have like Diplomacy or Hanabi,

Tuomas Sandholm (49:36.240)

things like that.

Lex Fridman (49:37.080)

These are like recreational games,

Lex Fridman (49:38.640)

but none of them are really acknowledged

Lex Fridman (49:42.040)

as kind of the main next challenge problem,

Tuomas Sandholm (49:45.840)

like chess or Go or Heads Up No Limit Texas Holdem was.

Lex Fridman (49:50.000)

So I don't really know in the game solving space

Lex Fridman (49:52.360)

what is or what will be the next benchmark.

Lex Fridman (49:55.400)

I kind of hope that there will be a next benchmark

Tuomas Sandholm (49:57.840)

because really the different groups

Lex Fridman (49:59.560)

working on the same problem

Tuomas Sandholm (50:01.120)

really drove these application independent techniques

Lex Fridman (50:05.120)

forward very quickly over 10 years.

Lex Fridman (50:07.480)

Do you think there's an open problem

Lex Fridman (50:09.120)

that excites you that you start moving away

Tuomas Sandholm (50:11.480)

from games into real world games,

Lex Fridman (50:15.000)

like say the stock market trading?

Tuomas Sandholm (50:17.200)

Yeah, so that's kind of how I am.

Lex Fridman (50:19.320)

So I am probably not going to work

Tuomas Sandholm (50:23.120)

as hard on these recreational benchmarks.

Lex Fridman (50:27.640)

I'm doing two startups on game solving technology,

Tuomas Sandholm (50:30.200)

Strategic Machine and Strategy Robot,

Lex Fridman (50:32.320)

and we're really interested

Tuomas Sandholm (50:34.160)

in pushing this stuff into practice.

Lex Fridman (50:36.560)

What do you think would be really

Tuomas Sandholm (50:43.160)

a powerful result that would be surprising

Lex Fridman (50:45.920)

that would be, if you can say,

Tuomas Sandholm (50:49.960)

I mean, five years, 10 years from now,

Lex Fridman (50:53.280)

something that statistically you would say

Tuomas Sandholm (50:56.480)

is not very likely,

Lex Fridman (50:57.920)

but if there's a breakthrough, would achieve?

Tuomas Sandholm (51:01.480)

Yeah, so I think that overall,

Lex Fridman (51:03.800)

we're in a very different situation in game theory

Tuomas Sandholm (51:09.000)

than we are in, let's say, machine learning.

Lex Fridman (51:11.760)

So in machine learning, it's a fairly mature technology

Lex Fridman (51:14.360)

and it's very broadly applied

Lex Fridman (51:16.480)

and proven success in the real world.

Tuomas Sandholm (51:19.680)

In game solving, there are almost no applications yet.

Lex Fridman (51:24.320)

We have just become superhuman,

Tuomas Sandholm (51:26.680)

which machine learning you could argue happened in the 90s,

Lex Fridman (51:29.600)

if not earlier,

Lex Fridman (51:30.640)

and at least on supervised learning,

Lex Fridman (51:32.960)

certain complex supervised learning applications.

Tuomas Sandholm (51:36.960)

Now, I think the next challenge problem,

Lex Fridman (51:39.000)

I know you're not asking about it this way,

Tuomas Sandholm (51:40.560)

you're asking about the technology breakthrough,

Lex Fridman (51:42.640)

but I think that big, big breakthrough

Tuomas Sandholm (51:44.240)

is to be able to show that, hey,

Lex Fridman (51:46.120)

maybe most of, let's say, military planning

Tuomas Sandholm (51:48.280)

or most of business strategy

Lex Fridman (51:50.080)

will actually be done strategically

Tuomas Sandholm (51:52.200)

using computational game theory.

Lex Fridman (51:54.120)

That's what I would like to see

Tuomas Sandholm (51:55.800)

as the next five or 10 year goal.

Lex Fridman (51:57.640)

Maybe you can explain to me again,

Tuomas Sandholm (51:59.520)

forgive me if this is an obvious question,

Lex Fridman (52:01.920)

but machine learning methods,

Tuomas Sandholm (52:04.000)

neural networks suffer from not being transparent,

Lex Fridman (52:07.840)

not being explainable.

Tuomas Sandholm (52:09.280)

Game theoretic methods, Nash equilibria,

Lex Fridman (52:12.400)

do they generally, when you see the different solutions,

Tuomas Sandholm (52:15.280)

are they, when you talk about military operations,

Lex Fridman (52:19.640)

are they, once you see the strategies,

Tuomas Sandholm (52:21.800)

do they make sense, are they explainable,

Lex Fridman (52:23.880)

or do they suffer from the same problems

Lex Fridman (52:25.840)

as neural networks do?

Lex Fridman (52:27.120)

So that's a good question.

Tuomas Sandholm (52:28.720)

I would say a little bit yes and no.

Lex Fridman (52:31.240)

And what I mean by that is that

Tuomas Sandholm (52:34.560)

these game theoretic strategies,

Lex Fridman (52:36.160)

let's say, Nash equilibrium,

Tuomas Sandholm (52:38.520)

it has provable properties.

Lex Fridman (52:40.320)

So it's unlike, let's say, deep learning

Tuomas Sandholm (52:42.360)

where you kind of cross your fingers,

Lex Fridman (52:44.440)

hopefully it'll work.

Lex Fridman (52:45.680)

And then after the fact, when you have the weights,

Lex Fridman (52:47.880)

you're still crossing your fingers,

Lex Fridman (52:48.920)

and I hope it will work.

Lex Fridman (52:51.160)

Here, you know that the solution quality is there.

Tuomas Sandholm (52:55.400)

There's provable solution quality guarantees.

Lex Fridman (52:58.040)

Now, that doesn't necessarily mean

Tuomas Sandholm (53:00.920)

that the strategies are human understandable.

Lex Fridman (53:03.480)

That's a whole other problem.

Lex Fridman (53:04.720)

So I think that deep learning and computational game theory

Lex Fridman (53:08.680)

are in the same boat in that sense,

Tuomas Sandholm (53:10.720)

that both are difficult to understand.

Lex Fridman (53:13.760)

But at least the game theoretic techniques,

Tuomas Sandholm (53:15.680)

they have these guarantees of solution quality.

Lex Fridman (53:19.840)

So do you see business operations, strategic operations,

Tuomas Sandholm (53:22.880)

or even military in the future being

Lex Fridman (53:26.040)

at least the strong candidates

Lex Fridman (53:28.320)

being proposed by automated systems?

Lex Fridman (53:32.760)

Do you see that?

Tuomas Sandholm (53:34.000)

Yeah, I do, I do.

Lex Fridman (53:35.040)

But that's more of a belief than a substantiated fact.

Tuomas Sandholm (53:39.640)

Depending on where you land in optimism or pessimism,

Lex Fridman (53:42.320)

that's a really, to me, that's an exciting future,

Tuomas Sandholm (53:45.720)

especially if there's provable things

Lex Fridman (53:48.760)

in terms of optimality.

Lex Fridman (53:50.560)

So looking into the future,

Lex Fridman (53:54.040)

there's a few folks worried about the,

Tuomas Sandholm (53:58.760)

especially you look at the game of poker,

Lex Fridman (54:01.200)

which is probably one of the last benchmarks

Tuomas Sandholm (54:03.360)

in terms of games being solved.

Lex Fridman (54:05.480)

They worry about the future

Lex Fridman (54:07.520)

and the existential threats of artificial intelligence.

Lex Fridman (54:10.520)

So the negative impact in whatever form on society.

Tuomas Sandholm (54:13.840)

Is that something that concerns you as much,

Lex Fridman (54:17.440)

or are you more optimistic about the positive impacts of AI?

Tuomas Sandholm (54:21.600)

Oh, I am much more optimistic about the positive impacts.

Lex Fridman (54:24.720)

So just in my own work, what we've done so far,

Tuomas Sandholm (54:27.560)

we run the nationwide kidney exchange.

Lex Fridman (54:29.920)

Hundreds of people are walking around alive today,

Lex Fridman (54:32.960)

who would it be?

Lex Fridman (54:34.080)

And it's increased employment.

Tuomas Sandholm (54:36.120)

You have a lot of people now running kidney exchanges

Lex Fridman (54:39.920)

and at the transplant centers,

Tuomas Sandholm (54:42.200)

interacting with the kidney exchange.

Lex Fridman (54:45.560)

You have extra surgeons, nurses, anesthesiologists,

Tuomas Sandholm (54:49.440)

hospitals, all of that.

Lex Fridman (54:51.400)

So employment is increasing from that

Lex Fridman (54:53.560)

and the world is becoming a better place.

Lex Fridman (54:55.320)

Another example is combinatorial sourcing auctions.

Tuomas Sandholm (54:59.040)

We did 800 large scale combinatorial sourcing auctions

Lex Fridman (55:04.040)

from 2001 to 2010 in a previous startup of mine

Tuomas Sandholm (55:08.240)

called CombineNet.

Lex Fridman (55:09.400)

And we increased the supply chain efficiency

Tuomas Sandholm (55:13.080)

on that $60 billion of spend by 12.6%.

Lex Fridman (55:18.080)

So that's over $6 billion of efficiency improvement

Tuomas Sandholm (55:21.440)

in the world.

Lex Fridman (55:22.240)

And this is not like shifting value

Tuomas Sandholm (55:23.760)

from somebody to somebody else,

Lex Fridman (55:25.240)

just efficiency improvement, like in trucking,

Tuomas Sandholm (55:28.200)

less empty driving, so there's less waste,

Lex Fridman (55:31.120)

less carbon footprint and so on.

Lex Fridman (55:33.440)

So a huge positive impact in the near term,

Lex Fridman (55:36.720)

but sort of to stay in it for a little longer,

Tuomas Sandholm (55:40.680)

because I think game theory has a role to play here.

Lex Fridman (55:43.080)

Oh, let me actually come back on that as one thing.

Tuomas Sandholm (55:45.320)

I think AI is also going to make the world much safer.

Lex Fridman (55:49.400)

So that's another aspect that often gets overlooked.

Tuomas Sandholm (55:53.760)

Well, let me ask this question.

Lex Fridman (55:54.920)

Maybe you can speak to the safer.

Lex Fridman (55:56.960)

So I talked to Max Tegmark and Stuart Russell,

Lex Fridman (55:59.960)

who are very concerned about existential threats of AI.

Lex Fridman (56:02.680)

And often the concern is about value misalignment.

Lex Fridman (56:06.240)

So AI systems basically working,

Tuomas Sandholm (56:11.880)

operating towards goals that are not the same

Lex Fridman (56:14.680)

as human civilization, human beings.

Lex Fridman (56:17.920)

So it seems like game theory has a role to play there

Lex Fridman (56:24.200)

to make sure the values are aligned with human beings.

Tuomas Sandholm (56:27.880)

I don't know if that's how you think about it.

Lex Fridman (56:29.960)

If not, how do you think AI might help with this problem?

Lex Fridman (56:35.280)

How do you think AI might make the world safer?

Lex Fridman (56:39.240)

Yeah, I think this value misalignment

Tuomas Sandholm (56:43.000)

is a fairly theoretical worry.

Lex Fridman (56:46.480)

And I haven't really seen it in,

Tuomas Sandholm (56:49.960)

because I do a lot of real applications.

Lex Fridman (56:51.840)

I don't see it anywhere.

Tuomas Sandholm (56:53.920)

The closest I've seen it

Lex Fridman (56:55.240)

was the following type of mental exercise really,

Tuomas Sandholm (56:57.920)

where I had this argument in the late eighties

Lex Fridman (57:00.720)

when we were building

Tuomas Sandholm (57:01.560)

these transportation optimization systems.

Lex Fridman (57:03.560)

And somebody had heard that it's a good idea

Tuomas Sandholm (57:05.360)

to have high utilization of assets.

Lex Fridman (57:08.160)

So they told me, hey, why don't you put that as objective?

Lex Fridman (57:11.400)

And we didn't even put it as an objective

Lex Fridman (57:14.720)

because I just showed him that,

Tuomas Sandholm (57:16.480)

if you had that as your objective,

Lex Fridman (57:18.480)

the solution would be to load your trucks full

Lex Fridman (57:20.320)

and drive in circles.

Lex Fridman (57:21.840)

Nothing would ever get delivered.

Tuomas Sandholm (57:23.000)

You'd have a hundred percent utilization.

Lex Fridman (57:25.120)

So yeah, I know this phenomenon.

Tuomas Sandholm (57:27.240)

I've known this for over 30 years,

Lex Fridman (57:29.680)

but I've never seen it actually be a problem in reality.

Lex Fridman (57:33.360)

And yes, if you have the wrong objective,

Lex Fridman (57:35.240)

the AI will optimize that to the hilt

Lex Fridman (57:37.800)

and it's gonna hurt more than some human

Lex Fridman (57:39.800)

who's kind of trying to solve it in a half baked way

Tuomas Sandholm (57:43.800)

with some human insight too.

Lex Fridman (57:45.480)

But I just haven't seen that materialize in practice.

Tuomas Sandholm (57:49.160)

There's this gap that you've actually put your finger on

Lex Fridman (57:52.720)

very clearly just now between theory and reality.

Tuomas Sandholm (57:57.080)

That's very difficult to put into words, I think.

Lex Fridman (57:59.680)

It's what you can theoretically imagine,

Tuomas Sandholm (58:03.240)

the worst possible case or even, yeah, I mean bad cases.

Lex Fridman (58:08.000)

And what usually happens in reality.

Lex Fridman (58:10.520)

So for example, to me,

Lex Fridman (58:11.960)

maybe it's something you can comment on having grown up

Lex Fridman (58:15.720)

and I grew up in the Soviet Union.

Lex Fridman (58:19.120)

There's currently 10,000 nuclear weapons in the world.

Lex Fridman (58:22.120)

And for many decades,

Lex Fridman (58:24.200)

it's theoretically surprising to me

Tuomas Sandholm (58:28.360)

that the nuclear war is not broken out.

Lex Fridman (58:30.880)

Do you think about this aspect

Tuomas Sandholm (58:33.760)

from a game theoretic perspective in general,

Lex Fridman (58:36.080)

why is that true?

Lex Fridman (58:38.440)

Why in theory you could see

Lex Fridman (58:40.720)

how things would go terribly wrong

Lex Fridman (58:42.600)

and somehow yet they have not?

Lex Fridman (58:44.280)

Yeah, how do you think about it?

Lex Fridman (58:45.600)

So I do think about that a lot.

Lex Fridman (58:47.240)

I think the biggest two threats that we're facing as mankind,

Tuomas Sandholm (58:50.320)

one is climate change and the other is nuclear war.

Lex Fridman (58:53.320)

So those are my main two worries that I worry about.

Lex Fridman (58:57.200)

And I've tried to do something about climate,

Lex Fridman (58:59.920)

thought about trying to do something

Tuomas Sandholm (59:01.320)

for climate change twice.

Lex Fridman (59:02.880)

Actually, for two of my startups,

Tuomas Sandholm (59:05.040)

I've actually commissioned studies

Lex Fridman (59:06.760)

of what we could do on those things.

Lex Fridman (59:09.480)

And we didn't really find a sweet spot,

Lex Fridman (59:11.040)

but I'm still keeping an eye out on that.

Tuomas Sandholm (59:12.680)

If there's something where we could actually

Lex Fridman (59:15.160)

provide a market solution or optimizations solution

Tuomas Sandholm (59:17.800)

or some other technology solution to problems.

Lex Fridman (59:20.960)

Right now, like for example,

Tuomas Sandholm (59:23.360)

pollution critic markets was what we were looking at then.

Lex Fridman (59:26.760)

And it was much more the lack of political will

Tuomas Sandholm (59:30.040)

by those markets were not so successful

Lex Fridman (59:32.840)

rather than bad market design.

Lex Fridman (59:34.640)

So I could go in and make a better market design,

Lex Fridman (59:37.080)

but that wouldn't really move the needle

Tuomas Sandholm (59:38.600)

on the world very much if there's no political will.

Lex Fridman (59:41.160)

And in the US, the market,

Tuomas Sandholm (59:43.600)

at least the Chicago market was just shut down and so on.

Lex Fridman (59:47.520)

So then it doesn't really help

Lex Fridman (59:48.760)

how great your market design was.

Lex Fridman (59:51.040)

And then the nuclear side, it's more,

Lex Fridman (59:53.560)

so global warming is a more encroaching problem.