Lex Fridman (47:54.760)
from a physics perspective, how much is determined,
John Hopfield (47:58.400)
already predetermined, how much is already deterministic
Lex Fridman (48:01.060)
about our universe, and there's lots of different things.
Lex Fridman (48:03.460)
And if you don't push quite that far, you can say,
Lex Fridman (48:07.560)
essentially, all of neurobiology, which is relevant,
John Hopfield (48:10.740)
can be captured by classical equations of motion.
Lex Fridman (48:13.740)
Right, because in my view of the mysteries of the brain
John Hopfield (48:18.960)
are not the mysteries of quantum mechanics,
Lex Fridman (48:22.160)
but the mysteries of what can happen
John Hopfield (48:24.840)
when you have a dynamical system, driven system,
Lex Fridman (48:28.840)
with 10 to the 14 parts.
John Hopfield (48:32.260)
That that complexity is something which is,
Lex Fridman (48:37.040)
that the physics of complex systems
John Hopfield (48:39.620)
is at least as badly understood
Lex Fridman (48:42.040)
as the physics of phase coherence in quantum mechanics.
Lex Fridman (48:46.520)
Can we go there for a second?
Lex Fridman (48:48.520)
You've talked about attractor networks,
Lex Fridman (48:51.720)
and just maybe you could say what are attractor networks,
Lex Fridman (48:54.800)
and more broadly, what are interesting network dynamics
Lex Fridman (48:58.600)
that emerge in these or other complex systems?
Lex Fridman (49:05.260)
You have to be willing to think
John Hopfield (49:06.320)
in a huge number of dimensions,
Lex Fridman (49:08.720)
because in a huge number of dimensions,
John Hopfield (49:11.000)
the behavior of a system can be thought
Lex Fridman (49:12.920)
as just the motion of a point over time
John Hopfield (49:15.920)
in this huge number of dimensions.
Lex Fridman (49:17.760)
All right.
Lex Fridman (49:19.340)
And an attractor network is simply a network
Lex Fridman (49:22.080)
where there is a line and other lines
John Hopfield (49:25.920)
converge on it in time.
Lex Fridman (49:28.320)
That's the essence of an attractor network.
John Hopfield (49:31.160)
That's how you.
Lex Fridman (49:32.000)
In a highly dimensional space.
Lex Fridman (49:34.760)
And the easiest way to get that
Lex Fridman (49:37.400)
is to do it in a highly dimensional space,
John Hopfield (49:40.760)
where some of the dimensions provide the dissipation,
Lex Fridman (49:44.960)
which, if I have a physical system,
John Hopfield (49:50.160)
trajectories can't contract everywhere.
Lex Fridman (49:53.680)
They have to contract in some places and expand in others.
John Hopfield (49:56.920)
There's a fundamental classical theorem
Lex Fridman (49:59.360)
of statistical mechanics,
John Hopfield (50:00.840)
which goes under the name of Liouville's theorem,
Lex Fridman (50:04.560)
which says you can't contract everywhere.
John Hopfield (50:08.600)
If you contract somewhere, you expand somewhere else.
Lex Fridman (50:12.400)
In interesting physical systems,
John Hopfield (50:15.240)
you've got driven systems
Lex Fridman (50:17.480)
where you have a small subsystem,
John Hopfield (50:19.240)
which is the interesting part.
Lex Fridman (50:21.720)
And the rest of the contraction and expansion,
John Hopfield (50:24.120)
the physicists would say it's entropy flow
Lex Fridman (50:26.000)
in this other part of the system.
Lex Fridman (50:30.880)
But basically, attractor networks are dynamics
Lex Fridman (50:35.520)
that are funneling down so that you can't be any,
Lex Fridman (50:40.360)
so that if you start somewhere in the dynamical system,
Lex Fridman (50:42.520)
you will soon find yourself
John Hopfield (50:44.120)
on a pretty well determined pathway, which goes somewhere.
Lex Fridman (50:47.120)
If you start somewhere else,
John Hopfield (50:48.120)
you'll wind up on a different pathway,
Lex Fridman (50:50.560)
but I don't have just all possible things.
John Hopfield (50:53.080)
You have some defined pathways which are allowed
Lex Fridman (50:56.640)
and onto which you will converge.
Lex Fridman (51:00.120)
And that's the way you make a stable computer,
Lex Fridman (51:01.920)
and that's the way you make a stable behavior.
Lex Fridman (51:06.280)
So in general, looking at the physics
Lex Fridman (51:08.760)
of the emergent stability in networks,
Lex Fridman (51:15.200)
what are some interesting characteristics that,
Lex Fridman (51:19.640)
what are some interesting insights
Lex Fridman (51:20.960)
from studying the dynamics of such high dimensional systems?
Lex Fridman (51:24.960)
Most dynamical systems, most driven dynamical systems,
John Hopfield (51:29.880)
are driven, they're coupled somehow to an energy source.
Lex Fridman (51:33.200)
And so their dynamics keeps going
John Hopfield (51:35.600)
because it's coupling to the energy source.
Lex Fridman (51:40.080)
Most of them, it's very difficult to understand at all
Lex Fridman (51:42.680)
what the dynamical behavior is going to be.
Lex Fridman (51:47.760)
You have to run it.
John Hopfield (51:49.240)
You have to run it.
Lex Fridman (51:50.600)
There's a subset of systems which has
Lex Fridman (51:54.080)
what is actually known to the mathematicians
Lex Fridman (51:57.280)
as a Lyapunov function, and those systems,
John Hopfield (52:02.000)
you can understand convergent dynamics
Lex Fridman (52:05.520)
by saying you're going downhill on something or other.
Lex Fridman (52:10.640)
And that's what I found with ever knowing
Lex Fridman (52:13.560)
what Lyapunov functions were in the simple model
John Hopfield (52:17.120)
I made in the early 80s, was an energy function
Lex Fridman (52:20.480)
so you could understand how you could get this channeling
John Hopfield (52:23.200)
on the pathways without having to follow the dynamics
Lex Fridman (52:28.080)
in infinite detail.
John Hopfield (52:31.880)
You started rolling a ball at the top of a mountain,
Lex Fridman (52:34.320)
it's gonna wind up at the bottom of a valley.
John Hopfield (52:36.480)
You know that's true without actually watching
Lex Fridman (52:40.440)
the ball roll down.
John Hopfield (52:43.120)
There's certain properties of the system
Lex Fridman (52:45.840)
that when you can know that.
John Hopfield (52:48.360)
That's right.
Lex Fridman (52:49.400)
And not all systems behave that way.
John Hopfield (52:53.640)
Most don't, probably.
Lex Fridman (52:55.240)
Most don't, but it provides you with a metaphor
John Hopfield (52:57.720)
for thinking about systems which are stable
Lex Fridman (53:00.720)
and who to have these attractors behave
John Hopfield (53:03.880)
even if you can't find a Lyapunov function behind them
Lex Fridman (53:07.920)
or an energy function behind them.
John Hopfield (53:09.880)
It gives you a metaphor for thought.
Lex Fridman (53:11.680)
Yeah, speaking of thought,
John Hopfield (53:17.200)
if I had a glint in my eye with excitement
Lex Fridman (53:21.000)
and said I'm really excited about this something
John Hopfield (53:25.600)
called deep learning and neural networks
Lex Fridman (53:28.440)
and I would like to create an intelligent system
Lex Fridman (53:32.440)
and came to you as an advisor, what would you recommend?
Lex Fridman (53:37.440)
Is it a hopeless pursuit to use neural networks
Lex Fridman (53:42.840)
to achieve thought?
Lex Fridman (53:44.920)
Is it, what kind of mechanisms should we explore?
Lex Fridman (53:48.760)
What kind of ideas should we explore?
Lex Fridman (53:52.040)
Well, you look at the simple networks,
John Hopfield (53:56.560)
the one past networks.
Lex Fridman (54:01.320)
They don't support multiple hypotheses very well.
John Hopfield (54:04.760)
Hmm.
Lex Fridman (54:06.960)
As I have tried to work with very simple systems
John Hopfield (54:09.960)
which do something which you might consider to be thinking,
Lex Fridman (54:12.960)
thought has to do with the ability to do mental exploration
John Hopfield (54:17.680)
before you take a physical action.
Lex Fridman (54:22.440)
Almost like we were mentioning, playing chess,
John Hopfield (54:25.480)
visualizing, simulating inside your head different outcomes.
Lex Fridman (54:30.440)
Yeah, yeah.
Lex Fridman (54:31.400)
And now you would do that in a feed forward network
Lex Fridman (54:37.400)
because you've pre calculated all kinds of things.
Lex Fridman (54:41.960)
But I think the way neurobiology does it
Lex Fridman (54:44.080)
hasn't pre calculated everything.
John Hopfield (54:49.360)
It actually has parts of a dynamical system
Lex Fridman (54:52.000)
in which you're doing exploration in a way which is.
John Hopfield (54:57.000)
There's a creative element.
Lex Fridman (55:01.760)
Like there's an.
John Hopfield (55:02.600)
There's a creative element.
Lex Fridman (55:04.680)
And in a simple minded neural net,
John Hopfield (55:13.000)
you have a constellation of instances
Lex Fridman (55:20.080)
of which you've learned.
Lex Fridman (55:23.040)
And if you are within that space,
Lex Fridman (55:25.760)
if a new question is a question within this space,
John Hopfield (55:32.800)
you can actually rely on that system pretty well
Lex Fridman (55:37.520)
to come up with a good suggestion for what to do.
John Hopfield (55:41.040)
If on the other hand,
Lex Fridman (55:42.000)
the query comes from outside the space,
John Hopfield (55:46.640)
you have no way of knowing how the system
Lex Fridman (55:48.440)
is gonna behave.
John Hopfield (55:49.280)
There are no limitations on what can happen.
Lex Fridman (55:51.440)
And so with the artificial neural net world
John Hopfield (55:55.300)
is always very much,
Lex Fridman (55:57.080)
I have a population of examples.
John Hopfield (56:01.020)
The test set must be drawn from the equivalent population.
Lex Fridman (56:04.740)
If the test set has examples,
John Hopfield (56:06.860)
which are from a population which is completely different,
Lex Fridman (56:11.100)
there's no way that you could expect
John Hopfield (56:14.420)
to get the answer right.
Lex Fridman (56:16.500)
Yeah, what they call outside the distribution.
John Hopfield (56:20.980)
That's right, that's right.
Lex Fridman (56:22.180)
And so if you see a ball rolling across the street at dusk,
John Hopfield (56:28.420)
if that wasn't in your training set,
Lex Fridman (56:33.300)
the idea that a child may be coming close behind that
John Hopfield (56:37.060)
is not going to occur to the neural net.
Lex Fridman (56:40.420)
And it is to our,
John Hopfield (56:42.500)
there's something in your biology that allows that.
Lex Fridman (56:45.580)
Yeah, there's something in the way
John Hopfield (56:47.620)
of what it means to be outside of the population
Lex Fridman (56:52.300)
of the training set.
John Hopfield (56:53.620)
The population of the training set
Lex Fridman (56:55.580)
isn't just sort of this set of examples.
John Hopfield (57:01.180)
There's more to it than that.
Lex Fridman (57:03.660)
And it gets back to my question of,
Lex Fridman (57:06.540)
what is it to understand something?
Lex Fridman (57:09.180)
Yeah.
John Hopfield (57:12.020)
You know, in a small tangent,
Lex Fridman (57:14.700)
you've talked about the value of thinking
John Hopfield (57:16.940)
of deductive reasoning in science
Lex Fridman (57:18.660)
versus large data collection.
Lex Fridman (57:21.820)
So sort of thinking about the problem.
Lex Fridman (57:25.300)
I suppose it's the physics side of you
John Hopfield (57:27.460)
of going back to first principles and thinking,
Lex Fridman (57:31.100)
but what do you think is the value of deductive reasoning
Lex Fridman (57:33.660)
in the scientific process?
Lex Fridman (57:37.740)
Well, there are obviously scientific questions
John Hopfield (57:39.820)
in which the route to the answer to it
Lex Fridman (57:42.980)
comes through the analysis of one hell of a lot of data.
John Hopfield (57:46.560)
Right.
Lex Fridman (57:49.180)
Cosmology, that kind of stuff.
Lex Fridman (57:50.500)
And that's never been the kind of problem
Lex Fridman (57:56.700)
in which I've had any particular insight.
John Hopfield (57:58.540)
Though I must say, if you look at,
Lex Fridman (58:01.660)
cosmology is one of those.
John Hopfield (58:04.180)
If you look at the actual things that Jim Peebles,
Lex Fridman (58:06.780)
one of this year's Nobel Prize in physics,
John Hopfield (58:10.140)
ones from the local physics department,
Lex Fridman (58:12.260)
the kinds of things he's done,
John Hopfield (58:13.760)
he's never crunched large data.
Lex Fridman (58:17.000)
Never, never, never.
John Hopfield (58:19.640)
He's used the encapsulation of the work of others
Lex Fridman (58:23.760)
in this regard.
John Hopfield (58:25.240)
Right.
Lex Fridman (58:27.820)
But it ultimately boiled down to thinking
John Hopfield (58:30.840)
through the problem.
Lex Fridman (58:31.700)
Like what are the principles under which
Lex Fridman (58:33.680)
a particular phenomenon operates?
Lex Fridman (58:35.840)
Yeah, yeah.
Lex Fridman (58:37.240)
And look, physics is always going to look
Lex Fridman (58:39.520)
for ways in which you can describe the system
John Hopfield (58:42.640)
in a way which rises above the details.
Lex Fridman (58:47.520)
And to the hard dyed, the wool biologist,
John Hopfield (58:53.840)
biology works because of the details.
Lex Fridman (58:56.760)
In physics, to the physicists,
John Hopfield (58:58.720)
we want an explanation which is right
Lex Fridman (59:01.160)
in spite of the details.
Lex Fridman (59:03.040)
And there will be questions which we cannot answer
Lex Fridman (59:05.560)
as physicists because the answer cannot be found that way.
John Hopfield (59:13.080)
There's, I'm not sure if you're familiar
Lex Fridman (59:15.240)
with the entire field of brain computer interfaces
John Hopfield (59:19.120)
that's become more and more intensely researched
Lex Fridman (59:24.040)
and developed recently, especially with companies
John Hopfield (59:25.920)
like Neuralink with Elon Musk.
Lex Fridman (59:29.080)
Yeah, I know there have always been the interests
John Hopfield (59:31.080)
both in things like getting the eyes
Lex Fridman (59:35.720)
to be able to control things
John Hopfield (59:38.320)
or getting the thought patterns
Lex Fridman (59:40.800)
to be able to move what had been a connected limb
John Hopfield (59:45.080)
which is now connected through a computer.
Lex Fridman (59:48.040)
That's right.
Lex Fridman (59:48.920)
So in the case of Neuralink,
Lex Fridman (59:51.320)
they're doing 1,000 plus connections
John Hopfield (59:54.600)
where they're able to do two way,
Lex Fridman (59:56.640)
activate and read spikes, neural spikes.