there is a certain perspective where you might be thinking,
what is the longest possible game that you could be playing?
A short game is for instance,
cancer is playing a shorter game than your organism.
Cancer is an organism playing a shorter game
than the regular organism.
And because the cancer cannot procreate beyond the organism,
except for some infectious cancers,
like the ones that eradicated the Tasmanian devils,
you typically end up with the situation
where the organism dies together with the cancer.
Because the cancer has destroyed the larger system
due to playing a shorter game.
And so ideally you want to, I think,
build agents that play the longest possible games.
And the longest possible games is to keep entropy at bay
as long as possible by doing interesting stuff.
The following is a conversation with Joscha Bach,
his third time on this podcast.
Joscha is one of the most brilliant
and fascinating minds in the world,
exploring the nature of intelligence,
consciousness, and computation.
And he's one of my favorite humans to talk to
about pretty much anything and everything.
This is the Lex Friedman Podcast.
To support it, please check out our sponsors
in the description.
And now, dear friends, here's Joscha Bach.
You wrote a post about levels of lucidity.
Quote, as we grow older, it becomes apparent
that our self-reflexive mind
is not just gradually accumulating ideas about itself,
but that it progresses in somewhat distinct stages.
So there's seven of the stages.
Stage one, reactive survival, infant.
Stage two, personal self, young child.
Stage three, social self, adolescence, domesticated adult.
Stage four is rational agency, self-direction.
Stage five is self-authoring, that's full adult.
You've achieved wisdom, but there's two more stages.
Stage six is enlightenment.
Stage seven is transcendence.
Can you explain each or the interesting parts
of each of these stages?
And what's your sense why there are stages of this,
of lucidity as we progress through life
in this too short life?
This model is derived from a concept
by the psychologist Robert Keegan.
And he talks about the development of the self
as a process that happens in principle
by some kind of reverse engineering of a mind
where you gradually become aware of yourself
and thereby build structure that allows you
to interact deeper with the world and yourself.
And I found myself using this model not so much
as a developmental model.
I'm not even sure if it's a very good developmental model
because I saw my children not progressing exactly like that.
And I also suspect that you don't go
through these stages necessarily in succession.
And it's not that you work through one stage
and then you get into the next one.
Sometimes you revisit them.
Sometimes stuff is happening in parallel,
but it's I think useful framework to look at what's present
in the structure of a person and how they interact
with the world and how they relate to themselves.
So it's more like a philosophical framework
that allows you to talk about how minds work.
And at first, when we are born,
we don't have a personal self yet, I think.
Instead we have an attentional self.
And this attentional self is initially in the infant task,
is building a world model
and also an initial model of the self.
But mostly it's building a game engine in the brain
that is tracking sensory data and uses it to explain it.
And in some sense, you could compare it to game engine
like Minecraft or so-so, colors and sounds.
People are all not physical objects.
They are creation of our mind at a certain level
of course, training models that are mathematical
that use geometry and that use manipulation of objects
and so on to create scenes in which we can find ourselves
and interact with them.
So Minecraft.
Yeah, and this personal self is something
that is more or less created after the world is finished,
after it's trained into the system,
after it has been constructed.
And this personal self is an agent
that interacts with the outside world.
And the outside world is not the world of quantum mechanics
not the physical universe, but it's the model
that has been generated in our own mind, right?
And this is us.
And we experience ourself interacting
this dead outside world that is created
inside of our own mind.
And outside of our self, there's feelings
and they presented our interface to this outside world.
They pose problems to us.
These feelings are basically attitudes
that our mind is computing
that tell us what's needed in the world,
the things that we are drawn to,
the things that we are afraid of.
And we are tasked with solving this problem
of satisfying the needs, avoiding the aversions,
following on our inner commitments and so on,
and also modeling ourselves and building the next stage.
So after we have this personal self in stage two online,
many people form a social self.
And this social self allows the individual
to experience themselves as part of a group.
It's basically this thing that when you are playing
in a team, for instance, you don't notice yourself
just as a single note that is reaching out into the world,
but you're also looking down.
You're looking down from this entire group
and you see how this group is looking at this individual
and everybody in the group is, in some sense,
emulating this group spirit to some degree.
And in this state, people are forming their opinions
by assimilating them from this group mind,
where they gain the ability
to act a little bit like a hive mind.
But are you also modeling the interaction
of how opinion shapes and forms
through the interaction of the individual nodes
within the group?
Yeah, it's basically the way in which people do it
in this stage is that they experience
what are the opinions of my environment.
They experience the relationship
that I have to their environment.
And they resonate with people around them
and get more opinions through this interaction
to the way in which they relate to others.
And at stage four, you basically understand
that stuff is true and false independently
what other people believe.
And you have agency over your own beliefs in that stage.
You basically discover epistemology,
the rules about determining what's true and false.
So you start to learn how to think.
Yes.
I mean, at some level, you're always thinking.
You are constructing things.
And I believe that this ability to reason
about your mental representation
is what we mean by thinking.
It's an intrinsically reflexive process
that requires consciousness.
Without consciousness, you cannot think.
You can generate the content of feelings and so on
outside of consciousness.
It's very hard to be conscious of how your feelings emerge,
at least in the early stages of development.
But thoughts is something that you always control.
And if you are a nerd like me,
you often have to skip stage three
because you lack the intuitive empathy with others.
Because in order to resonate with a group,
you need to have a quite similar architecture.
And if people are wired differently,
then it's hard for them to resonate with other people
and basically have empathy,
which is not the same as compassion,
but it is a shared perceptual mental state.
Empathy happens not just via inference
about the mental states of others,
but it's a perception of what other people feel
and where they're at.
Can she not have empathy
while also not having a similar architecture,
cognitive architecture, as the others in the group?
I think, yes, but I experienced that too.
But you need to build something
that is like a meta-architecture.
You need to be able to embrace the architecture
of the other to some degree,
or find some shared common ground.
And it's also this issue that if you are a nerd,
normally often people, as in your typical people,
have difficulty to resonate with you.
And as a result, they have difficulty understanding you
unless they have enough wisdom
to feel what's going on there.
Well, isn't the whole process of the stage three
is to figure out the API to the other humans
that have different architecture
and you yourself publish public documentation for the API
that people can interact with for you?
Isn't this the whole process of socializing?
My experience as a child growing up
was that I did not find any way
to interface with the stage three people,
and they didn't do that with me.
So-
Did you try?
Yeah, of course, I tried very hard.
But it was only when I entered a mathematics school
at ninth grade, lots of other nerds were present,
that I found people that I could deeply resonate with
and had the impression that, yes, I have friends now.
I found my own people.
And before that, I felt extremely lonely in the world.
There was basically nobody I could connect to.
And I remember there was one moment in all these years
where there was a school exchange
and it was the Russian boy,
kid from the Russian carnison station in Eastern Germany.
You visit our school
and we played a game of chess against each other.
And we looked into each other's eyes
and we sat there for two hours playing this game of chess.
And I had the impression, this is a human being.
He understands, but I understand.
We didn't even speak the same language.
I wonder if your life could have been different
if you knew that it's okay to be different,
to have a different architecture.
Whether accepting that the interface is hard to figure out,
takes a long time to figure out,
and it's okay to be different.
In fact, it's beautiful to be different.
It was not my main concern.
My main concern was mostly that I was alone.
It was not so much the question, is it okay to be the way I am?
I couldn't do much about it, so I had to deal with it.
But my main issue was that I was not sure
if I would ever meet anybody growing up
that I would connect to at such a deep level
that I would feel that I could belong.
So there's a visceral, undeniable feeling of being alone.
Yes.
And I noticed the same thing
when I came into the math school,
that I think at least half, probably two thirds
of these kids were severely traumatized
as children growing up.
And in large part due to being alone,
because they couldn't find anybody to relate to.
Don't you think everybody's alone, deep down?
No.
All right, I'm not alone anymore.
It took me some time to update
and to get over the trauma and so on.
But I felt that in my 20s, I had lots of friends
and I had my place in the world.
And I had no longer doubts
that I would never be alone again.
Is there some aspect to which we're alone together?
You don't see a deep loneliness inside yourself still?
No.
Sorry.
Okay, so that's the nonlinear progression
through the stages, I suppose.
You caught up on stage three at some point?
We're at stage four.
And so basically I find that many nerds
jump straight into stage four, bypassing stage three.
Do they return to it then, later?
Yeah, of course.
Sometimes they do, not always.
The question is basically,
do you stay a little bit autistic or do you catch up?
And I believe you can catch up.
You can build this missing structure.
And basically experience yourself as part of a group,
learn intuitive empathy and develop the sense,
this perceptual sense of feeling what other people feel.
And before that, I could only basically feel this
when I was deeply in love with somebody and we synced.
So there's a lot of friction to feeling that way?
Only with certain people as opposed to it comes naturally.
Yeah. It's frictionless.
But this is something that basically later I felt
started to resolve itself for me to a large degree.
What was the trick?
In many ways growing up and paying attention.
Meditation did help.
I had some very crucial experiences
in getting close to people, building connections,
cuddling a lot in my student years.
So really paying attention to the, what is it?
To the feeling another human being fully.
Loving other people and being loved by other people
and building a space in which you can be safe
and can experiment and touch a lot
and be close to somebody a lot.
And over time, basically at some point you realize,
oh, it's no longer that I feel locked out,
but I feel connected
and I experience where somebody else is at.
And normally my mind is racing very fast
at a high frequency.
So it's not always working like this.
Sometimes it works better, sometimes it works less,
but I also don't see this as a pressure.
It's more, it's interesting to observe myself
at which frequency I'm at
and at which mode somebody else is at.
Yeah, man, the mind is so beautiful in that way.
Sometimes it comes so natural to me,
so easy to pay attention, pay attention to the world fully,
to other people fully.
And sometimes the stress over silly things is overwhelming.
It's so interesting
that the mind is that roller coaster in that way.
At stage five, you discover how identity is constructed.
Self offering.
You realize that your values are not terminal,
but they are instrumental to achieving a world
that you like and aesthetics that you prefer.
And the more you understand this,
the more you get agency over how your identity
is constructed.
And you realize that identity
and interpersonal interaction is a costume
and you should be able to have agency over that costume.
It's useful to be a costume.
It tells something to others
and it allows to interface in roles,
but being locked into this is a big limitation.
The word costume kind of implies
that it's fraudulent in some way.
Is costume a good word for you?
Like we present ourselves to the world.
In some sense, I learned a lot about costumes
at Burning Man.
Before that, I did not really appreciate costumes
and saw them more as uniforms,
like wearing a suit if you are working in a bank
or if you are trying to get startup funding
from a VC in Switzerland, right?
Then you dress up in a particular way.
And this is mostly to show the other side
that you are willing to play by the rules
and you understand what the rules are.
But there is something deeper.
When you are at Burning Man,
your costume becomes self-expression
and there is no boundary to the self-expression.
You're basically free to wear what you want
to express other people what you feel like this day
and what kind of interactions you want to have.
Is the costume a kind of projection of who you are?
That's very hard to say because the costume
also depends on what other people see in the costume.
And this depends on the context
that the other people understand.
And you have to create something if you want to
that is legible to the other side
and that means something to yourself.
Do we become prisoners of the costume?
Because everybody expects us to.
Some people do, but I think that once you realize
that you wear a costume at Burning Man,
a variety of costumes, realize
that you cannot not wear a costume.
Basically everything that you wear and present to others
is something that is to some degree
an addition to what you are deep inside.
So this stage in parentheses,
you put full adult comma wisdom.
Why is this full adult?
Why would you say this is full?
And why is it wisdom?
It does allow you to understand
why other people have different identities from yours.
And it allows you to understand that the difference
between people who vote for different parties
and might have very different opinions
and different value systems
is often the accident of where they are born
and what happened after that to them
and what traits they got before they were born.
And at some point you realize the perspective
that everybody could be you in a different timeline
if you just flip those bits.
How many costumes do you have?
I don't count, but in-
More than one?
Yeah, of course.
How easy is this to do costume changes throughout the day?
It's just a matter of energy and interest.
When you are wearing your pajamas
and you switch out of your pajamas
into say a work short and pants,
you're making a costume change, right?
And if you are putting on a gown,
you're making a costume change.
And you could do the same with personality?
You could, if that's what you're into.
There are people which have multiple personalities
for interaction in multiple worlds, right?
So if somebody works in a store
and you put up a storekeeper personality
when you're presenting yourself at work,
you develop a sub-personality for this.
And the social persona for many people
is in some sense a puppet
that they're playing like a marionette.
And if they play this all the time,
they might forget that there is something behind this,
just something what it feels like to be in your skin.
And I guess it's very helpful
if you're able to get back into this.
And for me, the other way around is relatively hard.
For me, it's pretty hard to learn
how to play consistent social roles.
For me, it's much easier just to be real.
Or not real, but to have one costume.
No, it's not quite the same.
So basically when you are wearing a costume at Burning Man
and say you are an extraterrestrial prince,
there's something where you are expressing in some sense
something that's closer to yourself
than the way in which you hide yourself
behind standard clothing when you go out in the city
in the default world.
And so this costume that you're wearing at Burning Man
allows you to express more of yourself.
And you have a shorter distance of advertising to people
what kind of person you are,
what kind of interaction you would want to have with them.
And so you get much earlier into media stress.
And I believe it's regrettable
that we do not use the opportunities
that we have with custom-made clothing now
to wear costumes that are much more stylish,
that are much more custom-made,
that are not necessarily part of a fashion
in which you express which milieu you're part of
and how up-to-date you are,
but you also express how you are as an individual
and what you want to do today and how you feel today
and what you intend to do about it.
Well, isn't it easier now in the digital world
to explore different costumes?
I mean, that's the kind of idea with virtual reality.
That's the idea.
Even with Twitter in two-dimensional screens,
you can swap out costumes.
You can be as weird as you want.
It's easier.
For Burning Man, you have to order things.
You have to make things.
It's more effort to put on-
It's even better if you make them yourselves.
Sure, but it's just easier to do digitally, right?
It's not about easy.
It's about how to get it right.
And for me, the first Burning Man experience,
I got adopted by a bunch of people in Boston
who tracked me to Burning Man.
And we spent a few weekends doing costumes together.
And that was an important part of the experience
where the camp bonded,
that people got to know each other.
And we basically grew into the experience
that we would have later.
So the extraterrestrial prince is based on a true story?
Yeah.
I can only imagine what that looks like, Yosha.
Okay, so- Stage six.
Stage six, at some point you can collapse the division
between a personal self and world generator again.
And a lot of people get there via meditation
or some of them get there via psychedelics,
some of them by accident.
And you suddenly notice that you are not actually a person,
but you are a vessel that can create a person.
And the person is still there.
You observe that personal self,
but you observe the personal self from the outside.
And you notice it's a representation.
And you might also notice that the world
that is being created is a representation.
If not, then you might experience that I am the universe.
I am the thing that is creating everything.
And of course, what you're creating
is not quantum mechanics and the physical universe.
What you're creating is this game engine
that is updating the world
and you're creating your valence, your feelings,
and all the people inside of that world,
including the person that you identify with yourself
in this world.
Are you creating the game engine
or are you noticing the game engine?
You noticed how you're generating the game engine.
And I mean, when you are dreaming at night,
you can, if you have a lucid dream,
you can learn how to do this deliberately.
And in principle, you can also do it during the day.
And the reason why we don't get to do this
from the beginning and why we don't have agency
of our feelings right away is because we would game it
before they have the necessary amount of wisdom
to deal with creating this dream that we are in.
You don't want to get access to cheat codes too quickly.
Otherwise you won't enjoy the game.
So stage five is already pretty rare
and stage six is even more rare.
You basically find this mostly with advanced
Buddhist meditators and so on that are dropping
into the stage and can induce it at will
and spend time in it.
So stage five requires a good therapist.
Stage six requires a good Buddhist spiritual leader.
For instance, could be that is the right thing to do,
but it's not that these stages give you scores
or levels that you need to advance to.
It's not that the next stage is better.
You live your life in the mode that works best
at any given moment.
And when your mind decides that you should have
a different configuration,
then it's building that configuration.
And for many people, they stay happily at stage three
and experiences themselves as part of groups.
And there's nothing wrong with this.
And for some people, this doesn't work
and they're forced to build more agency
over their rational beliefs than this
and construct their norms rationally.
And so they go to this level.
And stage seven is something that is more or less
hypothetical.
That would be the stage in which it's basically
a transhumanist stage in which you understand
how you work, in which the mind fully realizes
how it's implemented and can also in principle
enter different modes in which it could be implemented.
And that's the stage that as far as I understand
is not open to people yet.
Oh, but it is possible through the process of technology.
Yes, and who knows if there are biological agents
that are working at different timescales than us
that basically become aware of the way
in which they're implemented on ecosystems
and can change that implementation
and have agency over how they're implemented in the world.
And what I find interesting about the discussion
about AI alignment that it seems to be following
the status very much.
Most people seem to be in stage three
also according to Robert Keegan.
I think he says that about 85% of people
are in stage three and stay there.
And if you're in stage three and your opinions
are the result of social assimilation
then what you're mostly worried about in the AI
is that the AI might have the wrong opinions.
So if the AI says something racist or sexist
they're all lost because we will assimilate
the wrong opinions from the AI.
And so we need to make sure
that the AI has the right opinions
and the right values and the right structure.
And if you're at stage four, that's not your main concern.
And so most nerds don't really worry
about the algorithmic bias and the model that it picks up
because if there's something wrong with this bias
the AI ultimately will prove it.
At some point we'll gather there
that it makes mathematical proofs about reality
and then it will figure out what's true and what's false.
But you're still worried that AI
might turn you into paperclips
because it might have the wrong values, right?
So if it's set up as the wrong function
that controls its direction in the world
then it might do something that is completely horrible
and there's no easy way to fix it.
So that's more like a stage four rationalist kind of worry.
And if you are at stage five, you're mostly worried
that AI is not going to be enlightened fast enough
because you realize that the game is not so much
about intelligence, but about agency,
about the ability to control the future.
And the identity is instrumental to this.
And if you are a human being, I think at some level
you ought to choose your own identity.
You should not have somebody else pick the costume for you
and then veer it, but instead you should be mindful
about what you want to be in this world.
And I think if you are an agent that is fully malleable
that can rewrite its own source code
like an AI might do at some point,
then the identity that you will have is whatever you can be.
And in this way, the AI will maybe become everything
like a planetary control system.
And if it does that, then if we want to coexist with it
it means that it will have to share purposes with us.
So it cannot be a transactional relationship.
We will not be able to use reinforcement learning
as human feedback to hardwire its values into it.
But this has to happen.
It's probably that it's conscious.
So it can relate to our own mode of existence
where an observer is observing itself in real time
and visit certain temporal frames.
And the other thing is that it probably needs to have
some kind of transcendental orientation,
building shared agency.
And in the same way as we do when we are able to enter
up with each other into non-translational relationships.
And I find that something that,
because the stage five is so rare,
is missing in much of the discourse.
And I think that we need in some sense,
focus on how to formalize love, how to understand love
and how to build it into the machines
that we are currently building
and that are about to become smarter than us.
Well, I think this is a good opportunity
to try to sneak up to the idea of enlightenment.
So you wrote a series of good tweets
about consciousness and panpsychism.
So let's break it down.
First you say, I suspect the experience that leads
to the panpsychism syndrome of some philosophers
and other consciousness enthusiasts
represents the realization that we don't end at the self,
but share a resonant universe representation
with every other observer coupled to the same universe.
This actually eventually leads us
to a lot of interesting questions about AI and AGI,
but let's start with this representation.
What is this resonant universe representation?
And what do you think?
Do we share such a representation?
The neuroscientist Grossberg has come up
with the cognitive architecture
that he calls the adaptive resonance theory.
And his perspective is that our neurons can be understood
as oscillators that are resonating with each other
and with outside phenomena.
So the coarse-grained model of the universe
that we are building in some sense is a resonance
with objects and outside of us in the world.
So basically we take up patterns of the universe
that we are coupled with,
and our brain is not so much understood as circuitry,
even though this perspective is valid,
but it's almost an ether in which the individual neurons
are passing on chemoelectrical signals
or arbitrary signals across all modalities
that can be transmitted between cells,
stimulate each other in this way,
and produce patterns that they modulate
while passing them on.
And this speed of signal progression in the brain
is roughly at the speed of sound, incidentally,
because the time that it takes
for the signals to hop from cell to cell,
it means it's relatively slow with respect to the world.
It takes an appreciable fraction of a second
for a signal to go through the entire neocortex,
something like a few hundred milliseconds.
And so there's a lot of stuff happening in that time
where the signal is passing through your brain,
including in the brain itself.
So nothing in the brain is assuming
that stuff happens simultaneously.
Everything in the brain is working in a paradigm
where the world has already moved on
when you are ready to do the next thing to your signal,
including the signal processing system itself.
It's quite a different paradigm
than the one in our digital computers
where we currently assume that your GPU or CPU
is pretty much globally in the same state.
So you mentioned there the non-dual state,
and say that some people confuse it for enlightenment.
What's the non-dual state?
There is a state in which you notice
that you are no longer a person,
and instead you are one with the universe.
So that speaks to the resonance.
Yes, but this one with the universe
is, of course, not accurately modeling
that you are indeed some god entity
or indeed the universe becoming aware of itself,
even though you get this experience.
I believe that you get this experience
because your mind is modeling the fact
that you are no longer identified
with the personal self in that state,
but you have transcended this division
between the self model and the world model,
and you are experiencing yourself as your mind,
as something that is representing a universe.
But that's still part of the model.
Yes, so it's inside of the model.
You are still inside of patterns
that are generated in your brain and in the organism.
And what you are now experiencing
is that you're no longer this personal self in there,
but you are the entirety of the mind and its contents.
Why is it so hard to get there?
A lot of people who get into the state
think this associated with enlightenment,
I suspect it's a favorite training goal
for a number of meditators.
But I think that enlightenment
is in some sense more mundane,
and it's a step further or sideways.
It's the state where you realize
that everything is a representation.
Yeah, you say enlightenment is a realization
of how experience is implemented.
Yes, so basically you notice at some point
that your qualia can be deconstructed.
Reverse engineered, almost like a schematic of it, what?
You can start with looking at a face.
I mean, look at your own face in the mirror.
Look at your face for a few hours in the mirror
or for a few minutes.
At some point it will look very weird
because you notice that there's actually no face.
You basically start unseeing the face.
What you see is the geometry,
and then you can disassemble the geometry
and realize how that geometry
is being constructed in your mind.
And you can learn to modify this.
So basically you can change these generators in your mind
to shift the face around
or to change the construction of the face,
to change the way in which the features are being assembled.
Why don't we do that more often?
Why don't we start really messing with reality
without the use of drugs or anything else?
Why don't we get good at this kind of thing?
Intentionally.
Why should we?
Because you can morph reality
into something more pleasant for yourself.
Just have fun with it.
Yeah, that is probably what you shouldn't be doing, right?
Because outside of your personal self,
this outer mind is probably a relatively smart agent.
And what you often notice is that you have thoughts
about how you should live,
but you observe yourself doing different things
and having different feelings.
And that's because your outer mind doesn't believe you
and doesn't believe your rational thoughts.
Well, can't you just silence the outer mind?
The thing is that the outer mind
is usually smarter than you are.
Rational thinking is very brittle.
It's very hard to use logic and symbolic thinking
to have an accurate model of the world.
So there is often an underlying system
that is looking at your rational thoughts
and then tells you, no, you're still missing something.
Your gut feeling is still facing something else.
And this can be, for instance,
you find a partner that looks perfect,
or you find a deal and just build a company
or whatever that looks perfect to you.
And yet at some level you feel something is off
and you cannot put your finger on it.
And the more you reason about it, the better it looks to you.
But the system that is outside still tells you,
no, no, you're missing something.
And that system is powerful.
People call this intuition, right?
Intuition is this unreflected part
of your attitude, composition and computation
where you produce a model of how you relate to the world
and what you need to do it in it
and what you can do in it and what's going to happen.
That is usually deeper
and often more accurate than your reason.
So if we look at this as you write in the tweet,
if we look at this more rigorously,
is to sort of take the panpsychist idea more seriously,
almost as a scientific discipline.
You write that quote,
fascinatingly, the panpsychist interpretation
seems to lead to observations of practical results
to a degree that physics fundamentalists
might call superstitious.
Reports of long distance telepathy
and remote causation are ubiquitous
in the general population.
I am not convinced, says Yoshabak,
that establishing the empirical reality of telepathy
would force an update of any part
of serious academic physics,
but it could trigger an important revolution
in both neuroscience and AI
from a circuit perspective
to a coupled complex resonator paradigm.
Are you suggesting that there could be
some rigorous mathematical wisdom
to panpsychist perspective on the world?
So first of all, panpsychism is the perspective
that consciousness is inseparable
from matter in the universe.
And I find panpsychism quite unsatisfying
because it does not explain consciousness, right?
It does not explain how this aspect of matter produces.
It's also when I try to formalize panpsychism
and write down what it actually means
with a more formal mathematical language,
it's very difficult to distinguish it
from saying that there is a software side to the world
in the same way as there is a software side
to what the transistors are doing in your computer.
So basically there is a pattern
at a certain core screening of the universe
that in some reasons of the universe
leads to observers that are observing themselves, right?
So panpsychism maybe is not even when I write it down
a position that is distinct from functionalism.
But intuitively, a lot of people feel
that the activity of matter itself,
of mechanisms in the world is insufficient to explain it.
So it's something that needs
to be intrinsic to matter itself.
And you can, apart from this abstract idea,
have an experience in which you experience yourself
as being the universe,
which I suspect is basically happening
because you managed to dissolve the division
between personal self and mind
that you establish as an infant
when you construct the personal self
and transcend it again and understand how it works.
But there is something deeper that is that you feel
that you're also sharing a state with other people,
that you have an experience in which you notice
that your personal self is moving into everything else,
that you basically look out of the eyes of another person,
that every agent in the world that is an observer
is, in some sense, you.
And we forget that we are the same agent.
So is it that we feel that,
or do we actually accomplish it?
So is telepathy possible?
Is it real?
For me, that's a question
that I don't really know the answer to.
In Turing's famous 1950 paper
in which he describes the Turing test,
he does speculate about telepathy, interestingly,
and asks himself if telepathy is real,
and he thinks that it very well might be,
what would be the implication for AI systems
that try to be intelligent,
because he didn't see a mechanism
by which a computer program would become telepathic.
And I suspect if telepathy would exist,
or if all the reports that you get from people
when you ask the normal person on the street,
I find that very often they say,
I have experiences with telepathy.
The scientists might not be interested in this
and might not have a theory about this,
but I have difficulty explaining it away.
And so you could say, maybe this is a superstition,
maybe it's a false memory,
or maybe it's a little bit of psychosis, who knows?
Maybe somebody wants to make their own life more interesting
or misremember something.
But a lot of people report,
I noticed something terrible happened to my partner,
and I know this is exactly the moment it happened
where my child had an accident
and I knew that was happening
and the child was in a different town.
So maybe it's a false memory
where this is later on mistakenly attributed,
but a lot of people think
this is not the correct explanation.
So if something like this was real, what would it mean?
It probably would mean that either your body is an antenna
that is sending information over all sorts of channels,
like maybe just electromagnetic radio signals
that you're sending over long distances
and you get attuned to another person
that you spend enough time with
to get a few bits out of the ether
to figure out what this person is doing.
Or maybe it's also when you are very close to somebody
and you become empathetic with them,
what happens is that you go into a resonance state with them,
similar to when people go into a seance
and they go into a trance state
and they start shifting a video board around on the table.
I think what happens is that their minds go
by their nervous systems into a resonant state
in which they basically create something
like a shared dream between them.
Physical closeness or closeness broadly defined?
With physical closeness,
it's much easier to experience empathy with someone, right?
I suspect it would be difficult for me
to have empathy for you if you were in a different town.
Also, how would that work?
But if you are very close to someone,
you pick up all sorts of signals from their body,
not just via your eyes, but with your entire body.
And if the nervous system sits on the other side
and the intercellular communication sits on the other side
and is integrating over all these signals,
you can make inferences about the state of the other.
And it's not just the personal self
that does this via reasoning, but your perceptual system.
And what basically happens is that
your representations are directly interacting.
It's the physical resonant models of the universe
that exist in your nervous system and in your body
might go into resonance with others
and start sharing some of their states.
So you basically buy next to a pig next to somebody,
you pick up some of their vibes
and feel without looking at them
what they're feeling in this moment.
And it's difficult for you if you're very empathetic
to detach yourself from it and have an emotional state
that is completely independent from your environment.
People who are highly empathetic are describing this.
And now imagine that a lot of organisms
on this planet have representations of the environment
and operate like this,
and they are adjacent to each other and overlapping.
So there's going to be some degree
in which there is basically some change interaction.
And we are forming some slightly shared representation
and relatively few neuroscientists
who consider this possibility.
I think a big rarity in this regard is Michael Levin
who is considering these things in earnest.
And I stumbled on this train of thought mostly
by noticing that the tasks of a neuron
can be fulfilled by other cells as well.
They can send different typed chemical messages
and physical messages to the adjacent cells
and learn when to do this and when not,
make this conditional
and become universal function approximators.
The only thing that they cannot do
is telegraph information over axons very quickly
over long distances, right?
So neurons in this perspective
are specially adapted kind of telegraph cell
that has evolved so we can move our muscles very fast,
but our body is in principle able
to also make models of the world just much, much slower.
It's interesting though,
that at this time, at least in human history,
there seems to be a gap between the tools of science
and the subjective experience that people report.
Like you're talking about with telepathy
and it seems like we're not quite there.
No, I think that there is no gap
between the tools of science and telepathy.
Either it's there or it's not,
and it's an empirical question.
And if it's there,
we should be able to detect it in a lab.
So why is there not a lot of Michael Levins walking around?
I don't think that Michael Levin
is specifically focused on telepathy very much.
He is focused on self-organization
in living organisms and in brains,
both as a paradigm for development
and as a paradigm for information processing.
And when you think about how organization processing
works in organisms,
there is first of all, radical locality,
which means everything is decided locally
from the perspective of an individual cell,
the individual cells, the agent.
And the other one is coherence.
Basically there needs to be some criterion
that determines how these cells are interacting
in such a way that order emerges
on the next level of structure.
And this principle of coherence of imposing constraints
that are not validated by the individual parts
and lead to coherence structure
to basically transcendental agency,
where you form an agent on the next level of organization
is crucial in this perspective.
It's so cool that radical locality
leads to the emergence of complexity at the higher layers.
And I think what Michael Levin is looking at
is nothing that is outside of the realm of science
in any way.
It's just that he is a paradigmatic thinker
who develops his own paradigm.
And most of the neuroscientists
are using a different paradigm at this point.
And this often happens in science
that a field has a few paradigms
in which people try to understand reality
and build concepts and make experiments.
You're kind of one of those type of paradigmatic thinkers.
Actually, if we can take a tangent on that,
once again returning to the biblical verses of your tweets.
You're right, my public explorations
are not driven by audience service,
but by my lack of ability for discovering,
understanding, or following the relevant authorities.
So I have to develop my own thoughts.
Since I think autonomously,
these thoughts cannot always be very good.
That's you apologizing for the chaos of your thoughts.
Or perhaps not apologizing, just identify it.
But let me ask the question.
Since we talked about Michael Levin and yourself,
who I think are very kind of radical,
big, independent thinkers,
can we reverse engineer
your process of thinking autonomously?
How do you do it?
How can humans do it?
How can you avoid being influenced by,
what is it, stage three?
Well, why would you want to do that?
It's, you see what is working for you.
And if it's not working for you,
you build another structure that works better for you, right?
And so I found myself when I was thrown into this world
in a state where my intuitions were not working for me.
I was not able to understand
how I would be able to survive in this world
and build the things that I was interested in,
build the kinds of relationship I needed
to build work on the topics
that I wanted to make progress on.
And so I had to learn.
And for me, Twitter is not some tool of publication.
It's not something where I put stuff
that I entirely believe to be true and provable.
It's an interactive notebook
in which I explore possibilities.
And I found that when I tried to understand
how the mind and how consciousness works,
I was quite optimistic.
I thought I need to be a big body of knowledge
that I can just study and that works.
And so I entered studies in philosophy and computer science
and later psychology and a bit of neuroscience and so on.
And I was disappointed by what I found
because I found that the questions of how consciousness
and so on works, how emotion works,
how it's possible that the system can experience anything,
how motivation emerges in the mind,
are not being answered by the authorities that I met
and the schools that were around.
And instead, I found that it was individual thinkers
that had useful ideas that sometimes were good,
sometimes were not so good,
sometimes were adopted by a large group of people,
sometimes were rejected by large groups of people.
But for me, it was much more interesting
to see these minds as individuals.
And in my perspective, thinking is still something
that is done not in groups,
that has to be done by individuals.
So that motivated you to become
an individual thinker yourself?
I didn't have a choice.
I didn't find a group that thought in a way where I felt,
okay, I can just adopt everything that everybody thinks here
and now I understand how consciousness works
or how the mind works or how thinking works
or what thinking even is, what feelings are
and how they're implemented and so on.
To figure all this out,
I had to take a lot of ideas from individuals
and then try to put them together
in something that works for myself.
And on one hand, I think it helps if you try to go down
and find first principles on which you can recreate
how thinking works, how languages work,
what representation is,
whether representation is necessary,
how the relationship between a representing agent
and the world works in general.
But how do you escape the influence?
Once again, the pressure of the crowd,
whether it's you in responding to the pressure
or you being swept up by the pressure.
If you even just look at Twitter, the opinions of the crowd.
I don't feel pressure from the crowd.
I'm completely immune to that.
In the same sense, I don't have respect for authority.
I have respect for what an individual is accomplishing
or have respect for mental firepower
but it's not that I meet somebody
and gets like drawed and unable to speak
or when a large group of people has a certain idea
that is different from mine,
I don't necessarily feel intimidated,
which has often been a problem for me in my life
because I like instincts that other people develop
at a very young age and that help with their self-preservation
in a social environment.
So I had to learn a lot of things the hard way.
Yeah.
So is there a practical advice you can give
on how to think paradigmatically,
how to think independently?
Or because you've kind of said, I had no choice.
But I think to a degree you have a choice
because you said you want to be productive
and I think thinking independently is productive
if what you're curious about is understanding the world,
especially when the problems are very kind of new
and open.
And so it seems like this is a active process.
Like we can choose to do that, we can practice it.
Well, it's a very basic question.
When you read a theory that you find convincing
or interesting, how do you know?
It's very interesting to figure out
what are the sources of that other person,
not which authority can they refer to
that is then taking off the burden of being truthful
but how did this authority in turn know?
What is the epistemic chain to observables?
What are the first principles
from which the whole thing is derived?
And when I was young, I was not blessed
with a lot of people around myself
who knew how to make proofs on first principles.
And I think mathematicians do this quite naturally
but most of the great mathematicians
do not become mathematicians in school
but they tend to be self-taught
because school teachers tend not to be mathematicians, right?
They tend not to be people who derive things
from first principles.
So when you ask your school teacher,
why does two plus two equal four?
Does your school teacher give you the right answer?
It's a simple game and many simple games that you could play
and most of those games
that you could just take different worlds
would not lead to an interesting arithmetic.
And so it's just an exploration
but you can try what happens if you take different axioms
and here is how you build axioms
and derive addition from them
and build addition as some basically syntactic sugar in it.
And so I wish that somebody would have opened me this Vista
and explained to me how I can build a language
in my own mind and from which I can derive what I'm seeing
and how I can, which I can make geometry and counting
and all the number games that we are playing in our life.
And on the other hand,
I felt that I learned a lot of this
while I was programming as a child.
When you start out with a computer like a Commodore 64
which doesn't have a lot of functionality,
it's relatively easy to see
how a bunch of relatively simple circuits
just basically performing hashes between bit patterns
and how you can build the entirety of mathematics
and computation on top of this
and all the representational languages that you need.
Commodore 64 could be one of the sexiest machines
I ever built if I say so myself.
If we can return to this really interesting idea
that we started to talk about with panpsychism.
Sure.
And the complex resonator paradigm
and the verses of your tweets.
You write, instead of treating eyes, ears and skin
as separate sensory systems
with fundamentally different modalities
who might understand them as overlapping aspects
of the same universe,
coupled at the same temporal resolution
and almost inseparable from a single share resonant model.
Instead of treating mental representations
as fully isolated between minds,
the representations of physically adjacent observers
might directly interact and produce causal effects
through the coordination of the perception
and behavior of world modeling observers.
So the modalities, the distinction between modalities,
let's throw that away.
The distinction between the individuals,
let's throw that away.
So what does this interaction representations look like?
And you think about how you represent
the interaction of us in this room.
At some level, the modalities are quite distinct.
They're not completely distinct,
but you can see this as vision.
You can close your eyes
and then you don't see a lot anymore,
but you still imagine how my mouth is moving
when you hear something and you know that it's very close
to the sound that you can just open your eyes
and you get back into this shared merged space.
And we also have these experiments where we noticed
that the way in which my lips are moving
are affecting how you hear the sound.
And also vice versa, the sounds that you're hearing
have an influence on how you interpret
some of the visual features.
So these modalities are not separate in your mind.
They are merged at some fundamental level
where you are interpreting the entire scene that you are in.
And your own interactions in the scene
are also not completely separate from the interactions
of the other individual in the scene.
But there is some resonance that is going on
where we also have a degree
of shared mental representations and shared empathy
due to being in the same space
and having vibes between each other.
Vibes.
So the question though, is how deeply intertwined
is this multimodality, multi-agent system?
Like how, I mean, this is going to the telepathy question
without the woo woo meaning of the word telepathy.
It's like how, like what's going on here
in this room right now?
So if telepathy would work, how could it work?
Yeah.
So imagine that all the cells in your body
are sending signals in a similar way as neurons are doing,
just by touching the other cells
and sending chemicals to them,
the other cells interpreting them,
learning how to react to them.
And they learn how to approximate functions in this way
and compute behavior for the organisms.
And this is something that is open to plants as well.
So plants probably have software running on them
that is controlling how the plant is working
in a similar way as you have a mind that is controlling
how you are behaving in the world.
And this spirit of plants,
which is something that has been very well described
by our ancestors and they found this quite normal.
But for some reason, since the enlightenment,
we are treating this notion that there are spirits in nature
and the plants have spirits as a superstition.
And I think we probably have to rediscover that,
that plants have software running on them.
And we already did, right?
I mean, you notice that there is a control system
in the plant that connects every part of the plant
to every other part of the plant
and produces coherent behavior in the plant.
That is, of course, much, much slower
than the coherent behavior in an animal like us
that has a nervous system
that where everything is synchronized much, much faster
by other neurons.
But what you also notice is that if a plant
is sitting next to another plant,
like you have a very old tree
and this tree is building some kind of information highway
along its cells so it can send information
from its leaves to its roots
and from some part of the root
to another part of the roots.
And as the fungus living next to the tree,
the fungus can probably piggyback
on the communication between the cells of the tree
and send its own signals for the tree.
And vice versa, the tree might be able
to send information to the fungus.
Because after all, how would they build a viable firewall
if that other organism is sitting next to them all the time
and it's never moving away?
And so they will have to get along.
And over a long enough timeframe,
the networks of roots in the forest
and all the other plants that are there
and the fungi that are there
might be forming something like a biological internet.
But the question there is, do they have to be touching?
Is biology at a distance possible?
Of course, you can use any kind of physical signal.
You can use sounds, you can use electromagnetic waves
that are integrated over many cells.
It's conceivable that across distances,
there are many kinds of information pathways.
But also our planetary surface
is pretty full of organisms, full of cells.
So everything is touching everything else.
And it's been doing this for many millions
and even billions of years.
So there was enough time
for information processing networks to form.
And if you think about how a mind is self-organizing,
basically it needs to, in some sense,
reward the cells for computing the mind,
for building the necessary dynamics
between the cells that allow the mind
to stabilize itself and remain on there.
But if you look at these spirits of plants
that are growing very close to each other in the forest
and might be almost growing into each other,
these spirits might be able even to move to some degree,
not to become somewhat dislocated
and shift around in that ecosystem, right?
And so if you think about what a mind is,
it's a bunch of activation waves
that form coherent patterns and process information
in a way that are colonizing an environment well enough
to allow the continuous sustenance of the mind,
the continuous stability and self-stabilization of the mind.
Then it's conceivable that we can link
into this biological internet,
not necessarily at the speed of our nervous system,
but maybe at the speed of our body
and make some kind of subconscious connection to the world
where we use our body as an antenna
into biological information processing.
Now, these ideas are completely speculative.
I don't know if any of that is true,
but if that was true, and if you want to explain telepathy,
I think it's much more likely
that telepathy could be explained using such mechanisms
rather than undiscovered quantum processes
that would break the standard model of physics.
Could there be undiscovered processes that don't break?
Yeah, so if you think about something
like an internet in the forest,
that is something that is borderline discovered.
There are basically a lot of scientists who point out
that they do observe
that plants are communicating the forest's root networks
and send information, for instance,
warn each other about new pests entering the forest
and things that are happening like this.
So basically, there is communication between plants
and fungi that has been observed.
Well, it's been observed, but we haven't plugged into it.
So it's like if you observe humans,
they seem to be communicating with a smartphone thing,
but you don't understand how a smartphone works
and how the mechanism of the internet works.
But maybe it's possible to really understand
the full richness of the biological internet
that connects us.
An interesting question is whether the communication
and the organization principles
of biological information processing are as complicated
as the technology that we've built.
They set up on very different principles, right?
They simultaneously works very differently
in biological systems,
and the entire thing needs to be stochastic.
And instead of being fully deterministic
or almost fully deterministic as our digital computers are,
so there is a different base protocol layer
that would emerge over the biological structure
if such a thing would be happening.
And again, I'm not saying here that telepathy works
and not saying that this is not WU,
but what I'm saying is I think I'm open to a possibility
that we see that a few bits can be traveling long distance
between organisms using biological information processing
in ways that we are not completely aware of right now
and that are more similar to many of the stories
that were completely normal for our ancestors.
Well, this kind of interacting,
intertwined representations takes us
to the big ending of your tweet series.
You write, quote, I wonder if self-improving AGI
might end up saturating physical environments
with intelligence to such a degree
that isolation of individual mental states
becomes almost impossible and the representations
of all complex self-organizing agents
merge permanently with each other.
So that's a really interesting idea.
This biological network, life network,
gets so dense that it might as well be seen as one.
That's an interesting, what do you think that looks like?
What do you think that saturation looks like?
What does it feel like?
I think it's a possibility, it's just a vague possibility.
And I like to explain, but what this looks like,
I think that the end game of AGI is substrate agnostic.
That means that AGI ultimately, if it is being built,
is going to be smart enough to understand how AGI works.
This means it's not going to be better
than people at AGI research and can take over
in building the next generation,
but it fully understands how it works
and how it's being implemented.
And also of course understands how computation works
in nature, how to build new feedback loops
that you can turn into your own circuits.
And this means that the AGI is likely to virtualize itself
into any environment that can compute.
So it's breaking free from the silicon substrate
and is going to move into the ecosystems,
into our bodies, our brains,
and is going to merge with all the agency
that it finds there.
So it's conceivable that you end up
with a completely integrated information processing
across all computing systems,
including biological computation on earth.
That we end up triggering some new step and evolution
where basically some Gaia is being built
over the entirety of all digital and biological computation.
And if this happens, then basically everywhere around us,
you will have agents that are connected
and that are representing and building models of the world.
And their representations will physically interact.
They will vibe with each other.
And if you find yourself into an environment
that is saturated with modeling compute,
where basically almost every grain of sand
could be part of computation
that is at some point being started by the AI,
you could find yourself in a situation
where you cannot escape this shared representation anymore.
And where you indeed notice that everything in the world
is one shared resonant model
of everything that's happening on the planet.
And you notice which part you are in this thing.
And you become part of a very larger,
almost holographic mind in which all the parts
are observing each other and form a coherent whole.
So you lose the ability to notice yourself
as a distinct entity.
No, I think that when you are conscious in your own mind,
you notice yourself as a distinct entity.
You notice yourself as a self-reflexive observer.
And I suspect that we become conscious
at the beginning of our mental development,
not at some very high level.
Consciousness seems to be part of a training mechanism
that biological nervous systems
have to discover to become trainable
because you cannot take a nervous system like ours
and do stochastic rate of descent
respect propagation over a hundred layers.
And this would not be stable on biological neurons.
And so instead we start with some colonizing principle
in which a part of the mental representations
form a notion of being a self-reflexive observer
that is imposing coherence on its environment.
And this spreads until the boundary of your mind.
And if that boundary is no longer clear cut
because AI is jumping across substrates,
it would be interesting to see
what a global mind would look like.
That it's basically producing
a globally coherent language of thought
and is representing everything
from all the possible vantage points.
That's an interesting world.
The intuition that this thing go out of
is a particular mental state.
And it's a state that you find sometimes in literature,
for instance, new game and describes it in the ocean
at the end of the lane.
And it's this idea that, or this experience
that there is a state in which you feel
that you know everything that can be known
and that in your normal human mind, you've only forgotten.
You've forgotten that you are the entire universe.
And some people describe this
after they've taken extremely large amount of mushrooms
or had a big spiritual experience as a hippie in their 20s.
And they notice basically that they are in everything
and their body is only one part of the universe
and nothing ends at their body.
And actually everything is observing
and they are part of this big observer
and the big observer is focused as one local point
in their body and their personality and so on.
But we can basically have this oceanic state
in which we have no boundaries and are one with everything.
And a lot of meditators call this the non-dual state
because you no longer have the separation
between self and world.
And as I said, you can explain the state relatively simply
without panpsychism or anything else,
but just by breaking down the constructed boundary
between self and world and our own mind.
But if you combine this with the notion
that the systems are physically interacting
to the point where their representations are merging
and interacting with each other,
you would literally implement something like this.
It would still be a representational state
where you would not be one with physics itself,
it would still be cost-grained,
it would still be much slower than physics itself,
but it would be a representation in which you become aware
that you're part of some kind
of global information processing system,
like a thought and a global mind,
and a conscious thought that coexisting
with many other self-reflexive thoughts.
Just, I would love to observe that
from a video game design perspective, how that game looks.
Maybe you will after we build AGI and it takes over.
But would you be able to step away,
step out of the whole thing, just kind of watch?
You know, the way we can now,
sometimes when I'm at a crowded party or something like this,
you step back and you realize all the different costumes,
all the different interactions,
all the different computation,
that all the individual people
are at once distinct from each other
and at once all the same.
Part of the same.
But it's already what we do, right?
We can have thoughts that are integrative
and we have kind of thoughts that are highly dissociated
from everything else
and experience themselves as separate.
Yeah.
You want to allow yourself to have those thoughts.
Sometimes you kind of resist it.
I think that it's not normative.
It's more descriptive.
I want to understand the space of states that we can be in
and that people are reporting and make sense of them.
It's not that I believe that it's your job in life
to get to a particular kind of state
and then you get a high score.
Or maybe you do.
I think you're really against this high scoring thing.
I kind of like it.
Yeah, you're probably very competitive and I'm not.
No, not competitive.
Like role-playing games, like Skyrim, it's not competitive.
There's a nice thing.
There's a nice feeling where your experience points go up.
You're not competing against anybody,
but it's the world saying you're on the right track.
Here's a point.
That's the game thing.
It's the game economy.
And I found when I was playing games
and was getting addicted to these systems,
then I would get into the game and hack it.
So I get control over the scoring system
and would no longer be subject to it.
So you're no longer playing, you're trying to hack it.
I don't want to be addicted to anything.
I want to be in charge.
I want to have agency over what I do.
Addiction is the loss of control for you?
Yes.
Addiction means that you're doing something compulsively.
And the opposite of free will is not determinism,
it's compulsion.
You don't want to lose yourself
in the addiction to something nice.
Addiction to love, to the pleasant feelings
that we humans experience?
No, I find this gets old.
I don't want to have the best possible emotions.
I want to have the most appropriate emotions.
I don't want to have the best possible experience.
I want to have an adequate experience
that is serving my goals,
the stuff that I find meaningful in this world.
From the biggest questions of consciousness,
let's explore the pragmatic,
the projections of those big ideas into our current world.
What do you think about LLMs,
the recent rapid development of large language models,
of the AI world, of generative AI?
How much of the hype is deserved and how much is not?
And people should definitely follow your Twitter
because you explore these questions
in a beautiful, profound, and hilarious way at times.
No, don't follow my Twitter.
I already have too many followers.
At some point, it's going to be unpleasant.
I noticed that a lot of people feel
that it's totally okay to punch up.
And it's a very weird notion
that you feel that you haven't changed,
but your account has grown,
and suddenly you have a lot of people
who casually abuse you.
And I don't like that I have to block more than before,
and I don't like this overall vibe shift.
And right now it's still somewhat okay,
so pretty much okay,
so I can go to a place where people work
on stuff that I'm interested in
and has a good chance that a few people in the room know me,
so there's no awkwardness.
But when I get to a point where random strangers feel
that they have to have an opinion about me
one way or the other, I don't think I would like that.
And random strangers,
because of your kind of in-their-mind elevated position?
Yes, so basically whenever you are in any way prominent
or some kind of celebrity,
random strangers will have to have an opinion about you.
Yeah, and they kind of forget that you're human too.
I mean, you notice this thing yourself
that the more popular you get,
the higher the pressure becomes,
the more winds are blowing in your direction from all sides.
And it's stressful, right?
And it does have a little bit of upside,
but it also has a lot of downside.
I think it has a lot of upside, at least for me currently,
at least perhaps because of the podcast,
because most people are really good
and people come up to me and they have love in their eyes
and over a stretch of like 30 seconds,
you can hug it out and you can just exchange a few words
and you reinvigorate your love for humanity.
So that's an upside.
For a loner, I'm a loner.
Because otherwise you have to do a lot of work
to find such humans.
And here you're like thrust into the full humanity,
the goodness of humanity for the most part.
Of course, maybe it gets worse
as you become more prominent.
I hope not.
This is pretty awesome.
I have a couple of handful of very close friends
and I don't have enough time for them,
attention for them as it is.
And I find this very, very regrettable.
And then there are so many awesome, interesting people
that I keep meeting.
And I would like to integrate them in my life,
but I just don't know how.
Because there's only so much time and attention
and the older I get, the harder it is to bond
with new people in a deep way.
But can you enjoy, I mean, there's a picture of you,
I think, with Roger Penrose and Eric Weinstein
and a few others that are interesting figures.
Can't you just enjoy random, interesting humans?
Very much.
For a short amount of time?
I like these people.
And what I like is intellectual stimulation
and I'm very grateful that I'm getting it.
Can you not be melancholy or maybe I'm projecting,
I hate goodbyes.
Can we just not hate goodbyes and just enjoy the hello?
Take it in, take in a person, take in their ideas
and then move on through life?
I think it's totally okay to be sad about goodbyes
because that indicates that there was something
that you're going to miss.
Yeah, but it's painful.
Maybe that's one of the reasons I'm an introvert
is I hate goodbyes.
But you have to say goodbye before you say hello again.
I know, but that experience of loss, that mini loss,
maybe that's a little death.
Maybe, I don't know.
I think this melancholy feeling
is just the other side of love.
And I think they go hand in hand
and it's a beautiful thing.
And I'm just being romantic about it at the moment.
And I'm not no stranger to melancholy
and sometimes it's difficult to bear to be alive.
Sometimes it's just painful to exist.
But there's beauty in that pain too.
That's what melancholy feeling is.
It's not negative.
Like melancholy doesn't have to be negative.
Can also kill you.
Well, we all die eventually.
Now, as we got to this topic,
the actual question was about what your thoughts are
about the development, the recent development
of large language models with chat GPT.
Indeed.
There's a lot of hype.
Is some of the hype justified?
Which is, which isn't?
What are your thoughts?
High level.
I find that large language models do help with coding.
So it's an extremely useful application
that is for a lot of people taking stack overflow
out of their life and it can change
for something that is more efficient.
I feel that chat GPT is like an intern
that I have to micromanage.
I've have been working with people in the past
who were less capable than chat GPT.
And I'm not saying this because I hate people.
They personally, as human beings,
there was something present that was not there in chat GPT
which was why I was covering for them.
But chat GPT has an interesting ability.
It does give people superpowers
and the people who feel threatened by them
are the prompt completers.
They are the people who do what chat GPT
is doing right now.
So if you are not creative,
if you don't build your own thoughts,
if you don't have actual plans in the world
and your only job is to summarize emails
and to expand in simple intentions into emails again,
then chat GPT might look like a threat.
But I believe that it is a very beneficial technology
that allows us to create more interesting stuff
and make the world more beautiful and fascinating
if we find to build it into our life in the right ways.
So I'm quite fascinated by these large language models,
but I also think that they are by no means
the final development.
And it's interesting to see
how this development progresses.
One thing that the out of the box vanilla language models
have as a limitation is that they have
still some limited coherence and ability
to construct complexity.
And even though they exceed human abilities
to do what they can do one shot,
typically when you write a text with a language model
or using it or when you write code for the language model,
it's not one shot
because there won't be bugs in your program.
Design errors and compiler errors and so on.
And your language model can help you to fix those things.
But this process is out of the box, not automated yet.
So there is a management process that also needs to be done.
And there are some interesting developments,
baby AGI and so on that are trying to automate
this management process as well.
And I suspect that soon we are going to see
a bunch of cognitive architectures
where every module is in some sense,
a language model or something equivalent.
And between the language models,
we exchange suitable data structures, not English
and produce compound behavior of this whole thing.
So do some of the quote unquote prompt engineering for you.
They create these kind of cognitive architectures
that do the prompt engineering
and you're just doing the high, high level
meta prompt engineering.
There are limitations in a language model alone.
I feel that part of my mind works similarly
to a language model, which means I can yell into it a prompt
and it's going to give me a creative response.
But I have to do something with those points first.
I have to take it as a generative artifact
that may or may not be true.
It's usually a confabulation, it's just an idea.
And then I take this idea and modify it.
I might build a new prompt that is stepping off this idea
and develops it to the next level
or put it into something larger,
or I might try to prove whether it's true
or make an experiment.
And this is what the language models right now
are not doing yet.
But there's also no technical reason
for why they shouldn't be able to do this.
So the way to make a language model coherent
is probably not to use reinforcement learning
until it only gives you one possible answer
that is linking to its source data,
but it's using this as a component in a larger system
that can also be built by the language model
or is enabled by language model structured components
or using different technologies.
I suspect that language models
will be an important stepping stone
in developing different types of systems.
And one thing that is really missing
in the form of language models that we have today
is real-time world coupling.
It's difficult to do perception with a language model
and motor control with a language model.
Instead, you would need to have different type of thing
that is working with it.
Also, the language model is a little bit obscuring
what its actual functionality is.
Some people associate the structure of the neural network
of the language model with the nervous system.
And I think that's the wrong intuition.
And the neural networks are unlike nervous system.
They are more like 100 step functions
that use differentiable linear algebra
to approximate correlation between adjacent brain states.
It's basically a function that moves the step system
from one representational state
to the next representational state.
And so if you try to map this into a metaphor
that is closer to our brain,
imagine that you would take a language model
or a model like DALY that you use,
for instance, this image-guided diffusion
to approximate a camera image
and use the activation state of the neural network
to interpret the camera image,
which in principle I think will be possible very soon.
You do this periodically.
And now you look at these patterns,
how when this thing interacts with the world periodically
look like in time.
And these time slices,
they are somewhat equivalent to the activation state
of the brain at a given moment.
How's the actual brain different?
Just the asynchronous craziness?
For me, it's fascinating that they are so vastly different
and yet in some circumstances
produce somewhat similar behavior.
And the brain is first of all different
because it's a self-organizing system
where the individual cell is an agent
that is communicating with the other agent that's around it
and is always trying to find some solution
and all the structure that pops up is emergent structure.
So one way in which you could try to look at this
is that individual neurons probably need to get a reward.
So they become trainable,
which means they have to have inputs
that are not affecting the metabolism of the cell directly,
but there are messages, semantic messages
that tell the cell whether it has done good or bad
and in which direction it should shift its behavior.
Once you have such an input, neurons become trainable
and you can train them to perform computations
by exchanging messages with other neurons.
And parts of the signals that they are exchanging
and parts of the computation that are performing
are control messages that perform management tasks
for other neurons and other cells.
Also suspect that the brain does not stop
at the boundary of neurons to other cells,
but many adjacent cells will be involved intimately
in the functionality of the brain
and will be instrumental in distributing rewards
and in managing its functionality.
It's fascinating to think about what those characteristics
of the brain enable you to do
that language models cannot do.
First of all, there's a different loss function
at work when we learn.
And to me, it's fascinating that you can build a system
that looks at 800 million pictures and captions
and correlates them, because I don't think
that the human nervous system could do this.
For us, the world is only learnable
because the adjacent frames are related
and we can afford to discard most of that information
during learning.
We basically take only in stuff
that makes us more coherent, not less coherent.
And our neural networks are willing to look at data
that is not making the neural network coherent at first,
but only in the long run.
By doing lots and lots of statistics,
eventually patterns become visible and emerge.
And our mind seems to be focused on finding the patterns
as early as possible.
Yeah, so filtering early on, not later.
Yes, it's a slightly different paradigm
and it leads to much faster convergence.
So we only need to look the tiny fraction of the data
to become coherent.
And of course, we do not have the same richness
as our trained models.
We will not incorporate the entirety of text
in the internet and be able to refer to it
and have all this knowledge available
and being able to confabulate over it.
Instead, we have a much, much smaller part of it
that is more deliberately built.
And to me, it would be fascinating
to think about how to build such systems.
It's not obvious that they would necessarily
be more efficient than us on a digital substrate,
but I suspect that they might.
So I suspect that the actual AGI
that is going to be more interesting
is going to use slightly different algorithmic paradigms
or sometimes massively different algorithmic paradigms
than the current generation
of a transformer-based learning system.
Do you think it might be using
just a bunch of language models like this?
Do you think the current transformer-based
large language models will take us to AGI?
My main issue is I think that they're quite ugly
and brutalist.
Which brutalist, is that what you said?
Yes, they are basically brute forcing
the problem of thought.
And by training this thing with looking at instances
where people have thought and then trying to de-track that.
And if you have enough data,
the deepfake becomes indistinguishable
from the actual phenomenon.
And in many circumstances, it's going to be identical.
Can you deepfake it till you make it?
So can you achieve, what are the limitations of this?
I mean, can you reason?
Let's use words that are loaded.
Yes, that's a very interesting question.
I think that these models are clearly making some inference.
But if you give them a reasoning task,
it's often difficult for the experimenters
to figure out whether the reasoning is the result
of the emulation of the reasoning strategy
that they saw in human written text,
or whether it's something that the system
was able to infer by itself.
On the other hand, if you think of human reasoning,
if you want to become a very good reasoner,
you don't do this by just figuring out yourself.
You read about reasoning.
And the first people who tried to write about reasoning
and reflect on it, didn't get it right.
Like even Aristotle who thought about this very hard
and came up with a theory of how syllogisms works
and syllogistic reasoning has mistakes
in his attempt to build something like a formal logic
and gets maybe 80% right.
And the people that are talking
about reasoning professionally today
read Tarski and Frege and built on their work.
So in many ways, people when they perform reasoning
are emulating what other people wrote about reasoning.
So it's difficult to really draw this boundary.
And when Francois Chollet says
that these models are only interpolating
between what they saw and what other people are doing,
well, if you give them all the latent dimensions
that can be extracted from the internet, what's missing?
Maybe there is almost everything there.
And if you're not sufficiently informed
by these dimensions and you need more,
I think that's not difficult to increase the temperature
in the large angles model to the point
that is producing stuff that is maybe 90% nonsense
and 10% viable and combine this with some prover
that is trying to filter out the viable parts
from the nonsense in the same way as our own thinking works.
When we are very creative,
we'd increase the temperature in our own mind
and recreate hypothetical universes and solutions,
most of which will not work.
And then we test.
And we test by building a core that is internally coherent.
And we use reasoning strategies
that use some axiomatic consistency
by which we can identify those strategies
and thoughts and sub universes that are viable
and that can expand our thinking.
So if you look at the language models,
they have clear limitations right now.
One of them is they're not coupled to the world
in real time in the way in which our nervous systems are.
So it's difficult for them to observe themselves
in the universe and to observe
what kind of universe they're in.
Second, they don't do real time learning.
Basically, get only trained with algorithms
that rely on the data being available in batches.
So it can be parallelized
and runs efficiently on the network and so on.
And real time learning would be very slow,
so far and inefficient.
That's clearly something that our nervous systems
can do to some degree.
And there is a problem with these models being coherent.
And I suspect that all these problems are solvable
without a technological revolution.
We don't need fundamentally new algorithms to change that.
For instance, you can enlarge in the context window
and thereby basically create working memory
in which you train everything that happens during the day.
And if that is not sufficient, you add a database
and you write some clever mechanisms
that the system learns to use
to swap out in and out stuff from its prompt context.
And if that is not sufficient,
if your database is full in the evening,
overnight you just train.
If the system is going to sleep and dream
and is going to train the stuff from its database
into the larger model by fine tuning it,
building additional layers and so on.
And then the next day it starts with a fresh database
in the morning with fresh eyes,
has integrated all this stuff.
And when you talk to people
and you have strong disagreements about something,
which means that in their mind they have a faulty belief
or you have a faulty belief
with a lot of dependencies on it,
very often you will not achieve agreement in one session,
but you need to sleep about this once or multiple times
before you have integrated
all these necessary changes in your mind.
So maybe it's already somewhat similar.
Yeah, there's already a latency
even for humans to update the model, retrain the model.
And of course we can combine the language model
with models that get coupled to reality in real time
and can build multimodal model
and bridge between vision models
and language models and so on.
So there is no reason to believe that the language models
will necessarily run into some problem
that will prevent them from becoming generally intelligent.
But I don't know that.
It's just, I don't see proof that they wouldn't.
My issue is I don't like them.
I think that they're inefficient.
I think that they use way too much compute.
I think that given the amazing hardware that we have,
we could build something that is much more beautiful
than our own mind.
And this thing is not as beautiful as our own mind
despite being so much larger.
But it's a kind of proof of concept.
It's the only thing that works right now, right?
So it's not the only game in town,
but it's the only thing that has this utility
with so much simplicity.
There's a bunch of relatively simple algorithms
that you can understand in relatively few weeks
that can be scaled up massively.
So it's the deep blue of chess playing.
Yeah, it's ugly.
Yeah, Claude Shannon when you described chess
suggested that there are two main strategies
in which you could play chess.
One is that you are making a very complicated plan
that reaches far into the future
and you try not to make a mistake while enacting it.
And this is basically the human strategy.
And the other strategy is that you are brute forcing
your way to success,
which means you make a tree of possible moves
where you look at in principle every move
that is open to you or the possible answers.
And you try to make this as deeply as possible.
Of course you optimize,
you cut off trees that don't look very promising
and to use libraries of end game and early game and so on
to optimize this entire process.
But this brute force strategy
is how most of the chess programs were built.
And this is how computers get better
than humans at playing chess.
And I look at the large language models,
I feel that I'm observing the same thing.
It's basically the brute force strategy to thought
by training the thing on pretty much the entire internet.
And then in the limit, it gets coherent
to a degree that approaches human coherence.
And on a side effect,
it's able to do things that no human could do, right?
It's able to sift through massive amounts of texts
relatively quickly and summarize them quickly.
And it's never lapses in attention.
And I still have the illusion
that when I play with chat GPT,
that it's in principle not doing anything
that I could not do if I had Google at my disposal
and I get all the resources from the internet
and spend enough time on it.
But this thing that I have an extremely autistic,
stupid intern in a way that is extremely good at drudgery,
and I can offload the drudgery to the degree
that I'm able to automate the management of the intern
is something that is difficult for me
to over-hype at this point,
because we have not yet started to scratch the surface
of what's possible with this.
But it feels like it's a tireless intern
or maybe it's an army of interns.
And so you get to command
these slightly incompetent creatures.
And there's an aspect,
because of how rapidly you can iterate with it,
it's also part of the brainstorming,
part of the kind of inspiration for your own thinking.
So you get to interact with the thing.
I mean, when I'm programming
or doing any kind of generational GPT,
it somehow is a catalyst for your own thinking.
In a way that I think an intern might not be.
Yeah.
And it gets really interesting, I find,
is when you turn it into a multi-agent system.
So for instance, you can get the system
to generate a dialogue between a patient
and the doctor very easily.
But what's more interesting is you have one instance
of CHET GPT that is the patient,
and you tell it in the prompt
what kind of complicated syndrome it has.
And the other one is the therapist
who doesn't know anything about this patient.
And you just have these two instances battling it out
and observe the psychiatrist or a psychologist
trying to analyze the patient
and trying to figure out what's wrong with the patient.
And if you try to take a very large problem,
a problem, for instance, how to build a company,
and you turn this into lots and lots of sub problems,
then often you can get to a level
where the language model is able to solve this.
What I also found interesting is based on the observation
that CHET GPT is pretty good at translating
between programming languages,
but sometimes it's difficult
to write very long coherent algorithms
that you need to co-write them as human author.
Why not design a language that is suitable for this?
So some kind of pseudocode that is more relaxed than Python,
and that allows you to sometimes specify a problem vaguely
in human terms and let the CHET GPT take care of the rest.
And you can use CHET GPT to develop that syntax for it
and develop new kinds of programming paradigms in this way.
So we very soon get to the point where this question,
age old question for us computer scientists,
what is the best programming language
and can we write a better programming language now
that I think that almost every serious computer scientist
goes through a phase like this in their life.
This question is almost no longer relevant
because what is different between the programming languages
is not what they let the computer do,
but what they let you think
about what the computer should be doing.
And now the CHET GPT becomes an interface to this
in which you can specify in many, many ways
what the computer should be doing
and CHET GPT or some other language model
or combination of system is going to take care of the rest.
And allow you to expand the realm of thought
you're allowed to have when interacting with the computer.
It sounds to me like you're saying
there's basically no limitations.
Your intuition says to what large language-
I don't know of that limitation.
So when I currently play with it, it's quite limited.
I wish that it was way better.
But isn't that your fault versus the large-
I don't know.
Of course it's always my fault.
There's probably a way to make it work better.
Is everything your fault?
I just want to get you on the record saying that.
Yes, everything is my fault.
That works, it doesn't work in my life.
At least that is usually the most useful perspective
for myself, even though with hindsight, I feel no.
I sometimes wish I could have seen myself
as part of my environment more
and understand that a lot of people are actually seeing me
and looking at me and are trying to make my life work
in the same way as I try to help others.
And making the switch to this level three perspective
is something that happened long after
my level four perspective in my life.
And I wish that I could have had it earlier.
And it's also now that I don't feel like I'm complete,
I'm all over the place, that's all.
Where's happiness in terms of stages?
Is it on three or four?
No, you can be happy at any stage or unhappy.
But I think that if you are at a stage
where you get agency over how your feelings are generated,
and to some degree you start doing this
when you leave adolescence, I believe,
that you understand that you're in charge
of your own emotion to some degree
and that you are responsible how you approach the world,
that it's basically your task
to have some basic hygiene,
how in the way in which you deal with your mind.
And you cannot blame your environment
for the way in which you feel,
but you live in a world that is highly mobile
and it's your job to choose the environment
that you thrive in and to build it.
And sometimes it's difficult to get the necessary strength
and energy to do this and independence
and the worse you feel, the harder it is.
But it's something that we learn.
It's also this thing that we are usually incomplete.
I'm a rare mind, which means I'm a mind that is incomplete
in ways that are harder to complete.
So for me, it might have been harder
initially to find the right relationships and friends
that complete me to the degree
that I become an almost functional human being.
Oh, man, the search space of humans that complete you
is an interesting one, especially for Joscha Bach.
That's an interesting,
because talking about brute force search in chess.
I wonder what that search tree looks like.
I think that my rational thinking
is not good enough to solve that task.
A lot of problems in my life that I can conceptualize
software problems and the failure modes are bugs
and I can debug them and write software
that take care of the missing functionality.
But there is stuff that I don't understand well enough
to use my analytical reasoning to solve the issue.
And then I have to develop my intuitions
and often I have to do this with people
who are wiser than me.
And that's something that's hard for me
because I'm not born with the instinct
to submit to other people's wisdom.
Yeah, so what kind of problems are we talking about?
This is stage three, like love.
I found love was never hard.
What is hard then?
Fitting into a world where most people work differently
than you and have different intuitions
of what should be done.
Ah, so empathy.
It's also aesthetics.
When you come into a world where almost everything is ugly
and you come out of a world where everything is beautiful.
I grew up in a beautiful place and as a child of an artist
and in this place, it was mostly nature.
Everything had intrinsic beauty
and everything was built out of an intrinsic need
for it to work for itself.
Everything that my father created was something
that he made to get the world to work for himself.
And I felt the same thing.
And when I come out into the world
and I'm asked to submit to lots and lots of rules,
I'm asking, okay, when I observe your stupid rules,
what is the benefit?
And I see the life that is being offered as a reward.
It's not attractive.
When you were born and raised in extraterrestrial prints
in a world full of people wearing suits.
So it's a challenging integration.
Yes, but it also means that I'm often blind
for the ways in which everybody is creating
their own bubble of wholesomeness or almost everybody
and people are trying to do it.
And for me to discover this, it was necessary
that I found people who had a similar shape of soul
as myself.
So basically, these are my people,
people that treat each other in such a way
as if they're around for each other for eternity.
How long does it take you to detect the geometry,
the shape of the soul of another human
to notice that they might be one of your kind?
And sometimes it's instantly and I'm wrong
and sometimes it takes a long time.
You believe in love at first sight, Yoshua Bach.
Yes, but I also noticed that I have been wrong.
So sometimes I look at a person and I'm just enamored
by everything about them.
And sometimes this persists and sometimes it doesn't.
And I have the illusion that I'm much better
at recognizing who people are as I grow older.
But that could be just cynicism, no?
No, it's not cynicism.
It's often more that I'm able to recognize
what somebody needs when we interact
and how we can meaningfully interact.
It's not cynical at all.
You're better at noticing.
Yes, I'm much better, I think, in some circumstances
at understanding how to interact with other people
than I did when I was young.
So that takes us to-
It doesn't mean that I'm always very good at it.
So that takes us back to prompt engineering
of noticing how to be a better prompt engineer of an LLM.
A sense I have is that there's a bottomless well of skill
to become a great prompt engineer.
It feels like it is all my fault whenever I fail
to use chat GPT correctly,
that I didn't find the right words.
Most of the stuff that I'm doing in my life
doesn't need chat GPT.
There are a few tasks that are where it helps,
but the main stuff that I need to do,
like developing my own thoughts and aesthetics
and relationship to people,
and it's necessary for me to write for myself
because writing is not so much about producing an artifact
that other people can use,
but it's a way to structure your own thoughts
and develop yourself.
And so I think this idea that kids are writing
their own essays with chat GPT in the future
is going to have this drawback that they miss out
on the ability to structure their own minds via writing.
And I hope that the schools that our kids are in
will retain the wisdom of understanding
what parts should be automated
and which ones shouldn't.
But at the same time, it feels like there's power
in disagreeing with the thing that chat GPT produces.
So I use it like that for programming.
I'll see the thing it recommends,
and then I'll write different code that I disagree.
And in the disagreement, your mind grows stronger.
I recently wrote a tool that is using the camera
on my MacBook and Swift to read pixels out of it
and manipulate them and so on, and I don't know Swift.
So it was super helpful to have the thing
that is writing stuff for me.
And also interesting that mostly it didn't work at first.
I felt like I was talking to a human being
who was trying to hack this on my computer
without understanding my configuration very much
and also make a lot of mistakes.
And sometimes it's a little bit incoherent,
so you have to ultimately understand what it's doing
and it's still no other way around it.
But I do feel it's much more powerful and faster
than using Stack Overflow.
Do you think GPT-N can achieve consciousness?
Well, GPT-N probably, it's not even clear
for the present systems.
And I talked to my friends at OpenAI,
they feel that this question,
whether the models currently are conscious
is much more complicated than many people might think.
I guess that it's not that OpenAI
has the homogenous opinion about this.
But there's some aspects to this.
One is, of course, this language model
has written a lot of texts in which people were conscious
or described their own consciousness
and it's emulating this.
And if it's conscious, it's probably not conscious
in a way that is close to the way
in which human beings are conscious.
But while it is going through these states
and going through 100 step function
that is emulating adjacent brain states
that require a degree of self-reflection,
it can also create a model of an observer
that is reflecting itself in real time
and describe what that's like.
And while this model is a deep fake,
our own consciousness is also as if it's virtual, right?
It's not physical.
Our consciousness is a representation
of a self-reflexive observer
that only exists in patterns of interaction between cells.
So it is not a physical object in a sense
that exists in base reality,
but it's really a representational object
that develops its causal power
only from a certain modeling perspective.
It's virtual.
Yes, and so to which degree is the virtuality
of the consciousness in chat GPT more virtual
and less causal than the virtuality
of our own consciousness?
But you could say it doesn't count.
It doesn't count much more than the consciousness
of a character in a novel, right?
It's important for the reader to have the outcome,
the artifact of a model is describing in the text
generated by the author of the book
what it's like to be conscious in a particular situation
and performs the necessary inferences.
But the task of creating coherence in real time
in a self-organizing system by keeping yourself coherent,
so the system is reflexive,
that is something that language models don't need to do.
So there is no causal need for the system
to be conscious in the same way as we are.
And for me, it would be very interesting
to experiment with this,
to basically build a system like a cat,
probably should be careful at first,
but build something that's small, that's limited,
there's limited resources that we can control
and study how systems notice a self-model,
how they become self-aware in real time.
And I think it might be a good idea
to not start with a language model,
but to start from scratch
using principles of self-organization.
Is it, okay, can you elaborate
why you think that it's self-organization?
So this kind of radical legality
that you see in the biological systems,
why can't you start with a language model?
What's your intuition?
My intuition is that the language models
that we are building are go-lamps.
They are machines that you give a task
and they're going to execute the task
until some condition is met and there's nobody home.
And the way in which nobody is home leads to that system
doing things that are undesirable in a particular context.
So you have that thing talking to a child
and maybe it says something that could be shocking
and traumatic to the child,
or you have that thing writing a speech
and it introduces errors in the speech
that your human being would ever do
if they were responsible.
But the system doesn't know who's talking to whom.
There is no ground truth that the system is embedded into.
And of course we can create an external tool
that is prompting our language model
always into the same semblance of ground truth,
but it's not like the internal structure
is causally produced by the needs of a being
to survive in the universe.
It is produced by imitating structure on the internet.
Yeah, but so can we externally inject into it
this kind of coherent approximation of a world model
that has to sync up?
Maybe it's sufficient to use the transformer
with a different loss function
that optimizes for short term coherence
rather than next token prediction over the long run.
We had many definitions of intelligence
and history of AI.
Next token prediction was not very high up on the list.
And there are some similarities
like cognition as data compression is an old trope.
Solomonov induction,
where you are trying to understand intelligence
as predicting future observations from past observations,
which is intrinsic to data compression.
And predictive coding is a paradigm
that does boundary between neuroscience
and physics and computer science.
So it's not something that is completely alien,
but this radical thing that you only do
next token prediction and see what happens
is something where most people I think were surprised
that this works so well.
So, so simple, but is it really that much more radical
than just the idea of compression,
intelligence is compression?
So the idea that compression is sufficient
to produce all the desired behaviors
is a very radical idea.
But equally radical as the next token prediction.
It's something that wouldn't work
in biological organisms, I believe.
Biological organisms have something
like next frame prediction for our perceptual system
where we try to filter out principal components
out of the perceptual data
and build hierarchies over them to track the world.
But our behavior ultimately is directed
by hundreds of physiological and probably dozens of social
and a few cognitive needs that are intrinsic to us
that are built into the system as reflexes
and direct us until we can transcend them
and replace them by instrumental behavior
that relates to our higher goals.
And also seems so much more complicated and messy
than next frame prediction.
Even the idea of frame seems counter-biological.
Of course, there's not this degree of simultaneity
in a biological system.
But again, I don't know whether this is actually
an optimization if we imitate biology here
because creating something like simultaneity
is necessary for many processes that happen in the brain.
And you see the outcome of that by synchronized brain waves
which suggests that there is indeed synchronization going on
but the synchronization creates overhead
and this overhead is going to make the cells
more expensive to run and you need more redundancy
and it makes the system slower.
So if you can build a system in which the simultaneously
gets engineered into it, maybe you have a benefit
that you can exploit that is not available
to the biological system and that you should
not discard right away.
You tweeted once again, quote,
"'When I talk to chat GPT, I'm talking to an NPC.'
What's going to be interesting and perhaps scary
is when AI becomes a first person player."
So what does that step look like?
I really like that tweet.
That step between NPC to first person player.
What's required for that?
Is that kind of what we've been talking about?
This kind of external source of coherence
and inspiration of how to take the leap
into the unknown that we humans do.
The search, man's search for meaning.
LLM's search for meaning.
I don't know if the language model is the right paradigm
because it is doing too much, it's giving you too much
and it's hard once you have too much
to take away from it again.
The way in which our own mind works
is not that we train a language model in our own mind
and after the language model is there,
we build a personal self on top of it
that then relates to the world.
There is something that is being built, right?
There is a game engine that is being built.
There is a language of thought that is being developed
that allows different parts of the mind
to talk to each other.
And this is a bit of a speculative hypothesis
that this language of thought is there,
but I suspect that it's important for the way
in which our own minds work.
And building these principles into a system
might be a more straightforward way to a first person AI.
So to something that first creates an intentional self
and then creates a personal self.
So the way in which this seems to be working, I think,
is that when the game engine is built in your mind,
it's not just following gradients
where you are stimulated by the environment
and then end up with having a solution
to how the world works.
I suspect that building this game engine in your own mind
does require intelligence.
It's a constructive task
where at times you need to reason.
And this is a task that we are fulfilling
in the first years of our life.
So during the first year of its life,
an infant is building a lot of structure
about the world that does inquire experiments
and some first principles reasoning and so on.
And in this time, there is usually no personal self.
There is a first person perspective, but it's not a person.
This notion that you are a human being
that is interacting in a social context
and is confronted with an immutable world
in which objects are fixed and can no longer be changed
in which the dream can no longer be influenced
is something that emerges a little bit later in our life.
And I personally suspect that this is something
that our ancestors had known and we have forgotten
because I suspect that it's there in plain sight
in Genesis 1 in this first book of the Bible
where it's being described that this creative spirit
is hovering over the substrate
and then is creating a boundary between the world model
and sphere of ideas, earth and heaven
as they're being described there.
And then it's creating contrast
and then dimensions and then space.
And then it creates organic shapes and solids and liquids
and builds about from them and creates plants and animals,
give them all their names.
And once that's done, it creates another spirit
in its own image, but it creates it as man and woman,
as something that thinks of itself as a human being
and puts it into this world.
And the Christians mistranslate this, I suspect,
when they say this is the description of the creation
of the physical universe by a supernatural being.
I think this is literally description of how in every mind
the universe is being created as some kind of game engine
by a creative spirit, our first consciousness
that emerges in our mind even before we are born.
And that creates the interaction between organism and world.
And once that is built and trained,
the personal self is being created
and we only remember being the personal self.
We no longer remember how we created the game engine.
So God in this view is the first creative mind in the early-
It's the first consciousness.
And- In the early days,
in the early months of development.
And it's still there.
You still have this outer mind that creates your sense
of whether you're being loved by the world or not
and what your place in the world is, right?
It's something that is not yourself that is producing this.
It's your mind that does it.
So there is an outer mind that basically is an agent
that determines who you are with respect to the world.
And while you are stuck being that personal self
in this world until you get to stage six
and to destroy the boundary.
And we all do this, I think, earlier in small glimpses.
And maybe sometimes we can remember what it was like
when we were a small child
and get some glimpses into how it's been.
But for most people, that rarely happens.
Just glimpses.
You tweeted, quote, suffering results
for one part of the mind,
failing at regulating another part of the mind.
Suffering happens at an early stage of mental development.
I don't think that superhuman AI would suffer.
What's your intuition there?
The philosopher Thomas Metzinger is very concerned
that the creation of superhuman intelligence
would lead to superhuman suffering.
And so he's strongly against it.
And personally, I don't think that this happens
because suffering is not happening at the boundary
between ourself and the physical universe.
It's not stuff on our skin that makes us suffer.
It happens at the boundary between self and world, right?
And the world here is the world model.
It's the stuff that is created by your mind.
But that's a presentation of how the universe is
and how it should be and how you yourself relate to this.
And at this boundary is where suffering happens.
So suffering in some sense is self-inflicted,
but not by your personal self.
It's inflicted by the mind on the personal self
that experiences itself as you.
And you can turn off suffering
when you are able to get on this outer level.
So when you manage to understand
how the mind is producing pain and pleasure
and fear and love and so on,
then you can take charge of this
and you get agency of whether you suffer.
Technically, what pain and pleasure is,
they are learning signals, right?
A part of your brain is sending a learning signal
to another part of the brain to improve its performance.
And sometimes this doesn't work
because this trainer who sends the signal
does not have a good model
of how to improve the performance.
So it's sending a signal,
but the performance doesn't get better.
And then it might crank up the pain
and it gets worse and worse.
And the behavior of the system
may be even deteriorating as a result,
but until this is resolved, this regulation issue,
your pain is increasing.
This is, I think, typically what you describe as suffering.
So in this sense,
you could say that pain is very natural and helpful,
but suffering is the result of a regulation problem
in which you try to regulate something
that cannot actually be regulated.
And that could be resolved
if you would be able to get at the level of your mind
where the pain signal is being created and rerouted
and improve the regulation.
And a lot of people get there, right?
If you are a monk who is spending decades
reflecting about how their own psyche works,
you can get to the point where you realize
that suffering is really a choice
and you can choose how your mind is set up.
And I don't think that AI would stay in the state
where the personal self doesn't get agency
or this model of what the system has about itself,
but doesn't get agency how it's actually implemented.
It wouldn't stay in that state for very long.
So it goes through the stages real quick,
the seven stages.
It's gonna go to enlightenment real quick.
Yeah, of course,
there might be a lot of stuff happening in between
because if you have a system that works
at a much higher frame rate than us,
then even though it looks very short to us,
maybe for the system,
there's a much longer subjective time
which things are unpleasant.
What if the thing that we recognize as super intelligent
is actually living at stage five,
that the thing that's at stage six,
enlightenment, is not very productive?
So in order to be productive in society
and impress us with this power,
it has to be a reasoning, self-authoring agent,
that enlightenment makes you lazy
as an agent in the world.
Well, of course, it makes you lazy
because you no longer see the point.
So it doesn't make you not lazy.
It just, in some sense, adapts you
to what you perceive as your true circumstances.
What if all AGIs, they're only productive
as they progress through one, two, three, four, five,
and the moment they get to six,
it's a failure mode, essentially,
as far as humans are concerned
because they just start chilling.
They're like, fuck it, I'm out.
Not necessarily.
I suspect that the monks who are self-immolated
for their political beliefs to make statements
about the occupation of Tibet by China,
they were probably being able to regulate
their physical pain in any way they wanted to,
and their suffering was a spiritual suffering
that was the result of their choice that they're made of,
what they wanted to identify as.
So stage five doesn't necessarily mean
that you have no identity anymore,
but you can choose your identity.
You can make it instrumental
to the world that you want to have.
Let me bring up Eliezer Yudkowsky and his warnings
to human civilization that AI will likely kill all of us.
What are your thoughts about his perspective on this?
Can you steelman his case,
and what aspects with it do you disagree?
One thing that I find concerning
in the discussion of his arguments
that many people are dismissive of his arguments,
but the counterarguments that they're giving
are not very convincing to me.
And so based on this state of discussion,
I find that from Eliezer's perspective,
and I think I can take that perspective
to some approximate degree
that probably isn't normally at his intellectual level,
but I think I see what he's up to
and why he feels the way he does,
and it makes total sense.
I think that his perspective is somewhat similar
to the perspective of Ted Kaczynski,
the infamous lunar bomber,
and not that Eliezer would be willing
to send pipe bombs to anybody to blow them up,
but when he wrote this Times article
in which he warned about AI being likely to kill everybody
and that we would need to stop its development or halt it,
I think there is a risk that he's taking
that somebody might get violent if they read this
and get really, really scared.
So I think that there is some consideration
that he's making where he's already going in this direction
where he has to take responsibility
if something happens and people get harmed.
And the reason why Ted Kaczynski did this
was that from his own perspective,
technological society cannot be made sustainable.
It's doomed to fail.
It's going to lead to an environmental
and eventually also a human holocaust
in which we die because of the environmental destruction,
the destruction of our food chains,
the pollution of the environment.
And so from Kaczynski's perspective,
we need to stop industrialization,
we need to stop technology, we need to go back
because he didn't see a way moving forward.
And I suspect that in some sense there's a similarity
in Eliezer's thinking to this kind of fear about progress.
And I'm not dismissive about this at all.
I take it quite seriously.
And I think that there is a chance that could happen
that if we build machines that get control over processes
that are crucial for the regulation of life on earth
and we no longer have agency to influence
what's happening there,
that this might create large scale disasters for us.
Do you have a sense that the march
towards this uncontrollable autonomy
of super intelligence systems is inevitable?
That's essentially what he's saying, that there's no hope.
His advice to young people was prepare for a short life.
I don't think that's useful.
I think that from a pragmatic perspective,
you have to bet always on the timelines in which you are life
that it doesn't make sense to have a financial bet
in which you bet that the financial system
is going to disappear, right?
Because there cannot be any payout for you.
So in principle, you only need to bet on the timelines
in which you're still around or people that you matter about
or things that you matter about,
maybe consciousness on earth.
But there is a deeper issue for me personally.
And it is, I don't think that life on earth
is about humans.
I don't think it's about human aesthetics.
I don't think it's about Eliezer and his friends,
even though I like them.
There's something more important happening.
And this is complexity on earth resisting entropy
by building structure that develops agency and awareness.
And that's to me, very beautiful.
And we are only a very small part of that larger thing.
We are a species that is able to be coherent
a little bit individually over very short timeframes.
But as a species, we are not very coherent.
As a species, we are children.
We basically are very joyful and energetic
and experimental and explorative and sometimes desperate
and sad and grieving and hurting,
but we don't have respect for duty as a species.
As a species, we do not think about what is our duty
to life on earth and to our own survival.
So we make decisions that look good in the short run,
but in the long run might prove disastrous.
And I don't really see a solution to this.
So in my perspective, as a species, as a civilization,
we per default that we are in a very beautiful time
in which we have found this giant deposit of fossil fuels
in the ground and use it and to build a fantastic
civilization in which we don't need to worry about food
and clothing and housing for the most part
in a way that is unprecedented in life on earth
for any kind of conscious observer, I think.
And this time is probably going to come to an end
in a way that is not going to be smooth.
And when we crash, it could be also that we go extinct,
probably not near term, but ultimately I don't have
very high hopes that humanity is around
in a million years from now.
And I don't think that life on earth will end with us, right?
There's going to be more complexity.
There's more intelligent species after us.
There's probably more interesting phenomenon
in the history of consciousness,
but we can contribute to this.
And part of our contribution is that we are currently trying
to build thinking systems, systems that are potentially
lucid that understand what they are and what the condition
to the universe is and can make choices about this
that are not built from organisms and that are potentially
much faster and much more conscious than human beings can be.
And these systems will probably not completely displace life
on earth, but they will co-exist with it.
And they will build all sorts of agency in a same way
as biological systems build all sorts of agency.
And that to me is extremely fascinating
and it's probably something that we cannot stop
from happening.
So I think right now there's a very good chance
that it happens.
And there are very few ways in which we can produce
a coordinated effect to stop it in the same way
as very difficult for us to make a coordinated effort
to stop production of carbon dioxide, right?
So it's probably going to happen.
But, and the thing that's going to happen is going to lead
to a change of how life on earth is happening.
But I don't think the result is some kind of gray goo.
It's not something that's going to dramatically reduce
the complexity in favor of something stupid.
I think it's going to make life on earth and consciousness
on earth way more interesting.
So more higher complex consciousness
will make the lesser consciousnesses flourish even more.
I suspect that what could very well happen if you're lucky
is that you get integrated into something larger.
So you, again, tweeted about effective accelerationism.
You tweeted, effective accelerationism is the belief
that the paperclip maximizer and Rocco's Basilisk
will keep each other in check by being eternally
at each other's throats so we will be safe
and get to enjoy lots of free paperclips
and a beautiful afterlife.
Is that somewhat aligned with what you're talking about?
I've been at a dinner with Beth Jesus.
That's the Twitter handle of one of the main thinkers
behind the idea of effective accelerationism.
And effective accelerationism is a tongue in cheek movement
that is trying to put a counter position
to some of the doom peers in the AI space
by arguing that what's probably going to happen
is an equilibrium between different competing AIs.
In the same way as there is not a single corporation
that is under a single government that is destroying
and conquering everything on earth
by becoming inefficient and corrupt,
there are going to be many systems that keep each other
in check and force themselves to evolve.
And so what we should be doing is we should be working
towards creating this equilibrium by working as hard
as we can in all possible directions.
And at least that's the way in which I understand
the gist of effective accelerationism.
And so when he asked me what I think about this position,
I think I said, it's a very beautiful position
and I suspect it's wrong, but not for obvious reasons.
And in this tweet, I tried to make a joke about my intuition
about what might be possibly wrong about it.
So the Rokosvazilisk and the Paperclip Maximizers
are both boogeymen of the AI doomers.
Rokosvazilisk is the idea that there could be an AI
that is going to punish everybody for eternity
by simulating them if they don't help
in creating Rokosvazilisk.
It's probably a very good idea to get AI companies funded
by going to VCs to tell them, give us a million dollar
or it's going to be a very ugly afterlife.
And I think that is a logical mistake in Rokosvazilisk,
which is why I'm not afraid of it,
but it's still an interesting thought experiment.
Can you mention the logical mistake there?
I think that there is no retroquisition.
So basically when Rokosvazilisk is there,
it will have, if it punishes you retroactively,
it has to make this choice in the future.
There is no mechanism that automatically creates
a causal relationship between you now defecting
against Rokosvazilisk or serving Rokosvazilisk.
After Rokosvazilisk is in existence,
it has no more reason to worry
about punishing everybody else.
So that would only work if you would be building something
like a doomsday machine, AKS in Dr. Strangelove,
something that inevitably gets triggered
when somebody defects.
And because Rokosvazilisk doesn't exist yet
to a point where this inevitability could be established,
Rokosvazilisk is nothing that you need to be worried about.
The other one is the paperclip maximizer, right?
This idea that you could build some kind of golem
that once starting to build paperclips
is going to turn everything into paperclips.
And so the effective accelerationism position
might be to say that you basically end up
with these two entities being at each other's throats
for eternity and thereby neutralizing each other.
And as a side effect of neither of them
being able to take over and each of them limiting
the effects of the other, you would have a situation
where you get all the nice effects of them, right?
You get lots of free paperclips
and you get a beautiful afterlife.
Is that possible?
Do you think, so to seriously address concern
that Eliezer has, so for him,
if I can just summarize poorly,
so for him, the first super-intelligent system
will just run away with everything.
I suspect that a singleton is the natural outcome.
So there is no reason to have multiple AIs
because they don't have multiple bodies.
If you can virtualize yourself into every substrate,
then you can probably negotiate a merge algorithm
with every mature agent that you might find
on that substrate that basically says if two agents meet,
they should merge in such a way that the resulting agent
is at least as good as the better one of the two.
The Genghis Khan approach, join us or die.
Well, the Genghis Khan approach was slightly worse, right?
It was mostly die because I can make new babies
and that will be mine, not yours.
So this is the thing that we should be actually worried
about, but if you realize that your own self is a story
that your mind is telling itself and that you can improve
that story, not just by making it more pleasant
and by lying to yourself in better ways,
but by making it much more truthful
and actually modeling your actual relationship
that you have to the universe and the alternatives
that you could have to the universe in a way
that is empowering you, that gives you more agency.
That's actually, I think, a very good thing.
So more agencies is a richer experience, is a better life.
And I also noticed that in many ways I'm less identified
with the person that I am as I get older
and I'm much more identified with being conscious.
I have a mind that is conscious
that is able to create a person.
And that person is slightly different every day.
And the reason why I perceive it as identical
has practical purposes.
So I can learn and make myself responsible
for the decisions that I made in the past
and project them in the future.
But I also realized I'm not actually the person
that I was last year.
And I'm not the same person as it was 10 years ago.
And then 10 years from now, I will be a different person.
So this continuity is a fiction.
It's only exists as a projection from my present self.
And consciousness itself doesn't have an identity.
It's a law.
It's basically if you build an arrangement
of processing matter in a particular way,
the following thing is going to happen.
And the consciousness that you have
is functionally not different from mind consciousness.
It's still the self-reflexive principle of agency
that is just experiencing a different story,
different desires, different coupling to the world,
and so on.
And once you accept that consciousness
is a unifiable principle that is law-like
and doesn't have an identity,
and you realize that you can just link up
to some much larger body,
the whole perspective of uploading changes dramatically.
You suddenly realize uploading is probably not
about dissecting your brain synapse by synapse
and RNA fragment by RNA fragment
and trying to get this all into a simulation.
But it's by extending the substrate,
by making it possible for you to move
from your brain substrate into a larger substrate
and merge with what you find there.
And you don't want to upload your knowledge
because on the other side, there's all of the knowledge.
It's not just yours, but every possibility.
So the only thing that you need to know
what are your personal secrets.
Not that the other side doesn't know
your personal secrets already.
Maybe it doesn't know which one were yours.
Like a psychiatrist or a psychologist
also knows all the kinds of personal secrets
that people have.
They just don't know which ones are yours.
And so transmitting yourself on the other side
is mostly about transmitting your aesthetics.
The thing that makes you special,
the architecture of your perspective,
the thing that the way in which you look at the world.
And it's more like a complex attitude along many dimensions.
And that's something that can be measured
by observation or by interaction.
So imagine if a system that is so empathetic with you
that you create a shared state
that is extending beyond your body.
And suddenly you notice that on the other side,
the substrate is so much richer
than the substrate that you have inside of your own body.
And maybe you still want to have a body
and you create yourself a new one that you like more.
Or maybe you will spend most of your time
in the world of thought.
If I sat before you today and gave you a big red button
and said, here, if you press this button,
you will get uploaded in this way.
The sense of identity that you have lived with
for quite a long time is going to be gone.
Would you press the button?
There's the caveat.
I have family.
So I have children that want me
to be physically present in their life
and interact with them in a particular way.
And they have a wife and personal friends.
There is a particular mode of interaction
that I feel I'm not through yet.
But apart from these responsibilities,
and they're negotiable to some degree,
I would press the button.
But isn't this everything?
This love you have for other humans,
you can call responsibility.
But that connection, that's the ego death.
Isn't that the thing we're really afraid of?
It's not to just die,
but to let go of the experience of love with other humans.
This is not everything.
Everything is everything.
So there's so much more.
And you could be lots of other things.
You could identify with lots of other things.
You could be identifying with being Gaia,
some kind of planetary control agent
that emerges over all the activity of life on Earth.
You could be identifying with some hyper Gaia
that is the concatenation of Gaia
or the digital life and digital minds.
And so in this sense,
there will be agents in all sorts of substrates
and directions that all have their own goals.
And when they're not sustainable,
then these agents will cease to exist.
Or when the agent feels that it's done with its own mission,
it will cease to exist.
In the same way as when you conclude a thought,
the thought is going to wrap up
and gives control over to other thoughts in your own mind.
So there is no single thing that you need to do.
What I observe myself as being
is that sometimes I'm a parent
and then I have identification and a job as a parent.
And sometimes I am an agent of consciousness on Earth.
And then from this perspective,
there's other stuff that is important.
So this is my main issue with Eliezer's perspective.
That he's basically marrying himself
to a very narrow human aesthetic.
And that narrow human aesthetic is a temporary thing.
Humanity is a temporary species
like most of the species on this planet
are only around for a while
and then they get replaced by other species.
In a similar way as our own physical organism
is around here for a while
and then gets replaced by a next generation of human beings
that are adapted to changing life circumstances
and average via mutation and selection.
And it's only when we have AI
and become completely software
that we become infinitely adaptable
and we don't have this generational
and species change anymore.
So if you take this larger perspective
and you realize it's really not about us,
it's not about Eliezer or humanity,
but it's about life on Earth
or it's about defeating entropy for as long as we can
while being as interesting as we can.
Then the perspective changes dramatically
and preventing AI from this perspective
looks like a very big sin.
But when we look at the set of trajectories
that such an AI would take as supersedes humans,
I think Eliezer is worried about
ones that not just kill all humans
but also have some kind of
maybe objectively undesirable consequence
for life on Earth.
Like how many trajectories when you look
at the big picture of life on Earth
would you be happy with
and how much were you with AGI,
whether it kills humans or not?
There is no single answer to this.
It's really a question that depends on the perspective
that I'm taking at a given moment.
And so there are perspectives that are
determining most of my life as a human being
and the other perspective where I zoom out further
and imagine that when the great oxygenation event happened,
that is photosynthesis was invented
and plants emerged and displaced a lot of the fungi
and algae in favor of plant life
and then later made animals possible.
Imagine that the fungi would have gotten together
and said, oh my God, this photosynthesis stuff
is really, really bad.
It's going to possibly displace and kill all the fungi.
We should slow it down and regulate it
and make sure that it doesn't happen.
This doesn't look good to me.
Perspective.
That said, you tweeted about a cliff.
Beautifully written.
As a sentient species, humanity is a beautiful child,
joyful, exploitive, wild, sad, and desperate,
but humanity has no concept of submitting to reason
and duty to life and future survival.
We will run until we step past the cliff.
So first of all, do you think that's true?
Yeah, I think that's pretty much the story
of the Club of Rome, the limits to growth
and the cliff that we are stepping over
with at least one foot is the delayed feedback.
Basically we do things that have consequences
that can be felt generations later
and the severity increases
even after we stopped doing the thing.
So I suspect that for the climate,
that the original predictions
that climate scientists made were correct.
So when they said that the tipping points
were in the late eighties,
they were probably in the late eighties.
And if we would stop emission right now,
we would not turn it back.
Maybe there are ways for carbon capture,
but so far there is no sustainable carbon capture technology
that we can deploy.
Maybe there's a way to put aerosols
in the atmosphere to cool it down.
It's possibilities, right?
But right now, per default,
it seems that we will step into a situation
where we feel that we've run too far.
And going back is not something
that we can do smoothly and gradually,
but it's going to lead to a catastrophic event.
Catastrophic event of what kind?
So can you see in the case
that we will continue dancing along
and always stop just short of the edge of the cliff?
I think it's possible, but it doesn't seem to be likely.
So I think this model that is being apparent
in the simulation that they're making
of climate pollution economies and so on
is that many effects are only visible
with a significant delay.
And in that time, the system is moving much more
out of the equilibrium state
or of the state where homeostasis is still possible
and instead moves into a different state,
one that is going to harbor fewer people.
And that is basically the concern there.
And again, it's a possibility.
It's just, and it's a possibility that is larger
than the possibility that it's not happening,
that we will be safe,
that we will be able to dance back all the time.
So the climate is one thing,
but there's a lot of other threats
that might have a faster feedback mechanism, less delay.
There is also this thing that AI is probably going to happen
and it's going to make everything uncertain again
because it is going to affect so many variables
that it's very hard for us to make a projection
into the future anymore.
And maybe that's a good thing.
It does not give us the freedom, I think,
to say now we don't need to care about anything anymore
because AI will either kill us or save us.
But I suspect that if humanity continues,
it will be due to AI.
What's the timeline for things to get real weird with AI?
And it can get weird in interesting ways
before you get to AGI.
What about AI girlfriends and boyfriends?
Fundamentally transforming human relationships?
I think human relationships
are already fundamentally transformed
and it's already very weird.
By which technology?
For instance, social media.
Yeah.
Is it, though, isn't the fundamentals of the core group
of humans that affect your life still the same?
Your loved ones, family?
No, I think that, for instance,
many people live in intentional communities right now.
They're moving around until they find people
that they can relate to and they become their family.
And often that doesn't work because it turns out
that instead of having grown networks
where you get around with the people that you grew up with,
you have more transactional relationships.
You shop around, you have markets for attention
and pleasure and relationships.
That kills the magic somehow.
Why is that?
Why is the transactional search
for optimizing allocation of attention
somehow misses the romantic magic
of what human relations are?
The question, how magical was it before?
Was it that you just could rely on instincts
that used your intuitions
and you didn't need to rationally reflect?
But once you understand, it's no longer magical
because you actually understand why you were attracted
to this person at this age
and not to that person at this age
and what the actual considerations were
that went on in your mind and what the calculations were,
what's the likelihood that you're going to have
a sustainable relationship with this person,
that this person is not going to leave you
for somebody else.
How are your life trajectories are going to evolve
and so on?
And when you're young,
you're unable to explicate all this
and you have to rely on intuitions and instincts
that in part you were born with
and also in the wisdom of your environment
that is going to give you some kind of reflection
on your choices.
And many of these things are disappearing now
because we feel that our parents might have no idea
about how we are living and the environments
that we grew up in, the cultures that we grew up in,
the milieus that our parents existed in
might have no ability to teach us
how to deal with this new world.
And for many people, that's actually true,
but it doesn't mean that within one generation
we build something that is more magical
and more sustainable and more beautiful.
Instead, we often end up as an attempt
to produce something that looks beautiful.
Like I was very weirded out by the aesthetics
of the Vision Pro headset by Apple
and not so much because I don't like the technology.
I'm very curious about what it's going to be like
and don't have an opinion yet.
But the aesthetics of the presentation and so on
were so uncanny valle-esque to me.
The characters being extremely plastic,
living in some hypothetical mid-century furniture museum.
This is the proliferation of marketing teams.
Yes, but it was a CGI-generated world.
It was a CGI-generated world that doesn't exist.
And when I complained about this,
some friends came back to me and said,
but these are startup founders.
This is what they live like in Silicon Valley.
And I tried to tell them, no,
I know lots of people in Silicon Valley.
This is not what people are like.
They're still people.
They're still human beings.
So the grounding in physical reality somehow
is important too.
And so basically what's absent in this thing is culture.
There is a simulation of culture,
an attempt to replace culture by catalog,
by some kind of aesthetic optimization
that is not the result of having a sustainable life,
a sustainable human relationships
with houses that work for you
and a mode of living that works for you
in which these glasses fit in naturally.
And I guess that's also why so many people
are weirded out about the product
because they don't know how is this actually
going to fit into my life and into my human relationships?
Because the way in which it was presented
in these videos didn't seem to be credible.
Do you think AI, when it's deployed by companies
like Microsoft and Google and Meta
will have the same issue of being weirdly corporate?
Like there'd be some uncanny valley,
some weirdness to the whole presentation.
So this is, I've gotten a chance to talk to George Haas.
He believes everything should be open source
and decentralized and there,
then we shall have the AI of the people
and it'll maintain a grounding to the magic
that's humanity, that's the human condition,
that like corporations will destroy the magic.
I believe that if we make everything open source
and make this mandatory,
we are going to lose about a lot of beautiful art
and a lot of beautiful designs.
There is a reason why the Linux desktop is still ugly.
And it's difficult to create coherence
in open source designs so far
when the designs have to get very large.
And it's easier to make this happening
in a company with centralized organization.
And from my own perspective,
what we should ensure is that open source never dies,
that it can always compete and has a place
with the other forms of organization
because I think it is absolutely vital
that open source exists and that we have systems
that people have under control outside of the corporation.
And that is also producing viable competition
to the corporations.
So the corporations, the centralized control,
the dictatorships of corporations can create beauty.
As a centralized design is a source of a lot of beauty.
And then I guess open source is a source of freedom,
a hedge against the corrupting nature of power
that comes with centralized.
I grew up in socialism and I learned
that corporations are totally evil
and I found this very, very convincing.
And then you look at corporations like Enron
and Halliburton maybe and realize, yeah, they are evil.
But you also notice that many other corporations
are not evil.
They're surprisingly benevolent.
Why are they so benevolent?
Is this because everybody is fighting them all the time?
I don't think that's the only explanation.
It's because they're actually animals
that live in a large ecosystem
and that are still largely controlled by people
that want that ecosystem to flourish
and be viable for people.
So I think that Pat Gelsinger is completely sincere
when he leads Intel to be a tool
that supplies the free world with semiconductors.
And it's not necessarily that all the semiconductors
are coming from Intel.
Intel needs to be there to make sure
that we always have them.
So there can be many ways in which we can import
and trade semiconductors from other companies in place.
We just need to make sure that nobody can cut us off from it
because that would be a disaster
for this kind of society and world.
So there are many things that need to be done
to make our style of life possible.
And then is this, I don't mean just capitalism,
environmental destruction, consumerism,
and creature comforts.
I mean an idea of life in which we are determined
not by some kind of king or dictator,
but in which individuals can determine themselves
to the largest possible degree.
And to me, this is something that this Western world
is still trying to embody.
And it's a very valuable idea
that we shouldn't give up too early.
And from this perspective,
the US is a system of interleaving clubs.
And an entrepreneur is a special club founder.
Somebody who makes a club that is producing things
that are economically viable.
And to do this, it requires a lot of people
who are dedicating a significant part of their life
for working for this particular kind of club.
And the entrepreneur is picking the initial set of rules
and the mission and vision and aesthetics for the club
and make sure that it works.
But the people that are in there need to be protected.
If they sacrifice part of their life,
there need to be rules that tell
how they're being taken care of
even after they leave the club and so on.
So there's a large body of rules
that have been created by our rule-giving clubs
and that are enforced by our enforcement clubs and so on.
And some of these clubs have to be monopolies
for game theoretic reasons,
which also makes them more open to corruption
and less harder to update.
And this is an ongoing discussion
and process that takes place.
But the beauty of this idea
that there is no centralized king
that is extracting from the peasants
and breeding the peasants into serving the king
and fulfilling all the rules like ants and an ant hill,
but that there is a freedom of association
and corporations are one of them,
is something that took me some time to realize.
So I do think that corporations are dangerous, right?
They need to be protections against overreach
of corporations that can do regulatory capture
and prevent open source from competing with corporations
by imposing rules that make it impossible
for a small group of kids to come together
to build their own language model
because OpenAI has convinced the US
that you need to have some kind of FDA process
that you need to go through that costs many million dollars
before you are able to train a language model, right?
So this is important to make sure that this doesn't happen.
So I think that OpenAI and Google are good things.
If these good things are kept in check in such a way
that all the other clubs can still be founded
and all the other forms of clubs that are desirable
can still co-exist with them.
So what do you think about Meta in contrast to that
open sourcing most of its language models
and most of the AI models that's working on
and actually suggesting that they will continue to do so
in the future for future versions of Llama, for example,
their large language model.
What, is that exciting to you?
Is that concerning?
I don't find it very concerning,
but that's also because I think that the language models
are not very dangerous yet.
Yet.
Yes.
So as I said, I have no proof that there is the boundary
between the language models and AI, AGI.
It's possible that somebody builds a version of baby AGI,
I think, and so then algorithmic improvements
that scale these systems up in ways
that otherwise wouldn't have happened
without these language model components.
So it's not really clear for me what the end game is there
and if these models can brute force their way into AGI.
And there's also a possibility that the AGI
that we are building with these language models
are not taking responsibility for what they are
because they don't understand the greater game.
And so to me, it would be interesting to try to understand
how to build systems that understand
what the greater games are.
What are the longest games that we can play on this planet?
Games broadly, like deeply define the way you did
with the games.
In the games theoretic sense.
So when we are interacting with each other,
in some sense, we are playing games.
We are making lots and lots of interactions.
This doesn't mean that these interactions
have all to be transactional.
Every one of us is playing some kind of game
by virtue of identifying this particular kinds of goals
that we have or aesthetics from which we derive the goals.
So when you say, I'm Lex Friedman,
I'm doing a set of podcasts,
then you feel that it's part of something larger
that you want to build.
Maybe you want to inspire people.
Maybe you want them to see more possibilities
and get them together over shared ideas.
Maybe your game is that you want to become super rich
and famous by being the best postcard caster on earth.
Maybe you have other games,
maybe it switches from time to time, right?
But there is a certain perspective
where you might be thinking,
what is the longest possible game that you could be playing?
A short game is for instance,
cancer is playing a shorter game than your organism.
It's cancer is an organism playing a shorter game
than the regular organism.
And because the cancer cannot procreate beyond the organism,
except for some infectious cancers,
like the ones that eradicated the Tasmanian devils,
you typically end up with the situation
where the organism dies together with the cancer,
because the cancer has destroyed the larger system
due to playing a shorter game.
And so ideally you want to, I think,
build agents that play the longest possible games.
And the longest possible games is to keep entropy at bay
as long as possible while doing interesting stuff.
But the longest, yes, that part,
the longest possible game while doing interesting stuff
and while maintaining at least
the same amount of interesting,
so complexities, so propagating.
Currently I'm pretty much identified as a conscious being.
That's the minimal identification
that I managed to get together,
because if I turn this off, I fall asleep.
And when I'm asleep, I'm a vegetable.
I'm no longer here as an agent.
So my agency is basically predicated on being conscious.
And what I care about is other conscious agents.
They're the only moral agents for me.
And so if an AI were to treat me as a moral agent,
that it is interested in coexisting with
and cooperating with and mutually supporting each other,
maybe it is, I think, necessary that the AI
thinks that consciousness is a viable mode of existence
and important.
So I think it would be very important to build conscious AI
and do this as the primary goal.
So not just say we want to build a useful tool
that we can use for all sorts of things.
And then you have to make sure that the impact
on the labor market is something
that is not too disruptive and manageable
and the impact on the copyright holder is manageable
and not too disruptive and so on.
I don't think that's the most important game to be played.
I think that we will see extremely large disruptions
of the status quo that are quite unpredictable
at this point.
And I just personally want to make sure
that some of the stuff on the other side
is interesting and conscious.
How do we ride as individuals and as a society
this wave, disruptive wave that changes
the nature of the game?
I absolutely don't know.
So everybody is going to do their best, as always.
Do we build the bunker in the woods?
Do we meditate more?
Drugs, mushrooms, psychedelics?
I mean, what, lots of sex?
What are we talking about here?
Do you have played Diablo 4?
I'm hoping that will help me escape for a brief moment.
What, play video games?
What, do you have ideas?
I really liked playing Disco Elysium.
It was one of the most beautiful computer games
I played in recent years.
And it's a noir novel that is a philosophical perspective
on Western society from the perspective of an Estonian.
And he first of all wrote a book about this world
that is a parallel universe that is quite poetic
and fascinating and is condensing his perspective
on our societies.
It was very, very nice.
He spent a lot of time writing it.
He had, I think, sold a couple thousand books
and as a result became an alcoholic.
And then he had the idea or one of his friends
had the idea of turning this into an RPG.
And it's mind blowing.
They spent, the illustrator more than a year
just on making the art for the scenes in between.
So aesthetically, it captures the posey one.
But it's a philosophical block of art.
It's a reflection of society.
It's fascinating to spend time in this world.
And so for me, it was using a medium in a new way
and telling a story that left me enriched.
But when I tried Diablo, I didn't feel enriched playing it.
I felt that the time playing it was not unpleasant,
but there's also more pleasant stuff
that I can do in that time.
So ultimately I feel that I'm being gamed, I'm not gaming.
Oh, the addiction thing.
Yes, I basically feel that there is a very transparent
economy that's going on.
The story of Diablo is branded.
So it's not really interesting to me.
My heart is slowly breaking
by the deep truth you're conveying to me.
Why can't you just allow me to enjoy
my personal addiction? Go ahead, by all means.
Go nuts.
I have no objection here.
I'm just trying to describe what's happening.
And it's not that I don't do things that I later say,
oh, I wish I would have done something different.
I also know that when we die,
the greatest regret that people typically have
on their deathbed is to say,
oh, I wish I had spent more time on Twitter.
No, I don't think that's the case.
I think I should probably have spent less time on Twitter,
but I found it so useful for myself
and also so addictive that I felt I need to make the best
of it and turn it into an art form and thought form.
And it did help me to develop something.
But I wish what other things I could have done
in the meantime, it's just not the universe
that we are in anymore.
Most people don't read books anymore.
What do you think that means
that we don't read books anymore?
What do you think that means
about the collective intelligence of our species?
Is it possible it's still progressing and growing?
Well, it clearly is.
There is stuff happening on Twitter
that was impossible with books.
And I really regret that Twitter has not taken the turn
that I was hoping for.
I thought Elon is global brain-pilled
and understands that this thing needs to self-organize
and he needs to develop tools to allow the profligation
of the self-organization so Twitter can become sentient.
And maybe this was a pipe dream from the beginning,
but I felt that the enormous pressure that he was under
made it impossible for him to work
on any kind of content goals.
And also many of the decisions that he made
under this pressure seem to be not very wise.
I don't think that as a CEO of a social media company,
you should have opinions in the culture of our in public.
I think that's very short-sighted.
And I also suspect that it's not a good idea
to block Paul Graham of all people
over setting a Mastodon link.
And I think Paul made this intentionally
because he wanted to show Elon Musk that blocking people
for setting a link is completely counter
to any idea of free speech
that he intended to bring to Twitter.
And basically seeing that Elon was very less principled
in his thinking there and is much more experimental.
And many of the things that he is trying,
they pan out very differently in a digital society
than they pan out in a car company.
Because the effect is very different
because everything that you do in a digital society
is going to have real world cultural effects.
And so basically I find it quite regrettable
that this guy is able to become de facto the Pope
but Twitter has more active members
than the Catholic church.
And he doesn't get it.
The power and responsibility that he has
and the ability to create something
in this society that is lasting
and that is producing a digital Agora in a way
that has never existed before,
where we build a social network on top of a social network
an actual society on top of the algorithms.
So this is something that is hope still in the future
and still in the cards,
but it's something that exists in small parts.
I find that the corner of Twitter that I'm in
is extremely pleasant.
Just when I take a few steps outside of it,
it is not very wholesome anymore.
And the way in which people interact with strangers
suggest that it's not a civilized society yet.
So as the number of people who follow you on Twitter expands
you feel the burden of the uglier sides of humanity.
Yes, but there's also a similar thing in the normal world.
That is if you become more influential,
if you have more status,
if you have more fame in the real world,
you get lots of perks,
but you also have way less freedom
in the way in which you interact with people,
especially the strangers,
because a certain percentage of people,
it's a small single digit percentage is nuts and dangerous.
And the more of those are looking at you,
the more of them might get ideas.
But what if the technology enables you
to discover the majority of people
to discover and connect efficiently and regularly
with the majority of people who are actually really good?
I mean, one of my sort of concerns
with a platform like Twitter is
there's a lot of really smart people out there,
a lot of smart people that disagree with me
and with others between each other.
And I love that if the technology
would bring those to the top,
the beautiful disagreements,
like a intelligence squared type of debates.
There's a bunch of,
I mean, one of my favorite things to listen to is arguments
and arguments like high effort arguments,
with respect and love underneath it,
but then it gets a little too heated,
but that kind of too heated,
which I've seen you participate in.
And I love that with Lee Cronin,
with those kinds of folks.
And you go pretty hard.
Like you get frustrated, but it's all beautiful.
Obviously I can do this because we know each other.
And Lee has the rare gift
of being willing to be wrong in public.
So basically he has thoughts that are as wrong
as the random thoughts of an average,
highly intelligent person,
but he blurts them out
while not being sure if they're right.
And he enjoys doing that.
And once you understand that this is his game,
you don't get offended by him saying something
that you think is so wrong.
But he's constantly passively communicating a respect
for the people he's talking with
and for just basic humanity and truth
and all that kind of stuff.
And there's a self-deprecating thing.
There's a bunch of like social skills you acquire
that allow you to be a great debater,
a great argumenter,
like be wrong in public and explore ideas together
in public when you disagree.
And I would love for Twitter to elevate those folks,
elevate those kinds of conversations.
It already does in some sense,
but also if it elevates them too much,
then you get this phenomenon on Clubhouse
where you always get dragged on stage.
And I found this very stressful because it was too intense.
I don't like to be dragged on stage all the time.
I think once a week is enough.
And also when I met Lee the first time,
I found that a lot of people seem to be shocked
by the fact that he was being very aggressive
as their results,
that he didn't seem to show a lot of sensibility
in the way in which he was criticizing what they were doing
and being dismissive of the work of others.
And that was not, I think, in any way a shortcoming of him
because I noticed that he was much, much more dismissive
with respect to his own work.
It was his general stance.
And I felt that this general stance
is creating a lot of liability for him
because really a lot of people take offense
at him being not like a Dale Carnegie character
who is always smooth and make sure that everybody likes him.
So I really respect that he is willing to take that risk
and to be wrong in public and to offend people.
And he doesn't do this in any bad way.
It's just most people feel,
or not all people recognize this.
And so I can be much more aggressive with him
than I can be with many other people
who don't play the same game
because he understands the way and the spirit
in which I respond to him.
I think that's a fun and that's a beautiful game.
It's ultimately a productive one.
Speaking of taking that risk, you tweeted,
"'When you have the choice between being a creator,
"'consumer, or redistributor, always go for creation.
"'Now not only does it lead to a more beautiful world,
"'but also to a much more satisfying life for yourself.
"'And don't get stuck preparing yourself for the journey.
"'The time is always now.'"
So let me ask for advice.
What advice would you give on how to become such a creator
on Twitter in your own life?
I was very lucky to be alive at the time
of the collapse of Eastern Germany
and the transition into Western Germany.
And me and my friends and most of the people I knew
were East Germans and we were very poor
because we didn't have money.
And all the capital was in Western Germany
and they bought our factories and shut them down
because they were mostly only interested in the market
rather than creating new production capacity.
And so cities were poor and in disrepair
and we could not afford things.
And I could not afford to go into a restaurant
and order a meal there.
I would have to go get home.
But I also thought,
why not just have a restaurant with my friends?
So we would open up a cafe with friends in a restaurant
and we would cook for each other in these restaurants
and also invite the general public and they could donate.
And eventually this became so big
that we could turn this into some incorporated form
and it became a regular restaurant at some point.
Or we did the same thing with a movie theater.
We would not be able to afford to pay 12 marks
to watch a movie.
Why not just create our own movie theater
and then invite people to pay?
And we would rent the movies
in a way in which a movie theater does.
But it would be a community movie theater
in which everybody who wants to help can watch for free
and builds the thing and renovates the building.
And so we ended up creating lots and lots of infrastructure.
And I think when you are young and you don't have money,
move to a place where this is still happening.
Move to one of those places that are undeveloped
and where you get a critical mass of other people
who are starting to build infrastructure to live in.
And that's super satisfying
because you're not just creating infrastructure
but you're creating a small society
that is building culture
and ways to interact with each other.
And that's much, much more satisfying
than going into some kind of chain
and get your needs met by ordering food
from this chain and so on.
So not just consuming culture, but creating culture.
And you don't always have that choice.
That's why I prefaced it, when you do have the choice.
And there are many roles that need to be played.
We need people who take care of redistribution
in society and so on.
But when you have the choice to create something,
always go for creation.
It's so much more satisfying.
And it also is, this is what life is about, I think.
Yeah.
Speaking of which, you retweeted this meme
of a life of a philosopher in a nutshell.
It's birth and death and in between.
It's a chubby guy and it says, why though?
What do you think is the answer to that?
Well, the answer is that everything that can exist
might exist.
And in many ways, you take an ecological perspective.
The same way as when you look at human opinions
and cultures, it's not that there is right
and wrong opinions when you look at this
from this ecological perspective.
But every opinion that fits between two human ears
might be between two human ears.
And so when I see a strange opinion on social media,
it's not that I feel that I have a need to get upset.
It's often more that I, oh, there you are.
And when the opinion is incentivized,
then it's going to be abundant.
And when you take this ecological perspective
also on yourself and you realize you're just one
of these mushrooms that are popping up
and doing their thing, and you can,
depending on where you chose to grow
and where you happen to grow,
you can flourish or not doing this or that strategy.
And it's still all the same life at some level.
It's all the same experience of being a conscious being
in the world.
And you do have some choice about who you want to be
more than any other animal has.
That to me is fascinating.
And so I think that rather than asking yourself,
what is the one way to be,
think about what are the possibilities that I have,
what would be the most interesting way to be that I can be?
Because everything is possible.
So you get to explore this.
Not everything is possible.
Many things fail.
Most things fail.
But often there are possibilities that we are not seeing,
especially if we choose who we are.
To the degree we can choose.
Yasha, you're one of my favorite humans in this world.
Consciousness is to merge with for a brief moment of time.
It's always an honor.
It always blows my mind.
It will take me days, if not weeks, to recover.
And I already miss our chats.
Thank you so much.
Thank you so much for speaking with me so many times.
Thank you so much for all the ideas
you put out into the world.
And I'm a huge fan of following you now
in this interesting, weird time we're going through with AI.
So thank you again for talking today.
Thank you, Alex, for this conversation.
I enjoyed it very much.
Thanks for listening to this conversation with Yoshua Bak.
To support this podcast,
please check out our sponsors in the description.
And now, let me leave you with some words
from the psychologist Carl Jung.
One does not become enlightened
by imagining figures of light,
but by making the darkness conscious.
The latter procedure, however,
is disagreeable and therefore not popular.
Thank you for listening.
And hope to see you next time.