The following is a conversation with Mark Zuckerberg
inside the metaverse.
Mark and I are hundreds of miles apart from each other
in physical space, but it feels like we're in the same room
because we're appeared to each other
as photorealistic Kodak avatars in 3D with spatial audio.
This technology is incredible and I think it's the future
of how human beings connect to each other
in a deeply meaningful way on the internet.
These avatars can capture many of the nuances
of facial expressions that we use,
we humans use to communicate emotion to each other.
Now I just need to work on upgrading my emotions,
expressing capabilities of the underlying human.
This is the Lex Friedman podcast.
And now dear friends, here's Mark Zuckerberg.
This is a great lighting change.
Wow, yeah, we can put the light anywhere.
And it doesn't feel awkward to be really close to you.
No, it does.
I actually moved you.
I moved you back a few feet before you got into the headset.
You were like right here.
I don't know if people can see this,
but this is incredible.
The realism here is just incredible.
Where am I?
Where are you, Mark?
Where are we?
You're in Austin, right?
No, I mean this place, we're shrouded by darkness
with ultra realistic face.
And it just feels like we're in the same room.
This is really the most incredible thing I've ever seen.
And sorry to be in your personal space.
We have done jiu-jitsu before.
Yeah, no, I was commenting to the team before
that I feel like we've choked each other
from further distances than it feels like we are right now.
I mean, this is just really incredible.
I don't know how to describe it with words.
It really feels like we're in the same room.
It feels like the future.
This is truly, truly incredible.
I just wanted to take it in.
I'm still getting used to it.
It's you, it's really you.
But you're not here with me, right?
You're there wearing a headset and I'm wearing a headset.
It's really, really incredible.
So can you describe what it takes currently
for us to appear so photorealistic to each other?
Yeah, so I mean, for background,
we both did these scans for this research project
that we have at Meta called Kodak Avatars.
And the idea is that instead of actually,
instead of our avatars being cartoony
and instead of actually transmitting a video,
what it does is we've sort of scanned ourselves
and a lot of different expressions
and we've built a computer model of sort of each
of our faces and bodies and the different expressions
that we make and collapse that into a Kodak
that then when you have the headset on your head,
it can, it sees your face, it sees your expression
and it can basically send an encoded version
of what you're supposed to look like over the wire.
So in addition to being photorealistic,
it's also actually much more bandwidth efficient
than transmitting a full video
or especially a 3D immersive video
of a whole scene like this.
And it captures everything, like the flaws,
like to me, the subtleties of the human face,
like even the flaws, that's like, that's all amazing.
It makes you, it makes it so much more immersive.
It makes you realize that like perfection isn't the thing
that leads to immersion.
It's like the little subtle flaws,
like freckles and like variations in color and just-
Yeah, wrinkles.
All stuff about noses. Asymmetry.
Yeah, asymmetry and just the different,
like the corners of the eyes,
like what your eyes do when you smile,
all that kind of stuff.
Yeah, eyes are a huge part of it.
Yeah, I mean, there's all the studies
that most of communication, even when people are speaking,
is not actually the words that they're saying, right?
It's kind of the expression and all that.
So, and we try to capture that with the kind of classical
expressive avatar system that we have.
That's the kind of more cartoon designed one.
You can kind of put those kinds of expressions
on those faces as well.
But there's obviously a certain realism
that comes with delivering
kind of this photorealistic experience that,
I don't know, I just think it's really magical.
I mean, this gets to kind of the core of what the vision
around virtual and augmented reality is
of like delivering a sense of presence
as if you're there together,
no matter where you actually are in the world.
And I mean, this experience,
I think is a good embodiment of that,
where it's like, I mean,
we're in two completely different states
halfway across the country.
And it just like, you know,
it looks like you're just sitting right in front of me.
It's pretty wild.
Yeah, I can't, I'm almost getting emotional.
It's like, it feels like a totally,
it's a fundamentally new experience.
Like for me to have this kind of conversations
with loved ones, it would just change everything.
Maybe just to elaborate,
so I went to Pittsburgh
and went through the whole scanning procedure,
which has so much incredible technology,
so software and hardware going on,
but it is a lengthy process.
So what's your vision for the future of this
in terms of making this more accessible to people?
You know, it starts off with a small number of people
doing these very detailed scans, right?
Which is, that's the version that you did
and that I did.
And, you know, before there were a lot of people
who have done this kind of a scan for,
we probably need to kind of over collect expressions
when we're doing the scanning,
because we haven't figured out
how much we can reduce that down
to a really streamlined process
and extrapolate from the scans that have already been done.
But the goal,
and we have a project that's working on this already,
is just to do a very quick scan with your cell phone
where you just take your phone,
kind of wave it in front of your face
for a couple of minutes,
you know, say a few sentences,
make a bunch of expressions,
but overall have the whole process
just be two to three minutes,
and then produce something that's of the quality
of what we have right now.
So I think that that's one of the big challenges
that remains.
And right now we have the ability to do the scans
if you have hours to sit for one.
And with today's technology,
I mean, you're using a meta headset that exists.
It's a product that's kind of for sale now.
You can drive these with that.
But the production of these scans in a very efficient way
is one of the last pieces that we still need to really nail.
And then obviously there's all the experiences around it.
I mean, right now we're kind of sitting in a dark room,
which is familiar for your podcast.
But I think part of the vision for this over time
is not just having this be like a video call.
I mean, that's fine.
It's cool, or it feels like it's immersive,
but you can do a video call on your phone.
The thing that you can do in the metaverse
that is different from what you can do on a phone
is doing stuff where you're physically there together
and participating in things together.
And we could play games like this.
We could have meetings like this in the future.
Once you get mixed reality and augmented reality,
we could have codec avatars like this
and go into a meeting and have some people physically there
and have some people show up in this photorealistic form
superimposed on the physical environment.
Stuff like that is going to be super powerful.
So we got to still build out all those kind of applications
and the use cases around it.
But I don't know, I think it's going to be a pretty wild
next few years around this.
I mean, I'm actually almost at a loss of words.
This is just so incredible.
This is truly incredible.
I hope that people like watching this can get a glimpse
of like how incredible it is.
It really feels like we're in the same room.
Like there is that, I guess there's an uncanny valley
that seems to have been crossed here.
It looks like you.
Yeah, I mean, I think there's still a bunch of tuning
that I think we'll want to do where different people
emote to different extents, right?
So I think one of the big questions is, you know,
like when you smile, how wide is your smile
and how wide do you want your smile to be?
And I think getting that to be tuned on a per person basis
is going to be one of the things
that we're going to need to figure out.
You know, it's like, to what extent do you want
to give people control over that?
Some people might try to, might prefer a version
of themselves that's more emotive in their avatar
than their actual faces.
You know, so for example, I always get a lot of critique
and shit for having like a relatively stiff expression.
But you know, I mean, I might feel pretty happy,
but just make a pretty small smile.
I mean, maybe, you know, for me, it's actually,
you know, it's like, I'd want to have my avatar
really be able to better express like how I'm feeling
than how I can do physically.
So I think that there's a question
about how you want to tune that.
But overall, yeah, I mean, we want to start
from the baseline of capturing how people actually emote
and express themselves.
And I mean, I think the initial version of this
has been pretty impressive.
And like you said, I do think we're kind of beyond
the uncanny valley here where it does feel
like you, it doesn't feel weird or anything like that.
I mean, that's going to be the meme
that the two most monotone people
are in a metaverse together.
But I think that actually makes it more difficult.
Like the amazing thing here is that the subtleties
of the expression of the eyes, you know,
people say I'm monotone and emotionless, but I'm not.
It's just this, maybe my expression of emotion
is more subtle usually like with the eyes.
And that's one of the things I've noticed
is just how expressive the subtle movement
of the corners of the eyes are
in terms of displaying happiness or boredom
or all that kind of stuff.
I am curious to see,
just because I've never done one of these before.
I've never done a podcast as one of these codec avatars.
And I'm curious to see what people think of it
because, you know, one of the issues that we've had
in some of the VR and mixed reality work
is it tends to feel a lot more profound when you're in it
than the 2D videos capturing the experience.
So I think that this one, because it's photorealistic,
may look kind of as amazing in 2D for people watching it
as it feels, I think, to be in it.
But we've certainly had this issue
where a lot of the other things just,
it's like you feel the sense of immersion when you're in it
that doesn't quite translate to a 2D screen.
But I don't know, I'm curious to see what people think.
Yeah, I'm curious to see if people could see that,
like my heart is actually beating fast now.
This is super interesting,
like that such intimacy of conversation
can be achieved remotely.
There's been, you know,
I don't do remote podcasts for this reason.
And this is like, breaks all of that.
This feels like just an incredible transition
to something else, to the different kind of communication.
Breaks all barriers, like geographic physical barriers.
You mentioned, do you have a sense of timeline
in terms of how many difficult things have to be solved
to make this more accessible,
to like scanning with a smartphone?
Yeah, I mean, I think we'll probably roll this out
progressively over time.
So it's not gonna be like we roll it out
and one day everyone has a codec avatar.
We wanna get more people scanned and into the system,
and then we wanna start integrating it
into each one of our apps, right?
Making it so that, you know,
I think that for a lot of the work style things,
productivity, I think that this is gonna make
a ton of sense.
In a lot of game environments, I mean, this could be fine,
but games tend to have their own style, right?
Where you almost wanna fit more
with the aesthetic style of the game.
But I think for doing meetings,
and one of the things that we get a lot of feedback
on work rooms where, you know,
people are pretty blown away by the experience
and this feeling that you can like be remote,
but feel like you're physically there
around a table with people.
But then, you know, we get some feedback
that people have a hard time with the fact
that the avatars are so expressive
and don't feel, you know, as realistic in that environment.
So I think something like this could make
a very big difference for those remote meetings.
And especially with Quest 3 coming out,
which is gonna be the first mainstream
mixed reality product, right?
Where you're really taking digital, you know,
expressions of either a person or objects
and overlaying them on the physical world.
I think the ability to do kind of remote meetings
and things like that,
where you're just remote hang sessions with friends.
I mean, I think that that's gonna be very exciting.
So yeah, rolling it out over the next few years.
It's not ready to be like a kind of mainstream product yet,
but we just wanna, we'll keep tuning it
and keep getting more scans in there
and keep, you know, and kind of rolling it out
into more of the features.
But yeah, I mean, definitely in the next few years,
you'll be seeing a bunch more experiences like this.
Yeah, I would love to see some celebrities scanned
and some non-celebrities.
I just, just more people to experience this.
I would love to see that.
This is something that, I mean, my mind is blown.
I'm literally at a loss of words
because it's very difficult to just convey
how incredible this is.
How like, how I feel the emotion, how I feel the presence,
how I feel like the subtleties of the emotion
in terms of like work meetings or any kind of,
in terms of podcasts, this is like, this is awesome.
And I don't even need your arms or legs.
Well, we gotta get that.
I mean, that's its own challenge.
And part of the question is also, so you have the scan,
then it takes a certain amount of compute to go drive that,
both for the sensors on the headset and then rendering it.
So one of the things that we're working through
is what is the level of fidelity that is optimal, right?
You could do the full body in kind of a codec
and that can be quite intensive.
But one of the things that we're thinking about
is like, all right, maybe you can kind of stitch
a somewhat lower fidelity version of your body,
but still have the main kind of, the major movements.
But your face is really the thing
that we have the most resolution on, right?
In terms of being able to read and express emotions.
I mean, like you said, if you move your eyebrows
like a millimeter, I mean, that really changes
the expression and what you're emoting.
Whereas, you know, I mean, moving your arm like an inch
probably doesn't matter quite as much.
So yeah, so I think that we'll,
we do want to get all of that into here
and that'll be some of the work over the next period as well.
So you mentioned Quest 3, that's coming out.
I've gotten a chance to try that too, that's awesome.
So how'd you pull off the mix?
So it's not just virtual reality, it's mixed reality.
Yeah, I mean, I think it's going to be,
it's going to be the first mainstream mixed reality device.
I mean, obviously we shipped Quest Pro last year,
but it was $1,500.
And part of what I'm super proud of is, you know,
we try to innovate not just on pushing the state of the art
and delivering new capabilities,
but making it so it can be available to everyone.
And, you know, we have this and it's coming out, it's $500.
And in some ways, I think the mixed reality
is actually better in Quest 3 than it was
than what we're using right now in Quest Pro.
So I'm really proud of the team
for being able to deliver that kind of an innovation
and get it out.
But, you know, some of this is just software
you tune over time and get to be better.
Part of it is you put together a product
and you figure out what are the bottlenecks
in terms of making it a good experience.
So we got the resolution for the mixed reality cameras
and sensors to be multiple times better in Quest 3.
And we just figured that that made a very big difference
when we saw the experience that we were able
to put together for Quest Pro.
And part of it is also that, you know,
Qualcomm just came out with their next generation chipset
for VR and MR, that we worked with them
on a kind of custom version of it.
But that was available this year for Quest 3,
and it wasn't available in Quest Pro.
So, you know, in a way, Quest 3,
even though it's not the Pro product,
actually has a stronger chipset in it
than the Pro line at a third of the cost.
So I'm really excited to get this in people's hands.
It does all the VR stuff that Quest 2
and the others have done too.
It does it better because the display is better
and the chip is better, so you'll get better graphics.
It's 40% thinner, so it's just more comfortable as well.
But the MR is really the big capability shift.
And part of what's exciting about the whole space right now
is, you know, this isn't like smartphones where,
you know, companies put out a new smartphone every year
and you can almost barely tell the difference
between that and the one the year before it.
Now for this, each time we put out a new headset,
it has like a major new capability.
And the big one now is mixed reality,
the ability to basically take digital representations
of people or objects and superimpose them on the world.
And basically, you know, I mean,
there's one version of this is you're gonna kind of have
these augments or holograms and experiences
that you can kind of bring into your living room
or a meeting space or an office.
Another thing that I just think is gonna be
a much kind of simpler innovation is that
there are a lot of VR experiences today
that don't need to be fully immersive.
And, you know, if you're playing a shooter game
or you're doing a fitness experience,
then sometimes people get worried
about swinging their arms around.
Like, am I gonna hit a lamp or something, you know?
And, you know, am I gonna run into something?
So having that in mixed reality actually is just
a lot more comfortable for people, right?
You kind of still get the immersion and the 3D experience,
and you can have an experience that just wouldn't be possible
in the physical world alone, but by being anchored to
and being able to see the physical world around you,
it's like, it just feels so much safer and more secure.
And I think a lot of people
are really gonna enjoy that too.
So yeah, I'm really excited to see how people use it,
but yeah, Quest 3 coming out later this fall.
Yeah, and I got to experience it
with other people sitting around,
and there's a lot of furniture.
And so you get to see that furniture
and get to see those people,
and you get to see those people
enjoy the ridiculousness of you swinging your arms.
I mean, presumably they're friends of yours,
even if they make fun of you.
There's a lot of love behind that,
and I got to experience that.
That's a really fundamentally different experience
than just pure VR with zombies coming out of walls.
Yeah, it's like someone shooting at you
and you hide behind your real couch
in order to duck the fire, yeah.
It's incredible how it's all integrated,
but also subtle stuff like in a room with no windows,
you can add windows to it,
and you can look outside as the zombies run towards you,
but it's still a nice view outside.
And so that's pulled off by having cameras
on the outside of the headset that do the pass-through.
That technology is incredible,
to do that on a small headset.
Yeah, and it's not just the cameras.
You basically need to,
you need multiple cameras to capture the different angles
and sort of the three-dimensional space,
and then it's a pretty complex compute problem
and AI problem to map that to your perspective,
because the cameras aren't exactly where your eyes are
because no two people's eyes
are gonna be in exactly the same place.
You kind of need to get that to line up,
and then do that basically in real time,
and then generate something that kind of feels natural,
and then superimpose whatever digital objects
you want to put there.
It's a very interesting technical challenge,
and I think we'll continue tuning this
for the years to come as well,
but I'm pretty excited to get this out,
because I think Quest 3 is gonna be the first device
like this that millions of people are gonna get.
That's mixed reality,
and it's only when you have millions of people
using something that you start getting
the whole developer community really starting
to experiment and build stuff,
because now there are gonna be people who actually use it.
So I think we'll get,
you know, we got some of that flywheel going with Quest Pro,
but I think it'll really get accelerated
once Quest 3 gets out there.
So yeah, I'm pretty excited about this one.
Plus there's hand tracking without,
so you don't need to have a control.
So the cameras aren't just doing the pass-through
of the entire physical reality around you.
It's also tracking the details of your hands
in order to use that for like gesture recognition,
this kind of stuff.
Yeah, we've been able to get way further
on hand recognition in a shorter period of time
than I expected.
So that's been pretty cool.
I don't know, did you see the demo experience
that we built around-
Piano?
Yeah, the piano, learning to play piano.
Yeah, it's incredible.
You're basically playing piano on a table,
and that's without any controller.
And like how well it matches physical reality
with no latency,
and it's tracking your hands with no latency,
and it's tracking all the people around you with no latency,
integrating physical reality and digital reality.
Obviously that connects exactly to this Kodak Avatar,
which is in parallel,
allows us to have ultra-realistic copies of ourselves
in this mixed reality.
It's, so like it's all converging
towards like an incredible digital experience
in the metaverse.
To me, obviously, I love the intimacy of conversation.
So even this is awesome.
But do you have other ideas of what this unlocks,
of like something like Kodak Avatar unlocks
in terms of applications,
in terms of things we're able to do?
Well, there's what you can do with avatars overall
in terms of superimposing digital objects
on the physical world.
And then there's kind of psychologically,
what does having photorealistic do?
So I think we're moving towards a world
where we're gonna have something
that looks like normal glasses,
where you can just see the physical world,
but you will see holograms.
And in that world, I think that they're gonna be
not too far off, maybe by the end of this decade.
We'll be living in a world
where there are kind of as many holograms
when you walk into a room as there are physical objects.
And it really raises this interesting question
about what are,
a lot of people have this phrase
where they call the physical world the real world.
And I kind of think increasingly,
the physical world is super important,
but I actually think the real world
is the combination of the physical world
and the digital worlds coming together.
But until this technology, they were sort of separate.
It's like you access the digital world through a screen.
And maybe it's a small screen that you carry around
or it's a bigger screen where you sit down at your desk
and strap in for a long session.
But they're kind of fundamentally divorced and disconnected.
And I think part of what this technology is gonna do
is bring those together into a single coherent experience
of what the modern real world is,
which is it's gotta be physical
because we're physical beings.
So the physical world is always gonna be super important.
But increasingly, I think a lot of the things
that we kind of think of can be digital holograms.
I mean, any screen that you have can be a hologram,
in any media, in any book, art.
It can basically be just as effective as a hologram
as a physical object, any game that you're playing,
a board game or any kind of physical game, cards,
ping pong, things like that.
They're often a lot better as holograms
because you could just kind of snap your fingers
and instantiate them and have them show up.
It's like you have a ping pong table show up
in your living room, but then you can snap your fingers
and have it be gone.
So that's super powerful.
So I think that it's actually an amazing thought experiment
of how many physical things we have today
that could actually be better as interactive holograms.
But then beyond that, I think the most important thing,
obviously, is people.
So the ability to have these mixed hangouts,
whether they're social or meetings
where you show up to a conference room,
you're wearing glasses or a headset in the very near term,
but hopefully by, for the next five years, glasses or so.
And you're there physically,
some people are there physically,
but other people are just there as holograms
and it feels like it's them who are right there.
And also, by the way, another thing
that I think is gonna be fascinating
about being able to blend together
the digital and physical worlds in this way
is we're also going to be able to embody
AIs as well.
So I think you'll also have meetings in the future
where you're basically,
maybe you're sitting there physically
and then you have a couple of other people
who are there as holograms
and then you have like Bob, the AI,
who's an engineer on your team who's helping with things
and he can now be embodied as a realistic avatar as well
and just join the meeting in that way.
So I think that that's gonna be pretty compelling as well.
So then, okay, so what can you do
with photorealistic avatars compared
to kind of the more expressive ones that we have today?
Well, I think a lot of this actually comes down
to acceptance of the technology.
And because all of the stuff that we're doing,
I mean, the motion of your eyebrows,
the motion of your eyes, the cheeks and all of that,
there's actually no reason why you couldn't do that
on an expressive avatar too.
I mean, it wouldn't look exactly like you,
but you can make a cartoon version of yourself
and still have it be almost as expressive.
But I do think that there's this bridge
between the current state of most of our interactions
in the physical world and where we're getting in the future
with this kind of hybrid physical and digital world
where I think it's gonna be a lot easier
for people to kind of take some of these experiences
seriously with the photorealistic avatars to start.
I'm actually really curious
to see where it goes longer term.
I could see a world where people stick to the photorealistic
and maybe they modify them to make them
a little bit more interesting,
but maybe fundamentally we like photorealistic things.
But I can also see a world that once people get used
to the photorealistic avatars
and they get used to these experiences,
that I actually think that there could be a world
where people actually prefer being able
to express themselves in ways
that aren't so tied to their physical reality.
And so that's one of the things
that I'm really curious about.
And I don't know,
in a bunch of our internal experiments on this,
one of the things that I thought
was psychologically pretty interesting
is people have no issues blending photorealistic stuff
and not, so we could have,
for this specific scene that we're in now,
we happen to sort of be in a dark room.
Part of that aesthetic decision, I think,
was based on the way you like to do your podcast,
but we've done experiences like this
where you have a cartoony background,
but photorealistic people who you're talking to,
and people just seem to just think
that that is completely normal, right?
It doesn't bother you.
It doesn't feel like it's weird.
Another thing that we've experienced with
is basically you have a photorealistic avatar
that you're talking to, and then right next to them,
you have an expressive kind of cartoon avatar.
And that actually is pretty normal too, right?
It's like, it's not that weird, right?
To basically being interacting with different people
in different modes like that.
So I'm not sure.
I think it'll be an interesting question.
To what extent these photorealistic avatars
are like a key part of just transitioning
from being comfortable in the physical world
to this kind of new modern real world
that kind of includes both the digital and physical,
or if this is like the long-term way that it stays.
That's a, I mean, I think that there are gonna be uses
for both the expressive and the photorealistic over time.
I just don't know what the balance is gonna be.
Yeah, it's a really good,
interesting philosophical question.
But to me in the short term, the photorealistic is amazing.
To what I would prefer, like you said the workroom,
but like on a beach with a beer,
just to see a buddy of mine remotely
on a chair next to me drinking a beer.
I mean, that as realistic as possible
is an incredible experience.
So I don't want any fake hats on him.
I don't want any just chilling with a friend,
drinking beer, looking at the ocean
while not being in the same place together.
I mean, that experience is just fundamentally,
it's just a high quality experience of friendship.
Whatever we seek in friendship,
it seems to be present there
in the same kind of realism I'm seeing right now.
This is totally a game changer.
So to me, I can see myself sticking with this
for a long time.
Yeah, and I mean, it's also, it's novel
and it's also a technological feat, right?
It's like being able to pull this off
is like, it's like a pretty impressive
and I think to some degree,
it's just this kind of like awesome experience.
Yeah, but I'm already, sorry to interrupt,
I'm already forgetting that you're not real.
Like this really, so it's novel.
This is just an avatar version of me.
That's a deep philosophical question, yes.
But I mean, but here's some of the,
so I put this on this morning
and I was like, all right,
it's like, okay, so my hair is a little shorter in this
than my physical hair is right now.
I probably need to go get a haircut.
And I actually, I did happen to shave this morning,
but if I hadn't, I could still have
this photorealistic avatar that is more cleanly shaven,
even if I'm a few days in.
Physically, so I do think that they're gonna start
to be these subtle questions that seep in
where the avatar is realistic in the sense of
this is kind of what you looked like at the time of capture,
but it's not necessarily temporally accurate
to exactly what you look like in this moment.
And I think that they're gonna end up being
a bunch of questions that come from that over time
that I think are gonna be fascinating too.
You mean just like the nature of identity of who we are?
Are we the people, you know how people do like,
like summer beach body where the people will be
for the scan, they'll try to lose some weight
and look their best and sexiest with the nice hair
and everything like that.
I mean, it does raise the question of,
if a lot of people are interacting
with the digital version of ourselves, who are we really?
Are we the entity driving the avatar or are we the avatar?
Well, I mean, I think our physical bodies
also fluctuate and change over time too.
So I think that there's a similar question of like,
which version of that are we?
Right, there's like the,
I mean, and it's interesting identity question
because all right, it's like, I don't know,
it's like weight fluctuates or things like that.
It's like, I think most people don't tend to think
of themselves as the, I don't know,
it's an interesting psychological question.
Some, maybe some people, maybe a lot of people
do think about themselves as the kind of worst version,
but I think a lot of people probably think
about themselves as the best version.
And then it's like what you are on a day-to-day basis
doesn't necessarily map to either of those.
So I think that that's, yeah,
there will definitely be a bunch of social scientists
and folks will have to, and psychologists are,
really, there's gonna be a lot to understand
about how our perception of ourselves and others
has shifted from this.
Well, this might be a bit of a complicated
and a dark question, but one of the first feelings I had
experiencing this is I would love to talk to loved ones.
And the next question I have is I would love to talk
to people who are no longer here that are loved ones.
So like, if you look into the future,
is that something you think about when people pass away,
but they can still exist in the metaverse?
You can still have, you know, talk to your father,
talk to your grandfather and grandmother and mother
once they pass away.
The power of that experience is one of the first things
my mind jumped to, because it's like, this is so real.
Yeah, I think that there are a lot of norms and things
that people have to figure out around that.
There's probably some balance where, you know,
if someone has lost a loved one and is grieving,
there may be ways in which, you know,
being able to interact or relive certain memories
could be helpful, but then there's also probably an extent
to which it could become unhealthy.
And I mean, I'm not an expert in that,
so I think we'd have to study that
and understand it in more detail.
We have, you know, a fair amount of experience
with how to handle death and identity
and people's digital content through social media already,
unfortunately, right?
Where there's, you know, unfortunately, you know,
people who use our services die every day
and their families often want to have access
to their profiles and we have whole protocols
that we go through where there are certain parts of it
that we try to memorialize,
so that way the family can get access to it,
so that way the account doesn't just go away immediately.
But then there are other things that are, you know,
important kind of private things that that person has,
like we're not gonna give the family access
to someone's messages, you know, for example.
So yeah, I think that there's some best practices,
I think, from the current digital world
that we'll carry over, but yeah,
I think that this will enable some different things.
Another version of this is how this intersects with AIs,
right, because, and one of the things
that we're really focused on is, you know,
we want the world to evolve in a way
where there isn't like a single AI super intelligence,
but where, you know, a lot of people are empowered
by having AI tools to do their jobs
and, you know, make their lives better.
And if you're a creator, right,
and if you run a podcast like you do,
then you have a big community of people
who are super interested to talk to you.
I know you'd love to cultivate that community
and you interact with them online
outside of the podcast as well,
but I mean, there's way more demand,
both to interact with you and I'm sure you'd love
to interact with the community more,
but you just are limited by the number of hours in the day.
So, you know, at some point, I think,
making it so that you could build an AI version of yourself
that could interact with people, you know,
not after you die, but while you're here to help,
you know, help people kind of fulfill this desire
to interact with you and your desire to build a community.
And there's a lot of interesting questions around that.
And, you know, that's obviously,
it's not just in the metaverse.
I think, you know, we'd want to make that work,
you know, across all the messaging platforms,
you know, WhatsApp and Messenger and Instagram Direct,
but, you know, there's certainly, you know,
a version of that where if you could have
an avatar version of yourself in the metaverse
that people can interact with and you could define
that sort of an AI version where, you know,
people know that they're interacting with an AI,
that it's not, you know, the kind of physical version
of you, but maybe that AI, even if they know it's an AI,
is the next best thing because they're probably not gonna,
you know, necessarily all get to interact with you directly.
I think that that could be a really compelling experience.
There's a lot of things that we need to get right about it,
that, you know, we're not ready to release the version
that a creator can kind of build a version of themselves yet
but we're starting to experiment with it
in terms of releasing a number of AIs
that people can interact with in different ways
and I think that that is also just gonna be
a very powerful, you know, set of capabilities
that people have over time.
So you've made major strides in developing
these early AI personalities with the idea
where you can talk to them across the meta apps
and have like interesting, unique kind of conversations.
Can you describe your vision there and these early strides
and what are some technical challenges there?
Yeah, so, I mean, a lot of the vision comes from this idea
that, you know, I don't think we necessarily want there
to be like one big super intelligence.
We want to empower everyone to both, you know,
have more fun, accomplish their business goals,
you know, just everything that they're trying to do
and, you know, we don't tend to have, you know,
one person that we work with on everything
and I don't think in the future we're gonna have,
you know, one AI that we work with.
I think you're gonna want a variety of these.
So there are a bunch of different uses.
If some will be kind of more assistant oriented,
there's a sort of the kind of plain and simple one
that we're building is called just meta AI.
It's simple, you can chat with it in any of your threads.
It doesn't have a face, right?
It's just kind of more vanilla and neutral
and kind of factual,
but it can help you with a bunch of stuff.
Then there were a bunch of cases
that are more kind of business oriented.
So let's say you want to contact a small business.
You know, similarly, you know,
that business probably doesn't want to have to staff someone
to man the phones and you probably don't want to wait
on the phone to talk to someone,
but having someone who you can just like talk to
in a natural way who can, you know, help you
if you're having an issue with a product
or if you want to make a reservation
or if you want to buy something online,
having the ability to do that
and have a natural conversation
rather than navigate some website
or have to call someone and wait on hold,
I think is going to be really good
both for the businesses and for normal people
who want to interact with businesses.
So I think stuff like that makes sense.
Then there are going to be a bunch of use cases
that I think are just fun, right?
So I think people are going to,
I think that there will be AIs that can tell jokes.
So you can put them into chat thread with friends.
I mean, I think a lot of this,
because we're like a social company, right?
I mean, we're fundamentally around
helping people connect in different ways.
And part of what I'm excited about is
how do you enable these kinds of AIs
to facilitate connection between two people or more,
put them in a group chat,
make the group chat more interesting
around whatever your interests are,
sports, fashion, trivia.
Video games, I love the idea of playing,
I think you mentioned Baldur's Gate.
An incredible game, just having an AI
that you play together with.
I mean, that could, that seems like a small thing,
but it could deeply enrich the gaming experience.
I do think that AIs will make the NPC
is a lot better in games too.
So that's a separate thing that I'm pretty excited about.
But yeah, I mean, one of the AIs that we've built
that just in our internal testing people have loved the most
is like an adventure text-based, like a dungeon master.
Yeah, nice.
And I think part of what has been fun,
and we talked about this a bit,
but we've gotten some like real kind of cultural figures
to play a bunch of these folks
and be the embodiment and the avatar of them.
So Snoop Dogg is the dungeon master,
which I think is just hilarious.
In terms of the next steps of,
you mentioned Snoop, to create a Snoop AI.
So basically AI personality replica, a copy,
well, not a copy, maybe inspired by Snoop.
What are some of the technical challenges of that?
What does that experience look like for Snoop
to be able to create that AI?
So starting off, creating new personas is easier
because it doesn't need to stick exactly
to what that physical person would want,
how they'd want to be represented, right?
It's like, it's just a new character that we created.
So even though there's a Snoop in that case,
he's basically an actor, right?
He's playing the dungeon master, but it's not Snoop Dogg,
right, it's whoever the dungeon master is.
If you want to actually make it so that you have an AI
embodying a real creator,
there's a whole set of things that you need to do
to make sure that that AI is not gonna say things
that the creator doesn't want, right?
And that the AI is gonna know things
and be able to represent things in the way
that the creator would want,
the way that the creator would know.
So I think that it's less of a question
around like having the avatar express them.
That I think we're, it's like we have our kind of V1 of that
that we'll release soon after Connect,
but that'll get better over time.
But a lot of this is really just about continuing
to make the models for these AIs
so that they're just more and more, I don't know,
you could say like reliable or predictable
in terms of what they'll communicate.
It's that way when you want to create the Lex assistant AI
that your community can talk to,
you can, you don't program them like normal computers,
you're training them, they're AI models,
not kind of normal computer programs,
but you wanna get it to be predictable enough
so that way you can set some parameters for it.
And even if it isn't perfect all the time,
you want it to generally be able to stay within those bounds.
So that's a lot of what I think we need to nail
for the creators.
And that's why that one's actually a much harder problem,
I think, than starting with new characters
that you're creating from scratch.
So that one I think will probably start releasing
sometime next year, not this year,
but experimenting with existing characters
and the assistant and games
and a bunch of different personalities
and experimenting with some small businesses.
I think that that stuff we'll be ready to do this year
and we're rolling it out basically right after Connect.
Yeah, I'm deeply entertained by the possibility
of me sitting down with myself and saying,
hey man, you need to stop the dad jokes or whatever.
I think the idea of a podcast
between you and AI assistant Lex podcast.
I mean, there is just even the experience
of a codec avatar being able to freeze yourself,
basically first mimic yourself.
So everything you do, you get to see yourself do it.
That's a surreal experience.
It feels like if I was like an ape looking at a mirror
for the first time, realizing like, oh, that's you.
But then freezing that and being able to look around
like I'm looking at you.
It's a, I don't know how to put it into words,
but it just feels like a fundamentally new experience.
Like I'm seeing maybe color for the first time.
I'm experiencing a new way of seeing the world
for the first time because it's physical reality,
but it's digital like, and realizing that that's possible.
It's just blowing my mind.
This is really exciting.
Cause I lived most of my life, you know,
before the internet and experiencing the internet,
experiencing voice communication, video communication.
You think like, well, there's a ceiling to this,
but this is making me feel like, oh, there might not be,
there might be that blend of physical reality
and digital reality.
It's actually what the future is.
Yeah, I think it's a weird experience.
It feels like the early days of like a totally new way
of living.
And like, there's a lot of people that kind of complain,
well, you know, the internet is not, that's not reality.
You need to turn all that off and go, you know, in nature.
But this feels like this will make those people happy.
I feel like, cause it feels real, the flaws and everything.
Yeah, well, I mean, a big part of how we're trying to design
this, these new computing products is that they should
be physical, right?
I think that's a big part of the issue with computers
and TVs and even phones is like, yeah, I mean,
maybe you can interact with them in different places,
but they're fundamentally like you're sitting,
you're still, and I mean, people are just not meant
to be that way.
I mean, I think you and I have this shared passion
for sports and martial arts and doing stuff like that.
We were just moving around.
It's like so much of what makes us people is like,
you know, you move around, you're not,
we're not just like a brain in a tank, right?
It's the where, you know, the human experience
is a physical one.
And so it's not just about having the immersive expression
of the digital world.
It's about being able to really natively bring that together
and I do really think that the real world is this mix
of the physical and the digital, right?
The digital is, there's too much digital at this point
for it to just be solid to a small screen,
but the physical is too important.
So you don't wanna just sit down all day long at a desk.
So I think that this is, yeah,
I do think that this is the future.
This is, I think the kind of philosophical way
that I would want the world to work in the future
as a much more coherently blended physical
and digital world.
There might be some difficult philosophical
and unethical questions we have to figure out as a society.
Maybe you can comment on this.
So the metaverse seems to enable sort of unlock
a lot of experiences that we don't have
in the physical world.
And the question is like, what is and isn't allowed
in the metaverse?
In video games, we allow all kinds of crazy stuff.
And in physical reality, a lot of that is illegal.
So where's that line, where's that gray area
between video game and physical reality?
Do you have a sense of that?
Well, I think, I mean, there are content policies
and things like that, right,
in terms of what people are allowed to create.
But I mean, a lot of the rules around physical,
I think you try to have a society
that is as free as possible,
meaning that people can do as much of what they want
unless you're gonna do damage to other people
and infringe on their rights.
And the idea of damage is somewhat different
in a digital environment.
I mean, when I get into some world with my friends,
the first thing we start doing is shooting each other,
which obviously we would not do in the physical world
because you need to hurt each other.
But in a game, that's like just, it's almost,
it's like just fun.
And even in like the lobby of a game, right?
It's like, it just, it's not even bearing on the game.
It's just kind of like a funny sort of humorous thing to do.
So it's like, is that problematic?
I don't think so because it's fundamentally,
it's not, you're not causing harm in that world.
So I think that the part of the question
that I think we need to figure out
is what are the ways where things could have been harmful
in the physical world that we will now be freed from that
and therefore there should be fewer restrictions
in the digital world.
And then there might be new ways
in which there could be harm in the digital world
that there weren't the case before.
So there's more anonymity, right?
It's when you show up to a restaurant or something,
it's like all the norms where you pay the bill at the end.
It's because you have one identity
and if you stiff them, then life is a repeat game
and that's not gonna work out well for you.
But in a digital world where you can be anonymous
and show up in different ways,
I think the incentive to act like a good citizen
can be a lot less and that causes a lot of issues
and toxic behavior.
So that needs to get sorted out.
So I think in terms of what is allowed,
I think you wanna just look at what are the damages.
But then there's also other things
that are not related to kind of harm,
less about what should be allowed
and more about what will be possible
that are more about the laws of physics, right?
It's like if you wanted to travel to see me in person,
you'd have to get on a plane
and that would take a few hours to get here.
Whereas we could just jump in a conference room
and put on these headsets
and we're basically teleported into a space
where it feels like we're together.
So that's a very novel experience
that it breaks down some things
that previously would have defied the laws of physics
for what it would take to get together.
And I think that that will create
a lot of new opportunities, right?
So one of the things that I'm curious about
is there are all these debates right now
about remote work or people being together.
And I think this gets us a lot closer
to being able to work physically in different places,
but actually have it feel like we're together.
So I think that the dream is that people will one day
be able to just work wherever they want,
but we'll have all the same opportunities
because you'll be able to feel
like you're physically together.
I think we're not there today
with just video conferencing
and the basic technologies that we have,
but I think part of the idea
is that with something like this,
over time you could get closer to that
and that would open up a lot of opportunities, right?
Because then people could live physically where they want
while still being able to get the benefits
of being physically or kind of feeling
like you're together with people at work,
all the ways that that helps to build more culture
and build better relationships and build trust,
which I think are real issues
that if you're not seeing people in person ever.
So yeah, I don't know.
I think it's gonna be,
it's very hard from first principles
to think about all the implications
of a technology like this and all the good
and the things that you need to mitigate.
So you try to do your best to kind of envision
what things are gonna be like
and accentuate the things that they're gonna be awesome
and hopefully mitigate some of the downside things.
But the reality is that we're gonna be building this out
one year at a time.
It's gonna take a while.
So we're gonna just get to see how it evolves
and what developers and different folks do with it.
If you could comment,
this might be a bit of a very specific technical question,
but Llama 2 is incredible.
It's the, you've released it recently.
There's already been a lot
of exciting developments around it.
What's your sense about its release
and is there a Llama 3 in the future?
Yeah, I mean, I think on the last podcast
that we did together,
we were talking about the debate that we were having
around open sourcing Llama 2.
And I'm glad that we did.
I think at this point,
the value of open sourcing a foundation model like Llama 2
is significantly greater than the risks.
In my view, I mean, we spent a lot of time
doing a very rigorous assessment of that
and red teaming it.
But I'm very glad that we released Llama 2.
I think the reception has been,
it's just been really exciting
to see how excited people have been about it.
And it's gotten way more downloads and usage
than I would have even expected.
And I was pretty optimistic about it.
So that's been great.
Llama 3, I mean, there's always another model
that we're training.
So, I mean, for right now, we train Llama 2
and we released it as an open source model.
And right now the priority is building that
into a bunch of the consumer products,
all the different AIs and a bunch of different products
that we're basically building as consumer products.
Because Llama 2 by itself, it's not a consumer product, right?
It's more of a piece of infrastructure
that people could build things with.
So that's been the big priority
is kind of continuing to fine tune
and kind of just get Llama 2
and it's little, the branches that we built off of it
ready for consumer products
that hopefully hundreds of millions of people
will enjoy using those products and billions one day.
But yeah, I mean, we're also working
on the future foundation models
and I don't have anything new or news on that.
I don't know exactly when it's gonna be ready.
I think just like we had a debate
around Llama 2 and open sourcing it,
I think we'll need to have a similar debate
and process to red team this
and make sure that this is safe.
But my hope is that we'll be able to open source
this next version when it's ready too.
But that's not, we're not close to doing that this month.
I mean, this is, that's just,
it's a thing that we're still somewhat early in working on.
Well, in general, thank you so much
for open sourcing Llama 2 and for being transparent
about all the exciting developments around AI.
I feel like that's contributing
to a really awesome conversation about where we go with AI.
And obviously it's really interesting
to see all the same kind of technology
integrated into these personalized AI systems
with the AI personas,
which I think when you put it in people's hands
and they get to have conversations with these AI personas,
you get to see like interesting failure cases,
like where the things are dumb
or they go into weird directions
or, and we get to learn as a society together,
what's too far, what's interesting, what's fun,
how much personalization is good, how much generic is good.
And we get to learn all of this.
And you probably don't know this yourself.
Like we have to all figure it out by using it, right?
Yeah, I mean, part of what we're trying to do
with the initial AI's launch
is having a diversity of different use cases
just so that people can try different things
because I don't know what's going to work.
I mean, are people going to like playing
in the text-based adventure games
or are they going to like having a comedian
who can add jokes to threads
or they can want to interact with historical figures?
We made one of Jane Austen and one of Marcus Aurelius
and I'm curious to see how that goes.
I'm excited for both.
Yeah.
As a big fan, I'm excited for both.
I have conversations with them.
I mean, yeah, that's, and I am also excited to see
the internet, I don't know if you heard,
can get kind of weird and I applaud them for it.
So I get to see it. I've heard that, yeah.
Yeah, so it'd be nice to see how weird they take it,
what kind of memes are generated from this.
And I think all of it is,
especially in these early stages of development,
as we progress towards AGI, it's good to learn
by playing with those systems and interacting with them
at like a large scale, like you said.
Yeah, totally.
I mean, that's why, well, so we're starting out with a set
and then we're also working on this platform
that we call AI Studio that's going to make it so that,
you know, over time, anyone will be able to create
one of these AIs, almost like they create
any other UGC content across the platform.
So I'm excited about that.
I think that to some degree, we're not going to see
the full potential of this until you just have
the full creativity of the whole community
being able to build stuff.
But there's a lot of stuff that we need to get right.
So I'm excited to take this in stages.
I don't think anyone out there is really doing
what we're doing here.
I think that there are people who are doing
kind of like fictional or consumer oriented
character type stuff, but the extent to which
we're building it out with the avatars
and expressiveness and making it so that they can interact
across all the different apps and they'll have profiles
and we'll be able to engage people on Instagram and Facebook.
I think it's just, it's going to be really fun.
Well, I'm still, so we're talking about AI,
but I'm still blown away this entire time
that I'm talking to Mark Zuckerberg
and you're not here, but you feel like you're here.
I've done quite a few intimate conversations
with people alone in a room and this feels like that.
So I keep forgetting for long stretches of time
that we're not in the same room.
And for me to imagine a future where I can,
with a snap of a finger, do that with anyone in my life,
the way we can just call right now
and have this kind of shallow 2D experience,
to have this experience like we're sitting next to each other
is like, I don't think we can even imagine
how that changes things, where you can immediately
have intimate one-on-one conversations with anyone.
That might like, in a way we might not even predict
changed civilization.
Well, I mean, this is a lot of the thesis
behind the whole metaverse is giving people
the ability to feel like you're present with someone.
I mean, this is like the main thing
I talk about all the time.
But I do think that there's a lot to process about it.
I mean, from my perspective, I mean, I'm definitely here.
We're just not, we're not physically in the same place.
It's not like you're not talking to an AI, right?
This is, so I think the thing that's novel
is the ability to convey through technology
a sense of almost physical presence.
So the thing that is not physically real
is us being in the same physical place.
But kind of everything else is.
And I think that that gets to this
somewhat philosophical question about
what is the nature of kind of the modern real world?
And I just think that that's, it really is this combination
of a physical world and the presence that we feel,
but also being able to combine that
with this increasingly rich and powerful
and capable digital world that we have
and all of the innovation that's getting created there.
So I think it's super exciting because I mean,
the digital world is just increasing in its capability
and our ability to do awesome things.
But the physical world is so profound
and that's a lot of what makes us human
is that we're physical beings.
So I don't think we wanna run away from that
and just spend all day on a screen.
And that's like, it's one of the reasons
why I care so much about helping to shape
and accelerate these future computing platforms.
I just think this is so powerful.
And it's, even though the current version of this
is like you're wearing a headset,
I just think this is gonna be by far the most human
and social computing platform that has ever existed.
And that's what makes me excited.
Yeah, I think just to linger on this kind of changing nature
of reality, like of what is real,
maybe shifting it towards the sort of consciousness.
So what is real is the subjective experience
of a thing that makes it feel real
versus necessarily being in the same physical space.
Because it feels like we're in the same physical space.
And that the conscious experience of it,
that's probably what is real.
Not like that the space time, like the physics of it,
like you're basically breaking physics
and focusing on the consciousness.
That's what's real.
Just whatever's going on inside my head.
But there were a lot of social and psychological things
that go along with that experience
that was previously only physical presence, right?
I think that there's like an intimacy, a trust.
There's a level of communication
because so much of communication is nonverbal
and is based on expressions that you're kind of,
you're sharing with someone
when you're in this kind of environment.
And before those things would have only been possible
had I gotten on a plane and flown to Austin
and sat physically with you in the same place.
So I think we're basically shortcutting
those laws of physics and delivering the social
and psychological benefits of being able to be present
and feel like you're there with another person,
which I think are real benefits to anyone in the world.
And I think that that, like you said,
I think that is gonna be a very profound thing.
And a lot of that is, that's the promise of the metaverse
and why I think that that's the next frontier
for what we're working on.
I started working on social networks
when they were primarily text.
Where the first version of Facebook,
your profile, you had one photo
and the rest of it was like lists of things
that you were interested in.
And then we kind of went through the period
where we were doing photos.
And now we're kind of in the period
where most of the content is video,
but there's a clear trend where over time,
the way that we want to express ourselves
and kind of get insight and content
about the world around us
gets increasingly just richer and more vivid.
And I think the ability to be immersed
and feel present with the people around you
or the people who you care about
is, from my perspective, clearly the next frontier.
It just so happens that it's incredibly
technologically difficult, right?
It requires building up these new computing platforms
and completely new software stacks to deliver that.
But I kind of feel like that's what we're here to do
as a company.
Well, I really love the connection you have
through conversation.
And so for me, this photorealism is really, really exciting.
I'm really excited for this future.
And thank you for building it.
Thanks to you and thanks to the amazing meta teams
that I've met, the engineers,
and just everybody I've met here.
Thank you for helping to build this future.
And thank you, Mark, for talking to me inside the metaverse.
This is blowing my mind.
I can't quite express.
I would love to measure my heart rate this whole time.
Would be hilarious if you're actually sitting
on a beach right now.
I'm not, I'm in a conference room.
Okay, well, I'm at a beach and not wearing any pants.
I'm really sorry about that for anyone else
who's watching me in physical space.
Anyway, thank you so much for talking today.
This really blew my mind.
It's one of the most incredible experiences of my life.
So thank you for giving that to me.
Awesome, awesome.
Glad you got to check it out.
And it's always fun to talk.
All right, I'll catch you soon.
See ya.
See you later.
This is so, so amazing, man.
This is so amazing.
This is so amazing.