This is the Nielsen Norman Group UX Podcast.
I'm Sunita Tenkala. As many of you know, extended reality, or XR, is an umbrella term
that encompasses virtual reality, augmented reality, and mixed reality. These technologies
essentially blend the digital and physical worlds, offering different levels of immersion
and interaction with digital content. Now, while these technologies have certainly advanced
technically, there's still much to learn about maximizing their capabilities. I had a really
great conversation with Jan Plas, professor and founding director of the Create Consortium and
co-director of the Educational Communication and Technology Program at New York University. He
discusses where XR can have a large impact in learning environments. He further talks about
the importance of understanding affordances with each immersive technology and shares examples of
successful applications in the context of learning and education, and further discusses the differences
of building out user experiences of 3D versus 2D spaces. You all are truly in for a treat. Enjoy.
Fun fact for those who are joining, Jan was actually my UX research methods professor while I was
getting my master's at NYU. Now, aside from teaching, you have so many other responsibilities,
some including leading two different labs, one being the Create Lab and co-directing the
Games for Learning Institute with Ken Perlin. Could you tell us a little bit about your background
and what are some of the current research that you're most focused on recently?
Yeah. Well, first of all, thank you for having me. Very exciting. My background is, I'm only bringing
this up because I get this question a lot. Do I have to study something to then do whatever I studied
directly? And my background wasn't at all like that. I studied out with maths and physics education
and thought I was going to be a teacher and then went into computer science and thought I was going
to contribute there. But I really enjoyed dealing with people and looking at how people benefited from
using computers for all different kinds of things, but of course, especially learning.
And so that has always been my guiding question. How can we use computers to augment
what people do and to enhance what we do and to help us learn better, help us do things better?
And for that, of course, you have to design the technology in a way so that it can do that,
right? Just because I design something doesn't mean it's working, as you know, in your line of work.
So we need to think about the user experience. And so I've been doing that for multimedia initially
and then games for learning and most recently XR for learning. And the question has always been the same.
How do we design those media to be effective for learning, creating, collaborating, etc.?
So one of the things you just mentioned was XR for learning. And that is a really booming area right now.
Before we get into the weeds and hear about some of the work that you're doing in that space,
I wanted to start off with some of the basics. What is XR? And I know that's a big question.
But what is XR?
Yeah, no, it's an important question because we use it so casually. And I think putting a definition
behind the terms is always useful. XR simply means extended reality. And it's a combination or mix of a
number of different technologies that we currently already have available as consumer devices that
actually differ quite significantly. And one of them is augmented reality. That's when I use computer
generated images, sounds, other things that I place into the real world. So I wear some goggles.
And when I look at you, I might see something about you or I might, you know, whatever it is that the
computer would generate about a person, about a space. If I look at a power plant, it might explain
to me how it works, right? Whatever there might be of interest to me, it augments the real world with
computer generated, you know, items, visual sounds, etc. Virtual reality, on the other hand,
completely immerses you into that space. So the world ceases to be there around you. We sometimes
forget that the world is around us, which is what we call a personal sense of presence in VR. And so
it's kind of this world where, you know, if you've seen Ready Player One, you know, everything is
computer generated. And then, of course, there are some people in there with their avatars,
if it's a collaborative system, who you can interact with. And then there's mixed reality.
That's when we add computational objects that are smart objects. So you can actually touch them,
you can do things with them. That becomes mixed reality, because now I'm adding actual objects,
where augmented reality just puts visuals in the real world. A mixed reality would also have
objects that they can touch and that the system is aware of. If I lift the cube, it knows, or the
mug, it knows where that mug is. Your research has focused a lot on the context of learning.
And what specifically got you interested in XR for learning? Yeah, it's kind of a technology
technology that that has, frankly, had many starts, right? XR, VR is certainly not new. It's been studied
for 40 years. But it's made, or it's trying to make a comeback where it's finally going mainstream,
we have devices that are consumer type devices. And so it brings together a number of really
interesting things. In fact, since I'm on this show, let me speak in these terms, it brings together
what Donald Norman wrote about the design of everyday things, right, the idea of what are affordances,
and how do you use affordances and the cognitive processing of information as a lens, right,
from that perspective. Then it brings together emotional design, which is kind of taking almost
the opposite perspective. Now it's the emotional world that you inhabit or that inhabits you,
and how we feel about the world around us and how that guides our actions. And then a little bit
of invisible computer, right? When all of the computing things go away, and you feel like you are
in a world that you believe in, right, you're immersed in it, you experience it almost like
the real world. And in some cases, we will experience it like the real world. And the fact
that there is a computer behind it is actually not of all concern anymore, which is very different
from what we're doing, right? We're very well aware of the fact that this is all computer-mediated.
And that fascinates me, because that opens up opportunities for learning that are quite remarkable,
where we have been, you know, we have many ideas of what good learning should look like. And we've
always been limited by screens, by keyboards, by mice implementing those.
Yeah. Why do you feel like it's, specifically, why do you feel like it's, in what ways do you see
it being a promising tool in educational experiences?
Yeah. I mean, first of all, I want to say that I would not claim that it is the only promising tool,
right? So any new medium will have, or any medium that we use for learning will have capabilities
that other media don't have in the other way around, right? So it's back to affordances. What
are the forms of virtual reality for learning? And one that I am especially interested in many,
many researchers in this area is that spatial arrangement of information that you find very
difficult on the screen. You can do it. There's screen-based virtual reality. It's not immersive.
It's on a screen. It's 3D, but you're not in it. So what if I can be in a 3D world? What if I can,
we're building, for instance, or have built simulations where you explore middle school biology,
you've built your own cell, plant cell, animal cell, procreatic cell. What if you can be there
and actually grab that organelle and construct the cell? What if I can be in the solar system and I
can go and stand on the moon and see what things look like from the moon? I mean, that is a completely
different type of learning that takes advantage of the spatial relations, that takes advantage of the
immersion and the sense of presence that emerges from that. It takes advantage of collaboration and,
of course, of emotional design, which is one of our research areas, emotional design for learning,
to induce the kind of emotions that are conducive to learning. And so it's a very unique medium to
bring all of this together. And now I'm talking about virtual reality. Augmented reality actually
has many other promises. Got it. Yeah. I mean, I feel like these kinds of simulations or these
experiences give learners an opportunity to experience a higher degree of empathy that's difficult from
written materials and lectures even. I'd love to hear a little bit about maybe examples of successful
XR-based educational experiences that either your lab is currently working on or created or you've come
across and why, and why you think they've been successful or what makes them successful?
Yeah. Yeah, absolutely. Well, let me start with one that we didn't create. So it was good to,
to realize there are many, many talented people to create amazing things. And this one is called
On the Morning You Wake. You may remember the scare when Hawaii thought there was an inbound ballistic
missile and 38 minutes, they thought that was real and they will come to an end. And there's a very
amazing group of, well, really VR filmmakers who got together, interviewed people and created
a virtual reality experience in which you can, well, obviously not quite experience that situation,
but experience how people who were there experienced it and follow them. And that was one of the most
moving experiences I've had to be in this immersed environment, you know, hearing about from
eyewitnesses what that felt like. And we did some research on that. And it very clearly showed that
virtual reality was a very, you know, much, we compared it to screens to, to tablet based viewing of
the same or of a version of this documentary. And so it was, it showed very clearly how virtual reality
can make that a much stronger emotional experience. So that was fantastic. On the morning you wake,
you should definitely look that up. And then that was more an experience rather than an interactive
participation. And as such, it was very, very powerful. And so that would help with the learning
approaches like experiential learning where you reflect after doing, right? So that's a, that's a very
excellent example for that. We're doing interactive environments as well, where you can collaborate
with others. And we had funding from Verizon social responsibility, and they funded a number of
projects to look at pushing the edge of immersive learning and XR. And so there are projects like
virtual science where they essentially immerse you into the cosmos and, and fast forward certain events
that would take millions of years and you can watch them. Very amazing. There's another one,
Mapper's Delight, where you can learn about rap music around the globe. That is, and the rap music
that you never heard of, because almost every country in the world seems to have people who make that
kind of music and they catalog that and make that available. There's another one called Unsung,
which tells untold stories of singers of color. And so you can learn about them and, and really
realize how, you know, broad the spectrum of singers is and different types of music. And then ours,
which we, we designed as part of this initiative, it was a competition that they put out a while ago.
It's called looking inside cells. And so in looking inside cells, you can
create your own cell based on the organelles and you need to pick the organelles. You get all
different kinds of organelles that may not relate to your cell and you need to pick out which ones
are part, let's say of a plant cell, and then you, you arrange those. And then once you've completed
that, you learn about mitosis, but instead of just watching it or in a textbook, you would just have
in the different phases of mitosis, you would, you know, see, um, or read about it here, you actually
do it. You help, um, duplicate the organelles in the cell, and then you help, um, grow, uh, have them
grow and then you help split the cell. So you weren't actively involved in that and that kind of,
and you can do that collaboratively with, with others. And so that kind of an experience is really
where I think, um, virtual reality can provide something you couldn't do in a classroom because when
my kids studied, uh, cell biology, they were doing little clay models of that. And as fun as that
was, I think we all might have a few more things to offer. Yeah. So, um, I'm curious with specifically
that, uh, experience that you created or the lab had created, um, were you able to test with,
uh, test the simulation with, um, students? Absolutely. That's what we do for those who
are part of the research space in virtual reality and learning. Um, those people will be aware of
the fact that the, like with almost every other medium that we use for learning, the research
evidence that virtual reality is effective for learning is actually not as clear as one would
think. And the reason for that is that what one person means by virtual reality is very different
from what another person means by it. And, and so we get a lot of results where there's one paper,
um, by my, um, friend and colleague, Richard Mayer and, and, uh, Justin Brown, um, that shows that
virtual reality in their study wasn't even as good as a PowerPoint slideshow. So you learned more from
the PowerPoints and you had lower cognitive load. That's usually the paper that I start with when I talk
about virtual reality and learning. Um, because it raises the question, how can that be? And the answer is,
well, you need to take affordances of virtual reality during learning when you design, and you
need to design your outcome measures in a way, how do we test what, you know, in a way that it
actually benefits, uh, learners who were in VR, you know, have something to show for. So if it's only,
um, naming, uh, um, the organelles of a cell, I can learn that from slide, right? But if it's about
spatial arrangement of those, then that's something that I now know that, uh, when I was in the virtual
reality environment. And so, um, that is exactly the study we ran. We compared, um, our virtual
reality tool, building a cell and doing mitosis to a PowerPoint slideshow. And lo and behold, um,
it is better to be in virtual reality for that. And there was more, higher learning outcomes for that
group. So, um, that's the kind of research we, we sometimes do media comparison. Um, but we're much more
interested and that's what we're doing right now to ask if I now use certain features in virtual
reality to induce emotions that, you know, put you in a positive state or a state of heightened
emotional arousal, would that, um, result in better learning outcomes? And that's a new project we're
running right now. And we're doing this actually right now, practically right now, um, where we use,
um, measures, biometric measures to measure your skin conductance and your heart rate variability
to get indications of your emotional state. And, uh, we give you different stimuli. We show you,
just like you say, with that, um, behavioral research, right? In our case, it would be more
your emotional response to seeing certain elements in virtual reality. Um, how can we use visual
features, sound and haptic features to induce emotions is currently our question. And then we
will put that into this learning environment and see if that enhances learning outcomes.
Wow. Super interesting. Um, I, do we have, do we get a sneak peek into what you've learned so far?
Well, um, we did a study actually a few years ago where we, uh, built in the past games to train your
cognitive skills, executive functions, which are some of the most important skills that guide our
cognition as well as our emotion regulation really is responsible for everything. Um, um, inhibitory
control, um, cognitive flexibility, working memory. And so we built games for that. And one is called
all you can eat tea. And there you have to feed aliens, um, different drinks, either milkshake or
cupcakes, and they change their minds all the time, which is how you practice your, your cognitive
flexibility. And we built a VR version of that where we, um, had the aliens in VR and instead of
just coming down on the horizon, they were coming toward you. And so they were getting bigger and
bigger and until you fed them. And, um, and we ran a study where we used a very common, um, user research,
uh, uh, method, uh, called think out loud, think about protocols or concurrent verbal protocols. And in
that we, um, just taught them to, uh, um, verbalize what they were thinking about, but we didn't give them
any specific instructions. And then later we looked at unsolicited utterances that related to the
emotional state. And we found that, um, in the virtual reality environment, there was a significantly
higher emotional response to those aliens coming toward, uh, the, the user. If you, if you would
have seen it, you would understand why this is the first paper where I actually was able to publish
expletives in a paper, because those were the kind of strong responses that people had to those aliens
coming. And so that was our first indication, um, how powerful, uh, VR is, uh, to, to induce emotions.
And now we're systematically exploring that. Um, let's say we know that warm colors and round
shapes induce positive emotions. So what if we use a warm, a brighter background, uh, color compared to a
more, more dimmed, uh, less warm background color, right? So those are the kind of, um, uh, conditions that
we're currently looking at. What if we use particle effect, uh, where there's sparkles around an object
and compare that to not having those. And I wish I could tell you, but we're actually in the middle
of the experiment. And so I don't have the, um, that's, that's totally fine. Um, what I find really
interesting though, is how, what you're describing, you know, there's certain visual design elements
that we know can exhibit or induce a positive, um, emotion. So there are some, you know, user
experience or visual design practices that we might apply in a, you know, traditional devices that we
use in 2d spaces. Um, I'm, I'm curious when you're designing these spaces, these different XR experiences,
do the same UX best practices follow, or do you feel like there's certain things that are different?
Yeah, that's an excellent question. I think there is always some overlap in, you know, the, the kind of
most basic, um, user experience, uh, design principles, like in terms of clarity of revealing
functionality, um, in terms of clarity of, of guiding the learner, what to do, those kinds of
things, those usability, um, uh, principles still apply, but how they implement it could be very
different. Um, when you design for a screen to put some texts to explain what's going on onto that
screen is what we do all the time, right? Sometimes that's the fastest way to, to, um, help the learner,
guide the learner. Um, in VR, we try to get rid of any texts that's more than a couple of
words because it just looks like I am supposed to be in a virtual space. And now there's a big,
you know, I don't know, announcement in, in, in text form. Right. And that, that breaks the whole,
um, uh, sense of presence of me being in that space because we don't have big billboards floating
around that says, Hey, click me or go this way. Right. Um, so there's, there's some very, very clear,
um, differences in, in VR, XR, uh, use less text. Um, but then there are others where coming back to
making it explicit, what you can do in this environment, the world around us doesn't have
labels everywhere, um, to say, click me or take me or so. And so what you need to think about in
virtual reality, if an object can actually be manipulated, which most of them should,
because otherwise it would feel weird. Um, then what does that mean? And how do you signal that to the
user? So the whole signifier and the whole, um, you know, making the user aware of the affordances,
um, so that they know what to do in the environment has to be rethought. And that's a fascinating
process, right? How do you guide the visual attention now that there's so much to see in
this environment that they might miss on the one important thing that you want them to do? Um,
or how do you do it that it's actually not up to you, what you want them to do. They get to do
whatever they want to do in a sandbox style, uh, environment, which is of course much harder to
design. So it's, it's fascinating how you need to develop a new language because we're still dealing
with controllers. We're still dealing with some of the things that will go away, but as long as I
have controllers and as long as I have a laser beam that I pointed things, and then I grabbed them by
pushing a button, I still need to explain that. So we often run small training sessions where we
built a custom tutorial to get you acquainted with how things work. Um, so you still know you're in a
tutorial and you're learning about the environment and then you go into the environment and then all
of that fades away rather than integrating all of those help features into the environment itself
and thereby breaking the sense of presence. That's been working very well for us and people sometimes
go through the tutorial in a couple of minutes and then they're done. And if you do need a little more
time, then you just spend more time in that tutorial. Um, eventually that will go away,
right? We're doing a lot of things now just because it's new for people to interact in such a world.
Um, and we'll get used to it. And especially when we have full finger control, which is already,
you know, available in some of the more expensive systems, but, um, where, where you just point at
things or actually grab things. Then we deal with other interesting questions. Like if in virtual reality,
I reach for my coffee mug and I tried to pick it up, how do I make sure that actually feels like
I'm picking up a mug? Right. And so now we're back into mixed reality type questions. And we have a
project here at NYU that, um, we're, we're, um, working on the proposal for where the question
becomes, can we work with robotics people, for instance? So then when you reach for something,
that there's something right there where you, where you reached for it in the shape that you're
expecting, right? Lots to think about, right. Um, lots more to explore as well. So I imagine
there's a lot of novelty and how in, in, you know, UX practices that we have, but there's also,
like you mentioned in the beginning, um, there's a lot that is standard and still applies within
these new contexts as well. Um, I had a question about accessibility and inclusivity. So as we know,
it's a, you know, vital consideration when building XR experiences, what do you feel like,
or what can we do to make sure accessibility and inclusivity are at the forefront of building these
kinds of immersive experiences? Yeah. We just ran a summer project on that where, um, we looked at,
um, the various types of, uh, or, or issues that come up with accessibility, right? And there's some
very simple issue, which are simply access issues, um, who has access to these devices. And then there
are, um, ability related issues, right? Which, which, um, expression of abilities do I have, or do I not
have as a particular learner or user and how should that be supported? And I think the first thing that I
have to say with, with that is that it's a mindset of all of us, whether we believe that this is
important, because if I design a virtual environment, I could design a lot of accessibility features into
it. If I made that from the very beginning, my goal, uh, but often we don't often we say, Oh,
let's just put something together. And then later we'll see what, um, what we can do in terms of
accessibility and that retrofitting rarely works, right? You need to think about that from the
beginning. Now, especially when it comes to, um, uh, visual perception and, uh, your site, um, it is
very difficult to create virtual reality environments where for people who, uh, um, have problems in that
regard, but that doesn't mean we shouldn't try to think about what that could look like. And there
are very interesting, um, uh, projects in the works where people, uh, collect sounds, right? So what, what
is, uh, for instance, the eclipse is coming in April 24, the next eclipse that we see in North America. Um,
I have a colleague who's been collecting the sounds of animals during the eclipse so that, um, people who
can't see the, the actual eclipse can hear what does it sound like when the eclipse is happening? Animals are
all of a sudden becoming quiet and then they all of a sudden start, um, making noises, making sounds
again. So there's, there's, um, a lot that we can think about and we have here at NYU, we have an
accessibility lab, which is, uh, specializing in those kinds of, uh, questions and we've been working
with them and, um, and we're asking the question, so what can one do? Um, and, and guiding that
question should be that we're not designing for others, right? Um, this is, um, ability or disability is a
spectrum on which we all are somewhere placed temporarily and it, it could take one stumble
and you all of a sudden have mobility issues, right? Or it could take other turns and all of a
sudden you have other, um, uh, experiences or conditions which place you elsewhere on the
spectrum. So it's in a way we're designing for ourselves or possibly our future selves. And we need
to keep that in mind. We're not designing for others. And, um, and virtual reality again, has a lot
of potential for that because we have the haptic sense in it. So, so strongly compared to other
environments, we have auditory information in it. Um, and we have the visual information and
there are very different input devices that we can use that are assistive devices that are already
available. I a hundred percent agree. And I agree with the point you make about it's a mindset and
developing that kind of mindset. Also, it takes time and, um, it takes time to invest. It takes
investment in that kind of research to show the kind of impact that it can have as well.
So it's great to see research labs, like taking that effort, people like you also thinking about
that right at the forefront as we're expanding these tools even further. So I wanted to kind of
wrap up and ask you for people who are interested in developing XR experiences, what advice do you have
for them? Yeah, I think my, my primary advice is going back to, I think how we started, which is to
say, um, and that's the advice that I would give for any medium. If you think you want to design
something for XR, for AR, for VR, for kind of this in between, uh, that we now see that, that allows
both XR, uh, AR and VR experiences. Think about the affordances of that technology and whether you
really take advantage of that with what you think you want to design. And that means spatial, that means
taking advantage of the sense of presence, including social presence, when there are others, that means
emotional design, that means high interactivity, high levels of interactivity. And, um, or in some cases,
the immersiveness alone might be enough, um, as in the, in the case for on the morning, you wake, but think
deeply about why this medium, because there are access issues. There are there issues of, you know,
it's not just cool that cool will go away. So you need to think very deeply about why it should be an
extended reality type of experience. And if you have a really good answer for that, then you should go
ahead and do it. And, um, I typically tell my students don't focus on the technical side of things,
work with an engineer who, who knows how to build the system. Um, the blue sky thinking about what should
learning really be like is something that often gets lost when we ask ourselves, what can I currently
implement with the skills that I have? Because I'm the designer and the developer. And so we find it much
better that you find someone who has that development skill and coding skill and can work in one of the
the environments, unity, unreal, web XR. Um, uh, but, but what we're lacking, I think still like in any
medium, in any learning environment is the people who are thinking big and are coming up with new ideas
grounded in the research. There's so much research available, but look at the research, uh, see what's
already there and then, um, implement some of these ideas that we've always had, but, um, that nobody could
ever build. And that will be so much, uh, in terms of innovation, so, so much of a benefit to all of us,
um, when, when people have this kind of blue sky thinking. So I think it's a super exciting world and
anybody who embarks in that process will, I think, find it really rewarding. Um, we certainly have.
My last question is you have talked about some incredible research that you've been working on.
Uh, what's the best way for people to follow that research and learn more?
Yeah. I mean, uh, we have a website create.nyu.edu. Um, you can see what projects we're currently doing.
Um, we can Google scholar provides all the articles that, um, we have published on that. If you need
access to any of them, you can email me, um, to, to get, um, a PDF for those. Um, and we just
got a contract to write a handbook, uh, of learning in virtual reality, uh, that, um,
which may, um, Peter Markransky and I will be editing. So, um, that will be coming out in MIT
press and 2025 probably. And we're very excited about that because that will, uh, provide a
research just like the handbook of games for learning that we did with MIT press provide a
resource for those who want to know what the research, uh, state of the art currently is.
Excellent. Thank you. Thank you. Thank you so much. Um, I could easily talk to you about this
for another hour. Um, but I really, really appreciate you taking the time to, um, talk
all XR with us. Um, I've, I've learned so much in this conversation. Thank you.
That was Jan Plass. Check out the show notes for links to anything we referenced so far,
but also remember that we have thousands of articles, videos, and reports on our website
about UX design research strategy, and even UX careers. That website is www.nngroup.com.
And if you enjoyed this show in particular, please follow or subscribe on your podcast platform
of choice. This show is hosted and produced by me, Samita Tenkala and all video editing and
post-production is by the Larimore production company. That's it for today's show until next
time. And remember, keep it simple.