This is the Nielsen Norman Group UX Podcast.
If you know anything about us,
then you know that we love research,
user research to be precise.
Our research has shown that understanding customers isn't
optional but essential if you want to succeed as a business.
That said, research is hard to do,
harder to refine, and even harder to manage at scale
when you have many people contributing and using that research.
But there is hope.
Today, you'll hear a conversation I had with Cara Pernice,
Senior Vice President at Nielsen Norman Group about how to manage
research processes and insights so that they
actually impact the designs that we are working so hard to create.
With that, here's Cara.
Hi, Cara.
Welcome to our podcast.
How are you doing today? How was your weekend?
It was fabulous. I actually did something really strange for these days,
which is my family came over.
We had like 11 people and we celebrated my birthday early.
I actually started crying when they had a birthday cake because I was like,
it's been so long since we've gotten together like that.
I just felt really like,
I can't believe this is happening and I really appreciated it.
Yeah, that's right. Your birthday,
I just remembered.
My happy birthday.
Your birthday is tomorrow.
That's very exciting.
Well, happy early birthday.
Thank you and happy belated birthday to you because we just had
a birthday.
Yes, thank you. I appreciate it.
Well, awesome. Well, yeah,
I'm glad your weekend was great and thank you for joining us today.
I'm curious because I feel like I know bits and pieces,
but I would love to know the full story, the full scoop.
How did you get into UX and what brought you to NNG specifically?
Oh, man, how much time do we have?
I got into UX in a very strange way.
The year was 1991 and when I graduated from college in Boston,
the cover of the Boston Globe said,
91 graduates, no hope.
I know, I majored in communications and public relations,
and I was about to take this job at
a communications agency on Newberry Street in Boston,
and my sister Susan was working for a really awesome company,
Lotus Development in Cambridge,
and she said,
Cara, you can't take that job.
You have to come work with me, this company is amazing.
She made it her mission to find a job that would maybe fit me,
someone who had taken a few psych and a few programming classes,
and found this job that required someone who had a master's in cognitive psychology,
and some other things I had never heard of before.
I had no business going for this job and I can't even
believe I got it after I think eight interviews.
They basically hired me because I could write
and I knew how to interview people from my journalism background.
Then my boss at the time,
Mary Kate Foley, just put a whole bunch of papers on my desk and said, read these.
The top one was by a guy named Ben Shneiderman,
and the one below it was a guy by the name of Jakob Nielsen in Denmark,
and I was like, wow, what is this?
So he's interviewing people,
but not really, he's watching them,
and I got sucked in and just started doing research.
I've been doing it ever since, that was 1991, and that was that.
I ended up at Nielsen Norman Group by way of probably SIGCHI and some other groups.
I used to see Jakob at different conferences,
and then he was my idol.
He was my screensaver for a while on my computer.
I don't know if I've ever told him that.
This person, Mary Beth Redker,
he was in the Boston area and he was asked for a tour of our usability labs,
and Mary Beth wanted to do it,
but she had a really big meeting and she said,
I know you are really interested in meeting Jakob and he wants a tour.
So I gave him a tour and after that,
we got to know each other and he asked me to review
some papers for CHI conferences and things,
and then he asked me to come work for him years ago, 20 years ago now.
So yeah, it's been quite an interesting journey here.
You mentioned SIGCHI. What is SIGCHI?
Just so we can spread the word.
Cool. Yeah. Thanks for asking for me to expand on that.
It's an older organization.
It's still around, but it stands for
Special Interest Group for Computer Human Interaction,
and it was where we used to nerd out a lot back in the early 90s.
That and UPA, Usability Professionals Association,
which has since been renamed UXPA.
So we always had these support groups in UX back in
the day and some of them still survive, but many more today.
Yeah. I can imagine, especially in the early days,
that it was probably not a lot of folks were in this field.
So getting a chance to really talk about the work that you do and not just the work that you do,
but being able to explain that work to others,
I imagine that it was really great to have that support.
Yeah, it totally was.
It was interesting, Therese,
because back then it was a support group
because we were always talking about things like fighting the good fight.
It's been fun to see over the years the change that has happened.
We still do fight the good fight,
but when you talk to somebody who's newer to the field and doing the work in UX,
they're not really talking about that as much.
They're just talking about how to do the work.
This idea that it's us against the developers or us against product is,
I don't think it's gone,
but our mentality is kinder and more open and more,
I shouldn't say kinder because we always were trying to do the right thing back then,
but there's more of an understanding of what UX is,
which I think paves a road for all functional groups to work better together.
Yeah. You mentioned that there have been changes over the years.
What have you learned about how teams do research specifically?
Because I know you mentioned that research was
something we're very passionate about even back then.
What has changed over time?
Yeah.
How UX research is done now?
A ton. I think a ton has changed.
Well, back then we did,
like I mentioned UPA, which is now UXPA,
and we always shared our research methods.
We do these talks and we'd talk about how you do international or how you share research,
or how you catalog your research in databases and things.
I think we were still figuring it out and it was like throwing spaghetti against a wall,
where today what's happening more is the spaghetti has already been on the wall,
and now we're figuring out how to name the spaghetti,
and how to throw it better,
and how to make it stick.
So it's things like terminology and almost branding and systemizing
our work has occurred, which is exciting.
Things like design thinking,
like empathizing, and brainstorming, and iterating.
We've always done that,
but we didn't call it something,
and we didn't make it simple for people to understand,
for people who weren't in our field to understand.
We're like Lean UX,
like naming it something and prescribing an order in which to do things has really helped us,
I think, grow and speak about what we
do more knowledgeably in a way that others can understand.
So I think that's huge.
Yeah. That's also a metaphor with the spaghetti,
because I think to take it even a step further,
we're keeping the spaghetti fresh too.
Ensuring that we're updating research
and making sure that we're revisiting older research to see what's still relevant,
what's still good.
Yeah, that's a really good one.
I mean, fresh spaghetti is important.
Sometimes we want to check out that fermented spaghetti.
One of the pet peeves I have always had in research is leaving data on the table.
Like when you do a research study and you write your research goals,
let's say, or your research questions with your team,
and we're always so busy that we will say we need to learn these three things.
So we do, let's say, a usability test and we learn
that stuff and we act on it and we move forward.
Well, when you do a research study,
there's a whole bunch of other stuff that the user has done or users have done
that a lot of times we don't really analyze or discuss or track.
I remember when I worked for IBM,
I would have people asking me to do a research study and I'd be like,
okay, we already have that data because we already did a study like that.
Can you read this first and then see?
It was like, I wouldn't say nine times out of 10,
but a lot of times people would say,
yeah, I guess I don't need to do this study because we do already know that,
or let's do this different study and use that one as a jumping off point.
I think that's part of what we're calling research ops today,
research operations, and really tracking
those things in a way that can be shared and found again.
Yeah. What exactly is research ops if you had to define it concisely?
Boy, you're making me do it concisely?
Okay. Well, how would you define research ops then?
I think research ops is about the components related to doing research.
It's basically looking at how do we plan research?
How do we refine and improve our craft of researching?
How do we also fit research in on projects?
How do we make sure that research and how we do it is actually
going to thrive and survive at our organization and in the world?
How do we educate and teach others about how to do research when we are doing research,
what that research is,
and what we learned in research studies?
So I feel like a lot of folks think research is about administration,
like admin-type tasks, and it certainly is.
Research ops will include your research repository,
skeletons for doing a field study,
what are the forms we use to make sure people
have consented to participating in a study.
So it is a lot of admin stuff.
But I feel like the ops part is even more than that, like the operations.
I learned about operations back in business school when I was getting my MBA,
and they really pounded in the history of operations,
and Eli Whitney and the cotton gin in the late 1800s,
and then later in World War II,
figuring out how do we really make things easy for people and expedite their work.
So I feel like when we think about operations,
it's really about efficiencies and economies,
and ensuring that what we're doing is making people work in a way that's productive and fast,
and with that, ensuring that we open up time for them for strategic thinking.
So it's the same principle that we talk about when we think about design systems.
I know you wrote a nice article about that not long ago,
and when you think about a design system,
someone might first say,
hey, you're going to make me use all these designs that you're taking away my creativity.
But the idea behind it is really consistency for our users,
but also these simpler widgets could be designed and refined and even coded,
encoded in a good way and put out there,
and then the designers, their minds,
their time can be opened up to think about more strategic things.
I think that's true with research too.
Even just the craft of research and understanding which method to use when,
and how to deal with the politics of those times when you write a research goal,
and you start to design a study with somebody,
and in the middle of that study design,
someone might say, oh,
and can you also learn how many people would want to do this?
You realize that is a research question that is not going to work in the study that we're doing,
so we need to track those things.
So basically, the idea of the operations of research,
and we want to be able to have systems in place so researchers can tackle
those problems instead of having to spend their time writing a script or finding
that form to make sure that we're getting consent in an ethical way, in a fair way.
Right. So essentially you're systematizing much like a design system,
systematizes the process of creating interfaces in that you have this place.
This place could be a lot of things,
but it could be an intranet or it could be maybe it's a shared drive of some sort.
But you basically have this place that you can use as a resource,
and you have things perhaps like the repository, like you said,
the research repository, the insights and saving them for the future.
But also what you mentioned,
which is the administrative details that often come with administering a study,
such as consent forms, such as maybe screening
surveys that people can customize to their needs.
You got it. Yeah. So it's tools, it's capabilities,
it's also your budget,
it's also your roles and your staff and communication.
So it's really everything.
It's really everything related to the system of doing user research.
Okay. So given that it is research ops,
who should be in charge of research ops?
Should it be the researchers or since like you said,
there's so many other moving parts, you've got budget,
you've got the people in charge,
you've got the actual maybe organizational structure.
So who should be the person to oversee that?
Or is it many people who oversee that?
It's a great question, Teresa.
I don't think there's like one answer to that.
When you think about it, some organizations don't even have a researcher and some
have tons of researchers that could be organized in a matrix way or a central way.
So it's going to really depend a lot on the organization,
its UX maturity, its size, its roles.
I think in an ideal world,
you're going to have people whose job is research ops,
like managing research ops.
They may even do something even more granular like manage an insight repository.
Like that's all they do is take these insights
that we learn about how people think over time and they may manage that.
So I think in an ideal world,
we've got a pretty big staff that can
really someone be in charge of the research repository,
and someone be in charge of some of these other things.
But I don't think that happens all that frequently.
So what happens pretty frequently,
I know from our classes is we'll have people who are just a team of
one and they're supposed to be doing interaction design,
information architecture, visual design, research.
To them, they may feel like,
oh my gosh, the idea of organizing research ops will put me over the edge.
However, it's usually going to help them.
So even a team of one should
find ways to be efficient with their work to reuse something they did well.
Then as an organization grows and a team grows,
you should have roles,
people who are responsible for doing these sorts of things.
Okay. Yeah. You mentioned design ops.
I remember sitting in on that class fairly recently as well.
It is interesting thinking about how design ops itself can be its own functional team,
of course, depending on the size of the organization and how
many UX professionals are in that organization.
So I guess my next question is,
how does it relate or not relate to design ops?
Since it seems like there is already,
and just for some context folks listening in,
design ops would be the operational aspect of doing design.
How we do design, the people behind those designs,
as well as the processes behind those designs,
and really breaking that down,
designing that very process,
which is pretty meta.
So on that note,
where does research ops fit in or does it not fit in?
Yeah. It's a good question, Therese, too.
I actually defer a bit to my colleague, Kate Kaplan,
who's been doing so much research on
design ops and teaches a really good class that we offer on that.
We've talked about this and she believes,
and I'm with her on this,
that research ops is really a part of design ops.
Because if we think design operations is really enabling consistent quality design,
really, I feel the only way you can do that is with sound,
thorough, good research.
So that to me, it's really just a part of design ops.
She calls design ops,
I think she calls it a curated potluck dinner or something like that.
Yeah. So I think that we're going to be
like some of the main courses on that potluck table,
research ops is. So it's a part of it.
Okay. That makes sense. Yeah.
I would definitely not recommend trying to design
your process of designing if research is not part of that process.
Totally. Yeah. We need it.
So I guess on the topic of things I would not do and maybe you would not do,
what are some common research ops mistakes that you see?
That's a great question, Therese.
So I've been talking to quite a few practitioners
recently about research ops and what they're doing and not doing in
preparation for a class that we're developing called research ops.
It's going to debut on September 1st this year, 2021.
Woohoo. Yeah, who is right?
Some of the things that I've seen,
I don't know that I'd call them trends,
but I'd certainly call them themes,
are things that folks are not doing that I think would help them.
One of them is not creating a research roadmap.
What will happen is folks will say,
well, we're working at Agile and I just as needed,
I will do research.
I'll just we'll do it as needed and when we can get people to listen,
and that just doesn't really work.
That just doesn't really work.
What you really want to do is talk to
your product owner or whomever is in charge of scheduling
basically and figure out where you can fit research in.
What I like to do is think about which areas and features are most
important to the business, to the organization, and to the users.
But then I like to look at another plane and that plane is,
how hard do we think this thing is going to be to design?
You never really know that.
You have to make an educated guess on that,
but an educated guess is better than no guess at all.
Then you decide based on
the importance and how difficult something might be to design,
and try to put those design phases,
those design sprints, whatever you call it at your organization,
try to put them early on and well before
the development sprint or before anyone's going to write an inch of code.
With that, you've allowed yourself enough time to actually iterate the design.
You're not creating these blocks that we create all the time or we accidentally
create or created for us that make it impossible to do
multiple rounds of research or to get anyone to listen to what we learn in research.
We want early research on stuff that's important,
that's hard to design,
and we want to iterate that design as many times as we
can and then have people write the code on it.
By the way, the developer and anyone writing code should be as
much as they can involved in the research as we go.
I feel like that's a pretty big mistake.
It has to do a lot with planning and scheduling.
I think that that's one of the hardest parts of doing UX.
I mean, doing UX research is that idea that I have to
convince someone to do the research and to listen to me.
It's less frequently about budget,
it's more about time.
I think schedule, I know roadmap is a big issue.
Another one which is related to that is really trying to
design research ops or even just a research repository,
trying to design it either alone or with just the UX folks.
Because then what happens is it's only a good tool for
the UX team and everyone else might be working in whatever,
air table, and we decide to work in dovetail
or we're working in considerably or something like that,
and no one knows that tool and they don't go to that tool,
or maybe they're not well-integrated,
and so it makes it even harder,
puts up a barrier that doesn't really need to be there.
I think doing it alone is another thing that
is kind of problematic today.
Yeah. I love that point you made too about the tool sometimes
itself being the barrier where maybe
somebody is interested in the research findings,
they're interested in learning.
But as soon as they encounter this new tool,
maybe they suddenly feel like an outsider
and this isn't intended for them,
and even if they are given the privileges to access that,
that's still another tool that person now has to learn,
and that can be intimidating and limits who gets exposure to that.
Exactly. Totally.
If you had to share
maybe a couple approaches that someone could
take when putting together something like
a research repository or research artifact repository,
what advice could you give when starting to create these types of systems?
I think choosing a system,
using a system is about looking at the content.
It's like designing a website actually.
It's like looking at the content that you
have and the content you believe you should have,
and then evaluating the tool based on that.
We know for research,
we need things like scheduling.
We need communication about when we're going to do research,
how and when that's going to be done.
We need to track our overall reports,
but then we need to track separate insights and maybe
even prioritize those things and then give a status about them.
So tagging is massive.
Dovetail is really cool for that kind of stuff.
Once you learn it, there's some tricks
to all of these tools that you have to overcome.
But there are a lot of different tools that I think will work today.
That's one of the reasons I wasn't dying to talk about
tools because they change so frequently,
and there are so many different features on all of them that what we
say today might not be true in three weeks.
Right.
But I think looking at what you
think you need from a tool is what's most important.
Right. Totally agree.
I guess for those folks who are now looking at their research process,
maybe thinking, I've never thought of my research process this way before.
How do you even start when trying to improve a UX research process?
What would you advise?
It's hard to be self-aware, isn't it?
You don't necessarily know what's wrong with your current process
if you're the one who created it and is living in it.
I think a little bit of outreach within your organization,
with your designers, developers, product,
whomever else might be involved to figure out what is
it that they seem to know and not know and need and not need.
Some of that's going to come from asking them,
and some of that's going to come from more observing.
Just like we know, that's how we do research.
We ask questions and then we also observe.
So I think some of that could be really helpful to figure out where the gaps are.
Rather than looking out there and going,
okay, what's the perfect example?
How does Facebook do this?
That's not necessarily going to apply to you.
So I think looking within,
trying to figure out what works and doesn't work,
and try to find some places where it's going to make a big impact.
If you know we're doing tons of research
and it seems like people aren't really involved in it,
and one of the reasons is either don't
understand it or they don't know when it's happening,
then maybe you focus there first.
You focus on making an easy to understand schedule and writing quick summaries,
boilerplate of what these basic research methods are that we do.
So people can maybe watch a video,
you can make a video of it or something like that.
I think find some pain points right now and try to fix those to start.
So you can see some wins and then also folks,
stakeholders around your organizations can see
some wins and understand why this is worthwhile.
Yeah. I think that's helpful.
I know that a lot of folks like me tend to see something like this,
get really excited and perhaps a little overambitious.
To use a metaphor that one of my colleagues used at one point,
I'm trying to remember exactly who it was,
but the metaphor was don't boil the ocean.
Try not to completely revolutionize your research process,
especially if there is a good standard process.
Maybe identifying some of those key moments can lead to
the biggest return on that reinvestment of time and effort.
Yeah, totally.
I think it's true,
boiling the ocean is too much and it'll make you crazy.
You want to be happy too.
You want to be able to feel like you made some progress.
That's an important reality to doing this kind of work.
If you're trying to really make a big change,
are you going to be a change agent in your organization?
You need to feel good about it and you need to
keep your energy up and if you keep feeling like you're failing,
you're not going to do a good job,
so make it possible.
Yeah, make it possible.
Achievable goals.
Yeah, and make it enjoyable too.
Make it something that you enjoy
contributing to and that others enjoy contributing to,
so that it's something that ultimately can be useful to others.
Yeah, totally. I think what's cool about that idea too,
is it's easy to make research findings enjoyable
by just opening up the world of what users do and say.
People say the craziest, funniest, interesting things.
If you've never done a usability test with your team,
the first time you do that,
you should just cherish it because they're going to love it.
They're going to be like, wow,
can you believe that?
This person is really someone we would be selling this to,
and I can't believe what they just said.
I never would have thought of that before.
Just pulling out interesting quotes and interesting things they did.
One of my favorite things used to be to grab a huge stack of videos.
These were videos, these were NTSC tapes that I would grab 20 of them,
and rewind them all to the same point,
and we do what we call the master design studio,
and I'd have everybody just watch three minutes
of 20 different people doing a task,
and then we'd start sketching.
This was in 1993.
We do stuff like this today,
but our technology is better and we're much better
about organizing it and talking about it and things like that.
But these things make it so fun,
and it makes it fun for the team to move forward.
It also makes them more confident.
If you start to do a design and you go back and look at
a couple of research studies right before,
you feel really grounded.
You feel really like, okay, yeah,
these are the problems that I need to solve,
and it humanizes it too.
Yeah. I like that,
thinking of it as a tool of empowerment,
not just for the researchers,
but for everyone who wants to solve
a design problem and feeling like they are equipped with the information
and the evidence that they need in order to make the best design possible.
Absolutely, yes.
You're so articulate, Therese.
Thank you.
You are.
Cara, it's already time.
I can't believe that time has flown by so fast.
Me neither.
Wow, Therese, you made it fun. Thank you.
Not scary at all.
I'm glad to hear.
Well, if others would like to follow you and
your work or want to learn anything else about research ops,
where could you point people to?
Sure. Well, I'm on LinkedIn.
I'm pretty active on LinkedIn.
Feel free to find me there.
I'm on Twitter, but less is active at Cara Ann there.
For research ops, we'll certainly be writing quite a bit.
We've got our class coming up and there's
a whole community of people that have been really moving this field forward.
The cool Slack channel and the leaders there,
Kate Tousie, Ruth Ellison,
Bridget Melser, Holly Cole, Emma Bolton.
There are tons of people that have been doing awesome work in the research ops area,
so you can also check their blogs and their workout too.
For sure, and we'll include all the links to that in the show notes.
So check that out if you want more info.
But otherwise, we're looking forward to your research ops class debuting September 1st.
So if you are interested,
definitely check that out at our website and ngroup.com.
We'd love to have you there.
Awesome. Well, thanks, Cara.
Have a great rest of your day.
And you, Therese. Thank you.
Thanks for listening to episode 12,
the last episode of this season.
Next month, we'll be starting season two of our podcast.
All thanks to you, our listeners.
We are so grateful for your support and if you want to show that support,
please leave a rating and subscribe no matter what platform you are on.
Also, you can find all of our reference courses and links in the episode notes.
But if you want to get updates emailed to you on articles and videos that we publish,
or any upcoming courses as they get released,
please sign up for our free email newsletter on our website at nngroup.com.
That's n-n-g-r-o-u-p.com.
Until next time or next season,
and remember, keep it simple.