This is the Nielsen Norman Group UX Podcast.
I'm your host, Therese Fessenden. On this monthly show, you'll hear interviews with
user experience thought leaders about fundamental UX topics, common industry questions,
and discussions on how great UX can truly transform the world.
Today, I'm joined by Kate Moran, Senior UX Specialist with NNG.
Kate's research spans a breadth of topics, and she's a pioneer in the industry for her
research insights. From discovering new browsing patterns with eye-tracking research, page parking,
tone of voice dimensions, and many other groundbreaking concepts.
She holds a master's degree in information science from UNC Chapel Hill and teaches
six certification classes on a number of different topics, including the topic of
today's podcast, Return on Investment, otherwise known as ROI.
On the show, Kate shares her insights from her latest research on benchmarking and ROI.
What to think about when communicating the business value of UX,
what to avoid when making metrics-based decisions, and how to ensure you're not letting the tail
wag the dog, so to speak, by letting data make your decisions for your team, rather than having
your team make decisions with the data. With that, I'm excited to welcome Kate Moran.
Thank you for joining us. How are you doing today? What has been exciting? What have you
been working on recently? Oh, I'm pretty good. You know,
like a lot of people, I've been hunkered down. But luckily, work has been busy, so it keeps me busy.
Yeah, I will say, also, I happen to have an English bulldog puppy sleeping next to me,
and she is a big snorer. So you'll have to let me know if that gets too loud.
Holy cow is my favorite. I will let you know. Don't worry.
Okay. So, yeah, you know, I feel like, like, I mean, you know, you know how it is, like,
with NNG, we've always got so many things that we're juggling at once. So, but I've been really
focused on reports, like, in general, this year, that's been kind of a focus, been doing a lot of,
like, research and a lot of report writing. And so that's something I'm focused on right now.
Yeah, so you've recently published one of them, if I remember correctly, the ROI report, right?
Yeah, it's called UX metrics and ROI. And essentially, it is a big collection of case
studies that describe how teams are using quantitative data in their day to day work.
So it's 44 case studies. And the really fun thing about that report is we've been updating it since
2006. So we've got legacy data going back to 2006 for how teams were improving their products and
quantifying that and collecting these UX metrics. That's really cool. And I know for some folks
hearing ROI sounds like a foreign language. So I would like to, for the benefit of others
listening, how would you define what ROI is exactly? Yeah, so the report is kind of about two
different but related things. And so the first kind of aspect of the report is about UX benchmarking.
So that is just collecting, you know, some sort of metric that represents the improvement
that you're making in the design, hopefully an improvement, usually an improvement.
But things like, you know, like time on task, like how efficient people are with the product,
or it could be something that you get from analytics, like your conversions, or it could be
returning visitors, just something that is a numerical representation of some aspect of the
experience. So that's kind of the first part of the report is we break down how that process works,
collecting those metrics, analyzing them. And we look at a lot of different case studies where
teams were doing that. So then the second sort of part of the report is ROI, so return on investment.
So that's sort of taking those metrics and showing how those metrics connect to business goals as
expressed as KPIs or key performance indicators. If that's not jargon-y enough for you.
That's perfect. Yeah, key performance indicators is, I think some folks might be a little bit more
versed in those in that it tends to impact a lot of people's jobs, like, yeah, certainly plays into
a lot of people's, you know, compensation, or not just compensation, but like how well or not well
the entire team is doing. Yeah, well, that's, that's what KPIs are. They're from a business
perspective, it's ways that you can sort of check in and see if the company or the organization
is meeting its goals. And usually, those are tied to revenue in some way. So a lot of times
we're talking about like profit or, you know, cost savings. And as you say, it is often tied
to how people are evaluated and like whether or not they're going to get bonuses or whether
they're going to get a new hire that they need, things like that. Yeah, it could be pretty
stressful, I imagine. And for some of the folks, I imagine that you researched ROI must have been
either something that went really well with the organization or stuff that stressed people
out. So could you give people any advice if they were, you know, trying to figure out how to
quantify their impact, not to lose their mind while doing it? Well, in that report, we have
some step by step instructions for doing that. But essentially, the thing that I really like
about looking at UX metrics and collecting ROI is that it requires you to think really deeply
about what is the value of UX and then sort of express that through quantitative data.
So you have to understand, you know, in most cases, we're not doing UX work because it's nice,
right? We might, you know, want to believe that that's why we're doing it. But in most cases,
we're doing it because it makes sense, because it makes business sense, and in many cases,
makes financial sense. The problem is that a lot of teams don't do the work of actually showing
that. They don't actually do the calculations. So, for example, McKinsey, about a year or so ago,
they did a big survey of different design teams. And they found that over 50% of the design teams
that they surveyed weren't doing that. They weren't quantifying their impact. And so as a result,
they had no way of showing how they were impacting the business. So the first thing to do is to think,
you know, what is the purpose of this product that you're working on? Or like, why does this
product exist? What value is it offering for people? You know, for like, let's imagine like,
you know, an intranet. So why does an intranet exist? To help employees find information that
they need, to make them efficient, to help them get their work done, and kind of support them.
So, okay, so that's the value of an intranet. So like, what are the things that we could measure
that would tell us whether or not it's doing its job? Probably a lot of efficiency things
are going to be important. So things like time on task, and also, you know, making sure people
can find the information that they want, and that it is the correct information. You know,
things like that. So there are UX metrics that we can collect that describe those things and help us
assess how well is this design working now. And then once you start doing this over time,
then you can look back and see, okay, how much progress have we made? How much better
have we gotten now? Kind of like looking back. So that's kind of our first place to start.
And then with the ROI calculations, it's really about figuring out, okay, I have these things,
these like aspects of the experience that I've measured. And that's on one side. And on the other
side, I have ROI metrics, or KPIs, things that the business values. So to sort of oversimplify
it a lot, it's about making that conversion kind of almost, I like to explain it to people like
converting units. So taking the improvement in the design and showing how that is all working,
is also something that contributes to cost savings. That's probably what we would look
for with something like an intranet. We would want to see, hey, look, we made people more efficient.
We helped them do their jobs more efficiently. How much value is that really adding to the business?
Got it. That makes a lot of sense. And I'm glad that you're talking about it like converting
units because I think people get very stressed about it and worry that if I don't justify this
in just the perfect equation, or if I don't get the value absolutely correct that this might not
be taken seriously. But I guess what are your thoughts on that? Are there things people should
keep in mind or mistakes that people make when they calculate? Or communicate ROI, I guess.
Right. Oh yeah. Well, I actually wrote an article on that, on those mistakes that people often make
when they're doing these calculations. I think part of the problem is that we call it ROI,
to be honest. If we could rebrand it, call it something different, there'd probably be
less confusion. But I think the problem is that people hear ROI and they think, okay,
this needs to be some super accurate, super specific financial calculation.
And the problem is that it will just never be that. It can't be that. We're trying to
predict what human beings are going to do. Just the way that we do these calculations,
it assumes that everything else in the world is going to stay constant. And we just don't know
that that's the case. You could make huge improvements to an e-commerce site and calculate
that it's going to translate to $2 million in added revenue every year. But then a huge
competitor comes out of nowhere, like Amazon decides to get involved in your product space,
whatever it is. And all of a sudden, that revenue is not there. We have to understand that it's not
a prediction. It's really a strategic exercise. Sometimes I tell people to think about it like
a thought exercise. It's really just trying to communicate what is the potential monetary value.
And sometimes it's monetary. Sometimes it's not. Sometimes you can just express something as time
saved. And that can be powerful. I think people need to remember we're not doing these ROI
calculations because they're fun. We're usually doing them maybe to communicate to our stakeholders
or our clients that there's something we need to do. There's a redesign that needs to happen.
There's some change we need to make, and it's going to be valuable. Or maybe after the fact,
we're trying to show those people, this is how much value we delivered for you. Or if you're
in a leadership position, you might do these kinds of calculations to help you prioritize and say,
look, I've got these six different UX projects I could work on. We're going to pick this one
because it's going to be potentially the most value. So we're always doing ROI calculations
for a reason. And so you should do as much work in that calculation as you need to.
Right? Does that make sense? You should do just enough to help accomplish your goal,
but don't do more than that. Got it. Okay. I like that idea partially because sometimes the thought
of doing massive financial predictions sounds terrifying. But yeah, you're right. It's a
thought exercise. It's meant for us to strategize. It's meant to be a tool for us. And this leads me
to another common theme I see in a lot of our conferences and classes that a lot of businesses
kind of, for lack of a better phrase, fetishize numbers. When they strategize, they often obsess
over these increased outcomes or increased performance at all costs. And sometimes I think
about that, especially nowadays, you see a lot of patterns that are somewhat dark patterns or
nefarious design choices for micro conversions, or just to get that additional 0.5% increase in clicks.
So I guess another question I would have is, do you have any advice for how teams can stay
growth oriented and avoid unethical behavior that might be incentivized by looking at
these numbers? Or should we deprioritize some metrics or should we prioritize others?
What do you think of this very big complicated question?
Well, I definitely know what you're talking about. I've definitely seen that in the teams
that I've helped. And also just kind of the culture around metrics and UX can sometimes
be toxic, unfortunately. I think part of the problem is that when we start collecting these
metrics, the power can kind of go to our heads a little bit. We can get a little excited about all
the data that we can find, especially with tools like analytics, where it's so easy to just pick
out all this information about what people are doing with your product in real life.
I think the problem comes in when teams let the metrics make their decisions for them
versus just looking at the metrics as a tool. And I think there's a distinction here between
using quantitative data to help inform design versus letting it lead design. Sometimes I think
this can be a cultural problem in organizations. It certainly seems to be something that's true
of people in leadership positions. They tend to care a lot about the data, care a lot about
quantitative data. And this is a sweeping generalization, of course, there's exceptions,
but they tend to be more impressed by the quantitative stuff. That's part of why I
recommend UX teams get involved with benchmarking and with calculating ROI is so they can sort of
speak the language of the business. But the problem really comes in when, as I said, teams
are letting the data make the decisions. And also I think when there's maybe too much emphasis on a
single metric to the extent that we sort of lose sight of the bigger picture. So I have two examples
for you. First is a non-UX example, and then I'll give you a UX example. So the non-UX example,
if you remember a couple of years ago, gosh, I don't know, was it five years ago? I'm getting so
old. Five, 10 years ago, something like that. Anyway, so you might've heard about the Wells
Fargo scandal that happened. Do you remember that being on the news? Somewhat, but if you
could refresh my memory since I'm also getting old. Fantastic. Yeah. So Wells Fargo, which for
anybody who's listening, who's not an American, this is an American bank. And they had a huge
scandal a few years ago where it came out that some of the account managers were creating
accounts for people without their permission. So they were creating these bank accounts for
their customers without their knowledge. And the problem was that these people were being charged
fees for those accounts. And also it's a violation of their privacy and security. So it was a huge
scandal. And I remember reading about it at the time that the problem ended up being that
the company was so fixated on one number on new accounts opened. I think it was like monthly new
accounts was their metric that they used. And they stressed it so much to the employees.
And they would fire people if they didn't meet those targets. So I know it was wrong of them,
but these account managers were saying, all right, I'm going to have to either I get fired or I
create these illegal accounts. So some of them were creating accounts. So that's just a really
dramatic example, I think, of this concept, this phrase that gets thrown around a lot, which is
what gets measured gets managed. So what we take the time to measure and to talk about and to use
as one of those KPIs, that does have consequences on the real life decisions that people will make.
So that's a non UX example, but it's definitely true in the UX world as well. So let's think
about a digital example. There was a trend that came out. Now this was a couple of years ago,
I feel like it was like two years ago, where we started to see these things that we call
manipulinks. I don't recommend manipulinks, but I'm saying yes, because I just remembered Kate's
video. Kate and Kim recorded this fantastic video. That's all I can say. Just look up manipulinks
and in group you'll find them. Well, thank you. We have fun with it, for sure, at least. So
manipulinks are our name for these little pop-ups that appear usually on e-commerce sites, sometimes
on content sites. But they're these little pop-ups that are basically trying to bully users into
signing up for their newsletter. And part of the way that they do this is they'll have an opt-out
link underneath that newsletter signup that says something like, no thanks, I don't like fun.
No thanks, I don't like deals. So the idea there is they're sort of putting words into the mouth
of the user and making them say something negative about themselves to sort of punish
them for not wanting to give their email address over to this company. Now, obviously from a UX
perspective, where we care about long-term relationships with our customers, that's not
ideal. Definitely not. So the question is, how do these teams do this and how are there so many
of them? Like now, if you hadn't noticed this before, you're probably going to start to see
them all over the place. I still see them constantly. I'm always finding new ones.
I think what's happening is I think these teams do see an increase in the number of people signing
up for those newsletters when they use these kinds of approaches. And some of the ones I've
seen are really bad. They will not even make it clear that someone's signing up for a newsletter.
They'll say something like, put your email in and you'll get 12 recipes for cooking over the weekend
or whatever it is. So they'll give you a little plug like that and they don't make it clear that,
yeah, you're going to get those 12 recipes. You're also then going to be subscribed to
all of our newsletters forever. So I think in some cases there's multiple things at play,
but I think teams are seeing that those approaches, they get results. They get a 12%
increase in newsletter signups. And if your boss is telling you, you have to get newsletter signups
at all costs, then that might become your priority. Now, the problem there obviously
is you're sacrificing your long-term relationship with your customer for a very short-term gain and
you're sort of bullying them into something they may or may not actually want or need.
So I think there's a short-sightedness there. I think there's a lack of understanding the context
there. And I think that's the danger with these metrics is they can't be the be all end all.
I couldn't have said it better myself. Looking forward into another bout of lockdown with the
pandemic slash next fiscal year, well, calendar year, I guess, lower risk tolerance is probably
going to impact a lot of UX decision-making. I think everyone's kind of trying to prepare
for worst case scenarios and it might get harder to innovate in this next coming year. Teams might
err more on the side of safety or you may be keeping existing designs as opposed to trying
to radically redesign or change things. So if you have any advice, what would it be to help teams
perform well, whether it's to meet those metrics or change their metrics, if that's applicable or
what do you recommend to teams as we look forward to the next year?
Well, that's definitely something that I am pretty anxious about. I'm sure most other people are too.
Nobody knows what's coming in 2021. The vaccine news sounds really promising, but this is like
an economic situation that nobody has seen before. So it is definitely a concern that,
especially depending on the industry that people are, whatever industry people are in,
some industries are going to be hit harder by this than others. And I am concerned that some
companies will start looking for places to cut their budgets. They'll start looking for places
where they can reduce overhead and get some savings. And I'm concerned for those teams
who haven't done this work yet. Whether or not they're doing this in a quantitative way or not,
if they haven't been making the argument, here's what UX delivers, here's why we're not a nice to
have thing, here's why we're an essential thing, I worry for those teams because I think that is
where we might see a little bit of regression. So one of the things that I'm working on in addition
to these reports is, I don't know how much I can say about this. It's kind of top secret, but it is
related to our UX maturity work that we've done in the past. So we love to study how teams, how
companies evolve and become more mature in terms of their approach to UX and their approach to
their relationship with their customers. So that's one thing that we're working on right now,
something upcoming related to that, but so it's very top of mind for me.
And top secret.
But it is true that, unfortunately, we don't see that teams always get more mature over time,
that companies always get more mature. That's what we like to think about. We like to imagine that
we're always working towards this end goal of someday every company will be super UX mature and
they'll understand that it's a necessity and valuable, but it doesn't always work that way.
Sometimes there is regression. We see teams and companies that were more mature and then for
various reasons, there's some backsliding. And that is one thing that I'm really anxious about
in the coming year is I hope we won't see a lot of companies falling back in that way. But I think
an important tool that can help people to fight that is trying to prove your value, trying to show
the company that you're worth keeping around. This is not something that all teams have to
worry about. It really tends to be more of a tool that you need if you are a little bit lower in
that UX maturity, if you feel like there is an already an established understanding of the value
of UX. But just last week, I was helping this woman who works for the United Kingdom's government.
She works on one of their digital products and she was like, I'm really struggling with figuring out
how to prove the value of UX and I'm struggling to do this calculation. So I ended up talking to
her for a while and probing a little bit deeper. And we ended up realizing she didn't need to do
the calculation because their government already has a really high UX maturity. They already see
the value. So this is not something that you always have to do. It's not something that
everyone has to do or that you have to do forever. It's just one of those things in your toolkit for
making sure that people understand the value of UX. Yes, that's some really nice advice as well.
And yeah, remember that you're really doing this for your team and for the company you work for.
And if you have that value known, then embrace it and keep chugging along, keep doing the good
work of UX. Yeah, definitely. Yeah. Well, Kate, if anyone wants to follow you, can you point them to
any social media or any related places to follow your work? Yeah. So you can always follow me on
Twitter. I'm at my name, Kate Moran, with two underscores in the middle. Sometimes it confuses
people. In retrospect, I probably should have just picked one underscore, but that's Kate
underscore underscore Moran. Of course, you can always visit nngroup.com to check out our reports
and our articles and all the videos that we mentioned today. Awesome. Thank you, Kate. I hope
you have a great rest of your day and I hope we get to talk to you again soon. Yeah. Thanks,
Therese. You too. Bye. Bye. For any of the resources we talked about today,
check out the show notes for links and the report that Kate's been working so hard on.
You can also subscribe to our weekly email newsletter, where we share free articles and
videos and upcoming opportunities to get a UX certification at one of our now virtual UX
conferences. To learn more, go to nngroup.com. That's n-n-g-r-o-u-p.com. Thanks for your support.
Until next time. And remember, keep it simple.