This is the Nielsen Norman Group UX Podcast.
I'm Therese Fessenden.
As we start a new year, we often like to reflect on our progress and improvements.
In fact, at the end of this episode, we're going to share some exciting news about improvements
we plan to make to this podcast, so stay tuned for that.
But, when it comes to resolving to improve, in general, there are just so many ways to
do that.
Last month, Dr. John Pagonis talked about how important it is to combine qualitative and
quantitative research, and to close the loop.
Today, we're going to feature a team that did just that, and how it led them to not only
improving their public-facing website, but it also won them a Nielsen Norman Group Intranet
Design Annual Award, which recognizes the world's best intranet designs.
Now, intranets, meaning internally-facing websites for employees or other organization members,
these aren't often the subject of most design awards.
And I get it.
It is fun to award the splashy public-facing websites.
But, these intranets are incredibly challenging to design, given how many different kinds of
users they often serve, and the resources that are often available to rise to those design
challenges.
Often, it's pretty slim.
So, we're excited to share an interview we had last year with two members of Princeton
University's award-winning intranet team.
Charlie Kreitzberg, Senior UX Advisor at Princeton University, and Christian Noble, Princeton's
Director of Digital Strategy.
In this episode, my colleague, UX Specialist Tim Noyesusser, interviews Christian and Charlie,
and they share with us how they got involved with UX, what they think makes their team successful,
and how they created two complementing frameworks that allowed them to make the most of their mixed
methods research, and further, make some pretty incredible design choices.
Yeah, thank you, Charlie.
Thank you, Christian, for being on the show today.
It's really an honor to have both of you here.
And maybe we can start off, and you can just tell us a little bit about yourselves and how
you ended up at Princeton University.
Okay, perhaps I should jump in first.
I followed a very torturous path to get there, and it goes way, way back to the 1960s when
computers were giant things with disks that were 36 inches around.
And I was 16 years old, and I was a music student, actually.
And I went up to Dartmouth College for the summer, and the first night that I was there, I locked
myself out of my dorm room and started wandering around the campus looking for help.
And I went to the only building that had a light on, which turned out to be the computer
center.
And I had never seen a computer before in my life.
And I was absolutely gobsmacked by this thing.
I think it was instant love.
And I really spent the rest of the summer teaching myself programming, working on the
computer there, which was open to all the students.
Actually, Dartmouth had created the first computer literacy program in the United States and invented
a language called BASIC, which you could teach yourself.
And so I started doing music on the computer, and that really led to a career in computing.
I went on to get an undergraduate degree in computing and a graduate degree in computer
science.
But the thing that I really would like to share with you, Tim, here is that when I started
working with computers, it actually changed my mind.
I had a different way of looking at the world, and I wanted to bring that to other people.
And that's what brought me into UX.
I was just so excited by this technology and the effect it could have on people that I decided
to switch into psychology, and I went to get my doctorate in cognitive psychology so I could
begin to explore the relationship between computer technology and people.
And that was basically the path that I took.
When I graduated, I started my own company, and I ran that for 25 years.
We did UX in the 1980s and 1990s and into the 2000s.
So it was just a very exciting time, and we were working really hard to teach people about
this.
And it wasn't until 2016 when I really kind of burned out running the company, and I said,
I don't want to be doing this anymore.
And this opportunity at Princeton opened up, and the opportunity to work in such a great
institution was something I just wanted to grab.
And so that's when I came in and met Christian.
Christian, how did you end up at Princeton University?
Well, I think if Charlie's path was meant to be, mine was definitely not meant to be.
You know those squiggly lines that we always think about when we think about career paths?
I think mine has certainly been like that.
I started in media, actually working for newspapers.
Well, when newspapers were at the very, very beginning, the late 90s, transitioning to the
web.
So I started at the Wall Street Journal.
I got hired there as a senior editor.
And the Wall Street Journal is what was then the online journal, but still today, WSJ.com.
It was in its first year, and it was great because it was the Wild West.
We were making stuff up every day.
And I remember, I really think back on those days very fondly because everybody there just
trying to publish the news in this really brand new, very untested format in mode.
None of us knew what we were doing.
We're making it up.
It was awesome.
It was a lot of fun.
And eventually, well, I stayed there for about, I don't know, eight, nine years, went to the
New York Times, also worked for the Philadelphia Inquirer.
In fact, actually, when we go back to the journal, it was, the journal had invited Jacob Nielsen
to come in and talk to us about usability.
You know, these, we were in such a nascent stage.
These, you know, any information we could get was like gold for us.
And I just remember sitting there listening to Jacob and thinking like, oh, wow, this is
like great.
It's just really opened my eyes up to this, this field called UX.
And I've been, you know, I want to say a fan or a lurker sort of ever since.
And when I came here to Princeton, met Charlie, all my UX dreams came true, I guess, so to
speak.
Because, you know, here was something I really kind of loved, but was not practicing.
And to be involved with a UX team and to really now be able to learn about it through actual,
you know, doing was just really a lot of fun.
And kinds of work that Charlie and I have done together, me from sort of a product side,
because really, I think, you know, we didn't call it back in the day when I first started
working in, you know, was the internet.
Certainly it was not, you know, the internet was not a sure thing in those days.
We didn't really call it product, but I think as time has gone on now, now we do see the
kind of work that I do, do as products.
So to have this real appreciation for UX and meet Charlie, who was doing UX, and both of
us coming together kind of as a team and trying to attack the same problem from slightly different
angles, I think has been just a really terrific experience and a lot of fun.
And hopefully we're doing good work and hopefully it's something that our users really love and
we're serving them really well.
So that's kind of how I got here.
When I first came to Princeton, I met Christian and another one of our colleagues, Jessica Monaco,
who's also a wonderful UX designer and visual designer.
And, um, our first project was also, it was actually to work on an intranet and that project
did not succeed.
Um, we got started, but it, it didn't never, it never really came to fruition, but it was
something we really, really wanted to do.
And, um, I'm not exactly sure why we tend to spark so well off each other.
One reason is I think we're both very direct and honest people.
There, there, there's not a lot of facade, at least when we're, we'd like to think we
have a little bit of political savvy, but the reality is we tend to say exactly what we
think.
And sometimes we think very different things and that's an enormous strength because nothing
makes me happier than when I field an idea and somebody says, I have a better way of doing
it.
And that expands my mind and another way of, uh, looking at things.
So that's been a very productive, uh, and very pleasant relationship for now, these six
years.
It's really admirable what you were able to accomplish at Princeton.
And Charlie, you mentioned the intranet that first didn't come to fruition, but how did
you reach that from first starting this project, which didn't come to fruition to actually then
designing one of the best intranets in the world?
Well, I think Christian really was the motivator, uh, on the second, our second attempt.
Uh, he spent two years going around the university, interviewing people, trying to understand what
people wanted.
And I kept coming to him and say, I want to be part of the team.
I want to be part of the team.
He kept saying, yeah, well, we'll, we'll, we'll get there and we'll get you in there.
And it took two years, but, you know, I think one of the things at core is that both of us
are absolutely committed more than anything else to the products that we're designing and producing.
We want to create beautiful, excellent, wonderful products with wonderful user experiences.
And there's really nothing that's more important than that.
And, and, and that I think is one of the things that, that holds us together and has, you know,
made this work.
And I should also say that on this, on, on the internet project that we were, we were just
delighted to receive the award.
But, um, I should also say that, you know, it wasn't just us two.
We had a wonderful team that we were working with, um, from, uh, the web development services
group at Princeton and, uh, Jessica Monaco, who was also a designer there.
And we had, uh, programmers and phenomenal project management.
And also my colleague, Susan Sparragan did a lot of usability testing and supported us
with some of the research.
So it really was a group effort.
And the, the team was just completely aligned around this.
And I, I don't remember one meeting where we had, you know, conflict or, or, you know,
an unpleasant experience.
We just, whatever the issues were, we addressed them, we worked them through in a constructive
way.
And I think that really did make a big difference, uh, in the end product.
Yeah.
I couldn't agree more because, um, you know, it is, it is a team effort and I think what's
really critical, um, and this applies to the, the, the work you do.
And then how you make that work a reality.
And that is to, to, to keep your ears open, to really, really listen, let somebody else
speak, keep your own mouth shut.
So for all the research work that I did, um, I really made sure that I, I, I got to every,
not every, you know, in a, in a quantity, but certainly in a quality from a quality aspect,
uh, the different user groups that, that customer groups that we were, we, we had to serve
and just let them tell me what, you know, we had a little method.
It's not, it's not like, you know, it was just completely free form.
We had a method, but, um, to just get them to talk about, you know, their pains and their
gains as we, you know, we typically talk about, uh, on the product side, uh, and really find
out, uh, where we needed to play this integral part.
And that's because, uh, there was no real, there was a need for an intranet, but there
was no demand for an intranet.
And we knew we had to approach this as if it was a brand new SAS product that say, I
mean, or at least analogous to that, um, nobody knew they needed until it was, you know, they
saw it and it was put in front of them.
And we knew it had to be like really super good because otherwise adoption wasn't going
to work.
And so that means not simply listening to customers, also listening to stakeholders and
doing what you can to align those needs.
That's, that's really, really critical.
You do need the support of your stakeholders as well as providing something great for customers.
And then in the development process, when you're sitting there with the team, the same
thing, you need to do the same thing.
You need to shut up and listen because you have in the room, a bunch of people, they're
very good at what they do.
They know, they know what to do.
And it's really important to make sure that they have a voice in what they're creating,
no matter what their, their, their job is.
So you may be a developer, but you know, we're not going to do just limit your voice to
development.
You have experience, uh, you know, customers from different perspectives.
You need to have an input into that and where we got to those points.
And of course they come up in every project where there are differences of opinion.
It's just about having these really super healthy arguments to make sure everybody can just
kind of really sit there and, and, and promote, uh, and advocate for their position, defend
it where they need to, and eventually come to a place where, uh, we know what we've got,
you know, cause we've really woodshedded the idea that, you know, as far as it can go
based on the research and, and of course, based on everybody's opinions and experience
and, and, and, um, abilities to create what, what comes out of that is something that's
really, really super useful for the end user and also meets those stakeholder needs.
Yeah.
If I could add one thing to that question, but team alignment is absolutely, I think the
most important thing.
If you're going to create a good user experience, uh, I, I have been literally on hundreds or
maybe thousands of UX projects having run, you know, this, uh, UX company for 25 years.
And, um, what I found was that what usually made projects go south was that the team was
not aligned around a common vision and that they really couldn't, uh, adjust their own
priorities and their own worldviews.
So that's become a really important thing.
And then at the end of the day, if you do have disagreement and you do have conflict, you
have to have a way of resolving it, which for me is that, you know, if Christian and I disagree,
Christian wins because he's the product manager and he has the larger responsibility.
Whereas I'm a UX lead and I have, I'm one of a number of factors, uh, that's responsible
for the way this is going to be produced.
But there's also the organizational strategy and there are the business processes and there
are the technical limitations and, and technical objectives and all of that has to be factored
into it.
So, uh, we were very fortunate in that way.
And, and with our development team, actually the director of web development services, Jill
Maraca came in there and was hands-on for the entire time of the, the, the, the project.
And so if a, you know, decision had to be made that we couldn't do it as a team, it would
either be Christian or Jill, depending on whether it was, you know, a, a technical decision or
a product decision.
We met a couple of months ago and we spoke about the Princeton EDU website.
You were telling me how you used a survey on your website to gain relevant insights.
Could you maybe tell us a little bit more about that project?
Sure.
Yeah.
The survey was for us to understand, uh, what are different audience groups we're doing
on our site?
Like what content was each audience group interested in?
So the university has a lot more than your average set of audience groups.
I mean, we, our small list is 16 and I mean, you can imagine what many of them are faculty,
staff, undergraduate students, graduate students, I mean, their parents, perspectives, and it
really can go on and on.
And, uh, university website, just, you know, the, the main EDU site, like any other organization,
any other institution, there's the front door.
That's where all those groups are walking through.
So we had to make sure that, uh, our, our IA, so, uh, information architecture, how you
organize the content on your site, and it's directly related to the navigation.
And so the findability of content, uh, the way people who come to your site understand
where things are and how they can find them is directly related to how you organize that
content.
And we, we speak of that, about that organization as an information architecture or IA.
It's so interesting to see your dynamics, your team dynamics, how you describe, how you work
together.
And when you, when you were talking about your team, what were some criteria that helped you
figure out what to design?
And I believe you mentioned a rubric that your team followed.
Could you maybe share a little bit about that, Charlie?
Yeah, I'd love to.
Um, it was really interesting because Christian and I each came to this project with something
we had developed ourselves that were quite different, but that turned out to work really,
really well together.
Christian's pair lab and my five, uh, five D rubric.
So as I've mentioned, I have been on hundreds and hundreds of UX projects, and I've gotten really
clear about the fact that team alignment, if that fails, um, then you're very unlikely to emerge with
a decent UX at the end of the, at the end of the project.
Uh, because there are so many, uh, different decisions that have to be made.
And if they're not made in a consistent kind of way, um, as one of my colleagues said, it's like
getting nibbled to death by ducks.
Each, each decision takes a little bit out of the integrity of your design.
Uh, so I became very interested in the fact that we don't have a common definition of UX.
Um, I've done a little research on this.
Uh, I asked people to define UX.
I get the most wonderful definitions, but when you look at it, they're not operational.
They're not workable.
Um, so I went back to some of the work that had been done.
Whitney Quisenberry, who, uh, produced, uh, something called the five E's, which was a
faceted view of, uh, user experience.
You might be familiar, some of the listeners with Peter Morville's Honeycomb, which was very
popular for a number of years.
And these all tried to take this complex multidimensional thing called user experience and produce
um, a definition that looked at the various dimensions and facets.
And, uh, I felt that could go another step.
So I did a whole lot of interviewing of people to try to find out what did they consider the
components of UX.
And out of that, um, I developed a schema called the five D rubric five D standing for
five dimensions.
So the five dimensions that we've found most important, uh, in describing UX comprehensively,
um, are the first one is empowerment.
The second one is efficiency.
The third one is intuition or intuitiveness, or sometimes we think of that as usability.
Uh, the fourth one is engagement.
And the final one is trust.
And these are very complex dimensions.
There's a lot to say about them and they can be customized to any specific, uh, project.
But what we found is that with those five dimensions, people feel satisfied that we're getting all
of the information that they want to give us about their feelings and their, their experience.
And, um, um, that then became a definition that we were able to share with the entire team, uh, with users and which could support also assessment and metrics.
We developed a bunch of online, very nice little assessment tools, which gave you both qualitative and quantitative measures.
And I am very much of a qualitative person when it comes to analytics, being a psychologist.
I want to interview people.
I want to learn about what people have to say.
So I am very much of a qualitative person in the way that I like to look at things.
And, you know, when you look at the way that, uh, UX is traditionally measured, it's often with something like the system usability scale, the SUS or the, uh, net promoter scale, the NPS, which does it on a scale of one to 10.
And system use, uh, system usability scale gives you a number from one to a hundred.
I look at that and I say, what does it mean?
I find something that has a value of 70 or 75.
It doesn't tell me what to do.
It just tell, I don't even know what the difference is between a 70 and an 80 in terms of any sort of absolute measure of, of UX goodness.
And so I actually have been turning to, um, places like Amazon where they use reviews, they use rating scales, they use dimensions.
These are all rubrics.
And what's most valuable are the comments that the people make.
And we have assessment tools now that we use Google forms for, or Microsoft forms.
They're, they're cheap, they're free, they auto score, and they take people less than five minutes.
And so that was the 5d rubric, which, uh, we are finding to be very, very valuable.
And we paired that with, um, Christian's pair lab, which was another way of gaining insight to what people were thinking.
Maybe Christian, you could explain a bit about that.
Yeah.
Thanks, Charlie.
And, and, you know, you want the qualitative side.
I need the quantitative side.
I need the numbers, right?
The numbers are really important to me.
Um, and, and, hey, listen, I share that view of yours that what is an 80, right?
Because if you don't have some context in it, if it's not in some way, a measure that you can really, really wrap your head around, um, it's hard to really work with.
And it's easy for people like, uh, say to, you know, look at that and manipulate it one way or another and stuff.
So I share that view.
And, um, what I was trying to do when I created this thing, I, I, I, the problem I was trying to solve that I had for many, uh, years in, in, in my experiences developing products was what do I do?
I mean, I don't know, I mean, okay, I do know, I mean, I have my intuition, I've got a bunch of data, you get a bunch of ideas or a bunch of features, and then somebody is going to ask you, what are you going to do first?
What's MVP?
And I think intuition's great.
And, um, I use it myself, but it only gets you so far.
And the chance of getting it wrong, sadly is, you know, is high and it can get higher when you have different inputs that are sort of dragging you in, in directions.
Certainly every project, uh, has a squeaky wheel of one form or another, maybe it's a stakeholder, maybe it's a customer and those squeaky wheels, they demand that oil and, you know, they tend to, to, to pull you or push you in different directions, which we're all may not, may not really be great for your customer or your product at the end of the day.
Um, so I thought about this a lot over the years and I came up with sort of an algorithm, you know, well, not sort of, but it is an algorithm, uh, that I was then able to create a tool around, which at the end prioritizes essentially ideas, but for me, product features, because I started the product, the beginning and after doing all the research, I have a list of, of feature ideas.
I need to know, uh, what is it that my, that my end users, my customers, my audiences really, really, really value over the things that they don't value kind of where, where, where, where does each, each part land?
And you can give to somebody a survey.
You can say, okay, here are 15 things.
Here are 20 things, put them in rank order by your most favorite to least favorite.
And you know what?
The top of that list is really probably pretty, pretty accurate.
The bottom of that list is, you know, probably pretty accurate of what they don't want.
And then you have this big, giant middle of mush, you know, like that instant oatmeal that you eat in the morning.
It's like, yeah, okay, what do I do with this?
And it's hard to make decisions off of that.
And I guess in some sense, the same with a product, uh, feature prioritization is true.
You've, you've really got to, uh, do what you can to get into the minds of people in, in a way that for me is quantifiable.
So what eventually became Paralab, uh, was, uh, an algorithm where I took ideas from social science and, and psychology, which is social science, uh, some machine learning and a couple of other things and said, uh, learn from those things.
If, what, what is the human mind really doing when it's trying to make choices, when it's trying to prioritize?
And it turns out that we're, our brains aren't really good when faced with big complex choice.
Um, the more you can reduce that complexity, the better we get at saying what it is we like and what it is we don't like.
And it turns out that the easiest choice is a choice between two things, a pair.
And so what Paralab does is it, is it takes a bunch of ideas and it splits them up into pairs and it puts them in front of customers, stakeholders, and it asks them, Hey, here's two ideas.
Which do you like more?
And, and, and by how much, and then I take all that data.
And by the way, it's also, uh, I figured out what, what really, really helps me is if I can segment that data by different customer groups or, you know, you could do it by anything.
Even, you know, I keep mentioning stakeholders and, and customers, which has been really, really useful to me because, uh, Paralab can show you or shows me where, uh, they agree, where they disagree.
It, uh, it, uh, uncovers hidden opportunities, uh, where you could possibly find things.
I guess if we were in a for-profit environment, which, you know, we're not at Princeton, but, uh, where you could monetize, uh, things that you didn't, didn't know were priorities for certain groups that, you know, that, that can be a path.
Uh, and it also puts out, uh, groups of, Hey, here are your high priorities, here are your medium priorities, here are your low priority things.
So I can, and they're comparable too.
So I can even say idea one, uh, is preferred by, uh, some segment group or, or by everybody, uh, 10 times more than say idea three and so forth.
And it, you can walk into a meeting with that, or let's say, you know, I, I've walked into meetings with that data in hand and having a language around the numbers.
Has changed the conversation and helped really bring everybody on board.
I think when we talk about inside Princeton, our internet, um, how did we get to a point where we were able to get buy-in from stakeholders and satisfy customers?
It's having those conversations with information or data or numbers, you know, with this quantitative data in which you, you can really discuss it in, in plain language.
Which has been incredibly, incredibly helpful.
Even with inside, uh, Princeton, the name, we did that with pair lab and we had, you know, a whole bunch of different names and sent them out to different people.
And this one popped up to the top and everybody likes it.
It's easy.
It's, it's memorable.
People know how to get to that URL, which is really a critical step in the user journey.
So it was, uh, you know, a really good kind of thing to be able to do quantitatively.
It was.
And in fact, it's got a qualitative aspect to it because it also crowdsources ideas.
And in, in the case of finding the name, for example, and we also found this with, with the feature prioritization, people suggested names and we feed them back into the study and then they get to vote on them again.
And, and so, because we know, uh, you know, when UX, what do we say in UX?
I am not the customer.
Right.
And that applies, I think very, very broadly.
I don't know all the answers.
I don't know all the, you know, all the say in this case, I don't know all the possible good ideas for names that we could have named this thing.
Uh, but my customers do and they suggested it and then we threw some back in, like we had a one bulletin helm, for example, which is a name of a tree.
Well, it was a tree because it was chopped down a long time ago on the Princeton campus, which was a very, very early bulletin board.
And I think it was in the 19th century and students would tack up, uh, posters on this, on this elm.
This, this actually turned out to be a dead elm.
They, they just use the trunk and somebody knew that and said, Hey, if you're, if you're building an intranet,
why don't we hearken back to a hundred plus years ago, something that was really important in Princeton's culture.
We fed it back in and it did okay.
You know, I think it ended up in the middle of the pack inside Princeton turned out to be number one.
So having that quantitative view upfront, being able to prioritize and compare things so that you can speak to people about them.
And then as, after we go through the process of, of building, having Charlie's 5d rubric at the end, and which to me says, Hey, did we do this right?
And where did we go wrong?
We can then feed that back into the development process.
And, and I think this is, this is one of the things Charlie taught me, which, um, really was, was a huge eyeopener.
And it applies to the 5d rubric, but, but the, I think the way we approach UX, thanks to Charlie is when you, when you go through your UX exercises and you find defects, you stop and create a loop.
Um, and Charlie, you're probably much better at articulating this than me, and maybe I should just shut up, but at least this is my, my view of, of what Charlie does.
Put, uh, take those defects, fix them, put them back in the loop and run them through UX and just keep doing that and keep doing that and keep doing that until you are really, really, really sure that, uh, you have figured out the solution to those defects.
And then, then you can move on.
The 5d rubric helps us really think through, uh, what those defects are and feed them back into the loop.
I don't know, Charlie, did I, did, did I articulate that, uh, accurately?
I think that is the basic, that sort of, uh, iteration and continuous improvement is the basis of user centered design in UX.
Um, I mean, we had one really good example of this when, uh, you know, we, we pulled together our first version of the intranet and we had done all of this great user research and we thought we had our assumptions clear and we understood user priorities.
And then we took the 5d rubric, which seems to be a really useful operational definition.
And, uh, basically we had, we gave them the, the, uh, early version.
We had them use it for a week or so, and then we asked them to fill out the, um, online forms.
And then we brought them together in a, in a focus group to discuss the, the comments that they made and what they had to say.
And we realized in the process of doing that, that we had made some serious errors of assumption and some of the designs we built were not what the users really wanted.
And that was amazingly clarifying.
And we went back and we said, wow, we missed that.
We did that wrong.
Let's fix that.
And we brought that back to the team and we said, here's what we found out.
And here's how we think we should fix it.
And we worked through the designs and we rebuilt it and we went back out there and people were delighted.
And that's because their concerns and their issues were dealt with in a structured way.
And that the analyses that we did gave us the qualitative information.
It wasn't just, oh, they think it's a 40 out of a hundred, but they said for every one of these dimensions, this is what I like.
This is what I don't like.
This is what I wish it would do.
And that guided us.
And that's the guidance that we as UX designers need an awful lot of.
Thank you, Charlie and Christian, for sharing all these insights and sharing all these stories.
And it's so interesting to see how you use those two frameworks that you developed, Charlie in the 5D rubric and Christian pair lab.
How you use these two frameworks to get quantitative and qualitative data, not only to work on your design, but also to align your team.
That's what you mentioned earlier, that this team alignment was a key part of your success to actually design something like your outstanding and your awarded intranet at Princeton.
Christian, you said you got this goldmine of information about different user groups.
I remember there was a surprising finding about your undergrad students, how they used your website.
Oh, sure.
They don't.
Okay, so one of the surprising things we learned is they really don't, except in certain cases.
And then we followed this up with some interviews, too, to confirm what we saw because, and maybe we want to talk about this, analytics tells you the what, analytics doesn't tell you the why.
So actually, I'll give you the whole picture here, where the analytics plus the UX combined to really inform us about what undergraduates were, I should say, not doing on the site.
And that is, they come to the site as perspectives.
That is definitely clear.
We saw that in the data.
They matriculate and they stop using the site.
What they do, we found out through user interviews, is very quickly, after they start their first semester, is they learn from their peers what apps and services they need in order to navigate the world of being a student, especially being a first-year student.
And they cobble together, essentially, this digital footprint.
That digital footprint does not include the course site.
And the exception, or we call, I'm sorry, Princeton.edu, we call it the course site.
The exception is when they appear on it, maybe we wrote a story about them, maybe a professor of theirs appeared or a friend of theirs appeared.
So there's a real social angle to the site for them, but it's very, very, very limited in scope.
And they go on their merry way through their four years, really not coming back to look at Princeton.edu, except in these other circumstances.
And the other things we found out, I think, which had been eye-opening, why we got to an intranet, was we were able to really, really clearly see what employees, and specifically staff and faculty, as separate segments, were doing.
And that was the first, I think, the genesis of the idea of, oh, we have a real need here, because the content on Princeton.edu is primarily geared for external audiences.
And yet, here we had this really, really clear set of behaviors that internal audiences were using the site for.
And we wondered, okay, if they're doing this here, can we do something better for them and give them, essentially, their own presence?
And that's what eventually became Inside Princeton.
So today, and this is, you know, in large part, thanks to that initial study, that pop-up study with the Google Analytics integration,
we're now able to provide, essentially, a site for external audiences, Princeton.edu, and then a site for internal audiences.
The content's different, and they're working for the audiences that they're designed for.
Maybe one last question.
If the audience had to leave with a single piece of wisdom from this episode, Charlie, maybe you go first.
What would you want it to be?
What is one single piece of wisdom?
Here's what I would like to say to them.
We have entered a new era of information technology.
Technology is everywhere.
Every single process, everything we do, there are little interactions.
Interactions that take place for people all through their days, all through their lives.
Every time there is an interaction, we create a user experience.
And that has made user experience really central.
We have been, to some extent, a craft with our own language.
We're really good at talking to each other.
And I think we do really good work as a discipline.
But we have not fully integrated into the larger processes of product development and product configuration and all of those kinds of things.
Now, UX is everybody's responsibility.
And what we really need to do is to find ways to operationalize user experience so that people who are not UX professionals can become full participants because UX is everybody's responsibility, not just the UX department.
How about you, Christian?
What would you like to give our audience on their way?
I think it's around listening.
And it has two dimensions in my mind.
So first is listening.
And I think the other way to look at it is basically keep your mouth shut.
And the reason is that you, as a single person, even as a team, your view of the world, your ability, your intuition, all these really, really great things that you bring to your product, to your project, that can get you really far, can only get you so far.
If you're not listening to your customers, if you're not listening to your stakeholders, you're going to miss something.
You're going to miss something really, really important.
Let them do the talking.
And, I mean, listen, we talk a lot about empathy in the process of UX.
And empathy really does start with listening, at least in my mind that it does.
So keep your ears open and your mouth shut in that area.
And the other one is to really partner, partner closely with UX and listen to your UX folks.
I think Charlie and I, we work well together.
But I think the things we've been able to accomplish, I think, are because we talk every week.
We have a standing meeting every week and we talk through stuff and we talk through the projects that we're working on.
We talk through other things that may not be related to the projects that we're working on.
For me, coming from the perspective of a product owner, I need UX really, really badly because in so many instances, UX acts to me as my conscience.
Charlie gives me a gut check so often.
He says, are you sure about that?
Are you sure about that?
And then as we start to talk through things, he'll challenge me and he'll get me to think.
And really where I think that UX has its greatest value is that it gets people to think, to really think through problems, think through them thoroughly so that at the end of the day, when you get to that end point, you have a much, much, much better result.
If you want to know more for Paralab, you can go to Paralab.io and I have a bunch of case studies on there you can look through.
And Charlie, where can people go to learn about 5D?
Well, I think we're on the Princeton site at ux.princeton.edu.
And also, I'm putting together a site, 5D-rubric.org, so that I can provide the material to other people who might want to use it.
Christian and Charlie, thank you so much.
You're an inspiring example of how product and UX can work together and build incredible products.
So thanks for coming on the NNG podcast and hopefully talk soon.
Tim and Therese, thank you for the opportunity.
I mean, Nielsen Norman Group has had an enormous influence on making UX really accessible to a very, very large group of people.
And, you know, the consistently very high quality, which I use all the time, I refer people to the materials that you're producing because they're always good.
And that's really great because we need more people to understand this UX maturity, I think, is the future of information technology.
I agree.
And, you know, let me mention one more thing about you.
I mentioned listening to Jacob years ago at the Journal, not long after I bought Don Norman's book, The Design of Everyday Things.
And I'm telling you, I have never looked at a doorknob the same since I read that book and it's on my bookshelf at home.
It's this eye opener.
I mean, just the stuff that all you guys are doing is really enriching for me, I know, for many, many other people.
And I'm also very, very thankful that you wanted to have us on.
Thank you very much.
That was Charlie Kreitzberg and Christian Noble.
We've included some links to Charlie's 5D rubric and Christian's Pair Lab in the show notes, as well as a link to our intranet design annual report, which features not only the Princeton team's great intranet, but many other winners as well.
We also have some links to other intranet and research-related resources in the show notes, which you can probably find on whatever podcast platform you're listening to.
Now, since we're talking about podcast platforms, now is a good time for our announcement.
This is our last audio-only episode.
But have no fear, because starting our next episode, we'll be uploading a video version for every installment.
So you'll be able to find all of those at our YouTube channel, which, if you don't already subscribe, just search for NN Group.
And whatever platform you're on, please do subscribe or follow so that you can be notified, whether that's a video or audio episode, whenever we publish the next installment.
But to learn more about UX in general, check out our website or sign up for our email newsletter at www.nngroup.com.
That's N-N-G-R-O-U-P dot com.
This show was hosted by Tim Noisesser and produced by me, Therese Fessenden.
All editing and post-production is by Jonas Zellner.
Music is by Tiny Music and Dresden the Flamingo.
That's it for today's show.
Until next time, remember, keep it simple.
Music is by Tiny Music and Dresden the Flamingo.