An example of a generative research interview
I always get asked about how I conduct research interviews, and each time I do my best to explain my techniques. I mention research plans, TEDW techniques, not asking about the future, and being careful not to ask leading questions.
However, I constantly struggle with how to describe my research interviews. Since I rarely follow a script for generative research interviews, it is hard for me to write down a list of questions because I generally base what I say on the participant’s response. I also tend to go slightly off-script during usability testing if I find an interesting thread to follow that might bring some solid insights.
After years of practicing, I feel most at home during research interviews. Yes, I definitely can still get nervous before a research interview, but generally, they are the highlight of my week. I think of research interviews as a window into someone’s life, and I am lucky enough to get a peek. For me, interviews are the coolest part of being a user researcher, which is why I get so frustrated not being able to answer one of my most received questions.
So, I decided to try something new that might help. Instead of writing out the basic structure of my user research sessions, I thought it might be more helpful to conduct an example research session and write out the exact script.
The idea
My fiance, a product manager, was kind enough to volunteer to be my research subject. We talk in-depth about quite a few topics, but we decided to talk about one of his passions: boardgames. In this example, I sit down with him to better understand how he decides to purchase a particular board game.
The interview
I only transcribed the first thirty minutes (out of about an hour), but it should still give a good idea. Want to listen as you read? Check out the sample interview here.
Introduction
N: My name is Nikki. I am working on a project right now to better understand how people think about purchasing board games and how they actually go about purchasing them. So I mean, user experience researcher at this company, and I wanted to understand your thoughts and your thought process on how you go about purchasing board games. So this session will only be about 30 minutes. And there are no right or wrong answers. So I want you to answer honestly. You’re not going to hurt anybody’s feelings with your feedback. Since we’re going to use this, your feedback and opinions improve our product and purpose. Do you have any questions before we begin, though?
C: That will make sense?
N: Okay, cool. Do you mind if I record this session before we start this?
C: Absolutely fine.
Diving in
N: Okay, cool. Awesome. So as I mentioned, I want to understand a little bit better how you think about purchasing board games. Um, so just to kick it off, could us think back to the last time you purchased a board game on your own, not a gift or anything, and kind of walk me through from when you first realized that you wanted or needed the board game all the way through actually purchasing it?
C: Yeah, sure. Um, so I really like board games. I’m not trying to say need as many as I have. But for me, it’s been like a hobbit in German. And I like to have kind of a diverse board game. Depending on who we play with, there are a few things that so I think maybe contextualize it a little bit. The last board game was called charter stone. I saw actually I was in a board game store said we had brunch with a friend, and we were walking past a board game store on the way back. And I dropped in. It was actually the box that first attracted me to it. I thought it was a really well-designed box. And aside to read it, and it was a legacy type board game. To me, it’s one of those that changes every time you play it. So what you do in one game will affect the next day. So normally, for I buy a board game, I actually also put alliances quite a popular website for games called BoardGameGeek, which is like the online home of board games. So on there, you’ll have forums discussing them, reviews for each game. And reviews will take things like complexity into account, learn the number of players, and the mechanics because there are many different mechanics in board games. So if you think of it a bit like a video game, you can, you know, you can build a base, or you can do a first-person shooter or car racing game. Like board games, you have different mechanics, and people prefer different mechanics on what they’re doing. So while I was in the store, I quickly looked it up. I saw it had a really good rating there. This one was made a little bit more of an impulse buy. And so I bought that one on the spot because there were the mechanics are involved. And it was a theme that I liked. The artwork was beautiful. And then at least initial reviews that I read on Board Game Geek really positive, so I bought it.
N: Wonderful. Awesome. So that was a lot of information. And so what I might do is I might actually go back and talk to you a little bit about what you what you just talked about. But when you say that you bought it, can you just describe that portion a little more. You were in a store?
C: I’d looked at a couple of ones. I picked it up and did the kind of browsing things on my phone. And just to see what Board Game Geek was saying what some of the big reviewers were saying about it. And it was generally positive. And so I picked up I wandered around the store for a little bit longer. It was more a case of what else might I buy? I said, there’s a board game store. There was nothing else typical that I really wanted that day or caught my eye. So I went and bought it was pretty straightforward.
N: Well, awesome. Going back way to the beginning of when you mentioned that you dropped into the board game store and saw it was there anything that kind of triggered you dropping into that board game store in particular.
C: It’s my local board game store. So so it’s the one that’s nearest to where I live. I bought a few games there before they’ve always really friendly, really nice, and there’s a huge selection, which is good. And we finished brunch, and it was literally two streets away from where we were eating, so it’s close by that as I might as well. And Saturday tends to be the day that I indulge in my hobbies. I have a couple of hobbies, so if I’m near the local store for that hobby, I maybe go in and just kind of browse and see what’s available. So that was it.
N: Okay, so it was location-based like you were close to it. So it was easy to stop by. And so you mentioned that the board game like the actual box caught your eye. And you said that it was like well designed, can you talk a little bit more about what you mean by well designed.
C: So this one’s actually quite a fun example. Because there’s almost nothing on the box, if you think about a box, you’d get with Apple, where you have an iPhone on the front, and then no information. It was kind of like that. The box is very clean, very white and actually stood out against a lot of the board games in the store, which is why I noticed it the first time. I then went to the back. There’s a brief summary of the mechanics, which I said were the kind of mechanics I liked. So kind of building strategy and building games, a nice preference of style, there are the little artworks, but there wasn’t that one was very clean and very crisp, and that I was pretty impressed by that. So I like legacy games. I like village building games. And obviously, at that point, it wasn’t enough for me to purchase it. I was really interested. And that’s when I went to Board Game Geek. And I got some information.
N: That’s really interesting. And so you said that that box was different because of maybe the clean and the lack of information on the box? Could you give an example maybe of what the board games normally have on the box?
C: Yeah, it depends on the box. You’d normally get a lot of information on the side of them. So you’ll normally have, it kind of depends. So on the front, you normally have obviously the title and some illustrations on the back or the side, you will tend to get kind of function information to how many players rough time how long it takes to play. And some of the mechanics that are involved with the dice base game is a card game is it, those kinds of things. So the mechanics I mentioned before. And then on the back, normally, you get a picture or an overlay of what’s included in the box, if I think about my favorite game, The Game of Thrones board game, or Imperial settlers. If you look on the back of that box, it gives you an overview of what it might look like when you’ve set. This box doesn’t do that because it’s more of a story mode. Because I said it’s a legacy game. You don’t know what cards are there. That’s actually one of the things when you open the box is a big thing says do not read the cards because you’re meant to play through the story mode. So your game evolves, depending on your decision. So you’re not allowed to say no, and it looks like and also by printing on the back of the box. It’s not representative because it’s going to look different for every player based on their decisions. So that one, I think, is very different from most of the games that I play. And I think after the election did a good way of giving a hint of mystery about what’s involved.
N: That’s super interesting. So you mentioned that, since there wasn’t a lot of information, you went on to Board Game Geek to go and kind of look up the reviews and the different mechanics and more information. But before we get into that, and how you went through that process with charter stone, do you also go on to Board Game Geek when you get all that information on the box?
C: Yes. Oh, Board Game Geek is a pretty good community. It’s quite a diverse community. It’s like the authoritative source of like board games online. Like that’s the place you go to check it. For example, if you’re playing a game and you’re not sure about a rule, or you feel that it’s two rules that are conflicting, almost everyone I know will go to the board game geek website before they go to the official, the official website because normally it’s easy to find other people obviously because they the official companies can test the game like 100,000 times. So people do it. And also, if someone’s got an official answer from a company, they will consolidate there. So Board Game Geek is pretty authoritative. It’s also very detailed in what you’re looking for. And the website’s not the prettiest, I’ll be honest, but it’s very functional. You find what you need to find, which is great. To go back to your question, yeah, I will use Board Game Geek pretty much every time. I will also use it sometimes if I’m at home and browsing online for a board game. I will use maybe five or six big reviewers that I will also see what they’ve said about again.
N: Cool. Awesome. Thank you. That’s really helpful. Could you talk about a time when you have seen a game online or in a store and purchased it without going to board game geek?
C: Yes, but only when I’ve played the game before. For example, if I played with a friend so Game of Thrones, the board game is actually my current favorite board game and has been for several years that I bought off Amazon. Immediately I finished I went to I got introduced to it by one of my other friends who went down we spent the day playing. I laughed. I was like, I’m buying this game because I played it. And I had the experience. And I use I guess I use this on Board Game Geek as a proxy, I proxy other people’s experiences to be like, would this apply to me? And if I played it with them, and I like it, I will buy it without checking Board Game Geek.
N: Okay. Interesting. So it seems that if you’ve had the experience of playing the game, you feel more comfortable purchasing it without visiting board game geek.
C: Yeah, because at that point in my experience, I know I like it. As I said, I think with Board Game Geek, I use it when it’s new for me. You can read the packaging, maybe less overcharged soon, because the packaging, but generally, you can get an overview of how the game works, doesn’t say how the game plays, which is very, which is a very different thing. And there’s a lot of board games out there. They’re not all fantastic. And there are multiple reasons that that will be the case. So I tend to trust the community of gamers about what they like what they do, like, there may be a chance I’m missing out on an awesome game that I would think was fantastic because the community doesn’t like it. But generally, I found that the games that I buy, the process of buying games generally seems to work for me.
N: Okay. And that’s the process of looking and looking it up on Board Game Geek.
C: Yeah, it’s about more validating. So I’ll come across it in bed. So a friend will mention a game, or I’ll see online I see in a bookstore for game geeks, which is my way of getting more information. What is the experience of playing this? Occasionally, I get games recommended to friends. And if a friend has recommended the game to me, I’ll check it out. But I probably still will check Board Game Geek if I haven’t played it myself. Okay. Okay. Even if it is a recommendation from a friend, you are more likely to. Yeah, I think it depends. I’ve got many friends who play board games, and they’ve all got different preferences. I mean, there’s a couple of friends who like or enjoy almost the same games. So if they recommended them, I probably would take it off the bat as a smart purchase.
N: So looking at board game geek since it’s interesting to see what other people say. Cool. Yeah. And you’ve mentioned a few times that you’re looking for, like the experience. Either like the experience that you had yourself personally or on board game geek, you’re kind of looking for what the experience might be. Can you talk a little more about how you figure out what that experience is, especially on board game geek?
C: Yeah, so Board Game Geek actually has a good way of quantifying this. So they have two main numbers. One is complexity. And one is geek rating. Complexity is given a rating between zero and five by how complex the game is. And then geek rating, which is how good it is how high it sits in the ranking. For example, I know my two of my favorite games, Game of Thrones and imperial settlers, I know where they sit on that rank. So I benchmark all the games against those. Of course, it depends on what I’m buying them for. If it’s like a party game, so the kind of game you played with people, then that’s different. So there’s a common one that people would know on things like Cards Against Humanity would be considered a party game, or resistance, which is ones where you have to pretend to be a secret agent, those kinds of things. For those of us who want an easier and a lower complex rating, because you the idea is that you find a group of people probably haven’t played this before, and you sit down and get started playing. But something like the Game of Thrones has a much higher complexity rating. And that’s a bit more for kind of people who like hardcore games. You will sit for five or six hours, probably more. As long as the corresponding geek rating score is high on it as well. You also get an overview of the mechanics. I don’t really like dice-based games. I don’t like the chance. I prefer strategy games where I can plan. Yeah, roll the dice. That’s when I’m not really into that. For example, those put me off because I’m not in control. I wouldn’t think I played a really good game instead of I got really lucky, which seems which is not as much my interest.
N: I don’t know if you have the answer to this question, but those dice games, do they vary in complexity as well, or could they potentially vary in complexity as well?
C: Good question. I don’t play as many dice games. So I don’t think it was well informed. It depends on the game. If the dice mechanic is one part of the mechanic in your game, you can still have a very high complexity rating. It’s a game where you just roll the dice, and then yet your outcome is probably lower. Monopoly is considered a reasonably straightforward game; everyone’s played Monopoly. But it’s pretty straightforward, right? You roll the dice; you move that far around. That’s a sensitive premise. And there’s a lot of luck involved in that, where you land. I don’t enjoy that as much. There’s a Battlestar Galactica board game made by the same company that makes Game of Thrones board game that has a dice mechanic in it. But many other things are going on at the same time. It just introduces the random element of space battle actually, or events happening. It’s a reasonably small mechanic, but it can be a bit annoying when it goes the wrong way. But it exists. And it’s a really good game. The dice mechanic is one that I’m not a huge fan of in it. But I understand in that situation, you have to have a degree of randomness. Otherwise the game, the game wouldn’t work. You don’t have that degree of randomness, for example, in the Game of Thrones board game. Right? You can be strategic and play it through and control as much as you can always control the other players around you.
N: Cool. All right. Awesome. Thank you. So kind of, kind of going back to, to that to the, to the two ratings, that they use the complexity and the geek rating. Yeah, the Geek score. Does that kind of roll up into like an overall rating? For the board game? Like, Is there like a zero to five stars? Zero to 10? stars?
C: Yeah, so the geek rating goes from zero to ten. I can’t remember where the ranking score comes from. It doesn’t come from a combination of complexity because you can have a game that is like 10, out of 10, and very complex, but people still love it. And it’s worth the complexity. So the complexity core doesn’t, as far as I know, factor into it. I think the geek score is the main driver where it sits in the ranking. For example, I know that Game of Thrones sits at like 7.6 on the geek score, and 3.65 and the complexity score. So I said that’s my benchmark. But for example, I know some games appear higher than Game of Thrones in the geek score, I just don’t enjoy playing. It’s a bit like there are games that rank high, but they just, they’re just not my style. Arkham Horror, for example, is one of them. It should be a similar game, but I don’t like it.
N: Okay, interesting. So that brings me actually into the next question that I have. Do you weigh both the complexity rating and the geek score? Pretty equally when you’re deciding? Or when you’re looking at something?
C: No, no the geek score for me is normally more important. The complexity rating is a lot of great material online, like, you know, how to play and guides for playing. So the complexity doesn’t really bother me. And the places only concerned for example, if you if I found a party game that was highly complex, I probably wouldn’t buy that because I don’t want to teach everyone. I would want to sit down and play and have fun. I don’t spend an hour explaining the rules to ten people. That’s not fun for anybody involved. It’s one of the reasons that Game of Thrones, for example, has such a large onboarding time. If you can spend maybe half an hour explaining the rules after they’ve watched a 20-minute tutorial video. And then after the first game, they’ll probably have how to play and the second game is gonna be loads of fun for them. Like Game of Thrones has a long onboarding time, but for me, it is well worth it. It’s a really great game. If you’re into the theme of hard to game like resistance or Exploding Kittens, you want to be able to explain it in 10 minutes, and the complexity rating is way lower than the complex games like Game of Thrones, Imperial settlers. Their complexity score is higher. There are lots of things going on with it. But it’s a great game for more hardcore gamers. I wouldn’t play that with my dad because my dad’s not as much into it. My friends who were in my board game group, they’re super into that kind of level. And then you’ve got some people that may be a bit high for, but there are other games that will sit in the middle.
N: Awesome. Cool. That’s, that’s perfect. If I’m understanding correctly, the Geek score is a little bit more important than the complexity rating.
C: Correct. And it depends on the situation.
N: Yeah. Okay. Cool. And so you mentioned that you benchmark generally against Imperial settlers and Game of Thrones. Can you talk about what you mean by like, benchmarking?
C: Yeah, it’s more just to anchor myself, so I know, hey, these are two games I really love. This is roughly what the community thinks of them in terms of complexity and in terms of rating. When I go and look at another game, either community saying these are better or worse. It’s just an anchor point really. I just take the games I know and try to compare them but it’s not always right. I said, there are games that rate higher, that I don’t enjoy as much. And some games really lower. So it’s more just a starting point.
N: Okay. Can you give me an example of a time where you’ve seen like a lower rating and have actually enjoyed or bought the game? Hmm, good question. It doesn’t have to be like an exact name.
C: Either way, I can tell you about a time that there was a game that I was ready to buy, but I didn’t enjoy it as much. Let’s see. Yeah, there’s one called Seven Wonders. That did not addict me or hook me as much as Game of Thrones or Imperial settlers. It’s a great game. I enjoy playing it, but never got into the rhythm as much as the other two. And I’m not totally sure why. Maybe I don’t know. Maybe I like the themes of Imperial settlers and Game of Thrones slightly more. But yeah, so that was a game that was rated higher by the board game community. But I know that I enjoy it less. It doesn’t entice me to play as much. I’m thinking hey, why don’t I play today? It’s never the game that jumps to mind like Imperial settlers or Game of Thrones. There is my default like, yeah, I really love these games.
N: Hmm. That’s really interesting that you say that. Because you’re saying it’s not that you necessarily enjoyed it less, but it doesn’t jump to your mind. And it doesn’t like entice you. And I know you said you just said like, I’m not exactly sure why. But I’m wondering like, is it because you’ve cited like mechanics and themes before as ways that you choose board games? And is it would it be something like that?
C: You get really bought into what you’re doing. So Game of Thrones is the one you play for five or six hours. You play a particular house, and I’ve read the books and TV shows. So if I’m and I lose a footman, I’m like, I know, I’ve lost the footman. Yeah. Similarly, with Imperial settlers, it’s a much shorter game than Game of Thrones. But you pay for your faction and you grow your empire. With Seven Wonders, great game really strategic, different ways of playing it. But the games are much shorter, I think like an hour tops, and there are the different factions and you get less bought into them? I think at least from last time I played, it’s not, you know, you don’t get as invested. Perhaps that’s what it is.
N: So when you say that you’re not as invested or as bought in? Can you explain a little bit more what you mean about being like invested in that game? Yeah, invest in the success of that faction that you’re playing, or that team that you’re playing? Or that civilization that you’re playing?
C: I guess there’s more. There’s more persistence on the End Game of Thrones in Imperial settlers, like your decisions feel like they take longer. I think when I think back, like, I deserve almost a story that goes with Imperial cycles in Game of Thrones, like you play through. And you can remember that time that someone did an amazing move. And like backstabbed in Game of Thrones, or Imperial status uses or there’s always certain like combinations of cards that your opponent played and the damage was really good. Like you learned a lot from it. Where Seven Wonders is very replayable it’s always a different game. But I very, I really remember any time I was like, Damn, that was awesome. Or I was like, Oh, right, that we were trying to get those from the other games. So maybe it’s that kind of book that I was thinking about.
N: Okay. Um, I don’t know if you can answer this. But if you could change something about seven wonders to make it more like the Game of Thrones games with the Imperial settlers games that you’re talking about? Like, what would you change?
C: I have not played in about a year. So it’s hard to remember. And sometimes I don’t want every game that I play to be as intense, for example. Seven Wonders, I think is, is a very strategic game. It removes a lot of the emotional elements that you may get in Game of Thrones. It’s so nice, I’m not sure I’ll change it. I do enjoy it. When I said I do enjoy it when I play it. It’s just a different type of enjoyment.
N: That’s interesting. Okay, so going back to the actual like decision that you made to buy it. So that seven wonders were rated higher?
C: Higher, yes.
N: And Game of Thrones?
C: Okay, I think complexity is definitely lower. Definitely low.
N: Um, okay, so it was higher on the geek rating. So that that kind of pushed, did that push you towards that?
C: Yeah. We bought this game because I was back in London with a few of my friends where we played a Game of Thrones in Paris, it was a lot together. And we considered buying another game. There was no point buying Imperial Settlers since we already had a copy of it. We went to the board game cafe that we used to go to in London, and we spoke to the guy there. And he actually said Seven Wonders is very popular. Consider it. We checked out on Board Game Geek and it is rated highly. So we took it home. And we spent most of the weekend playing. So I think it was a combination of things like we wanted something different than what we had before. Like I didn’t want to buy another copy of the game I already had. We wanted to do something different. And situationally, that’s when I had the opportunity to buy it. And I’ve seen someone mention it loads of times on Board Game Geek. So it’s a game that I was aware the board gaming community enjoyed. So it seemed like a good opportunity to buy it.
N: Okay, and so you bought that one. Yeah, took it home, and then you guys played it?
C: Yeah.
N: Okay, cool. Um, so do you think that there was anything that you could have found on Board Game Geek? Or like any sort of information that would have like helped you maybe not buy this game? Or you know, reconsider the purchase because it sounds like that geek rating weighed pretty heavily as well as like that recommendation.
C: It’s like it’s a very good game and I see why it’s rated highly in the community, but it’s not something I like. The whole idea of board games is that you find something that you like or enjoy anything that I would have made me change my mind not to buy that or buy something else. That’s a great question. i guess both
N: Let’s tackle not to buy that first. Or if it since it seems like this game doesn’t seem like a bad game to you if you want to think about something that was like more highly rated that actually, you didn’t end up liking, if you have an example of that maybe that would be easier to come up with.
C: Yeah, I honestly like all the games I have. I just like them for different reasons and in different situations. I can’t think of one honestly.
N: How about something similar to Seven Wonders had a higher rating, but you regret purchasing it if you have one of those.
C: There isn’t any game that I necessarily regret purchasing. I don’t play a number of games as often as I shouldn’t have maybe not got the full euro value out of the game that I’ve purchased. So Game of Thrones it’s not a cheap game to buy and with all the expansions probably over 100 euros, but I played over 100 games of it now. There are other games that I’ve played like five or six times that would have cost 50 euros. But the times I played them, I’ve enjoyed them. They’ve been, they’ve been a bit more situational.
N: Okay. So it doesn’t seem like there’s any sort of regret or like, I wish that I could return this.
C: No, there’s no.
N: How about Have you ever felt like you wish you could exchange it for a different game?
C: No, so I quite like having a collection. I liked having the diversity collection. I’m the same with books. Like I’ve got a large book collection.
Tear it apart!
I am no stranger to making mistakes, and I am sure I made plenty during this particular interview. I encourage you to use this example to highlight what you might do better and things you would like to avoid in your interviews!
Since I interviewed my fiance, I was a little more informal, but this is my general tone and flow. Here are a few things I missed and would change:
- Getting written NDA and consent from the participant
- Including warm-up questions. My favorites are: “what are your favorite hobbies?” “what do you do in your free time?” “have you tried anything new recently?”
- Saying “awesome” and “cool” quite a lot (internal facepalm)
- Avoiding the future-based questions
Basics on how to conduct card sorting
And why it is one of my favorite methods.
There are a few reasons I love card sorting so much. The method is simple, effective, and fun to participate in (for both the moderator and the participant). I was surprised when I realized I hadn’t written an article solely on card sorting. It deserves it’s time to shine.
I first learned about card sorting about six years ago. I was talking to a colleague, and we wanted to redesign our platform for a different set of users. We were currently supporting hotels and hotel staff, but we wanted to see if we would also help staff working in residential buildings (think high-rise apartment buildings in New York City).
Now, of course, there was a lot more that goes into a conversation like that then merely bringing up card sorting. We had done qualitative research with the residential building staff already. We were trying to think of how to pivot the platform. How might we use the existing skeleton to support the differences in workflow?
I was still relatively new to the field of user research, and it was back when I believed there were only two methodologies: discovery research and usability testing. I hadn’t done too much experimenting with other methods for a few reasons. I hadn’t learned about theses methodologies, I didn’t feel confident with them, and I had no real guidance on how to implement this.
We set up our goals for the research session. We wanted to accomplish a few things:
- Understand what current features made sense to these users, and which were missing
- How the users categorized the different features and information on the platform
- How users would imagine the platform to look in terms of navigation and nesting
- The flow users went through in their day-to-day
Many great goals. After a week, we started usability testing. Spoiler alert: it didn’t work. I tried to ask them how they would navigate the current platform and stumbled around with future-based questions that were most likely leading. At one interview, I remember getting out a piece of paper and just having the person write down a list of what they wanted.
Major fail.
We wanted a mix of understanding users’ mental models and information architecture. That is where card sorting shines. After that last interview, I went back to the drawing board, Google.
At the time, I didn’t know what information architecture was, but I eventually put together enough of the right search terms to find card sorting. After reading a few articles, we went to work.
What is card sorting?
Card sorting is an activity in which you give cards to a participant and have them order the information in a way that makes sense to them. These cards can have information written on them, can be blank or a combination of the two.
These three scenarios line up with the three different card sorting techniques there are. Each of them has a time and place. My favorite is mixed card sorting.
Closed card sorting
Closed card sorting is when each participant gets a set of cards with information already written on them. They are limited to using these cards. This approach is very evaluative and is best when the terminology or concepts are well-defined and established. It can give you apparent patterns on the cards. The significant cons for this, however, is that you might not fully understand the user’s mental model, as they have to conform to what you wrote on the cards.
Open card sorting
Open card sorting is pretty much the opposite of closed card sorting. Participants create categories and concepts of their own and then order them. Open card sort is great for exploratory work, and understanding how users relate to, organize, and define different concepts. It can lead to a better understanding of terms and definitions. However, the con for this approach is that the patterns are usually not as clear as with closed sorting.
Mixed card sorting
Mixed card sorting includes cards with predetermined information, but allows the participant to create new categories or concepts that may be missing. When I do mixed card sorting, participants can “edit” the pre-written information. Mixed card sorting is my favorite because it allows for both evaluative and generative work, although you may still run into the same cons of fewer distinct patterns.
How do you choose?
First off, determine if you are in a generative or evaluative phase of the project.
Would you feel comfortable/confident in defining terms, concepts, and categories for your users? Have you conducted previous research that would help you correctly identify them? If yes, you can then use closed card sorting to evaluate the patterns better.
If you are starting from scratch, or don’t feel confident in creating cards, I would recommend going with either open or mixed card sorting. This way, you can better understand how users are defining these areas, and then go ahead with a closed card sorting after that.
To decide between open and mixed is up to you. If you know that some concepts have been validated, but others not, then go for the mixed, if you want to go all out discovery, open will suit you best.
Regardless, make sure you always capture the running commentary as the participants are placing, writing, or organizing cards.
How to run a card sorting test
Many different components go into a card sort, so I split them up into three main steps. I also referenced articles that go more into detail when it makes sense, such as with defining the content of the cards.
Before the card sort
Now that we’ve talked about this magical methodology. Let’s run one.
Define your research goals. As wonderful as card sorting is, it has its time and place (as does every other methodology). To be effective, you need to make sure your research goals match up to your methods.
The common goals that align with card sorting are:
- Evaluating a users’ mental model on the information architecture of a product/service
- Understanding how concepts relate to each other in the mind of the user and the hierarchy of them
- Uncovering definitions, terms, or ideas that might be missing or misunderstood
Decide what card sorting method you will be using. Depending on the phase of your project, choose between open, closed, and mixed. If you aren’t sure, I always recommend mixed. If you are super tight on time, you can do a closed card sort and then, in the end, ask users what they would change.
Writing the cards. If you are doing a closed or mixed card sort, you have to write the cards. Knowing what to include in a card sort is the tricky part. What you write on the card depends on the project. The most important part is not to include more than 40 cards in your card sort. Since deciding what to write for a card sort is an extensive process, I wrote another article on that here. That article is all about how to write cards for card sorting.
Setting up the card sort
- Usually, card sorts last 60 minutes, so plan for that amount of time
- Recruit 15–20 participants for this kind of research
- Get the cards ready
- Make sure there is enough space to spread out all of the cards on a table or ensure there is enough wall space if they are hanging them
- If you are conducting your card sort online, make sure you have sent the link to the participants and tell them they will need internet access
- Get someone to help you during the session to take notes and observe what you might miss. They can also help with recording the meeting
- Go through the session to practice before with a colleague to make sure everything makes sense before inviting the participants
Moderating the card sort
- Give the participant the set of cards (or have them out on a remote tool). Walk them through what the session will be like and explain what you are looking for them to do. In each scenario, mention that any cards that are confusing or they are unsure about can be put to the side.
- Closed/mixed card sort: Tell the participants you are Explain that you are looking to understand how all of the items on the cards relate to each other. Explain that they can group the information in a way that makes the most sense to them.
- Open card sort: Tell the participant that you are trying to understand what should be in your product/service. Then explain, once they’ve finished naming the cards, that you want them to group the cards in a way that makes sense to them. Once the cards are grouped, you will ask the participant to name the grouping. Then explain that you will ask for a name for each group of cards once the participant has arranged them.
- Request that the participant thinks aloud during the session, so you can fully understand their thought process behind their categorizations. If the participant struggles with thinking aloud, be sure to prompt them regularly
- Thank the participant, give them a chance to ask any questions, and provide your contact information. Be sure to mention the timeline for any incentive.
- Finally, email the participant within 24 hours to thank them for their participation in the study and with any incentive you had offered
Analysis
Once you finish the card, it is time to analyze it! To keep the length of this article reasonable, I will write a separate article on how to analyze your card sorting results, including some templates.
Recommended tools and resources
- Notecards and pens
- Trello
- Optimal workshop
- UXSort
For more engaging discussion, please sign up for the User Research Academy slack group! And check out User Research Academy for free resources.
Concepts, processes, and terminology — oh my!
The important terms and ideas to understand as a user researcher.
I recently ran a webinar on the topic of transitioning from academia to user research. A lot of my webinar focused on the importance of understanding the teams and business you will be working in as a user researcher. Although you can create a perfect portfolio piece in a silo, it is imperative to have the ability to relate to how an organization runs. I have spoken to many UX research recruiters and what they look for, in addition to user research skills, is how you would integrate into a team.
It is challenging to demonstrate how you would fit into a team without a general understanding of the basics of how tech teams work. And without the jargon or terminologies that are common in the tech and user research industries. I recommend doing research on how tech teams work, how businesses work, and how user research can fit into organizations. This will make you stand out to recruiters and companies, as someone with a real passion and motivation to go above and beyond.
A note about business and user research
The real truth of the matter is that companies exist to make money. I hate always coming back to this fact, but it is reality. And we have to face reality. The majority of what teams are looking to accomplish in an organization exists to make that company money. We need to be okay with this fact and understand where user researchers fit into it.
User research enables teams to make the best decisions. When organizations make better decisions, it tends to impact essential business metrics for a company, such as conversion rate, revenue, retention rate, acquisition rate. When teams make informed decisions that create a positive impact on users, they can better contribute to the mission of making the company more money. So teams rely on you for fast research to make these better decisions and positively impact the organization as a whole.
This is really important to understand because it gives you context into what your mission is on a team, and in a larger organization, outside of the day-to-day tasks.
Now on to teams…
Fitting into a team is essential for user researchers. It is crucial to understand every single role you will be working with. These internal stakeholders will become your internal users. They will be reaching out to you with questions and digesting the research you deliver. By understanding what they are trying to accomplish, you can better empathize with them and tailor research insights in a way that is optimized for their day-to-day. This also helps you figure out where and when user research is relevant in a project.
Ideally, as a user researcher, you work with all different roles and departments. However, the colleagues I have worked with the most are:
- UX/UI Designers
- Product Managers
- Developers
- VP- or C-Level Executives (from Product, Engineering, & others)
- Account Managers
- Customer Support
- Content marketing
The list could go on, but these are the major roles I have seen could benefit from user research. Take the time to learn what each of these people does and what they are trying to achieve. With this knowledge, you can positively impact their lives with research, which will lessen the need to beg them for buy-in.
Processes, Jargon, and Terminology
I will not go in-depth with each piece of jargon, terminology, and process, but will provide a list of which you can use to then deep dive into. This also does not include every process, piece of jargon, or terminology out there, but it is a starting point for learning about the different concepts user researchers commonly encounter.
Processes:
By understanding processes, you can understand where user research fits in.
- Agile
- Scrum
- Sprints
- Ceremonies
- Retrospectives
- Backlogs
- Standups
- Kick-offs
- Lean methodologies
- Waterfall
- Kanban
- Requirements
- Product management
- Product roadmaps
Concepts:
By knowing these concepts, you can make sure you are including them and doing robust research
- Business metrics (ex: click-thru rate, conversion rate, retention rate)
- OKRs / KPIs
- Product analytics, by using tools such as Google Analytics and Firebase
- Validating/disproving hypotheses
- Prototypes, wireframes
- ResearchOps
- Usability Metrics
- A/B testing
- Workshop moderation/facilitation
- Iteration
- Ideation
- Design thinking
- Recruitment
- QA (quality assurance)
- How Might We Statements/reframing solutions
Terminology & Jargon
With knowledge of terminology and jargon, you can talk more confidently with many different departments about all you can help them with
- Generative research (or discovery research, problem-space research)
- Usability testing
- Diary studies
- Card sorting
- Guerrilla research
- Jobs to be Done
- Heuristic evaluation
- Competitive analysis
- Affinity diagrams
- Participatory design
- Mind map
- Customer journey map
- Mental models
- Empathy map
- User scenarios
- Surveys
- Screener surveys (link coming)
- Personas (and here)
- User stories
- Implementation
- Deployment
- JIRA
I hope these are helpful in getting you started with learning how tech and product teams/organizations work in the real world, as well as important terms to understand as a user researcher. The more familiar you are with these, the more you can conjecture how you could impact an organization as user researcher.
Especially big thanks to Nielsen Norman Group and Atlassian for documenting so many of these terms, making my life easier when finding great links!
If you are interested, check out more free content and my offerings on www.userresearchacademy.com and join our slack channel
If you liked this article, you may also enjoy:
User Research practice problems — updated
Because practicing whiteboarding, research plans, survey writing, and portfolio pieces makes perfect.
Updated March 2020
After teaching many classes and mentoring many students, the mantra “practice makes perfect” has become one of my most common sayings. During my courses and mentoring sessions, I only have so much time to go through practice examples. With that limitation, I thought it might be helpful to brainstorm a list of problems that many people can use to practice.
Some of the problems on this list are from my brain, and others from various things I may have seen over the internet. I’m sure some people have imagined these problems before me, and I will do my best to link to anything I take directly from another website. I will include these resources at the bottom of the article.
I will do my best to update this with new problems as much as possible. If you have any suggestions, please email me: nikki@userresearchacademy.com
How to use the list
The reason I put this list together is so you can practice various skills:
- Whiteboarding sessions (read how to tackle a whiteboard session)
- Writing research plans (read more about research plans, with a template)
- Survey writing (read about writing survey questions)
- Research portfolio pieces (how I structure a user research portfolio)
And anything else you think about!
You won’t have all the answers since these are static problems, and I can’t answer all the questions you may have about each problem. I also didn’t go into crazy detail for each problem. In this case, write down the questions you have or the assumptions you make. And then how you would tackle these. This is good practice regardless, as this happens a lot in real life!
Now on to the problems…
List of practice problems:
Problem statement
How can we better understand the way people travel?
Project brief:
Your client is a leading travel brand who sees new opportunities emerging in the digital space around how people plan travel. They want you to focus your research on one group of the travel experience, either leisure travel or business travel. The client expects you to deliver insights on how people travel, in order to start designing something relevant for users.
Problem statement
How can we better understand the way people adopt a pet?
Project brief:
Your client is a non-profit animal shelter who wants to better understand the process people go through when trying to adopt a new pet, specifically in the digital space. They want you to focus your research on people who have adopted a pet in the past 1–2 years. The client expects you to deliver insights on how people adopt pets, in order to start designing a digital solution relevant to users.
Problem statement
How can we better understand the way people digitally communicate with each other?
Project brief:
Your client is a messaging platform that wants to better understand how people digitally communicatee with each other. They want you to focus your research on people who use platforms such as iMessage, WhatsApp and Facebook Messanger to communicate with others on a daily basis. The client expects you to deliver insights on how people communicate with each other, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand the way people find their way around a new city?
Project brief:
Your client is a tourist agency who wants to better understand how people find their way around a new city. They want you to focus your research on people who have traveled, at least once, to a new city in the past six months. The client expects you to deliver insights on how people navigate a new city, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand the way people organize their wardrobe?
Project brief:
Your client is a retail platform that want to better understand how people organize their wardrobe. They want you to focus your research on people who purchase new clothing on a regular basis (at least once a month) and care about organizing their wardrobe. The client expects you to deliver insights on how people organize their wardrobe, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand the way people set and stick to their goals?
Project brief:
Your client wants to better understand how people set and stick to their goals. They want you to focus your research on people who have set a goal in the past six months and have either followed it through or have not been able to complete the goal. The client expects you to deliver insights on how people set and stick to their goals, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand the way people keep up to date on the weather?
Project brief:
Your client wants to better understand how people stay up-to-date with the weather. They want you to focus your research on people who currently use weather apps, or listen to the weather channel. The client expects you to deliver insights on how people stay up-to-date with the weather, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how to make women feel safe when traveling alone?
Project brief:
Your client wants to better understand how women currently travel alone, and the concerns they have. They want you to focus your research on women who have traveled alone at least once in the past six months. The client expects you to deliver insights on women traveling alone, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how people order food takeaway online?
Project brief:
Your client wants to better understand how people order food takeaway online. They want you to focus your research on people who have ordered food online at least three times in the past month. The client expects you to deliver insights on online food ordering, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how people decide and plan to move?
Project brief:
Your client wants to better understand how people decide to and plan to move. They want you to focus your research on people who have moved to a new house or apartment in the past six months. The client expects you to deliver insights about moving, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how to reduce homelessness in major cities?
Project brief:
Your client wants to better understand how to reduce homelessness in major cities. They want you to focus your research on major cities that are dealing with a homelessness crisis (ex: NYC, LA, SF). The client expects you to deliver insights about how to reduce homelessness, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how people find and keep roommates?
Project brief:
Your client wants to better understand how people find and keep roommates. They want you to focus your research on either someone who has just found a new roommate, or people who have been roommates for more than one year. The client expects you to deliver insights about how people find and keep roommates, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how people recycle?
Project brief:
Your client wants to better understand how people choose and understand recycling. They want you to focus your research on either people who have recycled consistently over the past six months or people who have tried recycling and gave up. The client expects you to deliver insights about how people choose and understand recycling, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how people buy clothes online?
Project brief:
Your client wants to better understand how people choose and buy clothes online. They want you to focus your research on people who have bought clothes online at least five times over the past six months and have returned at least three items. The client expects you to deliver insights about how people buy clothes online, in order to help them to start designing a relevant solution.
Problem statement
How can we better understand how people respond to and gather information about a pandemic? (COVID-19 bonus!)
Project brief:
Your client wants to better understand how people respond to a pandemic. They want you to focus your research on people who have lived through a pandemic and have experienced a lockdown/quarantine during that pandemic. The client expects you to deliver insights about how people respond to and gather information about a pandemic, in order to help them to start designing a relevant solution.
Resources
- 100 Example UX Problems by Jon Crabb
- 100 Days of Product Design by Susan Rits Design LLC
- Designercize by Mezzotent
- WeWork Interview Exercises
If you are interested, check out more free content and my offerings on www.userresearchacademy.com and join our slack channel
Which UX Research methodology should you use? [Chart included]
Start with this question, and all else will follow.
When I first started as a user researcher, I believed there were two types of user research: usability testing and discovery research. I was right, those two types of research do exist, but they are not the only methodologies a user researcher can use to answer questions. But, alas, I was young in my career and believed every problem could get solved in one of these two ways. Am I needed to test a prototype? Usability testing. Am I required to figure out the content? Usability testing or discovery research (depending on my mood). Do I have to figure out how a user feels? Discovery research.
For a while, it was all very black and white. It was a simple time, and one where I believed I could unlock all the answers by using these two methods. I didn’t have to think about information architecture, long-term studies, testing concepts, surveys, clickstreams, A/B testing, the list goes on. Sigh, it was an effortless time.
However, like most people on their career path, I finally hit a wall. I encountered a research question I could not solve with these methods, despite my trying. I did try, and I failed miserably at this.
We were trying to redesign the navigation of our platform and where different features appeared on each page. Notice, we didn’t start with a problem statement or a goal. We started with a solution of a redesign. Also, we already had the redesigned screens. With this in mind, I figured a usability test could solve all of our problems, or validate our pre-defined solutions.
In the end, we decided to do a usability test with the new navigation and layouts. It was clunky, but our participants got through. However, when we released the changes (not too far after the user tests), we found a lot of issues and had to revert to the old designs.
So what could we have done better?
There were many things we could have done better, but I’ll focus on the three biggest ones we could have changed:
- Define why. Defining the why is step one of any research project or initiative. We have to understand why we are doing the research. I often include the why from the user perspective, as well as the business perspective. The why can be a simple paragraph explaining the reasoning behind the project.
- Start with a problem statement. When you start any user research project, it is essential, to begin with a problem statement. A problem statement is a central question that has to be answered by the research findings. In this case, we started with a solution, which led us down the wrong path. By starting with a problem, you are much more likely to solve an actual user need. Learn more about reframing a solution to a problem statement.
- Consider the objectives. By starting with goals, we can understand what we are expecting to learn from the research. We can ask ourselves what we are trying to learn and what we would like the research to achieve. Objectives keep your research on track and align everyone with an expected outcome. Learn more about writing objectives.
Since we skipped straight to testing a solution, we missed these crucial steps. By doing this, we ended up with a suboptimal experience for our users because we didn’t understand what their problems were and what they needed. It is an easy trap to fall into, but easy to avoid by following those steps. Instead, the process should have looked much more like this:
Defining the basics
With the knowledge from above, the outcome of that research project would have been much different. However, I still had the idea that there were only two types of user research: usability testing or user interviews. If only I could have told myself about all the different methodologies out there. Instead of time traveling, I will rewrite history to showcase the process I would go through today to choose a method for the same project.
The project was: Redesigning our hotel concierge platform navigation and information architecture of our features for hotel concierge.
Step 1 — Define why you are doing the research.
We have seen some users struggling when trying to navigate through our platform. Through user research, we have observed users hitting the cmd+f option to find what they are looking for to make the process faster. They are unable to find nested information as they are not sure where to look in our navigation. Also, users have been employing several hacks as opposed to using the features we have built.
Step 2 — Define a problem statement.
I am a hotel employee trying to fulfill a request for more towels for a guest, but I am unable to find that specific feature, which makes me feel frustrated. With this, I have to enter a generic request, and then write in what I specifically need and hope someone reads my notes.
Step 3 — Define the objectives.
- Understand the general workflow of users and when they need access to particular pages/features
- Discover how users are currently using the product
- Uncover the limitations and pain points users are facing in different situations with the platform
- Learn about any potential improvements in workflow, information architecture, or missing features
Choosing between generative and evaluative research
Now here is where things became fuzzy, and where I want to dive deeper. Although now I have a good understanding of which methodologies would have been best for answering the question and objectives, it didn’t always come so intuitively.
Since then, I have seen four significant types of objectives:
- Understanding more deeply about people’s day-to-day lives, regardless of a product/service
- Discovering how a person uses a product/service and how that product/service impacts their daily lives (positively and negatively)
- Evaluating how a product/service works when a person is using it (pain points and joys)
- Assessing how a product/service could be structured to impact a person’s daily life positively
By breaking these objectives down, we can then choose between generative research and evaluative research.
- Generative research allows a deep understanding of who our users are (inside and outside of a product/service). We can learn what they experience in their everyday lives. It allows us to see users as human, beyond their interaction with a product/service.
- Evaluative research is about assessing how a product/service works when placed in front of a user. It isn’t merely about functionality, but also about findability, efficiency, and the emotions associated with using the product/service. Many people think evaluative research = usability testing, but it goes much further than that
- Hybrid research is a combination of generative research and evaluative research. The example I’ve used above can very well end up being a hybrid method. Hybrid research helps us simultaneously understand our users, as well as how a product/service is performing. Now, this is not magic and, since it covers both spaces at once, it does not go as deeply into a place of understanding or evaluation. I will focus on hybrid research in a future article, as it is a more advanced technique.
Once we understand what type of research we are looking to conduct, we can better understand which methodology ties best to our objectives. I have included a chart of methods I have used, and how they relate to the above goals I have realized.
If you are curious, for the research project I mentioned above, I would have suggested the following methods:
- Contextual inquiry
- Walk the store interview
- Card sorting
Now, none of this is an exact science, and your opinions may differ on this, which is fantastic! I would love to hear about how you approach this differently.
Interested in more user research? I teach an Introduction to User Research Course and am available for one-on-one mentoring. Check out the User Research Academy. Please join the User Research Academy Slack Community for more updates, postings, and Q&A sessions
5 ways to reframe a solution to a problem statement
And why this is essential for user research.
Very rarely do I get handed a problem statement. Whenever colleagues want to engage with me in a research project, they have already landed at a solution. I am not saying this is bad because solutions are a comfortable place to start. They are easy. However, they limit our potential and possibilities. I’ll give an example of something I hear frequently:
“We want to build a share feature for our users so they can share their trip ideas or details to friends and family. Can you research this for me?”
I’m sure this statement resonates with many user researchers. It is a tough situation to be in, and one that happens far too often. If this is the type of statement you are consistently getting from colleagues, you have an excellent opportunity to teach a new mindset. You can teach the mentality of approaching ideas from the problem-space before solution-space.
Why is reframing the solution important?
I’m going to pick on Google, for once, because I believe Google has enough self-confidence to take a small hit. Let’s talk about Google Glass.
To be upfront, Google Glass was a failure. The product did not succeed in the slightest. There are several different reasons for the flop, but we will focus on one in particular: Google Glass did not solve a problem for users.
There were many assumptions made about Google Glass, which were not validated with users. The conjectures led to many questions and, ultimately, to a product that did not correlate with the user’s needs or goals. There was a lack of clarity on:
- Who would use Google Glass
- When/where people would use Google Glass
- Why people would use Google Glass
- What value Google Glass brought to people
These are a lot of essential questions that were left unanswered. With this approach, there is a substantial likelihood that products will fail. If there is no value to users, they won’t use it.
And the best way to determine value? Start with a problem, not with a solution.
By understanding the problem-space before coming up with a solution, we allow for:
- Understanding a user’s actual problem and need
- Brainstorming ideas that are relevant for users
- More possibilities to help with a particular problem
- A higher likelihood of a successful product/feature
The best way to understand the problem space is by conducting generative research sessions with users. Although I won’t cover generative research here, please take a look at my comprehensive guide to generative research.
Picking apart the solution
We’ve established how important it is to start with a problem rather than a solution. However, many times, we are faced with a solution, as in the first example. Let’s take a look at that again:
“We want to build a share feature for our users so they can share their trip ideas or details to friends and family.”
Let’s assume there has not been any user research done on this topic of “sharing.” There are a few things that are not ideal with this statement:
- There is no clarity on what the problem is we are trying to solve
- We jump directly into a solution
- Users wanting to share their trip ideas or details is an assumption
- Sharing might not be the only or best solution
- Our thoughts should come from generative research
But, lets base this scenario in reality. We’ve received this solution and need to test it. We can go in one of two directions:
- Usability or concept test the solution
- Take a step back and consider the problem
I know the latter isn’t always possible, which is why I included the first option. Sometimes the idea/feature/product is too far down the production line, and we have to do damage control. The best we can do is usability test the solution to understand what we can improve in further iterations.
However, sometimes it isn’t too late, and we can take a step back and consider the problem. We can ask the question: “what problem is this solution trying to solve?”
Tools to reframe the solution -> problem statement
This kind of question may be much easier for researchers to think about since we are used to doing this daily. For our colleagues, it might not come as naturally. Luckily, there are a few different methods we can employ to help us reframe the solution.
‘How Might We’ (HMW) Statements
This method is very trendy in the UX world, and for a good reason. ‘How Might We’ statements allow us to reframe our insights and thoughts into a broader context. We use these types of statements because they can get us thinking about the problem from many different angles, and they open the door to creativity. They also ensure we are thinking about an actual user need versus just coming up with cool ideas. Let’s take the sharing example and recontextualize it with HMW statements:
Investigative Stories
Investigative stories enable us to become detectives when thinking about a problem. With this method, you employ a journalist-type view of the situation. By looking into all of the different aspects of the problem, you can get a holistic picture of what you are trying to tackle. After you answer all of the questions, you can create a story of your user, which helps to understand the actual user need/problem.
Unpacking Assumptions
This is a technique I have been practicing and teaching for many years now. It is one of my favorites as it plays very well into my Buddhism practice. Before I begin thinking about solutions, I list all of the different assumptions I/we have about the user. It consists of everything you think you know. If we take the example above, the list might look like this:
- Users want to share trip details with friends and family
- Users are booking travel for a group of people
- Users need other’s opinions on their trip options/details
- Users find their current method of sharing details painful
- Users want a new way to share trip details with others
The Six Thinking Hats Model
Dr. Edward de Bono originally introduced the ‘Six Thinking Hats’ model. This method examines problems/concepts from many different viewpoints to get a holistic understanding. Getting colleagues to engage in this kind of exercise can be complicated, but also extremely rewarding. The thinking hats/roleplaying game gets people into the minds of users. Each person wears a hat:
- White hat: Facts
- What information do we have?
- What hasn’t worked in the past?
- What information is missing that we need?
- What are the weaknesses? - Green hat: Creativity
- What are the other angles we are missing?
- What are the alternatives?
- What are the next steps? - Yellow hat: Benefits
- What are all the benefits of the different options?
- What solutions would work?
- What is the best-case scenario (for users and the business)? - Black hat: Risk
- What are the different risks of each option?
- What solutions would not work?
- What is the worst-case scenario (for users and the business)? - Red hat: Feelings
- How do the options make you feel (from the user’s perspective)?
- What do you like about the options?
- What don’t you like about the options? - Blue hat: Process
- Where are we now?
- What other work needs to be done?
- What is the next step?
Research Plan Template
One of the most significant pieces of advice I can give is to create a template that prompts colleagues to think of the problem statement before they even come to you. An additional way I do this is by sending a research plan template for colleagues to fill out before we meet to discuss features. This template starts with the problem we are trying to tackle, rather than the solution. With this, you can start discussions from the problem-space. Check out my research plan template.
Writing the problem statement
Once you have done these exercises, you can write some problem statements. I mentioned the ‘How Might We’ formula above, but I also use some other problem statement formulas:
- I am (persona/role) trying to (do X) but (barrier/problem) because (x), which makes me feel (emotion)
- I am a mom trying to book a flight ticket home for my daughter, but I don’t know her college schedule, so I am unsure which dates to book for her, which makes me feel frustrated.
- I am (a persona/in a situation) who needs a way to (user need) because (current problem)
- I am traveling with a group of friends and need a way to coordinate everyone’s schedules so that I can pick the best date/time for everyone
You don’t have to use the exact formula (I deviated above), but it is an excellent framework to keep in mind. Try to avoid proposing solutions during this phase, as it is an easy trap to fall into. Keep your focus on the problem and facts.
Overall, getting people to shift from the solution-space to the problem-space is a considerable challenge, but one worth confronting. If you are continually urging others to think about the problem before the solution, they will start to adapt to that mindset. By instilling these practices, and with some time, you can transform the mindset of an organization to approach ideas from the problem-space. Not only does this make your life as a user researcher easier, but it also turns the company into a much more user-centric culture.
If you liked this article, you may also find these interesting:
- User Research isn’t black & white
- How to write a generative research interview guide
- Benefits of internal user research
- User research plans — with template
Interested in more user research? I teach an Introduction to User Research Course and am available for one-on-one mentoring. Check out the User Research Academy.
Please join the User Research Academy Slack Community for more updates, postings, and Q&A sessions
Is it time we move beyond the NPS to measure user experience?
Why the NPS sucks and what to use instead.
I get an overwhelming sense of grief and anger when I see the Net Promoter Score (NPS) being used as a beacon of light for companies. I have heard so many people praising the NPS, and using it as a legitimate way to measure customer satisfaction and success of a company. In fact, sometimes, it is used as the only way to measure these metrics. But, I really believe, the NPS should not be the basis of making critical decisions.
As user experience and user research gain traction in the tech/product field, there are so many additional ways to gather feedback, measure satisfaction, and pinpoint where you are disappointing your users. The NPS feels like a very archaic metric that stuck around simply because it is easy. And, to be honest, the other suggested metrics in this article are more difficult than the NPS, but there is a reason behind that.
What exactly is the NPS?
If you have ever tried out a product or service, you have probably seen this question before:
On a scale of 0–10 how likely are you to recommend <product or service> to a friend or colleague?
Your NPS is the accumulation of people who use your product or service taking the time to rate whether or not they would recommend your product or service to others. Based on the score they give your product or service, they fall into one of three buckets:
With this information, you are able to calculate a score that you can measure across time. The formula to calculate the score is:
Net Promoter Score = % of Promoter respondents minus % of Detractor respondents
Remember, passives are kept out of calculations, so any scores of 7–8 essentially do not exist in terms of scoring.
As an example, let's say you received 100 responses to your survey:
- 40 responses were in the 0–6 range (Detractors)
- 30 responses were in the 7–8 range (Passives)
- 30 responses were in the 9–10 range (Promoters)
When you calculate the percentages for each group, you get 40%, 30%, and 30%.
To finish up, subtract 40% (Detractors) from 30% (Promoters), which equals -10%. Since the Net Promoter Score is always shown as a number (not percentage), your NPS is -10. NPS scores can range from -100 to +100.
In addition, the NPS appears to do things that other business metrics don’t or can’t. It is:
- A single question that is easy to understand
- Easy to calculate and measure
- Easy to track any changes over time
- Widely accepted in business
Why the NPS is a suboptimal metric
There are a few reasons why I think NPS is not the most ideal metric to use as a single metric. I am in no way saying, “never use the NPS again!” If you want to continue measuring NPS, just understand what you are really measuring and consider additional metrics to track alongside the NPS.
- Why is recommendation a sign of satisfaction? There are so many different reasons why one might or might not recommend a product/service to a family member, friend, or colleague. Sure, we could assume the mental model of “I am satisfied with this product/service, therefore I would recommend it,” but this still leaves many situations unanswered. What if it simply doesn’t make sense to recommend the product/service to these people because they never use anything like this? Or maybe you are unsure about the value since you haven’t used it a lot. Maybe you hate it, but someone else in your life would benefit from the product/service. There are many different contributing factors to recommendations, and they don’t always lead to product success and customer satisfaction
- Recommendations are contextual and subjective. Similar to the above statement, whether or not someone gives a recommendation can be completely subjective and based on context. Would I recommend a movie I just saw to others? Maybe. It depends on the person. If it was a comedy, I would potentially recommend it to my friends who like comedy, but not my friends who like action or thrillers. I could have also had a terrible night with bad popcorn, and lumpy theater seats, causing me to dislike the movie because of circumstance, rather than quality. Or I could have simply been having a bad day, and that impacted the experience I had with a product/service, causing me to give a bad score. There is a multitude of reasons why recommendations are extremely contextual and subjective
- NPS calculation overshadows success. The weird calculation of the NPS score does a great job of hiding improvement and success. The above example gave us an NPS of -10. Now, let’s say we work super hard to make some great changes to the product/service. We know we are on the right track because we are using user research insights to help give direction to these changes (biased, I know). So, it turns out we have increased those scores, and we receive many more 6’s and 7’s. For some reason, the NPS counts the 6’s as 0’s (detractors) and the 7’s don’t count (as they are passives). So, ultimately, our NPS is still -10, despite the improvements we have made. This simply does not make sense
- You do not know what type of user is responding. The NPS does not care or categorize the respondents to the survey. So, we don’t know if the people who are responding are power users, new users, or even our target personas for the product. If we have spent time building for certain types of personas but are getting scores from other people out of our scope, that can completely skew the results. It is also easy to skew the results in the other direction — you could be getting good reviews from people who you are not at all trying to target
- Cultural differences could impact your score. If you are a global company and are collecting responses from many different countries, it is extremely important to study the NPS data per country. There are vast differences in how cultures respond to these types of surveys, which can really mess up your data if you are lumping them together. It is best to keep an eye on the country or region your responses are coming from and to separate that out
- An 11-point scale is really large. The NPS gives a very large range of responses as an 11-point scale. In fact, it is one of the largest scales. The distinction between the numbers is not at all clear, and most likely does not make sense from person-to-person. You and I could have the same exact experience but I would give the product a score of a 7, and you give the score of a 6. What is the difference between 6 and 7? It is very hard for respondents to understand this difference, and to choose a meaningful response. It is almost the same as closing your eyes and picking between two numbers
- There is no data on ‘why’ someone gave a particular rating. Aside from all of these, my biggest gripe with NPS is that there is no understanding of why someone gave a particular score. Receiving a rating can be rather useless if you don’t understand the motivation behind the score. Without the ‘why,’ there is no way for a company to determine how to improve a product or what direction to move in. Perhaps it was an error beyond your control that led to a lower score. Without understanding the ‘why,’ you can spend much time and energy trying to guess what went wrong and how to fix it
- It is completely future-based. I do my best to avoid asking future-based questions. We want to focus on what people have done in the past, as most of us cannot predict our future behavior (think of all the gym memberships that go unused).
What to do instead (or in addition to)
Now that I have bashed the NPS as a single-source metric, I would love to give some alternatives to the NPS or to measure alongside the NPS. We all know people love numbers and measurements and there are some other questions we could be asking (hint: they might not be as easy, but could be more effective). You can even continue to include the positive/negative aspect of the NPS.
Frustration:
- How delighted or frustrated were you today? — Extremely delighted (+2), delighted (+1), meh (0), frustrated (-1), extremely frustrated (-2)
- In the past X weeks, have you felt frustrated with our product/service? — Yes (-1), Unsure (0), No (+1)
Helpfulness/happiness:
- Have we been helpful today? — Yes (+1), Unsure (0), No (-1)
- Have we made you happy today? — Yes(+1), Unsure (0), No (-1)
Loyalty:
- In the last X weeks, did you recommend us to a friend or family member? — Yes(+1), No(-1)
- Have you ever recommended us to a friend or family member? — Yes(+1), No(-1)
- Were you recommended to us by a friend or family member? — Yes (+1), No (-1)
- In the last X weeks, have you considered [canceling your subscription, switching to another provider, etc.]? — Yes (-1), No (+1)
Satisfaction:
- How satisfied are you with [company X]? — Very dissatisfied (-2), Dissatisfied (-1), Neither (0), Satisfied (+1), Very satisfied (+2)
- How easy was it to complete your order online? — Very easy (+2), Easy, (+1), Neither (0), Difficult (-1), Very difficult (-2)
Finally, always leave an open-ended text field asking something along the lines of “how could we improve?” This adds to the qualitative input you can receive to give you a better indication of what went wrong and the context of the situation. This will help in providing more clear direction towards product/service improvements.
Ultimately, user experience and satisfaction cannot be boiled down to one singular number. Let’s keep an open mind about what we measure, how we measure, and why we are using specific metrics, so we are best able to make better, well-rounded data-driven decisions. And, don’t forget to add in some qualitative research as well!
If you liked this article, you may also find these interesting:
- Asking about the future in user research
- A case for beta testing
- First-person surveys in user research
- Benchmarking in user research
Interested in more user research? I teach an Introduction to User Research Course and am available for one-on-one mentoring. Check out the User Research Academy.
Please join the User Research Academy Slack Community for more updates, postings, and Q&A sessions
Ten emotion heuristics: how to read a participant’s body language
As user researchers, we take words very seriously and place most of our importance on what our users are saying. While this is, in fact, where our attention should be placed, we should also consider the body language of our users during our research sessions. I’m not talking about “rage-clicking” or clear sighs of frustration (although those are important too), but more of the subtle body language.
A while back, I wrote about how important a user researcher’s body language is during an interview, and now I want to write about the other side: the user’s body language.
Why is this important?
One of the number one principles (at least one of mine) of user research is:
watch what users do versus just listening to what they say
When we focus all of our attention on what people are saying, we can miss what they are actually doing. And, unsurprisingly, what people say and what they do can be quite different.
I have a perfect example of this. I was conducting usability tests on a new flow we were thinking about implementing. One of the main tasks our users had to do was download multiple images at once. We didn’t make this easy and, previously, our users would have to hack downloading them at once. Once we finally had the resources to tackle this project, I was super excited to test our ideas.
We had one idea in particular we thought was a sure winner. I couldn’t wait to test it with users. We showed it at about ten usability tests and, luckily, I had my observation mindset on. Many of the users said they really liked it. They could finally download multiple images at once. HOWEVER, the majority of users, while telling me that they liked it, struggled with understanding the flow and completing the tasks. In fact, three users clicked on many different areas and appeared visibly frustrated, but still said it “wasn’t bad.”
Had I just been listening and using words as data, I would have pushed forward this idea. Instead, I noticed the struggles. During the interview, I was able to dig deeper into the frustrations beyond the surface. This allowed us to better understand where we needed to improve the UX.
It was after that particular test I started looking more into how to observe user’s behavior and body language during research interviews. I wanted more than retrospective or current self-reporting measures. I searched and found a method that relies on real-time observation of behavior and coding of participants’ facial expressions and gestures. Its creators, Eva de Lera and Muriel Garretta-Domingo call their method the “Ten Emotion Heuristics.”
The Ten Emotion Heuristics:
The heuristics are a set of guidelines to help assess what a user is feeling beyond self-reported measures. As mentioned above, there are times where users actions and words do not match up, and you can use the below heuristics as a way to understand what the user is really feeling, beyond the feelings they may be aware of.
- Frowning. If a user is frowning, it can be a sign of a necessity to concentrate, displeasure or of perceived lack of clarity
- Brow Raising. When users raise their brows, it can be a sign of uncertainty, disbelief, surprise, and exasperation. While surprise isn’t always negative, we don’t necessarily want our users to be surprised or uncertain of the experience on our platform
- Gazing Away. When a user gazes away from the screen, they may feel deceived, ashamed, or confused. They could also very possibly be bored with what is on the screen in front of them
- Smiling. A smile is a sign of satisfaction in which the user may have encountered something satisfying or joyful
- Compressing the Lip. Seeing a user compress their lips is a sign of frustration and confusion. I see this a lot when a user intends to do something, but it does not work, causing frustration and anxiety
- Moving the Mouth. If the user is speaking to themselves, trying to understand or complete a task, this indicates them feeling confused or lost in the experience
- Expressing Vocally. Vocal expressions such as sighs, gasps, coughs, as well as the volume of the expression, the tone or quality of the expression may be signs of frustration or deception.
- Hand Touching the Face. If a user is touching their face during the interview, they could be tired, lost, or confused. This can also indicate a high level of concentration and frustration with a task.
- Leaning Back on the Chair. When a user (or anyone, really) leans back in a chair, it is an indication they are having a negative emotion and would like to remove themselves from the situation. This generally shows a fairly high level of frustration
- 10. Forward Leaning the Trunk. Leaning forward and showing a sunken chest may be a sign of difficulty and frustration with the task at hand. At this point, the user may be close to giving up on a task or experience.
How to use these heuristics
The great thing about the ten emotion heuristics is that they are all 100% observable and cost-effective. The best thing you can do while learning these is to practice. Here is how I have learned to incorporate the emotion heuristics in every one of my interviews. These don’t have to be done step-by-step, but could be thought of that way!
- Memorize the different heuristics
- Practice the heuristics with others — both doing them and observing them
- While practicing with others, write down which heuristics you observe and compare notes
- Record each of your participants and assess heuristics AFTER the interviews — compare notes with a colleague on what heuristics you both found and at what points
- Observe and make note of heuristics during the interviews. See if you can dig deeper during those. Assess the interview after as well to see if you were accurate
- Rinse and repeat until you feel confident observing and noting the heuristics in real-time
Some things to note while you are practicing:
- Always record your participants during the interviews. Even when you are a “pro,” you might miss some instances. Make sure you can see their facial expressions in the recording!
- Have a colleague with you to compare notes (especially in the beginning)
- Take it slow! You won’t learn or notice these all over a short period of time
What can we do with this information?
There are a few different ways I like to use the emotion heuristics, and they all have really benefited my analysis of research studies.
- Observing behavior over words. What people tell you and how they act can be different. The emotion heuristics can give you an indicator of how someone is really feeling at a given moment
- Some negative emotion heuristics appear outside of the concepts that we are testing. This helps give an indication of where we might need to improve the overall user experience of the product, outside of what we are testing
- You can measure if there are trends with certain emotion heuristics across the experience. Are the majority of participants exhibiting particular emotions during one task or flow?
- If many negative emotion heuristics are surfacing during a task, flow, or experience, you can prioritize fixing that issue higher than others
- By identifying the different cues across the interview, you can rate whether the participant’s experience was overall positive or negative.
- Give scores to tasks and overall experiences, which can help with calling attention to issues and prioritization
Although it might sound simple, using these emotion heuristics can be quite tricky. You might get a participant who doesn’t display many expressions or a participant who displays too many at once to count. User research isn’t an exact science and you will never get the perfect participant. The best you can do is practice and observe these signals participants are putting out. They won’t be the answer to all your questions and, sometimes, they may lead you down the wrong path, but they are another tool to put in the user research toolbox.
—
If you liked this article, you may also find these interesting:
- ACV Laddering in UX Research: A simple method to uncover the user’s core values
- User Research Isn’t Black & White and how to navigate the grey area
- Benchmarking in UX Research to test how an app or website is progressing over time
- How to Assess Your Research Interviews: A framework to continuously improve
If you are interested, please join the User Research Academy Slack Community for more updates, postings, and Q&A sessions :)
Should user researchers give feedback to teams?
Where does user research feedback start?
The other day, I saw a bad prototype. It came up during a design critique. We were going to start user testing that very prototype the next day. I wrote down my notes and mentioned it to the designer.
The problems I saw were:
- Typos
- Simple design flaws (different colors for the same hierarchical information)
- Information that made no sense in the context
For me, these were easy things to fix. I wanted to give the designer this feedback for two reasons:
- To make the design cleaner, and to encourage attention to detail
- To ensure we were getting the right feedback from the users
The second reasoning behind this, for me, is more important. When I present a prototype to users, I would rather not spend the precious time we have with them confused about easily changed information. Or information that doesn’t matter in the test. It can be a big waste of time, and participants can get stuck on these small inconsistencies.
What happened in this case? Unfortunately, the designs didn’t get changed in time. We tested the older designs with the issues I mentioned. While we did get some great feedback on the user experience, there was some time wasted on those smaller inconsistencies. I had to explain them and wave them off as small mistakes that don’t matter. However, it made the interviewing experience seem less productive and prepared. Almost every user recognized the changes I had asked to be made.
Regardless, again, we still received great information from users, but it felt clunky to explain away the prototype. Now, I know prototypes are supposed to be far from perfect, but this felt beyond the usual prototype spiel I give.
So, my biggest question in this case: is it okay for the researcher to request the changes I did? Or is it on the researcher to perform the interview in a manner where these inconsistencies don’t matter to the user?
When should user researchers give feedback?
This particular case made me question when, and at what level, user researchers should give feedback to our teams. I generally give feedback during the following opportunities:
- During the idea/concept phase
- During the prototype phase
- After a design is completed
- Synthesis from research sessions
I’m not entirely sure if this list encompasses every opportunity, but it was my starting off point. I don’t want designers or other team members to think I am an expert in UI/UX (or any other field, for that matter) or that I am overstepping boundaries.
Here is how I give feedback at each of these steps:
- During the idea/concept phase. I do my best to ensure teams come to me with ideas very early on in the development process so we are able to test the viability of the idea with users. When they come to me with ideas, they are generally solutions, rather than problems. I ask them to come up with the problem they are trying to better understand and the questions they would ask during to find out more information. Some teams have come to me with a fully developed idea, which I knew would not stick with users, or solve any pain points. In this case, I was new to the company, so I was forced to test it with users. It proved the point that it is important to do some upfront user testing before we come with fully built solutions based on assumptions. Now, when people come to me with solutions, I request they go back to the drawing board and start with a user problem, and questions they would like to ask.
- During the prototype phase. This is similar to the example I gave above with the prototype. I try to get a look at all prototypes before we put them in front of users. I will have the designer walk me through each screen, and I will point out any small inconsistencies. This gives the designer a second pair of eyes on the designs and helps ensure the design and flow make sense. Prototypes can still be “messy,” as in low-fidelity, but they need to make sense. We don’t want to waste time having users comment on small things that are insignificant to the usability test.
- After a design is completed. This is where the whole feedback concept starts to get tricky for me. Once a design has completed user testing and is off into the wild world of being “live,” what do we do? Since it was already user-tested, do we have the right to give additional feedback? For this stage, I will wait a bit and then follow-up with any feedback we are receiving on the particular design (or feature). If the design did not go through user testing, I will test at this stage and do a heuristic evaluation to give some additional feedback to the designer.
- Synthesis from research sessions. And finally, synthesis. Maybe for some, this is the most straightforward but, for me, it can get complex. We all talk about how synthesis is one of the most important parts of the user research role, but rarely is it discussed in full. At this point, how I understand synthesis is as follows: we digest and analyze the research sessions, and then give “actionable recommendations” on what should come next. What does this mean? Are we telling people what to do? What are we recommending? This brings me to my next point…
At what level should we give feedback?
Since the first three opportunities for feedback are much more straightforward, I will focus on synthesis for this particular case. I have the following questions when it comes to giving feedback (specifically targeting synthesis):
- What should we be producing? Action items/recommendations?
- What are the action items/recommendations?
- How far should we go with action items/recommendations?
I truly believe user research isn’t about giving people answers, it is about giving people tools to better contextualize something. We aren’t meant to “tell people what to do.” So, with this in mind, what are we supposed to be writing in terms of recommendations?
When I tested the aforementioned solution (which was a huge feature), it was glaringly obvious our users would not find it helpful or useful. And they would not pay for it. There were a few aspects of it they liked, but, by in large, it wasn’t sticking with them. They simply were not interested, and would rather have the company work on other features or improvements.
When I received those results, I wasn’t entirely sure what to do. The solution was already half-way built, and the team had spent a good amount of time working on it. I decided to give my honest recommendation: stop working on this immediately and pivot to working on other, more impactful, areas. I gave some ideas on how we could change the idea to suit our users better, but it should not be a high priority. I was lucky to have enough people on my side to be met with little resistance.
However, when it comes to these tests, I always wonder what level we should give this feedback and these recommendations? I often state the recommendations as problems instead of solutions:
“User is unable to locate the “pay” button or move to the next step” versus “move the “pay” button to higher on the page or make it a bolder color.”
This still gives enough flexibility for someone to make a better decision, without me telling them exactly what to do. However, as I mentioned, I also made an honest recommendation of not continuing on with a product. I’m not entirely sure what the balance is, but I am sure there is one. But I would still love to know…
How do you give feedback as a user researcher?
If you liked this article, you may also find these interesting:
- Burnout as a User Researcher
- User Research Isn’t Black & White
- Benefits of Internal User Research
- How to Write a Generative Research Guide
If you are interested, please join the User Research Academy Slack Community for more updates, postings, and Q&A sessions :)
Assessing your user research career level, seniority, and path
Career paths are extremely important topics to think about, especially in the context of a more niche field, such as user research. While there is some information online, it is primarily for the broader field of UX and much more focused on the UX design career path. Planning the next 1, 3, or 5 years of your career is never easy. But trying to do so without a lot of guidance or mentorship can be even more difficult.
The reason I know this is because I have recently been trying to figure out what my next step is, and have also failed to plan effectively in the past. This lack of planning has led to confusing and less than ideal roles and situations. Knowing what your current level is, and where you want to go with your user research career, really helps you make sure you are in the best role for growth.
User research career levels
There are a few different levels of user research, and (fortunately) they generally follow the same kind of trajectory as other careers. I have also put a general amount of years of experience but this, of course, can vary. I have seen people in certain roles with much less, or more, experience than is generally “recommended.” I believe an employee’s skillsets and level of maturity are far more important in determining a career level than the number of years someone has been in the field.
I have also measured these against the following areas of impact a researcher could have on a company level:
- Operational: Deals with the day-to-day function of user research
- Organizational: Handles how the company understands and ingests user research
- Strategic: Aids the company in making strategic decisions based on user research
User researcher levels
- Research coordinator: Supports the product team throughout the research life cycle, including scheduling, recruiting efforts, participant communication, streamlining research operations and team communications
- Junior user researcher: Embedded in a team to carry out user research activities. They have some practical experience but need regular guidance and training to produce their best work and develop their skills. They generally work in combination with a more senior user researcher
- Mid-level user researcher: Embedded in a team and responsible for planning and carrying out user research activities. They are able to work independently on a team, without too much guidance
- Senior user researcher: Able to plan and lead user research activities in larger teams and on more complex services. They build user-centered practices in new teams and align user research activities with wider plans to inform service proposition. They may supervise and develop other user researchers to assure and improve research practice
- Lead user researcher: Leading and aligning user research activities across several teams. They ensure that teams take a user-centered, evidence-based approach to service design and delivery. They develop and assure good user research practices
- Head of user research: Leads user researchers in an organization and attracts and builds talent. They are an expert practitioner who can define and assure best practice, influence organizational strategy, and priorities, and collaborate with colleagues across a company.
Different career paths
There are two main career paths for user researchers, and they are similar to those of other industries. In general, you can go one of two paths:
- The individual contributor
- The manager
The biggest question I would ask when determining one of these two paths is to understand: “do I want to help others develop the skills I have?” or “do I want to continue to hone my skills as a research practitioner?”
As a manager, of course, you may still be able to engage in the more tactical side of the job (ex: actually conducting research), but usually, you operate as a people manager, mentor, and strategic partner. As an individual contributor, you will most likely be able to contribute at an operational, organizational, and strategic level. The biggest difference here is whether or not you are mentoring and managing others.
If you feel okay stepping away from the day-to-day and are looking to mentor others in the field, management might be your best bet. If you want to become a super expert in your field, I would stick with being an individual contributor. Of course, the best way is to simply try. As of right now, I have been a senior-level individual contributor and have finally decided to make the leap to management. Let’s see :)
How to assess your level
Now, these descriptions and charts are all generic. They have to be. There is no one description that will be perfect for all. Many user researchers have unique paths and experiences, so it is hard to make generalizations. However, for the purpose of this article, I do my best to help you categorize and see where you can grow.
Here are the steps I take when assessing my level:
- Audit my own skills. I always start by listing out all of my skills and my level of confidence in these skills (“low, medium, high” is totally fine). I also list the skills I would really like to learn next, that I find important for moving to the next level
- Look at the skills in the different levels. I then look through the skills in the level I think I am, and then the levels directly below/above. I also stalk people on LinkedIn at the level I think I am, and the surrounding levels. I see what their experience and skill sets are, and then compare that to mine. I also do a lot of networking and talking to researchers at all levels to get a more concrete idea. Actually talking to other researchers is the best way to do this!
- Consider my past experience. As I mentioned before, skills are one thing and experience is another. I always consider my past experiences, roles, and responsibilities. For example, when I was starting out, I was a UX Research intern but was expected to operate as a more junior/mid-level. With this, I was able to gather different skills and experiences that pushed me almost straight from Intern to mid-level. It is important to take this into consideration.
- Think about my level of maturity. This is especially important when you start to get into the senior/lead positions in the field. It is extremely important that you strongly consider how mature you are as an individual. More and more people will be depending on you on a consistent basis, and you will have much more pressure and feedback coming your way. It is okay if you have not yet had the training to best handle these situations, but they are very critical to consider when you move into these roles. You will also have a level of responsibility for others if you choose the management track
I recommend doing this at least twice a year, as it will give you a better understanding of where you are currently and where you can go in the future. Not only is this great practice to do for yourself but, if you are looking to go towards the managerial track, it will be great practice for your future reports!
What is another aspect of your career that is great to assess regularly? Defining your user research philosophy!
If you liked this article, you may also find these interesting:
- How to Assess Your User Research Interviews
- How to Break Into User Research
- A Week in the Life of a User Researcher
- How to Write a Generative Research Guide
If you are interested, please join the User Research Academy Slack Community for more updates, postings, and Q&A sessions :)
How to Break into User Research
Becoming a user researcher is not easy.
One of the number one questions I get every week are people asking me how to break into the field of user research from another role or right after graduating. I speak with people from all different disciplines, some closer to user research, such as marketing, psychology or design, and others further away, such as accountants or writers.
One thing I love about user research is that the skills you need to break into a field are about relating to and empathizing with humans. For me, it has been one of the most rewarding jobs I could ever have imagined. Through user research, I can have a positive impact on both users and team members. Another great thing? I genuinely believe most people can become great user researchers, without paying loads of money for a degree or certificate.
To prove this, here is my story of how I broke into this field and some tips to go with it.
How I got into user research
Getting into user research was one of the least straight-forward paths I have taken, and that is often the case for most people breaking into this field. There is no one magic course to take or one perfect path that will guarantee you an entry into any user research job. It is one of the more difficult specialties to get into because of the indirect pathway, especially when even the entry-level jobs seem to have a mountain of requirements. A lot of my students ask if they should go back to school to get a Masters. With this degree, they believe there will be a higher chance of them getting noticed. That isn’t necessarily the case.
Most of the time, I would, quite honestly, say no, you don’t need to go back to school and get your MA. The only exception is if the degree encompasses other interests and potential career paths. For example, one of my students is potentially interested in becoming a user researcher, but also might end up in policy research. In this case, it might make sense for him to pursue this degree. Aside from that case, I would argue that it is not necessary to go back to school to get into user research.
I was able to get into user research through an internship in New York City. I had just finished my MA degree in psychology (yes, I know I said you don’t need a higher degree to get a job, but wait for it), and decided I didn’t want to continue with pursuing my Ph.D. Instead, I wanted to join the world of user research. My MA degree was in clinical psychology and came with some knowledge of statistics. The role I finally received was as a user research intern, with a qualitative focus, at a tech company. I had never worked at a tech company before, and let me tell you, my Master’s degree could never have prepared me for the experience.
How did I go from an MA in psychology to a UX research role?
I took a few different steps when I started looking at and applying to different UX research positions. Below is (what I remember) of my crazy process of diving into this unknown world:
- Look at 100 user research job postings and scour the responsibilities. I found the most common to be conducting research sessions, usability testing, note-taking, recruiting. Then I tried my best to make my previous experience sound as relevant as possible in this context. It wasn’t easy. Yes, I had experience recruiting participants and some interviewing experience, but usability testing was out of the realm of my knowledge, as well as understanding how a tech company worked.
- Stalk all the well-known user researchers on LinkedIn. I read about their day-to-day descriptions of what they did. Reading their responsibilities helped me understand the different skills I needed, outside of what recruiters posted on job descriptions. It also helped me know if this was something I wanted to do with my life
- Find companies that have an established research team or a senior researcher. This way, you can learn. I was the only UX researcher at my first internship and had to leave after eight months because I needed a mentor. I was lucky enough to find one in my next role. He helped me take my researching skills to the next level and is one of the reasons I have succeeded in this field
- Apply to a million jobs. I think I applied to something like 67 jobs when I was first starting. Some of them, I would have never got in a million years, but it was worth applying. What is the worst that can happen? You never hear from them again, or they say no thanks. I disregarded some of the “requirements” and I think you should do the same. In the beginning, I looked at and applied to roles where they wanted 1–2 years of experience. Even if they say they want an MA or MSc degree, APPLY ANYWAY. As Dory might say, just keep applying.
- Go to meetups and meet user researchers. Connect with others in the field and ask them how they got into user research (there are many weird ways). Networking is also fantastic for finding internships or potential opportunities. Hate networking? Check out my guide to networking as a user researcher.
- Many positions were looking for a portfolio or a case study. Now we get to the hardest part. I had NO idea what a case study for user research was. After some googling, I understood this was an example of work. Well, I hadn’t done any UX work that I could show. I started working on a few personal projects. This, right here, is my number one tip: If you have never done user research before, the best thing you can do is read up on it and START DOING RESEARCH. Pick an idea and try to map out how it would work at a company. I created a competitive analysis, wrote a research plan, did some discovery research, card sorting, usability tests, and came up with some insights. Pick something you are interested in and do a research project on this. I post weekly prompts on Medium and feel free to respond to those.
Practicing user research is an essential part of creating a portfolio and getting “experience.”
Yes, I honestly look back on those pieces now and laugh, BUT, I got an internship, and a few other interviews because of those small projects. I put in a lot of amount of effort into three case studies, and it helped me when I went to apply and interview. They weren’t perfect, not even close, but they showed initiative and my passion for learning and being a part of this field. What else do you want for an intern or junior role?
I have some final thoughts on why a degree won’t help you as a user researcher. The most critical and best way to learn user research is to be in an environment with user research. It is absorbed by practice, not by theory. Unfortunately, at this moment, user research is not being taught this way anywhere. There may be aspects of this in an MA program, but I believe the best thing you can do for yourself is to get into an environment in which you are learning user research through doing and observing. MA programs will give you a theory to work off of, but the practical experience isn’t there.
I prefer aspiring researchers to spend two years trying to get into an apprenticeship or internship than spend two years learning, and then try to get into an entry-level research position.
Check out my website for some resources to get you started on user research (such as templates and guides): www.userresearchacademy.com.
As always, feel free to leave any questions or comments.
Burnout as a User Researcher
Even blogging can feel like social media.
It’s about 6:50am and I have been staring at this blank Medium page for days. I have merely managed the title and tagline. Recently, it has been a struggle to write articles, both for Medium and my contract writing. This is contrary to how I usually feel about writing. I normally love writing, it energizes me. I know everyone struggles from writers block, and feeling tired, but this feels beyond that. For some reason, something in me has shifted and, suddenly, my word choice feels stupid, my sentences don’t feel as witty and my thoughts feel incredibly scattered.
This has also managed to seep into other areas of my life, such as work. The other day, I sat in a strategy planning meeting for Q3. I’m a senior user researcher, and I have been doing this job now for about seven years, but as I sat in that meeting trying to wrap my head around building business cases, revenue impact and prioritizing backlogs, I felt utterly like an imposter.
Earlier this week, I started a writing assignment, which was to produce an article on generative research. Generative research is my bread and butter, my go-to, my expertise, but I sat there and stared at my blank canvas of a word document for a good two hours before closing it. I felt an immense amount of pressure for it to be perfect, and shame for barely being able to type 350 words out of my 1,500 word minimum.
Also, I’ve been generally irritable, cynical and sensitive. This has been super fun for my friends and boyfriend.
Why am I sharing this?
Sometimes I feel like blogging can be similar to Instagram or Facebook. Usually, I don’t write about how insecure I might feel in a meeting, or how I can feel so lost and overwhelmed at work. When we read articles, it can seem like the author really has their sh*t together. They rarely mess up interviews, or struggle with producing actionable insights. It can feel like everyone else knows everything. And, while I will always agree that reading is a form of learning and sharing knowledge, at times, the vast amount of knowledge can become so overwhelming.
Nobody knows everything. We are all out here trying to do our best, but, at times, it can feel extremely important to convince others that we are smart, talented and always on top of everything.
In addition, user research can be a particularly emotional job. I have spoken to many researchers and a lot of us are empathetic introverts. Being an empathetic introvert can be difficult in a job that requires you to constantly empathize with, talk to and be around others. This is the exact reason I left the world of therapy (and then, consequently joined the world of tech therapy).
When you empathize with someone you, essentially, are sharing their same feelings. For instance, if a person is struggling or upset, you are more likely to feel that way as well. We are often trying to understand problems with products and where people are having a hard time so we often see people feeling frustrated by our product. This is how we, as researchers, are able to relate to and advocate for our users, but it can also leave us feeling drained. Sharing research can be equally as difficult as you are constantly highlighting problem areas and issues for the team. Although I always call these areas for improvement, or opportunities, that doesn’t always fool people.
What are some “symptoms?”
There are some red flags that can help you determine if you are sliding down the slippery slope of burnout. Although everyone is different, and expresses reactions to being stressed in various ways, here are some symptoms I have recognized:
- Unable to focus. You have difficulty focusing on one thing at a time, and end up just doing a lot of scattered work, and, ultimately, not completing tasks
- Desire to avoid. I am an avid to-do list user, and absolutely love ticking tasks off of my to-do list…sometimes to the point of putting a task I already finished on the list, just in order to complete it. When I feel burnt out, my to-do list just keeps piling up, and I do my best to not even look at it, or add new tasks to it
- Isolatation. When these feelings of burnout come, all I want to do is sleep and work from home. I don’t want to be around people, or share my thoughts with others. This can also cause me to withdraw from my personal relationships and social life
- Personal life becomes more difficult. I’m like Monica from Friends most days, but when I am burnt-out, I don’t even want to tidy up my apartment. I pick more fights with people I care about, I don’t eat as well or commit to my daily meditations. Everything feels too heavy and arduous
- Consistently feeling demotivated. That job you used to care about? It suddenly becomes significantly less important to get anything done, or contribute positively. Even the easy tasks feel insurmountable, and everything feels too difficult
- Reduced performance. The feeling of not being able to do anything right that I mentioned in the beginning of this article, that is a huge warning sign for me. This could surface as missing deadlines, feeling “spaced out” (missing phone meetings or being late to meetings) or being less engaged in your day-to-day role. It is a heavy burden to feel like you are unable to perform well
- Increased cynicism. Now, my humor is generally full of cynicism, and that is perfectly fine, but I mean an elevated cynicism, which can actually end up in being mean. Suddenly I feel like I am the debbie downer, and the person who is more focused on the negative, or how difficult something is. Nothing is possible, and everything is annoying
What can you do to avoid/help burnout?
I have developed a few ideas on how to help alleviate the feelings of burnout, both before and after the symptoms start.
More research-specific ideas:
- Do research with team members! You don’t have to be the only one listening, taking notes, synthesizing…in fact, you shouldn’t be. User research really is a team sport, and sharing the experience with others really helps diffuse the high emotions in the session. You have other perspectives and can talk through what you just experienced with others
- Do emotion-neutralizing exercises. I do my best to meditate daily in order for me to stay aware of my emotions, and control how those emotions surface. In addition, I have meditated after particularly difficult research sessions, and have also meditated before sessions if I have been having a tough day beforehand. Another exercise I use is to write out any feelings before and after the research session. Both of these allow me to leave additional emotional baggage outside my sessions
- Avoid back-to-back research sessions. Although this is sometimes unavoidable, I find breaking up the session allows me to decompress and “restart” my emotions before the next session. That way, I am ready to handle whatever is coming for the next session, and am not carrying around anything from the previous session. It also gives me a chance to do any neutralizing exercises (or get something to eat)
- Listen to the research a few different times. I listen to my research recordings right after a research session, and then, again, a few days (or a week) later. I find relistening right after the session to be much more emotionally charged, as I am still very attuned to how that user is feeling. of course, I think it is very important to listen to a session again right after (I also use this to take notes), but I also think giving yourself some emotional distance by listening later also helps put issues and problems into perspective
Overall ideas:
- Reach out to your team members and manager — For me, this has been the most beneficial and important. You aren’t alone in this, and talking with others helps you feel more connected. Not only that, but you can let your manager know how you are feeling and they can help you work through some of these issues. It is always better to share this information rather than others thinking you are “slacking” or that you don’t care
- Be nice to yourself — not every single day will be perfect. Some days will be more difficult than others and you won’t always be successful in controlling your emotions. We can’t always be on top of everything at once, so make sure to give yourself a break if something goes wrong
- Take care of your body and mind. Go on vacation when you need to, take a mental health day or work from home if you need some space. I always try to eat well, exercise and get enough sleep. Now, this doesn’t always happen (see the point above), but I naturally feel better when I do!
These symptoms might sound dramatic or floofy (one of my favorite made-up words), but they really can be real for some people. Research, as well as many other jobs, can be extremely emotionally draining. When we are feeling unwell, we end up doing more harm to ourselves and others, so it is important to always assess how we are feeling, and be okay accepting however you feel — the best thing you can do is acknowledge and move forward!
Being a good user researcher goes beyond empathy
15 traits and skills valuable to a user researcher.
I’m not going to lie, being a user researcher is a really cool job: I have a variety of different activities I could be tackling in a given week or day, I have the opportunity to make a long-lasting positive impact, I can talk to people to more deeply understand them (fascinating), I write research plans and I also get to run fun brainstorming workshops with cookies and pizza.
It’s super enjoyable, but, I have found, it does take a certain type of person to be a user researcher. While I have some natural tendencies towards the job, I have had to evolve some of my other qualities. For instance, I am very much introverted. On the surface, you might not be able to tell, but it is more about the fact that being around a bunch of people and listening intently can be quite draining for me. Because of this, I have had to balance how I approach work, such as:
- No more than three research sessions in one day
- Try to avoid moderating too many back-to-back workshops
- Attempt to make Monday and Tuesday my “meeting days” in order to have fewer team meetings throughout the week
- Take some days to work from home sometimes, in order to recharge (when I have a more meeting-free day and don’t have research sessions scheduled)
That being said, there are some more obvious qualities that a user researcher should tend towards (or work on). We all know empathy and friendliness are key, but what are the lesser known characteristics that benefit user researchers?
User research personality traits
- Perceptiveness
We need to take in everything that is going on around us, especially during research sessions. By being perceptive, we are able to filter out the noise, and hone in on the most important pieces of what participants are telling us. This trait allows us to ask crucial questions on the spot, instead of realizing the opportunity was missed later on - Open-mindedness and calmness
Looking for patterns in data, in order to deliver insights, is critical in making a successful user researcher, but it is extremely important to not jump to conclusions when synthesizing research. We need to remain calm and open to all possibilities, in order to see grey areas others may miss. This also applies to realizing we can’t understand or do everything perfectly - Neutrality
Frankly, we aren’t really meant to have opinions but, instead, are meant to share all the facts with others in the most unbiased way. During research, we don’t really respond to what participants are saying. There have been times I have wanted to laugh, cry and hang up the call, but it is our duty to listen, dispassionately to what others are saying…although, this doesn’t mean you act like a robot - People management
As a researcher, I interact with many people from product owners to designers, developers to marketing, finance/legal to branding. Sometimes there can be a good amount of chasing and babysitting, to ensure research gets done properly (and isn’t pulled in too late to a project). I create bi-weekly meetings with product owners and designers so that research isn’t left behind. I also created a one-pager on how to work with a user researcher - A whole lotta patience
Similar to above, with chasing and educating, there is a need for a degree of patience. User research isn’t concrete enough as a field to simply be known (or understood correctly). In addition, it can get boring sitting through hours of interviews with people who are telling you, seemingly, similar stories or taking the same actions on tasks. However, we need the patience to look through each interview with a lens of fascination and potential. Also, be patient with last minute cancellations! - Mental juggling
Specifically during research sessions, we do a lot of different things simultaneously: observing, listening, understanding, forming questions, time-keeping, empathizing, knowing when to dig deeper, managing observers, etc. This is a lot of stuff to do at once, so we need to have the mental capacity to juggle all of this competing information. I use meditation to help with focus and clearing my mind. That, and lots of practice! - Be compelling
Often, we need to convince others of the value of user research, as well as evangelizing the voice of the user. In order to do this, we need persuasion, but without the negative connotation that word usually entails. It is important to understand the fears or misconceptions of user research and then present research projects and findings in a way that quells those fears - Having a touch of romanticism
Researchers generally want to make the world a better place and to improve people’s lives. A degree of romanticism is necessary in allowing us to forge forward through common obstacles and struggles. At the end of the day, all we want to do is help others, both companies and users
User research skills
- Ability to collaborate
As mentioned above, researchers can touch many different departments and are often considered a “service” for teams to use. In this sense, we need to be able to collaborate with all different areas of a company, whether that be product, tech, marketing, finance, legal, etc. Making yourself as approachable as possible, and taking the time to understand someone’s previous experience with user research enables you the opportunity to collaborate even more - Adaptability and admittance
User research can be unexpected — a connection could be lost, a participant could be hard to handle, a method may not be working well, someone could question and dissolve your research findings through something you didn’t know. We have to be flexible in changing methods and working with what is directly in front of us in any given moment. I now expect the unexpected and am happy to admit I did something wrong or that I simply don’t know - Teaching
One of the best things I have done at every company or consultant position I held was to teach others how to perform basic user research (primarily usability tests). Being able to explain concepts to people in a way that enables them to understand not only helps you educate people in an area with many misunderstandings, but can also come in handy for facilitating brainstorming sessions or fielding the many questions a user researcher gets asked when presenting findings (ONLY 10 PEOPLE?!) - Learning quickly
Especially in the freelance/consultancy world, we need to learn quickly: learn about the people we are working with, how teams are structured, the processes we need research to fit into, the domain we are a part of, different types of technology, new research methods. We don’t need to become experts in everything, but it is important to be able to recognize commonalities between what you know and things you are trying to learn, as well as putting in the time to properly absorb information necessary to your performance - Have a good memory
This might seem obvious, extremely helpful. This is key when moderating interviews, as you can memorize the script you need, making the session much more natural for you and the participant. There is a lot of information to be stored, and you are bound to forget some details, but having a good memory enables you to make connections between research participants (and even projects) that others might miss - Writing
I write a lot in my day-to-day activities, such as recruitment emails, scheduling emails, presentations, research summaries, personas, scenarios, one-pagers, budgets, research plans, screeners, surveys…the list goes on. Sometimes, I even help with UX copywriting for my designers (we don’t have that role filled yet). Being a structured, concise yet friendly writer can really aid you, and save you time with editing - Storytelling
And, of course, storytelling…what does that even me? To me, I care about people’s stories because I think they are an integral part of understanding someone and then communicating research to others. In order to effectively get others to understand our users, and the impact our product has on them, I need to string together the stories users tell me in order to convey meaning. If you are interested in more, I talk about storytelling for a research portfolio here
Don’t have these skills or traits?
That is okay, and doesn’t mean you can’t be a user researcher, or try it out. I have just realized these are very helpful qualities and abilities to have. If you fall short on some of these, they can definitely be learned and sharpened over time. Not all of us are perfect and, as a researcher, of course I talk about the ideal. We all have our own strengths, and areas of improvement. As with anything in user research, it just takes practice!
A case for beta testing
How User Research & Beta Testing Intersect
What is beta testing?
Beta testing is a wonderful, and often underused tool, that allows you to ascertain user feedback for a live product. Since beta testing only requires you putting the product into a small number of percentage of your users, you can correct any bugs or improve UX before you roll out your product to everyone. Essentially, you use these beta testers to see whether they are actually using the product/feature, how they are using it and what bugs, improvements or innovations come up. If beta testers aren’t using certain features, the chances are your regular user base won’t be either. It is during this stage, you are able to find out why through beta testing.
I don’t necessarily believe beta testing and user testing are different things. In fact, I think they fall under the same umbrella and is more of a continuous process in which testing at a particular point of the software release is seen as doing beta testing. However, many people do beta testing without user research and vice versa, such as in analyzing software performance, product development pipeline and marketing analysis. Usability testing shows you if a concept is usable, while beta testing shows you if/how people will actually use the live product.
The biggest difference is that beta testing comes in when the product is actually live, rather than concept or prototype.
Why should you beta test?
I can honestly tell you beta testing has many different benefits, and is a great concept to bring to your company. Since it can sit in the realm of user research, but also in marketing and sales, it can lead to cross-departmental collaboration and an overall better experience for your customers. Here are a few benefits I have seen from my past experience with beta testing:
- The ability to do continuous research
- Cycle of always testing new features
- Finding bugs or UX improvements before they become a large-scale problem
- Show users you care deeply about their opinion
- Build a community of engaged users that will give you feedback on your new ideas, features and releases, which, in turn, helps them feel valued
Just writing this list gets me excited to start a beta program at my next full-time job. A lot of established companies, and even startups, are absolutely at a place where they can start a beta program, which means they can start doing continuous user research, which is the key to a healthy company. Happy users = happy company.
When should you beta test?
Beta testing is a wonderful concept for a few different scenarios:
- If you are looking to get started with user research and already have a live product
- If you want to test and learn about different features before complete rollout
- If you have very specific questions you want answered about a certain feature or flow in order to make informed decisions
- If you are interested in tracking analytics in terms of usage before rolling out to your entire user base
- If you are trying to find bugs or issues with flow on new releases, features or products
The most important part to note is that beta testing is done when a product is live, and that includes products that are just at the MVP (minimal viable product) stage — in fact, it is encouraged to beta test products at the MVP stage.
Start a beta testing user research program
There is no one way to start a beta program (or anything, really), so I am simply speaking from my experiences of my past. It can take some time to get it started, especially if you are starting 100% from scratch, but the benefits outweigh all of the upfront work. Once you have your beta program established, all you have to do is simply maintain it, and enjoy all the user feedback.
- Define the goals and rules of the beta program
What do you want from this beta program? Will you be engaging these users on a weekly, monthly or quarterly basis? How will you be engaging them? Will it be simply for new features, or for innovation and new product releases? How often will you require them to give feedback?
For example, at one company, we reached out to beta testers when we had a new feature we wanted to test before we did a massive rollout. There wasn’t necessarily a continuous timeframe we strictly followed, but, I would say, we spoke to users just about every month or two about this. The goal of the beta program was to grow a community of users we could always reach out to in order to test new features. After some time, we were also able to engage these users about new ideas (concept testing), before we even got to live code. - Create a beta program sign-up
Figure out a place where you can ask users if they are interested in joining the beta program, and allow them to sign up, whether that be via a website or another online tool. Where ever the sign up lives, it should state the following:
- Explain the benefits of joining the beta program
- Detail the requirements of the beta program
We required out users to give us feedback on certain features or product rollouts every two weeks. It obviously depended on the scale of the beta test, but we set a very clear timeframe. I would schedule the feedback sessions in advanced, and make sure to email them with reminders.
In addition, I made sure the benefits were very clear: they were able to see all new features/products before the masses, their feedback visibly made changes to our product, they could help inform our product roadmap and, every quarter, I sent a gift basket to our beta testers (once we got too many, I rotated, and tried to get everyone at least once a year) - Invite people!
Once the internal parts are figured out, it is time to invite users to become beta testers. I spoke to account management and marketing to understand who they thought the best beta testers would be. I had also been doing user research, and marking down who I thought great candidates were. I emailed all of them directly with a link to our sign up, and offered to speak to them via phone call, if they had questions. I would also ask users at the end of research sessions if they were interested in joining. Once people join, try to segment them into groups, such as “power users,” “very willing to give feedback,” or “trouble with technical issues” — that way, you have an idea of which users to potentially select for certain beta tests - Beta test!
Now comes the fun part, you have your users (which, again, may take a little time) and now it is time to beta test! You can choose to beta test a feature with all your beta testers or you can test different versions with different groups of beta testers. Once you have decided, compose an email telling them what to expect once you turn the feature on. Don’t give them too much, as you want to encourage them to explore on their own, but let them know you are readily available for questions and feedback.
Make sure to remind users when they are scheduled for feedback sessions, and how you plan to gather their feedback. We did 95% of our feedback sessions over remote video conferencing - Receive feedback
As mentioned, I created feedback meetings with our users after a predetermined amount of time. I recorded the meetings (with permission, of course), took notes and then compiled all the feedback into research summaries. I would send the research companies to relevant and interested parties, such as the scrum team working on the feature/release, as well as marketing or sales - Continue the cycle
Once you have created and fostered a community of users, you will find how easy it is to conduct continuous user research. You will be able to reach out to users for more than just beta testing, and engage them in concept testing, usability testing and even generative research. It is a really powerful tool to make your users feel heard, and to encourage them to share their thoughts and opinions with you
Just a word of caution — beta testers are most likely going to be power users of your product. They aren’t going to be completely representative of your user base and population. Therefore, keep in mind that not all their feedback will be helpful for other users. We used beta testing in addition to user research, which meant we got a really holistic picture. Don’t simply rely on beta testing, but use it to get started on continuous user research!
[embed]https://upscri.be/ux[/embed]
UX Research Roundup — 2018
2018 was a long year. Thinking back to January 2018 makes my brain feel like I am trying to remember something from five years ago. However long, 2018 brought a lot of interesting learnings and thoughts in user research. As the years go by, user research evolves and grows in ways I never thought possible when I first entered the field. It is a fascinating and exciting area to work in. I’m very much looking forward to 2019 and what that brings, but wanted to roundup my favorite UX Research (and some non-uxr) moments from 2018.
UX Research Roundup 2018 (and some extras)
- 14 Random Ways to Be Kind at Work because, our colleagues are also our users
- Analyzing Usability Test Data: A step-by-step guide to getting the most from your usability tests
- The beginning of Slack for ResearchOps, an incredible community dedicated to understanding and improving Research Operations in companies
- A simple Prioritization Matrix (very easy to use) to help inform design decisions
- An interesting take on why Personas Are Garbage (not that I agree completely with this message, but it is an interesting perspective), and the other side: How to Create Lean Personas
- Moderating Usability Tests with Simultaneous Translation — something I will be using at my new Full-time role!
- Quantifying Qualitative Research (Video & Text — amazing)
- Join the Cognitive UXD newsletter, a fascinating combination of UX and Psychology
- Don’t design for yourself, do research and design for your customers!
- UX Curator newsletter is my favorite newsletter, no matter the year. It consists of 4 of the top UX stories of the day delivered in a simple and concise email
- Some arguments on Why UX is Dead…or is it?
- NPS is a Waste of Time, Use These Metrics Instead — I love when people go against something widely used and then give suggestions on what else to use
- Facebook Research: Tips from the First Decade
- How Properly Wasting Time at Work Increase Productivity — I love articles like this! I think we all try to do too much at once, make it a goal for 2019!
- Design & UX Research Forums were created from the ResearchOps Community. So much invaluable and wonderful information in one place, it is mind-blowing!
- How to Build a Usability Testing Lab Anytime, Anywhere
- 50 Interview Questions for B2B SaaS UX Research and also really great general tips. These questions are wonderful for generative research and creating (non-garbage) personas or customer journey maps
- Cognitive Biases & the Questions You Shouldn’t Be Asking
- Sharing UX Research Results by Comic
- How to Read a Mental Model Diagram and literally everything by Indi Young is amazing (she’s my UX research celebrity crush)
Some shameless self-promotion
- Storytelling for a UX Research Portfolio
- UX Research Toolkit Roundup
- My User Research Course!!
- How Might We Be Wrong
- An Introverted UX Researcher’s Guide to Solo Networking in Product/Tech
- Check out my Skillshare course on how to create a user research plan
Looking ahead to 2019
- UXInsight Conference 2019 (I will be going!)
- Strive UX Research Conference 2019
- User Research London Conference 2019 (I will be attending this too!)
- UX Research Australia 2019
What will happen in 2019 for UX Research? I predict more time and value placed in deeply understanding customers, defined user research processes that work alongside agile environments, specialization in research fields/verticals, scaling user research teams will become easier and, finally, I’m hoping for a comprehensive user research tool that gives us everything we could ever wish for and more.
Happy 2019 everyone!
[embed]https://upscri.be/ux[/embed]
How might we be wrong?
Why User Research isn’t a magical unicorn.
As user researchers, we spend a lot of time and effort on convincing people of the value we could bring. Because of this, we are sometimes hesitant to consider how our results may not hold the best and truest insights. It isn’t necessarily stemming from an egotistical wish to always be right, but more fear-based. We don’t want to lose the buy-in we fought so hard for. If our results are pristine, actionable, innovative, “right,” then we could potentially lose what we worked so hard for: the right to do research.
As I tell many clients and students, user research isn’t the answer. It isn’t a magical unicorn that farts perfect, rainbow-colored insights. User research gives us an understanding of our users so we can make informed decisions. I didn’t say the right decision, but informed decisions. It gives us a direction (or several) that is based off of our knowledge of our users. However, as many times as we say this, user research can often be looked at as the one thing to solve them all (any LOTR fans out there?). When research fails to live up to this impossible standard, stakeholders can get frustrated and shut it down.
We need to ease up on our expectations of research, and use it as guidance as opposed to the one and only true path. With this, user researchers can analyze their results without fear and find where the research may be less valid.
The Concept of Validity
Validity is how well a test or research measures what it was intended to measure. It also the extent to which the results can be generalized to a wider population.
Validity has always been controversial in qualitative research, and many user researchers have abandoned this concept completely, as it sits too much in the world of quantitative data and statistical analysis. As qualitative user researchers, we simply don’t have the large numbers to back up a concept like validity in the same way quantitative data does.
However, validity is important in understanding how you might be wrong about a result. This concept is called a validity threat. They are other possible explanations or interpretations, something Huck and Sandler called “rival hypotheses.” What validity threats boil down to are alternative ways of understanding your data.
How does this occur? It can be for many reasons (have to love confounding variables), such as your participants not presenting their actual views (social desirability bias), missing data points that support/disprove your hypothesis, asking leading questions, or simply having certain biases towards the people or ideas you are researching.
In short, a lot can go wrong, which is why, I believe, it is important to rethink the meaning of validity in terms of qualitative research, instead of ignoring it completely. What do we need to look into in order to redefine validity in terms of qualitative user research?
Two Most Common Threats: Researcher Bias & Participant Reactivity
As humans, we are inherently biased in a given situation. Even if you do everything in your power to remove these biases, they will continue to linger on an unconscious level. This can impact user research when conclusions/results are based on data that fit the researcher’s existing theories, goals or perceptions and the selection of insights that “stand out” to a researcher. These insights or quotes may catch a researcher’s eye, but it is understanding why those pieces of data are particularly important. In order to know this, you have to be aware of your biases and how you will deal with them. Is it by writing them down before you analyze your data or by having a few researchers look through the results in order to see how different people are approaching the insights?
Similarly to bias, the user research will always have an influence on the participant, causing a degree of reactivity from the user. Quite obviously, eliminating the impact a researcher may have on a participant is impossible, so the goal is not to eliminate, but to understand what the potential impact may have been and use it when analyzing results. There are some measures a researcher can take to decrease a participant’s level of reactivity, such as not asking leading questions, dressing similarly to the interviewee, sitting on the same level and using open body language. Like with bias, it is important to understand how you could be influencing the participant and how that may affect the results.
Write a Memo
In order to help ourselves better understand what threats we face, we can do a simple exercise that involves writing down the answers to the following questions:
- What do you think are the most serious threats for this research objective? Why are these the most serious threats? What are the main ways you might interpret the data incorrectly? Be confused about the insights? How could you interpret results incorrectly?
- What do you think are the most serious threats other people (colleagues, stakeholders) may have? Why do you feel this way?
- How do you think you can best assess and mitigate these different threats, for yourself and others? How might others be able to help you assess and mitigate the threats?
What Else Can Help?
There are several other ways qualitative researchers can explore and impact validity:
- Long-term studies: By continuously interviewing and observing participants, you are able to learn more about them, beyond what they tell you in a 60-minute window. You can understand users much more deeply, and start to grasp which insights or quotes may be shallow or random. This goes against many company’s desire to do research quickly, but, if you have the buy-in and time, the more research you do, the more valid and generalizable your conclusions will be
- Rich data: It is wonderful when someone can take notes for you during an interview, or when you can take notes yourself, however, people tend to drift in and out of focus during interviews, so it is difficult to pick out everything a participant says. Note-taking can be an early form of bias — someone writes down what they deem as important — a better alternative is to get a transcript, record the interview and have several people taking notes to compare later
- Respondent validation: Humans do a great job of assuming…as in we do it a lot, but we aren’t actually very good at it. If a participant explained something to you, and you were a little lost or confused, don’t just assume you will figure it out later. Always ask participants to clarify, when it makes sense to. This is the easiest way of ruling out the possibility of misunderstanding what a participant meant. This can also help you understand your biases and misconceptions
- Don’t ignore data: I know we all want to get it right, especially when we are getting pressure from above, and we want to deliver the most favorable results. With this type of tunnel-vision, we can easily miss or ignore data that goes against the more flattering hypothesis. Even if the data sucks and tells you your company is doing everything wrong, it’s better to report that than waste time. Another note on this, make sure you have a few hypotheses lined up, not just the ones everyone wants you to validate — write these down like your biases — that way you are looking at the data from several different standpoints
- Comparing data: Talk to your users at different times of day, in different settings and in different ways. This helps rule out confounding variables. For example, if you are a health company and speak to all of your users first thing in the morning, you may hear a very different story from them than those same users later at night, when they are tired and would much rather reach for potato chips than your juice cleanse. Ensure you aren’t replicating the exact settings for all your users, and you will get data you can more confidently generalize.
Even with these approaches, there is no way to truly know if your qualitative results are perfectly valid (in fact, they will never be, as is the same with quantitative data). But, this is a step towards a more open-minded approach when thinking about the validity and qualitative research. User research isn’t supposed to be known for sure, as nothing in life is. We need to give researchers a break and let them approach their data with the potential that it isn’t always the right answer. That way, we can work together towards more informed and less forced solutions for our users.
[embed]https://upscri.be/50d69a/[/embed]
Storytelling for a UX research portfolio
The Hero’s Journey.
UX Research portfolios are hard
One of the most common questions I get from my UX students is as follows: “how do I create a research-based portfolio?” It is not a question only my students struggle with, but, I believe, most user researchers have also encountered problems with. In a time when user experience is generally thought of design and research together, hand-in-hand, it can be difficult to portray one without the other.
I ran into this problem five years back, when applying to my first UX Research role. My interviewer requested a portfolio piece to discuss for when I came to the office. I had about 48 hours. Looking back, those 48 hours were spent in a combination of drinking every caffeinated beverage (+points if it also had alcohol), scouring the internet for answers and, ultimately, sitting on my couch, watching HGTV and wishing I had taken the route of interior design.
I delivered a portfolio piece, but it was a weird mix of research and design. It wasn’t perfectly structured, in fact it was far from, and it even included a gradient background. However, it was an internship role, and I convinced them enough to land the job. Luckily, I was spoiled, and wouldn’t have to worry about my research portfolio for a few years down the line. But, when those few years past and reality hit, I knew I needed to upgrade the gradient.
Components of a research portfolio
Since user researchers, who are solely focused on research, don’t have designs to showcase as deliverables (although we may have worked very closely with designers), we have to make sure we capitalize on a few components within our portfolios. I have narrowed them down to three:
- Compassion: understanding your audience and evangelizing users
- Skills: demonstrating knowledge of the field and how to gather/synthesize insights to be actionable (my least favorite word)
- Storytelling: ability to effectively communicate research goals, methods, findings and outcomes to anyone and everyone
Ah, storytelling. Everyone wants a storyteller. I must admit, the people I regard as highly intelligent (both emotionally and intellectually) are wonderful storytellers. It is an extremely captivating and useful skill to have. I quickly learned, in my day-to-day job, how important storytelling is. You are able to help others understand the user and to communicate findings in an extremely human, and digestible, way. This is all well and good, until you are no longer in front of your audience, but instead, they are reading about it on a piece of paper (and have about 64 more portfolios to get through by the end of day).
So, how do we do it? The Hero’s Journey
I made several passes at my portfolio, trying to put emotions, insights and methodologies into technical, research-y (I’m a writer, I can make things up) words. Each time, the delete button reigned supreme. Considering I am a fiction writer, I was frustrated. I had the worst case of continued writer’s block. So, instead of trying, another time, to force a square peg into a round hole, I adapted the most popular storytelling/story writing technique to the user research process: The Hero’s Journey.
The process, and I simplified it from above, goes: inciting incident, rising action, crisis, climax, denouement, end. I tried to map user research to each of these points and ended with:
- Inciting incident: what led to the research being needed and what business (or other) problems the research is looking to solve? This is, essentially, the back story.
- Rising action: what is the approach the research is taking (the methodologies, the participants) and, most importantly, why did you choose that approach and those participants. The why is the story, and the most important part, which shows how you think through the best way to get answer to the above problems.]
- Crisis: what is something that went wrong leading up to or during the research? What was the most difficult part? This is less of a post-research reflection and more of speaking through difficult participants or unsupportive stakeholders. This gives a little flavor to the portfolio, and pulls the reader in, before presenting your findings. Also, everyone has problems, so it is relatable.
- Climax: after all the preparation, the pitfalls and the research, what did you actually find? I like to include specific quotes, audio clips or video clips, when possible, as well as photos (sticky notes count!). Do everything you can to humanize the findings in a way that flows naturally — are there findings that are more important? Put them up front. Do two findings complement each other? Put them together.
This is also a great place to speak about how you actionized these findings. - Denouement: this is a winding down from the action, and where I like to show any deliverables I helped to work on, such as personas or journey maps. If applicable, I will also include any design iterations based on the research. It is also important to speak about how you socialized the research, such as through emails, presentations, etc.
- End: are there any next steps, such as usability testing? Or a new research project that stemmed from this one? Are there more questions that need to be asked? I love talking about how continuous of a process user research can be.
This section is a place to reflect on the project, any final thoughts you had, who you worked with or what you may have done differently.
While I may end up adding a bit too much cheekiness, or detail, based off this approach, it may also be more fun to read!
Hopefully, if you are struggling with putting together your UX Research portfolio, this will serve as a helpful outline or breakdown, and get you to think beyond UX. It certainly helped me look at my portfolio approach more creatively.
I would love to hear any UX Research portfolio tips & tricks you have!
[embed]https://upscri.be/50d69a/[/embed]
Want stakeholder buy-in? Present the MVR (Minimum Viable Research)
As a user researcher, I guarantee you have hit a point (or multiple points) in which you had to convince, or beg, stakeholders to either 1. Do research in general, or 2. Conduct certain research projects. Oftentimes, in this role, you are not only evangelizing the user’s voice and trying to create empathy for users, you are simultaneously trying to get some empathy for yourself and evangelize user research. It can be quite exhausting. You can find yourself up against barriers such as: concern that user research will take forever, cost millions of dollars in Amazon gift cards or the product team has a good enough intuition to design products that users love.
What can you do? Put on your running shorts and prepare to become an Olympic Hurdler (I had to Google that, and I’m not even sure that is the right term).
How does a user researcher hurdle? By treating your stakeholders like users.
While there are different approaches to getting buy-in, there is one I really think is particularly effective, especially in our field: performing user research on stakeholders by understanding the ‘why’ behind the pushback.
When a stakeholder says “no” to your user research project, it can feel deflating and frustrating, causing you to become defensive. Would you ever get defensive while speaking with users during an interview? Hopefully not. So, why would you do that to stakeholders? Essentially, they are your users too. While they aren’t using the product/feature in question, they are still an audience you need to understand and cater to.
When a user gives you feedback during an interview, you ask them why they feel or think a certain way. Why wouldn’t that be the same for stakeholders? You want to create an open space for, both, your stakeholders and users, in which you are able to facilitate a conversation filled with empathy and understanding. If that isn’t user research at it’s finest, I don’t know what is.
What are some questions to ask your stakeholders?
- Why do you feel negatively about user research?
- This will help you understand the best ways to quell their fears - Tell me what happened the last time you did user research.
- This will help you understand the fears or expectations stakeholders may have - What is your ideal timeline and approach for this project?
- With this, you are able to find ways to insert user research into their timeline - What are the biggest barriers to conducting user research on this project
- You can begin to understand how to best fit research in so it overcomes these specific barriers - If we could manage to do any user research on this project, what would it be?
- Opens the conversation up to discover what they think user research might mean for this project - What could be some ideal outcomes of user research on this project?
- Highlights the potential positive outcomes of user research
Once you’ve identified the root of the “no,” you can begin to cater your MVR plan to addressing the concerns.
Minimal Viable Research in 3 Steps
Now you have identified stakeholders biggest pain points and needs with user research, time to build some personas! Kidding. The next part is understanding what the business goals are, which generally encompass metrics I spoke about in another article, such as revenue, acquisition rates and retention rates. If you are able to understand why stakeholders are pushing back on user research (step 1) and then, in parallel, uncover the business goals and metrics most important to them (step 2), you can then create a research plan that both mitigates their worries and shows how research can impact the metrics they care about (step 3).
For example, as the new user researcher at Dog City (a purely fictional company, but what a fantastic place to work) I would, obviously, like to start conducting user research. I go to my team, excited to be the new, shiny employee who brings value, and, I find, they are hesitant as to where to begin and how user research works. Naturally, I dig deeper and discover they believe user research isn’t a priority and we don’t have time to do these projects. Step one complete.
With this knowledge, I know I have to present a fast option that will deliver high impact to the business. My next mission is to head over to higher level stakeholders in order to understand what the business goals are for this particular quarter at Dog City. Lo and behold, we are focused on conversion rates. Step 2 complete.
I then have some work to do. I take a look at the current checkout flow and perform a ‘mini’ heuristic evaluation myself. I find a small amount of previous research has been done and has shown users having a hard time during checkout because it is hard to complete the forms. Finally, I look into Google Analytics and see users are abandoning at two distinct points. What does this lead to?
An MVR plan of usability testing the checkout flow with seven users over the course of a week, which would yield high-impact and actionable recommendations with two results: placating the team by showing them research doesn’t have to take forever and appealing to management who are looking to increase conversion rates. Step three complete.
(BONUS) Step 4: Rinse and repeat until people are banging down your door (or hovering over you, in the more common open-office layout) with their own research project requests. Go forth, Dog City user researcher, and take over the world!
There are many ways to go about this and, of course, it always depends on the company, stakeholders, teams, etc, but I would love to hear about how you jumped over the hurdles of getting buy-in!
[embed]https://upscri.be/50d69a/[/embed]
Are we actually designing with humans in mind?
And the high you get from social media.
Social Media & Your Brain on Drugs
I speak for most of the population when I admit, there is a certain high you get from social media, whether it is a like, posting a particularly beautifully filtered photo, scrolling through Instagram to see frenemies “enjoying life” and chuckling to yourself, knowing they aren’t actually that happy. I know I’m not the only one who has done this. That rush and those feelings you experience is literally your brain on drugs.
The dopamine response
What does that mean? Obviously you haven’t taken any drugs, but, what I am trying to say is, the stimulus of what I mentioned above, of social media, induces a very similar reaction to that of drugs. Whenever you post or receive a like or comment, even when you are giving away likes or comments, you get a hit of dopamine.
What is dopamine exactly? It’s a neurochemical known as the “reward module” which is released after we perform certain actions or behaviors, such as exercising. It gives us a rush of happiness, accomplishment and, kind of, a sense of purpose in this crazy world. Dopamine also plays a role in habit formation and addiction. It’s great in helping us form that habit of going back to the gym constantly, but not so great if it uses its powers to compel us to constantly check social media. The combination between the unpredictability of whether or not you will get a like or comment on your photo with the dopamine-induced response when you do, simply keeps us addicted to social media. You might even start getting that giddy feeling when you hear your phone buzz.
So what?
Tech companies understand what causes dopamine to trigger in our brains, and they use all those techniques and hacks in their products, causing us to be in a never ending cycle of compulsion loops. The randomness of a like, you never know when you are going to get it, or a notification, is addicting and we compulsively check our phones and apps to see if we have received a dose of social affirmation.
Instagram is curated, every social media platform is curated, you don’t see people posting about their mediocre meals, back pain or bad hair days (for the most part). This curated life (what a great name for a TV show, heyo Netflix), creates an enormous amount of pressure on people to be happy. So much of our image of what happiness is is based on social media and that feed full of grinning, perfect people living the best life. These apps create a contagious fabulousness we are all are tricked into ascribing to.
While there is that rush of dopamine that comes from these likes and endless scrolling, there is also this dissonant feeling of failure, time wasted, being left out, and that we aren’t achieving all that others are. We strive harder to fill in the gap and post photos that show we are as glorious and happy as all of our frenemies, we receive likes, we get a rush, we see others posting their fabulousness and we continue to compete. It is a chemical hijack of our brains that creates an artificial baseline of what happiness really is and feels like.
Luckily, there is another type of “happy” neurotransmitter: serotonin. This is a different type of happiness that doesn’t come in a rush from faux-validation through animated heart icons. Instead, serotonin reduces levels of anxiety and depression, as opposed creating rushes of happiness. It feels like a more calm and content satisfaction, like observing a beautiful sunset, sitting on a beach or finishing a delicious novel. Generally, when we achieve goals that cause our brain to emit serotonin, they are goals or moments filled with intention and mindfulness.
What app that you recently used made you feel calm or helped you make progress towards a goal that, in turn, made you feel satisfied and content? I can think of a few, one of them being Headspace, an app for meditation. Makes sense.
Are we really designing for humans?
When we think about being user-centric, user-friendly and crafting experiences based on human-centered design, I am here to challenge: what does this actually mean to us as researchers, designers, developers and all product people, hell, all tech and business people? Are we designing with humans in mind when we create these dopamine-inducing user experiences? To me, it doesn’t feel authentic when the products and services we are designing are creating a false castle of happiness built on a sand foundation of sadness and worthlessness.
Maybe we can still sell our products and services, but more mindfully. Is there a way we can start taking steps towards curating user experiences that create a more calm contentedness that comes from the activation of serotonin? Can we actually achieve real human-centered and empathy-based user experiences while selling products and services or are we doomed to this compulsion loop of dopamine forever?
[embed]https://upscri.be/50d69a/[/embed]