How to assess your research interviews
A framework to continuously improve research sessions
As researchers, we are hyper-focused on the actual research interviews we conduct. We spend a substantial amount of time prepping for each meeting to give it our all when we face each participant. Right after, we rush to gather our insights, to remember all the juicy details, and to inform our teams to the best actionable steps. We strive to make a positive impact.
However, we spend so much time prepping and synthesizing and tend to evaluate how each research interview went significantly less. I still hate listening to myself, and it sometimes makes me cringe when I hear my voice. The art of reviewing our interviews and judging how well we did as a moderator, during the interview, is simply something that doesn’t happen often enough. I’ve found reviewing interviews and writing feedback has helped me to evolve my researching skills immensely.
How to evaluate your research sessions
After each interview I conduct, I listen again to the full session and make notes on how I can improve in future interviews. I’ll especially look for times when I make a glaringly obvious mistake, such as asking a leading question, assuming what a participant meant or asking a complicated, double-barrelled question.
In addition to keeping these basic best practices in mind, I have also been using an adaptation of Steiner Kvale’s criteria of a good interviewer to give myself a more accurate assessment. For each of these topics, I give myself a rating of 1–5: one meaning I did not fulfill this principle at all during the interview, and five meaning I went above and beyond to satisfy that principle. I then include some notes in each one of what I did well, and what I could do better the next time.
Ten Principles of a good interview
- Familiarity with the topic of the conversation. You have immersed yourself in the problem space and have researched the domain you are about to enter, including industry trends and jargon, as well as being aware of potential competitors. If applicable, and you are conducting usability tests, you have a functional knowledge of the prototype or product you will be testing. You are familiar with the concepts, interactions, patterns of a product and know what to expect when a user clicks on specific elements. For example, I didn’t do enough due diligence, as I was juggling a few usability tests at once. I ended up just as confused and surprised as the participant when the prototype wasn’t working in the way we expected.
- The interview was structured. You start the conversation by explaining what the participant can expect and layout the purpose of the discussion. I tell my students to write down introductions and try to memorize them before starting a research project. The beginning of a research session can make or break the entire interview. If the participant feels you are robotic and reading from a script, they may have a hard time opening up to you. On the flip side, if you don’t adequately explain what the research session is about, you leave the participant in the dark, which can feel very unnerving.
- Everything was clear. The questions you asked were simple, easy, and short, which is especially crucial in usability testing. But first, let me go into question structure. We want our questions to be as open-ended as possible. I use the TEDW method, “tell me about…/explain…describe…walk me through.” These open-ended questions lead to stories and conversations, which can give us much needed context and right insights, versus asking them a continuous stream of yes/no questions.
With usability testing, tasks need to be clear and directive. I create small scenarios behind each task. Here is a great example to illustrate this concept: Ikea wanted to understand how people found products on their website. They were usability testing by asking users to “find a bookcase.” Virtually every user went to the search box and typed in the word “bookcase.” They weren’t getting any valuable information, so switched the task to a scenario-based request: “You have over 100 books strewn around your apartment, in boxes, and cluttering various shelves. Find a way to organize them.” Participants responded in a very different way: clicking through various categories, searching for terms like ‘storage solutions’ or ‘shelves.’ Very rarely did someone go to the search bar and type ‘bookcase.’ This scenario is much more natural and relatable than “find a bookcase” and results in innate action. - The tone was gentle. You allow participants to finish what they say. My number one pet peeve is when people interrupt participants during research sessions. There is nothing you can do to make someone stop talking more than continually interrupting them. I often wait three seconds after the interviewee has stopped talking, ensuring their thought is finished. I find these three seconds especially important on remote video conferencing, as there is often a delay. If you must interrupt the participant, you can do so gently, with apologies. The only time I will interrupt is if we got utterly off-topic, and I am concerned about time. In this scenario, I will say: “I’m so sorry for interrupting, and I am super interested in hearing more about this topic. However, in the interest of time, could we switch back to this other topic, and I will try to leave time, in the end, to come back to what we are talking about.”
- There was a sense of empathy. You listened to the interviewee with all the attention you have. For those 30, 60 or 90 minutes, whatever the participant says and does is the most crucial thing in the world. We must mirror this feeling in our responses. When a participant is going off-topic but is passionate about the idea, you can allow them to vent, while acknowledging their feelings. Even if what they are saying seems mundane, it is still imperative for us to care about the person in front of us. In a way, we assume the role of a therapist, by being understanding and warm. When we build empathy for participants, we can walk in their shoes, and understand what is significant for them. With this, we can give context to the insights we gain, and appreciate the person we are speaking. On this level, we can create truly impactful solutions to their problems.
- Responses felt aligned with the flow of the interview. You responded to what is important to the interviewee. While we want to stay true to our research objectives, it is also essential to respond to the participant’s thoughts. I find that this is especially relevant for generative research, and is why I tell students to write a very sparse interview guide. Participants will always bring up what is most important to them when you ask a proper, open-ended question. Your job is to follow them on that path and allow them to be the leaders in the conversation. We help create boundaries with a strong introduction. The best thing you can do is respond to what the participant just said, by asking them why or asking them to explain further. I feel user research is a bit like improv — you are thinking on your feet and base your next question off of what that person just said. We aim to dig further into what a person has just said, especially with words you don’t understand or are highly subjective. Which is pretty much any descriptive word (easy, flexible, difficult) and any feelings a user expresses (frustration, sadness, etc.). If participants mention a potential solution, understand the why behind it and what problem it would solve for them. A great example is from one of my students — the participant kept saying that she used reviews or recommendations from other parents to help find after-school activities. This opinion was the perfect point to drill deeper: how did this impact her? What kind of information was she seeking? How did this help her? An example of the last time this happened. By staying on the same path the participant is walking down, we can understand the most critical information, which can result in more meaningful insights.
- References to the participant’s language. You acknowledge and refer back to what has previously been said, using the same style the participant is using. At times, moderators will ask the same question over and over again to get an answer they want, which ends up giving false data. If you find yourself asking questions multiple times, reassess the flow of the interview. Frequently, you will ask the first question, and the participant will respond with a long answer, with multiple paths to go down. I write down keywords, in the user’s language, that help spark my memory later in the conversation. After the user has finished their thought, I will go back to the first keyword and go through one-by-one in asking the user to explain more about what he/she meant.
- Correct interpretation. You were able to summarize the conversation, using the participant’s language, and did not assume, or put words into the participant’s mouth. For example, a participant said he would “appreciate more on-the-go information on his phone in terms of news.” The moderator responded, saying, “you would want more content tailored to your iPhone.” That response was a paraphrase of the participant’s thoughts. Another example is when a participant explains a feeling such as, “it made me feel upset, and I was yelling at the screen…” and the moderator says, “it sounds like you were angry.” The participant did not state that they were angry, and now we have put words in their mouths. This experience can be uncomfortable for participants and makes them feel unheard. Ensure you are using the participant’s exact words when reiterating what they just mentioned.
- A sense of curiosity. You show a natural inclination to understand people and dig deeper. No matter what the subject is, or what the person is talking about, you have the motivation to understand them. You crave the ins-and-outs of why a participant is saying or doing something, and you dig deeper past the surface-level of explanations.
- Silence was utilized correctly. You have used silence to your advantage. This last point can be related to being gentle and sensitive during the interview, but I think it deserves a separate point. Being silent, and allowing the person to talk is an extremely beneficial skill. Deep listening is one of the best ways to get people to talk to you.
While I am giving myself a score on the above principles, I also keep other questions in mind to ensure the overarching flow and structure of the interview went well. So, I am not only evaluating myself in core areas, such as empathy and silence but also how the conversation went. I keep the following questions in mind and write down short answers to each.
Additional questions for evaluating how an interview went
- To what extent are the participant’s answers spontaneous, vibrant, specific, and relevant?
- Are the interviewer’s questions shorter and the participant’s answers longer?
- Does the interviewer follows up and clarify the meanings of the participant’s answers, particularly emotions?
- Did the interviewer interpret the meaning of the participant’s answers throughout the interview without asking?
- Did the interviewer attempt to verify his/her assumptions of the participant’s answers?
- Was the interview self-communicating through natural, story-like answers?
When we take the time to evaluate our interviewing techniques, we can level-up our user research skills beyond what we can imagine. It is a great practice to get in the habit of, and can be a humbling experience!