A case for beta testing
How User Research & Beta Testing Intersect
What is beta testing?
Beta testing is a wonderful, and often underused tool, that allows you to ascertain user feedback for a live product. Since beta testing only requires you putting the product into a small number of percentage of your users, you can correct any bugs or improve UX before you roll out your product to everyone. Essentially, you use these beta testers to see whether they are actually using the product/feature, how they are using it and what bugs, improvements or innovations come up. If beta testers aren’t using certain features, the chances are your regular user base won’t be either. It is during this stage, you are able to find out why through beta testing.
I don’t necessarily believe beta testing and user testing are different things. In fact, I think they fall under the same umbrella and is more of a continuous process in which testing at a particular point of the software release is seen as doing beta testing. However, many people do beta testing without user research and vice versa, such as in analyzing software performance, product development pipeline and marketing analysis. Usability testing shows you if a concept is usable, while beta testing shows you if/how people will actually use the live product.
The biggest difference is that beta testing comes in when the product is actually live, rather than concept or prototype.
Why should you beta test?
I can honestly tell you beta testing has many different benefits, and is a great concept to bring to your company. Since it can sit in the realm of user research, but also in marketing and sales, it can lead to cross-departmental collaboration and an overall better experience for your customers. Here are a few benefits I have seen from my past experience with beta testing:
- The ability to do continuous research
- Cycle of always testing new features
- Finding bugs or UX improvements before they become a large-scale problem
- Show users you care deeply about their opinion
- Build a community of engaged users that will give you feedback on your new ideas, features and releases, which, in turn, helps them feel valued
Just writing this list gets me excited to start a beta program at my next full-time job. A lot of established companies, and even startups, are absolutely at a place where they can start a beta program, which means they can start doing continuous user research, which is the key to a healthy company. Happy users = happy company.
When should you beta test?
Beta testing is a wonderful concept for a few different scenarios:
- If you are looking to get started with user research and already have a live product
- If you want to test and learn about different features before complete rollout
- If you have very specific questions you want answered about a certain feature or flow in order to make informed decisions
- If you are interested in tracking analytics in terms of usage before rolling out to your entire user base
- If you are trying to find bugs or issues with flow on new releases, features or products
The most important part to note is that beta testing is done when a product is live, and that includes products that are just at the MVP (minimal viable product) stage — in fact, it is encouraged to beta test products at the MVP stage.
Start a beta testing user research program
There is no one way to start a beta program (or anything, really), so I am simply speaking from my experiences of my past. It can take some time to get it started, especially if you are starting 100% from scratch, but the benefits outweigh all of the upfront work. Once you have your beta program established, all you have to do is simply maintain it, and enjoy all the user feedback.
- Define the goals and rules of the beta program
What do you want from this beta program? Will you be engaging these users on a weekly, monthly or quarterly basis? How will you be engaging them? Will it be simply for new features, or for innovation and new product releases? How often will you require them to give feedback?
For example, at one company, we reached out to beta testers when we had a new feature we wanted to test before we did a massive rollout. There wasn’t necessarily a continuous timeframe we strictly followed, but, I would say, we spoke to users just about every month or two about this. The goal of the beta program was to grow a community of users we could always reach out to in order to test new features. After some time, we were also able to engage these users about new ideas (concept testing), before we even got to live code. - Create a beta program sign-up
Figure out a place where you can ask users if they are interested in joining the beta program, and allow them to sign up, whether that be via a website or another online tool. Where ever the sign up lives, it should state the following:
- Explain the benefits of joining the beta program
- Detail the requirements of the beta program
We required out users to give us feedback on certain features or product rollouts every two weeks. It obviously depended on the scale of the beta test, but we set a very clear timeframe. I would schedule the feedback sessions in advanced, and make sure to email them with reminders.
In addition, I made sure the benefits were very clear: they were able to see all new features/products before the masses, their feedback visibly made changes to our product, they could help inform our product roadmap and, every quarter, I sent a gift basket to our beta testers (once we got too many, I rotated, and tried to get everyone at least once a year) - Invite people!
Once the internal parts are figured out, it is time to invite users to become beta testers. I spoke to account management and marketing to understand who they thought the best beta testers would be. I had also been doing user research, and marking down who I thought great candidates were. I emailed all of them directly with a link to our sign up, and offered to speak to them via phone call, if they had questions. I would also ask users at the end of research sessions if they were interested in joining. Once people join, try to segment them into groups, such as “power users,” “very willing to give feedback,” or “trouble with technical issues” — that way, you have an idea of which users to potentially select for certain beta tests - Beta test!
Now comes the fun part, you have your users (which, again, may take a little time) and now it is time to beta test! You can choose to beta test a feature with all your beta testers or you can test different versions with different groups of beta testers. Once you have decided, compose an email telling them what to expect once you turn the feature on. Don’t give them too much, as you want to encourage them to explore on their own, but let them know you are readily available for questions and feedback.
Make sure to remind users when they are scheduled for feedback sessions, and how you plan to gather their feedback. We did 95% of our feedback sessions over remote video conferencing - Receive feedback
As mentioned, I created feedback meetings with our users after a predetermined amount of time. I recorded the meetings (with permission, of course), took notes and then compiled all the feedback into research summaries. I would send the research companies to relevant and interested parties, such as the scrum team working on the feature/release, as well as marketing or sales - Continue the cycle
Once you have created and fostered a community of users, you will find how easy it is to conduct continuous user research. You will be able to reach out to users for more than just beta testing, and engage them in concept testing, usability testing and even generative research. It is a really powerful tool to make your users feel heard, and to encourage them to share their thoughts and opinions with you
Just a word of caution — beta testers are most likely going to be power users of your product. They aren’t going to be completely representative of your user base and population. Therefore, keep in mind that not all their feedback will be helpful for other users. We used beta testing in addition to user research, which meant we got a really holistic picture. Don’t simply rely on beta testing, but use it to get started on continuous user research!
[embed]https://upscri.be/ux[/embed]