This week, I’ve been studying the “conversion research” section. If you’re a brand new business, it is recommended that you read these three books.
- Startup Owners Manual by Steve Blank
- Lean Startup by Eric Ries
- Lean Analytics by Alistair Croll
The ResearchXL Framework is introduced which is a six-step framework that helps you to assess user experience, identify problem areas, identify functional problems, and also draw insights. I go into greater detail on these in a prior post, so I’ll stick to the main points:
- Heuristic Analysis — Focuses on clarity, friction, anxiety and distraction.
- Technical Analysis
- Digital Analytics
- Qualitative Research
- User Testing
- Mouse Tracking Analysis
Peep in this section recommends using a five point scale to prioritize areas of concern.
- 5 Star — This rating is for critical usability, conversion, or persuasion issues that will be encountered by many visitors to the site or has high impact. Implementing fixes or testing is likely to drive significant change in conversion and revenue.
- 4 Star — This rating is for a critical issue that will be encountered by many visitors to the site or has a high impact.
- 3 Star — This rating is for a major usability or conversion issue that may not be viewed by all visitors or has a lesser impact.
- 2 Star — This rating is for a lesser usability or conversion issue that may not be viewed by all visitors or has a lesser impact
- 1 Star — This rating is for a minor usability or conversion issue and although is low for potential revenue or conversion value, it is still worth fixing at a lower priority.
There are two criteria that are more important than others when giving the star score:
- Ease of implementation (time/complexity/risk) — Sometimes the data tells you to build a feature, but it takes months to do it, so it’s not something you’d start with.
- Opportunity Score (subjective opinion on how big of a lift you might get) — This depends on how many users are exposed to the issue, and how close to the money the issue is. Let’s say you see that the completion rate on the checkout page is 65%. That’s a clear indicator that there’s lots of room for growth, and because this is a money page (payments taken here), any relative growth in percentages will be a lot of absolute dollars.
Follow The Money — The biggest thing in testing is to get as close to the money as possible and so you can immediately measure and impact the bottom line of increasing revenue/profits.
Measuring the Effectiveness of a Testing Program — In order to avoid wasting time by repeating the same testing mistakes, it’s vital to continually self-assess the effectiveness of your testing programs. The three metrics recommended by CXL are:
- Testing Velocity
- Percentage of tests that provide a win
- Impact per successful experiment
A common theme I’ve seen is to get close to the customer, through direct interaction and also through customer-facing staff. By using quality surveys, you’ll be able to glean insight directly from customer feedback that help craft better messaging and marketing campaigns. Surveys can be done directly by phone, via email, or web exit to name a few.
Web Exit Surveys — If you want to increase the conversions on your app or website, you have to figure out who your primary target audience is, what they want, what matters to them and what the sources of friction are for them that prevent them from purchasing. Don’t fall into the trap that “everybody” is your target audience. If you fall into that trap, then you will kill your ability to boost conversions.
Qualitative research is mostly learning about WHO the customers are, what they want, and the language that they use. This is critical for copywriting, understanding friction, learning what matters to them about the products you sell, and so on.
Conversions are all about relevancy — if what you offer and how you present it matches their state of mind, you have gained a customer. IF you customer is “everybody,” you’re making it extremely difficult for yourself. Nobody will identify with “everybody.”
Who to Survey? Survey people who still freshly remember their purchase and the problems/concerns, aka “friction, they experienced in the process of purchasing or signing up. Only talk to your recent first-time customers (who have no prior relationship or experience that could bias their responses). You also want to filter out repeat buyers or people who bought a long time ago. If you ask somebody who made the purchase 6 months or more ago, they have long forgotten and might feed you with false information.
How Many People to Survey? The best online surveys are qualitative (open-ended questions). We still need a good number of responses to get an adequate overview. If you only survey 10 people, a few people’s opinion is going to drastically skew the picture and it will make it harder to identify false patterns.
The quantity recommended by CXL is somewhere between 100–200 people. You don’t need more than 200 because the answers get repetitive and don’t offer additional insight. You also want to limit “asks” like a survey to customers in order to keep strong relationship equity with them. This is similar to an opinion poll. Any answers less than 100 and you won’t have enough answers to draw conclusions from. If you do only have 100 responses though, 100 is better than 10 or 20. So do the best with what you’ve got. Some feedback is better than no feedback.
Since you are primarily running a survey for qualitative purposes, and you are not going to need to make numerical comparisons between two data sets, this means you can feel reasonably comfortable with fewer responses. The fewer sessions you gather, the wider your margin of error becomes. For example at 90% confidence, the margin of error looks like this for sample sizes less than 1,000:
As you can see, with 200 responses your margin of error is close to ±6%. That’s perfectly acceptable for qualitative surveys. This is mostly about spotting patterns and learning about the voice of the customers, not exact percentages. Also, note that there are diminishing returns with the larger the number.
