Measuring UX success

Measuring success

Measuring success is generally said to be an easy thing to do. Are your sales increasing? Are more people subscribing to your content? Is your user base increasing?

But sometimes your success isn’t that easy to measure. Even if your product doesn’t rely on sales, or is free to use, you still have to justify your decisions to stakeholders. Developers’ time is expensive. Your time is expensive. Adding new features is expensive, so understanding why these features are needed will help justify the time to build them.

 

 

Who needs to be involved from each team:

• Product Manager
• Performance Analyst
• User Researcher
• Other team members (optional)

 

Which questions to ask yourself:

1. Goal: Why are you building it?
2. Hypothesis: What do you think will happen when you release it?
3. Measure: How do you measure it?
4. Action: What are you going to do about it?

 

If you measure something and it’s not tied to a goal, in turn changing your behaviour, you’re wasting your time.

 

Step 1 Goal: Why are you building it?

There must be an overarching reason for doing it (“because we’ve been told to” is not a good enough reason…). Make sure your goal is based on user needs, not just business needs, or you could be in trouble. As a lot of our products tie in together, some goals may overlap.

It may be one main goal (sell more products), or two or three. Try to not do over three to begin with, as you can always add more later.

 

Step 2 Hypothesis: What do you think will happen when you release it?

It may just be a gut feeling. Guts matter, you’ve just got to test them. Instincts are experiments. Data is proof. But remember to tie it in to a goal. What goal are you hoping this will help fulfill?

 

Step 3 Measure: How do you measure it?

Your KPI has to be measurable, but it doesn’t have to be measured in Google Analytics or something similar.

Don’t be afraid to use qualitative user research to measure.

A lot of products won’t be openly available to the public straight away and therefore won’t have the user base for quantitative measuring.

Step 4 Action: What are you going to do about it?

This depends on what you’re measuring and what the results are. Do you need to make a change to the site? Do you need to investigate further with qualitative user research? Do you need to do an A/B test? Can you leave the monitoring running as an ongoing operational metric?

You should already be doing this with user research. When you get new information, you do something with it.

 

Setting KPIs when you’re starting out…

After your Discovery phase, you should have enough info to know what the goal/s for your project is. Talk to the rest of your team.

You should also have enough info to work out the hypothesis for the features which go towards your MVP and beyond. You don’t need to know all of these at the start. Your project evolves and grows, and so do your KPIs.

 

You’re drawing a line in the sand, not carving it in stone 

When you’re starting out on a brand new project you’re chasing a moving target, because you don’t really know how to define success for your product.

Adjusting your goals and how you define your key metrics is fine, as long as you understand WHY you’re changing them and not just doing it to “hit your targets”. Are your users using your product in a different way than you expected? Fine, that warrants a metric change.

 

Remember: Without an initial line in the sand, you have no benchmark for learning.

 

Choosing the right metric…

Think of it as the way to find the answer to a question.

Talk to your User Researcher and your Performance Analyst. Sometimes other team members may be able to help as well, like Data Analysts.

A fortnightly chat seems to work well as a start but do whatever works best for your team.

 

A good metric is…

Comparative

• Being able to compare a metric to other time periods, groups of users, existing site, other metrics, etc, helps you understand if you’re improving

Understandable

• If people can’t remember and discuss it, it’s much harder to implement the changes needed

Changes the way you behave

• If over 50% of your users say they don’t need a feature, don’t build it. If an A/B test shows one version performs better, make that one.

Qualitative or quantitative metric

Qualitative metrics are unstructured, anecdotal and very revealing.

• Surveys and open text field polls
• Open card sorts
• Interviews and observations, etc

Quantitative metrics involve numbers and statistics, and provide hard numbers

• Analytics
• Closed card sorts
• Tickbox polls
• A/B or MVT tests, etc   

 

Vanity vs actionable metrics

Vanity metrics make you feel good about what you do, but don’t change how you act

• Time on site (what did people actually do?)
• Number of visits (count unique visits instead)
• Number of emails collected (this does not mean they engage with what you send)

Actionable metrics change your behaviour by helping you pick a course of action.

• Percent of users who regularly engage with content
• Number of users acquired over a time period (due to social media campaign)

 

Exploratory vs reporting metrics

Exploratory metrics are speculative and try to find the unknown insights. This is where most our products are right now.

• Which version works best
• How many users want/need this
• Testing hypothesis

Reporting metrics keep you abreast of normal, managerial, day-to-day operations. This is more for when you know what you need to measure

• Improving engagement
• Improving sales

 

Leading vs lagging metrics   

Leading metrics gives you a predictive understanding of the future. You can only really use this when you have enough data to work with already.

• Users submitting evidence may predict users attending committees
• Users putting things in baskets may predict number of sales
• Queries to switchboard can tell you what you need to fix

Lagging metrics explain the past, and can give an indication of if there is a problem

• Queries to switchboard over a period
• Drop offs in a journey funnel

 

Correlated vs causal metrics

If two metrics change together, they’re correlated. Correlations can help you predict what will happen.

• Sales of ice cream and number of drownings both go up at the same time, but changing one won’t change the other.

If one metric causes another metric to change, they’re causal. If you find a causal relationship between something you want and something you can control, then you can change the future. It’s rarely 100% though. One metric can be caused by multiple others.

• Beta pages had a unique visits spike in July. Caused by a burst in social media and news, in turn caused by new photos.

 

Don’t…

• …assume that the data is clean – does 30% of your users actually spend less than one second on your site, or are they bots?
• …forget to normalise – if you’re creating a list of the most popular tubestops for commuters, remember to take into account the total number of travellers to each stop, otherwise you’ll just get a list of busy tubestations
• …forget to look at outliers – those 21 users who visit your site 100 times a day are either your biggest fans, or bots. Either way, check them out.
• …forget to exclude outliers – those 21 users are great from a qualitative perspective but remove them before you build a general model.
• …ignore seasonality – Christmas time is not a “normal” time. It will skew your data
• …ignore size when reporting growth – when you’re just starting out, your parents visiting does technically count as doubling your user base…
• …succomb to data vomit – a dashboard isn’t much use if you don’t know where to look
• …ignore data from other sources – are the schools who visit most often also really active on Instagram?
• …focus on the noise – the human brain is built to spot patterns where there are none. Capture everything, but focus on what’s important.

 

 

Template

I created this template to help teams start specifying what and how to measure success. This template has successfully been adopted by both public and private sector companies as a way to decide on the importance of large and small releases and new features.

We are going to launch [feature], which meets [goal], because we think it will [hypothesis] (and [hypothesis]). We will rate the success of this by using [method] to measure/compare [metric].

 

For example, this was used by the Parliamentary Digital Service to help measure success of their features when relaunching the UK Parliament public website.

We are going to launch Committee Roles, which meets “Find out how a member represents me”, because we think it will increase number of contacts. We will rate the success of this by using analytics to compare clicks on email links and phone numbers with existing clicks, and using a members office survey to compare number of contact with existing numbers.  

 

TEMPLATE – Who may be best placed to advise on what

We are going to launch [feature], which meets [goal], because we think it will [hypothesis] (and [hypothesis]).  – Product Manager / Team

We will rate the success of this by using [method] to measure/compare [metric]. – User Research / Performance Analyst