CUSTOMER VALIDATION

How to Run Experiments for Product Market Fit

Use Pirate Metrics to guide you towards increased customer satisfaction, retention, and referral

Erik van der Pluijm
Erik van der Pluijm
2019-03-29 | 8 min read

Find the people that can’t do without your product. Photo by Carson Arias on Unsplash

When you have validated your idea and validated Problem Solution Fit, the next important milestone for your idea is Product Market Fit.

What is Product Market Fit?

There are many different conflicting definitions of Product Market Fit. Most are quite ambiguous, and make it difficult to understand when you have achieved it.

Product/market fit means being in a good market with a product that can satisfy that market.

- Marc Andreessen

While this definition by Marc Andreessen makes perfect sense, it is impossible to measure. What is a ‘good market’? What does it mean that the product can ‘satisfy’ a market?

If you have to ask whether you have Product/Market Fit, the answer is simple: you don’t.

- Eric Ries

The above statement by Eric Ries is profound, but it is no help to you if you are among those that ‘had to ask’…

So first of all, we need a definition that allows us to objectively measure if Product Market Fit is achieved.

Achieving product/market fit requires at least 40% of users saying they would be “very disappointed” without your product.

- Sean Ellis

40% of users seems like something that is measurable. There are for instance several surveys startups can use to get an idea from their users how they are doing. The question that gets asked is usually a variation of this one:

How would you feel if you could no longer use product X?

  • Very disappointed
  • Slightly disappointed
  • Couldn’t care less (it’s not that useful)
  • N.A. (I no longer use product X)

Asking this to your users will definitely give you an indication of how they think they feel about not being able to use the product. But it is still a question about a hypothetical situation, and as Rob Fitzpatrick teaches us in the Mom Test, people can’t really answer those questions truthfully.

In the end, feedback, surveys, and comments are great to have, but the only real test is behaviour shown while using the product or service.

It would be great to have experiments that you can run besides surveys, right?

How do you know you achieved Product Market Fit?

To test if you have achieved Product Market Fit, you can run several experiments.

The indication that 40% of customers indicate that they would be very disappointed if they no longer have access to a product can be surveyed, but it can also be measured in terms of retention. After all, if people want to continue to use something, they should be seen to continually use it.

This means, that Product Market Fit is for a big part depending on how well you can satisfy your user’s needs, and how happy they are with your solution. This satisfaction is the driver.

AARRR!

To enable measuring retention, it helps to use the AARRR — Pirate Metrics model. The letters AARRR stand for:

  • Acquisition. You acquire the user. This usually means a sign up.
  • Activation. The user uses your product. This means a first visit or login.
  • Retention. The user continues to use your product. This indicates they like your product.
  • Referral. The user likes your product so much he refers other new users.
  • Revenue. The user pays you.

For Product Market Fit, especially the Retention and Referral stages of the model are important. It’s not per se about revenue yet. It’s about how easy it is for you to keep users and to have people refer others to your product or service.

To prepare for your experiments, first map out your AARRR funnel. Try to figure out what you know or don’t know about conversion rates. Where do new users come from before they enter the ‘acquisition’ bucket? How many of them become ‘activated’? How many refer the product or service to others?

To prove you have achieved Product Market Fit, you should be able to retain at least 40% of your users as active users. (The period of activity may vary depending on the nature of your product or service, i.e. some things you use every day, others once a month — it’s not the frequency that counts, but being the preferred solution).

Steps towards Product Market Fit

Coming from Problem Solution Fit, the first step towards Product Market Fit is to achieve some (very) early traction. Can you go from your ~100 initial ambassadors to ~1000 enthusiastic users? And can you figure out how happy they are to use the product? This is your baseline.

Next, you need to learn how to influence and hopefully increase Retention. Retention is key in terms of Product Market Fit. You need users to be advocates of your product or service, and you must learn how to keep them happy and make them even more enthusiastic.

Finally, you need to learn how to increase Referral rates. Being able to influence this is key to lower your CAC (Cost of Acquiring a Customer) and is a good indication of how happy people are with your service. After all, people hardly tell their friends to use apps they don’t really like — even when they might still use those apps if they have no choice.

MVP

Where in the Idea Validation it was enough to have a rough sketch, and you could get away with a prototype for Problem Solution Fit Validation, now you really need to deliver on key features. That means, building a first functional product: an MVP.

There are many confusing definitions of what an MVP is, but I like to think of it as the minimum product that you can build to prove Product Market Fit.

Right after Problem Solution Fit, the first versions of that MVP will be rough (and won’t achieve the levels of enthusiasm in your users that you’re looking for) — working towards Product Market Fit you’ll get a better understanding of what users really look for in the product or service, and your MVP will develop.

Every feature you add or tweak needs to increase user happiness in terms of retention and referral rates. Most new features will start their lives in the form of small experiments, trying to gauge the effect of a change before actually implementing it.

Keep the ‘minimal’ aspect of MVP in mind! There will be a lot of varied and confusing feedback coming from users, and simply building everything they ask for will not help you. The feedback needs to be deconstructed and you need to look for the biggest common denominators, the real reasons behind the comments. These will show up as risky assumptions when defining your experiments.

How to run a Product Market Fit experiment

Especially in this stage, it is vital to have a good experiment setup and use the experiment canvas as a guide. There are more variables to be aware of than in the earlier stages, and it can be tempting to ‘just try’ a large number of different things — which can lead to going in circles, not getting clear signals to guide your development, which is the whole point of doing experiments.

The point of running experiments is to get clear signals.

(Note: This doesn’t always mean that an experiment should lead to an improvement! It could be that you run an experiment to figure out if a certain feature really is important, and test it by removing it. If the output changes dramatically, now you know that feature is important, and you know it’s important to keep supporting it.)

To get clear signals, you need to test the most important hypotheses, and to do that, you need to look at the riskiest assumptions.

It is vital to keep asking what users are unhappy about, and why they are unhappy about that. Come up with assumptions of the reasons behind what users tell you, behind the behaviours you observe. Use the Riskiest Assumption canvas to sort through your assumptions for the best one to test.

Examples of risky assumptions for product market fit:

  • People prefer this solution to competing solutions or workarounds
  • People enjoy the solution so much they want their friends to use it
  • The ambassadors we found are not ‘unique’, there are many more potential users we can find
  • People come back to use our product and promote it to their go-to solution
  • Adding a ‘share’ button will lead to a higher referral rate
  • Adding notifications will lead to higher number of active users, which will lead to higher retention

Example hypotheses for product market fit:

  • 40%+ of the 100+ users we survey indicate that they would be very disappointed if the product or survey was discontinued.
  • 40%+ of the 100+ users we survey indicate that they shared or want to share the solution with their friends
  • 40%+ of 100+ people we interview are willing to give us 3 email addresses of friends
  • 40%+ of our users are active daily/weekly/monthly users
  • 40%+ of our users use the ‘refer to friend’ function in the app
  • Adding new feature X increases retention rates by 5% in 1 month

Example methods for product market fit:

  • Face to face interviews — get a good idea of what they like or hate.
  • User tests — what are people actually doing? What do they think while using the product?
  • Campaigns — make campaigns on e.g. facebook that tests how people respond to particular feature sets.
  • Referral experiments— ask your users to refer others (with or without a reward) and see if they do.
  • Fake features — Add new features in the product as only a button with a ‘coming soon’ screen behind it with an option for feedback, and see how many users actually click it.

Keep in mind that the main reason to run experiments in this stage is still not to grow (although it would be nice!) but to learn how you can grow and to increase satisfaction.

A good analytics setup is vital. You need it to be able to quantify if you are working towards Product Market Fit.

The tools to use are a good AARRR model, built in a spreadsheet for instance, that has baselines for retention and referral; the customer journey and personas made earlier (as reference); surveys; user tests; and of course, analytics.

So, if you are working on a digital product or service, it pays to spend time and effort in setting up your analytics across your Pirate Metrics funnel, and to be able to get a grasp of where your users come from, what channels are the best, and what your most happy and loyal users actually do when they use your system. If you don’t know that, you are basically flying blind.

Remember that getting from a freshly validated Problem Solution Fit to a solid Product Market Fit will in most cases take quite some time. This is where you find out if your idea and the solution you built on it actually have a shot at becoming the next big thing. If you crack the code of making users fall in love with your product or service you’re in business!

Keep experimenting!