Business Experiments
Taking Intelligent Risks
(Also known as Pilot Projects and Pilot Experiments)
Your boss has asked you to develop a new service that will boost customer retention, and you've come up with several ideas that you think might work.
The problem is, you're not sure which ones will have the biggest impact. And you don't have the time and resources to implement each idea fully. So, which idea should you move forward with?
This is one situation where it's useful to set up business experiments. By running these, you can test ideas on a small scale, before you take big risks or commit significant resources to a larger project.
In this article, we'll discuss how to conduct business experiments, and we'll look at how you can learn from them.
Why Experiment?
Many businesses don't grow unless they innovate, and innovation comes from developing and implementing new ideas. However, innovation usually carries a risk of some kind: fundamental assumptions may be wrong, new products may not work sufficiently well, customers may not accept your ideas, and so on.
This is why it's so useful to run business experiments. By testing ideas, you can understand how they might perform on a larger scale. You can also learn from your failures and mistakes, and refine your ideas to make them better.
Business experiments can involve anything from very basic tests (for example, changing the welcome message on your website or testing a new process for dealing with telephone enquiries) to more-complex projects (for example, testing a new product or service with a small number of customers).
Business Experiments and Risk
Business experiments can sometimes seem fraught with uncertainty and risk. After all, there's often no way to know whether they will succeed.
However, that's kind of the point. When you do a business experiment, you're opening yourself up to risk in a very controlled way. Although you clearly want to avoid failure, you're actually willing to fail, so that you can avoid more costly mistakes in the future, and so that you can develop and exploit high risk opportunities.
In the short term, business experiments may seem too small and too risky to be worth the effort. However, when you look at the benefits of these over the long term, you'll see that it's often more costly not do to them. By being willing to learn - both from the ideas that succeed and from the ones that fail - you'll uncover the successful ideas that will help your business or organization be the very best that it can be.
Tip 1:
Depending on your role, you may need to get permission from your boss or senior management team before you conduct a business experiment, even if the risks are minimal. Use your best judgment here.
Tip 2:
If you need to plan how to get support for your experiments, see our article on Sales Skills for Non-Salespeople, and our Bite-Sized Training session, Sell Your Idea!
Tip 3:
Business experiments won't be suitable in all situations - for example, where you're dealing with a very small market, or where success can be copied very quickly. Here, you'll have to make a decision based on your instincts and on the information that you have available.
Tip 4:
Before you conduct a business experiment in a "live" environment, you may also want to test your ideas with users and customers through interviews, focus groups, or surveys, where appropriate.
How to Organize an Effective Experiment
To organize an effective business experiment, consider using the approach below. This is based on a process developed by innovation expert, Thomas H. Davenport.
1. Create a Hypothesis
As with scientific experiments, the first part of the process is to create a hypothesis - this is your prediction about what will happen if your test is a success. For example, "Sending out two extra mailers a month will increase overall revenue" or "Moving the navigation to the left-hand side of the website will keep users on the website for longer."
As you create your hypothesis, think about the following questions:
- Can you measure what happens when you make the change? (You need to be able to "pass or "fail" the test, based on your hypothesis.)
- Does it fit with your team and organization's overall strategy, goals and values?
- Does it add value to what you do?
- If the answer to any of these questions is "no," then you may want to amend your hypothesis or consider not doing the experiment at all.
- Finally in this step, identify how you'll measure the success of your experiment. Ask yourself, "What does 'success' look like?"
What you measure will depend on your industry and on the type of experiment you're conducting, and it can include metrics such as sales and revenues, new enquiries, visits to key web pages, or more general factors such as ease of use, or feedback from customers or team members.
Again, also think about how these measurements will affect your overall goals and strategy. For instance, you might define success as "Increasing website page views." However, this won't be viewed as a success if it causes a drop in overall revenue. (This might happen because people have stopped going to your online store.)
2. Design Your Experiment
Now you need to think about how you'll conduct your business experiment. Outline what you'll be testing and how long you'll be testing it for.
Testing a new product or process is relatively simple - you just need to run the experiment for a specific time, and then use the metrics you identified in the previous step to evaluate its success.
However, if you're testing a modification to an existing product or process, you need to compare "like for like" scenarios, so that you can measure the success of your hypothesis. You can do this by using a "control group" and a "treatment group." (This is sometimes called A/B testing or split testing.)
Start by controlling as many variables as you can - that is, try to minimize change wherever possible.Your control group is what you use for your baseline measure. Try to avoid making any changes with this group during your experiment. Do the same with your treatment group, except for the change that you want to test. You can then compare your control group with your treatment group to test your hypothesis.For instance, imagine that your hypothesis is, "Opening until 10 p.m. will increase profits in retail stores." You choose two stores as the treatment group; these will open until 10 p.m. Your other stores are your control group; these won't change their opening hours. Then, after two weeks, you'll compare each store to see if opening until 10 p.m. increased profits during the test period.
In some cases, it will be impractical to use a control group and treatment group simultaneously. (You wouldn't be able to run the test above if you only had one store, for example.)When this happens, you can use carefully-selected past data as your control group, and then use the data that you get when you make the change as your treatment group. In the example of the store with only one site, you could compare sales this year with sales for the same period last year.
Overall, keep your experiment as simple as you can. The more complex the experiment, the more it will cost to do, the more risks it will involve, and the more time it will take to analyze the results. However, you need to make sure that your experiment will provide meaningful data.
Tip 1:
When you use a control group and a treatment group, test one factor at a time. For example, if you tested extended opening hours and brighter lighting, you wouldn't know how which of these affected profits. (You can test other factors in further experiments.)
Tip 2:
To get a better understanding of the risks associated with a business experiment, use tools like Risk Analysis and the Risk Impact/Probability Chart.
Tip 3:
If you need to design and build a prototype service or product to test, don't underestimate the amount of effort that will need to go into this. If you produce something that is sub-standard, your experiment is more likely to fail.
3. Run the Experiment
Once you've designed your business experiment, it's time to take action and run it.
Inform everyone in your organization who is affected by the test, before it goes live. They should know how you'll be conducting it, and why it's taking place.
Once the test is live, monitor it to make sure that nothing happens that could distort your findings. For instance, factors such as bad weather could affect sales in the example we highlighted above. If events like this do happen, consider re-running the experiment or removing some of the data from your final analysis.
If you're testing new products and services with customers, you may want to let them know that they're using a test product. This is a useful way to protect your reputation if things go wrong. It also encourages people to give feedback.
Tip:
You may also want to make sure that people who are affected negatively by the experiment are compensated appropriately.
4. Analyze Results and Follow Up if Necessary
Once you've conducted your experiment, take some time to analyze the results thoroughly.First, compare actual performance against your hypothesis:
- What was supposed to happen and what did happen?
- Why was there a difference?
- Was the experiment a success? (Don't be too disappointed if it wasn't!)
- What unexpected consequences happened as a result of your experiment? How could you manage or take advantage of these?
Then think about how you can use what you've learned. Would conducting further experiments help you get even more value from your idea?For instance, would opening stores until 9 p.m. instead of 10 p.m. make them more profitable? Or would modifying a product further result in even better feedback from customers?
In some cases, you can take action on what you've learned at the same time that you continue to conduct further experiments. Other times, you may have to conduct follow-up experiments before you can understand the best way forward.
Note:
There is a whole range of statistical tests that you can use to analyze the results of business experiments.
When you use these, bear in mind that it's often very difficult to control external factors in business experiments. This means that significance/confidence levels are often much lower in business experiments than they can be in laboratory experiments.
Key Points
Business experiments help you test your ideas and gather further information, before you commit significant resources to a large project. They can be anything from very basic tests to complex projects involving prototype products or services.To conduct an effective business experiment, do the following:
- Create a hypothesis.
- Design your experiment.
- Carry the experiment out.
- Analyze results, and follow up if necessary.
Sometimes, you can implement what you've learned while you continue to conduct further experiments. Other times, you may have to conduct follow-on experiments before you can understand the best way forward.