Why experiment?

People in organizations are often afraid of running experiments. I have observed ca. 70% of people responsible and accountable for products are reluctant to validate their assumptions by experimenting. There are several reasons for this hesitancy.

The most common is misunderstanding what exactly the Agile experiment means. The second one is the negative result of the experiment. I always answer that this is fine that we have such a result! It means that there is no need to develop a solution that no one wants or does not fulfill users’ expectations.

Users & Customers

To understand the nature of experiments, the first thing is to know users. Interestingly, still the majority of Scrum Teams know nothing or little about their users.

I would recommend applying a proto-persona and validation. Questions that can be asked:

  1. Who is our user/customer?
  2. What problems are we trying to solve for our users/customers? What do they need?
  3. What would the outcome be for our users/customers?
  4. What would the impact be on our product?

Outcomes not outputs

We need to differentiate outcomes from outputs. Outputs are results of our activities like a number of features/solutions, bugs, story points. These measures do not indicate about the value delivered. Organizations have been mainly measuring outputs. At least, delivery departments.

Outcomes are measurable changes in user/customer behavior. These are the results of outputs. Examples of outcomes: conversion, churn, product usage, customer usage, customer/user satisfaction, etc.

Measuring value

The topic of outcomes is related to measuring value. Without measuring value, how do we know that we actually delivered any outcomes? More information can be found on Scrum.org and my website.

Running experiments

The shortest answer to the question “why experiment?” is to validate your assumptions. Again, the majority of organizations make assumptions about the customers/users’ usage, solving their problems or fulfilling their needs without any single trace of evidence. We assume that our proposed features, services, solutions, the product will work! In other companies, especially those who do not measure value, there is even no conversation about customers/users’ needs. Lack of measuring value. This is a vicious circle.

Spreading the agile mindset that encourages frequent inspection and adaptation is the first step. It will help to understand the need for experimentation. I am often asked how to convince business and product people to allow Scrum Teams to run an experiment. Use these questions/statements below:

  1. How do we know if this solution will deliver value for customers? If the answer is vague, continue.
  2. The experiment we mean is short and low-cost. We do not need to spend Sprints for validation. Share examples of quick, low-cost experiments: fake feature, paper/digital prototype, observations, user interviews, landing page.
  3. Calculate. By short experiments, I mean hours, not months.

Example:

Let’s compare:

One day experiment (provide costs) => immediate data about the value or in a short period of time.

or

Delivering the entire solution in 3 months (provide costs) => data after 3 months or later about the value.

Is it worth risking and investing such an amount of money without any validation?

4. Write hypotheses. Combine them with a business problem statement, expected outcomes, user benefits and measures.

5. What if the experiment brings a negative result? This is fantastic news! We do not need to spend 3 months (replace by your estimation), people effort and company money to deliver something that is not valuable.

6. Who will be the subject of the experiment? Ideally, real users/customers. When you do not have direct access to them, try to find people in your environment who are users (or potential users of the solution) and match your personas. You can ask your colleagues who are not involved in the product delivery.

experiments

Experiments are valuable

The next step is to design an experiment. You can combine a few Lean UX practices. Example: interviews + observations + prototypes. Useful information about Lean UX can be found here.

Ideally, you want UX people involved in this process. If you work with Scrum, a UX person should be a part of the Scrum Team. The entire Scrum Team might run experiments.

Conclusion

It is worth investing a few hours or days in validating our assumption on real users to prove if you are right or wrong with your solution. Delivering the entire solution without any validation is costly and unnecessarily risky. Think in terms of outcomes, not outputs. Involve the whole team to help you run experiments. Measure value with Evidence-Based Management and make decisions based on facts.

If you’d like to experience and know more about Scrum and experimentation, the Professional Scrum with UX class is all about this topic.

This article was written by a human, me, not AI.

Facebook
Twitter
LinkedIn

Leave a Reply

Your email address will not be published. Required fields are marked *

More To Explore

output vs outcome title

Outputs or outcomes?

While working with organizations on defining their product value, I have noticed that outputs are often confused with outcomes. Products and companies are mostly focused on outputs and company impacts, while outcomes seem to be ignored.

Read More »

Key Value Metrics

Measuring Value is one of the most essential activities in product management. For instance, for Product Owners and Managers, this process is a background for making informed decisions about the future of the product and assessing the current value and state of a product.

Read More »