Magdalena Firlit logo
Magdalena Firlit logo

Why experiment?

People in organizations are often afraid of running experiments. I have observed ca. 70% of people responsible and accountable for products are reluctant to validate their assumptions by experimenting. There are several reasons for this hesitancy.

The most common is misunderstanding what exactly the Agile experiment means. The second one is the negative result of the experiment. I always answer that this is fine that we have such a result! It means that there is no need to develop a solution that no one wants or does not fulfill users’ expectations.

Users & Customers

To understand the nature of experiments, the first thing is to know users. Interestingly, still the majority of Scrum Teams know nothing or little about their users.

I would recommend applying a proto-persona and validation. Questions that can be asked:

  1. Who is our user/customer?
  2. What problems are we trying to solve for our users/customers? What do they need?
  3. What would the outcome be for our users/customers?
  4. What would the impact be on our product?

Outcomes not outputs

We need to differentiate outcomes from outputs. Outputs are results of our activities like a number of features/solutions, bugs, story points. These measures do not indicate about the value delivered. Organizations have been mainly measuring outputs. At least, delivery departments.

Outcomes are measurable changes in user/customer behavior. These are the results of outputs. Examples of outcomes: conversion, churn, product usage, customer usage, customer/user satisfaction, etc.

Measuring value

The topic of outcomes is related to measuring value. Without measuring value, how do we know that we actually delivered any outcomes? More information can be found on Scrum.org and my website.

Running experiments

The shortest answer to the question “why experiment?” is to validate your assumptions. Again, the majority of organizations make assumptions about the customers/users’ usage, solving their problems or fulfilling their needs without any single trace of evidence. We assume that our proposed features, services, solutions, the product will work! In other companies, especially those who do not measure value, there is even no conversation about customers/users’ needs. Lack of measuring value. This is a vicious circle.

Spreading the agile mindset that encourages frequent inspection and adaptation is the first step. It will help to understand the need for experimentation. I am often asked how to convince business and product people to allow Scrum Teams to run an experiment. Use these questions/statements below:

  1. How do we know if this solution will deliver value for customers? If the answer is vague, continue.
  2. The experiment we mean is short and low-cost. We do not need to spend Sprints for validation. Share examples of quick, low-cost experiments: fake feature, paper/digital prototype, observations, user interviews, landing page.
  3. Calculate. By short experiments, I mean hours, not months.

Example:

Let’s compare:

One day experiment (provide costs) => immediate data about the value or in a short period of time.

or

Delivering the entire solution in 3 months (provide costs) => data after 3 months or later about the value.

Is it worth risking and investing such an amount of money without any validation?

4. Write hypotheses. Combine them with a business problem statement, expected outcomes, user benefits and measures.

5. What if the experiment brings a negative result? This is fantastic news! We do not need to spend 3 months (replace by your estimation), people effort and company money to deliver something that is not valuable.

6. Who will be the subject of the experiment? Ideally, real users/customers. When you do not have direct access to them, try to find people in your environment who are users (or potential users of the solution) and match your personas. You can ask your colleagues who are not involved in the product delivery.

experiments
Experiments are valuable

The next step is to design an experiment. You can combine a few Lean UX practices. Example: interviews + observations + prototypes. Useful information about Lean UX can be found here.

Ideally, you want UX people involved in this process. If you work with Scrum, a UX person should be a part of the Scrum Team. The entire Scrum Team might run experiments.

Conclusion

It is worth investing a few hours or days in validating our assumption on real users to prove if you are right or wrong with your solution. Delivering the entire solution without any validation is costly and unnecessarily risky. Think in terms of outcomes, not outputs. Involve the whole team to help you run experiments. Measure value with Evidence-Based Management and make decisions based on facts.

If you’d like to experience and know more about Scrum and experimentation, the Professional Scrum with UX class is all about this topic.

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn

Leave a Comment

Your email address will not be published.

More To Explore

agile transformation

Why sometimes Agile Transformations fail?

Based on my experience working in Agile environments since 2010, I did some research and general observations on the topic. Regardless of the business domains or product area, there are some common rules that may devastate your efforts towards Agility.

Read More »
transparency

Fight for Transparency

A significantly increasing number of companies strive for Agility. They intend to progress with Agile Transformation.
Based on my research from several dozen organizations, what might prevent them from benefiting from the Scrum (or just an Agile mindset) is lack of Transparency. In these circumstances, any change seems to be impossible to proceed.

Read More »
Scroll to Top