What is an Experimental Program and Who is Involved? (Experimental Program Series: Guide 02)

In my previous postI briefly described how leading companies are using experiments to optimize their products and services and evolve them to the point where they feel elegant, efficient, and magical. These companies have developed mature experimentation programs (ExPrs), including the infrastructure, people, and processes for using experiments in areas such as product development and marketing.

But most companies are at an earlier stage of embracing experimentation. Employees at these companies may not need convincing about the value of running experiments, but they typically don’t have a roadmap for how to integrate experiments to improve their operations. The goal of this series is to provide that roadmap.

In this post I would like to define what an experimental program is and discuss which stakeholder groups need to participate in the ExPr to make it successful.

A experimental program is the mechanism by which a company uses randomized controlled experiments to generate positive business results. An ExPr consists of the people, processes, and infrastructure for conducting experiments at a company.

Technical readers may be familiar with some of the technology required to run experiments, including software for splitting users/visitors/etc. (e.g. experimental units) in treatment groups and for calculating the results of completed experiments.

But ExPrs consist of more than just the technology needed to conduct experiments.

An experimental program can only be successful if the right people are involved. ExPrs require the participation of a cross-functional set of stakeholders – individuals and teams from different business units, data science, and technology. These people are responsible for driving experimentation by conceiving, planning, implementing, and analyzing experiments.

How exactly this ideation, planning, implementation and analysis is completed is the process part of an experimentation program. Processes describe how stakeholders collaborate with each other and with technology to generate positive business outcomes through experimentation. These processes describe how experimental ideas are generated, collected, prioritized, and implemented. They also describe how experimental results are evaluated and what change management is required to operationalize the results of successful experiments.

Subsequent posts in this series will focus on the processes and technological components of ExPrs. The rest of this post will focus on people.

Which stakeholder groups should be involved in setting up an experiment program? I categorize these groups based on their degree of direct involvement in an ExPr. Primary stakeholders are directly involved in the planning and execution of experiments. Secondary stakeholders are not directly involved, but should be kept informed to maximize the likelihood of acceptance of experimental results.

One of the most challenging but exciting parts of running an effective experimentation program is that it requires cross-functional participation at multiple levels of the organization. By cross-functional, I mean that the effort requires participation from individuals and teams across business units (particularly those business units seeking to optimize their outcomes), data science, and technology.

Let us briefly describe the roles of these cross-functional groups:

1. Business units

Business stakeholders are the people who own the process/product/outcome that they want to optimize through experimentation. For example, a product team wants to increase the average revenue per sale for a particular product.

The business stakeholders can be seen as the “customers” or “users” of an experiment program. They invest in the ExPr to improve an outcome that is measured by a metric that their business area cares about.

Business stakeholders may or may not have the experience or expertise to understand how experiments can be used to achieve the outcomes they care about. However, they are the experts in their functional area, products, processes, users, etc. Business stakeholders without this experience may need to be convinced of the benefits of experimentation. A common cause for concern is when an experimental variation can destroy value. In this case, it is up to data science to present the relevant tradeoffs by discussing guardrail metricsample size distributions, etc.

However, business stakeholders who do have experience with experimentation, or who at least believe in the power of experimentation to drive change in business, can be very effective partners in launching an ExPr.

2. Technique

Effective experimentation depends on reliable and trustworthy data. This data is typically generated when a user interacts with some part of the organization’s technology stack. A “user” can be internal, such as an employee of the company, or external, such as a paying customer or website visitor. Similarly, the organization’s technology stack can be the proprietary software that the company has built itself, or it can be third-party software that the company uses (for example, data from an email marketing tool or log data exported from the salesperson’s phone system used by the sales team), or a combination of both.

Engineering should be involved in the ExPr because this team best understands the systems that will be involved in the actual experiments. For example, when running experiments that involve parts of the tech stack that have not been involved in experiments before, engineers are needed to ensure that the technology is equipped with the appropriate telemetry and that these logs are accessible so that the experiment results can be analyzed. In more mature experimentation programs, engineering teams may work to integrate their systems with their company’s centralized experimentation platform (we’ll talk more about this in future posts).

In each of these cases, engineering management needs to be at the table to ensure the required efforts are prioritized on their roadmaps.

3. Data science

Data science plays two important roles in an effective experimentation program.

The first role is tactical. Data scientists are the experts in the statistical domain. Their background in experimental design and analysis is used to design and rigorously evaluate the results of individual experiments. This expertise helps organizations protect themselves from common statistical problems such as Simpson’s paradox, substandard experiments, and to peepIn this role, data scientists help determine the parameters of individual experiments, such as sample size requirements (How long do we need to perform this experiment?) and experimental populations (How do we determine who should participate in this experiment?).

The second role is strategicThe data science team should be responsible for advance the experiment program by influencing the cross-functional group of stakeholders (in a collaborative way). The main idea behind this is that data science is closer to the business than engineering, and closer to engineering than to the business. The data science team should act as a bridge between these two stakeholder groups, helping to translate business goals into technical engineering requirements and then helping to validate that what engineering builds meets the requirements needed to run effective experiments.

This second role should be fulfilled by data science managers or product managers who are part of a data science team.

Secondary stakeholders

An experimental program can also be a series of secondary stakeholders – individuals or teams who are in some way affected by the planned experiments, but do not actively participate in moving the experiments forward.

An example of secondary stakeholders are business units that are collectively responsible for the business outcomes we hope to improve through experimentation.

For example, let’s say the sales function in a B2C company wants to reduce costs. They collaborate with the data science team and decide to test certain automated, low-touch sales strategies for subsets of their prospects. But let’s say they share responsibility for a metric like the number of qualified prospects in the funnel with the marketing function. In this case, the marketing team is a secondary stakeholder in the ExPr. The marketing team should be notified of planned experiments that impact the KPIs they are jointly accountable for, as well as the results of experiments that have been completed.

Secondary stakeholders should be actively informed about the decisions and results of an ExPr. Failure to inform secondary stakeholders may result in experimental results not being fully embraced by the organization.

In summary, an experimentation program is the mechanism by which a company uses randomized controlled experiments to generate positive business results. It consists of the people, processes, and infrastructure for conducting experiments at a company.

The people involved in an ExPr consist of a cross-functional group of stakeholders operating at multiple levels within an organization. These stakeholders can be divided into two groups.

Primary stakeholders consist of people from individual business units, engineering, and data science. These individuals work together to plan, implement, analyze, and (hopefully) operationalize experimental results.

Secondary stakeholders need to be informed about the progress of experiments, but are not actively involved in running experiments. Keeping secondary stakeholders informed is important to ensure that experimental results are accepted and embraced by the entire company.

In the next post, we’ll discuss an interaction model that describes how these stakeholders can interact to move experiments forward. If you’d like to be notified when I publish this series, sign up below and I’ll email you every post I publish.

The opinions expressed here are my own and do not reflect the views or opinions of my employers.