Office Hours: The Importance of Experimentation

By Katherine L. Milkman

How do you tackle decisions at your organization about changing a policy or procedure in an area where you lack expertise? Consulting an expert is one smart, common approach. But how often do you consider implementing ideas experimentally to find out if they really work or if they are, perhaps, worse than the status quo? My bet is that you don’t think about experimentation often enough.

One terrific aspect of my job is that I sometimes get calls from managers trying to make policy decisions and seeking expert guidance. The story of one such call highlights why it is important for companies not only to seek expert opinions but also to experimentally evaluate the success of new strategies.

In 2008, I was approached by an executive at a large company who was interested in my research on how people can be nudged to make fewer short-sighted decisions (e.g., to save more for retirement, exercise more frequently, and eat healthier foods). At his company, most employees were enrolled in a 401(k) plan, but a significant minority were not setting aside the money they would need for retirement. He wanted to change their behavior.

Editor’s note: Watch “In a Minute: Katherine Milkman”—our video interview with the professor:

Because social norms marketing works in so many settings, we thought it could enhance our simplified 401(k) plan enrollment mailings. We proposed including a message about the percentage of peers in an employee’s age group at the company who were already enrolled in the 401(k) plan.Recent marketing research has shown that conveying information about one’s peers predictably influences attitudes and behavior. For example, telling homeowners that their neighbors use less electricity than they do reduces electricity consumption. This type of “social norms marketing” is probably effective for two reasons. First, learning your peers are engaging in a behavior can be informative; if everyone else is buying iPhones, iPhones must be great, right? Second, many people may prefer to conform in order to “fit in”; if everyone else is using an iPhone, I don’t want to be the one weirdo still using a different device!I immediately pulled together a superstar group of collaborators (John Beshears of Harvard Business School, James Choi of Yale School of Management, Brigitte Madrian of Harvard Kennedy School and David Laibson from Harvard Economics) to brainstorm. An obvious solution we proposed was to send “simplified enrollment” mailings to nonsavers, which allowed employees to start contributing a fixed percentage of their pay to the company’s 401(k) plan with a preselected asset allocation by returning a short, simple reply form. Such simple mailings have been shown to significantly boost savings in past research. The question then became: How could we best design these simplified enrollment forms to maximize 401(k) sign-ups?

Many managers would stop there. Expert advice was sought and a solution found. However, we successfully argued that an experimental evaluation was critical. Past research had shown huge benefits from simplified enrollment mailings, so we knew we would see an enrollment uptick. However, we wouldn’t know whether the inclusion of peer information had helped or possibly hurt; it had never before been used to encourage savings. Therefore, we strongly encouraged an experimental evaluation of our social norms messaging idea.

We were surprised by what we found—which is exactly why experimentation with new ideas is critical. While our simplified enrollment mailings produced a dramatic increase in 401(k) enrollment, as expected, all mailings were not equally effective. Including peer information was actually discouraging to employees, producing a lower signup rate than a standard simplified enrollment mailing. Learning that a higher percentage of peers in your age group were saving further reduced enrollment. These effects were driven by employees earning below average salaries in a given region of the organization. They apparently found it demotivating to learn that their peers were already saving more than they were for retirement. Perhaps the social comparison reminded these employees of their relatively low economic status and made the overall message of the mailer a discouraging one.

If we hadn’t run an experiment, the company we worked with would have rolled out simplified enrollment mailings including peer information, observed a boost in enrollment and continued using similar mailings in the future. Because our test demonstrated that peer information was harmful in this setting, not only could this company optimize its future messaging strategies, but we all learned an important broader lesson about when social norms marketing can backfire: in settings where upward social comparisons make low-status groups feel hopelessly behind.

Experimentation is key to learning in organizations, and yet too many companies are satisfied to simply collect before/ after data that may not tell them what components of a given policy did or did not perform as hoped or whether some unrelated, simultaneous change in the economy or the organization produced the detected benefits (or harms). Only through careful experimentation can causal inferences be drawn about the success or failure of new ideas.

One company I work with, Evive Health, has taken this philosophy particularly to heart. Evive sends reminders to the employees of its client companies whenever they are due to receive a health test or procedure (e.g., flu shots, colonoscopies). Since I began working with Evive, they have completely revamped their business model to hard-code experimental testing into their mailing algorithms so new design ideas are constantly assessed experimentally, helping them optimize the effectiveness of their messages.

My hope is that after reading this article, you will change your approach to tackling new challenges whenever possible. From now on, don’t only consult with others (and ideally experts), but consider implementing new ideas experimentally to find out if they really work.

Katherine L. Milkman is the James G. Campbell Jr. Assistant Professor of Operations and Information Management.


Wharton Magazine - Background

Type to Search

See all results