Testing Different Versions of Language can Improve Your Communications

When writing or designing content for the public, do you test different versions to see if some are more effective than others?

Running randomized experiments with multiple wording or design options is an example of the practice of A/B testing. If you’re not already doing it, you may want to incorporate it into your work in order to better serve the public. It can assist in decision-making about wording and design to help people better understand what you’re trying to communicate and guides people to take actions that you want them to take. It also provides a neutral voice to help resolve internal disagreements.

Case Study: Helping People Self-select Programs

At Code for America, we use A/B testing extensively in our work with governments.

In our work with the State of Minnesota to create an integrated application for public benefits, there was an option for people to apply for “Housing Support,” which was selected by a large number of people who were not eligible for the program. To help potential applicants better understand what this program actually consists of, the team did A/B testing and found that using the title “Group housing” resulted in “a 50% drop in clients who chose that program, with a corresponding 300% increase in its approval rate.” In other words, “using a more descriptive name for the program helped clients self-select more accurately.”

This change directly helped the county governments’ case workers by dramatically reducing the high volume of ineligible applications that they have to process—while not doing much to discourage eligible people from applying.

Using A/B Testing to Resolve Internal Disagreements

One particularly useful application of A/B testing is in resolving internal debates about different wording or design options.

It’s not uncommon for different staff members to disagree about what should be shown to the public. When that happens, what do you do? Who “wins”? The person who keeps arguing for the longest time? The highest person on the org chart?

In a webinar by the U.S. General Services Administration, Kristal Byrd discusses a time when “we came to a difference of opinion again about our survey question that is geared toward getting users feedback on the content of a page … Rather than continue to go back and forth and have long discussions, it was agreed that we would use our customer survey tool.”

“That was kind of like a very diplomatic way of handling it,” she said. “I feel really proud that we’re at that point in our organization where we can rely on the data to make some important decisions and when this data comes back from these two questions we’ll be able to sit down and have a conversation about which question is more effective and we can make a decision based on that.”

Getting Started

Many tools exist that can help you do A/B testing on your website(s), often working with a variety of different content management systems. Check out the documentation for your content management system as well as the websites of these different tools to see what options may be available to you. If you send out emails via a newsletter, many popular mailing services have options for A/B testing.

While A/B testing is especially easy to do in the digital realm, you can also do it to some extent with printed resources, such as by using different URLs or phone numbers for different versions of a printed document that has some sort of call to action. While you likely won’t be able to get the large sample size or detailed analytics that you’d get with digital experimentation, you can still gain valuable insights. Also, beyond rigorous quantitative A/B testing, you can still use qualitative research methods to try to gain some insight into how people respond to your creations.

By incorporating A/B testing into your work, you can bring data to the table to help improve how you communicate with the public and ultimately deliver better services.

Greg is the Associate Director for Human-Centered Government at Code for America, where he is leading efforts to support public servants with resources and training on the organization’s principles and practices for how government can and should serve the public in the digital age.

Images are screenshots from the referenced Code for America report and General Services Administration webinar.

Photo by Josh Sorenson via pexels.com