In the private sector, products and services are tested with end users (e.g., usability testing) with little to no restriction. With government products and services, though, getting approval for usability testing is more complex. Some of the reasons for this complexity are understandable: Usability testing takes labor hours, tax dollars, and time commitments (often uncompensated) from end users, who may belong to vulnerable populations.
A less understood obstacle to usability testing is the Paperwork Reduction Act (PRA), which was developed to prevent unnecessary burden and waste. The PRA is dense and easy to misunderstand, leading to an enduring myth in the federal government that the PRA prohibited usability testing. On November 21, 2024, OMB released a memo clarifying that the PRA does not prohibit usability testing and provided guidance for complying with PRA when conducting user research. The memo breaks down some scenarios in which PRA approval is necessary, but it still may be confusing for colleagues who lack a research background. So, let’s talk about the criteria in more detail.
Are you collecting PII?
If you are planning a usability test (i.e. showing a website, form or product to someone and asking their feedback), the first question to consider is whether your team is collecting PII (personally identifiable information, such as names, account numbers and Social Security numbers). If you are collecting PII for purposes other than contacting and communicating with the study’s participants, then PRA approval is needed.
How many people?
One of the simplest criteria to consider when conducting usability research is the number of people who will participate in the study. If it is fewer than 10 people, no PRA approval is needed. However, if you are conducting a usability study with fewer than 10 people AND collecting PII for purposes other than communication with the participants, then PRA approval is needed.
Is this a survey?
The OMB memo specifies that usability tests do not need PRA approval when they ask “structured, semi-structured, or open-ended questions” while someone conducts a task and a researcher observes. A simpler way to think about this is to ask, “Is this a survey?” If your study does not involve sending a form with mostly multiple-choice questions for someone to complete on their own, then it most likely is not a survey. If it is a survey, though, it needs PRA approval.
Who sees the results?
If you are collecting enough data that you can generate reports and visuals or make statistical inferences to share with the public, then PRA approval is needed. This type of study usually involves large sample sizes, which excludes most usability studies.
Who does this impact?
Impact and applicability of the study’s results matter when determining if PRA approval is required. If the results of your study will be used for any of the following purposes, then PRA approval is called for:
- Measuring program effectiveness
- Measuring outcomes of programs
- Evaluating metrics or KPIs of specific programs
- Adjudicating benefits or programs
Most studies that aim to evaluate programs (as opposed to a specific service, website or form) are typically quantitative in nature and have larger samples sizes than the average usability study.
If you need PRA approval, don’t panic. Start early.
If you do need PRA approval to conduct your study, the best advice is to start early in the process and work closely with your agency’s PRA staff. The more information you have about your study’s design, purpose and scope, the quicker the process will go. For some studies, expedited PRA approval is possible.
As helpful and encouraging as it is, the OMB memo is still a wall of text. For those of us who do not process information well in text format, I have made a basic flow chart to guide people through determining if PRA approval is needed for their study. I highly recommend that people still read the memo and consult their agency’s PRA staff for any final decisions, but a visual may be a more comfortable starting point for some.
Simple flow chart displaying various criteria for studies needing PRA approval
Ann Aly (pronounced like Ali) is a UX and civic tech practice leader with a background in academic research, music, and education. She combines these experiences to lead teams improving federal government services, emphasizing communal leadership and transparency. Ann holds a PhD and MA (both in Linguistics) from UCLA, and an MA (Spanish and Portuguese) and BA (Music) from Florida State University. When she’s not asking too many questions, Ann enjoys woodworking, gardening, and exploring the Shenandoah Valley woodlands near her home.
Leave a Reply
You must be logged in to post a comment.