I think it's safe to say that the most mundane part of my job is putting together a rock solid participant screener. Anticipating all the exciting research to be had once we've found the right participants is what pushes me through this process. In the past few weeks I've been in Screener Land--a dry place where the inhabitants dress in black and line up obediently against a white landscape-- and thought I would share some techniques on screener development.
1. Be explicit with logistics and instructions.
In the screener I include a participant matrix with desired quotas, and an introduction to the study. I wrap up each screener with a detailed invitiation script that includes: further details about the study, incentive information, recording, directions, and homework assignment (if applicable). I expect the recruiter to communicate all research details to the participant because often we are going into people's homes with a video camera. I've been in several situations where participants were caught off guard: "this wasn't expected." This American Life aired a piece about the fine print in contracts, referencing a myth that popstars like Madonna are highly demanding because they include line items like "all blue M&Ms must be removed from the candy jar" in their contracts. The reality is that these line items are included to ensure that the folks who are setting up the concert actually follow all directions to a tee: if blue M&Ms are in the candy jar, what other line items were overlooked?
2. Write down who you want to recruit, and how samples might diverge based on behaviors, attitudes, and demographics.
When I talk with recruiters, I usually describe the desired sample in high-level ways: "Mostly young women who eat oatmeal every morning because (Group 1) they like the taste (Group 2) they believe it's good for them." I often find that this initial description (devoid of specifics, but that which captures the essence of who we want to recruit) is what sticks with recruiters, so be sure you are being adequately descriptive. Beyond that, I jot down on a sheet of paper how I might describe my sample in three different categories:
- Do they buy instant oatmeal or steel cut?
- How many times a year do they go to Whole Foods?
- How likely would they be to try a new kind of oatmeal?
- They believe that instant oatmeal doesn't provide the same health benefits as steel cut
- They rank the taste of oatmeal higher than health benefits
- They agree that oatmeal has changed over the past 5 years
- Mix of ages between 18-45
- 80% female
- Income range must fall between...
3. Don't reveal too much about what you want to study (unless you need to). Avoid making respondents hyper-aware of what population they might represent.
When it comes to the topic at hand (oatmeal), I like to broaden it to a larger theme (breakfast food) and begin with questions that will eventually lead to oatmeal eaters. I try to continue the theme throughout the line of questioning.
BAD: We are working in conjunction with a client who wants to talk with oatmeal eaters.
GOOD: We are working in conjunction with a client who wants to know more about what you think about breakfast foods.
Avoid questions that compel a Yes or No response.
BAD: Do you eat oatmeal?
GOOD: I'm going to read a list of breakfast foods, which ones do you eat at least twice a week..."
Introduce options that may not be relevant to your study (in particular to behavior and attitude questions) to avoid serial participants.
BAD: I'm going to read a list of statements about breakfast foods...which ones apply to you?
GOOD: I'm going to read a list of statements about what is important to as a part of your morning routine...which ones apply to you?
4. Make edits, and let others make edits for you. Pilot your screener.
My process involves the unforgivable hardcopy print out. And dare-I-say, I opt for one-sided in this step. It's important for me to spread out the entire screener on a large table and think-aloud as I proceed through the screener logic. With my trusty orange Sharpie, I make changes big and small that will reduce confusion and ambiguity for the recruiter. With orange marker all over the screener, I hand it off to someone else on the research team who may be marginaly invovled in the project: they ALWAYS have great suggestions that leave me slapping my forehead. ALWAYS. Oh the power of working in teams! I then pilot the screener with someone who isn't in research so they can tell me if a particular question makes no sense.
5. Stay involved in the recruit, this allows you to ask more precise questions.
When I work with third party recruiters, I like to include "HOLD" questions that compell the recruiter to be in touch with me. This method works for me because third party recruiters I've worked with tend to recruit during the day anyway when I am accessible at my desk. These hold questions compel the recruiter to do their due diligence and find the right participants for the research. This technique also allows me to be a bit more nuanced in my line of questioning (i.e., more open-ended questions as opposed to multiple choice).
6. Validate your participants during the research session.
Knowing that I have the right person doesn't end when I receive a schedule of participants. The first line of questioning I employ during the research aims to validate some of the assertions the participant made in the screener itself. Face-to-face questioning often yields more candid responses that will help you identify if someone is a true representative of your population, or someone who simply slipped through the cracks.