A framework for recruiting qualified participants for qualitative and quantitative research.
You’ve found a perfect participant for your research study. He answered your recruiting questionnaire and has all the right qualifications. Of course, you invite him to participate in the study. But halfway through the research interview, you discover a problem. This guy is saying all the right buzzwords, but his answers lack detail. When you ask for more depth, you find shallow responses. You get the feeling that this guy doesn’t really know what he’s talking about at all. Sure enough — there’s nothing to prove he has the qualifications you were looking for. The data you’ve collected becomes worthless.
Recruiting the right participants can be difficult. The people you interview are the foundation of your qualitative research. Collected data is only as good as the people that provide it. Even a well-designed study is not valuable without the right participants. Interviewing unqualified participants is a waste of time and money. And if undetected, these participants can invalidate your study results. But how do you identify the best people and screen unqualified ones? With a few well-placed buzzwords an unqualified participant can trick even experienced researchers.
There are a number of steps you can take to prevent the scenario described above. The following is a guide that takes you through all the steps you can take to ensure you’re collecting the best data possible from genuinely qualified participants. This includes what do to before recruiting, during recruiting, during an interview, and after an interview.
I typically handle research and recruiting internally. This has some notable benefits: we control the relationship with our participants and can recruit cost-efficiently. However, it also requires us to verify the qualifications of the participants and the validity of the collected data. We regularly get research applicants who stretch the truth or simply lie about their qualifications. These occurrences are far from unusual across the industry.
Having a solid framework in place before recruiting helps ensure participant quality. Without clear guidelines, it can become difficult to discern which participants are truly qualified. You want to define a source of truth that allows you to identify who is qualified to participate and who only wants to make a quick $100.
Compensation for research participants starts at $50 to $100 for an hour-long session. The allure for applicants is understandable. Some people sense easy money and want to apply, even if it entails lying. A waiter can pretend to be a retail store manager, and a store associate can pretend to be a restaurant manager. When they’re filling out a questionnaire online it’s easy to cheat. For researchers, learning to deal with this is simply part of the process.
Lowering the compensation reduces the incentive for fraudulent behavior. You can emphasize non-monetary benefits, such as the social benefits or the impact of the study. However, reducing compensation creates another problem: it decreases the overall number of applicants. I never use this approach.
Measure of Authenticity
Some demographics have common measures of authenticity, such as certifications or social media profiles. If something similar is common in the desired demographic, it is worth asking for this before the interview. For example, if you’re recruiting digital marketers, you can safely assume that every qualified applicant has a LinkedIn profile. LinkedIn profiles generally accurately reflect a person’s qualifications. Asking for this in the screener provides solid assurance of the candidate’s legitimacy. LinkedIn is a great source of truth if your research revolves around people who are employed. It’s difficult to fake and it prevents multiple applications from the same person.
To illustrate, at my current company we recruit sales associates who have worked in retail for several years. LinkedIn is a great place to test this claim. If someone has an active LinkedIn profile demonstrating work history and more than 50 connections, there’s a very good chance the person is real. If a person has no picture, one listed job, and no connections, then the person could have made a LinkedIn account just for you. This method is not foolproof, but we’ve found it to be a solid reference to evaluate applicants.
Knowing What to Look for
To qualify participants you need to be a subject-matter expert. If you don’t have a solid understanding of the subject, it’s hard to determine what participants are qualified. But where do you start?
Before beginning the recruiting the process it is extremely helpful to establish a reference for verified information. You probably already have a person in your network who knows something. Maybe someone knows someone who knows something. This person could be in your company, your personal network, Twitter, LinkedIn, or at a client’s company. Finding an expert in your network lets you glean information from a trusted source. Reach out to some of these experts to learn from their experience.
Speaking with experts allows you to establish baseline qualifications and expectations of your upcoming participants. Having a set of data allows you to evaluate the information you collect during from your research applicants and participants. It gives you a great lens through which to evaluate their responses.
Here are some good questions to ask as you’re developing your recruiting lens:
- What background do people in this target demographic have?
- What are their motives for taking their job/position/hobby/interest?
- What industry-specific knowledge would most of these people possess?
- What knowledge distinguishes average performers from exceptional performers?
It is worth going into great detail with these baseline interviews. This information will serve as a reference and as a guide for all following research. Creating a foundation of verified information will allow you to create a set of highly specific questions. These can be used to quickly ascertain if participants are qualified or phony.
Deciding Where to Recruit
Where you recruit participants can have a big impact on how many unqualified applicants you receive. Sourcing candidates merit an entire article on its own. Here are a few essential options:
Recruiting through online classified ads, such as Craigslist
- For many demographics, you’ll get both a high volume of applicants. Because these are completely public there’s also a high chance of getting unqualified applicants.
Direct outreach to select participants via LinkedIn or email
- This will produce a very small volume of participants. People are often shy to respond to cold outreach for research studies. However, you can pick your perfect participants.
Direct introductions to select participants through your networks
- This will produce a limited volume of applicants but provides much higher quality. These participants will have the implicit trust of the people referring them.
Over time, building out your own database of participants is key to efficient and accurate recruiting.
Your existing qualified research participants are also a great source. They can provide you with additional participants. A store associate knows more store associates, and a nurse knows more nurses. This is commonly referred to as Snowball Sampling. It’s a great way to recruit more candidates who have a very high chance of being qualified. Referrals can be incentivized too. For each referral, a participant can receive a third or half of his interview compensation. When an excellent participant refers to other excellent participants this is money well spent.
This is method is not guaranteed. If compensation is involved there will be a temptation to cheat. We’ve had referrals accidentally forward us emails from participants that said “just tell these guys you’re working at Chanel and they’ll pay you $100 for a call.” Of course, the person didn’t work at Chanel. In many cases, it’s best to direct referrals to the same screening questionnaire that the original applicants used. This will capture their information and give you the assurance they’re truly qualified.
Maintain a database of previous study participants and qualified applicants is a great help. Applicants that couldn’t participate in an initial study can be pulled in. Participants can provide referrals. Moreover, they can participate in future studies without a new recruiting effort. Keep notes for how articulate and helpful participants were, so you can refine your pool of future participants.
An effective screening test must give you enough information to determine if the applicants fit your target criteria. It should not reveal what answers or demographics you’re seeking. You don’t want to give applicants the buzzwords that will make them seem qualified. The wording and questions should be structured so they don’t hint at the right answers.
For example, let’s say you’re looking to recruit customer service representatives that work at car rental companies. If you recruit for “customer service reps” you will get a large number of applications. However, you will also get customer service reps that don’t work in the car rental industry due to the generic title. Explicitly recruiting for “car rental customer service reps” reduces the number of applications. It also increases the risk that unqualified applicants pander to your criteria because the title reveals the exactly desired applicant.
Writing a great questionnaire is tricky. Making questions too specific gives away the answer. Making questions too generic doesn’t qualify participants. The questions need to specialize enough so the answers aren’t a google search away or simply common sense. At the same time, the questions need to be general enough to be easily answered by your target demographic.
As a general rule of thumb, keep questionnaires as short as possible while sufficiently qualifying applicants. During recruiting it’s tempting to use shorter questionnaires. They get more completions. They require less effort and give applicants fewer opportunities to drop out. However, there are some very notable benefits to using slightly longer questionnaires. More questions provide more opportunity to evaluate an applicant’s qualifications. With longer questionnaires, it’s also worth checking that an applicant’s responses are consistent throughout. The answers from the questionnaire can also be used during interviews to double-check the participant’s qualifications.
The worst thing that can happen in research is that false information is accepted as true. If the bad information is used to guide the product development process, the costs will easily eclipse any time and compensation lost in the recruiting and research process.
Formulating questions is a precise skill. To illustrate this, let’s recruit people who frequently purchase luxury apparel online.
We might ask:
How frequently do you purchase items online?
This question isn’t effective because it is completely open to the applicant’s interpretation. It is very subjective. One person might consider weekly purchases as frequent. Another person might consider monthly purchases as frequent. Asking people for their averages or predictions is also problematic. People are remarkably poor at estimating their own behavior. (Think about new year’s resolutions and people predicting their exercise habits!)
A better approach is to ask for a specific behavior. Asking for the specific number of purchases in the recent past has the lowest error risk. It provides you with relatively objective data and lets you make your assessment of purchase frequency.
We can make the question more precise:
How often did you buy items online in the past two weeks?
- Less than 6 times (eliminate applicants who choose this)
- 5–7 times
- 8–10 times
- 11–13 times
- 14 or more
In this scenario, applicants will go for those specified middle choices to avoid being filtered out.
Here’s why: the options use inconsistent wording and banding, which hints at the desired answers. Notice that the middle three answers are very specific, while the end points are very broad. This makes it appear that 6 to 14 is the desired recruiting range. It is best to avoid broad phrasing such as “less than” or “more than”. Such imprecise wording implies that the recruiter does not really care about that particular option. Meanwhile, very specific numbers and ranges suggest the recruiter does care.
Below is a better way to phrase the question. It uses consistent wording and banding, and provides multiple undesired answers. This makes it much more difficult to guess what the “right” answers are.
How often did you buy items online in the past two weeks?
- 0 (eliminate)
- 1–2 (eliminate)
- 3–4 (eliminate)
- 14 or more
However, this question still makes it obvious that the recruiter is looking for people who buy online. Generally, screeners do not ask for attributes that are not desired. (Adding such questions would require applicants to take more time, and possibly scare away qualified applicants.)
We can make it more difficult for applicants to guess the desired answer. The following questions deliberately obscure the right response:
Where do you buy items online? (select all that apply)
- Home Depot
- Neiman Marcus (eliminate applicants who did not choose this)
Still, this question makes it possible for a cheater to pass the screener. For example, an applicant could simply select every option to ensure they are not filtered out. The fundamental issue with this question structure is that the predefined answers provide a small group of potentially correct answers, making it much easier to guess at a correct answer.
One creative solution is to offer made-up choices. In our example, we could throw in a retail chain that doesn’t actually exist. If applicants claim to shop there frequently, we know something isn’t right.
The safest approach is to use an open-ended question to filter out all unqualified applicants by not providing any hints:
Do you buy items online?
If yes, where do you buy items online?
(eliminate applicant if an answer does not include Neiman Marcus)
After you’ve collected the first group of applicants, you can take an additional step to verify their qualifications. When you reach out to schedule interview times, you can send along a small secondary screening questionnaire to inform the interview. This needs to be done carefully so you don’t filter out good candidates.
In this secondary questionnaire, you can ask questions that build on the first questionnaire. If the answers are not consistent, you can still disqualify applicants before doing a full interview. One tricky way to detect pandering applicants is to ask contradictory questions. For example: in the first questionnaire you can ask if they work in the hospitality industry, and ask if they work in retail in the secondary questionnaire. If applicants answer “yes” both times, there’s good chance they’re just telling you what they think you want to hear.
Before the Interview
By now you have a thoroughly screened group of research participants. Don’t ruin the research now. Don’t tell participants about the interview contents or subject of the study before the interview. You want participants going into the study with a fresh perspective and their real experience. That’s why they are being compensated. Sharing too much before the interview gives unqualified participants the chance to do research. This could make them sound qualified in an interview when they actually aren’t. Using their interview data could then spoil your thoughtful research efforts.
During the interview
Ideally, all unqualified participants are filtered out before the interviews. But some will slip past your screener. Once the interview begins, participants should be verified quickly.
Start with very specific questions with objective answers. Here are a few examples of verifications questions for a retail sales associate:
- What is the process for checking in for a shift?
- What is the name of the POS system?
- What is the address of your store?
- What are the store neighbors?
- What are the store hours?
- What is the compensation?
These are questions that you can research in advance. They should be easy to answer for a qualified participant.
Simple social observations go a long way in an interview. There are a number of warning signs, such as if the participant shows notable hesitance or vagueness in answering questions. If the participant gives short, generic, uninformative answers, you might fly through most of your prepared questions in the first 10 minutes. That’s a strong signal the candidate is not informed about the topic.
Pay attention to inconsistencies in the conversation. For example, it’s a red flag if an early answer states “ I love working with customers,” and the person later says “I don’t waste my time with the customers if they’re not going to buy”. It’s difficult to maintain a lie consistently throughout an hour-long interview. Asking questions against the screening questionnaire or the person’s LinkedIn can also be insightful.
Don’t wait until the end of an interview to check if the participant is as qualified as anticipated. Don’t waste time and money. If the first few questions demonstrate that the participant misrepresented himself in the screener, end the interview early. Simply tell them: “This has been very informative, and we’ve already learned everything we needed. We are going to end the interview early. Thank you for your time.”
Managing a deceitful participant is tricky. It is not worth continuing the research process with a candidate. The data is false and detrimental to your study. Depending on the terms specified in your recruiting ad, it may be possible to withhold payment for an interview that was terminated early. However, it’s usually easiest to pay the participant and move on. If someone was rude enough to cheat in your screener, there’s a good chance their manners won’t improve if payment is withheld.
There will be scenarios when a participant has been honest but doesn’t benefit your research. Maybe he isn’t talkative. Maybe there was a recruiting error and he doesn’t match desired demographic or information. You may not want to interview that participant again, but they did not deceive the recruiting process. In these situations, the participant should absolutely be paid.
After the interview
Having a few bad interviews isn’t the end of the world. As long as unfit candidates and invalid data are filtered out, research is well worth the investment. After all, research expenses are marginal compared to the cost of misguided product development.
After you’ve completed your interviews, maintain a database of qualified and unqualified participants. You can leverage great participants for future research or referrals. Likewise, you can track and avoid unqualified applicants and participants in the future.
These frameworks and tricks help you ensure you’re getting the best possible participants for your research studies. You now have tools to screen bad applicants before the interview begins, identify qualified applicants, and double or triple check qualifications during the research process. There’s no need to get tricked. Great research data requires great participants.