Navigating External Data: A Market Research Primer

In my blog post Data-Driven Decision-Making: A Review of RSM’s webinar, I discussed the importance of gathering external data to complement internal data for decision-making. This post provides a quick overview of market research and how it is used to gather that data.

Why External Data Matters

Most organizations recognize the importance of collecting and analyzing internal data for effective decision-making. Nonprofits draw from a broad range of information sources, including accounting systems, donor systems such as Raiser’s Edge, marketing campaign trackers like Mail Chimp, websites with registration systems or eCommerce functionality, and volunteer systems.

This information is critical to decision-making. However, focusing exclusively on internal data is like an aircraft pilot trying to fly in the clouds without instruments. It’s easy to become disoriented and end up spinning downward out of control.

External data provides a compass to the organization. It tells you the size of the problem or opportunity you are addressing, what factors are most likely to make an impact, and whether or not you are making progress.

It also provides answers to questions that surface through internal data. For example, internal data can identify poor donor retention rates, but only external data can tell you why the problem exists.

Sources of External Data

External data comes from a variety of sources.

Anecdotal Information

Anecdotal information is the most common source of feedback to a nonprofit. Your development team may hear that donors aren’t familiar with your organization’s name or don’t understand its mission. Team members staffing an arts event may hear positive (or negative) feedback from patrons about the organization’s performances.

It is important to systematically collect this information. Informal feedback provides good directional guidance and can be an early warning sign if things aren’t going well. However, anecdotal information is also inherently biased. Basing your decision-making exclusively on anecdotal information might lead you to miss important points of view.

Secondary Data

Secondary data is information about your market that is assembled by other sources. It is another common information source for nonprofits (and for profits) to use for decision-making. For example, a shelter might track homelessness statistics collected by the city or county. An arts organization might benefit from a national study conducted by an arts foundation about the impact of arts on a community’s health.

This information is helpful for educating donors and the community about needs, setting broad strategies, and sometimes for assessing progress. However, it doesn’t tell you much about your own organization’s impact or market perception. To get this type of insight, you must have primary data.

Primary Data

Primary data is information gathered directly from the people you want to influence, whether they are donors, members, patrons, parents or any other audience. Successful organizations build systems to systematically collect primary data from key audiences. For example, a youth sports organization might send a feedback form to all parents at the end of each season. An environmental organization might interview individuals who participated in a park clean-up to find out how to improve the program in the future.

Primary data gathered by the organization provides important insights. The advantage to this source of information is that it is easy to track over time to see trends and take action. The disadvantage is that the audience may not be as honest in their responses if they feel individuals impacted by negative feedback might find out who made the comments.

When the stakes are high, primary data gathered by a third-party organization, or independent market research, might be the best solution. While it can be expensive, carefully-conducted, independent market research provides the most accurate external data for decision-making.

When to Use Independent Market Research

Every organization should collect anecdotal feedback, track secondary data and gather primary data through in-house systems. These three sources are critical to cultivating systematic listening within a nonprofit organization. While these sources are subject to bias, when used together they can provide solid directional guidance relative to low-risk decision-making. However, if the cost associated with a poor decision is high, independent market research should be added to the mix.

For example, several years ago I worked with an arts organization that was planning a significant financial investment into a new program. The program was designed to attract young patrons. It was based on their observations, or anecdotal feedback, about the behavior of young patrons in their theater. However, because the investment was significant, they chose to test the key assumptions upon which the program was based to reduce the risk of failure.

That turned out to be a wise decision, as our research study showed that some of their assumptions were flawed. Based on the feedback, they adjusted the program to more accurately target the factors influencing purchasing behavior, and the program was a success. Investing in research delayed implementation by several months and cost the organization several thousand dollars. However, if they had launched the program based on existing data alone, the program would not have been as successful.

The primary purpose of independent market research is to reduce risk. It acts as an insurance policy by turning some assumptions into facts. Use market research to reduce risk when a decision relies heavily on internal data or assumptions and the cost of failure would be significant to the organization.

Picking the Right Market Research Tool

There are many different tools internal and external researchers can use to gather primary data. The most common methods include interviews, focus groups and surveys. Different delivery mechanisms help organizations reach different audiences. For example, website-based surveys effectively capture reactions from an unknown group people who use a website, while email surveys are more effective at reaching a known population. As you consider the type of tool to use, begin by determining whether you need to use a quantitative tool or a qualitative tool.

Qualitative Studies

Qualitative studies focus on generating a deeper understanding of market perceptions and opinions through conversations. They typically involve a small set of participants who may or may not be representative of the overall population. However, qualitative approaches provide the researcher with an opportunity to explore information in greater detail. Interviews and focus groups are common quantitative approaches.

Qualitative approaches are best when directional guidance is sufficient.

Quantitative Studies

By contrast, quantitative studies focus on generating more statistically representative, numbers-based data. Accurate quantitative research relies on the selection of a representative subset of the population that is of sufficient size to be statistically reliable. Because of the size of the population and the desire to have comparable data, the researcher asks defined questions, with limited opportunity to probe responses in greater detail. The most common quantitative approaches include mail, email, telephone and intercept surveys.

Quantitative approaches are best when an organization needs reliable information in order to make a decision.

Often, researchers use the two types of studies together. For example, consider a professional association exploring membership options. They might start with focus groups, a qualitative approach, designed to determine which benefits mean the most to members, and follow that with a survey, a quantitative approach, to assess how many prospective members would join if the package were offered.

Common Sources of Bias

Both qualitative and quantitative research approaches are subject to bias. In fact, it is almost impossible to completely avoid bias in research design. However, an understanding of the most common forms of bias can help identify and prevent potential problems, particularly with internally-conducted research.

The most common types of bias include:

Selection and sampling biases

Most quantitative approaches assume a random sampling of the population. If researchers only conduct research at certain times or in certain places, the sample population may not reflect the actual population. Researchers who sample on a convenience basis, polling only people who are easily accessible during their work day, are more likely to be subject to this bias.

Another common source of this type of bias is the selection process. Consider, for example, a survey sent to your email box. Your decision about whether to answer may depend on how you feel about the organization. If the nonprofit is assessing public opinion and broad groups of people opt out, this can distort the results.

Interviewer or moderator bias

In qualitative studies, moderator bias is a common concern. This happens when a moderator interprets participant comments, modifies follow-up questions, or responds in non-verbal ways, based on his or her perception of the topic.

It also happens during data analysis when a person allows their personal opinion to influence their interpretation of either qualitative of quantitative data. Internal market research functions are particularly prone to this type of bias.

Design bias

Bias in the design process occurs in many ways. Often, the source is the question itself, which may lead the respondent to answer in a particular way, or in the array of responses from which the respondent may choose. For example, the question “How significantly will the following action impact you?” assumes that the respondent will be impacted by the action. If he or she might otherwise have answered that it would not impact them at all, they may feel more likely to interpret an impact with this form of question. Similarly, if a survey asks “Which of the following do you read?” and does not offer a “None of the above,” or similar response, it may be prompting inaccurate responses.

Confusing language or a lack of parallel construction in survey questions can also result in inaccurate responses. For example, if a survey uses a scale of one (low) to ten (high) on most questions, but changes it on others, respondents may miss the shift.

Measurement and response biases

These issues arise when the respondent shapes their responses in a way that they suspect will please the researcher or beneficiary. For example, it is common for researchers asking about intent to donate or volunteer to receive overly positive response rates. While respondents may respond with rosy intents in the survey process, the actual participation rates are often considerably lower. Studies asking about socially undesirable behaviors, such as violence or criminal activities, are particularly prone to this type of bias.

The Cost of External Data Gathering

Gathering external data has a cost. Pooling anecdotal information and researching secondary data require staff time. Internally-generated primary data storage and analysis requires a technology investment. Hiring an independent market research firm can be costly. However, the cost of failing to gather data can be much higher.

I recommend asking the following questions:

• What assumptions are you making in your planning process (such as public perception)?
• Which key factors will determine the success or failure of your organization (such as donor satisfaction)?
• What is the risk associated with failure?
• What is the potential return if the assumptions are correct?

Use the answers to determine:

• What type of data to gather,
• Who should gather it,
• What research tools to use, and
• What level of expenditure is appropriate.

Have a Question?

Ask me. I have been conducting market research studies for almost 20 years. During that time, I have worked with nonprofits of all sizes to develop affordable approaches that deliver actionable data. Leave a comment on this blog or email me. I would be happy to help.

If you like this post, please subscribe to my blog. You’ll get a monthly summary of the articles I write.

Thank you for reading Upturn Strategies!