Spring 2003

Recently, I participated in a radio interview about the polls on the Iraq situation and how they seemed to vary depending on what people were asked. The host of the show finally said, "It seems to me that it's all in the way you ask the question."

"He clearly meant this to disparage polling in general. Yet one of the strengths of modern survey research is that asking questions in different ways can yield a more nuanced view of what people know and feel.

A second variation is where the response to the wording reflects real underlying nuances in the respondent's opinion. For example, respondents reply differently to the questions: "Do you approve or disapprove of the practice of abortion?" and "Do you favor or oppose women having the right to choose to have abortions if they want to?"

Back to top

Opinion Dynamics is frequently asked to conduct research among hard-to-reach populations, such as high-level executives and other well-paid professionals. Among the most difficult to reach are physicians, who are typically highly paid and for who time is a premium. Physicians tend to be selective when it comes to research because they are frequently asked to participate in studies. (For example, some physicians will do surveys about the efficacy of particular medicines or procedures, but will not do them if they seem oriented toward "marketing" or other purposes.) Often, traditional research with physicians is either very costly or simply not feasible.

Given the importance of understanding physician attitudes, behaviors, and experiences to so many organizations, researchers must find ways to overcome the hurdles and research this population. Traditional methods of interviewing, such as large scale structured interviewing from our calling center, or standard focus groups simply don't work very well. To interview physicians, and other hard-to-reach populations, we must be more creative in our approach.

ODC has conducted a number of studies in the last two years focusing on physicians. In two cases in particular, we worked with organizations with very limited budgets, yet for whom we were still able to successfully obtain valuable information from physician populations:

Case Study One: For a physician affiliated with Massachusetts General Hospital conducting research on medical practices, we conducted three focus groups with doctors in different parts of the country. The recruiting, which was overseen by the physician himself, was conducted largely through contacts - the physician contacted others he was personally familiar with in each location, who then worked on his behalf to contact others in those locations who met the screening criteria. The keys to success in this project included the following:

• The participants were generally young enough to appreciate the monetary incentive attached to the project, and lacked family obligations that might take up their time, but were experienced enough to provide useful information.
• The participants were notified in advance of the general topic, and provided as much information as possible to assure them of the importance of the topic, without seriously affecting the results of each group.
• The groups were held in a location convenient for most; in particular, a hospital conference room in each location.

  Case Study Two: For medical researchers at the Department of Ambulatory Care and Prevention at Harvard Medical School, ODC conducted in-depth interviews with physicians participating in a program to reduce antibiotic usage in children. In this case, three methods were used to recruit the doctors.

• Physicians were first sent letters by the medical researchers indicating that we would be calling, and providing information about the nature of the calls.
• Secondly, the recruitment was conducted by only a few key ODC staff members, specifically experienced in recruiting physicians for research purposes. Once again, our recruiters provided potential participants with as much information as possible about the nature of the calls, to reassure participants of the importance of the topic.
• Third, in addition to offering a monetary incentive, physicians were offered tickets to a Boston Red Sox game as an alternative, something many wanted but would not necessarily purchase for themselves.

It should be stressed that we recruited physicians for these studies using methods that are not always desirable or possible in other situations. However, they demonstrate that even with a limited budget, it is possible to collect information from hard-to-reach populations.

For more information on this topic, contact Lawrence Shiman at 800-966-1254 or via e-mail at lshiman@opiniondynamics.com.

Back to Top

Association executives, particularly those with smaller operating budgets, presume to know their members based entirely on qualitative assessments or sometimes, guess work. If renewal and new member enrollment rates and non-dues remain acceptable, that guess work becomes "brilliant intuition". If membership-marketing decisions based on "we know our members" produce unsatisfactory results, this is often blamed on "exogenous factors beyond our control". In truth, outcome may not be a reliable measure of how well an association understands its members. Success based on a flawed information process can lead to failure if the process is replicated.

Reliable information is frequently judged costly to many membership organizations. However, a less costly research methodology can be very attractive. One such option is Internet based on-line member surveys.

Yet, are on-line surveys a reasonable research methodology for membership-based associations? Perhaps, but like any other research methodology, Internet surveys must adhere to the basic tenants of survey research if they are to produce valuable results.

1. Will your members respond to an on-line survey?
To ensure an adequate response rate it is vital to make certain that members have convenient access to the Internet and are familiar with using the web.

It is also beneficial to host the survey on a secure website with the ability to accept direct data entry and monitor results. The survey should be graphically appealing and user friendly, while permitting appropriate skip patterns, split sampling and question rotation.

2. Will those who respond accurately reflect the entire membership?
High response rates are best achieved when members receive e-mailed invitations including a working hyperlink to the survey. If your e-mail capture rate is high (50% or better) and well distributed throughout your membership base, the pool of respondents will represent the sample of members. However, do not assume the numbers of respondents represent all members.

If older members, who tend to be less Internet savvy, did not provide e-mail addresses, then the data would not reflect their views.

3. Is the survey instrument a legitimate research tool?
A flood of responses via the Internet will not produce useful data if the questionnaire is improperly designed. The scope of the survey (not too long, not too short), question wording and ordering, number of open-ended questions, rotation of pick lists and other survey attributes are just as critical for on-line studies as for any other type of survey. Regardless of the methodology, a biased survey instrument will produce biased results.

4. Can you properly analyze the data to get full value from the numbers?
In most cases, simple topline data is not sufficient to set dues or product price points, determine multi-layered member satisfaction, design new offerings, alter a convention program or conceive a new marketing approach. To fully utilize survey data, advanced statistical techniques such as factor, cluster and regression analysis need to be implemented to identify the range of needs, wants and expectations of various segments of your membership. A well-designed survey can uncover satisfaction drivers and retention factors among various subgroups of your membership, but not without this higher-level analysis.

Conclusion
Internet surveys can be a reasonable method to acquire real learning at a reduced cost. It is vital to remember, however, that if a situation is not suited to Internet research due to insufficient Internet access among members or other methodological factors then using the Internet for a member survey could confuse, rather than clarify your understanding of your members. In that case, you are better served using the best survey mechanism for your circumstance rather than opting for the Internet simply to save some money in the near term.

To ensure an adequate response rate it is vital to make certain that members have convenient access to the Internet, and are familiar with using the web.

For more information on this topic, contact Glenn Cowan in our Washington, DC office at 202-347-3344, or via e-mail at gcowan@opiniondynamics.com.

Back to Top

Our seven-year association with the Fox News Channel offers us a unique opportunity to mine a rich historical database of public opinion on a range of interesting issues. The recent decline in President George W. Bush's job and personal favorability ratings led us to construct a comparison between the present occupant of the White House and former President, Bill Clinton. While we only have data from President Clinton's second term in office, it is still instructive to look at the path of public opinion for each president's tenure. What emerges is an interesting dichotomy in public sentiment.

In general, it's clear that the American public passed their judgments quite differently on the two presidents. While President Clinton suffered erosion in his personal favorability, his job performance ratings held fairly steady, even through the rigors of the impeachment process. By contrast, we have recently witnessed a drop in President Bush's job performance ratings (from 77% a year ago to 55% at the two-year mark), but a less precipitous drop in his personal rating (about 10 points).

Despite this difference, it is interesting to note that the trend or slope of personal favorability ratings is similar for both presidents.

Figure 1.
Personal Favorability, ratings of Clinton and Bush presidencies

Looking at the job performance for the two presidents at their respective "two-year" marks, we find that Bill Clinton has a 14-point edge on President Bush (69% to 55%) despite the fact that President Clinton was mired in the impeachment controversy at his two-year milepost. Of course, to date, President Bush has had to endure a sluggish economy and the war in Iraq. Nevertheless, the difference in job performance ratings is informative.

Figure 2.
69% approved of the job Clinton was doing at the two-year mark compared to 55% for Bush.

Our most recent Fox News/Opinion Dynamics Poll (March 11-12, 2003) offers us the chance to compare the two presidents in a slightly different light. Although President Clinton is more than two years removed from office, we find that he still provokes sharp divisions in the electorate. As the next chart indicates, the former president garners about as many positive as negative personal ratings. In fact, the last time President Clinton received a positive personal rating above 50% was in mid-July of 1999 (52%). By contrast, while President Bush's personal favorability has declined over the last six months or so, he still enjoys a positive personal favorability ratio of better than 2-to-1 (62% favorable/29% unfavorable).

All in all, these data point to quite different patterns of assessment by American voters regarding two quite different presidents. As both presidents encountered difficult times, voters expressed their disappointment with President Clinton in a personal way and with President Bush in a job performance way.

The FOX News/ODC Poll is a bi-weekly poll conducted to capture public opinions about current issues.

For more information on this topic contact Ernie Paicopolos at 800-966-1254 or via e-mail at epaicopolos@opiniondynamics.com.

Back to Top