MAC-Team

Surveys

Article Index

How to collect your survey data?manga guide to statistics

Will your survey be written or oral? Is there going to be a number where people can call to register their results? Are you going to have a post office box to which completed surveys should be mailed? You need to decide whether it's going to be administered by people known to the audience and whether it will be done in person, by phone, or by mail. Remember that the more personal you make it, the higher the return rate will be. Surveys that are delivered cold have a return rate of only two to three percent, unless they're on a very hot topic for the community you're surveying.

Keep in mind whom you want to survey. Does your public feel more comfortable writing or speaking? Will it be efficient to leave surveys somewhere for people to pick up at their will, or should you do something to make sure they get one? If your survey is to be administered orally, will people feel honored or annoyed about being asked for their opinions?

Personal Interviews
An interview is called personal when the Interviewer asks the questions face-to-face with the Interviewee. Personal interviews can take place in the home, at a shopping mall, on the street, outside a movie theater or polling place, and so on.

 Advantages  Disadvantages
  •  The ability to let the Interviewee see, feel and/or taste a product.
  • The ability to find the target population. For example, you can find people who have seen a film much more easily outside a theater in which it is playing than by calling phone numbers at random.
  • Longer interviews are sometimes tolerated. Particularly with in-home interviews that have been arranged in advance. People may be willing to talk longer face-to-face than to someone on the phone.
  • Personal interviews usually cost more per interview than other methods. This is particularly true of in-home interviews, where travel time is a major factor.
  • Each mall has its own characteristics. It draws its clientele from a specific geographic area surrounding it, and its shop profile also influences the type of client. These characteristics may differ from the target population and create a non-representative sample.

Telephone Surveys 
Surveying by telephone is a popular interviewing method. This is made possible by nearly universal phone coverage in many countries.

Advantages  Disadvantages 
  • People can usually be contacted faster over the telephone than with other methods. If the Interviewers are using CATI (computer-assisted telephone interviewing), the results can be available minutes after completing the last interview.
  • You can dial random telephone numbers when you do not have the actual telephone numbers of potential respondents.
  • CATI software, such as The Survey System, makes complex questionnaires practical by offering many logic options. It can automatically skip questions, perform calculations and modify questions based on the answers to earlier questions. It can check the logical consistency of answers and can present questions or answers choices in a random order (the last two are sometimes important for reasons described later).
  • Skilled interviewers can often elicit longer or more complete answers than people will give on their own to mail, email surveys (though some people will give longer answers to Web page surveys). Interviewers can also ask for clarification of unclear responses.
  • Some software, such as The Survey System, can combine survey answers with pre-existing information you have about the people being interviewed.
  • Many telemarketers have given legitimate research a bad name by claiming to be doing research when they start a sales call. Consequently, many people are reluctant to answer phone interviews and use their answering machines to screen calls.
  • The growing number of working women often means that no one is home during the day. This limits calling time to a "window" of about 6-9 p.m. or on week-ends (when you can be sure to interrupt dinner or a favorite TV program).
  • You cannot show or sample products by phone. 

Mail Surveys
(or using postal/distribution channels for a paper format).

Advantages  Disadvantages 
  • Mail surveys are among the least expensive (depending on country)
  • This is the only kind of survey you can do if you have the names and addresses of the target population, but not their telephone numbers.
  • The questionnaire can include pictures - something that is not possible over the phone.
  • Mail surveys allow the respondent to answer at their leisure, rather than at the often inconvenient moment they are contacted for a phone or personal interview. For this reason, they are not considered as intrusive as other kinds of interviews.
  • Time! Mail surveys take longer than other kinds. You will need to wait several weeks after mailing out questionnaires before you can be sure that you have gotten most of the responses.
  • In populations of lower educational and literacy levels, response rates to mail surveys are often too small to be useful. This, in effect, eliminates many immigrant populations that form substantial markets in many areas. Even in well-educated populations, response rates vary from as low as 3% up to 90%. As a rule of thumb, the best response levels are achieved from highly-educated people and people with a particular interest in the subject (which, depending on your target population, could lead to a biased sample).
  • One way of improving response rates to mail surveys is to mail a postcard telling your sample to watch for a questionnaire in the next week or two. Another is to follow up a questionnaire mailing after a couple of weeks with a card asking people to return the questionnaire. The downside is that this doubles or triples your mailing cost. If you have purchased a mailing list from a supplier, you may also have to pay a second (and third) use fee - you often cannot buy the list once and re-use it. 

Another way to increase responses to mail surveys is to use an incentive. One possibility is to send a bill (money)/stamps along with the survey (or offer to donate it to a charity specified by the respondent). If you do so, be sure to say that the reward is a way of saying "thanks," rather than payment for their time. Many people will consider their time worth more than a few stamps. Another possibility is to include the people who return completed surveys in a drawing for a prize. A third is to offer a copy of the (non-confidential) result highlights to those who complete the questionnaire. Any of these techniques will increase the response rates.

Remember that if you want a sample of 1,000 people, and you estimate a 10% response level, you need to mail 10,000 questionnaires. You may want to check with your local post office about bulk mail rates - you can save on postage using this mailing method. However, most researchers do not use bulk mail, because many people associate "bulk" with "junk" and will throw it out without opening the envelope, lowering your response rate. Also bulk mail moves slowly, increasing the time needed to complete your project.

Computer Direct Interviews
These are interviews in which the Interviewees enter their own answers directly into a computer. They can be used at malls, trade shows, offices, and so on. The Survey System's optional Interviewing Module and Interview Stations can easily create computer-direct interviews. Some researchers set up a Web page survey for this purpose.

 Advantages  Disadvantages
  • The virtual elimination of data entry and editing costs.
  • You will get more accurate answers to sensitive questions. Recent studies of potential blood donors have shown respondents were more likely to reveal HIV-related risk factors to a computer screen than to either human interviewers or paper questionnaires. The National Institute of Justice has also found that computer-aided surveys among drug users get better results than personal interviews. Employees are also more often willing to give more honest answers to a computer than to a person or paper questionnaire.
  • The elimination of interviewer bias. Different interviewers can ask questions in different ways, leading to different results. The computer asks the questions the same way every time.
  • Ensuring skip patterns are accurately followed. The Survey System can ensure people are not asked questions they should skip based on their earlier answers. These automatic skips are more accurate than relying on an Interviewer reading a paper questionnaire.
  • Response rates are usually higher. Computer-aided interviewing is still novel enough that some people will answer a computer interview when they would not have completed another kind of interview.
  • The Interviewees must have access to a computer or one must be provided for them.
  • As with mail surveys, computer direct interviews may have serious response rate problems in populations of lower educational and literacy levels. This method may grow in importance as computer use increases. 

Email Surveys
Email surveys are both very economical and very fast. More people have email than have full Internet access. This makes email a better choice than a Web page survey for some populations. On the other hand, email surveys are limited to simple questionnaires, whereas Web page surveys can include complex logic.

 Advantages  Disadvantages
  • Speed. An email questionnaire can gather several thousand responses within a day or two.
  • There is practically no cost involved once the set up has been completed.
  • You can attach pictures and sound files.
  • The novelty element of an email survey often stimulates higher response levels than ordinary "snail" mail surveys. 
  • You must possess (or purchase) a list of email addresses.
  • Some people will respond several times or pass questionnaires along to friends to answer. Many programs have no check to eliminate people responding multiple times to bias the results. The Survey System's Email Module will only accept one reply from each address sent the questionnaire. It eliminates duplicate and pass along questionnaires and checks to ensure that respondents have not ignored instructions (e.g., giving 2 answers to a question requesting only one).
  • Many people dislike unsolicited email even more than unsolicited regular mail. You may want to send email questionnaires only to people who expect to get email from you.
  • You cannot use email surveys to generalize findings to the whole populations. People who have email are different from those who do not, even when matched on demographic characteristics, such as age and gender.
  • Email surveys cannot automatically skip questions or randomize question or answer choice order or use other automatic techniques that can enhance surveys the way Web page surveys can.
  • Many email programs are limited to plain ASCII text questionnaires and cannot show pictures. Email questionnaires from The Survey System can attach graphic or sound files. Although use of email is growing very rapidly, it is not universal - and is even less so outside the USA (three-quarters of the world's email traffic takes place within the USA). Many "average" citizens still do not possess email facilities, especially older people and those in lower income and education groups. So email surveys do not reflect the population as a whole. At this stage they are probably best used in a corporate environment where email is common or when most members of the target population are known to have email.
 

Internet/Intranet (Web Page) Surveys
Web surveys are rapidly gaining popularity. They have major speed, cost, and flexibility advantages, but also significant sampling limitations. These limitations make software selection especially important and restrict the groups you can study using this technique.

Advantages  Disadvantages 
  • Web page surveys are extremely fast. A questionnaire posted on a popular Web site can gather several thousand responses within a few hours. Many people who will respond to an email invitation to take a Web survey will do so the first day, and most will do so within a few days.
  • There is practically no cost involved once the set up has been completed. Large samples do not cost more than smaller ones (except for any cost to acquire the sample).
  • You can show pictures. Some Web survey software can also show video and play sound.
  • Web page questionnaires can use complex question skipping logic, randomizations and other features not possible with paper questionnaires or most email surveys. These features can assure better data.
  • Web page questionnaires can use colors, fonts and other formatting options not possible in most email surveys.
  • A significant number of people will give more honest answers to questions about sensitive topics, such as drug use or sex, when giving their answers to a computer, instead of to a person or on paper.
  • On average, people give longer answers to open-ended questions on Web page questionnaires than they do on other kinds of self-administered surveys.
  • Some Web survey software, such as The Survey System, can combine the survey answers with pre-existing information you have about individuals taking a survey.
  • While growing every year, Internet use is not universal. Internet surveys do not reflect the population as a whole. This is true even if a sample of Internet users is selected to match the general population in terms of age, gender and other demographics.
  • People can easily quit in the middle of a questionnaire. They are not as likely to complete a long questionnaire on the Web as they would be if talking with a good interviewer.
  • If your survey pops up on a web page, you often have no control over who replies - anyone from Antartica to Zanzibar, cruising that web page may answer.
  • Depending on your software, there is often no control over people responding multiple times to bias the results.
  • At this stage we recommend using the Internet for surveys mainly when your target population consists entirely or almost entirely of Internet users. Business-to-business research and employee attitude surveys can often meet this requirement. Surveys of the general population usually will not. That said, Internet surveys did about as well, and in some cases better, than other methods in predicting the outcome of the 2012 U.S. presidential election.

 

 

Even when Internet users may not closely match your target population, a Web page survey may be your best choice if you want to show video or both sound and graphics. A Web page survey may be the only practical way to have many people view and react to a video.

In any case, be sure your survey software prevents people from completing more than one questionnaire. You may also want to restrict access by requiring a password (good software allows this option) or by putting the survey on a page that can only be accessed directly (i.e., there are no links to it from other pages).