Designing Customer Surveys That Work

by Richard E. Zimmerman, Linda Steinmann and Vince Schueler


By completing 12 basic steps of survey design,
you can boost the odds of gathering useful,
timely and accurate customer information.

One of the first principles of quality management is listening to your customers. Customer surveying is an essential listening tool that can yield information about customer expectations, customer satisfaction and strategies for improvement.

Sound survey design is mostly common sense-it is a basic job skill you can learn. By completing 12 basic steps of survey design, you can boost the odds of gathering useful, timely and accurate customer information from your surveys, whether you conduct them yourself or work with a survey research specialist.

Quality surveys are built on two types of expertise: Your business process knowledge plus a survey specialist's technical knowledge. As a business process participant, you are responsible for several of the steps that profoundly affect the scope, type and quality of any survey effort. As Figure 1 illustrates, the survey expert adds the highest value in later design stages.

This article provides step-by-step guidelines to assist you and your organization in developing and implementing an effective survey. You need to make 12 key decisions in three areas: establishing goals and objectives, constructing the survey instrument and finalizing survey logistics (see Figure 2).

Establishing goals and objectives

First things first-while many novice survey designers begin by writing questions or determining sample size, these activities should occur downstream. The most important, and often the most challenging, part of survey design is clarifying survey intent and scope. This is where your business knowledge is essential. Survey specialists can't read your mind. Only you know what is most important to your business or program.

Step 1: Determine survey purpose. The most critical part of survey design is a clearly defined statement of purpose and a well-structured view of what you will do with your newly acquired information. Surveys are decision-making tools. They have little value if you are not clear on the decisions your survey will support. Completing the following phrases formalizes your intent.
Use decision:
Why I want to do a survey because
I intend to use the information I am seeking by
The information to be gathered will enable me to decide
I am prepared to implement change as a result of this survey because


Your responses provide a compass, like the mission statement for your work unit, that will keep the project focused. A clear purpose guides the remaining survey design elements and emphasizes action based on results. One of the worst outcomes is collecting information that is never acted upon.

Step 2: Identify who will use the results and how they communicate. Surveys are communication tools. It is important to understand who will use the survey results and what type of information they respond to. Avoid the common pitfall of assuming others share your tastes in data. Although marketing staff may be comfortable with focus group results, a technical audience may want statistics. Leaders often want both numbers and human-interest perspectives when making decisions. Thus, a quantitative survey paired with selected case studies can be the most compelling for this group.

Completing the following statements helps you define the survey audience.
User decision:
The users of survey results include ...
The group(s) I will provide with information include ...
They want the information in order to ...
The way(s) they like to receive information include ...

Step 3: Identify what specific information is needed and when. This next level of refinement identifies and double-checks the information needed from the survey process. Completing the following statements helps you prioritize your information needs.
Criteria decision:
The information I really need is ...
I need this information because ...
My top-priority information needs are ...

List the criteria that will be used in making the decisions identified above. Identify the information that will enable you to make decisions and take action.

Be specific. Identify every element of data that will be collected and its added value. Avoid the tendency to gather information that is merely interesting or might be good to know.

Develop specific research questions that need to be answered. For example, if you are looking for information on a proposed product, identify the new product's elements and the concerns you have about each of them. Also, consider how your choice of survey information will protect the participants' privacy.

Once you have prioritized your information needs, determine when surveying should begin and how often it should occur. Answering the following questions will help you decide.
Timing decision:
How soon will I need information from this survey?
How often does significant change occur in my process, my customers or their lives or in the environment in which this process operates?
How often do I need updated information to manage my process or program?
How often do survey users need updated information?


The survey start date may be driven by important decisions you need to make, or by how much time you are willing to devote to planning a long-term survey effort.

Determine how often to survey by weighing the potential benefit derived from resurveying vs. the investment in resources-both yours and your customers'. As a rule, the time period between surveys should be short enough to give you reliable information, yet long enough that your customers will not feel bothered. Several factors drive survey frequency, including survey length and complexity, changes in your customer base, changes in your service or product delivery process, and seasonality of services or products.

Your survey planning should include approaches for retaining the data from each survey and tracking the results to show trends over time. This emphasizes the need for excellent data-requirements definition and database design because comparisons over time can be tricky. Some questions to consider include the following:
Data-tracking decision:
How will we track performance over time?
How will we provide staff access to survey data while protecting confidential information?
What if our criteria requirements change over time?


Step 4: Determine resource requirements. Ideally, resources devoted to your survey will be driven by the importance of the decisions you will make based on the data. Consider the following questions when scoping your resource requirements:
Resource decision:
What is the value of the information I am seeking?
What are the potential consequences of the decisions I will make based on this information?
What is the cost of not having survey data?
What staff and other resources are currently available?
What staff and other resources do I need?


Then determine the resource requirements by identifying the staff and financial resources that can be devoted to the project. It is important to agree about resource issues to ensure you have the capability to deliver a survey that meets your overall expectations. You may have internal resources, such as trained interviewers or data analysts. If not, consider whether you are willing to contract out for these services. Limitations on resource requirements will shape the entire survey design process.

Step 5. Determine who should conduct the survey. Even if you have the in-house capability to conduct a survey, sometimes it is important to use a neutral third party. The main benefits are credibility and candor. In some cases, survey users or respondents may believe that your organization lacks objectivity in collecting or reporting survey results. Survey users may find a study conducted by an independent third party more credible than one you conduct yourself. In other cases, survey respondents may withhold information if they are speaking to a seemingly "interested party" about sensitive issues.

Even if you select an outside consultant, be sure to complete the survey design steps that follow. Your answers to the remaining questions will help the firm you select complete the work with excellence.

Constructing the survey instrument

Now you have entered the stages where it is important to tap into internal or external survey expertise. Depending on the size and complexity of your survey, you may need to involve a survey expert throughout the remaining steps of the survey planning process.

Step 6: Determine who has the information you need. Once you have identified information and presentation needs, you should consider whom to survey. This is called selecting the "sample frame." In most cases, you will gather data from one or more subsets of the entire universe of people you can contact. By making an intelligent choice of whom to survey, your results will be accurate and your survey costs will be lower. Consider the following questions:
Audience decision:
Who has the information I need?
Is my customer base made up of two or more discrete groups, or are all my customers fairly similar?
Who decides whether to use my products or services?
Do I have current addresses or phone numbers?
The key group(s) I need information from include ...
Shall I randomly sample from my entire customer base or survey only certain types of customers?


It is important to understand who your sample represents in order to draw appropriate conclusions from the data. If you wish to generalize from the sample to the population it represents, the sample must be randomly selected. Random selection tends to assure the most representative sample (and will probably generate the highest credibility among users), but is not necessary or desirable in every case. Depending on your overall customer base or the decisions you are facing, you may want to concentrate your survey on a particular segment rather than a random selection of your entire customer base. This is especially true if your customer base is characterized by distinct segments rather than homogeneity.

Step 7: Select an appropriate survey type. There are several types of survey formats you could use to obtain the information needed:
Face-to-face interviews are often conducted to explore issues that are not easily understood and when information cannot be gathered from a written or telephone survey. Face-to-face interviews also allow the interviewer to assess nonverbal responses and ask follow-up questions. These interviews are often used to gain insight into customer needs and requirements as well as to determine which services are most important to them.
Telephone surveys consist of a series of questions and answers presented in strict sequence by a trained interviewer. Some survey organizations use computer-aided telephone interviewing software. The strength of telephone surveys lies in their interactive nature-respondents can ask for clarification about the questions, and surveyors can quickly spot any problems with survey administration.
Written surveys are the most commonly used survey type because of their lower cost and labor requirements. A written survey allows the customer more time to respond, but there is no opportunity to clarify intent. Because no interviewer is available to coach the respondent through the questions, it is very important for questions to be ordered properly and for clear directions to be given.
Focus groups are small-group conversations moderated by a neutral facilitator. They can provide valuable in-depth, detailed reactions to new concepts and ideas. Questions tend to be open-ended. The results are largely qualitative and cannot be generalized to the entire population unless extreme care is taken when recruiting respondents. One use of focus groups is to provide a quick and fairly inexpensive way to identify the issues to be addressed in a subsequent mail-out or telephone survey.
Key informant interviews are face-to-face or telephone interviews with leaders in a field whose responses can be assumed to apply generally to others in that field. Open-ended or quantitative questions can be used.


The survey method you select depends largely on the time available, customer characteristics, the information you wish to obtain and the costs you are willing to incur. Consider the following questions when choosing a survey type:
Survey type decision:
What format will get the needed information at the lowest cost and effort, and within the time frame I need?
How will limitations of survey types impact my ability to make key decisions?

See Figure 3 for a comparison of data-collection methods.

Mail surveys can suffer from low response rates. Ways to boost survey returns include:
Reducing respondent barriers to survey return through such means as short surveys (faster to fill out) and postage-paid, preaddressed mailers (easier to mail).
Using postcard or telephone reminders to follow up with respondents who have not returned their surveys.
Providing incentives, such as gifts or chances to win prizes, for returning surveys.


In most cases, surveys personally and courteously handed to customers during the service-delivery process have higher response rates. Surveys available for pick-up at the service-delivery location are often only completed by customers who have had a very favorable or a very unfavorable experience. This, of course, creates a different type of bias.

Telephone surveys provide the fastest turnaround and allow you to troubleshoot problems not detected in the pretest. Telephone surveys are best used when all segments of the survey population are known to have telephones and should be conducted during the time of day when respondents will be available to talk, which may be after 5:00 p.m.

Step 8: Design the survey questions. Survey results will be meaningless unless you ask the right questions. Ideally they are based on the steps outlined above and on the survey and analysis approaches employed. Consider the following when designing questions:
Question decision:
How is this question relevant to the purpose of the survey?
Is the question clearly worded?
Will the question be easily understood by my customers?


The question must contain enough specifics so the respondent can give a meaningful answer. Compare, for example, the following examples: How would you rate our service? and How would you rate the timeliness of our computer-repair services? Answers to the former will be difficult to act upon, while answers to the latter tell you clearly what, if anything, needs to change.
Types of questions: Two types of questions are generally asked on written surveys-open questions and closed questions. Open questions allow the customer to respond to a question in his or her own words. They can be a rich source of information, especially when you may not know all the possible answers that respondents might choose. However, you must consider how you will analyze the results of open questions, for you will need a coding system to quantify open-question responses.

Closed questions offer the customer a choice of specific responses from which to select. Multiple choice, rating scale and yes/no questions are examples of closed questions.

The questions you ask your customers must be properly worded in order to achieve good end results. Try to begin all of your questions with how, what, when, where, why or do. Avoid the following:
Leading questions-They inject interviewer bias.
Compound questions-They may generate a partial or nonresponse.
Judging questions-They can lead to guarded or partial responses.
Ambiguous or vague questions-They produce meaningless responses.
Acronyms and jargon-These may be unknown to the respondent.
Double negatives-They may create misunderstanding.
Long surveys-They discourage respondent participation.


Step 9: Design the data analysis and reporting format. As a final check on the instrument, build the tables and charts, and fill them with test data. Ask the question, "If these tables and charts are filled in, will I be able to make the decisions I've planned?" Other useful questions to address include:
Analysis decision:
What type of analysis will I need to perform? How simple or sophisticated should it be?
Who will analyze the data? How long will it take?
For automated analysis, what format must the survey results use?


You may need to iterate between steps 8 and 9 to ensure you will have the data needed to meet your survey needs.

Finalize survey logistics

Step 10: Determine sample size and selection. Sample size is driven by many factors, including: the survey type you have chosen, the complexity and relative homogeneity of your target audience, and the margin of error you and your users are willing to tolerate in this survey. The selection of the sample size grows in importance as the audience size and diversity increase, and as the importance of the decisions made from this information increases. Sometimes users of the survey information will want you to assure them that you have sampled adequately from certain groups they care about. Completing the following phrases will clarify your sample size.
Sample decision:
How many completed surveys do I need?
How many people must be contacted in order to get the needed number of completed surveys?
How will we reach them?


Large samples are not necessary for many surveys. A sample of 40 to 50 surveys is often quite adequate, and 100 suffices in most cases. At this point in the process you should consult a survey expert to assure that your approach is well-grounded. You may need to iterate between steps 6 and 10 to ensure you are reaching the right number of desired respondents.

Step 11: Determine data-entry methods. Once survey data are collected, ensure that the data are accurately recorded so they are ready for analysis. Questions to ask include the following:
Data-entry decision:
Who will perform the data entry?
How much time will it take?
Will we use a spreadsheet program, a database program or a survey-analysis software program?
How will we assure data quality and accuracy?


Specific design tools are available to increase the ease and accuracy of data entry. Consult a survey specialist for ideas.

Step 12: Pretesting the survey. Before distributing a survey to your customers, always pretest the survey. No matter how confident you are in the survey design, pretesting will help you:
Determine whether your instructions are understood.
Identify questions that may be misunderstood or that are poorly worded.
Determine whether rating scales are understood.
Determine how long it will take your customers to complete the survey.
Determine the customers' overall level of interest in completing the survey.


One method to pretest your survey is through a pilot group process. Pilot group participants should be representative members of your survey's target audience. The pretest should be administered in person, with participants understanding their role.

You will want to look for questions not answered, several responses given for the same question and comments written in the margins. All are signals that the question may not have been understood and may need revision.

You should conduct follow-up group discussions with your survey team to gain additional insight into the strengths and weaknesses of your survey product and to identify suggested improvements.

Do your homework

Survey planning can be completed quickly and without a lot of formality. Doing your homework will ensure the most useful survey results at the lowest cost. Survey goals and objectives are the key to the entire effort. Writing the questions is a very important and visible element in your survey.

To have a successful survey, you should concentrate on the planning and design, and let the steps we've offered lead you to the necessary questions and format. You can bring in and collaborate with experts as you need them to help you produce a quality product.

Acknowledgments

Peter Rosik of Research Services Associates in Gig Harbor, Washington.

About the authors

Richard E. Zimmerman is director, management consulting, in the Manufacturing Services Group of the Pulp, Paper and Packaging sector of Weyerhaeuser. For 1995­p;1996, he was director of the Washington Performance Partnership, an agency-level function targeted at improving management practices.

Linda Steinmann is a performance budgeting specialist at the Washington State Office of Financial Management. She has 11 years' experience in the legislative and executive branches of Washington state government.

Vince Schueler is an independent consultant specializing in customer research, program evaluation and performance measurement in Olympia, Washington. He has more than 15 years' experience conducting and using mail, telephone and key informant surveys in the public and private sector.