Design Survey Model

Employee surveys are usually designed to measure or assess the views or perceptions of employees regarding a specific event or condition in the organization or work environment, e.g. to find out how happy the employees are with the remuneration policy, or to determine how committed and engaged they feel towards the organization at a specific point in time.

Wrapp

For a survey to accurately and reliably measure or assess such an event or condition, the survey must be deliberately designed to systematically cover the different facets or constituent parts of the event or condition that is to be measured. The 3 steps outlined below will provide you with a simple, structured, and failsafe approach that will make designing a survey a breeze.

To illustrate the 3-step survey design process, we will use a Cafeteria Happiness survey as an example – we are going to design a survey to measure employee happiness with the cafeteria facility and services at their fictional place of work.

Step 1: Determine the Objective (purpose) of the Survey

The first and most important step in the survey model design process is to sit down with all the stakeholders (i.e. the managers or clients who have expressed a need for the survey) and to brainstorm the objective or purpose of the survey, i.e. what do they want to use the survey for, what do they want to achieve with the survey. To illustrate, research shows that companies with high levels of engaged employees perform better than companies with low levels of engaged employees, so if you would like to improve the engagement levels in an organization, it would make sense to use an employee engagement survey to (purpose:) isolate and identify the factors that inhibit or drive engagement in the organization, so you know where to focus and what to fix.

To use the Cafeteria Happiness survey as an example – a cafeteria that serves outstanding food may not necessarily motivate employees to work harder or to be more productive, but a cafeteria that provides really terrible food could be a major source of unhappiness that could demotivate and/or derail employees, so it will make sense to get some understanding of (purpose:) what the employees like and don’t like about the current cafeteria services and what can be done to make it even better. A secondary objective will be to provide the employees with an opportunity to comment or make suggestions regarding the cafeteria services – they may have some good suggestions, and besides, asking them for their opinions will reinforce the perception that management cares and are prepared to listen.

Step 2: Break it down into Main Topics

Once you have agreement amongst the stakeholders regarding the purpose or objective of the survey, the next step will be to get a better understanding of what exactly needs to be assessed or measured via the survey by breaking it down into a number of topics or related themes. If you are doing an engagement survey such as Engaged@Way-of-Work, the themes can be split into the 9 drivers of engagement – from Strategic Alignment and Manager Intent to Employee Voice, Personal Development and Talent Fit.

So as part of Step 2, brainstorm with the stakeholders the main aspects or facets (i.e. topics) that need to be assessed, what they mean, and why each is important, and write down a short definition and/or a few bullet points to summarize the intent, meaning and purpose of each topic.

To use the Cafeteria Happiness survey again as an example – the main topics that need to be measured and/or addressed could arguably include:

• Topic 1: Food – whether the quality and variety of the food is good enough;
• Topic 2: Service – whether the service is professional and timely and the staff courteous and friendly;
• Topic 3: Facility – whether the physical venue, fixtures and fittings are acceptable, clean and fit-for-purpose.

Step 3: Define the Survey Questions

The third and last step will be to break the topics down into ideally 3 – 5 survey questions each. As a general rule, try to come up with at least 3 survey questions for each topic so their responses (1) can, where necessary, be quantified and averaged into a score for the topic, and (2) the topic with its survey questions can be statistically validated at a later stage if required. Topics that cannot be broken down into a couple of survey questions should where possible be combined with other topics so they can be collectively measured and quantified.

The process is quite straightforward: take each of the main topics, review the topic’s definition, and break the topic down where practical into a couple of related aspects that could be measured separately.

To use the Cafeteria Happiness survey as example, the first topic (Food) can be broken down into 3 different aspects that should be measured:

1. Aspect 1: Variety – whether there are variation and balance in the menus;
   
   
2. Aspect 2: Quantity – whether the portions are adequate;
   
   
3. Aspect 3: Quality – whether the food is well prepared using quality ingredients.
   
   

All that remains now is to define at least one survey question for each of the aspects as identified. To use the Cafeteria Happiness survey again as an example, one or more survey questions have been defined for each of the Food topic aspects:

1. Aspect 1: Variety
   Question 1: The menus are varied and changed on a daily basis
   Question 2: The meals have a good balance of proteins, vegetables, and carbohydrates
   
   
2. Aspect 2: Quantity
   Question 3: The food portions are adequate
   
   
3. Aspect 3: Quality
   Question 4: The meals are well prepared
   Question 5: Good quality ingredients are used for the meals
   
   

And now repeat the process for the remainder of the topics as well (Service and Facility) – break each topic down into different aspects where possible, and define at least one survey question for each of the aspects.

Wrapping up the Survey Model

All the hard work has now been done. To wrap up the survey model so it is ready for stakeholder sign-off before you capture it on the Engage Survey platform, document the content of the 3 steps in a Word or Excel document and make it available to all the stakeholders so they can review and suggest any final changes. Or even better, use the Excel Take-on File template that you can download from the Onboarding section of the Resources site (see Excel Take-on File for more on this).

To summarize the wrapping up of the survey model:

• Survey name – come up with a more suitable name for the survey, e.g. Acme Corporation Cafeteria Happiness Survey;
• Survey constructs & descriptions – for context in the analytics dashboard, define short descriptions for each of the main topics as defined in Step 2 (Note: main topics are called survey constructs in the analytics dashboard);
• Survey questions – the questions can now be combined per survey construct (main topic), as shown below, and can optionally be shuffled around a bit to break the pattern and to force respondents to evaluate each question on its own merits.

Just bear in mind that the design of the survey model – with its constructs and survey questions – is separate from the survey questionnaires that the respondents will receive; Engage Survey will allow you to create more than one survey questionnaire layouts for the same survey model, and to include any number or permutation of survey questions in each of the different survey questionnaire designs.

See the appropriate sections in the Resources and Onboarding sites for more on defining a survey model via the Engage Survey platform, designing the survey questionnaire, finalizing the survey communications, and creating and launching a survey campaign.

Issues to Consider when Formulating the Survey Questions

Before you finalize the survey questions, maybe first consider the following issues and decide how you want to deal with each:

a. Should the different topics be quantified or not?

If you are using survey questions with a rating scale (Disagree to Agree) in a topic, and you want the Engage Survey system to calculate an overall score for the topic so you can (1) compare the scores at business unit/team or demographic level, and/or (2) track changes to the topic over time via follow-up surveys, you will have to make the topic a Quantified one. By selecting the Quantified option when you define the topic in Engage Survey, the system will automatically convert the responses for all rating-type survey questions (single-option Likert and dropdown questions) to a score out of 10, and then average the scores of all such survey questions for the topic. See Calculate Survey Scores for more on this.

Please note that you can still add open-ended (comment-type) questions to a Quantified topic if you would like respondents to provide you with free-format suggestions and comments regarding the topic.

Make the topic a Non-quantified one if you don’t necessarily want to convert the topic’s survey questions to a score that you can use for comparison or future tracking purposes. Non-quantified topics can include any combination of multi-option questions (where respondents can select more than one option in response to a question) as well as open-ended (free-format) and Net Promoter Score (NPS) questions (e.g. to determine satisfaction levels or likelihood to recommend – see NPS Questions for more on this.

b. Should survey questions be phrased as Questions or as Statements?

You will also have to decide whether you want to phrase the survey questions as statements or as questions (note: the Cafeteria Happiness survey's questions are all phrased as statements). Statements are easier to formulate since they use standardized Likert options that will require the respondents to indicate to what degree they Disagree or Agree with each of the statements (see Question 2 in the example below).

Phrasing the survey questions as Questions, though, will require more descriptive question-specific Likert options, and will be a bit more work (see Question 1 in the example below).

Survey questions that are phrased as Questions with descriptive, contextual Likert options will, if done properly, be easier to understand and quantify (see example below) and could arguably yield more accurate and reliable responses.

Note: the images/icons used for the different Likert options in the examples above are optional and can be user-defined.

c. Should the survey model be anonymous or not?

Survey models will by default be Anonymous when they are created, which means the survey results will only be shown if there are more than the minimum threshold number of responses (default minimum of 4) for the business unit/team or as set by the current filter. It also means that survey responses cannot be viewed or tracked at individual respondent level, but only at business unit level. Defining a survey model as Anonymous will guarantee the anonymity of the survey respondents, and should also be communicated as such in order to improve the survey response rates. This will be the preferred mode in case of engagement, culture, inclusion, or other employee feedback surveys where the idea is to identify and isolate trends and patterns at business unit and demographic level.

Survey models can alternatively be defined as Non-anonymous, in which case the responses of individual respondents can be viewed and analyzed where required. This will be particularly useful in cases of surveys that track performance or employee-specific experiences, e.g. feedback regarding an onboarding or training session. Care should however be taken to make sure that the respondents understand that their survey responses will only be confidential and not anonymous.

d. Will there be a need for different questions for some respondents?

Most employee surveys are designed to ask the same survey questions to all the respondents, which will make it easier to compare the data and to track changes over time.

There may be cases, however, where there is a need to ask different survey questions to different groups of respondents based on a previous survey response (choice) they have made, in which case conditional branching logic will have to be used. The conditional branching feature will allow you to guide respondents to survey questions that are relevant to them and/or their circumstances, and will result in a more personalized survey experience for the respondents concerned. See Including Conditional Branching in a Survey Questionnaire for more on this.

Best Practices with respect to Survey Design

The following suggestions and tips will help you to design a better survey model.

• Keep the survey short – to prevent respondents from getting survey fatigue and either abandoning the survey halfway without completing it, or completing it mechanistically, keep the survey as short as possible, ideally not longer than 30 or 40 questions, or if you really must 50 survey questions.
• Restrict survey questions to a single item – avoid asking two different things in the same question, e.g. Are you happy with the quality and the quantity of food? Respondents who are happy with the quality but unhappy with the quantity would not know how to respond.
• Include at least one open-ended question – including an open-ended question towards the end of the survey questionnaire will provide the respondents with the opportunity to voice their opinions and to express themselves.
• Restrict the number of open-ended questions – try to restrict the number of open-ended questions in a survey to 2 or 3, or maximum 4. Open-ended questions make the survey much longer, so if possible make them optional (i.e. not compulsory) and include them at the end of the survey questionnaire. Open-ended responses have to be analyzed and interpreted to be of any value, so adding too many open-ended questions will place an unreasonable burden on the analysis of the responses.
• Include demographic questions at the end of the survey – if demographic questions (e.g. How old are you?) are to be included in the survey, always include them right at the end of the survey. Demographic questions are invasive and personal in nature and may derail respondents if asked at the beginning of a survey. The preferred option in any event will be to not include any demographic questions at all, but to instead use pre-populated surveys where the demographic data are captured before the launch of the survey (see Online surveys via a targeted (pre-populated) survey links for more on this).
• Don’t allow survey constructs with only 1 question – trying to quantify a survey construct with a single survey question will not be of much value, so to improve the statistical validity of a quantified survey construct, try to include where possible at least 3 survey questions per survey construct. Alternatively, combine survey constructs that have only one or two questions with another construct. 
• Don’t ask the same thing twice – unless you deliberately want to test the reliability of a respondent’s responses, do not include survey questions that ask the same thing more than once, even if you rephrase the questions so they appear different.
• Don’t ask leading questions – to make sure the responses are not skewed, take care when phrasing the survey questions that you don’t ask any loaded or leading questions that can result in a biased response, e.g. Would you agree that the cafeteria provides outstanding food?
• Stay away from technical terms and jargon – phrase the survey questions specifically with the target audience in mind, so do not use any technical jargon or complex terms that may be open to incorrect interpretation.
• Don’t use absolute terms – try not to use words such as ‘always’ or ‘every’ in a survey question that may bias or prejudice the survey responses.
• Keep reverse score questions separate in the survey questionnaire – if you have to ask reverse score questions, include them in a separate section of the survey questionnaire; mixing reverse score questions with normal questions will confuse the respondents and may have a negative impact on the reliability of the survey responses.
• Be consistent with Likert scales – try to use the same number of Likert options (ideally 5) for all single-choice survey questions, and keep the scales the same, e.g. Disagree Strongly on the left and Agree Strongly on the right.