Designing Effective Surveys

Surveys, also known as questionnaires, are valuable tools for collecting data from individuals. They involve presenting respondents (the people taking the survey) with a series of questions and recording their answers. This data can reveal demographic information (such as age, gender, race, and income) as well as numerically scaled data on people’s satisfaction, opinions, impressions, and attitudes about a particular subject. As Bezuidenhout et al. (2014) and Brink et al. (2018) note, surveys are designed to encourage honest responses and build trust with participants.

How Surveys Work

Quantitative surveys often rely on checklists and rating scales to simplify and quantify responses, making it easier to analyse attitudes and behaviours. These surveys involve asking questions, which is also interpreted as a structured form of interviewing. While similar to everyday conversations, interviews in surveys differ significantly as the interviewer leads the questioning, while the respondent has limited opportunity to ask questions. Although the interviewer guides the process after consent is given, the respondent retains control over whether and when to end the interview.

It’s crucial to have a strong rationale for each question. This rationale should stem from prior research, relevant literature, or a testable hypothesis. In robust academic research, these sources often converge, ensuring that each question has a clear and justifiable purpose.

Traditionally, surveys were printed documents where information was recorded directly (Bezuidenhout et al., 2014; Brink et al., 2018). However, digital alternatives have become increasingly popular, allowing researchers and respondents to use tablets or smartphones for data collection (Bezuidenhout et al., 2014).

Designing Good Survey Questions

Designing an effective survey requires careful consideration. Factors such as question wording, text format, and even paper colour can influence how respondents answer (Brink et al., 2018). Creating a survey from scratch requires significant effort, attention to detail, and extensive testing. This process can take months, as researchers work to ensure their questions meet rigorous validity and reliability standards.

Fortunately, researchers can often use pre-existing, validated questions, provided they cite the original authors appropriately. Established tools, such as the Myers-Briggs Type Indicator (MBTI), demonstrate the value of leveraging tried-and-tested questionnaires. Developed during World War II and first published in 1962, the MBTI is a psychometric tool designed to classify personality types. While it has faced criticism, such as inconsistent results during re-testing (Krznaric, 2013), it is generally considered a reliable and valid instrument.

Unfortunately, the MBTI is not available without special training and can only be used under licence. Many popular surveys are not available free of charge or can only be used with special permission or under licence. Fortunately, there are several sources that offer free access to questionnaire templates . Microsoft and Google, for example, have freely available tools and templates which can be used. Alternatively, SurveyMonkey, SurveyExpression or Bristol Online Surveys (BOS) offer free access to templates and limited use of their services.

Before creating new questions, it's wise to explore existing ones, especially if they have been rigorously tested. Using established questions is not only acceptable but often advantageous, provided the original authors are properly credited. Many researchers are often willing to share their surveys, particularly if their work is cited appropriately.

Validity and Reliability

A significant advantage of utilising pre-existing questions is that they may have already been evaluated for validity. Validity refers to the extent to which the questions accurately measure what they intend to measure (Cresswell, 2013). There are several key types of validity.

1. Validity refers to the extent to which the questions accurately measure what they intend to measure (Cresswell, 2013). There are several key types of validity:

   
   

2. Content Validity: This assesses whether the questions comprehensively cover all aspects of the subject matter being investigated.

   
   

3. Construct Validity: This examines whether the questions effectively measure the underlying concepts or theoretical constructs that the research aims to explore.

   
   

4. Face Validity: This refers to whether the questions, at first glance, appear to measure what they are supposed to measure.

Establishing face validity often involves piloting the questionnaire with a small group of participants to gauge their interpretation of the questions. A high level of agreement (e.g., over 70%) between the researcher's intended meaning and participant understanding indicates strong face validity. Internal validity can also be demonstrated through detailed descriptions of the question development process or by explaining the relationship between the questions and relevant theory.

1. External validity, on the other hand, addresses the generalisability of the findings to a broader population (Cresswell, 2013):

   
   

2. Concurrent Validity: This assesses whether the survey results correlate with results obtained from other established measures of the same concept.

   
   

3. Predictive Validity: This examines whether the survey scores can accurately predict future outcomes or behaviours.

If you choose to create your own questions, it is essential to evaluate them against these different types of validity.

Reliability, another critical concept, refers to the consistency of results (Cresswell, 2013). Reliable questions should produce similar results if administered in different settings or at different times. Key aspects of reliability include:

1. Question Consistency: This ensures that the questions consistently measure the same underlying constructs.

   
   

2. Test Stability: This evaluates whether individuals provide similar responses if they take the same survey again at a later time.

   
   

3. Consistency in Test Administration: This ensures that the questions are asked and the responses are recorded in a consistent manner, minimising errors.

For example, a question like “In which city do you currently live?” is highly reliable, as most respondents will interpret and answer it consistently. In contrast, a subjective question such as “How much money does it take to be happy?” is less reliable due to varying interpretations of happiness and monetary value. Being able to demonstrate high levels of both validity and reliability significantly strengthens the credibility and trustworthiness of the data collected.

In quantitative research using surveys, the goal is to obtain responses to the same set of questions from all participants. This promotes objectivity by reducing the influence of researcher bias. Questionnaires are often favoured for their cost-effectiveness, convenience, and ability to reach large and geographically dispersed populations. Self-completion questionnaires are particularly well-suited for these purposes.

Critical Evaluation and the Importance of Question Quality

Despite the convenience of existing tools, it is crucial to critically evaluate the validity and reliability of any questionnaire used. Robson (2011) cautions against overemphasising the importance of questionnaires simply because they generate quantitative data, suggesting that responses may be influenced by factors such as politeness, boredom, or social desirability rather than genuine beliefs or behaviours.

To maximise both reliability and validity, questions should adhere to three key principles, as suggested by Alreck and Settle:

Specificity: Questions should precisely target the information needed, ensuring a strong connection between the question and the research objectives.

Clarity: Questions should be easily understood by all respondents, avoiding jargon, technical terms, or ambiguous language. The vocabulary used should be appropriate for the target audience.

Brevity: Questions should be concise and straightforward, using simple language and avoiding complex sentence structures. Compound or compound-complex sentences should be broken down into simpler questions.

By carefully considering these factors and critically evaluating the chosen instrument, researchers can ensure that their surveys yield meaningful, reliable, and valid data, contributing to robust and trustworthy research findings.