Nishmeet is a Research Analyst at Outline India. He has previously worked as a Research Intern with Center for WTO Studies, IIFT New Delhi on Asia Pacific Economic Cooperation (APEC) and is particularly interested in International Trade. His expertise lies in number crunching and handling large data sets along with designing. Nishmeet completed his Masters in Economics from the South Asian University and holds a Bachelors in Economics from Khalsa College, Delhi University.
Pre-Testing Survey Questionnaires
A cohesive, well-structured and easy to understand questionnaire is a key component in the process of collecting valid, reliable, unbiased and complete data. It ensures that our questions measure the concepts or behaviors we want them to measure and thus the data produced is a true representation of the study objective.
Pre-Test is an important step to check the reliability and validity of a questionnaire before starting actual data collection. It helps one identify questions that are either difficult to comprehend or misleading for the participants. It also leads to early detection of problems with the questionnaire, that might lead to biased answers.
In research, the usage of standardized questionnaires and procedures may lead to deviations and errors in measurement because they are based on certain implicit assumptions such as
- Questions are understood by all respondents in a consistent manner.
- Questions retrieve the required information.
- The wording of questions provides the respondents with all the necessary information to be able to answer them in a way required by the researcher.
- In case of interviewers, the questions are read as worded without any difficulty.
In order to reduce the probability of collecting data with errors pre-testing or piloting the questionnaire is recommended.
The process of pre-testing involves completion of a few interviews by a group of experienced interviewers and researchers. The results from these surveys are then analysed case by case, to ascertain questions that worked and those that failed to attain their intended objective. As on offshoot, with the advent of using digital devices or CAPI software for data collection, pre-testing exercise enables one to check for programmes skips, validations and overall functionality of the survey questionnaire.
The literature on social surveys has identified both qualitative and quantitative methods for pre-testing survey questionnaires.
- Expert Review: Research experts in questionnaire designing or subject matter study the questionnaire for potential problems and give recommendations for possible improvements.
- Focus Groups: Small group of dummy respondents brought together to investigate their understanding and comprehensibility of the research topic and a few survey questions.
- Cognitive Interviews: It is similar to focus groups but involves deeper understanding of the process of response/answer formulation by the respondents. In this method, interviews are done with additional questions which aim to probe the understanding of how the respondent arrives at an answer.
- Behavior Coding: It is a technique in which a coding system is used to understand the problems in interpreting or asking of survey questions. It can provide a quantitative summary of problems at the question level, respondent level or interviewer level.
- Latent Class Analysis: It helps in mitigating or estimating the effect of measurement errors in surveys. The technique involves identifying unobserved/latent variables that explain systematic relationship between observed variables. The method involves estimation of conditional/ unconditional probabilities to ascertain the relationship
- Field Test: The technique involves conducting pilot interviews and testing the dummy data to fit the analysis plan, analysis of data for key variables or observation of any missing variables or values. It also helps check the validity and reliability of the underlying statistical model.
The aforementioned methods are the ones commonly used by researchers in pre-testing. Additionally, it is always important to follow-up the pre-testing exercise by a comprehensive report comprising of question by question review of results, with recommendations. The report must also have information about the subject population and recruitment process, the techniques used to pre-test and the expertise and number of interviewers. It is also essential to include limitations of testing in the report.
Do reach out to us, for any questions/concerns on thoughts at www.outlineindia.com