BigSurv18 program
Wednesday 24th October Thursday 25th October Friday 26th October Saturday 27th October
Respondents Responding Well or So I Recall… Exploring Memory, Mode, and Behaviour in Surveys |
|
| Chair | Dr Linsay Gray (MRC/CSO Social & Public Health Sciences Unit, University of Glasgow) |
| Time | Friday 26th October, 16:00 - 17:30 |
| Room: | 40.035 S. Graus |
Different Strokes for Different Folks: An Assessment of Mode Effects in a Student Population
Dr Rebecca Powell (RTI International) - Presenting Author
Dr Antje Kirchner (RTI International)
Dr Austin Lacy (RTI International)
Mr Johnathan Conzelmann (RTI International)
One method to increase sample representativeness and reduce potential bias due to nonresponse is to offer respondents different modes of survey completion. This helps ensure all sample members have an opportunity to complete the survey in a mode they have access to or are more willing to participate in. Research indicates that different types of modes (e.g., web and phone), reach more diverse people and ensure higher data quality (e.g., de Leeuw 2005) as different types of people tend to respond via different modes (i.e., potential mode selection effect). However, surveying respondents through different modes may result in differences in responses and data quality when comparing the same question across the modes (i.e., potential mode measurement effect).
While survey data allows us to compare differences in estimates between two survey modes, without an external criterion (i.e., a “truth”) it is unclear which of the estimates are more accurate without assumptions. High quality administrative data available for both respondents and nonrespondents can be considered such an external criterion and can be used for multiple purposes. First, administrative data that includes demographic data about the sample members can be used by researchers to isolate the mode measurement effect by adjusting for the mode selection effect. Second, administrative data can also be used to evaluate the amount of measurement error within each mode by comparing this gold standard with the survey responses.
In this paper, we test several hypotheses for mode effects (e.g., lower measurement error in self-administered modes for sensitive questions) using data from the 2012 National Postsecondary Student Aid Study (NPSAS:12) to examine mode effects in a survey completed either via telephone or web. In addition to survey data, NPSAS:12 collects a wide variety of data on students through administrative records—such as government data on education finances. To assess mode measurement effects on the survey responses, we compare survey responses with the administrative data for our sample of over 11,000 students on 24 variables of interest. First, we account for mode selection effects by adjusting the data using covariate balancing. Then, we calculate the mode measurement effects by comparing discrepancy indicators and a measure of direction and magnitude for each mode. Mode effects are investigated for four different student populations: (1) undergraduates who completed the full survey, (2) undergraduates who completed the abbreviated survey, (3) graduates who completed the full survey, and (4) graduates who completed the abbreviated survey.
Preliminary results indicate two main findings. First, there is a limited presence of mode effects for respondents who completed the full survey, and there are no reporting differences across modes for respondents who completed the abbreviated instrument. Second, there are differences between graduate and undergraduate students. Specifically, undergraduate students tend to have more measurement error when completing the survey via telephone, while graduate students tend to have more measurement error when completing the survey online. This paper will develop hypotheses, explore these findings, and provide researchers with a potential framework for calculating and adjusting for mode effects in their own studies.
Memory Bookmarking: Using Multimodal Real-Time Data to Facilitate Recall
Ms H.Yanna Yan (University of Michigan)
Dr Frederick Conrad (University of Michigan) - Presenting Author
Understanding human behavior is key goal across the social sciences, but accurately measuring the people’s everyday behaviors is often difficult. To collect information about people, social scientists frequently ask survey questions requiring retrospective self-reports. Despite the popularity of this approach, answering this kind of question can impose considerable burden on respondents, even over 24 hours as is the case in typical time-use surveys. Not only is recall difficult it is often inaccurate.
To address the recall problem, this paper provides a first report on a new measurement approach called “Memory Bookmarking” (MB). Using the MB approach researchers collect real-time information via smartphone about respondents’ momentary circumstances and then feed this information back to respondents as memory cues in a follow-up time-use survey. Such “memory bookmarks” (i.e., cues) should enable respondents to recall the details of the cued event and other related events, much like an actual bookmark allows readers to open the book to a meaningful location and see what is on the previous and next pages.
In the study reported here, we first measure respondent’s time-use - all of their activities over the previous 24 hours - using a blank, online time-use survey (“time diary”). Upon completing the empty diary, respondents are randomly assigned to one of the five experimental conditions distinguished by whether or not they are asked to provide cues on the day before completing a second time diary and, if so, how they are asked to “describe” their current situation at the time they are signaled: (A) control, i.e., no cues are solicited, (B) respondents are asked to text a verbal description of what they are doing, (C) respondents are asked to text verbal descriptions of what they are doing and how they are feeling, (D) respondents are asked to send a photograph depicting their activity or physical surroundings, and (E) respondents send GPS data from their smartphone indicating their location. Respondents in Conditions B - E are signaled four times in a day and asked to provide memory cues in their assigned medium. Respondents’ time-use is measured again using either a blank time diary (group A) or a customized time dairy in which the cues they provided the day before are embedded (group B-E).
The two waves of time-use measurement enable us to assess the benefits of memory bookmarks by looking at change on several measures from recall with no cues (wave one) to recall with cues (wave two). And it allows us to examine in which modes the cues promote better recall and thus better data quality. These comparisons will yield two major insights: 1) whether and how feeding back real-time event descriptions improves data quality; and 2) the extent to which respondents comply with a potentially intrusive task. This study brings together organic data such as photographs of one’s situation or GPS coordinates with conventional self-reports to facilitate people’s recall of what they do and how they feel.
What Are the Effects of "Forcing" Respondents to Behave in Certain Ways?
Dr David Vannette (Qualtrics Methodology Lab) - Presenting Author
Dr Mario Callegaro (Google)
Dr Yongwei Yang (Google)
Dr Steven Snell (Qualtrics)
The validity and reliability of survey data often rely on respondents giving careful and appropriate responses. As a result survey designers have increasingly requested tools that enable greater control over the survey response process. Software vendors have complied by providing tools that can do things like validate content provided in open-ended text responses, constrain or penalize ‘speeding,’ and even ‘force’ or ‘request’ responses to questions from respondents. While many researchers may infer that using these tools to shape responses by respondents produces higher quality data, very little existing research has looked at what the actual effects of these tools are on both the respondent experience and ultimately data quality. In this study, we use paradata, self-reports from respondents, and a variety of measures of data quality, to assess the effects and effectiveness of efforts to shape respondent behavior in surveys. Additionally, we examine big data in the form of a large corpus of survey metadata from a survey software vendor to understand the prevalence and effects of these efforts at scale. The results of this work will inform existing research and theory about the survey response process and questionnaire design. Importantly, there may be heterogeneity in how respondents from different sampled populations respond to these efforts that is important to measure and understand. In this study, we examine populations of respondents that come as purchased samples from online panel vendors and also samples that have not been sourced from an online panel to identify if there are differences in these key populations of survey respondents. We hope that this work will be valuable to researchers across disciplines and applications of survey research and will provide tools for optimizing both data quality and respondent experience.