BigSurv18 program
Wednesday 24th October Thursday 25th October Friday 26th October Saturday 27th October
More Than Words Can Say: Leveraging Data Science Methods to Get the Full Story about Survey Respondents |
|
| Chair | Professor Gabriele Durrant (University of Southampton) |
| Time | Friday 26th October, 09:45 - 11:15 |
| Room: | 40.213 |
Combining High-Volume Paradata With Survey Data to Understand Respondent, Instrument, and Interviewer Effects on Response Latencies
Professor Patrick Sturgis (University of Southampton) - Presenting Author
Professor Gabi Durrant (University of Southampton)
Dr Olga Maslovskaya (University of Southampton)
Professor Ian Brunton-Smith (University of Surrey)
It is well known that interviewers are key to determining the quality and cost of data collection in household interview surveys. They influence the likelihood that sample members will respond and, therefore, the bias and precision of estimates. They also affect the answers that respondents give in ways that can make them more, or less accurate. These effects have been robustly demonstrated in the existing literature. In this paper we consider a different and less well studied outcome; the extent to which interviewers affect the time respondents take to answer questions. Because interview length is often used as an indicator of both response quality and fieldwork costs, the answer to this question is consequential. We address this question by linking high volume paradata on response latencies from Understanding Society, a large face-to-face household interview survey, to survey data on interviewers and respondents to understand the causes of variability in response latencies. The linked data file contains over 3 million records and has a complex, hierarchical structure. We apply a cross-classified mixed-effects location scale model to response latencies in the UK Household Longitudinal Study. The model extends the standard two-way cross-classified random-intercept model by specifying the residual variance to be a function of covariates and additional interviewer and area random effects. This extension enables us to study interviewers’ effects on not just the ‘location’ (mean) of response latencies, but also on their ‘scale’ (variability). The combined data file enables us to estimate complex interactions between interviewers, respondents, questions, and response styles on response latencies.
Eight Seconds From Opine to Click - Respondent and Question Effects on Response Times in a Large-Scale Web Panel
Professor Oliver Serfling (Faculty of Society and Economics, Rhine-Waal University of Applied Sciences) - Presenting Author
The median web survey participant needs 8 seconds to read, comprehend, and select on one out of up to ten alternative answer options - but what variables determine the respondents’ answer speeds? Based on rational choice theory and stratified samples drawn from 3.5 million users with more than 100 million responses from a German web survey, linear and logistic regressions with panel data were run to test hypotheses on respondent behavior. This study identifies significant respondent, questionnaire, and interaction effects on their response times.
The findings show that respondents’ sociodemographic characteristics such as age, gender, marital status, and education have significant effects on their response times. Age has an inverse U-shaped effect, with minimal response times in the thirties. The same nonlinear pattern is found with respect to participation experience, measured by the number of polls conducted in the past and their length of panel membership. Moreover, full-time employees and married respondents are the fastest, with divorced and self-employed responding slowest. Additionally, the response time is inversely related to the number of years of schooling. This education effect is exacerbated when interacted with the complexity of the polling question, as measured by its word count.
With regards to questionnaire effects, the results reveal increased response times for wordier questions and answer options. Furthermore, the response time increases with the average word size in the text. However, the amount of text in answer options can be faster processed if it is split on a larger number of answer options and if there is a larger variance in the word length. Results on the relationship between education and the number of answer options indicate that less-educated participants tend to reduce their cognitive effort by selecting an answer at random - leading to lower response times than those with higher levels of education.
Examining the time when the survey took place, participants are found to react slower than average throughout the weekend and between 11pm and 4am, with the slowest being between 1am and 2am. They are quickest when surveyed between Tuesdays and Thursdays and between 9am and 10am, with considerably below-average response times from 7am to 3pm.
By providing a “don’t know” option (DK) in a survey question, response times are increased by approximately one second, leading us to conclude that the average user evaluates the question and answers carefully before choosing the “don’t know” option. We show that this DK effect can be split up into (1) a response time enhancing effect of the provision of a DK option that exceeds the effect of an additional answer option and (2) additional time to reason. However, there is also evidence that longer questions may slightly prompt the use of the DK option.
The results contribute to the survey research literature on respondent behavior and have practical implications for the design of online surveys. Additionally, they can act as building blocks, leading towards an assessment of the reliability of users and their answers in an anonymous interview.
Using Paradata to Interpret an Autoforward Experiment
Mr Jeldrik Bakker (Statistics Netherlands) - Presenting Author
Dr Marieke Haan (University of Groningen)
Dr Peter Lugtig (Utrecht University)
Professor Barry Schouten (Statistics Netherlands / Utrecht University)
Paradata are automatically collected data that are about the process of answering the survey. These kinds of data are generated by the actions of the respondent and the online software then stores these data, for example: the break off points, the number of mouse clicks, and the response times. In this study, we use paradata to interpret an autoforward experiment in a mixed-device online health survey. In surveys with auto forward functionalities, respondents are directly brought to the next question after selecting a response option. In general, auto forwarding (AF) leads to shorter completion times because respondents have to click less compared to manual forwarding (MF). Especially for smartphone respondents, AF could be helpful to reduce their burden and prevent them from dropping out: AF only requires one ‘touch’ per question. Therefore, in this study response time is the key variable of interest.
We implemented an AF experiment in the Health Survey 2017 of Statistics Netherlands. Former respondents of Statistics Netherlands were invited to participate in the survey. We used a stratified sample with the following variables: age (16-29, 30-49, and >50), and device use (smartphone or tablet). Subsequently, systematic sampling was used to obtain a national representative sample. Respondents were randomly assigned to the AF-condition or the MF-condition. Due to this sampling strategy we were able to conduct analyses for all devices: desktop (n=360), tablet (n=592), and smartphone (n=536). During data collection, paradata were collected which resulted in over 1 million available observations for response time analysis.
In the presentation we will focus on response times within the AF and MF design, and analyze these data for all devices used for survey participation: PC, tablet or smartphone. The benefits and difficulties of using an autoforward design in an online survey will be discussed. In addition, we will show the challenges we encountered when preparing these paradata for analysis.
Research-Driven Product Development With Surveys, Big Data, and Machine Learning: A Google AdWords Case-Study
Dr Inna Tsirlin (Google) - Presenting Author
Dr Soeren Preibusch (Google)
Businesses and marketing agencies use online advertising to promote their goods and services. Their advertising goals span from building awareness, to encouraging potential buyers to explore offerings, to closing a sale. Google AdWords is a flagship online advertising platform where enterprise users can create, manage, monitor, and optimise their online advertising. In AdWords, advertising campaigns can be created to target the entire marketing continuum from considering a product to purchasing it.
When creating a new campaign, AdWords asks the users for their Marketing Objective (MO) to guide them to the specific features designed to help their campaign succeed. The product development and design of this MO selection step was built on research that has combined survey methods, big data log analysis, and machine learning. Here we report on the practical application of big data plus surveys in Google’s research on AdWords.
During the exploration phase, we assessed users’ understanding of the now-replaced user interface. We used a survey to collect advertisers’ marketing goals and then mapped their stated goals to the MOs they actually selected in the product when creating new campaigns. We found that many campaigns were created with goals that did not match the advertisers’ stated goals, which indicated a lack of understanding of the MOs in the product. Log analysis of micro-interactions in the UI also pointed to confusion with the MO selection process.
During the redesign phase, we surveyed small to large businesses to collect marketing goal descriptions in free-form from business owners advertising online (with AdWords and other tools) (N = 3k). Analysis used a combination of a deep neural net trained as a conversational agent and a modified k-means clustering algorithm. This analysis identified the main types of the marketing goals and also the wording advertisers were most likely to use.
During the evaluation phase, a redesign of the user interface where advertisers state their MOs was deployed for a small random sample of users. A combination of qualitative user studies, behavioural patterns from logs, and in-product surveying showed improved advertiser understanding: goals were selected faster, fewer users were dissatisfied with the meaning of the marketing objectives, and confusion was reduced.
From our Google AdWords case-study in research-driven product development with surveys, big data and machine learning, we distil recommendations from mixed-methods approaches for practitioners and best practices for implementation.