Online survey platforms enable faster and more cost-effective data collection across public, private, and academic sectors, but they also introduce opportunities for fraud. The persistent, evolving nature of fraud requires constant adaptation to preserve data quality. Our prior research examined data quality amongst the leading pay-for-data collection platforms, where only 10.2% of MTurk respondents provided acceptable-quality data. This study aims to demonstrate how follow-up verification surveys can improve the security and efficiency of incentive payouts by verifying respondent demographic information to reduce fraud.
An orthopedic activity level assessment survey was disseminated via MTurk, incorporating updated quality checks to refine fraud detection, yielding over 5,000 responses. The survey administrator received emails regarding survey compensation issues, prompting a standardized follow-up survey designed to verify previously reported demographic information sent in response to email communications. Responses from the original survey were match merged with the follow-up survey data, and a descriptive comparative analysis was conducted to identify inconsistencies.
The preliminary findings indicate that follow-up surveys can successfully identify fraud. After omitting duplicate emails and survey submissions, 28 emails were received regarding issues with incentives. Of those, only 50% of respondents completed the follow-up survey. Among those who completed, 1 had a zip code discrepancy, 2 had birth year discrepancies, and 1 had an email inconsistency, compared to previously reported information. Of the 13 who did not complete the follow-up survey, nearly half of the distinct MTurk Worker IDs shared the same email in the original survey.
While minimal discrepancies were found among respondents who completed the follow-up survey, more substantial fraud was evident among those who initiated contact for incentives but failed to complete the follow-up survey. This study reveals the potential of follow-up surveys serving as secondary checks to detect fraud, allowing practitioners to implement automated demographic consistency checks before releasing incentives.
Think You Can Fake It? We'll Make You Verify It: Utilizing Follow-up Surveys to Detect and Prevent Fraud in Online Crowdsourced Data
Category
Student Abstract Submission