New Book Examines ‘The Problem with Survey Research’

Social scientists and marketers expressed dismay when the U.S. House of Representatives recently voted to eliminate the Census Bureau’s American Community Survey, a source of data on income, housing, education, labor force and other demographics.

However, a University of Illinois at Chicago researcher maintains that this and other surveys, polls, interviews, and focus groups produce unreliable results.

In “The Problem with Survey Research” (Transaction Publishers, 2012), George Beam, UIC associate professor of public administration, argues that survey research produces only responses to questions.

“If you want to find out what’s really going on, don’t ask,” Beam advises researchers. Reliable information, he says, can be acquired only by observation, experimentation, formal modeling, document analysis, comparison, and using multiple sources of data.

“When all you have are answers, it’s impossible to know which, if any, are correct,” Beam said. “Low response rates, wording, setting, the matter of who’s asking — all affect the responses. Researchers can’t know whether the answers correspond to respondents’ actual behaviors or their true beliefs.

Beam notes that researchers recognize the drawbacks of survey research, but continue conducting it because it is institutionalized.

In the case of the American Community Survey, Beam said, “unreliable answers determine how more than $400 billion in national and state funds are distributed. Answers are skewed by the sensitivity of questions like, ‘What time do you leave for work? Do you need help in going shopping? Your income?’ And by the threat of a $5,000 fine for not responding.

“The problem extends beyond the academic and government realms,” he said. “The media also rely on polls and interviews.”

In his book, Beam explains the ubiquity of survey research, its documented unreliability, the ambiguous design of oral and written survey instruments, and how settings and sources affect responses.

Beam suggests more scientific methods of research in his final chapters. In an appendix, he discusses surveys that are not meant to produce usable information, but simply allow respondents to speak their minds, receive an advertising message, or support a candidate or business.

UIC ranks among the nation’s leading research universities and is Chicago’s largest university with 27,500 students, 12,000 faculty and staff, 15 colleges and the state’s major public medical center. A hallmark of the campus is the Great Cities Commitment, through which UIC faculty, students and staff engage with community, corporate, foundation and government partners in hundreds of programs to improve the quality of life in metropolitan areas around the world.