pollsters are rethinking how they operate after a string of astonishing misses around the globe this year<u+2014>from incorrectly calling the brexit vote in the u.k., the peace accord with rebels in colombia and now the u.s. presidential election . pollsters say a confluence of changes are making their jobs more difficult: people are changing how they communicate, moving from landlines to cellphones and the internet. that makes it harder to generate large random samples. plus, fewer people are willing to answer surveys. as a result, pollsters must more heavily weight the answers they get, which requires making assumptions that don<u+2019>t always prove true<u+2014>especially on the variable of who will show up to vote. <u+201c>i would say, as a businessman, <u+2018>yeah, we have to be concerned about the fact that the business i work in<u+2014>people are going to question its efficacy,<u+201d> said fred yang, a partner at hart research associates, a democratic polling firm in washington. the industry<u+2019>s trade association said this week it would conduct a review of the 2016 election to better understand what happened. the outcome also raises questions about the research businesses rely on to test new products and measure customer behaviors, since many of the same survey methods are used for market research. <u+201c>a corporate market research project, you don<u+2019>t know if your polling is shit because there<u+2019>s no election day,<u+201d> dan wagner, head of democratic research firm civis analytics, which also conducts nonpolitical surveys, said in an interview prior to the election. in politics, <u+201c>there<u+2019>s a day where you<u+2019>re going to find out whether you were right or whether you<u+2019>re an idiot.<u+201d> two decades ago, more than one-third of u.s. households contacted for a survey agreed to answer questions, according to the pew research center. now, that number is around 9%. <u+201c>it<u+2019>s basically gotten more difficult to be accurate,<u+201d> said patrick ruffini, head of republican research firm echelon insights. <u+201c>it doesn<u+2019>t take much for everybody to be a little bit<u+2014>or dramatically<u+2014>off about what the outcome is going to be.<u+201d> about half of all american households rely exclusively on cellphones, but reaching those people is expensive and time consuming because researchers can<u+2019>t call those people using auto-dialers. a law aimed at telemarketers requires anyone calling a cellphone to dial all 10 digits by hand. increasing costs have pushed many researchers toward new survey methods, primarily online, that aren<u+2019>t as well understood. a study by pew earlier this year found inaccuracies in online surveys that were hard to explain. <u+201c>when you had landlines, it was the easiest thing in the world,<u+201d> says michigan pollster steve mitchell. <u+201c>now it requires a great deal of work<u+201d> to weight the sample according to expected voter turnout. too much weighting can introduce errors, iowa-based pollster j. ann selzer says. <u+201c>the less you touch your data the better off you are,<u+201d> she said. <u+201c>every decision that you make about massaging this and weighting that, there can be attached to that a judgment call or a bias.<u+201d> for instance, ms. selzer says that four years ago, education level wasn<u+2019>t an important indicator for voting preference. this year it was. most pollsters based their models on what happened in the 2012 election, which ms. selzer says opened the door for problems. <u+201c>the assumption that the electorate would look the same and act the same was wrongheaded from the get go.<u+201d> mr. wagner says people who respond to polls may be fundamentally different from those who don<u+2019>t. in this case, mr. wagner<u+2019>s early analysis indicates that low-educated white people who are willing to answer polls don<u+2019>t seem to reflect the opinions of the group as a whole. mr. mitchell, in michigan, says his voter turnout model is what gave him trouble and it was exacerbated by the difficulty of reaching voters. mr. mitchell normally uses only landlines. but landlines reach far more older people than young people, so he weights their answers accordingly. then, on the thursday before election day, mr. mitchell began incorporating data from a new source: surveys done by people who responded to banner ads on mobile phones. mr. mitchell hoped the method would make it easier to reach young people. it collected more young people than mr. mitchell anticipated. but he neglected to weight the sample back down to his original estimates. because more young people preferred mrs. clinton, mr. mitchell<u+2019>s poll showed that her support had increased from 3 points to 5 points. mr. trump leads in michigan by 0.3%, but the associated press still deems the race too close to call. <u+201c>i would say the state of the craft remains challenging and challenged,<u+201d> said mr. yang, whose firm directs the wall street journal/nbc news poll with the republican firm public opinion strategies. <u+201c>but it wasn<u+2019>t just because of tuesday. it<u+2019>s been like this for a while,<u+201d> because of the difficulties of building a sample that accurately reflects the voting public and of estimating which people will actually vote. mr. yang saw little evidence that any one method for building a sample<u+2014>those done by phone, those without live interviewers or those that rely on a range of data sources<u+2014>proved itself better than any other. the one exception, he said, might be a method employed by the los angeles times and its partner that assembled a panel of 3,200 respondents and re-interviewed them regularly. that method was criticized during the campaign for consistently showing mr. trump ahead. the final estimate was that mr. trump would win by 3 points. but that poll wasn<u+2019>t entirely accurate, either, because it is mrs. clinton who is on pace to win the popular vote, by about 1 percentage point. chris wilson, chief executive of wilson perkins allen opinion research, said his skepticism of polling has led his firm more deeply into data analytics<u+2014>using voter files and other databases, as well as predictive models, to get a sense of public opinion. traditional polling <u+201c>is not going to survive,<u+201d> he said. <u+201c>it<u+2019>s going to go away in its current form.<u+201d> write to ryan knutson at ryan.knutson@wsj.com and aaron zitner at aaron.zitner@dowjones.com
pollsters face hurdles in changing landscape - wsj
ryan knutson aaron zitner