Experts say caller ID is destroying our understanding of jobs market

The UK’s Office for National Statistics used to knock on doors and ask people if they were employed, as a traditional way of compiling the country’s labor market data.

But when COVID-19 forced social distancing in 2020, door-knocking was out. At one point during the pandemic, the survey was reliant on how many phone numbers it could find for a representative sample — and also on people actually answering their mobile phones when an unknown number called.

Response rates plunged. By 2023, the data had become so patchy the ONS was forced to suspend its unemployment reading as it desperately sought to repair a key release that helps inform government policy, influences interest rates and drives billions of dollars of investment decisions. An ensuing crisis of confidence in Britain’s data ended with the resignation of its national statistician, Ian Diamond, last month.

Getting households to pick up the phone is just one of many obstacles facing the world’s stats geeks, including government agencies tasked with collecting numbers on everything from joblessness to agricultural output. Just as cellphone users swipe away unwanted callers, people screen visitors with video doorbells and an increasing number live in apartment blocks impenetrable to data collectors, compounding a collapse in survey responses.

That’s left statisticians across the world scrambling to update collection methods stuck in the 20th century without introducing new 21st century biases. The solution that agencies and pollsters, from the ONS to the Bureau of Labor Statistics in the U.S., have zeroed in on: online surveys.

These polls are less costly than telephone or face-to-face surveys, more convenient for respondents, and statistics agencies are using AI to target nonresponders. Yet the big analog-to-digital switch in economic data is throwing up new problems for the indicators global investors and policymakers rely on.

“There’s no magic mode,” said Steve MacFeely, chief statistician at the Organisation for Economic Cooperation and Development. “They’re well aware of the risk and are doing it because they have no other choice.”

Problems with the new wave of online surveys include fresh biases in the data, confusion over questions, gaining trust amid a barrage of spam and fraud in people’s inboxes, and the fact that households may be just as likely to ignore an email as a phone call.

“What you gain in coverage/response, you possibly lose in measurement quality or content features,” said Eric Harrison, deputy director of the European Social Survey.

Problems with the UK’s labor market figures serve as a cautionary tale.

Britain’s efforts to create a new online-led labor market survey came after the response rate to its household survey collapsed to below 15%, forcing the ONS to temporarily withdraw its key labor market estimates in late 2023. BOE rate-setters complain it is making it far more difficult to set interest rates, even with the reinstated figures.

A full transition to an online questionnaire in the UK may not happen until 2027 after teething problems in testing. Early results showed the new online survey was producing significantly different readings from the existing survey that relies on telephone calls and knocking on doors, people familiar with the situation said.

One potential issue is that respondents aren’t able to ask an interviewer to clarify a question. The ONS is looking at an AI chatbot that could help.

Another key risk from online surveys is that it creates new unknown biases that statisticians have to counteract. In the UK, the existing labor force survey struggled to get enough young workers to answer. For an online questionnaire, it could be the opposite problem.

In the U.S., Joanne Hsu, director of consumer surveys at the University of Michigan, believes that being behind the computer screen may also skew how people respond to surveys.

The Michigan survey’s closely watched figures on economic sentiment and inflation expectations in the U.S. fully transitioned to a web-only mode in 2024. While it helped to double the survey’s reach to nearly 1,500 responses, Hsu noticed that respondents are more likely to give extreme answers on the web — especially for inflation expectations, which recently surged to the highest in three decades.

While the U.S. has seen a smaller drop off in responses to many of its official surveys, the rate for the Current Population Survey underpinning labor statistics has fallen by almost 20 percentage points in a decade to 68%. For the Job Openings and Labor Turnover Survey or JOLTS report, a business survey, response rates have tumbled to almost 30%.

Former U.S. Bureau of Labor Statistics commissioner William Beach said that U.S. data could soon be in the same state as Britain’s moribund labor statistics without action to modernize its surveys.

Labor force surveys “are dying, they’re decaying, they’re in very serious trouble,” Beach recently said on Bloomberg’s Odd Lots podcast. “Unless we modernize that survey, we will see a time when we will be like the British, unable to publish portions of it that just don’t have sufficient sample for statistical release.”