How reliable are BLS labor force estimates?

USA 20 May 2015. The jobless rate and the labor force participation rate are two of the most widely reported monthly figures released to the public by the Bureau of Labor Statistics. Both are based on information derived from the Current Population Survey involving approximately 60,000 U.S. households that are randomly selected and interviewed by the Census Bureau. As is true of all the data from the CPS, the jobless rate and the participation are subject to sampling errors that derive from surveying only a relatively small number of households rather than every household in the United States. The CPS data are not headcounts of the kind that derive from the Decennial Census. They are estimates.

To achieve the 90 percent confidence level with data drawn from the CPS, the BLS applies a test that requires that any month-to-month change in a given estimate must be greater than 1.6 times the relevant standard error for that estimate.  Failing that test means that the change is not statistically significant, is not reliable. That cautionary note is included in every monthly CPS report on the U.S. labor force.
For example, to achieve statistical significance the month-to-month change in the jobless rate for men 20 years of age and older must be at least 0.25 percentage point (PP). For those men, the change between March and April this year was not statistically significant.

The BLS has published a table that shows whether the March-to-April 2015 change is statistically significant for 32 labor force estimates including employment status (7), unemployment rate (13), reason for unemployment (4), duration of unemployment (4), employed workers at work part-time (4). Only three of the 32 changes – the change in the unemployment rate for Asians and for persons with a bachelor’s degree or higher and the change in the number of persons unemployed for less than 5 weeks – were statistically significant. None of the other 29 changes met the test of statistical significance including the “drop” in the jobless rate to 5.4 percent and the “rise” in the participation to 62.8 percent that were widely reported in the media as if they were statistically reliable.

In sharp contrast the record of year-to-year changes is most impressive. Of the same 32 dimensions of the labor force, year-to-year change for 25 met the test of statistical significance.

The BLS table is accessible at http://www.bls.gov/web/empsit/cpssigsuma.htm.

A simple rule of thumb: the smaller the population the greater the standard error for any month-to-month change in the estimated unemployment rate for that population. For teenagers, any such change in the estimated unemployment rate must reach +/-1.63 PP for statistical significance. For all persons 25 years of age and older it is +/-0.18PP.  Another is pay more attention to year-to-year change.

Business news reporters and news anchors should consider keeping a copy of that table on hand when they draw information from the CPS, reminding themselves from time to time that an estimate is not an actual headcount. Otherwise in rushing to be the first among their competitors to report information from the CPS they are likely to mislead the American public.

How reliable are BLS labor force estimates? One problem with reliability relates to the manner in which the data are collected. Today enumerators contact the households selected for the survey by telephone. Years ago Census enumerators were dispatched directly into those households where they could make a rough judgment as to whether the respondent was being truthful based on the responses given to the survey questions and conditions in the household.

Another is that the CPS does not include questions regarding the legal status of persons who are foreign born. For that reason the household respondent does not have to openly lie about any undocumented alien who lives in that household. All she/he has to do is to say nothing. If such a person living there is hiding from immigration authorities and has a job the official estimate of the number of persons employed is understated and the jobless rate is overstated. There are an estimated 11 million undocumented aliens in the United States today.  

Additionally, since 2001 the CPS survey has been based on interviews involving approximately 60,000 households. Since then, the number of U.S. households has climbed from 108.2 million to 123.2 million. For that reason alone the standard error associated with every estimate is larger today than in 2001.

Reliability is compounded when the users of those estimates act as if any month-to-month change can be reported without regard to the standard error associated with that change.  One is reminded of the anecdote from Statistics 101 in which a foolhardy young man ignored the warning signs and drowned in a river of an average depth of 3 feet.

Edward J. O’Boyle is Senior Research Associate with Mayo Research Institute
www.mayoresearch.org     318-381-4002     edoboyle737@gmail.com