Today was a pretty big day for U.S. manufacturing data, with the release of an advance August survey by the respected research firm Markit.com, and of the Philadelphia Federal Reserve bank’s closely watched gauge of manufacturing activity in some of the middle Atlantic states. There’s lots more monthly manufacturing data to come in the next few days, so it’s important to understand that these types of reports in particular can present a seriously misleading picture of manufacturing’s health. They’re by no means completely worthless. But their results need to be taken with a big grain of salt.
Here’s why: The Markit reports – as well as the surveys released each month by various regional Federal Reserve banks and the monthly results of the Institute for Supply Management – present the results of what are called diffusion indices. That is to say, they show the percentage of the firms surveyed that report a better performance in whatever is being measured over a certain timeframe (e.g., overall business conditions, employment, new orders), a worse performance, or about the same performance.
The big problem with diffusion indices: They usually suffer what statisticians call “survivorship bias.” They only survey the performance of companies that are in existence during the particular time period being examined. Because they don’t, therefore, say anything about how the number of companies in existence has changed over time, much less about the actual production or employment levels of these companies, they create a picture of overall manufacturing activity in the location in question that’s at best highly incomplete.
Before I was even aware of the formal concept of survivorship bias, I learned about it in practice from talking with manufacturing executives who belonged to my former organization, the U.S. Business and Industry Council. Especially as the current recovery took hold, I would regularly ask them how their own companies were faring. The response I often heard? “Never (or rarely) better!”
So I then asked them why they remained in the Council, which has long warned of major competitive weaknesses in domestic industry. I expected most to emphasize continuing uncertainty about the future, and a reluctance to read too much into the present – especially considering that virtually none was adequately prepared for the Great Recession. And uncertainty was certainly mentioned.
Even more often, however, these business owners and managers explained that they were benefitting significantly from the demise of weaker domestic rivals. Because so many so many firms that were perfectly viable and competitive during normal times were suddenly drowned by the worst U.S. downturn since the Great Depression, the remaining companies were able to win the business formerly held by the losers once growth and even stability began to return.
As the USBIC members put it, they were taking advantage of pie that was growing much more slowly – if it was growing at all – because so much more of it was up for grabs than ever before. But they remained deeply concerned about the future overall size of the pie, which they correctly believed was crucial both to a return to genuine overall U.S. economic health and to national security.
A February article of mine on the Institute for Supply Management’s monthly manufacturing index showed how its survivorship bias was producing results often sharply at odds with the national manufacturing production figures put out by the Federal Reserve, which are far from perfect, but which do attempt to measure actual industrial output and its rise and fall. I haven’t tracked the relationship between the Markit and regional Fed manufacturing results on the one hand, and industrial production on the other, but it’s entirely reasonable to suppose that a notable gap exists there, too.
These diffusion index surveys can contain some useful information about the momentum of domestic manufacturing. And the regional Feds regularly ask manufacturers in their geographic districts interesting and important questions about specific policy and other challenges and opportunities they face. (This morning’s Philly Fed report included some fascinating information about how manufacturers in that region are reacting to Obamacare.) That’s why I follow and report on them. But understanding the true state of domestic manufacturing requires looking at a much wider range of material. Relying solely or even largely on diffusion index surveys is one of those tempting shortcuts that can lead you seriously astray.
RealityChek 