The Baffling World of Automobile Surveys

New York Times Blogs

October 31, 2012

When Consumer Reports issued its 2012 Annual Auto Reliability Survey on Monday, the big news was that Ford had slid in the rankings, coming to rest in 27th place out of 28 brands included in the analysis.

That news also placed a focus on Jaguar, which was at the bottom of the list for the second consecutive year. It all seemed so clear—unless you also were familiar with several studies done by market-research firm J.D. Power & Associates. Then you might say: “Wait a minute. I thought Jaguar had done well recently.”

In fact, it had done well in one of those studies. The J.D. Power 2012 Initial Quality Study put Jaguar in the No. 2 spot—tied with Porsche—just behind the first-place Lexus.

If that seems puzzling in light of the Consumer Reports study, it also can seem strange that Jaguar was 28th out of 32 brands in a different J.D. Power study, the 2012 Vehicle Dependability Study.

Lest anyone think Jaguar is a fluke, consider Toyota’s youth brand, Scion, whose results were the opposite of the Jaguar results. Scion was ranked first in Consumer Reports’ reliability survey but was 26th in J.D. Power’s Initial Quality Study.

Different approaches

The differences are due to the fact that each survey looks at different factors, according to information from officials at the two companies. The studies ask different questions and vary in whether they include new models, redesigned models or old models.

Monday’s Consumer Reports study predicts the reliability of 2013 vehicles—those that are in the dealerships right now. “What we are looking at is all of the vehicles that are on sale now and predicting the reliability of those vehicles,” said Jake Fisher, director of automotive testing. “We want to tell you what cars are reliable now if you went out and bought a car today.”

It shows how well new cars that are on sale will hold up based on the most recent three model years of data, provided the model has not been redesigned for 2013.

J.D. Power’s Initial Quality Study is a snapshot taken at a different period of time than Consumer Reports’ study. And the J.D. Power study looks at initial quality, not dependability.

The Initial Quality Study looks at problems owners have with their new vehicles in the first 90 days of ownership. It broadly defines problems as things that have “caused a complete breakdown or malfunction, or where controls or features may work as designed but are difficult to use or understand.”

It gives equal weight to problems that are defects, which can be fixed, and elements that are working as intended yet rankle owners; the latter cannot be fixed immediately, if ever.

The Consumer Reports survey asks respondents whether they have had any problems with their cars “in the past 12 months that you considered serious because of cost, failure, safety or downtime.”

J.D. Power’s Vehicle Dependability Study, meanwhile, measures problems experienced by the original owners of three-year-old vehicles. It can be valuable to consumers in the market for a used car. The 2012 dependability study looks at 2009 vehicles and problems their original owners had with those vehicles in the previous 12 months.

In that regard alone, the J.D. Power dependability study is different from the Consumer Reports predicted-reliability study. The J.D. Power study looks at one model year over a three-year period, while the Consumer Reports survey looks at three model years. In the case of a 2013 model, for example, the magazine looks at the 2012, 2011 and 2010 model years.

Quality vs. dependability

As for the J.D. Power studies of initial quality and vehicle dependability, “We see a strong correlation between how a model performs in IQS and how they perform in VDS,” said Raffi Festekjian, director of automotive research at J.D. Power & Associates. Those that perform well coming out of the gate are more likely to sustain that quality and dependability over those next three years than those that begin with poor initial quality, he added.

The key in comparing these two J.D. Power studies is to look at different years. A consumer would need to compare the 2012 Vehicle Dependability Study to the 2009 Initial Quality Study to see how the 2009 models held up over three years.

While there can be differences between the Consumer Reports and J.D. Power studies, there also can be similarities. Ford is an example. The automaker’s brand fell last year and this year in the Consumer Reports reliability study. This year, it’s 27th out of 28 brands. It also fell in the J.D. Power initial quality study to 27th out of 34 brands.

According to past and current interviews with researchers at both companies, Ford fell in both surveys for the same reasons: problems with the MyFord/MyLincoln Touch control system and the rough-shifting PowerShift automatic transmission. To see how the 2012 Ford models fare in the J.D. Power vehicle dependability study, we’ll need to wait until the 2015 study comes out.

As for consumers trying to figure out how to use this sometimes conflicting, sometimes similar information, the comforting ideal would be to find a brand or model that does well in each of the studies—although that might be a challenge.