No test dummies

(Fortune Magazine) -- On a fact-finding mission to learn more about the auto business, Alan Mulally, the new CEO of Ford Motor (Charts, Fortune 500), traveled to an old drag strip in East Haddam, Conn., to see how Consumer Reports tests cars. Mulally spent four hours at the 327-acre site and got a model-by-model analysis of Ford's product line from CR's head of auto testing, David Champion.

Overall, Champion said CR rates Ford's models as the best made in Detroit, but he gave lower marks for the newest one, the Edge. "It is disappointing," Champion told Mulally. "The interior fit and finish is poor, the steering woolly, and the design of the tailgate makes it very hard to lift."

Top tester David Champion at Consumer Reports' specially built hill.

Testing the stopping distance of a Ford Mustang on a slick road.

More from Fortune

Will Mmmhops be a hit? NBA confirms L.A. Clippers sale to ex-Microsoft CEO Steve Ballmer FBI and SEC probe into Carl Icahn and golfer Phil Mickelson

FORTUNE 500 Current Issue Subscribe to Fortune

Map See more

Take a tour of the auto-testing facility

The new Salem witch trials Congress should blame itself, not imaginary 'price gougers,' for the run-up in gasoline prices, says Fortune's Alex Taylor. (Read the column.)

British-born and unfailingly polite, Champion delivers even his harshest opinions cheerfully, and Mulally, according to a spokesman, was thankful for the "unvarnished feedback." Other automakers aren't so grateful. When Consumer Reports discovered, after polling its readers, that Mercedes-Benz is the least reliable car brand sold in America, a Mercedes spokeswoman complained that "there's something wrong with the survey." But Champion was not to be swayed. In reply, he observed that Mercedes' quality is so bad that a brand-new Mercedes SUV is less reliable than a nine-year-old Lexus LS 400.

There aren't many people around the auto industry who can make such a remark without fear of reprisal, but Champion and his magazine are an unusual case. They run the nation's largest independent car-testing operation, backing up their experience at the test track with data from readers. They don't have to worry about angry manufacturers canceling their advertising in the magazine because Consumer Reports, published by Consumers Union, is funded almost entirely by its readers. And their judgment has been repeatedly upheld. Aggrieved automakers have occasionally sued, but CR has never lost a case or paid a penny in a settlement.

Consumer Reports treasures its reputation and apologizes faster than Don Imus when it makes a mistake. Earlier this year it retracted the results of crash tests conducted by an outside lab on infant car seats. After the lab reported that ten of the 12 seats tested provided poor protection in a collision, it turned out that it had run the side-impact tests at 70 miles per hour - about twice the rate intended. CR promptly disclosed the error and notified each of its subscribers.

Consumer Reports has a circulation of 4.3 million, and another 2.7 million readers pay up to $26 per year to get it online. Auto industry executives travel to East Haddam about once a month to find out what Champion and his team have to say. Besides Mulally, guests have included Daimler (Charts) chairman Dieter Zetsche and General Motors (Charts, Fortune 500) product guru Bob Lutz. When a BMW X-5 SUV threatened to roll over during a routine test, Champion discussed the problem with BMW engineers. After they replicated the test in Germany, they made a software change in the electronic stability control in production models.

To keep its tests free from influence, Consumer Reports takes unusual precautions. While other magazines (Fortune included) get test cars from manufacturers that may have been specially prepared, Consumer Reports buys its cars anonymously from local dealers. This year its tab will be nearly $3 million for 84 cars. After thorough testing, it resells the cars to employees at discounts of 20% to 30%. Sometimes it has to struggle to move a slow-selling car, just the way a dealer would. Champion found no takers for a $42,000 Jeep Commander and received just $26,943 for it in a trade-in.

Once it buys a car, CR puts 5,000 to 6,000 miles on it over as much as ten months, performing more than 50 different tests. It leaves little to chance. To measure mileage, for instance, it installs a fuel meter on the gas line rather than relying on the accuracy of a fuel pump or the vehicle's onboard computer. Then it performs three tests: one each to replicate city and highway driving, and a third mixed-use trip that lasts five hours. Instead of just blazing a trail through the woods to gauge the off-road capability of SUVs, CR paved a hill with concrete and studded it with 300 tons of boulders. To evaluate headlights, CR's engineers used to stay up late on moonless nights to perform their tests outside. They still do, but they recently erected a headlight test building so that they can work indoors too.

All the track work at CR is handled by nine experienced engineers. Most come from the auto industry; Champion himself spent 17 years as an engineer for Land Rover and Nissan. Champion would like to dispel the notion that his staff consists of lab-coated toaster testers whose own cars are indestructible Dodge Darts. The auto test team has broadened its coverage in recent years beyond sedans and minivans, and Champion and others can be seen happily steering Porsche 911s and Dodge Vipers around the handling course.

At the end of each day Champion hands off the test work to less experienced but more typical drivers. He assigns all the engineers and some other staff members different vehicles to drive home. Each contains a logbook with a checklist for comments. One item is how intuitive the car is to operate. Champion has his own opinion. "If it is a BMW with iDrive [a multifunction, single-knob control device], the answer is, Not at all," Champion says. "I've spent a week with a BMW and still not remembered how iDrive works."

The ultimate amateur test drivers are Consumer Reports' own magazine and website subscribers, 1.3 million of whom responded to the 2006 auto survey about their automobiles. They were asked to note any serious problems encountered in the past year and to categorize them in 17 trouble areas. CR requires at least 100 responses for each model before the results are published. The findings are used to generate a score for predicted reliability and another for owner satisfaction, which are published alongside the road-test numbers compiled by the auto team.

With all that data, navigating the results in the magazine can be trying. To win a check-rated recommendation, a vehicle must perform well in the East Haddam testing, rate average or better in reader surveys, and if crash-tested, provide good overall protection based on insurance industry and government studies. But while the cars are ranked according to their road-test scores, CR will usually give a recommendation only after it has accumulated a year of reliability information. The reason is that some brands like Volkswagen score well in road tests but fare poorly in reader surveys. So a high-scoring new car with no reliability data may be listed above a less proficient but reliability-rated older one - and still not get CR's check-rated seal of approval.

With safety, fuel economy, and reliability among CR's guiding principles, readers shouldn't be surprised that all ten of the cars identified as top picks in the 2007 auto issue come from Japanese automakers. Nor is it surprising that Consumer Reports is often accused of a pro-Japanese bias. But Japan doesn't get a free pass. In its April issue the magazine observes that "because a car is Japanese doesn't mean it's a great car," and Champion is happy to pass along a scathing critique of the Lexus GS450h, which he downgrades for squeaks, rattles, a tiny trunk, and a $65,000 pricetag.

In any event, second-guessing goes hand-in-hand with any ranking system. Just look at American Idol. The difference at Consumer Reports is that more than just bragging rights is involved in rating cars. Fortunately for the magazine, the toaster testers in East Haddam stand behind every call it makes. Dissenters will have to find their own bully pulpit.