(FORTUNE Magazine) – You thought you understood them fairly well. You read their answers on market-research forms and peered at them in focus groups through one-way mirrors. When they said they were "satisfied" or "very satisfied" on survey questionnaires, you went to bed happy at night. Why then did they defect, one by one, to your competitors? Why, oh why, did your customers hand you such a nasty surprise?

If you are asking yourself these kinds of questions, you are part of a crowd. Unable to fathom what really motivates their customers, more and more companies are failing their central mission of pleasing the people who matter the most. Scores in the annual American customer satisfaction index, which measures how happy people are with the products and services they purchase, tumbled 2% overall from last year. Also, the index, which follows this story on page 108, shows that only a handful of the more than 200 companies surveyed rated substantially higher with customers than in 1995.

It seems that traditional market research no longer works with an increasingly diverse and fickle customer base. The methods marketers have relied on for decades, perfunctory written and phone surveys, simply skim the surface of the shifting customer zeitgeist. Says Larry Keeley, president of the Doblin Group, a Chicago-based design and consulting firm: "These surveys are nothing more than tracking studies, designed to measure if customers are a little more or a little less pleased with you than they were last year."

Even standard focus groups may have outlived their usefulness. People often lie about their real feelings if the rest of the group disagrees with them, and their memories of their encounter with your company may not be fresh. Then there's that one-way glass through which companies watch their customers. "It's a nasty sort of voyeurism," says Michael Barry, vice president of GVO, a product definition and development company in Palo Alto. "The people behind the mirror start laughing about how stupid the people in the focus group are and stop listening to what they actually have to say."

Surely there must be a better way. Wise heads in the arcane world of customer research are onto something called storytelling. These folks advocate far more probing research than ever before, advising companies to elicit real-life stories from customers about how they behave and what they truly feel. The notion may seem like a leap into the unknown, but at least some companies have discovered that these storytelling methods work. Great service and, ultimately, breakthrough products have resulted. Kimberly-Clark built a new $500 million diaper market using in-depth customer research. At Intuit, storytelling customers helped its software writers revolutionize the way people all over the U.S. handle their money. Clothing maker Patagonia, soliciting true tales about how customers live and use their gear, manages to keep its product line ahead of the demand curve.

At the heart of this new brand of customer research is a search for subtle insights into human behavior--not only the emotion-laden anecdotes but also unspoken impulses. Just think, for example, of the last time you made eye contact with an attractive stranger. A whole range of feelings washed over you, and at that moment it would be hard to argue with the hoary notion that at least 80% of all human communication is nonverbal.

Getting at that other 80%, where much of what people truly think and feel is buried, can be tricky. But Harvard business school professor Gerald Zaltman, who has spent the past five years attempting to milk the real truth out of people, may have found a door to people's inner dictates. In an ordinary-looking office building on the Harvard campus, Zaltman has set up a working laboratory of metaphors. He broadly defines the term: "A metaphor is a definition of one thing in terms of another, and people can use them to represent thoughts that are tacit, implicit, and unspoken."

Zaltman draws metaphors out of consumers by asking them to spend a few weeks thinking about how they would visually represent their experiences with a company. He asks them to cut pictures from magazines that somehow convey those experiences. Then a couple dozen of his survey subjects come to his lab and spend a few hours telling stories about all of the images they chose and the connections between them.

While Zaltman has used this technique to glean insights on everything from cars to candy bars, it can be especially revealing when it's applied to products that customers have particularly strong feelings about. Take pantyhose. "Women in focus groups have always said that they wear them because they have to, and they hate it," says Glenda Green, a marketing research manager at Du Pont, which supplies the raw material for many pantyhose manufacturers. "We didn't think we had a completely accurate picture of their feelings, but we hadn't come up with a good way to test them."

Du Pont turned to Zaltman, and a parade of women soon descended upon the metaphor lab with their magazine clippings. Someone brought a picture of a spilled ice-cream sundae, capturing the rage she feels when she spots a run in her hose. Another arrived with a picture of a beautiful woman with baskets of fruit. Other photos depicted a Mercedes and Queen Elizabeth. "As we kept probing into the emotions behind the choice of these photos, the women finally began admitting that hose made them feel sensual, sexy, and more attractive to men," says Green. "There's no way anyone would admit that in a focus group."

There are still some intriguing mysteries here. It's clearly an old-fashioned notion, this idea of women binding themselves in nylon to impress the guys. And Green admits that consumers have, at best, a like-hate relationship with the product. Stocking manufacturers have begun to use the research Zaltman provided, tailoring ads to appeal less to women's executive personas and more toward their sexy, cocktail-dress side. One company has also enclosed inspirational quotes in each package of hose, while another hired a numerologist to rename its offerings.

Kimberly-Clark scraped away at similarly deep-seated emotions when it attempted to reinvent the diaper business some years back. Every time another kid graduated to underpants, the company lost a customer for life. Tough diapering challenges faced the parents holding the toddlers over the toilet too. "They knew about diapers because diapers were what they were trying to get out of, and they knew underwear because that's what they wanted to get into," recalls Dudley Lehman, who heads the diaper business for Kimberly-Clark. "They could express all their problems, but they couldn't articulate what they needed to solve them."

For years, a number of people at the company had been toying with the idea of producing training pants. This transition product would look like underwear and fit like it too, yet still keep accidents on the inside. But would parents buy it? Kimberly-Clark assigned a small team, many of whom were deep in the throes of toilet-training their own children, to the task of probing the market for the answer. When they sat down in the homes of their customers to hear real-life stories, they discovered a few things. "The stress in toilet training came from parents' feelings of failure, and you'd never get people to admit that in a focus group," says GVO's Michael Barry, who has done consulting work for Kimberly-Clark. "The worst thing for a parent is to have someone gape in horror and then ask, 'Oh, is your child still in diapers?!'"

After listening to this embarrassing story over and over again, the team realized that these parents viewed diapers as clothing that signified a particular stage of child development, not as waste-disposal fodder. Clothing has meaning, and the message diapers sent to the toilet trainees was disastrous. "It was like taking a dog to a spot where it had gone to the bathroom all its life and then telling him that he couldn't do it anymore," says Lehman. So the company rolled out its Huggies Pull-Ups training pants in limited areas at first, then nationally in 1991. By the time the competition caught up, the company was selling $400 million worth of Pull-Ups a year.

At Intuit, a software company born of the laments of ordinary people about what a pain in the posterior it is to balance a checkbook, the key to success has been relieving customers' anxiety about money and computers. "People don't buy technology," says Intuit Chairman Scott Cook. "They buy products that improve their lives."

One key to improving your life is to simplify. If customers have to call in an army of 12-year-olds to figure out how to add new numbers to a personal-finance program, they'll turn the computer off and write it in longhand. "For years, we've been hearing people tell us, 'The computer should just tell me how to do it,' " explains Shannon Pekary, an Intuit software engineer. "We decided to take them extremely literally, and now there's audio that gives instructions at each step of the way."

When it comes to story gathering at Intuit, noncustomers play a large role. Scott Cook tested one of his first beta products on members of the Palo Alto Junior League, many of whom had never used a computer. Today people who don't use Intuit products make up 80% of the research pool at the company's usability labs. Employees often tag along to their homes for a guided tour of their bill files and check drawers, just to see what kinds of financial systems people build for themselves when left to their own devices.

Follow Me Home, in fact, is the formal name for one major Intuit research method. The plan, a kind of formalized stalking, has company marketers and engineers peer over customers' shoulders to see how they fare in their attempts to get Intuit software installed in their home computers. Lately, researchers have taken to leaving microcassette recorders behind, so that anytime customers get irritated they can simply press Record and file their complaint.

Slavish attention to customer stories has led to many product improvements. Some come as something of a shock, like the realization that small-business owners were using Quicken home-finance software to keep their books. Apparently all the accounting software at the time forced users to learn double-entry bookkeeping. Odd requirement, Intuit's marketers concluded, given that it's the owners themselves--not trained accountants--who keep the books most of the time. Who among these entrepreneurs had the time or the inclination to learn the difference between debits and credits? From that realization, QuickBooks was born, and today it's used in thousands of businesses around the world.

In the early Nineties, the company added a personal inventory feature to Quicken Deluxe so users could determine whether their possessions were properly insured. Intuit was thinking of killing the feature until a man in California wrote in to tell the company a story. He had bought Quicken a few days before brushfires began to break out near his house. As they got closer, he realized he had a crisis on his hands. He shoved Quicken into his laptop and began running around the house trying to document everything in it. When the fires got closer, he fled with only his floppy in hand. His house was spared, and so too was the inventory feature. Not many people are likely to need it under such dire circumstances, but this man's tale provided valuable input into the feature's potential worth.

Nobody's perfect, of course, and Intuit fell flat on its face in early 1995 when customers called to tell stories about how bugs in its TurboTax product were making a mess of their income tax returns. Intuit's technical support reps didn't take the problems seriously enough, the company continued to ship the software, and it ended up footing the bill from the IRS for its customers' interest and penalties. Two years later, having stepped up reps' attention to customer stories a few notches more, it's hard to imagine a similar recurrence.

At Patagonia, an outdoor-sports apparel company in Ventura, California, customer storytellers surf at the "Point," right outside the front door of headquarters. Founder Yvon Chouinard, who spends at least six months a year at the ends of the earth testing his company's gear himself, has made a point of hiring several of these customers so they could share their war stories in-house. He refers to them affectionately as his "dirtbags," people who spend so much time outside that it shows under their fingernails.

The dirtbags help drive product development. A few years ago, for instance, Bob McDougall, a resident water-sports expert, flipped his craft on a stretch of rapids in British Columbia. Everything in the boat, including his shoes, went rushing down the river. He managed to swim to the riverbank and then realized he was going to have to climb out of the canyon and hike eight miles back to his car--barefoot. He made it back in one piece, driven mostly by the urge to design the ultimate canyon-climbing, mountain-hiking, waterproof river shoe.

Patagonians collect such war stories from far-flung customers as well, and use them as a marketing tool. Many of their wares are sold through a biannual catalogue that is unique among its peers. Instead of spending millions to shoot glossy spreads of unthinkably beautiful models, the company relies on its customers to pose while wearing Patagonia duds in exotic locales. This pictorial road map of customer adventures makes for great marketing, but it has another role as well. The fact that it places customers' stories front and center proves that their opinions and experiences are valued, and they respond in droves. "We have trained them to believe that we are serious about responding to their feedback and improving our products," says Randy Harward, the company's director of quality.

By now, you may be wondering how to introduce the concept of customer storytelling at your company. One trick: Avoid the abstract concept, and home in on the practical advantages of knowing what makes people tick. "Managers get too caught up with explaining the brilliance of their own theories," says Jim Champy, the head of consulting at Perot Systems. "You have to start with the proof, not the theory, and the proof comes from stories. Tell the boss that you want to share a story or two about how you want your customers to experience the company, and you'll be surprised how quickly he'll perk up."

Here's another selling point: Unlike traditional market research, it may not cost a lot to gather the right kind of stories from your customers. The Follow Me Home program at Intuit is hardly the company's largest research expense. GVO managers believe they can deliver new insight with as few as 25 interviews--if they're done right. Gerald Zaltman claims that he stops hearing much that's new after 15 or 20 people have passed through his metaphor lab. But no matter what it costs, it's hard to place a value on a single great idea. After all, a truly incredible story is ultimately priceless.

REPORTER ASSOCIATE Joyce E. Davis