(Fortune Magazine) -- Ever since American Funds launched its first portfolio back in 1934, the company has taken an unlikely path to success. In an industry that seems to run on hype as much as performance, American Funds has shunned the spotlight. It never advertises to individuals, and its fund managers rarely grant interviews.

So it was no surprise that the Los Angeles-based firm didn't make a fuss when earlier this year it became the nation's No. 1 manager of stock and bond funds. American quietly took the title from Vanguard in April after bumping Fidelity from the No. 2 perch in June 2005.

As of May 31, the investment house had $844 billion in its stock and bond funds, vs. $842 billion for Vanguard and $774 billion for Fidelity, according to Financial Research Corp.

But just as American settles into its new throne, there are signs of trouble in the kingdom. Investment returns at some of its most popular funds are slipping. And while American seems to be aware of the problem - it has begun shuffling fund managers as part of a restructuring to deal with the firm's girth - it refuses to do the single best thing to protect existing shareholders: close ballooning funds to new investors.

"They've gotten too big," blasts former Vanguard chairman John Bogle, a longtime critic of the fund industry who now runs the Bogle Financial Markets Research Center. "I'd expect below-average performance in the years ahead."

The first thing to understand about American Funds is that it doesn't follow the industry playbook. The company offers just 29 mutual funds, compared with more than 200 at Fidelity. And unlike most of its competitors, which sell to individuals directly, American markets its funds only through brokers and financial advisors. Of course, that's been a key to its remarkable growth in recent years as investors sought professional help following the 2000 - 2002 bear market.

What's most distinctive, however, is its management style. At most investment shops, funds are run either by a single manager or by a team that makes decisions by committee. But American - or more precisely, Capital Research & Management, the firm that oversees American Funds - has its own way.

It uses a hybrid approach first developed in the late 1950s by Jon Lovelace, son of firm founder Jonathan Bell Lovelace. The typical American fund has three to nine "portfolio counselors" who operate autonomously. They share ideas, but each makes his or her own buy and sell decisions with a portion of the fund's assets. Each fund, in effect, is really a collection of several small portfolios.

In theory, this structure allows American to accommodate asset inflows better than most fund families. As money comes in, American can parcel it out to a number of managers or even add another portfolio counselor if flows get really heavy. "Our approach is scalable," says Chuck Freadhoff, the firm's spokesman.

But the scale has gotten immense. The fund complex has attracted some $300 billion into its stock and bond funds in the past five years - more than 30% of all inflows for the entire industry. American now has seven of the ten largest mutual funds, including the biggest, Growth Fund of America, with $144 billion in assets.

And it may be just a coincidence, but performance has been slipping as the cash pours in. American's long-term results are still stellar. On average, American funds rank in the top-20th percentile of their categories based on ten-year returns, according to Morningstar.

But over the past five years - the period in which American experienced the great flood of new money - the average fund slips to the 37th percentile of its category. It drops to the 40th percentile over three years. Simply put, American seems to be slouching toward mediocrity.

Investment Company of America and Washington Mutual Investors, the firm's third- and fourth-largest funds, each with about $80 billion in assets, are both slumping hard. Washington Mutual's ranking falls from the 25th percentile over ten years to the 53rd percentile over five years to the 78th percentile over three years.

American's Freadhoff insists that the deluge of dollars has not hampered performance. "We don't see any correlation between assets and returns," he says. He adds that the firm's results are partially distorted because it has increasingly introduced additional share classes that have higher expenses, hurting its relative rankings.

That may be, but Capital Research has quietly been making changes to cope with all the new cash. For years the firm's portfolio managers and analysts have met in small groups each week to discuss investment ideas.

The meetings typically included ten to 12 people, but it became increasingly difficult to keep them that small as the firm - which now numbers more than 150 investment professionals - grew in size. Starting a few years ago, the typical meeting had 20 to 30 people. "It was harder and harder to get into small groups," says Drew Taylor, a vice president at Capital Research. "Meetings weren't as dynamic and deep."

As a solution, the firm formally divided the management teams of its funds earlier this year into two units, Capital World Investors and Capital Research Global Investors. The new structure means fund managers and analysts can meet in small groups more easily.

As Taylor sees it, the move was "a natural step in the evolution of our organization" and was done "to manage the money for shareholders as effectively as we can." It has also prompted speculation that Capital Research could be preparing for a more drastic step: splitting the firm into two companies. "It's something we've discussed," says Taylor.

Regardless, the new arrangement is bound to have big implications. For one, the firm has shuffled several top fund managers. Notably, it has dropped Gordon Crawford from the roster at Fundamental Investors.

Crawford is the most recognizable of the firm's managers, thanks to his role in pushing Steve Case off the board of AOL Time Warner (Charts) (parent of Fortune's publisher) and his close relationships with media and entertainment industry executives such as Ted Turner. Crawford will remain at the firm and on his other funds, including Growth Fund of America, while Brady Enright and Ronald Morrow replace him at Fundamental Investors.

Second, the two groups won't share investment ideas. The obvious question, says Morningstar analyst Paul Herbert, is whether any previously fruitful relationships among sets of managers and analysts have been severed. Capital Research's Taylor insists they haven't and maintains that the new setup will make interactions among the analysts and managers more productive.

One thing American has yet to do is shut a fund to new investors - something fund watchers at Morningstar and elsewhere have been suggesting for a couple of years. Typically, fund companies close funds when they get bloated.

Even Fidelity, which has long been criticized for leaving funds open too long, shut Contrafund earlier this year after several years of lackluster returns. (It also closed Magellan several years ago, but not before performance dropped off.) Yet all of American's funds remain open, and the company says it has no plans to close any of them. "They're a marketing machine," charges Bogle.

"We take the question of size very seriously," responds Freadhoff. "But at this point, we don't believe size has reached a point that our shareholders are disadvantaged by it."

American Funds has a long, proud history of proving skeptics wrong. Still, investors may want to remember the words that one wise investor shared a decade ago: "A fat wallet is the enemy of superior investment results." Who said that? Warren Buffett.

____________________________

The other story at Time Warner.

Starbucks is overheating.