Can you outsmart the market?

(Money Magazine) -- As you try to figure out how to put your money to work in a market that veers from depression to mania, you're confronted with two opposing facts that lead to opposing investing strategies.

Fact No. 1: Outwitting the market is tough.

There's a good chance you've learned this the hard way. Remember when you were a tech stock genius, circa 1999? Or how you didn't really gain confidence in the bull market that started in 2002 until about 2005, only to be really confident just in time for the market's top in 2007?

If experience hasn't schooled you, the numbers can: Over the past 15 years, about 60% of the mutual funds that invest in blue-chip stocks failed to beat the S&P 500 index, the frequently cited proxy for "the market."

And that counts only the funds that performed well enough to survive those 15 years. Many didn't. Managers of bond and foreign-stock funds also have a rough time beating benchmarks.

Even during the recent bear market, the sort of moment when pros are supposed to prove their worth by sidestepping the weakest investments, the indexes trumped most managers, data from Standard & Poor's show.

And consider what happened to Bill Miller, the famed investor who beat the S&P 500 for 15 straight years: He walked his Legg Mason Value Trust fund right into the propellers of the financial crisis by betting on AIG and Freddie Mac.

The obvious conclusion: Stock picking, whether you do it yourself or pay a pro to do it for you, is a mug's game. You're better off buying and holding a cheap, diversified, and consistent index fund, which passively invests in the stocks listed on a broad market benchmark.

Fact No. 2: At times the market can act incredibly stupidly.

If you own, say, the Vanguard S&P 500 index fund, you can take comfort in knowing that it performed relatively well in a disastrous 2008. But surely the more salient fact is that it lost 37% of your money that year and has so far earned back just 40% of that loss.

Twice in less than a decade, in fact, investors who did the smart thing and "owned the market" have seen huge chunks of their wealth destroyed by bursting asset bubbles, and it's not as if there were no warnings. Dotcom excesses were a punch line long before tech flamed out. And the media were churning out headlines about a real estate bubble in 2005.

The obvious conclusion: Passive investing - whether through indexing or a buy-and-hold strategy with other kinds of funds - is for suckers. You need a strategy that shields you from the consequences of irrational exuberance.

There's just one problem: fact No. 1.

We're going to revisit the theoretical and practical cases for passive investing. They're still very strong. But two increasingly popular ideas that suggest a more-active passive approach deserve a closer look: The first is that traditional index funds have a design flaw that leaves you vulnerable to bubbles, so maybe it's time to employ a better mousetrap. The second is that there are simple, objective measures that can tell you when stocks as a class of assets are simply too risky.

After considering those claims, we'll spell out three strategies for navigating a market that sometimes loses its mind - without your losing your own trying to second-guess it.

The case for index funds and the case for buy and hold aren't necessarily identical, but they share a common intellectual root: the idea that markets are what economists call "efficient."

The stock market is made up of lots of buyers and sellers who have access to the same information. If you think Microsoft is cheap at $29 a share, you're betting that you know something the trader selling it to you from Fidelity or Goldman Sachs or some hedge fund hasn't yet figured out. That's a tough bet to win more than half the time.

The efficient-markets hypothesis also says that if some pattern or formula emerges that leads to higher returns with no extra risk, it will disappear quickly as investors swoop in to exploit it. Result: Stock prices move around randomly. And if you want to get a higher return than the market, you have to take on more risk.

Also tilting the scale toward indexing is what Vanguard founder John Bogle once called the "cost matters" hypothesis. Since actively managed funds slice an average of 1.4% a year from returns in fees (plus high unseen trading costs), it's hard for them to beat an index fund charging less than 0.2%.

But what if markets aren't so smart after all? As Time columnist Justin Fox explains in his book "The Myth of the Rational Market," the idea of an efficient market has come in for plenty of refinements, exceptions, and lately outright attack.

Economists Eugene Fama (the pioneer of efficiency theory) and Kenneth French have shown that there are a few patterns in stock returns: Small stocks beat big ones, and stocks with low valuations, known as value stocks, beat the high-priced shares of fast-growing companies.

Meanwhile, behavioral economics has identified persistent psychological biases that cause investors to make substantial errors in pricing assets. For example, Yale economist Robert Shiller and Wellesley's Karl Case surveyed homebuyers in cities experiencing a real estate bubble and found that many expected houses to appreciate a wildly unrealistic 10% per year for 10 years.

Equity investors are prone to a similar backward-looking optimism: How often did you hear in the '90s that stocks would earn a consistent double-digit return and would always beat bonds?

Then there are those bubbles. By some accounts, they aren't bolts from the blue but are recurring features of markets, particularly lightly regulated ones. The late - and newly fashionable - economist Hyman Minsky described a cycle in which stable markets eventually create their own instability. As investors get used to steady returns, they start borrowing to amp up their gains (think Lehman Brothers and its 30-to-1 leverage) and buying increasingly dubious stuff (think subprime mortgages and CDOs).

It seems you ought to be able to insulate against such bursts of insanity, even if you concede that trying to find the next Bill Miller, and then knowing when to dump him, isn't the answer. These two ideas are gaining currency:

The "better" index. Traditional indexes are by market capitalization, which means that the more the price of a stock rises relative to other equities, the bigger the slice of the index it becomes.

In the tech bubble, owning a broad index fund would have had you betting bigtime on the bubbliest stocks. In 1997 technology shares were just 15% of the assets in the Vanguard 500 index fund. By 1999 they were about one-third.

"Traditional cap-weighted indexes overweight the overvalued and underweight the undervalued," says money manager Rob Arnott. He's developed an alternative "fundamental" index, which owns a cross section of the market and divides up its holdings by measures of economic values, such as dividends and sales, rather than the market's weighting.

Arnott has licensed mutual and exchange-traded funds - index funds that you buy and sell like stocks - to track his creations. Wharton professor Jeremy Siegel, author of "Stocks for the Long Run," is an adviser to an investment firm implementing a similar strategy. The claim is that the funds naturally avoid bubbles. Arnott's FTSE RAFI 1000 index would have beaten the S&P 500 by five percentage points a year since 1999, assuming it had actually existed back then.

The caveat: The new indexes lean toward value stocks, which do seem to have a performance edge. The not-so-academic debate is whether that's because the market initially underprices them or because they're riskier. In other words, was the extra return a free lunch, a reward for sticking your neck out, or something in between?

It's certainly the case that tilting toward value isn't a guarantee you'll miss every crash. Arnott's index had as much as the S&P 500 in financial stocks just before the banking crisis hit, and it lost a bit more than the established index did in 2008.

Yes, the neo-indexers have a smart investment case and a lot of historical data to back them up. But if growth turns out to beat value over the next 30 years, they may not deliver what an ordinary index fund can all but promise: the market's return, and better performance than most other funds.

The valuation test. Even if you invest only via broad index funds, you still must decide how much of your assets you want to hold in equities overall. In general, that should be determined by how much risk you want to take. But if you check some basic market statistics, you can easily see that stocks are more expensive at some times than they are at others. It seems like common sense to trim back when an asset class is dear, and hold a little extra when it's cheap.

Andrew Smithers, head of the London investment research firm Smithers & Co., argues that the 10-year price/ earnings ratio - sometimes known as the Shiller P/E, after the Yale economist - is one way to reliably gauge how expensive stocks are. It looks at what the market is willing to pay for stocks based on average earnings over the past decade in order to smooth out short-term bumps. When the long-term P/E is high, as you can see in the graphic below, future returns have tended to be low. (You can find this P/E at irrationalexuberance.com.)

The question is, Why should this keep happening? After all, Shiller laid out his P/E argument in a bestselling book a decade ago. Savvy investors ought to have tried to take advantage, and the effect should have gone away.

But Smithers says that isn't likely. Rallies don't change course on any kind of reliable schedule, and even professional investors find it hard to wait out an overpriced market. In the years leading up to the 2000 crash, refusing to buy booming tech stocks cost more than one fund manager his job. "It's a wild and woolly market in which career risk dominates," says Jeremy Grantham of the Boston investment firm GMO.

The caveat: Princeton economist Burton Malkiel, author of "A Random Walk Down Wall Street," the book that popularized efficient-market theory, concedes that market P/Es do seem to have had some predictive power. "But it's loose," he adds. And he doesn't see any evidence that individual investors will do better than the pros in choosing when to buy and sell.

Most people who time the market do a terrible job at it: Cash flows into stock funds hit peaks in early 2000 and ebbed as the market hit bottom. This kind of activity is incredibly costly. From 1989 through 2008, the S&P 500 gained a bit more than 8% a year, but the average equity fund investor earned less than 2% thanks to lousy timing, according to the research firm Dalbar.

Okay, you're saying, but now I know that following the herd is a mistake, so I'll change my ways. Unfortunately, that's difficult to do in real time, even if in hindsight the correct course is clear. "This is a problem behavioral finance helps us understand," says economist French. "We're all overconfident, and one of the sources of that is the simplicity we see when we look backwards."

Holding out during a long boom, even in an "obvious" bubble, can be excruciating. The Shiller P/E went above 20 for the first time in decades in 1992. Alan Greenspan coined the term "irrational exuberance" in 1996. By 1999, when P/Es topped 40, your neighbor who had half his portfolio in Cisco didn't look like a lemming; he looked like a guy who drove a nicer car than yours and whose kids went to private school.

When a rally keeps on going, bearish investors decide that they can't fight the tape, as the Wall Street saying goes, and buy back in at the worst possible moment. "When you time, you have to be right twice: on the sale and on the purchase," says Malkiel.

So the stupid market is devilishly hard to outsmart. What do you do about that?

Learn the right lesson from a burst bubble. Many market timers argue that the crashes of 2000 and 2008 show that stock prices flash signals warning of steep losses ahead. But here's a simpler and easy-to-follow lesson: The market can deliver a steep loss at any time. And those losses can mean poor results even over a fairly long run. Right now, U.S. stock returns are still negative over the past 10 years. Japan's market (not counting dividends) is about where it was in 1984.

So even if you remain a believer in buy and hold, one moral of the past decade is that you may want to buy and hold less in U.S. equities than conventional wisdom dictated in the 1990s. Stay diversified in cash, bonds, and overseas stocks.

And above all, save enough so that you don't have to depend on high returns for equities. Even the stern anti-timer Malkiel says that the dramatic rise in valuation over the past several decades suggests lower returns in the future than the average 9.6% of the past.

Rebalance. Malkiel recommends rebalancing your portfolio once a year. That keeps your risk exposure where you want it, but it also means you're making small bets against market sentiment. Say you decide that the best asset mix for you is 50% in stocks and 50% in bonds. If stocks have a huge rally and go up 40%, while bonds rise by 5%, your portfolio will now be 57% in stocks and just 43% in bonds. To get back to even, you'll have to sell the stocks the market loves at the moment and buy bonds.

Conversely, if the equity market falls, you'll be forcing yourself to buy stocks just when the herd is shunning them. This automated, unemotional contrarian strategy works against pure efficient-market theory. But it also seems to work, period.

Not that it's foolproof. As the financial adviser and writer William Bernstein says, an investor who rebalanced into Japanese stocks starting in 1990 would have been sorry.

If you time, keep it limited, rare, and cheap. We're not all built psychologically for the laissez-faire approach to investing. Some of us like to test our predictive acumen; others can't stand the idea that they should do absolutely nothing to prevent a large loss. But the more you trade, the more likely you are to make a wealth-destroying mistake.

So set parameters. First, because there's a good chance you are wrong, never go all the way into stocks or all the way into cash. Baylor University finance professor William Reichenstein says you might vary your stock allocation by no more than 10 percentage points in either direction. If you generally have 50% in stocks, you'd go as high as 60% when they looked cheap, and as low as 40% when they seemed very expensive. Don't do so based on the news of the day; change your allocation only in response to significant shifts in valuation.

Smithers thinks that investors far from retirement can usually ignore even these, with a few exceptions like 1999. And, he advises, you need to make this decision yourself. Money managers can't be relied on to buck the trend. They're part of the reason markets get overpriced.