(MONEY Magazine) – Humans have a remarkable ability to detect patterns. That's helped our species survive, enabling us to plant crops at the right time of year and evade wild animals. But when it comes to investing, this incessant search for patterns causes more heartache than anything else.

We see that value funds have stunk for years, so we dump them and pile into fashionable growth stocks like Intel and Cisco--right before they hit the skids.We buy a stock because some guy at a barbecue recommended it, and everything he talks about seems to go up--but this one plunges. We put every dime in stocks after hearing that they've trounced bonds forever--only to see bonds zoom past stocks this year.

Our incorrigible search for patterns leads us to assume that order exists where it often doesn't. Many of us believe, for example, that it's possible to foresee where the market is heading or whether a particular stock will continue to rise. In reality, these things are far more random and unpredictable than we like to admit.

Remarkably, scientists are now finding that this tendency to look for patterns is hardwired into the human brain. Psychologists have long known that if rats or pigeons knew what the Nasdaq is, they might be better investors than most humans are. That's because, in some ways, animals are better than people at predicting random events. If, for instance, you set up two lights in a laboratory and flash them in a random sequence, humans will persistently try to predict which of the two lights will flash next. Stranger still, they'll keep trying even when you tell them that the flashing of the lights is purely random. Let's say you flash a green light 80% of the time and a red one 20% of the time but keep the exact sequences random. (A run of 20 flashes could look something like this: GGGGRGGGGGGGRRGGGGGR.) In guessing which light will flash next, the best strategy is simply to predict green every time, since you stand an 80% chance of being right. That's what rats or pigeons generally do in a similar experiment that rewards them with a crumb of food whenever they correctly guess the next outcome.

But humans are apparently convinced that they're smart enough to predict each upcoming result even in a process they've been told is random. On average, this misguided confidence leads people to get the right answer in this experiment on only 68% of their tries. In other words, it's precisely our higher intelligence that leads us to score lower on this kind of task than rats and pigeons do.

The man with two brains

A team of researchers at Dartmouth College, led by psychology professor George Wolford, has been studying why it is that we think we can predict the unpredictable. Wolford's team ran light-flashing experiments on "split-brain patients"--people in whom the nerve connections between the hemispheres of the brain have been surgically severed as a treatment for epilepsy. Here's the group's key discovery, which was recently published in the Journal of Neuroscience: When the epileptics viewed a series of flashes that they could process only with the right side of their brains, they gradually learned to guess the most frequent option all the time, just as rats and pigeons do. But when the signals were flashed to the left side of their brains, the epileptics kept trying to forecast the exact sequence of flashes--sharply lowering the overall accuracy of their predictions.

Wolford's conclusion: "There appears to be a module in the left hemisphere of the brain that drives humans to search for patterns and to see causal relationships, even when none exist." His research partner, Michael Gazzaniga, has christened this part of the brain "the interpreter." Wolford explains: "The interpreter drives us to believe that 'I can figure this out.' That may well be a good thing when there is a pattern to the data and the pattern isn't overly complicated." However, he adds, "a constant search for explanations and patterns in random or complex data is not a good thing."

The dance of happenstance

Trouble is, the financial markets are almost--though not quite--as random as those flashing lights. On CNBC and countless websites, investment strategists and other so-called experts scan the momentary twitches of the market and predict what will happen next. Far more often than they're right, they're wrong--and the Dartmouth discovery about the interpreter in our brains helps explain why. These pundits are examining a chaotic storm of data and refusing to concede that they can't understand it. Instead, their interpreters drive them to believe they've identified patterns upon which they can base predictions about the future.

Meanwhile, the interpreters in our own brains impel us to take these seers more seriously than their track records deserve. As Berkeley economist Matthew Rabin has pointed out, just a couple of accurate predictions on CNBC can make an analyst seem like an ace, because viewers have no way to sample the analyst's entire (and probably mediocre) forecasting record. In the absence of a full sample, our interpreters take over and lead us to see the analyst's latest calls as part of a pattern of success.

The interpreter also helps explain what's called the gambler's fallacy--the belief that if, say, a coin has come up heads several times, then it's "due" to come up tails. (In fact, the odds that a coin will turn up tails are always 50%, no matter how many times in a row it's come up heads.) The gambler's fallacy is as common on Wall Street as hairballs under a couch: Some pundits will say emerging markets are sure to rebound because they've been doing badly for years, while others say tech stocks will crash because they've risen so much. In reality, the market makes mincemeat out of most of our predictions; apparent trends often foretell little about the future.

In its constant search for patterns, the interpreter also tricks investors into believing that hot performance streaks are sure to persist. Based on a few months of scorching returns, investors piled into Internet stocks late last year--and are now sitting on returns as cold as liquid nitrogen. What's happening here is simple: As soon as a pattern seems to emerge in the market, the interpreter in our brains sees it as part of a predictable trend--rather than a random happenstance that may never be repeated.

Finally, I think the Dartmouth research helps solve another puzzle. Even when we have only a small sample of our own performance at risky tasks--a few yanks on a one-armed bandit or a handful of big scores on tech stocks--we tend to decide either that we know what we're doing or that we're on a lucky streak. We almost never conclude that our success is the result of chance alone. Dutch psychologists Willem Wagenaar and Gideon Keren have found that professional gamblers, when accounting for their wins and losses, greatly overestimate the role of skill, attributing just 18% of the outcome of each bet to random chance.

Similarly, when a day-trader makes a fat profit off a stock after doing no research and owning it for only seconds, he's likely to conclude that he's an analytical genius or has an uncanny feel for the market. In truth, that profit is probably an accident--but his mind won't allow him to see things that way.

Mind over matter

So how can you keep your brain from giving you a garbled view of the investment world? You could disable your interpreter once and for all by having a neurosurgeon separate your brain's two hemispheres, and then by scrutinizing investment information in the leftmost part of your field of vision. That way, only the right half of your brain would be able to process investment data, and the interpreter would be shut down. However, it won't be easy talking a surgeon into carving your cranium open for this, and watching CNBC out of the far corner of your eye might be a pain. So here are some less drastic options.

Don't obsess. In one of his most startling findings, George Wolford of Dartmouth says people in his experiments earned higher scores when they were distracted with a "secondary task" like trying to recall a series of numbers they'd recently seen. In other words, interruptions improved their performance by preventing the interpreter in their brains from seeking spurious patterns in the data. Likewise, continually monitoring your results will probably make them worse--as you fool yourself into seeing trends that aren't there and trade too much as a result. If you're spending more than a few hours a month on investing, you're not only taking valuable time away from the rest of your life, but you're almost certainly hurting your returns.

Remember what's at stake. John Staddon, a professor of psychology at Duke, says rats or pigeons will generally bet on the option that has had the highest probability of success over time. But, notes Staddon, "humans will consistently do that only when the stakes are large and the consequences really matter." So you'll make better financial decisions if you convince yourself that there's no such thing as a small or casual investment. Just think of the thousands of dollars you could squander--and the blissful retirement you could jeopardize--with a few careless stock picks.

Track your forecasts. Whenever you've got a strong opinion about where a stock, or the market, is headed, jot it down and note the date. This will keep you from conveniently forgetting your failed forecasts and may provide you with a humbling reminder of your limitations as a soothsayer. And whenever some analyst seems to know what he's talking about, remember that pigs will fly before he'll ever release a full list of his past forecasts, including the bloopers.

Defy the chaos. Not everything about investing is chaotic, however; a few things really are predictable. On average, over time, investors who keep costs low (either through index funds or buy-and-hold stock portfolios) are mathematically certain to outperform in- vestors who trade too frequently or buy funds with high expenses. So before you focus on your returns--which are entirely unpredictable--make sure that your investments are not overpriced.

Diversification is another principle that defies chaos. Consider the danger of investing almost exclusively in tech stocks. Many investors who bet heavily on the sector in 1982--the last time it was this hot--loaded up on market darlings such as Alpha Microsystems, Commodore, Tandy, Vector Graphic and Wang Laboratories, which later tanked. If you diversify--by owning a wide range of U.S. and foreign stocks and bonds--you virtually eliminate the chance that a few duds like these will ruin your financial future. Broad diversification is still the best insurance against the risk of making an investment mistake. And there's nothing random about that.

Jason Zweig can be reached at fundamentalist@moneymail.com.