New York (CNN/Money) - During the past three years, real estate has been a shelter in the storm.

Since 2001, home prices have gained about 6.3 percent annually, according to the National Association of Realtors. And in dozens of hot markets, from San Francisco to Providence, RI to Topeka, KS, homeowners have seen double-digit price increases over the past year.

Next to the seeming flimsiness of stocks, real estate looks rock solid. For the past 40 years, home sales prices have outpaced inflation by one or two percentage points per year, and there has never been a national decline in real estate values.

But that's just part of the picture. When you drill down to local markets, instead of steady rises, you may find vertiginous spikes followed by stomach-churching drops. What's more, when busts hit, it can take years -- maybe even a decade -- for individuals who bought at the top of the market to recoup their investment.

To see how grim it can get, we looked at annual sales figures for 138 metro areas across the country during the past three decades to spot where local bubbles burst, what drove prices into the cellar and how long it took for property owners to recoup their money. Here are some of the factors that can kill a real estate boom.

It's obvious. Jobs equal workers. Without work, residents leave, and home sales dry up.

Consider the case of southern California. Once home to a thriving defense industry, military cutbacks hit the region especially hard in the early 1990s. Some 1 million individuals left the area, according to Ingo Winzer, president of The Local Market Monitor, a real estate consulting firm that tracks housing prices nationwide.

In Los Angeles, home prices shed 21 percent of their value between 1989 and 1996, with the typical house selling for $172,900. (The peak was $214,800 in 1989 following a five year, 77-percent jump.)

An exodus can hit smaller communities, too. Syracuse, NY once boasted 250,000 residents back in the 1950s, when it was a thriving industrial city. No longer. Many of those jobs are gone and Syracuse lost a full 10 percent of those inhabitants from 1990 to 2000, when its population dropped to 147,000 residents. Home prices, not surprisingly, fell too. Half of all property owners in the county who sold homes in 1997, for example, sold at a loss. Vacant buildings were not uncommon. (At one point, there were more than 1,000 empty dwellings.)

Ask housing experts about local busts and one of the first places they'll mention is Houston, TX. When the oil market was kicked in the teeth back in the mid-1980s, home prices in this city tumbled fast. In just three years, from 1985 to 1988, the typical home price dropped by 21 percent -- or from $78,600 to $61,800.

Related Stories • Did you pay too much for your house? • Real estate or stocks? • Milking the bubble • Rev up your resale value

"Prices fell so much that people owed more on than their mortgages than their homes were worth," said David Weil, an economics professor at Brown University. " They'd drive to the bank and drop off their keys to their homes and just leave."

Houston isn't the only city where home prices have fallen when the local economy languishes badly.

Take the stock market crash of 1987, which hit New York City's financial industry hard. Prices peaked at $183,000 in 1988, and anyone who bought then had to wait until after 1997 to get to even money.

Another victim? Hartford, CT. From 1984 to 1988, the typical home price soared 92 percent to $167,600 from $87,400. Then the insurance industry started laying off or moving out. Hartford's population growth slowed to zero. And home prices starting falling. In fact it wasn't until last year that someone who bought at the 1988 price would have made their money back.

Are markets that have soared quickly especially prone to a bust? That's a question no doubt troubling many homeowners. But the answer isn't simple.

Certainly, there have been plenty of hot markets that suddenly turned sour. Consider Honolulu, Hawaii, for example. Back in 1995, the average tab for a house in this community hit a record $360,000 -- a whopping 122 percent increase from the decade before.

Then suddenly, prices began to drop. By 1999, a $360,000 island retreat was being unloaded for $290,000, a 19 percent discount, according to NAR. Prices started to finally rise in 2000, but anyone who bought at the island's real estate peak didn't recoup their money until this year.

Hawaii's housing woes were tipped off by several factors, not the least of which was the decline in the Japanese economy, which squelched real-estate investment in Hawaii.

Honolulu was also in trouble in part because few fundamentals, other than investment dollars -- were pushing the market. In fact, during the boom years, the island's population was climbing at a 1 percent rate, too low to justify the massive run-up in housing values.

Bottom line: it's important to look at what drives housing spikes before you assume there will be a catastrophe, said Winzer.

"People tell you that housing never goes down, but that's just not true -- you try to sell a house when interest rates have gone up," said Stephen Cauley, associate director of the Ziman Center for Real Estate, Anderson School at UCLA.

To illustrate his point, Cauley points to the early 1980's, when double-digit interest rates were being used to fight inflation. That made the cost of borrowing money for a home almost prohibitively expensive.

"It was horrendous for the housing market," said Cauley. "There were no transactions."

By 1982, the number of existing home sales had slid to 1.92 million, the lowest number on record, according to NAR. Many markets -- notably Detroit, Providence, Chicago and Philadelphia -- saw home prices stay flat or fall between 1979 and 1982.

These days, of course, high interest rates seem a distant threat, though they are beginning to creep up. Current mortgage rates are hovering just above 6 percent for a fixed, 30-year loan. But even if rates go up a full percentage point, rates are still low, said Cauley.

How will all this play out? If history is any guide, there won't be one big pop, the kind that usually come with stock-market crashes. But that doesn't make it any less painful.