NEW YORK (Money Magazine) -- Summer's winding down, and you've still got vacation on the brain: the quiet of the country, the crisp mountain air, the lull of the ocean.

In the hope of holding on to those images a bit longer, you may have begun toying with the idea of a second home.

You're not the only one. "A lot of people go on vacation, look at real estate fliers and suddenly decide they want to buy a house," says David Hehman, CEO of Escapehomes.com.

On the other hand, you know from your primary home's market that real estate is not the no-brainer investment it once was.

And so you wonder: Are you letting your heart run away from your head? Should you even be considering a vacation home?

Probably not, unless you can accept that you're no longer going to get rich quick by buying a second home. Between 2001 and 2006, the median home price in the 30 top vacation markets rose 120 percent, according to David Stiff, senior economist for Fiserv Lending Solutions - nearly twice as much as the appreciation in primary markets.

And second-home prices are even more vulnerable to a downtun - nobody needs a second home.

Most house-sale numbers for this spring aren't in yet. But in the first quarter, "sales volume in vacation areas dropped a lot, " Stiff reports.

For a prime example of how quickly markets can turn, consider Naples, Fla., which showed a nearly 40 percent appreciation between the fourth quarters of 2004 and 2005. Yet between June 2005 and June 2006, sales volume dropped 48 percent, according to the Florida Association of Realtors. And median prices, meanwhile, fell 8 percent.

Michael Costa, a 39-year-old equity trader in New York City, can tell you all about it. In July 2004 he bought a house in Naples, largely because it was one of the hottest markets he could find.

By the time he listed the house in fall 2005, the market was already deflating. The property took 10 months to sell, and he got 19 percent less than his original asking price. Costa made money, but not nearly as much as he would have months earlier.

"I feel lucky because I think it's going to get worse," he says.

So far, however, Naples' rapid decline seems to be an anomaly. Like primary-home sellers, second-home sellers have been loath to drop prices, Stiff says, preferring instead to keep their houses on the market longer.

As a result, prices in many vacation areas have continued to rise, though at a slower pace.

In Cape Cod, prices were up only 3 percent between the first quarters of 2005 and 2006 compared with 5 percent the prior year.

Lake Tahoe's El Dorado County was up only 7 percent vs. 26 percent the year before.

It's not all grim news. The moral of this market: If you are going to buy, do it for love, not money. Do it because it's a place you adore and will want to use for years to come. That way, even if prices dip - or plummet - you will be able to enjoy it while you wait out the fluctuations.

"Right now you should be thinking consumption rather than investment," says Nicolas Retsinas, director of Harvard University's Joint Center for Housing Studies.

By that token, if you've always dreamed of owning a piece of oceanfront or having a slope-side condo for ski season, you may actually find today an appealing time to buy.

For one thing, the deflating bubble has shifted the power in the market from seller to buyer. In most markets, houses are taking longer to sell, and consequently owners are offering incentives and showing a willingness to negotiate.

"If you find a house you like, you can almost watch for it to go down in price," says Hehman.

Meanwhile, the rental market is recovering from a prolonged slump brought on by the ownership craze. Rising interest rates have made renting more feasible than buying for many. Rents are rising in a number of markets, and occupancy rates are at five-year highs.

So while you'll pay more to finance a home, you may have better luck renting it than you would have had a year ago.

Still, don't let the fresh air and margaritas kidnap your common sense. It's important to be realistic, especially in an uncertain market.

First off, make sure you truly like the community. For that, there's no substitute for spending time in a prospective town. "The more you know an area, the better you can eliminate some of the risk of buying," says Hehman. Rent for one season, and you'll also find out about up-and-coming areas and projects that could affect your investment.

Second reality check: Can you afford this? "The basic rule of thumb is that your housing costs - including those for your primary home - should be a third of your overall income," says Jay Mastilak, a certified financial planner and senior vice president for PNC Investments. Include mortgage, maintenance, property taxes, association fees, insurance and travel expenses.

It's also worth noting that higher rates on mortgages and home-equity loans may cancel out the advantages of buying at a narrow price cut. So do the math.

The next thing to figure out is whether you'll spend enough time at the house to justify the expense. Come up with a back-of- the-envelope estimate by adding up annual expenses and dividing that sum by the total nights you'll spend there. Don't be surprised if you find it's cheaper to stay at a five-star resort.

But maybe you're thinking of offsetting the cost by renting? Make sure your work will be as easy as you're hoping: Consult property management firms about when, how long and for how much you could expect to rent the property.

You should also get a sense of the long-term sale potential in the community. While you may accept that your home won't sell for twice what you paid anytime soon, you don't want to lose money either.

Ask local agents for appreciation averages for the past 10, 20 and 30 years; ideally, you'll see a positive number.

Once you've done all this homework, the fun part (read: open houses) begins. The bright side of this market is that you can - finally - take your time shopping for the right place. This year Birmingham residents Bill and Walker Jones, 55 and 53, began building a house near Big Sky ski resort in Montana after 10 years of vacationing nearby. Bill is sure they're making the right investment, and at the right time: "I think we'll look back and see this as one of the best decisions we've ever made."

Latest home prices in 151 markets

How much home can you afford?