BEND, Ore. (CNN/Money) – Are two houses better than one? A growing number of second-home owners seems to think so.

The hottest segment of an already hot real estate market is bubbling up in coastal towns, mountain towns and other vacation spots.

"Things have really picked up in 2004," said Lynda Traverso, a real estate agent on Sanibel Island, near Fort Myers, Fla. where home prices have risen noticeably over the past couple of years.

Most buyers, she said, are from the Midwest or the East Coast. "I see a lot of people in their 40s and 50s who are taking money they would have invested in the stock market and buying vacation homes."

Property values on the Jersey Shore, meanwhile, have been bid up by buyers in New York, Philadelphia and Washington D.C.

This all comes as no surprise to Clark Thompson, CEO of EscapeHomes.com, a site that matches buyers and sellers with real estate agents who specialize in the vacation segment. Traffic on the site has increased more than 20 percent over the past year, and Thompson's own second home, in Palm Springs, Calif., has nearly doubled in value since he bought it less than four years ago.

On Wednesday, EscapeHomes.com unveiled its latest list of top 10 destinations for buying a second home, based on traffic on the site and inquiries from buyers over the past six months. (See Table)

The second-home market is difficult to track because, on paper, second-home transactions are no different than those for primary residences, said Walter Molony, a spokesperson for the National Association of Realtors (NAR).

To get at just how strong these markets are, David Stiff, a senior economist at Fiserv CSW, turned to Census Bureau statistics on seasonal housing units.

Stiff found 13 counties in which at least 10 percent of single-family homes are seasonal.

Vacation home havens Counties with a high proportion of vacation homes, as measured by Fiserv CSW, showed a median home-price increase of 37% over the past three years, vs. 28% for other markets. Market  3-year gain  Cape Cod, MA (Barnstable County)  64%  Island Beach, NJ (Ocean County)  63%  Cape May, NJ (Cape May County)  56%  Lake Tahoe, CA (El Dorado County)  52%  Boca Raton, FL (Palm Beach County)  41%  Brigantine Island, NJ (Atlantic County)  41%  Gulf Coast, FL (Sarasota County)  37%  Jupiter Island, FL (Martin County)  37%  Punta Gorda, FL (Charlotte County)  34%  Sanibel Island, FL (Lee County)  32%  Berkshire Mts., MA (Berkshire County)  29%  Naples, FL (Collier County)  29%  Sedona, AZ (Coconino County)  23%   Gains from 2Q 2000 to 2Q 2003  Source:  Fiserv CSW

Among these counties, the median price appreciation was 37 percent between the second quarters of 2000 and 2003. In counties where seasonal homes represented less than 10 percent of housing, meanwhile, the median appreciation was only 28 percent.

In Cape May County in New Jersey, where seasonal housing accounts for 48 percent of the market, home prices increased 56 percent between the second quarters of 2000 and 2003. In Barnstable County Massachusetts, better known as Cape Code, 32 percent of all homes are seasonal, and prices are up 62 percent.

Prices in the Lake Tahoe area appreciated about 50 percent, while prices in and around Fort Meyers and Naples increased 32 percent and 29 percent respectively.

Said the NAR's Malony, who bought his second home -- a cabin on 5 acres in West Virginia -- in 1995: "The conventional wisdom when we bought was don't expect to make a lot of money on this and plan to hold onto it for decades." The land alone is now worth the 10 times what he paid for it.

When shopping for a second home, you'll want to take extra care to get to know the area. If it's not close to home, this will take more effort on your part. Thompson recommends visiting the at least three times and working with a real estate agent who knows the market.

Also keep in mind that if you're financing the property, you may have some extra hurdles than you did with your primary home.

Because lenders typically view second homes as a higher risk, you may pay a slightly higher interest rate, see higher fees and be required to hand over a larger down payment, said Mindy Neubauer, a spokesperson for Lendingtree.com. "Many lenders require 20 percent or more down," she said.

According to NAR's Molony, financing a second home has gotten a little easier in recent years. "It used to be that lenders required shorter terms and wouldn't give you a fixed rate," he said.

There are also tax considerations. The interest you pay on your second home's mortgage is tax deductible, assuming you don't rent it out for more than two weeks a year. But, any profit you make on the property is subject to capital gains, unless of course you eventually sell your primary residence, move into your second home and live there for at least two years

Finally, there's homeowners insurance to consider. Insurance may be more expensive because you're not there full time or because your remote retreat is too far from the nearest fire station.

"Our cabin is worth less than half our primary residence yet we're paying a lot more," said Molony. "And our premium just went up 56 percent."