Luxury property sales in London are bouncing back now that the country's elections are over and the Labour Party lost.

Real estate brokers in the U.K. breathed a sigh of relief after last week's parliamentary election left the Conservative Party in power. With the Labour Party losing the vote, the prospect of a "mansion tax" has diminished and prices are set to grow further.

The tax on properties worth more than £2 million ($3.1 million) was at the center of Labour's election campaign, spooking potential investors.

"It would have been a serious problem if Labour got in," Charles McDowell, an agent specializing in London prime property, said. "We went from very pessimistic to incredibly optimistic in a week," he added.

The Labour party also pledged to introduce a tax on overseas earnings for the super-rich. Many many London luxury property buyers are foreigners.

Many high-end property buyers held off ahead of the election. Sales in the central London boroughs of Westminster, and Kensington and Chelsea were down 9% last year, data from Capital Economics show.

"These clients are now resuming their search and I think we will see an increase in activity," McDowell said.

His clients are the global rich, who want to be in central London -- even if they don't live there all the time. "When you have a certain amount of money, London is one of the places you want to buy property in, along with the south of France or a skiing chalet," he said.

He said these buyers are very sensitive to tax increases, such as the one proposed by Labour. "What our clients don't want is annual tax to go up. They'd rather pay one lump sum," he said.

Related: Where the super-rich are buying

Analysts say the prices are not likely to jump dramatically just because of the election results. "The Conservative victory is an overall a positive for the housing market. But we wouldn't want to overstate the impact, and house prices are not set to surge on this outcome," Matthew Pointon, Capital Economics' property economist, said.

But as investors cheer more certainty and stability, there is one big worry on the horizon. Prime Minister David Cameron has pledged to give voters an "in or out" referendum on British membership of the European Union by 2017.

The risk of a British exit from the EU -- or "Brexit" -- is likely to make some investors nervous.

"An exit from Europe may have implications not only for the value of Sterling but also on inward investment; which could in turn affect commercial property values," said Ian Whittock from real estate consultancy Knight Frank.