NEW YORK (CNN/Money) - You pay a lot of taxes at the federal, state and local levels. Chances are you're not in love with any of them. But if you had to choose, which would you consider to be the least fair?

Apparently, a plurality of us think the local property tax is the worst, according to Gallup's most recent Tuesday Briefing analysis.

Of the more than 500 adults Gallup surveyed, 35 percent said their property tax was the least fair. Twenty percent, meanwhile, named the federal income tax, 14 percent cited state income tax, another 14 percent gave the thumbs down to their state sales tax, and 12 percent identified the federal Social Security tax.

Property tax is often the bugaboo of homeowners, but never more so than in a rapidly rising housing market.

Since property tax is based on the assessed value of your home, if you live in a neighborhood with appreciating home prices, your property tax bill is likely to go up, and potentially in a big way.

When home values rise significantly, many localities may reduce the property tax rate (the amount charged for, say, every $1,000 in home value). Nevertheless, despite rate reductions the total tax bill often still goes up notably as it has in many areas across the country, said Harley Duncan, executive director of the Federation of Tax Administrators.

Besides rising home values, Duncan said, another reason homeowners may be paying more in property taxes now is because of a reduction in federal and state aid to local governments and schools between 2002 and 2004.

While state income tax revenues have been increasing over the past several qaurters, the amount they had collected by June 30, 2004 was just back to 2000 levels.

So when it comes to giving aid to local governments, "revenue increases haven't made up for what we lost in the past two years," Duncan said.

One reason taxpayers may perceive that the property tax -- which is the main source of revenue for local governments -- is the most unfair is because increases are usually sudden and dramatic, suggested Bill Ahern, spokesperson for the Tax Foundation, a nonprofit policy research group that advocates, among other things, a simplified tax system.

By contrast, Ahern said, "Income and sales taxes are both collected in dribs and drabs."

Another reason may be that property tax isn't tied to a realized stream of money but rather to paper gains, he said. You're likely to pay more in tax when you earn more income or when you buy more things. But with property tax, you pay more when your home increases in value, regardless of any changes to your income or whether you ever realize that increased gain when you sell.

Then there's the issue of whether you believe your home has been assessed fairly. There is some recourse if you believe it hasn't. To learn more, click here.

-------------------------

Which states and cities are the most and least tax-friendly? Click here for the story.