NEW YORK (CNNMoney.com) -- Manhattan and the metropolitan area of Stamford/Norwalk, Conn. have the highest paying taxpayers among U.S. counties and cities.
That's according to a recent report by the Tax Foundation, which analyzed IRS data from tax year 2004 to measure the average federal income tax burden as a percent of adjusted gross income (AGI).
The Tax Foundation is a nonprofit fiscal policy research group that advocates for, among other things, a flat tax.
Among counties, taxpayers in New York County (Manhattan) pay the highest average percentage of AGI (20 percent) in federal income taxes, or $25,875.
Among major metropolitan areas, taxpayers in Stamford and Norwalk, Conn. pay the highest average percentage of AGI (22 percent), or $41,496.
On the other end of the spectrum, Starr County in Texas is the lowest ranked county in terms of federal tax burden. It had a negative tax burden of 4.13 percent, meaning taxpayers there got back an average $867.
Several factors determine taxpayers' federal income tax liability in a given city or county.
For starters, the federal income tax system is progressive: those who make more pay a higher percentage of their income in taxes than those who make less.
High-cost-of-living areas typically have more high earners. But federal income tax rates don't adjust for cost-of-living differences.
Investment income is another factor influencing the rankings. Taxpayers with more taxable investments may have higher overall AGIs than taxpayers with fewer if any investments. But if a taxpayer makes most of his income from investments, his federal tax burden may be lower as a percentage of AGI, because capital gains and dividends are taxed at lower rates than salaries and wages.
State and local income taxes play a role as well. Since they are deductible on your federal return, if you live in an area with high state and local income taxes you will have a bigger deduction than if you lived in a low tax area.
On the flipside, however, those who live in high tax areas (especially in New York, California and Connecticut) are more susceptible to the Alternative Minimum Tax (AMT), which imposes a higher tax burden than the regular income tax code. That's because a number of tax breaks - including the state and local tax deduction - are disallowed under the AMT.
Lastly, family size and housing prices can make a difference in your federal tax burden, too. The more kids you have, the more federal tax breaks you get. And the higher housing prices are, the more likely you are to pay more in mortgage interest and get a bigger deduction as a result. 
