NEW YORK (CNNMoney) -- Billionaire Warren Buffett can pay more taxes if he wants to, but some "ordinary millionaires" say they're already paying plenty.
Buffett has been outspoken in saying the super-rich should pay more. Others in that club -- like Bill Gates -- agree. And President Obama has been pushing the "the Buffett Rule", a guiding principle that aims to ensure the rich pay as much a share of their income as the middle class.
Even so, some millionaires say Buffett doesn't speak for them.
"There is more of a difference between my financial position as a multi-millionaire and Buffett's than there is between mine and a guy that makes minimum wage," one CNNMoney reader said. "Why am I grouped with him and why does he feel he can speak for me?"
Only 24% of millionaires said higher taxes on higher incomes is the fairest way to go, according to recent survey from Spectrem Group, a research firm specializing in finances of affluent Americans. The biggest chunk of millionaires, 44%, think a flat rate tax across all income brackets is the fairest system.
One millionaire CNNMoney reader said that for the past five years, his tax rate (including state income taxes) has ranged from 40% to 55% -- which he thinks is more than enough money to be forking over to Uncle Sam.
"We are not a country that was founded on entitlements, but for some reason or another people now believe that they are entitled to freebies," he said. "Of course being paid not to work is a lot easier than actually working."
The idea of a millionaire tax also strikes a nerve among millionaire small business owners. Their taxes wouldn't necessarily increase under a Buffett Rule, because it would likely target those who get most of their income from investments, on which they pay just the 15% capital gains rate.
Still, some business owners don't want to hear they should be paying more.
Leigh Bortins, for example, owns a startup business selling homeschooling education curriculums and providing seminars for families. She files it as an S-Corp, meaning her business income is taxed at personal rates.
Bortins reports $1 million to $2 million in profit on her tax return each year, and she said the nearly $1 million she paid in taxes this year on her 2010 income could have allowed her to hire 10 more employees (at an average salary of $50,000) and she still could have given $500,000 in taxes to the government.
Another business owner said paying more taxes when you're in your 80s -- like Buffett -- is a lot different than paying more when you're younger and just starting a business.
"The Buffett Rule works great if you're 80+ years old and already successful," said , said Jon Hoch, the owner of a power equipment company. "At 40 years old, the vast majority of my profits are reinvested back into my business. As an S-Corp, I pay huge quarterly taxes even though I reinvest my the majority of our profits to increase our inventory. "
Even some non-millionaires are against higher taxes on the rich, saying it would discourage the drive to get ahead.
"I am not a millionaire but I am trying my best to become one and I am teaching my children to work hard, be innovative, and always reach for your dreams," said one CNNMoney reader. "Why should I support crushing those dreams because some people don't aspire for that dream and want me to subsidize their lack of ambition?"
On the other side of the spectrum, some taxpayers who haven't even hit the millionaire tax bracket yet are willing to give more money to Uncle Sam.
Lucas Wacker and his wife bring in a total taxable income of a little over $62,000 but he says they live a comfortable enough life to be able to give a little extra to the government if it would help pull the economy out of this current mess.
"I feel if it would honestly make a difference, I could easily pump another 2-3% into taxes without having to change my lifestyle at all," he said. "I don't have any children yet, but I would rather skip a trip to dinner and the cinema once a month to have to pay taxes than have my unborn children have the worries of unemployment."
Wacker said that five years ago, when he was only making $17,000 a year, he still lived a good life. Now, at $40,000 a year he lives a better life but says he and his wife spend money on unnecessary things, like eating out too often.
"Americans are too selfish now-a-days -- they do not see the greater good," he said. "We are in a pretty bad spot right now and we all have responsibilities." 
Carlos Rodriguez is trying to rid himself of $15,000 in credit card debt, while paying his mortgage and saving for his son's college education.
Susan Carson and Laura DeLallo make $225,000 and have half a million in retirement savings, but their sprawling portfolios is proving hard to manage.
| Overnight Avg Rate | Latest | Change | Last Week |
|---|---|---|---|
| 30 yr fixed | 3.80% | 3.88% | |
| 15 yr fixed | 3.20% | 3.23% | |
| 5/1 ARM | 3.84% | 3.88% | |
| 30 yr refi | 3.82% | 3.93% | |
| 15 yr refi | 3.20% | 3.23% |
Today's featured rates:
