Raise taxes now -- the elders of the economy say so

NEW YORK (CNNMoney.com) -- First it was Greenspan. Now one by one, other elders of the economy are speaking out against deficits, and they're making the surprising argument for higher taxes.

Former Federal Reserve chairman Alan Greenspan was first and has taken the most extreme position, arguing that all of the Bush tax cuts of 2001 and 2003 should be allowed to expire.

Greenspan, no fan of big government and an initial backer of the Bush tax cuts, allows that higher taxes now could lead to slower economic growth, but has said that chipping away at the deficit is more important.

Joining him -- at varying degrees -- are David Stockman, former budget director in the Reagan White House, and former Treasury Secretaries Robert Rubin and Paul O'Neill.

The White House and most Democrats have argued for keeping the tax cuts in place for most households, but letting them expire for those earning more than $250,000, about 2% of the country.

Extending the tax cuts for everyone would cost the government $3.7 trillion over 10 years. Taxing the high-earners would get back about $700 billion of that.

David Stockman joins Greenspan at the far end, saying the nation can't afford to extend the tax cuts now. "You have to pay your bills. I say we can't afford the Bush tax cuts," Stockman told NPR this weekend.

On CNN's Fareed Zakaria GPS this weekend, Rubin who worked in the Clinton White House, and Paul O'Neill, who worked for George W. Bush, also talked about the need for higher taxes.

Rubin supports the White House position of extending cuts for most households, yet raising them on the wealthy. Simply, he worries about raising taxes too broadly while the economic recovery is still shaky.

That said, he is also a proponent of the estate tax, which expired this year, and is slated to return to 2001 levels -- a $1 million exemption and 55% top rate.

Said Rubin: "I would put an estate tax in place right now, immediately. I would increase the tax on the higher brackets."

O'Neill said one reason he was fired as Treasury Secretary was his take on tax cuts: "I was strongly opposed to the Bush tax cuts in 2003. I didn't think we could afford another tax cut." (video)

The issue now is not the Bush tax cuts, said O'Neill, who doesn't think the economy is in terrible shape. Rather, it's the need for fundamental tax reform, preferably one that is much simpler and emphasizes investment and growth over immediate consumption.

"If we let the Bush tax cuts expire, it's still the same stupid tax system."