Fed Chairman Ben Bernanke testifies before the House Budget Committee last week.
NEW YORK (CNNMoney) -- By now, Federal Reserve Chairman Ben Bernanke must be used to being a punching bag.
Both Mitt Romney and Newt Gingrich have vowed to replace him if they're elected president. Texas governor and former presidential candidate Rick Perry claimed it would be "treasonous" for Bernanke to use Federal Reserve policy to stimulate the economy before the election.
But criticism of the Fed also extends into Congress. Some Republican lawmakers are scolding Bernanke for what they feel is a case of him overstepping his bounds and telling them how do their job.
In a House hearing last week, Rep. Scott Garrett took issue with a white paper published by the Federal Reserve last month that analyzes several potential housing policies.
"I was truly taken aback," said Garrett, a Republican from New Jersey, claiming the paper backed policies pushed by the Obama administration. "Is this an invitation now to Congress that we should be issuing resolutions to what the monetary policy that the Fed should be doing? If so, I'm sure there's a lot of members who would like to engage in it."
But despite the criticism, Bernanke does his best to not step into the political fray. At a Senate hearing Tuesday, Congress members from both sides of the aisle tried to push Bernanke to endorse their point of view. Instead, he repeatedly offered economic forecasts without backing specific budget proposals.
"I've often said I'm in favor of the law of arithmetic," he said at one point. And at another, "I'm a little bit leery of taking positions."
A Republican appointed to the Fed by President George W. Bush, Bernanke stood by then Treasury Secretary Henry Paulson in 2008 to urge Congress to act swiftly to stabilize financial markets.
Other than those emergency efforts to address the crisis, he has largely refrained from supporting fiscal proposals from either Republicans or Democrats.
That stands in stark contrast to his predecessors, who were often more outspoken. Some were even praised for their advice.
Alan Greenspan enthusiastically backed the Bush tax cuts in 2001 -- a move Greenspan himself has since admitted was a mistake.
While Republicans are now wagging their finger at the Fed's thoughts on housing, back then, lawmakers were glad to have Greenspan's comments on fiscal policy.
And that's nothing new. Congress has often sought the advice of Fed chairmen. In the early 1980s, Paul Volcker urged Congress to support deficit cuts even deeper than those planned by President Reagan.
A Democrat appointed by Jimmy Carter, Volcker later opposed financial deregulation and was passed over by President Ronald Reagan for a second appointment to the Fed. Greenspan, who took his place, advocated for deregulation openly before Congress.
But instead of using his position to promote specific fiscal policies, Bernanke repeatedly offers this same, vague advice to Congress: focus on cutting the deficit over the long run, but don't derail the still-fragile economic recovery in the meantime.
"I would say 'do no harm' is an important piece of advice I would offer you," he said Thursday.
Ultimately, Fed watchers said Bernanke's advice is mostly economic, not political.
"Talking to Congress about how the deficit affects the economy and how addressing it affects the economy is not in the realm of politics, it's in the realm of economics," said Ellen Zentner, senior U.S. economist at Nomura Securities.
And as for that pesky white paper last month?
"The Fed is concerned about housing because it's an area key to the economy," said Diane Swonk, chief economist for Mesirow Financial. "The Fed feels justified in voicing those concerns. That doesn't mean they're taking a partisan stand."
Meanwhile, it's not as if Congress doesn't try to meddle in monetary policy.
In September, four top Republican Congressmen -- Sen. Mitch McConnell, Rep. John Boehner, Sen. Jon Kyl, Rep. Eric Cantor -- sent a letter to the Fed, urging the central bank not to enact more stimulative policies that they feared could trigger inflation in the future.
|What we want Apple to unveil at WWDC|
|Millennials squeezed out of buying a home|
|7 traits the rich have in common|
|Big Data knows you're sick, tired and depressed|
|Your car is a giant computer - and it can be hacked|
|Overnight Avg Rate||Latest||Change||Last Week|
|30 yr fixed||4.01%||4.01%|
|15 yr fixed||3.18%||3.18%|
|30 yr refi||4.02%||4.01%|
|15 yr refi||3.19%||3.19%|
Today's featured rates:
|Latest Report||Next Update|
|Home prices||Aug 28|
|Consumer confidence||Aug 28|
|Manufacturing (ISM)||Sept 4|
|Inflation (CPI)||Sept 14|
|Retail sales||Sept 14|