Put the pitchforks down. Blaming the Federal Reserve for America's widening income inequality isn't fair.

Yes, income inequality is a major problem, one that was worsened by the Great Recession and the sluggish recovery.

And yes, the Fed's response to the crisis -- unprecedented amounts of easy money -- did boost the stock market. That, in turn, made many rich Americans even better off.

But those arguing that the Fed intensified inequality miss the boat: The Fed was the only game in town in 2008 because Congress didn't act more forcefully to save the U.S. economy. And if the Fed didn't step up to the plate, income inequality would likely be far worse today.

"Given the fiscal policy we had, what could the Fed do? I think they did the best thing they could," said Dean Baker, co-head of the left-leaning Center for Economic Policy and Research. "We wouldn't be happier with fewer jobs and a lower stock market."

Fed had no choice: In a perfect world, the U.S. would have responded to the financial crisis with a powerful combination of tax cuts and spending programs designed to breathe life back into the economy. Think: big tax rebates for businesses and consumers and lots of major infrastructure projects.

Yet there wasn't much appetite for a large amount of fiscal stimulus, which has far more of a direct impact on the economy than the monetary stimulus that the Fed does. Part of the apprehension for fiscal stimulus was understandably driven by concerns about running up the deficit. Of course, America's bloated balance sheet is Congress' fault, not the Fed's.

Lawmakers ended up passing a very modest stimulus package in 2008 and a larger one in 2009. In retrospect, each proved to be too timid given the severity of the recession.

Related: The Bernie Sanders plan to fix income inequality

Fed stepped up: That left the Fed to do the only thing it could do: Lower interest rates to near zero for the first time ever and buy trillions of dollars of bonds.

"The Fed became the fourth branch of government -- and arguably the most functional branch," said Kristina Hooper, U.S. investment strategist at Allianz Global Investors.

Thanks in large part to what the Fed did, risky assets like stocks rebounded from the crisis. That was good news for affluent Americans who own stocks but also the millions of people who have exposure through pension funds and 401(k) plans.

Related: Who's getting rich off the stock market?

'Flawed' tool: However, the labor market took much longer to recover, deepening the income inequality problem.

"If the Fed did nothing and unemployment was still at 10%, it's hard to argue that would have been good for income distribution," said Douglas Holtz-Eakin, the former director of the Congressional Budget Office.

The sluggish recovery shows how monetary policy is a "flawed" tool, Hooper said.

"We may have been able to have a more robust jobs recovery by now had we focused on fiscal stimulus as opposed to monetary," she said.

Related: U.S. and Israel have worst inequality in developed world

'Best weapon' against poverty: That's a point echoed by Ben Bernanke himself in a blog post this week.

"If fiscal policymakers took more of the responsibility for promoting economic recovery and job creation, monetary policy could be less aggressive," the former Fed chief wrote.

Bernanke pointed to a paper published this week by the Brookings Institute that argued monetary stimulus "does not necessarily increase inequality." The paper, by Josh Bivens, also found that positive employment effects are "by far the largest impact of monetary policy changes on inequality."

That's critical for the inequality question. Bernanke believes a healthier jobs market "is the best weapon we have against poverty."

Rising home values help average Americans: It's also important to remember that the Fed's emergency policies weren't just good for the stock market. They also boosted the battered housing market by making mortgages cheap and allowing consumers to refinance at very low rates.

While lifting the stock market may increase inequality, lifting home prices reduces inequality because home equity represents a big chunk of middle-class wealth, Bivens argues. He pointed to how housing makes up 62.5% of household wealth for the middle class, versus 28% for the 80th to 99th percentiles and less than 9% for richest 1% of Americans.

Don't forget that income inequality is not a problem that was born during the crisis. In fact, as Bernanke points out, it's been a growing issue in the U.S. over the past several decades due to a variety of factors, including globalization and demographics.

Related: America's poor are 'envy of the world,' says richest Congressman

Rushing a rate hike could backfire: None of this means the Fed can't help prevent income inequality from widening further.

If the Fed raises interest rates before the jobs market gets back to full health, it would "disproportionately harm the wages of low and moderate-wage workers," Bivens wrote.

Hopefully Fed officials keep income inequality concerns in mind as they debate their next moves.