Just about everybody agrees that the housing market is finally recovering -- but don't expect big price gains.

Nearly two-thirds of the nation's housing markets will see price declines for the year through next June, according to analytics firm Fiserv (FISV). Overall, the gains will be just 0.3%.

One big factor that could weigh on prices: The fiscal cliff.

If Congress can't agree on a deal to halt a series of tax increases and spending cuts, a recession is likely, and that would hit the housing recovery hard.

In addition, if the Bush-era tax cut on capital gains is allowed to expire -- allowing the rate to increase to 20% from 15% on Jan. 1 -- it would take a significant bite out of the profits high-end sellers would realize and give them less to spend on buying a new home, said Celia Chen, an economist and housing market analyst for Moody's Analytics.

"Even people who do have the resources to buy homes will be more nervous," she said.

Related: Home prices: Your local forecast

But even if we avoid the fiscal cliff, there are other factors weighing on home prices.

In order to raise more tax revenue, Congress is considering putting a cap on the mortgage interest tax deduction, a key tax break aimed at encouraging homeownership -- mainly among the upper-middle class.

The fiscal cliff's impact on homeowners

Most of the benefit of this deduction goes to wealthier households. Mortgage borrowers with incomes of $250,000 or more realize an average annual tax savings of $5,460, according to the Tax Policy Center. Meanwhile, those making less than $40,000 a year, save just $91.

Capping the deduction would discourage buyers from buying bigger, more expensive homes, said Chen.

But it's not just the high-end of the market that could get squeezed.

With Congress distracted by the fiscal cliff, there is a real chance that the Mortgage Debt Forgiveness Act of 2007 could expire come January 1. If the act were to lapse, struggling homeowners will have to start paying income taxes on the portion of their mortgage that is forgiven in a foreclosure, short sale or principal reduction.

Related: Most affordable cities for homebuying

That means homeowners will be on the hook for thousands of dollars in taxes that they likely can't afford. That will force more people who could have sought a less damaging alternative, like a short sale, to choose foreclosure instead.

Fiserv's estimates assume that about half of the fiscal cliff tax hikes and spending cuts will occur, said Stiff. The forecast does not take into account any change to the mortgage interest deduction. Should that deduction expire, Stiff said home prices might be even weaker over the short-term.

Fiserv expects home prices to start heating up again next fall. Between June 2013 and 2014, it expects prices to climb 3.4% and to continue to grow at an annual rate of about 3.3% over the five years through June 2017.