The housing bust wrecked the finances of a lot of families, but it hit minorities especially hard. And now, those minority families are missing out on the housing recovery.

Hispanic households on average lost nearly half of their home equity between 2007 and 2010, according to a paper by the Urban Institute. Black households lost 28% and white households lost just 24%.

The housing bust hit minorities harder partly because they came to the home ownership party later, according to Caroline Ratcliffe, a co-author of the Urban Institute paper.

"Hispanics in particular did a lot of home buying just before the recession hit," she said.

That meant that many bought at or near the top of the market. And when their home equity vanished, it made them more vulnerable than other groups to foreclosure.

Related: Best deals on real estate

Now that home values are rebounding, it will benefit white households much more. Their home ownership rate is close to 75%. Black home ownership is about 45% and for Hispanics, it's 47%.

"Due to disproportionate loss of home ownership among people of color, the racial wealth gap is likely to grow further as families that have lost their homes will see no benefit from the recovering home prices," said James Carr, senior fellow with the Center for Economic Progress.

Minorities also haven't been able to get back into the housing market because banks have made it tougher and costlier for risky borrowers to get mortgages.

Minority families are facing several barriers to becoming home buyers again.

Most home loans are now backed by Fannie Mae, Freddie Mac and the Federal Housing Administration. All have made loans more expensive for borrowers with low credit scores.

Related: 76% of Americans living paycheck to paycheck

Additional fees for buyers with low credit scores can add up -- more than $100 a month when compared to someone with a high credit score getting a conventional loan.

Even worse, FHA borrowers now have to pay mortgage insurance premiums for the entire term of the loans, even during the last few years when the risk of default is very small.

Government programs favor the well off over the poor, said Ratcliffe.

"The federal government spends billions in tax credits to promote long-term asset growth," she said. "Tax-free retirement savings, the mortgage interest deduction and other programs primarily benefit high income Americans.

The wealth gap could widen if the pathway to home ownership is blocked for too many minorities, said Lewis Ranieri, founder of Ranieri Partners.

"We need a system that funds first time homeowners and trade up buyers and it shouldn't be only for middle class whites," he said.