NEW YORK (CNN/Money) - Home builders, realtors and others are preparing to fight a Bush administration plan that would require Fannie Mae and Freddie Mac to increase financing of homes for low-income people, a home builder group said Thursday.

The National Association of Home Builders, along with the National Association of Realtors and the Mortgage Bankers Association, are drafting a letter to Alphonso Jackson, secretary of the Department of Housing and Urban Development (HUD), arguing that middle-income home buyers are the ones that will get hurt by the proposed plan, the NAHB told CNN/Money.

In April, the HUD proposed new rules that would raise the percentage of loans bought by the two government-sponsored enterprises (GSEs) that finance borrowers whose incomes are at or below the median for their area, according to the Wall Street Journal .

But the groups will warn in the letter that the proposed rules requiring the two GSEs to finance more "affordable housing" may have "unintended consequences," hurting some poor and middle-income people struggling to afford houses, the Journal said.

Fannie and Freddie, which use their ability to borrow cheaply in the government agency bond market to help middle-to-low income people buy homes, would be compelled to provide more funds to low-income home buyers by slashing their financing of middle-income home buyers, David Crowe of the NAHB told the paper.

The points being raised by the groups have also mirrored objections raised by Fannie (FNM: Research, Estimates) and Freddie (FRE: Research, Estimates). Both GSEs said they favor more efforts to promote affordable housing, but say HUD has made some unrealistic assumptions about how much more the GSEs can do over the next few years, the Journal said.

Interested parties have until July 2 to comment on the rules. The letter to Jackson will ask for a further 60 days of public comment, after which HUD would decide whether to push ahead with its rule or revise it, the paper added.

The NAHB said the groups plan to send out the letter later this week.