We'd like to return these bad loans, please

NEW YORK (CNNMoney.com) -- The foreclosure document fiasco has already caused a major headache for U.S. banks -- and that headache may soon escalate into a migraine.

But the additional pain isn't coming from the Obama administration or state attorneys general, both of whom have stepped up pressure on the banks. Nor is it coming from individuals who allege their homes were wrongly foreclosed on.

It's coming from institutional investors who bought home loans that had been bundled together by banks and then sold off as Residential Mortgage Backed Securities, or RMBS.

These investors thought they were buying solid investments. But the recent robo-signing debacle shed light on document problems in foreclosures, revealing problems with the underlying paperwork and quality of the loans.

So, investors are trying to force banks to repurchase the securities. That would leave banks exposed to billions of dollars in potential losses if plaintiffs are able to force repurchases based on the errors.

Of course, repurchase requests aren't new. Banks have been receiving them for years. But they could be facing an onslaught now.

"It's really going to be up to the courts to decide on this one ... Certainly investors will try to use this to avoid losses," said Joseph Mason, a professor at Louisiana State University, who studies legal risk in finance and banking.

Earlier this week, the New York Fed, along with investor firms BlackRock and the Pacific Investment Management Company, sent a letter to Bank of America (BAC, Fortune 500) alleging that its subsidiary, Countrywide, has failed to properly service loans totaling $47 billion.

The letter doesn't amount to legal action, but it does put BofA on notice that litigation may be down the road.

Bank of America is among the most exposed of all the banks due to the sheer scale of its business. It sold a staggering $1.2 trillion in loans Fannie Mae and Freddie Mac between 2004 and 2008, and thus far has received repurchase requests on $18 billion of those loans.

But it says only $2.5 billion in losses have resulted from those requests.

"The risk is relatively sealed on this ... the issue is how long the fight will take," Bank of America CFO Charles Noski said during the company's third quarter earnings call.

Each bank has a different internal process, but in many cases losses are avoided by providing supplementary documents that prove the claim is invalid.

BofA estimates it is more than two-thirds through the wave of repurchase requests on loans made at the height of the mortgage free-or-all, and Moynihan has vowed to fight unjustified repurchase demands in order to protect shareholder interests.

"If you think about people who are going to come back saying I bought a Chevy Vega, but I want it to be a Mercedes with a 12 cylinder. We're not putting up with that, and we will be very ardent to protect the shareholders interest," Bank of America CEO Brian Moynihan said during the earnings call.

But Bank of America's outstanding repurchase claims have climbed in each of the last four quarters, reaching a peak of $12.9 billion in the third quarter of 2010.

Worries over the bank's exposure have kept its stock under pressure, sending shares to a 52-week low on Thursday.

JP Morgan Chase (JPM, Fortune 500) reported paying out $1.5 billion in repurchases in the third quarter, an increase of $1 billion over last year. And the bank added $1 billion to its repurchase reserve in anticipation of an increase in requests. But that remains a relatively small figure compared to the bank's $24.3 billion in third quarter revenue.

Not every bank is in the same position as Bank of America and Chase. Wells Fargo (WFC, Fortune 500), for one, insists that its exposure is relatively low.

"These issues have been somewhat overstated and to a certain extent, misrepresented in the marketplace," Wells Fargo CFO Howard Atkins said during the bank's earnings call on Wednesday.

Executives on the call insisted the bank does not anticipate an increase in repurchase requests, and that the number of requests had fallen in the third quarter.

If repurchase requests were to spike, the bank says its $1.3 billion repurchase reserve liability is adequate.