NEW YORK (CNN/Money) - Wondering why the S&P 500 is having so much trouble making any headway lately? Hang it on banks.

Thanks to the fall-off in the Treasury market, the financial sector is running into serious headwinds. Investors worry that, despite all the Fed officials' yelping to the contrary, Treasury investors are correctly forecasting that the revival in the economy will be so strong that the Federal Reserve is going to have to hike the overnight funds rate before too long. That would take away the banks' low cost of money, cutting into their profitability.

More immediately, investors worry about what sort of mortgage trouble the banks have gotten into. The big backup in bond yields has sent mortgage rates sharply higher: Wednesday, the Mortgage Bankers Association reported that the average rate on a 30-year mortgage last week was 6.22 percent, up sharply from the June low of 5.06 percent.

The rate rise has led mortgage activity to drop to a 14-month low, according to the MBA. Outfits that have big mortgage-origination businesses like Washington Mutual, Wells Fargo and Countrywide have naturally seen a fall-off.

Banks that loaded up on the tradable baskets of mortgages called mortgage-backed securities may also be hurting. According to the Federal Reserve, large commercial banks had 17 percent less in mortgage-backed securities in their portfolios in the week ended Aug. 13 than they did in the beginning of June. Some of that is due to the fall-off in those securities' value, most is due to some heavy selling.

The good news is that commercial loan activity is showing signs of finally picking up -- but it isn't happening fast enough to save the banks in the current quarter.

Analysts are going to need to start taking down their third-quarter earnings expectations soon, and even then, at least some of the banks may have to issue warnings. That means investing in the sector right now is a bit like playing Russian roulette -- you might get lucky and pick the ones that haven't run into any problems, but if you're wrong you're portfolio is going to be in trouble.

These are chances that most professional investors don't want to take. And because the financial sector represents such a big portion -- over 20 percent -- of the S&P 500, it makes it very hard for the index to break higher.