Forget oil. It's time to worry about banks.

Shares in some of the world's biggest banks are plunging. Financial stocks in the S&P 500 are down more than 11% so far this year. That's worse than oil, energy stocks, and even the emerging markets index. European banks have fallen even further.

Deutsche Bank (DB) has lost 31% so far this year, Unicredit (UNCFF) is down 35%, and Credit Suisse (CS)is 30% down. Barclays (BCLYF), BNP Paribas (BNPQF), Societe General (SCGLF), and UBS (UBS) have all lost about 20% since the beginning of 2016.

Bank earnings have generally been disappointing. Credit Suisse shares hit a 24-year low after it posted its first loss since 2008, and BNP Paribas suffered a 50% drop in net income in the fourth quarter.

Even JPMorgan Chase (JPM), which had a bumper quarter, warned of a "challenging" start to 2016 because of volatile markets.

There is growing fear about the state of the world's economy. China is slowing down, and the American economy is losing momentum. Emerging markets are struggling to keep growing -- and some have slumped into recession.

Banks are being hit particularly badly. Here are three reasons why:

1. Low interest rates

Low interest rates in most major economies are cutting into banks' profit margins, and investors are losing hope for a change any time soon.

The Federal Reserve raised interest rates for the first time in nearly a decade last December, and has indicated that rates could rise four times this year.

But investors don't believe it, and the financial markets are now pricing in zero hikes in 2016.

Related: Cash is more popular than stocks or bonds

The Bank of England signaled Thursday that rates are likely to stay at record lows for another year. The Bank of Japan has just introduced negative interest rates, and the European Central Bank is widely expected to take action soon to drive eurozone rates even lower.

"Risk of a Fed backtrack and negative rates across the board from major central banks (are) not good for (the) sector," said Mike van Dulken, head of research at Accendo Markets.

2. Oil pressures

Collapsing crude oil prices are also weighing on the financial sector. Many banks have invested heavily in big oil projects, only to see the companies behind them go bust as prices collapsed. At least 42 North American oil companies filed for bankruptcy since the beginning of 2015.

"The decline in oil prices to largely unforecasted levels makes for a large question mark over all the lending done to support the industry during boom times," said Nicholas Colas, chief market strategist at brokerage Convergex.

Related: Relax. We are not heading for a global meltdown

Wells Fargo (WFC), JPMorgan Chase, and Citigroup (C) are all sitting on billions in loans to the energy sector. They're setting funds aside to cover expected losses.

3. Regulations and fines

Many banks are also still struggling with the fallout of bad behavior. Fines and settlements are piling up.

Banks are now also facing tougher regulations, which are forcing them to scale back their investment banking businesses.

They are required to set aside more capital for rainy days, and are banned from trading with their own money -- so-called proprietary trading.

The new rules were designed to make banks safer and prevent future crises, but the banks say they are also making them less profitable.

BNP Paribas, Credit Suisse, Deutsche Bank and Barclays all recently announced a shift away from investment banking, in an attempt to cut costs.