That's the message from a number of fund managers and analysts ahead of Friday, when JPMorgan Chase(JPM) and Wells Fargo(WFC) report first quarter results. Citigroup(C), Bank of America(BAC), Goldman Sachs(GS), and Morgan Stanley(MS) will report their earnings next week.
It's a very different story than the end of 2013, when many investment strategists predicted that a vigorously improving economy would translate into robust bank profits.
"The group is very economically sensitive," said Peter Kovalski, who manages the $64 million Alpine Financial Services Fund.
What are the possible drags?
If weather really put a damper on the economy in the first half of the quarter, bank profits could take a hit if there was a slowdown in demand for new consumer loans.
And still rock-bottom interest rates seemed to do the banks more harm than good. Higher rates would mean more opportunities for banks to boost returns from loans.
While near-zero interest rates helped propel a mortgage boom in recent years, particularly for refinancings, Kovalski said that trend seems to be over.
The Wall Street firms could also suffer from lower trading revenue. Ultra-low rates most likely led to a continued slump in bond trading revenue in the first quarter.
Commodities trading has also taken a hit and isn't likely to turn around soon given the slowdown in demand from China and other developing nations.
But even though the stock market ended the quarter flat, revenue from stock trading on behalf of institutional clients was decent, said Marty Mosby a bank analyst with Guggenheim Securities.
Still, Wall Street's trading engine isn't as what it used to be since the financial crisis.
Once fruitful businesses such as proprietary trading -- trading with banks' own money for profit -- have been mostly shuttered thanks to tighter federal regulation of financial companies.
Michael Lewis: Trading 'should be cheaper'
And even though the mega legal settlements paid out by banks in the first quarter for their behavior during the housing bubble may seem like a negative, that's not necessarily the case. Instead, the settlements are a sign that the banks are finally putting the past behind them.
"Investors have gotten comfortable that the settlements will be covered," Mosby said. "Banks are getting to a place where capital positions are pretty strong so from an investor standpoint, it really has a minimal impact."
What to look for from the "Big Six"
For JPMorgan, analysts are cautiously optimistic that the nation's largest bank by assets can move past its legal problems. The stock is flat for 2014, but is still up 20% over the past year, a time period that included numerous multi-billion dollar settlements, including a record fine of $13.5 billion in November over mortgage-backed securities.
Investors seem pleased with Wells Fargo, which is up over 7% year-to-date. Even with a slowdown in mortgage originations, Wells has a healthy mortgage servicing business, and many analysts, including Mosby, are bullish on its prospects.
Citigroup hasn't had an easy go as of late. Its shares tumbled in recent weeks after the bank announced that the Fed rejected its dividend and share repurchase plan last month. Still, Patrick Kaser, a fund manager with Brandywine Global, said the stock could bounce back if there's a rebound in emerging markets, where Citi has a big consumer banking presence.
Analysts appear to be viewing Bank of America favorably. The firm has been following through on various cost cutting measures and its shares are up over 5% so far in 2014.
Goldman Sachs shares have taken a beating though. They are down around 12% this year. The bank is still considered the crown jewel of Wall Street, but Goldman Sachs has come to rely more heavily on fees from investment banking as many of its lucrative trading operations have slimmed down due to the tougher regulations.
Meanwhile, analysts say Morgan Stanley has done a good job building up its wealth management arm, which is more stable than other businesses that are more subject to market volatility.
But despite the overall gloom about first quarter bank earnings, there are encouraging signs for the future.