The stock market pulled back a bit Thursday, but the Dow, S&P 500 and Nasdaq are still not far from their all-time highs.

It seems safe to say that the Trump trade is still on. Wall Street remains hopeful about the potential for tax reform, deregulation and economic stimulus from President Trump -- even if the timing for all of this could get pushed to later this year, or early 2018.

But many stocks have been left out of the Trump rally. As of the closing bell on Thursday, there were 168 companies in the S&P 500 whose stock prices have declined since the inauguration on January 20.

And many of these companies are in sectors that were expected to be big beneficiaries of Trump policies, particularly oil and finance.

Energy companies Transocean (RIG), Range Resources (RRC), Anadarko Petroleum (APC) and Helmerich & Payne (HP) have all plunged more than 25% after Trump took office. That makes them among the worst 10 performers in the S&P 500 since late January.

Noble Energy (NBL), Halliburton (HAL), Devon Energy (DVN), Schlumberger (SLB) and Marathon Oil (MRO) have all dropped by more than 15% too.

Related: Comey firing poses new risk to Trump rally

Banks have also been left out of the big market rally. Shares of Capital One (COF), Discover Financial Services (DFS), Fifth Third (FITB), MetLife (MET) and scandal-plagued Wells Fargo (WFC) have all fallen.

So have shares of Goldman Sachs (GS), the alma mater of several key members on Trump's financial team -- most notably Treasury Secretary Steven Mnuchin and National Economic Council director Gary Cohn.

Other companies that have surprisingly lagged? Real estate firms. Trump, after all, made his name (and his wealth) from being a landlord.

But having one of their own in the Oval Office has done little good for real estate investment trusts. There are 14 REITs in the S&P 500 whose stock prices have fallen since January 20.

Several mining companies have fallen too, even though many on Wall Street predicted that Trump's inflationary policies -- and unpredictability -- could be good for gold.

Related: Retail's train wreck continues

Trump also hasn't done any wonders for retailers, despite a surge in consumer confidence since his election.

Bed Bath & Beyond (BBBY), Kohl's (KSS), Victoria's Secret owner L Brands (LB), Macy's (M) and Target (TGT) have all fallen in the past few months. But a lot of that has to do with the shifting landscape in retail and the dominance of Amazon (AMZN) and Walmart (WMT).

Media companies aren't benefiting either, even though Trump's ability to constantly generate headlines has been great for clicks and ratings.

Shares of CBS (CBS), Comedy Central owner Viacom (VIAB) and Trump's favorite media company Fox (FOXF) have all fallen since he took office.

For now, the broader market is still holding up well -- thanks in large part to strong results from tech leaders like Apple (AAPL), Google parent company Alphabet (GOOGL), Amazon (AMZN) and Microsoft (MSFT).

The S&P 500 has gained more than 5% since Trump moved into the White House.

Related: Wall Street wants tax reform more than anything else from Trump

But Craig Sterling, head of US equity research for Pioneer Investments, said that investors need to keep in mind that other parts of the market could pull back too if Trump is unable to push any of his economic proposals through Congress.

And the longer that the James Comey firing and other circus-like aspects of the White House dominate the news cycle, the tougher it will be for Trump to actually get anything meaningful accomplished.

"The Trump trade was ahead of itself," Sterling said. "The market priced in a near certainty of lower taxes, less regulation and higher economic growth."

But Wall Street is quickly learning, like the rest of America, that nothing is certain in Washington.