U.S. stocks closed lower Thursday, after investors fretted that the Federal Reserve might end its intervention in the markets sooner than expected.

The minutes from the Fed's December meeting showed some members were weighing whether the central bank should wrap up its bond buying program before the end of this year.

"People thought bond buying would go through 2013 at least," said Douglas DiPietro, managing director at Evercore. "Now it looks like some members of the committee are getting more cautionary about that."

In mid-December, the Fed announced plans to expand its controversial stimulus program and said it would keep accommodative policies in place until the unemployment rate falls to 6.5% or inflation exceeds 2.5% a year.

Stocks had bounced around the breakeven line for most of the day, but moved solidly lower after the Fed minutes were released.

The Dow Jones industrial average, the S&P 500 and the Nasdaq lost between 0.2% and 0.4%.

Related: Wall Street's biggest headwind: Washington

Investors had already lost the exuberance that dictated stock buying on the first trading day of the year. All three stock indexes shot up more than 2.5% Wednesday after lawmakers reached a last minute deal to avert the so-called fiscal cliff.

Now investors are turning their attention to the prospect of more wrangling over the U.S. budget in the weeks to come.

Bank stocks, which were among the biggest gainers on Wednesday, mostly traded lower Thursday. Goldman Sachs (GS), Morgan Stanley (MS), Wells Fargo (WFC) Bank of America (BAC) and JPMorgan Chase (JPM) dropped between 0.5% and 1%. Only Citigroup (C) closed higher.

On Thursday, Investors also got a fresh snapshot of the health of the labor market. The private sector added 215,000 jobs in December, according to ADP, while the government reported first-time jobless claims rose 10,000 to 372,000 in the latest week.

Still those jobs reports are simply a prelude to the government's closely watched monthly labor report, due Friday morning. Economists surveyed by Briefing.com expect job growth to have continued at a modest pace in December, with employers adding 150,000 jobs.

In corporate news, Family Dollar (FDO) shares plunged after the discount retailer reported earnings that missed forecasts and issued a weak outlook.

Shares of Gap (GPS) moved higher, after the retailer reported stronger-than-expected same store sales and announced a new $1 billion stock buyback.

Limited (LTD) shares slid after the owner of Victoria's Secret said same store sales fell short of forecasts.

Shares of spam maker Hormel (HRL) rallied after the company announced it was paying $700 million to acquire the Skippy brand from Unilever (UL).

Shares of SunPower (SPWR) spiked after Warren Buffett's Berkshire Hathaway (BRKA) invested between $2 billion and $2.5 billion in two of the solar company's projects.

Related: Fear & Greed Index shows greed on the rise

European markets closed in mixed territory after posting gains of 2% Wednesday, while Asian markets ended higher. Hong Kong's Hang Seng advanced 0.4%, while the Australia ASX All Ordinaries index added 0.8%. Markets in Tokyo and Shanghai were closed for an extended New Year's holiday.

The dollar rose against the euro and the British pound, but lost ground versus the Japanese yen.

Oil prices and gold prices edged lower.

The price on the 10-year Treasury fell, pushing the yield up to 1.91% from 1.84% late Wednesday.