U.S. stocks ended lower Wednesday after Federal Reserve meeting minutes raised worries the central bank might scale back its bond buying program.

The Dow Jones Industrial Average fell 108 points, or 0.8%. The S&P 500 fell 1.2% and the Nasdaq lost 1.5%.

Following its January meeting, the Fed's Open Market Committee said the central bank would continue buying bonds and keep interest rates near zero until the job market improves "substantially." But the minutes from that meeting showed that some Fed officials were in favor of slowing the purchases sooner.

According to the minutes, "a number" of committee members argued that an ongoing review of the controversial program "might well lead the committee to taper or end its purchases before it judged that a substantial improvement in the outlook for the labor market had occurred."

Steven Ricchiuto, chief economist at Mizuho Securities USA, wrote in a note to clients that the minutes illustrate the Fed's "discomfort with being involved in non-conventional policy."

However, he added that the economy is probably not strong enough yet to justify the Fed ending its easing programs anytime soon.

Investors were already primed to sell stocks after the major indexes closed Tuesday at new 5-year highs. Despite the Wednesday's losses, stocks are still up between 5% and 6% so far this year.

"Investors are having a gut check, so to speak, as to whether these levels are justified," said Bruce McCain, chief investment strategist at Key Private Bank.

The latest economic data have been positive, but not strong enough to live up to investors' high expectations, he added.

A report on the housing market came in mixed. The number of new homes breaking ground in January fell from the month before, although the number of applications for new building permits rose.

The report came on the same day that homebuilder Toll Brothers (TOL) reported revenue and earnings that fell short of analysts' expectations.

Related: Bonds are riskier than stocks

A separate report on the Producer Price Index showed wholesale prices rose 0.2% in January, slightly less than economists had expected.

In the commodities market, oil prices sank 2% on rumors that a large player was being forced to sell crude futures.

"There's a lot of chatter going around that a commodity hedge fund is in trouble and unwinding positions," said David Lutz, head ETF trader at Stifel Nicolaus. But he discounted the rumors, saying that a 2% drop would not be enough to force a fund to liquidate positions. "It's unlikely that someone was long and wrong," said Lutz.

Gold prices also declined, falling below $1,600 an ounce for the first time since August 2012.

Related: Fear & Greed Index still in 'extreme greed'

On the corporate front, Office Depot (ODP) announced that it would merge with OfficeMax (OMX) in a deal valued at $1.2 billion. Shares of OfficeMax fell 10%, while Office Depot's stock sank 20%. Both stocks surged Tuesday on reports that a deal was imminent.

Mining firm BHP Billiton (BBL) said lower commodity prices and a weaker U.S. dollar in the second half of 2012 more than offset its stronger volumes. The company also announced that CEO Marius Kloppers will retire in May, and will be succeeded by Andrew Mackenzie, who previously headed the company's division specializing in non-ferrous metals.

Rio Tinto (RIO)shares fell, amid tensions surrounding its copper mine in Mongolia. The Mongolian government owns 34% of the mine and is demanding a greater share of profit.

SodaStream (SODA) shares dropped after the company's revenue forecast was slightly more conservative than analysts' had hoped.

Tesla (TSLA) reported a bigger-than-expected quarterly loss after the closing bell. Shares of the electric car maker fell in after-hours trading.

Controversial nutritional supplements company Herbalife (HLF) also announced better-than-expected earnings and sales figures Tuesday afternoon. But shares fell Wednesday after Herbalife executives did not entertain many questions on a conference call with analysts. Herbalife has been in the spotlights since well-known hedge fund manager Bill Ackman accused the company late last year of operating a pyramid scheme.

European markets ended mixed. London's FTSE (UKX) edged higher, while the DAX (DAX) in Germany and CAC (CAC40) in France fell. Asian markets ended higher. The Shanghai Composite advanced 0.6%, the Hang Seng added 0.7% and Japan's Nikkei gained 0.8%.

Related: Check gas prices in your state

Gas prices rose for the 34th day in a row, fueling concerns about whether higher prices at the pump will take a bite out of consumer spending in February. Nationwide, a gallon of unleaded gasoline averaged $3.766 Wednesday, according to AAA. Gas prices have risen 34 cents, or 10%, since the beginning of February.

The price on the 10-year Treasury stayed steady with its yield at 2.03%.

The U.S. dollar fell against the euro but gained against the British pound and the Japanese yen.