NEW YORK (CNNMoney.com) - Consumer confidence fell to its lowest since after last year's hurricanes in May, hurt by higher gasoline prices and concerns about the labor market, a report showed Tuesday.

But some economists said the news wasn't all bad, noting that despite the drop in confidence consumers seem to be adjusting to higher gas prices. And they noted that the job market remains relatively strong, and that wage growth has picked up.

The Conference Board said its index of consumer sentiment slipped to 103.2, from an upwardly revised reading of 109.8 in April. Economists polled by Briefing.com on average had forecast that the index would likely fall to 100.9.

"Consumer confidence, which reached a four-year high in April, lost ground in May," Lynn Franco, director of the Conference Board Consumer Research Center, said in the report.

"Apprehension about the short-term outlook for the economy, the labor market and consumers' earning potential has driven the Index down to levels not seen since the aftermath of the hurricanes last summer."

Consumers indicated some concern about their job prospects. Those saying jobs are "plentiful" decreased to 28.6 percent from 29.4 percent in April, while those claiming jobs are "hard to get" increased to 20.5 percent from 19.7 percent, the report said.

Looking ahead, Franco said the Index bears close watching, "as it is often a harbinger of times to come." According to the report, those consumers expecting business conditions to worsen increased to 13.2 percent from 9.3 percent in April.

The outlook for the labor market was also less optimistic. Those expecting more jobs to become available in the coming months decreased to 14.6 percent from 15.4 percent in April while the proportion of consumers anticipating their incomes to increase in the months ahead declined to 16.6 percent from 18.0 percent in April.

Confidence is watched closely since consumer spending accounts for nearly 70 percent of the nation's economy, though some analysts say sentiment and spending aren't as closely correlated as they once were.

Many economists also look to Wal-Mart (Research) as a barometer of consumer spending power given its status as the world's largest retailer, averaging more than 100 million shoppers a week. Over the weekend, Wal-Mart said higher gas and utility prices were starting to take a toll on its customers.

At the same time, the fact that the May decline in consumer confidence was smaller-than-expected could also mean that consumers are beginning to price in the impact of the escalating gas and energy prices on their household budgets, said Steve Cochrane, economist with Moody's Economy.com.

"Certainly you don't want to dismiss the drop but it's not particularly negative," Cochrane said. "Some individuals are factoring in the longer-term effect of paying more at the pump and they will adjust their budgets, paring back here and there to accommodate for gas."

Even so, in this environment Cochrane said there are two key reasons to be very concerned about households at the lower-end of the income strata.

"Low-income households have no wiggle room in their budgets," he said. "These consumers may get squeezed further if they've taken out adjustable home mortgages because monthly mortgages payments are expected to rise in the second-half of the year."

Michael Niemira, chief economist and director of research for the International Council of Shopping Centers, also described the May report as "good news and bad news."

The good news, he said, is that at 103.2, that number is still higher that the Index's long-term historical average of 97.2. "The bad news is in the direction it's currently heading," Niemira said.

"Consumers have been adjusting their budgets for some time already," he said. " We recently surveyed consumers asking them how gas prices have impacted their discretionary spending power. A third of them said they were cutting back on eating out. So the paring back hasn't yet hit the demand for goods."

What is the threshold where consumers will start to retrench further?

"It's typically not the case until there's a recession," Niemira said. "I would argue that income growth is far more important to watch right now. The average weekly wage growth has been a lot faster this year versus a year ago. So consumers are benefiting from expansion in the economy and this should help to offset the energy and gas price worries."

----------

University of Michigan index shows weakness in consumer sentiment. Click here for more.

For more economic news, click here.