NEW YORK (CNN/Money) - The U.S. economy is in the early stages of another sugar high, fueled by tax cuts, low interest rates and mortgage refinancing.

But while those very same forces drove consumer spending throughout 2003, many economists worry the buzz will wear off by second half of this year unless job growth picks up.

Consider first the impact of the tax cuts.

Personal income tax refund checks are higher this tax season, thanks to the residual effects of last year's tax cut, as many businesses withheld federal taxes at the old, higher rates through the second half of 2003.

Even so, refunds have been running far below forecasts. Through March 18, the government had paid about $95.7 billion in refunds, compared with about $86.4 billion at the same point in 2003, a gain of 10.7 percent.

The Treasury Department and many private forecasters had expected a gain of about 30 percent, a boost that would have put an extra $56 billion or so in consumers' pockets. At the current rate, refunds will be only about $20 billion higher than last year, or about 0.2 percent of GDP.

Some economists believe the extra surge of refunds is simply lagging, in part because wealthier people -- who got most of the benefit of last year's tax cut -- have more complicated returns and are taking longer to file.

"I think we'll see a clear acceleration of refund payments as we get closer to the April deadline, when the checks really start to flow," said Anthony Chan, chief economist at Banc One Investment Advisors. "People tend to procrastinate."

Another boost -- almost equally as large as that of the refund checks -- could come when people who aren't getting refunds realize their tax payments are lower than they were last year, Chan and other economists say.

Regardless, the impact of lower tax bills on the economy will depend on how much of that money people actually spend -- in question as consumers feel the pinch of more expensive gas prices.

"These are not incredibly large numbers," said Jim O'Sullivan, senior economist at UBS. "Along with the lower tax payments, refunds could add a half-percentage-point of growth to GDP in the first half of the year -- not a dramatic effect, but a bit of a plus."

Pundits have been predicting the decline of the housing market since at least 2002, when an economic recovery was expected to push interest rates higher, sapping demand for mortgages.

Instead, interest rates fell to new record lows in 2003, triggering a $2.5-trillion boom in mortgage refinancing, according to National Association of Realtors chief economist David Lereah.

Related stories Mortgage rates remain low Fed on hold again What are we worried about? Time to jack up interest rates? Payrolls disappoint again

About $140 billion of that refi boom came in the form of cold, hard cash, much of which consumers put back into the economy. Though such numbers are likely unattainable this year, mortgage rates have fallen again lately, keeping the refi boom going a little while longer; Lereah expects about $40 billion in cash-out refis this year.

"Housing will still contribute to the overall economy in a positive way and will help consumer spending," Lereah said.

But most economists expect interest rates to rise gradually this year, dulling housing's effect on the economy.

Some also worry that rising rates could end up pinching those consumers who took advantage of low rates to go deeper into debt -- especially those who spent all the money from their cash-out refis.

"People used to build wealth through building equity in their homes," Merrill Lynch chief U.S. strategist Richard Bernstein, a vocal bear, wrote in a research note on Monday. "Today, people prefer to speculate on the price of their house by using historically high levels of leverage."

YOUR E-MAIL ALERTS Follow the news that matters to you. Create your own alert to be notified on topics you're interested in. Or, visit Popular Alerts for suggestions. Manage alerts | What is this?

Most economists still don't worry much about consumer debt, since many people have locked in low rates, and they believe the labor market will get even stronger as the year goes on, pushing wages higher.

In a research note last week, Goldman Sachs said the withdrawal of temporary stimuli could slow the annualized pace of GDP growth down to about 3 percent in the second half of the year.

That's not exactly economic apocalypse -- it's actually close to the long-term trend rate of growth -- but the lack of extra stimulus will mean the economy could be even more vulnerable if jobs don't start growing soon.

"Consumer spending is likely to become much more dependent on jobs and confidence by the third quarter," Scott Hoyt, director of consumer services at Economy.com, wrote in a note last week. "If labor markets have not turned, boosting confidence by then, the risk of a significant slowing in consumer spending will be very high."