(FORTUNE Magazine) – When oil hits $40 a barrel, as it did in mid-May, alarm bells start to go off. After all, the last time oil jumped over $40 was in 1990, during the first Gulf war, after which the economy suffered a prolonged slump. And during the oil shocks of the 1970s, oil was well above $40 a barrel (in today's dollars) for five long years.

So let's just cut to the chase: How worried should we be about oil prices derailing this recovery?

Judging purely from economic models, the answer is not all that much. According to the Federal Reserve, a $10-a-barrel rise in the price of oil, if sustained over a year or so, would reduce GDP by a mere 0.2%. That's as close to insignificant as it comes, particularly when the economy is growing at a rate of 4% or better. And at this stage prices aren't a lot higher than they were last spring when they peaked near $38 a barrel.

Still, companies and consumers are starting to feel the pinch. While industries like the airlines and trucking have significant exposure to oil prices and are starting to pass on the price hike to consumers, virtually every firm in the economy is affected to some degree by higher oil prices. (think of heating bills, shipping costs, or travel). Wal-Mart chief Lee Scott recently raised concern that higher prices at the pump would reduce his customers' ability to spend. Residents in Santa Barbara are so distressed by gas staying above $2 a gallon that they want Arnold Schwarzenegger to declare a state of emergency. And higher gas prices act like an instant tax increase. Economists use a rule of thumb that goes like this: For every penny per gallon that gas prices rise, $1 billion gets sucked out of the economy. Consumers were expecting $15 billion to $20 billion back in their pockets from tax cuts this April; gas meanwhile has risen $0.35 since the start of the year, a cost of $35 billion.

Even more worrisome, though, are two rather major wildcards--war and psychology--that can't be factored into any Fed model. First off, we're in the middle of a war in Iraq, the world's supply of oil is very tight, and everyone is jittery about a possible disruption. In early May extremists targeted a pipeline in Karbala, Iraq, stopping the daily flow of about one-quarter of that country's oil production.

Then there's the other piece of the puzzle: the psychological effect of higher oil prices. At some point, rising crude prices could change the way companies and consumers behave, and that could significantly increase inflationary expectations and inflation itself. In other words, if everyone expects prices to keep rising, they may buy as much gas as they can today, and bid up prices--then, presto, there's a panic. If that happens, $40 a barrel may look like the good old days.

--Anna Bernasek