(FORTUNE Magazine) – Three years ago energy problems appeared to be the economic equivalent of smallpox--a once frightful scourge that had been eradicated. By late 1998 the price of oil had tumbled to just $10 a barrel, and Americans were falling all over themselves in their rush to buy energy-hogging SUVs and McMansions. Who could blame them? The Economist proclaimed we were "drowning in oil" and predicted $5-a-barrel prices before long. Meanwhile, petroleum giants like Royal Dutch/Shell based their five-year business plans on oil's remaining below $14 a barrel. And a U.S. Department of Energy official confidently told a Senate committee that prices were unlikely ever to return to their historic average of $17.

Oops. In February 1999 the price of oil suddenly began to climb--and climb and climb--and it didn't stop until last October. By then, oil prices had almost quadrupled, hitting $37 a barrel, by far the most dramatic run-up since, well, Jimmy Carter sidled up to the camera in the Oval Office wearing that cardigan. Analysts termed the recent spike a blip and said it wouldn't last. Some blip, eh? Prices have abated from their peak but remain a lot closer to $30 a barrel than to $20. And at that level energy costs are having an impact on the already wobbly U.S. economy. Consumer spending on energy has risen from $250 billion to $350 billion in just two years. That's equivalent to a $100 billion tax on consumers, vaporizing around 1% of GDP. "Expensive oil has been a brake on world growth since mid-2000, and it's continuing to hold us back," says David Malpass, Bear Stearns' global economist.

So what happened? Basically, the legions of experts who make calculations of the millions of barrels of oil used each day overlooked one itty-bitty thing--OPEC. Without a lot of fanfare, the Organization of the Petroleum Exporting Countries appears to have transformed itself from a klutzy cartel into a surprisingly savvy market presence. Over the past two years or so, OPEC members have looked like a veteran dance troupe, moving in step to control prices and production.

It's easy to understand why no one was taking OPEC seriously. Although it controls 75% of all proven reserves and supplies roughly 40% of the world's oil, the organization has been about as functional as the Bundy family on Married...With Children. For the past two decades OPEC was beset by woes ranging from political subterfuge to outright war (Iran and Iraq). Attempts at reform were a joke. For example, in the mid-1980s, members were desperate to stamp out cheating and hired an international accounting firm to police quotas--but then individual countries refused to provide the firm one iota of information.

Three factors have helped OPEC pump itself up into a muscular market force. First, the pain inflicted by low oil prices in the mid-1990s galvanized the group. In an unprecedented move beginning in 1996, archrivals Saudi Arabia and Iran formed an alliance that remains the cornerstone of the new OPEC. Their truce means that the extremes of OPEC are cooperating--Saudi Arabia, the group's most powerful and moderate force, and Iran, the political renegade. "Too many people are looking at OPEC through the rear-view mirror," says Matt Simmons of Simmons & Co., a Houston-based energy investment bank. "There's resolve in their eyes never to go back to the days of cheap oil."

A second critical factor was the election of Hugo Chavez as President of Venezuela in early 1999. Until that time, Venezuela had been trying to win market share from Saudi Arabia in the U.S. It was a nasty battle that resulted in both countries' flooding the market just as the Asian crisis hit and demand around the world instantly dried up. On Feb. 2, 1999, Chavez and Saudi Arabia's Minister for Oil met, and Chavez agreed to cut back Venezuela's daily production by 500,000 barrels. From that day, oil prices began to rise. "OPEC has entered the age of reason," says Raja Sidawi, chairman of the Energy Intelligence Group. "It's evolved into an efficient management system."

The third factor is Team OPEC's new price-control mechanism, introduced by member countries last year. Using a basket of many different grades of crude oil, plus estimates for world supply and demand, OPEC applies a formula to keep oil within a designated band of $22 to $28 a barrel. Instead of meeting once a quarter in a highly charged political atmosphere to debate what to do, OPEC members now gather almost monthly for a technical meeting. In effect, the new formula has taken the politics out of OPEC and introduced an objective, scientific rule of thumb. In the past two years the cartel has changed its production levels at least a dozen times, representing perhaps its longest period ever of effective cooperation. And the strong oil price only serves to reinforce OPEC's discipline.

Not everyone is convinced that the cartel has turned itself into a management team that even Jack Welch could admire. In fact, OPEC has yet to face a true test of its resolve--and believe us, it will come soon enough. Although OPEC's newfound spirit of cooperation has brought prices back, right now the cartel is operating close to full capacity. That means even if countries wanted to cheat on their quotas, they couldn't. Some analysts think that's the main reason OPEC seems so effective today. "If OPEC had any extra capacity, they'd be cheating, and oil would be at $20 a barrel," says Amy Myers Jaffe, senior energy analyst at Rice University's Baker Institute for Public Policy. Jaffe and others believe OPEC's determination will be challenged when additional capacity comes online. In the past decade, low oil prices discouraged investment throughout the industry. Both OPEC and non-OPEC producers abandoned exploration and drilling plans, and global capacity stagnated. Now that prices have recovered, investment is picking up. But there tends to be a lag. Jaffe calculates that it usually takes two years for new drilling to result in additional supply.

While supply is increasing, demand for oil may be softening. The economic slowdown in Japan and Asia is already well under way, and Europe is weakening too. Analysts predict that demand for oil, rather than continuing to grow at between 2% and 3%, will be flat next year and well into 2003. "OPEC will have a more difficult time in 2002 to maintain discipline," says Jaffe. "The higher the price stays now, the worse it will be." That's because high prices unleash forces that work to lower the price over time by encouraging development of the oil supply and discouraging demand. Faced with softer demand and greater supply, will OPEC hang tough and cut back its production even more to keep the price up?

In the past cartels have tended to exert market power for brief periods before falling apart. And indeed, OPEC has a delicate balancing act. It has to find the price that benefits member countries yet at the same time doesn't hurt world demand or encourage too much supply.

Many analysts believe it's only a matter of time before the new OPEC unravels. According to Bear Stearns' oil analyst Frederick Leuffer, prices will drop as low as $18 a barrel by next year. He points out that the only time that oil has been above $17 a barrel in the past 60 years has been when there was a war involving at least one OPEC country or a member facing political strife or embargoes. Today none of those conditions exist, yet the price remains near $30. For Leuffer, that means the current situation is a fluke that will soon correct itself.

For others this unprecedented period in OPEC's history is proof of something new at work. "They know what their job is now, and they're doing it," says Sidawi. Analysts like Sidawi argue that OPEC is more sophisticated now; it looks ahead at the global demand-and-supply issues and plans accordingly. And they argue that as long as the political situation remains stable in Saudi Arabia, the driving force behind the new OPEC, its price-band mechanism will help the cartel adjust to any new market conditions. The fact that a new U.S. Administration appears more open to higher oil prices doesn't hurt either. "Bush and OPEC are on the same page," says Seth Dunn, an oil analyst at the Worldwatch Institute. "There's less of a tendency for Bush to be overly antagonistic to OPEC."

The good news for the rest of us is that even if OPEC's success continues, prices are expected to come down somewhat in the near term from $28 a barrel. Most analysts believe an average price of around $25 a barrel may be more typical as the world economy slows. The futures market, too, is predicting lower prices a year from now. The June 2002 contract for West Texas Intermediate oil is priced at $26. (A caveat: The futures market has consistently underestimated the actual price of oil during the past two years.)

But prices almost certainly will continue to fluctuate. Indeed, analysts like Steve Strongin at Goldman Sachs reckon that oil is now three times as volatile as it was in the past. The main reason: Problems with storage and transport--the result of chronically low investment--can create disruptions that lead to wild price swings. Even a born-again OPEC can't do much about that.

But if the cartel has really gotten its act together, average prices in the longer term still aren't likely to dip much below $20. Sure, that's cheap--but a long way from the $10 a barrel most of us took for granted only three years ago.

FEEDBACK: abernasek@fortunemail.com