(FORTUNE Magazine) – Oh, what a lovely war it's been for security contractors in Iraq. Taking on the many military tasks outsourced by the stretched-thin Pentagon, they have profited from a battle-zone gold rush: an instant billion-dollar industry with a 20,000-person workforce. But now the business of war is starting to look more like the war itself: a long, hard slog.

With the transfer of power to Iraq's interim government, the rules governing this mercenary marketplace are subject to change. Payrolls and other costs have soared, while business has slowed. Contractors whose business skills haven't proven as sharp as their battle skills are in trouble.

Most of the soldiers of fortune in Iraq are not classic mercenaries who move from one war to the next. Most work for security firms founded for this war and headed by former elite troopers: Army Rangers, Deltas, Green Berets, and Navy SEALs. And the majority of those firms are classic undercapitalized startups.

Their venture capitalist, in effect, was the Coalition Provisional Authority (CPA). In May 2003 a tiny $2 million firm called Custer Battles LLC managed to land a $16 million contract to guard Baghdad International Airport. Its bankers wouldn't finance such risky business, so the CPA fronted the startup money: $2 million in cash, stuffed by partner Mike Battles into a duffel bag and transported to Lebanon, where he started hiring. The CPA's imprimatur quickly led to other jobs. Within six months Custer Battles employed more than 1,000 people and accelerated annual revenue to $75 million.

Similarly, Global Risk Strategies was a small London-based firm before it got its feet wet in Afghanistan and picked up early CPA contracts. It instantly became a big company by recruiting 500 Gurkha soldiers from Nepal and 500 former members of Fiji's army. Because labor from Third World countries was cheap (those employees might earn as much in a month as a former Green Beret made in a day), the profit margins for contracts that utilized them could be huge. For security firms that got in early, there was gold in the rubble of Iraq.

But after insurgents began to step up attacks last fall, more established firms began coming into the Iraq business theater. The U.S. had committed $18.4 billion to the rebuilding of Iraq. The worse hostilities got, the more evident it became that much of that sum--say, 10% to 20%--would be paid to the firms that guarded the rebuilders.

In came the venerable ArmorGroup of London, whose predecessor firm, Defense Systems Limited, was the first of this breed of company, founded back in 1981. Blackwater, a younger (founded in 1996) but substantial American company, stepped up its business. Corporate-security firms like Kroll, which had never seen much more hostile action than takeover battles, also arrived. It was like being a tech firm in 1998, when, if you were in the business, you had to be in Silicon Valley. If you were in the security business in 2004, you had to be in the Sunni Triangle.

The economics of the business immediately changed, as the newcomers began raiding the first movers for employees. They lured many of the firms' finest with what mercenaries respond to best: money. Standard wages for PSD (personal security detail) pros were previously running about $300 a day, according to people who know this market. Once Blackwater started recruiting for its first big job, guarding Paul Bremer, the rate shot up to $600 a day. Global Risk no longer had a lock on the market for Gurkhas, whose monthly wages rose from $800 to as high as $2,000 today.

The big firms didn't grab all the business by any means, but they squeezed the margins and exacerbated small firms' biggest problem: a shortage of people with management skills. When a business leapfrogs from $1 million to $100 million in a matter of months, it is a severe test for even a seasoned entrepreneur. These firms were being run by special-ops jocks--knuckle-draggers, in the parlance of that fraternity--and a Green Beret who knows all about securing a building may know nothing about securing credit lines. Even the can-do spirit at the core of every one of these guys could become a business liability. When clients asked if a firm could manage telecom or food service or construction, in addition to security, the answer was usually, Hell, yes.

When firms with little management infrastructure diversified into areas they had no business trying, they only added to chaos. Mike Battles says his firm did fine branching out into logistics and engineering but came a cropper in another side venture he'd rather not name. Suffice it to say that a two-week contract turned into a five-month money-losing quagmire and a lesson learned: "We said 'never again' to areas where we didn't have an existing capability."

Battles and partner Scott Custer come out of Army special ops, but they also had business experience before Iraq. Many security contractors, to their detriment, did not. "These former military professionals are very mission-oriented, but they don't fully understand the business implications of their actions," says Tom Charron, whose Sallyport Global Holdings consults with companies doing business in Iraq. "Some got in over their heads."

The military mentality also led some firms to refuse to set up joint working agreements when they had the chance to do so, in areas like procuring critical supplies. The economic advantages of working together were overridden in part by service rivalries that continue into the private sector. Security firms are often founded on buddy networks out of a particular armed-forces branch. Blackwater is teeming with ex-Navy SEALs, starting with the two top leaders of the firm, founder Erik Prince and CEO Gary Jackson. Alums of the Army Delta Force founded Chicago-based Triple Canopy. "There are almost these clubs of former special-ops folks," says John Peters, CFO of Triple Canopy. "A business will get started and the word goes down the line, 'You need to be involved in this--you should be involved.'"

Competition keeps old rivalries burning. The firms regularly raid the others for talent and level accusations against one another about who is to blame for the latest ratcheting up of pay. The target of their ire last year was Erinys International, which won a $40 million contract to guard Iraq oil installations and then, being a startup, had to poach management from other firms to fulfill it. Now contractors are muttering darkly about a firm called Aegis Defense Services. In June this small London outfit won a $293 million contract to coordinate reconstruction security. This was a stunner, since Aegis chief Tim Spicer had slipped into obscurity for the past six years, after getting caught shipping 30 tons of arms to Sierra Leone in apparent violation of a UN weapons embargo. Security contractors were apoplectic that Tim Spicer was back and that his firm was conducting dawn raids on their people. "They're out there saying, 'How about another $50,000?'" sputters one executive. "To ramp up their business, they're ramping up the cost for everyone else."

The pattern of rivalry extends to the industry's dual capitals: London, where former members of Britain's elite Special Air Service really got the business going in the early 1980s, and Washington, where old U.S. soldiers never die--they join Beltway security contractors. The British like to view themselves as the keepers of standards. "We've had people come in whose main qualification was being a nightclub bouncer," says David Claridge, managing director of Janusian Security Risk Management. "We show them out and later find they've been employed by one of the other companies." The British also pride themselves on low-key, "defensive" security and are scornful of some of the freewheeling Americans' "offensive" brand.

In fact, this is probably the one industry in the world whose market leaders are begging for more regulation. Says ArmorGroup's chief operating officer, Christopher Beese: "Our company has had a 40-page code of ethics and a 15-page code of conduct for over 15 years; we set high standards for ourselves. But when others go around hanging out of white SUVs waving guns at people, this business needs to be regulated."

His wish has just been granted. With the transfer of power from the Coalition Provisional Authority to the Iraqi interim government, security firms must now be licensed. They must show the Iraq Ministry of the Interior that they have adequate insurance; they must submit to semiannual audits; and they must satisfy a host of other requirements to prove they're substantive, law-abiding businesses.

Will that make any difference? "I would hope so, given that the authorities now have a basis for disqualifying firms," says Richard Fenning, chief operating officer of London-based Control Risks Group. But, he adds, the circumstances and market forces may do the real winnowing out. "Lots of people have set themselves up with lifestyle-altering contracts, and lots didn't know what they were getting themselves into. As the situation has deteriorated in the last few weeks and months, you get the sense they're saying, 'This is not very much fun; I want to go home.'"

Some are looking, just as surely as the U.S., for exit strategies. Firms that staffed up excessively, anticipating that the bonanza would go on and on, are having cash-flow problems. Part of the trouble is the snail's pace of reconstruction. Only about $5 billion of the $18.4 billion has been allocated, and far less spent, slowing the expected flood of security work to a trickle. Just when contractors had learned how to manage their growth spurts, now they have to figure out how to handle reversals of fortune. Custer Battles is out as Baghdad Airport security contractor; Global Risk is in. (Mike Battles says the firm chose not to rebid.) Custer Battles's workforce is down about 300 from its peak, yet it still must pay security managers hired for construction projects that remain unstarted. "We've felt the pain," says Battles, who believes it's only temporary and who sees a silver lining. "The lull has enabled us to improve our management systems and catch up with our growth."

These firms' skimpy management systems are also a reason to fear that if the new market dynamics don't get them, the auditors in Washington will. The lush contracts of the early gold rush--hastily executed and spottily documented--are finally being closely examined in the departments of Defense, State, and elsewhere. The security firms are caught in the same dragnet as Halliburton, the alleged serial overcharger, and the two contractors implicated in the Abu Ghraib scandal, Titan Corp. and CACI International. There could be big financial, or even legal, repercussions. At a recent congressional hearing, Pentagon auditors said about $186 million has been withheld from Halliburton because of evidence it billed for 36% more meals than actually provided.

"These firms operated out of sight for a long time in Iraq; now they're coming under greater scrutiny," says Peter Singer, an expert on this sphere at the Brookings Institution. "After an Internet-like boom, the industry may be starting to see the end of the frontier of its expansion. The tectonic plates may be shifting under it, and these firms don't know what's going to happen."

FEEDBACK jhelyar@fortunemail.com