Research in Motion founder Mike Lazaridis told customers: "We've let many of you down."
NEW YORK (CNNMoney) -- RIM's BlackBerry service was fully restored around the world early Thursday morning after what the company called its largest-ever network disruption.
Details on the outage, which began Monday and affected millions of users around the globe, came out during a conference call Thursday morning with top executives from BlackBerry maker Research in Motion.
RIM (RIMM) founder and co-CEO Mike Lazaridis said the BlackBerry service's infrastructure "suffered a hardware error," and then the problem cascaded. A backup system "did not work the way we intended."
Lazaridis said RIM is "working with vendors to fix the particular error that occurred Monday," but he declined to name specific vendors.
The BlackBerry service outage started in Europe, the Middle East and Africa. By Tuesday it spread to South America. That night, RIM assured customers that the glitch had been identified and was "now being resolved" -- but on Wednesday it got worse as customers in the United States and Canada were hit. The outage primarily affected text messaging and Internet access, leaving some voice calling services operational.
Earlier Thursday, Lazaridis posted a taped message on the company's website, saying, "We've let many of you down."
At a teleconference on Wednesday, executives pledged that despite a huge data backlog, RIM will not drop any e-mails -- all messages will eventually be delivered.
The reps dodged several questions about make-good efforts for customers.
It's a severely black mark for RIM, which is having a rough year to say the least. Its sales are flagging, investors are agitating for a management shakeup, the stock is down almost 60% year-to-date.
But BlackBerry's user base is strong: The service had more than 70 million global subscribers as of Sept. 15, a whopping 40% growth over the previous year. It's not a good moment to alienate them.
"The timing couldn't be worse," research firm Trefis wrote in a note to subscribers, citing increased competition from Google's (GOOG, Fortune 500) Android phones and Apple's (AAPL, Fortune 500) iPhone, whose new 4S iteration goes on sale Friday.
The disruptions also give ammunition to disgruntled investors, who want to remove and replace RIM's board. Small activist shareholders including Canada's Jaguar Financial Corp. are pushing the company to sell itself.
Angry customers: Perhaps most damning is the effect on RIM's customer base, as the outage may shove on-the-fence customers over the edge. After all, service disruptions have been at least an annual occurrence for the past several years.
"I have been an analyst for 25 years and have watched RIM wrestle with this same outage problem time after time," Jeff Kagan, an independent telecom and technology analyst based in Atlanta, wrote in a research note.
In 2007, RIM co-CEO Jim Balsillie attributed a BlackBerry e-mail outage to a "process error" that he promised "won't happen again." February 2008 brought another outage; December 2009 brought two in less than a week. Both U.S. and international users complained of disruptions in March 2010.
"For the first time I am now saying yes... this new outage affects RIM more than in the past," Kagan wrote. "This will not be the final nail in RIM's coffin, but where in the past RIM always came back, this will give customers yet another reason to look at other devices."
Room for competition to swoop in: RIM's rivals are acutely aware of that, and they're "salivating" at this week's events, said Ramon Llamas, senior research analyst on mobile devices at IDC.
"This is supposed to be BlackBerry's bread-and-butter, so when secure messaging breaks down for an unspecified period of time, people will certainly vent their frustrations," Llamas said.
It will be tough to mollify those customers quickly, as systemic failures can be complicated to diagnose and fix.
"This is not Antennagate, where it's a device thing and you can get a plastic cover to fix it," Llamas said, referring to Apple's famous problem with its iPhone 4 hardware. "This is a system-wide and growing breakdown."
If service problems continue to plague RIM, it will undercut any positive moves on the device front. Customers won't care how good the hardware is if the phone simply won't work.
"It's a shame, because the BlackBerry 7 devices are strong," Llamas said. "But even with great phones, people will start factoring in these experiences. And they may not be soon to forget."
RIM doesn't have much room for error left. Its last quarter was grim, with sales missing forecasts and RIM's new devices drawing mixed reviews. RIM's co-CEOs assured analysts and investors that the company is in the middle of a turnaround and will show improvement in the current quarter, which ends in November.
Next week they'll have a high-profile platform for their efforts: RIM's annual U.S. developer conference, DevCon Americas, kicks off Tuesday in San Francisco. RIM plans to show off its forthcoming line of smartphones built around QNX, its next-generation operating system, and its "PlayBook 2.0" update for its struggling tablet.
But at this point, PlayBooks and QNX phones may not be enough to win over cranky users, who have stormed Twitter with grumpy #DearBlackberry messages. Next week, they'll have the chance to deliver them in-person to RIM executives.
-- CNNMoney's Aaron Smith and CNN's Katy Byron contributed to this story. 
