The task was unlike any AT&T's workers had seen before.
Restoring service to a critical cell site on top of a 22-story building with no power in a flood zone presented a big problem: Building codes prevented on-site generators from being installed on the roof, and there was no hope of getting power restored anytime soon.
But 2 Wall Street in New York's financial district, whose cell site serves the New York Stock Exchange, needed to be brought back online quickly after Superstorm Sandy knocked out power everywhere in Manhattan below 39th Street.
"We could prepare for almost every possible situation, but this was unique," said Tom DeVito, AT&T's general manager of the New York and New Jersey region.
The solution DeVito and AT&T's engineers devised to restore service was to bring a gas-powered generator to the sidewalk and snake a 3,000-foot power cord from the generator, up the staircase and to the cell tower. It was a shoestring and bubblegum approach, but it was successful.
Related story: How ConEd turned New York City's lights back on
There are countless stories of makeshift solutions similar to 2 Wall Street in the wake of Sandy. AT&T (T) carried thousands of feet of cable up 180 Water Street's 26 stories to restore service there. And the nation's second-largest wireless network rolled out temporary "Cells on Wheels" at food and beverage stations set up by the mayor's office around the city, so that people could charge their smartphones and make calls while getting a bite to eat.
About 11,000 AT&T engineers were deployed, working 10 to 12 hour days without a day off for about two weeks to restore service to the region.
"This was one of the biggest mobilization efforts in AT&T's history," DeVito said. "This truly was an unprecedented storm."
AT&T's own New York headquarters were flooded, forcing the company to set up a makeshift command center in the back of its flagship retail store in Times Square. A conference room in the back of the store became DeVito's "war room," where the region's two dozen engineering managers devised a triage process to get the area's most cell sites back online.
Of course, AT&T is not the only telecom that had to undergo herculean efforts to restore cell phone service in New York, New Jersey and Connecticut. But Verizon (VZ) and Sprint (S) declined requests to talk about their post-Sandy operations.
Related story: Verizon says Sandy could have 'significant' impact on profit
City Hall, the New York Federal Reserve and the New York Stock Exchange were among the first sites to get cell service restored. Then, DeVito and his team labeled the remaining off-line cell sites with an A, B, C or D, prioritizing sites by the ease of their restoration, the number of people served by each tower, and the different characteristics of each site.
Every cell site in the world is unique in some way, but that's particularly true in New York, where building codes, height, access and other factors can vary tremendously. Some had generators that needed to be fueled. Others had air conditioning units that needed to be restored. Others had battery backups that were failing. Each of the cell sites in the area needed its own particular plan.
There are some exceptions in the hardest-hit regions, but practically every AT&T cell site in the storm area has now been restored.
AT&T said it has put several redundancies in place to ensure that its network doesn't become too overloaded again when a number of sites go down, yet DeVito said AT&T was probably as prepared as it could have been. Still, he conceded that Ma Bell may need to continue to harden its infrastructure in the wake of a continuing trend of severe weather.
"The storm of century seems to be happening every year now; that does not go unnoticed," DeVito noted. "When we see risk, we have to plan for the worst and hope for the best."
But there are times when all the planning of the world just can't overcome Mother Nature.
