President Obama said he spoke with Summers earlier Sunday and accepted his decision.
"I will always be grateful to Larry for his tireless work and service on behalf of his country, and I look forward to continuing to seek his guidance and counsel in the future," the president said in a statement.
Current Fed Chairman Ben Bernanke's term is set to end in January. Before the appointment can be made official, the Senate is tasked with holding confirmation hearings and voting on the nomination.
While Summers is no stranger to high-profile roles and has a close relationship with Obama, forged in the depths of the financial crisis, he would have been a controversial pick.
During his time as Treasury Secretary in the Clinton administration, Summers earned his reputation as a deregulator. Summers urged Congress to repeal parts of a Depression-era rule called Glass-Steagall, which prevented financial institutions from simultaneously acting as investment banks, commercial banks and insurance companies. This move paved the way for banks to become larger than ever.
Also in the late '90s, Summers pushed back against a movement to regulate complex derivatives. Some say, both those decisions played a role in precipitating the 2008 financial crisis.
If Obama had picked Summers, one of his economic mentors, the confirmation hearings could have been dicey. In July, a group of Democratic Senators sent a letter to Obama urging him to choose top contender Janet Yellen, who currently serves as vice chair at the Fed.
"Our nation badly needs a chairman with a solid record as a bank regulator," the letter said.
In his letter to the president announcing his withdrawal, Summers wrote that he "reluctantly concluded" that the confirmation process would be "acrimonious and not serve the interests of the Federal Reserve, the Administration, or ultimately, the interests of the nation's ongoing economic recovery."
Summers served as President of Harvard, and more recently, headed Obama's National Economic Council. But he lacked monetary policy experience in comparison with Yellen. According to CNNMoney surveys, Yellen is by far the preferred candidate among economists due to her years of Fed experience. If appointed, she would be the first woman to lead the organization in its 100-year history.
Summers' withdrawal is expected to delay the president from announcing a nominee at least for another week.
The Fed leadership transition comes at a critical time.
The U.S. central bank has indicated it is ready to wind down its economic stimulus, via its $85-billion-a-month bond-buying program.
Further down the road, the new Fed chairman will also have to grapple with the complicated task of unwinding five years of monetary stimulus. Since late 2008, the Fed has accumulated more than $3 trillion in Treasuries and mortgage-backed securities, in its bid to support the U.S. economy.
Outgoing chairman Bernanke leaves behind a legacy of bank bailouts and unprecedented monetary stimulus that some economists argue prevented a total collapse of the U.S. economy.
He also leaves the Fed a more transparent institution. Under his leadership, the central bank began holding press conferences to explain its policies to the public. It also adopted a policy known as "forward guidance," in which the Fed alerts the public of its expectation for future interest rates.
The Fed has recently said it expects its key interest rate to remain near zero until 2015.