Billionaire hedge fund managers slammed the White House's handling of the Federal Reserve chair nomination process thus far and are disappointed that Larry Summers won't get the job.

"I think the process, specifically talking about Larry Summers and the decision not to move forward with him as Fed chairman, is horrendous," said Glenn Dubin, CEO and chairman of Highbridge Capital, at the Bloomberg Markets 50 Summit in New York Tuesday.

Dubin went on to say he thinks Summers, who pulled out of the Fed chair race earlier this month due to the likelihood of a contentious Senate confirmation process, would have made a "tremendous" Fed chief.

Fellow billionaire hedge fund manager Mark Lasry, CEO of Avenue Capital Group, echoed Dubin's sentiments about Summers, and added that "it is sad that he had to pull out."

Ben Bernanke's term as Fed chair ends in January. Bernanke has not commented on his future plans. But it is has been widely reported that he does not want to return and that President Obama is also interested in appointing a new Fed chair.

Related: Warren Buffett's pick for Federal Reserve chair is...

But not all hedge fund managers were upset with the withdrawal of Summers. Bruce Richards, CEO of Marathon Asset Management, noted that "it would have been worse for [Obama] to stay with Summers and then watch him get rejected" during the Senate confirmation process.

"If you're the president, you go with what you need to get through," said Richards, adding that Obama has plenty of other battles to fight, including a looming government shutdown and the rapidly approaching federal debt limit.

Janet Yellen, who currently serves as vice chair at the Fed, is now viewed as the only viable choice to succeed Bernanke. Yellen has already won support from key Senate Democrats and more than 450 economists, who submitted letters urging President Obama to nominate her over Summers for the Fed's top job.

Richards added that he thinks Yellen is "awesome" and will make a better chairman than Summers or Donald Kohn, a former Fed vice-chairman who has also been mentioned as a dark horse candidate for the job.

If Yellen is nominated and confirmed, she would be the first female chair in the history of the Federal Reserve. Currently, just 17 of 177 central bank leaders around the world are women, according to the Central Bank Directory.

The lack of women at the head of central banks has made gender a hot topic in the Fed chair discussions. It inevitably came up during the conversation among hedge fund managers.

Though he seemed to favor Yellen, Richards referred to her as "this woman," only to get called out by Bloomberg anchor Stephanie Ruhle, who was moderating the panel.

Richards corrected himself and said "this person" But the slip-ups didn't end there.

Dubin, while expressing disenchantment with the political process, said "the president felt that Larry Summers was the right man for the job."

He ultimately concluded "all three candidates, including the lady, would make a great Fed chairman."