NEW YORK (CNN/Money) - With scandals hitting the headlines, regulators looming large and average returns proving lackluster, hedge fund managers are probably feeling a bit like Rodney Dangerfield this year ... no respect.

Hedge funds, which are limited to investors with a net worth of $1 million or more, are not well understood by the public because they are private pools of capital meant only for wealthy investors and not required by law to say much of anything about how they're run.

When hedge funds are covered in the mainstream press, it's usually after something bad has happened, such as a manager defrauding investors or a fund blowing up based on bad trades. This year produced a particularly sordid saga in the Bayou Group, a hedge fund whose two chief executives pleaded guilty to defrauding their investors for years.

Since they're limited to wealthy investors, why should members of the public even care? Because more and more, pension funds, which manage the retirement money of ordinary Americans, are among those sophisticated investors. (In fact, The Boston Herald reported that the pension plan of Boston's subway workers invested in Bayou).

Now, a well known brokerage firm, Lehman Brothers, is ensnared in a hedge fund drama; it is suing hedge fund Wood River, claiming the fund lured Lehman into a fraudulent stock transaction, and worried investors are having a tough time getting their money back.

Also, the industry has grown dramatically in recent years as bad performance by the broader stock and bond markets have driven investors to look for new alternatives. The number of hedge funds has spiked; there are now an estimated 8,000 hedge funds managing $1 trillion worldwide.

As more people squeeze into the markets and more managers jump on the same trades, it's getting even harder for hedge funds to gain "alpha," or excess returns above the market's gains.

"The opportunity to make money easily is not there as it was two or three years ago," said Benjamin Bornstein, who manages three hedge funds from his Newport Beach, Calif. firm Prospero Capital. "The quality of the average fund is going down because there are too many out there, it's too easy to start one and regulatory requirements aren't going to (include) the small guys."

Adding insult to injury for managers trying to defend the industry's reputation is the fact that, judging by indexes that measure hedge fund performance, this hasn't been a great year for hedge funds overall.

While some strategies that aren't correlated to the broader markets have done exceptionally well by comparison -- and indeed, the term "hedge fund" relates to an extremely broad range of investment strategies that are hard to compare to one another -- Chicago-based Hedge Fund Research, whose HFRI Index tracks the performance of 1,600 hedge funds, says hedge funds have returned 7.30 percent for the year to date, net of fees.

True, those numbers look pretty good when compared to the broad markets. The S&P 500 was only up 1.39 percent through September and is down 2.83 percent through yesterday. But for some hedge fund strategies, it's been a lean year, particularly when compared to previous years, when hedge funds trounced traditional indexes.

Joshua Rosenberg, president of Hedge Fund Research, acknowledged the tough market environment for hedge funds, adding that what's making the market environment difficult this year is "the same thing we have been seeing for the past five or six quarters -- difficult overall market conditions, combined with historically low levels of volatility, have created an environment very unfavorable to hedge funds who traditionally love volatility."

He added that the third quarter, in which hedge funds returned 5.6 percent, was the strongest quarter for hedge funds in some time.

"The traditional markets have obviously stunk; whether you're in stocks or bonds it's been a pretty uninspiring year," said Mike Hennessy, managing director at Morgan Creek Capital, a North Carolina-based fund that invests in hedge funds on clients' behalf, adding that while hedge fund returns have been "decent but uninspiring," savvy investors have been able to make money by picking the right strategies.

Hennessy pointed out that some strategies are doing extremely well this year, including energy in the U.S., as well as funds that focus on international and emerging markets.

In addition to the struggle to eke out returns in difficult conditions, managers are also facing the specter of increasing regulation. Starting in February 2006, hedge fund managers will be required to register with the Securities and Exchange Commission as investment advisers.

And with the recent uptick in high-profile fraud cases, industry watchers say that even though the incidence of fraud in the hedge fund industry is still low compared to the number of funds, regulators will be watching it more closely.

While the Bayou saga has tarnished the industry's reputation somewhat, long-time investors and managers say that there could be some benefit from the fallout, especially since that the Bayou fraud and others have revealed that the amount of due diligence taking place even at some institutional shops is not what it should be.

"Every time there is a fund that blows up, you see some really sharp people in there. I think a lot of what is happening, people say, 'so and so is in there, so that gives me comfort,'" said Morgan Creek's Hennessy, who has been investing in hedge funds for many years, through Morgan Creek and in his previous role as vice president at the University of North Carolina's endowment fund, which had a successful hedge fund investing program.

"I've seen people say, 'So and so is in there and it's good enough for me,'" he added. "That's just ridiculous. Maybe (that firm) hasn't done any due diligence either."

Larry Smith, chief investment officer at Third Wave Global Investors, a Greenwich, Conn., hedge fund, and former CIO at Credit Suisse Asset Management, said that as the industry matures, investors will toughen up their standards, which he hopes will raise barriers to entry to the industry.

"The idea that you could give money to someone coming off a (bank's proprietary trading desk) who has never run a business is yesterday's story," he said. "People who are good investors need to recognize that it's not enough anymore. Investors, if nothing else, have now been awakened to make sure that the hedge funds they invest in have the proper structure and proper checks and balances in place."

Indeed if they don't, they could end up like Henny Youngman, right?

---------

Refco's chief exec charged with fraud: More here.

What went wrong at Bayou? Find out here.