NEW YORK (CNN/Money) - Four of the biggest U.S. mutual fund firms have engaged in illegal trading practices that may have cost mutual fund investors billions of dollars in the past four years, New York Attorney General Eliot Spitzer said.

Between 1999 and 2003, mutual fund managers at Bank One (ONE: Research, Estimates), Janus (JNS: Research, Estimates), Strong Financial and Bank of America (BAC: Research, Estimates) Nations Funds illegally gave a hedge fund special privileges to make illegal trades, in return for fees, Spitzer said in a civil complaint filed Wednesday with the New York Supreme Court.

Spitzer did not offer specific figures about the costs to ordinary shareholders of these trades, which allow some traders to take advantage of market inefficiencies, but he cited academic studies that said the trades cost investors "billions of dollars per year."

"What we have is a fundamental violation of the rights of shareholders and a violation of the duty of mutual funds to protect the interests of long-term shareholders," Spitzer told a press conference.

Massachusetts' top securities regulator also said his office is probing possible illegal trading practices among the mutual fund companies.

More on the scandal • Fund investors weigh options • How Mom & Pop lost out • Fund industry investigated • Mass. probes Prudential • Mutual fund inquiry spreads

"There is enough of an interest on our part to ask the questions of practice and procedure at mutual funds. We are interested in this and we are compiling information," William Galvin, the Secretary of the Commonwealth of Massachusetts told Reuters when asked if he is conducting a similar probe.

Bank of America and Strong told CNN/Money they are cooperating with Spitzer's investigation but had no other comment. Bank One said it would look into the complaint's allegations and cooperate with Spitzer. Janus told Reuters it was reviewing the complaint and cooperating with Spitzer's office.

Wednesday's complaint was not filed against the fund managers but against Canary Capital Partners, a hedge fund based in Secaucus, N.J. Spitzer said Canary and its managing principal, Edward Stern, agreed to pay $30 million in restitution and a $10 million penalty to settle the charges against it, which included allegations that it had performed illegal trades of mutual fund shares.

Canary and Stern neither admitted nor denied wrongdoing.

Read the complaint here New York vs. Canary Capital Partners, et al. (PDF)

Stern is the son of Leonard Stern, CEO of Secaucus-based Hartz Mountain Industries, which until recently owned and operated pet-food maker Hartz Mountain Corp. and Stern Publishing, which published the Village Voice and other regional, alternative newspapers.

Though now out of the publishing and pet-food businesses, the Stern family continues to own real estate, and the Hartz Mountain Industries Web site credits Edward Stern with running the group's "financial and investment activities."

Stern's hedge fund, Canary, was accused by Spitzer of "late-day trading," the term for buying mutual fund shares at their closing price after the close of U.S. stock market trading at 4:00 p.m. ET, allowing the firm to profit from corporate news released after the closing bell. The law requires mutual-fund trades made after the close to be processed at the next day's share value, to keep late traders from having an advantage over others.

"Late-day trading is like being permitted to bet on yesterday's horse races," Spitzer said. "You already know who's going to win."

Despite the illegality of these trades, Spitzer alleged, Bank of America gave Canary special software that allowed it to circumvent the rules and make after-hours trades at closing prices, in return for "millions" in fees.

Canary also was charged with "market timing" trades, designed to exploit the effect of time-zone lags on international mutual fund shares.

Japanese stocks, for example, close 14 hours before U.S. stocks. If Japanese stocks fall, then an international mutual fund holding Japanese shares will still reflect those lower prices when they close 14 hours later. If traders are sure Japanese stocks will rise on the next trading day, they can buy the international fund, with its 14-hour-old, "stale" net asset value (NAV), and cash in the next day, when the NAV rises.

Such trading is not illegal, but Spitzer said many fund managers, including Janus, are aware of the damage it can do to long-term investors and say in their prospectuses that they will try to stop such trades. Instead, Spitzer alleged, Janus and other fund managers took payments to allow such trades.

"Mutual funds, who have a fiduciary duty to shareholders, who have made the representation that they will do what's reasonable to stop these trades, and then they take payments for allowing these trades to take place -- they're in deep trouble," Spitzer said.

The attorney general did not indicate whether other charges will be filed and said he had no idea yet of the full extent of such trades, but he said his office's investigation is ongoing.

"My office will take all reasonable steps to ensure that the ill-gotten gains of those who engage in this conduct are returned to investors, that wrongdoers are held responsible, and that the appropriate reforms are implemented to halt this egregious activity," Spitzer added.

Hedge funds are largely unregulated funds that typically require large initial investments and regularly use leverage and short positions. They gained favor during the three-year bear market for their ability to profit on falling stock prices.

Bank One has the 16th-largest mutual fund group in the United States and the third-largest bank-sponsored fund group. Janus has $149.8 billion under management. Nations Funds has $134 billion under management, and Strong has $42.1 billion under management.

The Investment Company Institute, a Washington, D.C., lobbying group representing fund managers, said, "Mutual funds and the ICI support strict compliance with and vigorous enforcement of the law on behalf of mutual fund shareholders."