NEW YORK (CNN/Money) - Merrill Lynch agreed Tuesday to pay $13.5 million in fines to New Jersey and Connecticut for failing to supervise a group of brokers who engaged in improper market timing of mutual funds.

"We have agreed to resolve this matter with the state of New Jersey, Connecticut and NYSE," Merrill Lynch spokesman Mark Herr said, after The New York Stock Exchange's regulatory unit ordered the firm to pay.

New York-based Merrill, the country's largest retail brokerage, agreed to pay a $10 million civil penalty to the state of New Jersey, where the brokers operated from the firm's Fort Lee office. Merrill also agreed to pay $3.5 million to Connecticut.

Merrill also said it fired the three employees implicated in the scam "for violating the firm's policy prohibiting market timing, after they misled the firm about their activities and concealed behavior they had been told to stop." In addition, two supervisors were fined and yet another was demoted.

The office of New Jersey Attorney General Peter Harvey, which led the investigation, identified the brokers as Christopher Chung, Kevin Brunnock and William Savino. Brunnock, contacted by phone in Fort Lee, declined to comment. The other brokers could not be reached.

"While this was an isolated incident, we take it very seriously and are determined to enforce our policies against the market timing of mutual funds," Merrill said in a statement. Merrill did not name the brokers.

According to the NYSE, the brokers executed thousands of short-term mutual fund transactions from the firm's Fort Lee, N.J., office between January and April of 2002.

After some mutual funds complained, Merrill Lynch told the brokers to stop engaging in market timing transactions in the clients' accounts. But the brokers continued their trades, concealing them through outside accounts, and Merrill Lynch didn't realize it because the firm did not check to make sure the brokers complied, according to the NYSE.

Connecticut authorities assisted in the investigation, and the brokers' market timing activity "had a detrimental affect on Connecticut investors," said Richard Harris, spokesman for Connecticut Governor M. Jodi Rell.