ICAP has also agreed to pay a fine of $87 million to the U.K.'s Financial Conduct Authority and the U.S. Commodity Futures Trading Commission.
The three men facing charges are Darrell Read of New Zealand and Daniel Wilkinson and Colin Goodman of the United Kingdom. They were not in custody as of Wednesday, and law enforcement officials did not offer details on the prospects for their extradition.
"We're working very closely with our law enforcement counterparts around the world," the Justice Department's Mythili Raman said.
The U.S. filed criminal charges in December against two former traders from Swiss bank UBS(UBS) -- Tom Hayes of the U.K. and Roger Darin of Switzerland -- accused of attempting to manipulate rates. UBS, Barclays(BCS) and Royal Bank of Scotland(RBS) have paid hundreds of millions of dollars in fines in connection with the scandal, and more than a dozen other banks remain under investigation.
Libor rates, which serve as a benchmark for transactions all over the world, are created each day by polling a panel of large banks on their cost of borrowing. By deliberately engineering false submissions, traders at these firms could move rates to benefit their positions, according to court documents.
Brokerages like ICAP connect buyers and sellers in various kinds of financial products, including traders at large banks, in exchange for commissions. Brokers also provide "color," or market information, to clients.
Law enforcement officials say the three ICAP brokers assisted Hayes, the UBS trader, in rigging Libor rates denominated in yen between 2006 and 2011 by offering misleading information to banks formulating their submissions. They are also accused of lobbying Libor submitters at other banks to make submissions based on Hayes' wishes.
The CFTC said the suspects also helped other traders manipulate rates "on limited occasions."
Hayes is said to have been the brokers' largest client between 2006 and 2009, with Read and Wilkinson earning "a significant part" of their compensation based on his commission fees. Goodman allegedly received bonuses through the desk Wilkinson supervised to reward him for his assistance in the scheme, and is said to have been known among colleagues as "Lord Libor."
The defendants were also allegedly promised champagne and free meals in exchange for helping to fix rates.
ICAP said Wednesday that it was cooperating with regulators, and that it planned to improve its risk and compliance procedures.
"We deeply regret and strongly condemn the inexcusable actions of the brokers who sought to assist certain bank traders in their efforts to manipulate YEN Libor," ICAP CEO Michael Spencer said in a statement.
"It is deeply regrettable that the actions of these individuals have compromised the efforts of our 5,000 employees around the world."
The five men indicted in the U.S. so far represent just a small fraction of the finance professionals who have been implicated in the Libor scandal so far.
At UBS alone, at least 45 people were involved in or aware of attempts to manipulate rates, according to U.K. authorities. At Royal Bank of Scotland, 21 employees were said to be involved, and a number of traders at Barclays attempted to rig rates as well, according to settlement documents.
Prosecutors in the U.K., where Hayes is also facing charges, said in June that he conspired with colleagues at JPMorgan, RBS, Deutsche Bank, Rabobank, HSBC and brokers RP Martin and Tullet Prebon in addition to ICAP. Hayes is also accusing of attempting to manipulate rates while working at Citigroup(C).
The Justice Department's Raman said the investigation into Libor-rigging remains ongoing.
"It's one of the largest financial fraud investigations the department has ever done," she said.
-- CNNMoney's Chris Isidore contributed to this report