Video More video Hoffa tells CNN's Dana Bash his group split with the AFL-CIO because unions aren't growing and policies aren't working. (July 25) Play video

Video More video AFL-CIO president John Sweeney addressed the conference. (July 25) Play video

CHICAGO (CNN) - The Teamsters and the Service Employees International Union voted Monday to withdraw from the AFL-CIO.

The announcement came on the first day of the 50-year-old labor federation's annual meeting.

James Hoffa of the Teamsters and Andy Stern of SEIU told reporters they each notified AFL-CIO President John Sweeney on Monday morning in a letter after their executive boards unanimously agreed to disaffiliate.

"What John Sweeney has been doing has not been working and it's time to try something new," said Hoffa, general president of the International Brotherhood of Teamsters.

He predicted to CNN that seven other international unions would soon bolt from the AFL-CIO and join the Teamsters.

"It is not done lightly. We have extended a number of propositions and ideas to the AFL-CIO to make sure that we could change the tide of the AFL-CIO.

"We have been disappointed over the last 10 years that we have seen a decline in membership," he said, referring to Sweeney's decade-long tenure as president.

Though the union suggested "a number of changes," including a $5 million rebate to help the Teamsters reorganize in their core industries, "they said no," Hoffa said. "Their idea is to keep throwing money at politicians."

He added, "What was being done at the AFL-CIO is not working; we are going to do something new. That is our message."

Among the new things the Teamsters will try, Hoffa said, will be becoming "more bipartisan. ... We're not going to be afraid to back a Republican."

Two other unions were also considering splitting from the AFL-CIO.

A spokeswoman for UNITE/HERE, which represents textile and hotel workers, said she would not announce Monday whether it is quitting the umbrella organization.

A spokeswoman for the United Food and Commercial Workers International Union said she did not know what was being planned.

"All the signs are that they are going to split off," former Labor Secretary Robert Reich told CNN.

The dissident unions have a combined membership of nearly 5 million workers -- about a third of the AFL-CIO's total membership. Their leaders have accused Sweeney of putting more money into political campaigns at the expense of union organizing.

"We are not trying to divide the labor movement, we are trying to rebuild it," SEIU President Andy Stern said Sunday.

When the AFL-CIO formed in 1955, about 35 percent of American workers belonged to unions. Today, unions represent about 8 percent of the private-sector workforce.

The AFL-CIO has weighed in heavily behind Democrats in national politics, with member unions contributing tens of millions to party candidates in 2004, according to federal campaign finance records.

"In unity, there is strength," Reich said. "And if the AFL-CIO splits, the fear is at least that the entire labor union movement will be less powerful than it is today -- and that's not very powerful."

Reich, who led the Labor Department during the Clinton administration, said American workers are likely to be worse off "if the labor unions just find themselves fractured and angry." But if the dissidents can get more workers to sign union cards, he said, "it's possible that the labor union movement, when the dust clears, is actually stronger."

In some ways the split of major unions from the AFL-CIO may have more symbolic impact than operational impact for unions and their members, said Gary Chaison, professor of industrial relations at Clark University in Massachusetts.

"Quite bluntly, to an auto worker at General Motors or an airline worker facing demands for concessions, this doesn't mean a hell of a lot," said Chaison. "Their concern is 'can my union deliver at the bargaining table?' I can't think of any union that would really be hurt at the bargaining table."

But Chaison said the split is important as a sign of the troubles that have been plaguing organized labor for decades: waning numerical strength and lost leverage in dealings with employers.

"I do think employers will be emboldened by this move. It's a signal that labor is on the ropes," he said. "The decline of organized labor over the last 20 years now is really shown by the events of the last day or two."

Chaison said that the loss of dues paid by members of the four major unions which may leave the AFL-CIO could cut its operating budget by up to a third.

"I think it (the AFL-CIO) will survive as long as it has key unions still in it like Steelworkers and Auto Workers and especially the Communication Workers," said Chaison. "But it's going to be a loss for the AFL-CIO in terms of income. It's not going to be effective in terms of political activity for quite a while."

Sweeney, who was president of the SEIU before becoming the federation's leader in 1995, said the federation's remaining 50-plus unions "will make major decisions about changing workers' lives this week, no matter what happens."

"It's a shame for working people that, before the first vote has been cast, four unions have decided that if they can't win, they won't show up for the game," Sweeney said in a written statement.

He said the boycotting unions "should come argue for their ideas and listen to others. That's how democracies work." He called it "fundamentally wrong to use working people's issues as a fig leaf for a power struggle."

But SEIU Secretary-Treasurer Anna Burger said the AFL-CIO "has listened ... but, in the end, has not heard us." She called the differences "unresolvable."

The coalition includes three other unions -- the Laborers International Union, the Carpenters Union and the United Farm Workers. The UFW and the Laborers are not planning to boycott the convention, and the Carpenters Union is not an AFL-CIO member.

-- CNN Producer Grace Ramirez contributed to this report.

--------------------------------------------------

Fewer workers are getting flexible schedules -- click here.