By Jennifer Liberto@CNNMoneyMay 17, 2013: 3:13 PM ET
The Internal Revenue Service scandal couldn't have come at a worse time for the nation's 2 million federal workers, who are desperate to end furloughs and get their first raise in three years.
IRS workers were front and center at a Friday congressional hearing about employees who wrongly singled out conservative groups seeking tax breaks. Workers were called "foolish" and "incompetent," and blamed for "horrible customer service."
Top IRS chiefs are doing damage control by blaming a small group of low-level federal workers, saying they made "foolish mistakes" by using politically loaded terms such as "tea party" and "patriot" to hold up groups applying to become charities.
"On the one hand you are arguing today that the IRS is not corrupt but the subtext of that is you are saying -- look, we are just incompetent," said Rep. Peter Roskam, an Illinois Republican, to departing Acting IRS Commissioner Steven Miller during the hearing. "And I think it is a perilous pathway to go down."
So far, union groups have laid low. Several unions didn't respond for requests for comment.
Colleen Kelley, president of the National Treasury Employees Union that represents IRS workers, said in a letter to employees this week that the watchdog report showed no one "intentionally did anything wrong, and I believe that to be the case."