They're worried about their jobs, high debt levels and rising living expenses, according to a survey released Tuesday by the Employee Benefit Research Institute.
Only 13% of workers surveyed said they "feel very confident" that they will be able to retire comfortably — less than half the percentage reported in 2007.
Nearly half -- 49% -- said they were "not too" or "not at all" confident.
A large chunk of the workers surveyed have little or no retirement savings. Of those who provided estimates, 57% reported household savings and investments of less than $25,000, which included 28% of respondents who said they had less than $1,000. Only 24% reported savings of $100,000 or more.
Debt is standing in the way of saving. More than half of workers reported having a problem with their level of debt, while only about half of those surveyed said they could definitely cover $2,000 worth of unexpected expenses within the next month.
"Many lack even a short-term cushion," Matt Greenwald, president of Greenwald & Associates, a market research firm which conducted and co-sponsored the survey, said in a release.
Though economic conditions are improving, workers may also be waking up to just how much they actually need to save for retirement, according to EBRI.
Workers fear that cuts to Social Security could leave them picking up a bigger share of retirement expenses. Nearly 70% of workers reported a lack of confidence that Social Security would be kept at current benefit levels.
Spiraling health care costs and long-term care expenses are also a growing concern.
In 2013, 29% of respondents expressed concern with their ability to cover medical costs in retirement, up from 24% last year. Nearly 40% said they were worried about paying for long-term care, compared to 34% last year.