U.S. unemployment is down. Consumer confidence is up. Inflation is low.

Things are improving, yet Americans are still worried. The economy is voters' top concern ahead of the midterm elections next month, ranking ahead of national security, according to a recent Politico poll.

Only 42% of those surveyed by CNN late last month thought the economy was in good shape. While that's the highest share since January 2008 and an improvement from the 29% who felt this way a year ago, it's still weak overall.

Let's take a look at what's going right: The unemployment rate is below 6% for the first time since 2008. Job openings are back to 2001 levels. Consumer confidence is at its highest point since before the recession, and inflation remains a tame 1.7%.

Related: What do women want in a husband? A job!

Sounds great, but it's taken the country a long time to get to this point, said Richard Curtin, chief economist of the Thomson Reuters/University of Michigan Survey of Consumers.

The recovery has also been stronger for some than others. Young adults are still having a tough time starting their careers, while older Americans are having difficulty shifting into retirement after their nest eggs were destroyed during the Great Recession.

"It's taking so long to recover and it's been so uneven," he said. "It's been more than five years since the end of the recession."

Related: Obama's midterm message: Believe me, we're better off

Although the unemployment rate has fallen rapidly in the past two years, it remains at a relatively elevated level, said Justin Wolfers, a senior fellow at the Peterson Institute for International Economics. The average jobless rate in the decade before the Great Recession hit in December 2007 was 4.9%.

Americans also don't feel any better off. While more people may have jobs, they aren't bringing home fatter paychecks. Wages and income have remained stagnant for years, making it tough for folks even though inflation is low. Median household income, which stood at $51,939 last year, is back to 1995 levels.

Consumers expect a median income boost of 1.1% over the next year, Curtin said. But that won't keep up with their inflation expectations of 2.8%.

"American households, on average, are still struggling with their living standards slowly eroding," he said.

Not everyone, however, is suffering from flat-lining wages ... and that's also why the average American remains worried about the economy. The rich are seeing both their income and wealth rise. The wealthiest 5% of American households held 63% of all wealth in 2013, up from 54% in 1989, according to a recently released Federal Reserve survey.

"Rising inequality is why Main Street doesn't feel like it's benefiting from the full fruits of the recovery," Wolfers said.