The oil bust is ending the boomtown party in America's energy states.

After leading the nation's comeback story during the recovery, many energy-focused states lost a lot of their momentum in 2014, according to a new report from the Commerce Department.

A barrel of oil was worth $100 in August, but fell to $54 by the end of last year due to a global glut in the oil supply. For many states such as South Dakota, Oklahoma and Wyoming, that's bad news.

"The fall in oil and gas prices has had huge effects in states like that," says Michael Canes, an distinguished fellow at the Logistics Management Institute in Virginia. He added: "People take a big hit, that's just a sad fact of life."

Related: Oil boomtown on the brink of bust?

Energy states got hit hard: South Dakota -- a poster child of America's oil drilling renaissance -- grew 3.1% in 2013, but only 0.6% last year.

Even North Dakota -- one of the country's fastest growing states -- saw a marked decline in its gross domestic product, a broad measure of economic activity, from 9.7% in 2013 to 6.3% in 2014.

Energy companies have already cut over 60,000 jobs this year, according to the Labor Department. Those job losses hurt people the most in these states.

Many experts say cheaper gas prices are good for the overall economy, giving Americans extra spending money. But Americans are spending those savings yet. And that perceived, widespread benefit comes at the expense of America's oil-producing region.

Oklahoma, Wyoming and Montana took similar hits to their economic growth. Their economies are still one-dimensional. They depend on oil to drive most other businesses.

Related: Canada's economy is a disaster from low oil prices

Colorado and Texas stay strong: But there's two outliers to the trend in energy states: Colorado and Texas. Both of their economies grew more in 2014 than in 2013.

Texas' economy expanded 5.2% in 2014, a big rise from its 3.7% gain from the prior year.

The Lone Star state is no longer a one-star economy. Since its big oil bust in the 1980s, Texas has added several industries to its repertoire: technology, health care and education, among others.

It's also expanded its reach into solar, wind and renewable energy production. That's an important development, says Canes, an energy expert.

Cities like Austin and San Antonio have become hot beds for an array of jobs too.

Related: Fastest growing U.S. cities: Texas is king

Texas is the nation's second largest state economy, and its unemployment rate is only 4.2%, well below the national average of 5.5%.

Those stellar job figures are rivaled by Colorado, which has the same unemployment rate and a diverse economy. Colorado is less energy focused than others, but it's been put under the energy spotlight because of the Keystone Pipeline debate.

Colorado has added thousands of jobs in non-energy sectors during the recovery, like construction, education and tourism.

Related: Recreational pot: $53 million in tax revenue to Colorado

Outlook going forward: Last year's precipitous drop in oil prices is still impacting states.

Big energy companies are slashing jobs and oil rig counts continue to fall. Oil has rebounded slightly since bottoming out in March. A barrel of oil is now $61. But it's still far from the triple-digit days that helped spark the drilling boom.

Some people aren't waiting to see if the turnaround comes to pass.

"I'll bet a lot of those people are starting to leave," says Canes. "I do think people will leave North and South Dakota."