During his nomination hearing to become Secretary of the Department of Housing and Urban Development Thursday, Ben Carson assured his questioners that he would not seek to gut the agency or take rental assistance away from the millions of people who rely on it.

That gave some reassurance to lawmakers and housing advocates who were concerned about Carson's lack of experience and about comments he has previously made about HUD and public assistance in general. (In a 2015 op-ed piece for the Washington Times, Carson criticized a HUD rule meant to desegregate neighborhoods calling it "social engineering.")

But Carson will inherit a public housing system that's in crisis. Hundreds of thousands of people are on waiting lists hoping to find a home and a budgetary shortfall of $25.6 billion for much-needed repairs, according to HUD. And experts say Congress is unlikely to increase funding for public housing, and HUD, which funds the programs nationwide, estimates that approximately 10,000 units of public housing are lost each year to decay and disrepair.

Funding has been diminishing for years. In 2016, the federal government allocated $1.9 billion to fund capital improvements in public housing nationwide, nearly a billion dollars less than it did in 2000.

Related: Government-backed mortgages are about to get cheaper

At the same time, many cities, including New York, are rapidly being gentrified and filled with luxury housing, which is making it more difficult for low income Americans to find affordable places to live.

"There is a rental affordability crisis in the United States," Julian Castro, HUD's outgoing secretary told CNNMoney. "Having public housing units is a buffer against those rising rents."

"We ought to do everything that we can to maintain the ability of low-income families to live close to good job opportunities, close to good schools, close to good transit options," he added.

Carson, a retired neurosurgeon who grew up in poverty, noted in his opening statement Thursday that he understood the issues facing millions of low-income residents in public housing. "I have actually in my life understood what housing insecurity was," he said.

About 2.2 million Americans live in public housing, and more than 7.3 million receive government subsidies like Section 8 vouchers which are increasingly becoming the preferred form of housing assistance.

During his hearings, Carson said it would be "cruel and unusual" to do away with this assistance without a proper replacement in place.

One possible alternative: In June, House Speaker Paul Ryan released his A Better Way plan to improve opportunity and poverty in America, which called for work requirements and time limits for public housing residents.

Related: What about the black working class?

Shola Olatoye, the chair of the New York City Housing Authority, which manages 178,318 units of housing, called the assumption that public housing residents are not working an "outdated" stereotype. "Section 8 and public housing has been proven to be the single most important factor in getting people out of poverty," Olatoye said. The average tenure of a NYCHA resident is about 14 years, and most NYCHA residents are either working or elderly, Olatoye said.

Diane Yentel, the president of the National Low Income Housing Coalition called Ryan's proposed requirements "burdensome." "The vast majority of people living in subsidized housing are elderly, disabled, caring for a disabled family member or are working and don't make enough," she said.

In a statement after Carson's hearing, Yentel said she was "pleased" to hear Carson say he would support fair housing programs and many other HUD initiatives, but she's still concerned about potential funding cuts.

Victor Bach, a senior housing analyst at the Community Service Society of New York, a housing advocacy group, said a litmus test for Carson would be whether or not he lobbies to include public housing as part of a national infrastructure investment. "We would like to see public housing infrastructure included along with bridges and roads," Bach said.

Rebuilding American infrastructure was a big part of Trump's campaign and something that Democrats have latched on to as a potential bright spot in an otherwise partisan Congress. But Trump has said little about how he would handle public housing specifically.

Related: 5 startups tackling urban America's biggest issues

So far, the housing that has been built to replace demolished housing in cities like Atlanta and Chicago looks very different than the densely populated, institutional brick buildings Americans often think of as "the projects." Those older structures were demolished to make way for smaller, modern, mixed income developments that often include amenities like yards and shops.

But in cities like New York, where public housing buildings are densely populated, demolishing current housing stock to replace it with smaller units is not an option, said Olatoye. "We're a vertical city. We're not going to demolish a 26-story building."

NYCHA also has a waiting list of 250,000 people.

In order to remain fiscally solvent ,NYCHA is experimenting with a number of new initiatives, including leasing its land to private developers who will build affordable and market rate apartments to generate the needed funds for things like repairs to the older housing stock.

"We have to preserve this housing because it houses our city's workers, but we have to adapt to our reality," Olatoye said.