If you thought getting a cab in the rain was tough in Manhattan, try renting an apartment there.

The median rent climbed to $3,380 in May, and the borough's vacancy rate was 1.65%, according to a report by real estate appraisal firm Miller Samuel for Douglas Elliman Real Estate.

"Another way to think of it is 98.35% of the units in Manhattan are rented at any given time," said Jonathan Miller, president of Miller Samuel.

High costs and low inventory are making it tough to rent across the Big Apple.

Steadily rising rents are creating tension between landlord and tenants. "When new rentals surge, that means a lot of tenants are rejecting offers and looking elsewhere," said Miller.

But affordability can be hard to come by anywhere in New York City. The median rent in Brooklyn was $2,933, while rent in the northwest part of Queens was $2,597.

Related: Why your rent check just keeps going up

High churn is creating stress on the market, Miller said. "You have people scrambling, they want to be here in the city. There is job growth, there is a lot going on here. But the jobs that are needed to be filled are by people who can't afford to pay the rent."

Developments in Queens and Brooklyn are becoming condos instead of rental apartments, Miller said, adding to the inventory shortage. "Land prices have risen so much ... the math doesn't work for rental anymore."

Low inventory also means less incentives for renters. "Concessions being offered by landlords, like free rent, have nearly evaporated."

And it doesn't look like price relief is coming anytime soon.

"I don't see the narrative changing in the foreseeable future," Miller said. "It's a very complex problem; a city can't grow if new entrants to the city can't afford to live there."