The billionaire chairman of one of the world's biggest luxury groups is losing a lot of sleep over rising inequality and the destruction of the middle class.

Johann Rupert, chairman of Cartier owner Richemont (CFRHF), made a number of dystopian warnings during a speech at a Financial Times conference in Monaco.

He forecast that robots would "put hundreds of millions of people out of work," which would widen the gap between rich and poor and stoke social unrest.

"It's really what keeps me awake at night ... How is society going to cope with structural unemployment and the envy, hatred and the social warfare? Because the people with money will not wish to show it," he said.

Super wealthy clients could be targeted as unemployment surges, he said. That would make selling luxury goods more difficult.

"We cannot have 0.1% of 0.1% [of rich individuals] taking all the spoils. And folks, those are our clients. But it's unfair and it is not sustainable," he said to an audience that appeared stunned by his remarks. "So I don't know what new social pact we'll have, but we'd better find one."

Related: Why is everyone in Monaco so darn rich?

The South African has been working in the luxury retail industry since 1976. He and his family are estimated by Bloomberg to be worth about $7 billion, and are among the 20 biggest shareholders in Richemont.

Based in Switzerland, the group owns over a dozen luxury brands, including Montblanc, Chloe, Net-a-Porter and Van Cleef & Arpels.

Rupert said he had been thinking about these issues since reading about the rise of machines in the workplace.

And he appears to be taking his own gloomy predictions seriously.

"We are in for a huge change in society. Get used to it, and be prepared," he said. "Hopefully we will survive it, because we're planning for it."

Richemont did not return requests for comment.