If the phone rings and the caller says he represents the IRS, be suspicious.

That's the warning from federal authorities, who on Thursday said a nationwide phone scam has stolen $1 million from thousands of unsuspecting people.

The impostor claims to be an Internal Revenue Service representative and tells "intended victims they owe taxes and must pay using a pre-paid debit card or wire transfer," an IRS inspector general office said.

Related: 10 tax audit red flags:

"The scammers threaten those who refuse to pay with arrest, deportation or loss of a business or driver's license."

The IRS has received more than 20,000 reports about the scam.

J. Russell George, the Treasury inspector general for tax administration, called it "the largest scam of its kind that we have ever seen."

According to the inspector general, IRS officials typically first reach out by mail rather than phone, and don't demand immediate payment by debit card, credit card or wire transfer.

Related: 7 surprising 2014 tax facts

The IRS said people who receive such calls or other suspicious requests should contact the IRS.

Tax-related phone scams are among the "dirty dozen" fraud techniques the IRS warned about earlier this tax season. It also warned of phishing emails, preparer fraud and claims a preparer can offer "free money."