When you owe the IRS tens of thousands of dollars in past-due taxes, you can expect to owe big penalties and interest.

And soon you could even lose your U.S. passport.

Lawmakers are in the final stages of negotiation over a bill to provide funding for U.S. highways and transit programs. One provision in it would let the federal government revoke, deny or limit a U.S. citizen's passport if the person owes more than $50,000 in "seriously delinquent tax debt," including penalties and interest.

It would only apply to those people whom the IRS has filed a lien or levy against and who have not yet worked out a repayment plan to satisfy the debt. An exception could be made for anyone who is actively disputing their case or who needs to travel for emergency or humanitarian purposes.

Should the provision pass next month, it would go into effect on Jan. 1, 2016.

Related: Millions of Americans might need passports to fly domestic

You may wonder what past-due taxes and passports have to do with funding transportation infrastructure. The answer is pretty much nothing, except that the measure is estimated to raise nearly $400 million over a decade. And that can defray some of the bill's cost since lawmakers decided, among other things, not to raise the federal gas tax.

"This is going to have an extraordinary impact [in terms of getting people to pay up]," said Los Angeles-based tax lawyer Dennis Brager, a former IRS trial attorney who thinks the measure is too draconian.

For starters, Brager said, it's not that hard to owe $50,000 because penalties and interest can add up very quickly.

Second, people faced with a big tax bill tend to get overwhelmed and postpone dealing with it for awhile. But the IRS typically issues a levy or lien six months after sending the initial bill. So even if a delinquent taxpayer eventually does come forward to set up a repayment plan, they may lose their passport for a time anyway.

And third, Brager said, while the measure allows for certain emergency exceptions -- perhaps letting someone travel to be with an ill parent in another country -- who's to say how long it will take the State Department to rule on that exception?

Related: You've never seen IRS penalties like these

One group who may be most vulnerable to the provision are the roughly 8 million Americans living abroad, who rely on their passports far more heavily than their domestically based compatriots.

"Their financial affairs are, typically, far more complex than their counterparts back home," said Nigel Greene in a statement. Greene runs the deVere Group, one of the world's largest independent financial advisory companies.

The relatively new Foreign Account Tax Compliance Act (FATCA) -- which requires other countries' financial institutions to report back to the IRS on accounts held by U.S. taxpayers -- may add to that complexity, Greene added.

"I would urge U.S. citizens abroad ... to ensure that their financial affairs are in order and compliant by the New Year," he said.