In fact, nearly 60% of those in the top 1% of taxpayers at the start of any 10-year period between 1987 and 2010 had dropped out by the 10th year. That's from a study by the U.S. Treasury Department.
And the Tax Foundation found this: Of those who reported income of more than $1 million between 1999 and 2007, about half only reported income that high for one year.
Make no mistake: They still may be rich. But their incomes fluctuate greatly.
Obama: Budget 'is about our values'
"There are a lot of people who are [at the top] only once in any given period," said Roberton Williams, a fellow at the Tax Policy Center.
Transience is also a characteristic of those at the very tippy top. Among the top 400 taxpayers between 1992 and 2008, nearly three quarters appeared only once during those 17 years, according to IRS data.
--They may start to bring in less taxable income once they retire.
--They may have a bad year on their investments and claim losses, which can offset their capital gains.
--They may change the composition of their income, so that more of it is coming from tax-exempt investments, which don't have to be reported on one's 1040.
Indeed, the federal tax return offers no clue to a person's net worth.
For example, the size of retirement accounts and the value of property that has not been sold are not reported on tax returns. While those assets may throw off some taxable income, such as rent, their underlying value is a better measure of wealth.
That's why there's a lot less fluctuation in the top ranks of the wealthy than there is among the highest income households.
Bill Gates' income in any given year may be topped by that of a hedge fund manager, Williams noted. But his wealth remains vast enough to keep him among the world's richest for a very long time.
This story is part of a CNNMoney series exploring Americans' real tax burden. We'll look at who pays the most and who pays the least; and why two people with the same income can have very different tax bills. We'd love to hear how you feel about your tax burden at #YourEconomy.