Despite a national debate over college affordability, pay just keeps going up for presidents of private colleges.

Median compensation rose nearly 6% to $436,429, according to a new report from the Chronicle of Higher Education. It looks at salary and benefits for leaders at 497 private, non-profit colleges during 2013, the latest year data is available.

Columbia University's Lee Bollinger topped the list, earning $4.6 million. That includes a $1.2 million base salary, and other benefits like bonus pay and the cost of his campus residence.

Bollinger is the longest-serving president at an Ivy League school. He came to Columbia in 2002 by way of the University of Michigan, where he was president for six years.

His total earnings in 2013 also included nearly $1.3 million of deferred compensation that had been put aside over the course of his tenure at Columbia. It's common for colleges to offer this pay as a retention incentive.

Related: Why America's most expensive college is a bargain

Consultants say that there are a "finite amount of people" for these jobs and it's not uncommon for college presidents to hop from one institution to another, said Sandhya Kambhampati, a reporter at the Chronicle.

Four other presidents earned more than $2 million in salary and benefits during 2013: Amy Gutmann at the University of Pennsylvania, Nido Qubein at High Point University, Richard Joel at Yeshiva University, and Nicholas Zeppos at Vanderbilt University.

It's worth noting that even multi-million compensation packages can make up just a small portion of a school's expenses. At Columbia, for example, Bollinger earned about $1,300 for every $1 million the college spends.

"The salaries of college presidents generally have a small effect on tuition at the vast majority of institutions," said Robert Kelchen, a professor of higher education at Seton Hall University.

Related: College costs $43,921 this year

Despite some big pay packages, the president is not always the top-paid employee at nearly a quarter of private colleges. Coaches were the top earners 7% of the time. At Vanderbilt, for example, the football coach made more than the president in 2013.

No one thing can be blamed for rising college costs, Kelchen said. The biggest expenses are typically related to instruction: faculty salaries and benefits, facilities and IT. While salaries have generally kept pace with inflation, the cost of health care benefits have grown. Schools have also added non-teaching staff members, like advisers and counselors.