The 10 richest colleges in the U.S. have endowments of at least $10 billion.

With $36.4 billion, Harvard has the largest endowment by far.

But the average investment return for colleges and universities across the country took a big hit in 2015, falling to 2.4% from 15.5% the year before, according to a new report from the National Association of College and University Business Officers.

It's not a complete surprise considering last year's weak market performance. And it's not as bad as 2012, when endowments averaged a loss of 0.3%.

Related: Billionaire gives Harvard record $400 million gift

But what's most concerning is that the disappointing year drags down the average 10-year return below what most colleges say they need to maintain, said William Jarvis, the executive director at the Commonfund Institute, an asset management firm that sponsored the report. That might lead colleges to use less money from their endowments each year.

"If we're now living in a low-return environment, the question that comes up is whether you need to adjust your spending rate downward," Jarvis said.

On average, colleges spent about 4% of their endowment last year, which covered about 10% of their operating budget. Those with an endowment larger than $1 billion funded about 17% of spending, according to the report.

Related: Stanford and Harvard got over $1 billion in donations

Endowment investment allocations did not change much since the year before. On average, more than half of an endowment's assets are in alternative strategies -- like derivatives, hedge funds and private equity. For smaller endowments, the share of funds in these more risky investments is less. About 19% of assets are in international stocks and 16% are in domestic, across the board.

The top 10 largest college endowments:

1. Harvard University: $36.4 billion
2. Yale University: $25.6 billion
3. University of Texas system: $24.1 billion
4. Princeton University: $22.7 billion
5. Stanford University: $22.2 billion
6. Massachusetts Institute of Technology: $13.5 billion
7. Texas A&M University system: $10.5 billion
8. Northwestern University: $10.2 billion
9. University of Pennsylvania: $10.1 billion
10. University of Michigan: $10 billion