Index funds revolutionized the investing world. Built on the idea that markets are efficient and the best way to outperform is to reduce costs, index funds have grown to manage trillions of dollars in assets.

In fact, nearly one in every $5 invested in stock funds is invested in an index fund, up from less than one in every $10 in the year 2000. There are a number of reasons why: Investors are becoming more aware of costs, exchange-traded funds (of which the majority is indexed) are booming, and ongoing fee wars have made index funds more competitive than ever with their actively managed peers.

But there is a cost to low costs.

The issue: At the annual Daily Journal meeting last week, Charlie Munger, Warren Buffett's right-hand man at Berkshire Hathaway (BRKA), fired a warning on the rapid rise of index funds: "Index funds will be permanent owners who can never sell. That will give them power they are not likely to use well."

Related: 3 Oil Stocks You Can Buy and Hold for the Next 20 Years

Other notable investors, like legendary value investor Mario Gabelli, similarly contend that index funds weaken corporate governance. Index fund managers rarely vote against incumbent management teams. In fact, it became headline news in 2013 when index manager Vanguard voted against directors at Hewlett-Packard (HPQ) in a sign of just how rare it is for indexes to take an active stance.

Index funds merely track an index. They don't make active stock picks; rather, they buy or sell stocks simply to match the holdings of the index they seek to track. And the indexes, like the popular S&P 500 index, are not actively managed. The S&P 500 index primarily weighs its holdings by their market capitalization -- the sum of the value of all of a company's outstanding shares. A passive investor's stance on a stock is largely moot when the fund can't buy or sell it.

Lost power: Put index funds' influence in perspective by looking at Apple (AAPL), the single largest company in the world by market capitalization. Its top three institutional investors are prominent index fund managers Vanguard, State Street, and BlackRock. Combined, these three institutional owners control 12% of its shares and, naturally, 12% of its proxy votes. That's a lot of power to put in so few hands.

Related: 3 Things I Learned Investing Thousands of Dollars of The Motley Fool's Money

Scrolling through Vanguard's votes, you'll find very few occasions in which it voted against management. Fortune noted that Vanguard voted against separating the CEO and board chairman positions, except when management was in favor of the proposal. Certainly, there have to be at least a handful of companies that would be better run if the CEO weren't also the chairman of the board. After all, a board's No. 1 job is to hire and fire a company's CEO. That's a tough call for a board to make when the current CEO attends every meeting.

This is largely an industrywide phenomenon. The largest funds don't take an aggressive stance with their proxy votes, which some would say is a direct result of a conflict of interest.

Fund managers have a fiduciary responsibility to their investors, but they also want to win the lucrative business of managing corporate retirement plans, which can generate billions of dollars in assets under management from the largest employers. Alas, C-suite executives aren't interested in putting employee savings in funds that will vote against them at the next annual shareholder meeting.

Related: 3 Up-and-Coming Restaurant Stocks to Watch

The bottom line: Index funds are excellent products. Vanguard is a beautifully designed company, owned by its investors and managed to minimize costs at every turn. But as they grow, index funds will need to be kept in check.

As you think about which funds you wish to invest in, look first at fees, but don't ignore funds' form N-PX filings, which detail just how "activist" or "passive" each fund is when it comes to voting on corporate matters on your behalf. We all lose when fund companies don't take an active role in corporate governance.

Jordan Wathen has no position in any stocks mentioned. The Motley Fool recommends Apple, Berkshire Hathaway, and BlackRock.