Wall Street no longer takes American Express.

Shares of AmEx (AXP) plunged 13% Friday after the credit card company issued a weak outlook for 2016 AND 2017 in its latest earnings report.

It appears that investors no longer have much, if any, faith in management. The stock is now trading at its lowest levels since December 2012. It is down more than 20% already this year -- and that follows a nearly 25% drop in 2015.

According to FactSet Research, 13 Wall Street analysts cut their price target for AmEx on Friday.

The stock's nosedive may even be testing the patience of legendary investor Warren Buffett.

AmEx is one of the so-called "Big Four" investments of Berkshire Hathaway (BRKB) -- the conglomerate run by Buffett. Wells Fargo (WFC), Coke (KO)and IBM (IBM) are the others.

But the value of Berkshire's AmEx stake has fallen by about $2 billion already this year!

Related: AmEx is just one of several laggards in the Berkshire portfolio

It may be tough for AmEx to recover anytime soon. The company is still reeling from the loss of its credit card partnerships with Costco (COST) and JetBlue (JBLU) last year.

Earlier this month, mutual fund giant Fidelity announced it was ditching the co-branded credit card it had with AmEx and Bank of America (BAC) in favor of a new one with Visa (V) and U.S. Bancorp (USB).

AmEx chairman and CEO Kenneth Chenault said in the company's earnings release late Thursday that there is "intense competition" in the payments industry and that "revenue growth has not accelerated as we anticipated."

As a result, AmEx is taking a closer look at its expenses in order to keep its profits from totally collapsing. Chenault said that the company plans to cut $1 billion in costs by the end of 2017.

But lower costs may not be enough.

"Something more significant needs to be done to address the challenges facing their business," said Jason Arnold, an analyst with RBC Capital Markets, who has an underperform rating on the stock -- basically a sell.

"The management team hasn't addressed the competitive environment. The steps they've taken have been small," Arnold added.

AmEx is losing ground to big banks like Citigroup (C) and JPMorgan Chase (JPM) as well as rival card networks Visa and MasterCard (MA).

If AmEx is unable to turn things around, there could be pressure on the board to make some drastic moves.

Bill Carcache, an analyst with Nomura, wrote in a report Friday that the continued weakness at AmEx may "fuel speculation around the viability of its business model."

And Chenault, who has been CEO since 2001, could find himself on the hot seat.

ValueAct, an investment firm that often buys stakes in companies with the hopes of shaking things up behind the scenes, took a position in AmEx last year.

Related: Activist investor takes stake in AmEx

Many investors believe ValueAct was instrumental in helping to push former Microsoft (MSFT) CEO Steve Ballmer aside in 2013, for example. And ValueAct CEO Jeffrey Ubben was named to the board of media giant Fox (FOXA) last September.

Ubben has been mum about AmEx -- and he was not immediately available for comment for this story. There are even rumors that ValueAct actually has already sold its stake. But it's possible that other activists might start circling AmEx.

So AmEx should consider itself lucky that Warren Buffett is still a big fan. But it might be more helpful if he started buying more AmEx stock.

Berkshire has increased its stake in Wells Fargo and IBM in the past few years ... but it hasn't boosted its AmEx position in a long time.