(FORTUNE Magazine) – One of the highlights of the year for Brad Martin is surely the golf outing, named after him, that occurs every autumn in Birmingham, Ala. The express purpose of the Brad Martin Golf Invitational, held on the picturesque fairways of the Robert Trent Jones--designed Oxmoor Valley courses, is to raise scholarship money for children of employees of Saks Inc., owner of the venerable Saks Fifth Avenue department store chain. The event's other purpose, say some Saks vendors, is to allow them to fete Martin--a charming Southerner who came to retailing by way of Tennessee politics--and the company he has run for seven years. "Brad is very much the focal point," says one vendor who attended last year. "His name is everywhere." At the silent auction last year, attendees had the chance to bid on a basketball signed by Julius "Dr. J" Erving (he's a member of the Saks board) and a couple of rounds of golf with Martin. (Minimum bid: $10,000. Value, according to the Saks website: "Priceless.") At the big dinner on the last night, said another person who was there, a minister led the group in prayer. "God bless you all," the minister concluded. "And God bless Brad Martin."

These days Martin needs all the help he can get. Saks Inc. has been racked by scandal over the past several months, as allegations of improper collection of "markdown money"--the funds that department stores deduct from vendors' payments when goods don't sell briskly enough--have prompted investigations by the SEC, federal prosecutors, and the company's own board, as well as lawsuits from purportedly shortchanged vendors. With inflammatory accounts swirling in the press, the company missed the April deadline to file its annual financials, and in June both Moody's and S&P downgraded its debt. Three high-level executives--including Martin's brother Brian, the onetime general counsel--have been fired, while the CFO has been stripped of financial reporting duties.

Yet all that is only the latest of Martin's troubles. Under his leadership the company's stock has dropped more than 20%, while competitors like Neiman Marcus and Nordstrom have enjoyed strong returns (see chart). Sales for Saks-branded stores (Saks Inc. currently includes seven other retail chains, which deliver more than half of the company's revenue) teeter only 9% above where they were in 2000, with operating income down 13%. So how did a once-enviable retail swan get into this ugly mess? Company officials declined to be interviewed, agreeing only to respond in writing to a limited set of questions. In those answers the company blamed "the tragic events of Sept. 11, 2001, [for] a dramatically negative effect ... particularly in New York City and other gateway cities." But Martin himself admitted, in a prepared statement, that "centralizing key operations more quickly would have created a more efficient and effective structure," and that "key hiring decisions" and "more aggressive ... business planning" could have helped. In any case, Martin and his board are surely feeling the heat now. Vulture investors, including private-equity firms Bain Capital and KKR, are reportedly circling the company, putting its future in doubt.

Can Martin possibly turn things around--or is he about to lose control of the company? For 43 of the first 45 years of Saks Fifth Avenue's history, it enjoyed the stability of one master: Adam Gimbel. A cousin of the man who co-founded the store in 1924, Gimbel had impeccable fashion sense. Under his stewardship Saks Fifth Avenue adopted the opulent Art Moderne style for its flagship store and created the concept of in-store boutique specialty shops. Saks became synonymous with luxury and exclusivity.

Martin's heritage is less refined. A glad-handing former Tennessee state legislator, Martin got into the retail game by developing strip malls. His first foray into department stores was a mid-market, five-outlet chain in Tennessee called Proffitt's, where his then-wife, Jean, liked to shop. But Martin knew how to make a deal. He took Proffitt's public in 1987 and a few years later went on a shopping spree, buying seven regional department store chains--including Parisian, based in Birmingham, Ala.; Herberger's in St. Cloud, Minn.; and Carson Pirie Scott in Milwaukee--from 1994 to 1998. As the Proffitt's empire grew to 237 stores, with $3.5 billion in sales, investors were handsomely rewarded: The stock rose 900% from 1987 to 1998, far outpacing the returns of department store competitors Dillard's and May.

Saks Fifth Avenue was Martin's crowning acquisition, bought in 1998 after an aggressive expansion that failed to produce much in profits. "Given the choice between owning Saks Fifth Avenue, a $2.5 billion business with an extraordinary brand appeal and real estate position, vs. owning a marginal regional department store, I'll take Saks," he told FORTUNE back then. "It's a once-in-a-lifetime opportunity." The company quickly adopted the Saks name.

Wall Street hated the deal. Concerned that Martin had overpaid for his new jewel--that the deal was more about ego than sound economics--investors stampeded out of the stock, which fell from $40 to $33 in five days. Seventh Avenue, meanwhile, was stunned. How could a ragtag company like Martin's possibly be a good fit with an empress of the grand emporiums? Retailing is notoriously fast-paced and competitive, with success often dependent on gut instinct and fashion savvy. Luxury retailing is even more nuanced. Martin, who tended to leave leadership intact after buying a chain, declared that he would continue his hands-off approach. "I'm not going to run Saks Fifth Avenue," he said. At the same time, he appeared to understand that overseeing Saks brought new expectations. "I have to elevate my game," he told the Wall Street Journal in 1998.

But that next level never materialized. He showed little interest in hobnobbing with the fashion elite in a business where relationships with designers are crucial. ("Being visible--I don't think it matters that much," he told Women's Wear Daily in 1998.) He failed for years to find a strong replacement for Rose Marie Bravo, a fashion whiz who had been a key link to designers and had departed the Saks chain in 1997. Management became a merry-go-round: The Saks Fifth Avenue division is now on its third CEO since Martin took over. "The leadership changes have caused a lot of recalibration to new strategies and new personalities," says Arnold Aronson, a managing director at retailing consultancy Kurt Salmon, who ran Saks as CEO in the 1980s.

Underperforming stores were allowed to linger, while resources were diverted to private-label lines that failed to take off--and worse, undercut the chain's reputation for chic. It all showed in the numbers. At the time Martin acquired the Saks stores, sales per square foot were running at $361, according to A.G. Edwards; that compared with about $440 for Neiman Marcus stores. Today that figure has dropped $12 per square foot for Saks, to $349; Neiman's sales have risen to $555 per square foot.

Into this less-than-pretty picture came a bona fide scandal. In a surprise press release, Saks Inc. announced on March 3 that its audit committee was investigating alleged improper collections of so-called markdown money in its Saks Fifth Avenue chain. Citing an estimated $21.5 million in collections from fiscal years 1999 to 2003, the company said it expected to restate earnings.

The reaction was sweeping. The SEC opened an investigation, as did the U.S. Attorney for the Southern District of New York. The fashion press was thrown into an uproar, as article after article shined a spotlight on the very concept of markdown money. The irony is that markdown collections are widespread, and widely accepted, in the retail industry. While Saks may have been no more reliant on it than others--and some say less so--its revelation made it the poster child for the suddenly distasteful industry convention.

The practice started innocently enough a couple of decades ago, when department stores, feeling pinched, started asking their vendors to give back a little if their merchandise had to be marked down in order to move. A department store buyer has a target margin he is supposed to meet, but sometimes a product fails to fetch a high enough price to match that level of profitability. At the end of the quarter the buyer may go back to a vendor and, after the fact, try to negotiate a lower price on the goods. The difference between the old and new prices--the "markdown money"--is deducted from the store's next order. Vendors not only have to rely on the profitability assertions of buyers but also have pressure to comply. "Basically what the retailers do to their vendors is say, 'If you don't give us the markdown money, you're not going to get any orders next season,'" says Robert D'Loren of UCC Capital, an investment-banking firm that focuses on the fashion world. "It's almost extortion. If they were in a different business, they'd call it racketeering." Another disputed area: chargebacks. If a vendor fails to meet delivery standards that have been spelled out in advance--on a certain type of hanger, say, with the shipping label in a certain spot--the firm owes the store a preset amount. Once again, the vendor is often at the mercy of the store's assertions.

How much money do vendor chargebacks and markdowns bring Saks? The company won't say. But competitor Nordstrom has filed financials that give some clue to how critical those payments are to the business: Price reductions from vendors came to almost $48 million in 2004 at Nordstrom; net earnings over the same period were $393 million. "Every single [department store] has come to depend on this money," notes one vendor. "It's like heroin. They need markdown money and violation charges to make their numbers now."

Saks's problem, of course, was admitting it had improperly calculated markdowns. The chain already faced one civil lawsuit from a former vendor of Michael Kors apparel that made similar claims. Another followed from a licensee of Oscar de la Renta, whose attorney, Donald Kreindler, has organized a coalition of vendors to address the practice of markdown charges throughout the industry.

But for Saks and Martin there was still more trouble ahead. Saks Inc. announced in April that it would not be filing its annual report on time, as it continued to examine accounting issues. A few weeks later the board issued another bombshell: The company's chief accounting officer, the Saks chain's chief administrative officer, and Martin's brother Brian (now a senior vice president) were being fired; CFO Douglas Coltharp was stripped of responsibility for accounting and financial reporting, and he and Martin would both have their annual bonuses either reduced or eliminated. The announcement also tied Brian Martin and the accounting chief to a mysterious 2002 internal investigation of markdown money, though it gave no further details.

And just when it seemingly could get no worse, Saks received a notice from a convertible bondholder in June that the failure to file timely financials put the company into technical default. Saks's credit rating was downgraded until the company shelled out $585 million to pay off affected bond claims.

A growing drumbeat of investors is saying enough is enough. Wayne Archambo of investment firm BlackRock, which owns more than four million Saks shares, confronted Martin during a company conference call in March, suggesting it was in shareholders' best interest to consider selling the company. On another conference call, in May, Parker Phillips of hedge fund Bondurant Management was blunter: "Why should we have any confidence that this management team is the right team to lead the company?"

"The team is focused on doing the right thing for the business, the right thing for the shareholder," Martin responded. Saks's largest investor, Mason Hawkins of Southeastern Asset Management, remains supportive of the CEO. And there have been some signs of progress. Last fall the company announced the closings of 12 underperforming Saks chain stores. Saks Fifth Avenue head Fred Wilson (the former CEO of Donna Karan) has lured new products from celebrities like Bono and Elizabeth Hurley and cut off the uncool private-label lines.

But Martin's corporate strategy has clearly changed. He and the board have begun dismantling the company, selling off the Proffitt's and McRae's chains last month for $622 million, and inviting bids for six of the company's remaining mid-market chains. In its written statement the company disputes the term "dismantling," describing its moves as "strategic options ... in the long-term best interests of our shareholders." Nonetheless, if the six chains are sold off, that would leave Martin only the Saks Fifth Avenue division and higher-end chain Parisian.

Might the entire company be swallowed up and Martin's full team replaced? The $5.1 billion that Neiman Marcus recently fetched in a buyout bid by Warburg Pincus and the Texas Pacific Group has piqued private-equity interest in the tarnished Saks brand. In its written response, the company declined to comment on the prospect of a buyout, but the rumors are flying. So is Saks Inc. stock. Despite the markdown scandal, shares have moved up more than 30% since March, from under $16 to above $20.

Martin, meanwhile, remains in his bunker. At the end of June the board approved generous new long-term incentives for him and other top executives. Martin also gets 40,000 additional shares if, among other things, he completes a certain Sarbanes- Oxley requirement on internal financial controls by 2006--in other words, if he abides by the law. That gives disgruntled investors an additional incentive of their own to embrace whatever takeover proposals may arise. Perhaps that's par for the course.

FEEDBACK eflorian@fortunemail.com

REPORTER ASSOCIATE Doris Burke