(FORTUNE Magazine) – WAY UP atop a Manhattan skyscraper, inside the investment banking firm of D.E. Shaw & Co., is a tiny hexagonal room staffed by six people who look fresh out of college. They stare at computer screens that cover the walls, clicking their mice, occasionally talking calmly to one another or into their phones. It is utterly unlike the typical Wall Street trading floor (where scores of shouting, sweating Mylanta chuggers raise a Dantesque din). In fact, it is utterly unimpressive--except for the fact that Shaw's trading volume is sometimes equal to 5% of the total volume of the New York Stock Exchange.

D.E. Shaw & Co. is the most intriguing and mysterious force on Wall Street today. It's the ultimate quant shop, a nest of mathematicians, computer scientists, and other devotees of quantitative analysis who use their arcane sciences to monitor the world's financial markets and squeeze profits out of places most people would never think of looking. It's the answer you'll get if you ask the question, What's the most technologically sophisticated firm on the Street? Many investment banks and trading firms, of course, have stocked their ranks with hot-rod quants plying their Ph.D.s to figure out the markets. But D.E. Shaw doesn't just have lots of "rocket scientists." This place is run by rocket scientists, starting with the 44-year-old chief executive and founder, David Shaw, Ph.D., a former Columbia University computer science professor who used to design and build supercomputers for a living.

Few people outside the world of finance have ever heard of D.E. Shaw, and until recently that suited him just fine. The firm's primary business--quantitative trading, using techniques like statistical arbitrage--is very esoteric stuff, as we'll see, and its other investment banking activities are done in obscure markets. But now Shaw and his scientists and mathematicians are pursuing a hugely ambitious plan to venture far beyond Wall Street. If they succeed--anything but a sure thing--D.E. Shaw could become as well known as Charles Schwab, if not Bill Gates. Shaw envisions nothing less than using his firm's computational muscle to offer free E-mail to every man, woman, and dweeb on the Internet (the rollout begins in March), and later to add online personal financial services, including home banking. He sounds more like the academic he used to be than the mogul he's become when he talks about his business plan. What he's aiming to do, he says, is "to identify ways in which technology has the potential to fundamentally transform our world, and to play a significant role in bringing about that transformation."

Shaw formed his company eight years ago as a hedge fund--an investment partnership designed to use everything from derivatives to short selling to make money in any market environment--with $28 million in startup capital. The most prominent of the handful of original investors were Donald Sussman of Paloma Partners, another hedge fund, and Continental Casualty Co., owned by Loews Corp. Shaw started with six employees and leased his first office space in a loft near Greenwich Village. ("We have the distinction," he notes, "of being the only investment bank to be started above a communist bookstore.") The company made its first trade about six months after startup, turned a profit, and has been making money ever since. Today, D.E. Shaw & Co. has about 300 employees and more than $600 million in gross capital, ranking it among the top 25 securities companies in the country.

HOW HAS the company performed? Very nicely. According to one person familiar with his operations, D.E. Shaw has been averaging 18% annual returns since its inception. Numbers like that put Shaw well above the average, if not into the stratosphere, of money managers. But what really attracts hedge fund investors to Shaw is the fund's low volatility relative to the broader market. Last year, through December 1, the take was around 16%, well behind the S&P 500. But in 1994, a year when most hedge funds either lost money or barely broke even, Shaw delivered a 26% return.

Getting this kind of hard information about D.E. Shaw isn't easy. In fact, getting practically any information about the company is difficult. The paragraph you just read may well be the most detailed account of Shaw's finances that's ever been published. Shaw's penchant for secrecy is legendary; his attitude toward the press has been indifferent. The company not only refuses to discuss specifics of its investment performance--it even declines to reveal simple facts such as the names of all of its original partners, though they are listed on a publicly available document. (True, you have to track down the limited partnership papers at the Secretary of State's Office in Albany, New York.)

To make sure nothing gets out that isn't supposed to get out, Shaw has all his employees sign nondisclosure agreements, and these gag orders do their job well. Two former employees who initially agreed to be interviewed by Fortune changed their minds after sleeping on the decision, both citing the nondisclosure agreements. Only one would say anything at all, to wit: "I only have positive things to say anyway."

THE SECRECY is understandable when it comes to the firm's proprietary technology--what Shaw calls "our life's blood." Shaw's market-beating algorithms are so secret, even limited partners such as Morgan Miller (one of Shaw's earliest investors and an executive at National Spinning Co.) aren't entirely sure what's going on behind the curtain. "With most of the investments I have, I understand exactly what's going on. I don't with David," says Miller. "It does bother me in a way. But it's something I can live with." Shaw himself will give only a coarse description of the statistical-arbitrage trading strategies his mathematicians have invented, which are designed to exploit tiny pricing mismatches, taking a little profit here and a little more there, in the hope that by the time you add it all up, you come out well ahead. Shaw is quick to point out that this strategy is "market neutral," meaning the goal is finding these little profit pockets without actually betting on the direction of the market. "Although to be fair," he says, "if we did know how to do that, we would." (For more on statistical arbitrage, see "How Quants Like Shaw Make Their Money" on the next page.)

Given his reputation as an amalgam of Einstein, Midas, and Rasputin, Shaw in person turns out to be surprisingly unpretentious. Consider the unassuming way he explains himself and his firm. "Our goal," he says, carefully picking his words to get it right the first time, "is to look at the intersection of computers and capital, and find as many interesting and profitable things to do in that intersection as we can." He doesn't live extravagantly, either. A favorite restaurant is a Brazilian buffet joint across the street from his offices--"The last time I was here a cockroach was walking across the floor," he tells a dinner companion.

Shaw got a computer science Ph.D. from Stanford in 1980, very close--temporally and geographically--to the epicenter of the computer revolution. It took him eight years to earn his doctorate because he took a few years off to start and run a successful computer software company. He wasn't the only wallah-in-training at Stanford in those days. Leonard Bosack and Andreas Bechtolsheim, co-founders of Cisco Systems and Sun Microsystems, respectively, and Jim Clark, founder of Silicon Graphics and now chairman of Netscape--all were either faculty members or fellow students. "It was a fascinating time to be at Stanford, absolutely incredible," says Jerry Kaplan, a prominent Silicon Valley entrepreneur who was a research associate there at the same time. Although Shaw didn't have a reputation as a superstar, Kaplan recalls, "he was considered one of the core, competent people."

Fresh from Stanford, Shaw got a job as an assistant professor of computer science at Columbia, where he did research on massively parallel supercomputing. In 1986, Morgan Stanley lured him into its analytical and proprietary trading technology group, offering him six times his professor's salary and an environment where there were much greater financial resources at his disposal than he'd ever see in academia. The shift to Wall Street was a natural, says Shaw. "Finance is really a wonderfully pure information-processing business," he says. But he soon started coming up with his own ideas about applying computer technology to financial markets, and although he speaks admiringly of Morgan today, he found the place too restrictive. He wanted to build his own company, as he puts it, "designed from the beginning from a computer science perspective." A year and a half after joining Morgan, he quit to form D.E. Shaw & Co.

Despite all his success as a pathbreaker in these arcane forms of proprietary trading, Shaw--incredibly--says that if he were starting out today, he'd probably stay out of the business altogether, simply because it's gotten so much harder to break into. As competition has increased, profit opportunities have gotten even tinier. Over the past eight years, D.E. Shaw has spent roughly $100 million developing its technology and uncovering new ways to beat the market. State-of-the-art computer hardware is everywhere, but the real soul of the operation is the software. Trying to re-create the systems it has come up with would cost much more today than the original investment. "For us, proprietary trading is still very important and very good," Shaw says, "but that's partly because we lucked out." In other words, they got into the statistical arbitrage game early, when it cost only $100 million to play. THE COMPANY already devotes two-thirds of its personnel to newer services aimed at outside customers, unlike proprietary trading, which is all for its own account. For the past couple of years, Shaw has been using his $100 million technological toolbox to build a financial services division. Among other functions, this new client-seeking enterprise specializes as a market maker for obscure, complex securities like Japanese warrants and convertible bonds. One reason these issues are attractive to Shaw is that they're hard for most people to figure out--that is, those without his computational firepower find them more trouble than they're worth. Perhaps a more important reason, says Andrew Schwaeber, a trader of Asian securities at Highbridge Capital who has experience dealing with D.E. Shaw both as customer and broker, is that Shaw got into the Japanese warrant and convertible business at a time when the business was declining and other securities firms were scaling back because they weren't making enough money. The market opened up, Shaw jumped in, and now, says Schwaeber, "they're doing it better than anybody."

Okay, so Shaw built his firm into the quant shop other quant shops dream about, and since then has blazed a path toward bona fide investment bankdom. Then what's a smart outfit like D.E. Shaw doing flirting with that crowded no-profit zone, the Internet? "I'm embarrassed to tell anybody, just because everything in the newspaper seems to have the word Internet," Shaw admits. But, media drenching notwithstanding, he thinks cyberspace really is a place where he can do business--and where he can compete with the likes of Microsoft, Sun Microsystems, MCI, and every other high-tech company frantically trying to capture a market no one can even measure yet.

THE FIRST Net project is a spinoff called Juno, a free E-mail service. The simple, sensible pitch of Juno is right there in its advertising material, scheduled for direct mail, broadcast, and print onslaught in March: "E-mail was meant to be free." Users of the service don't pay a dime--not if they just want to send and receive E-mail to or from anyone anywhere on the Internet. "It has a huge potential," says Phoebe Simpson, an online analyst at Jupiter Communications. "There's a huge market for people who want to simplify the E-mail process, who go online just for their E-mail." The profits in this scheme will theoretically come through ads displayed onscreen as people use the software. "It's like network TV," says Charles Ardai, Juno president--a former pulp mystery writer with a background in marketing. "You want to watch Seinfeld? You get a few ads." Optional fee-based services may be added later, like online shopping or the ability to attach graphics or spreadsheet files to your E-mail. "If you want to send pictures of your kids, God bless you, but it may not be free," says Ardai.

Even more audacious than Juno is the second venture, called Farsight. Essentially, Farsight's ambition is to become your online banker, broker, insurance salesman--whatever you do with your money, they want you to do it with them on their Web page. The service isn't ready for outsiders to test, although the prototype Shaw demonstrated for Fortune looks slick and easy to use. Shaw says he's lined up some partners; he's also talking to commercial banks and mutual fund companies, but (there's that secretiveness again) he won't say who. Will it succeed? Remember, a successful online personal finance enterprise has been the elusive dream of the entire banking industry for more than a decade. It's something no one from Citibank to Microsoft has figured out how to do yet. D.E. Shaw's Farsight could be the online personal finance breakthrough everyone's been waiting for. Or not.

On a recent evening, Shaw whirls into his lobby, apologizing profusely. He's a half-hour late for an appointment because of an urgent phone call from Richard Gephardt. The House minority leader wanted to ask a fellow liberal Democrat what would really happen on Wall Street if the government defaulted on its debts. ("I told him it wouldn't be good," says Shaw.) Shaw is also an acquaintance of President Clinton's--they met while Clinton was still governor of Arkansas--and has served on presidential advisory bodies. Back in college he was an early anti-Vietnam protester.

Yet another persona. Could this quant among quants, with his massively parallel business ambitions, also have an incipient case of Potomac fever? Shaw laughs at the suggestion. He can't imagine going into politics. Or leaving the private sector, period. Or being bought out. "The most fun I've ever had has been serving as CEO of this company," he says. More he won't say, just as he's not about to tell anyone how he beats the market.

Reporter Associate Tricia Welsh