(FORTUNE Magazine) – Stephen Carter was studying law in 1978 when the Supreme Court's ruling in the celebrated Bakke case imposed limits on affirmative action. At the time, he and other black students at Yale felt that ''someone seemed to be pointing at us and saying, 'You don't belong here.' '' Hurt and infuriated, Carter and his friends picketed a meeting of the Yale Political Union, a student organization that was holding a debate on affirmative action. When his group was invited to join in, Carter angrily took the podium and attacked the assembly -- mostly white, ''immaculately attired'' undergraduates -- for demanding colorblind admissions. He left elated: ''We were, for a shining moment, in our glory.'' A few years later, as a young lawyer looking for a teaching job, Carter saw that triumphal moment in a more critical light. He had put together an impressive resume: member of the Yale Law Journal, practice with a good law firm, a clerkship for Justice Thurgood Marshall. He soon found that any school would be happy to have a black professor with his credentials, but the offers left him soured. He wanted to be judged on his merits, and was insulted to find himself viewed as ''the best black.'' He and his black colleagues, he realized, ''were not prepared to discuss or even to imagine life without preferences, a world in which we would be challenged to meet and beat whatever standards for admission and advancement were placed before us.'' , That change of heart -- and the tentative way he expresses it -- conveys a great deal about the balanced, middle-of-the-road position that Carter stakes out in his garrulous, repetitive, disorganized but ultimately engaging Reflections of an Affirmative Action Baby (Basic Books, $23). His book gains interest because it focuses on an issue critical to the debate over Clarence Thomas's nomination to the Supreme Court. To make his points, Carter draws heavily on personal experience. Born to a middle-class black family -- his father taught at Cornell's business school -- he was a good student in high school and at Stanford, but it took the added weight of affirmative action to get him into Yale Law School. ''Were my skin not the color that it is,'' Carter remarks with characteristic candor, he would never have had the chance to compete with the best. He obviously did so, with considerable success: He is now a professor at Yale Law School. While he refuses to deny ''the beneficiary's side of the saga'' of affirmative action, Carter believes that the saga is rapidly -- and deservedly -- coming to an end. The policy's unpopularity in the nation is growing, the Republican Party is preparing to make a campaign issue of it in 1992, and its unfairness to whites amounts to ''an insistence that all members of the disfavored dominant group bear the mantle of the oppressor. ((It)) becomes almost a punishment for the sin of being born the wrong color.'' Much more important, he argues, is affirmative action's ''stunning irrelevance.'' Racial preference programs, he argues, are increasingly dominated by the children of the middle class -- even though data on income show that middle-class blacks, unlike those still trapped in the inner cities, need no such help. The fault for that skewed result lies not just with affirmative action but also with society at large, which, Carter says, ''prefers its racial justice cheap.'' Affirmative action makes it easier for society to reject a costlier, more sweeping civil rights agenda -- especially improving early education and medical care -- that would do most to change the lives of poor blacks. Carter also points out that the current defense of affirmative action by orthodox black leaders argues for having blacks in high places because they bring with them ''a distinctive voice, a vision of the world,'' that would otherwise be lacking. He calls that view a new kind of racism because it postulates that there is only one way to be black, that ''if you know the color of somebody's skin, you know what the person values and what causes the person supports.'' Carter says the Supreme Court accepted this idea of a monolithic black world view when it upheld FCC regulations favoring minority ownership of radio and TV stations -- a ruling that ''places government imprimatur on racial generalizations.'' Choosing his words with a caution that too often muddies his points, Carter faults today's black leaders (whom he almost never names) both for insisting passionately on affirmative action and for ostracizing dissenters who resist the notion that blacks in authority must conform to a ''correct'' way. By imposing such a loyalty test -- by arguing that ''biology implies ideology'' -- black leaders deprive their people of the contribution fine minds could make by bringing better answers to the black predicament. At this dangerous but important moment in black history, Carter believes, blacks cannot afford the luxury of insisting on ''a single, unchallengeable answer.'' CARTER'S BOOK evidently went to press before Thomas was nominated for the Court. Carter puts him in that group of black dissenters whose refusal to accept that single, unchallengeable answer has aroused the enmity of conventional leaders. In their view, Carter says, Thomas's principal sin is ''to combine a black skin with an unabashed conservatism.'' That is anathema to the leaders, even though on many issues -- abortion, school prayer, the rights of criminals -- black Americans are more conservative than much of the nation. This effort of the leadership to silence debate ''solves no problems; it simply limits the range of possible solutions.'' No conservative himself, Carter suggests that his own political views would probably brand him a left-liberal -- but he'd prefer no label at all. In the end this iconoclast advocates refocusing affirmative action on its ''innocent roots'' by offering training and other opportunities only to the most economically disadvantaged blacks. For blacks as a whole, Carter concludes, the job now at hand is to face the tests of the market -- not to try to discredit standards of excellence, but to accept them and meet them. ''We can complain about it, or we can help ourselves.''