March
2007
The Need for Lists - the Power of Junk
Interesting article by a college president questioning the usefulness of the US News and World report College Rankings.
There are a few threads to follow: Sarah Lawrence decides to stop using SAT scores, without affects on the students applying. However US News will calculate an arbitrary average SAT score for that school without regards to reality. So the school decides to consider dropping out, i.e. supplying even fewer statistics to US News which will then fill in again for them. So if they decide not to participate, they will be participated by a news organization highly dependent on the success of those rankings.
So lies are being uses as statistics. A national ranking system turns out to be anything but. A national news magazine ends up losing all integrity. And a college is being under the gun to stand up to principle, or to be broadly misrepresented by those with the power. Who ends up good coming out of this?
Thank you for writing this article. I hope that we get some sort of retraction from US News & World Report, but if there isn’t - shame on you.
Too bad we’re all so dependent on lists - that we need rankings to make informed choices.
(Original article posted here.)
The Cost of Bucking College Rankings
By Michele Tolela Myers
Sunday, March 11, 2007Like most college presidents, I have seen many prospective students and their parents show up on campus in recent months, clutching their well-worn copies of U.S. News & World Report’s rankings issue. U.S. News has smartly tapped into students’ need to sort out colleges and universities in a rational way. Parents, who face increasing college costs, understandably want to know where best to make that expensive investment.
U.S. News benefits from our appetite for shortcuts, sound bites and top-10 lists. The magazine has parlayed the appearance of unbiased measurements into a profitable bottom line.
The problem is that the U.S. News college rankings are far from reliable.
Turns out that some of their numbers are made up. I know that firsthand. Two years ago, we at Sarah Lawrence College decided to stop using SAT scores in our admission process. We didn’t make them optional, as some schools do. We simply told our prospective students not to bother sending them. We determined that the best predictors of success at Sarah Lawrence are high school grades in rigorous college-prep courses, teachers’ recommendations and extensive writing samples. We are a writing-intensive school, and the information produced by SAT scores added little to our ability to predict how a student would do at our college; it did, however, do much to bias admission in favor of those who could afford expensive coaching sessions.
Since we dropped the SAT altogether, we no longer provide SAT information to U.S. News & World Report. Our two years’ experience with this practice has been very good. Faculty members report that our students continue to be terrific. Their average high school grades, high school ranks and grades in Advanced Placement courses have not changed.
But this principled decision has put us in jeopardy. I was recently informed by the director of data research at U.S. News, the person at the magazine who has a lot to say about how the rankings are computed, that absent students’ SAT scores, the magazine will calculate the college’s ranking by assuming an arbitrary average SAT score of one standard deviation (roughly 200 points) below the average score of our peer group.
In other words, in the absence of real data, they will make up a number. He made clear to me that he believes that schools that do not use SAT scores in their admission process are admitting less capable students and therefore should lose points on their selectivity index. Our experience, of course, tells us otherwise.
But the story does not stop here. When I reported this conversation at Sarah Lawrence, several faculty members and deans suggested that perhaps it was time to stop playing ranking roulette and opt out of the survey. A few colleges explore this option each year, but most don’t follow through (Reed College is one of the few exceptions I know of), because, like unilateral disarmament, unilateral withdrawal from the U.S. News ranking system is dangerous.
We discovered how dangerous it can be through a presentation U.S. News made at the 2006 meeting of the North East Association for Institutional Research. There, the magazine indicated that if a school stops sending data, the default assumption will be that it performs one standard deviation below the mean on numerous factors for which U.S. News can’t find published data. Again, making up the numbers it can’t get.
The message is clear. Unless we are willing to be badly misrepresented, we had better send the information the magazine wants. We haven’t yet decided what we will do. But if we don’t go along, we understand we will be harmed because many students will assume that Sarah Lawrence is much less selective than it actually is.
The reality is that the magazine’s rankings issue has a large circulation and that parents and students rely on these rankings to make a college choice that has enormous educational and financial implications. This gives the magazine the power to keep colleges playing the game it sets and controls.
Why should we care if we lose our place in their rankings? Because ultimately, so many people take these rankings seriously. I would at least like to let them know how misleading the whole affair is.
The writer is president of Sarah Lawrence College.