More Enron-ization of rankings.

As TaxProf Blog reports (here), UMKC has had an outside audit (here) to uncover rankings shenanigans.  What should we learn from examples like this one?  The same thing we should've learned decades ago: people respond to incentives.  For the best examples of perverse incentives, see this Freakonomics post.

Update on my Gonzaga piece on law schools, lying, and Enron.

After I published this piece, Robert K. Vischer, the dean of the University of St. Thomas Law School (Minnesota), gently suggested that I might want to clarify St. Thomas's restatement of its employment data: 

[Y]ou refer to St. Thomas Law School as having done “the right thing and ‘fessed up when it discovered that it had been misrepresenting its placement data.”  This is an important paper on an important topic, and I’m glad you’re tackling it head on.  I fear, though, that this line may lead readers to lump St. Thomas in with other law schools that have shown a pattern of deliberate misrepresentation.  Our oversight was a one-time occurrence that resulted from an erroneous entry of data.

Specifically, on line 169 of our Class of 2010 U.S. News Employment data report, the number of graduates known to be employed at graduation was correctly listed as 51 graduates (or 32.9% of our 155 total number of 2010 graduates). Unfortunately, on line 164 of the report, we incorrectly listed 125 graduates (or 80.6%) as employed at graduation.  U.S. News listed that incorrect number in its law school rankings.  We immediately contacted U.S. News to alert them to the error. The nine month graduation rate of 86.5% was correct in the rankings.

We did make a one-time mistake, and we paid the price by being unranked for a year.  I’m hoping to avoid leaving the impression that we are another example of law schools trying to fudge the truth over a period of years.

I'm happy to help Dean Vischer clarify his school's role.  There's a huge difference between making a mistake and engaging in a pattern of deception.

Thanks, Wall Street Journal Law Blog and TaxProf Blog!

Nice to be in the spotlight (here and here).

A template for law schools' press releases, now that the USNWR rankings are in.

Well, the rankings are out, and the annual "let's figure out what to say in our press release" dilemma begins.  Here's a template for some law schools to consider.

Dear [stakeholder or possible donor]:

We are pleased to tell you that our law school ranking [increased more than we expected] [increased a negligible amount, but we're going to tout the change as if it were huge] [went down far enough that we've called in sick for the foreseeable future] [went down a negligible amount that will provide us with a disproportionate amount of grief for far too long] [stayed the same].

We are, of course, the best school in [the country--if we're Yale] [the region] [the state] [the city] [the part of the city in which we are located].  Our quality is reflected by [our reputational ranking] [our admissions numbers] [our placement rate] [our bar passage rate] [the ability of our marketing people to think creatively].

Even though we're [superb] [excellent] [very good] [really, really decent] [surprised that we're still here], we have committed to doubling our efforts to [track down unemployed alumni on various social media sites so that we can fill in the now-less-easy-to-cheat-on placement statistics] [rethink our curriculum in changing times] [write imaginative press releases that make us look as if we're moving in the right direction] [figure out how to steal budget money from other units on campus].

We thank you for your support, and we look forward to giving you even better news next year.

Of course, it's easier for me to write a post like this one given that we did just fine in the rankings.  But some of the press releases that are pouring in are just too funny to ignore.

I am addicted to lawprofblawg.

And here's a perfect example of why I'm so addicted to it.

Brian Leiter has it exactly right when it comes to faculty credentials.

See here.

Quelle surprise! Law firms manage their profits per partner for the rankings.

See here.  At what point do we stop measuring things that don't really matter, rather than chase the tail of the rankings dog?

USNWR likely to change its rankings methodology for placement.

Hat tip to TaxProf Blog, with this post (here).

I'm obviously no fan of the rankings or how law schools use them as a measure of quality, but I'm even more disappointed in schools that "manage" their employment numbers by, well, lying about them.  What else could USNWR do but find a workaround for the gaming that's been going on?

Chicken, meet egg.

The ABA Journal is reporting (here) USNWR's reaction to being blamed, at least in part, for the frenzy to buy high UGPA and LSAT scores with merit scholarships.  Last Sunday, the New York Times reported (here) that many of the students who received merit scholarships didn't retain those scholarships after the first year.  The consequences of losing a merit scholarship at an expensive school can be dire. 

It's true that the USNWR rankings don't cause students to lose their scholarships.  (The cause is a combination of a scholarship's requirements, the grading curve, and the individual student's abilities--as well as any particular personal crisis that the student might be having during the first year.) 

It's equally true that schools are buying "high-numbers" students as a way to improve their rankings, given how much UGPA and LSAT figures can drive the rankings (here) and how few other ways there are to game the system.   It's really difficult to affect the peer and lawyers/judges rankings (notwithstanding all of the glossy brochures that we all get in the fall); in this economic environment, placing students in real jobs at graduation is also really difficult.  Bar passage isn't a large enough factor in the rankings.  What does that leave?  Ah, yes:  "the numbers."

I can't remember a time before the rankings.  (I can't remember a lot of things, though, so I'm not worried about this particular lapse in my memory.)  At some earlier point, didn't we select our students based on our predictions of the applicants' success in our schools?  We used UGPAs and LSATs for those predictions, but we also used non-numerical predictors of academic success.  Schools may still use those, but they sure do pay attention to how "the numbers" are going to look at matriculation.

No matter how we admitted students before, "the numbers" and the USNWR rankings are now officially a chicken-egg problem.

Seriously?

I'm always disappointed when law students go ballistic over a downward move in the USNWR rankings (see, e.g., here, discussing Emory's brouhaha).  I know, I know:  the higher-ranked the school, the easier it is to get a foot in the door at BigLaw firms.  I get that, and the fact that the USNWR rankings matter so much to BigLaw is a big flaw in the BigLaw recruiting process.  So shame on those BigLaw firms that only look at GPA cutoffs and rankings.  They're missing out on some stellar people who don't fit the traditional BigLaw profile but who would do wonderful work for them.

But let's face it:  USNWR is a magazine.  It sends out surveys to a very small sample size (for you non-statistics geeks, that means that the surveys aren't providing very reliable information), and those surveys count for way too much of the final scores.  The rest of the scores are based on criteria that are too easily manipulated.

So basing one's view of one's law school on the scoring system that a magazine uses to sell issues is a bit like caring that one's speakers go to 11.

If the dean at Emory resigned just because of the rankings, that's a sad commentary on the misplaced importance that people place on this single way of viewing the array of ABA-accredited law schools.  On the other hand, I know from first-hand experience how much fun it is to transition back to a faculty after having been a dean.  So, welcome back to the life of a law professor, David.  You'll enjoy it.

Yep. I was a chump.

See here.  This TaxProf Blog post points out how much creative accounting folderol goes into reporting employment stats of law schools to U.S. News. 

As I've said before (see here), lying on these questionnaires isn't much different from the "earnings management" that went on at Enron and the other like-minded companies.  Refusing to lie puts schools at a huge disadvantage, but lying just results in a race to the bottom in accuracy while pursuing a race to the top in "100% employed" reports.

Reason number n why Andy Morriss and Bill Henderson rock.

Over at the Legal Profession Blog, Bill Henderson has posted an explanation of what it would take for Stanford Law School to overtake Yale in the U.S. News rankings (see here).  In this post, Bill explains that he and Andy Morriss create an annual simulation model of those rankings, and that this model has helped them figure out what Stanford would have to do in order to become #1 in the rankings.  The price tag for becoming #1?  Somewhere between $350 million and $1.8 billion.

Bill's conclusion?  No one could say it better:

The legal profession, especially our students, have some big problems at the moment.  And society's are even larger.  The best law school is one that prepares its students to solve these problems.  This requires a careful balance of innovative teaching and scholarship.  The U.S. News rankings don't capture these metrics.  In fact, they obscure them and create incentives for truly destructive behavior.  By and large the deans are trapped.  From my own perspective, I don't think even one law school in the US News Tier 1 has reached even 10% of its potential to educate and solve problems.  Too many one-professor silos.  Too much ego.

I am sorry to moralize.  But someone needed to say it.  Let's focus on some problems worth solving.  At the end of the day, it will be worth it.

Bill, I'm glad you said it.  BRAVO.

Do law school grades mean anything?

This morning's New York Times has a front-page article (here) about law schools raising their curves to make their students more attractive as job applicants.  (Hat tip to Scott Unger and my dad.)

That change reminds me of Ellen Degeneres's routine about the difference between "upright" and "reclined" on airplane seats (here).  We can call something whatever we want, but that doesn't change the nature of the thing.  Even if the curve goes higher (say, a curve around a B+ instead of a B-), it's still an ordering of all of the people in the course.

Curved grades tell people how a given person did on a given exam on a given day, compared to the other people who took the exam that day.  It's rather like George Carlin's old joke about partial scores in sports ("Notre Dame, 6").  It's information, but it's not complete information.

Employers use grades and the U.S. News rankings of schools because they believe that those two data points are approximations of how "good" the job candidates are, even though grades are actually not such a good shadow variable.  Very few clients ask their lawyers to answer hypotheticals for three hours.  There are a lot of the intangibles that make a good lawyer good, such as common sense, emotional intelligence, and teamwork, and grades don't necessarily correlate with those characteristics.  I keep telling employers that taking a little more time on the front end (for example, by asking job candidates to talk about how they'd approach a particular situation) would pay off in the long run (retention of good candidates).

But until employers are willing to spend more time looking at more than mere transcripts, I can understand how tempting it is to raise a school's curve.

Why not go all the way, though?  Some schools make it downright difficult to get anything but As.  Playing with curves is just the start of something big--why not grade for mastery instead of for comparison purposes?  Instead of curving grades, establish clear standards for what performance would constitute an A, what lesser performance would earn a B, etc.  Then grades might actually be useful to the outside world.

In the meantime, folks might want to read Dan Keating's article on grades:  Daniel Keating, Ten Myths About Law School Grading, 76 Wash. U. L.Q. 171 (1998).  It's a fun read.

Update (6/25) (hat tip to my buddy Seymour Serebnick):  see the NYT's letters to the editor about the original story here.

Brian Leiter's latest letter to USNWR is a must-read.

See here.

Newsflash: small, newly established law schools not in National Law Journal's top 25

Thanks to my home Google page, I read TaxProf Blog regularly, so I clicked on the links (see here and here) to the National Law Journal's Law School Report (here).  The basic methodology:  "[NLJ has] ranked the top 50 law schools by the percentage of 2009 juris doctor graduates who snagged jobs at NLJ 250 firms by Sept. 30, 2009."

What I like:  the ranking at least gives users some useful information about the link between going to a particular school and how many of that school's grads go to BigLaw; my alma mater (Stanford) did well, even though many older grads (25th reunion this year!) are BigLaw refugees; and there are thoughtful articles that accompany the report (see Emily Spieler's thoughts here). 

The only drawback is that some newer schools (think Boyd) won't make the cut for a while, if ever:  we're still new-ish, as law schools go (in two more years, the school could have a Bar Mitzvah), and we don't have the benefit of having a plethora of branch offices of BigLaw firms in Nevada, although we do place graduates in many other states.

All in all, though, the Law School Report makes for juicy reading.

Will the crash in BigLaw hiring be a boon for lesser-"ranked" law schools?

We won't know for a while, but if some law schools didn't place a lot of their grads in BigLaw firms before the crash, and the graduates at those "lesser" schools find more jobs than those do who hail from "elite" schools, will we see a big change in the USNWR rankings in a couple of years, when the data from the placement crash play out?

Schadenfreude.

Schadenfreude is, by definition, rather satisfying.  See this Above the Law post (here), which also contains a nifty Marine Corps reference ("Welcome to the suck.")

Roger Williams University's latest study of faculty productivity outside the USNWR top "40"

See here for the whole study.  My favorite part is reproduced below (hat tip to Brian Leiter and to Paul Caron):
THE TOP 40 SCHOOLS [AFTER USNWR'S TOP 40] 
1. University of San Diego (13.85)
2. Florida State University (13.30)
3. University of Richmond (9.56)
4. University of Missouri - Columbia (8.82)
5. Illinois Institute of Technology (Chicago-Kent) (8.71)
6. University of Nevada - Las Vegas (8.54)
    Case Western Reserve University (8.51)
8. University of Cincinnati (8.15)
9. Brooklyn Law School (8.00)
10. Pepperdine University (7.45)
11. Roger Williams University (6.76)
      University of Florida (6.73)
13. Hofstra University (6.56)
14. Arizona State University (6.44)
15. University of Pittsburgh (6.25)
16. Seattle University (6.10)
      Temple University (6.06)
18. Seton Hall University (5.92)
19. University of Houston (5.84)
      Loyola Law School - Los Angeles (5.82)
21. DePaul University (5.78)
      Rutgers (State University of New Jersey) - Newark (5.78)
23. University of Tennessee (5.47)
      University of Kentucky (5.42)
      University of Miami (5.41)
26. Villanova University (5.19)
      University of Louisville (5.14)
28. Rutgers (State University of New Jersey) - Camden (5.06)
29. Loyola University Chicago(4.97)
30. Capital University (4.61)
      Indiana University - Indianapolis (4.56)
32. Pennsylvania State University (4.52)
33. Willamette University (4.35)
      Wayne State University (4.33)
35. University of Connecticut (4.29)
36. Santa Clara University (4.14)
      University of Oregon (4.13)
      St. John’s University (4.13)
      Michigan State University (4.11)
40. Marquette University (4.00)

It's nice to see that folks recognize how prolific our faculty's scholarship has been.  Our faculty productivity is one of many reasons I love it here.

More foolishness coming about because of the USNWR rankings

This story's been brewing for a while now: the rankings are costing law students in terms of higher tuition. See this link from the TaxProf Blog here and Robert Morse's point of view here.

Legal Profession Blog is, of course, right again.

See the various posts about the new, flawed SuperLawyers rankings here and here.