Using data to compare institutions

A thread in the ASSESS listserv recently turned to questioning the value of collecting comparable data from comparable institutions.

For me, the biggest reason for caution about comparisons may be 'Bowen's Law' (also known as the revenue theory of costs).

Sponsored by the Carnegie Commission on Higher Education, Howard Bowen tested the hypothesis that higher education instruction has a production function, i.e., that there is some normal relationship between spending and outcomes. When someone remarks that "I can't run a program with that budget; it's too little money," or enviously comments about another program, "With that endowment and budget, they must be wasting a lot of money," or simply spends time collecting comparable data in order to compare their program with the norm (can we boast our results are 5% better than those achieved by other programs that have about our same budget? can we use below average budgets to justify a request for an increase?) --- all of those imply a faith in a production function.

But Bowen found no such evidence to support the belief in a production function. When he controlled for institutional type, context and reputation, he still found that spending per student at the institutions in that homogenous group was all over the map. And when Bowen analyzed how each instructional dollar was spent, that spending varied just as widely. His findings supported the revenue theory of costs:
1. Higher education institution raise all the money they can
2. Spend all the money they get
3. And spend each dollar in ways largely dictated by how they spent it last year.

To put it another way, when you see variations in spending, it's likely a function of the varying histories of fund-raising. So cost comparisons are not a rational guide for budgeting.  And I think that casts doubt on the value of seeking a norm among 'comparable' institutions in educational performance. What does it mean to say that a group of institutions are comparable?

For me, the important reasons to compare data among institutions are at the extreme ends of any distribution.

As a taxpayer, I'm interested in seeing pressure or even sanctions brought to bear on institutions that are doing a miserable job using my money to teach core abilities and skills that most institutions can teach well.

And, at the other end of the distribution, I want to know what kinds of excellence are indeed possible. One of the corollaries of Bowen's Law is that almost anything is possible, at almost any budget level, if ideas, circumstances, and strategies are appropriate and if one works in that direction slowly and methodically enough (remember element #3 of Bowen's Law - change comes slowly)  But knowing that anything is possible, and seeing that some institution has actually achieved at an extraordinary level - those are two different things.

Twenty or thirty years ago, I talked with someone at a NJ institution (Ramapo?) whose labor studies program had recently begun teaching a BA in labor studies to union members by sending faculty to teach courses in union halls.  I asked how they had started the program. He replied, "When we heard that a labor studies program at a NY university was doing it, we couldn't not do it. Of course our program is not like theirs. We had to do it our way. But that's why we did it."

Now that was a valuable use of comparisons!

What do you think? Under what circumstances is it valid to collect test data, budget data or other data to compare institutions or academic programs?

*****

Bowen, Howard R. (1980) The Costs of Higher Education: How Much Do Colleges and Universities Spend per Student and How Much Should They Spend? San Francisco: Jossey-Bass.