11. Evaluating TLT: Suggestions to date, and some old beliefs

For the next couple weeks, I'll be writing about evaluation of eLearning, information literacy programs, high tech classrooms, and other educational uses of technology.

Actually, I've been commenting on evaluation in many of the prior posts, so let's begin with a restatement of suggestions I've made over the last 2 months in this blog series:

1. Focus on what people are actually doing with help from technology (their activities, especially their repeated activities).
2. Therefore, when a goal for the technology investment is to attract attention and resources for the academic program, gather data about whether program activities are establishing a sustainable lead over competitors, a lead that attracts attention and resources.
3. When the goal for technology use is improved learning, focus on whether faculty teaching activities and student learning activities are changing, and whether technology is providing valuable leverage for those changes. (Also assess whether there have been qualitative as well as quantitative improvements in outcomes.)
4. When the goal is improved access (who can enter and complete your program), measure not only numbers and types of people entering and completing but also study how the ways faculty, staff and students are using technology make the program more (or less) accessible and attractive (literally).
5. When the goal is cost savings, create models of how people use their time as well as money. And focus on reducing uses of time that are burdensome, while maintaining or improving uses of time that are fulfilling.
6. When the goal is time-saving, also notice how the saving of time may transform activities, as in the discussion of Reed College in the 1980s, where saving time in rewriting essays led to subtle, cumulative changes in the curriculum and, most likely, in the outcomes of a Reed education.
7. Gains (and losses) in all the preceding dimensions can be related. So your evaluation plan should usually attend to many, or all, of these dimensions, even if the rationale for the original technology use focused on only one. For example, evaluations of eLearning programs should examine changes in learning, not just access. Evaluations of classroom technology should attend to accessibility, not just learning.

Years ago, I might have looked at a list like this, and also agreed that:

1. Evaluation should assess outcomes. (how well did we do in the end?)
2. Evaluation should therefore be done as late as possible in the life of the initiative or project, in order to give those results a chance to become visible (and to resolve startup problems that might have initially obscured what the technology investment could really achieve).
3. Corollary: When writing a grant proposal, it's helpful to wait until you've virtually completed the project plan and budget before calling in someone like me to write an evaluation plan for you. Just ask the evaluator to contribute the usual boilerplate by tomorrow; after all, evaluation plans are pretty much alike, right?
4. Corollary #2: If the project succeeds, it will be obvious. If it fails, evaluation can be a threat. So, when developing a budget for your project or program, first allocate every available dollar for the real work. Then budget any dollars that remain for the evaluation.

Do those last four points sound familiar? Have any of those four ideas produced evaluation findings that were worth the time and money? (Tell us about it.) When you plan a project, what purposes do you have for the evaluation? In a couple days, I'll suggest some alternative ideas for evaluation.

PS. This Friday, October 23, at 2 PM ET, please join Steve Gilbert and me online for a live discussion of some of these "Ten Things". Please register in advance by going to this web page, scrolling down to October 23, and following the instructions. And, to help us plan the event, tell us which ideas you'd especially like us to discuss.