K. Evaluation should be mainly formative and should begin immediately.

Earlier, I described some old beliefs about program evaluation. I used to assume that evaluation of TLT had to be summative ("What did this program accomplish? Does that evidence indicate this program should be expanded and replicated? continued? or canceled?"). The most important data would measure program results (outcomes). You've got to wait years to achieve results (e.g., graduating students) and the first set of reults may be distorted by what was going on as the program started up. Consequently, I assumed, evaluation should be scheduled as late as possible in the program.

Many people still believe these propositions and others I mentioned earlier this week. I still get requests: "The deadline for this grant program is in two days. Here's our draft proposal. Out of our budget of $500,000, we've saved $5,000 for evaluation. If you're able to help us, please send us a copy of your standard evaluation plan."

Yug!

Stakeholders need to see what's going on so they can make better, less risky decisions about what to do next. Getting that kind of useful information is called “formative evaluation.” (By the way, a stakeholder is someone who affects or who is affected by a program: its faculty, the staff who provide it with services, its students, and its benefactors, for example.)

In the realm of teaching and learning with technology (TLT), formative evaluation is even more important than in other realms of education. The program is likely to be novel and rapidly changing, as technologies and circumstances change. So the stakeholders are on unfamiliar ground. Their years of experience may not provide reliable guides for what to do next. Formative evaluation can reduce their risks, and help them notice and seize emerging opportunities.

Formative evaluation also can attract stakeholders into helping the evaluator gather evidence. In contrast, summative evaluation is often seen as a threat by stakeholders. “The summative evaluation will tell us we're doing well (and we already know that). Or perhaps the evaluator will misunderstand what we're doing, creating the risk that our program will be cut or canceled before we have a chance to show what this idea can really do. And no one reads those summative reports anyway unless they're looking for an axe to use on a program. So, no, I don't want to spend time on this and, if I'm forced to cooperate, I have no reason to be honest.” In contrast, formative evaluations should be empowering - a good evaluation usually gives the various stakeholders information they need in order to get more value from their involvement with the program.

What many folks don't realize is that formative evaluation requires different kinds of data than summative evaluation does.

Summative evaluation usually must focus on data about results -- outcomes. But outcomes data by itself has little formative value. If you doubt that, consider that a faculty member has just discovered that the class average on the mid term exam was 57.432. Very precise outcomes data. But not very helpful guidance for figuring out how to teach better next week.

In contrast, a formative evaluation of an educational use of technology will often seek to discover a) what users are actually doing with the technology, and b) why they acted that way (which may have nothing to do with the technology itself). (For more guidance on designing such evaluations, see "The Flashlight Approach" and other chapters of the Flashlight Evaluation Handbook.

Corollary #1: The right time to start evaluating is always "now!" Because the focus is likely to be on activities, not technology, the evaluation of the activity can begin before new technologies or techniques go into use. Baseline data can be collected. And, even more importantly, the team can learn about factors that affect the activity (e.g. 'library research') long before new technology (e.g. new search tools) are acquired. This kind of evaluation can yield insights to assure that the new resources are used to best effect starting on day 1 of their availability.

Corollary #1: When creating an action plan or grant proposal, get an evaluator onto your planning team quickly. An experienced, skillful evaluator should be able to help you develop a more effective, safer action plan.

Corollary #2: when developing your budget, the money and effort needed for evaluation (what the military might call 'intelligence gathering”) may be substantial, especially if your program is breaking new ground.


What are the most helpful, influential evaluations you've seen in the TLT realm? Did they look like this? What kind of information did they gather? Next week, I'm discuss how our current infatuation with learning objectives has overshadowed some very important kinds of evidence, and potentially discouraged us from grabbing some of the most important benefits of technology use in education, benefits that can't be measured by mass progress on outcomes.

Resources for workshops on assessment and evaluation

1. Do something.
2. Pause.
3. Gather some more information.
4. Then decide what to do next.

Sounds easy. A lot of what we do at the TLT Group is helping academics with #3, whether you call it "assessment," "evaluation," "scholarship of teaching and learning," "accreditation self-studies," or something else. We provide Flashlight Online so people can gather their own information. We do external evaluations. Of equal importance, we help educate faculty and other staff so that they can ask the right questions for themselves: questions whose answers are likely to be useful in deciding what to do next.

We continually revise and expand our educational resources, including workshop materials and the Flashlight Evaluation Handbook.

For example, we just added some new material to our list of 'frequently made objections' to assessment. If you're doing an evaluation, or training others, and get some push back, it's time for you to do steps 1-4 above: why are people really objecting? This web page should help you ask them the right questions.

And, if you run workshops or give talks about evaluation, a little humor can help lighten the mood while making a point. Here are some cartoons and stories you might find laughable.

We assembled these resources for our subscribing institutions. If you find them useful and your institution isn't supporting the TLT Group's work, perhaps you could pursue a subscription.

Working on ARQ (Asking the Right Questions)

"Asking the Right Questions" (ARQ) is a set of brief hybrid workshop materials for faculty. The goal: figure out how to improve online discussion, or your lecture slides, or your use of clickers, or your online assignments -- some kind of teaching and learning with technology in your course -- by asking your students some pointed questions about that activity.

Each workshop module is only 5-15 minutes long, short enough to insert in a faculty meeting or brownbag lunch, or to use in a brief online workshop. The ARQ workshops can be led by faculty development or IT support staff, or by faculty who have been through the workshop previously, tried what they learned, and liked the results.

A group of us (join us!) are developing and testing these materials, and the collection grows and changes almost every week. Frank Parker tells me that ARQ materials have been tried out at Johnson C. Smith University. Meanwhile I've been tweaking the materials for improving online discussions.

I'm also using Flashlight Online 2.0 to develop new survey templates (and a new ARQ workshop) on improving the use of clickers, and another ARQ workshop (with Flashlight Online 2.0 materials) for getting more out of your use of electronic portfolios. I'm working with IUPUI on the eportfolio materials; if you like this ARQ approach and want to join in that work please let me know. I don't yet have any partners for work on the clicker workshop materials. Interested?

I'm also thinking that it's time to go after some grant money to speed up the development and testing process, and to evaluate the materials: can this approach really engage more faculty and help build a culture of evidence? Maybe an institution with an accreditation a few years off, or a consortium or association, would like to go after such a grant, and use us as a contractor to help?

Personal notes: After traveling every week since late September, I'm off the road until January. Arthroscopic surgery on my left shoulder coming up, the Tuesday after Thanksgiving. No heroic athletic injury involved: just a bone spur that's been making me increasingly sore. The surgery should clean that up.

Before that, however, Leslie and I are headed for New York City to celebrate Thanksgiving. Chris (our son) and Monique have invited us and also her parents to dinner. This is our first opportunity to meet Monique's parents. We're all pretty excited about this particular holiday.

Asking the Right Questions (cont'd)

We ran our first subscriber and member workshop on "Asking the Right Questions" (ARQ) last week. As you may know, ARQ is a growing collection of materials for brief hybrid workshops, all aimed to show instructors new ways to gather information from their own students in order to improve teaching and learning with technology in their courses.

Last Tuesday's ARQ workshop focused on how to gather evidence that could be used to improve participation in online discussion. The workshop teaches faculty how to frame their own questions, and also how to adapt Flashlight Online model surveys to collect evidence from their own students. The workshop materials also include an eClip to help run the session, handouts, and feedback forms.

We went through the workshop and then critiqued it; the version you can see online today is the result of that critique. I'm sure it will continue to change as we get more experience with the materials. Everyone on Tuesday said they were considering offering the workshop at their home institutions.

This coming Tuesday (Oct. 16 at 3 PM ET) we'll try a workshop on personal response systems (e.g, clickers). Here too we're going to look into how asking students the right questions can show instructors and institutions how to get more value from the PRS itself. The ARQ materials include sample Flashlight Online surveys: one to do a needs assessment of faculty, and the other for faculty to adapt for use with their own students. (If you're interested in Flashlight Online 2.0, take a look -- I used the beta test version to write these surveys.)

If you're at a TLT Group subscriber institutions or are yourself a TLT Group member, I hope you can join us! Maybe you'll get interested enough to help us develop more ARQ materials. Click the link for our calendar; the ARQ workshop is at the top - click to register. It's only open for our subscribers and members but for them (you, I hope), it's free.

Finding an Important Topic to Study

I'm working on workshop materials for our "Asking the Right Questions" (ARQ) program - locally facilitated, brief workshops to help academic staff at our subscribing institutions learn how to collect focused feedback from their students in order to improve teaching and learning with technology in their courses. When I say "brief," I mean that each module is only 5-20 minutes long: short enough to be an agenda item in a faculty meeting, a brownbag lunch activity, or a short online event. the workshop materials are designed so that local staff or peer mentors will have an easy time running them. We'll also offer our subscribers free online workshops to train their facilitators: staff in information technology units, faculty development units, teaching&learning centers, libraries and, of course, faculty members who would lead a peer group through a few of these workshops.

This blog entry is a draft for a workshop to help academic staff take a second step in focusing their inquiries. This workshop builds on the work participants would have done in ARQ Workshop #1.

The goal of this workshop is to learn more about identifying research questions that are most worth the effort to answer. (It takes just as much work to answer a trivial question as to answer an important one.)

In ARQ Workshop #1, participants identified a technology used persistently in their courses, a technology that seemed valuable yet in some way problematic. Then they identified an activity -- a pattern of action that was enabled or supported by that technology - that was also of obvious importance and yet somehow problematic.

What additional characteristics make an activity important to study? After we list a few criteria, we ask participants to identify an activity worth studying in their own courses. And, if they like, they can modify or add to this list of criteria. (If you have such improvements, please e-mail me! We'll improve these materials and acknowledge your contribution.)

• Focus on an activity that 's pervasive, important, and capable of being significantly improved through adroit use of technology. For example, it's important for faculty to understand what students are thinking, in order to adjust teaching on the fly and improve learning. Personal response systems (a.k.a. 'clickers', 'classroom voting systems,' 'online surveys') are spreading, and can help instructors get a more precise sense of what all their students are thinking.
  
• Focus on an activity that can be improved incrementally so that, as you learn more, you can keep improving the activity and its outcomes. That improvement usually comes from learning more and more about how to exploit the activity. For example, over time faculty might learn how to ask more powerful questions using a personal response systems, and how to use the PRS to facilitate Peer Instruction.
• Focus on an activity that is defined enough that you can actually study and improve it without making that research into a career. "Studying how to moderate online discussions" is a broader and more challenging topic than "studying how to get students to pay attention to one another's comments before they post a new comment online."
  
• Focus on an activity that is also important to other courses that these students take. Your study about how to improve that activity are more likely to be of interest to your colleagues: they could use your methods to improve their own teaching, for example. And, these kinds of incremental improvements in an activity become more accepted and effective when the change occurs 'across the curriculum.' Students learn to expect the changed activity and, as they engage in its repeatedly, they become more skilled. So faculty who teach them later on can try more ambitious improvements.
  

Your task: identify at least one such activity, one that's very important for you to illuminate by gathering student feedback.

HINT: List lots of such technology-supported activities - ten or more. Imagine asking your students questions that address your concerns as well as your hopes. Which potential study scares and excites you the most? Is there a potential inquiry that makes you think, "Now that I've thought about this, I can't not learn more about what's really going on here!" (For more on this hint, see this short essay in the Flashlight Evaluation Handbook.)

PS The outcomes of that activity, the activity itself, and the technology are what we call a "triad." 'Triads' are a simple grammar for conceptualizing some important things that go on in a course: technology, activities, and outcomes.

Does What You Learn in Second Life Transfer to Real Life?

"Second Life" is an example of an immersive environment, a simulated world in which participants can move around a landscape, communicate, create things, etc. On EDUCAUSE Connect I have been listening to an excellent podcast:
http://www.educause.edu/ELI072/Program/
12402?PRODUCT_CODE=ELI072/GS07
The moderator, Alan Levine of the New Media Consortium, asked participants at one point the question in the title of this posting. I think participants' responses are quite interesting, diverse, and all to the point.

I'd echo one of the participants in asking, "Does what you learn on campus transfer to real life?" The two questions are parallel. Their common answer, "Learning often transfers less than students and faculty assume. 'What is learned in the classroom stays in the classroom (if it stays at all.)'

I especially like some of the educational lessons that are bundled under the heading, "teaching for understanding." For example, if you learn about something in context A and are only tested on that knowledge or skill in Context A, you're not real likely to be able to apply what you learned, months or years later, to context B. You're more likely to be able to apply what you've learned in some unfamiliar context later on if you were taught about the idea in more than one context, and then tested on your ability to apply it in familiar and unfamiliar contexts.

In short, students benefit when they're given practice and feedback as they 'transfer' learning from one sphere to another. The same is almost certainly true of learning in Second Life. We may look at a student designing jewelry in Second Life and hope that the lessons transfer to real life. But if that transfer is an explicit agenda, and if the learning and assessment are based on transfer, transfer is more likely to happen. A virtual world is a different sort of world, but its still a world and what we know about learning in other spheres applies to humans in virtual worlds, too.

PS If your institution is a TLT Group subscriber, you want to take a look at the chapter in the Flashlight Evaluation Handbook on evaluating educational activities in Second Life; it's Section VI.W. It's still primitive, but it's developing, and we even have a little item bank on this topic in Flashlight Online. We'd love to work with a subscriber or two on studies in this arena, and use the work to develop this further.

The Flashlight Approach to Evaluation, Summarized

For a variety of reasons, educators, their units and institutions now often gather evidence in order to evaluate 'stuff'. By 'stuff' we mean tools, resources, and facilities such as course materials, computer software, classroom design, blended courses, distance learning programs, network infrastructure, libraries, ePortfolios, ... By "evaluate" we mean a purposeful gathering of the evidence needed in order to make better choices about what to do next.

These days when people think about evaluation, their thinking often begins and ends with 'outcomes assessment.' But information about outcomes is rarely enough, by itself, to show how to improve those outcomes. (For more on this point, click here.) The Flashlight approach to evaluation is designed to provide the best insights for improving outcomes for the least effort.

The Flashlight approach has a number of elements, of which these are the most important:


1. Activities: Focus on what people do with the 'stuff' at issue. For example, if you want to get more value from personal response systems (some of which are also known as 'clickers'), you first need to discover what faculty and students are actually doing with the clickers. For instance, are the clickers being used to create structured discussion about difficult ideas? to take attendance? to test memorization? Each of those patterns of use (which we refer to as 'activities') will create different benefits, costs, damage...

2. The dark side: Consider people's fears and concerns about the stuff, not just their hopes and goals. For example, if clickers are being used to take attendance, are students sending friends to class with a handful of clickers?

3. Motives, incentives, and disincentives: Value is created by the way people use stuff. So the best clues for increasing that value come from learning why people use the stuff as they do. For example, if you do a workshop on the value of using clickers for conceptual learning, and 30% of the participants don't start using clickers that way, you should investigate the reasons. It might be rooted in their personal approaches to teaching, their disciplines, your training, reactions of students, their facilities, ... No matter what you discover, it's almost always going to be useful in helping you figure out whether and how to get more value from clickers.

4. Education is not a machine: Even when the same faculty member teaches two sections of the same course, and has taught them for years, what students do and learn will differ. Add clickers, or any other technology that increases options for faculty and students, and that variation will probably increase. That's one reason we emphasize unique uses perspective for evaluation, and not just uniform impact approaches.

5. Collaboration: Whose choices influence how the stuff is used? Whose choices could change how the stuff is used? Those are the people who should be involved in helping design your study. If you need help in gathering data (e.g., getting good response rates to surveys), then they can help. They can also help you make sure that your questions and language are clear and compelling.

6. Start evaluating now: To improve outcomes (including costs), change activities. (Buying new stuff is just a means to change activities). And it's always the right time to begin studying an activity. So start evaluating now.
For example, if you're interested in using clickers to foster conceptual learning, start evaluating conceptual learning (including faculty development) now, whether or not you're using clickers yet, whether or not you're experienced in their use yet, whether or not you're considering replacing one kind of clicker with another. For example, if you are considering buying new stuff, your findings can help you
* choose products;
* remove barriers to effective use even before the technology becomes available, and
* provide baseline data for measuring the impact of the new stuff.
In short, whether or not it's time to buy, develop, or replace stuff, it's always time to begin studying activities that use, or that would use, that stuff.

The whole Flashlight Evaluation Handbook is designed to flesh out these and related ideas using examples and specialized guides. The TLT Group can also provide coaching (send us your draft plans and we can talk), collaborators in doing such studies, or we can even do them for you. So take a look at the Handbook and contact us if you'd like to talk (301-270-8311; flashlight@tltgroup.org).

PS. If your institution is a subscriber, feel free to use this summary for workshops. If your institution is not yet a subscriber, we would appreciate it if you would ask permission to use this material.