Week 3: Evaluating Programs and Human Performance Technology

Have you ever planned a lesson without thinking about how you would evaluate the effectiveness of the instruction or of the learning? I must admit that I’ve been guilty of this from time to time.

When I transitioned from a teacher to a librarian, my bad habit got a lot worse. I would think to myself “I can figure it out later”, or I would slap down an evaluation method on a lesson plan to satisfy my boss without putting any thought into it.

It has only been through the course of my studies and this week’s readings that I have realized the importance of evaluation, even in the library. I will never be able to determine if students are grasping the concepts and learn the skills that I’m trying to teach without some form of evaluation.

Brinkerhoff’s Success Case Model

The Brinkerhoff Success Case Model examines instances where an intervention or training worked and seeks to find out what made them successful. The Success Case Model has five steps:

1.      The evaluator must plan the case study. Determine the program’s purpose, and settle on issues such as time frame and budget.

2.      Construct a visual impact model. This model includes program goals and desired results of the program.

3.      Conduct a survey research study to identify best and worst cases.

4.      Schedule and conduct in-depth interviews of success cases and even a few nonsuccess cases.

5.      Write up your findings from the evaluation and communicate the results to stakeholders.

The Success Case Model is one that I could easily see myself using as a librarian. To improve my library program, I could use this evaluation method to find success cases in which librarians are effectively teaching information literacy skills to students.

I could plan a case study in which I consult with my principle and ELAR teachers to determine the major goals and outcomes of the library program related to information literacy and reinforcing classroom TEKS. I can then identify campuses both within and outside of my district that have built successful library programs that have already accomplished my program’s desired objectives. I feel that it’s also important to visit campuses with similar demographics to my school.

I feel that I could also go beyond the forty-five minute phone interviews and even conduct campus visits to see the successful case in action. After my findings, I can present the information to my stakeholders (administration and teachers involved) and review our findings.

Patton’s Utilization Focused Evaluation

Reiser and Dempsey describe the key to the utilization focused evaluation that “the utility of an evaluation is to be judged by the degree to which it is used” (2012). The major steps are as follows:

1.      Conduct a readiness assessment

2.      Identify the “primary intended users” and develop a working relationship with them

3.      Conduct a situated analysis

4.      Identify the primary intended uses

5.      Focus the evaluation

6.      Design the evaluation

7.      Collect, analyze, and interpret the evaluation data

8.      Continually facilitate evaluation use

9.      Conduct a metaevaluation

The utilization focused evaluation model has a lot of features that can be used with any other evaluation model. In the library field, continuous evaluation is key to ensure student learning. I’m constantly evaluating the programs available, resources, instructional design of my lessons, and collaboration efforts. This model is one that I could use in all areas of the library.

What Questions Should Instructional Design Evaluation Address?

·        Is there an adequate return on investment?

o   First, I view return on investment as student growth and performance. Therefore, this question seeks to ask if the program is helping students learn in a measurable way. If it is, then it is worth the investment. If the program is not effective or only shows minimal growth, then perhaps funds and/or resources should be allocated to another program to assist in student learning.

·        Should student feedback be considered?

o   I feel that programs and evaluation methods are put in place, but so often we never ask the people we’re trying to serve for their opinions. In this case, the students. Any instructional design should have a component of student feedback. If the students are not enjoying the instruction, then the instruction is ineffective in my opinion.

·        How does this instructional design impact student engagement?

o   This question is one that I feel is important. Throughout our readings, it has become apparent that student engagement is crucial to long term student learning. So why should we ask about student engagement during the evaluation process?

Performance Problems and Non-Instructional Solutions

As I read about performance problems and non-instructional support, I liked the idea of providing performers with what they need when they need it. This made me think of my library setting, and how sometimes the students don’t necessarily need me to provide instruction to solve their problems.

For instance, I have students constantly asking me questions about book recommendations, how to perform a search in the school catalog, and how to use some of the resources we have in the library. I teach a lot of this information during the first six weeks; however, students forget. Therefore, I feel that a non-instructional solution would be best in this situation as I don’t always have time to reteach skills to several different students who all need help with different skills.

I could create a series of online instructional videos that students can access whenever they have questions or concerns regarding basic library skills or resource use. For example, if students forget how to use an app that we’re working with, they can go online and watch a brief overview video. Our library is stocked with enough iPads so that each student can use one when they come to the library.

Of course, this would not replace instruction but simply be a supplement in a situation where instruction may not be required.

Reiser, R.A., & Dempsey, J.V. (2012). Trends and issues in instructional design and technology (3rd ed.). Boston: Pearson.