In this paper Williams links questions about evaluation with questions about the use and design of learning objects. The first three pages raise a number of stimulating questions that are then discussed within subsequent pages of the paper. The questions are relevant to issues, problems, practices, and theories that may hinder successful design, implementation, and evaluation of learning objects. Williams has the reader use an actual learning object, a pan balance, as an object for learning about evaluating learning objects. He observes that "once the questions regarding audience and their values and criteria are addressed, evaluation methodology is relatively straight forward." Assuming that this can be done, he focuses on steps of an evaluation sensitive to particular audience needs and values.

To set the stage for this discussion, Williams explores current thinking about evaluation; its relations to learning objects; and the nature of participant-oriented evaluation. Some of his basic points include the following: All evaluation boils down to the same end: "…comparing what something is to what it ought to be, in order to facilitate a judgment about the value of that thing." The challenge is to define the values of dimensions by which to describe or to decide "what" the object "ought to be." After values to be used in the evaluation process have been clarified, standards or principles need to be set for the evaluation process.

One approach to a contextualized evaluation is the participant-oriented approach; basic to all models within this category is that they address diverse values in a fair and systematic way. Williams proceeds from this latter point to show how blending Stufflebeam's CIPP (context, input, process, product) model with Patton's user-focused approach creates a powerful meta-model by which to design evaluations that meet the greatest needs of the most people at each stage of William's proposed evaluation model. The implications for application of the combined model are fleshed out with a hypothetical example employing the aforementioned pan-balance-as-learning-object. Shortly stated, Williams argues that the history of solid work in the field of evaluation suggests that the following three steps always be components of the design process for evaluation learning objects:

1) identify who will use and evaluate the learning objects;
2) assess how users define relevant learning objects and the criteria by which users judge them;
3) collect and use data about how the learning objects measure up to those criteria to make evaluation judgments in accordance with established meta-evaluation standards.

Shifting from questions about evaluation methodologies, Williams connects the developed themes to questions about learning objects. The concerns of instructors, students, and instructional support persons are discussed. While it is recognized that most research and development resides in the domain of instructional support services (including the work of instructional designers, librarians, technical specialists, etc.), Williams emphasizes that learners are "key users of evaluations of learning objects" and it is with this in mind that his paper continues.

He notes that at the heart of the evaluation task is figuring out 1) the users' interest; 2) the user's criteria for judging "evaluands;" and 3) the questions the evaluation might be organized around. Williams observes that one of the central reasons for attending to how diverse audiences define learning objects comes from a principle that context determines the value of a given learning object. Context makes learning objects meaningful to users; thus, evaluation must attend to this fact about learning objects and their judges.

Williams recognizes that value and interests conflicts may come into play, and for that reason, he notes that an evaluation process should allow users to make their decisions in concert with those of fellow users. The remainder of the paper provides guidelines for achieving such an outcome. An elaborate discussion of two alternative approaches to evaluating of learning objects based on a participant-model is provided: one is external and comprehensive and the other internal, immediate, and continuous.

For the full text, see http://reusability.org/read/chapters/williams.doc

Back to main Effectiveness page ....