• Key names to remember: Tyler, Provus, Metfessel & Michael, Hammond, Bloom & Krathwohl

• Tyler’s Eight-Year study
• NAEP
• School accountability
• Competency testing
• Criterion-referenced tests
• Federal project monitoring

• Establish goals/objectives
• Classify goals/objectives
• Define in behavioral terms
• Find situations
• Select measurement techniques
• Collect data
• Compare data with behavioral objectives

• Acquire information
• Develop work habits
• Develop effective thinking
• Internalize social attitudes
• Keep physical health
• Develop life philosophy

• Danger of reliance on standardized test results
• Importance of screening or filtering goals by philosophical, social, and pedagogical sources
• Use logical and empirical methods to evaluate goals

• Examine the cogency of the rationale behind the goal/objective
• Examine the consequences on accomplishing the goal/objective
• Consider higher-order values to determine if goal/objective conflicts

• Collect group data on value positions
• Arrange expert/panel reviews
• Conduct content studies to determine value positions
• Conduct pilot study to see if goal is attainable (demonstration projects)

• Involve stakeholders
• Formulate cohesive model of goals/obj.
• Translate into communicable form
• Select instruments for measuring
• Carry out observations
• Analyze data
• Interpret data
• Develop recommendations

• Expanded concept of alternative instruments for data collection

• Agree on standards (objectives)
• Determine if discrepancy exits between performance and standards
• Use discrepancy information to improve, maintain, or terminate program

• Definition
• Installation
• Process
• Product
• Cost-benefit

• When discrepancy exists determine
• Why
• What corrective actions are possible
• Which corrective action is best
• Added problem-solving approach to objectives-oriented evaluation

• Three dimensions of the cube
• Needs of the client
• Characteristics of the client
• The principle actors
• Expanded thinking about the range of educational objectives

• Helped develop taxonomies of objectives in the cognitive and affective domains

• Developed objectives-oriented approach for curriculum improvement

• Goals themselves must be evaluated
• Goals are often rhetoric and do not reflect the real objectives
• Important outcomes may not fall in categories of goals/objectives
• Goal free evaluation will help reduce bias and increase objectivity

• Evaluator is not aware of program goals
• Goals do not narrow focus of evaluation
• Focus on actual rather than intended outcomes
• Minimal contact between evaluator and manager/staff
• Increased likelihood of noting unanticipated side-effects

• Simple, easy to understand
• Produces relevant information
• Well-developed processes
• Extensive literature
• Clarification of intended outcomes
• Face validity
• Reveals value basis for judging
• Advances in outcome measurement

• Lacks evaluative component
• Lacks standards to judge discrepancies
• Neglects value of objectives
• Ignores possible alternatives
• Neglects transactions
• Neglects context
• Ignores unintended outcomes
• Omits evidence of value
• Promotes linear, inflexible approach

• May appear seductively simple
• Factory model
• Does not question worth of goals
• Staff activities may not match stated goals
• Non-articulated or non-interpretable goals
• Overemphasis on testing

• Primary focus is to serve the needs of decision-makers (administrators, policy makers, boards, practitioners, etc.)
• The concerns, informational needs and criteria for effectiveness of decision-makers guide the evaluation
• Names to remember: Stufflebeam, Alkin, Patton, Wholey

• Context: Planning decisions (what needs are to be addressed; defining objectives for the program)

Input: Structuring decisions (what resources are available; what alternative strategies should be considered; what plan has the best potential

• Process: Implementing decisions (how well is the plan being implemented; what are the barriers; what revisions are needed
• Product: Recycling decisions (what results were obtained; were needs reduced; what should be done with the program

• Objective: To define the context, identify populations, assess needs, diagnose problems
• Method: system analysis, survey, document review, hearings, interviews, tests, Delphi
• Relation to decision-making: decide on setting, goals/objectives, planning, and providing basis for judging outcomes

• Most basic kind of evaluation
• Provides rationale for determining objectives
• Identifies and defines the limits of the domain to be served and the sub-parts
• Uses empirical and conceptual analysis, theory, authoritative opinion to judge basic problems which much be solved

• Objective: identify and assess system capabilities, alternative strategies, implementation designs, budgets
• Method: resource analysis, feasibility analysis, literature searches, exemplary program visits, pilot projects
• Decision: selecting sources, structuring activities, and basis for judging implementation

Looks at resources, time, budgets, potential barriers, and consequences of not overcoming barriers, relevance of design to objectives

Methods for input evaluation often lacking in education. May include committee deliberations, review of professional literature, consultants, experimental projects

Formality of input evaluation depends on whether change is large or small and whether high or low information grasp is available to support change

• Objective: identify/predict defects in design or implementation and record and judge procedural activities
• Method: monitoring, describing process, interacting, observing

Decision: for implementing and refining program design and procedures, process control, information to use in interpreting outcomes

• Provides periodic feedback to those responsible for implementation
• Maintains a record of procedures as they occur

Four essential features: full-time evaluator, instruments for describing process, regular feedback, frequent updating of process evaluation design

• Objective: describe and judge outcomes, relate them to objectives (and CIP), and interpret worth
• Method: operationally measuring criteria, collecting stakeholder judgments, quantitative/qualitative
• Decision: to continue, terminate, modify, refocus, and present record of effects (intended/unintended; positive/negative)

• Purpose to measure and interpret attainments at end of project cycle (and as often as needed during)
• Operationally measures objectives and compares to predetermined standards
• Interprets outcomes using context, input, and process information

• Focus the evaluation
• Collect information
• Organize information
• Analyze information
• Report information
• Administration of the evaluation

• Systems assessment: information about the state of the program (context)
• Program planning: assist in selecting particular programs likely to be effective (input)
• Program implementation: determine whether program was introduced to appropriate group in manner intended

Program improvement: how a program is functioning, whether objectives are being achieved, whether unanticipated outcomes are appearing (process)

• Provus’s Discrepancy model - some aspects are management-oriented
• Patton - utilization-focused evaluation
• Wholey - focus on practical uses of evaluation

• Choice of objectives and assigned priorities
• Choice of strategy and design specifications
• Guidance for implementation
• Termination, continuation

• Orderly systems approach that gives focus to evaluation
• Stresses importance of utility of information
• Shows value of early evaluation
• Simple heuristic tool that helps generate important questions to be answered
• Easy to explain

• Supports evaluation of every component
• Provides evaluation focus as programs grows and changes
• Stresses timely use of feedback

• Inability to respond to issues that clash with concerns of decision-makers
• Indecisive leaders unlikely to benefit
• Gives preference to top management (hired gun mentality)
• Possibility of unfair or undemocratic evaluation

• Can result in costly and complex evaluations
• Assumes that important decisions can be clearly identified in advance and that clear alternatives can be specified
• Assumes orderliness and predictability of decision-making process

• Increasingly stressed
• Also performance measurement
• Shifts focus from activities to results
• Has expanded to include:
• financial accountability
• program products or outputs
• adherence to standards of quality
• participant-related measures (demographics)
• key performance indicators
• client satisfaction

• Clearer definition of intended outcomes
• Barometer to assess progress
• Information for improvement
• Motivator to staff
• Helps position organization as successful

• Merely attending/participating is not an outcome
• Outcome is not an output
• outputs are about the program
• outcomes are about the participants
• Outcomes benefit/change participants’
• knowledge status
• attitude conditions
• skills behaviors
• values

• Inputs
• Activities
• Outputs
• Initial outcomes
• Intermediate outcomes
• Long-term outcomes

• Multiple benefits- focus, feedback, needs, comparisons, tracking, image....
• Clarification of intended outcomes
• Helps improve services, increases accountability, better decisions

• Is doable, even for small programs
• Commitment at top is essential
• Must identify own outcomes, indicators and relevant data
• Create written logic model of inputs, activities, outputs, outcomes
• Tap many perspectives to identify outcomes

• Does not always require new data collection
• May need outside assistance with technical expertise
• Use trial run for measurement system
• It takes time (6 months to data collection, often 3-5 years to see outcomes)
• Monitor and improve measurement system

• Initial preparation
• get ready
• choose outcomes
• specify indicators
• prepare to collect data
• Trial run
• collect data
• try analysis and reporting
• Implementation
• adjust measurement system
• use findings

• Funders play key role in focus on outcomes. Guide, do not impose.
• Help develop capacity.
• Provide training, technical assistance.
• Agree on terminology, methodology, timetables.
• Provide perspective on plans.
• "Walk the talk" - eliminate requirements that do not match outcome focus.

• Recognize potential for harm - can shift focus, penalize programs with hard to measure outcomes, promote "creaming"
• Focus on good faith effort, not findings
• Align outcomes with funding priorities
• Recognize need for baseline data
• Be cautious of arbitrary criterion levels
• Do not compare programs to judge "best", compare with self
• Funding decisions consider more than just outcomes

Many things outcomes measurement does not do - monitor resources/outputs, measures the "right" outcomes, explain particular levels of outcomes, "prove", answer judgment questions

Measuring and improving program-level outcomes does not by itself improve broader level outcomes, there are many other influences.

• Harder-to-measure outcomes
• Increasing input from research studies
• Sharing useful outcomes, indicators,tools, and methods
• Developing computer capacity
• Judging appropriateness of outcomes, indicators, and measures
• Establishing reasonable ranges for performance/benchmarks
• Align outcomes with change efforts