L. R. Gay

Ary

Borg and Gall

Coburn

Fine: Qualitative Studies

Fine: Quantitative Studies

L. R. Gay

Hittleman and Simon

Romig

Criteria for Synthesis Research

Ward

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

L. R. Gay Research Evaluation Criteria

Gay, L. R. (1996). Educational Research: Competencies for Analysis and Application.

 

INTRODUCTION

Problem

Is there a statement of the problem?

Is the problem "researchable;" can it be investigated through the collection and analysis of data?

Is background information on the problem and its educational significance presented/discussed?

Does the problem statement indicate the variables of interest and their specific relationships?

When necessary, are variables directly or operationally defined?

 

Review of Related Literature

Is the review comprehensive? Are all references cited relevant to the problem under investigation?

Are most of the sources primary?

Have the references been critically analyzed and the results of various studies compared /contrasted?

Is the review well organized, i.e., does it logically flow in such a way that the references least related to the problem are discussed first and the most related references discussed last?

Does the review conclude with a brief summary of the literature and its implications for the problem?

Do the implications discussed form an empirical or theoretical rationale for the hypotheses?

 

Hypotheses

Are specific questions to be answered listed or specific hypotheses to be tested stated?

Does each hypothesis state an expected relationship or difference? Is each hypothesis testable?

 

METHOD

 

Subjects

Are the size and major characteristics of the population and of the sample studied described?

If a sample was selected, is the method of sampling clearly described?

Is the method of sample selection one that is likely to result in a representative and unbiased sample?

Did the researcher avoid the use of volunteers?

Does the sample size meet the suggested guideline for minimum sample size appropriate for the method of research represented? (i.e. survey=10%; 25-30 subjects for experimental)

 

Instruments

Is the rationale given for instrument selection and each described in terms of purpose /content?

Are the instruments appropriate for measuring the intended variables?

Is evidence presented that indicates that each instrument is appropriate for the sample under study?

Is instrument validity discussed and coefficients given if appropriate?

Is reliability discussed in terms of type and size of reliability coefficients and subtest reliabilities?

If an instrument was developed specifically for the study, are the procedures involved in its development and validation described?

If an instrument was developed specifically for the study, are administration, scoring, and interpretation procedures described?

 

Design and Procedure

Is the design appropriate for answering the questions?

Are the procedures described in sufficient detail to permit them to be replicated?

If a pilot study was conducted, are its execution and results described?

Are control procedures described and were potentially confounding variables accounted for?

 

RESULTS

Are appropriate descriptive statistics presented?

Was the probability level at which the results of the tests of significance were evaluated specified in advance of the data analysis?

Is there evidence that the researcher avoided violating the required assumptions for parametric use?

Are the tests of significance described appropriate given the hypotheses and design?

Was every hypothesis tested and significance interpreted using the appropriate degrees of freedom?

Are the results clearly presented? Are the tables and figures well organized and easy to understand?

Are the data in each table and figure described in the text?

 

DISCUSSION/CONCLUSIONS/RECOMMENDATIONS

Is each result discussed in terms of the original hypothesis to which it relates?

Is each result described in terms of its agreement or disagreement with previous results of others?

Are generalizations consistent with the results?

Are the possible effects of uncontrolled variables on the results discussed?

Are theoretical and practical implications of the findings discussed?

Are recommendations for future action and for future research made?

Are the suggestions for future action based on practical significance (i.e. has the author avoided confusing statistical and practical significance)?

 

ABSTRACT OR SUMMARY

Is the problem restated?

Are the number and type of subjects and instruments described?

Is the design used identified? Are procedures described?

Are the major results and conclusions restated?

 

TYPE SPECIFIC EVALUATION CRITERIA

 

Qualitative

Is each data collection strategy described?

Was more than one data collection strategy used?

Were the data collections strategies appropriate given the purpose of the study?

Were strategies used to strengthen the validity and reliability of the data (e.g. triangulation)?

If participant observation used, are strategies for minimizing observer bias /observer effects described?

Are the researcher’s reactions reported?

If the researcher’s reactions are given, are they differentiated from descriptive field notes?

Are data coding strategies described and examples of coding given?

Are conclusions supported by data, (e.g. are direct quotes used to illustrate the points made)?

 

Historical

Were the sources of data related to the problem mostly primary?

Was each piece of data subjected to external criticism?

Was each piece of data subjected to internal criticism?

 

Descriptive Research: Questionnaire Studies

Are questionnaire validation procedures described?

Was the questionnaire pretested?

Are pilot study procedures and results described?

Are directions to questionnaire respondents clear?

Does each item in the questionnaire relate to one of the objectives of the study?

Does each questionnaire item deal with a single concept?

When necessary, is a point of reference given for questionnaire items?

Are leading questions avoided in the questionnaire?

Are there sufficient alternatives for each questionnaire item?

Does the cover letter explain the purpose and importance of the study and give the potential responder a good reason to cooperate?

If appropriate, is confidentiality of responses assured in the cover letter?

Was the percentage of returns approximately 70% or greater?

Are follow-up activities described?

If the response rate was low, was any attempt made to determine any major differences between responders and nonresponders?

 

Descriptive Research: Interview Studies

Were the interview procedures pretested?

Are pilot study procedures and results described?

Does each item in the interview guide relate to a specific objective of the study?

When necessary, is a point of reference given in the guide for interview items?

Are leading questions avoided in the interview guide?

Does the interview guide indicate the type and amount of prompting and probing that was permitted?

Are the qualifications and special training of the interviewers described?

Is the method that was used to record responses described?

Did the researcher use the most reliable, unbiased method of recording responses possible?

Did the researcher specify how the responses to semi-structured and unstructured items were quantified and analyzed?

 

Descriptive Studies: Observation Studies

Are observational variables defined?

Were observers required to observe only one behavior at a time?

Was a coded recording instrument used?

Are the qualifications and special training of the observers described?

Is the level of observer reliability reported?

Was the level of observer reliability sufficiently high?

Were possible observer bias and observer effect discussed?

Was observation of the subjects the best approach for data collection as opposed to unobtrusive measures?

 

Correlation Research: Prediction Studies

Is a rationale given for selection of predictor variables?

Is the criterion variable well defined?

Was the resulting prediction equation validated with at least one other group?

 

Correlation Research: Relationship Studies

Were variables carefully selected, (i.e. was the shotgun approach avoided)?

Is the rationale for variable selection described?

Are conclusions and recommendations based on values of correlation coefficients corrected for attenuation or restriction in range?

Do the conclusions avoid suggesting causal relationships between the variables?

 

Causal-Comparative Research

Are the characteristics or experiences that differentiate the groups (independent variable) clearly defined or described?

Are critical extraneous variables identified?

Were any control procedures applied to equate the groups on extraneous variables?

Are causal relationships found discussed with due caution?

Are plausible alternative hypotheses discussed?

 

Experimental Research

Was an appropriate experimental design selected?

Is a rationale for design selection given?

Are sources of invalidity associated with the design identified and discussed?

Is the method of group formation described?

Was the experimental group formed in the same way as the control group?

Were groups randomly formed and the use of existing groups avoided?

Were treatments randomly assigned to groups?

Were critical extraneous variables identified?

Were any control procedures applied to equate groups on extraneous variables?

Were possible reactive arrangements (e.g. Hawthorn effect) controlled for?

 

earth1.gif (1170 bytes)

3class.gif (3925 bytes)

openbk.gif (597 bytes)

3sched.gif (3294 bytes)

giftbx.gif (1750 bytes)

comm.gif (289 bytes)

3learn.gif (3515 bytes)

Home

Introduction

Syllabus

Schedule

Course
Requirements

Web
Board

Links

| Professional Site | Personal Site | Pretest | Evaluation |