L. R. Gay Research Evaluation Criteria
Gay, L. R. (1996). Educational Research:
Competencies for Analysis and Application.
INTRODUCTION
Problem
Is there a statement of the problem?
Is the problem "researchable;" can it be
investigated through the collection and analysis of data?
Is background information on the problem and its
educational significance presented/discussed?
Does the problem statement indicate the variables of
interest and their specific relationships?
When necessary, are variables directly or operationally
defined?
Review of Related Literature
Is the review comprehensive? Are all references cited
relevant to the problem under investigation?
Are most of the sources primary?
Have the references been critically analyzed and the
results of various studies compared /contrasted?
Is the review well organized, i.e., does it logically flow
in such a way that the references least related to the problem are discussed first and the
most related references discussed last?
Does the review conclude with a brief summary of the
literature and its implications for the problem?
Do the implications discussed form an empirical or
theoretical rationale for the hypotheses?
Hypotheses
Are specific questions to be answered listed or specific
hypotheses to be tested stated?
Does each hypothesis state an expected relationship or
difference? Is each hypothesis testable?
METHOD
Subjects
Are the size and major characteristics of the population
and of the sample studied described?
If a sample was selected, is the method of sampling
clearly described?
Is the method of sample selection one that is likely to
result in a representative and unbiased sample?
Did the researcher avoid the use of volunteers?
Does the sample size meet the suggested guideline for
minimum sample size appropriate for the method of research represented? (i.e. survey=10%;
2530 subjects for experimental)
Instruments
Is the rationale given for instrument selection and each
described in terms of purpose /content?
Are the instruments appropriate for measuring the intended
variables?
Is evidence presented that indicates that each instrument
is appropriate for the sample under study?
Is instrument validity discussed and coefficients given if
appropriate?
Is reliability discussed in terms of type and size of
reliability coefficients and subtest reliabilities?
If an instrument was developed specifically for the study,
are the procedures involved in its development and validation described?
If an instrument was developed specifically for the study,
are administration, scoring, and interpretation procedures described?
Design and Procedure
Is the design appropriate for answering the questions?
Are the procedures described in sufficient detail to
permit them to be replicated?
If a pilot study was conducted, are its execution and
results described?
Are control procedures described and were potentially
confounding variables accounted for?
RESULTS
Are appropriate descriptive statistics presented?
Was the probability level at which the results of the
tests of significance were evaluated specified in advance of the data analysis?
Is there evidence that the researcher avoided violating
the required assumptions for parametric use?
Are the tests of significance described appropriate given
the hypotheses and design?
Was every hypothesis tested and significance interpreted
using the appropriate degrees of freedom?
Are the results clearly presented? Are the tables and
figures well organized and easy to understand?
Are the data in each table and figure described in the
text?
DISCUSSION/CONCLUSIONS/RECOMMENDATIONS
Is each result discussed in terms of the original
hypothesis to which it relates?
Is each result described in terms of its agreement or
disagreement with previous results of others?
Are generalizations consistent with the results?
Are the possible effects of uncontrolled variables on the
results discussed?
Are theoretical and practical implications of the findings
discussed?
Are recommendations for future action and for future
research made?
Are the suggestions for future action based on practical
significance (i.e. has the author avoided confusing statistical and practical
significance)?
ABSTRACT OR SUMMARY
Is the problem restated?
Are the number and type of subjects and instruments
described?
Is the design used identified? Are procedures described?
Are the major results and conclusions restated?
TYPE SPECIFIC EVALUATION CRITERIA
Qualitative
Is each data collection strategy described?
Was more than one data collection strategy used?
Were the data collections strategies appropriate given the
purpose of the study?
Were strategies used to strengthen the validity and
reliability of the data (e.g. triangulation)?
If participant observation used, are strategies for
minimizing observer bias /observer effects described?
Are the researcher’s reactions reported?
If the researcher’s reactions are given, are they
differentiated from descriptive field notes?
Are data coding strategies described and examples of
coding given?
Are conclusions supported by data, (e.g. are direct quotes
used to illustrate the points made)?
Historical
Were the sources of data related to the problem mostly
primary?
Was each piece of data subjected to external criticism?
Was each piece of data subjected to internal criticism?
Descriptive Research: Questionnaire Studies
Are questionnaire validation procedures described?
Was the questionnaire pretested?
Are pilot study procedures and results described?
Are directions to questionnaire respondents clear?
Does each item in the questionnaire relate to one of the
objectives of the study?
Does each questionnaire item deal with a single concept?
When necessary, is a point of reference given for
questionnaire items?
Are leading questions avoided in the questionnaire?
Are there sufficient alternatives for each questionnaire
item?
Does the cover letter explain the purpose and importance
of the study and give the potential responder a good reason to cooperate?
If appropriate, is confidentiality of responses assured in
the cover letter?
Was the percentage of returns approximately 70% or
greater?
Are followup activities described?
If the response rate was low, was any attempt made to
determine any major differences between responders and nonresponders?
Descriptive Research: Interview Studies
Were the interview procedures pretested?
Are pilot study procedures and results described?
Does each item in the interview guide relate to a specific
objective of the study?
When necessary, is a point of reference given in the guide
for interview items?
Are leading questions avoided in the interview guide?
Does the interview guide indicate the type and amount of
prompting and probing that was permitted?
Are the qualifications and special training of the
interviewers described?
Is the method that was used to record responses described?
Did the researcher use the most reliable, unbiased method
of recording responses possible?
Did the researcher specify how the responses to
semistructured and unstructured items were quantified and analyzed?
Descriptive Studies: Observation Studies
Are observational variables defined?
Were observers required to observe only one behavior at a
time?
Was a coded recording instrument used?
Are the qualifications and special training of the
observers described?
Is the level of observer reliability reported?
Was the level of observer reliability sufficiently high?
Were possible observer bias and observer effect discussed?
Was observation of the subjects the best approach for data
collection as opposed to unobtrusive measures?
Correlation Research: Prediction Studies
Is a rationale given for selection of predictor variables?
Is the criterion variable well defined?
Was the resulting prediction equation validated with at
least one other group?
Correlation Research: Relationship Studies
Were variables carefully selected, (i.e. was the shotgun
approach avoided)?
Is the rationale for variable selection described?
Are conclusions and recommendations based on values of
correlation coefficients corrected for attenuation or restriction in range?
Do the conclusions avoid suggesting causal relationships
between the variables?
CausalComparative Research
Are the characteristics or experiences that differentiate
the groups (independent variable) clearly defined or described?
Are critical extraneous variables identified?
Were any control procedures applied to equate the groups
on extraneous variables?
Are causal relationships found discussed with due caution?
Are plausible alternative hypotheses discussed?
Experimental Research
Was an appropriate experimental design selected?
Is a rationale for design selection given?
Are sources of invalidity associated with the design
identified and discussed?
Is the method of group formation described?
Was the experimental group formed in the same way as the
control group?
Were groups randomly formed and the use of existing groups
avoided?
Were treatments randomly assigned to groups?
Were critical extraneous variables identified?
Were any control procedures applied to equate groups on
extraneous variables?
Were possible reactive arrangements (e.g. Hawthorn effect)
controlled for?
