This is the first of several guides
that will be published on the Internet this semester. Be sure to come back
because I will add links to new sites over the course of the next few months,
making it easy for you to cross reference topics. All guides will be linked
to the Overview and Readings WEB sites and placed in the Blackboard system.
EDF
5481 METHODS OF EDUCATIONAL RESEARCH
FALL
2001
WHAT METHODS
DO
In Methods of
Educational Research, we study several "quantitative" and "qualitative"
methods--or, in my preferred terminology, more
or less structured research designs for
collecting data. We will deal with empirical research, i.e., tangible data
which is assessed using the evidence of our senses. While these kinds of
data collection methods are not the only way in which we "know" things,
they have particular utility for testing research hunches and hypotheses
in a variety of fields.
Research Methods
are most often used for two major purposes:
(1) To
establish "facts" or recurring regularities
in the environment. Examples of facts include:
-
The incidence of
violence on high school campuses.
-
The percentage of
adult Koreans with access to the Internet at least once a week.
-
How many American
adults over 18 engage in physical exercise.
(2) To
test (and, more surreptitiously, establish) causal explanations for established
facts. Most theories address explanations
for factual material. Explanations typically
assert causal relationships among variables of interest.
For example:
-
Students who engage
in explosive violence on high school campus have been bullied at that school.
-
Science or technology
professionals have greater access to the Internet at work than other workers
do.
-
Men more often engage
in physical exercise than women.
Establishing facts
is hard enough! The measures we use may be contaminated by response bias
(e.g., many people tend to agree with any general statement. As a result,
you may not know whether your scale measures the desired construct--or
"agreement response set.") The population you studied may be relatively
small and harbor unique characteristics that are not typical of your true
population of interest. For example, it is risky to generalize from studies
of college undergraduates to corporate workers. You may have measured the
wrong dimensions or omitted key facets of your topic (example: you thought
you were measuring positive attitudes toward performance--but instead
you measured emotions about competition).
METHODS AND CAUSE: A PRELIMINARY
STATEMENT
As soon as we
try to establish causal precedence, things become even more difficult.
For every pair of factors that we see locked in a causal relationship:
-
We could mistake
the direction of causality. For example,
recent work on parents who physically discipline (spank!) their children
found that nearly 19 out of every 20 parents use some form of physical
discipline. The rare children whose parents never spanked them were found
to have exemplary behavior. The assumption was that parental discipline
patterns influenced children's behavior. BUT,
isn't it possible that "exemplary children" never even tempted their parents
to use physical discipline in the first place? That is, the true causal
variable here was the behavior of CHILDREN, rather than parents.
-
Any apparent causal
relationship occurs because a third factor caused both the orginal "cause"
and also the "effect." In other words, the relationship is "spurious,"
and not a "true" or "real" causal relationship. For
example, several decades ago, researchers found that American high school
students who smoked cigarettes had lower grades. Their conclusion was that
something about smoking caused lower grades. Leaving aside the reversed
causal possibility (your grades were so awful, you began smoking to relieve
the stress), later scholars found that the "true cause" was parental social
class. High school students who came from poorer backgrounds were both
more likely to smoke cigarettes and also had lower grades. Once
parental background was controlled, student cigarette smoking no longer
predicted grade point average. Spurious relationships appear in experimental
studies too; for example, your results may be due to anxiety aroused by
being in a testing situation or an artifact of a particular treatment manipulation.
-
Your results were
caused by alternative causal variables, leaving your original causal explanation
suspect. For example, I recently found
that the level of basic science knowledge in American adults was somewhat
higher among men than among women. People who have read this material conclude
that women are just less knowledgeable about science. HOWEVER,
I later discovered that much of the difference occurred not because
women gave more incorrect answers than men, but because women gave
more "I don't know" responses than men did. Issues such as self-efficacy
become more important in giving "I don't know" responses than incorrect
ones.
A considerable amount
of scholarship consists of formulating and testing alternative causal explanations
for "factual material," that is, teasing out how and why regularities occur.
Methodology is critical in the research enterprise. Some alternative explanationsare
methodological artifacts: for example, a limited population; an unrepresentative
sample; biased questionnaire items or tests; or incomplete experimental
treatments. Others are conceptual issues that can only be tested using
thorough methods of data collection.
STAGES OF METHODOLOGIES
In this course, we will study several
different types of research designs. However, all these designs also share
some common similarities and a well-planned sequence of activities. I will
mention some basic ones now, and we will study these issues in more depth
over the semester:
-
Being able to
develop a research problem
-
Deciding on the
unit of analysis (individual? group? organizations?) and taking
measures that are consistent with that unit of analysis
-
Deciding how to
sample one's chosen units
-
Deciding how to
measure one's concepts via choice of method (experiment? survey
questionnaire, including "tests"? archival search? etc.)
-
Once a method has been chosen, deciding
on actual measures and procedures, such as questionnaire items
or experimental treatments
-
Pilot testing
one's measures and double-checking the results
-
Moving "into the
field" to collect data
-
Making contact with subjects and respondents,
including Institutional Review Boards (Human Subjects Committees) and any
organizational representatives.
-
Training field staff
-
Supervising or conducting data collection
-
Reducing the collected
data to manageable size by selecting coding categories and coding
the data
|
DEVELOPING A RESEARCH
PROBLEM
|
Most of us, when
we begin to write up professional research, like to start writing our papers
like storytellers. We discuss an interesting recent research finding. We
describe a compelling social problem. Very often, the "meat" of our study
does not even emerge until the fifth typewritten page. Besides making it
very difficult for your reader, who must scrutinize your vivid prose for
several pages to learn what it is that you will even study and the topic
of your research, this written procrastination serves as a signal that
you are not really sure what your research is about!
When I work with
students on research projects, I am adamant that somewhere on the first
page of writing, a student must tell me:
What
the project is about. Anxiety and testing
results? Hormone fluxuations and sports participation? Motivation tools
and sports team performance?
Why
this project is important. Why it is a subject worthy of study.
Will it cure a social problem? Will it diagnose a learning disability?
Will it help individuals achieve a higher performance? Will it extend scholarship
in the discipline?
What
specifically will be done in this study.An
examination of how gender and educational type and level influence science
knowledge in survey data? An experiment with social identity threat and
pain tolerance? An observational study of group dynamics on football teams?
This combination
of elements constitutes your research problem statement: the general
area of your research, why
it is important, and what
specifically you will study.
Your research
problem statement will also address:
Your key conceptual variables
and definitions of these variables.
Postulated
causal relationships among these variables
(or, conceptual hypotheses).
Writing a research
problem statement will be THE MOST DIFFICULT ASSIGNMENT you will have all
semester, and you will rewrite it a few times over the next several weeks.
HOW TO GET
STARTED
If you are having
trouble conceptualizing a research problem, you are not alone. This is
typically the most difficult stage of conducting research. Further, in
less structured research, you may be constantly revising the research problem
as you gather data, and you may do so in any kind of research if you encounter
surprising and unanticipated results. Nevertheless, here are several "tried
and true" ways to begin.
CONCEPTUAL
AND DEDUCTIVE APPROACH. You are thoroughly
familiar with the literature in your area (say, self-regulated learning)
and you are aware of gaps where theory has not yet been tested, or where
theoretical predictions contradict one another, or you derive your research
problem from some basic theoretical assumptions. For example, perhaps you
compare the reading assessment scores of elementary school children taught
via "whole learning" versus "phonetics".
CURIOSITY.
Intrigued by regularly occuring "fact," you wish to know more about why
and how that fact occurs. You may be dissatisfied with previous explanations.
For example, why does educational level affect basic science knowledge?
Is it the type of college major? Stimulating an interest in science? "Weeding
out" the less intelligent? Holding a scientific or technical job?
You may encounter
a suprising, unanticipated "serendipitous" finding that begs for an explanation.
Your guesses about why this anomalous result occurred become the basis
of defining your research problem. Example, several decades ago, researchers
on achievement motivation discarded women subjects because their results
did not "fit" the researchers' paradigm. Encountering this unexplained
quirk in a footnote in my textbook, I have been examining issues
in gender ever since.
IT'S
THE MONEY, HONEY. Your major professor
or your client defines the research problem and you conduct the study.
In my experience, working for a client can be the most difficult way to
begin because the client often has a very fuzzy idea at best of what they
want to know or do. You often end up defining, or at the least, clarifying
and refining the research problem for the client.
STILL
STUCK? CHECK THIS SITE OUT!
KNOW YOUR TOPIC
There is no substitute
for knowing your topic well. Both your textbooks have excellent chapters
that describe literature searches. Online search engines and journal or
abstract services cut the time involved tremendously and alert you to new
source of information. Check out the links to various organizations (many
of them sponsor journals) in the RESOURCES section of our Blackboard course.
Collect as many
relevant study designs as you can. I have a file cabinet filled with survey
research questionnaires on all kinds of different topics.
Talk with your
clients, speak with members of your proposed subject pool. Find out which
aspects of your research problem are the most important to them.
One way to continue
working on your research project is to start a flow chart (HINT: take a
look at the Inspiration computer program,
available in our Learning Resource Center). Diagram your key variables
and the types of relationships among variables that you expect to find.
Such a chart will alert you to the concepts you need to measure.
Each global concept,
such as "reading assessment" or "instructional design plan" has a number
of variable components and alternative definitions. Be alert to this multiplicity
of definitions and make clear what your definition is, what your key variales
are, and what is or is not an instance of your definition.
A variable
is a characteristic or factor that has values that vary, for example, levels
of education, intelligence, or physical endurance.
A variable has at least two different
categories or values. If all cases have the same score or value, we call
that characteristic a constant, not a variable.
CONCEPTUAL
VARIABLES are what you think the entity really
is or what it means. YOU DO NOT DISCUSS MEASUREMENT AT THIS
STAGE! Examples include "achievement motivation" or "endurance" or "group
cohesion". You are describing a concept.
On the other hand, OPERATIONAL
VARIABLES (sometimes called "operational definitions") are how
you actually measure this entity, or the concrete operations,
measures or procedures that you use to measure the variable.
You usually begin your research problem
with CONCEPTUAL VARIABLES and the relationships among them. One of the
few exceptions is if your actual purpose is
to study a particular operational variable, for example, perhaps you want
to study the validity of the FCAT test, the achievement assessment test
that kindergarten through twelve grade students in Florida must take each
year.
We will spend considerable time in the
next week examing causal issues. For right now, you need to know about
independent and dependent variables.
Causes
are called
INDEPENDENT VARIABLES.
If one variable truly causes a second,
the
cause is the independent variable.
Speaking more statistically, variation in the independent variables comes
from sources outside our causal system or is "explained" by these sources.
Independent
variables are often also called explanatory
variables or predictors.
Effects
are called
DEPENDENT VARIABLES.
Statistically
speaking, we "explain" the variation in our dependent variable.
Dependent variables
are also sometimes called outcome
or criterion
variables.
A research problem
will describe the causal relationships between independent and dependent
variables and explain how these relationships come to be.
|
A
DOZEN METHODOLOGICAL CLICHÉS TO GET US STARTED
|
-
1.
Good research takes time. "Overnight polls" have terrible
response rates and dubious generalizability. Experiments must be pilot
tested, for example, to see if subjects even noticed your treatment manipulations
(manipulation checks). Ethnographies
can take months, or even years. No method can be done in a hurry. Even
if you are only in the field a short time, allow enough time for
planning and pilot testing in your research.
-
2.
No one study disproves (or worse yet, "proves") anything.While
we like to think of the "definitive experiment," each study has strengths
and weaknesses. Perhaps one cannot generalize well to a known population
of individuals (groups) or situations (NOTE: this is EXTERNAL
VALIDITY). Perhaps there are alternative explanations about
what caused the outcomes (NOTE: this addresses INTERNAL
VALIDITY). An aggregate of studies are usually needed to make
strong assertions about the phenomena under study.
-
3.
Always,
ALWAYS pilot test before you go into
the field. This way you will catch
problems with the experimental manipulations, difficulties with field observational
categories, strange ways that respondents interpret your survey research
questions and much more. If you are using any type of questionnaire, be
sure that you pilot test at least once by reading questions aloud (even
if the questionnaire will be self administered).
-
4.
Try to measure your variables as many ways as practicably possible. You
want to rule out alternative explanations for your results. Do you want
your survey results to represent acquiescence response set instead of substance?
Of course not! Do you want your experimental findings to reflect experimenter
demand effects instead of treatment effects? Of course not!
The process of measuring the same concept
in different ways is sometimes called "TRIANGULATION."
It is one way to try to ascertain CONSTRUCT VALIDITY
(i.e., whether your operationalized variables really measure the construct
you envisioned--or something else entirely).
-
5.
Trust subjects and respondents. Listen to what they
are trying to tell you. Your respondents may be trying to tell you
(subtly, nicely) that they can't understand your questions (your colleagues
had no trouble with the professional jargon). Your subjects may see the
experimental task as ridiculous although they will try to "help out" by
completing it anyway. Debrief. You are bound by ethics to do so anyway.
Ask your subjects what they thought was the purpose of your experiment.
Ask a random subsample of respondents to answer the question in their own
words or why they answered the way they did (Schuman's "RANDOM
PROBE" technique).
-
6.
Watch your defined population. Who does it represent?
Undergraduate educational psychology students only? All undergraduate college
students? High school Spanish students at an upper income facility? Football
coaches at AA universities? Graduate students enrolled in distance learning
courses? You almost certainly will want to make generalizations later on.
-
7.
Try to avoid dichotomies in your measurements whenever possible. Likewise,
don't collapse an interval level variable (e.g., years of education) into
an ordinal one (unequal educational categories) if at all possible. The
computer can aggregate several categories into one category in a matter
of seconds (e.g., 9 through 11 years of formal education can be recoded
as "some high school"). However, you cannot go the other way: "some high
school" cannot be turned into a definitive number.
Try learning to think in conceptual
continuums, degrees of "more" or "less" rather than "either" "or." Although
your manipulated treatments in an experiment may be categorical, even the
manipulations can be "levels" or degrees of a treatment.
-
8.
Consider how you will analyze your data once you have collected it.
I know that some of you have not yet taken a course in statistics. Therefore--consult
a statistician or a friend/student who has elected several statistics courses.
If you do an experiment and want to use analysis of variance, you will
need interval level (or "sort of numeric" anyway) measures for your dependent
variables. Regression also typically requires interval dependent variables
(see your statistics instructor for variations on this theme). If all your
variables are nominal you will be very limited in the analytic methods
you can use.
-
9.
It is very difficult, and sometimes impossible, for sophisticated means
of data analysis to compensate for poor data collection.
If your response rate is poor, your results probably cannot be generalized
to any known population. Some behavioral scientists engage in elaborate
weighting schemes so their data appear to be "representative" but the problem
is that we seldom know how those who responded differed from those who
refused or could not be located. If your measures are contaminated by response
bias, you will not be able to disentangle effects without gathering more
data. If you find out later that you neglected to measure important variables,
you may no longer have access to your population to collect more information.
-
10.
There is no such thing as "value free" research. Researchers
are human beings who are the captives of their culture. This includes considering
only research that produces "statistically significant" differences as
"important" (consider for a moment what this perspective did to the field
of "sex differences" if ONLY research finding
differences got published).
So what is there to do? Try to understand
your own values and how they might introduce biases into your research.
Safeguard against your own biases. Don't do your own interviewing, open
code your own data, or serve as your own experimenter. Trade off with another
graduate student or seek funding to hire people for these positions.
Talk with men (if you are female) or
women (if you are male). Show your research design to friends from other
cultural backgrounds to see if your ideas--or your treatments--or your
questionnaire items--might be misconstrued.
-
11.
Your research will take at least twice as long and be at least three times
as much trouble as you ever thought it would before you got started. Subjects
don't show up and must be rescheduled. The survey lab goes broke while
you are in the field (this one really happened to me). You must locate
someone who speaks the language fluently. Allow time for the Human Subjects
Committee to examine your design. You had to go out of town and the client
pilot-tested on the wrong population (I had this one happen too.) MORAL:
try to trouble shoot as much as you can at the very beginning!
-
12. DO THE BEST YOU
CAN WITH WHAT YOU GOT. No study (including mine or yours)
will be perfect. You almost certainly will have a less than ideal level
of funding (bake sale level, maybe?). You will have less than ideal assistance
and your time will be constrained. If we all waited for perfection, we
would never study anything. So, relax and enjoy!
Susan Carol Losh August
28 2001
This page was built with
Netscape Composer
and is best viewed with
Netscape Navigator
600 X 800 display resolution.
ASSIGNMENT
TWO
