PLEASE NOTE: IF ANY QUESTIONS ARE POSTED TO ME THAT WOULD CHANGE OR CLARIFY THE CONTENT OF THIS SITE, I WILL CORRECT IT HERE.

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

GENERAL GUIDE:   EXAM ONE ALSO SEE:  GUIDE 1: INTRODUCTION GUIDE 2: VARIABLES AND HYPOTHESES GUIDE 3: RELIABILITY, VALIDITY, CAUSALITY, AND EXPERIMENTS GUIDE 4: EXPERIMENTS & QUASI-EXPERIMENTS GUIDE 5: A SURVEY RESEARCH PRIMER

COVERAGE

DIFFERENCES FROM THE BOOKS

BASIC CONCEPTS

APPLICATIONS

SAMPLE QUESTIONS

This exam covers chapters 1-6 in Wiersma and chapters 1-3, 6,  8 (pp. 180-186 ONLY), and 9 in McMillan, and all lectures, videos, demonstrations, and course Web sites through Guide 4 and associated links, including any material I have placed in Blackboard/CourseInfo.

You are also expected to be familiar with the Obedience documentary film shown September 26.

Exam One is 100 points and should take about one hour to complete. It counts 25 percent toward your final grade.

In some cases you will be asked to choose the sections of a question that you answer, e.g., select three out of four sections. The purpose of this is to allow you to show off the areas that you know the best. DO NOT answer all choices in such instances. No extra credit! I only grade the first number of designated selections if you answer all the selections in these cases. So what can happen is that (for example, in a 3 out of 4 selection question) you get parts 1, 2 and 4 right, but I only grade parts 1, 2, and 3, so your credit is lower than if you had simply answered 1, 2 and 4.

The exam is a mix of multiple choice, fill in the blank, and short essay questions. You may add a SHORT explanation to any short-answer question.
 
 

WHAT WON'T BE ON THE EXAM

Exam coverage from the texts is selective. For example, although both Wiersma and McMillan have excellent chapters on doing literature reviews, I will not have questions about them on this exam. These are valuable chapters to use in your own research, regardless of any exam questions.

I will not have articles for you to analyze as McMillan does in his excellent examples. I DO expect you to know that you must state your research problem early in the Introduction, I will not test you on this part of the chapter.

I do not expect you to memorize which disciplines are "qualitative" and which are "quantitative," although I DO expect you to know the general differences between these two types of research.
 
 

WHAT WILL BE ON THE EXAM

I expect you to know the differences among reliability, construct validity, internal validity, and external validity. In fact, there will be a series of multiple choice questions with examples of each.

I want you to know about threats to internal and external validity, and the enhancements your designs can make to both these types of validity. You will have short essay and multiple choice questions in these areas.

I expect you to be able to place variables in nonexperimental studies in causal order (independent, intervening, dependent). There will be a series of these questions. You will need to be able to describe the rule that you used to causally order the variables  too.

I expect you to know about properties of variable category systems and levels of measurement in data. You will have some examples of these.

I expect you to recognize different types of experimental and quasi-experimental designs and how these relate to threats to internal validty and external validity.

You need to be able to identify different kinds of hypotheses.

Similar to the experimental critique in Assignment 2 , you need to be able to trouble shoot and problem solve different kinds of experimental and non experimental designs.

In general, your books and I agree on terms and usage. Here are a few exceptions.
 
 

1. INTERVENING VARIABLE

Recall that here is an important case where we differ:

I define an intervening variable as one that links the independent variable to the dependent variable. Thus, an intervening variable is part of a causal chain:

INDEPENDENT VARIABLE -------> INTERVENING VARIABLE ------> DEPENDENT VARIABLE

PLEASE USE THIS DEFINITION FOR THIS COURSE.

This definition is also consistent with the way that most statisticians use the term in Structural Equation Modeling.

Using this definition, intervening variables are NOT confounded. In fact, by disaggregating the effects of an independent variable, intervening variables actually unconfound an originally confounded variable.

Using this definition, intervening variables absolutely CAN be measured. They are critical to use in non experimental research designs. They can specify what it is about the dependent variable that is important.

EXAMPLE: educational level is a cause of science attitudes because educational level influences how many science courses someone has taken, and in turn, the number of science courses affects science attitudes.

Intervening variables inform us about causal sequences or chains, thus explaining the causal process of a phenomenon.

EXAMPLE: educational level -----> occupational type -----> income level

As you can see, intervening variables, both conceptually and operationally, are very important for all but the very simplest causal assertions. It will be useful to you to practice thinking in these causal chains and speculating about precisely which intervening variables are the most critical in outcomes.

2. REACTIVITY (subject effects)

Both our Wiersma book and I AGREE that reactivity designates changes that people make in their behavior when they know that they are being studied. This is almost certainly due to increased self-awareness and self-monitoring. People also experience evaluator or evaluation apprehension because they may fear that their behavior in the laboratory somehow will be deficient so they change their behavior. The experimental laboratory is probably the most reactive because people have come for an experiment and they know their behavior is being watched, although it can occur in other study designs too. That is why so many experimenters use deception. They are trying to divert subject attention so that the "true behavior under study" is not altered.

DIVERGENCE. Wiersma lists reactivity as a threat to external validity, that is, the ability to generalize to other situations. While I don't disagree, I think the far greater threat is to INTERNAL validity. Reactivity introduces an alternative causal explanation for our results: they occurred, not because of the experimental manipulation, but because people were so self-conscious that they changed their behavior. Reactivity may also statistically interact with the experimental manipulation. For example, if the treatment somehow impacts on self-esteem (say you are told that the stories you tell to the TAT pictures indicate your leadership ability), reactivity may be a far greater problem.

3. QUASI-EXPERIMENT

Our Wiersma book defines quasi-exeriments as research using treatments and intact groups (i.e., groups pre-existing to any treatment or intervention). While I agree that interventions with intact groups probably are quasi-experimental 75-80 percent of the time, there is the other 20-25 percent, outlined in the paragraph below.

I use a more inclusive definition of quasi-experiment: If your study has different levels of treatments or interventions, and people or groups are assigned to those treatments WITHOUT random assignment, you have a quasi-experiment.

In a "true experiment," subjects are randomly assigned to treatment or intervention groups using a coin flip or some other probability, non human judgment method. Randomization is what makes true experiments so strong in internal validity. It means that on the average at the beginning of a study, all your treatment groups are about the same or "pre-equivalent." Thus, in a posttest only design, we can reasonably state that the differences we find are due to the treatment (all things equal, or "ceterus paribus"). For example, randomization controls for self-selection bias, history, instrumentation, regression effects, and other threats to internal validity.

Randomization just isn't always possible. Some treatment groups are initially formed on the basis of performance (high, medium, low, for example), some variables (e.g., bipolar depressive disorder) just aren't experimentally induced. Individuals who are not in intact groups could enter the treatment levels in your quasi-experiment through self-selection, because of their particular performance category (that bottom quartile in Intramural sports), or because a researcher has "paired" individuals a priori that she or he believes are somehow similar.

What are important are: (1) HOW subjects entered the groups in the first place; (2) What happens in the group; and (3) The length of time groups pre-existed prior to interventions.

If subjects are randomly assigned to groups in the first place (which often happens in school and universities for classes where there are many equivalent sections), the tasks to be performed are virtually identical in each group, and the pre-intervention time is short (probably a few weeks at most), THEN if you randomly assign groups to conditions, you probably have a true experiment.

Consider some alternatives:

• Subjects are assigned to groups using pre-existing knowledge about the subjects and the groups consequently differ on variables related to the study. Sports teams grouped by ability, "tracking" systems in schools, and enlisted versus officers in the military are three examples.
• Even if random assignment originally places subjects in groups, their curricula and itineraries may be different, thus providing subjects in different groups with different experiences.
• Finally, bosses and teachers differ in their approaches, again providing subjects in different groups with different experiences which diverge further as time goes on.

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS 

Conceptual variable
Operational definition
Operational variable

Independent and dependent variables 
Rules for assigning causality in non experimental data 

Recognize examples of the following and be able to define the following:

LEVELS OF MEASUREMENT
Nominal variable
Ordinal variable
Interval variable
Ratio variable

Reliability
Bias
Construct Validity
Internal Validity
External Validity

SEE YOUR WIERSMA BOOK, PAGE 104! and

HINT: CHECK OUT THIS CLASS WEB SITE FOR THESE TERMS 

Experimental reality (sometimes called artificiality)
Mundane reality

Manipulation check

Solomon Four Group Designs are more expensive because they require more subjects and conditions than other types of experimental treatments. But, many researchers believe the advantages are worth the expense. Notice how in addition to controlling other threats to internal validity, this design also controls for pretest effects, and pretest/treatment effects.

Hypothesis
Conceptual Hypothesis
Operational Hypothesis
Null hypothesis

Confounding or confounded variable: a multidimensional variable, for example, educational level which measures social class, cognitive sophistication, and exposure to diversity.

Randomization
Control group
Treatment group
"Levels of treatment" (THIS DIFFERS FROM: Variable Levels of Analysis)
Experimental demand ("demand effects")
"Organismic" variable (or naturalistic variable)
Pretest
Pilot test
Interaction effect (also known as "statistical interaction")
Repeated measures design (Wiersma sometimes calls it "time series")

IMPORTANT NOTE: While randomization or random assignment of subjects or groups to treatments/interventions is very important for internal validity, notice it says NOTHING about how you obtained your subjects or groups in the first place. Thus randomization is NOT connected to external validity.

Randomization is different from simple random sampling so do not confuse them (although both are probability-based methods).

  HERE for a fast review.
 

Given a pair of variables in nonexperimental data, can you designate which one is the independent variable, which the intervening variable, and which the dependent variable? Can you give the rule behind your decision?

Can you decide if a causal relationship is asymmetric or symmetric?

Can you decide accurately whether a variable is nominal, ordinal, or interval-ratio level?

Do you know the difference between a conceptual variable or definition and an operational variable or definition? When writing a research report, where is the appropriate spot to use each kind of variable or definition?

What are the key components of an informative research problem statement?

Can you write a clear hypothesis linking two variables? Can you convert this to a null hypothesis (if the data allow)?

What distinguishes a "true" experiment from a "quasi" experiment?

What does it mean to say a variable is "confounded?" Give an example and explain where the problem lies.
 
 

Place the following three variables in this causal order (the original order is alphabetical):
independent variable, intervening variable (IN-CLASS DEFINITION), and dependent variable.

Provide TWO RULES FOR CAUSAL ORDER IN NON EXPERIMENTAL STUDIES that you used to order the variables.

Place the independent variable on the far left, then the intervening variable, and the dependent variable on the far right:

• 1. Education, Income, Occupational Type
• 2. Anxiety test score, Gender, Type of invention film watched (violent versus not)
• 3. Baseball performance measure, Coaching style, Use of visualization technique

Which of the following is an example of reactivity?

[   ] A. Analyzing the relationship between high school grades and SAT scores.
[   ] B. Having someone fill out a survey and walk on a treadmill in the Sports Psychology lab.
[   ] C. Observing the behavior of pedestrians following a confederate against a "don't walk" sign.
[   ] D. Scoring instances of teamwork during field observations of a football team.

Only in the "B" condition is the person acutely aware that their behavior is being closely scrutinized.

External validityis important because:

[   ] A. We want our measures to be stable and predictable.
[   ] B. We want to be able to generalize our results to other persons and situations.
[   ] C. We want to make sure that the treatment variable caused the effects we observed.
[   ] D. Generally it is NOT important for a study to have high external validity.

"A" is about reliability, "C" is about internal validity and "D" is just not true.

The study design used students from undergraduate Educational Psychology classes as subjects who were randomly assigned to experimental treatments. This design poses a threat to:

[   ] A. Construct validity
[   ] B. Internal validity
[   ] C. External validity
[   ] D. Reliability

Since we don't know the variables or how they were measured, we can't say anything about construct validity or reliability ("A" or "D"), since the study used randomization, it addresses "B" or internal validity. The "grab" or convenience sample means that at best we can only generalize about FSU Educational Psychology students, thus "C" or external validity is a problem.

The Lion County School District has decided it wants to test a new method of reading. Here is what they decide to do:

In Childs High School, Mr Torkleson teaches two classes. He teaches Class A with the same methods that he currently uses. He teaches Class B with the new method. The Principal of Childs High decides which class receives the new method. Students taught with the new method score significantly higher on the F-SCAT Assessment test.

A. Is this study a true or a quasi experiment? How do you know?

B. Did the study use a Solomon 4 Group design?

C. Briefly describe one piece of information that you would find useful to evaluate this study that was not provided in this description. Tell WHY this information would be useful for you to evaluate the study.

D. Describe TWO DIFFERENT threats to internal validity that you see in this study design. Briefly describe what each threat is (e.g., testing, regression effects, etc.) and then tell WHY this is a problem for the internal validity of this design.

NOTE: EXAM 1 WILL HAVE TWO PROBLEM SOLVER QUESTIONS.

Briefly describe the SOLOMON FOUR GROUP DESIGN.
Which research method can use this design (experiment? survey? ethnography? etc.)?
Diagram the SOLOMON FOUR GROUP DESIGN (Wiersma's terminology is fine.)
Briefly describe two ways that this design can improve internal validity.

For each of the following questions, indicate (1) whether the variable is nominal, ordinal, or interval-ratio and (2) IN ONLY ONE SHORT SENTENCE describe the reason behind your decision:
 

1. Age in years	2. Stanford-Binet Intelligence Test Scores
3. Gender of teacher	4. Class ranking (e.g., Valedictorian, Salutatorian, etc.)

BRIEFLY define AND give an example of ANY THREE (and ONLY three) of the following terms:

1. Conceptual hypothesis
2. Construct validity
3. Experimental mortality
4. Posttest only experimental design
5. Quasi-experimental design

HERE IS AN INSTANCE WHERE (1) YOU WRITE ON ONLY THREE TERMS and (2) YOU MUST ANSWER BOTH PARTS OF THE QUESTION: THE DEFINITION and THE EXAMPLE TO RECEIVE FULL CREDIT.

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW

Susan Carol Losh  September 29, 2001
This page created with Netscape Composer and
is best viewed with Netscape Navigator
800 X 600 display resolution.