(2) 1 page article reviews

Internet: http://www.mpr-online.de © 2002 IPN Kiel

Misinterpretations of Significance:

A Problem Students Share with Their Teachers?

Heiko Haller1 & Stefan Krauss2

Abstract

The use of significance tests in science has been debated from the invention of these

tests until the present time. Apart from theoretical critiques on their appropriateness for

evaluating scientific hypotheses, significance tests also receive criticism for inviting mi-

sinterpretations. We presented six common misinterpretations to psychologists who

work in German universities and found out that they are still surprisingly widespread –

even among instructors who teach statistics to psychology students. Although these mi-

sinterpretations are well documented among students, until now there has been little

research on pedagogical methods to remove them. Rather, they are considered “hard

facts” that are impervious to correction. We discuss the roots of these misinterpretations

and propose a pedagogical concept to teach significance tests, which involves explaining

the meaning of statistical significance in an appropriate way.

1. Introduction

The current debate about null hypothesis significance testing (often referred to as

NHST) reminds us of a struggle that ends in an impasse between the critics and the

defenders. The widely reported criticisms of NHST address – among other issues – its

weight in social science, its mathematical principle, its ease of misinterpretation and its

mindless use (for a review on the “significance test debate” see, for instance, Nickerson,

2000). At present, both parties seem to have won: On the one hand, the critics of NHST

1 Free University of Berlin, Dept. of Educational Science and Psychology, Otto-von-Simson-Straße 19, 14195 Berlin; E-Mail: [email protected] 2 Max Planck Institute for Human Development, Lentzeallee 94, 14195 Berlin; E-Mail: [email protected]

2 MPR-Online 2002, Vol. 7, No. 1

because much of their critique is substantial and largely uncontradicted (Sedlmeier,

1996; Carver, 1993) and on the other, the defenders because NHST is still taught to

psychology students at universities as the method for evaluating scientific hypotheses.

This article does not comment on this debate (for recent contributions concerning the

debate on NHST in this journal see Sedlmeier, 1996; Iseler, 1997; Sedlmeier, 1998; Ha-

ger, 2000). Rather, we take the teaching of NHST as a given fact and focus on impro-

ving it. Unfortunately, literature suggests that after a statistics course the avarage stu-

dent cannot describe the underlying idea of NHST (Falk and Greenbaum, 1995; Gige-

renzer and Krauss, 2001). What is mastered is the mere calculation of a significance

test. Yet, in our view the teaching of NHST can only be justified if students are able to

grasp the meaning of what they are doing.

The lack of understanding NHST can be demonstrated by asking students the follo-

wing question: “What does the statement that a statistical measure is significant at the

5% level mean?” It is not very likely to receive the correct answer, namely: “The proba-

bility of the available (or of even less likely) data, given that the null hypothesis is true,

is less than 5%.” Tragically, this answer is not just an answer to an inconsequential sta-

tistical question. Rather, this answer represents the understanding of the underlying

idea of the whole inference approach.

An overview of students’ misinterpretations of this question reveals that there are ba-

sically two main classes: The first class contains “meaningless answers” like: “It means,

that the measure lies 5% above the random-percentage”. The second class of misinter-

pretations contains the belief that with NHST the probability of hypotheses can be as-

sessed.3 Furthermore, there are individual cases deviating from these classes, such as the

statement that significance says something about the replicability of a significant test

result. It is important to note that most students showing such misinterpretations have

mastered the formal procedure of significance testing.

The aim of this article is twofold: In the first section, we describe possible sources of

the widespread confusion. The commonly accepted view seems to be that psychology

students – who are not interested in statistics – show a natural confusion about an in-

herently difficult concept that invites misunderstandings regardless of the way it is

3 For a collection of wrong beliefs stemming from a written undergraduate examination (Vordiplom) in

psychology see Gigerenzer and Krauss (2001, pp. 53-54).

H. Haller & S. Krauss: Misinterpretation of Significance 3

taught. In the first section, our questions are: Do instructors of methodology (i.e.,

people who are responsible for teaching NHST to psychology students) address and cla-

rify the meaning of NHST in their lectures? Furthermore: Are they aware of the correct

interpretation of a significant test result at all? Our empirical results reveal that cla-

rifying the meaning of NHST to psychology students in Germany is usually not a mat-

ter of statistics education. We found that most methodology instructors do not even

know the correct interpretation but rather share the misconceptions of their students.

Although this lack of insight – at least among students – can be considered a well-

known fact, there is astoundingly little pedagogical effort to eliminate these misconcep-

tions. In the second section, we provide a pedagogical concept of how to teach signifi-

cance testing that is explicitly designed to do away with the typical misinterpretations.

Where do the Misconceptions come from?

As previous experimental studies show, almost all students (Falk & Greenbaum,

1995) and even most academic psychologists (Oakes, 1986) have no real insight into the

meaning of a significant test result. Does this confusion principally arise in students’

minds or is it supported – or even implemented – from outside? For the implementation

assumption, we basically have two suspects, namely (1) statistical textbooks or (2) sta-

tistics teachers.