Bias and Noise: Tools for Classroom Triage

Imagine you teach four archery classes of five students each. At the end of the class, you have a test to see how well students have learned to shoot an arrow. The results of each class are as follows.

As a teacher, you might ask yourself why the results are different in each class. There’s actually an answer. Economist Daniel Kahneman details a very similar scenario in his book “Noise.” The two classes we need to examine to understand what’s going on are Class B and Class C.

• Class B has a bias problem. The student outcomes in this class are all similar, but the outcome isn’t what is desired.

• Class C has a noise problem. There seems to be no pattern to student results, which is also not desirable.

Class A also has bias, but there’s no problem. And Class D has both bias and noise. But the key here is to understand the fundamental difference between noise and bias.

Biased and Noisy Classrooms

Any class in any subject can have bias and noise problems. On a multiple choice math quiz, students could typically give a certain answer, or answers could be scattered. In a writing class, papers could exhibit a recurring grammar error, or they could have weaknesses from formatting to logical flow. Or finally, a choir could make a timing error on the same piece of music, or they could consistently have blending and tuning issues.

The point is, noise and bias are everywhere in education, and being able to identify and distinguish them is vital to improving your students’ learning experience. And we’ll examine each kind of problem in turn.

Biased Problems

It’s important to understand that the use of the word “bias” here does not refer to student predispositions or idiosyncrasies that alter the way one approaches things. Nor does it refer to the predispositions or idiosyncrasies of the teacher.

Additionally, the term “bias” here does not immediately infer that a result is bad. Let’s take the archery scenario again, but instead set the test up this way:

• Give each student two targets to shoot at (Target 1 and Target 2).

• Tell students that both targets count as the same amount of points, students just need to hit the bullseye in one of them.

If Class A all chooses Target 1 instead of Target 2 and all of the students still get a bullseye, we would say that Class A is biased toward Target 1, but that would not be negative.

Instead, “bias” here indicates student results being clustered. This can be either desirable or undesirable. If on a multiple choice test students answer B, and that’s the correct answer, that would be a desirable bias.

But what do you do when you have a bias problem? What if your students are all producing similar results, but those results aren’t what you’re hoping for?

If this is case, it’s important to examine the course curriculum—what is being taught. When you have a bias problem, then students understand what is being taught, but there may be gaps or errors in the content they’ve learned.

To once again borrow from our archery scenario, students in Class B clearly all understood the same things. They had similar skills, knew the same principles, but something was missing. It could be they weren’t taught how to properly calibrate the sights on the bow or how to adjust aim for wind conditions. Either way, the issue was not a question of whether or not students learned, it’s with what they learned.

Standardized math tests illustrate this issue profoundly well. In a four option multiple choice math test, it’s very common to see the following set of answers:

• One correct answer (we’ll call this Answer A)

• One incorrect answer that follows the correct process except for one detail (Answer B)

• Two incorrect answers that approach the problem in entirely wrong ways (Answers C and D)

If students tend to cluster toward any answer but A, not only can you tell that students have a similar gap in their knowledge, but you can actually identify where that gap is. If they cluster around Answer B, you can pinpoint the exact misstep the students are making. If they are biased toward C or D, the issue is that they don’t actually understand the problem itself—and you can tell by their response what they think the problem is.

Again, here it’s clear to see that a bias problem is not an issue with whether or not the students understand, it’s an issue with what the students understand.

Noisy Problems

As you may be able to guess, noisy problems are not an issue with what is being taught, but rather how it’s being taught. You could teach all the right things, but if you don’t teach them well, your students still won’t have desirable outcomes.

In the archery image, Class C had scattered outcomes. This is not because what was being taught was incorrect, it’s because what was being taught was not understood. When students don’t understand what is being taught, they’re left with the same amount of knowledge as they had before, so outcomes are random.

Essentially, if teaching is not done well, student results are left to chance.

If you asked a chemist, a lawyer, and a teacher to each build you a house, you could guarantee that they would all make mistakes along the way, but you could also guarantee that they would make different mistakes. This is precisely because none of them know how to build a house—or more accurately, none of them know all the same things about building a house. This is how noise works.

When you have a noise problem in your class, it’s important to examine how you run your classroom. Are you using effective teaching methods? Are you structuring class time in a way that allows students to demonstrate what they know? Are your students asking questions?

Asking questions like these will help you identify where your weak points are as a teacher, and allow you to improve upon them. As a result, student outcomes should become less noisy.

The Overlap

It’s a little bit of a misnomer to say that problems are exclusively either noisy or biased. Almost always, problems are both. This means, that when our students aren’t producing the outcomes we are hoping for, the results probably most closely resemble Class D in the archery example.

It is not uncommon for groups of students to misunderstand the same concept taught in class. On a test, this may exhibit as a bias problem, but they way you solve it is through better teaching. So, yes, effective teaching reduces both noise and bias. And the same is true for better curricula—good content contextualizes problems, which means that student outcomes are less noisy.

This means that reducing noise helps you identify bias, and identifying bias helps you reduce noise. It’s a virtuous cycle that every student needs their teachers to engage in.

The benefits of separating noise and bias are two-fold, and we’ll explore each of them in turn.

First, it increases the teacher’s ability to influence the class positively. It can be very discouraging and frustrating when it’s clear that students don’t understand something in a class. When students struggle, it can fill teachers with self-doubt and aggravated confusion. And it can be tempting to push responsibility onto the student, thinking that they just need to study more or harder. But when a teacher can understand that student outcomes are largely influenced by what is being taught and how it’s taught, it gives the teacher the ability to change those outcomes for the better.

Not only is this obviously the better conclusion to arrive at for the student, but it’s also better for the teacher. As class outcomes improve, fewer issues crop up, which benefits the teacher because they end up spending less of their time in difficult or frustrating situations.

Second, it gives the teacher a place to start. By separating bias and noise, teachers can more easily pick a starting point for improving their class outcomes. Since reducing noise and bias is cyclical in nature, teachers just need an entry point to effectively benefit their classes. Separating noise and bias gives teachers two entry points to helping every class that struggles with undesirable outcomes. Any class that is having struggling as a whole can be helped by asking this question:

Is this a problem with what is being taught, or how it’s being taught?