How is Understanding Measured?

Spoiler alert: It isn't.

Conceptual understanding in math has served as a dividing line between those who teach in a conventional or traditional manner (like myself), and those who advocate for progressive techniques. I have written about this topic many times, but thought I’d discuss how one measures or tests for understanding. And the short answer is “You don’t; you test for proxies.”

The most reliable proxies involve procedural fluency and factual mastery but also involve some degree of mathematical reasoning. Simply stated, a mark of understanding is being able to solve all sorts of variations of problems.

Yet, the education establishment often proceeds from the belief that “Getting answers does not support conceptual understanding.” Critics of the process of getting answers view it as “inauthentic” and “algorithmic”. For example, if we ask what is the perimeter of a rectangle, with sides that are 5 and 7 inches, say, the student applies the formula he has, memorized:  = perimeter and comes up with 24 inches. This is viewed as “rote learning” and held not to demonstrate “deeper understanding”.  Even variations on the problem, are held in disdain, such as: A rectangle has a side that is 7 inches with a perimeter of 24 inches. What is the length of the other side?

But we have to be aware of how novices learn. As Sweller and others have written in their research, novices learn how to solve problems from worked examples with subsequent problems varied in a process called “scaffolding”. Those who believe that such scaffolding is inauthentic and algorithmic, instead give students problems that can’t be readily solved by formulas or previously learned procedures. These are called “rich problems”.

What exactly are rich problems? A definition that I heard recently goes: “A problem that has multiple entry points and has various levels of cognitive demands. Every student can be successful on at least part of it."

My definition differs a bit: “One-off, open-ended, ill-posed problems that don’t generalize and which supposedly lead students to apply/transfer prior knowledge to new or novel problems.” 

Here is an example of such a problem: “What are the dimensions of a rectangle with a perimeter of 24 units?” A student who may know how to find the perimeter of a rectangle but cannot provide answers to this (and there are infinitely many) is viewed as not having “deep understanding”. 

This view holds that the so-called “inauthentic” and “algorithmic” problems do not gauge understanding because the practice, repetition and imitation of known procedures is merely "imitation of thinking”.  Peter Liljedahl, who has written about building “thinking” classrooms, holds that if students who do such “mimicking’ are not really thinking.

But imitation is the key to learning as one goes up the scale from novice to expert—and it’s harder than it looks. As anyone knows who has learned a skill through initial imitation of specific techniques, such as learning a dance step, bowling, golfing,—watching something and doing it are two different things. What looks easy often is more complicated than it appears.

So too with math. The final accomplishment often does not resemble how one gets there. Students are making the small steps necessary to develop better mathematical thinking. Simply put, no one leaps directly from novice to expert.

Dylan Wiliam, Emeritus Professor of Educational Assessment at the University College of London), sums it up nicely.  “For novices, worked examples are more helpful than problem-solving even if your goal is problem-solving”

I have heard people say “Calculation is the price we used to have to pay to do math. It's no longer the case. What we need to learn is the mathematical understanding.”

And often on the heels of this statement I will hear that they had done well in math all through elementary school, but when they got to algebra in high school they hit a wall.  Or, similarly, they did great in high school, and hit a wall with calculus.

There is much information that we do not have from such statements.  Was the education they received really devoid of any kind of understanding and all rote?  Of the people who get A’s in math in high school who are really math zombies and cannot progress to the next level  how much of what they experienced is due to concepts not explained well, emphasis on procedures only, and grade inflation? And to what extent are these problems the result of the obsession over understanding?

I have a wish-list for future studies based on communication I’ve had with people in math education:

• To what extent does success in high school math programs correlate with success in higher level math and science courses in college? (Differentiated by regular track vs AP/IB/honors track)

• For successful math students in high school, and college math what did they do that’s different than those who were successful in high school but did not do well in college math?

• And a corollary of such a study would be: What percent of the student population has had math tutors, or been enrolled in learning centers?  For such students what are the basic teaching techniques used for math?

• What effect has the emphasis on understanding been on students who have been identified as having a learning disability?

• Is there evidence that such emphasis has resulted in students being labeled as having learning disabilities?

I am interested in any studies you may know of that would shed light on these questions. I will leave you with a summary of what I believe are key regarding understanding in math:

• There are levels of understanding that one attains along the route from novice to expert.

• Attaining procedural fluency and conceptual understanding is an iterative process of which practice is key.

• Whether understanding or procedure comes first ought to be driven by subject matter and student need — not by educational ideology.