Feedback in Complex Environments

The experience of striking a tennis ball offers rich feedback.

As you're waiting for the ball, you can feel your body position through a sense called proprioception. You can see the ball leave your opponent's racquet and see your opponent's body position. As you move into position to return the ball, your brain is subconsciously (and possibly consciously) monitoring your movement to make it match the ball's movement. As you strike the ball, you can feel it solidly hit the sweet spot on the racquet or dully clank off the frame. As the ball sails back towards your opponent, you can see its entire trajectory: whether it passes over the net, whether it goes in or out, whether it moves your opponent out of their position or not.

Many sports are like this. A basketball player gets feedback before, during, and after their shot. They might not know what they need to change, but whether the ball went in the basket or not is obvious – if it went in, something went right; if it didn't, something went wrong.

Many other fields and social systems, however, lack this feedback structure. Like basketball players, radiologists have a clear goal: detect cancer. But in many cases – perhaps in all cases, I don't know – the radiologist never gets to know whether their preliminary diagnosis was right or wrong. Based on the radiologist's reading, a biopsy might be performed. But these biopsy results come out weeks or even months later. And they are analyzed by a different specialist. It's easy to see how a basketball player or a tennis player gets better, but how does the radiologist get better? How would basketball players learn to shoot if they never saw whether their shot went in?

From my limited interactions with the medical system, it's unclear when or how patient outcomes are ever collected. I had minor surgery on my eye, many years ago. The surgery didn't fundamentally solve the problem, but the problem was minor enough, and I moved elsewhere. No one asked me for a follow-up report, so they had no idea whether the surgery worked.

I've had a persistent musculature/tendon issue for years and visited several doctors and many physical therapists. When these offices discharged me, did they consider me to be "cured"? I can tell you that I wasn't. Was anything written down at all about the outcome of my case and the many other cases they confront? I don't know.

Of course, medical assistance for other ailments that I've had has improved my clinical outcome. But, again, how would they actually know that my condition has improved and how would this feedback (along with feedback about all other clinical outcomes) change their clinical practice?

So the first step to learning from feedback is to get feedback. Without it, it's hard to improve.

Teachers face a similar problem in a different way. They can get immediate feedback on what their students can do. The popular "Exit Ticket" method, where a teacher gives students a short, ungraded quiz at the end of the class, provides an immediate measure of learning. And they can get feedback in the medium term, too. Students take tests and mid-terms and finals.

But these measures have limits. The ultimate outcomes of education go beyond a semester or a year – it's meant to impact cognition well into the future. And these outcomes encompass many items that we don't test for: character development, collaborative problem solving ability, clarity of thinking, emotional self-regulation, positive self-esteem, practical know-how, etc.

More importantly, it's hard to match inputs to outcomes. Say that my teacher did something to improve my long-term outcomes. What was it? The choice of textbook? The fun activities? The hard problem sets? Some magical mixture of everything?

Medical treatment, by contrast, is simple. Patients usually take one treatment for one ailment at a time. And the treatment itself – the drug, the surgery, etc. – is the presumed cause of the medical outcome; not the color of clothing the doctor was wearing, or whether they smiled at you when you came in, or whether you liked the same baseball team.

The second step is to match the feedback to inputs of some kind, at least tentatively. If you can't match a meaningful outcome with some inputs (or combination of inputs), improvement through practice becomes very hard.