Tyranny of Intuition (part one)

I've long had a fascination with intution and how often it can be wrong. I think it stems back to when I first learned in grade school that a feather and a lead ball dropped in a vacuum would fall at the same rate. Confounding factors lead us all to have the intuition that this is not true. I remember we all argued with the teacher at first, and we were all wrong of course.

This realization that intuition can be so utterly wrong is I think what drew me to science and drove me to a degree in Physics, where I especially was drawn to that most anti-intuitive of areas: Quantum Mechanics. That was cool stuff.

In software development I've seen intuition get in the way in two major areas - and when you're a better developer you realize these exist and you combat them. The first is that you know ahead of time what code needs to be optimized. You don't. You know once the code gets exercised under real or close-to-real circumstances and you can measure the bottlenecks. The second is in thinking you know what type of user interface your customer is going to want. You don't. You will after you see them flail around with one you've built (the more experience you have the closer you'll get the first time of course).

The thing is, complex systems (which includes aggregations of people) behave in ways that often are not intuitive. Often confounding factors cloud your judgement of how they'll behave. The more you know and deeper you can think about the interactions the better. That's the good news.

The "problem" of intuition, as I see it, is that we have to rely on it. Reality is too complex to be able to think through everything in full. Even though our intution is going to be wrong, we often have to rely on it.

Because of that, I think the problem of intuition isn't one that can be solved, but it's one of those things where at least acknowledging the problem will help you avoid some pitfalls.

Experience teaches us when to trust our intuition. We won't always be right, so we should always be on the lookout for when intuition might be leading us astray.

While any given person can combat over-reliance on intuition, it's the operation of intuition in populations as a whole and its effect on public policy decisions that makes it a tyranny.

More thoughts on intuition coming in the next post.