Tuesday, February 23, 2010

Everybody was A/B Testing - Hee-yah!

(Sung to the tune of "Everybody was Kung-Fu Fighting")

Last month I read an article in The Economist (here: How to combat the natural tendency to procrastinate) about a study on procrastination that centered around farmers in Africa, and sought to answer a question about how policy could be arranged to increase fertilizer use (the usage of which is in the farmers' self interest).

The economists devised a scheme in which farmers paid the full market price for fertiliser, but had it delivered to their homes by a non-governmental organisation at no additional cost. A subset received this “discount” at harvest time, while another group were also offered free delivery, but only when planting time was imminent. Still others were offered a 50% subsidy on the market price, an approach commonly taken by governments to encourage fertiliser use. As the model of time-inconsistent preferences predicted, the offer of free delivery early in the season pushed up usage of fertiliser by 11 percentage points over a control group who were not offered anything. The same discount late in the season, however, had a statistically insignificant effect. A 50% subsidy later in the season, a much costlier policy than free delivery, pushed up usage by about as much as the early discount.

Something that really struck me about this was it seemed to parallel very strongly the type of A/B testing that is becoming ever more popular on the web (my latest venture is steeped in that world, so I've got it on the brain anyway).

I've read about plenty of studies where a group opts in to a study, maybe for some payment, and then plays some games that are meant to uncover some aspect of psychology. There's a control group, etc.

But what struck me here is that these were farmers just going about their business. I'm not sure if they were privvy to the study that was being conducted. But even if they were, they weren't "opting in" in the usual way. This gives it a striking resemblance to A/B testing online, which happens without the knowledge of the web user. They aren't taking surveys or playing a game; just going about their business with a specific behavior being tested and analyzed for the purpose of optimizing some specific outcome.

In the world of public policy it also points to a more practical mindset that could be effective if used wisely. It's more a scientific mindset than a political one, so maybe there is little hope that meaningful public policy would happen this way (although I can't help but note that it is obviously related to the States vs. Federal question within our own US system).

It's clear that in most ways the web isn't inventing something completely new with this A/B testing thing. But the web does provide unprecedented opportunities to use and learn from A/B testing.

Even though not invented there, I do think that lessons in A/B testing that will be learned from the rapidly innovative online world will find their way back into the non-online world more and more. Although it should also be a two way street. I'm sure some online marketers could learn some good lessons from the observed behavior of African farmers too.

If nothing else, the spread of a mindset that we don't necessarily know the answers at the start, and that identifying a goal to test and optimize for would be a great thing. In the world of public policy, this could make a world of difference.

Maybe it could even help get us off a road paved with good intentions, and instead onto a path toward the goals we're actually aiming at.

Monday, February 15, 2010

Time Contraction

The last months I've been coding away in my home office with time flying past me almost unnoticed. Two things really brought home what effect this is having on my time perception.

One is that my wife has pointed out a number of times lately where I've told somebody that something happened "a couple weeks ago" when in fact it could have happened as much as a couple months ago.

The other was that when we went to Florida for a family visit last week for 5 days I came back feeling like I had taken about 2 weeks off. This even though I brought my laptop and still did some work.

I generally don't mind this time contraction, especially in the Winter. What's great about it is that it is caused getting a lot done and enjoying it. "Time flies when you're having fun" as they say.

It's great to measure the passage of time by how much new stuff is working rather than how many times the earth has spun around.