This isn't about the usual "long tail" you hear about, but about something similar that for some reason I find amusing - having just received a check for $39.78.
Back in 1998-99 I wrote a book on a VPN networking protocol called L2TP (Layer-2 Tunneling Protocol). I had written (in C/C++) our L2TP (as well as PPTP and L2F) implementations at my first startup, New Oak Communications, and then had gotten involved in the IETF process around the standardization of L2TP.
I wasn't looking to write a book, but based on some I-D's I had written on extensions to the base L2TP at the time, I was contacted by an editor in an email and asked if I'd think about writing a book. I had no delusions this would be a best seller, and said yes expecting (correctly) that it wouldn't really pay back in money for the time I spent on it, and that it would be a useful experience.
This makes me think of the long tail in two ways. First, this is obviously a niche subject. I wrote it for the audience of software developers who would be implementing the L2TP protocol, and secondarily for those that might be involved in the network planning around deploying its use. Yeah, there's gonna be a lot of those.
Honestly, I completely forget I even wrote a book unless one of two things happen. First, someone says (for some random reason) something about "your book" to me. It usually takes me a good 5 seconds to know what they're even talking about. Second, twice a year when I get the royalty statement. I say "statement" because it only becomes a royalty "check" when it accrues > $25 due to me.
So that's the second, and most significant, way it seems like a "long tail" to me. Because a full 11 years later the royalty stuff still dribbles in on this seriously niche-subject book. I wonder how much longer the poor publisher will need to keep sending me these statements.
It's most interesting to me just to see that 11 copies sold the second half of 2009. It must've been the holiday season. I'm really surprised it is more than zero at this point.
It's sold 4046 copies over these 11 years (technically only 10.5 so far), which is pretty cool. All in all it was worth the experience. The most surreal thing was one day years ago coming home to a small package that came in the mail from the publisher that contained a few copies of the Japanese translation. That was worth it just for the joke from my mother that she understood about as much of it as the original in English.
The thing I'm happiest about is the positive reviews on Amazon that it got. That's what I was most worried about back then, having direct access, and for all to see, to someone's potential negative take on something that I put some real effort into. That was a new thing back then, and it was scary.
Anyway, whenever this statement comes it just makes me chuckle. I was pretty sure I wouldn't break the $25 threshold again and get a check. I've got to imagine though that this one really will be the last one.
Wednesday, April 7, 2010
Saturday, April 3, 2010
MongoDB Sequences
I came across an issue today with MongoDB, the first one where SQL would have had a simple answer - sequences.
If you're familiar with SQL you know that this is very simple to do by declaring a column an index in a table. With MongoDB there's no built-in capability for this (as of 1.4.0, which I'm using now - I wouldn't be surprised to see some built-in sequence capability in a future release).
MongoDB does have a case that guarantees order, a "capped" collection, but it's not really meant for this purpose.
I found some hints online about how this could be done. Since none of those were satisfying and I ended up coming up with my own relatively simple way, I thought I'd share it as food for thought.
The approach is to have a javascript function saved to the database that can be called from the client to do our bidding. The client calls db.eval() to invoke this function to insert the object for us.
To set this up, I created a collection named "sequences" where the "_id" of each entry is the name of another collection in the database that I want to be sequenced. The table just needs to have an entry with an initial value for the sequence (take your pick) before the function to insert ever gets called.
For instance, if I had a collection named "foo" I would start with an entry like:
To insert an object into the database I invoke db.eval() passing the function the name of the collection to insert into, and the object to be inserted.
The function that does the insert is:
I'm returning the sequence that was allocated (which I keep track of in my use case in the case where the insert was successful) and the error information associated with the insert. That way if the insert failed for some reason (like an index uniqueness constraint violation) I still find out about it.
Some caveats are probably in order.
I'm not in a sharded environment (yet), and when I am I suspect I will have to revisit this.
This also isn't the most efficient approach for high performance because db.eval() monopolizes mongod, so depending on the database usage pattern this could be pretty disruptive. On the other hand, this mongod behavior effectively acts as a lock and means calling this function will be atomic. I'm going to wait and see if this is actually a performance issue in my environment, however, before implementing an approach that brings more complexity into the application.
Whatever the case, I thought this was an interesting way to solving the problem to think about as it's a pretty straightforward analog to the SQL sequence functionality.
By the time I need to do it differently, maybe there will be a native way to do so in MongoDB.
If you're familiar with SQL you know that this is very simple to do by declaring a column an index in a table. With MongoDB there's no built-in capability for this (as of 1.4.0, which I'm using now - I wouldn't be surprised to see some built-in sequence capability in a future release).
MongoDB does have a case that guarantees order, a "capped" collection, but it's not really meant for this purpose.
I found some hints online about how this could be done. Since none of those were satisfying and I ended up coming up with my own relatively simple way, I thought I'd share it as food for thought.
The approach is to have a javascript function saved to the database that can be called from the client to do our bidding. The client calls db.eval() to invoke this function to insert the object for us.
To set this up, I created a collection named "sequences" where the "_id" of each entry is the name of another collection in the database that I want to be sequenced. The table just needs to have an entry with an initial value for the sequence (take your pick) before the function to insert ever gets called.
For instance, if I had a collection named "foo" I would start with an entry like:
{_id: "foo", seq: 1}To insert an object into the database I invoke db.eval() passing the function the name of the collection to insert into, and the object to be inserted.
The function that does the insert is:
function(coll, obj) {
var s = db.sequences.findOne({_id: coll});
s.seq++;
db.sequences.save(s);
obj['seq'] = s.seq;
db[coll].insert(obj);
return {'seq':s.seq,
'error_status':db.runCommand("getlasterror")};
}
I'm returning the sequence that was allocated (which I keep track of in my use case in the case where the insert was successful) and the error information associated with the insert. That way if the insert failed for some reason (like an index uniqueness constraint violation) I still find out about it.
Some caveats are probably in order.
I'm not in a sharded environment (yet), and when I am I suspect I will have to revisit this.
This also isn't the most efficient approach for high performance because db.eval() monopolizes mongod, so depending on the database usage pattern this could be pretty disruptive. On the other hand, this mongod behavior effectively acts as a lock and means calling this function will be atomic. I'm going to wait and see if this is actually a performance issue in my environment, however, before implementing an approach that brings more complexity into the application.
Whatever the case, I thought this was an interesting way to solving the problem to think about as it's a pretty straightforward analog to the SQL sequence functionality.
By the time I need to do it differently, maybe there will be a native way to do so in MongoDB.
Wednesday, March 31, 2010
Homemade Anti-siphon Check Valve (zip-tie + latex glove)
A tale about bootstrapping your startup.
The weather lately has been super rainy, which unfortunately comes with some water in our basement. After our sump pump broke down I was able to switch in a utility pump and move the float-switch from the sump pump to that and it worked well.
The one problem I had, though, was that the utility pump uses a standard garden hose and neither I, nor my local hardware stores, had an attachment to make it not siphon when it wasn't running.
What this means is that once the utility pump emptied the well it is in and switched off, all the water that was in the hose would flow back down and fill the well half way back up. Obviously this means that half the work the utility pump was doing was put into moving that half-well of water back out. Basically it was working twice as much as it needed to.
I put up with this for a bit, but it stayed in the back of my head because this is a problem that should be solvable.
Today I hit upon it - using a latex glove to block the end of the hose when the water stopped so that air couldn't enter the end of the hose and let the water flow backward.
Here's the recipe:
I'm mainly posting this because I didn't have much luck searching for homemade solutions online. So maybe this method will catch on. Just sent me a $1 royalty any time you use it. ;-) My guess is the "parts" probably total about $0.05 in cost.
Here's what happens. When the water turns on, the pressure expands the middle finger part and the water flows up and out the other fingers. When the water stops, the lack of pressure lets the glove contract back into place, with the middle finger back around the hose. When the water wants to flow backward in the hose, the negative pressure sucks the tip of the middle finger back into the hose where it forms a tight seal around the end of the hose and stops it.
Of course, you could use duct-tape instead of a zip-tie - but I like to use zip-ties whenever possible.
Here's why this is better than store-bought! I was able to do this on the end of a hose that had been cut to length, with an irregular shaped end and no threads.
I just couldn't resist a little video to show clearly how it works. This is the one I did around the hose with the cut end. Yes, it is giving you the finger.
Direct link: here
The funny thing about this whole exercise is that it reminded me of what it's like bootstrapping a startup.
There's always a way to solve a problem. The solution you think of first you might not be able to get access to or afford. But remember that you may just have a latex glove and some zip-ties laying around that might even do the job better.
The weather lately has been super rainy, which unfortunately comes with some water in our basement. After our sump pump broke down I was able to switch in a utility pump and move the float-switch from the sump pump to that and it worked well.
The one problem I had, though, was that the utility pump uses a standard garden hose and neither I, nor my local hardware stores, had an attachment to make it not siphon when it wasn't running.
What this means is that once the utility pump emptied the well it is in and switched off, all the water that was in the hose would flow back down and fill the well half way back up. Obviously this means that half the work the utility pump was doing was put into moving that half-well of water back out. Basically it was working twice as much as it needed to.
I put up with this for a bit, but it stayed in the back of my head because this is a problem that should be solvable.
Today I hit upon it - using a latex glove to block the end of the hose when the water stopped so that air couldn't enter the end of the hose and let the water flow backward.
Here's the recipe:
- cut the tips off all the fingers of a latex glove, except the middle finger;
- put the latex glove over the hose with the end of the hose down as far into the middle finger as you can get it;
- stretch it fairly tight and ziptie the wrist of the glove around the hose to hold it in place.
I'm mainly posting this because I didn't have much luck searching for homemade solutions online. So maybe this method will catch on. Just sent me a $1 royalty any time you use it. ;-) My guess is the "parts" probably total about $0.05 in cost.
Here's what happens. When the water turns on, the pressure expands the middle finger part and the water flows up and out the other fingers. When the water stops, the lack of pressure lets the glove contract back into place, with the middle finger back around the hose. When the water wants to flow backward in the hose, the negative pressure sucks the tip of the middle finger back into the hose where it forms a tight seal around the end of the hose and stops it.
Of course, you could use duct-tape instead of a zip-tie - but I like to use zip-ties whenever possible.
Here's why this is better than store-bought! I was able to do this on the end of a hose that had been cut to length, with an irregular shaped end and no threads.
I just couldn't resist a little video to show clearly how it works. This is the one I did around the hose with the cut end. Yes, it is giving you the finger.
Direct link: here
The funny thing about this whole exercise is that it reminded me of what it's like bootstrapping a startup.
There's always a way to solve a problem. The solution you think of first you might not be able to get access to or afford. But remember that you may just have a latex glove and some zip-ties laying around that might even do the job better.
Wednesday, March 24, 2010
Pulling back the veil
Today we started to reveal what our company is working on. Jonathan, CEO & Founder, went into it in an interview here: http://www.adexchanger.com/platforms/yieldbot/
Pretty exciting to see info out there describing what we've been working on for months. Just as exciting is the data that we're starting to get from our private Beta.
Publicly talking about the product + customers using the platform. Two great tastes that taste great together.
Pretty exciting to see info out there describing what we've been working on for months. Just as exciting is the data that we're starting to get from our private Beta.
Publicly talking about the product + customers using the platform. Two great tastes that taste great together.
Monday, March 15, 2010
The Web is still in Beta
You hear the 2.0 and 3.0 labels used so much to describe "The Web" that you can actually start to believe it.
But it got me thinking, if you step back and look at the state of the web as you would a product, can it even be considered to be 1.0 yet?
I think the 1.0/2.0/3.0 mentality put too much emphasis on the "techy" point of view, and not enough on that of "normals" (See Chris Dixon's post "Techies and normals").
I'd make a case that what is commonly referred to as "Web 1.0" (late 90's - the "bubble" period) should be considered "Alpha". In this phase everyone was trying straightforward ports of functionality that you can do offline and bringing it online. The result was too much investment, weak business plans, etc., etc. Most people bothering to read this probably know it well.
For most of the past decade and up to now we've had massive adoption of online services and much broader demographics coming online and using services (such as the distribution that Facebook has).
There's no doubt that the online experience has gotten better and better, and ever more useful services are becoming available (and ever more intricate and involved time sinks). But I think this period should be considered "Beta", and that's still where we are.
I think the main problem is that the web is still a set of disjointed tools that us creators are still trying to figure which should be built, and regular users are still trying to find.
That's a fine state to be in, it just isn't "2.0".
There's a few areas that stick out to me in particular (some around booking travel or the struggles of the "news" industry), and one we're building a company around is online advertising.
The Alpha phase of the Web saw flashing gifs, popups, and "punch the monkey" type ads. The Beta phase has seen Google massively capture the value of the advertising dollar with a truly revolutionary model around a service everyone uses (search).
The problem has been the sucking sound of money flowing away from the publishers in this model. Before the web hits 1.0 it needs a model where publishers can have a substainable business. We're creating a platform that will do just that, putting that sucking sound into reverse, and allowing publishers to capture the value of their audience. Our little part of getting the Web to 1.0. :-)
But it got me thinking, if you step back and look at the state of the web as you would a product, can it even be considered to be 1.0 yet?
I think the 1.0/2.0/3.0 mentality put too much emphasis on the "techy" point of view, and not enough on that of "normals" (See Chris Dixon's post "Techies and normals").
I'd make a case that what is commonly referred to as "Web 1.0" (late 90's - the "bubble" period) should be considered "Alpha". In this phase everyone was trying straightforward ports of functionality that you can do offline and bringing it online. The result was too much investment, weak business plans, etc., etc. Most people bothering to read this probably know it well.
For most of the past decade and up to now we've had massive adoption of online services and much broader demographics coming online and using services (such as the distribution that Facebook has).
There's no doubt that the online experience has gotten better and better, and ever more useful services are becoming available (and ever more intricate and involved time sinks). But I think this period should be considered "Beta", and that's still where we are.
I think the main problem is that the web is still a set of disjointed tools that us creators are still trying to figure which should be built, and regular users are still trying to find.
That's a fine state to be in, it just isn't "2.0".
There's a few areas that stick out to me in particular (some around booking travel or the struggles of the "news" industry), and one we're building a company around is online advertising.
The Alpha phase of the Web saw flashing gifs, popups, and "punch the monkey" type ads. The Beta phase has seen Google massively capture the value of the advertising dollar with a truly revolutionary model around a service everyone uses (search).
The problem has been the sucking sound of money flowing away from the publishers in this model. Before the web hits 1.0 it needs a model where publishers can have a substainable business. We're creating a platform that will do just that, putting that sucking sound into reverse, and allowing publishers to capture the value of their audience. Our little part of getting the Web to 1.0. :-)
Saturday, March 13, 2010
Tyranny of Intuition (part two)
In my first post on intuition I mentioned how it fascinated me how often our intution is wrong. This fascination started with the natural sciences, but when it comes to social and political science it becomes even more fascinating. Mostly because it stops being personal and reliance on intuition across the population I think leads to bad policy decisions.
I thought it would be interesting to touch on some cases off the top of my head where intuition leads us (or has led us) down mistaken paths.
Availability of pornography increases sexual crimes. I was reminded of this one this week when I saw an article (http://www.the-scientist.com/article/display/57169/) on this topic citing a study that found the opposite. It's not the first study I've seen referenced that found the same. Same goes for violent movies/games/music and crime rates.
Constantly praising kids will raise their self esteem. We went through quite a period of "all kids need to win" which I think was a real problem. I've read a lot in recent years about how the opposite actually is true. Kids know when they're being gamed, and the praise is devalued if they know they didn't do much to get it. They lose their incentive to try harder. They will also give up earlier on hard problems because they don't know what it means to be challenged. This is an area that seems to be correcting, and the Pixar movie The Incredibles did a great job on this theme. Self-esteem is important of course - and more important is earning it. For some reason our intuition was that we could create self-esteem out of thin air.
College financial aid makes college more affordable. Intuitively it makes sense, if you ignore (purposely or not) fundamental dynamics of economics. Means-tested assistance does make sense, but the system we have has clearly gotten out of control. College education inflation has persisted over time well above the general inflation rate because it has turned into a government subsidized industry. I actually think this one will change. It might be painful, like the real estate bubble bursting, but by the time my daughter is of college age (17 years!) I expect this to be much different. Technology will eventually undercut the traditional model, and I think you're starting to see that.
The price of real estate will always go up. We're living the effects of that mistaken intuition (or maybe just what was previously a massively-shared assumption), but how can you not mention it?
Our neighborhoods are more dangerous now than when I was a kid. To see the levels of protection applied to today's kids you'd think so anyway. In second grade (7 years old) I walked more than a mile to school, along and across major roads, in a city. Now we have parents in the suburbs waiting with their kids for the bus at the end of their driveway. Of course I think we mostly know that it's the availability of information about bad things happening that has actually gone up. We're bombarded by media that sells us the fear that is mostly what we'll respond to with buying.
Paper money is worth something. Maybe this is less an intuition, and more a shared scam we've mostly bought into. But all past fiat currencies have been hyperinflated out of existence.
The heads of our government are less greedy than the heads of our corporations. I think we have this intuition because we naturally want to believe the someone is looking out for us, and that government (or at least our US government) is automatically benevolent. The reality is that people are mostly all the same. This intuition causes us to not be critical enough of the power government assumes for itself.
I want to finish up with the punchline - why I think we live under a tyranny of intuition and why it's probably inevitable (which, admittedly, is disappointing). It's this:
* Most people are fine not looking past their intuitions;
* They will elect representatives who cater to those intuitions;
* In many cases these intuitions are wrong.
As a result, the major policy decisions that contol the ways we live our lives and limits our freedom are and will continue to be mainly driven by the mistaken intutions of the majority of the population.
It's not a happy story, but it's a conclusion I've been coming to more and more these days. Could I be wrong? I'd actually hope so. Is my intuition that most people are fine not looking past their intuitions wrong? My experience says no - and the level of discourse of recent policy discussions that I've witnessed reinforces this for me.
Well, at least I got that off my chest.
I thought it would be interesting to touch on some cases off the top of my head where intuition leads us (or has led us) down mistaken paths.
Availability of pornography increases sexual crimes. I was reminded of this one this week when I saw an article (http://www.the-scientist.com/article/display/57169/) on this topic citing a study that found the opposite. It's not the first study I've seen referenced that found the same. Same goes for violent movies/games/music and crime rates.
Constantly praising kids will raise their self esteem. We went through quite a period of "all kids need to win" which I think was a real problem. I've read a lot in recent years about how the opposite actually is true. Kids know when they're being gamed, and the praise is devalued if they know they didn't do much to get it. They lose their incentive to try harder. They will also give up earlier on hard problems because they don't know what it means to be challenged. This is an area that seems to be correcting, and the Pixar movie The Incredibles did a great job on this theme. Self-esteem is important of course - and more important is earning it. For some reason our intuition was that we could create self-esteem out of thin air.
College financial aid makes college more affordable. Intuitively it makes sense, if you ignore (purposely or not) fundamental dynamics of economics. Means-tested assistance does make sense, but the system we have has clearly gotten out of control. College education inflation has persisted over time well above the general inflation rate because it has turned into a government subsidized industry. I actually think this one will change. It might be painful, like the real estate bubble bursting, but by the time my daughter is of college age (17 years!) I expect this to be much different. Technology will eventually undercut the traditional model, and I think you're starting to see that.
The price of real estate will always go up. We're living the effects of that mistaken intuition (or maybe just what was previously a massively-shared assumption), but how can you not mention it?
Our neighborhoods are more dangerous now than when I was a kid. To see the levels of protection applied to today's kids you'd think so anyway. In second grade (7 years old) I walked more than a mile to school, along and across major roads, in a city. Now we have parents in the suburbs waiting with their kids for the bus at the end of their driveway. Of course I think we mostly know that it's the availability of information about bad things happening that has actually gone up. We're bombarded by media that sells us the fear that is mostly what we'll respond to with buying.
Paper money is worth something. Maybe this is less an intuition, and more a shared scam we've mostly bought into. But all past fiat currencies have been hyperinflated out of existence.
The heads of our government are less greedy than the heads of our corporations. I think we have this intuition because we naturally want to believe the someone is looking out for us, and that government (or at least our US government) is automatically benevolent. The reality is that people are mostly all the same. This intuition causes us to not be critical enough of the power government assumes for itself.
I want to finish up with the punchline - why I think we live under a tyranny of intuition and why it's probably inevitable (which, admittedly, is disappointing). It's this:
* Most people are fine not looking past their intuitions;
* They will elect representatives who cater to those intuitions;
* In many cases these intuitions are wrong.
As a result, the major policy decisions that contol the ways we live our lives and limits our freedom are and will continue to be mainly driven by the mistaken intutions of the majority of the population.
It's not a happy story, but it's a conclusion I've been coming to more and more these days. Could I be wrong? I'd actually hope so. Is my intuition that most people are fine not looking past their intuitions wrong? My experience says no - and the level of discourse of recent policy discussions that I've witnessed reinforces this for me.
Well, at least I got that off my chest.
Friday, March 12, 2010
Tyranny of Intuition (part one)
I've long had a fascination with intution and how often it can be wrong. I think it stems back to when I first learned in grade school that a feather and a lead ball dropped in a vacuum would fall at the same rate. Confounding factors lead us all to have the intuition that this is not true. I remember we all argued with the teacher at first, and we were all wrong of course.
This realization that intuition can be so utterly wrong is I think what drew me to science and drove me to a degree in Physics, where I especially was drawn to that most anti-intuitive of areas: Quantum Mechanics. That was cool stuff.
In software development I've seen intuition get in the way in two major areas - and when you're a better developer you realize these exist and you combat them. The first is that you know ahead of time what code needs to be optimized. You don't. You know once the code gets exercised under real or close-to-real circumstances and you can measure the bottlenecks. The second is in thinking you know what type of user interface your customer is going to want. You don't. You will after you see them flail around with one you've built (the more experience you have the closer you'll get the first time of course).
The thing is, complex systems (which includes aggregations of people) behave in ways that often are not intuitive. Often confounding factors cloud your judgement of how they'll behave. The more you know and deeper you can think about the interactions the better. That's the good news.
The "problem" of intuition, as I see it, is that we have to rely on it. Reality is too complex to be able to think through everything in full. Even though our intution is going to be wrong, we often have to rely on it.
Because of that, I think the problem of intuition isn't one that can be solved, but it's one of those things where at least acknowledging the problem will help you avoid some pitfalls.
Experience teaches us when to trust our intuition. We won't always be right, so we should always be on the lookout for when intuition might be leading us astray.
While any given person can combat over-reliance on intuition, it's the operation of intuition in populations as a whole and its effect on public policy decisions that makes it a tyranny.
More thoughts on intuition coming in the next post.
This realization that intuition can be so utterly wrong is I think what drew me to science and drove me to a degree in Physics, where I especially was drawn to that most anti-intuitive of areas: Quantum Mechanics. That was cool stuff.
In software development I've seen intuition get in the way in two major areas - and when you're a better developer you realize these exist and you combat them. The first is that you know ahead of time what code needs to be optimized. You don't. You know once the code gets exercised under real or close-to-real circumstances and you can measure the bottlenecks. The second is in thinking you know what type of user interface your customer is going to want. You don't. You will after you see them flail around with one you've built (the more experience you have the closer you'll get the first time of course).
The thing is, complex systems (which includes aggregations of people) behave in ways that often are not intuitive. Often confounding factors cloud your judgement of how they'll behave. The more you know and deeper you can think about the interactions the better. That's the good news.
The "problem" of intuition, as I see it, is that we have to rely on it. Reality is too complex to be able to think through everything in full. Even though our intution is going to be wrong, we often have to rely on it.
Because of that, I think the problem of intuition isn't one that can be solved, but it's one of those things where at least acknowledging the problem will help you avoid some pitfalls.
Experience teaches us when to trust our intuition. We won't always be right, so we should always be on the lookout for when intuition might be leading us astray.
While any given person can combat over-reliance on intuition, it's the operation of intuition in populations as a whole and its effect on public policy decisions that makes it a tyranny.
More thoughts on intuition coming in the next post.
Subscribe to:
Posts (Atom)