Wednesday, August 17, 2011


I spent a good deal of time wrestling with getting MySQL installed on my new MacBook Air (OSX Lion), and in the name of saving someone else from repeating that amount of frustration I want to share the insights gained.

The problem I was having was that the install all went fine (using Macports) but the post-install setup of the database, using mysql_install_db, was failing. When it seemed to succeed, it left me with an empty users table and inability to gain access (or set the password for the root mysql user).

I found this: which perfectly described the situation I was in.

In the comments of that I got to this stackoverflow answer which was even better.

The first problem I had, for whatever reason, was related to permissions, and that got me past that.

However, it continued to fail.

If you're currently in that situation, I feel your pain. And here is the nugget of info you might need.

The mysql_install_db script attempts to not overwrite tables that already exist when it is called. I believe the mechanism used to do this is a little too coarse. The scripts use the @@warning_count variable to try and detect if a table already exists, setting other variables, like in this case I hit @had_user_table.

After having done an uninstall/install to get me back to clean, I ran the mysql_install_db script again, but also got warnings. It probably doesn't matter, but I specifically got this:

110816 23:42:06 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.

It may have been these, or it may have been some other warnings that were silent, but I think the (empty) user table got created but the initial entries for the root user didn't get populated. Once the empty table exists, subsequent executions of mysql_install_db then avoid writing new entries into the table. The net result being you never get the initial entries for the root user that you need.

The simple fix to this was to modify the mysql_system_data_tables.sql file (which for me was in the /opt/local/share/mysql5/mysql/ directory) to change:

INSERT INTO user SELECT * FROM tmp_user WHERE @had_user_table=0;

to be:

INSERT INTO user SELECT * FROM tmp_user;

This is basically setting it up to ignore the fact that it encountered warnings the table existed and instead you're telling it that you know better.
If you're going to do this, you should make sure your table really is empty. Run the server with the --skip-grant-tables option and run mysql and verify that the following:

SELECT * FROM mysql.user;

returns no results.

That we use MySQL for such a very small part of our platform at Yieldbot made it that much more painful. That it kept me from what was going to be my focus for the night, working with HBase, was all the more ironic.

Happy troubleshooting!

Wednesday, July 13, 2011

Are Amazon Spots Struggling?

[Update - heard directly from AWS and they report that the graph issue is a known bug and unrelated to the problem we were seeing getting m2.2xlarge instances. Hopeful we'll get to the bottom of the underlying problem]

[Update #2 - once we shared the info around time and type, AMZN was able to verify that there was a temporary issue involving only the m2.2xlarge, only in our region, and right in the window we happened to want them. Bit unlucky, but we should be all good now :-) ]

This seems pretty coincidental with the AWS announcement yesterday with per-AZ priced spot instances. At yieldbot we do some heavy lifting with clusters of m2.2xlarge servers, and haven't been able to get any spot instances for hours. Luckily we fall back to reserved instances if we have to, we'd just rather not pay the higher price. :-)

Here's a snapshot of the price history taken a few minutes ago for the m1.xlarge instances. Note that us-east-1b and us-east-1d seem to have stopped a while ago. us-east-1a and us-east-1c goes close to current, and our testing verified we could get us-east-1c instances.

Now check out the graph for the m2.2xlarge instances, taken about the same time. Availability seems to have fallen off a cliff hours ago, particularly in us-east-1c which appears to have not had availability for this instance size on the spot market for about 12 hours now.

The AWS health dashboard currently shows all green, which at some level is understandable. Spot instances are, after all, an as-available resource. The interesting thing here though is that the price shows no spikes, no price movement indicating an approaching shortage. The market didn't get expensive, there just stopped being a market.

Maybe AWS suddenly needed to add some instances to the pool of available reserved instances? Happily, reserved instances did come up and with no delays.

But we can't wait to start saving money with the spot instances again.

Anyone out there seeing the same or similar problem?

Friday, April 22, 2011

Amazon Saved Us a Bunch of Disk Space

Someone should start a webpage where people can post the beneficial things that happened to them as a result of the Amazon "cloudocalypse", just for the contrarians out there. Here's our entry.

We use MongoDB for a good piece of our platform, and have been meaning to swap in a backup of the database as the primary.

Because we had created some large collections a few months back that were subsequently dropped (for which MongoDB doesn't reclaim disk space) we found ourselves in a situation of pretty inefficient disk utilization. We'd recently bumped up to 90% on the database disk volume.

At the same time the backup with the same size volume was running at about 40%.

It was on our list to swap the backup in as the primary, but we hadn't gotten to it yet. Yesterday Amazon decided for us that it was time. And now we have all this free disk space in our database volume.

Thanks Amazon! :^)

Monday, October 11, 2010

team++ ftw!

Today's big news out of the Yieldbot offices is the addition of Soren Macbeth (@sorenmacbeth) to the team. Ok, so we don't have offices. If we did though...

Taking a step from being 2 people to being 3 people is huge. Taking that step like we just did, priceless. And now I get to work with @jonathanmendez *and* @sorenmacbeth every day. Woot!

In my startup experiences the early DNA of the team is the difference. Which is why having Soren join us just has "win" written all over it. From our first conversation it was clear our development philosophies were in alignment. I can't wait for Soren's mark to be made on the code base and on the product. Exciting times ahead!

Thursday, September 23, 2010

A Small Thought on Meaning, Humans, and Machines

I was attending Mongo Boston this week when a simple typographical oddity sent me on an interesting (and luckily brief) train of thought.

The simple occurence of "mb" on a slide to indiciate "megabytes" (it's in this presentation). First amusement because generally 'm' is milli and 'b' is bits. Of course expressing something in millibits is generally useless. So of course a human reading it (unless you're weird like me) doesn't give it a second thought.

Then I thought about the difference in how quickly I (and everyone else) make the right assumptions on what 'mb' in this context really is intended to mean.

And therein lies the huge challenge on the machine side if you're doing data mining and analytics. How do you infer intention automatically and properly, especially when you don't know ahead of time what type of mistakes or inaccuracies are going to be involved? Just the type of thing I've been having fun figuring out over the past year - at least for a particular problem space.

But something else interesting crystallized for me too at a human level. That this auto-correction that we all do so that we don't sit there confused by how one would break something into 200 millibit chunks is also the cause of many of our problems.

We need to be able to jump to conclusions in order to get past "mb" without having to ask a presenter for clarification, but we also are prone to jump to conclusions when we shouldn't - particularly in judging attributing some intention to someone else's action.

Two side of the same coin, one useful and the other problematic. Mostly we can't have one without the other though. Although I guess with practice we hopefully get better at figuring out when to jump and when not to jump. Though I think way too many people never figure out when not to jump.

If there were any typos above I hope your brain autocorrected them for you so you didn't notice.

Wednesday, September 8, 2010

The Best Wisdom is Portable

By which I mean that it can be put in multiple contexts and keep its status as wisdom.

I'm reminded of this over and over reading Seth Godin's book "Linchpin". I was expecting it would be a good book, but only a few pages in I was as excited about reading the book as maybe I've ever been starting off on a book. It just resonated so well with me, gave better voice than I could to vague thoughts that I've had, and went in brilliant directions I would have never thought of.

I'm now most of the way through, and I look back and it seems like almost at every page I wanted to stop and jot down at least one quote. Though I haven't (until now) because I couldn't bring myself to stop the flow of reading it.

The highly portable quote I hit today:

"Great bosses and world-class organizations hire motivated people, set high expectations, and give their people room to become remarkable."

This is one of the thoughts that so obviously resonates with me right now as we're thinking about putting together the team for our startup to take it to the next level.

But I feel like its true wisdom is that it resonates just as strongly in the completely different context of being a parent. It just strikes a chord with how I think about how we want to approach raising our almost-2-year-old daughter.

All children are motivated learners; are naturally curious. So you get that for free. Setting high expectations and giving her room to become remarkable is where the magic can happen. I get the impression that too many parents forgot that the last 20 years, and I hope that it's changing now.

This isn't the first case that this portability of wisdom in Linchpin hit me, and I think that is the real genius of the book.

Wednesday, September 1, 2010

Mosques, Corporations, and the First Amendment

Earlier this year I agreed with the Supreme Court majority decision on Citizens United that on First Amendment grounds Corporations and Unions could not be limited in their funding of campaigns.

The common argument made, which I disagreed with, was the simple one that the ruling made was that "corporations are people" and that this was ridiculous. And that the First Amendment should only, and was only meant to, apply to individuals.

I felt that the inclusion of "the press" in the original document indicates that these Rights, while grounded in the rights of the individual, did actually extend to the aggregations of individuals as well.

Fast forward to today and we have the "ground zero mosque" issue. Here as well I feel that under the First Amendment, this *organization* has the right to build their Community Center/Mosque wherever they see fit, and many are arguing the same. And it isn't because I/we think "churches are people" (or that "congregations are people"). Note, no single person is building or funding the community center. The first amendment here is getting invoked properly as applied to this group of people, and this should kill the "corporations aren't people" line of attack on the Supreme Court decision on Citizens United.

To put it in perspective, imagine the precedent had the Supreme Court ruled the opposite way on Citizens United. The decision would have been that the government can severely restrict the rights of aggregations of people, so long as the rights of the individuals are preserved in some more narrow sense.

Using this logic you could then say that the government could have the right to put restrictions on where a mosque get built, perhaps "within reason" or some vague qualifier. For instance, it would be easy to argue using this logic that the government could require the mosque be built 10 blocks away from the proposed location, as there would be little argument that it would put the individual members in a position of not being able to enjoy "free exercise" of their religion - it would be just a little extra commute (for some of them) after all. If you disagree with that, imagine it again but 1 or 2 blocks instead of 10.

Now one argument would be that churches exist for the express purpose of the exercise of religion and so it is more obvious that their rights more directly derive from the rights of the individual members. But then in the Citizens United case the Corporation in question was one that was created for the express purpose of putting out a political message. So its rights too were very directly derivative of the rights of the individuals that funded it.

Anyway, this is really a complex issue, much more so than "the Supreme Court thinks Corporations are people" line of reasoning.

Personally, I definitely do think that there is an overall problem of money and politics mixing. As I wrote about recently I think that transparency is a great part of the solution, as you saw in that recent case of Target dipping its toe into political contributions and the public backlash.

If we don't have full transparency yet on the money trail in politics we absolutely should have every measure in place to be fully transparent. The power of transparency in this age of information technology I think should not be underestimated.

I also think that it would be healthy to explore options to amend the Constitution to be more specific about rules around elections in particular since they are an aspect of the direct running of government. The bars are somewhat high to get that started, and they should be. But that mechanism is there for us to make changes that we deem important enough (but as Prohibition showed, we can certainly be temporarily pretty stupid about things).

But as far as the first amendment goes, I'm happy to have it consistently apply to individuals and to the organizations that individuals choose to create and belong to.