Leaderboard Hack-A-Thon Post-mortem
In my last post about our internal studio Hack-A-Thon, I concluded with:
At our upcoming Hack-A-Thon, I’m going to be working on rewriting the internals of our open source leaderboard code. More specifically, I want to change the API to be more readable and self documenting as well as to take advantage of transactions to get consistent snapshots of leaderboard data. After that I’m going to re-run the performance metrics to see how transactions affects the leaderboard data retrieval. Next up will be updating the public documentation. Finally I’ll release a new library of the leaderboard code. And then I’m going to do all of that for the PHP, Java, and Scala ports. That’s the plan at least.
What was it they say about the best laid schemes of mice and men?
AWRY WE GO
I started working with the leaderboard code in Ruby at 4 PM on the day of the Hack-A-Thon. As stated above, I wanted to change the API to be more readable and self documenting as well as to take advantage of transactions to get consistent snapshots of leaderboard data. This was particularly important because leaderboards are constantly being updated and re-ranked in real-time and without transactions, the data returned to the client could have changed, resulting in stale/inaccurate data. After that I re-ran the performance metrics to see how transactions affected the leaderboard data retrieval. The good news is that transactions didn't affect leaderboard data retrieval. At least in the planned usage scenario where you'd be pulling data from a leaderboard read slave. I do want to get concurrent read/write performance metrics as well. How about that for a follow-up blog post? I digress. Next up was updating the public documentation. Finally I released a new library with the updated leaderboard code. I actually waited a few weeks to do this, although there was no specific reason I didn't cut a new library package at the end of the Hack-A-Thon. All of the initial leaderboard development work took me through 11 PM, with a couple of hours of non-working in-between to go to the gym, grab dinner at home, shower, and back to the studio. My trusty "companion cube" was with me as well. I had a game plan based on my revision history to the leaderboard code to tackle the rewrite in the next language, Scala.
The scala-leaderboard API, documentation and performance metric updates obviously went quicker. I finished those updates around 4:30 AM. 12 hours into the Hack-A-Thon, being up for nearly 24 hours at this point and I was still going strong. Let's randomly pick another language, PHP.
That's right folks, I decided to tackle the updates to the PHP leaderboard library. At 5 AM. We're partying now, right? Speaking of dick killing parties (watch that video for one of the best ROFLMAO scenes from Bulletstorm). So yeah, the next time someone invites you to a PHP party at 5 AM, you know what kind of party it is. I digress. I basically got sucked into a vortex of hellfire and doom trying to get the PHP environment I somehow had set up on my home machine on my work machine. I worked at it for an hour or so. I greeted the sunrise. And then I took a 40 minute nap. Around 7:30 AM, people were starting to filter into the studio and a bunch of us decided to go for breakfast. I just needed a recharge. After breakfast I worked at the PHP environment for another couple of hours, flipped my desk over (not really), and rallied for the final push of leaderboard updates in the final language, Java.
I got through the API updates to the Java leaderboard library. I still have yet to finish off the documentation and performance metric updates.
All-in-all, the Hack-A-Thon was a success.
During the Hack-A-Thon, another colleague did his own port of the leaderboard library and so we have a good start on the erlang leaderboard library. Recently another colleague did a port of the leaderboard library to NodeJS and so we have the node leaderboard library.