Leancode

One thing about running servers is they’ll drive you insane with worry, unless you’ve set up a system to notify you if anything is amiss. I’ve been using the beta of site24×7 to monitor chartpart, and it’s a nice service. The service can check your site at 15 minute intervals. It can trigger an email to you on downtime, of course, but also when response time crosses certain thresholds, or content changes by a certain % (I’m not certain the last type is working correctly yet).

Here is an interesting chart I grabbed today from site24×7, showing response time of chartpart.com over the last week.

You can clearly see my switch a few days ago from a shared host (textdrive $12/mo) to a dedicated server (aplus $50/mo). The best case response time is similar, but the worst case on the shared host can be quite bad when other users of the same box are generating loads.

Another related change that happed at the same time: on the shared host I was running the rails app under lighttpd. In setting up the dedicated server, I switched to the hot new thang: apache mod_proxy_balancer + mongrel.

Shared hosts are great for prototyping a new site, but either a VPS/VM (with dedicated resources) or dedicated host is essential for production deployment.

The response time graph is now a thing of beauty, and I can sleep better knowing I’ll get alerted if it goes south on me. And I have nice graphs to quickly get a sense of historical performance, if anything changes.

Chartpart may bounce around for the next few hours. The new dedicated server is now all configured (more on that later), and the switch over has been triggered.

The domain is registered at godaddy.com, but the actual DNS records for chartpart.com are hosted at zoneedit right now. It may take some time for all these changes to replicate around. Both the old and new servers are up and running, so while the DNS change is replicating, you also get this interesting effect where some people will be directed to the old server, some to the new, while the DNS change is in flight.

To read more about how all this works, read the “what’s going on behind the scenes” question in the internic FAQ. Many players and some non-trivial coordination.

Amazon has a new web service in limited Beta, Amazon EC2 [techcrunch take here], a compute-on-demand facility that’s very interesting and different from the standard hosting models already out there. You pay for exactly the computing power you need: $0.10/CPU-hr [correction: it's a bit more at $0.10 per uptime hour], plus Amazon S3’s storage pricing: $0.15/GB/mo storage, $0.20/GB transfer. An “instance” (the Xen VM your app runs in) is resourced like a 1.7Ghz Xeon CPU with 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth (bursting to 1Gb)

This looks particularly interesting for a small web business like leancode. Why? Because there’s so much variability in how much computer power I might need. I’m trying to attack niches — meaning, I’m throwing small pieces of spaghetti at the wall. If I deploy with EC2, and the service doesn’t take off, I pay very little. If the service does take off, EC2 scales transparently up to (the equvalent of) the computing power of a dedicated server, with all the connectivity and hardware reliability of a large entity like Amazon. At that point, the pricing is more expensive than having a dedicated server — but I only pay if I’m already succcessful.
On the business side, what’s so interesting here is how this can lower the barriers to entry for small, yet-unproven web services. We can launch them, be set up for success, but have less downside (at least in computing cost) for failure.
On the technical side, Amazon is pioneering another interesting element — you set up the server by creating a filesystem image (called an AMI-Amazon Machine Image) which is ultimately loaded into a Xen Virtual Machine running on Linux boxes at Amazon. Unlike true VM hosting, Amazon does not guarantee the persistance of anything other than the original image. If you want to save something, you can’t just save it to the filesystem. You have to save it via web services to the S3 service or something else.

But my first project here — chartpart — does not require any persistant storage yet (other than a cache), EC2 is already a fit, and a very attractive one. If they get out of limited beta, I’ll seriously consider the move. Amazon EC2, along with the other Amazon Web Services, are successfully leading a potentially radical shift in how we think about computing resources as a true, scalable cloud of services. And that will create lots of opportunities, for those large and small alike.

A key concern with providing hosted charts is the ability to scale up — if successful, I’ll have to serve out a potential torrent of chart images (which are appearing on other people’s web sites, but which are hosted on chartpart).

And I’ll get no direct revenue from all this activity — only from those drawn to subscribe to premium services. So delivering high performance (to the client) at low cost (to myself) is a big challenge.

Since this is a very partitionable problem (more on that later), caching is a key part of how I’ll try to achieve this.
But, beyond that, load balancing is another aspect. So I wondered — are hosted services available such that I can start with a cheap account (shared host, or dedicated box if not), and load balance several of them as I grow?

I could not find any shared hosts that offered load balancing. Please comment if you know of any — I would think that’d be a benefit to clients and the hosting company. But I did find several that offered load balancing with dedicated servers.

One promising one appears to be aplus.net — they offer “value” dedicated servers at $49/mo, to which load balancing can be added after the fact. Their current standard pricing for load balancing is $159 setup and $220/mo for up to 5 servers. I like this, because I can start with a single $49/mo server, then grow to a $500/mo 5-server load balanced setup. Anyone have experience with this kind of setup, and have anything good or bad to say about aplus or another host?

It turns out my Vonage problems were unrelated to Bittorrent/Azureus.  More than one customer in our area was having problems with calls to a Vonage number resulting in ‘circuit busy’ or other messages.  The problem seems to have originated with Vonage, and resolved itself after a few days.  Nice timing for this kind of problem, being around that company’s attempted IPO and free Skype calls in the US/Canada this calendar year.
So the .torrent below is still up with at least one seed, and I’ll try to keep that going until a decision about what direction to take RORVM — continue delivering it as a finished VM, or turn it into capistrano script to configure a VM from parts.

A few things I learned from putting out RORVM last week:

• When you publish a torrent, you can publish with a “torrent tracker” site, or do it “trackerless” (dht). I used Azureus to do it the trackerless way, and I think that was the wrong choice. Going trackerless reduces the ability for others to find the torrent, which then reduces the number of other seeders to help distribute it. So I’ll try a tracker for the next one, but I’m still not sure of the best tracker to use, though.
• For a “Linux Appliance VM”, which mine is, you could imagine another means of distribution. Instead, download a standard VM image (say a straight Ubuntu Breezy 5.10 server install — or just the CD iso and install yourself) — and then re-configure the VM with a tool like Ruby’s Capistrano. So my instructions would change to 1. download ruby+capistrano to your PC. 2. download a straight Ubuntu VM. 3. download my capistrano script to re-configure the VM. Wouldn’t require a torrent on my part. But the cost of a few more steps to get it running.
• If you do try to download my current VM image as it stands, you might find no seeds available. I was keeping Azureus running on my home box to make sure we had at least one seed, but I’ve had a problem — Vonage. I’ve been making full use of the Comcast cable connection, running a lot of stuff over it, including my home phone. Well, with the 40K of upload traffic that was consumed by seeding my torrent, we started having problems with “line is busy” or other errors in calling our Vonage number. Not 100% sure it’s cause-and-effect, and don’t know why Vonage isn’t falling over to the backup number, but I’m blaming it on the torrent for now, and may need to find another way.

I’ve used a bunch of CMS systems over the years: perforce + raw HTML, wikis, zope, sharepoint, textpattern, etc.

But with leancode, I needed to strike a balance where the site had to be great for time-oriented information (blogging, project news) while being functional enough for static pages. Wordpress seemed like a good fit, but it required some customization. Here’s what was done:

1. Followed textdrive’s instructions for installing wordpress
2. Chose the andreas08 theme from themes.wordpress.net because it combined a navbar based on the sites ‘pages’ with a clean, functional design.
3. Followed instructions for using a wordpress ‘page’ as the site front page including making blog appear in non-root folder, creating a ‘blog’ page, and changing the permalink definition as described
4. Installed the static front page plugin for wordpress (again, a wget from within the target directory on the hosting site), created a ‘home’ page for it in the wordpress panel, and activated the plugin so that this home page became the new front for leancode.com
5. Went to the theme’s directory (under wp-content/themes/andr*), and edited header.php to remove the extra list element in the navbar for the duplicate ‘home’ entry on the menu
6. Changed the wordpress page order to reflect the home, blog, about order I wanted.

The result of all this is a wordpress install which is only slightly customized (theme with one tweak, and one plugin), but which otherwise can be managed from the wordpress control panel. And it looks and behaves enough like a website done with a more sophisticated CMS.
Some paths I went down and rejected:

• Having a static front page which is not a wordpress ‘page’ This would have meant hand-maintaing the nav menu, and having to edit content outside of wordpress. The method I eventually chose has eliminated those special cases.
• Giving wordpress its own directory. I think I could have got this to work, but since the whole site is in wordpress, it didn’t buy me anything right now.

Things that still bother me

• The URL for the blog is leancode.com/blog, but the URL for the feed is leancode.com/feed. I would have liked to see clear separation of the blog feed (e.g. leancode.com/blog/feed) from the other content on the site that one might subscribe to

Any suggestions on this or other aspects of the site design are welcome …

Thanks to all the creators of wordpress, andreas for the theme, and all the others involved in the deep stack of open source software who make this stuff work.