30 second timeouts, part 2

This is a follow-up to my earlier post about the 30 second timeout rule and its implications for programs like movable type.

Thank you, everyone who offered feedback. I read every reply and tried to address every point raised. I'm working from notes here and responding to a lot of issues at once, so forgive me if I don't credit everyone who offered an idea individually.

clarifications

First of all, I'm not trying to ban MT, and I don't ever want to be pushing people from one CMS to another every time there's a problem. Banning MT would be disastrous for cornerhost's financial prospects, so it's not at all what I was trying to say. MT is simply the most popular piece of software that routinely conflicts with the 30 second execution rule that happens to be very effective at protecting the servers.

Nor did I mean to imply that MT wasn't mature. I just meant that there are other blogging tools that are also mature and don't have MT's particular problem. Wordpress blogs get just as much comment spam as MT blogs, and as Chuck Welch pointed out, there are hosts out there that have had to ban wordpress sites for hogging resources.

how MT degrades

The scalability problems in MT and wordpress are very different. MT uses static html files so that it's easy for thousands of people to view the same page at once. Wordpress (or MT in dynamic page mode) has to make a database connection for each page view and then execute a bunch of database queries, so a massive influx of traffic can tie up the database and really slow things down.

Mt degrades in a different way. Because all the pages on an MT site are rendered as html files, then pages need to be rebuilt every time something changes. For example, if someone post a comment on an entry, then the front page and any archive or category page that shoes that entry has to be updated, because those pages all show the number of comments at the bottom of the entry. The more comments and entries there are, the longer this takes. However, MT often runs much slower than it should, even accounting for these requirements. For example, someone told me last night that the time required for posting a comment to his site went from about 4 seconds to about 40 over the course of a couple days.

If you think about the graph of an exponential function, this makes sense. Things run smoothly, almost flat for a while, and then abruptly curve upward. I haven't jumped into the MT source code to prove this, but I suspect that the comment posting algorithm in MT displays this behavior: you grow too big and suddenly you hit a tipping point, and suddenly your site takes forever to do anything. Like I said, I haven't proved this, but it seems to fit the data.

Comment spam + Slow MT = dead server

Then along comes a spammer, ready to post 500 or even just 5 comments to a blog. Since the spam comes in faster than a "slow" MT site can deal with it, the mt-comment.cgi and mt-trackback.cgi processes pile up in memory. And remember, these aren't little tiny CGI scripts any more. Each one has inflated into a server-crippling resource hog of it's own.

Each server at cornerhost has Dual Xeon 2.8 GHz processors. These are FAST machines. But a program with exponential complexity can tie it up just the same. Think of the exponential graph as representing the amount of hardware resources necessary to complete the task: it shoots up so fast that it doesn't matter how powerful your computer is.

Not every MT site is affected

I haven't looked through the MT code to prove that the algorithm they use has this problem, that's just my theory. Some sites, even older ones, haven't tipped, and may not ever tip. I don't have enough data to nail down the cause. All I know is that SOMETIMES mt goes nuts, and then even moderate comment traffic can take down a server. I usually call these comment spam attacks, but of course it can happen even with normal commenting by real users.

Six Apart is not very cooperative

Someone made the suggestion of working with Six Apart to solve this problem. This is troublesome. If you look at their license, they are somewhat hostile to any kind of outside commercial support. For example, a host is not permitted to preinstall MT for users (only Six Apart can do that) and of course they are actively competing against hosting companies with Typepad. They're aware that MT has this problem. I really can't offer any explanation as to why it still happens in even the latest versions. Either they want to fix it but don't know how, or they don't want to fix it. And since MT is proprietary software, nobody else can (legally) fix it either.

But I really don't hate MT

MT causes me more problems than just about anything else, but I don't really hate it or want to ban it. For the most part, the problem is manageable. I much prefer it when people use other tools, but I understand that people like MT and want to use it and for the most part I'm happy to host MT sites, as long as I can keep the server running.

MT speedups and workarounds

There are some ways to improve MT performance. They have a paper up here:

http://www.sixapart.com/pronet/articles/how_to_speed_up.html

I'm not convinced these solutions address the core problem (the algorithm) but they can help with the symptoms.

Enable background tasks

Some people have told me that enabling background tasks (see the "how to speed up" link above) prevents their sites from running up against the 30 second timeout. I suspect that while the pages appear to run faster, the background tasks may still pile up. However, I haven't had a chance to really look into this yet. In any case, it does seem to improve user experience and should prevent duplicate comment attempts due to timeouts.

FastCGI?

Someone asked about FastCGI.

FastCGI keeps the mt process in memory deliberately, so as to avoid launching a new copy of perl with each page request. If the core problem is (as I am suggesting) an exponential-time algorithm, then shaving a few milli-seconds off the startup for perl wouldn't help much. It can only offer a small, linear improvement, and again the servers are very fast to begin with.

However, I am considering offering FastCGI for other reasons people who use certain web frameworks keep asking for it, so I may offer this at some point in the future, as an add-on service or new account type.

The 30 Second Rule

Let me explain the 30 second rule.

The way it works is there's a script called vengeance that sits in memory, continually monitoring all the processes running on the machine. It knows the difference between a program you're running through the shell (pine or emacs or vim) and one that's being run from cron or apache.

If a script is older than about 15 seconds or so, vengeance will renice it, which means give it a lower priority in the operating system. Then at the 30 second point, the process is simply killed.

For a long time, I turned off the "kill" part. Why? Because so many people complained that MT was getting killed. So for a time, all vengeance did was renice processes. Unfortunately, this just gives you a lot of low-priority scripts that still hog the server. I now whitelist mt.cgi itself, but the mt-comments and mt-trackback scripts still get killed.

The thirty-second rule isn't a cureall (plenty of other things can go wrong) but the servers run a whole lot better, for longer, when it's in place.

Is the rule too strict?

I chose the number 30 because that's what pair.com used to do back when I was a customer there, and pair's servers run pretty well. But the number is just an arbitrary cutoff point.

In general, it just shouldn't take 30 seconds for a high speed modern computer to put a comment on a blog.

Of course there are other scripts that time out, and some of these are harmless. Vengeance doesn't consider the actual resources used by the script.

Nor does it consider the environment. The degree to which a script is tolerated should vary depending on the server load, and other factors. If the server is really busy, it makes sense to shorten the timeout, and if the server is calm, then things can be allowed to run longer.

There's a field of technology called fuzzy logic that's very good at problems like this, and I'm planning to incorporate a simple fuzzy inference engine into a new replacement for vengeance. This is actually a lot easier than it sounds, and it should make the system much more lenient while still keeping the servers stable.

blacklisting spammers

Another approach that strikes at the symptom, but which could benefit a whole lot of people, is a blacklist for comment spammers. I've been hesitant to do this in the past because it's way too easy for the blacklist to be wrong. However, a good blacklist would certainly ward off comment spam, which would ease the burden of the slow MT sites. I haven't made up my mind about this yet, and am very open to feedback.

better analysis of incidents

One problem with vengeance is that it doesn't notify people that their scripts have been killed. It's actually pretty easy to identify a script's owner, even when it's running through apache without SuExec. I consider it a bug that nobody is notified when a script is killed. The data just goes into a logfile, which I look at occasionally, but the information could definitely be put to better use.

The only real obstacle is that I don't want to send someone 500 emails if one of their sites goes nuts. It makes more sense to filter this information automatically and tie it into some kind of ticket system. It's simply a matter of having the time and resources to make it happen.

and finally: rantelope

Even though I'm happy with diversity and will continue to let people host whatever software they want, I also know that I'd have a very strong competetive advantage if there was a tool that I knew inside and out. I'd be able to offer much more in-depth support, even down to the level of optimizing and debugging the code, and doing the initial installation and setup work.

Once upon a time, I decided to write a blogging tool called rantelope for just these reasons, but the project never got very far. It's been on the back burner ever since, and is certainly not ready for prime time.

However, lately I've been thinking more and more of the idea of building rantelope by porting and extending an existing tool (probably wordpress or drupal). This would give me the chance to understand each part of the system without having to create it from whole cloth.

coming soon...

All of this is a lot of work, so it's going to take a while to make it happen. I'm still working out what's coming first, and how all this balances with the many other projects on my plate. In the meantime, at least for a while, the 30 second rule is staying put.