cornerblog

migration issues

2011-05-07T15:51:00.012-04:00

Here's the list of issues I'm aware of. (Marking these off as I go along)

~~control panel not working~~
~~odd mailbox permissions on argon~~
~~database connection problems from argon~~
~~out of memory errors with php~~
~~perl missing ImageMagick libraries~~
~~broken link to phpmyadmin~~
~~suexec problems on argon~~
~~SpamAssassin not running~~
~~bad imaps/pop3s certificates~~

Update: If you use an IMAP mail client and suddenly can't see your folders: get rid of the ~/mail/ prefix in your server settings. The new IMAP server uses this by default.

Update: If you have previously compiled any custom binaries on the old servers, you may need to recompile them. The new boxes are running a 64-bit version of linux so your 32-bit programs will need to be recompiled.

Update: the ssh fingerprints for the new boxes are at:

https://secure.sabren.com/cornerhost/prints.txt

Update: SquirrelMail settings and preferences were lost. This actually happened a couple days ago. My merging script accidentally wiped out everything on secure.sabren.com (the control panel, the main cornerhost site, etc). I restored most of it from backups, but the SquirrelMail settings are stored in an odd corner of the file system that wasn't part of my backup script. Nobody's mail got lost, but any settings configured inside squirrelmail are gone for good. The main casuality seems to be address books. Sorry. :/

migration complete

2011-05-07T06:11:00.002-04:00

All files have been moved over and everything is up and running.

Please let me know if you run into any problems.

migration underway

2011-05-07T03:36:00.002-04:00

All five old servers are now down so I can migrate last minute file changes.

cutover starting 3:00AM EST

2011-05-06T23:53:00.003-04:00

I'll be switching everything over to the new data center between 3 and 5 AM tonight, Eastern time.

The servers will be down while I do a final sync of updated files, though I don't expect them to be offline for the full two hours.

As a user, you shouldn't need to do anything, unless you use your own external DNS. In that case, you need to update to the new IP addresses:

   # old mercury and manganese users:
   neon.sabren.com (50.28.14.224)

   # old scandium, titanium, and vanadium users:
   argon.sabren.com (50.28.14.226)

Data Center Move in Progress

2011-05-06T17:46:00.003-04:00

The move to the new data center is happening this weekend.

Your server name will change, though I will be leaving the old names around as aliases for the time being:

old box:	new box:
mercury	neon
scandium	argon
titanium	argon
vanadium	argon
manganese	neon

I'm doing my best to make the transition as painless and transparent as possible, though there will be some short periods of downtime as I move each server.

I'll post more as things progress over the weekend.

db issues resolved

2011-05-06T17:44:00.003-04:00

The database appears to be working normally again.

Both data centers denied there was any throttling going on, and as far as I could tell the transfers were not saturating the network connection, but in any case, the transfers are complete and the problem appears to be resolved.

database issues today

2011-05-06T11:20:00.003-04:00

A number of users began reporting database issues yesterday, which have been getting progressively worse.

My best guess at the moment is that there is some kind of network throttling going on, triggered when I began transferring everyone's files over to the new boxes.

I've got tickets in at both data centers to try and resolve the issue.

control panel / db / etc upgraded

2011-04-08T15:39:00.004-04:00

I've moved the database / control panel server to the new data center. (You may already have noticed a speedup in your mysql powered pages - this thing is fast!)

I decided to upgrade some of the software pieces along the way, and some apps, like the old wiki and the innercircle list broke in the process. These should be restored (or replaced) by Monday.

I've postponed migration of the other servers for a bit. This is going to happen sometime in the next month, but I wanted to give people a chance to test their sites on the new boxes before they goes live.

The new boxes have updated versions of apache, php (5.2.16), python (2.7.1) and ruby (1.9.2), and unless there's a huge demand, I'm going to drop support for all previous versions (except python 2.4, on which the operating system relies).

I currently have a considerable backlog of support requests. If you have an urgent issue that I haven't replied to, please re-send it to me at [michal at sabren dot com] with the word "urgent" in the subject.

More later...

Server Consolidation / Upgrade

2011-04-05T13:43:00.003-04:00

Just a heads up that I'm moving all cornerhost accounts to a new data center this week.

I'm merging some of the servers together, so your server name and IP address are going to change, although I will keep the old names around as aliases for the time being.

I will be syncing files on the new and old machines until the moment of transition, so downtime should be limited to a few minutes.

The new boxes have twice the RAM of the current machines, far more hard drive space, and six CPU cores compared to the four we have now.

I'll post more information (including the new names and IP addresses) once I have it.

mercury down

2011-02-22T13:44:00.002-05:00

Mercury is suffering from a pretty heavy load at the moment. If I can't get it working in the next 15 minutes or so, I'm going to reboot.

outage at the data center

2010-05-03T01:19:00.002-04:00

I'm unable to reach any of the servers except manganese. The data center says:

We appear to be experiencing a network problem at this time. We are working with Network Operations to identify and resolve the problem. Please stand by for updates.

Hopefully this will be resolved shortly.

manganese php virus

2010-02-03T12:30:00.002-05:00

Several customers have reported that their sites were infected with a virus that was adding an iframe to their pages.

Numerous sites on manganese were compromised.

I've got a program running to remove every instance of the iframe, and I'm working to find and close the security hole that let this happen.

Outage

2010-01-14T14:58:00.002-05:00

Due to a cashflow issue, the data center was unable to bill the usual company credit card, and has disconnected all our servers. I've gone in and paid with some alternate credit cards, and service should be restored within the hour.

mercury offline this morning

2009-09-11T08:48:00.003-04:00

The data center has taken mercury offline due to some sort of attack this morning. They said:

Your server is experiencing a large inbound attack. We placed your server under guard protection, however the attack exceeded the guard threshold and we were forced to null route your IP. The null route will remain in effect until traffic has reached levels manageable by guard.

I'll post more when I know more.

mercury back online

2009-05-19T18:04:00.003-04:00

Mercury is back online and the misbehaving firewall has been disabled.

Sorry, everyone. Needless to say, that security group won't be messing with my servers again.

Still waiting...

2009-05-19T16:51:00.002-04:00

The firewall does seem to be out of control, blocking everything. This issue has been passed to a level 2 tech at the data center...

Mercury down... (maybe?)

2009-05-19T15:54:00.000-04:00

I signed up for a security audit and ongoing monitoring on mercury... It appears they've installed some kind of overly paranoid monitoring system that is blocking legitimate customers at the firewall level.

I seemed to have triggered this myself and can't even tell if the server is up anymore.

I have a ticket open with the planet and we're getting this sorted out ASAP.

mercury restored

2009-03-25T01:28:00.002-04:00

Mercury is back online. I've updated the mod_security rules and enabled suhosin, which adds an extra layer of security.

One thing suhosin does is completely remove the possibility of remote includes for php. So if you have an include() call that starts with "http://" it will no longer work. What you need to do is put the include file on your account and replace the "http" path with the corresponding local file system path.

mercury taken offline

2009-03-25T00:34:00.002-04:00

Apparently there was a security breach on Mercury, and the data center was unable to contact me. They disabled the server several hours ago.

I am working to resolve this issue right now.

mercury down

2009-03-13T16:38:00.002-04:00

Mercury just stopped responding and I'm unable to bring up a remote terminal.

I've opened a ticket for the data center to investigate.

update: it's back online now. Was down for about 8 minutes.

what to do if your old PHP script broke

2009-03-10T12:29:00.001-04:00

Hi guys,

The number 1 problem cropping up for people whose PHP scripts have stopped working correctly since the upgrade is that they rely on the old, insecure way that PHP handled variables passed in from the query string in the URL.

This old behavior is one of the major causes of security holes in PHP. Basically, if the program is not well, written, it may be possible for an attacker to change variables that were not meant to be passed in through the URL, and possibly take control of your site.

Although this only affects a very small percentage poorly written scripts, the PHP community has come up with a much smarter alternative, which is to move these variables into special associated array variables like $_GET, $_POST, and $_REQUEST.

So, if your script depends on a variable named $foo being passed in through the URL, like so...

http://whatever/script.php?foo=bar

... Then you should put this line at the top of your script:


  $foo = $_GET['foo'];      // read from query string

Similarly, you can read from a form post with:


  $foo = $_POST['foo'];     // read from form post

If you prefer to accept both GET and POST requests, you can use this line, which handles either method:


  $foo = $_REQUEST['foo'];  // read from either

Add one of these lines to the top of the file for each variable in your form. (Make sure you put it INSIDE the <?php ... ?> area!)

Having said that, I realize this is short notice, so as an alternative, you can revert to the old behavior for your entire site by adding this line to your .htaccess file:


   php_flag register_globals on

However, at some point in the future, I will probably disable this option completely... So if you don't feel comfortable editing your php scripts yourself, please get in touch with me and ask for help.

For more information on this PHP issue, see here:

http://us2.php.net/register_globals

Thanks!

mail to comcast to be fixed shortly

2009-03-07T14:56:00.001-05:00

It seems comcast is rejecting mail from the new machines until we get reverse dns records (PTR records) set up. I've put a request in to the data center so hopefully this should be fixed this weekend.

SMTP and control panel login fixed

2009-03-06T10:02:00.004-05:00

The SMTP server is working correctly now on all boxes.

(For the curious, the problem was that even though SASL was installed and configured correctly, saslauthd wasn't actually running by default anymore.)

The control panel is also working correctly now. (Though the logfile data is still missing).

post upgrade issues, revised

2009-03-06T09:01:00.003-05:00

fixed:

logfiles have been restored
SSL for mail is now working correctly

in progress

control panel is rejecting logins
SMTP seems to be rejecting logins for some people

(These are two unrelated issues as far as I can tell. The control panel will be fixed shortly. SMTP is still a mystery.)

new ip addresses

2009-03-05T13:24:00.003-05:00

If anyone needs the new IP addresses, they are:

mercury.sabren.com.     IN      A       75.125.245.26
scandium.sabren.com.    IN      A       75.125.250.26
titanium.sabren.com.    IN      A       174.133.41.42
vanadium.sabren.com.    IN      A       174.133.41.90
manganese.sabren.com.   IN      A       174.133.33.154

control panel working

2009-03-05T13:19:00.000-05:00

The control panel should be working again (except for logfile reports).

all servers online

2009-03-05T10:44:00.002-05:00

All servers are now back online.

Still working on:

SSL for mail
old logfiles
getting the control panel hooked up

manganese fully functional

2009-03-05T09:36:00.002-05:00

Mail for manganese (and all servers except titanium) is back online.

Titanium is still chugging along...

manganese and titanium partially restored

2009-03-05T08:50:00.001-05:00

I've turned on the web server for manganese and titanium.

I'm still waiting on mail to finish syncing before people can log back in. Looks like it's getting near the end on both machines.

known issues

2009-03-05T07:07:00.004-05:00

Known issues for anyone looking at the new servers:

SSL isn't quite working yet for mail. (but you can use the webmail interface
Your web logfiles may or may not be there yet.
Control panel changes won't take effect immediately (if you can log in at all).

All of this should be fixed shortly.

scandium and vanadium online

2009-03-05T07:06:00.002-05:00

Scandium and Vanadium are back online.

Titanium and Manganese look like they're pretty close.

mercury online

2009-03-05T06:31:00.003-05:00

Mercury is back online.

The DNS change appears to have gone through as well. Please let me know if you're having any issues.

The other servers are still waiting on rsync. I'll bring them online as they finish. Hopefully it shouldn't be more than another hour or two.

final sync in progress

2009-03-05T04:10:00.002-05:00

The final rsync is in progress. All accounts are locked to prevent file changes during this time (including web,mail,and ftp access).

This will take a couple hours. Speed varies depending on the server.

I am also initiating the DNS changes for all domains.

down for the upgrade

2009-03-05T03:39:00.000-05:00

It's about 3:30AM eastern time, and I'm taking the servers down for maintenance now.

reminder: upgrades tonight

2009-03-04T11:20:00.001-05:00

Just a reminder that the servers will be down for several hours late tonight while I do the final rsync and switch us over to the new boxes.

Hydrogen is already switched over so I'll be posting updates on cornerhost.net as things develop.

innercircle downtime

2009-03-03T08:54:00.002-05:00

~~The innercircle list is down while I figure out how to get mailman to recognize my old data...~~

Never mind. It's back. :)

don't use www for testing

2009-03-02T08:54:00.002-05:00

Just a quick note: the www prefix isn't set up for the test domains. Just use http://yourdomain.com.new-xx.sabren.com/whatever

Test your site on the new servers!

2009-02-28T22:06:00.002-05:00

The new boxes are almost ready.

There are a few things left to install, but most sites should now be operational.

My plan is to make the cutover later this week (probably Wednesday night). In the meantime, please test out your site and help me identify any remaining issues.

To see your site on the new server, append new-xx.sabren.com to your domain, where xx is the short name of the server:

if you're on...	... then add this:
mercury	.new-hg.sabren.com
scandium	.new-sc.sabren.com
titanium	.new-ti.sabren.com
vanadium	.new-v.sabren.com
manganese	.new-mn.sabren.com

So, for example, if your site is:

http://tempytantrum.com/ (on mercury)

You should go to:

http://tempytantrum.com.new-hg.sabren.com/

I've tested a variety of the major software packages (wordpress, drupal, MT, etc) and they all appear to be working, but of course I haven't tested every possible configuration or plugin, so please take this opportunity to check your site out before the switch.

Known issues:

The files on the new boxes are a day or two old. I'll refresh them tonight or tomorrow, and again right before the switch.
The new servers will probably not send not send mail correctly to your cornerhost accounts (they'll deliver locally).
You cannot log into the server with ftp/ssh yet (to keep people from making changes that will get overwritten with the next rsync). If you need to log in, please contact me and I'll give you a temporary password.
python2.2 is no longer available. If your site uses python2.2, please try switching to /usr/bin/python2.5 or /usr/bin/python2.4 (test this on the *current* box)
FastCGI sites are not working yet.

Feel free to contact me directly with any issues you turn up, or post on innercircle to avoid duplicate bug reports.

dropping php 4 (?)

2009-02-26T06:21:00.003-05:00

Can anyone give me a reason to *not* completely drop PHP4 ?

I was planning to keep both side by side, but I'm thinking about making PHP5 the one and only option.

I'm going to have test domains up very shortly and you'll have a couple days to test out your site on the new machines before the cutover.

In the meantime, I'm just curious if anyone is already aware of a specific (non-hypothetical) reason that you require PHP4.

ns1.sabren.com migrated to new boxes

2009-02-18T23:53:00.004-05:00

I've moved the primary nameserver (ns1.sabren.com) over to the new box.

Secondary nameserver (ns2.sabren.com) is on a separate group of machines and will not be updated.

I will continue to run the current ns1 machine for the next 48 hours or so while the top-level DNS change goes through.

This update should not affect customer accounts in any way.

db.sabren.com rebooted

2009-02-18T22:11:00.001-05:00

I had to reboot the server for a quick configuration change (updating the hostname).

MySQL clock adjusted to EST

2009-02-18T18:17:00.001-05:00

The clock on the new server was set to Central time by default.

I just adjusted to Eastern time, per the Cornerhost standard.

MySQL 5.0 Now Running

2009-02-10T05:29:00.004-05:00

Thanks for your patience, everyone.

We are now running MySQL 5.0 on the new hardware.

I've been spot checking customer sites at random and everything seems to be working. Please let me know if you run into any trouble.

One minor issue for people who access the database from outside of the cornerhost network: I have not updated the DNS entry for db.sabren.com, and don't intend to do so until I have apache working on the new boxes. If you need direct remote access to the server for some reason, you can use the IP address directly: 174.133.41.82

MySQL down for upgrade (finally)

2009-02-10T03:03:00.001-05:00

MySQL is down for the upgrade (finally). It will be back up in a few hours.

Upgrade Still Going On

2009-02-09T08:25:00.002-05:00

8:30 am and the transfer of database files is still going. I worked out the issue with rsync, and now that it's copying all 600+ databases it's taking much, much longer. It ran all night and will probably continue for another couple hours. So... I'll wait until tonight to take it down for the final switchover.

Upgrade Postponed to Sunday Night

2009-02-08T05:01:00.004-05:00

Well, somehow I managed to screw up a simple rsync.

Once all 600+ databases finished copying, there were only 108 databases on the new box.

The old server is up and running again. I'm rerunning the rsync and will attempt the switchover again tomorrow night.

Reminder: DB Upgrade Tonight

2009-02-07T23:59:00.002-05:00

Just a quick reminder that the database will be down tonight for the MySQL upgrade.

There's already a copy of all databases on the new machine... I'm updating the copy right now (looks like it'll take another hour or two), and then will shut down the current box for one final up-to-the-minute update before running the MySQL upgrade routine and bringing the new server live.

Database service will be fully restored by morning.

MySQL upgrade this Saturday Night

2009-02-03T12:41:00.001-05:00

I will bring the new database server live this Saturday night, probably around 1am EST (so really very early Sunday morning.)

In addition to the new hardware, I'm upgrading MySQL itself from 4.1 to 5.0.

(If enough people are interested, I'll schedule a further upgrade to 5.1 later on...)

There are some minor backward incompatibilities between 4.1 and 5.0 ... I don't expect these to affect anyone here, but if you're using a lot of hand-coded SQL, you might want to look over the list of changes between 4.1 and 5.0

Upgrade Specs

2009-01-30T13:29:00.002-05:00

Here are the hardware specs for the old and new boxes:

Component	Old	New
CPU	Dual Proc. Single Core Xeon 2.8	Dual Proc. Dual Core Xeon 5130
Hard Drive	2 x 146GB IDE (72,00 RPM)	2 x 146GB SAS (10,000 RPM)
RAM	2-3 GB	4GB
OS	RHEL 3	CentOS 5

upgrades coming

2009-01-29T19:42:00.002-05:00

There have been a lot of complaints about the database server and manganese lately. I'm just writing this to let you know I hear you. :)

I've opened a ticket with the Planet about getting some much-needed upgrades.

Rather than just upgrading the existing boxes, my plan is to purchase completely new machines with better processors, faster hard drives, and a more modern operating system (CentOS 5).

I also plan to make the transition to MySQL 5.1 and the latest PHP5 during this upgrade. I also plan to drop older python versions in favor of 2.5, 2.6, and 3.0 ...

(If necessary, I may keep one of the old boxes around temporarily for people whose sites are dependent on the older technologies. Please let me know if you know your site is incompatible with these changes.)

I'll let you know more detail on the specs and timeline once I work out the deal with the planet.

duplicate mail on manganese

2009-01-20T17:06:00.002-05:00

Several users on manganese have complained about duplicate incoming emails.

My best guess is that there is a bottleneck on procmail and spamassassin causing requests for these tools to pile up.

I think I have a permanent solution to this problem, but it will take some time to develop. As a temporary workaround, I've re-enabled the mail queue for manganese, and will just manage the queues by hand to avoid backlogs.

working on vanadium

2008-11-07T13:45:00.003-05:00

Vanadium's been running pretty slow lately. It doesn't seem to be any one specific thing. It just looks like the hard drive has more work than it can handle.

I'm going to offload mail processing to the secondary hard drive. It may take a few hours. I'll do what I can to keep things as responsive as possible manually in the meantime.

mercury fixed

2008-09-07T16:52:00.000-04:00

Whoops. No attack. My fault. Fixed.

The problem spread to vanadium briefly, too.

I changed the top #! line in all my admin scripts to use /bin/env (so that I could standardize on one python executable.)

Turns out Apache doesn't supply any environment variables to log handlers, and refuses to serve content if the logger doesn't run.

Sorry for the trouble.

Update

2008-09-07T16:38:00.000-04:00

Not an attack. problem with logging system. Should be fixed momentarily.

From Michal

2008-09-07T16:26:00.000-04:00

It's also affecting manganese and appears to be an attack.

apache outage on mercury

2008-09-07T16:10:00.002-04:00

I got an alert that apache is down on mercury.

Load is normal. Apache is running, but not responsive. Investigating.

twisted/python help wanted

2008-08-26T16:17:00.001-04:00

In case anyone's interested, I've posted a small programming job up at oDesk:

http://www.odesk.com/jobs/Python-Twisted-Developer_~~3525773673aad59e

This is a work at home on your own schedule thing. The mission is to get vengeance (the system monitor here at cornerhost) working with twisted.

(I've been using oDesk for overseas contractors. I'm open to private bids if you're in the US. Price range varies depending on your experience. For anything over $20/hr I'd prefer to work out a fixed price. Feel free to contact me with questions.)

server response times and new issue tracker

2008-08-15T11:37:00.003-04:00

Someone asked on the innercircle list about load issues on vanadium. This was my response. I thought I'd post it here, too.

With so much happening on the servers, questions like this are kind of hard to answer. I generally have a hunch what the problem is when something goes wrong, and on vanadium I have my suspicions about one site in particular.

Short answer: I'll play around with this site and see if I can bring it in line. I added an issue for this here:

https://secure.sabren.com/roundup/issue11

Long answer: To really handle the general case, we need to be able to separate special cause problems ("comment spammers are sending 10 million spams a second and crippling the server") vs general problems (average traffic on all sites has grown and high loads have become the norm).

It seems like it would be easy to tell the difference, but it really isn't. What we really need is a statistical model that shows how each server behaves in general, not just at one particular moment (during a comment spam attack or whatever).

I've been reading a lot of quality and statistics books (six sigma, etc) and trying to wrap my head around this stuff.

From now on, I'm treating server performance as mission critical (right up there with support response time). Things are a long way from perfect on both ends, but I'm working on it.

BTW, if you didn't notice that URL, I've set up a new issue system based on roundup. :)

You guys can log in with your cornerhost accounts and make comments or sign up for email updates on issues by adding your name to the "nosy" list. This is a work in progress and only for "public" issues right now, but eventually it'll handle support too. The top level link is here:

https://secure.sabren.com/roundup/

I also wrote a post about the tracker over on my blog.

15 Lifetime Accounts Available

2008-08-11T19:40:00.003-04:00

I'm doing a very brief lifetime hosting offer.

It's open to anyone, but there are only 15 accounts available.

Price is $475 USD.

See here:

http://cornerhost.com/lifetime/

quick reboot of all boxes at 1AM EST

2008-07-16T23:28:00.001-04:00

I'm rebooting all machines in 90 mins (around 1AM EST) minutes for a linux kernel upgrade. Shouldn't take more than a minute or two.

working on mercury (again)

2008-07-16T16:42:00.002-04:00

Despite my best efforts, Mercury is still getting slammed with extremely high loads.

I've hired a part time Linux consultant to help. His name is Harry Roberts [http://codeb.us/] and he's already hard at work.

In fact, he quickly discovered that SpamAssassin is basically overloading the hard drive with all the IO work it does updating its Bayesian filters.

As an immediate fix, he's working on moving all the .spamassassin directories on mercury to the second hard drive. We're also looking into a centralized Bayesian database for users who don't train their filters individually.

Hopefully this should lead to some major performance improvements sometime today.

Thanks for your patience.

working on mercury

2008-07-14T15:32:00.004-04:00

Mercury is experiencing extremely heavy loads, causing delays with both web and mail. I'm aware of the problem and am working on it.

update The problems were sendmail related and seem to have been caused by several bad forwarding addresses. I tracked these down and fixed them, and Mercury is once again running smoothly.

mod_security rules updated

2008-07-04T23:01:00.003-04:00

FYI, I updated the mod_security rules on all machines, and also customized the 406 error page to show why a rule got triggered and how to disable just that rule. (example)

New Policies for Catchall Rules

2008-06-23T12:15:00.002-04:00

Catchall Forwarding
No longer Allowed At Cornerhost

In order to help prevent cornerhost's servers from blacklisting by ISPs and other organizations, I'm no longer allowing forwarding of catchall addresses.

Catchalls are the rules that tell the email server what to do when someone sends mail to an arbitrary address that doesn't have an explicit rule of its own.

Spammers will often try out random addresses at your domain. Addresses that don't bounce will wind up on their lists.

Sadly, many ISPs make no distinction between forwarding spam and sending spam, and catchall forwarding results in frequent blacklisting of the cornerhost servers, so it is no longer allowed.

You may still create rules to forward specific addresses. The new policy only deals with catchalls.

Existing Catchall Forwards
Will Expire On July 7

I've already updated the control panel to reflect the new policy.

Any catchall forwards left on the morning of July 7 will be disabled and redirected to your main cornerhost mailbox.

Please log in at http://cornerhost.net/ as soon as possible and update your rules.

Even if you don't forward your catchall rule, you should still consider turning it off and only accepting mail for addresses you actually use. This will almost certainly cut down on the amount of spam you receive.

The Default Catchall Rule
Is Now "Bounce"

I have also changed the default catchall rule from "go into your main cornerhost mailbox" to "bounce".

This it to cut down on the amount of spam you receive.

From now on, when you add a new domain (or set up a new account) you will have to configure your mail rules before you start receiving mail. (I've added a reminder to the "domain added" message in the control panel.)

Again, this part only affects new domains.

An Alternative for Gmail/Yahoo

If you happen to be forwarding all your mail to Gmail or Yahoo, a better alternative is to store the mail here, and then have them check the box. Instructions are here:

CheckMailFromGmail
CheckMailFromYahoo

(As far as I can tell, this isn't possible with Hotmail. Sorry.)

As always, if you have any questions or feedback, please let me know.

outage

2008-05-30T14:03:00.006-04:00

2:05pm EST It seems all our machines just dropped off the net at once. I'm guessing this is some kind of data center issue. I'm investigating now.

2:15 EST - My best guess is that the culprit is a MySQL overload. The MySQL server is unresponsive, and my auto-reboot attempt failed. Currently waiting on the data center to restart

The other boxes seem to be working, except of course for MySQL connections.

2:20 EST - Apparently it's not MySQL but an internal networking problem at the data center. They said:

Our network engineers have identified an issue with our network currently trying to get it resolved. Your patience is appreciated in this matter.

Hopefully, this should be back to normal soon.

3:10pm Still out. I just talked to a customer support guy who confirmed they're still working on the issue. This issue is affecting the entire data center, not just the cornerhost servers. Unfortunately, there's nothing we can do but wait for them to solve the problem. :/

4:30pm The issue appears to have been resolved.

call for php5 module requests

2008-05-25T12:11:00.001-04:00

I've been asked to enable PDO_MySQL for PHP5, so I'm planning to build another RPM in the not too distant future.

If there are any other modules you need included or just want to try out, please let me know ASAP. (I won't promise to include everything, but I'll do my best)

PHP5 upgraded to 5.2.5

2008-04-18T13:00:00.001-04:00

I've updated PHP5 to version 5.2.5.

I've also enabled the mcrypt, mbstrings, jpeg, and png extensions.

If you're not using php5 and would like to try it, you can add this line to your .htaccess file (script and shell plans only):

AddHandler php5 .php

top 5 priorities

2008-03-23T23:36:00.002-04:00

I haven't done a status report in a while. Here are my top 5 current priorities:

Automated Mail Queue Management

During the recent mail queue crisis on manganese, I wrote a script to help manage the sendmail queues. I promised to get this running around the clock to prevent further issues, but I haven't finished yet. I've just been running it manually so I can keep an eye on things. Getting this 100% automated is my top priority.

Mod Security Improvements

I've had a number of complaints about 406 errors cropping up when they shouldn't - the result of over-eager mod_security rules. The latest versions of mod_security allow blocking individual rules at the .htaccess level, but will require some changes to the server configuration.

In particular, any "SecFilterEngine off" lines in your .htaccess files will have to be replaced, so I need to coordinate the upgrades to make sure nobody's site starts throwing a 500 error.

I also want to get you the audit logs for your individual sites so you can see exactly what's being blocked.

Catchall Rule Changes

Currently, new domains have their email rule set to act as a catchall. This used to be pretty convenient, but these days it leads to people getting a lot of spam. I want to change the default to bounce, and add some new user interface elements in the control panel to help you set up mail rules if you need them.

Also, as I mentioned previously, I want to disable forwarding catchall addresses to outside mail services, as this is a great way to get our servers blacklisted for relaying spam. I've worked up a plan for implementing this over time and making sure everyone has the information they need to make this transition. Expect to hear more on this soon.

Fix all known Control Panel Bugs

I've got a list of bugs piling up in the Trac and elsewhere. I want to take a week or two and get these resolved. Since the code is now open source, I plan to write up reports on some of these in a blog or something to start building a body of knowledge on where things are and how to get things done in the code.

Update PHP5

I've got a new build of php5 with mcrypt and mbstring (extensions that several apps, including phpMyAdmin, complained were missing). This is almost ready to deploy. I just need to double check that I haven't dropped anything from the current build, since I had to recreate the rpm build script by hand.

Anyway, those are my top 5. I'll keep you posted as things progress.

the queues look good

2008-03-01T23:41:00.003-05:00

Things seem to be running smoothly again with the mail queues. Pretty much everything has been delivered.

Some people reported getting copies of mail they had already received. I'm not sure why the duplicates were in the queue, but now that things are cleared out, that shouldn't happen anymore.

I was working on a queue management script as I went along, and have dramatically improved my ability to deal with this kind of problem in the future. However, I'd much rather prevent it from happening in the first place, and I think one of the most important steps I can take here is to change the policy of enabling catchall rules by default, and very likely preventing the catchall addresses from being forwarded. I'm still working out the details on that, but expect to hear more soon.

mail queue triage

2008-02-29T01:17:00.000-05:00

The queue size and server load on manganese are under control, but I am still struggling with the massive backlog of undelivered mail.

There are several broad categories of mail to deal with, in order of importance:

mail that goes to a local mailbox that is checked on a regular basis.
mail that is forwarded to external mailboxes
mail that goes to a local mailbox that is rarely or never checked

The first and third categories (local mail) should be the quickest to process, but unfortunately they are mixed in with mail that needs to be forwarded.

Right now, I'm building a report of "last read" dates for each mailbox so I can separate out users who check their and move local mail for these users to the front of the queue.

The next step will be to sort forwarded mail by destination server, so that I can uncover any specific services that are slow to accept mail and deal with them appropriately.

The idea is for this to be a standard policy going forward, so (for example) if you are currently forwarding your mail to gmail or yahoo, you will get much faster service by having the mail delivered to a local box and having the outside service poll that box through pop3.

manganese queue again

2008-02-28T13:22:00.002-05:00

Well, the mail queue on manganese grew to 6000 entries again overnight.

Things are slow on the server, but I'll have a report within the hour with a "top 20" list of domains with the worst problems.

I can already see at least three accounts that got a ton of spam trapped in the queue from last night, so I'm going to deal with those now while the reporting system does its thing.

mail queue status

2008-02-27T18:41:00.003-05:00

I'm currently breaking mail queues on all machines into smaller queues of 500 to 1000 messages each. This is proving to be a rather long process, and may take several more hours.

Manganese in particular had over 50,000 messages in the queue. Vanadium and mercury both have about 20,000 - though these messages appear to be mostly old spam that has piled up over time.

My plan is to have this queue-splitting process running periodically from now on so that slow mail (such as mail to unreachable hosts) automatically gravitates to the secondary queues and sending can be retried less often. This in turn should speed up delivery of healthy mail since the server has less to do, and also free up CPU resources for everything else.

I'm just letting the queue splitters run for now. There's not much else I can do until they finish.

manganese update

2008-02-27T00:33:00.003-05:00

The mail queue on manganese is still having troubles.

Part of the problem is that as the queue gets large, it takes longer and longer to process it. Sendmail itself has this problem, but it also affects my ability to find, select, move, and delete messages.

Sendmail stores each message as up to three separate files, each of which has to be parsed. Simply finding all messages to a particular domain, or running a report to see what domains are getting the most messages can take over an hour.

Normally, the queues are quite small so this isn't a problem, but when the queue gets big, it can take a long long time to fix things just because everything is so slow.

I'm just as sick of dealing with this as you guys, so I'm updating my queue management script to caches all the data in a sqlite database. This will make it much easier for me to inspect the queue, and dramatically speed up batch commands like creating secondary queues.

As for what's actually causing the problem on manganese, it looks like it might be a combination of rejected mail forwards (people forwarding all their mail to another server) and a possible bottleneck when it comes to spamassassin.

I'm isolating the blocked forwards into separate queues for further analysis as I go along (again this is very slow), and I've also increased the number of spamassasin demons.

The box should be back up to speed by tomorrow. It may be a day or two before the mail clears out in the secondary queues. (I'll let you know if I isolate your mail.)

update on slow manganese mail

2008-02-22T09:52:00.001-05:00

I've been getting reports of slow/undelivered mail on manganese. I'm not convinced any mail is actually being lost. Rather, I think it's just getting trapped in the queue for prolonged periods of time.

For some reason, the mail queue on manganese seems to be clogged up repeatedly lately.

I'm going to see if I can get a separate queue working to isolate these slow messages from normal mail. This should fix the problem permanently.

Meanwhile, I'll just manually chop the queue up and see if I can get these messages delivered.

mod security glitches - RESOLVED

2007-11-27T18:45:00.001-05:00

I just tried to upgrade mod_security to the latest version, and ran into quite a few unexpected problems on the live servers.

I'm rolling back the change now, so if you just started getting 500 errors for no reason, it should be back to normal in a few minutes.

update: everything is back to normal now.

minor control panel update

2007-11-07T19:12:00.001-05:00

I just rolled out a new version of the control panel. You shouldn't notice anything different though - It's mostly xhtml cleanups from migrating the templates over to a new template system called genshi.

This version also fixes a problem where changes to a site's configuration were not being saved correctly.

Let me know if you run into any problems with it.

phpMyAdmin upgrade

2007-11-06T15:23:00.000-05:00

I've upgraded phpMyAdmin to the latest version (2.11.2). It's giving a couple warning messages (which I'll get fixed soon), but otherwise seems to be working a lot better than the old version.

For one thing, it no longer says you can't create tables when you can.

Let me know if you have any problems.

control panel glitches fixed

2007-10-16T10:52:00.000-04:00

The control panel broke for a bit this morning.

My fault. I rolled out an update to the control panel last night, and it looks like the log directory got deleted.

It's fixed now, and I'll update my code to prevent that in the future.

titanium okay

2007-10-13T14:28:00.000-04:00

~~Titanium's been having trouble the past hour or so. The data center attempted a reboot. I'm currently unable to log in myself, but I will get this resolved ASAP and post back here.~~

Update: it came back up just as I posted this. I'm monitoring it closely to make sure it stays up.

manganese unblocked

2007-09-18T12:41:00.001-04:00

FYI, the block on manganese was cleared early this morning. Thanks, Microsoft!

please turn off outgoing catchall rules

2007-09-17T19:41:00.003-04:00

If you have a catchall address set to forward all your mail to an external address, please turn it off, and only forward the addresses you actually need.

Catchall forwarding allows too much spam to go through, and it damages the reputation of the server.

I don't really like doing this, but I don't have a choice here. I'm going to make a more formal announcement of this change soon, and then I'm going to disable this feature completely.

(Again, this is only for catchalls. Forwarding individual addresses is still fine.)

manganese blocked by hotmail

2007-09-17T19:41:00.001-04:00

Manganese is being blocked by hotmail. I've opened a ticket with microsoft, so hopefully this will be corrected soon.

lifetime hosting offer ending soon

2007-09-14T14:44:00.000-04:00

I've decided to withdraw the lifetime hosting offer at the end of this month.

I didn't do a good job of pushing it, and about half the slots are still open.

But not for long.

It's a one time payment of $475 for a shell account that lasts as long as cornerhost stays in business. (Normally, the shell plan costs $200 a year)

The shell plan includes access to the linux shell (of course), and also allows use of one long-running / FastCGI process (for running django, rails, etc).

And of course, you get access to any upgrades that come along in the future.

The offer is limited to the 50 remaining slots, or the end of the month - whichever comes first. I don't have any plans to do this again in the future.

So... If you want in, you can either email me directly to upgrade, or sign up a new account in the usual place:

http://cornerhost.com/signup/

(Select the shell account from the dropdown and just mention in the comments field that you want a lifetime account.)

address change for ns2.sabren.com

2007-08-28T21:02:00.000-04:00

The IP address for the NS2.SABREN.COM nameserver is changing to 70.86.213.2.

This should not make any difference whatsoever to you, but if you notice anything strange going on with DNS, please let me know.

If for some strange reason you're relying on NS2 having a specific IP address... Well, I'd sure like to know why.. :)

please log in from cornerhost.net from now on

2007-08-21T16:24:00.000-04:00

I'm preparing to remove the control panel login box from cornerhost.com. Please update your bookmarks and log in from here instead:

http://cornerhost.net/

I know this will be an inconvenience for some users, and I apologize for the trouble.

The reason for the change is to separate the sales content aimed at potential customers from system content aimed at current customers.

For example, the off-site status blog is mirrored at cornerhost.net, so you can see the latest news and updates every time you log in.

If you don't know your password, but it's stored in firefox, you can recover it with:

  Tools / Options  / Security / Show Passwords...
  Scroll down to http://cornerhost.com/
  Click "show passwords"

As far as I know, you can't recover passwords from IE ~~or Safari~~, so for the time being you can still use the old login form.

If you don't know or can't find your password, please contact me so I can reset it for you.

Update for Safari Users:

Nathan Vannahme pointed out that: Safari on the Mac uses the system Keychain, so you can view its stored passwords with the Keychain Access app.

mysql was down for about 10 minutes

2007-08-11T02:23:00.001-04:00

Well, the waking Michal up when MySQL goes down idea works. :)

I got paged when MySQL started giving a "too many connections" error around 2:00AM EST. I logged in and got MySQL back up within 5 minutes.

Whatever the cause, it seems to have hit rather suddenly. I have a tool logging the mysql status every minute. There were only a handful of connections until 1:55, and then suddenly no word for nine minutes. So either there's a bug in my logging system or MySQL suddenly started taking forever to connect.

2007-08-11 01:53:02,112:INFO:root:found   4 threads. doing nothing.
2007-08-11 01:54:01,890:INFO:root:found   4 threads. doing nothing.
2007-08-11 01:55:01,580:INFO:root:found   3 threads. doing nothing.
2007-08-11 02:04:02,209:WARN:root:got 'too many connections' error; restarting

It's interesting that the automated attempt to restart failed. I'll have to look into that.

In any case, everything's working fine now. Total downtime seems to have been about 10 minutes.

prototype OpenID server

2007-08-10T20:32:00.000-04:00

Bradley Peters rocks! :)

Following the changes to the python openID example server that he posted to innercircle, I now have a prototype OpenID server that authenticates you based on your cornerhost username and password.

Still *very* experimental, and the URL will definitely change at some point, but you can try it out for yourself here:

https://hydrogen.sabren.com:4430/

help wanted: simple openID server

2007-08-08T00:50:00.000-04:00

I'd like to open Trac access to any cornerhost user, so you guys can post bug reports.

But: I don't want to open it up to the whole world. Several people have asked me about setting up an OpenID server based on the cornerhost logins, and Trac has an OpenID plugin, so authorizing people based on OpenID logins seems like a good way to go.

The problem is I don't know how to make the server.

I made a client just fine : http://withoutane.com/openid/

... But the server example was much more confusing.

There's an example server in the openid python library, which you can see here:

http://www.openidenabled.com/openid/libraries/python

... But it doesn't actually prompt you for a password, and I just haven't had time to connect the dots.

The code for checking a password at cornerhost boils down to one line:

   clerk.match(User, username=usr).getBeaker().ispasswd(usr,pwd)

But for the life of me, I can't figure out how to get that hooked into an OpenID server.

Can somebody help?

Extended code for the login is in BeakerAuth.py.

mock data tutorial for cornerops/duckbill

2007-08-04T00:04:00.000-04:00

And... Another tutorial, showing how to set up mock data for use with cornerops. Includes screenshots of the admin control panel and the billing database I use when I manage your account.

how to set up the cornerhost source code

2007-08-03T19:39:00.001-04:00

There's now a walkthrough for setting up the cornerhost source code. If you're looking to participate, this is the place to start. (I tested it both on my OS X box and a cornerhost shell account, but please let me know if there's anything confusing/missing/etc.)

cornerops trac and subversion access

2007-07-31T10:52:00.000-04:00

You can now download the code for the cornerhost billing system and control panel!

And... Thanks to the work of Chris Braybrook, we have a nice shiny Trac installation for browsing the source, tracking bugs, and posting documentation:

https://secure.sabren.com/trac/cornerops/

(Granted, the documentation part is a bit lacking at the moment, so this is mostly for the adventurous and the early adopters.)

More to come: I've held off posting vengeance and the other admin tools until I get the code cleaned up.

Anyway, check it out!

By the way, the Trac install is read-only without a password, but it's open to everybody: if you want access, just drop me an email.

Incident Report for 2007-07-30

2007-07-30T19:06:00.000-04:00

What went wrong?
7 Hour MySQL Outage

What was the fix?
I manually killed every MySQL process and restarted the service.
MySQL came back up immediately.

What initially caused the problem?
Unclear. MySQL has locked up like this a few times in the past.
The data center rebooted Vanadium 30 mins prior to first report of the problem.
I don't know whether that was related.

Why did it take so long to fix?
I was asleep. I was up all night working on the new cornerhost subversion repository.
I went to bed late this morning, and woke up around 4pm to discover the problem.

Shouldn't Vengeance have Solved this Automatically?
Yes. Vengeance is supposed to restart mysql if it goes down.
It tried repeatedly to issue the command: '/etc/init.d/mysqld restart' but this command failed.

Shouldn't someone or something have woken me up?
Yes. At the very least, I should have gotten an SMS note on my cell phone.

Could the Data Center have Solved this?
Ideally. They monitor quite a few services for me, but MySQL doesn't appear to be
among the services they monitor. I just opened a ticket with them to see if they can
provide a solution.

Could anyone else have solved this?
Yes, there are one or two people who have emergency access to kill services,
but they are not trained system administrators.

Could the initial problem have been prevented?
I don't know. I don't know how to duplicate the problem.

What steps will be taken to prevent this in the future?
1. Today: I have opened a ticket inquiring about MySQL monitoring from the data center.
2. This Week: I will install Nagios to add a second layer of monitoring, and SMS/IM me if a problem occurs.
3. This Week: I will fix vengeance to allow escalating approaches to repairing services.

cornerhost going open source

2007-07-19T20:41:00.000-04:00

I just posted a long piece on my blog about taking cornerhost (the control panel, billing system, etc, open source).

I know a lot of you are pretty active in the open source community, or have taken an interest in helping out with cornerhost in the past.

Anyway, I'm still in the planning stages right now, and could really use some advice and feedback.

http://withoutane.com/rants/2007/07/opening-up

py2.5 mysqldb now uses mxDateTime

2007-04-24T12:02:00.000-04:00

I've installed mxBASE for python2.5

As per the standard DB API 2.0, MySQLdb will now correctly return mxDateTime objects for date and time columns. If you have scripts that work with date and time columns, you should double check that they work with these objects.

another new feature: drop database

2007-04-20T09:45:00.001-04:00

If you're still in the mood to delete things, you can now also drop databases from the cornerhost control panel.

Simply click 'edit databases', then click 'delete' next to the database you want to drop, and confirm the drop on the next page.

new feature: delete mailbox

2007-04-20T07:18:00.001-04:00

I've added a new feature to the control panel.

If you click on "edit mailboxes", you can now delete old mailboxes that you are no longer using.

Also, this page now shows what addresses map to each box (like the edit rule page, but grouped by mailbox rather than domain).

Have fun deleteing all those pop_user_xxx boxes. :)

dns list redesign in control panel

2007-04-16T12:55:00.001-04:00

I just published a new layout for the DNS page in the control panel, which shows all the record information explicitly, as opposed to a simplified view that hid the information.

It's a lot more cluttered than it used to be, but I think it's actually more usable.

What do you think? I could use some feedback on this, especially from anyone who tends to work a lot with DNS.

(click "list domains" in the control panel to see the page)

Thanks!

working on control panel

2007-03-29T20:42:00.000-04:00

The control panel will be down for the next hour or so while I integrate some changes. Apologies for the short notice.

2007-02-28T19:52:00.000-05:00

The certificate for secure mail on mercury was about to expire. I added a new one good until 2017.

If you don't know what this means, you probably don't need to worry about it. :)

subject= /C=US/ST=GA/L=Atlanta/O=Sabren Enterprises
Inc/CN=mercury.sabren.com
notBefore=Mar  1 00:42:21 2007 GMT
notAfter=Feb 26 00:42:21 2017 GMT
MD5 Fingerprint=30:91:77:E0:B9:17:14:92:B2:3F:2B:C1:E0:8D:E7:7F
-----BEGIN CERTIFICATE-----
MIICXzCCAcigAwIBAgIBADANBgkqhkiG9w0BAQQFADBqMQswCQYDVQQGEwJVUzEL
MAkGA1UECBMCR0ExEDAOBgNVBAcTB0F0bGFudGExHzAdBgNVBAoTFlNhYnJlbiBF
bnRlcnByaXNlcyBJbmMxGzAZBgNVBAMTEm1lcmN1cnkuc2FicmVuLmNvbTAeFw0w
NzAzMDEwMDQyMjFaFw0xNzAyMjYwMDQyMjFaMGoxCzAJBgNVBAYTAlVTMQswCQYD
VQQIEwJHQTEQMA4GA1UEBxMHQXRsYW50YTEfMB0GA1UEChMWU2FicmVuIEVudGVy
cHJpc2VzIEluYzEbMBkGA1UEAxMSbWVyY3VyeS5zYWJyZW4uY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQCtw81fs1Kx9v7iFghAtwrMr1xy8h7ZrIwYsjUr
+ZhN/EpiMYfGFbHsZH1B1bJMHfiyUtgf9VrUDNt/InN7TFDl3KOsluvxD/lmsLov
kvsNLDpvy72Lnxb77WaNGjKAXKQ3bsLf0XmuH8rs1EGmJKjsKaC7ANhwjadDOHnM
Cd/MkwIDAQABoxUwEzARBglghkgBhvhCAQEEBAMCBkAwDQYJKoZIhvcNAQEEBQAD
gYEAnGWCpGvMAIO0yZllZcK1p9OFwAWTRONcXLUzAgXxQaAbQDRrA6AeiqAoCGn5
L6UcAT3Fvj2D7bgAn7QDMk6cbDtl0BfhKsvuyU0bU3RQNKufjph7G6dgScRwd6Vs
POwCJzB5p0EKPYiQ4eNtyusnmxtS9rZGm4ogJ22aHtcpqQo=
-----END CERTIFICATE-----

Certs for all machines are listed here:

https://secure.sabren.com/cornerhost/certs.html

hydrogen upgrade tonight at 2am central

2007-01-27T18:50:00.001-05:00

The upgrade to the database server has been scheduled for 2AM central tonight. Again, it should only be down a few minutes.

cornerblog

migration issues

migration complete

migration underway

cutover starting 3:00AM EST

Data Center Move in Progress

db issues resolved

database issues today

control panel / db / etc upgraded

Server Consolidation / Upgrade

mercury down

outage at the data center

manganese php virus

Outage

mercury offline this morning

mercury back online

Still waiting...

Mercury down... (maybe?)

mercury restored

mercury taken offline

mercury down

what to do if your old PHP script broke

mail to comcast to be fixed shortly

SMTP and control panel login fixed

post upgrade issues, revised

new ip addresses

control panel working

all servers online

manganese fully functional

manganese and titanium partially restored

known issues

scandium and vanadium online

mercury online

final sync in progress

down for the upgrade

reminder: upgrades tonight

innercircle downtime

don't use www for testing

Test your site on the new servers!

dropping php 4 (?)

ns1.sabren.com migrated to new boxes

db.sabren.com rebooted

MySQL clock adjusted to EST

MySQL 5.0 Now Running

MySQL down for upgrade (finally)

Upgrade Still Going On

Upgrade Postponed to Sunday Night

Reminder: DB Upgrade Tonight

MySQL upgrade this Saturday Night

Upgrade Specs

upgrades coming

duplicate mail on manganese

working on vanadium

mercury fixed

Update

From Michal

apache outage on mercury

twisted/python help wanted

server response times and new issue tracker

15 Lifetime Accounts Available

quick reboot of all boxes at 1AM EST

working on mercury (again)

working on mercury

mod_security rules updated

New Policies for Catchall Rules

Catchall Forwarding No longer Allowed At Cornerhost

Existing Catchall Forwards Will Expire On July 7

The Default Catchall Rule Is Now "Bounce"

An Alternative for Gmail/Yahoo

outage

call for php5 module requests

PHP5 upgraded to 5.2.5

top 5 priorities

Automated Mail Queue Management

Mod Security Improvements

Catchall Rule Changes

Fix all known Control Panel Bugs

Update PHP5

the queues look good

mail queue triage

Catchall Forwarding
No longer Allowed At Cornerhost

Existing Catchall Forwards
Will Expire On July 7

The Default Catchall Rule
Is Now "Bounce"