mercury back online

Mercury is back online and the misbehaving firewall has been disabled.

Sorry, everyone. Needless to say, that security group won't be messing with my servers again.

Still waiting...

The firewall does seem to be out of control, blocking everything. This issue has been passed to a level 2 tech at the data center...

Mercury down... (maybe?)

I signed up for a security audit and ongoing monitoring on mercury... It appears they've installed some kind of overly paranoid monitoring system that is blocking legitimate customers at the firewall level.

I seemed to have triggered this myself and can't even tell if the server is up anymore.

I have a ticket open with the planet and we're getting this sorted out ASAP.

mercury restored

Mercury is back online. I've updated the mod_security rules and enabled suhosin, which adds an extra layer of security.

One thing suhosin does is completely remove the possibility of remote includes for php. So if you have an include() call that starts with "http://" it will no longer work. What you need to do is put the include file on your account and replace the "http" path with the corresponding local file system path.

mercury taken offline

Apparently there was a security breach on Mercury, and the data center was unable to contact me. They disabled the server several hours ago.

I am working to resolve this issue right now.

mercury down

Mercury just stopped responding and I'm unable to bring up a remote terminal.

I've opened a ticket for the data center to investigate.

update: it's back online now. Was down for about 8 minutes.

what to do if your old PHP script broke

Hi guys,

The number 1 problem cropping up for people whose PHP scripts have stopped working correctly since the upgrade is that they rely on the old, insecure way that PHP handled variables passed in from the query string in the URL.

This old behavior is one of the major causes of security holes in PHP. Basically, if the program is not well, written, it may be possible for an attacker to change variables that were not meant to be passed in through the URL, and possibly take control of your site.

Although this only affects a very small percentage poorly written scripts, the PHP community has come up with a much smarter alternative, which is to move these variables into special associated array variables like $_GET, $_POST, and $_REQUEST.

So, if your script depends on a variable named $foo being passed in through the URL, like so...

http://whatever/script.php?foo=bar

... Then you should put this line at the top of your script:

  $foo = $_GET['foo'];      // read from query string

Similarly, you can read from a form post with:

  $foo = $_POST['foo'];     // read from form post

If you prefer to accept both GET and POST requests, you can use this line, which handles either method:

  $foo = $_REQUEST['foo'];  // read from either

Add one of these lines to the top of the file for each variable in your form. (Make sure you put it INSIDE the <?php ... ?> area!)

Having said that, I realize this is short notice, so as an alternative, you can revert to the old behavior for your entire site by adding this line to your .htaccess file:

   php_flag register_globals on

However, at some point in the future, I will probably disable this option completely... So if you don't feel comfortable editing your php scripts yourself, please get in touch with me and ask for help.

For more information on this PHP issue, see here:

http://us2.php.net/register_globals

Thanks!

mail to comcast to be fixed shortly

It seems comcast is rejecting mail from the new machines until we get reverse dns records (PTR records) set up. I've put a request in to the data center so hopefully this should be fixed this weekend.

SMTP and control panel login fixed

The SMTP server is working correctly now on all boxes.

(For the curious, the problem was that even though SASL was installed and configured correctly, saslauthd wasn't actually running by default anymore.)

The control panel is also working correctly now. (Though the logfile data is still missing).

post upgrade issues, revised

fixed:

• logfiles have been restored
• SSL for mail is now working correctly

in progress

• control panel is rejecting logins
• SMTP seems to be rejecting logins for some people

(These are two unrelated issues as far as I can tell. The control panel will be fixed shortly. SMTP is still a mystery.)

new ip addresses

If anyone needs the new IP addresses, they are:

mercury.sabren.com.     IN      A       75.125.245.26
scandium.sabren.com.    IN      A       75.125.250.26
titanium.sabren.com.    IN      A       174.133.41.42
vanadium.sabren.com.    IN      A       174.133.41.90
manganese.sabren.com.   IN      A       174.133.33.154

control panel working

The control panel should be working again (except for logfile reports).

all servers online

All servers are now back online.

Still working on:

• SSL for mail
• old logfiles
• getting the control panel hooked up

manganese fully functional

Mail for manganese (and all servers except titanium) is back online.

Titanium is still chugging along...

manganese and titanium partially restored

I've turned on the web server for manganese and titanium.

I'm still waiting on mail to finish syncing before people can log back in. Looks like it's getting near the end on both machines.

known issues

Known issues for anyone looking at the new servers:

• SSL isn't quite working yet for mail. (but you can use the webmail interface
• Your web logfiles may or may not be there yet.
• Control panel changes won't take effect immediately (if you can log in at all).

All of this should be fixed shortly.

scandium and vanadium online

Scandium and Vanadium are back online.

Titanium and Manganese look like they're pretty close.

mercury online

Mercury is back online.

The DNS change appears to have gone through as well. Please let me know if you're having any issues.

The other servers are still waiting on rsync. I'll bring them online as they finish. Hopefully it shouldn't be more than another hour or two.

final sync in progress

The final rsync is in progress. All accounts are locked to prevent file changes during this time (including web,mail,and ftp access).

This will take a couple hours. Speed varies depending on the server.

I am also initiating the DNS changes for all domains.

down for the upgrade

It's about 3:30AM eastern time, and I'm taking the servers down for maintenance now.