MySQL move complete

The MySQL move is complete and (as far as I can tell) problem-free. Total database downtime was about 30 minutes.

I updated the /etc/hosts files on all the hosting servers to use the new IP address, so any scripts at cornerhost should already be connecting directly to lithium.

If you're set up to access MySQL from outside my network, mercury is working as a proxy server and will tunnel your requests to the new machine.

One minor issue acffects people who have specifically requested access to MySQL from a fixed IP address oustside cornerhos:

I think everyone who has access from a specific IP address also has access from mercury, but just in case I missed someone... If you get a permission denied error accessing from home, the fix is to add a line to your hosts file that looks like this:

66.139.78.174  db.sabren.com

The host file is in /etc/hosts on unixlike systems, and in various places for various win32s. Google suggests this page if you need to hunt down the proper directory.

If anyone has any *other* database problems, please let me know ASAP!

Reminder: MySQL move tonight

Just a reminder that MySQL will be broken for a few hours tonight while I move it to the new server. (probably around 2:00AM EST)

analog changes

BTW, three recent changes for the nightly analog reports in the control panel:

• Reports now show the last seven days, not the last seven logfiles. Usually these two things mean the same, but if a site did't have traffic for a particular day, then there wouldn't be a logfile, and the report would show older data. This is fixed.
• If a site has NO logfiles for the past seven days, you'll get a message saying there was no traffic.
• The FLOOR value for the various options has been set to 1, so you'll now see all referrers, etc. I may make this configurable later, but probably not any time soon.

The changes are working on some servers already and will work on the rest tomorrow.

BTW, This fixes the longstanding "I'm getting someone else's logfiles!" error for new sites that practically everyone has reported at some point in the past. :)

[The someone else was always the default apache logfiles]

mysql move thursday night

The MySQL server (db.sabren.com) will move from mercury to its own bigger/faster/cooler server sometime "late, late thursday night" (EST).

There WILL be some downtime, but hopefully not more than an hour or two. A tcp/ip tunnel from mercury to the new machine will keep things running while the DNS change goes through, so nobody needs to change their code.

Unfortunately, I wasn't clever enough to name the new server dubnium...

NEW new public key for mercury

D'oh! Not only did I install the wrong private key, the certificate was only good for 30 days... This one is good for the next two years:

-----BEGIN CERTIFICATE-----
MIICXzCCAcigAwIBAgIBADANBgkqhkiG9w0BAQQFADBqMQswCQYDVQQGEwJVUzEL
MAkGA1UECBMCR0ExEDAOBgNVBAcTB0F0bGFudGExHzAdBgNVBAoTFlNhYnJlbiBF
bnRlcnByaXNlcyBJbmMxGzAZBgNVBAMTEm1lcmN1cnkuc2FicmVuLmNvbTAeFw0w
MzAzMDUwNDA1MTJaFw0wNTAzMDQwNDA1MTJaMGoxCzAJBgNVBAYTAlVTMQswCQYD
VQQIEwJHQTEQMA4GA1UEBxMHQXRsYW50YTEfMB0GA1UEChMWU2FicmVuIEVudGVy
cHJpc2VzIEluYzEbMBkGA1UEAxMSbWVyY3VyeS5zYWJyZW4uY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQDJmiJnPEhAk8nakPWXuT4kYQsuLCgxLPKBFVLP
c3P0S6ur8QT8Kgtn+o5nAHZjGp03kf9Rdr95qKd+KkVoZioMjCqns6BKLOWZ6FN6
QKTYZ9dIOx/Zsn8m502hXWHzH6QmJCltqrbi8ykjSLM7gKYi2eOHEqvMYhboiCeF
oNnj7wIDAQABoxUwEzARBglghkgBhvhCAQEEBAMCBkAwDQYJKoZIhvcNAQEEBQAD
gYEASqxzcFJHpwQiklZBWZcwcteFTvH5bxuAHIqGlHvVUCoDJgSx/g+/CnGN7fD+
cgufShqvhfKGGmKDVS/knNJp2N4ZvpdVK1gOLpKHawvK7hANGtcYw2083v8433Wt
ov1cFZj8kp0YbU+rjuC+m+gE8hSABLRDtyApKhZHtAenr6c=
-----END CERTIFICATE-----

titanium problem solved

I was going nuts here trying to figure out why titanium, which has hardly any traffic would spontaneously be overrun with so many requests, and then shut down.

Turns out someone had a php script that managed to request itself via http, thus creating an infinite loop and tying up the server.

Kids, don't try this at home.

Or, at least, don't try it on my servers. :)

new public key for mercury

Mercury users:

The SSL certificate for secure POP3, IMAP, and SMTP expired 3/2 on mercury. Here is the new certificate, which won't expire:

-----BEGIN CERTIFICATE-----
MIICXzCCAcigAwIBAgIBADANBgkqhkiG9w0BAQQFADBqMQswCQYDVQQGEwJVUzEL
MAkGA1UECBMCR0ExEDAOBgNVBAcTB0F0bGFudGExHzAdBgNVBAoTFlNhYnJlbiBF
bnRlcnByaXNlcyBJbmMxGzAZBgNVBAMTEm1lcmN1cnkuc2FicmVuLmNvbTAeFw0w
MzAzMDQxOTM4MjhaFw0wMzA0MDMxOTM4MjhaMGoxCzAJBgNVBAYTAlVTMQswCQYD
VQQIEwJHQTEQMA4GA1UEBxMHQXRsYW50YTEfMB0GA1UEChMWU2FicmVuIEVudGVy
cHJpc2VzIEluYzEbMBkGA1UEAxMSbWVyY3VyeS5zYWJyZW4uY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQC9VXkVJi9SVyRs0+fmFprC5OsgY63Y9Rd2cBEf
J0cw6L6yBu1rda1/EnKXaNgBqYrm/ESOvISNTqdF2rxZDKbPJxkBv6B+LAAXArnR
tq0vzoTyge4k0cZXiW5B54aWacXjF+Fz6KoWzFw3PLp9QofitqT37bzFPK8AKx7c
tbbK4wIDAQABoxUwEzARBglghkgBhvhCAQEEBAMCBkAwDQYJKoZIhvcNAQEEBQAD
gYEAk+qINZYRyFeqEfQ5kzaS2vnScmK0tPvCCqoN0S6GfFwdkApSqvim5sQ+U5ch
1H2lo57IE/hGpACbS3hJH0PHmkrA0wGF6c7xYQaWhjxzg0Bpz1KV8KI0c9Hv826R
6QF7w1Vzv0+j1Lazk49deTPo5W0WDAwMZG6f5ibJvoUYXlw=
-----END CERTIFICATE-----

I'll do this for the other machines in a few months.

titanium outages and new "immune sytem"

First: titanium is up and running again. [It was down for two several-hour periods early sunday morning and later sunday evening]

The logfiles said it was basically being swamped with hits, and finally reached the max number of clients. But that's not supposed to shut down apache forever, so I still need to track down what's actually happening.

But right now I'm more concerned about my own response time.

A while back I wrote a script to check the status on each server every five minutes and email me if there's a problem.

Twice today I opened my inbox after a few hours to find a zillion messages telling me titanium was down. (Not to mention all the emails from the poor users ON that machine!) Not good.

So I did some hacking. Now, if the apache connection is refused or times out, the script will attempt to kill apache and restart. I just manually killed titanium again and ran the checker. It brought apache back to life.

I'll track down what's actually going wrong, but hopefully, if it happens again tomorrow, the outage should be corrected automatically within 5 minutes. (Same for all the machines)