vanadium issue resolved (?)

I discussed the apache problem with the planet security team and they were able to block the flood of packets that was causing the problem. It seems the problem was either a denial of service attack or something very much like it.

I'll keep an eye on the server but it looks like the problem is solved.

vanadium update

Still working with the planet to diagnose the lag time on vanadium. It seems that apache is clogging up with unread requests. I've set apache's timeout between IP packets to 10 seconds (vs the default of 5 minutes) and this seems to help prevent the clogging symptom. Still no word on the core problem though.

vanadium network issue

Several people have complained that web and mail service on vanadium have been unreachable over the past few days, though the load has been light and the server has responded quickly for me. They are able to access my other servers with no problem.

This leads me to believe there may be a network or routing problem at the data center. I've got a support request in and will post an update whenever I hear something.

scandium downtime friday

Here's what happened to scandium on friday:

A database change I made for the latest control panel update broke the script that creates web and log directories for a new site. This is a very old script that I don't yet have under automated tests.

That would not normally cause a problem, except somebody created a site and asked to have their error logs turned on (and thus sent to the directory that didn't get created)

Unfortunately apache crashes and refuses to start unless it can open all logfiles. I fixed it on friday but didn't manage to track down the root cause until just a little while ago.

I've made some small changes to make sure this particular script will give an early warning should it ever break again. Getting all my old scripts under test is still an ongoing process.