Thanks

Okay. I'm ordering two new machines, to at least let me control the damage if a system goes down again in the future, and rackspace will help me rebuild the existing one from scratch.
 
Meanwhile, I just wanted to say thanks again to everyone for being so understanding about this. I have great customers! (Well, except for two of them that aren't around anymore....) [and if by chance you missed all this, scroll down to the bottom of the page and see what all the fuss is about]

check your old email!

If you were one of the forty people whose passwords were compromised (you'll know because you won't be able to access your account), I have sent you an email... These emails went to the address you gave me when you signed up -- oftentimes old email addresses that are not at your domain. (I can't send the password to the mail at the domain, because I've locked you out of it)....
 
There are four people I have not been able to send emails to, because the only email address I have for you is at your domain.... So, instead of just putting your password back and risk letting the hacker in, or... I will try to contact you by phone... In which case, check your voice mail.
 
(seems the worst damage caused here -- other than the colossal inconvenience to some of my customers, for which I DEEPLY apologize -- was to my sleep schedule and phone bill... Nothing like a few international calls to top off a great day...)

seven ways to protect your PC

Mark Pilgrim sent me a link to his list of 7 ways to protect your home Windows PC for free. As he says, choose all 7....

psyBNC

psyBNC is EVIL. What is psyBNC?

If you know nothing about bncs, a bnc is short for a 'bouncer.' A bnc acts as a proxy for irc, allowing you to hide your real IP address and use a vhost (vanity host - something like 'this.is.a.l33t.vhost.com'). What are the advantages of this? Well, mainly there's just one important one: It'll stop stupid packet kiddies from trying to knock you off the network. Everyone hates getting disconnected, and with a bnc on a decent shell, you should be pretty immune. Remember though: the kiddies can still nuke you, but it is assumed that the shell provider has a high-bandwidth line that allows it to withstand the numerous packets. If your shell is on a 56.6, you'll still be screwed.

In other words, script kiddies run this so that they can flame people on IRC and have the other script kiddies attack their web host.
 
Can anyone tell me how this could possibly be a win-win situation? If I see bnc running on an account, I will close the account. Period. End of story.
 
I make a point of this because every trouble account I've ever had - including the two hackers - has run this software. It's quite possible that these guys were not hackers after all, but just annoyed someone who really was.
 
...
 
I've had exactly two orders today... Both from people wanting to run this software.

what about my home machine?

Someone just asked me if one of the hackers could have accessed his home machine. I'd say this is very unlikely.... It doesn't even look as if the hackers actually got to use any of the stolen passwords before I changed them, but even if they had, your home machine would most likely not be affected.
 
However.... That does NOT mean that you're safe. Even if you think nobody knows who you are or has any reason to bug you, that does not mean you cannot be hacked. There are plenty of script kiddies out there who simply search through hundreds of IP's looking for vulnerabilities. If you use a windows machine, and you've never taken steps to secure it, you're very likely in danger of an attack. In fact, you may have been attacked already without even knowing about it.
 
The place to go for securing your windows machine is Steve Gibson's Shields UP! page, which will actually examine your machine remotely, the same way a hacker would, and report on the results.

sftp

I turned off ftp access earlier this morning... I'm turning it back on for now.... but if you have a shell account, I strongly urge you not to access your account through ftp anymore. Instead, use sftp or scp. There's a win32 implemenation of each over at the PuTTY site... I'm going to see what I can do about setting up sftp for non-shell users.
 
BTW, I will not be allowing telnet access anymore. Use ssh instead. (PuTTY is an ssh client for windows)

going down the list

Thanks everyone for being so understanding about the break-in. As you can imagine, I've got a ton of email to deal with right now. Top priority is contacting the 40 users whose passwords were logged. As I'm looking over each account manually before contacting the owner, this is may take some time. However, I will be restoring access to all accounts today and as quickly as I possibly can.

cornerblog

I bought this blogspot site a while back. Time to put it to good use. It makes sense to have an off-site blog... Just in case.

hackers = rough night

What started as a quick insomnia-induced email check turned into several hours of pure terror.
 
I got hacked.
 
I had two people sign up for shell accounts yesterday... I was a little suspicious, because neither had a domain. I have had problems with people using the system to host IRC bots before, so I generally am very skeptical of this situation. Basically, if you've got a domain with a blog on it, I'm much more comfortable giving you a shell account. On two occasions, I think I may have lost legitimate customers because I was "overly cautious" about their domains.... So I figured I'd just let it go this time, and just watch the accounts.
 
Big mistake.
 
One was kind enough to pay $200 with what I suspect is a stolen credit card (I haven't settled the transaction)... Yesterday evening, both of these users attacked my system.
 
The hackers overwrote several important linux utilities with corrupted versions (presumably so that they could better hide their trails). They also managed to log passwords for 40 legitimate users through some sort of packet sniffing on the FTP, telnet, and POP3 ports.
 
Thanks to the incredible all-night support from the folks at rackspace (thanks, Jim!), all known security holes are now patched in the server. The two users have been locked out, and the 40 compromised passwords have been changed.
 
If you find that you can't log in today, that's why. Your site should continue to run, and your mailboxes will continue to work - you are just temporarily locked out. I am working to correct this as quickly as possible, but I will have to check each account manually to ensure that it hasn't been further compromised. [So far, there is no evidence that any of the accounts were actually accessed...]
 
Also: I do not store credit card information on the server, so there is no need to worry about that.
 
Here's what I'm doing to correct the problem:

• Even though the appropriate patches have been applied, there is no guarantee that this has solved the problem. Therefore, I will be rebuilding the system from scratch on a new hard drive, and restoring data for all users. (There will probably be an upgrade involved to red hat 7.1 or .2)
• CVS accounts will be moved to a separate machine later today. Most likely, I will disable unencrypted pserver access.
• The cornerhost site, user database, billing system, etc will be moved to a third machine later today.
• Telnet, ssh 1 and unencrypted pop3/imap/ftp will no longer be allowed (after a very short transition period)... These will be replaced with ssh 2 and other secure alternatvies.
• I will be reading linux security news sites on a daily basis from now on. (In fact, you can expect to see a new "favorites" feature on linkwatcher for just this reason)
• I'm going to contact the bank that issued the possibly-stolen visa.
• I will be talking to my lawyers in the real near future.
• I will never again sell a shell account to someone who lacks an existing website, unless I can otherwise verify that they're legit.

Okay. Time to grab some coffee and start letting customers back in. I'm really sorry about this, everyone.