Some wang(s) cracked our mail server at work Sunday. Fortunately, nothing but time was lost in the process. Chris has some good ideas for tackling the work that followed. What he forgot to mention was: Be Paranoid. Since, I'm often an asshole, so I know how to think like one. Expect any/all common helpful commands to be compromised. I think in retrospect the best thing to have done would have been to immediately cut the power to the box. Once down we could have then booted off a CD and inspected what happened safely. (fsck be damned. You should be using a journaled filesystem anyway...) I wasn't thinking quite that sharply, but I did avoid a few landmines. We saw that shutdown was altered, and while poking around we triggered some process that was trying to "rm -rf *" part of the filesystem. Fortunately, we umounted the vital partitions first (after kill -9'ing a lot of now suspect processes.) The timing was quite amazing. We are in the process of moving offices, and in the middle of a huge release with a completely unrealistic timeframe. All our furniture was moved yesterday. This morning the phone guys helped out by accidentally tearing out our T1 and phone system. At that point I starting to get nervous... what next a meteor impact? The new install kicks ass. This isn't the way I wanted to do it, but I'm glad it's done. We now have a shitpile of storage, and a much more modern distribution. I also replaced horde with Squirrel Mail. It's vastly simpler to install/maintain and provides some nice additional features.