So as you can tell, this maintenance is taking a lot longer than I expected. Nothing's broken, it's just that what I thought was going to be about an hour and a half process turned into a bit of a slog. So here's the deal. Some of it technical, some of it not, but all of it honest.
Our hosting provider is Linode. (This isn't a Linode rant, they're awesome!) Before around an hour ago, we were running on their Linode 4096 plan, which runs $80 per month. Since City of Hero's shutdown, though, our traffic is a fraction of what it was, so we really don't need that much horsepower to run the site. (But we'll still have it, I'll get to that in a minute...) So for a while now, I've been wanting to downgrade the server to a Linode 2048 plan instead, which is $40 per month. I knew that would be kind of an ordeal, so I put it off until tonight.
So tonight, I buckled down and started working on it. The first order of business is that I would have to shrink some of the filesystems. No problem, we had a lot of space not being used, so that should easy peasy. But I got to thinking about it, and I didn't like the layout of our filesystems. For you Unix geeks out there, I had a root filesystem (/), /home, /var/www (all of our web site data), /var/database (all of our mysql databases), and /var/vcs (our Mercurial code repositories).
At first, I thought, what the heck, let's just put the whole kit and kaboodle on the root filesystem. That was the first thing that extended the outage time. It's not like there's any huge advantage to us for having them split up, and it's all redundant virtual storage anyway so it's not like we'll ever just have one filesystem fail, so I started copying /var/www over to our root filesystem. Around 45 minutes later, it was copied. Great! So I started copying the others, starting with /var/database.
Around that time, I got to chatting with Codewalker, who mentioned that it might be worthwhile to have the root filesystem by itself and /var separate, so that if the /var filesystem fills up, the server won't crash to the point where we can't log in remotely. He's right, so all of a sudden, I was almost back to square one. We decided to consolidate our web directories, database directories, user home directories, and version control system directories all on /var, and have that be a monster filesystem, with root and everything else that's not /var on its own, much smaller, filesystem.
So with that, I deleted what I had copied over to /var on the root filesystem, resized it back down, and increased the old /var/www filesystem, moved www down a level under /var, and started copying the other stuff. After some configuration changes and filesystem resizing, another two hours had gone by. I fired up the server, and I started getting a bunch of error messages. Crud! What happened? I had stupidly forgotten to copy the files from /var on the root filesystem to the new /var filesystem I had created. Okay, no sweat, back into single-user mode, copy over, boot back up, everything works.
Now, it's about five hours in, and I'm only now getting to the whole reason I wanted to do this stuff to begin with: to downgrade the server. So I pop into Linode and choose the new sizing option. It tells me that I need to use my free upgrade before I can downgrade the server. Um... free upgrade?
Turns out, at some point in the not-too-distant past, Linode changed their plans around, doubling the RAM and increasing the disk capacity on all of their servers. But it didn't just happen automatically, since it required a server reboot and some down time to migrate everything. So at least for a while, we've been on a Linode 4GB RAM plan, but have only had 2GB of physical memory allocated to us. That's right, folks, we've only been using half of our allocated resources, grrr!
So I had to upgrading the server (for free!) to a Linode 4GB server, a bit ironically, so that I can complete the downgrade of the server (to half price!) to a Linode 2GB server. Done, then a quick kickoff of the downgrade (which actually took a lot less time than I thought, probably since I had already resized the filesystems and Linode's request queue was much smaller by then), and now we're finished,
booyah!
I'm really sorry that it has taken this long to get everything straightened out, but I'm also
really glad I'll be saving $40 per month and we'll be operating with the exact same resources we had before.