THR Status and Updates

Outage, weird security cert message

Derek — Wed, 09 Jan 2013 17:22:03 +0000

Working on it.

Fixed

Now, if you still have problems connecting, use a different web browser, or clear your cache for the last few hours and you should connect.

The Next Step

I think I need to find someone to sacrifice a chicken and cleanse the servers of evil spirits or something…

Wednesday AM Database Upgrade

Derek — Wed, 09 Jan 2013 00:13:06 +0000

If you’re trying to access THR on Wednesday morning and you got here via an error message, then you’re probably experiencing our database upgrade.

Best case: we’ll be down for 15 minutes, the everything will be running again.

Worst case? Something will break, and it’ll be daylight before you see the server again.

Sorry for the inconvenience.

Thanks for your patience.

Tuesday Weirdness

Derek — Tue, 08 Jan 2013 18:00:08 +0000

Database work is going on, which sometimes means reboots. When this happens:

The database completely goes away for somewhere between 30 seconds and a minute.
After it comes back up, it won’t answer the web servers for another minute or two while it familiarizes itself with our data.

This will happen a few times today. If it happens to you, please try again in a minute or two.

Tuesday Morning Outage

Derek — Tue, 08 Jan 2013 09:52:31 +0000

Short version: we’re doing some maintenance to keep up with our current load and will be down for a while.

Longer version: we’ve seen much more demand this last couple of weeks, and as a result at peak times users are having to wait 30 seconds for pages to load. To alleviate the waits, we are being forced to make some structural changes to our database, and these changes make THR unavailable until they complete. So we started at 4:30 AM so we could perform the changes at the slowest time of the day.

THR will be back up today, but don’t expect it to take less than an hour. Understand it may take a few hours — we just can’t predict it.

Once the process is complete, THR should again load properly.

Thanks for your patience.

And we’re back up. Thanks again.

Short Outages this weekend

Derek — Sun, 06 Jan 2013 21:10:13 +0000

Demand on THR’s database server has been higher than normal, and as a result we’ve seen some increased slowness at times. As we move on into SHOT season later this month, load will get even higher, so I’m working on optimizing things in advance of the show.

I’m tweaking throughout the weekend. If you see a brief outage, or a database error message, it’s probably because I just restarted the database and you decided to connect at exactly the wrong second. Please just reload the page and THR should come right back up.

Thanks for your patience.

12/30/2012 Database Maintenance

Derek — Sun, 30 Dec 2012 09:47:59 +0000

The last time we ran maintenance routines things didn’t complete properly. So far this morning things are slow, so we’re running the process again.

We started around 4:30am Eastern Standard Time. Expect things to take a total of two to three hours. Edited to add: it’s taking longer than I expected. We’ll be back soon.

Sorry for the inconvenience, and Happy New Year!

12/18 Outage

Derek — Tue, 18 Dec 2012 12:52:50 +0000

11:08am EST Status: Back up

Please note that the updates below are based on my understanding of the problem — that only one database table needs to be checked. The estimates are based on the number of records processed, though even in optimal conditions the repair will take a bit longer than these numbers imply.

10:30am EST Update:

Indexes 1-6 (I believe that’s all of them) on the largest table are apparently repaired, and the database tables are being copied. Hold tight folks – we’re close, but I can’t estimate how close.

Assuming, of course, that nothing else goes wrong.

10:02am EST Update: 94% complete

9:59am EST Update: 85%

9:51am EST Update: 67% complete

9:44am EST Update: 57% complete

9:41am EST Update: 47% complete

9:34am EST Update: 35% complete

9:29am EST Update: the repair looks like it’s about 25% complete

The Problem:

Our load over the last few days, unsurprisingly, has been about double what we’re used to. Things were slowing down, and back-end processes (like the process that updates search, and the backup routines) were starting to conflict with each other as well, so basic database optimization was required.

I triggered the appropriate maintenance to run at our slowest time this morning, and the process failed because it ran out of disk space. The server went idle while waiting on me to free up some disk space, but I couldn’t fix the disk space issue due to some weird combination of the backup software I’m running, the database software, and the zero disk space problem. I’ve found a patch that should stop this from happening in the future, but right now the immediate problem is getting the database server running again.

I’ll keep y’all updated, but I will have the server up as soon as possible.

Early Sunday Database Work

Derek — Sat, 17 Nov 2012 16:21:41 +0000

If you’re trying to connect early in the morning on 11/18/2012, then you’re likely seeing problems due to a maintenance task we have running.

The database server has been running a while, and we’re helping it out by defragmenting the database. This takes more than an hour, but less than two, unless something fails. Come back a little later and everything should be fine.

Sorry for the inconvenience. We try and do this once a month or so, and it’s just time again.

Quick Reboot

Derek — Wed, 17 Oct 2012 16:37:49 +0000

Many have noticed our nightly backup window. I’m working to reduce that significantly, but doing so requires some system updates, including a reboot.

I’m rebooting in mid-day in order to diagnose things over the rest of the day. THR should be back up within minutes>

Going down: 13:38 EST

Back up: 13:40 EST

Now we’ll have about a half-hour of slowness as the query cache is recreated.

Sorry about the interruption.

Fighting slowness this morning

Derek — Mon, 01 Oct 2012 14:45:00 +0000

Something we implemented a couple of years ago was Sphinx Search. It is much faster than MySQL’s built-in text search, and is something that’s been generally trouble-free. Until today.

Normally it indexes our database in the morning and takes a minute or two to crank through all 8.whatever million posts in our database. Then every 5 minutes it runs again in update mode to process the newest posts.

For some reason it’s taking much longer than normal to run today. I’m working on fixing it, but that’ll require a reboot, which will mean 3-4 minutes downtime unless I discover a deeper problem.

I’ll keep you updated.