If you’re trying to access THR on Wednesday morning and you got here via an error message, then you’re probably experiencing our database upgrade.
Best case: we’ll be down for 15 minutes, the everything will be running again.
Worst case? Something will break, and it’ll be daylight before you see the server again.
Sorry for [...]
Short version: we’re doing some maintenance to keep up with our current load and will be down for a while.
Longer version: we’ve seen much more demand this last couple of weeks, and as a result at peak times users are having to wait 30 seconds for pages to load. To alleviate the waits, we [...]
Something we implemented a couple of years ago was Sphinx Search. It is much faster than MySQL’s built-in text search, and is something that’s been generally trouble-free. Until today.
Normally it indexes our database in the morning and takes a minute or two to crank through all 8.whatever million posts in our database. Then every [...]
I try and announce these outages in advance, but performance this morning was just atrocious.
I expected the outage to take about an hour, but it’s running beyond that. I’ll post more as I learn more.
Sorry for the outage.
12:12PM EST UPDATE: everything is still running cleanly, but slower than normal. The cause seems [...]
There are a number of issues to be resolved while I’m at the datacenter, but I expect that the total outage will last less than 2 hours.
The tasks I’ll be working on include:
Putting new memory in one of the firewalls. I’m running a pair of firewalls in fail-over mode, and the fail-over transition works well and takes [...]
THR is back up. Thanks for your patience.
In the long run we’ll be doing this periodically, like early in the morning on the first day of every month.
We’re doing this one manually though, as the database could really stand to be defragmented after the recent [...]