Something we implemented a couple of years ago was Sphinx Search. It is much faster than MySQL’s built-in text search, and is something that’s been generally trouble-free. Until today.
Normally it indexes our database in the morning and takes a minute or two to crank through all 8.whatever million posts in our database. Then every […]
Got in touch with a consultant and he recommended working directly with the files on disk, rather than through the database server.
Working that solution now, so THR’s web servers can’t connect to the database server and I’m not sure what message you’re seeing.
Either this will work, or I’ll restore from backup.
My fingers […]
If you look on the right there’s a sign-up form for a new mailing list I set up today.
This list will be used for maintenance and outage notifications only, so if you want to subscribe you’ll probably receive less than one e-mail per month.
More relevant to today though: as soon as the server is […]
Here’s the best information I’ve got on the situation:
There was a bug in the kernel of the database server’s operating system. This bug kicks in when large files are accessed over particular hardware. The reports I’ve seen suggest that this problem really only appeared on HP hardware, and since we don’t run […]
I try and announce these outages in advance, but performance this morning was just atrocious.
I expected the outage to take about an hour, but it’s running beyond that. I’ll post more as I learn more.
Sorry for the outage.
12:12PM EST UPDATE: everything is still running cleanly, but slower than normal. The cause seems […]
I’m posting this here not because it’s relevant to THR’s userbase, but because Google doesn’t seem to have any results listed that relate to this problem.
For years I have periodically pulled backups from my colocated servers to an off-site location using a program called Rsnapshot. This has worked reliably for years, but […]
My colocation facility gave me a heads-up that they were replacing one of their giant UPSes yesterday, but that no problems were expected.
It turns out they had a problem, and my cluster lost power this morning around 6:30am EST. Things are configured for automatics failover, but what I think happened is the servers came […]
There are a number of issues to be resolved while I’m at the datacenter, but I expect that the total outage will last less than 2 hours.
The tasks I’ll be working on include:
Putting new memory in one of the firewalls. I’m running a pair of firewalls in fail-over mode, and the fail-over transition works well and takes […]
Over the last 60 days the database has become defragmented — so much so that the average page load time has increased by about a half-second.
So, this was an opportunity to defragment the database the restart the server after some basic updates. No big deal, just down-time.
Thanks for your patience.
Summary: 110 minutes […]