The alternate fix appears to have worked, though I’m still watching it in case we run into residual issues.
Thanks for your patience and support.
Got in touch with a consultant and he recommended working directly with the files on disk, rather than through the database server.
Working that solution now, so THR’s web servers can’t connect to the database server and I’m not sure what message you’re seeing.
Either this will work, or I’ll restore from backup.
My fingers [...]
If you look on the right there’s a sign-up form for a new mailing list I set up today.
This list will be used for maintenance and outage notifications only, so if you want to subscribe you’ll probably receive less than one e-mail per month.
More relevant to today though: as soon as the server is [...]
Here’s the best information I’ve got on the situation:
The Cause
There was a bug in the kernel of the database server’s operating system. This bug kicks in when large files are accessed over particular hardware. The reports I’ve seen suggest that this problem really only appeared on HP hardware, and since we don’t run [...]
I try and announce these outages in advance, but performance this morning was just atrocious.
I expected the outage to take about an hour, but it’s running beyond that. I’ll post more as I learn more.
Sorry for the outage.
12:12PM EST UPDATE: everything is still running cleanly, but slower than normal. The cause seems [...]
I’m posting this here not because it’s relevant to THR’s userbase, but because Google doesn’t seem to have any results listed that relate to this problem.
For years I have periodically pulled backups from my colocated servers to an off-site location using a program called Rsnapshot. This has worked reliably for years, but [...]
My colocation facility gave me a heads-up that they were replacing one of their giant UPSes yesterday, but that no problems were expected.
It turns out they had a problem, and my cluster lost power this morning around 6:30am EST. Things are configured for automatics failover, but what I think happened is the servers came [...]
There are a number of issues to be resolved while I’m at the datacenter, but I expect that the total outage will last less than 2 hours.
The tasks I’ll be working on include:
Putting new memory in one of the firewalls. I’m running a pair of firewalls in fail-over mode, and the fail-over transition works well and takes [...]
Over the last 60 days the database has become defragmented — so much so that the average page load time has increased by about a half-second.
So, this was an opportunity to defragment the database the restart the server after some basic updates. No big deal, just down-time.
Thanks for your patience.
Summary: 110 minutes [...]
I need to reconfigure some hardware, and this will result in downtime.
I expect that connectivity should be affected for no more than 20 minutes at a time, but 3 outages during the day are likely, and more are possible.
Work should be completed by 6PM EST. I hope to get started early, but time [...]
Sign up for THR Maintenance Alerts
