Something we implemented a couple of years ago was Sphinx Search. It is much faster than MySQL’s built-in text search, and is something that’s been generally trouble-free. Until today.
Normally it indexes our database in the morning and takes a minute or two to crank through all 8.whatever million posts in our database. Then every 5 minutes it runs again in update mode to process the newest posts.
For some reason it’s taking much longer than normal to run today. I’m working on fixing it, but that’ll require a reboot, which will mean 3-4 minutes downtime unless I discover a deeper problem.
I’ll keep you updated.