So I woke up this morning to the joyful sight of 1786 emails waiting for me. They were all automated, from the various scripts that run every minute on the server. Apparently at some point last night after I had gone to bed, one of these scripts (the search index updater, as it turns out) started going nuts and eating up all the RAM (memory). I have a different script which watches for when this is happening, and it automatically reboots the server when memory gets down to a certain level. Usually this will fix random, sporadic issues with slow memory leaks in long running processes. But the problem was that after reboot, the search index updater script was run again as usual, and apparently whatever condition caused it to go nuts last time still existed, so it then proceeded to do the exact same thing, around in a loop, rebooting and freaking out, reboot, freak out, barf, rinse, repeat.

So I have now disabled the search index updater script until I can find out why it's suddenly going off the rails like that - I haven't changed anything in there lately, so I have no idea what's going on yet. I am using a third party search index called Sphinx, which is very fast and good but somewhat opaque in terms of its inner workings. I'll have to see if I can replicate the issue on my home workstation. Being able to reproduce the bug is 50% of the way to fixing it, usually.

I think this is not related to the hard drive issue that we had recently. That was hardware, this is (I assume) purely software. The RAID array is reporting optimal status. Nor is it related to the ongoing DNS propagation issues.

It's weird that all this should be happening just in the middle of my house move. Almost enough to turn me from being an atheist into being someone who not only believes, but harbors a niggling suspicion that I'm being played with.

Apologies for the further downtime - I think everything is up again, but obviously the search index won't be updated until I can find out why the update script was going haywire, and that may take some time as everything I own is in cardboard boxes, and I need to sort out my life somewhat before I can return to full time development. Sorry about that! At least the internet at the new house was working as promised when we got here (quite amazing - I don't think I've ever had cable internet 'just work' like that - so there you go, not everything is going wrong... touch wood).

I'm getting rather tired of having to write "thanks for your patience", but it still seems to apply...

Neil