NeilGunton

Unfortunately I have had to take crazyguyonabike offline in order to move the box to a new datacenter. Short story is that there is a problem with one of the RAID drives, and I was not getting the remote help I needed from the people at the datacenter in Las Vegas where the machine has been colocated for several years. This, combined with a series of mysterious, sporadic and unexplained network outages have finally made me decide to bite the bullet and just move the thing to be closer to where we are living, so that I can work on it myself when I need to. So, I have found an ISP here in Corvallis OR (Peak Internet) which seems reasonable. The big downer to all this is that I need to take the server completely down for a few days while it is shipped to me. I just took it down today, and the current datacenter people should be in the process of figuring out shipping, boxing up etc. I hope the machine will be with me sometime early next week, assuming we are able to do 2-day shipping without it being completely extortionate. I will then replace the failed hard drive, and install the server in the new datacenter. We are approaching capacity on the current set of hard drives, so I will need to buy new drives (8 of them, yikes) eventually, but I've decided to try to leave that until a bit later. For now I will focus on just getting the server to the new colo, fixing the drive, rebuilding the replacement, and getting things back online again asap. Then I will do the system rebuild with all new drives sometime down the line, when I am able to give users of the site more notice and everything can be planned.

I'm sorry I wasn't able to give much notice today, but things have not been moving quickly at all with the failed drive, and I have been frustrated that we couldn't just do this apparently simple thing (probably would have taken about 10 minutes if they could simply take the drives out one by one and look at serial numbers, since I had the S/N of the failed drive in a log file, but no, instead they insisted that I used a very obscure command line utility to make an LED light up on the front of the server so they could tell which drive it was - the catch being that I couldn't do this, since the drive had completely gone and so didn't even have a device number, which the command line utility needed, but telling them this was like talking to a brick wall). Anyway, the server has been running in a degraded state now (with the one drive down) since mid October, and if the other drive that was paired with the failed one also went down, then we would lose everything (it's RAID10, for anyone who's interested). Well, we wouldn't lose everything permanently, since I have MySQL replication both to a remote backup server in Germany and to my home workstation, and the pics are also replicated every minute, so worst case we would lose the last minute's worth of pic uploads if the server went down hard. But then I would need to rebuild the system from scratch, which is always a pain. I am keen to get it working properly again, and in the new datacenter, so once I had the new place lined up, I decided to just get it done asap rather than waiting around. I know this will be bad for some people who are on the road at the moment and trying to update their journals, but I just have to bite that bullet for a few days of downtime. If/when I get bigger then I can perhaps afford to buy hot spare servers, but I'm not remotely there yet. I don't want to use the backup server in Germany as a hot spare, because I am in the middle of moving house at the moment, so my own workstation may well be taken offline at any time to go to the new place. So, if the German server was the only one hosting the live website, it would have no backups. If it went down, we would potentially lose everything that had changed since the last snapshot, and I don't want to risk that. So, distasteful as it is, we have a few days of downtime, sorry about that. It doesn't happen very often, fwiw. Feel free to let me know I'm a crappy admin, unprofessional etc, I probably deserve it, but I'm just trying to fix what I regard as a critical problem as quickly as possible.

Thanks for your patience,

Neil

statrixbob

Nonsense. You do a great job providing a great service. Stuff happens from time to time and it sounds like you are dealing with it without the benefit of a lot of cooperation.

Aloha!

Bob

mm718

+1

NeilGunton

Thanks for the votes of confidence... the timing on all this could have been better, as we are also in the middle of moving house. In fact we just moved all our stuff a few days ago from Eureka CA up here to Corvallis OR. We thought we had found a good house, but it turned out not so great (won't go into details here), so we have been kind of up in the air the last few days, looking for another house. Add to this the fact that the rental market in this town is not great - it's a big college town, and the college has grown by leaps and bounds in recent years, with OSU seemingly unloading a lot of the accommodations load onto the town rental market. With so many students a lot of the available rentals are either really ratty and depressing, or else high end and out of our reach. So we're here in this temporary place, everything in cardboard boxes... then to have the website going down at the same time... it's a bit much. But that's life! You deal.

Update on the server: They told me it would cost $50 for ground, $150 for 2 day or $250 for overnight, so I opted for overnight. I don't know if that's a rip off, but I am really in no position to quibble over shipping costs. They're probably being straight about it but I have no way of telling, basically they have to ship it anyway so I have to get it done and worry about costs later. Apparently they missed the shipping window today (had to be by 1:30pm), so as I have already paid their invoice they should presumably be able to get it out the door tomorrow, Thursday... so assuming "overnight" really means what it says (and for $250 it should) then I might have the server here sometime on Friday. If it comes in the morning, which overnight to businesses often do (I think), then I might even be able to get the drive rebuilt and the server back up that same day. This is another reason I went for overnight - otherwise it would go out over the weekend, which is always a pain.

Anyway, I'll keep you posted here and on twitter...

Thanks again for your patience,

Neil

chriskmurray

It is a great resource that charges nothing and only has a small bar to the side asking for donation, a few days down time is more than acceptable. Keep on keeping on and good luck with the move.

Aushiker

+ 1

Andrew

stephenjubb

As a Web Developer myself I am qualified to give opinion.
You give a service for which you do not charge (it's voluntary), no one can complain, you provide the touring community a great benefit.

You have good hardware backup and replication and I think have done at low cost. You have also wrote the web server software yourself if I remember correctly.

There is only yourself and you are clearly passionate about the site, even better you tour yourself. You are a combined engineer, site admin and web developer. A skillset that professionally would cost a lot to hire.

The only point of failure has been the internet service provider, them not wanting to remove the drives, perhaps is from fear of being sued if something goes wrong (I'm not saying you would) just as a culture Americans from observing from the UK, like to sue. Such a situation can hamper companies being able to help, as possibly may be the cause in this case. I don't know not living there I would like third party opinion.

You have given a clear explanation of the situation and feel bad about the server going down.

You have a solution to the problem.

You mention about the lack of planning to minimise server down time. Looking at it the server went into a degraded state mid october, right when you are moving house. The quickest solution is to get the drive replaced and plan a redeployment later.

That is what you did, but the Internet Service provider were not playing ball, which is strange, would love to know why. Either way without a sufficient explanation from them it is better to move the server.

The only way you could really have planned for such a situation is contingency planning and that takes big bucks so as a free service could not be done.

So based on my clinical analysis there really is no fault on your part whatsever.

cheers

Steve Jubb

stephenjubb

Forgot to add also you are spending serious money to get the server moved. Another reason not your fault

stephenjubb

Have you thought of issuing a redirect when people get to www.crazyguyonabike.com to a webpage just informing them?

If you need space I can help.

Appreciate that may cause problems with the DNS updates for www.crazyguyonabike.com but your going to have update it anyway.

Tandem Tom

I have REALLY enjoyed And benefited from Neil's site. This seems like a good time to consider a donation!!

keenancook

+1 - I was thinking the same thing.

BigBlueToe

Neil, thank you for the notice, and big thanks for all you do in providing such a wonderful site. It's time for me to make another donation!

Al Cyone

Or just an "Under Re-Construction" page with links to this forum?

I think I can host it under my own plan (with DreamHost).

Gus Riley

I have used CGOAB to chronicle three tour journals so far. I have used CGOAB for research and entertainment for many years. It has proved invaluable in my trip planning; one of the very best sources for bicycling touring on the internet. I am pledging my donation ASAP.

staehpj1

I hope the moves go well Neil. That includes the server move and your moving house. I too will say that you do a great service to the bicycle touring community and I appreciate your efforts.

lee kenney

Thank goodness ,my morning was a mess , first "wiggo" getting hit by a car, then no cgoab to sooth my cycling fever but what to my surprise . Bike forums has the answer to one of my morning concerns.We must be patient , thanks Neil.

mr geeker

neil, down time is good, it allows you to keep your house in order. as a side note, i cant call you a crappy web host, no matter if you want me to or not

zan0416

Neil, welcome to Corvallis! I live in Corvallis, let me know how I can help you with your move.

We use Peak at our business for internet connectivity, I highly recommend them.

Dan

NeilGunton

Thanks, I have redirected DNS to the backup server and cobbled together a temporary page which tries to explain what is going on. Unfortunately I'm having a problem getting it to work with https, but standard http links should work, like https://www.crazyguyonabike.com/. The DNS might take a while to propagate to some, since whatever DNS you use might still have the old address cached. If you want to see it directly, then you can go here:

https://5.9.19.51/

The server has now been shipped, and UPS seems to confirm on the status page that it is being sent "Next Day Air", so I'm really hoping that all works as it should, and we actually see the server sometime tomorrow. Also hoping that everything will just go smoothly on the rebuild and installation in the new datacenter, but part of me wants to say "yeah, right, when has that ever gone smoothly"! I can think of many things that could go awry - e.g. UPS damages the server somehow in shipping, or the old datacenter forgot to include the spare drive, or even the server simply refuses to start up when I power it on. Oh well, no point worrying about stuff before it happens, I'll just keep fingers crossed and update here & on the website "maintenance" page as significant events occur.

Thanks again for the good wishes,

Neil

sapote

hello welcome to corvallis. wonderful you are here. really great biking community you are becoming part of. why'd you choose Corvallis?

DAMO9966

Neil,

Have you considered moving to the cloud? Pricing keeps dropping and no hardware hassles.

Dave

NeilGunton

Thanks! My wife got a job offer here, and we've thought for a while that the place works well "on paper" in several ways, e.g. easy access to Portland, close to Eugene too, easy access to Oregon coast, temperate climate (ok lots of rain, but otherwise no extreme weather), no tsunamis, no hurricanes, no tornadoes, no extreme cold, no extreme heat, no wildfires. Ok, we're going to see a really big earthquake at some point when the Cascadia Subduction Zone finally ruptures again, but I think we'll be in better shape here than we would have been in Eureka, which is right near the Mendocino Triple Junction (where the San Andreas meets up with the Mendocino Fault and the Cascadia Subduction Zone - one of the most geologically active places in the country, and overdue for an 8+ earthquake). And there is the risk of occasional flooding from the river, but we're not in the flood plain as far as I know. And we have access to all the natural wonders of Oregon, and it's a very bike friendly city, and with the students (always pros and cons with them) we get a town that has a fair amount of culture and amenities. We can drive up to the major urban center easily (90 minutes to Portland on I5 for Chiho's Japanese grocery store). Finding an ok rental house has been hard, but just today we finalized our second attempt (first attempt we thought would be good, but there are, um, some issues with the landlady which I won't go into here). The new house is on Princess, just off Walnut. So there is that busy road very close by, but at least we're not right on it. And the house has some crucial aspects for our two cats: A sliding door in the back (so I can put in one of those patio door inserts with a cat flap to give them access to outside, which they need since they were both adopted as adult strays - they just go nuts if they can't go out) and the back yard, while small, also has ready-made holes in the fences so they can get out and explore (the rear has access to the greenway). So hopefully things are working out - they even gave us 1/2 off on the first two months full rent! Yay, it's been really expensive moving up here so we could use a break. And the server is on its way, so here's hoping things are finally turning around.

Oops, I'm probably writing way too much off-topic stuff here, sorry about that. A lot's been happening and you're getting a kind of brain dump, my apologies.

Neil

NeilGunton

Yeah, that's an obvious question given these hardware issues. I don't have a fully rational response to that, except to say that I still don't really trust the cloud concept. I guess maybe I'm just old school, but I like to have full control over the server, down to the metal. I like that I can spec a really nice RAID10 setup with 8 drives, giving me very predictable IO performance. I also like that I'm not at the mercy of random outages in what are extremely complex, multi-layered systems that I have no control over. My site is of a size that can very well be hosted on a single box, in fact this box can do a lot more without breaking a sweat. The problem with this downtime is really unusual, having to bring everything to a screeching halt just so I can send the server to a different physical datacenter seems like an argument against hosting your own hardware, but it really doesn't happen very often. Also, remember that there have been big outages with all the cloud providers lately, so nothing is immune to downtime. At least my downtime is under my control, and I know how to fix it. Then there's just the simple security of knowing exactly where your data is. But of course there are arguments to be made the other way too - as I said, it's not a fully rational thing, more like an intuitive preference for being in full control right down to the hardware. I do feel like I actually get better performance this way, with everything running on a single local box, rather than relying on all those abstraction layers and distributed services that AWS etc entail. I know that when I've used websites that are hosted on the cloud, they often just don't seem as responsive and "zippy" as one that is hosted on a dedicated server.

Neil

reed523

It's all greek to me. All I know is you provide an awesome service and this is as good a spot as any to say THANKS AGAIN! for your site.

AdamDZ

Besides the fact that I am going through withdrawal not getting my daily fix of journal reading... I can easily wait until the site is fixed good luck Neil, don't feel bad, you're doing awesome job and provide great service. People, consider donating or even subscribing. I subscribed monthly since this site beats pretty much any bike related reading.

Cloud? Just another word for bunch of servers sitting at a datacenter somewhere. How often Amazon Cloud services go down? If I had the choice of having my own server sitting at a location that I have access to I'd take that over any cloud.