This box completely freaked out hardware-wise (while I was away on vacation, to make timing worse) and took quite a bit to fix it.

It started rebooting randomly whenever a large amount of network traffic occurred. For example, it regularly helps back up data over the network. (The data is stored on several different boxes, at different locations, so no one of them is irreplacable.) A week ago, such backups started triggering a reboot. Odd!

I suspected, of course, a hardware problem, specifically the motherboard. Less likely, perhaps, the network card. Though the motherboard was suspect #1.

Brought it by the shop for a diagnosis. For some odd reason, they decided only a Linux-friendly repair tech could perform the hardware diagnostics. And they only had one Linux-friendly repair tech. A dignostic which was supposed to be done "early this afternoon" ended up taking over a day to do. I wasn't appraised nor included in this decision, just informed of it after a significant, unexplained delay.

Sigh. Just check the hardware. Leave the OS to the rest of us.

The repair tech got around to performing the actual hardware diagnosis just minutes before closing: "I'm done. You can pick it up now." "How late are you open?" "Five more minutes." "I'm 45 minutes away." "You can get it tomorrow when we open at ten."

Grrrrr.

Anyway, they came back and said the hardware -- motherboard, memory, CPU et al -- were both hunky and dory. Right. Nonetheless, they persuaded me to do a number of things I now later regret, such as purchasing and installing a new network card and futzing endlessly with BIOS. None of these, of course, made the slightest difference.

In the end I remembered it's always safer to (secretly) regard most people as probably incompetant and wrong most the time, and decided to ignore their adamant advice that -- believe me, there is no chance the motherboard is at fault -- and replace the motherboard, as I had suspected all along.

That did the trick.

Until I decided to put back the old (vindicated, I thought) network card.

Suddenly everything went black. The system didn't boot. It didn't even beep upon startup. The power lights all went on, the fan powered up, but no response from the motherboard.

Eeek!

I put back the new network card again.

Still nothing.

I checked all the seatings, the connections, all the jumper settings. Nothing wrong as far as I could see.

Great.

Now my theory is that perhaps the old network card really was, somehow, toast, in a subtle way -- perhaps some kind of short or crossover had occurred -- that it had somehow damaged the previous board's PCI bus, and that putting it into the new board had an even more devestating effect.

Or perhaps that something awful has happened to the CPU.

I take out the network card entirely. No improvement. I even disconnect some of the auxilliary power lights and the reset switch, leaving only the on/off switch connector. Still nothing.

Back to the shop it goes. They say: "Oh, yes, this is easy. Your video card is toast." They replace the video card. No, that's not it. "Ah, we just need to reset BIOS." They pull out the battery and replace it to trigger a reset. No, not that either.

The box is so simple, it confuses them: There's nothing else to blame. No network cards now. No peripherals of any kind other than a keyboard. Only one big memory card. Ah, the Linux-friendly tech has seen this before. He knew how to get around it. We'll have him take a look at it.

When does he come in? Tomorrow.

I feel like Lileks, trying to install HDTV.

The next day, very late in the day, the Linux-friendly tech fiddles with it. Pulls things out, puts them back. Twiddles a few switch settings. Things start working again. He is unable to explain why, or what changed. He suspects a switch was set differently than before.

Hmmm... that's possible, I think: I left a number of the selectors on "default" or "auto-sense" rather than set them explicitly for the desired bus speed. Perhaps one of those was choosing a bad value.

(Of course, I could have tried that, too. But because it had gone from working to not-working, permanently, by just switching a PCI card, I thought something more significant must surely have gone wrong.)

Once again, I have to wait yet one more day to pick it up, because he has agained called me just minutes before closing.

Yeah, perhaps I just should have bought a new box. In the end that might have been simpler, hindsight being as it is.

But nothing I considered crucial to business was on this box -- just this blog and some redundant data. It itself is just a failover and backup box in case something more crucial goes wrong on another. And I'm loathe to spend another $1-2K for another rackmount server, or even $600ish for a cheap low-end box, when less than two hundred can replace the motherboard. Call me frugal.

Even if it means losing my blog for a week.

No offense to you all, but this is just a hobby.

And I could have set the blog up on another box and rerouted things, but at each point it was: Oh, just this one thing, taking less than a day, and your box will be fixed. First it was a diagnostic which should have added hours, not taken until the next day. Then it was a network card, which didn't pan out. Then it was a motherboard which did, briefly. Then another three-day visit to the tech (one to wait for him, one for him to look at it, and then a pickup on the third day.)

Of course, it shouldn't have been a week. Once I was back in town, I should have just bought a new motherboard, changed it, and been done with the matter. That would have taken only a few hours, rather than the week it ended up taking, several days of which were lost because (a) I tend to look to others more in hardware matters than software (I do not view myself as expert in the former, though perhaps I should, given that I was right on this one and the techs were deeply wrong), and (b) I was persuaded, by people I thought had more knowledge than I, to do the exact wrong thing.

Oh, and another lesson: Motherboard diagnostics are not as good nor reliable as the techs who do them think they are. There are problems they don't or won't turn up, no matter how much the repair technicians will insist to the contrary and warn you you're making a mistake otherwise. In the end, I concluded they're about as likely to be right on a given topic as the saleskids at Best Buy.

Even the final repair would have been quicker, if I hadn't assumed I had done something horribly wrong, and allowed that perhaps just a bit more fiddling might be worth a go. I'll remember that.

One of life's little lessons. Sorry for the outage. I'm back.