Tag Archives: grlug

RAID is not Backup – My Experience

It’s a common theme you read on many sysadmin forums – ‘RAID is not backup!’ I have always agreed with that statement, but it didn’t hit home until recently.

A little over a month ago, I was on site in Kentucky to switch some T1 lines around. When I got there, I noticed one of the drives on their server had failed. I requested a replacement drive from the corporate office. Since I was stuck on hold with the telco during the data line switchover, I ran a backup of the server. The next morning, the replacement drive had not arrived. I left instructions to just swap the drive out when it did show up, and started on my drive back up to Michigan, with the backup tape in my laptop case.

A few hours later, my phone rings. “Jim, I switched out the drive, and now everyone says all their files are missing.” I walk through a couple of checks, and come to the conclusion that this is pretty much the worst case scenario – one of the other RAID drives failed during the array rebuild, and took the entire array down. Worse yet, I have the only full backup tape, and I’m on the road almost halfway between Michigan and Kentucky. A long weekend was in store for me.

Fortunately, the server that went down was ‘only’ their file & print server, and not the Exchange server or only DC for that domain. Another plus was that the server was down over a (relatively slow) weekend, as opposed to the middle of the week. To work around some of the issues, DHCP services were moved to the primary router at the Kentucky site, and DNS was repointed to Michigan. Users could still access email and the terminal system. Corporate IT began building a new server in Michigan, so I could start restoring data as soon as I got back.

After 6 hours of restoring the tape, the replacement server was mostly back up and running, with users losing less than 12 hours of saved work, and no email. Printing was an issue on the new server, as it was loaded with newer drivers that caused problems for some of the older PCs.

Lesson learned: RAID is not backup. As drive capacities become larger, the likelyhood of having additional drives fail during the rebuild increases. To help work around this, build your RAID arrays with at least 1, preferably 2 hotspare drives for automatic failover, and configure your server to send email or text alerts when it detects hardware issues.

UNetbootin – Create Linux & Utility Bootable Flash Drives

Out of blank CDs? Is your burner giving you nothing but coasters?

With UNetbootin, you can download almost any of the common Linux distros or  various utilities such as NTPasswd or SystemRescueCD, all from within one small program, and create a bootable USB flash drive. It also works with any GRUB or isolinux based CD image.

Not only does it automatically download the distro of your choice, but it also provides version history, and builds for x86 or x64.

Cheese with that Whine?

If you are reading the blog of a Microsoft developer, you shouldn’t be surprised when he posts examples using Bing. While I’m all for open source, and convincing people to switch, an MS blog probably isn’t the place to complain that Bing throws a warning in Konquerer and that Google Maps “just works”, especially when Bing works fine in Firefox, and the only reason you’re seeing that warning is because you are using a somewhat niche browser (yes, I realize that it’s essentially Webkit, and that if Chrome and Safari work, then it should, too).

As he responds to one of the comments – “If I link to Google, people scold me for not linking to Bing. And now, if I link to Bing, people scold me for not linking to Google. …  I guess the only safe thing is to not link to anything.”


Sony Drops Linux Support on Slim PS3

With the release of the PS3 Slim, Sony has announced that the new SKU will no longer support the ‘Other OS’ install option. It is going the way of PS2 support before it, and being cut in the name of cost savings. Sure, Linux on the PS3 was crippled under a hypervisor which prevented direct access to the GPU, but it still marked a step in the right direction for opening up a nice hardware platform to the masses for something besides games & movies.

This change, fortunately, does not affect existing PS3s, but raises the question – What if one day for some reason, Sony just decided that they aren’t including the hypervisor anymore in firmware updates? Would those of us with Linux installed have any way of pulling that data off the drive?

Ubuntu Media Center PC

I finally had it with Vista on my media center PC. Slow startups, it never saved my resolution for the TV correctly, and frequently locked up while loading videos. It’d been a while since I tried Ubuntu (normally I use Fedora), so I decided to load up Jaunty and see how it worked.

First, the bad. It did not load the accelerated drivers for the Intel onboard video. While it sucks for 3D, it was more than enough to run XBMC smoothly under Windows. Under Ubuntu, videos would play fine, the interface was just dog-slow. I could’ve researched and fixed the problem, then I remembered I had an old GeForce 4 that Vista didn’t support. I slapped that in there, loaded up the nVidia drivers, and XBMC ran smooth as butter.

Now, the good. Wireless. Freaking amazing, I did not expect it to work right after an install with a Linksys WUSB54GSC I had, but after logging in, I was presented with a list of wifi networks, and connected to mine with no problems whatsoever.

There was some Samba weirdness. For some reason, it initially only showed my other Windows PCs, and my OpenFiler server didn’t show up in the Network list at all, even though they were all in the same workgroup. Changing my workgroup settings seemed to fix that.

On the whole, I’m very satisfied with how XBMC on Ubuntu turned out to be.

Rebuilding Software RAID5 on an Atom

The replacement drive for my 1TB RAID5 finally came in last night. I had never actually had to do a rebuild with the md toolset before (I’ve always had hardware support in the past). I wasn’t able to find any rebuild option in OpenFiler’s web interface, so I ssh’ed in.

First, I looked at the partition structure of one of the remaining working drives with fdisk -l /dev/sdd, then I matched this partition setup on the replacement drive. Then I added the new drive back into the array with mdadm /dev/md1 -a /dev/sde1, and monitored the rebuild process occasionally by looking at /proc/mdstat.

The total time for the array rebuild was just shy of 4 hours and 30 minutes, on an Atom N270.

Hacking the Panda GateDefender 8100

The Panda GateDefender 8100 is a Linux-based filtering, AV, and intrusion prevention system.  The system I have to work with has a P4 3.4GHz CPU, 1GB of DDR400, and an 80GB SATA drive.  It has one available PCI slot, 2 unused SATA ports, and a CF slot.

While you can get a display on the VGA connector, its not an actual console. To get a console, you’ll need a serial cable. Open up a terminal emulator and set the port to 57600,8,N,1, then power on the machine. To get into the BIOS, press the tab key when prompted.

If you look at the motherboard, you’ll see JP1 is labeled CMOS Reset. I’ve tried this, and it did not seem to actually reset anything, nor did JP8 (NMI) or JP2 (???). I used a tool called CmosPwd to recover the actual password (its ‘adnap17’).  Once in the BIOS, you can move on to getting more access to the box.

Continue reading