Archive for September, 2010

Dell PowerVault MD3000i balancing virtual disks between RAID controllers

Friday, September 17th, 2010

The Dell PowerVault MD3000i iSCSI SAN has a pair of redundant RAID controller modules designed to maximise availability of the mission critical data held in your Storage Area Network. These RAID controller modules can work in a pseudo active/active mode through a bit of a work around, however there seems to be some confusion on the Dell mailing lists and forums as to exactly what is required in order to balance the load across the two RAID controller modules.

There are references to creating two sets of both the physical disks via the disk groups (the actual RAID sets) and the virtual disks (the part of each RAID set exported as a LUNs) in order to split the load between the two RAID controller modules, as each can only be active on a single RAID controller module at once.

It turns out that it isn’t necessary to create separate disk groups in order to split the physical disks across the two RAID controller modules, as each RAID controller module has access to all of the disks in the system and both of the RAID controller modules can talk to the same disks at once.

Instead, what you should do (if possible) is two create a pair of virtual disks in your disk group (or multiple disks groups if you want to utilise different RAID levels for different virtual disks). When more than one virtual disk exists, the Dell PowerVault MD300i will automatically balance the virtual disks across the RAID controller modules in a round robin fashion in order to try and spread the read/write load between the two RAID controller modules.

Although each RAID controller module is master for it’s own virtual disk, it can still pick up the virtual disk(s) off the other RAID controller module should one fail and so still provides a redundant, highly available storage system.

You can view the details of how the virtual disks are currently distributed across the RAID controller modules by going to “Support” and then “View Storage Array Profile” in the Dell Modular Disk Storage Manager software. The “Virtual Disks” tab lists the “Preferred owner” as well as the “Current owner” for each virtual disk.

The dreaded scrolling GRUB GRUB GRUB on startup

Thursday, September 16th, 2010

Rebooting a server is always stressful, particularly when you don’t have immediate physical access to it. Of course, when the server inevitably doesn’t come back up you need to either get directly on the local console or connect in via KVMoIP and one of the worst things you can see is “GRUB GRUB GRUB” scrolling past endlessly.

This is a sign that stage 1 of the GRUB bootloader which is stored in the Master Boot Record (MBR) has somehow become corrupted and do GRUB can’t start. There is no way to even get into the GRUB command line and boot the system manually or even troubleshoot further as the problem is with stage 1 and not stage 2.

As I ran into this on a CentOS machine, I used a netinstall CD with the virtual media feature on an IP KVM attached to the server to boot into rescue mode and chroot into the operating system installed on the drive in question. I could then identify the /boot hard drive number and partition from /boot/grub/menu.lst ready to re-install GRUB and point it at the stage 2 files.

Simply run /sbin/grub to get to a version of the GRUB command prompt and then (assuming /boot/grub/menu.lst references root {hd0,0) for each of the menu options) just run:

root (hd0,0)
setup (hd0)

You should see a series of messages about looking for stage 1.5 and 2 files and then that it has successfully embedded. Congratulations, GRUB has now been re-installed and simply rebooting your machine should take you straight into your operating system as normal.

Python setuptools and get_python_version is not defined

Sunday, September 12th, 2010

If you run into the below error when using setuptools (setup.py), then it’s quite possible that you’re using an outdated version of Python’s setuptools. In particular, the python-setuptools package in the CentOS yum repository is too old.

Traceback (most recent call last):
File “setup.py”, line 19, in ?
setup(**metadata)
File “/usr/lib64/python2.4/distutils/core.py”, line 149, in setup
dist.run_commands()
File “/usr/lib64/python2.4/distutils/dist.py”, line 946, in run_commands
self.run_command(cmd)
File “/usr/lib64/python2.4/distutils/dist.py”, line 966, in run_command
cmd_obj.run()
File “/usr/lib/python2.4/site-packages/setuptools/command/bdist_rpm.py”, line 28, in run
_bdist_rpm.run(self)
File “/usr/lib64/python2.4/distutils/command/bdist_rpm.py”, line 377, in run
self.move_file(rpm, self.dist_dir)
File “/usr/lib/python2.4/site-packages/setuptools/command/bdist_rpm.py”, line 20, in move_file
getattr(self.distribution,’dist_files’,[]).append(
NameError: global name ‘get_python_version’ is not defined

Luckily, this is quite easy to fix; simply remove the RPM and download the latest version from http://pypi.python.org/pypi/setuptools then just run it with “sh” as if it was a normal shell script.