Archive for March, 2010

DomU kernel panic under Xen with R1Soft CDP agent

Sunday, March 28th, 2010

I ran into a weird problem on a freshly installed server today where one of the DomU virtual machines kept locking up with a kernel panic message:

BUG: soft lockup – CPU#1 stuck for 10s!

All of the other Xen DomU virtual machines running on the box were fine, so I was pretty sure it wasn’t a hardware bug. The only possible solutions that I could come up with were

  1. My somewhat aggressive stripping down of the Dom0 host
  2. Zimbra (as the Java process was always the one referenced as being stuck
  3. The kernel module for the R1Soft CDP backup agent that I had installed a few hours ago.

I’ve run Zimbra on a CentOS 5 Xen VM before (although not 5.4 admittedly), so didn’t think that would be the problem (anyway, user land software should never be able to cause a kernel panic – in theory at least ;-))
Again, the other Xen DomU virtual machines were fine, so hopefully the configuration on the Xen Dom0 host machine itself shouldn’t be causing the problem.

Just to be safe, I updated the kernel-xen package from the default 2.6.18-164.el5 to the latest 2.6.18-164.15.1 which was only released a few days ago and doesn’t appear to be on all of the mirrors yet!
With this done and the virtual machine restarted I had to run r1soft-cki to load an updated kernel module for the R1Soft CDP backup agent and the virtual machine has been stable for several hours now. Fingers crossed it stays that way!

Weirdest Linux command ever

Thursday, March 25th, 2010

This morning I had to run possibly the most bizzare Linux command ever whilst stripping down a CentOS server:

yum remove troursers

That really brightened up my day and made me smile. 🙂

Activity Monitor in Snow Leopard

Thursday, March 18th, 2010

If you’re anything like me, then you like to keep your computer nice and neat and organised; this includes sorting all of my applications into categories so that they don’t clutter the place up in one big list.

In Mac OS X 10.6 (aka Snow Leopard), this presents a bit of a problem as Activity Monitor can no long be moved as the path to activitymonitord is now hard coded for some reason. If you do move it, then Activity Monitor appears to start and then just hangs.

If you fire up the Console application to take a look at the syslog, then you’ll see messages along the lines of:

18/03/2010 22:20:34 com.apple.launchd[1] (com.apple.ActivityMonitor[1716]) posix_spawn(“/Applications/Utilities/Activity Monitor.app/Contents/MacOS/activitymonitord”, …): No such file or directory
18/03/2010 22:20:34 com.apple.launchd[1] (com.apple.ActivityMonitor[1716]) Exited with exit code: 1
18/03/2010 22:20:34 com.apple.launchd[1] (com.apple.ActivityMonitor) Throttling respawn: Will start in 10 seconds

There is a similar problem with the System Preferences application if you are trying to install custom preference panes such as Growl where the install window will hang with similar looking console messages:

18/03/2010 22:50:46 com.apple.launchd[1] (com.apple.systempreferences.install) Throttling respawn: Will start in 10 seconds
18/03/2010 22:50:56 com.apple.launchd[1] (com.apple.systempreferences.install[2390]) posix_spawn(“/Applications/System Preferences.app/Contents/Resources/installAssistant”, …): No such file or directory
18/03/2010 22:50:56 com.apple.launchd[1] (com.apple.systempreferences.install[2390]) Exited with exit code: 1

This is only a problem when installing new preference panes, and the System Preferences will work fine normally when moved.

Hopefully this will save someone the headache of trying to diagnose this. I was on the verge of doing a re-install, having only just installed OS X in the first place and started moving everything to be how I like it!

cPanel/WHM and yum-updatesd

Monday, March 15th, 2010

In my continuing fight with yum-updatesd, I found that on servers with cPanel/WHM installed it was crashing with mysterious Python errors:

root@tma03 [/etc/yum]# yum-updatesd –debug –no-fork
Traceback (most recent call last):
File “/usr/sbin/yum-updatesd”, line 35, in ?
import dbus
File “/usr/lib64/python2.4/site-packages/dbus/__init__.py”, line 1, in ?
from _dbus import *
File “/usr/lib64/python2.4/site-packages/dbus/_dbus.py”, line 48, in ?
from proxies import *
File “/usr/lib64/python2.4/site-packages/dbus/proxies.py”, line 2, in ?
import introspect_parser
File “/usr/lib64/python2.4/site-packages/dbus/introspect_parser.py”, line 1, in ?
import libxml2
File “/usr/lib64/python2.4/site-packages/libxml2.py”, line 215
pass
^
TabError: inconsistent use of tabs and spaces in indentation

In the end, I fixed this by forcing a re-install of the dbus-python and libxml2-python RPMs from the official CentOS mirrors. In my case, this was:

rpm –force -hUv http://mirrors.freethought-internet.co.uk/centos/5/os/x86_64/CentOS/dbus-python-0.70-7.el5.x86_64.rpm http://mirrors.freethought-internet.co.uk/centos/5/os/x86_64/CentOS/libxml2-python-2.6.26-2.1.2.8.x86_64.rpm

Adjust using your mirrors, distribution, version, architecture and RPM revision as appropriate.
Also, always remember to check for updates after manually re-installing RPMs from the OS repository.

I have no idea what cPanel/WHM does to break Python like this or if updates to cPanel are going to break yum-updatesd again in the future, but I have fixed this on several cPanel 11.25 boxes now!
I know cPanel likes to mess with system files, but these RPMs aren’t in the exclude list that the cPanel installer adds to /etc/yum.conf and forcing a re-installation of them seems to fix yum-updatesd.

E-mail notifications from yum-updatesd

Monday, March 15th, 2010

After wrestling with yum-updatesd in a rather confused manner for some time, I finally discovered why it wasn’t e-mailing me notifications of updates; it needs the dbus service (messagebus) to be running, even if you are not using emit_via=dbus and with dbus_listener=false.

With the messagebus service running so that yum-updatesd could connect to the dbus service it didn’t need to use, it is just a case of setting emit_via=email and then adjusting the email_from, email_to and smtp_server settings in /etc/yum/yum-updatesd.conf appropriately.

WHMCS v4.2.1 and Google Checkout

Monday, March 15th, 2010

The Google Checkout module has been updated in the new v4.2.1 version of WHMCS and the release notes have a slight omission; the Google Checkout module no longer works out of the box with the default settings in your Google Checkout account.

In order to fix this, go to Settings and then Integration and then under “Shopping cart post security” untick the box next to “My company will only post digitally signed carts”.

According to WHMCS, this is due to this feature causing problems. It is probably worth reading the Google Checkout knowledge base article about the potential security holes that disabling this feature will introduce.

WHMCS will check the amount on the XML Google send back that it uses to mark the appropriate invoice(s) as paid, so you won’t need to manually verify the details as the knowledge base article suggests; if someone fiddles with the data sent you send to Google from your invoice screen then the invoice will only be marked as partially paid.

@mail in Plesk 9.3

Friday, March 5th, 2010

I recently upgraded a server to Plesk 9.3 only to find that it broken @mail. The list of available webmail clients no longer has @mail in it and trying to remove the psa-atmail RPM failed with a script error.

In the end, I had to force the removal of the RPM with the –noscripts option and then go and manually delete /var/www/atmail as well as any entries in /etc/psa-webmail/atmail, /etc/psa-webmail/atmailcom, /etc/psa/webmail/atmail and /etc/psa/webmail/atmailcom before I could get the RPM to re-install correctly.
Even then, the RPM install failed with “Unable to get options for atmail webmail” and I had to run “/usr/local/psa/admin/bin/webmailmng –install –name=atmail” to get it back in to the list of available webmail clients.

It seems that this has all come about with Parallels moving the webmail config files from /etc/psa-webmail/ to /etc/psa/webmail/ and botching up the RPMs somehow.