Posts Tagged ‘Xen’

Citrix XenServer XS62E015 update failing to apply

Monday, June 16th, 2014

If you’re using XenServer 6.2, you may have some problems installing the XS62E015 update from CTX140808.

You go through the normal update procedure – download from the CTX140808 Citrix knowledge base article, extract it and upload the XS62E015.xsupdate file to the pool, then apply UUID c8b9d332-30e4-4e5e-9a2a-8aaae6dee91a to the pool, which promptly fails with:

The uploaded patch file is invalid. See attached log for more details.
log: error parsing patch precheck xml: expected one of these character sequence: “required”, found “error” line 4

It turns out that Citrix issued two updates for the same issue – one for Citrix XenServer 6.2 without SP1 installed and one for Citrix XenServer with SP1 installed. The slight snag is that both of these updates show up in the list of available updates in Citrix XenCentre and the knowledge base articles don’t mention that the two updates patch the same problem, but in two different version of Citrix XenServer.

If you are running Citrix XenServer 6.2 with SP1 installed, then you need to install the XS62ESP1003 update from CTX140416 (UUID c208dc56-36c2-4e91-b8d7-0246575b1828). Once XS62ESP1003 has been installed, XS62E015 will also show up in the list of installed updates.

XenConvert fails to import OVF to XenServer

Saturday, January 11th, 2014

I recently ran into a problem when trying to P2V a server with the Citrix XenConvert tool.

XenConvert would successfully create the local VHD file for the disk image as well as the OVF file for the server and then copy them to XenServer, however at the vert last hurdle (actually importing the OVF into XenCenter), it would then fail with a rather unhelpful “Failed to import the OVF Package” message.

The relevant part of the XenConvert.txt log would look something like this:

Physical to OVF Package stopped at <date/time>
Physical to OVF Package lasted <duration> seconds
Source is <path>.
Destination is <XenServer IP>.
OVF to XenServer started at <date/time>
Importing OVF Package…
Failed to import.
Failed to import the OVF Package.
OVF to XenServer stopped at <date/time>
OVF to XenServer lasted <duration> seconds
Physical to XenServer stopped at <date/time>
Physical to XenServer lasted <duration> seconds

It turns out that the OVF package has in actual fact been successfully imported in to XenServer, it’s just that for whatever reason XenCovert fails to remove the “hidden” flag from the resulting Virtual Machine.

You can see the hidden Virtual Machine in XenCenter by selecting “Hidden Objects” from the “View” menu.

In order to make the virtual machine permanently visible, you need to switch to the console and find the UUID of the virtual machine using:

xe vm-list

Once you have the UUID, you can remove the “hidden” flag from the virtual machine with the following command:

xe vm-param-remove uuid=<UUID> param-name=other-config param-key=HideFromXenCenter

Obviously you need to replace “<UUID>” with the UUID that you found in the output of the vm-list command.

You should now have a normal, fully functioning Virtual Machine which you can see without “Hidden Objects” enabled in XenCetner.

SolusVM displaying wrong disk usage statistics for Xen PV VMs

Sunday, December 29th, 2013

Recently some of our Xen PV VMs started to show strange disk space usage statistics in SolusVM – despite there being plenty of space left on the disk in the VM, SolusVM was reporting that the disk was nearly full!

I struggled to find any public information about this, but apparently it is a known problem with SolusVM and the version of the “df” utility used in RHEL/CentOS 6.5. There have been some slight changes to the way that df displays its output and this causes SolusVM to interpret the disk usage figures incorrectly.

SolusLabs have posted a workaround for this at and I’m reproducing it here for posterity as well as in the hope that it will get indexed:

Fix for Xen PV disk usage showing 100% when using a CentOS 6.5 host node

The “df” output in CentOS 6.5 has changed. You may notice this when you upgrade you’re host node from CentOS 6.4. All the Xen PV virtual servers will show 100% disk space used.

On the affected host node edit /usr/local/solusvm/data/advanced.conf and add the following line:


Then run these commands:

wget -O /usr/local/solusvm/www/command.php
wget -O /usr/local/solusvm/core/solusvmc-xen
chmod 6777 /usr/local/solusvm/core/solusvmc-xen

Now when you reboot a virtual server (with the reboot button) the disk usage should update correctly.

Citrix XenServer boot from CD/DVD

Sunday, September 8th, 2013

If you’re using Paravirtualised (PV) guest Virtual Machines on Citrix XenServer, then it’s not immediately obvious how to boot the guest from a CD/DVD should you need to. In the “Boot Options” tab of the Properties for the Virtual Machine, the “Boot Device” drop down menu will be greyed out, with no obvious way of enabling it.

The actual method of booting a Virtual Machine from a CD/DVD is a little less obvious – go to the Virtual Machine in question, switch to the console tab and select the CD/DVD from the drop down list at the top. Make sure that the Virtual Machine is shutdown, then click on the “VM” menu, then “Start/Shut down” and select “Start in Recovery Mode”. The Virtual Machine will now boot from whichever CD/DVD you’ve selected.

Missing kernel initial RAM disk with SolusVM and Xen

Thursday, April 28th, 2011

If for any reason your /boot/solus-vmlinuz symlink as well the /boot/solus-initrd.img initial RAM disk are missing or incorrect in Dom0 on one of your SolusVM Xen slaves, then you can force SolusVM to regenerate them using the latest Xen enabled copy of the kernel installed on the server using the following command in Dom0 on the slave:

php /usr/local/solusvm/includes/xenkernel.php

This not only re-creates the /boot/solus-vmlinuz symlink to the appropriate vmlinuz file, but also builds the necessary /boot/solus-initrd.img initial RAM disk to boot your DomU machines.

Of course, if you are using PyGrub then you don’t use these files in Dom0 🙂

Intel VT Virtualisation Technology on Dell PowerEdge servers

Saturday, June 19th, 2010

Somewhat annoyingly, Dell seem to like to disable Intel’s VT (Virtualisation Technology, sometimes called VMX) in the BIOS on their Dell PowerEdge servers, which means that you can’t use the Xen hypervisor to virtualise Microsoft Windows Server without changing this setting, which requires a reboot of the server to take effect.
You can use omreport from the Dell OpenManage Server Administrator software to check whether or not you have Intel Virtualisation Technology enabled.
If you haven’t got OpenManaged Server Administrator installed, then you can enable the Dell yum repository for CentOS/Red Hat systems and install it with:

wget -q -O – | bash
yum -y install srvadmin-base
/opt/dell/srvadmin/sbin/ start

Once you’ve got the Dell OpenManage Server Administrator services running, you can take a look at what processor is installed in your system and what the current BIOS settings are with:

omreport chassis processors
omreport chassis biossetup

The two attributes that you’re looking for are Processor Virtualization Technology (which needs to be enabled) and Demand-Based Power Management (which needs to be disabled).

If you need to change them, then you can do this with:

omconfig chassis biossetup attribute=cpuvt setting=enabled
omconfig chassis biossetup attribute=dbs setting=disabled
omreport chassis biossetup again and then once you’ve rebooted the server you can start taking advantage of the hardware virtualisation provided by Intel’s Virtualisation Technology.

DomU kernel panic under Xen with R1Soft CDP agent

Sunday, March 28th, 2010

I ran into a weird problem on a freshly installed server today where one of the DomU virtual machines kept locking up with a kernel panic message:

BUG: soft lockup – CPU#1 stuck for 10s!

All of the other Xen DomU virtual machines running on the box were fine, so I was pretty sure it wasn’t a hardware bug. The only possible solutions that I could come up with were

  1. My somewhat aggressive stripping down of the Dom0 host
  2. Zimbra (as the Java process was always the one referenced as being stuck
  3. The kernel module for the R1Soft CDP backup agent that I had installed a few hours ago.

I’ve run Zimbra on a CentOS 5 Xen VM before (although not 5.4 admittedly), so didn’t think that would be the problem (anyway, user land software should never be able to cause a kernel panic – in theory at least ;-))
Again, the other Xen DomU virtual machines were fine, so hopefully the configuration on the Xen Dom0 host machine itself shouldn’t be causing the problem.

Just to be safe, I updated the kernel-xen package from the default 2.6.18-164.el5 to the latest 2.6.18-164.15.1 which was only released a few days ago and doesn’t appear to be on all of the mirrors yet!
With this done and the virtual machine restarted I had to run r1soft-cki to load an updated kernel module for the R1Soft CDP backup agent and the virtual machine has been stable for several hours now. Fingers crossed it stays that way!

On the warpath

Sunday, August 3rd, 2008

Hey, guess what? In typical Edward style, it has been ages since I updated this thing and I still haven’t got the site working fully or using a custom skin. I have an excuse… I am busy/lazy (take your pick!) 😉

I got back from going to see “The Dark Night” a couple of hours ago. It’s a bloody good film but ‘m annoyed that I saw the plot twist coming right from the start – being a bit of a batman fan, I recognised the character as soon as a bloke called Harvey showed up with what correctly I guessed a double headed coin that he uses to “make his own luck” 😉

When I got back I set about unlocking Naomi’s old Sony Ericcson phone that she kindly gave me to replace my ageing Panasonic GD67 (circa 2003 and a hand me down via Matt) that does interesting things including randomly turning itself off.

6 weeks ago I requested a NUC (network unlock code) from Vodafone which they said would take between 5 days and 6 weeks to generate. So I patiently waited and when 6 weeks rolled around on Friday, I sent them a very angry e-mail telling them to pull their collective fingers out. This morning I got a reply telling me that they had generated a new code on the 24th of June (4 days after I had e-mailed them jumping through numerous hoops to prove that I hadn’t stolen the phone from Naomi) but that they “forgot” to e-mail it to me.

Incompetence or company policy not to co-operate with NUC requests? I’d go for the second one personally.

Phone unlocked, I set about learning how to use it and telling Naomi it was finally working. It seems quite nice but it’s such a change it’s going to take a lot of getting used to. At least it’s well built (unlike the Panasonic) and has fairly intuitive menus (unlike the Panasonic). It’s still not a patch on my 2G iPhone though 😉

My good mood was destroyed shortly after my unlocking success when I attempted to order a pair of power supplies from Rapid Electronics (you don’t get a link you bunch of incompetent tits!). My order was very simple, 2 power supplies delivered to one address but billed to another. Yet, despite providing a text field for your billing address, it’s not editable. At first I thought this was a browser issue so I changed from Safari to Firebox but still it didn’t work, They seem to want me to call their sales line for what is clearly a complicated procedure for them (instead of standard everywhere else) so instead I sent them an indignant e-mail telling them to learn how to build web-sites. I don’t expect that it will make the blindest bit of difference but it calmed me down somewhat.

If anyone from Rapid is reading this, the solution to your problem involves a 9mm bullet and the head of everyone in your web design department 😉

In an attempt to restore my mood (or possibly swing me ever deeper into a pit of despair) I set about the task of conducting a post-mortem on the two RouterBOARD 433AH boxes that were playing up when I tried to put them in Node4 almost a month ago.

At the time, one of the power supplies decided to fry itself and then the box running on the remaining power supply started behaving erratically. When setting it up I had been able to log into it fine but now my SSH session would hang as soon as the management shell spawned after authenticating.

Because of this we made the decision that we couldn’t trust any of the equipment and aborted the install until we had been able to examine what the problem was.

After about five minutes I managed to work out the really stupid and annoyingly simple reason for the SSH sessions hanging… there is an incompatibility between Mac OS X’s and the management shell in RouterOS version 3.x. The simple fix for this is to re-seize the Mac OS X terminal window as soon as you log in. This forces some kind of refresh which fires the RouterOS management shell back into life.

It seem that MikroTik have no intention of fixing this annoying bug and are blaming it on Apple instead 🙁

With that fixed (or at least explained), as soon a I can order some power supplies from a slightly more competent supplier than Rapid then maybe we can finally get this equipment installed.

One good thing did come out of the problems with the RouterBOARDs, I stumbled across the wiki page detailing the new support for Xen virtulization in RouterOS 3.11, a fantastic feature that I’ve been campaigning for for a long time but somehow missed the announcement for! I will have to have a play with it some time, but for now my bed beckons as tomorrow, more DIY awaits… 🙁