CentOS 5 and Xen
Older Information
I've cleaned out a ton of stuff from this page. If you want to see it pre-cleaning use the link below.
The Future Of Xen & CentOS
I've spent a decent amount of time over the last 2 years learning the ins & outs of Xen virtualization and it seems, much to my chagrin, that Red Hat has decided to drop Xen support in the upcoming EL 6 release.
I'm not sure if anyone actually uses this page (it's been accessed over 4,000 times as of 14MAY11) but I'll probably not make many updates to it at all in the future. I'm either migrating to KVM or Citrix's Xen solution.
Overview
This page is completely geared towards CentOS as I'm running CentOS as the host operating system & CentOS for all of my virtual hosts.
The point of this page is to have someone who's never used Xen before to be able to have virtual machines up and running in under an hour by following the Setting Up A Xen Virtual Host section.
Please keep in mind that this isn't intended as a complete guide to using or configuring Xen and will by no means make you an expert on the subject. This will get you where you need to be quickly and enable you to learn more about Xen virtualization.
Who knows, you might even become somewhat proficient using Xen after going through this guide.
Any messages with errors, inconsistencies, omissions or improvements are more than welcome. See the Duncan page for contact information.
What Is Xen?
The Xen® hypervisor, the powerful open source industry standard for virtualization, offers a powerful, efficient, and secure feature set for virtualization of x86, x86_64, IA64, PowerPC, and other CPU architectures. It supports a wide range of guest operating systems including Windows®, Linux®, Solaris®, and various versions of the BSD operating systems.
Terminology
- Hypervisor : Xen itself, it controls all interaction between the domains and the hardware, including booting Domain0.
- Domain0 (Dom0) : The privileged GNU/Linux domain started by the hypervisor which has access to the physical machine's hardware and also has the ability to start, stop & otherwise manipulate the other domains.
- DomainU (DomU) : An unprivileged domain without access to the physical hardware. This is your standard virtualized machine and what you'll eventually install after you've got Xen running on your system.
What I'm Doing With It
I have in my possession a computer that's far more than anything I realistically need as a single machine. It's a quad core Intel Xeon X3220 @ 2.4GHz & sports a beefy 8GB of RAM.
So, what does that mean? It means that I'm going to have this machine host several virtual machines that will be everything from web servers, database servers & more.
Issues I've Had
Xen & SELinux
Ah, SELinux; You love to hate it and you... well, you probably just hate it. I came across an issue when building a new Xen Dom0 before disabling SELinux & I had this error appear :
Error: Disk isn't accessible No handlers could be found for logger "xend"
My solution was to just disable SELinux, as usual. I edited /etc/sysconfig/selinux and changed my policy to read :
SELINUX=disabled
I've found a blog post detailing a fix, which I haven't personally tried.
Network Connectivity
I was having an issue where networking would just suddenly die for all of my DomU hosts. After reading up on this on several forum (ugh) & blog (not as ugh) posts this is apparently caused by the Dom0 reaching outside of its memory zone & into the DomU.
This was remedied by updating Xen's configuration as well as the amount of memory supplied to the Dom0 during boot.
I set dom0-min-mem to 256 megabytes in xend-config.sxp :
[root@sgt1xen1 ~]# grep 256 /etc/xen/xend-config.sxp (dom0-min-mem 256)
I also set dom0_mem=256M in my Grub configuration. I haven't seen if this setting is carried over automatically during a kernel upgrade yet.
title CentOS (2.6.18-194.17.1.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-194.17.1.el5 dom0_mem=256M module /vmlinuz-2.6.18-194.17.1.el5xen ro root=LABEL=/ module /initrd-2.6.18-194.17.1.el5xen.img
Setting Up A Xen Virtual Host
Installing Xen
You'll first need to install the hypervisor as well as a kernel that will work underneath it.
yum install kernel-xen xen
Booting Xen Kernel
After installing your Xen kernel you'll need to set it as the default kernel to boot into, this is specified in /etc/grub.conf. You'll need to update the default= line to read as the Xen kernel, in this case it'll be 0 since it's the first one. Update to point at a Xen kernel if necessary & reboot.
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda2
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console
title CentOS (2.6.18-128.2.1.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.2.1.el5
module /vmlinuz-2.6.18-128.2.1.el5xen ro root=LABEL=/ rhgb quiet console=ttyS0,9600n8 console=tty1
module /initrd-2.6.18-128.2.1.el5xen.img
title CentOS (2.6.18-92.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-92.el5 ro root=LABEL=/ rhgb quiet console=ttyS0,9600n8 console=tty1
initrd /initrd-2.6.18-92.el5.img
Scripted Creation of Xen DomU Hosts
I wasn't really happy with the complexity of the standard host creation tools so I wrote a short script to build new Xen DomU hosts with minimal effort. It's basically the rest of this section boiled down to a few flags on the command line.
Here I'm creating a machine with the DomU name of lamprey, 20gb drive, 512mb RAM, MAC address of 0:16:3e:de:ad:03 & stored in /vm/lamprey.
newxen -d 20 -h lamprey -m de:ad:03 -r 512 -x /vm/lamprey
Once the installation is complete the Xen configuration files for the host will be saved in /vm/lamprey
Manually Creating Xen DomU Hosts
Grabbing Xen Install Files
For a 64-bit machine :
wget -O /tmp/centos_5.img http://mirror.centos.org/centos/5/os/x86_64/images/xen/initrd.img wget -O /tmp/centos_5 http://mirror.centos.org/centos/5/os/x86_64/images/xen/vmlinuz
For a 32-bit machine :
wget -O /tmp/centos_5.img http://mirror.centos.org/centos/5/os/i386/images/xen/initrd.img wget -O /tmp/centos_5 http://mirror.centos.org/centos/5/os/i386/images/xen/vmlinuz
Where DomU Filesystems Live
On my Dom0 system I've created a dedicated filesystem that lives under /vm/ where I keep my virtual hosts, one to a directory. Let's make one for our new host, Stilgar.
mkdir /vm/stilgar
Creating A Disk Image
Modify the red text to specify the size of the drive you want to create in gigabytes.
NOTE: this command allocates all the space for the drive image at once, it will not grow. This is to prevent disk space contention issues on the Dom0 host if multiple DomU systems start to grow out their disk images.
disk_gigabytes=20 cd /vm/stilgar dd if=/dev/zero of=xvda.img oflag=direct bs=1G count=$disk_gigabytes
Alternately we can create a drive image that will grow as our needs require up to the maximum we specify. Since we're not allocating the space we need beforehand we could come into disk space contention issues in the future if we need space that's not available.
disk_gigabytes=20 cd /vm/stilgar dd if=/dev/zero of=xvda.img oflag=direct bs=1G seek=$disk_gigabytes count=1
Xen Configuration File
This is done in two stages. The first stage is an installation & the second is the every-day configuration.
Installation Configuration
I've saved the following file as /etc/xen/stilgar. Below you'll see a few sections marked in red.
- mac= : this determines the MAC address of your virtual machine. You'll want to make sure that it's unique on your network. 00:16:3e is unique to Xen systems and has been reserved as such, you'll want to ensure that the last three aren't repeated on your local network.
- ks= : this is the location of your kickstart file, I've pointed this at the one I host on my files site. You'll want to change this to i386 if you're running a 32-bit system.
- name = : this is the name that the hypervisor knows the system as
- disk = : this is where the disk image resides
kernel = "/tmp/centos_5" ramdisk = "/tmp/centos_5.img" extra = "text ks=http://files.duncanbrown.org/linux/xen/kickstart-x86_64.ks" name = "stilgar" memory = "512" disk = [ "tap:aio:/vm/stilgar/xvda.img,xvda,w", ] vif = [ "mac=00:16:3e:01:01:01,bridge=xenbr0" ] vcpus=1 on_reboot = "destroy" on_crash = "destroy"
Kickstart Files
I keep a copy of my kickstart files available online in case anyone wants to refer to them for their own Xen installs. The CentOS mirror referenced is a very high-speed server that I've clocked > 1MB/s transfer rates with. Since it's just a minimal install this shouldn't take long at all.
Start Your Installation
The following command will create your DomU host as well as connect you to the console with -c. The installation should go smoothly, though any errors will appear in your terminal. If you've ever performed a network installation of RedHat, Fedora or CentOS there really aren't any surprises to be found. It's the text-mode installer.
xm create -c /etc/xen/stilgar
The -c flag tells Xen to connect to the console. CTRL-] will disconnect you.
Final Configuration
Now that you've finished your base installation you'll want to have your DomU start when the Dom0 host boots, this is achieved by saving a configuration file in /etc/xen/auto/. I saved the following as /etc/xen/auto/stilgar.
name = "stilgar" memory = "2048" disk = [ "tap:aio:/vm/stilgar/xvda.img,xvda,w", ] vif = [ "mac=00:16:3e:01:01:01,bridge=xenbr0" ] bootloader="/usr/bin/pygrub" vcpus=1 on_reboot = "restart" on_crash = "restart"
Controlling Virtual Hosts
The xm command allows you to control your hosts as well as connect to their consoles.
While there's alot more to xm than this, you can read more about it on it's man page.
Start a DomU instance based on a config file
xm create [DomU_config_file]
Start a DomU instance based on a config file and immediately attach to the console
xm create -c [DomU_config_file]
Show DomU hosts running
xm list
Connect to the virtual serial console of a DomU, CTRL-] to disconnect.
xm console [DomU]
Pause a DomU
xm pause [DomU]
Un-pause a DomU
xm unpause [DomU]
Shut down a DomU safely
xm shutdown [DomU]
Reboot a DomU safely
xm reboot [DomU]
Save the current condition of a DomU to a statefile
xm save [DomU] save_statefile
Restore a DomU from a statefile
xm restore save_statefile
Pull the plug on a DomU
xm destroy [DomU]
Give an overview of current running DomU systems, example display can be seen at http://files.duncanbrown.org/linux/xen/xm_top.png
xm top
Prebuilt Xen Image
This is a 64-bit CentOS 5 Xen DomU machine that I've been using as a template. The default root password is changeme. It's only a 5GB drive in order to conserve disk space. You can add additional disks.
Additional Xen Usage
DomU Display Output
VNC
VNC is a method used to access a GUI over a network. Xen has the ability to create a virtual framebuffer for remote VNC access of DomU systems.
Configuration For Dom0
If you want remote hosts to be able to access the DomU hosts over VNC you'll need to update /etc/xen/xend-config.sxp. The default in CentOS is to disable remote VNC connections, you'll need to update the vnc-listen directive.
# The interface for VNC servers to listen on. Defaults # to 127.0.0.1 To restore old 'listen everywhere' behaviour # set this to 0.0.0.0 #(vnc-listen '127.0.0.1')
I've changed mine to read as follows :
(vnc-listen '0.0.0.0')
You'll also see a vncpasswd directive, you can leave that blank or set it for a global VNC password. In the next section you'll see how you can set individual VNC passwords on individual DomU systems.
(vncpasswd 'mylamepassword')
Configuration For DomU
You'll want to set the virtual framebuffer to use VNC as well as give it a unique port and optionally a password. The example below assigns the host the VNC port of 5910 with the password mylamepassword.
If you don't supply anything for vncdisplay it will automatically assign the next available port starting at 5900.
You add this to the Xen DomU configuration on the Dom0.
vfb = [ 'type=vnc,vncdisplay=10,vncpasswd=mylamepassword' ]
Next you'll need to update /etc/grub.conf on the DomU host and remove console=xvc0.
title CentOS (2.6.18-164.el5xen)
root (hd0,0)
kernel /vmlinuz-2.6.18-164.el5xen ro root=/dev/VolGroup00/LogVol00 console=xvc0
initrd /initrd-2.6.18-164.el5xen.img
Storage
CD/DVD Drives
Add this to your disk= configuration :
'phy:/dev/cdrom,xvdb:cdrom,r'
CD/DVD Images (.iso)
If you want to have an .iso file attached on boot you'll just need to add it to your Xen configuration file for that particular DomU. I've marked what you may want to change in case /dev/hdd is already in use.
disk = [ 'tap:aio:/vm/stilgar/xvda.img,xvda,w', 'file:/path/to/your/dvd.iso,hdd:cdrom,r' ]
