CentOS 6.3下安装KVM虚拟机案例

在我决定自己在CentOS 6.3下动手安装kvm虚拟机的时候，我最先去的官网找安装的步骤，没花多大力气就找到了《Getting KVM to run on your machine》这篇文章，按照上面的步骤开始安装。根据这篇文章，安装kvm虚拟机并运行，只需要以下三个步骤：

    1. /usr/local/kvm/bin/qemu-img create -f qcow2 vdisk.img 10G
    2./usr/local/kvm/bin/qemu-system-x86_64 -hda vdisk.img -cdrom /path/to/boot-media.iso -boot d  -m 384
    3./usr/local/kvm/bin/qemu-system-x86_64 vdisk.img -m 384

很多人在执行第1个步骤的时候，都会很顺利，不会遇到问题。大多数情况下，都会卡在第2个步骤上。在执行第2个步骤的时候，遇到的第一个问题是找不到qemu-system-x86_64命令；第二个问题就是看到”VNC server running on `::1:5900’“这个提示，google了半天也不行。

首先来说第一个问题，qemu-system-x86_64是在安装qemu（注意不是qemu-kvm）时生成的命令，而CentOS下默认安装的是qemu-kvm包，对应的命令是qemu-kvm。这个信息在上面提到的文章中也会说明，但是接着遇到的问题就是找不到qemu-kvm这个命令。qemu-kvm这个命令在/usr/libexec/目录下。对红帽系列系统比较熟的话，很容易找到qemu-kvm这个命令所在的目录，只需要通过查看rpm -ql qemu-kvm的输出即可，以后遇到类似的问题，也可以通过rpm -ql命令找到。

如果你是在桌面环境下的话，直接执行命令”vncviewer :5900“就可以继续安装过程，如果你在远程ssh连接的shell执行vncviewer命令的话，会报下面的错误：

TigerVNC Viewer for X version 1.1.0 – built Apr 29 2013 11:33:36
Copyright (C) 1999-2011 TigerVNC Team and many others (see README.txt)
See http://www.tigervnc.org for information on TigerVNC.
vncviewer: unable to open display “”

因为vncviewer需要在桌面环境下执行。

另一种方式就是在执行qemu-kvm命令的时候，加上“-vnc :0”这样就不会有这个提示了，你可以利用tightvnc这样的工具来连接到5900端口继续安装过程，这种情况的话系统不需要图形界面就可以了。

当然还有些人可能使用的方式中要在服务器段配置vncserver，这种情况的下，CentOS需要安装的rpm包为tigervnc和tigervnc-server，其中vncviewer这个命令就位于tigervnc包中。具体的安装过程参见下面两篇文章：

http://wiki.centos.org/HowTos/VNC-Server#head-76401321dae4d80916a7fd7e710272a9b85c9485

http://www.linuxidc.com/Linux/2013-04/82510.htm

在启动vncserver服务的时候，你可能遇到下面的问题：

WARNING: The first attempt to start Xvnc failed, possibly because the font
catalog is not properly configured.  Attempting to determine an appropriate
font path for this system and restart Xvnc using that font path …
Could not start Xvnc.

/usr/bin/Xvnc: symbol lookup error: /usr/bin/Xvnc: undefined symbol: pixman_composite_trapezoids
/usr/bin/Xvnc: symbol lookup error: /usr/bin/Xvnc: undefined symbol: pixman_composite_trapezoids

解决这个问题，只需要执行下面的命令即可：

yum install pixman pixman-devel libXfont

远程连接vncserver的工具我用的是tightvnc，这个工具是免费的，非常好用。

更多CentOS相关信息见CentOS 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=14

Getting KVM to run on your machine

You will need (see download section):

• kvm-release.tar.gz

• A VT capable Intel processor, or an SVM capable AMD processor

• qemu prerequisites:

  o zlib libraries and headers o SDL libraries and headers o alsa libraries and headers (optional alsa support: disabled by default but can be enabled with –enable-alsa) o gnutls libraries and headers (optional VNC TLS support: enabled by default but can be disabled with –disable-vnc-tls) o kernel headers (on Fedora, the kernel-devel package)

On a debian etch system you can install the prerequisites with:

apt-get install gcc libsdl1.2-dev zlib1g-dev libasound2-dev linux-kernel-headers pkg-config libgnutls-dev libpci-dev

Note: When building from git, you also need gawk.

Please report problems (and successes) to the mailing list. [edit] Unpacking and configuring kvm components

You may wish to take a look at the [“Kernel-optimizations”] page. There exists a [attachment:kvm-26-alt-grab.diff.gz patch] which will change the SDL keygrab combination from ctrl-alt to ctrl-alt-shift. It was written primarily to deal with the heavy use of ctrl-alt-delete in NT-based VMs.

If you are using a patched kernel (e.g. a recent -mm kernel or the kvm git tree), configure the kernel normally, boot into it, and:

tar xzf kvm-release.tar.gz
cd kvm-release
./configure --prefix=/usr/local/kvm --with-patched-kernel
make
sudo make install
sudo /sbin/modprobe kvm-intel
# or: sudo /sbin/modprobe kvm-amd

If you’re not running a patched kernel:

tar xzf kvm-release.tar.gz
cd kvm-release
./configure --prefix=/usr/local/kvm
make
sudo make install
sudo /sbin/modprobe kvm-intel
# or: sudo /sbin/modprobe kvm-amd

Note: if sound doesn’t play in the guest vm you can add –audio-drv-list=”alsa oss” to ./configure as explained in http://www.linux-kvm.com/content/sound-problem-solved

Creating a disk image for the guest:

/usr/local/kvm/bin/qemu-img create -f qcow vdisk.img 10G

Installing a guest operating system:

 sudo /usr/local/kvm/bin/qemu-system-x86_64 -hda vdisk.img -cdrom /path/to/boot-media.iso -boot d  -m 384

(kvm doesn’t make a distinction between i386 and x86_64 so even in i386 you should use qemu-system-x86_64)BR

If you have less than 1GB of memory don’t use the -m 384 flag (which allocates 384 MB of RAM for the guest). For computers with 512MB of RAM it’s safe to use -m 192, or even -m 128 (the default)

Running the newly-installed guest:

sudo /usr/local/kvm/bin/qemu-system-x86_64 vdisk.img -m 384

or a slightly more complicated example, where it is assumed that bridged networking is available on tap0; see [“Kernel-optimizations”] for some setup hints:

/usr/local/kvm/bin/qemu-system-x86_64 -hda xp-curr.img -m 512 -soundhw es1370 -no-acpi -snapshot -localtime -boot c -usb -usbdevice tablet -net nic,vlan=0,macaddr=00:00:10:52:37:48 -net tap,vlan=0,ifname=tap0,script=no

(kvm doesn’t make a distinction between i386 and x86_64 so even in i386 you should use qemu-system-x86_64)

If you’re on Debian Etch, substitute kvm for qemu-system-x86_64 (thanks to fromport, soren and mael_). See also the entries under the label “Ubuntu” on the HOWTO page. qemu-system-x86_64

If you’re on Fedora/RHEL/CentOS (and installed a kvm package and not built kvm yourself from source) then substitute qemu-kvm for qemu-system-x86_64

choose the right kvm & kernel version

Three Components

To make it work, you need to get the right version for three components:

• Linux Module
• User Space Application
• Guest Virtio Driver

[edit] Linux Module

KVM come with a linux module to support full virtualization. The linux module is just three files: kvm.ko, kvm_intel.ko, kvm_amd.ko. You can install them just like you install drivers for your video card. The good news is, you might do not need it install it. The 2.6.20 version has already included those linux modules. Depends on your distribution configuration, it might not be installed, or as a module or built-in. Here is a table listing the relation:

• 2.6.20 kvm-12
• 2.6.21 kvm-17
• 2.6.22 kvm-22

If you are not sure your linux distribution contain it or not, use this command:

modprobe -l | grep kvm

The linux module can be built from source code. This is also the recommanded way to get the right version of linux module. Compiling from the source code, and than make install should make the linux module inserted into your /lib/modules/linux-uname -r. Simply make it in use by:

modprobe kvm
modprobe kvm_intel
or
modprobe kvm
modprobe kvm_amd

User space application

Compile from source code, you can get it. Otherwise, refer to the previous section.

Guest virtio driver

There was no special requirement for guest operating system if you are not using para-visualized disk ornetwork adapter. If you are using them, make sure you get virtio_pci.ko, virtio_rng.ko, virtio_blk.ko, virtio_net.ko. They are in 2.6.25 or later kernel. There is also a option to backport them.

Refer to Virtio for more information

How To Migrate From Vmware To KVM

The vmware system consists of two disks in raw format: the old boot disk and the second one. It is Windows 2000 Server guest OS.

1. Create empty new boot image (use dd).
2. Boot stand alone OS from any other disk with old boot disk and new boot image connected (that is at least 3 disks).
3. Sign up, make partition and format the new disk (do not make volume).
4. xcopy /e /c /r /h /k /o /x /y old_boot_disk:new_boot_disk:
5. Copy boot.ini from new boot disk to the second one (that is not boot disk).
6. Edit boot.ini on the second disk to boot from rdisk(1)
7. Swap disks: new <-> second.
8. Boot setup from CD and go on Repair Procedure (after licence agreement) for the new boot disk.
9. fdisk new boot image and toggle Boot flag.

System Message: WARNING/2 (data/kvm-howto.txt, line 141)

Block quote ends without a blank line; unexpected unindent.

10. Swap disks back: second <-> new.
11. Boot from the new disk.
12. Remove vmware tools and devices.
13. Reboot and be happy.

That’s all.

16. 19. To swap disks and edit boot.ini (items 5,6,7,10) should be excluded if the boot disk is named C:, that is right in most cases. Those items are for strange case, when boot disk is D: and second one is C:.

Setting guest network

Guest (VM) networking in kvm is the same as in qemu, so it is possible to refer to other documentations about networking for qemu. This page will try to explain how to configure the most frequent types of network needed. [edit] User Networking

Use case:

• You want a simple way for your virtual machine to access to the host, to the internet or to resources available on your local network.
• You don’t need to access your guest from the network or from another guest.
• You are ready to take a huge performance hit.
• Warning: User networking does not support a number of networking features like ICMP. Certain applications (like ping) may not function properly.

Prerequisites:

• You need kvm up and running
• If you don’t want to run as root, the user you want to use needs to have rw access to /dev/kvm
• If you want to be able to access the internet or a local network, your host system must be able to access the internet or the local network

Solution:

• simply run your guest with “-net nic -net user”, e-g:

  System Message: WARNING/2 (data/kvm-howto.txt, line 175)

  Literal block expected; none found.

qemu-system-x86_64 -hda /path/to/hda.img -net nic -net user

Notes:

• The IP address can be automatically assigned to the guest thanks to the DHCP service integrated in QEMU
• If you run multiple guests on the host, you don’t need to specify a different MAC address for each guest
• You can still access one specific port on the guest using the “-redir” option.This means e.g. if you want to transport a file with scp from host to guest, start the guest with “-net nic -net user -redir tcp:5555::22”. Now you redirecting the host port 5555 to the guest port 22. After starting up the guest, you can transport a file with e.g. “scp -P 5555 file.txt root@host-ip:/tmp” from host to guest. Remember that you must use the ip address of your host to connect to the guest, because you are redirecting a local port to the guest.

private virtual bridge

Use case:

• You want to set up a private network between 2 or more virtual machines. This network won’t be seen from the other virtual machines nor from the real network.

Prerequisites:

• You need kvm up and running

• If you don’t want to run as root, the user you want to use needs to have rw access to /dev/kvm

• You need the following commands installed on your system, and if you don’t want to run as root, the user you want to use needs to be able to sudo the following command:

  System Message: WARNING/2 (data/kvm-howto.txt, line 195)

  Literal block expected; none found.

/sbin/ip /usr/sbin/brctl /usr/sbin/tunctl

Solution:

• You need to create a bridge, e-g:

  sudo /usr/sbin/brctl addbr br0
  

• You need a qemu-ifup script containing the following:

  #!/bin/sh
  set -x
  
  switch=br0
  
  if [ -n "$1" ];then
          /usr/bin/sudo /usr/sbin/tunctl -u `whoami` -t $1
          /usr/bin/sudo /sbin/ip link set $1 up
          sleep 0.5s
          /usr/bin/sudo /usr/sbin/brctl addif $switch $1
          exit 0
  else
          echo "Error: no interface specified"
          exit 1
  fi
  

• Generate a MAC address, either manually or using:

  #!/bin/sh
  # generate a random mac address for the qemu nic
  # shell script borrowed from user pheldens @ qemu forum
  echo $(echo -n DE:AD:BE:EF ; for i in `seq 1 2` ;
  do echo -n `echo ":$RANDOM$RANDOM" | cut -n -c -3` ;done)
  

• Run each guest with the following, replacing $macaddress with the value from the previous step:

  qemu-system-x86_64 -hda /path/to/hda.img -net nic,macaddr=$macaddress -net tap
  

Notes:

• If you don’t want to run as root, the qemu-ifup must be executable by the user you want to use

• You can either create a system-wide qemu-ifup in /etc/qemu-ifup or use another one. In the latter case, run:

  qemu-system-x86_64 -hda /path/to/hda.img -net nic,macaddr=$macaddress -net tap,script=/path/to/qemu-ifup
  

• Each guest on the private virtual network must have a different MAC address

public bridge

WARNING: The here shown method, will not work with most(all?) wireless drivers, as these do not support bridging.

Use case:

• You want to assign an IP address to your virtual machines and make them accessible from your local network
• You also want performance out of your virtual machine.

Prerequisites:

• You need kvm up and running

• If you don’t want to run as root, the user you want to use needs to have rw access to /dev/kvm

• You need the following commands installed on your system, and if you don’t want to run as root, the user you want to use needs to be able to sudo the following command:

  System Message: WARNING/2 (data/kvm-howto.txt, line 260)

  Literal block expected; none found.

/sbin/ip /usr/sbin/brctl /usr/sbin/tunctl

• Your host system must be able to access the internet or the local network

Solution 1: using distro sysconfig script

• Edit /etc/sysconfig/network-scripts/ifcfg-eth0

  o comment out BOOTPROTO o Add BRIDGE=switch

• Create /etc/sysconfig/network-scripts/ifcfg-br0

  o The content should be:

  System Message: WARNING/2 (data/kvm-howto.txt, line 274)

  Literal block expected; none found.

  DEVICE=switch BOOTPROTO=dhcp ONBOOT=yes TYPE=Bridge

• /etc/init.d/network restart

• The bridge br0 should get the ip address (either static/dhcp) while the physical eth0 is left without ip address.

Solution 2: manual

• You need to create a bridge, e-g:

  sudo /usr/sbin/brctl addbr br0
  

• Add one of your physical interface to the bridge, e-g for eth0:

  sudo /usr/sbin/brctl  addif br0 eth0
  

• You need a qemu-ifup script containing the following:

  #!/bin/sh
  set -x
  
  switch=br0
  
  if [ -n "$1" ];then
          /usr/bin/sudo /usr/sbin/tunctl -u `whoami` -t $1
          /usr/bin/sudo /sbin/ip link set $1 up
          sleep 0.5s
          /usr/bin/sudo /usr/sbin/brctl addif $switch $1
          exit 0
  else
          echo "Error: no interface specified"
          exit 1
  fi
  

• Generate a MAC address, either manually or using:

  #!/bin/sh
  # generate a random mac address for the qemu nic
  # shell script borrowed from user pheldens @ qemu forum
  echo $(echo -n DE:AD:BE:EF ; for i in `seq 1 2` ;
  do echo -n `echo ":$RANDOM$RANDOM" | cut -n -c -3` ;done)
  

• Run each guest with the following, replacing $macaddress with the value from the previous step:

  qemu-system-x86_64 -hda /path/to/hda.img -net nic,macaddr=$macaddress -net tap
  

Notes:

• If you don’t want to run as root, the qemu-ifup must be executable by the user you want to use

• You can either create a system-wide qemu-ifup in /etc/qemu-ifup or use another one. In the latter case, run:

  qemu-system-x86_64 -hda /path/to/hda.img -net nic,macaddr=$macaddress -net tap,script=/path/to/qemu-ifup
  

• Each guest on the network must have a different MAC address:

  iptables
  

you can also connect your guest vm to a tap in your host. then setting iptables rules in your host to become a router + firewall for your vm. vde

another option is using vde (virtual distributed ethernet). performance

Data on benchmarking results should go in here. There’s now a page dedicated to ideas for improving Networking Performance.

set up a network console

To set up a network console, add the following command to /etc/rc.d/rc.local:

/sbin/modprobe netconsole netconsole=@/eth0,12345@10.0.0.1/00:E0:81:2B:0C:C1

Where the mac address is the destination nc listener and 12345 is the port number for listenning.

To run the log client, issue the command:

nc -dul 12345

on the client machine, 12345 is the port number from above.

It also helps to disable sync logging by changing /etc/syslog.conf from:

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

to:

*.info;mail.none;authpriv.none;cron.none                -/var/log/messages

and to add:

echo 9 > /proc/sysrq-trigger

to /etc/rc.d/rc/local

How to assign devices with VT-d in KVM

Assigning device to guest

1. Modifying kernel config:

   • make menuconfig
   • set “Bus options (PCI etc.)” -> “Support for DMA Remapping Devices” to “*”
   • set “Bus options (PCI etc.)” -> “Enable DMA Remapping Devices” to “*”
   • set “Bus options (PCI etc.)” -> “Support for Interrupt Remapping” to “*”
   • set “Bus options (PCI etc.)” -> “PCI Stub driver” to “*”
   • exit/save

2. build kernel:

   • make
   • make modules_install
   • make install

3. reboot

4. unbind device from host kernel driver (example PCI device 01:00.0)

   • lspci -n

   • locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9

     ...
     01:00.0 0200: 8086:10b9 (rev 06)
     ...
     

   • echo “8086 10b9” > /sys/bus/pci/drivers/pci-stub/new_id

   • echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind

   • echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

5. load KVM modules:

   • modprobe kvm
   • modprobe kvm-intel

6. assign device:

   • /usr/local/bin/qemu-system-x86_64 -m 512 -boot c -net none -hda /root/ia32e_rhel5u1.img -pcidevice host=01:00.0

VT-d device hotplug

KVM also supports hotplug devices with VT-d to guest. In guest command interface (you can press Ctrl+Alt+2 to enter it), you can use following command to hot add/remove devices to/from guest:

• hot add:

  pci_add pci_addr=auto host host=01:00.0

• hot remove (e.g bdf is 00:06.0 in guest):

  pci_del pci_addr=6

Notes

• VT-d spec specifies that all conventional PCI devices behind a PCIe-to-PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
• If the device doesn’t support MSI, and it shares IRQ with other devices, then it cannot be assigned due to host irq sharing for assigned devices is not supported. You will get warning message when you assign it. Notice this also apply to the devices which only support MSI-X.

Changing disks in the cdrom drive

Qemu provides a way to change iso in the cdrom via the monitor interface.

QEMU 0.10.5 monitor – type ‘help’ for more information (qemu)

The commands you’ll want to use, info block, eject, and change. First we need to determine which block device is the cdrom you are interested in. Issue the info block command and look for cdrom devices.

(qemu) info block
ide0-hd0: type=hd removable=0 file=/dev/null ro=0 drv=host_device encrypted=0
ide1-cd0: type=cdrom removable=1 locked=0 [not inserted]
floppy0: type=floppy removable=1 locked=0 [not inserted]
sd0: type=floppy removable=1 locked=0 [not inserted]

ide1-cd0 is the only cdrom device in this example and there isn’t any media inserted. To change the cdrom, we issue the change command supplying the device name (ide1-cd0) and the path to the new iso file.

(qemu) change ide1-cd0 /tmp/dsl-4.4.10.iso
(qemu) info block
ide0-hd0: type=hd removable=0 file=/dev/null ro=0 drv=host_device encrypted=0
ide1-cd0: type=cdrom removable=1 locked=0 file=/tmp/dsl-4.4.10.iso ro=0  drv=raw encrypted=0
floppy0: type=floppy removable=1 locked=0 [not inserted]
sd0: type=floppy removable=1 locked=0 [not inserted]
(qemu)

In case you already have a cdrom in the drive you need to eject the current iso first before issuing the change command. The eject command takes the name of the cdrom block device.

(qemu) eject ide1-cd0
(qemu) info block
ide0-hd0: type=hd removable=0 file=/dev/null ro=0 drv=host_device encrypted=0
ide1-cd0: type=cdrom removable=1 locked=0 [not inserted]
floppy0: type=floppy removable=1 locked=0 [not inserted]
sd0: type=floppy removable=1 locked=0 [not inserted]
(qemu) change ide1-cd0 /tmp/fedora11.iso
(qemu) info block
ide0-hd0: type=hd removable=0 file=/dev/null ro=0 drv=host_device encrypted=0
ide1-cd0: type=cdrom removable=1 locked=0 file=/tmp/fedora11.iso ro=0 drv=raw  encrypted=0
floppy0: type=floppy removable=1 locked=0 [not inserted]
sd0: type=floppy removable=1 locked=0 [not inserted]

use virtio

Using virtio_net For The Guest NIC

The following is done using Debian Lenny, with the 2.6.25 kernel installed from Sid.

• install the guest OS as per normal, using rtl8139 or e1000 for the guest NIC
• boot into the guest as per normal
• edit /etc/apt/sources.list to add a sid repo
• install the 2.6.25 kernel
• boot into the guest using the 2.6.25 kernel
• edit /etc/initramfs-tools/modules and add virtio, virtio_pci, virtio_ring, virtio_net, virtio_blk
• update the initramfs using update-initramfs -u
• shutdown the guest
• change the -net nic option to include model=virtio
• boot the guest

Throughput Tests Using iperf

To see what the throughput differences would be like, I ran a bunch of iperf tests from a Debian guest. Host Server Config

• Tyan h2000M motherboard
• 2x dual-core Opteron 2220 CPUs @ 2.8 GHz
• 8 GB DDR2-667 ECC SDRAM (2x 2 GB sticks per CPU)
• 3Ware 9650SE-16ML SATA-II RAID Controller
• 12x 500 GB SATA-II harddrives in a single RAID-6 array
• Intel PRO/1000-MT PCIe quad-port gigabit NIC (configured as 4-port bond0, used as the kvm bridge)
• Debian Lenny installed, updated June 04, 2008, with kvm-69 and kernel 2.6.25 (kernel from Sid)

Guest VM Config

• 1 CPU
• 2 GB RAM
• 100 GB virtual harddrive (via LVM)
• Debian Lenny with kernel 2.6.25 from Sid

System Message: WARNING/2 (data/kvm-howto.txt, line 524)

Bullet list ends without a blank line; unexpected unindent.

/usr/bin/kvm -name mail -daemonize -localtime -usb -usbdevice tablet -smp 1 -m 1048 -vnc :04 -pidfile /var/run/kvm/mail.pid -net nic,macaddr=00:16:3e:00:00:04,model=virtio -net tap,ifname=tap04 -boot c -drive index=0,media=disk,if=virtio,boot=on,file=/dev/mapper/vol0-mail

iperf Server Config

iperf -s -w 65536 -p 12345 -I 5

iperf Client Configs

* [1] iperf -c <server> -w 65536 -p 12345 -t 60
* [2] iperf -c <server> -w 65536 -p 12345 -t 60 -d
* [3] iperf -c <server> -w 65536 -p 12345 -t 60 -P 4

Results

These are averaged over 3 runs.

To the host (virtio)

• [1] 92 Mbps

To a server connected to a gigabit port on the same switch (virtio)

• [1] 834 Mbps
• [2] 519 Mbps out, 531 Mbps in
• [3] 906 Mbps combined

To a server connected to a gigabit port on the same switch (e1000)

• [1] 296 Mbps
• [2] 259 Mbps out, 62 Mbps in
• [3] 302 Mbps combined