解决方法:
概述
kexec是一个快速启动机制,允许通过已经运行的内核的上下文启动一个Linux内核,不需要经过BIOS。BIOS可能会消耗很多时间,特别是带有众多数量的外设的大型服务器。这种办法可以为经常启动机器的开发者节省很多时间。
kdump是一个新的,而且非常可信赖的内核崩溃转储机制。崩溃转储数据可以从一个新启动的内核的上下文中获取,而不是从已经崩溃的内核的上下文。当系统崩溃时,kdump使用kexec启动到第二个内核。第二个内核通常叫做捕获内核(capture kernel),以很小内存启动,并且捕获转储镜像。
第一个内核保留了内存的一部分,第二个内核可以用来启动。注意,在启动时,kdump保留了一定数量的重要的内存,这改变了红帽企业Linux 5最小内存需求。为了计算系统需要的真正最小内存,可以参看 http://www.RedHat.com/rhel/details/limits/ 上列出的最小内存需求,加上kdump使用的内存数量,以决定真正的最小内存的需求。
因为第一个内核的内存内容已经被保留,所以kexec可以不经过BIOS,启动捕获内核。这是内核崩溃转储的根本。
怎样配置kdump
1.确认kexec-tools已经安装:
#rpm -q kexec-tools
2.配置/etc/kdump.conf文件,指定vmcore将被转储的路径。可以通过scp拷贝到另一个服务器,也可以是裸设备,或者本地的文件系统。
3.修改一些启动参数,为捕获很保留一块内存。对于i386和x86_64架构,编辑/etc/grub.conf,在内核行的末尾添加 crashkernel=128@16M。
下面是一个带有kdump选项的/etc/grub.conf文件:
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,0)
# kernel /boot/vmlinuz-version ro root=/dev/hda1
# initrd /boot/initrd-version.img
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Client (2.6.17-1.2519.4.21.el5)
root (hd0,0)
kernel /boot/vmlinuz-2.6.17-1.2519.4.21.el5 ro root=LABEL=/ rhgb quiet crashkernel=128M@16M
initrd /boot/initrd-2.6.17-1.2519.4.21.el5.img
4.修改之后,重启系统。128M内存(从16M开始)不被正常的系统使用,为捕获内核保留。注意,free -m的输出会显示内存比不加参数时少了128M,这就是我们所期望的。
注意:可以使用小于128M,但是只使用64M做测试被证实是不可靠的。
5.现在,保留内存已经设置了,打开kdump初始脚本,启动服务:
# chkconfig kdump on
# service kdump start
6.可以通过kexec加载内核镜像,让系统准备捕获一个崩溃时产生的vmcore。可以通过sysrq强制系统崩溃:
# echo “c” > /proc/sysrq-trigger
这造成kernel panic,紧跟着系统重启kdump内核。当启动进程进入到启动kdump服务器时,vmcore将会被拷贝到你在/etc/kdump.conf文件中指定的位置。
注意:
终端frame-buffer和X将运行不正常。在运行一些类似于在内核配置上添加了”vga=791″或者运行X的系统,在通过kexec启动内核时,终端显示将不清楚。记住,kdump内核仍旧能够创建转储。当系统重启,显示将会恢复到正常状态。
Kexec is a fastboot mechanism that allows booting a Linux kernel from the context of an already running kernel without going through the BIOS. The BIOS can be very time consuming, especially on big servers with numerous peripherals. This can save a lot of time for developers who end up booting a machine numerous times.
Kdump is a new kernel crash dumping mechanism and is very reliable. The crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. Kdump uses kexec to boot into a second kernel whenever the system crashes. This second kernel, often called a capture kernel, boots with very little memory and captures the dump image.
The first kernel reserves a section of memory that the second kernel uses to boot. Be aware that the kdump reserves a significant amount of memory at the boot time, which changes the actual minimum memory requirements of Red Hat Enterprise Linux 5. To compute the actual minimum memory requirements for a system, refer to http://www.RedHat.com/rhel… for the listed minimum memory requirements and add the amount of memory used by kdump to determine the actual minimum memory requirements.
Kexec enables booting the capture kernel without going through BIOS hence the contents of the first kernel’s memory are preserved, which is essentially the kernel crash dump.
Installing Kdump
Verify the kexec-tools package is installed:
# rpm -q kexec-tools
If it is not installed proceed to install it via yum:
# yum install kexec-tools
Specifying Kdump Location
The location of the Kdump vmcore must be specified in /etc/kdump.conf. Not specifying the vmcore location will result in undefined behavior. You can either dump directly to a device, to a file, or to some location on the network via NFS or SSH.
Dumping Directly to a Device
You can configure Kdump to dump directly to a device by using the raw directive in kdump.conf. The syntax to be used is
raw devicename
For example:
raw /dev/sda1
Please be aware that this will overwrite any data that was previously on the device.
Dumping to a file on Disk
Kdump can be configured to mount a partition and dump to a file on disk. This is done by specifying the filesystem type, followed by
the partition in kdump.conf. The partition may be specified as a device name, a filesystem label, or UUID in the same manner as /etc/fstab.
The default directory in which the core will be dumped is /var/crash/%DATE/ where %DATE is the current date at the time of the cash dump. For example:
ext3 /dev/sda1
will mount /dev/sda1 as an ext3 device and dump the core to /var/crash/, whereas
ext3 LABEL=/boot
will mount the device that is ext3 with the label /boot. On most Red Hat Enterprise Linux installations, this will be the /boot directory. The easiest way to find how to specify the device is to look at what you’re currently using in /etc/fstab. The default directory in which the core will be dumped is /var/crash/%DATE/ where %DATE is the current date at the time of the crash dump. This can be changed by using the path directive in kdump.conf. For example:
ext3 UUID=f15759be-89d4-46c4-9e1d-1b67e5b5da82
path /usr/local/cores
will dump the vmcore to /usr/local/cores/ instead of the default /var/crash/ location.
Dumping to a Network Device using NFS
To configure kdump to dump to an NFS mount, edit /etc/kdump.conf and add a line with the following format:
net <nfs server>:/nfs/mount
For example:
net nfs.example.com:/export/vmcores
This will dump the vmcore to /export/vmcores/var/crash/[hostname]-[date] on the server nfs.example.com. The client system must have access to write to this mount point.
Dumping to a Network Device using SSH
SSH has the advantage of encrypting network communications while dumping. For this reason this is the best solution when you’re required to dump a vmcore across a publicly accessible network such as the Internet or a corporate WAN.
net user@<ssh server>
For example:
net kdump@crash.example.com
In this case, kdump will use scp to connect to the crash.example.com server using the kdump user. It will copy the vmcore to the /var/crash/[host]-[date]/ directory. The kdump user will need the necessary write permissions on the remote server.
To make this change take effect, run service kdump propagate, which should result in output similar to the following:
Generating new ssh keys… done,
kdump@crash.example.com’s password:
/root/.ssh/kdump_id_rsa.pub has been added to
~kdump/.ssh/authorized_keys2 on crash.example.com
Specifying Page Selection and Compression
On large memory systems, it is advisable to both discard pages that are not needed and to compress remaining pages. This is done in kdump.conf with the core_collector command. At this point in them the only fully supported core collector is makedumpfile. The options can be viewed with makedumpfile –help. The -d option specifies which types of pages should be left out. The option is a bit mask, having each page type specified like so:
zero pages = 1
cache pages = 2
cache private = 4
user pages = 8
free pages = 16
In general, these pages don’t contain relevent information. To set all these flags and leave out these pages, use a value of -d 31. The -c tells makedumpfile to compress the remaining data pages.
#core_collector makedumpfile -d 1 # throw out zero pages (containing no data)
#core_collector makedumpfile -d 31 # throw out all trival pages
#core_collector makedumpfile -c # compress all pages, but leave them all
core_collector makedumpfile -d 31 -c # throw out trival pages and compress (recommended)
Keep in mind that using the -d and -c options will marginally increase the ammount of time required to gather a cores.
Adding Boot Parameters
Modify some boot parameters to reserve a chunk of memory for the capture kernel. For i386 and x86_64 architectures, edit /etc/grub.conf, and append crashkernel=128M@16M to the end of the kernel line.
Note: It may be possible to use less than 128M, but testing with only 64M has proven unreliable. This is an example of /etc/grub.conf with the kdump options added:
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,0)
# kernel /boot/vmlinuz-version ro root=/dev/hda1
# initrd /boot/initrd-version.img
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Client (2.6.17-1.2519.4.21.el5)
root (hd0,0)
kernel /boot/vmlinuz-2.6.17-1.2519.4.21.el5 ro root=LABEL=/ rhgb quiet crashkernel=128M@16M
initrd /boot/initrd-2.6.17-1.2519.4.21.el5.img
Testing
After making the above changes, reboot the system. The 128M of memory (starting 16M into the memory) is left untouched by the normal system, reserved for the capture kernel. Take note that the output of free -m shows 128M less memory than without this parameter, which is expected.
Now that the reserved memory region is set up, turn on the kdump init script and start the service:
# chkconfig kdump on
# service kdump start
This should load the kernel image image via kexec, leaving the system ready to capture a vmcore upon crashing. To test this, force-crash the system using sysrq:
# echo “c” > /proc/sysrq-trigger
This causes the kernel to panic, followed by the system restarting into the kdump kernel. When the boot process gets to the point where it starts the kdump service, the vmcore should be copied out to disk to the location you specified in the /etc/kdump.conf file.
NOTE: Console frame-buffers and X are not properly supported. On a system typically run with something like “vga=791” in the kernel config line or with X running, console video will be garbled when a kernel is booted via kexec. Note that the kdump kernel should still be able to create a dump, and when the system reboots, video should be restored to normal.