Boot on sgarzare's blog

QEMU 4.2 mmap(2)s kernel and initrd

sgarzare@redhat.com (Stefano Garzarella) — Sun, 22 Dec 2019 17:46:44 +0100

In order to save memory and boot time, QEMU 4.2 and later versions are able to mmap(2) the kernel and initrd specified with -kernel and -initrd parameters. This approach allows us to avoid reading and copying them into a buffer, saving memory and time.

The memory pages that contain kernel and initrd are shared between multiple VMs using the same kernel and initrd images. So, when many VMs are launched we can save memory by sharing pages, and save time by avoiding reading them each time from the disk.

This feature is automatically used for some targets with ELF kernels (e.g. x86_64 vmlinux ELF image with PVH entry point), but it is not available with compressed kernel images (e.g. bzImage).

Patches

The patches that implement this feature are merged upstream and released with QEMU 4.2.

The main change was to map kernel and initrd into the memory, instead of reading them, using g_mapped_file_*() APIs.

Benchmarks

We measured the memory footprint and the boot time using a standard QEMU build (qemu-system-x86_64) with a PVH kernel and initrd (cpio):

Initrd size: 3.0M
Kernel (vmlinux)
- image size: 28M
- sections size [size -A -d vmlinux]: 18.9M

Julio Montes did a very good analysis and he posted the results here: https://www.mail-archive.com/qemu-devel@nongnu.org/msg633168.html

Memory footprint

We used smem to measure USS and PSS:

USS (Unique Set Size): amount of memory that is committed to physical memory and is unique to a process; it is not shared with any other. It is the amount of memory that would be freed if the process were to terminate.
PSS (Proportional Set Size): This splits the accounting of shared pages that are committed to physical memory between all the processes that have them mapped.

$ smem -k | grep "PID\|$(pidof qemu-system-x86_64)"
  PID User     Command                         Swap      USS      PSS      RSS
24833 qemu     /usr/bin/qemu-system-x86_64        0    71.6M    74.3M    105.2

This is the memory footprint analysis, increasing the number of QEMU instances:

                           Memory footprint [MB]
     QEMU             before                 after
 # instances        USS     PSS           USS     PSS
      1           102.0   105.8         102.3   106.2
      2            94.6   101.2          72.3    90.1
      4            94.1    98.0          72.0    81.5
      8            94.0    96.2          71.8    76.9
     16            93.9    95.1          71.6    74.3

Boot time

We measured the boot time using the qemu-boot-time scripts described in this post.

This is the boot time analysis:

                                   Boot time [ms]
                          before                  after
 # trace points
 qemu_init_end:           63.85                   55.91
 linux_start_kernel:      82.11 (+18.26)          74.51 (+18.60)
 linux_start_user:       169.94 (+87.83)         159.06 (+84.56)

Conclusions

Mapping into memory the kernel and initrd images allowed us to save about 20 MB of memory when multiple VMs are started and allowed us to speed up the boot by about 10 milliseconds.

Note: both gains are strictly related to images size.

How to measure the boot time of a Linux VM with QEMU/KVM

sgarzare@redhat.com (Stefano Garzarella) — Sat, 24 Aug 2019 15:03:30 +0200

The stefano-garzarella/qemu-boot-time repository contains a Python perf-script and (Linux, QEMU, SeaBIOS) patches to measure the boot time of a Linux VM with QEMU/KVM.

Using I/O writes, we can trace events to measure the time consumed during the boot phase by the different components:

firmware
- SeaBIOS
- qboot
optionrom
- linuxboot [bzImage]
- pvh [vmlinux + PVH entry point]
Linux kernel

We extended the I/O port addresses and values defined in qboot/benchmark.h adding new trace points to trace the kernel boot time.

In the repository you can find patches for Linux, QEMU, and SeaBIOS to add the I/O writes in the components involved during the boot, and a Python perf-script useful to process the data recorded through perf using perf-script’s built-in Python interpreter.

Trace points

The benchmark.h file contains the following trace points used in the patches:

QEMU
- qemu_init_end: first kvm_entry (i.e. QEMU initialized has finished)
Firmware (SeaBIOS + optionrom or qboot)
- fw_start: first entry of the firmware
- fw_do_boot: after the firmware initialization (e.g. PCI setup, etc.)
- linux_start_boot: before the jump to the Linux kernel
- linux_start_pvhboot: before the jump to the Linux PVH kernel
Linux Kernel
- linux_start_kernel: first entry of the Linux kernel
- linux_start_user: before starting the init process

Custom trace points

If you want to add new trace points, you can simply add an I/O write to LINUX_EXIT_PORT or FW_EXIT_PORT I/O port with a value (> 7) that identifies the trace point:

    outb(10, LINUX_EXIT_PORT);

The perf script output will contain Exit point 10 line that identifies your custom trace point:

 qemu_init_end: 143.770419
 fw_start: 143.964328 (+0.193909)
 fw_do_boot: 164.71107 (+20.746742)
 Exit point 10: 165.396804 (+0.685734)
 linux_start_kernel: 165.979486 (+0.582682)
 linux_start_user: 272.178335 (+106.198849)

How to use

Clone qemu-boot-time repository

REPOS="~/repos"
cd ${REPOS}
git clone https://github.com/stefano-garzarella/qemu-boot-time.git

Apply patches to Linux, QEMU and SeaBIOS

Trace points are printed only if they are recorded, so you can enable few of them, patching only the components that you are interested in.

Linux

Apply the patches/linux.patch to your Linux kernel in order to trace kernel events

cd ${REPOS}/linux
git checkout -b benchmark
git am ${REPOS}/qemu-boot-time/patches/linux.patch

QEMU

Apply the patches/qemu.patch to your QEMU in order to trace optionrom events

cd ${REPOS}/qemu
git checkout -b benchmark
git am ${REPOS}/qemu-boot-time/patches/qemu.patch

mkdir build-benchmark
cd build-benchmark
../configure --target-list=x86_64-softmmu ...
make

You can use qemu-system-x86_64 -L ${REPOS}/qemu/build-benchmark/pc-bios/optionrom/ ... to use the optionrom patched.

SeaBIOS

Apply the patches/seabios.patch to your SeaBIOS in order to trace bios events

cd ${REPOS}/seabios
git checkout -b benchmark
git am ${REPOS}/qemu-boot-time/patches/seabios.patch

make clean distclean
cp ${REPOS}/qemu/roms/config.seabios-256k .config
make oldnoconfig

You can use qemu-system-x86_64 -bios ${REPOS}/seabios/out/bios.bin ... to use the SeaBIOS image patched.

qboot

qboot already defines trace points, we just need to compile it defining BENCHMARK_HACK

cd ${REPOS}/qboot
make clean
BIOS_CFLAGS="-DBENCHMARK_HACK=1" make

You can use qemu-system-x86_64 -bios ${REPOS}/qboot/bios.bin ... to use the qboot image.

Enable KVM events

The following steps allow perf record to get the kvm trace events:

echo 1 > /sys/kernel/debug/tracing/events/kvm/enable
echo -1 > /proc/sys/kernel/perf_event_paranoid
mount -o remount,mode=755 /sys/kernel/debug
mount -o remount,mode=755 /sys/kernel/debug/tracing

Record the trace events

Start perf record to get the trace events

PERF_DATA="qemu_perf.data"
perf record -a -e kvm:kvm_entry -e kvm:kvm_pio -e sched:sched_process_exec \
            -o $PERF_DATA &
PERF_PID=$!

You can run QEMU multiple times to get also some statistics (Avg/Min/Max)

qemu-system-x86_64 -machine q35,accel=kvm \
                   -bios seabios/out/bios.bin \
                   -L qemu/build-benchmark/pc-bios/optionrom/ \
                   -kernel linux/bzImage ...
qemu-system-x86_64 -machine q35,accel=kvm \
                   -bios seabios/out/bios.bin \
                   -L qemu/build-benchmark/pc-bios/optionrom/ \
                   -kernel linux/bzImage ...
qemu-system-x86_64 -machine q35,accel=kvm \
                   -bios seabios/out/bios.bin \
                   -L qemu/build-benchmark/pc-bios/optionrom/ \
                   -kernel linux/bzImage ...

Stop perf record

kill $PERF_PID

Process the trace recorded using the qemu-perf-script.py

Note: times printed in milliseconds

perf script -s ${REPOS}/qemu-boot-time/perf-script/qemu-perf-script.py -i $PERF_DATA

in trace_begin
sched__sched_process_exec     1 55061.435418353   289738 qemu-system-x86
kvm__kvm_entry           1 55061.466887708   289741 qemu-system-x86
kvm__kvm_pio             1 55061.467070650   289741 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=1

kvm__kvm_pio             1 55061.475818073   289741 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=4

kvm__kvm_pio             1 55061.477168037   289741 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=3

kvm__kvm_pio             1 55061.558779540   289741 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=5

kvm__kvm_pio             1 55061.686849663   289741 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=6

sched__sched_process_exec     4 55067.461869075   289793 qemu-system-x86
kvm__kvm_entry           4 55067.496402472   289796 qemu-system-x86
kvm__kvm_pio             4 55067.496555385   289796 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=1

kvm__kvm_pio             4 55067.505067184   289796 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=4

kvm__kvm_pio             4 55067.506395502   289796 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=3

kvm__kvm_pio             4 55067.584029910   289796 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=5

kvm__kvm_pio             4 55067.704751791   289796 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=6

sched__sched_process_exec     0 55070.073823767   289827 qemu-system-x86
kvm__kvm_entry           0 55070.110507211   289830 qemu-system-x86
kvm__kvm_pio             0 55070.110694645   289830 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=1

kvm__kvm_pio             1 55070.120092692   289830 qemu-system-x86      rw=1, port=0xf5, size=1, count=1, val=4

kvm__kvm_pio             1 55070.121437922   289830 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=3

kvm__kvm_pio             1 55070.198628779   289830 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=5

kvm__kvm_pio             1 55070.315734630   289830 qemu-system-x86      rw=1, port=0xf4, size=1, count=1, val=6

in trace_end
Trace qemu-system-x86
1) pid 289738
 qemu_init_end: 31.469355
 fw_start: 31.652297 (+0.182942)
 fw_do_boot: 40.39972 (+8.747423)
 linux_start_boot: 41.749684 (+1.349964)
 linux_start_kernel: 123.361187 (+81.611503)
 linux_start_user: 251.43131 (+128.070123)
2) pid 289793
 qemu_init_end: 34.533397
 fw_start: 34.68631 (+0.152913)
 fw_do_boot: 43.198109 (+8.511799)
 linux_start_boot: 44.526427 (+1.328318)
 linux_start_kernel: 122.160835 (+77.634408)
 linux_start_user: 242.882716 (+120.721881)
3) pid 289827
 qemu_init_end: 36.683444
 fw_start: 36.870878 (+0.187434)
 fw_do_boot: 46.268925 (+9.398047)
 linux_start_boot: 47.614155 (+1.34523)
 linux_start_kernel: 124.805012 (+77.190857)
 linux_start_user: 241.910863 (+117.105851)

Avg
 qemu_init_end: 34.228732
 fw_start: 34.403161 (+0.174429)
 fw_do_boot: 43.288918 (+8.885757)
 linux_start_boot: 44.630088 (+1.34117)
 linux_start_kernel: 123.442344 (+78.812256)
 linux_start_user: 245.408296 (+121.965952)

Min
 qemu_init_end: 31.469355
 fw_start: 31.652297 (+0.182942)
 fw_do_boot: 40.39972 (+8.747423)
 linux_start_boot: 41.749684 (+1.349964)
 linux_start_kernel: 122.160835 (+80.411151)
 linux_start_user: 241.910863 (+119.750028)

Max
 qemu_init_end: 36.683444
 fw_start: 36.870878 (+0.187434)
 fw_do_boot: 46.268925 (+9.398047)
 linux_start_boot: 47.614155 (+1.34523)
 linux_start_kernel: 124.805012 (+77.190857)
 linux_start_user: 242.882716 (+118.077704)

QEMU 4.0 boots uncompressed Linux x86_64 kernel

sgarzare@redhat.com (Stefano Garzarella) — Fri, 23 Aug 2019 15:26:54 +0200

QEMU 4.0 is now able to boot directly into the uncompressed Linux x86_64 kernel binary with minimal firmware involvement using the PVH entry point defined in the x86/HVM direct boot ABI. (CONFIG_PVH=y must be enabled in the Linux config file).

The x86/HVM direct boot ABI was initially developed for Xen guests, but with latest changes in both QEMU and Linux, QEMU is able to use that same entry point for booting KVM guests.

Prerequisites

QEMU >= 4.0
Linux kernel >= 4.21
- CONFIG_PVH=y enabled
- vmlinux uncompressed image

How to use

To boot the PVH kernel image, you can use the -kernel parameter specifying the path to the vmlinux image.

qemu-system-x86_64 -machine q35,accel=kvm \
    -kernel /path/to/vmlinux \
    -drive file=/path/to/rootfs.ext2,if=virtio,format=raw \
    -append 'root=/dev/vda console=ttyS0' -vga none -display none \
    -serial mon:stdio

The -initrd and -append parameters are also supported as for compressed images.

Details

QEMU will automatically recognize if the vmlinux image has the PVH entry point and it will use SeaBIOS with the new pvh.bin optionrom to load the uncompressed image into the guest VM.

As an alternative, qboot can be used to load the PVH image.

Performance

Perf script and patches used to measure boot time: https://github.com/stefano-garzarella/qemu-boot-time.

The following values are expressed in milliseconds [ms]

QEMU (q35 machine) + SeaBIOS + bzImage
- qemu_init_end: 36.072056
- linux_start_kernel: 114.669522 (+78.597466)
- linux_start_user: 191.748567 (+77.079045)
QEMU (q35 machine) + SeaBIOS + vmlinux(PVH)
- qemu_init_end: 51.588200
- linux_start_kernel: 62.124665 (+10.536465)
- linux_start_user: 139.460582 (+77.335917)
QEMU (q35 machine) + qboot + bzImage
- qemu_init_end: 36.443638
- linux_start_kernel: 106.73115 (+70.287512)
- linux_start_user: 184.575531 (+77.844381)
QEMU (q35 machine) + qboot + vmlinux(PVH)
- qemu_init_end: 51.877656
- linux_start_kernel: 56.710735 (+4.833079)
- linux_start_user: 133.808972 (+77.098237)
Tracepoints:
- qemu_init_end: first kvm_entry (i.e. QEMU initialization has finished)
- linux_start_kernel: first entry of the Linux kernel (start_kernel())
- linux_start_user: before starting the init process

Patches

Linux patches merged upstream in Linux 4.21:

https://lkml.org/lkml/2018/12/14/1330

QEMU patches merged upstream in QEMU 4.0:

qboot patches merged upstream: