In order to save memory and boot time, QEMU 4.2 and later versions are able to
mmap(2) the kernel and initrd specified with -kernel and -initrd parameters.
This approach allows us to avoid reading and copying them into a buffer,
saving memory and time.
The memory pages that contain kernel and initrd are shared between multiple VMs using the same kernel and initrd images. So, when many VMs are launched we can save memory by sharing pages, and save time by avoiding reading them each time from the disk.
This feature is automatically used for some targets with ELF kernels (e.g. x86_64 vmlinux ELF image with PVH entry point), but it is not available with compressed kernel images (e.g. bzImage).
Patches
The patches that implement this feature are merged upstream and released with QEMU 4.2.
The main change was to map kernel and initrd into the memory, instead of
reading them, using g_mapped_file_*() APIs.
Benchmarks
We measured the memory footprint and the boot time using a standard QEMU
build (qemu-system-x86_64) with a
PVH kernel
and initrd (cpio):
- Initrd size: 3.0M
- Kernel (vmlinux)
- image size: 28M
- sections size [
size -A -d vmlinux]: 18.9M
Julio Montes did a very good analysis and he posted the results here: https://www.mail-archive.com/qemu-devel@nongnu.org/msg633168.html
Memory footprint
We used smem to measure USS and PSS:
- USS (Unique Set Size): amount of memory that is committed to physical memory and is unique to a process; it is not shared with any other. It is the amount of memory that would be freed if the process were to terminate.
- PSS (Proportional Set Size): This splits the accounting of shared pages that are committed to physical memory between all the processes that have them mapped.
$ smem -k | grep "PID\|$(pidof qemu-system-x86_64)"
PID User Command Swap USS PSS RSS
24833 qemu /usr/bin/qemu-system-x86_64 0 71.6M 74.3M 105.2
This is the memory footprint analysis, increasing the number of QEMU instances:
Memory footprint [MB]
QEMU before after
# instances USS PSS USS PSS
1 102.0 105.8 102.3 106.2
2 94.6 101.2 72.3 90.1
4 94.1 98.0 72.0 81.5
8 94.0 96.2 71.8 76.9
16 93.9 95.1 71.6 74.3
Boot time
We measured the boot time using the qemu-boot-time scripts described in this post.
This is the boot time analysis:
Boot time [ms]
before after
# trace points
qemu_init_end: 63.85 55.91
linux_start_kernel: 82.11 (+18.26) 74.51 (+18.60)
linux_start_user: 169.94 (+87.83) 159.06 (+84.56)
Conclusions
Mapping into memory the kernel and initrd images allowed us to save about 20 MB of memory when multiple VMs are started and allowed us to speed up the boot by about 10 milliseconds.
Note: both gains are strictly related to images size.