这是本节的多页打印视图。 点击此处打印.

返回本页常规视图.

快速开始

cloud hypervisor 快速开始

1 - 下载

cloud hypervisor 下载

下载

github 下载地址:

https://github.com/cloud-hypervisor/cloud-hypervisor/releases/

x86

x86平台下载:

mkdir -p ~/work/soft/cloudhypervisor/bin
cd ~/work/soft/cloudhypervisor/bin
wget https://github.com/cloud-hypervisor/cloud-hypervisor/releases/download/v50.0/ch-remote
wget https://github.com/cloud-hypervisor/cloud-hypervisor/releases/download/v50.0/cloud-hypervisor
wget https://github.com/cloud-hypervisor/cloud-hypervisor/releases/download/v50.0/ch-remote-static
wget https://github.com/cloud-hypervisor/cloud-hypervisor/releases/download/v50.0/cloud-hypervisor-static

chmod +x ./ch-remote
chmod +x ./cloud-hypervisor
chmod +x ./ch-remote-static
chmod +x ./cloud-hypervisor-static

# 验证一下版本
./cloud-hypervisor --version
./cloud-hypervisor-static --version
./ch-remote --version
./ch-remote-static --version

版本输出为:

$ ./cloud-hypervisor-static --version
cloud-hypervisor v50.0.0
$ ./cloud-hypervisor --version
[1]    10062 segmentation fault  ./cloud-hypervisor --version
$ ./ch-remote --version
ch-remote v50.0.0
$ ./ch-remote-static --version
ch-remote v50.0.0

cloud-hypervisor 不能用.这是因为带 static 的二进制文件是静态编译版本，所有依赖库都已经打包在可执行文件里；不带 static 的则是动态链接版本，运行时需要系统上存在相应的共享库。如果系统缺少或版本不匹配，就可能出现 segmentation fault。

方便起见,还是直接用 static 版本比较好.

mkdir -p ~/work/soft/cloudhypervisor/bin
cd ~/work/soft/cloudhypervisor/bin
wget https://github.com/cloud-hypervisor/cloud-hypervisor/releases/download/v50.0/ch-remote-static -O ch-remote
wget https://github.com/cloud-hypervisor/cloud-hypervisor/releases/download/v50.0/cloud-hypervisor-static -O cloud-hypervisor

chmod +x ./ch-remote
chmod +x ./cloud-hypervisor

# 验证一下版本
./cloud-hypervisor --version
./ch-remote --version

arm64

类似,下载 cloud-hypervisor-static-aarch64 和 ch-remote-static-aarch64．

安装

将二进制文件所在的目录加入到 path 就可以了,这样用起来比较方便.

vi ~/.zshrc

加入内容:

# cloud hypervisor
export PATH="$PATH:$HOME/work/soft/cloudhypervisor/bin"

生效并检验:

source ~/.zshrc

ch-remote --version
cloud-hypervisor --version

升级

貌似只要直接覆盖原来版本的文件即可．

2 - 构建

cloud hypervisor 编译和构建

也可以不下载,自己从源码开始构建.

参考: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/building.md

准备

机器信息:

  • 操作系统: debian13
  • 内核: 6.12

准备工作:

mkdir -p $HOME/work/soft/cloudhypervisor

export CLOUDH=$HOME/work/soft/cloudhypervisor

cd $CLOUDH

安装依赖包:

sudo apt-get update
sudo apt install git build-essential m4 bison flex uuid-dev qemu-utils musl-tools

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

rustup target add x86_64-unknown-linux-musl

构建

clone 代码并开始构建:

mkdir -p ~/work/code/cloud-hypervisor
cd ~/work/code/cloud-hypervisor

git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git
cd cloud-hypervisor
# 最新的 release tag
git checkout v50.0
# 采用静态链接
cargo build --release --target=x86_64-unknown-linux-musl --all
sudo setcap cap_net_admin+ep ./target/x86_64-unknown-linux-musl/release/cloud-hypervisor

检查构建之后的版本：

./target/x86_64-unknown-linux-musl/release/cloud-hypervisor --version
cloud-hypervisor v50.0

ls -lh ./target/x86_64-unknown-linux-musl/release/cloud-hypervisor 
-rwxrwxr-x 2 sky sky 4.0M Nov 18 15:23 ./target/x86_64-unknown-linux-musl/release/cloud-hypervisor

静态链接也才4M,没大多少.

3 - 启动

启动 cloud hypervisor

参考:

https://www.cloudhypervisor.org/docs/prologue/quick-start/#booting-linux

在 linux 上启动.

Cloud Hypervisor 支持直接内核启动（x86-64内核需使用支持PVH构建的内核）或通过固件/firmware启动（可选Rust Hypervisor固件或名为 CLOUDHV/CLOUDHV_EFI 的edk2 UEFI固件）。

最新版本的Rust Hypervisor固件及edk2存储库中均提供固件文件的二进制构建版本。

固件的选择取决于您的客户操作系统选项，可能需要进行一些实验性测试。

固件启动

Cloud Hypervisor 支持引导包含运行云工作负载所需全部组件的磁盘镜像，即云镜像。

准备工作

首先安装 qemu-utils,稍后要用到其中的 qemu-img 和 mkdosfs :

sudo apt install qemu-utils
sudo apt install dosfstools
sudo apt install mtools

以下示例命令将下载Ubuntu云镜像，将其转换为 Cloud Hypervisor 可用的格式，并生成引导该镜像所需的固件。

mkdir -p ~/work/soft/cloudhypervisor/fireware
cd ~/work/soft/cloudhypervisor/fireware

wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img

qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw

wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.5.0/hypervisor-fw

处理完成后的文件列表:

$ ls -lh    

total 2.1G
-rw-rw-r-- 1 sky sky 618M Jun 25 09:39 focal-server-cloudimg-amd64.img
-rw-r--r-- 1 sky sky 2.2G Oct 16 16:54 focal-server-cloudimg-amd64.raw
-rw-rw-r-- 1 sky sky 133K Nov 30  2024 hypervisor-fw

Ubuntu云镜像默认不提供密码，因此首次启动时必须使用 cloud-init 磁盘镜像进行镜像定制。本脚本可生成基础 cloud-init 镜像，为镜像预设默认用户名/密码为 cloud/cloud123 。仅需在首次启动时添加此磁盘镜像即可。脚本同时通过 --net “mac=12:34:56:78:90:ab,tap=” 选项，基于 test_data/cloud-init/ubuntu/local/network-config 配置信息分配默认IP地址。随后系统将根据 network-config 详情启用匹配MAC地址的网络接口。

准备需要的脚本, 在代码仓库根目录下的 scripts 中:

https://github.com/cloud-hypervisor/cloud-hypervisor/tree/v48.0/scripts

后面还要使用到 test_data/cloud-init/ubuntu/ 目录下的文件, 因此最好是把整个代码仓库下载下来,方便后面使用

mkdir -p ~/work/code/cloud-hypervisor/
cd ~/work/code/cloud-hypervisor/

git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git
cd cloud-hypervisor
git checkout v50.0

执行:

~/work/code/cloud-hypervisor/cloud-hypervisor/scripts/create-cloud-init.sh

输出如下:

+ rm -f /tmp/ubuntu-cloudinit.img
+ mkdosfs -n CIDATA -C /tmp/ubuntu-cloudinit.img 8192
mkfs.fat 4.2 (2021-01-31)
+ mcopy -oi /tmp/ubuntu-cloudinit.img -s test_data/cloud-init/ubuntu/local/user-data ::
+ mcopy -oi /tmp/ubuntu-cloudinit.img -s test_data/cloud-init/ubuntu/local/meta-data ::
+ mcopy -oi /tmp/ubuntu-cloudinit.img -s test_data/cloud-init/ubuntu/local/network-config ::

这会生成一个 ubuntu-cloudinit.img 文件,大小为 8M:

$ ls -lh /tmp/ubuntu-cloudinit.img
-rw-rw-r-- 1 sky sky 8.0M Oct 16 17:12 /tmp/ubuntu-cloudinit.img

注意这个目录是 /tmp,重启之后文件很可能就不在了,方便起见, 还是复制到

mkdir -p ~/work/soft/cloudhypervisor/images
mv /tmp/ubuntu-cloudinit.img /home/sky/work/soft/cloudhypervisor/images

启动

注意: 从 v49 版本开始需要设置 cloud-hypervisor 二进制的特权, 执行命令:

sudo setcap cap_net_admin+ep /home/sky/work/code/cloud-hypervisor/cloud-hypervisor/target/x86_64-unknown-linux-musl/release/cloud-hypervisor

否则后面启动时会报错:

cloud-hypervisor:   0.016376s: <main> ERROR:/home/sky/work/code/cloud-hypervisor/cloud-hypervisor/src/lib.rs:23 -- Fatal error: VmBoot(VmBoot(DeviceManager(CreateVirtioNet(OpenTap(TapOpen(ConfigureTap(Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" })))))))
Error: Cloud Hypervisor exited with the following chain of errors:
  0: Error booting VM
  1: The VM could not boot
  2: Error from device manager
  3: Cannot create virtio-net device
  4: Failed to open taps
  5: Open tap device failed
  6: Unable to configure tap interface
  7: Operation not permitted (os error 1)

注意启动命令中的路径要用绝对路径,不能用 ‘~/’ 之类,否则会报错无法找到文件.

/home/sky/work/code/cloud-hypervisor/cloud-hypervisor/target/x86_64-unknown-linux-musl/release/cloud-hypervisor \
	--kernel /home/sky/work/soft/cloudhypervisor/fireware/hypervisor-fw \
	--disk path=/home/sky/work/soft/cloudhypervisor/fireware/focal-server-cloudimg-amd64.raw path=/home/sky/work/soft/cloudhypervisor/images/ubuntu-cloudinit.img \
	--cpus boot=4 \
	--memory size=1024M \
	--net "tap=,mac=,ip=,mask="

输出为:

cloud-hypervisor: 481.142000µs: <main> WARN:vmm/src/vm_config.rs:356 -- Deprecation warning: No IP address provided. A default IP address is assigned. This behavior will be deprecated soon.
cloud-hypervisor: 506.863000µs: <main> WARN:vmm/src/vm_config.rs:363 -- Deprecation warning: No network mask provided. A default network mask is assigned. This behavior will be deprecated soon.

Ubuntu 20.04.6 LTS cloud hvc0

cloud login:

输入前面设置的用户/密码 (cloud / cloud123), 登录进去, 如果要退出,执行 sudo shutdown now 命令关闭启动的 microVM 即可.

此时因为没有配置好网络, 所以登录进去 microVM 后会发现没有网络可用.

配置microVM的网络

没成功,等下继续.

首先要在主机上创建并配置 TAP 接口:

sudo ip tuntap add name vm-tap0 mode tap user $(whoami) 
sudo ip link set vm-tap0 up
sudo ip addr add 192.168.1.127/24 dev vm-tap0

此时在主机上看到的 vm-tap0 设备为:

$ ip addr
......

10: vm-tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 3e:7b:2e:ca:9a:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.127/24 scope global vm-tap0
       valid_lft forever preferred_lft forever

注意：上面的 ip 命令在重启后会失效。若需持久化配置，可将其写入主机的网络配置文件（如 /etc/network/interfaces 或使用 systemd-networkd/NetworkManager)。

注意2: 如果配置有问题,可以先删除,再重新配置. sudo ip link delete vm-tap0

/home/sky/work/code/cloud-hypervisor/cloud-hypervisor/target/x86_64-unknown-linux-musl/release/cloud-hypervisor \
	--kernel /home/sky/work/soft/cloudhypervisor/fireware/hypervisor-fw \
	--disk path=/home/sky/work/soft/cloudhypervisor/fireware/focal-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img \
	--cpus boot=4 \
	--memory size=1024M \
	--net "tap=vm-tap0,mac=12:34:56:78:90:22,ip=192.168.1.129,mask=255.255.255.0"


/home/sky/work/code/cloud-hypervisor/cloud-hypervisor/target/x86_64-unknown-linux-musl/release/cloud-hypervisor \
	--kernel /home/sky/work/soft/cloudhypervisor/fireware/hypervisor-fw \
	--disk path=/home/sky/work/soft/cloudhypervisor/fireware/focal-server-cloudimg-amd64.raw path=/home/sky/work/soft/cloudhypervisor/images/ubuntu-cloudinit.img \
	--cpus boot=4 \
	--memory size=1024M \
	--net "tap=tap0,mac=12:34:56:78:90:22,ip=192.168.1.10,mask=255.255.255.0"

镜像启动

Cloud Hypervisor 也支持直接内核启动。对于 x86-64 架构，需要一个 vmlinux ELF 内核（编译时包含 PVH 支持）。为了支持开发，有一个自定义分支；不过只要所需选项已启用，任何近期的内核都可以使用。

构建 linux 内核

要构建内核：

cd ~/work/code/cloud-hypervisor/

git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-6.12.8 linux-cloud-hypervisor
cd linux-cloud-hypervisor
make ch_defconfig

sudo apt install libelf-dev
sudo apt install bc

注意要执行 sudo apt install libelf-dev 命令安装 libelf,否则会报错:

<stdin>:1:10: fatal error: libelf.h: No such file or directory

构建 x86-64 kernel:

KCFLAGS="-Wa,-mx86-used-note=no" make bzImage -j `nproc`

构建完成后,linux 内核保存在 ./arch/x86/boot/compressed/vmlinux.bin,大小为 45M:

$ ls -lh ./arch/x86/boot/compressed/vmlinux.bin
-rwxrwxr-x 1 sky sky 45M Nov 18 15:50 ./arch/x86/boot/compressed/vmlinux.bin

rootfs 磁盘镜像可以用之前下载的 ubuntu 镜像, 之前存放在目录 ~/work/soft/cloudhypervisor/fireware 下

也可以再次下载和转换:

cd ~/work/soft/cloudhypervisor/fireware

wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img # x86-64
wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img # AArch64
qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw # x86-64
qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-arm64.img focal-server-cloudimg-arm64.raw # AArch64

启动 vm:

/home/sky/work/code/cloud-hypervisor/cloud-hypervisor/target/x86_64-unknown-linux-musl/release/cloud-hypervisor \
	--kernel /home/sky/work/code/cloud-hypervisor/linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \
	--console off \
	--serial tty \
	--disk path=/home/sky/work/soft/cloudhypervisor/fireware/focal-server-cloudimg-amd64.raw \
	--cmdline "console=ttyS0 root=/dev/vda1 rw" \
	--cpus boot=4 \
	--memory size=1024M \
	--net "tap=,mac=,ip=,mask="

4 - 命令

cloud hypervisor 命令

参考:

https://www.cloudhypervisor.org/docs/prologue/commands/

cloud-hypervisor

通过 help 命令可以得到参数和参数说明:

cloud-hypervisor --help

  • --api-socket <api-socket>

    HTTP API socket (UNIX domain socket): path=</path/to/a/file> or fd=<fd>

  • --balloon <balloon>

    Balloon parameters

    “size=<balloon_size>,deflate_on_oom=on|off”

  • --cmdline <cmdline>

    Kernel command line

  • --console <console>

    Control (virtio) console: “off|null|pty|tty|file=</path/to/a/file>,iommu=on|off”

    [default: tty]

  • --cpus <cpus>

    boot=<boot_vcpus>,max=<max_vcpus>,topology=<threads_per_core>:<cores_per_die>:<dies_per_package>:<packages>,kvm_hyperv=on

    [default: boot=1,max_phys_bits=46]

  • --debug-console <debug-console>

    off|pty|tty|file=</path/to/a/file>,iobase=<port in hex>

    [default:off,iobase=0xe9]

  • --device <device>...

    Direct device assignment parameters

    “path=<device_path>,iommu=on|off,id=<device_id>,pci_segment=<segment_id>”

  • --disk <disk>...

    Disk parameters

    “path=<disk_image_path>,readonly=on|off,direct=on|off,iommu=on|off,num_queues=<number_of_queues>,queue_size=<size_of_each_queue>,vhost_user=on|off,socket=<vhost_user_socket_path>,bw_size=,bw_one_time_burst=,bw_refill_time=,ops_size=<io_ops>,ops_one_time_burst=<io_ops>,ops_refill_time=,id=<device_id>,pci_segment=<segment_id>,rate_limit_group=<group_id>,queue_affinity=<list_of_queue_indices_with_their_associated_cpuset>,serial=<serial_number>”

  • --event-monitor <event-monitor>

    File to report events on: path=</path/to/a/file> or fd=<fd>

  • --firmware <firmware>

    File to report events on: path=</path/to/a/file> or fd=<fd>

  • --fs <fs>...

    virtio-fs parameters “tag=<tag_name>,socket=<socket_path>,num_queues=<number_of_queues>,queue_size=<size_of_each_queue>,id=<device_id>,pci_segment=<segment_id>”

  • --initramfs <initramfs>

    Path to initramfs image

  • --kernel <kernel>

    Path to kernel to load. This may be a kernel or firmware that supports a PVH entry point (e.g. vmlinux) or architecture equivalent

  • --landlock

    enable/disable Landlock.

  • --landlock-rules <landlock-rules>...

    Landlock parameters “path=<path/to/{file/dir}>,access=[rw]”

  • --log-file <log-file>

    Log file. Standard error is used if not specified

  • --memory <memory>

    “size=<guest_memory_size>,mergeable=on|off,shared=on|off,hugepages=on|off,hugepage_size=<hugepage_size>,hotplug_method=acpi|virtio-mem,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>,prefault=on|off,thp=on|off”

    [default: size=512M]

  • --memory-zone <memory-zone>...

    User defined memory zone parameters

    “size=<guest_memory_region_size>, file=<backing_file>, shared=on|off, hugepages=on|off, hugepage_size=<hugepage_size>, host_numa_node=<node_id>, id=<zone_identifier>, hotplug_size=<hotpluggable_memory_size>, hotplugged_size=<hotplugged_memory_size>, prefault=on|off”

  • --net <net>...

    Network parameters

    “tap=<if_name>,ip=<ip_addr>,mask=<net_mask>,mac=<mac_addr>,fd=<fd1,fd2…>,iommu=on|off,num_queues=<number_of_queues>,queue_size=<size_of_each_queue>,id=<device_id>, vhost_user=<vhost_user_enable>, socket=<vhost_user_socket_path>, vhost_mode=client|server, bw_size=,bw_one_time_burst=, bw_refill_time=, ops_size=<io_ops>, ops_one_time_burst=<io_ops>, ops_refill_time=, pci_segment=<segment_id>offload_tso=on|off, offload_ufo=on|off, offload_csum=on|off”

  • --numa <numa>...

    Settings related to a given NUMA node

    “guest_numa_id=<node_id>, cpus=<cpus_id>, distances=<list_of_distances_to_destination_nodes>, memory_zones=<list_of_memory_zones>, pci_segments=<list_of_pci_segments>”

  • --pci-segment <pci-segment>...

    PCI Segment parameters

    “pci_segment=<segment_id>, mmio32_aperture_weight=, mmio64_aperture_weight=”

  • --platform <platform>

    num_pci_segments=<num_pci_segments>, iommu_segments=<list_of_segments>, iommu_address_width=<bits>, serial_number=<dmi_device_serial_number>, uuid=<dmi_device_uuid>, oem_strings=<list_of_strings>

  • --pmem <pmem>...

    Persistent memory parameters

    “file=<backing_file_path>,size=<persistent_memory_size>,iommu=on|off,discard_writes=on|off,id=<device_id>,pci_segment=<segment_id>”

  • --pvpanic

    Enable pvpanic device

  • --rate-limit-group <rate-limit-group>...

    Rate Limit Group parameters

    "bw_size=<bytes>, bw_one_time_burst=<bytes>, bw_refill_time=<ms>, ops_size=<io_ops>, ops_one_time_burst=<io_ops>, ops_refill_time=<ms>, id=<device_id>"

  • --restore <restore>

    Restore from a VM snapshot.

    Restore parameters

    "source_url=<source_url>, prefault=on|off, net_fds=<list_of_net_ids_with_their_associated_fds>"

    source_url should be a valid URL (e.g file:///foo/bar or tcp://192.168.1.10/foo)

    prefault brings memory pages in when enabled (disabled by default)

    net_fds is a list of net ids with new file descriptors. Only net devices backed by FDs directly are needed as input.

  • --rng <rng>

    Random number generator parameters “src=<entropy_source_path>,iommu=on|off”

    [default: src=/dev/urandom]

  • --seccomp <seccomp>

    [default: true] [possible values: true, false, log]

  • --serial <serial>

    Control serial port: off|null|pty|tty|file=</path/to/a/file>|socket=</path/to/a/file>

    [default: null]

  • --tpm <tpm>

    TPM device “(UNIX Domain Socket from swtpm) socket=</path/to/a/socket>”

  • --user-device <user-device>...

    Userspace device socket=<socket_path>,id=<device_id>,pci_segment=<segment_id>

  • -v...

    Sets the level of debugging output

  • --vdpa <vdpa>...

    vDPA device

    “path=<device_path>,num_queues=<number_of_queues>,iommu=on|off,id=<device_id>,pci_segment=<segment_id>”

  • -V, --version

    Print version

  • --vsock <vsock>

    Virtio VSOCK parameters

    “cid=<context_id>, socket=<socket_path>, iommu=on|off, id=<device_id>, pci_segment=<segment_id>”

  • --watchdog

    Enable virtio-watchdog

  • -h, --help

    Print help

​

ch-remote

通过 help 命令可以得到参数和参数说明:

ch-remote --help
命令 命令描述
add-device Add VFIO device
add-disk Add block device
add-fs Add virtio-fs backed fs device
add-net Add network device
add-pmem Add persistent memory device
add-user-device Add userspace device
add-vdpa Add vDPA device
add-vsock Add vsock device
boot Boot a created VM
coredump Create a coredump from VM
counters Counters from the VM
create Create VM from a JSON configuration
delete Delete a VM
info Info on the VM
nmi Trigger NMI
pause Pause the VM
ping Ping the VMM to check for API server availability
power-button Trigger a power button in the VM
reboot Reboot the VM
receive-migration Receive a VM migration
remove-device Remove VFIO and PCI device
resize Resize the VM
resize-zone Resize a memory zone
restore Restore VM from a snapshot
resume Resume the VM
send-migration Initiate a VM migration
shutdown Shutdown the VM
shutdown-vmm Shutdown the VMM
snapshot Create a snapshot from VM
help Print this message or the help of the given subcommand(s)

命令行选项:

选项 选项描述
--api-socket <api-socket> HTTP API socket path (UNIX domain socket)
-h, –help Print help
-V, –version Print version