Using bhyve on FreeBSD

07-07-2015

Disclaimer: this post was created on 07-07-2015 for FreeBSD 10.1, it might not work on other platforms or versions.

I wanted to write an article about bhyve for a long time now, and fortunately I recently had the time to do just that. Bhyve has the potential to became a very sophisticated, and advanced hypervisor.

Introduction

I’m not sure since when, but bhyve now supports libvirt. That’s great news, given that libvirt is a universal tool which helps in managing VMs. There are quite a few hypervisors compatible with libvirt, for example KVM, Xen, Linux containers (LXC) and a bunch of others. On the other hand, there are only a few graphical fronteds compatible with it, for example Virtual Machine Manager (virt-manager), which is a general purpose VM management tool. This can be installed to a computer and the hypervisors can be managed from there. Virt-manager is generally good for a few hypervisors, with 10-20 VMs, but managing a cloud infrastructure is quite a challenge with it. There a few tools compatible with libvirt, which are targeted at clouds or greater deployment.

In this article, I will cover the Virtual Machine Manager (virt-manager). This tool allows for the basic management of VMs, such as installing, starting, deleting etc. The goal of this document is to introduce the installation and configuration of bhyve to a FreeBSD host, libvirt, and the virt-manager to a remote computer, then to explain how you can connect from the virt-manager to the remote host. This setup allows the management of VMs remotely. This article is customized for CentOS guest installation as a guest, however other Linux guests could be installed and used in a similar way.

As its current state, libvirt is not fully compatible with bhyve and therefore interacting with VMs with libvirt is not possbile. However, this article provides instructions for installing and managing VMs with bhyve itself.

Additional informations for this article

I use the following setup:

  • FreeBSD 10.1-RELEASE-p13 - as the host computer
  • Arch Linux current - as the remote computer
  • These two computers are on one network

bhyve

So what is exactly bhyve? From the offical website:

bhyve, the “BSD hypervisor” is a hypervisor/virtual machine manager developed on FreeBSD and relies on modern CPU features such as Extended Page Tables (EPT) and VirtIO network and storage drivers.

It has been released on 20th January, 2014, so it’s a quite new, “shiny” technology. Therefore it is possible that a few bugs remain, or some feature do not work yet. If you encounter some, please report it. bhyve supports FreeBSD, OpenBSD, NetBSD and Linux distributions as a guest. Bhyve gets loaded and acts like KVM. This means, that bhyve itself is loaded into the kernel and that allows it to provide good performance to the virtual machines.

In order to boot Linux as a guest, it is important to have the sysutils/grub2-bhyve installed.

Installing and using bhyve

bhyve is by default installed on FreeBSD, it just needs to be loaded. Load the bhyve kernel module so:

# kldload vmm

In order to load automatically while booting up the system. It appends a new line to the /boot/loader.conf file:

# echo 'vmm_load="YES"' >> /boot/loader.conf

Then, build sysutils/grub2-bhyve from ports:

# portmaster sysutils/grub2-bhyve

Or install the package:

# pkg install sysutils/grub2-bhyve

To get further information about bhyve:

# man bhyve

Installing and starting a CentOS guest

If the guest needs to be be connected to the network, a tap interface and a bridge between the physical interface and the tap has to be created. My physical interface is re0, it is important to check and correct the commands accordingly.

# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
  net.link.tap.up_on_open: 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm re0 addm tap0
# ifconfig bridge0 up

Create the file, which will be used as the disk for the guest:

# truncate -s 8G centos.img

A Device map has to be created in order to clarify, which file should be mapped to which virtual device. In the example, both the centos.img, the device.map file and the CentOS-7-x86_64-DVD-1503-01.iso (which is the CentOS installer) was placed in one folder. The content of the device.map file here is:

(hd0) ./centos.img
(cd0) ./CentOS-7-x86_64-DVD-1503-01.iso

To start up a Linux guest, two steps must be taken. First, point GRUB to the kernel and initrd, and then the guest can be started. The following line starts the GRUB, points to the previously created device.map file, selects the root device, allocates memory and assigns a name.

# grub-bhyve -m device.map -r cd0 -M 2048 centos

Now the GRUB needs to be pointed to the vmlinuz and initrd files. First, check which devices are present:

grub> ls
(hd0) (cd0) (cd0,msdos2) (host)

Check, which files are present in the CD:

grub> ls (cd0)/
CentOS_BuildTag EFI/ EULA GPL images/ isolinux/ LiveOS/ Packages/ repodata/ RPM
-GPG-KEY-CentOS-Testing-7 RPM-GPG-KEY-CentOS-7 TRANS.TBL

In the isolinux folder:

grub> ls (cd0)/isolinux
boot.cat boot.msg grub.conf initrd.img isolinux.bin isolinux.cfg memtest splash
.png TRANS.TBL upgrade.img vesamenu.c32 vmlinuz

Then locate vmlinuz and initrd:

grub> linux (cd0)/isolinux/vmlinuz
grub> initrd (cd0)/isolinux/initrd.img

Boot the installer:

grub> boot

The Linux kernel is now loaded, so it is possible to start the guest.

# bhyve -AI -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./centos.img -s 4:0,ahci-cd,./CentOS-7-x86_64-DVD-1503-01.iso -l com1,stdio -c 4 -m 2048M centos

CentOS CLI installer configured

If everything went right, the CentOS CLI installer has been started. After being installed, the instance has to be destroyed!

# bhyvectl --destroy --vm=centos

Then the installed guest can be started:

# grub-bhyve -m device.map -r hd0,msdos1 -M 2048M centos

Also the Linux guest is installed, and a correct grub.cfg is present.

grub> ls (hd0,msdos1)/grub2/
themes/ device.map i386-pc/ locale/ fonts/ grubenv grub.cfg
grub> configfile (hd0,msdos1)/grub2/grub.cfg

Grub loaded with configuration

Start the VM (without the installer ISO):

# bhyve -AI -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./centos.img -l com1,stdio -c 4 -m 2048M centos

The config file, installed by the CentOS installer, applies the settings to GRUB and the choose option appears. When the OS is loaded:

# uname -a
Linux centos.local 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

The previosly created tap0 interface is present in the guest, and it is connected to the network. After the network is configured, the VM is reachable over SSH. After shutting down the guest, it is important to destroy the instance!

# bhyvectl --destroy --vm=centos

In order to find out, which VMs are running, the /dev/vmm/ shoud be checked.

# ls /dev/vmm/

Installing and running FreeBSD as a guest

Naturally, it is possible to run FreeBSD as a guest. That’s a bit easier task than running Linux. A disk image file is needed to be created:

# truncate -s 16G freebsd.img

Then running the installer from the downloaded FreeBSD installer ISO with a built in script:

# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-10.1-RELEASE-amd64-dvd1.iso freebsd

This will start up the guest, booting from the ISO and showing the guest on the terminal. From here, the installation could be performed.

libvirt

As I have mentioned before, libvirt is a very powerful tool that allows us to interact with hypervisors and management consoles. It is compatible with a bunch of hypervisors, and also with a lot of management tools. It not just makes the local management easier, but with a few hypervisors, it can be used remotely over SSH.

Installing libvirt on FreeBSD is pretty straightforward. devel/libvirt has to be either compiled from ports, or installed from packages.

Compiling from source:

# portmaster devel/libvirt

It is important to configure libvirt with bhyve support! Installing from packages:

# pkg install devel/libvirt

libvirt provides a command line tool, called “virsh” to interact with hypervisors. In order to start libvirt, the daemon has to be started:

# service libvirtd onestart

To start the libvirtd service automatically, when the host boots up, the following line needs to be added to the /etc/rc.conf file:

libvirtd_enable="YES"

Unfortunately starting a VM (or domain, with virsh syntax) with virsh, freezes the libvirt daemon and the VM actually does not start. In other words, at its current state, it is not possible to use bhyve with libvirt. I hope that this bug will be solved soon. Please also note, that in the example, FreeBSD 10.1-RELEASE-p13 is used, and a possible fix could have been submitted and could be present in 11-CURRENT!

To start the instance with virsh, a centos.xml file has been created in the folder /usr/local/etc/libcirt/qemu with contents stated on the website of libvirt. Then the file has to be initialized:

# virsh define centos.xml

When it is initialized, the VM could be started:

# virsh start centos

This is where it freezes. Starting a VM remotely with virt-manager will eventually work, when this issue has been fixed.

virt-manager

The Virtual Machine Manager (virt-manager) is an easy-to-use interface for managing VMs. It is compatible with libvirt and enables the user to manage bhyve (and a bunch of other hypervisors) remotely, or locally. The developer of virt-manager is Red Hat, and the first stable version has been released on 7th of September, 2014. I have been using this application for a long time to interact with KVM, and I find it a very handy tool.

Despite the fact that its not working as it’s current state with bhyve, here is the guide on how to continue. Because virt-manager interacts with libvirt and libvirt is unable to start a VM on bhyve, starting up a VM from virt-manager is not working. The first step is to install the virt-manager with the package manager of the distribution. On Arch Linux:

# pacman -S virt-manager

Once installed, it is possible to connect to the bhyve running on the FreeBSD.

Adding bhyve+SSH connection to virt-manager

The connection has been added to the virt-manager

Connection details of bhyve on remote host

Summary

Despite the fact, that bhyve and libvirt have a few bugs, and do not work right now, my hopes are high conserning this combo. I find FreeBSD a very good operating system, which has just needed a fine hypervisor. I hope that the developers of bhyve and libvirt will find the problem soon and will fix it. Until that, it is possible to use bhyve from command line and access the guests from there until the SSH has been configured.

##. Comments

It is important for me to stress, that this article is written to inform, and to be a guide to accomplish something. I am not responsible for any data losses, or for anything that goes wrong. It is your responsibility to understand the meaning of the commands and to determine the outcome.

It is also very important to have a backup of everything.

Thanks, sources I used

I would like to thank everybody, who contributed to bhyve, libvirt and virt-manager, also the FreeBSD community, RedHat and everyone else who made this possible. I would like also to thank to artsies for proofreading this document. For this article, I used the following sources:

Thank you for reading.