This post is mostly a story about my new VM server setup. However, it features a fun twist that reminds me how important it is to consider the layers upon layers of added security tools that exist in Linux that could interfere with a normal setup.
The Context
Around a month and a half ago I got this new server in my infrastructure, an old decommissioned Dell Precision T1700. I was pleasantly surprised to learn that it has an Intel Xeon with 8 cores at 3.5GHz (which makes it the most powerful server I own aside from my main workstation).
I decided to move a couple of services off of my VPS, which had been struggling under the load that Matrix Synapse puts on it. So far, aside from the CPU spending roughly half of its time in I/O idle (I never replaced the hard drives in it, they have lived a good life, and I have no irreplaceable data on them, so I'm just waiting for a failure before buying new ones), the server is fine. It is so fine in fact that I wanted to add more onto the pile of things it would have to deal with.
A week or so ago I was doing a lab at school where the teacher provided us with a Virtualbox Appliance VM image so we could have all of the tools and setup for the lab. Since I was running everything on my Thinkpad, which already had libvirt/virt-manager installed, I wanted to see if I could convert the OVA into QCow2 or better import the OVA as a VM.
A Tangent on OVAs and Virt-Manager
As it turns out, an OVA is simply a TAR archive with three files in it : the disk image, a XML file (something dot ovf) containing the VM settings, and a third on that is optional and which purpose I am not entirely sure of.
Long story short I struggled the entire lab to try and make the VM work on QEMU with Libvirt, but failed. I ended up running VirtualBox for the parts I really needed to get running (I mostly listened to the lab, which was mostly a lesson, but I'm digressing).
See, first of all, Libvirt's virtual machine assistant has this funny quirk where if you want to change things like the chipset used or the firmware (which defaults to BIOS), you have to specify one little option before creating the VM. Otherwise, it's set forever (unless you really know what you are doing and love editing XML).
During the whole ordeal I figured I could offload the garbage of running libvirt onto a remote server... or my new server.
A couple of SSH tweaks and copying public keys later and I had a connection from my laptop to the libvirt daemon running on my Dell Precision server. It was just the beginning.
Debugging the Lolipop in the Machine
You work at a factory. You are in charge of maintenance on a lot of complicated machines with a lot of moving parts. During maintenance for one of the machines, you figure out the issue comes from an authorization module. You bypass the module to let the machine run, but it does not. Fiddling more in the gears, you realize there is a second, separate authorization module with a lolipop stuck in it. That module overrides the first one, and someone probably dropped their lolipop into it. You spent five hours on this, because someone dropped a lolipop on a module you didn't know existed.
You never expect a lolipop in an industrial machine. Yet, sometimes, you should.
When I tried to create a UEFI machine through Virt-Manager's wizard, after figuring out I needed to configure before creating, I ran into errors saying that access to the NVRAM variable files for that machine was unauthorized. I was running libvirt as root, and connecting through a user that was in the libvirt group. I figured, which not try and set the permissions for that file to 777 so I was sure I could find it ? I still hit a "Permission Denied" wall.
After some research online I figured out that there is currently a bug in the apparmor profiles brought in alongside Libvirt where the authorizations granted to libvirtd when it starts do not allow it to read /var/lib/libvirt/qemu/nvram/*.fd. The fix that is currently released on Ubuntu (but not even Debian sid yet) only modifies /etc/apparmor.d/abstractions/libvirt-qemu to apply this patch
--- /etc/apparmor.d/abstractions/libvirt-qemu.orig 2022-01-22'
time: '18:22:57.000000000 +0000
+++ /etc/apparmor.d/abstractions/libvirt-qemu 2022-02-25'
time: '13:54:22.075405809 +0000
@@ -85,7 +85,7 @@
/usr/share/misc/sgabios.bin r,
/usr/share/openbios/** r,
/usr/share/openhackware/** r,
- /usr/share/OVMF/** r,
+ /usr/share/OVMF/** rk,
/usr/share/ovmf/** r,
/usr/share/proll/** r,
/usr/share/qemu-efi/** r,
@@ -249,5 +249,8 @@
/ r, # harmless on any lsb compliant system
/sys/bus/nd/devices/{,**/} r,
+ # required for QEMU accessing UEFI nvram variables
+ /**/nvram/*_VARS.fd rwk,
+
# Site-specific additions and overrides. See local/README for details.
#include <local/abstractions/libvirt-qemu>
Then you just systemctl restart apparmor and systemctl restart libvirt and you get going.
The funny thing I found out later on is that it's not the only apparmor-related bug in libvirt, apparently.
The Aftermath
In the end, I just wanted to use Virt-Manager and make my little toy VM, but I got sidetracked into the depth of libvirt, and its side interactions with Apparmor.
I have managed to install a little remote VM that runs decently and that I hope to turn into my practice VM for things like Hack The Box or Try Hack Me. Or, at least, whenever I have time for these again.