io7m | single-page | multi-page
1. Orientation
Previous
Waxmill User Manual 0.0.2
Up
3. Command-Line Interface
Next
2

# Usage

- 2. Usage
  - 2.1. Waxmill
  - 2.2. Paths And Environment
  - 2.3. Virtual Machines
  - 2.4. Defining A Virtual Machine
  - 2.5. Adding Devices
  - 2.6. Realizing A Virtual Machine
  - 2.7. Boot Configurations
  - 2.8. Running A Virtual Machine
  - 2.9. Process Supervision

2.1

## Waxmill

1
The primary means by which users use the Waxmill package is via the waxmill command-line tool.
2.2

## Paths And Environment

1
The waxmill command-line tool will attempt to read a configuration file specified by the WAXMILL_CONFIGURATION_FILE environment variable. The configuration file, by convention, should be stored at /etc/waxmill/config.xml. The default configuration file is as follows:
2

### 2.2.2 Configuration File

    <?xml version="1.0" encoding="UTF-8" ?>

    <Configuration xmlns="urn:com.io7m.waxmill.config:1:0">
      <Paths>
        <Path type="ZFSExecutable"
              value="/sbin/zfs"/>
        <Path type="GRUBBhyveExecutable"
              value="/usr/local/sbin/grub-bhyve"/>
        <Path type="BhyveExecutable"
              value="/usr/sbin/bhyve"/>
        <Path type="BhyveCtlExecutable"
              value="/usr/sbin/bhyvectl"/>
        <Path type="CuExecutable"
              value="/usr/bin/cu"/>
        <Path type="VirtualMachineConfigurationDirectory"
              value="/etc/waxmill/vm"/>
        <Path type="VirtualMachineRuntimeDirectory"
              value="/storage/vm"/>
      </Paths>
    </Configuration>

3
The configuration file explicitly specifies the locations of various system utilities [1], and also specifies the location of the virtual machine configuration directory and the virtual machine runtime directory. The virtual machine configuration directory is a directory consisting of XML configuration files for individual virtual machines. The virtual machine runtime directory is a directory containing one directory per virtual machine, each of which stores resources needed by virtual machines at run-time such as ZFS volumes, sockets, and lock files.
2.3

## Virtual Machines

1
A virtual machine, in Waxmill, carries a universally-unique identifier that is used to refer to the virtual machine during its entire lifetime. The identifier of a virtual machine cannot be changed after it is defined; this property of identifiers is useful to track virtual machines if they are migrated to new physical machines.
2
A virtual machine is first defined, then configured with one or more devices, then configured with one or more boot configurations, then realized, and finally run.
2.4

## Defining A Virtual Machine

1
Defining a virtual machine involves taking a description of the virtual machine and recording that description in the virtual machine configuration directory. This is typically accomplished using the vm-define command. Once a virtual machine is defined, it will appear in the list of defined virtual machines (which can be inspected using the vm-list command).
2

### 2.4.2 Defining A Virtual Machine

    $ waxmill vm-define \
      --name 'com.io7m.example4' \
      --cpu-count 4 \
      --comment 'An example machine' \
      --memory-gigabytes 1 \
      --memory-megabytes 512

    $ waxmill vm-list
    # ID                                     Name
    800a2dad-3367-4a98-879a-0fac219f55f4     com.io7m.example2
    a866f9e4-f6c1-4ecf-ad49-000925eefa0a     com.io7m.example4
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example
    7c2bcb79-f20c-47fa-812b-8994f17f97b2     com.io7m.example5

3
The definition of the created machine can be exported using the vm-export command. The command will simply print the XML configuration file to the standard output. This can be used to manually inspect the configuration of the machine, or to copy the configuration to a new physical machine.
4

### 2.4.4 Exporting A Virtual Machine

    $ waxmill vm-export --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a
    <?xml version="1.0" encoding="UTF-8"?>
    <wxm:VirtualMachines xmlns:wxm="urn:com.io7m.waxmill.vm:1:0">
        <wxm:VirtualMachine id="a866f9e4-f6c1-4ecf-ad49-000925eefa0a" name="com.io7m.example4">
            <wxm:Comment>An example machine</wxm:Comment>
            <wxm:CPUTopology sockets="1" threads="1" cores="4"/>
            <wxm:Memory gigabytes="1" megabytes="512"/>
            <wxm:Devices>
                <wxm:HostBridge vendor="UNSPECIFIED">
                    <wxm:DeviceSlot bus="0" slot="0" function="0"/>
                </wxm:HostBridge>
            </wxm:Devices>
            <wxm:BootConfigurations/>
            <wxm:Flags>
                <wxm:Flag name="DisableMPTableGeneration" enabled="false"/>
                <wxm:Flag name="ForceVirtualIOPCIToUseMSI" enabled="false"/>
                <wxm:Flag name="GenerateACPITables" enabled="true"/>
                <wxm:Flag name="GuestAPICIsX2APIC" enabled="false"/>
                <wxm:Flag name="IncludeGuestMemoryInCoreFiles" enabled="false"/>
                <wxm:Flag name="RealTimeClockIsUTC" enabled="false"/>
                <wxm:Flag name="WireGuestMemory" enabled="false"/>
                <wxm:Flag name="ExitCPUOnPAUSE" enabled="true"/>
                <wxm:Flag name="YieldCPUOnHLT" enabled="true"/>
            </wxm:Flags>
        </wxm:VirtualMachine>
    </wxm:VirtualMachines>

5
Similarly, the vm-import command can be used to import one or more virtual machines from a given XML configuration file.
6
bhyve virtual machines have a number of configurable boolean flags. The vm-set command can be used to turn these flags on and off. For example, using PCI passthru devices requires wiring the memory of the guest virtual machine.
2.5

## Adding Devices

1
A virtual machine needs to be configured with one or more devices in order to actually do useful work. Typically, a virtual machine will have one or more storage devices (emulated disks), one or more network devices (emulations of ethernet interfaces), and a console device (to allow access to the operating system console from the host machine). A device is attached to exactly one device slot at any given time. A device slot is a PCI slot value consisting of a 3-tuple (bus, pcislot, function), where bus ∈ [0, 255], pcislot ∈ [0,31], and function ∈ [0,7]. The concrete notation used to denote a device slot is bus:pcislot:function, so the string 1:5:3 denotes a device slot on bus 1, slot 5, function 3. In practice, virtual machines will typically specify all devices to use bus 0 and function 0, and only vary the slot value.
2
The waxmill command-line tool contains various subcommands to create and attach devices to virtual machines.
3

### 2.5.3 Device Commands

- vm-add-ahci-disk
- vm-add-ahci-optical
- vm-add-lpc-device
- vm-add-virtio-disk
- vm-add-virtio-network-device

4
The types of devices attached to a virtual machine will depend on the level of support that the guest operating system running on the virtual machine has for virtualization. Operating systems that have explicit support for virtualization (typically every major operating system with a release more recent than about 2007) will perform best using Virtio devices. Older operating systems may need to be presented with the illusion of having access to AHCI devices. Please see the documentation for each of the commands for details.
5
The Waxmill package can optionally automatically manage storage devices backed by ZFS volumes if requested. If, on a virtual machine with ID X, a storage device that is specified to use a zfs-volume backend is attached to slot a:b:c, the Waxmill package will automatically create a ZFS volume at /storage/vm/X/disk-a:b:c (where /storage/vm is the configured virtual machine runtime directory) when realizing the virtual machine. This is merely a convenience feature; if the ZFS volume already exists, it will be used and the Waxmill package will not attempt to modify it.
2.6

## Realizing A Virtual Machine

1
Realizing a virtual machine involves reading the description of the virtual machine from the virtual machine configuration directory and then creating any resources that are needed at run-time such as ZFS volumes. The vm-realize command is used to realize virtual machines:
2

### 2.6.2 Realizing A Virtual Machine

    $ waxmill vm-realize \
      --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a \
      --dry-run true
    /sbin/zfs create -V 10gb storage/vm/a866f9e4-f6c1-4ecf-ad49-000925eefa0a/disk-0_1_0

    $ waxmill vm-realize \
      --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a

    $ waxmill vm-realize \
      --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMRealize: ZFS volume storage/vm/a866f9e4-f6c1-4ecf-ad49-000925eefa0a/disk-0_1_0 exists and has the correct size

2.7

## Boot Configurations

1
Before a virtual machine can actually be started, it is necessary to define at least one boot configuration. A boot configuration is a named set of instructions that describe how the virtual machine is to be started up. The instructions may include the names of disk images that are to be inserted into emulated optical drives, the type of boot loader to be used, and/or how the boot loader is to locate an operating system kernel on any of the storage devices attached to the virtual machine. Because of the complexity of the information described within a boot configuration, there is no terse command analogous to vm-define that can be used to create a boot configuration. Instead, the boot configuration must be written in an XML file and added to the virtual machine with the vm-update-boot-configurations command. A simple configuration, named install, that boots OpenBSD from an official installer image inserted into the optical drive at device slot 0:4:0 using grub-bhyve might look as follows:
2

### 2.7.2 Example OpenBSD Install Boot Configuration

    <?xml version="1.0" encoding="UTF-8" ?>

    <BootConfigurations xmlns="urn:com.io7m.waxmill.vm:1:0">
      <BootConfigurationGRUBBhyve name="install">
        <BootDiskAttachments>
          <BootDiskAttachment>
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
            <StorageBackendFile path="/storage/images/openbsd/install67.iso"/>
          </BootDiskAttachment>
        </BootDiskAttachments>
        <GRUBBhyveKernelOpenBSD>
          <BSDBootDevice kernelPath="/6.7/amd64/bsd.rd">
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
          </BSDBootDevice>
        </GRUBBhyveKernelOpenBSD>
      </BootConfigurationGRUBBhyve>
    </BootConfigurations>

2.8

## Running A Virtual Machine

1
Once a virtual machine has been realized and has at least one boot configuration, it can be run using the vm-run command:
2

### 2.8.2 Running A Virtual Machine

    $ waxmill vm-run \
      --machine 538a90e4-d50d-4511-8643-ae418279bac4 \
      --boot-configuration install
    rdmsr to register 0x140 on vcpu 0
    rdmsr to register 0x34 on vcpu 0
    Unhandled ps2 mouse command 0xe1
    Unhandled ps2 mouse command 0x88

3
The above command attempts to run the virtual machine with ID 538a90e4-d50d-4511-8643-ae418279bac4, using the boot configuration named run. The waxmill command-line tool process is replaced with that of the bhyve virtual machine, and the bhyve process continues to run in the foreground until the user shuts it down. The vm-run command is designed to be executed by a process supervision system.
2.9

## Process Supervision

1
The Waxmill package is designed to be usable in environments that use a process supervision system. For example, assuming a virtual machine 3bab9914-dfe2-4a37-9c19-c7e9b06c854b has already been configured, it's possible to create runit run and finish scripts such as the following:
2

### 2.9.2 runit run

    #!/bin/sh
    exec /usr/local/bin/waxmill vm-run \
      --machine 3bab9914-dfe2-4a37-9c19-c7e9b06c854b \
      --boot-configuration normal

3

### 2.9.3 runit finish

    #!/bin/sh
    exec /usr/local/bin/waxmill vm-kill \
      --machine 3bab9914-dfe2-4a37-9c19-c7e9b06c854b \

4
These scripts will cause 3bab9914-dfe2-4a37-9c19-c7e9b06c854b to be started and supervised, and the kernel resources associated with the virtual machine will be properly cleaned up when the service is shut down or restarted.

## Footnotes

1
The Waxmill package always prefers to rely on explicit configuration data rather than reading untrusted data from the environment (such as the PATH environment variable).

#### Footnote References

- Footnote Reference 2.2.3

1. Orientation
Previous
Waxmill User Manual 0.0.2
Up
3. Command-Line Interface
Next
io7m | single-page | multi-page