io7m | single-page | multi-page

# Waxmill User Manual 0.0.3

ID | 402d2620-c5c9-4330-bc01-5430645815d6
Date | 2020-07-08

- 1. Orientation
  - 1.1. Overview

- 2. Usage
  - 2.1. Waxmill
  - 2.2. Paths And Environment
  - 2.3. Virtual Machines
  - 2.4. Defining A Virtual Machine
  - 2.5. Adding Devices
  - 2.6. Realizing A Virtual Machine
  - 2.7. Boot Configurations
  - 2.8. Running A Virtual Machine
  - 2.9. Process Supervision

- 3. Command-Line Interface
  - 3.1. Overview
  - 3.2. help
  - 3.3. schema
  - 3.4. version
  - 3.5. vm-add-ahci-disk
  - 3.6. vm-add-ahci-optical
  - 3.7. vm-add-e1000-network-device
  - 3.8. vm-add-framebuffer-device
  - 3.9. vm-add-lpc-device
  - 3.10. vm-add-passthru-device
  - 3.11. vm-add-virtio-disk
  - 3.12. vm-add-virtio-network-device
  - 3.13. vm-console
  - 3.14. vm-define
  - 3.15. vm-delete-boot-configurations
  - 3.16. vm-delete-devices
  - 3.17. vm-delete
  - 3.18. vm-export
  - 3.19. vm-import
  - 3.20. vm-kill
  - 3.21. vm-list-with-name
  - 3.22. vm-list
  - 3.23. vm-realize
  - 3.24. vm-run
  - 3.25. vm-set
  - 3.26. vm-update-boot-configurations

- 4. API
  - 4.1. JavaDoc

1

# Orientation

- 1. Orientation
  - 1.1. Overview
    - 1.1.1. Design Principles
    - 1.1.2. Plain Text Configuration
    - 1.1.3. Memory Safety, Type Safety
    - 1.1.4. Process Supervision

1.1

## Overview

1.1.1

## Design Principles

1
The Waxmill package provides a command-line interface and Java API for creating, configuring, and running bhyve virtual machines. The package follows a number of design principles in order to ensure stability and reliability.
1.1.2

## Plain Text Configuration

1
The Waxmill package stores configuration data for virtual machines in a strict, strongly-typed, versioned XML format. This ensures that virtual machine configurations created by older versions of the Waxmill package can continue to be understood by newer versions of Waxmill, or can be safely migrated to newer formats without any loss of information. Waxmill unconditionally validates all input against a schema (including data that the package wrote itself), and will refuse to work with malformed and/or invalid data. Additionally, the Waxmill package uses a hardened XML parser configuration that is expected to be immune to entity expansion attacks.
1.1.3

## Memory Safety, Type Safety

1
The Waxmill package is written in Java 11, using only a couple of native methods (fork() and execve()), and is therefore likely to be immune to buffer overflows and other memory-safety related bugs that plague the typical present-day system languages. The codebase is, additionally, subjected to extensive static checking on every build. Binary builds are expected to be byte-for-byte reproducible. Extensive use of static typing is used throughout the codebase to enforce invariants statically where possible, and the code is annotated with precondition/invariant/postcondition checks in order to induce bugs to cause early failures instead of silently corrupting state. The codebase also has a high-coverage test suite which exhaustively tests the various API methods and the entire command-line interface.
1.1.4

## Process Supervision

1
The Waxmill package is designed to work under process supervision. The supervision of system services has been a solved problem for decades although many pieces of server software are still not designed to work well under process supervision. Many virtual machine management systems still work with incredibly fragile rc-style scripts and pidfiles that instantly become stale and dangerous on virtual machine crashes. The Waxmill package provides a single command to start a virtual machine in the foreground such that the virtual machine can be supervised using systems such as daemontools, or runit. These tools can guarantee that a virtual machine always starts from a clean slate, and will always be reliably restarted should it crash.
2

# Usage

- 2. Usage
  - 2.1. Waxmill
  - 2.2. Paths And Environment
  - 2.3. Virtual Machines
  - 2.4. Defining A Virtual Machine
  - 2.5. Adding Devices
  - 2.6. Realizing A Virtual Machine
  - 2.7. Boot Configurations
  - 2.8. Running A Virtual Machine
  - 2.9. Process Supervision

2.1

## Waxmill

1
The primary means by which users use the Waxmill package is via the waxmill command-line tool.
2.2

## Paths And Environment

1
The waxmill command-line tool will attempt to read a configuration file specified by the WAXMILL_CONFIGURATION_FILE environment variable. The configuration file, by convention, should be stored at /etc/waxmill/config.xml. The default configuration file is as follows:
2

### 2.2.2 Configuration File

    <?xml version="1.0" encoding="UTF-8" ?>

    <Configuration xmlns="urn:com.io7m.waxmill.config:1:0">
      <Paths>
        <Path type="ZFSExecutable"
              value="/sbin/zfs"/>
        <Path type="GRUBBhyveExecutable"
              value="/usr/local/sbin/grub-bhyve"/>
        <Path type="BhyveExecutable"
              value="/usr/sbin/bhyve"/>
        <Path type="BhyveCtlExecutable"
              value="/usr/sbin/bhyvectl"/>
        <Path type="CuExecutable"
              value="/usr/bin/cu"/>
        <Path type="VirtualMachineConfigurationDirectory"
              value="/etc/waxmill/vm"/>
        <Path type="VirtualMachineRuntimeDirectory"
              value="/storage/vm"/>
        <Path type="IfconfigExecutable"
              value="/sbin/ifconfig"/>
      </Paths>
    </Configuration>

3
The configuration file explicitly specifies the locations of various system utilities [1], and also specifies the location of the virtual machine configuration directory and the virtual machine runtime directory. The virtual machine configuration directory is a directory consisting of XML configuration files for individual virtual machines. The virtual machine runtime directory is a directory containing one directory per virtual machine, each of which stores resources needed by virtual machines at run-time such as ZFS volumes, sockets, and lock files.
2.3

## Virtual Machines

1
A virtual machine, in Waxmill, carries a universally-unique identifier that is used to refer to the virtual machine during its entire lifetime. The identifier of a virtual machine cannot be changed after it is defined; this property of identifiers is useful to track virtual machines if they are migrated to new physical machines.
2
A virtual machine is first defined, then configured with one or more devices, then configured with one or more boot configurations, then realized, and finally run.
2.4

## Defining A Virtual Machine

1
Defining a virtual machine involves taking a description of the virtual machine and recording that description in the virtual machine configuration directory. This is typically accomplished using the vm-define command. Once a virtual machine is defined, it will appear in the list of defined virtual machines (which can be inspected using the vm-list command).
2

### 2.4.2 Defining A Virtual Machine

    $ waxmill vm-define \
      --name 'com.io7m.example4' \
      --cpu-count 4 \
      --comment 'An example machine' \
      --memory-gigabytes 1 \
      --memory-megabytes 512

    $ waxmill vm-list
    # ID                                     Name
    800a2dad-3367-4a98-879a-0fac219f55f4     com.io7m.example2
    a866f9e4-f6c1-4ecf-ad49-000925eefa0a     com.io7m.example4
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example
    7c2bcb79-f20c-47fa-812b-8994f17f97b2     com.io7m.example5

3
The definition of the created machine can be exported using the vm-export command. The command will simply print the XML configuration file to the standard output. This can be used to manually inspect the configuration of the machine, or to copy the configuration to a new physical machine.
4

### 2.4.4 Exporting A Virtual Machine

    $ waxmill vm-export --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a
    <?xml version="1.0" encoding="UTF-8"?>
    <wxm:VirtualMachines xmlns:wxm="urn:com.io7m.waxmill.vm:1:0">
        <wxm:VirtualMachine id="a866f9e4-f6c1-4ecf-ad49-000925eefa0a" name="com.io7m.example4">
            <wxm:Comment>An example machine</wxm:Comment>
            <wxm:CPUTopology sockets="1" threads="1" cores="4"/>
            <wxm:Memory gigabytes="1" megabytes="512"/>
            <wxm:Devices>
                <wxm:HostBridge vendor="UNSPECIFIED">
                    <wxm:DeviceSlot bus="0" slot="0" function="0"/>
                </wxm:HostBridge>
            </wxm:Devices>
            <wxm:BootConfigurations/>
            <wxm:Flags>
                <wxm:Flag name="DisableMPTableGeneration" enabled="false"/>
                <wxm:Flag name="ForceVirtualIOPCIToUseMSI" enabled="false"/>
                <wxm:Flag name="GenerateACPITables" enabled="true"/>
                <wxm:Flag name="GuestAPICIsX2APIC" enabled="false"/>
                <wxm:Flag name="IncludeGuestMemoryInCoreFiles" enabled="false"/>
                <wxm:Flag name="RealTimeClockIsUTC" enabled="false"/>
                <wxm:Flag name="WireGuestMemory" enabled="false"/>
                <wxm:Flag name="ExitCPUOnPAUSE" enabled="true"/>
                <wxm:Flag name="YieldCPUOnHLT" enabled="true"/>
            </wxm:Flags>
        </wxm:VirtualMachine>
    </wxm:VirtualMachines>

5
Similarly, the vm-import command can be used to import one or more virtual machines from a given XML configuration file.
6
bhyve virtual machines have a number of configurable boolean flags. The vm-set command can be used to turn these flags on and off. For example, using PCI passthru devices requires wiring the memory of the guest virtual machine.
2.5

## Adding Devices

1
A virtual machine needs to be configured with one or more devices in order to actually do useful work. Typically, a virtual machine will have one or more storage devices (emulated disks), one or more network devices (emulations of ethernet interfaces), and a console device (to allow access to the operating system console from the host machine). A device is attached to exactly one device slot at any given time. A device slot is a PCI slot value consisting of a 3-tuple (bus, pcislot, function), where bus ∈ [0, 255], pcislot ∈ [0,31], and function ∈ [0,7]. The concrete notation used to denote a device slot is bus:pcislot:function, so the string 1:5:3 denotes a device slot on bus 1, slot 5, function 3. In practice, virtual machines will typically specify all devices to use bus 0 and function 0, and only vary the slot value.
2
The waxmill command-line tool contains various subcommands to create and attach devices to virtual machines.
3

### 2.5.3 Device Commands

- vm-add-ahci-disk
- vm-add-ahci-optical
- vm-add-lpc-device
- vm-add-virtio-disk
- vm-add-virtio-network-device

4
The types of devices attached to a virtual machine will depend on the level of support that the guest operating system running on the virtual machine has for virtualization. Operating systems that have explicit support for virtualization (typically every major operating system with a release more recent than about 2007) will perform best using Virtio devices. Older operating systems may need to be presented with the illusion of having access to AHCI devices. Please see the documentation for each of the commands for details.
5
The Waxmill package can optionally automatically manage storage devices backed by ZFS volumes if requested. If, on a virtual machine with ID X, a storage device that is specified to use a zfs-volume backend is attached to slot a:b:c, the Waxmill package will automatically create a ZFS volume at /storage/vm/X/disk-a:b:c (where /storage/vm is the configured virtual machine runtime directory) when realizing the virtual machine. This is merely a convenience feature; if the ZFS volume already exists, it will be used and the Waxmill package will not attempt to modify it.
2.6

## Realizing A Virtual Machine

1
Realizing a virtual machine involves reading the description of the virtual machine from the virtual machine configuration directory and then creating any resources that are needed at run-time such as ZFS volumes. The vm-realize command is used to realize virtual machines:
2

### 2.6.2 Realizing A Virtual Machine

    $ waxmill vm-realize \
      --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a \
      --dry-run true
    /sbin/zfs create -V 10gb storage/vm/a866f9e4-f6c1-4ecf-ad49-000925eefa0a/disk-0_1_0

    $ waxmill vm-realize \
      --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a

    $ waxmill vm-realize \
      --machine a866f9e4-f6c1-4ecf-ad49-000925eefa0a
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMRealize: ZFS volume storage/vm/a866f9e4-f6c1-4ecf-ad49-000925eefa0a/disk-0_1_0 exists and has the correct size

2.7

## Boot Configurations

1
Before a virtual machine can actually be started, it is necessary to define at least one boot configuration. A boot configuration is a named set of instructions that describe how the virtual machine is to be started up. The instructions may include the names of disk images that are to be inserted into emulated optical drives, the type of boot loader to be used, and/or how the boot loader is to locate an operating system kernel on any of the storage devices attached to the virtual machine. Because of the complexity of the information described within a boot configuration, there is no terse command analogous to vm-define that can be used to create a boot configuration. Instead, the boot configuration must be written in an XML file and added to the virtual machine with the vm-update-boot-configurations command. A simple configuration, named install, that boots OpenBSD from an official installer image inserted into the optical drive at device slot 0:4:0 using grub-bhyve might look as follows:
2

### 2.7.2 Example OpenBSD Install Boot Configuration

    <?xml version="1.0" encoding="UTF-8" ?>

    <BootConfigurations xmlns="urn:com.io7m.waxmill.vm:1:0">
      <BootConfigurationGRUBBhyve name="install">
        <BootDiskAttachments>
          <BootDiskAttachment>
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
            <StorageBackendFile path="/storage/images/openbsd/install67.iso"/>
          </BootDiskAttachment>
        </BootDiskAttachments>
        <GRUBBhyveKernelOpenBSD>
          <BSDBootDevice kernelPath="/6.7/amd64/bsd.rd">
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
          </BSDBootDevice>
        </GRUBBhyveKernelOpenBSD>
      </BootConfigurationGRUBBhyve>
    </BootConfigurations>

2.8

## Running A Virtual Machine

1
Once a virtual machine has been realized and has at least one boot configuration, it can be run using the vm-run command:
2

### 2.8.2 Running A Virtual Machine

    $ waxmill vm-run \
      --machine 538a90e4-d50d-4511-8643-ae418279bac4 \
      --boot-configuration install
    rdmsr to register 0x140 on vcpu 0
    rdmsr to register 0x34 on vcpu 0
    Unhandled ps2 mouse command 0xe1
    Unhandled ps2 mouse command 0x88

3
The above command attempts to run the virtual machine with ID 538a90e4-d50d-4511-8643-ae418279bac4, using the boot configuration named run. The waxmill command-line tool process is replaced with that of the bhyve virtual machine, and the bhyve process continues to run in the foreground until the user shuts it down. The vm-run command is designed to be executed by a process supervision system.
2.9

## Process Supervision

1
The Waxmill package is designed to be usable in environments that use a process supervision system. For example, assuming a virtual machine 3bab9914-dfe2-4a37-9c19-c7e9b06c854b has already been configured, it's possible to create runit run and finish scripts such as the following:
2

### 2.9.2 runit run

    #!/bin/sh
    exec /usr/local/bin/waxmill vm-run \
      --machine 3bab9914-dfe2-4a37-9c19-c7e9b06c854b \
      --boot-configuration normal

3

### 2.9.3 runit finish

    #!/bin/sh
    exec /usr/local/bin/waxmill vm-kill \
      --machine 3bab9914-dfe2-4a37-9c19-c7e9b06c854b \

4
These scripts will cause 3bab9914-dfe2-4a37-9c19-c7e9b06c854b to be started and supervised, and the kernel resources associated with the virtual machine will be properly cleaned up when the service is shut down or restarted.
3

# Command-Line Interface

- 3. Command-Line Interface
  - 3.1. Overview
  - 3.2. help
  - 3.3. schema
  - 3.4. version
  - 3.5. vm-add-ahci-disk
  - 3.6. vm-add-ahci-optical
  - 3.7. vm-add-e1000-network-device
  - 3.8. vm-add-framebuffer-device
  - 3.9. vm-add-lpc-device
  - 3.10. vm-add-passthru-device
  - 3.11. vm-add-virtio-disk
  - 3.12. vm-add-virtio-network-device
  - 3.13. vm-console
  - 3.14. vm-define
  - 3.15. vm-delete-boot-configurations
  - 3.16. vm-delete-devices
  - 3.17. vm-delete
  - 3.18. vm-export
  - 3.19. vm-import
  - 3.20. vm-kill
  - 3.21. vm-list-with-name
  - 3.22. vm-list
  - 3.23. vm-realize
  - 3.24. vm-run
  - 3.25. vm-set
  - 3.26. vm-update-boot-configurations

3.1

# Overview

- 3.1. Overview
  - 3.1.1. Usage
  - 3.1.2. Verbosity
  - 3.1.3. @ Syntax
  - 3.1.4. Exit Code
  - 3.1.5. EBNF

3.1.1

## Usage

1
The waxmill package provides a command-line interface and API for creating, configuring, and running bhyve virtual machines. The base waxmill command is broken into a number of subcommands which are documented over the following sections.
2

### 3.1.1.2 Command-Line Overview

    INFO com.io7m.waxmill.cmdline.Main: Usage: waxmill [options] [command] [command options]

      Options:
        --verbose
          Set the minimum logging verbosity level.
          Default: info
          Possible Values: [trace, debug, info, warn, error]

      Use the "help" command to examine specific commands:

        $ waxmill help help.

      Command-line arguments can be placed one per line into a file, and the file
      can be referenced using the @ symbol:

        $ echo help > file.txt
        $ echo help >> file.txt
        $ waxmill @file.txt

      Commands:
        help                              Show detailed help messages for commands.
        schema                            Export schemas.
        version                           Show the application version.
        vm-add-ahci-disk                  Add an AHCI disk to a virtual machine.
        vm-add-ahci-optical               Add an AHCI optical drive to a virtual machine.
        vm-add-e1000-network-device       Add an e1000 network device to a virtual machine.
        vm-add-framebuffer-device         Add a framebuffer device to a virtual machine.
        vm-add-lpc-device                 Add an LPC device to a virtual machine.
        vm-add-passthru-device            Add a PCI passthru device to a virtual machine.
        vm-add-virtio-disk                Add a virtio disk to a virtual machine.
        vm-add-virtio-network-device      Add a virtio network device to a virtual machine.
        vm-console                        Connect to the console of a virtual machine
        vm-define                         Define a new virtual machine.
        vm-delete                         Delete a virtual machine.
        vm-delete-boot-configurations     Delete boot configurations from a virtual machine.
        vm-delete-devices                 Delete devices from virtual machines.
        vm-export                         Export virtual machine descriptions.
        vm-import                         Import virtual machine descriptions.
        vm-kill                           Kill a virtual machine.
        vm-list                           List the available virtual machines.
        vm-list-with-name                 List the virtual machines with the given name.
        vm-realize                        Realize a virtual machine.
        vm-run                            Run a virtual machine.
        vm-set                            Set virtual machine configuration flags.
        vm-update-boot-configurations     Add/replace boot configurations in a virtual machine.

      Documentation:
        https://www.io7m.com/software/waxmill/documentation/

3.1.2

## Verbosity

1
All subcommands accept a --verbose parameter that may be set to one of trace, debug, info, warn, or error. This parameter sets the lower bound for the severity of messages that will be logged. For example, at debug verbosity, only messages of severity debug and above will be logged. Setting the verbosity to trace level effectively causes everything to be logged, and will produce large volumes of debugging output.
2

### 3.1.2.2 Log Levels

Name | Description
--- | ---
trace | All messages greater than or equal to trace level (highest)
debug | All messages greater than or equal to debug level
info | All messages greater than or equal to info level
warn | All messages greater than or equal to warn level
error | All messages greater than or equal to error level

3.1.3

## @ Syntax

1
The waxmill command-line tool uses jcommander to parse command-line arguments, and therefore supports placing command-line arguments into a file, one argument per line, and then referencing that file with @. For example:
2

### 3.1.3.2 @ Syntax

    $ waxmill vm-list
    # ID                                     Name
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example

    $ waxmill vm-define --name com.io7m.example2

    $ waxmill vm-list
    # ID                                     Name
    800a2dad-3367-4a98-879a-0fac219f55f4     com.io7m.example2
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example

    $ (cat <<EOF
    vm-define
    --name
    com.io7m.example3
    EOF
    ) > args.txt

    $ waxmill @args.txt

    $ waxmill vm-list
    # ID                                     Name
    800a2dad-3367-4a98-879a-0fac219f55f4     com.io7m.example2
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example
    6730354e-bcba-4636-ab07-ad853c375f7d     com.io7m.example3

3.1.4

## Exit Code

1
All subcommands, unless otherwise specified, yield an exit code of 0 on success, and a non-zero exit code on failure.
3.1.5

## EBNF

1
The specification gives grammar definitions in ISO/IEC 14977:1996 Extended Backus-Naur form.
3.2

# help

- 3.2. help
  - 3.2.1. Name
  - 3.2.2. Description
  - 3.2.3. Example

3.2.1

## Name

1
help - Show help on any command
3.2.2

## Description

1
The help command shows extended help information for other commands.
2

### 3.2.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.2.3

## Example

1

### 3.2.3.1 Example

    $ waxmill help help
    INFO com.io7m.waxmill.cmdline.Main: Usage: help [options] command

      Options:
        --verbose
          Set the minimum logging verbosity level.
          Default: info
          Possible Values: [trace, debug, info, warn, error]

      The "help" command, executed without arguments, shows the names of all
      commands including the names and types of all of their parameters.

      The "help" command can also be passed the name of a command. Commands
      may include extended help messages such as the one you are reading
      right now.

      Example: waxmill help help

    $ ./waxmill help vm-console
    INFO com.io7m.waxmill.cmdline.Main: Usage: vm-console [options]

      Options:
        --configuration
          The path to the configuration file (environment variable:
          $WAXMILL_CONFIGURATION_FILE)
        --dry-run
          Show the commands that would be executed, but do not execute them.
          Default: false
      * --machine
          The ID of the virtual machine
        --verbose
          Set the minimum logging verbosity level.
          Default: info
          Possible Values: [trace, debug, info, warn, error]

      The "vm-console" command attempts to connect to the primary console
      for the given virtual machine. In practice, this means executing the
      "cu"[0] program on the device node that represents the console for
      the VM. The command will fail with a diagnostic error message if the
      virtual machine has no console, or has multiple consoles.

      [0] https://www.freebsd.org/cgi/man.cgi?query=cu&sektion=1

3.3

# schema

- 3.3. schema
  - 3.3.1. Name
  - 3.3.2. Description
  - 3.3.3. Example

3.3.1

## Name

1
schema - Export XSD schemas
3.3.2

## Description

1
The schema command lists the available XSD schemas available for the various XML formats used by the waxmill tools. The command can dump the text of specific schemas when passed the identifier of a schema.
2

### 3.3.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--id | URI | false | The schema identifier
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.3.3

## Example

1

### 3.3.3.1 Example

    $ waxmill schema
    urn:com.io7m.waxmill.config:1:0
    urn:com.io7m.waxmill.vm:1:0

    $ waxmill schema --id urn:com.io7m.waxmill.config:1:0
    <?xml version="1.0" encoding="UTF-8" ?>

    <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                xmlns:wxmc="urn:com.io7m.waxmill.config:1:0"
                targetNamespace="urn:com.io7m.waxmill.config:1:0">

      <xsd:annotation>
        <xsd:documentation>
          A schema describing the format of Waxmill configuration files.
        </xsd:documentation>
      </xsd:annotation>

      <xsd:simpleType name="PathType">
        <xsd:annotation>
          <xsd:documentation>
            The type of configuration paths.
          </xsd:documentation>
        </xsd:annotation>

        <xsd:restriction base="xsd:string">
          <xsd:enumeration value="BhyveExecutable">
            <xsd:annotation>
              <xsd:documentation>
                The path to the bhyve executable. On FreeBSD, this is /usr/sbin/bhyve by default.
    ...

3.4

# version

- 3.4. version
  - 3.4.1. Name
  - 3.4.2. Description
  - 3.4.3. Example

3.4.1

## Name

1
version - Show the Waxmill version
3.4.2

## Description

1
The version command shows the Waxmill application version.
2

### 3.4.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.4.3

## Example

1

### 3.4.3.1 Example

    $ waxmill version
    Waxmill 0.0.1-SNAPSHOT

3.5

# vm-add-ahci-disk

- 3.5. vm-add-ahci-disk
  - 3.5.1. Name
  - 3.5.2. Description
  - 3.5.3. Example

3.5.1

## Name

1
vm-add-ahci-disk - Add an AHCI disk to a virtual machine
3.5.2

## Description

1
The vm-add-ahci-disk command adds a disk to a virtual machine. The disk is an emulation of an AHCI controller attached to a SATA hard-drive and, as such, requires a driver in the guest operating system that is capable of supporting AHCI.
2

### 3.5.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--backend | Storage Backend | true | A specification of the AHCI storage device backend to add
--comment | String | false | A comment describing the new disk
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--machine | UUID | true | The ID of the virtual machine
--open-option | OpenOption | false | The options that will be used when opening the storage device.
--verbose | Log Level | false | Set the minimum logging verbosity level.
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

3

### 3.5.2.3 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

4

### 3.5.2.4 Storage Backend Syntax

    path           = ? Any UNIX-like path ? ;
    size           = ? Any non-negative multiple of 128000 (bytes) ? ;
    fileBackend    = "file" , ";" , path ;
    zfsBackend     = "zfs-volume" , [ ";" , size ] ;
    storageBackend = fileBackend | zfsBackend ;

    Examples:
      file;/tmp/xyz
      zfs-volume
      zfs-volume;1280000000

5

### 3.5.2.5 Open Options

Name | Description
--- | ---
NO_CACHE | Do not use caching (O_DIRECT).
SYNCHRONOUS | Writes to the file are synchronous (O_SYNC).
READ_ONLY | The file is opened read-only.

3.5.3

## Example

1

### 3.5.3.1 Example

    $ waxmill vm-add-ahci-disk \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --backend file;/tmp/disk.img

3.6

# vm-add-ahci-optical

- 3.6. vm-add-ahci-optical
  - 3.6.1. Name
  - 3.6.2. Description
  - 3.6.3. Example

3.6.1

## Name

1
vm-add-ahci-optical - Add an AHCI optical drive to a virtual machine
3.6.2

## Description

1
The vm-add-ahci-optical command adds an optical drive to a virtual machine.
2
Optical drives do not have any storage backends assigned to them by default. The storage backend for an optical drive must be specified by a boot configuration.
3

### 3.6.2.3 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--comment | String | false | A comment describing the new disk
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

4

### 3.6.2.4 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

3.6.3

## Example

1

### 3.6.3.1 Example

    $ waxmill vm-add-ahci-optical \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0

3.7

# vm-add-e1000-network-device

- 3.7. vm-add-e1000-network-device
  - 3.7.1. Name
  - 3.7.2. Description
  - 3.7.3. Example

3.7.1

## Name

1
vm-add-e1000-network-device - Add an e1000 network device to a virtual machine
3.7.2

## Description

1
The vm-add-e1000-network-device command adds a network device to a virtual machine. The device is an emulation of an Intel e82545 network device and, as such, requires a driver in the guest operating system that is capable of supporting that device.
2

### 3.7.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--backend | Network Backend | true | A specification of the network device backend to add
--comment | String | false | A comment describing the new disk
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

3

### 3.7.2.3 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

4

### 3.7.2.4 Network Backend Syntax

    digit           = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;
    hexDigit        = "a" | "b" | "c" | "d" | "e" | "f" ;
    lowerAlpha      = ? [a-z] ? ;
    upperAlpha      = ? [A-Z] ? ;
    groupCharacter  = lowerAlpha | upperAlpha | "_" ;
    groupName       = { groupCharacter } ;
    groupList       = groupName , { "," , groupName } ;

    macPart         = hexDigit , hexDigit ;
    macAddress      = macPart , ":" , macPart , ":" , macPart , ":" , macPart , ":" , macPart , ":" , macPart ;
    tapDeviceName   = "tap" , '{' digit '}' ;
    vmnetDeviceName = "vmnet" , '{' digit '}' ;
    tapDevice       = "tap" , ";" , tapDeviceName , ";" , macAddress , [ ";" groupList ] ;
    vmnetDevice     = "vmnet" , ";" , vmnetDeviceName , ";" , macAddress , [ ";" groupList ] ;

    Examples:
      tap;tap23;d9:63:c0:d9:09:e8
      tap;tap0000000000001;d9:63:c0:d9:09:e8
      tap;tap23;d9:63:c0:d9:09:e8;wwwUsers,mailUsers,ntpdUsers
      vmnet;vmnet23;d9:63:c0:d9:09:e8
      vmnet;vmnet00000000001;d9:63:c0:d9:09:e8
      vmnet;vmnet23;d9:63:c0:d9:09:e8;wwwUsers,mailUsers,ntpdUsers

3.7.3

## Example

1

### 3.7.3.1 Example

    $ waxmill vm-add-e1000-network-device \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --backend tap;tap23;d9:63:c0:d9:09:e8

3.8

# vm-add-framebuffer-device

- 3.8. vm-add-framebuffer-device
  - 3.8.1. Name
  - 3.8.2. Description
  - 3.8.3. Example

3.8.1

## Name

1
vm-add-framebuffer-device - Add a framebuffer device to a virtual machine
3.8.2

## Description

1
The vm-add-framebuffer-device command adds a framebuffer device to a virtual machine.
2

### 3.8.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--comment | String | false | A comment describing the new device
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--height | Integer | false | The framebuffer height
--listen-address | String | false | The VNC server listen address
--listen-port | Integer | false | The VNC server listen port
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.
--vga-configuration | VGAConfiguration | false | The guest VGA configuration
--wait-for-vnc | Boolean | false | Will cause the machine to wait for a VNC connection before booting
--width | Integer | false | The framebuffer width
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

3

### 3.8.2.3 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

4

### 3.8.2.4 Address Syntax

    ipv4Part = ? [0 .. 255]{1,3} ? ;
    ipv4     = ipv4Part , ".", ipv4Part , "." , ipv4Part , "." , ipv4Part , ;
    ipv6     = ? Any https://tools.ietf.org/html/rfc4291 IPv6 address ? ;
    name     = ? Any host name ? ;
    address  = ipv4 | ipv6 | name ;

    Examples:
      127.0.0.1
      [::1]
      [2600:1417:6000:198::255e]
      localhost
      www.example.com

3.8.3

## Example

1

### 3.8.3.1 Example

    $ waxmill vm-add-framebuffer-device \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --wait-for-vnc true \
      --width 1024 \
      --height 768 \
      --listen-address [::1] \
      --listen-port 5901 \
      --vga-configuration OFF

3.9

# vm-add-lpc-device

- 3.9. vm-add-lpc-device
  - 3.9.1. Name
  - 3.9.2. Description
  - 3.9.3. Example

3.9.1

## Name

1
vm-add-lpc-device - Add an LPC device to a virtual machine
3.9.2

## Description

1
The vm-add-lpc-device command adds an LPC device to a virtual machine. The bhyve documentation describes such devices as "LPC PCI-ISA bridges with COM1 and COM2 16550 serial ports and a boot ROM". In practice, LPC devices are used to provide console access to virtual machines, and UEFI firmware to virtual machines that are booting from UEFI.
2

### 3.9.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--add-backend | TTY Backend | true | A specification of the TTY device backend to add
--comment | String | false | A comment describing the new device
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

3

### 3.9.2.3 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

4

### 3.9.2.4 TTY Backend Syntax

    path         = ? Any unix-like path ? ;
    port         = "com1" | "com2" | "bootrom" ;
    stdioBackend = "stdio" , ";" , port ;
    fileBackend  = "file" , ";" , port , ";" , path ;
    nmdmBackend  = "nmdm" , ";" , port ;
    ttyBackend   = stdioBackend | fileBackend | nmdmBackend ;

    Examples:
      stdio;com1
      file;bootrom;/tmp/xyz
      nmdm;com2

3.9.3

## Example

1

### 3.9.3.1 Example

    $ waxmill vm-add-lpc-device \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --add-backend stdio;com1

    $ waxmill vm-add-lpc-device \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --add-backend file;bootrom;/usr/local/share/uefi-firmware/BHYVE_UEFI.fd

    $ waxmill vm-add-lpc-device \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --add-backend nmdm;com1

3.10

# vm-add-passthru-device

- 3.10. vm-add-passthru-device
  - 3.10.1. Name
  - 3.10.2. Description
  - 3.10.3. Example
  - 3.10.4. See Also

3.10.1

## Name

1
vm-add-passthru-device - Add a PCI passthru device to a virtual machine
3.10.2

## Description

1
The vm-add-passthru-device command adds a PCI passthru device to the virtual machine. Using a PCI passthru device requires wiring the guest virtual machine's memory - see the vm-set command.
2

### 3.10.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--add-backend | TTY Backend | true | A specification of the TTY device backend to add
--comment | String | false | A comment describing the new device
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--host-device-slot | Device Slot | true | The host device.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

3

### 3.10.2.3 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

3.10.3

## Example

1

### 3.10.3.1 Example

    $ waxmill vm-add-passthru-device \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --host-device-slot 1:2:3

3.10.4

## See Also

1
See bhyve PCI Passthrough on the FreeBSD wiki.
3.11

# vm-add-virtio-disk

- 3.11. vm-add-virtio-disk
  - 3.11.1. Name
  - 3.11.2. Description
  - 3.11.3. Example

3.11.1

## Name

1
vm-add-virtio-disk - Add a Virtio disk to a virtual machine
3.11.2

## Description

1
The vm-add-virtio-disk command adds a disk to a virtual machine. The disk is a Virtio block storage device and, as such, requires a driver in the guest operating system that is capable of supporting Virtio.
2

### 3.11.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--backend | Storage Backend | true | A specification of the Virtio storage device backend to add
--comment | String | false | A comment describing the new disk
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--machine | UUID | true | The ID of the virtual machine
--open-option | OpenOption | false | The options that will be used when opening the storage device.
--verbose | Log Level | false | Set the minimum logging verbosity level.
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

3

### 3.11.2.3 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

4

### 3.11.2.4 Storage Backend Syntax

    path           = ? Any UNIX-like path ? ;
    size           = ? Any non-negative multiple of 128000 (bytes) ? ;
    fileBackend    = "file" , ";" , path ;
    zfsBackend     = "zfs-volume" , [ ";" , size ] ;
    storageBackend = fileBackend | zfsBackend ;

    Examples:
      file;/tmp/xyz
      zfs-volume
      zfs-volume;1280000000

5

### 3.11.2.5 Open Options

Name | Description
--- | ---
NO_CACHE | Do not use caching (O_DIRECT).
SYNCHRONOUS | Writes to the file are synchronous (O_SYNC).
READ_ONLY | The file is opened read-only.

3.11.3

## Example

1

### 3.11.3.1 Example

    $ waxmill vm-add-virtio-disk \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --backend file;/tmp/disk.img

3.12

# vm-add-virtio-network-device

- 3.12. vm-add-virtio-network-device
  - 3.12.1. Name
  - 3.12.2. Description
  - 3.12.3. Example

3.12.1

## Name

1
vm-add-virtio-network-device - Add a Virtio network device to a virtual machine
3.12.2

## Description

1
The vm-add-virtio-network-device command adds a network device to a virtual machine. The device is a Virtio network device and, as such, requires a driver in the guest operating system that is capable of supporting Virtio.
2

### 3.12.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--backend | Network Backend | true | A specification of the network device backend to add
--comment | String | false | A comment describing the new disk
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | true | The slot to which the device will be attached.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.
--replace | Boolean | false | true if an existing device in the given slot should be replaced.

3

### 3.12.2.3 Device Slot Syntax

    bus        = ? [0 .. 255] ? ;
    slot       = ? [0 .. 31] ? ;
    function   = ? [0 .. 7] ? ;
    deviceSlot = bus , ":" , slot , ":" , function ;

    Examples:
      0:0:0
      255:31:7

4

### 3.12.2.4 Network Backend Syntax

    digit           = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;
    hexDigit        = "a" | "b" | "c" | "d" | "e" | "f" ;
    lowerAlpha      = ? [a-z] ? ;
    upperAlpha      = ? [A-Z] ? ;
    groupCharacter  = lowerAlpha | upperAlpha | "_" ;
    groupName       = { groupCharacter } ;
    groupList       = groupName , { "," , groupName } ;

    macPart         = hexDigit , hexDigit ;
    macAddress      = macPart , ":" , macPart , ":" , macPart , ":" , macPart , ":" , macPart , ":" , macPart ;
    tapDeviceName   = "tap" , '{' digit '}' ;
    vmnetDeviceName = "vmnet" , '{' digit '}' ;
    tapDevice       = "tap" , ";" , tapDeviceName , ";" , macAddress , [ ";" groupList ] ;
    vmnetDevice     = "vmnet" , ";" , vmnetDeviceName , ";" , macAddress , [ ";" groupList ] ;

    Examples:
      tap;tap23;d9:63:c0:d9:09:e8
      tap;tap0000000000001;d9:63:c0:d9:09:e8
      tap;tap23;d9:63:c0:d9:09:e8;wwwUsers,mailUsers,ntpdUsers
      vmnet;vmnet23;d9:63:c0:d9:09:e8
      vmnet;vmnet00000000001;d9:63:c0:d9:09:e8
      vmnet;vmnet23;d9:63:c0:d9:09:e8;wwwUsers,mailUsers,ntpdUsers

3.12.3

## Example

1

### 3.12.3.1 Example

    $ waxmill vm-add-virtio-network-device \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --device-slot 0:4:0 \
      --backend tap;tap23;d9:63:c0:d9:09:e8

3.13

# vm-console

- 3.13. vm-console
  - 3.13.1. Name
  - 3.13.2. Description
  - 3.13.3. Example

3.13.1

## Name

1
vm-console - Connect to the console a virtual machine
3.13.2

## Description

1
The vm-console command attempts to connect to the primary console for the given virtual machine. In practice, this means executing the cu program on the host device node that represents the console for the VM. The command will fail with a diagnostic error message if the virtual machine has no console, or has multiple consoles.
2

### 3.13.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--comment | String | false | A comment describing the new disk
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--dry-run | Boolean | false | Show the commands that would be executed, but do not execute them.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.13.3

## Example

1

### 3.13.3.1 Example

    $ waxmill vm-console \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd \
      --dry-run true
    /usr/bin/cu -l /dev/nmdm_de17174a-1b3c-4ebc-80af-cb239aa18cbd_B

    $ waxmill vm-console \
      --machine de17174a-1b3c-4ebc-80af-cb239aa18cbd
    Connected

    OpenBSD/amd64 <931.461.60231.14.vt920> (tty00)

    login:

3.14

# vm-define

- 3.14. vm-define
  - 3.14.1. Name
  - 3.14.2. Description
  - 3.14.3. Example

3.14.1

## Name

1
vm-define - Define virtual machines
3.14.2

## Description

1
The vm-define command defines a new virtual machine. The command will define the configuration of the virtual machine but will not create any of the underlying resources such as ZFS filesystems and/or volumes. The command will, by default, generate a new unique machine ID unless one is manually specified with the --machine option.
2

### 3.14.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--comment | java.lang.String | false | A comment describing the new virtual machine
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--cpu-count | int | false | The number of CPU cores in the virtual machine
--machine | UUID | false | The ID of the new virtual machine
--memory-gigabytes | long | false | The size in gigabytes of the virtual machine's memory (added to --memory-megabytes)
--memory-megabytes | long | false | The size in megabytes of the virtual machine's memory (added to --memory-gigabytes)
--name | String | true | The name of the new virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.14.3

## Example

1

### 3.14.3.1 Example

    $ waxmill vm-define \
      --name 'com.io7m.example4' \
      --cpu-count 4 \
      --comment 'An example machine' \
      --memory-gigabytes 1 \
      --memory-megabytes 512

    $ waxmill vm-define \
      --name 'com.io7m.example5' \
      --cpu-count 4 \
      --comment 'An example machine' \
      --memory-gigabytes 1 \
      --memory-megabytes 512 \
      --machine 7c2bcb79-f20c-47fa-812b-8994f17f97b2

    $ waxmill vm-define \
      --name 'com.io7m.example5' \
      --cpu-count 4 \
      --comment 'An example machine' \
      --memory-gigabytes 1 \
      --memory-megabytes 512 \
      --machine 7c2bcb79-f20c-47fa-812b-8994f17f97b2
    ERROR com.io7m.waxmill.cmdline.Main: com.io7m.waxmill.exceptions.WXMExceptionDuplicate: A virtual machine already exists with the given ID.
      ID:                  7c2bcb79-f20c-47fa-812b-8994f17f97b2
      Machine (A) Name:    com.io7m.example5
      Machine (A) Source:  <unspecified>
      Machine (B) Name:    com.io7m.example5
      Machine (B) Source:  file:///etc/waxmill/vm/7c2bcb79-f20c-47fa-812b-8994f17f97b2.wvmx

3.15

# vm-delete-boot-configurations

- 3.15. vm-delete-boot-configurations
  - 3.15.1. Name
  - 3.15.2. Description
  - 3.15.3. Example

3.15.1

## Name

1
vm-delete-boot-configurations - Delete boot configurations for a virtual machine
3.15.2

## Description

1
The vm-delete-boot-configurations removes boot configurations from a given virtual machine.
2

### 3.15.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--file | Path | true | A file containing boot configurations
--machine | UUID | true | The ID of the virtual machine
--name | Boot Configuration | false | The names of boot configurations
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.15.3

## Example

1

### 3.15.3.1 Example

    $ waxmill vm-delete-boot-configurations --machine 538a90e4-d50d-4511-8643-ae418279bac4 --name nonexistent
    ERROR com.io7m.waxmill.cmdline.internal.WXMCommandVMDeleteBootConfigurations: No boot configuration exists with the given name: nonexistent

    $ waxmill vm-delete-boot-configurations --machine 538a90e4-d50d-4511-8643-ae418279bac4 --name install
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMDeleteBootConfigurations: Deleted boot configuration: install

3.16

# vm-delete-devices

- 3.16. vm-delete-devices
  - 3.16.1. Name
  - 3.16.2. Description
  - 3.16.3. Example

3.16.1

## Name

1
vm-delete-devices - Delete devices from a virtual machine
3.16.2

## Description

1
The vm-delete-devices removes devices from a given virtual machine. If any of the specified devices are referenced by a boot configuration on the virtual machine, the boot configuration must first either be deleted, or modified such that it no longer refers to the device.
2

### 3.16.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--device-slot | Device Slot | false | The devices to remove
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.16.3

## Example

1

### 3.16.3.1 Example

    $ waxmill vm-delete-devices --machine 538a90e4-d50d-4511-8643-ae418279bac4 --device-slot 0:10:0
    ERROR com.io7m.waxmill.cmdline.internal.WXMCommandVMDeleteDevice: Device does not exist: 0:10:0

    $ waxmill vm-delete-devices --machine 538a90e4-d50d-4511-8643-ae418279bac4 --device-slot 0:3:0
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMDeleteDevice: Deleted device 0:3:0

    $ waxmill vm-delete-devices --machine 538a90e4-d50d-4511-8643-ae418279bac4 --device-slot 0:2:0
    ERROR com.io7m.waxmill.cmdline.internal.WXMCommandVMDeleteDevice: One or more boot configurations are using the device.
      Device:              0:2:0
      Boot configurations: [run]

3.17

# vm-delete

- 3.17. vm-delete
  - 3.17.1. Name
  - 3.17.2. Description
  - 3.17.3. Example

3.17.1

## Name

1
vm-delete - Delete defined virtual machines
3.17.2

## Description

1
The vm-delete command deletes the configuration for a virtual machine. It will not stop any running instance of the given virtual machine, nor will it delete the underlying resources such as ZFS volumes.
2

### 3.17.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--verbose | Log Level | false | Set the minimum logging verbosity level.
--machine | UUID | true | The ID of the virtual machine.

3.17.3

## Example

1

### 3.17.3.1 Example

    $ waxmill vm-list
    # ID                                     Name
    800a2dad-3367-4a98-879a-0fac219f55f4     com.io7m.example2
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example
    6730354e-bcba-4636-ab07-ad853c375f7d     com.io7m.example3

    $ waxmill vm-delete --machine 6730354e-bcba-4636-ab07-ad853c375f7d

    $ waxmill vm-list
    # ID                                     Name
    800a2dad-3367-4a98-879a-0fac219f55f4     com.io7m.example2
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example

    $ waxmill vm-delete --machine 6730354e-bcba-4636-ab07-ad853c375f7d
    ERROR com.io7m.waxmill.cmdline.Main: com.io7m.waxmill.exceptions.WXMExceptionNonexistent: The specified machine does not exist.
      Machine: 6730354e-bcba-4636-ab07-ad853c375f7d

    ERROR com.io7m.waxmill.cmdline.Main: Caused by: java.nio.file.NoSuchFileException: /etc/waxmill/vm/6730354e-bcba-4636-ab07-ad853c375f7d.wvmx

3.18

# vm-export

- 3.18. vm-export
  - 3.18.1. Name
  - 3.18.2. Description
  - 3.18.3. Example

3.18.1

## Name

1
vm-export - Export defined virtual machines
3.18.2

## Description

1
The vm-export command exports the configuration data for a virtual machine. This command is the inverse of the vm-import command.
2

### 3.18.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--verbose | Log Level | false | Set the minimum logging verbosity level.
--machine | UUID | true | The ID of the virtual machine.

3.18.3

## Example

1

### 3.18.3.1 Example

    $ waxmill vm-export --machine 538a90e4-d50d-4511-8643-ae418279bac4
    <?xml version="1.0" encoding="UTF-8"?>
    <wxm:VirtualMachines xmlns:wxm="urn:com.io7m.waxmill.vm:1:0">
        <wxm:VirtualMachine id="538a90e4-d50d-4511-8643-ae418279bac4" name="com.io7m.example">
            <wxm:CPUTopology sockets="1" threads="1" cores="1"/>
            <wxm:Memory gigabytes="0" megabytes="250"/>
            <wxm:Devices>
                <wxm:HostBridge vendor="UNSPECIFIED">
                    <wxm:DeviceSlot bus="0" slot="0" function="0"/>
                </wxm:HostBridge>
                <wxm:AHCIDiskDevice>
                    <wxm:DeviceSlot bus="0" slot="1" function="0"/>
                    <wxm:StorageBackendZFSVolume/>
                </wxm:AHCIDiskDevice>
                <wxm:AHCIOpticalDiskDevice>
                    <wxm:DeviceSlot bus="0" slot="2" function="0"/>
                </wxm:AHCIOpticalDiskDevice>
                <wxm:VirtioNetworkDevice>
                    <wxm:DeviceSlot bus="0" slot="4" function="0"/>
                    <wxm:TAPDevice name="tap23" address="a3:26:9c:74:79:34"/>
                </wxm:VirtioNetworkDevice>
            </wxm:Devices>
            <wxm:BootConfigurations>
                <wxm:BootConfigurationGRUBBhyve name="run">
                    <wxm:GRUBBhyveKernelOpenBSD>
                        <wxm:BSDBootDevice kernelPath="/bsd">
                            <wxm:DeviceSlot bus="0" slot="2" function="0"/>
                        </wxm:BSDBootDevice>
                    </wxm:GRUBBhyveKernelOpenBSD>
                </wxm:BootConfigurationGRUBBhyve>
            </wxm:BootConfigurations>
            <wxm:Flags>
                <wxm:Flag name="DisableMPTableGeneration" enabled="false"/>
                <wxm:Flag name="ForceVirtualIOPCIToUseMSI" enabled="false"/>
                <wxm:Flag name="GenerateACPITables" enabled="true"/>
                <wxm:Flag name="GuestAPICIsX2APIC" enabled="false"/>
                <wxm:Flag name="IncludeGuestMemoryInCoreFiles" enabled="false"/>
                <wxm:Flag name="RealTimeClockIsUTC" enabled="false"/>
                <wxm:Flag name="WireGuestMemory" enabled="false"/>
                <wxm:Flag name="ExitCPUOnPAUSE" enabled="true"/>
                <wxm:Flag name="YieldCPUOnHLT" enabled="true"/>
            </wxm:Flags>
        </wxm:VirtualMachine>
    </wxm:VirtualMachines>

3.19

# vm-import

- 3.19. vm-import
  - 3.19.1. Name
  - 3.19.2. Description
  - 3.19.3. Example

3.19.1

## Name

1
vm-import - Import defined virtual machines
3.19.2

## Description

1
The vm-import command imports configuration data for a virtual machine. This command is the inverse of the vm-export command.
2

### 3.19.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--verbose | Log Level | false | Set the minimum logging verbosity level.
--file | Path | true | A file containing one or more virtual machine configurations.

3.19.3

## Example

1

### 3.19.3.1 Example

    $ cat vm.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <wxm:VirtualMachines xmlns:wxm="urn:com.io7m.waxmill.vm:1:0">
        <wxm:VirtualMachine id="538a90e4-d50d-4511-8643-ae418279bac4" name="com.io7m.example">
            <wxm:CPUTopology sockets="1" threads="1" cores="1"/>
            <wxm:Memory gigabytes="0" megabytes="250"/>
            <wxm:Devices>
                <wxm:HostBridge vendor="UNSPECIFIED">
                    <wxm:DeviceSlot bus="0" slot="0" function="0"/>
                </wxm:HostBridge>
                <wxm:AHCIDiskDevice>
                    <wxm:DeviceSlot bus="0" slot="1" function="0"/>
                    <wxm:StorageBackendZFSVolume/>
                </wxm:AHCIDiskDevice>
                <wxm:AHCIOpticalDiskDevice>
                    <wxm:DeviceSlot bus="0" slot="2" function="0"/>
                </wxm:AHCIOpticalDiskDevice>
                <wxm:VirtioNetworkDevice>
                    <wxm:DeviceSlot bus="0" slot="4" function="0"/>
                    <wxm:TAPDevice name="tap23" address="a3:26:9c:74:79:34"/>
                </wxm:VirtioNetworkDevice>
            </wxm:Devices>
            <wxm:BootConfigurations>
                <wxm:BootConfigurationGRUBBhyve name="run">
                    <wxm:GRUBBhyveKernelOpenBSD>
                        <wxm:BSDBootDevice kernelPath="/bsd">
                            <wxm:DeviceSlot bus="0" slot="2" function="0"/>
                        </wxm:BSDBootDevice>
                    </wxm:GRUBBhyveKernelOpenBSD>
                </wxm:BootConfigurationGRUBBhyve>
            </wxm:BootConfigurations>
            <wxm:Flags>
                <wxm:Flag name="DisableMPTableGeneration" enabled="false"/>
                <wxm:Flag name="ForceVirtualIOPCIToUseMSI" enabled="false"/>
                <wxm:Flag name="GenerateACPITables" enabled="true"/>
                <wxm:Flag name="GuestAPICIsX2APIC" enabled="false"/>
                <wxm:Flag name="IncludeGuestMemoryInCoreFiles" enabled="false"/>
                <wxm:Flag name="RealTimeClockIsUTC" enabled="false"/>
                <wxm:Flag name="WireGuestMemory" enabled="false"/>
                <wxm:Flag name="ExitCPUOnPAUSE" enabled="true"/>
                <wxm:Flag name="YieldCPUOnHLT" enabled="true"/>
            </wxm:Flags>
        </wxm:VirtualMachine>
    </wxm:VirtualMachines>

    $ waxmill vm-import --file vm.xml
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMImport: Imported 1 virtual machines

    $ waxmill vm-import --file vm.xml
    ERROR com.io7m.waxmill.cmdline.Main: com.io7m.waxmill.exceptions.WXMExceptionDuplicate: A virtual machine already exists with the given ID.
      ID:                  538a90e4-d50d-4511-8643-ae418279bac4
      Machine (A) Name:    com.io7m.example
      Machine (A) Source:  file:///tmp/vm.xml
      Machine (B) Name:    com.io7m.example
      Machine (B) Source:  file:///etc/waxmill/vm/538a90e4-d50d-4511-8643-ae418279bac4.wvmx

3.20

# vm-kill

- 3.20. vm-kill
  - 3.20.1. Name
  - 3.20.2. Description
  - 3.20.3. Example

3.20.1

## Name

1
vm-kill - Kill a running virtual machine
3.20.2

## Description

1
The vm-kill command kills a running virtual machine.
2

### 3.20.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--dry-run | Boolean | false | Show the commands that would be executed, but do not execute them.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.20.3

## Example

1

### 3.20.3.1 Example

    (Formatted for legibility)

    $ waxmill vm-kill \
      --machine 538a90e4-d50d-4511-8643-ae418279bac4 \
      --dry-run true

    /usr/sbin/bhyvectl \
      --vm=538a90e4-d50d-4511-8643-ae418279bac4 \
      --destroy

3.21

# vm-list-with-name

- 3.21. vm-list-with-name
  - 3.21.1. Name
  - 3.21.2. Description
  - 3.21.3. Example

3.21.1

## Name

1
vm-list-with-name - List defined virtual machines with a given name
3.21.2

## Description

1
The vm-list-with-name command lists the identifiers of all of the virtual machines that have a given name. The command raises an error if no virtual machines have the given name.
2

### 3.21.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--name | Machine Name | true | The name of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.21.3

## Example

1

### 3.21.3.1 Example

    $ waxmill vm-list-with-name --name com.io7m.example
    538a90e4-d50d-4511-8643-ae418279bac4

    $ waxmill vm-list-with-name --name nonexistent
    ERROR com.io7m.waxmill.cmdline.internal.WXMCommandVMListWithName: No virtual machines exist with the given name.
      Name:  nonexistent

3.22

# vm-list

- 3.22. vm-list
  - 3.22.1. Name
  - 3.22.2. Description
  - 3.22.3. Example

3.22.1

## Name

1
vm-list - List defined virtual machines
3.22.2

## Description

1
The vm-list command lists the defined virtual machines.
2

### 3.22.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.22.3

## Example

1

### 3.22.3.1 Example

    $ waxmill vm-list
    # ID                                     Name
    800a2dad-3367-4a98-879a-0fac219f55f4     com.io7m.example2
    538a90e4-d50d-4511-8643-ae418279bac4     com.io7m.example
    6730354e-bcba-4636-ab07-ad853c375f7d     com.io7m.example3

3.23

# vm-realize

- 3.23. vm-realize
  - 3.23.1. Name
  - 3.23.2. Description
  - 3.23.3. Example

3.23.1

## Name

1
vm-realize - Realize a virtual machine
3.23.2

## Description

1
The vm-realize command realizes a virtual machine. Realizing a virtual machine may be unnecessary if you have already manually created all of the resources (such as ZFS volumes) that the machine needs to run. The vm-run command will validate that all of the required resources exist when attempting to start a virtual machine.
2

### 3.23.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--dry-run | Boolean | false | Show the commands that would be executed, but do not execute them.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.23.3

## Example

1

### 3.23.3.1 Example

    (Formatted for legibility)

    $ waxmill vm-realize \
      --machine 538a90e4-d50d-4511-8643-ae418279bac4 \
      --dry-run true

3.24

# vm-run

- 3.24. vm-run
  - 3.24.1. Name
  - 3.24.2. Description
  - 3.24.3. Example

3.24.1

## Name

1
vm-run - Start a virtual machine
3.24.2

## Description

1
The vm-run command starts a virtual machine. The command, internally, uses execve() to replace the waxmill process with that of the bhyve virtual machine. The virtual machine stays running in the foreground. The command is intended to be used to start virtual machines under process supervision systems such as daemontools or runit.
2
The vm-run command will, when executed, will first generate and write the necessary configuration files into the virtual machine runtime directory for the given machine, and then generate and execute the necessary shell commands to start the virtual machine. For example, if the selected boot configuration uses GRUB bhyve, the required grub.cfg and device.map files will be generated and saved into the directory. The --dry-run option will cause the command to generate those required files, but will display the shell commands necessary to start the virtual machine on the standard output instead of actually executing them.
3

### 3.24.2.3 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--boot-configuration | Boot Configuration | true | The name of the boot configuration that will be used.
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--dry-run | Boolean | false | Show the commands that would be executed, but do not execute them.
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.24.3

## Example

1

### 3.24.3.1 Example

    (Formatted for legibility)

    $ waxmill vm-run \
      --machine 538a90e4-d50d-4511-8643-ae418279bac4 \
      --boot-configuration install \
      --dry-run true

    /usr/local/sbin/grub-bhyve \
      --device-map=/zroot/storage/vm/538a90e4-d50d-4511-8643-ae418279bac4/grub-device.map \
      --root=host \
      --directory=/zroot/storage/vm/538a90e4-d50d-4511-8643-ae418279bac4 \
      --memory=250M \
      538a90e4-d50d-4511-8643-ae418279bac4

    /usr/sbin/bhyve \
      /home/rm/git/com.github/io7m/waxmill/bhyve \
      -P \
      -A \
      -H \
      -c cpus=1,sockets=1,cores=1,threads=1 \
      -m 250M \
      -s 0:0:0,hostbridge \
      -s 0:1:0,ahci-hd,/zroot/storage/vm/538a90e4-d50d-4511-8643-ae418279bac4/disk-0_1_0 \
      -s 0:2:0,ahci-cd,/tmp/openbsd-6.6-amd64.iso \
      -s 0:3:0,lpc \
      -l com1,/dev/nmdm_538a90e4-d50d-4511-8643-ae418279bac4_B \
      -s 0:4:0,virtio-net,tap23,mac=a3:26:9c:74:79:34 \
      538a90e4-d50d-4511-8643-ae418279bac4

3.25

# vm-set

- 3.25. vm-set
  - 3.25.1. Name
  - 3.25.2. Description
  - 3.25.3. Example

3.25.1

## Name

1
vm-set - Set virtual machine configuration flags
3.25.2

## Description

1
The vm-set command sets configuration flags for virtual machines.
2

### 3.25.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--machine | UUID | true | The ID of the virtual machine
--verbose | Log Level | false | Set the minimum logging verbosity level.
--wire-guest-memory | Boolean | false | Enable/disable wiring of guest memory.

3.25.3

## Example

1

### 3.25.3.1 Example

    $ waxmill vm-set \
      --machine 538a90e4-d50d-4511-8643-ae418279bac4 \
      --wire-guest-memory true

3.26

# vm-update-boot-configurations

- 3.26. vm-update-boot-configurations
  - 3.26.1. Name
  - 3.26.2. Description
  - 3.26.3. Examples

3.26.1

## Name

1
vm-update-boot-configurations - Add or update boot configurations for a virtual machine
3.26.2

## Description

1
The vm-update-boot-configurations adds or updates boot configurations within a given virtual machine. The boot configurations are specified using an XML format. The precise format of the boot configurations elements are given in the urn:com.io7m.waxmill.vm:1:0 schema [1].
2

### 3.26.2.2 Parameters

Parameter | Type | Required | Description
--- | --- | --- | ---
--configuration | Path | false | The path to the configuration file (environment variable: $WAXMILL_CONFIGURATION_FILE)
--file | Path | true | A file containing boot configurations
--machine | UUID | true | The ID of the virtual machine
--update | Boolean | false | Update existing boot configurations
--verbose | Log Level | false | Set the minimum logging verbosity level.

3.26.3

## Examples

1
The Waxmill package currently supports booting virtual machines using UEFI or grub-bhyve.
2
An example boot configuration that attaches an OpenBSD installer CD to device 0:4:0, and boots the installation kernel using grub-bhyve:
3

### 3.26.3.3 OpenBSD GRUB-bhyve

    <?xml version="1.0" encoding="UTF-8" ?>

    <BootConfigurations xmlns="urn:com.io7m.waxmill.vm:1:0">
      <BootConfigurationGRUBBhyve name="install">
        <BootDiskAttachments>
          <BootDiskAttachment>
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
            <StorageBackendFile path="/storage/images/openbsd/install67.iso"/>
          </BootDiskAttachment>
        </BootDiskAttachments>
        <GRUBBhyveKernelOpenBSD>
          <BSDBootDevice kernelPath="/6.7/amd64/bsd.rd">
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
          </BSDBootDevice>
        </GRUBBhyveKernelOpenBSD>
      </BootConfigurationGRUBBhyve>
    </BootConfigurations>

4
An example boot configuration that attaches an OpenBSD installer CD to device 0:4:0, and boots machine using UEFI:
5

### 3.26.3.5 OpenBSD UEFI

    <?xml version="1.0" encoding="UTF-8" ?>

    <BootConfigurations xmlns="urn:com.io7m.waxmill.vm:1:0">
      <BootConfigurationUEFI name="install"
                             firmware="/usr/local/share/uefi-firmware/BHYVE_UEFI.fd">
        <BootDiskAttachments>
          <BootDiskAttachment>
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
            <StorageBackendFile path="/storage/images/openbsd/install67.iso"/>
          </BootDiskAttachment>
        </BootDiskAttachments>
      </BootConfigurationUEFI>
    </BootConfigurations>

6

### 3.26.3.6 Example

    $ cat boot.xml
    <?xml version="1.0" encoding="UTF-8" ?>

    <BootConfigurations xmlns="urn:com.io7m.waxmill.vm:1:0">
      <BootConfigurationGRUBBhyve name="install">
        <GRUBBhyveKernelOpenBSD>
          <BSDBootDevice kernelPath="/bsd.rd">
            <DeviceSlot bus="0"
                        slot="4"
                        function="0"/>
          </BSDBootDevice>
        </GRUBBhyveKernelOpenBSD>
      </BootConfigurationGRUBBhyve>
    </BootConfigurations>

    $ waxmill vm-update-boot-configurations --machine 538a90e4-d50d-4511-8643-ae418279bac4 --file boot.xml
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMUpdateBootConfigurations: Parsed 1 boot configurations
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMUpdateBootConfigurations: Added new boot configuration: install

    $ waxmill vm-update-boot-configurations --machine 538a90e4-d50d-4511-8643-ae418279bac4 --file boot.xml
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMUpdateBootConfigurations: Parsed 1 boot configurations
    ERROR com.io7m.waxmill.cmdline.internal.WXMCommandVMUpdateBootConfigurations: A boot configuration already exists with the given name: install

    $ waxmill vm-update-boot-configurations --machine 538a90e4-d50d-4511-8643-ae418279bac4 --file boot.xml --update
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMUpdateBootConfigurations: Parsed 1 boot configurations
    INFO com.io7m.waxmill.cmdline.internal.WXMCommandVMUpdateBootConfigurations: Updated boot configuration: install

4

# API

- 4. API
  - 4.1. JavaDoc

4.1

## JavaDoc

1
Please see the included JavaDoc for API reference documentation.

## Footnotes

1
The Waxmill package always prefers to rely on explicit configuration data rather than reading untrusted data from the environment (such as the PATH environment variable).

#### Footnote References

- Footnote Reference 2.2.3

1
Use the schema command to view the urn:com.io7m.waxmill.vm:1:0 schema.

#### Footnote References

- Footnote Reference 3.26.2.1

io7m | single-page | multi-page