Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support crucible disks in propolis-standalone #486

Merged
merged 2 commits into from
Aug 11, 2023
Merged

Support crucible disks in propolis-standalone #486

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f2810b
Add crucible config to propolis-standalone
faithanalog Feb 24, 2023
e2e461f
Split up README content for server and standalone
pfmooney Aug 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

214 changes: 30 additions & 184 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Propolis

Propolis is a rust-based userspace for illumos bhyve.
Propolis VMM userspace for use with illumos bhyve.

## Prerequisites

Expand All @@ -12,189 +12,35 @@ Propolis works best (and its CI tests run) on AMD hosts, but it can also be used
to run VMs on Intel hosts. Live migration is primarily supported on AMD hosts
but may work on Intel hosts as well.

## Building

To build, run:

```bash
$ cargo build
```

## propolis crate

The main propolis crate is structured as a library providing the building
blocks to create bhyve backed VM instances. It also provides a number of
emulated devices that can be exposed to guests (e.g. serial port, virtio
devices, NVMe).

## propolis-server

Propolis is mostly intended to be used via a REST API to drive all of its
functionality. The standard `cargo build` will produce a `propolis-server`
binary you can run:

### Running

```
# propolis-server run <config_file> <ip:port>
```

Note that the server must run as root. One way to ensure propolis-server has
sufficient privileges is by using `pfexec(1)`, as such:

```
# pfexec propolis-server run <config_file> <ip:port>
```

### Example Server Configuration

**Note**: the goal is to move the device config from the toml
to instead be configured via REST API calls.

```toml
bootrom = "/path/to/bootrom/OVMF_CODE.fd"

[block_dev.alpine_iso]
type = "file"
path = "/path/to/alpine-extended-3.12.0-x86_64.iso"

[dev.block0]
driver = "pci-virtio-block"
block_dev = "alpine_iso"
pci-path = "0.4.0"

[dev.net0]
driver = "pci-virtio-viona"
vnic = "vnic_name"
pci-path = "0.5.0"
```

## propolis-cli

Once you've got `propolis-server` running you can interact with it via the REST
API with any of the usual suspects (e.g. cURL, wget). Alternatively, there's a
`propolis-cli` binary to make things a bit easier:

### Running

The following CLI commands will create a VM, start the VM, and then attach to
its serial console:

```
# propolis-cli -s <propolis ip> -p <propolis port> new <VM name>
# propolis-cli -s <propolis ip> -p <propolis port> state <VM name> run
# propolis-cli -s <propolis ip> -p <propolis port> serial <VM name>
```

## propolis-standalone

Server frontend aside, we also provide a standalone binary for quick
prototyping, `propolis-standalone`. It uses a static toml configuration:

## Running

```
# propolis-standalone <config_file>
```

Example configuration:
```toml
[main]
name = "testvm"
cpus = 4
bootrom = "/path/to/bootrom/OVMF_CODE.fd"
memory = 1024

[block_dev.alpine_iso]
type = "file"
path = "/path/to/alpine-extended-3.12.0-x86_64.iso"

[dev.block0]
driver = "pci-virtio-block"
block_dev = "alpine_iso"
pci-path = "0.4.0"

[dev.net0]
driver = "pci-virtio-viona"
vnic = "vnic_name"
pci-path = "0.5.0"
```

Propolis will not destroy the VM instance on exit. If one exists with the
specified name on start-up, it will be destroyed and created fresh.

Propolis will create a unix domain socket, available at "./ttya",
which acts as a serial port. One such tool for accessing this serial port is
[sercons](https://github.com/jclulow/vmware-sercons), though others (such as
"screen") would also work.

### Quickstart to Alpine

In the aforementioned config files, there are three major components
that need to be supplied: The guest firmware (bootrom) image, the ISO, and the
VNIC.

Since this is a configuration file, you can supply whatever you'd like, but here
are some options to get up-and-running quickly:

#### Guest bootrom

The current recommended and tested guest bootrom is available
[here](https://buildomat.eng.oxide.computer/public/file/oxidecomputer/edk2/image_debug/6d92acf0a22718dd4175d7c64dbcf7aaec3740bd/OVMF_CODE.fd).

Other UEFI firmware images built from the [Open Virtual Machine Firmware
project](https://github.com/tianocore/tianocore.github.io/wiki/OVMF) may also
work, but these aren't regularly tested and your mileage may vary.

#### ISO

Although there are many options for ISOs, an easy option that
should work is the [Alpine Linux distribution](https://alpinelinux.org/downloads/).

These distributions are lightweight, and they have variants
custom-built for virtual machines.

A straightforward option to start with is the "virtual" `x86_64` image.

The "extended" variant contains more useful tools, but will require a
modification of the kernel arguments when booting to see the console on the
serial port. From Grub, this can be accomplished by pressing "e" (to edit),
adding "console=ttyS0" to the line starting with "/boot/vmlinuz-lts", and
pressing "Control + x" to boot with these parameters.

#### VNIC

To see your current network interfaces, you can use the following:

```bash
$ dladm show-link
```

To create a vnic, you can use one of your physical devices
(like "e1000g0", if you have an ethernet connection) as a link
for a VNIC. This can be done as follows:

```bash
NIC_NAME="vnic_prop0"
NIC_MAC="02:08:20:ac:e9:16"
NIC_LINK="e1000g0"

if ! dladm show-vnic $NIC_NAME 2> /dev/null; then
dladm create-vnic -t -l $NIC_LINK -m $NIC_MAC $NIC_NAME
fi
```

#### Running a VM

After you've got the bootrom, an ISO, a VNIC, and a configuration file that
points to them, you're ready to create and run your VM. To do so, make sure
you've done the following:
- [build propolis](#Building)
- run the [propolis-server](#propolis-server)
- create your VM, run it, and hop on the serial console using [propolis-cli](#propolis-cli)
- login to the VM as root (no password)
- optionally, run `setup-alpine` to configure the VM (including setting a root
password)
## Components

Programs:
- [propolis-server](bin/propolis-server): Run a Propolis VM instance, operated
via REST API calls (typically by
[omicron](https://github.com/oxidecomputer/omicron))
- [propolis-cli](bin/propolis-cli): CLI wrapper interface for `propolis-server`
API calls
- [propolis-standalone](bin/propolis-standalone): Simple standalone program to
run a Propolis VM instance, operated via a local config file

Libraries:
- [propolis-client](lib/propolis-client): Rust crate for `propolis-server` API
- [propolis](lib/propolis): Represents the bulk of the emulation logic required
to implement a userspace VMM. Both `propolis-server` and
`propolis-standalone` are built around this.

## Internal Crates

These are not meant as committed public interfaces, but rather internal
implementation details, consumed by Propolis components.

- bhyve-api: API (ioctls & structs) for the illumos bhyve kernel VMM
- dladm: Some thin wrappers around `dladm` queries
- propolis-server-config: Type definitions for `propolis-server` config file
- propolis-standalone-config: Type definitions for `propolis-standalone` config file
- propolis-types: Publically exposed (via `propolis-server`) types, intergral
to the `propolis` library
- viona-api: API (ioctls & structs) for the illumos viona driver

## License

Expand Down
66 changes: 66 additions & 0 deletions bin/propolis-server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Propolis Server

## Running

Propolis is mostly intended to be used via a REST API to drive all of its
functionality. The standard `cargo build` will produce a `propolis-server`
binary you can run:

```
# propolis-server run <config_file> <ip:port>
```

Note that the server must run as root. One way to ensure propolis-server has
sufficient privileges is by using `pfexec(1)`, as such:

```
# pfexec propolis-server run <config_file> <ip:port>
```

## Example Configuration

**Note**: the goal is to move the device config from the toml to instead be
configured via REST API calls.

```toml
bootrom = "/path/to/bootrom/OVMF_CODE.fd"

[block_dev.alpine_iso]
type = "file"
path = "/path/to/alpine-extended-3.12.0-x86_64.iso"

[dev.block0]
driver = "pci-virtio-block"
block_dev = "alpine_iso"
pci-path = "0.4.0"

[dev.net0]
driver = "pci-virtio-viona"
vnic = "vnic_name"
pci-path = "0.5.0"
```

## Prerequisites

When running the server by hand, the appropriate bootrom is required to start
guests properly. See the [standalone
documentation](../propolis-standalone#guest-bootrom) for more details. Details
for [creating necessary vnics](../propolis-standalone#vnic) can be found there
as well, if exposing network devices to the guest.

## CLI Interaction

Once you've got `propolis-server` running you can interact with it via the REST
API with any of the usual suspects (e.g. cURL, wget). Alternatively, there's a
`propolis-cli` binary to make things a bit easier:

### Running

The following CLI commands will create a VM, start the VM, and then attach to
its serial console:

```
# propolis-cli -s <propolis ip> -p <propolis port> new <VM name>
# propolis-cli -s <propolis ip> -p <propolis port> state <VM name> run
# propolis-cli -s <propolis ip> -p <propolis port> serial <VM name>
```
4 changes: 3 additions & 1 deletion bin/propolis-standalone/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ toml.workspace = true
tokio = { workspace = true, features = ["io-util", "rt-multi-thread"] }
serde = { workspace = true, features = ["derive"] }
propolis.workspace = true
crucible-client-types = { workspace = true, optional = true }
propolis-standalone-config = { workspace = true }
erased-serde.workspace = true
serde_json.workspace = true
Expand All @@ -31,7 +32,8 @@ slog-dtrace.workspace = true
slog-bunyan.workspace = true
slog-term.workspace = true
num_enum.workspace = true
uuid.workspace = true

[features]
default = []
crucible = ["propolis/crucible-full", "propolis/oximeter"]
crucible = ["propolis/crucible-full", "propolis/oximeter", "crucible-client-types"]
Loading
Loading