Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resources: Add documentation about extending disk images #148

Open
wants to merge 3 commits into
base: stable
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 119 additions & 0 deletions _pages/documentation/gem5-stdlib/8-extending-disk-images.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
---

layout: documentation
title: Extending Pre-Built Disk Images Using Packer
parent: gem5-standard-library
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why's this bundled in with the standard library? I don't think that's the right parent.

doc: gem5 documentation
permalink: /documentation/gem5-stdlib/extending-disk-images
author: Harshil Patel

---

gem5 Resources provides pre-built generic Ubuntu 24.04 disk images that you can extend and modify using Packer.

### Steps

1. **Download the Base Disk Image**

Obtain the base disk image from gem5 Resources. Links to the base images are available for [x86](https://resources.gem5.org/resources/x86-ubuntu-24.04-img), [ARM](https://resources.gem5.org/resources/arm-ubuntu-24.04-img), and [RISC-V](https://resources.gem5.org/resources/riscv-ubuntu-24.04-img). Download the image from the `versions` tab.
Harshil2107 marked this conversation as resolved.
Show resolved Hide resolved

2. **Unzip the Downloaded Image**

Unzip the downloaded image using `gunzip`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a comment on a gripe of mine and not a request for any change but I've heard you say this before so feel I should mention it in case you don't already know;

You don't need gnuzip to decompress a gzipped file:

gzip disk.iso # creates a compressed file 'disk.iso.gz'
gzip -d disk.iso.gz # decompresses 'disk.iso.gz', creating a new 'disk.iso'

I know gunzip is popular but I always found it odd to have these as two separate applications when one does just fine (FYI: gunzip is literally just gzip with the -d flag hardcoded).


```sh
gunzip /path/to/image
```

3. **Calculate the `sha256sum` of the Unzipped Image**

Calculate the `sha256sum` of the unzipped image:

```sh
sha256sum /path/to/unzipped/image
```
Harshil2107 marked this conversation as resolved.
Show resolved Hide resolved

4. **Create a Packer Script**

Write a Packer script to use the above disk image as a base image and extend it. Below is an example Packer script with placeholders:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the new npb/gapbs disk images are merged, it could be helpful to point toward the Packer scripts for those as well. It would give people more examples to look at.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also be good to add a link to the directories for the base Ubuntu x86/Arm disk images to provide more examples of Packer scripts.

You could also tell readers to simply use those directories as more of the process is automated, making it easier to make disk images.


```hcl
packer {
required_plugins {
qemu = {
source = "github.com/hashicorp/qemu"
version = "~> 1"
}
}
}

variable "image_name" {
type = string
default = "x86-ubuntu-24-04" # Update with your desired image name
}

variable "ssh_password" {
type = string
default = "12345" # Update if different for the base image
}

variable "ssh_username" {
type = string
default = "gem5" # Update if different for the base image
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does someone find out what the password and user name is for a base image?

}

source "qemu" "initialize" {
accelerator = "kvm"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I need to have a KVM enabled machine? Does this work even if my disk image is X86 but my host is ARM?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is set by the creator of the base image. I edited the comments to mention that the 22.04 and 24.04 images on gem5 resources have this user and password.

boot_command = [
"<wait120>",
"gem5<enter><wait>",
"12345<enter><wait>",
"sudo mv /etc/netplan/50-cloud-init.yaml.bak /etc/netplan/50-cloud-init.yaml<enter><wait>",
"12345<enter><wait>",
"sudo netplan apply<enter><wait>",
"<wait>"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should explain this magic.

] # This boot command logs in and re-enables the network so that Packer can connect via SSH
cpus = "4"
disk_size = "5000"
format = "raw"
headless = "true"
disk_image = "true"
iso_checksum = "sha256:# sha256sum of the base image"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need this? It's rather annoying they need to get the md5 of the disk image just to put it here. There's no way to disable the checksum?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it serves as a check that the file is not curropted.

iso_urls = [""] # Path to the base image
memory = "8192"
output_directory = "" # Output directory path
qemu_binary = "/usr/bin/qemu-system-x86_64"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you need to update this for each of the 3 base images?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

qemuargs = [["-cpu", "host"], ["-display", "none"]]
shutdown_command = "echo '${var.ssh_password}'|sudo -S shutdown -P now"
ssh_password = "${var.ssh_password}"
ssh_username = "${var.ssh_username}"
ssh_wait_timeout = "60m"
vm_name = "${var.image_name}"
ssh_handshake_attempts = "1000"
}

build {
sources = ["source.qemu.initialize"]

Harshil2107 marked this conversation as resolved.
Show resolved Hide resolved
Comment on lines +102 to +104
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be worth explaining what quemu is and in what capacity you're using it here to work with the disk images

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a small description in the begining of the section.

provisioner "shell" {
execute_command = "echo '${var.ssh_password}' | {{ .Vars }} sudo -E -S bash '{{ .Path }}'"
scripts = ["# path to the post-install script that will extend the disk image"]
expect_disconnect = true
}
}
Comment on lines +110 to +115
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the most important part of the whole thing and you really brushed over it.

I'd make this more tutorially and have a good example here of what a script can do here to setup a disk image.

Like, I have an application i want to be in the disk image. How do I put it there? How to I setup it up correctly so it runs when the US finishes booting?

```

After modifying the above Packer script with the necessary information, you can run the Packer file.

5. **Run the Packer File**

To run the Packer file, use the following commands:

```sh
sudo apt-get update && sudo apt-get install packer # Installing Packer
Comment on lines +124 to +125
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you need to install qemu and stuff like riscv-system-qemu to have this work cross-platform?


packer init /path/to/packer/script
packer build /path/to/packer/script
```

**Note:** If you want to view the terminal of the image during the build process, you can use a VNC viewer. Packer will provide a VNC port during the build, which you can connect to using a VNC viewer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be good to show an example of Packer output with the VNC port so it's easier for people to find, but it's not absolutely necessary

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd clarify what you mean by "terminal of the image". If I'm reading this I'd think i was already in the terminal. What it the "image's" terminal.

I think maybe the problem is you've not really explained what packer is doing here. Is it running in quemu and therefore has it's own terminal output?