THIS DOCUMENT CONTAINS DESCRIPTION: BOTH QEMU AND RPI3 (ACTUAL HARDARE); ALSO DEVELOP ON YOUR LOCAL MACHINES
READING TIME: 30 MIN
Get the code:
git clone https://github.com/fxlin/p1-kernel
- rpi3: raspberry pi 3, a credit card size computer
- baremetal: write & run code directly on hardware (rpi3, real or emulated)
- kernel: the baremetal code you will develop to run on (real/emulated) hardware
- kernel binary/image: a single file, which contains the compiled kernel program and data
Note:
-
Recommended configurations are underscored.
-
How to connect to CS server(s): see here.
-
VSCode: optional. It's available on Win/OSX/Linux. It can be used for any configuration below.
| Your local machine runs: | If develop remotely on CS servers (recommended) | If develop on local machine (only if you know what you are doing) |
|---|---|---|
| Windows | WSL for SSH shell | WSL for toolchain. gdbserver could be tricky. |
| Linux | SSH shell | Native toolchain + console |
| Mac | Terminal for SSH shell | HomeBrew |
Note:
-
Recommended configurations are underscored.
-
How to connect to CS server(s): see here.
-
VSCode: optional. It's available on Win/OSX/Linux. It can be used for any configuration below.
| Your local machine runs: | Develop remotely on CS servers | Develop locally |
|---|---|---|
| Windows | WSL for SSH shell; then download (scp) kernel binary to local | WSL for toolchain |
| Linux | SSH shell; then download (scp) kernel binary to local | Native toolchain + console |
| Mac | Terminal for SSH shell | HomeBrew (untested) |
These are compiler, linker, etc. for us to generate the kernel code. Use the one provided by Ubuntu.
Note: Below is necessary only when you develop kernel code on your local machine. The server already has the toolchain installed
$ sudo apt install gcc-aarch64-linux-gnu
$ sudo apt install gdb-multiarch
$ aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
This is where you run the kernel code.
This is required no matter you develop on local machines or on the server.
Grab the QEMU source. Our QEMU is based on upstream v4.2 with custom aarch64 debugging support.
git clone https://github.com/fxlin/qemu-cs4414.git qemu
cd qemu
./configure --target-list=aarch64-softmmu -enable-sdl # -sdl builds in UI support over X
make -j`nproc`
export PATH="$(pwd)/aarch64-softmmu:${PATH}"
If successful, this will result in QEMU executables in ./aarch64-softmmu/. The last line above adds the path to our search path.
If you encounter compilation errors (e.g. unmet dependencies), make sure you run all apt get commands above.
Now try QEMU & check its version. The supported machines should include Rpi3
$ qemu-system-aarch64 --version
QEMU emulator version 5.0.50 (v5.0.0-1247-gaf6f75d03f-dirty)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers
patched for cs4414/6456 aarch64 kernel hacking
$ qemu-system-aarch64 -M help|grep rasp
raspi2 Raspberry Pi 2B
raspi3 Raspberry Pi 3B
Test QEMU with Rpi3 baremetal code (NOTE: this repo is for validating your toolchain & QEMU build; it is NOT our course project)
git clone https://github.com/fxlin/raspi3-tutorial.git
cd raspi3-tutorial
git checkout b026449
cd 05_uart0
make
qemu-system-aarch64 -M raspi3 -kernel kernel8.img -serial stdio
If everything works fine, you should see QMEU print out:
My serial number is: 0000000000000000
Note: the test program runs an infinite loop which will cause high CPU usage on your host machine. Kill the test program timely.
On Linux (e.g. connecting to the course server from your local machine):
On Windows (local WSL, NOT using the course server). The emulated display requires the QEMU to be built with graphics support (e.g. a WSL or native Windows build)
Move to the QEMU cheatsheet.
 |
 |
|---|---|
| Required: An Rpi3 board (Model B or B+) link | Required: A USB-serial cable Amazon. Connection inside the dongle: black-GND; green-TXD; white-RXD; red-VCC. |
-
Required: A micro SD card. The capacity can be humble (e.g. 4GB). The speed does not matter much. The one I used was $6. Rpi's official page about uSD
-
Required: SD card reader. To be plugged in your PC for loading kernel to the micro SD card. A cheap one can be $7 on Amazon
-
Recommended: A micro USB cable for powering Rpi3.
Older versions of Raspberry Pi are not going to work with this tutorial because all lessons are designed to use a 64-bit processor that supports ARMv8 architecture, and such processor is only available in the Raspberry Pi 3. Newer versions, including Raspberry Pi 3 Model B+ should work fine.
Raspbian is a Debian-based Linux distro. It's the official OS for Rpi3. Why we need Raspbian? 1. to test USB to TTL cable connectivity initially. 2. after installing Raspbian, the SD card is formatted in the right way. All the proprietary binary blobs needed to boot Rpi3 are also in place.
Load the SD card with Raspbian OS. Follow the official instructions.
Rpi3 <-- a USB-serial cable ---> PC (running a temrinal emulator)
After you get a serial cable, you need to test your connection. If you never did this before I recommend you to follow this guide It describes the process of connecting your Raspberry PI via a serial cable in great details. Basically, you run Raspberry's official OS to ensure the hardware setup is fine.
Linux users: minicom recommended.
sudo minicom -b 115200 -o -D /dev/ttyUSB0 -C /tmp/minicom.log
Note: your PC may give different names to the USB-serial dongle, e.g. /dev/ttyUSB1. Find it out by looking at dmesg output.
Windows users (including WSL): PuTTY recommended. A sample configuration below.
Note: your PC may give different names to the USB-serial dongle, e.g. COM4. Find it out by looking at Windows Device Manager.
We recommend you power Rpi3 through its micro USB port. Perhaps use a flip switch on the other side of the USB power for power cycling Rpi3. The guide above also describes how to power your Raspberry Pi using a serial cable. RPi OS works fine with such kind of setup, however, in this case, you need to run your terminal emulator right after you plug in the cable. Check this issue for details.
Rpi3 <-- micro USB ---> PC
Rpi3 <-- micro USB ---> Wall charger
Power cycling Rpi3, you should see Linux kernel console output on PC terminal.
This is my desktop when I hack with the Rpi3 kernel.
On powering up, Rpi3 looks for the following files on boot partition of the SD card.
- bootcode.bin: the proprietary bootloader for enabling SDRAM. This comes with Raspbian.
- start.elf: the proprietary firmware loaded by the bootloader. Using the updated Raspbian OS. This comes with Raspbian.
- fixup.dat: needed to use 1GB of memory. This comes with Raspbian.
- config.txt: to be parsed by start.elf and decide boot behavior. It offers a great deal of options which is pretty cool. A default one comes with Raspbian. This file is to be customized by us
- kernel8.img: our kernel.
Summary: we need to change config.txt (once) and kernel8.img (every time we re-compile kernel) on the SD card.
Plug the SD card to PC via the card reader. Open config.txt which is on the boot partition. The following two lines are crucial. Add them to config.txt.
arm_64bit=1
enable_uart=1
Note: multiple online tutorials advise options like kernel_old=1 or arm_control. You do NOT need those. With our options in config.txt above, Rpi3 will load the kernel named kernel8.img to 0x80000. Check the official doc for config.txt above. Look for kernel_address.
Ref: the official doc for config.txt.
... to ensure our toolchain works fine.
git clone [email protected]:fxlin/raspi3-tutorial.git
cd raspi3-tutorial
git checkout b026449
cd 05_uart0
make
Note: the repo above (raspi3-tutorial.git) is NOT our project repo. It's someone's code for testing rpi3 hardware. We are just using for testing ONLY.
Copy kernel8.img to the SD card. Eject the SD card from PC. Plug the SD to Rpi3. Make sure the serial connection is good and terminal emulator on your PC is ready. Power cycle Rpi3. You should see something like:
(Your serial number may be different)
Viola! You just built your first baremetal program for Rpi3!