Skip to content

Latest commit

 

History

History
525 lines (397 loc) · 16.5 KB

de10pro-instructions.adoc

File metadata and controls

525 lines (397 loc) · 16.5 KB
Raw

This document describes the workflow for the CHERI-BGAS project.

TL;DR

Run the following commands to boot a CHERI-BGAS system (assuming all tools are installed, disk images and filenames match, etc…​)

Read the later sections for more details

Prepare the RISCV FPGA soft-core and bring up FreeBSD on the DE10Pro’s ARM HPS system

# host machine
quartus_pgm -m jtag -o P\;cheri-bgas-socfpga-25012022-12_02.hps.rbf@2 && picocom -b 115200 /dev/ttyUSB0
# DE10Pro's ARM HPS ...
# u-boot
fatload mmc 0:1 1000 cheri-bgas-socfpga-25012022-12_02.core.rbf
fpga load 0 1000 ${filesize}
bridge enable
fatload mmc 0:1 0x2000000 efi/boot/bootaa64.efi
fatload mmc 0:1 0x8000000 socfpga_stratix10_de10_pro2.dtb
bootefi 0x2000000 0x8000000
# FreeBSD loader
set currdev=disk0s2:
load /boot/kernel/kernel
include /boot/lua/loader.lua
# FreeBSD prompt...

Boot FreeBSD on the RISCV soft-core

first terminal: RISCV gdb stub

# ARM HPS FreeBSD prompt
setenv RISCV_GDB_STUB_FMEM_DEV /dev/fmem_sys0_debug_unit
setenv RISCV_GDB_STUB_PORT 81
RISCV_gdbstub/src/main

second terminal: run cu to connect to the RISCV soft-core uart

# ARM HPS FreeBSD prompt
cu -l /dev/ttyu0
# enter when the gdb commands are sent in the third terminal

# ... RISCV FreeBSD prompt

third terminal: gdb session to drive the RISCV soft-core

# ARM HPS FreeBSD prompt
cd /root/riscv-freebsd-boot
gdb bbl-riscv64
# gdb prompt
target remote localhost:81
load devicetree.wrapped.elf
load kernel-riscv64.GFE-NODEBUG
load
set $a0 = 0
set $a1 = 0x80000000
continue
# see output of second terminal

Contents

Prepare the SD card

On a DE10Pro CHERI-BGAS system, we want a FreeBSD kernel to run on the ARM HPS system, and a CheriBSD kernel for the RISCV FPGA soft-core. As a first step to booting a software ecosystem on the DE10Pro CHERI-BGAS setup, we prepare a disk image to put on an SD card. We will use cheribuild to prepare our environment. Start by cloning cheribuild and run it with the llvm target to prepare the compiler needed for later builds:

./cheribuild.py llvm

1. Preparing the RISCV environment

(At the time of writing) Booting CheriBSD on our RISCV soft-core requires:

  • a device tree binary for the RISCV soft-core system

  • a build of BBL

  • a build of CheriBSD with an embedded rootfs

1.1. Device Tree Binary for the RISCV soft-core system

Follow the instructions over here to obtain a devicetree.wrapped.elf.

1.2. RISCV BBL

Run the following cheribuild command:

./cheribuild.py bbl-gfe-baremetal-riscv64-purecap

A bbl image is created cheri/output/sdk/bbl-gfe/riscv64-purecap/bbl.

1.3. RISCV CheriBSD kernel with embedded rootfs

Run the following cheribuild commands:

./cheribuild.py cheribsd-riscv64-purecap
./cheribuild.py disk-image-mfs-root-riscv64-purecap
./cheribuild.py --cheribsd-mfs-root-kernel/build-fpga-kernels cheribsd-mfs-root-kernel-riscv64-purecap

A riscv CheriBSD kernel with embedded rootfs is created cheri/output/kernel-riscv64-purecap.CHERI-PURECAP-GFE.

2. Preparing the ARM HPS environment

(At the time of writing) We want to boot a FreeBSD "host" setup on the ARM HPS system that will drive the RISCV soft-core. On this ARM host system, we will want to have all the necessary files to simply boot FreeBSD correctly, and the necessary tools to properly interact with the soft-core system. Here, we first go through files to be embedded in the final rootfs, and we then use cheribuild to create both the ARM FreeBSD kernel and the disk image with the files of interest.

2.1. Preparing the relevant device trees

2.1.1. Device tree for the ARM HPS

>>TODO<<

2.1.2. Device tree overlay exposing the soft-core system to the ARM HPS

To allow the ARM HPS system to communicate with the RISCV soft-core on the FPGA, we need to expose the various RISCV debug unit, uart, interrupts…​ to HPS. To do this, we build a device tree overlay as follows:

>>TODO<<

The instructions described here are currently relying on having build a FreeBSD ARM kernel generated using cheribuild. Ideally, we could amend these with a way to build dtbos which would only rely on dtc.

>>TODO<<

Create the following files:

  • build.sh :

WORKDIR=<path to your cheri build source root>
$WORKDIR/build/freebsd-aarch64-build/bmake-install/bin/bmake -m $WORKDIR/freebsd/share/mk/ SYSDIR=$WORKDIR/freebsd/sys DTC=$WORKDIR/build/freebsd-aarch64-build/$WORKDIR/freebsd/arm64.aarch64/tmp/obj-tools/usr.bin/dtc/dtc

  • Makefile :

DTSO=	fpga-cheri-bgas.dtso

.include <bsd.dtb.mk>

  • fpga-cheri-bgas.dtso :

/dts-v1/;
/plugin/;

#include <dt-bindings/interrupt-controller/arm-gic.h>

/ {
    compatible = "altr,socfpga-stratix10";
}

&{/soc} {
    fmem_h2f_sys_sel@f9030000 {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "cheri,fmem";
        reg = <0xf9030000 0x1000>;
        status = "okay";
        region0 {
            name = "selector";
            reg = <0x0 0x1000>;
        };
    };
    fmem_sys1@f9010000 {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "cheri,fmem";
        reg = <0xf9010000 0x3000>;
        status = "okay";
        region0 {
            name = "sys1_debug_unit";
            reg = <0x0 0x1000>;
        };
        region1 {
            name = "sys1_interrupts";
            reg = <0x1000 0x1000>;
        };
        region2 {
            name = "sys1_misc";
            reg = <0x2000 0x1000>;
        };
    };
        /*
        region3 {
            name = "sys1_uart";
            reg = <0x3000 0x1000>;
        };
        */
    uart1@f9013000 {
        #address-cells = <1>;
        #size-cells = <1>;
        current-speed = <115200>;
        compatible = "ns16550a";
        interrupts = <GIC_SPI 25 IRQ_TYPE_LEVEL_HIGH>;
        reg = <0xf9013000 0x1000>;
        clock-frequency = <50000000>;
        reg-shift = <2>;
    };
    fmem_sys1_h2f_addr_ctrl@f9014000 {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "cheri,fmem";
        reg = <0xf9014000 0x1000>;
        status = "okay";
        region4 {
            name = "sys1_h2f_addr_ctrl";
            reg = <0x0000 0x1000>;
        };
    };
    fmem_sys0@f9000000 {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "cheri,fmem";
        reg = <0xf9000000 0x3000>;
        status = "okay";
        region0 {
            name = "sys0_debug_unit";
            reg = <0x0 0x1000>;
        };
        region1 {
            name = "sys0_interrupts";
            reg = <0x1000 0x1000>;
        };
        region2 {
            name = "sys0_misc";
            reg = <0x2000 0x1000>;
        };
    };
        /*
        region3 {
            name = "sys0_uart";
            reg = <0x3000 0x1000>;
        };
        */
    uart0@f9003000 {
        #address-cells = <1>;
        #size-cells = <1>;
        current-speed = <115200>;
        compatible = "ns16550a";
        interrupts = <GIC_SPI 17 IRQ_TYPE_LEVEL_HIGH>;
        reg = <0xf9003000 0x1000>;
        clock-frequency = <50000000>;
        reg-shift = <2>;
    };
    fmem_sys0_h2f_addr_ctrl@f9004000 {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "cheri,fmem";
        reg = <0xf9004000 0x1000>;
        status = "okay";
        region4 {
            name = "sys0_h2f_addr_ctrl";
            reg = <0x0000 0x1000>;
        };
    };
    fmem_h2f_dflt@80000000 {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "cheri,fmem";
        reg = <0x80000000 0x60000000>;
        status = "okay";
        region0 {
            name = "h2f_dflt_1G";
            reg = <0x0 0x40000000>;
        };
        region1 {
            name = "h2f_dflt_512M";
            reg = <0x40000000 0x20000000>;
        };
    };
};

You can generate the dtbo fron the dtso by running build.sh.

2.2. Preparing relevant files for the FreeBSD rootfs

Extra files can be added to the final FreeBSD rootfs by placing them in <path to your cheri build source root, a.k.a. /home/gameboo/devstuff/cheri>/extra-files/\* prior to running the command which builds the rootfs.

In our FreeBSD rootfs, we will want (in addition to all the files already included by the FreeBSD build):

  • the RISCV environment we prepared in Preparing the RISCV environment, i.e.

    • Device Tree Binary for the RISCV soft-core system

    • RISCV BBL

    • RISCV CheriBSD kernel with embedded rootfs

  • the files necessary to allow the ARM HPS to communicate with the RISCV FPGA soft-core, i.e.

  • possibly your ssh keys to help ssh-ing into the ARM HPS system (this can be automatically added when building the FreeBSD disk image)

  • optionally, in /etc/rc.conf add

ifconfig_<interface name, a.k.a. dwc0>="inet <ip for the ARM HPS system, e.g. 192.168.10.10/24>"
defaultrouter="<ip of a host machine to act as a router, e.g. 192.168.10.1>"

  • optionally, in /etc/resolv.conf add

nameserver 8.8.8.8
nameserver 208.67.222.222
nameserver 208.67.220.220

2.3. Build an ARM FreeBSD kernel and disk image

Once the <path to your cheri build source root, a.k.a. /home/gameboo/devstuff/cheri>/extra-files/> folder is prepared as described in Preparing relevant files for the FreeBSD rootfs, you can run the following command to build a FreeBSD kernel and a disk image:

./cheribuild.py --source-root=<path to your cheri build source root, a.k.a. /home/gameboo/devstuff/cheri> --freebsd/repository=https://github.com/CTSRD-CHERI/freebsd-morello --freebsd/git-revision=stratix10 --freebsd/toolchain=system-llvm freebsd-aarch64 disk-image-freebsd-aarch64

Note: we explicitly use the https://github.com/CTSRD-CHERI/freebsd-morello FreeBSD fork on the stratix10 branch as it contains the fmem driver necessary to expose the AXI memory-mapped device described in the device tree overlay

Once the command is done, a <path to your cheri build source root, a.k.a. /home/gameboo/devstuff/cheri>/output/freebsd-aarch64.img disk image should be available. This image can be flashed onto an SD card by running (most likely as root to allow access to the SD card device):

dd if=<path to your cheri build source root, a.k.a. /home/gameboo/devstuff/cheri>/output/freebsd-aarch64.img> of=<path to your SD card device, a.k.a. /dev/sdX> bs=4M

Note: you can mount the FAT partition from the SD card and copy files to it on a linux system easily. The rootfs partition uses UFS which cannot be easily written from a linux system. You will want a FreeBSD system to change files on it.

2.4. Build an ARM u-boot bootloader

You can build u-boot by running this script. Running the script in ./ generates (amongst others) the following files:

  • a u-boot image ./u-boot-socfpga/u-boot-dtb.img

  • a first stage memory image ./u-boot-socfpga/spl/u-boot-spl-dtb.ihex

3. Finalising the FAT partition

To enable the later steps, the FAT partition on the SD card should contain:

Prepare the RISCV FPGA soft-core system

1. Embed a u-boot bootloader in the FPGA bitfile, and get an hps and a core slice

The first stage memory image u-boot-spl-dtb.ihex built in Build an ARM u-boot bootloader should be embedded in the startix 10 FPGA configuration.

This assumes a built DE10Pro-cheri-bgas sof file. Add explanations…​ 

BOOTLOADER=<path to u-boot-spl-dtb.ihex>
SOF=<path to your stratix 10 *.sof file>
OUTNAME=socfpga
quartus_pfg -c $SOF -o hps=ON -o hps_path=$BOOTLOADER $OUTNAME.rbf

2. Push the hps rbf slice to the board and get a usb terminal going

From the host machine driving the DE10pro board:

RBF=<path to your stratix 10 *.hps.rbf file>
quartus_pgm -m jtag -o P\;$RBF@2 && picocom -b 115200 /dev/ttyUSB0

3. Push the core rbf slice from the sdcard to the FPGA and setup the AXI bridges

From the u-boot prompt on the ARM HPS system:

fatload mmc 0:1 1000 <FAT partition path to your stratix 10 *.core.rbf file>
fpga load 0 1000 ${filesize}
bridge enable

3.1. side note - other useful u-boot commands:

printenv
usb start
usb info
fatload usb ...

Boot FreeBSD on the ARM HPS system

1. Get to the BSD loader

From the u-boot prompt on the ARM HPS system:

fatload mmc 0:1 0x2000000 <FAT partition path to your *.efi FreeBSD loader file>
fatload mmc 0:1 0x8000000 <FAT partition path to your *.dtb device tree file>
bootefi 0x2000000 0x8000000

2. Actually boot FreeBSD

From the FreeBSD loader prompt:

load <disk0s1>:</path/to/kernel>
set currdev=<disk0s2:>
include <ROOTFS path to loader script, e.g. /boot/lua/loader.lua>
boot

2.1. side note - other useful FreeBSD loader commands:

Misc:

show
fdt ls

To specify the usb drive as the rootfs on FreeBSD boot:

ufs:diskid/DISK-20090815198100000s2a

Get a FreeBSD prompt on the RISCV FPGA soft-core system

1. Get a GDB session to the RISCV FPGA soft-core

Once FreeBSD is booted on the ARM HPS system:

  • Run RISCV gdb stub from an ARM HPS system FreeBSD prompt. RISCV gdb stub can be cloned from https://github.com/CTSRD-CHERI/RISCV_gdbstub.git and built simply using make.

  • Once a RISCV gdb stub session is running on the ARM HPS system, connect a riscv gdb session. This should be done from a machine which has access to:

    • a RISCV bbl bootloader

    • a RISCV FreeBSD kernel

    • a device tree for the RISCV FPGA soft-core system (see https://github.com/CTSRD-CHERI/DE10Pro-softcore-devicetree.git) Typically, you would run on the same host machine driving the DE10Pro board gdb-multiarch <path to RISCV bbl> (running gdb-multiarch wih a riscv binary as argument will ensure a riscv-gdb session) and connect to the RISCV gdb stub running (on port 81) on the ARM HPS system with target remote <DE10Pro board ip or hostname>:81

2. Boot the RISCV FPGA soft-core system

From the gdb session to the RISCV FPGA soft-core:

  • load the RISCV system’s device tree (the one generated from https://github.com/CTSRD-CHERI/DE10Pro-softcore-devicetree.git is wrapped in an elf container which will load at address 0x80000000): load <path to the elf container for the RISCV system’s device tree>

  • load the RISCV FreeBSD kernel: load <path to the RISCV FreeBSD kernel>

  • load the bbl bootloader: load

  • set the a0 and a1 argument registers to bbl with the hart id and device tree address respectively:

    • set $a0 = 0

    • set $a1 = <device tree load address, e.g. 0x80000000>

  • send the RISCV core running: continue

3. Connect a prompt to the RISCV FPGA soft-core system UART

From an ARM FreeBSD prompt, connect to the uart using cu -l /dev/ttyu0 (this /dev/ttyu0 device should have been detected by virtue of having booted the ARM FreeBSD kernel with the previously mentioned device tree overlay).