Skip to content

Latest commit

 

History

History
205 lines (137 loc) · 7.15 KB

xdma_install.md

File metadata and controls

205 lines (137 loc) · 7.15 KB
Raw

XDMA Driver Installation and Frequently Asked Questions

XDMA is a Linux kernel driver for using DMA and/or User-defined interrupts for AWS FPGAs. Please see XDMA README for details.

Table of Contents

  1. Q: How do I know if the XDMA driver is available and installed?
  2. Q: How do I get the source code of the xdma driver and compile it?
  3. Q: How can I make sure the installed driver will be preserved following a kernel update?
  4. Q: What PCIe Vendor-ID and Device-ID does XDMA driver support
  5. Q. Install of XDMA driver fails on AMI version 1.5.x or Later

Q: How do I know if the XDMA driver is available and installed?

The XDMA driver could be already installed in the latest releases of various Linux distributions and in AWS FPGA Development AMI available at AWS Marketplace

To make sure the driver is installed run next command from your linux shell: $ lsmod | grep xdma

Running this will return XDMA only if the driver is installed and running.

If running, the XDMA driver exposes multiple device channels. Each XDMA channel has a write and read device filename (e.g. /dev/xdmaX_h2c_Y and /dev/xdmaX_c2h_Y), and supports the Linux character device APIs. User-defined interrupts/events are accessed through the /dev/xdmaX_event_Y device files. Where X is a number from 0-7 to represent different FPGAs in the system, and Y is used to represent the available DMA channels and event interfaces for a given FPGA.

The developer can use these DMA channel and interrupt device files directly from Linux userspace application.

Q: How do I get the source code of the xdma driver and compile it?

Amazon xdma driver is included in AWS FPGA SDK for integration with other Linux distributions, please follow the next set of steps:

Step 1: Make sure you have gcc and linux kernel source code installed in your machine:

For AmazonLinux,RedHat,Centos

  $ sudo yum groupinstall "Development tools"
  $ sudo yum install kernel kernel-devel

NOTE:

  • If the kernel is updated, you will have to REBOOT your instance to allow loading the new kernel. The build will not work if the kernel version and kernel source version do not match.
  • If you see an error that looks like make: *** /lib/modules/4.14.88-88.76.amzn2.x86_64/build: No such file or directory. Stop. make sure that you have installed matching kernel and kernel-devel versions and rebooted to load the new kernel.

For Ubuntu

  $ sudo apt-get install make
  $ sudo apt-get install gcc

For Suse

  $ sudo zypper install make
  $ sudo zypper install gcc
  $ sudo zypper update
  $ sudo zypper install kernel-devel

Step 2: Clone the git repo locally under my_fpga_dir for example:

  $ mkdir -p <my_fpga_repo>
  $ cd <my_fpga_repo>
  $ git clone https://github.com/aws/aws-fpga

Note: the above mentioned git call would fail if the local git repository already exists.

Step 3: Enter the directory and compile the code:

  $ cd <my_fpga_repo>
  $ cd aws-fpga/sdk/linux_kernel_drivers/xdma
  $ make

If the make command was successful, you would find xdma.ko.

Step 4: Copy the driver to the modules directory:

For AmazonLinux, RedHat, Open Suse, and Centos:

The next set of steps will install the kernel driver so it gets loaded everytime the machine boots/reboots:

$ sudo make install

To load the driver without rebooting:

$ sudo modprobe xdma

To unload the driver:

$ sudo rmmod xdma

To uninstall the driver so it no-longer gets loaded everytime the machine boots/reboots:

$ sudo make uninstall

NOTE: steps 3 and 4 would need to be repeated for every kernel update.


Q: How can I make sure the installed driver will be preserved following a kernel update?

Step A Get DKMS

For CentOS7 : sudo yum install dkms.noarch

Step B: Move the xdma source package to the /usr/src/ directory

so dkms can find it and build it for each kernel update: Append the version number (you can find the current version number in the release notes) of the source code to the directory name. For example, version 1.0.0 is shown in the example below:

$ sudo cp -R <my_fpga_repo/aws-fpga/sdk/linux_kernel_drivers/xdma /usr/src/xdma-1.0.0

Step C: Create the dkms configuration file with the following values, substituting your version of xdma.

  $ sudo vim /usr/src/xdma-1.0.0/dkms.conf
  PACKAGE_NAME="xdma-drv"
  PACKAGE_VERSION="1.0.0"
  CLEAN="make -C ./ clean"
  MAKE="make -C ./ BUILD_KERNEL=${kernelver}"
  BUILT_MODULE_NAME[0]="xdma"
  BUILT_MODULE_LOCATION="."
  DEST_MODULE_LOCATION[0]="/updates"
  DEST_MODULE_NAME[0]="xdma"
  AUTOINSTALL="yes"

Step D: Add, build, and install the xdma module on your instance with dkms.

  $ sudo dkms add -m xdma -v 1.0.0
  $ sudo dkms build -m xdma -v 1.0.0
  $ sudo dkms install -m xdma -v 1.0.0

Step E: Rebuild the initramfs so the correct module is loaded at boot time. $ TBD

Step F: Verify that the xdma module is installed using the modinfo xdma.

  $ modinfo xdma
  filename:       /home/centos/aws-fpga-staging/sdk/linux_kernel_drivers/xdma/xdma.ko
  license:        GPL v2
  version:        2017.1.47
  description:    Xilinx XDMA Classic Driver
  author:         Xilinx, Inc.
  rhelversion:    7.3
  srcversion:     0498A67CF8B4F126703BA5D
  vermagic:       3.10.0-514.10.2.el7.x86_64 SMP mod_unload modversions 
  parm:           sgdma_timeout:timeout in seconds for sgdma, default is 10 sec. (uint)
  parm:           poll_mode:Set 1 for hw polling, default is 0 (interrupts) (uint)


Q: What PCIe Vendor-ID and Device-ID does XDMA driver support?

Initially, XDMA supports PCIe VendorID:DeviceID 1d0f:f001 (the ID's of the DMA example). To use the device driver on another CL, please add the Vendor ID and Device to the device table in xdma_mod.c

For example:

static const struct pci_device_id pci_ids[] = {
        ...
        { PCI_DEVICE(0x1d0f, 0xf001), },
        ...
        { 0, }
};

where 0x1d0f is the vendor ID for Amazon and 0xf001 is the device ID for the cl-dram-dma example CL.

Be sure to remake and re-install the XDMA driver after modifying the device table.

Amazon encourages pull requests to this github to add CL ID's to the driver, so there is no need to fork the driver or SDK.

Q: Install of XDMA driver fails on AMI version 1.5.x or Later

DEVAMI 1.5.0 or Later instances come with preinstalled Xilinx Runtime Environment (XRT). XOCL driver come pre-installed with XRT. This prevents XDMA driver install. Please remove XOCL driver module and then proceed to install XDMA driver.

To check if XOCL driver is running

lsmod | grep xocl

To Remove XRT and XOCL driver

sudo systemctl stop mpd
sudo yum remove -y xrt xrt-aws

XDMA driver install can proceed once XRT is removed.