Notebooks/vnx-basic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic application\n",
    "\n",
    "This notebook shows you how to use the *basic* Vitis network example that can be generated from this repository.\n",
    "The design provides network connectivity using User Datagram Protocol (UDP) as the transport protocol.\n",
    "\n",
    "The assumptions made is this notebook are:\n",
    "1. You have an Alveo card configured with one of the supported shells\n",
    "1. You have generated the xclbin file for the *basic* example\n",
    "1. You have a 100 GbE capable NIC\n",
    "1. Both the Alveo card and the 100 GbE capable NIC are in the same host\n",
    "1. Alveo card is connected to the NIC, either directly or with a any network equipment, using any of the interfaces.\n",
    "\n",
    "Let's have a look at the *basic* design.\n",
    "\n",
    "There are 4 Kernels:\n",
    "* CMAC: provides the translation between physical signals to AXI4-Stream interface\n",
    "* Network layer: provides a bridge between raw Ethernet packets and the application using UDP as transport layer\n",
    "    * ARP provides translation between MAC and IP addresses\n",
    "    * ICMP provides ping capabilities\n",
    "    * The UDP module has a 16-entry table with socket information that needs to be filled in before running\n",
    "* krnl_mm2s: reads data from memory and packetize it setting tdest and tlast appropriately\n",
    "* krnl_s2mm: read data from the stream and copy it to memory\n",
    "\n",
    "![](../img/udp_network_basic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the followings cells we will:\n",
    "* Import the necessary pynq and python packages, define current device and xclbin file\n",
    "* Explore kernels in the design\n",
    "* Check physical link\n",
    "* Change Alveo card IP address and ping it\n",
    "* Configure socket table and populate it to the UDP module\n",
    "* Create UDP socket in the host\n",
    "* Allocate Alveo buffers\n",
    "* Move data from the HOST through the network (NIC) to the Alveo card\n",
    "* Move data from the HOST through the Alveo card to the network (NIC)\n",
    "* Free resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages and program FPGA\n",
    "In this section we need to import the `pynq` and python packages that will be used in the rest of this notebook. We also import the `vnx_utils.py` file with helper functions to set up the vnx examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "try {\n",
       "require(['notebook/js/codecell'], function(codecell) {\n",
       "  codecell.CodeCell.options_default.highlight_modes[\n",
       "      'magic_text/x-csrc'] = {'reg':[/^%%microblaze/]};\n",
       "  Jupyter.notebook.events.one('kernel_ready.Kernel', function(){\n",
       "      Jupyter.notebook.get_cells().map(function(cell){\n",
       "          if (cell.cell_type == 'code'){ cell.auto_highlight(); } }) ;\n",
       "  });\n",
       "});\n",
       "} catch (e) {};\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pynq\n",
    "import numpy as np\n",
    "from _thread import *\n",
    "import threading \n",
    "import socket\n",
    "from vnx_utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to define the current device, only if there is more than one Alveo card on the host. First let's check how many devices are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0) xilinx_u250_gen3x16_base_2\n",
      "1) xilinx_u50_gen3x16_xdma_201920_3\n",
      "2) xilinx_u280_xdma_201920_3\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(pynq.Device.devices)):\n",
    "    print(\"{}) {}\".format(i, pynq.Device.devices[i].name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* If there are more than one Alveo card available, you should pass the `device` argument to the `pynq.Overlay` class.\n",
    "* The xclbin variable should point to the `xclbin` file for the *basic* example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "currentDevice = pynq.Device.devices[2]\n",
    "xclbin = '../basic.intf3.xilinx_u280_xdma_201920_3/vnx_basic_if3.xclbin'\n",
    "ol = pynq.Overlay(xclbin,device=currentDevice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore kernels in the design\n",
    "This design was built for both interfaces. Therefore, we will see each kernel repeated twice. One for the interface 0 and another for the interface 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ol.ip_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check physical link\n",
    "After the dynamic region of the Alveo card is programmed with the basic example we can start interacting with the design.\n",
    "\n",
    "Let's check if the Alveo card has detected link with the network equipment. To do so, we will use one of the helper functions `link_status`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Link interface 0 {'cmac_link': True}; link interface 1 {'cmac_link': False}\n"
     ]
    }
   ],
   "source": [
    "print(\"Link interface 0 {}; link interface 1 {}\".format(ol.cmac_0.link_status(),ol.cmac_1.link_status()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Alveo card IP address and ping it\n",
    "By defaul the Alveo IP address is `192.168.0.5` and MAC address is `00:0A:35:02:9D:E5`. Let's change it to `197.11.27.12`, in this example we are using interface 0\n",
    "\n",
    "After the IP address is changed, we can ping the Alveo card. The first attempts will fail but the remaining should work.\n",
    "\n",
    "**Make sure you have configured the IP address of the 100 GbE capable NIC to be in the same subnetwork as the Alveo card**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'HWaddr': '00:0a:35:02:9d:0c', 'inet addr': '197.11.27.12', 'gateway addr': '197.11.27.1', 'Mask': '255.255.255.0'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alveo_ipaddr = '197.11.27.12'\n",
    "print(ol.networklayer_0.set_ip_address(alveo_ipaddr, debug=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Check 100 GbE capable NIC configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "enp115s0f1 Link encap:Ethernet  HWaddr a7:e5:2a:b9:01:c5  \n",
      "          inet addr:197.11.27.55  Bcast:197.11.27.255  Mask:255.255.255.0\n",
      "          UP BROADCAST MULTICAST  MTU:1500  Metric:1\n",
      "          RX packets:0 errors:0 dropped:0 overruns:0 frame:0\n",
      "          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0\n",
      "          collisions:0 txqueuelen:1000 \n",
      "          RX bytes:352573447 (352.5 MB)  TX bytes:52137528 (52.1 MB)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ifconfig enp115s0f1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Run ping to get ARP set up\n",
    "Some of the attempts will fail but the remaining should work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PING 197.11.27.12 (197.11.27.12) 56(84) bytes of data.\n",
      "64 bytes from 197.11.27.12: icmp_seq=3 ttl=128 time=0.111 ms\n",
      "64 bytes from 197.11.27.12: icmp_seq=4 ttl=128 time=0.048 ms\n",
      "64 bytes from 197.11.27.12: icmp_seq=5 ttl=128 time=0.037 ms\n",
      "\n",
      "--- 197.11.27.12 ping statistics ---\n",
      "5 packets transmitted, 3 received, 40% packet loss, time 3999ms\n",
      "rtt min/avg/max/mdev = 0.037/0.065/0.111/0.033 ms\n"
     ]
    }
   ],
   "source": [
    "!ping -c 5 $alveo_ipaddr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure socket table and populate it to the UDP module\n",
    "\n",
    "In this section we will configure the socket table in software and populate it to the UDP module in the Alveo card. \n",
    "1. Define a couple of connections \n",
    "1. The socket table is populated to the UDP module in the Alveo card using the `.populate_socket_table()` helper function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Sockets: 16\n",
      "HW socket table[  0], ti: 197.11.27.12\ttp: 50446\tmp: 60133\tv: 1\n",
      "HW socket table[  1], ti: 197.11.27.12\ttp: 38746\tmp: 62781\tv: 1\n",
      "HW socket table[  2], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[  3], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[  4], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[  5], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[  6], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[  7], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[  8], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[  9], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[ 10], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[ 11], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[ 12], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[ 13], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[ 14], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n",
      "HW socket table[ 15], ti: 0.0.0.0\ttp:     0\tmp:     0\tv: 0\n"
     ]
    }
   ],
   "source": [
    "sw_ip = '197.11.27.55'\n",
    "ol.networklayer_0.sockets[0] = (sw_ip, 50446, 60133, True)\n",
    "ol.networklayer_0.sockets[1] = (sw_ip, 38746, 62781, True)\n",
    "\n",
    "ol.networklayer_0.populate_socket_table(debug=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create UDP socket in the host\n",
    "In this part we will open an UDP socket in the host (software) to be able to communicate with the Alveo card through the network.\n",
    "\n",
    "We will use the python socket API to do so, the socket will be binded to the port `38746`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "SW_PORT = ol.networklayer_0.sockets[1]['theirPort']\n",
    "sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP\n",
    "sock.bind(('', SW_PORT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Allocate Alveo buffers\n",
    "\n",
    "We need to allocate the buffers on the global memory of the Alveo card to be able to pull and push data from them. We also will initialize the the sending buffer, `mm2s_buf`, with random data. This random data will be sent later to the network.\n",
    "\n",
    "1. Define alias for the application kernlels (`mm2s` and `s2mm`)\n",
    "1. Define size and shape of the buffers\n",
    "1. Allocate the buffers\n",
    "1. Initialize sending buffer with random data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "mm2s = ol.krnl_mm2s_0\n",
    "s2mm = ol.krnl_s2mm_0\n",
    "\n",
    "size = 1408 * 100\n",
    "shape = (size,1)\n",
    "\n",
    "if hasattr(ol, 'HBM0'):\n",
    "    mm2s_buf = pynq.allocate(shape, dtype=np.uint8, target=ol.HBM0)\n",
    "    s2mm_buf = pynq.allocate(shape, dtype=np.uint8, target=ol.HBM0)\n",
    "else:\n",
    "    mm2s_buf = pynq.allocate(shape, dtype=np.uint8, target=ol.bank1)\n",
    "    s2mm_buf = pynq.allocate(shape, dtype=np.uint8, target=ol.bank1)\n",
    "\n",
    "mm2s_buf[:] = np.random.randint(low=0, high=((2**8)-1), size=shape, dtype=np.uint8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Move data from the HOST through the network (NIC) to the Alveo card\n",
    "\n",
    "In this section we will write data to the socket, which will send such data from the host to the Alveo card using the network.\n",
    "\n",
    "Start streaming to memory mapped kernel, we need to specify how much data to expect from the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2mm_wh = s2mm.start(s2mm_buf,size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize a buffer with random data, define the packet size and compute how many packets we need to send to transmit the whole buffer to the network. Write the data from the buffer into the socket in `BYTES_PER_PACKET` chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "udp_message_global = np.random.randint(low=0, high=((2**8)-1), size=shape, dtype=np.uint8)\n",
    "BYTES_PER_PACKET = 1408\n",
    "num_pkts = size//BYTES_PER_PACKET\n",
    "alveo_port = ol.networklayer_0.sockets[1]['myPort']\n",
    "for m in range(num_pkts):\n",
    "    udp_message_local = udp_message_global[(m * BYTES_PER_PACKET) : \\\n",
    "                        ((m * BYTES_PER_PACKET) + BYTES_PER_PACKET)]\n",
    "    sock.sendto(udp_message_local, (alveo_ipaddr, alveo_port))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Wait for the s2mm kernel to receive all the data\n",
    "* Move data from global memory to HOST memory\n",
    "* Compare what was sent against what was received and print out result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Host sending data through the network and the host getting data from kernel was a: SUCCESS!. Total data transmitted 140,800 bytes to ('197.11.27.12', 62781)\n"
     ]
    }
   ],
   "source": [
    "s2mm_wh.wait()\n",
    "s2mm_buf.sync_from_device()\n",
    "\n",
    "msg = \"SUCCESS!\" if np.array_equal(udp_message_global, s2mm_buf) else \"FAILURE!\"\n",
    "print(\"Host sending data through the network and the host getting data from kernel \\\n",
    "was a: {}. Total data transmitted {:,} bytes to {}\" .format(msg,size,(alveo_ipaddr, alveo_port)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Move data from the HOST through the Alveo card to the network (NIC)\n",
    "\n",
    "We need to create a new thread to read data from the socket, this mainly because there will be multiple packets. This is done in the `socket_receive_threaded` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_lock = threading.Lock() \n",
    "# thread function \n",
    "def socket_receive_threaded(sock, size): \n",
    "    BYTES_PER_PACKET = 1408\n",
    "    shape_global = (size,1)\n",
    "    shape_local = (BYTES_PER_PACKET,1)\n",
    "    recv_data_global = np.empty(shape_global, dtype = np.uint8)\n",
    "    data_partial = np.empty(shape_local, dtype = np.uint8)\n",
    "    num_it = (size // BYTES_PER_PACKET)\n",
    "    global mm2s_buf\n",
    "    sum_bytes = 0\n",
    "    connection = 'None'\n",
    "    for m in range(num_it):\n",
    "        res = sock.recvfrom_into(data_partial) \n",
    "        recv_data_global[(m * BYTES_PER_PACKET) : ((m * BYTES_PER_PACKET) \\\n",
    "                        + BYTES_PER_PACKET)] = data_partial\n",
    "        sum_bytes = sum_bytes + int(res[0])\n",
    "        connection = res[1]\n",
    "    msg = \"SUCCESS!\" if np.array_equal(mm2s_buf, recv_data_global) else \"FAILURE!\"\n",
    "    print (\"Kernel sending data to the network and the host getting data from network\"\n",
    "    \" was a: {}. Total data received {:,} bytes from {}\".format(msg,sum_bytes,connection))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Copy random data to Alveo global memory\n",
    "* Acquire thread \n",
    "* Launch threaded function\n",
    "* Start kernel memory mapped to stream kernel, this will start sending UDP packets from the Alveo card to the NIC\n",
    "* Threaded function will start receiving data and it will print out result once ALL data was collected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Kernel sending data to the network and the host getting data from network was a: SUCCESS!. Total data received 140,800 bytes from ('197.11.27.12', 62781)\n"
     ]
    }
   ],
   "source": [
    "mm2s_buf.sync_to_device()\n",
    "print_lock.acquire() \n",
    "start_new_thread(socket_receive_threaded, (sock,size,))\n",
    "mm2s_wh = mm2s.start(mm2s_buf,size, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Free resources\n",
    "Delete buffers and free Alveo card"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "del mm2s_buf\n",
    "del s2mm_buf\n",
    "del udp_message_global\n",
    "pynq.Overlay.free(ol)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------------------\n",
    "Copyright (c) 2020-2021, Xilinx, Inc."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}