How can I overcome Raspberry Pi Pico W WiFi UDP packet loss for data rates > 427 kBytes/s?

[frohro]

I am reading 24-bit I2S data from an ADC at 96 ks/s and sending it over WiFi using UDP on the Raspberry Pi Pico W. I am having problems with packet loss of about 1%, which for my SDR application is unacceptable. I know UDP is not guaranteed to deliver packets, but it is not the network that is causing the problem. The Pico cannot reliably send the UDP data as fast as I want it to (not very fast).

I am sure I isolated the problem I have been struggling with to the Raspberry Pi Pico WiFiUdp. I am sure it is not my router or network because I have tested the network at more than 10x the amount of data per second with 0% packet loss, but when the data is coming from the Pico W, then I get significant packet loss when I try to go faster than 427 kBytes/s ( delay less than 2 ms). I can't even get that data rate without significant packet loss if I am doing anything more complicated than my test program below. For example, if I remove the memset(buffer, 0x00, bufferSize); below, it has half the packet loss (about 1% as opposed to 2%).

#include <Arduino.h>
#include <WiFi.h>
#include <WiFiUdp.h>

// WiFi settings
const char* ssid = "Frohne-2.4GHz";
const char* password = "";

// UDP settings
IPAddress remoteIP(192, 168, 1, 101);
unsigned int remotePort = 12345;
WiFiUDP Udp;

// Packet settings
const int bufferSize = 1468;
const int packetNumberSize = sizeof(uint32_t);

// Packet number
uint32_t packetNumber = 0;

void setup() {
    Serial.begin();
    // Initialize WiFi
    WiFi.begin(ssid);
    while (WiFi.status() != WL_CONNECTED) {
        delay(1000);
        Serial.println("Connecting to WiFi...");
    }
    Serial.println("Connected to WiFi");

    // Initialize UDP
    Udp.begin(12345);
}

void loop() {
    // Create a buffer for the packet
    uint8_t buffer[bufferSize];

    // Fill the buffer with zeros
    memset(buffer, 0x00, bufferSize);

    // Set the packet number in the buffer
    memcpy(buffer, &packetNumber, packetNumberSize);

    // Send the packet
    Udp.beginPacket(remoteIP, remotePort);
    Udp.write(buffer, bufferSize);
    Udp.endPacket();

    // Increment the packet number
    packetNumber++;

    // Wait for a short period before sending the next packet
    // delay(1);
}

I have already gone to sending only 24-bit samples instead of 32, but my sample code is simpler with 32-bit samples, so that is what I'm showing here. I could get by without USB. Is there a way I could disable USB to remove USB interrupts? What other ways could I improve this? I saw that others using the Pico SDK were able to get significantly higher data rates over WiFi than the Arduino-Pico framework. Should I investigate that? What other advice do you have?

In case you are interested, the code I have been using to test the data rate is in Python and I will quote it below. I can also supply the code I was using to test the network if anyone wants. I am measuring the data rate using the Ubuntu System Monitor, which I have tested pretty extensively and found to be reliable.

import socket
import struct
import wave
import numpy as np
import cProfile
import matplotlib.pyplot as plt
import time

# Constants
PORT = 12345
PACKET_SIZE = 1468 # 4 bytes for packet number + 183 * 8 bytes for int32_t pairs
SAMPLE_RATE = 96000 # 96 ks/s
CHANNELS = 2 # Stereo
NUM_PACKETS = 10000
NUM_PLOT_POINTS = 1000

def main():
    # Create a UDP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024*1024)
    sock.bind(('', PORT))

    # Prepare to receive packets
    packets = []

    print("Listening for packets on port 12345...")
    while len(packets) < NUM_PACKETS:
        data, addr = sock.recvfrom(PACKET_SIZE)
        start = time.time()
        packet_number = struct.unpack('<I', data[:4])[0] # Little endian

        # Unpack the stereo audio data into a list of tuples (right, left)

        audio_data = struct.unpack('<366i', data[4:]) # Little endian


        # Pair up the integers as left and right audio samples
        audio_data_pairs = list(zip(audio_data[1::2], audio_data[::2]))
        packets.append((packet_number, audio_data_pairs))
        time_per_statement = time.time() - start
        
    # print(f"Time per statement: {time_per_statement} seconds")
    # Sort packets by packet number
    packets.sort(key=lambda x: x[0])

    # Check for missing or out-of-order packets
    expected_packet_number = packets[0][0]
    previous_packet_number = None
    missed_packets = 0
    for packet in packets:
        packet_number = packet[0]
        if previous_packet_number is not None:
            if packet_number < previous_packet_number:
                print(f"Packets not sorted correctly. Previous packet number {previous_packet_number}, current packet number {packet_number}")
            elif packet_number != previous_packet_number + 1:
                print(f"Packet missed. Expected packet number {previous_packet_number + 1}, but received packet number {packet_number}")
                missed_packets += 1
        previous_packet_number = packet_number
        expected_packet_number += 1
    print(f"Total missed packets: {missed_packets}")
    unique_packets = NUM_PACKETS - missed_packets
    print(f"This is {unique_packets/float(NUM_PACKETS)*100:.8f}% of the total packets received.")
    # if missed_packets != 0:
    #     print(f"This is {NUM_PACKETS*244/SAMPLE_RATE/missed_packets:.2f} seconds of audio data per problem.")

    # # Prepare the audio data for writing to a .wav file
    # Since the audio data is now a list of tuples, we need to flatten it
    audio_data_sorted = np.array([item for sublist in [item[1] for item in packets] for item in sublist], dtype=np.int32)

    # Separate the left and right channels
    left_channel = np.array([item[0] for sublist in [item[1] for item in packets] for item in sublist], dtype=np.int32)[:NUM_PLOT_POINTS]
    right_channel = np.array([item[1] for sublist in [item[1] for item in packets] for item in sublist], dtype=np.int32)[:NUM_PLOT_POINTS]
    time_array = np.arange(len(left_channel))/SAMPLE_RATE
    plt.figure(figsize=(10, 6))
    plt.plot(time_array, left_channel, label='Left Channel')
    plt.plot(time_array, right_channel, label='Right Channel')
    plt.legend()
    plt.xlabel('Time (s)')
    plt.title('Audio Data Received from Pico W')
    plt.show()

    # # Write to a .wav file
    # with wave.open('output.wav', 'wb') as wav_file:
    #     wav_file.setnchannels(CHANNELS)
    #     wav_file.setsampwidth(4) # Assuming 16-bit samples
    #     wav_file.setframerate(SAMPLE_RATE)
    #     wav_file.writeframes(audio_data_sorted.tobytes())

    # print("Audio data written to output.wav")

if __name__ == "__main__":
    cProfile.run('main()', 'my-script.profile')

[earlephilhower]

I suggest turning on full debugging Tools->Debug Level->All and setting Debug Port->Serial to see if there are packets being thrown away before transmission, and why.

The interesting bit would be here where potentially it might not be able to allocate an LWIP pbuf and if so the endPacket call will fail:

arduino-pico/libraries/WiFi/src/include/UdpContext.h

Lines 398 to 437 in 783bee5

	err_t trySend(const ip_addr_t* addr, uint16_t port, bool keepBufferOnError) {
	size_t data_size = _tx_buf_offset;
	pbuf* tx_copy = pbuf_alloc(PBUF_TRANSPORT, data_size, PBUF_RAM);
	if (tx_copy) {
	uint8_t* dst = reinterpret_cast<uint8_t*>(tx_copy->payload);
	for (pbuf* p = _tx_buf_head; p; p = p->next) {
	size_t will_copy = (data_size < p->len) ? data_size : p->len;
	memcpy(dst, p->payload, will_copy);
	dst += will_copy;
	data_size -= will_copy;
	}
	}
	if (!keepBufferOnError) {
	cancelBuffer();
	}
	if (!tx_copy) {
	DEBUGV("failed pbuf_alloc");
	return ERR_MEM;
	}
	if (!addr) {
	addr = &_pcb->remote_ip;
	port = _pcb->remote_port;
	}
	err_t err = udp_sendto(_pcb, tx_copy, addr, port);
	if (err != ERR_OK) {
	DEBUGV(":ust rc=%d\r\n", (int) err);
	}
	pbuf_free(tx_copy);
	if (err == ERR_OK) {
	cancelBuffer(); // no error: get rid of buffer
	}
	return err;
	}

Also, I would suggest checking the return values of the UDP calls. They will let you know if they have failed before getting anything to the radio.

[frohro]

I just changed the number of packets sent to 10000 and got 9998 through, so it is possible to get more through sometimes. I'm going to look over my test code and try to figure out where it is wrong. :-)

[frohro]

@earlephilhower , It seems to be the first few packets that are lost. I changed the number of packets sent from 10000 to 1000 to 100 to 10, and always seemed to loose one or two. I also simplified my code by trimming out the old testing things. I still have not seen all the packets come through, but I did get 9999 out of 10000 once. I also got 9 out of 10 once. Maybe this helps locate the problem. Often with only ten packets sent, there is only one that gets through.
Here is my new CYW43shim.cpp:

/*
    WiFi <-> LWIP driver for the CYG43 chip on the Raspberry Pico W

    Copyright (c) 2022 Earle F. Philhower, III <[email protected]>

    This library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Lesser General Public
    License as published by the Free Software Foundation; either
    version 2.1 of the License, or (at your option) any later version.

    This library is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    Lesser General Public License for more details.

    You should have received a copy of the GNU Lesser General Public
    License along with this library; if not, write to the Free Software
    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
*/

#include "CYW43shim.h"
extern "C"
{
#include <cyw43.h>
#include <cyw43_stats.h>
}
#include <pico/cyw43_arch.h>
#include <Arduino.h>
#include "/home/frohro/Documents/PlatformIO/Projects/SDR-TRX/src/test_missed_packets.h"

// From cyw43_ctrl.c
#define WIFI_JOIN_STATE_KIND_MASK (0x000f)
#define WIFI_JOIN_STATE_ACTIVE (0x0001)
#define WIFI_JOIN_STATE_FAIL (0x0002)
#define WIFI_JOIN_STATE_NONET (0x0003)
#define WIFI_JOIN_STATE_BADAUTH (0x0004)
#define WIFI_JOIN_STATE_AUTH (0x0200)
#define WIFI_JOIN_STATE_LINK (0x0400)
#define WIFI_JOIN_STATE_KEYED (0x0800)
#define WIFI_JOIN_STATE_ALL (0x0e01)

netif *CYW43::_netif = nullptr;

struct netif *__getCYW43Netif()
{
    return CYW43::_netif;
}

CYW43::CYW43(int8_t cs, arduino::SPIClass &spi, int8_t intrpin)
{
    (void)cs;
    (void)spi;
    (void)intrpin;
    _netif = nullptr;
    bzero(_bssid, sizeof(_bssid));
}

bool CYW43::begin(const uint8_t *address, netif *netif)
{
    (void)address;
    _netif = netif;
    _self = &cyw43_state;

    if (!_ap)
    {
        _itf = 0;
        cyw43_arch_enable_sta_mode();
        cyw43_wifi_get_mac(_self, _itf, netif->hwaddr);

        auto authmode = CYW43_AUTH_WPA2_AES_PSK;
        if (_password == nullptr)
        {
            authmode = CYW43_AUTH_OPEN;
        }

        // Not currently possible to hook up igmp_mac_filter and mld_mac_filter
        // TODO: implement igmp_mac_filter and mld_mac_filter
        // Implement cyw43_set_allmulti(_self, true) using exposed ioctl call (may not be functional in SDK 1.5?)
        uint8_t allmulti_true[] = {'a', 'l', 'l', 'm', 'u', 'l', 't', 'i', 0, 1, 0, 0, 0};
        cyw43_ioctl(&cyw43_state, CYW43_IOCTL_SET_VAR, sizeof allmulti_true, allmulti_true, CYW43_ITF_STA);
        // Add MDNS multicast MAC addresses manually, thanks Wikipedia!
        uint8_t mdnsV4[] = {0x01, 0x00, 0x5E, 0x00, 0x00, 0xFB};
        cyw43_wifi_update_multicast_filter(&cyw43_state, mdnsV4, true);
#if LWIP_IPV6
        uint8_t mdnsV6[] = {0x33, 0x33, 0x00, 0x00, 0x00, 0xFB};
        cyw43_wifi_update_multicast_filter(&cyw43_state, mdnsV6, true);
#endif

        if (_bssid[0] | _bssid[1] | _bssid[2] | _bssid[3] | _bssid[4] | _bssid[5])
        {
            if (cyw43_arch_wifi_connect_bssid_timeout_ms(_ssid, _bssid, _password, authmode, _timeout))
            {
                return false;
            }
            else
            {
                return true;
            }
        }
        else
        {
            if (cyw43_arch_wifi_connect_timeout_ms(_ssid, _password, authmode, _timeout))
            {
                return false;
            }
            else
            {
                return true;
            }
        }
    }
    else
    {
        _itf = 1;
        cyw43_arch_enable_ap_mode(_ssid, _password, _password ? CYW43_AUTH_WPA2_AES_PSK : CYW43_AUTH_OPEN);
        cyw43_wifi_get_mac(_self, _itf, netif->hwaddr);
        return true;
    }
}

void CYW43::end()
{
    _netif = nullptr;
    cyw43_deinit(&cyw43_state);
}

uint16_t CYW43::sendFrame(const uint8_t *data, uint16_t datalen)
{
    static uint32_t  totalPackets = 0;
    Serial.printf("datalen: %d\n", datalen);
    if (datalen == 1500)
    {
        totalPackets++;
    }
    Serial.printf("totalPackets: %lu\n", totalPackets);
    if (0 == cyw43_send_ethernet(_self, _itf, datalen, data, false))
    {
        return datalen;
    }
    return 0;
}

uint16_t CYW43::readFrame(uint8_t *buffer, uint16_t bufsize)
{
    // This is the polling method, but we hand this thru the interrupts
    (void)buffer;
    (void)bufsize;
    return 0;
}

and my new app:

#include <Arduino.h>
#include <WiFi.h>
#include <WiFiUdp.h>
#include "test_missed_packets.h"

// WiFi settings
const char *ssid = "Frohne-2.4GHz";
const char *password = "";

// UDP settings
IPAddress remoteIP(192, 168, 1, 101);
unsigned int remotePort = 12345;
WiFiUDP Udp;

// Packet settings
const int bufferSize = 1458;
const int packetNumberSize = sizeof(uint32_t);

void setup()
{
    Serial.begin();
    // Initialize WiFi
    delay(10000);  // Give time for serial monitor to show up.
    WiFi.begin(ssid);
    while (WiFi.status() != WL_CONNECTED)
    {
        delay(1000);
        Serial.println("Connecting to WiFi...");
    }
    Serial.println("Connected to WiFi");

    // Initialize UDP
    Udp.begin(12345);
    
    // Create a buffer for the packet
    uint8_t buffer[bufferSize];

    for (int i = 0; i < 10; i++)
    {

        // Fill the buffer with zeros
        memset(buffer, 0x00, bufferSize);

        // Set the packet number in the buffer
        memcpy(buffer, &i, packetNumberSize);

        // Send the packet
        if (!Udp.beginPacket(remoteIP, remotePort))
        {
            Serial.println("Failed to begin packet");
            return;
        }
        if (!Udp.beginPacket(remoteIP, remotePort))
        {
            Serial.println("Failed to begin packet");
            return;
        }
        if (!Udp.write(buffer, bufferSize))
        {
            Serial.println("Failed to write packet");
            return;
        }
        if (!Udp.endPacket())
        {
            Serial.println("Failed to end packet");
            return;
        }

        // Wait for a short period before sending the next packet
        delay(0);
    }
    Serial.println("Done sending packets");
}

void loop()
{
}

[frohro]

Hi @earlephilhower ,
I don't have a lot of experience programming as I only do it off an on once in a while. If it were not for GitHub Copilot, I would be in a lot of pain when I did any programming. I'm having difficulty seeing how your code only counts packets from the app as you warned me about earlier. My code shows there are several other packets with different lengths coming through even after my program says done and I see no other 1500 length packets. Maybe some of those packets are actually mine, split up or maybe your code is counting some other packets that come along from other WiFi duties. How do we tell? Did I read your code correctly? Here it is below for your convenience.

int fails = 0;
int calls = 0;
uint16_t CYW43::sendFrame(const uint8_t* data, uint16_t datalen) {
    calls++;
    if (0 == cyw43_send_ethernet(_self, _itf, datalen, data, false)) {
        return datalen;
    }
    fails++;
    return 0;
}

[earlephilhower]

It doesn't matter where they're coming from. Get the starting count, do your work, get the ending count. If the delta ending count < number of packets you sent then LWIP dropped them somewhere. There will only be occasional non-app packets going over the interface, and they will be few and far between in the scheme of things in this toy app.

I just ran 10,000 packets and I got exactly as expected (with the delay() commented completely out) in terms of packets over the interface (2 Ethernet frames out of 10,000 non-app related). Ethernet sent 10K packets. At the server I got 9,890 of them which is about the same ratio as before so I don't have any evidence it's a startup thing :

16:42:22.833 -> BEGIN: calls, fails: 5, 0
16:42:38.266 -> END: calls, fails: 10007, 0

In all this testing (with or without delay()s) I have never seen LWIP UDP > Ethernet Frames so I believe that's not where you should be looking.

(FWIW, don't print stuff from the sendFrame, it will affect timing and could possibly block. There's no need to print from there, just track it and use the app do dump. I suggest undoing your other changes in the lwip driver and just using the micro change suggested.)

At this point I can't say if a) it's a CYW43 driver problem, b) it's a CYW43 binary blob problem, or c) it's a radio thing and you should expect to drop 1% of Ethernet frames over the air. To go further you'll need to build a plain pico-sdk app doing this torture test (no delay between packets) and verify it fails and see if the RPi guys have any ideas.

[frohro]

@earlephilhower , I removed the print statements from the shim and verified that they were causing the packet loss there, so we now agree.

[earlephilhower]

One other thing would be to compare the lwipopts.h file for those higher BW apps vs. the one here. It was designed to keep memory usage low and maybe there's a low-impact way of increasing BW w/o blowing up RAM usage.

[earlephilhower]

Wow, that's seriously impressive. It doesn't use the SDK interfaces/timers or PIO or IRQ handler, nor LWIP, nor even the SDK-provided binary-only firmware for the WIFI chip. Guy's a savant and rewrote his own IP stack and CYW43 driver/port. But, right now, it's not really compatible with anything other than that one app I think. Definitely not something that could go here w/o a lot of work.

There are some IPerf apps in the SDK that do use LWIP, so you might look at them (in the pico-examples repo). That would give you a better idea of what LWIP and this core could be possible (and in the worst case you can do your own LWIP calls if the WiFiUDP class isn't cutting it for you).

Maybe you're hitting fragmentation at some point in the stack? The TCP_MSS here in the core 1460 whereas you're sending down UDP packets of 1468. Granted, the define is TCP_MSS so it may not actually affect UDP, but it seems like it would be simple enough to drop that size down to, say, 1440, and see if you get any different behavior.

[frohro]

I reduced the packet size to 1436, but no help there.

[frohro]

Hi @earlephilhower ,
I have been trying to get the iperf example to build. I thought it should using the same VScode platformio.ini that I have been using because calling the pico-sdk is possible using the Arduino-Pico framework, but I get this:

src/iperf.cpp:89:5: error: '_start_tcp_server_defaultlwiperf' was not declared in this scope
   89 |     _start_tcp_server_defaultlwiperf(&iperf_report, NULL);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*** [.pio/build/pico/src/iperf.cpp.o] Error 1

I tried adding to the platformio.ini

lib_deps = lwIP

but that didn't work either. I thought I'd play around with the example as you suggested to see what it could do. I assume that was what you were suggesting I do.

I still think I need to disable some interrupts, but I guess that would not work, because you haven't mentioned it even though I did. Any other suggestions?

[earlephilhower]

I've only used plain cmake to run the examples. They're not going to run under the Arduino core, and I don't think that P.IO supports building against the raw SDK.

Cmake is pretty straightforward and "just works" in my experience. You do need to set the board to the pico_w (double-check the definition, it's been a while) and define the SSIDs, but cmake followed by a make -j will get you going.

[frohro]

Ok, I'll work on that. When I get it going I assume I test to see how fast UDP goes there. Then I assume it may give me an idea how to speed things up on the Arduino Pico framework.