nengo_fpga.Simulator runs 1-2 time-steps ahead of nengo.Simulator

[arvoelke]

import numpy as np
import matplotlib.pyplot as plt

import nengo
from nengo_fpga.networks import FpgaPesEnsembleNetwork

n = 1
neuron_type = nengo.SpikingRectifiedLinear()
rate = 100
sim_t = 35

# 1) Nengo Ensemble implementation
with nengo.Network() as model1:
    a1 = nengo.Ensemble(
        n_neurons=n,
        dimensions=1,
        neuron_type=neuron_type,
        gain=np.zeros(n),
        bias=rate*np.ones(n),
    )
    
    p1 = nengo.Probe(a1.neurons, synapse=None)
    
# 2) Equivalent FpgaPesEnsembleNetwork implementation
with nengo.Network() as model2:
    a2 = FpgaPesEnsembleNetwork(
        fpga_name='de1',  # <-- change to pynq to get 1 time-step difference
        n_neurons=n,
        dimensions=n,
        learning_rate=0,
    )
    a2.ensemble.gain = np.zeros(n)
    a2.ensemble.bias = rate*np.ones(n)
    a2.ensemble.neuron_type = neuron_type
    a2.connection.solver = nengo.solvers.NoSolver(np.eye(n))
    a2.connection.synapse = None
    
    p2 = nengo.Probe(a2.output, synapse=None)

with nengo.Simulator(model1) as sim1:
    sim1.run_steps(sim_t)

with nengo_fpga.Simulator(model2) as sim2:
    sim2.run_steps(sim_t)

plt.figure()
plt.plot(sim1.data[p1], label="nengo.Simulator")
plt.plot(sim2.data[p2], label="nengo_fpga.Simulator")
plt.xlabel("Timesteps")
plt.ylabel("Output")
plt.legend()
plt.show()

The difference is one time-step for pynq (not shown) and two time-steps for de1 (shown above).

However if we change the second simulator from nengo_fpga.Simulator to nengo.Simulator we get the expected result:

[bmorcos]

I believe this is due to the fact that we use the first simulator step to initialize everything... does that sound right @xchoo?

[xchoo]

Maaaaaaaaybe? I'll add a card to investigate this

[arvoelke]

Wouldn't it be the other way around then (if the first time-step is effectively a no-op, wouldn't the FPGA be a step behind rather than ahead)?

[xchoo]

It might be that for nengo_fpga we consider timestep 0 to be the first timestep, whereas in nengo, it's timestep dt as the first timestep? (iirc)

[xchoo]

If anything, nengo_fpga should run 1 timestep behind (because of the UDP socket delay). This requires further investigation.

[tbekolay]

in nengo, it's timestep dt as the first timestep? (iirc)

Yes, the first recorded timestep is t=dt.

[xchoo]

Actually, @arvoelke that looks like a 2 timestep difference?? 😲

[arvoelke]

Yes you are right. There is a two time-step difference for de1 (original post edited) and a one time-step difference for pynq (shown below):

[bmorcos]

I was looking into a related issue where we were missing the packet from the first timestep,

• nengo/nengo-pynq#4
• nengo/nengo-de1#3

The solution here was to initialize the remote (device) side time to dt instead of starting at 0.

Fixing this closed the step offset by 1, this means PYNQ now matches the reference simulator, and the DE1 is off by one instead of two. I'm guessing there is a similar issue with sim time in the DE1 C driver which may account for the second step offset, though I haven't looked yet.

I modified Aarons script above and got these plots for reference:

[bmorcos]

Just putting this here as reference for when I come back to this: initializing curr_t=2*dt brings the DE1 inline with Nengo, but this doesn't seem like the correct solution. I haven't tracked down where the second timestep offset is coming from yet.

[arvoelke]

I haven't tracked down where the second timestep offset is coming from yet.

It might help by looking at what happens if there is input on the very first time-step, and only that one time-step. I vaguely recall this producing a weird result on one of the boards. The gain will need to be made non-zero.

Looking at non-spiking might also help make it easier to spot. For spiking, you would want to make the input large enough to integrate into a spike given the gain on the neuron (e.g., 1/dt assuming a gain of 1 on the spiking ReLU).

[bmorcos]

I'm posting an update here with some additional info as it's related and I'm not sure it warrants a separate issue.

It appears as though there were actually two facets to this issue:

• The data on the FPGA being labelled with the proper timestep (this was addressed by initiallizing curr_t = dt as per above and corroborated by being able to just set curr_t = 2*dt on the DE1)
• The FPGA and host having input/output data in sync, which was not visible given the test code above that does not have any host interactions.

For the latter, I haven't done an in depth investigation yet but I want to note some preliminary observations for posterity. I noticed this trying to port the keyword spotter to the PYNQ FPGA:

• The first output from the FPGA was 0
• The input to the FPGA ensemble object was correct, so there is an issue within the NengoFPGA pipeline.
• Adding an initial dummy input, then throwing away the first (corresponding) output appears to fix the problem.
• We need to examine both the host and client codes to determine where we are accidentally adding this delay by doing a send/recv/update in the wrong order, or perhaps we've omitted an initialization step or something.
• This may be (correctly) manifested as the FPGA being off by one time step given the DE1 implementation, and although the PYNQ appears to be correct, it is in fact off by one time step as observed with the initial output of 0?

@xchoo is currently investigating NengoFPGA socket performance, so chances are things will change, we should potentially just wait on his results/rework before digging into this. We can use our workaround of adding a dummy initial step for now.

[bmorcos]

A more pointed hypothesis for when we come back to this:

• The first input is lost to connection/setup. Without padding, the first output is zero and the second output corresponds to the second input.
• The FPGA appears to be a step behind because the order of operations of compute and communication on the device side (single_pes_net.py) so we are operating on stale data. The helper socket class bundles send and receive into a single call, the current implementation is send/recv-update-step. We want to split this to be recv-update-step-send.