Trouble learning

[4iar]

This is an extremely cool project!

I am having trouble getting the network to learn tasks though.

To get started with the package (v0.2.4), I tried throwing some OpenAI gyms at it. I used the breakout_stdp.py example script and changed the environment from 'BreakoutDeterministic-v4' to 'CartPole-v0', then modified inpt (n=4, shape=[4,1]) and out (n=2) to correspond to the CartPole task.

I ran 650 iterations of the task but saw no improvement in the accumulated reward. I've tried experimenting with some of the different learning/update rules with no success. I've also tried some of my own custom OpenAI gyms but found no success either. What other things could I try to get the network to learn the task? I am also new to neural networks & reinforcement learning so maybe there are some obvious things I've missed.

Cheers!

Some other info:

I ran into two issues and have mentioned them here just in case they are related:

1. Running the script throws the following error:

Traceback (most recent call last):
  File "run.py", line 36, in <module>
    environment.reset()
  File "/Users/x/bindsnet/bindsnet/environment/__init__.py", line 237, in reset
    self.preprocess()
  File "/Users/x/bindsnet/bindsnet/environment/__init__.py", line 268, in preprocess
    self.obs = subsample(gray_scale(self.obs), 84, 110)
  File "/Users/x/bindsnet/bindsnet/datasets/preprocess.py", line 13, in gray_scale
    return cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
cv2.error: OpenCV(4.1.0) /Users/travis/build/skvark/opencv-python/opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::(anonymous namespace)::CvtHelper<cv::impl::(anonymous namespace)::Set<3, 4, -1>, cv::impl::(anonymous namespace)::Set<1, -1, -1>, cv::impl::(anonymous namespace)::Set<0, 2, 5>, cv::impl::(anonymous namespace)::SizePolicy::NONE>::CvtHelper(InputArray, OutputArray, int) [VScn = cv::impl::(anonymous namespace)::Set<3, 4, -1>, VDcn = cv::impl::(anonymous namespace)::Set<1, -1, -1>, VDepth = cv::impl::(anonymous namespace)::Set<0, 2, 5>, sizePolicy = cv::impl::(anonymous namespace)::SizePolicy::NONE]'
> Invalid number of channels in input image:
>     'VScn::contains(scn)'
> where
>     'scn' is 1

It looked related to the different input format.

I temporarily worked around this by commenting out the subsample line in the preprocessing step: https://github.com/Hananel-Hazan/bindsnet/blob/0.2.4/bindsnet/environment/__init__.py#L268

2. The plots only show a single time step (big horizontal bars), rather than spikes/voltages across the entire trial (e.g. like in this fig: https://i.imgur.com/Sltk55I.png).

[Huizerd]

How did you encode the state of the environment into spikes for the network to process? You need a way to allow the network to distinguish between different states. One way of doing this is using so-called place or grid cells, which also happen to be present in animal brains. The core idea is that you need to (uniformly) distribute some Poisson neurons over your state space, and make their firing rate dependent on the distance between their center and the current state of the agent. In this way, different states will lead to different firing patterns, and the network is able to learn something. See an example of an implementation in this paper. If I remember correctly, most encodings in BindsNET are mainly suitable for pixel intensities.

[djsaunde]

@4iar We (the BINDS lab) haven't managed to get SNNs to learn any interesting in OpenAI gym environments. Reinforcement learning with SNNs is a rather open research question! We did manage to convert trained artificial neural networks to SNNs while maintaining performance on Breakout, however (link). I would suggest getting familiar with RL, and seeing whether ideas from other approaches can be used for SNN training.

We'll look into those errors; there have been a few major PRs lately that may have broken some things.

[4iar]

Thanks for the super helpful replies and papers @Huizerd and @djsaunde!

I also saw a CartPole example in PR #269 that implements place cells, which is really useful!

What's the best way to get started with training an ANN and converting it to an SNN on my own tasks? I saw a few scripts in the experiments repo bindsnet_experiments/experiments/conversion. I also saw a more recent script convert_and_save_snn.py there. Wasn't sure which script + version of BindsNET to start playing with

[djsaunde]

Hi @4iar,

No problem. The bindsnet/conversion module contains code for converting PyTorch ANNs with only ReLU activations to SNNs. The networks that result from the conversion can then be used to do the same task that the PyTorch ANN was trained to do, typically with little loss in performance.

This script is perhaps the simplest example we have at the moment. It builds and trains a fully-connected multi-layer ANN to classify the MNIST digits, converts it to an SNN, and evaluates it on the MNIST test data. However, there have been a lot of changes to BindsNET since we've written the script, so it will likely require some debugging.

[4iar]

Awesome, cheers @djsaunde!

[0xnurl]

For anyone reaching this thread looking for an up-to-date version of the ANN->SNN MNIST script, I updated it to work with BindsNET 0.3.1 (pending PR, but project seems to be no longer maintained)

BINDS-LAB-UMASS/bindsnet_experiments#9

[Hananel-Hazan]

Thank you @0xnurl for your efforts and sharing the PR. You are right, this part of the project is no longer maintain and nor longer supported. But if you have something in mind please reach out and we will think on a way forward.