Using pyFAST in Google CoLab and GitHub Actions

[andreped]

I have previously been able to use pyFAST in Docker containers, meaning that pyFAST can be used on headless servers.
Deployment was then performed in pyFAST using fast.Pipeline.

Recently, FAST added headless support to runPipeline which would mean that I could replace the temporary custom pyFAST fast.Pipeline approach I had with a more generalized CLI method.

I therefore wanted to test using this CLI in two relevant use cases:

1. In a GitHub Actions workflow for unit testing (CPU only - run in containers hosted by GitHub)
2. In a CoLab Jupyter Notebook to produce interactive demos

For the first one, as there is no GPU available, I used PoCL, which at least made it possible for me to import pyFAST without any errors. However, when running some simple tests, I got some errors, which I did not get on my local setup - likely due to the usage of PoCL. I therefore put my focus on CoLab instead.

For the latter, I tried to use PoCL as well (also choosing CPU runtime), just to see if I could get a CPU-only approach working. I ran some simple runPipeline and python tutorial tests. None of these worked. One of them executed without errors, but it did not seem like it actually started running.

I have made a gist to reproduce these experiments, which could be valuable to test PoCL further with pyFAST on CoLab.

Of course, in CoLab, we do actually have access to a GPU, so PoCL may not be required, however, setting up OpenCL/OpenGL in CoLab seemed tricky. I could make a proper attempt though, but I was not tempted.

I guess the main reason is that pyFAST is not really compatible with PoCL, which may not be that surprising.

Anyways, I'm making this thread for discussion as I'm debugging this further.

[andreped]

I managed to get pyFAST working on CoLab using the same install setup as I used for setting up pyFAST in Docker (see here).
So that is great! That means that we can make pyFAST demos in CoLab which is a great way for others to learn about FAST :]
Note that the nuclei segmentation pipeline still didn't start properly. No idea why. Any ideas?

I made the gist openly available for others to try.

But of course, we do not have access to a GPU when running pyFASTs on CPU clusters such as using GitHub-hosted containers through GitHub Actions. Any alternative to PoCL that I could try?

[smistad]

The errors you get with pocl are build errors. Meaning that pocl fails to compile some of FAST's OpenCL code during execution. This is most likely due to bugs in pocl.

You can try newer versions of pocl (build yourself or use newer Ubuntu) or you can try Intel's CPU only OpenCL runtime

[andreped]

I managed to fix the issue. After decompression, I needed to place the extracted x64/ directory at the appropriate location. Now clinfo finds the platform.

However, I now run into another issue when running tests (for both runPipeline and using pyFAST):

INFO [133801215184576] Found 1 OpenCL platforms.
INFO [133801215184576] 1 platforms selected for inspection.
INFO [133801215184576] Platform 0: Intel(R) Corporation
INFO [133801215184576] This platform has 1 available devices in total
INFO [133801215184576] Looking for GPU devices only.
INFO [133801215184576] 0 selected.
INFO [133801215184576] NO suitable GPU found! Trying to find other devices
INFO [133801215184576] Found 1 OpenCL platforms.
INFO [133801215184576] 1 platforms selected for inspection.
INFO [133801215184576] Platform 0: Intel(R) Corporation
INFO [133801215184576] This platform has 1 available devices in total
INFO [133801215184576] Looking for all types of devices.
INFO [133801215184576] 1 selected.
INFO [133801215184576] Inspecting device 0 with the name            Intel(R) Xeon(R) CPU @ 2.20GHz
INFO [133801215184576] The device was accepted.
INFO [133801215184576]            Intel(R) Xeon(R) CPU @ 2.20GHz has 2 compute units
INFO [133801215184576] The device            Intel(R) Xeon(R) CPU @ 2.20GHz got a score of 2
INFO [133801215184576] The platform Intel(R) OpenCL was selected as the best platform.
INFO [133801215184576] A total of 1 devices were selected for the context from this platform:
INFO [133801215184576] The best device was:            Intel(R) Xeon(R) CPU @ 2.20GHz
ERROR [133801215184576] Terminated with unhandled exception: clCreateContext

See updated gist.

[smistad]

Strange. Now it is able to find the platform and the CPU, but fails when initializing OpenCL..

[andreped]

Strange indeed. But I added the updated gist, so you can debug that further on CoLab, when you have the time.

When this is properly working, I will try to run some tests on GitHub-hosted servers. If that works, then I believe it opens some new opportunities :]

[andreped]

As it seemed like I was potentially using the wrong Intel OpenCL package, I now tried installing the Intel OpenCL oneAPI runtime instead to see if I am able to use FAST on a CPU-only server.

After installation, it seems to be able to find the platforms, but we get RuntimeError: clCreateImage when attempting to run any FAST pipelines with runPipeline or pyFAST scripts.

To reproduce the error, see gist on CoLab.

[lomshabhishek]

On Mac M1 as well I faced similar issue

Traceback (most recent call last):
  File "/Users/abhisheklomsh/Downloads/pyfast_file.py", line 46, in <module>
    .run()
  File "/Users/abhisheklomsh/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fast/fast.py", line 3440, in run
    return _fast.ProcessObject_run(self, executeToken)
RuntimeError: Thread stopped because: clCreateImage