Python Support for Adapters, Processors, & Sinks

[mtaylor-os]

The Idea:
Python integration with Apache StreamPipes allowing Python to be used to create custom Adapters, Processors, and Sinks.

Background:
Having worked in industry for many decades now, and seeing where SCADA systems are going, it feels like StreamPipes could be a strong player in the data manipulation workflow if it has improved Python support.

Most SCADA systems already have Python support (Ignition, some iFix) or planning on it (DeltaV) so there's a large group of capable engineers that already understands basic Python scripting. However, the one thing all these SCADA systems lack is robust data processing...aka compute abilities. They take data from PLCs, display it, trend it, and alarm on it. But our organization, like many across the globe, are moving towards computed data analysis such as statistical analysis on a process variable (think 6-Sigma), inference and prediction (think Artificial Intelligence), and energy/green-house-gas reduction (think mass + energy balance). All these calculations are challenging to do in the native SCADA system...I've tried. If Apache StreamPipes provided more robust Python integration it would elevate the platform above NodeRed and NiFi...the two current contenders in this field and what many controls engineers currently lean towards with compute needs arise.

[tenthe]

Hello @mtaylor-os,

thank you for starting this disucssion.
I would love a deeper integration with Python, especially for processors to allow users to easily implement their own logic.
Currently you can only do this with functions. These are slightly less flexible for users of the StreamPipes GUI, but Python developers can use them to process historical and streaming data. Did you already try this out?
Are there certain use cases where the functions are not sufficient for your requirements?

Regarding adapters & data sinks. Are there any specific implementations that you miss? These could initially also be implemented with Java, as users should not notice the difference.

Cheers,
Philipp

[mtaylor-os]

I did not try out the functions yet, partly because I've not yet found good documentation on where they can and cannot be used and what are their limitations. Regardless, the limited documentation I did find did not appear to say I could use them to create processors, adapters, and data sinks.

Our use cases are many, but boil down to adding compute abilities to our existing SCADA systems. First we'd need to ingest data from the SCADA into StreamPipes, which means StreamPipes would need to serve as an OPC-UA client or modbus-tcp client and pull data in. Then do the compute work such as long-term trending, continuous regression analysis, or advanced energy usage modelling with multiple data sources. StreamPipes would then need to provide that computed data back to the SCADA system, which means StreamPipes would need to be an OPC-UA server or modbus-tcp server. Finally the SCADA system would pull data from one of those StreamPipe servers.

Based on the documentation, StreamPipes does not appear to currently support the above workflow. Because many of our controls engineers have experience with Python due to its ubiquity, and those same engineers think of coffee when I mention Java, the thought was Python could be used to fill in the gaps...and hence the question.

Appreciate the help!

[bossenti]

Hi @mtaylor-os,

really appreciate your input here!

Have you already found our tutorials on using functions in Python?
https://streampipes.apache.org/docs/docs/python/dev/tutorials/3-getting-live-data-from-the-streampipes-data-stream/
https://streampipes.apache.org/docs/docs/python/dev/tutorials/4-using-online-machine-learning-on-a-streampipes-data-stream/
https://streampipes.apache.org/docs/docs/python/dev/tutorials/5-applying-interoperable-machine-learning-in-streampipes/
This might at least give you an idea of how functions can be used.
In combination with our support for OPC-UA (OPC-UA Adapter & OPC-UA Sink) and Modbus (Modbus Adapter).
In addition, since you are talking about pulling data from StreamPipes with a SCADA system, you can access data from StreamPipes via our REST API. Either directly using the provided endpoints or using one of our clients (Java, Python, Go). Since Python is most suitable for you, you can find examples here: https://streampipes.apache.org/docs/docs/python/dev/tutorials/2-extracting-data-from-the-streampipes-data-lake/

Based on my understanding, it should be possible to implement your desired workflow or use case with these concepts.
Nevertheless, supporting the generation of adapters, processors and sinks via Python is something we should definitely tackle.
Once again, we really appreciate your input and sharing your use case and requirements. It really helps us. We would be happy to further discuss what our Python support could look like if the above is not feasible for you.

[mtaylor-os]

@bossenti,
Thank you kindly. I did come across the OPC-UA Sink link you provided, but the documents say it is for storing into an Apache CouchDB and not OPC-UA so I was a bit confused. Also it didn't directly say an OPC-UA server would be created, and because OPC-UA is a bidirectional protocol it's entirely possible the OPC-UA Sink is actually a client that writes to an OPC-UA server, rather than being a server itself. Can you confirm which of these it is?

In your ONNX python tutorial it shows how to push computed data back to StreamPipes, which looks usable. Is there also a methodology to push data back to StreamPipes from the REST API? That would allow for a clean separation of concerns from any python compute backend and StreamPipes input/output to SCADA.

Many thanks for your time and energy!

[bossenti]

the documents say it is for storing into an Apache CouchDB and not OPC-UA so I was a bit confused.

Damn, that's simply an error in the documentation. I'll fix it.
The OPC-UA sink is only capable to write to a OPC-UA server. This is done by using a OPC-UA client library, StreamPipes itself does not provide or operate (as) an OPC-UA server. However, in can serve as an intermediate between two OPC-UA servers being able both reading and writig from/to an OPC-UA server.

No, it's not possible to directly push data to StreamPipes via it's REST API directly. This is due to the concept that data always comes via an adapter into StreamPipes. However, you could use the HTTP server adapter. This spins up a HTTP server providing you an endpoint to which data can be pushed which is then available in StreamPipes. Please keep in mind that this adapter has mainly been used for development and testing purposes.
What is your idea behind such an approach? If you don't mind sharing, we migh have a better solution available.

Hope this helps! 🙂

[mtaylor-os]

Dang, I was really hoping to decouple our SCADA system from StreamPipes by having StreamPipes be an OPC-UA server. If StreamPipes can only serve as an OPC-UA client then new tags must be made on the SCADA system, then StreamPipes has to be updated to write to those tags.

The idea behind all this is to use our SCADA system to continue collecting some 200k-300k tags as it has always done. Then StreamPipes could pull a few of those tags, manipulate* them in some way, and provide the calculated results as a new set of tags. Our SCADA system could then read those new tags and historize them, alarm on them, send emails on them, all the things our controls engineers already do with the SCADA system.

• = calculate moving averages over days, calculate range/mean/mode/median/stdev statistics, perform linear regression to estimate when an event will happen in the future, etc.

[bossenti]

To which extend are your SCADA systems flexible how or from where they pull data? Is a push approach also feasible?

So from my understanding, StreamPipes can provide you with the functionalities you described. The only missing piece is the data transfer from StreamPipes to the SCADA systems right?
Do you have anything in mind how this would be possible with improved Python support?

[mtaylor-os]

SCADA systems are inherently large, complex, and expensive beasts. As such they typically target mainstream communication protocols commonly in use for industrial control systems. Such communication protocols that can be used include Allen Bradley specific protocols, Siemens specific protocols, ModbusTCP, and OPC-UA. With a few rare exceptions, pushing data to a SCADA runs contrary to how they are designed so support is limited or unavailable.

It sounds like StreamPipes may be able to provide the functionality, yes. Of course testing will confirm that in due time. The only missing piece is indeed the ability to pull data from StreamPipes to our SCADA systems.

Because controls engineers commonly use Python, they are familiar with it...not so with Java. This means that improved Python support would allow controls engineers across the globe do things like create a new data adapter that can interface to some specialty sensor (for example our ultrasonic flow meters that use a simple but proprietary TCP protocol), create new processors (like how we are looking to create an energy consumption calculator to hone in on high energy users), and custom data sinks (we would love to output data to a different SCADA system another plant uses, and hence allow integration across plants).

We do very much appreciate the information and support, thank you.

[bossenti]

It sounds like StreamPipes may be able to provide the functionality, yes. Of course testing will confirm that in due time. The only missing piece is indeed the ability to pull data from StreamPipes to our SCADA systems.

Please keep us in the loop how it works out for you. We appreciate any feedback or are happy to help if you encounter any problems. In terms of pulling data from StreamPipes, I would suggest to use the REST-API.

create new processors (like how we are looking to create an energy consumption calculator to hone in on high energy users), and custom data sinks (we would love to output data to a different SCADA system another plant uses, and hence allow integration across plants).

This is in the end what a function can provide you with. Of course they have their downsides and you don't have a reusable pipeline element or sink in the end, but you can at least achieve the desired outcome. Functions the option I'd choose for our use case in a POC.