The program periodically reads monitoring data from an OpenEMS Energy Management System and stores these as measurements in an InfluxDB V2 database.
For OpenEMS, see: https://github.com/OpenEMS/openems
OpenEMS-based systems usually come already with a powerful User Interface including vizualization and monitoring.
monitorEMS is intended for users who want to
- integrate EMS monitoring or EMS data with monitoring of other systems, such as a Heat Pump, Home Automation or Weather forecast Systems.
- add monitoring features which are currently not (yet) available in OpenEMS, such as battery monotoring.
- have the historical data of their Energy Management System under their own control
monitorEMS uses a generic approach for mapping openEMS to Influx datapoints (see below). This allows monitoring virtually all accessible OpenEMS data just by configuration.
For Visualization, some Grafana dashboards are included (see Visualization).
In order to use the program, you need
- an Energy Management System, based on OpenEMS
(monitorEMS has been developed and tested with a Fenecon Home 10 system, running FEMS) - a computer on which Docker is running (Windows PC, Linux, e.g. Raspi, NAS, ...), which is available for 24/7 operation.
(This project has been deployed on a Synology DS220+ Disk Station) - An Influx DB V2.x (eventally running as Docker container)
- A Grafana instance for visualization (eventally running as Docker container)
If you are not already running a Docker Engine, see Install Docker Engine.
On NAS systems, Docker is usually available as App in the NAS OS.
InfluxDB is a time series database. monitorEMS currently only supports the free V2.x version of InfluxDB!
Grafana is an open source analytics and vizualization platform.
The following description assumes running all applications as Docker containers.
You may alternatively run monitorEMS in a virtual Python environment from the command line (see below).
| Step | Action |
|---|---|
| 1. | Install and configure an InfluxDB V2 (https://docs.influxdata.com/influxdb/v2/install/) |
| 2. | In InfluxDB, create a new bucket. The monitorEMS default configuration assumes "EMS" |
| 3. | In InfluxDB, create an API Token with write access to the bucket |
| 4. | Prepare a config folder on a suitable location of the machine where Docker is installed. |
| 5. | Download monitorEMS_tpl.json and store it as monitorEMS.json in the config folder prepared in step 4 |
| 6. | Adjust the configuration file (see Configuration) Essential adjustments are: - Hostname or IP-Address of the EMS - URL for the Influx DB - Influx Org, Bucket and Token for EMS data |
| 7. | Run the latest monitorems Docker image in a Docker container. The container's /app/config directory must be mapped to the directory created in step 4.From the command line, this can be done with docker run --name monitorems --mount type=bind,src=<configDir>,dst=/app/config signag/monitorems:latestwhere <configDir> must be replaced by the full path of the directory created in step 4. |
| 8. | Check the container log for correct operation or any errors (you may need to wait for the next period which will be indicated in the log) |
| 9. | In the Influx Data Explorer, check that data are being written to the EMS bucket. |
| 10. | Set up the Grafana dashboards, supplied with this package (see Grafana Setup). |
Configuration for monitorEMS needs to be provided in a specific configuration file.
By default, a configuration file "monitorEMS.json" is searched in the given sequence under $ROOT/config, $HOME/.config or under /etc.
Here, $ROOT is the project root directory and $HOME is the home directory of the user running the program.
For testing in a development environment, primarily the location $ROOT/tests/data is searched for a configuration file.
Alternatively, the path to the configuration file can be specified on the command line.
The Docker image expects a configuration file "monitorEMS.json" under /app/config which needs to be mapped to a directory in a container-external file system.
The following is an example of a configuration file:
A template can be found under
$ROOT/config in the installation folder.
{
"measurementInterval": 120,
"emsURL": "http://femsxxxxx:80",
"emsUsername": "x",
"emsPassword": "user",
"InfluxOutput": true,
"InfluxURL": "",
"InfluxOrg": "",
"InfluxToken": "",
"InfluxBucket": "EMS",
"csvOutput": false,
"csvDir": "",
"emsData": [
{
"measurement": "ems_sum",
"component": "/_sum",
"channel_root": "/",
"csvFile": "monitorEMS_sum.csv",
"query": "/rest/channel/_sum/[CEGP].*",
"tag_datapoints": [],
"field_datapoints": [
"ProductionActivePower",
"ProductionActiveEnergy",
"ConsumptionActivePower",
"ConsumptionActiveEnergy",
"GridActivePower",
"GridBuyActiveEnergy",
"GridSellActiveEnergy",
]
},
{
"measurement": "ems_battery_cells",
"component": "/battery?",
"channel_root": "/Tower?Module?Cell???",
"csvFile": "monitorEMS_battery_cells.csv",
"query": "/rest/channel/battery\\d{1}/Tower\\d{1}Module\\d{1}Cell\\d{3}V.*",
"field_datapoints": [
"Voltage"
]
}
]
}| Parameter | Description |
|---|---|
| measurementInterval | Measurement interval in seconds. (Default: 120) |
| emsURL | URL for EMS. E.g. for FEMS: "http://femsNNNNN:80", where NNNNN is the FEMS-number |
| emsUsername | User name for authentifacion to EMS (usually "x") (see REST-Api Controller) |
| emsPassword | Password for authenticion to EMS (usually "user") |
| InfluxOutput | Specifies whether data shall be stored in InfluxDB (Default: false) |
| InfluxURL | URL for access to Influx DB |
| InfluxOrg | Organization Name specified during InfluxDB installation |
| InfluxToken | Influx API Token (see Getting started) |
| InfluxBucket | Bucket to be used for storage of EMS data |
| csvOutput | Specifies whether EMS data shall be written to a csv file (Default: false) |
| csvDir | Path to the directory where csv files shall be located |
| emsData | List of configuration sets for Influx measurements to be created. For explanation of the different elements, see Mapping of OpenEMS to InfluxDB Data Points, below. For an example, see monitorEMS_tpl.json. See also How to find Configurations for Data Points Of Interest. |
| - measurement | Influx Measurement to be used for data points of this configuration |
| - component | Pattern to be applied to the OpenEMS Component-ID to identify Influx Tag Key and Tag Value e.g. pattern battery?, applied to the Component-IDbattery0,identifies "battery" as Tag Key and "0" as Tag Value. |
| - channel_root | Pattern to be applied to the OpenEMS Channel-ID to identify Influx Tag Keys, Tag Values and Field Keys e.g. pattern /Tower?Module?Cell???, applied to/Tower0Module3Cell012Voltageidentifues the Influx Tags {"Tower": "0", "Module": "3", "Cell": "012" } and "Voltage" as Influx Field Key. |
| - csvFile | Name of the csv file to which CSV data shall be written, if activated |
| - query | Query to be used to query data from a REST-API endpoint. Regular expression shall be used to fetch all required OpenEMS datapoints in one query. This is necessary in order to avoid REST-call limitations. On the other hand, the regular expression shall be specific enough to limit the result as much as possible to the intended data points, in order to avoid unnecessary large result sets. |
| - tag_datapoints | JSON list of datapoints resulting from the query to be used as Influx Tags |
| - field_datapoints | JSON list of datapoints to be used as Influx Fields |
In order to understand how monitorEMS analyzes the response from OpenEMS REST-API endpoints and how these are transformed into Influx measurement data points, it is necessary to look at the OpenEMS REST-API and the resulting JSON structure:
The API syntax is (using authorization header):
http://<hostname|IP>:80/rest/channel/<Component-ID>/<Channel-ID>
For example, the URL to query the voltage of a specific battery cell is:
http://femsxxxxx:80/rest/channel/battery0/Tower0Module0Cell002Voltage
with the JSON result
{
"address": "battery0/Tower0Module0Cell002Voltage",
"type": "INTEGER",
"accessMode": "RO",
"text": "",
"unit": "V",
"value": 3313
}OpenEMS REST-API endpoints also support regular expressions.
So, the URL
http://femsxxxxx:80/rest/channel/battery\d{1}/Tower\d{1}Module\d{1}Cell\d{3}V.*
returns the voltage for all cells of all modules of all towers of all batteries:
[
{
"address": "battery0/Tower0Module4Cell008Voltage",
"type": "INTEGER",
"accessMode": "RO",
"text": "",
"unit": "V",
"value": 3314
},
{
"address": "battery0/Tower0Module4Cell004Voltage",
"type": "INTEGER",
"accessMode": "RO",
"text": "",
"unit": "V",
"value": 3313
},
{
"address": "battery0/Tower0Module4Cell000Voltage",
"type": "INTEGER",
"accessMode": "RO",
"text": "",
"unit": "V",
"value": 3314
},
{
"address": "battery0/Tower0Module2Cell012Voltage",
"type": "INTEGER",
"accessMode": "RO",
"text": "",
"unit": "V",
"value": 3313
},
...
]
monitorEMS now considers each element of Component-IDs (e.g. "battery") and Channel-IDs (e.g. "Tower", "Module", "Cell") as Influx Tag Keys and the values behind these keys as Influx Tag Values.
The part of the Component-ID after the last 'tag key' ("Voltage" in the example above) is considered as Influx Field Key and the "value" element of the result is considered as Influx Field Value.
A unique set of Tag Values (In the example: {"battery", "Tower", "Module", "Cell"}) is assigned a specific Influx measurement.
Its name can be freely chosen in monitorEMS, for example "ems_battery_cells" for the example above.
In addition to the Tags from Component-IDs and Channel-IDs, it is also possible to treat specific openEMS datapoints as tags.
For example "SerialNumber", which is a datapoint resulting from the REST endpoint
battery0/Tower0Module[N]SerialNumber
It should not be treated as Influx Field but as Tag, adding information to "ems_battery_modules" measurements with characteristic Tags {"battery", "Tower", "Module"}, thus allowing to track the SerialNumber for each module together with module-related fields such as temperatures from different sensors.
This is quite important, because after adding new modules on top of a tower, the module number of the 'old' modules will change (OpenEMS counts from top to bottom) but the SerialNumber remains unchanged.
To uniquely identify Tag Keys and Field Keys from Component-IDs and Channel-IDs, the monitorEMS configuration uses patterns with ? identifying positions of Tag values.
The examples, below, refer to "ems_battery_cells" measurement from the example, above:
- "component" pattern
battery0
battery? - "channel_root" pattern
Tower0Module2Cell012Voltage
Tower?Module?Cell???
Here, the end of the "channel_root" pattern-string marks the begin of the Influx Field Key, here "Voltage".
This option should be used when experimenting and testing with configurations for new data points before a configuration is exposed to the Docker container.
The program has especially a -t option for testing which forces a cycle to start immediately without waiting for the configured interval.
In order to run monitorEMS (or other Python programs of this package) as Python program, proceed as follows:
(Python 3.10 or later and Git must have been installed)
| Step | Action |
|---|---|
| 1. | In the command shell of your system navigate to a suitable parent ($PARENT) folder |
| 2. | Clone the monitorEMS repositorygit clone https://github.com/signag/monitorEMS |
| 3. | Create and activate virtual Python environmentcd monitorEMSpython -m venv .venv.venv\Scripts\activate (Windows)source .venv/bin/activate (Linux) |
| 4. | Install necessary programs:pip install -r requirements.txt |
| 5. | Start program:cd monitorEMSpython monitorEMS.py -hThis should show the usage description (see Usage) |
| 6. | Now, you can create a $PARENT/monitorEMS/tests/data folder, stage the configuration file and run a test withpython monitorEMS.py -t -v |
(Not required when running the Docker image)
usage: monitorEMS.py [-h] [-t] [-s] [-l] [-L] [-F] [-p LOGFILE] [-f FILE] [-v] [-c CONFIG]
This program periodically reads data from an openEMS Energy Management System
and stores these as measurements in an InfluxDB database.
If not otherwises specified on the command line, a configuration file
monitorEMS.json
will be searched sequentially under ./tests/data, ./config, $HOME/.config or /etc.
This configuration file specifies credentials for EMS access,
the data to read, the connection to the InfluxDB and other runtime parameters.
The configuration file also defines the EMS datapoints which will be stored as measurements.
-h, --help show this help message and exit
-t, --test Test run - single cycle - no wait
-s, --service Run as service - special logging
-l, --log Shallow (module) logging
-L, --Log Deep logging
-F, --Full Full logging
-p LOGFILE, --logfile LOGFILE
path to log file
-f FILE, --file FILE Logging configuration from specified JSON dictionary file
-v, --verbose Verbose - log INFO level
-c CONFIG, --config CONFIG
Path to config file to be usedThis package includes a small tryEmsApi.py Python program which can be used to evaluate the different OpenEMS REST-API endpoints and the data they disclose.
Before usage, copy this file to $PARENTDIR/tests and name it according to your needs. ($PARENTDIR/tests is ignored by git)
- Set 'emsURL' to the URL of your system
- Set 'component' to one of the components shown in your OpenEMS configuration
- Adjust the name of the 'csvFile' to be produced
- Run the program in the virtual environment for monitorEMS. It will create the CSV file under an 'emsOutput' subdirectory.
- From the CSV file, select the data points to be monitored
- Adjust the regular expression in 'channel' to a more restrictive variant
- Use the resulting output to set up an 'emsData' entry in the 'monitorEMS.json' configuration file