Autostart in background / Docker image

[joddye2]

Hi!

With "python -m TheengsGateway" from terminal works like a charm. If I close the window, publishing finishes (the TG runs on raspbian ).

Is there an easy way for autostart and working in background without live terminal?

Possible to dockerize this whole thing?

[1technophile]

Hi,

Here is some methods for this:
https://www.dexterindustries.com/howto/run-a-program-on-your-raspberry-pi-at-startup/

I will be happy to have your feedback on these different methods (or another).

[joddye2]

I looked around some methods but most of them refers to use "something.py"....but we starts the gw with python mod not a script. I tried to use the ble_gateway.py and init.py without success.

[1technophile]

Also just to know which models are compatible with TheengsGateway could you share your Raspberry Pi model please?

[joddye2]

Hello!

I'm using RPI Zero 2 W with the latest Raspberry PI OS Lite

[koenvervloesem]

This works for me on a Raspberry Pi 4:

Install TheengsGateway as root (see later for why this is):

sudo pip3 install TheengsGateway

Then create this systemd service script in /etc/systemd/system/TheengsGateway.service:

[Unit]
Description=Theengs Gateway
After=network.target
[Service]
Restart=always
Type=simple
ExecStart=/usr/bin/python3 -m TheengsGateway -H host -u user -p password

[Install]
WantedBy=multi-user.target

Use your host, username and password for the MQTT broker.

Then reload the service files and start and enable the system service:

sudo systemctl daemon-reload
sudo systemctl enable --now TheengsGateway.service

Check whether the gateway is running:

systemctl status TheengsGateway

Check the logs if something's wrong:

journalctl -xe

It should also be possible to adapt this to run as a non-root user (and then you can install the TheengsGateway package as a non-root user too), but I'll have to try this first.

[Werner-G]

Since I installed TheengsGateway as root, my theengsgw.conf wasn't use anymore. I don't understand which -h etc.-tags do you mean?

[joddye2]

Since I installed TheengsGateway as root, my theengsgw.conf wasn't use anymore. I don't understand which -h etc.-tags do you mean?

optional arguments:
-h, --help show this help message and exit
-H HOST, --host HOST MQTT host address
-P PORT, --port PORT MQTT host port
-u USER, --user USER MQTT username
-p PWD, --pass PWD MQTT password
-pt PUB_TOPIC, --pub_topic PUB_TOPIC
MQTT publish topic
-st SUB_TOPIC, --sub_topic SUB_TOPIC
MQTT subscribe topic
-sd SCAN_DUR, --scan_duration SCAN_DUR
BLE scan duration (seconds)
-tb TIME_BETWEEN, --time_between TIME_BETWEEN
Seconds to wait between scans
-ll {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
TheengsGateway log level

[Werner-G]

OK, understand. I learnt to stop service before reload the deamon. Everythings works fine. I love this Gateway since this a good companion to zigbee2mqtt which I use for all my Zigbee-sensors.

[joddye2]

I've experienced some issues with this method.
After the "enable" command starts the service, but when a restart occured won't start.

Status:
Apr 28 21:37:48 raspberrypi4 systemd[1]: TheengsGateway.service: Scheduled restart job, restart counter is at 5.
Apr 28 21:37:48 raspberrypi4 systemd[1]: Stopped Theengs Gateway.
Apr 28 21:37:48 raspberrypi4 systemd[1]: TheengsGateway.service: Start request repeated too quickly.
Apr 28 21:37:48 raspberrypi4 systemd[1]: TheengsGateway.service: Failed with result 'exit-code'.
Apr 28 21:37:48 raspberrypi4 systemd[1]: Failed to start Theengs Gateway.

[joddye2]

Problem solved!

After=network-online.target

[jerome6994]

Use your host, username and password for the MQTT broker.

host : ip_broker:1883 ?
username: none
password: none

So I can write the script like this ?
[Unit] Description=Theengs Gateway After=network.target
[Service] Restart=always Type=simple ExecStart=/usr/bin/python3 -m TheengsGateway -H ip_broker:1883 -u -p
[Install] WantedBy=multi-user.target

[koenvervloesem]

You can just drop the -u and -p parameters if you're not using authentication.

[jerome6994]

OK but failed

but I think it comes from further upstream of the script
with this python3 -m TheengsGateway -h https://theengs.github.io/gateway/install/install.html#install-theengs-gateway I have

I have a Proxmox debian 11 light VM

[1technophile]

Hello,

Is bluez installed, try maybe:
sudo apt-get install bluez

[jerome6994]

I just installed bluez

I resumed the process
I'm logged in as root, does that matter?
It's just that I haven't put any sudo right yet

it works on debian 10 or 11 ?

then it is said to do
python -m TheengsGateway -H "192.168.1.17" -u "username" -p "password"

but it's not rather
python3 -m TheengsGateway -H "192.168.1.17" -u "username" -p "password"

[1technophile]

Once bluez is installed you can try again:
python3 -m TheengsGateway -h

[1technophile]

It should work on both debian versions 10 and 11

[1technophile]

then it is said to do
python -m TheengsGateway -H "192.168.1.17" -u "username" -p "password"
but it's not rather
python3 -m TheengsGateway -H "192.168.1.17" -u "username" -p "password"

It depends on your python installation. The gateway requires python 3 installed. I would try both.

[jerome6994]

I just made the order and here is the rendering
python3 -m TheengsGateway -h

On my machine I believe it's python3 by default

I want to start from scratch with a fresh VM

[1technophile]

That's good, you can now try to launch it
python3 -m TheengsGateway -H "HAAS IP address" -u "username" -p "password"
Username and password can be created into Haas Users section (it will apply directly to the broker)

[jerome6994]

So I just opened the service just now by doing :
python3 -m TheengsGateway -H "broker IP"
because I did not put a user on the brocker which is outside of HASS

I just turned off my VM and turned it on again and everything works great

remains to be seen how we integrate the various BT devices concerned in the gateway and then in HA

[pnieweglowski]

It is possible to run Theengs in docker.

FROM python:3.10.4-bullseye
RUN apt-get update && apt-get upgrade -y
RUN pip3 install TheengsGateway
RUN apt-get install -y bluez
ENV current /config
ENV HOME $current
WORKDIR $current
CMD ["python3","-m","TheengsGateway"]

Keep in mind, that you have to run it in privileged mode, mount dbus, config dir and with network: host.

[1technophile]

@pnieweglowski thanks for sharing this, how do you manage the configuration on this use case? Do you access to the docker to change it.

[mihsu81]

@pnieweglowski, Here's the log. I guess there aren't wheels available for a particular CPU architecture and cmake and ninja are needed in order to build one locally. On the same board but with Tinker Board Debian 10 V3.0.11 also takes a long time because it has to build the missing wheels.

Collecting TheengsGateway
  Downloading TheengsGateway-0.1.5.tar.gz (118 kB)
     |████████████████████████████████| 118 kB 752 kB/s
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/pip-standalone-pip-cdqab7c2/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f97lw3sl/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools cmake wheel scikit-build ninja
       cwd: None
  Complete output (67 lines):
  Collecting setuptools
    Downloading setuptools-62.1.0-py3-none-any.whl (1.1 MB)
  Collecting cmake
    Downloading cmake-3.22.4.tar.gz (31 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Collecting scikit-build
    Using cached scikit_build-0.14.1-py2.py3-none-any.whl (76 kB)
  Collecting ninja
    Downloading ninja-1.10.2.3.tar.gz (26 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting packaging
    Using cached packaging-21.3-py3-none-any.whl (40 kB)
  Collecting distro
    Using cached distro-1.7.0-py3-none-any.whl (20 kB)
  Collecting pyparsing!=3.0.5,>=2.0.2
    Using cached pyparsing-3.0.8-py3-none-any.whl (98 kB)
  Building wheels for collected packages: cmake, ninja
    Building wheel for cmake (pyproject.toml): started
    Building wheel for cmake (pyproject.toml): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 /tmp/tmp5_lcavjl build_wheel /tmp/tmp8kt_gghh
         cwd: /tmp/pip-install-a8nra4nl/cmake_6be77b0c3ff14fc7840fa871e41d5d27
    Complete output (9 lines):
      File "/tmp/pip-build-env-3w0mwcx1/overlay/lib/python3.7/site-packages/skbuild/setuptools_wrap.py", line 613, in setup
        cmkr = cmaker.CMaker(cmake_executable)
      File "/tmp/pip-build-env-3w0mwcx1/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 141, in __init__
        self.cmake_version = get_cmake_version(self.cmake_executable)
      File "/tmp/pip-build-env-3w0mwcx1/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 96, in get_cmake_version
        "Problem with the CMake installation, aborting build. CMake executable is %s" % cmake_executable
    Traceback (most recent call last):

    Problem with the CMake installation, aborting build. CMake executable is cmake
    ----------------------------------------
    ERROR: Failed building wheel for cmake
    Building wheel for ninja (pyproject.toml): started
    Building wheel for ninja (pyproject.toml): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 /tmp/tmpvtmgb26k build_wheel /tmp/tmp7f9gzwqo
         cwd: /tmp/pip-install-a8nra4nl/ninja_aab29cc3600240b9b65b8b41fd22292d
    Complete output (9 lines):
      File "/tmp/pip-build-env-6iziev3a/overlay/lib/python3.7/site-packages/skbuild/setuptools_wrap.py", line 613, in setup
        cmkr = cmaker.CMaker(cmake_executable)
      File "/tmp/pip-build-env-6iziev3a/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 141, in __init__
        self.cmake_version = get_cmake_version(self.cmake_executable)
      File "/tmp/pip-build-env-6iziev3a/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 96, in get_cmake_version
        "Problem with the CMake installation, aborting build. CMake executable is %s" % cmake_executable
    Traceback (most recent call last):

    Problem with the CMake installation, aborting build. CMake executable is cmake
    ----------------------------------------
    ERROR: Failed building wheel for ninja
  Failed to build cmake ninja
  ERROR: Could not build wheels for cmake, ninja, which is required to install pyproject.toml-based projects
  WARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.
  You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/ea/08/5a38455181c25d7d2cbd03f12c863888d16671262baa8cdff4bdcaf72c71/TheengsGateway-0.1.5.tar.gz#sha256=64ebf42756e2a77b9bb075c16b7e756c9bd78b4825fb3b59c15803a04bca7d5d (from https://pypi.org/simple/theengsgateway/). Command errored out with exit status 1: /usr/bin/python3 /tmp/pip-standalone-pip-cdqab7c2/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-f97lw3sl/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools cmake wheel scikit-build ninja Check the logs for full command output.

[pnieweglowski]

@mihsu81 cmake's wheel doesn't contain binary wheel for arm32 https://pypi.org/project/cmake/
But for your platform, you can use piwheels project,

FROM python:3.9
RUN apt-get update && apt-get upgrade -y
RUN pip3 install --extra-index-url=https://www.piwheels.org/simple TheengsGateway
RUN apt-get install -y bluez
ENV current /config
ENV HOME $current
WORKDIR $current
CMD ["python3","-m","TheengsGateway"]

If you want to compile it from scratch, docker buildx for crossplatform operations can be used, eg. building on your pc, running on Tinker

[pnieweglowski]

@1technophile I created config file and mount it to docker. I run most of services using docker-compose, but docker will work too.

$ tree TheengsGateway/
TheengsGateway/
├── build
│   └── Dockerfile
├── config
│   └── theengsgw.conf
└── readme.md

2 directories, 3 files
$ cat TheengsGateway/config/theengsgw.conf 
{
    "ble_scan_time": 5,
    "ble_time_between_scans": 5,
    "discovery": false,
    "host": "ha.lan",
    "log_level": "INFO",
    "pass": "xxxxxx",
    "port": 1883,
    "publish_all": true,
    "publish_topic": "home/TheengsGateway/BTtoMQTT",
    "subscribe_topic": "home/TheengsGateway/commands",
    "user": "zzzzzzzz"
}
$ cat docker-compose.yml
version: '3'
services:
  theengs:
    container_name: theengs
    build:
      context: TheengsGateway/build
    image: tg:latest
    network_mode: host
    privileged: true
    restart: unless-stopped
    volumes:
      - /home/ha/TheengsGateway/config/:/config/
      - /var/run/dbus/:/var/run/dbus/

[1technophile]

Added it here https://hub.docker.com/repository/docker/theengs/gateway/general thanks to the work of @maretodoric

[mihsu81]

@pnieweglowski @1technophile I've created an HA add-on, but It doesn't seem to scan for BLE devices.
Any suggestions?
https://github.com/mihsu81/addon-theengsgw

[mihsu81]

@1technophile Finally got to the bottom of this issue. It's caused by an outdated "libseccomp2" in Debian Buster. debuerreotype/docker-debian-artifacts#106 and itzg/docker-minecraft-server#755 (comment).
You'll need to add the buster-backports repo and update libseccomp2.

#Add the signing keys to verify the new packages
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 04EE7237B7D453EC 648ACFD622F3D138
#Add the Buster backport repository to apt sources.list
echo 'deb http://httpredir.debian.org/debian buster-backports main contrib non-free' | sudo tee -a /etc/apt/sources.list.d/debian-backports.list
sudo apt update
sudo apt --reinstall install libseccomp2 -t buster-backports

Or just upgrade to the latest Raspbian based on Bullseye. (https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2022-04-07/2022-04-04-raspios-bullseye-armhf-lite.img.xz)

@pnieweglowski Thanks a lot for your suggestion regarding https://www.piwheels.org/simple. I've added it as source for TheengsGateway and it sure makes things simpler.

[1technophile]

Great, the installation worked!
Do you think we could create the mqtt user during this install process and add it to the configuration ?

[mihsu81]

I don't know if I understood correctly your question, but you're asking if this add-on would be able to create a user in another add-on running a MQTT broker? I only meant it as a wrapper around TheengsGateway which would allow me to run it on my HA installation. It expects the user to have a MQTT broker and credentials available. :)

[1technophile]

Yes, if the addon could create the user automatically into Home Assistant (the MQTT broker uses HASS users), as a consequence, there will be no need to enter any configuration.
This would requires to have the broker addon as a prerequisite with auto-discovery activated.
Not sure how this is possible.

By the way, what you have done is already a great step!

[mihsu81]

Ok, that makes it much clearer now. I completely forgot that the Mosquitto Broker Add-on can authenticate HA users.
It might be possible, but I'll make no promises. 😁

[RadianM]

In case it helps anyone in the situation I found myself in, koenvervloesem wrote :

It should also be possible to adapt this to run as a non-root user (and then you can install the TheengsGateway package as a non-root user too), but I'll have to try this first.

I had already installed everything as user Pi and only later decided to create a daemon to start on boot. The adaptation required was to add User=pi in [service] i.e.

[Service]
Restart=always
Type=simple
User=pi
Environment="PYTHONPATH=$PYTHONPATH: /home/pi/.local/lib/python3.9/site-packages"
ExecStart=/usr/bin/python3 -m TheengsGateway

I'm not sure if the Environment needed to be set but I'd already found that and set it to be sure.

[JeewanBahra]

@RadianM This approach worked amazingly for me so thank you :)

[RadianM]

Thank CoPilot
Seriously - the number of times it comes up with a solution when I'm struggling with nix!