Speedup phoniebox code

[laclaro]

Hi,

first of all: this is a great project! I don't like mopidy, because of it's bulkiness, therefore I stick to the slim MPD-version. However, on my raspberry pi nano the handling of rfid-triggering and playout is very slow (annoying lag, making it not suitable to children) and also on rpi3 I notice a lag.

Therefore I dug a bit in the codes of rfid_trigger.sh and playout_controls.sh.

I think we should think about the approach used in the scripts. I identified some things that decrease the speed until playout considerably.

1. there are VERY many SD-card accesses. This is maybe the main issue. Maybe change the approach to read the config once, keep it in memory and only update if something changed (e.g. by python-handling).
2. some tasks are not necessary for EVERY run (maybe only once, such as the initialization checks "if directory/file exists"). Commenting increases speed!
3. Large case-statements to handle everything in one single script. Maybe introduce another command-line option to avoid having to check all cases or directly call different scripts from within python.
4. strange and very complicated implementation of second-swipe-functionality (because of earlier use of vlc?).
5. Cool "resume"-feature decreases speed on rpi nano by a lot!

I tinkered a bit with the code to find this out. Is there the possibility to have a bash script run as daemon, such that all the script files are kept in memory already? Otherwise it would be clever to read all settings from python to avoid excessive I/O on swipe.

Best,

Henning

[MiczFlor]

Hi @laclaro
point taken. The thing that puzzles me most is that the delays came with the introduction of the spotify version - and affects the classic (non-spotify) version, too.
I had a look through the code, too, to see if there are any http requests in the core shell because of spotify. But didn't find any. And don't know really what the speed loss comes from. Because there was a dramatic drop after spotify came in.
I doubt it is the "if file exists" explains the delays. If you have the time, give it a shot and comment them out - and see if it helps.
Generally speaking, the code has grown from "make it work" to "make it work" - and would benefit from a rethinking based on what we know now.
Feel free to come up with ideas, I will help where I can.
Thanks for your research and feedback, micz

[laclaro]

Hi,

of course I was mostly commenting things and swiping again, so the things mentioned above did slow my rpi nano. I was testing with a short beep played by daemon_rfid_reader.py when the card is recognized and the same short beep as sound mapped to the card.

If I have some time, I may think on this matter. I think we have to change the settings-changing-mechanism of the web-interface. Does it just write the setting to one of the files?

Best,

Henning

[laclaro]

I extended the daemon and provided a script to handle the new settings files. I decided to go with two files:

• One global settings file for phoniebox (settings/phoniebox.conf)
• One for the card assignments and folder.conf info (shared/Card_Assignments.txt).

It is work in progress still but provides playout-functionality and second_swipe.
To test you will have to install python ConfigParser via pip. There is also a package called python-configparser (note the missing capitals!). Also we need python-mpd.

~$ touch ~/RPi-Jukebox-RFID/shared/Card_Assignments.txt
~$ python change_setting.py assigncard 123456789 MPD_Audiofolder
~$ sudo python daemon_rfid_reader_new.py
-> swipe

The code is here: https://github.com/laclaro/RPi-Jukebox-RFID

There is work to do on changing the web-interface to use the change_setting.py script (that's at least what I was thinking of) and to extend the daemon to provide all the missing features of your bash-scripts.

Help is welcome!

Best,

Henning

[MiczFlor]

Hi Henning @laclaro
you beat me to it :) after your post I started to minimize shell SD reading and migrated all settings into one shell script which in turn writes a settings/global.conf

• I like your change_setings.py approach an would like to work towards one "global" script for read and write settings values.
• One question on my mind: does your solution (in progress) work with the Spotify version AND the Classic version or only one of these? Because that would be a show stopper.
• Is there any way you could provide a pull request to my code base and I will park your solution in a separate branch to work from?
• And one more question: what does the Card_Assignments.txt contain? Does it support subfolders?
  All the best and double thumbs up!
  micz

[laclaro]

Hi,

thank's for taking this serious ;)

First of all: I did not test with the spotify-version, since I don't like mopidy too much (I found it to slow for the pi, but only tried it with a large music database).
Second: I think there is no reason to assume that it does not work, since the protocol mopidy uses is mpd's, so everything (except blocked commands like "listall") should work. According to the paths in mopidy's database one has maybe to prepend a "Files/" or similar compared to plain mpd.

Since I wanted to merge all card-settings to one single file my Card_Assignments.txt contains sections for each card that read like

[231412138169]
cardid = 231412138169 <-- this is essentially redundant information, but in case one wants to change the "section" style...
uri = Tiergeräusche/Eule
shuffle = 0
repeat = 0
resume = 0
elapsed = 0
playstatus = stop

This is essentially the folders.conf and the shortcuts combined.

If we wanted to migrate slowly, I could write my code such that the daemon only handles the things rfid_trigger.sh took care of and then calls playout_controls.sh with the cardid.

Once we decided how we move forward I will do a pull request with a working version in that direction.

Best,

Henning

[laclaro]

One quick comment: the daemon may handle everything that involves playing songs, so essentially replacing playout_controls.sh by something that is for useful features beyond that, e.g. wifi toggles, etc.

I already implemented the checks for the folder.conf options and second swipe. What I did not yet work on is a coherent script, that can be used by the web UI to manage the cards and options.

If something is changed in the config or card assignments from within the web UI, the deamon could be triggered to reload the config files by some small file that is created by php or so.

Best regards

Henning

[MiczFlor]

Hi @laclaro
I leave this to your imagination and planning for now :) I have stuff on my local machine that creates a global conf from all the local setting files in the settings folder. Not sure if and how I will merge this with develop and or master.
However, don't wait for me, I wait for you...

[laclaro]

I created a new repository (https://github.com/laclaro/python-phoniebox) for my work on the python-phoniebox module that is separated in the PhonieboxDaemon.py and the Phoniebox.py. This separation was intended to make it easy to create third-party python scripts (new web interface?) to import the Phoniebox module to perform config-changes or assign new card settings.

In comparison with the plain MiczFlor-version it is much faster on my rpi3. Especially the bash-version starts lagging if you swipe too quickly.

The daemon is configured by a single config file phoniebox.conf and is able to use the RFID-assignments set by the web interface (I call this "legacy" card assignments) and updates the card assignments regularly during runtime, so new assignments get passed to the new python daemon. My daemon writes the read "legacy" card assignments to a single file. I would like to have the web interface use this format on the long run. However, the PhonieboxDaemon keeps the settings and assignments in memory during runtime thus reducing SD-write operations that slow down the software.

But since I am physicist and not a python developer, it is still lacking a couple of features like webstream support (has to be set up manually) and an additional script to natively change the config.

You are welcome to help me out.

Best,

Henning

[MiczFlor]

Hi @laclaro
thanks for the contribution. I haven't had the chance to look at it yet.
Could you create a pull request and I will create a new branch working on this? The last change of this scale was the removal of VLC as the player and replace it with MPD. From experience, it is an advantage to have many eyes on the task. Otherwise there is the "works for me" problem, which usually covers a third of the tasks that need to work - to make it accessible for everyone.
All the best, micz

[andreasbrett]

This separation was intended to make it easy to create third-party python scripts (...) to import the Phoniebox module to perform config-changes or assign new card settings.

Love that idea! Will definitely do so in the mqtt module.

[michnixweiss]

Sorry for digging this old thread up.
But I am using am Raspi Zero W, which is slow booting and lagging. So I searched for some improvements.
What do I have to do to test / use the Python versions?

Best regards

[laclaro]

https://github.com/MiczFlor/RPi-Jukebox-RFID/tree/develop/scripts/python-phoniebox

You have to run the PhonieboxDaemon.py (you may use systemd).

[michnixweiss]

Hey @laclaro
Thanks for reply.
Is your daemon replacement for startup-scripts.sh.sh?
So I disable it in systemd?

[laclaro]

startup-scripts.sh basically starts playoutcontrols which is not used anymore, correct.

[s-martin]

https://github.com/MiczFlor/RPi-Jukebox-RFID/tree/develop/scripts/python-phoniebox

You have to run the PhonieboxDaemon.py (you may use systemd).

python-phoniebox was an early attempt to switch to python, but is not used anymore.
I'm not even sure it works right now.

There's a branch future3 to rewrite Phoniebox in python, but it's not usable in production yet.

[MiczFlor]

Hi @michnixweiss
the code of @laclaro is a good starting point if you want to get your hands dirty. It covers quite a lot of functionality.
But, as @s-martin says, if you want to be in tune with Phoniebox's future, take a look at the future3 branch. https://github.com/MiczFlor/RPi-Jukebox-RFID/tree/future3/develop
It's more or less a total rewrite. It's meant to be stable and ready for testing end of September.

[michnixweiss]

Thanks @laclaro, @s-martin, @MiczFlor for your answers.

Do you have any suggestions to get the Pi Zero W start faster? The best I got is about 51sec to startup sound, which is the signal for my kids that its ready for use.
I "tried" the points mentioned here:

• https://raspberrypi.stackexchange.com/questions/63162/how-to-boot-raspberry-pi-very-quickly-into-linux
• 🐛 | 3 minute start-up time #1110
• https://himeshp.blogspot.com/2018/08/fast-boot-with-raspberry-pi.html?m=1
• https://silocitylabs.com/post/2019/06/13/ultra-minimal-raspbian-image-for-pi-zero-and-zero-w/
• https://splittscheid.de/phoniebox-schneller-booten-fehler-beheben/

Most performance from compiled mpd, uninstalling/disabling unused stuff and switch to dropbear.

I think about trying piCore or DietPi. What dependencies are needed (except python and php).