Picframe Overlay

[helgeerbe]

Actual I use electron to create an overlay for picframe. It's a transparent window running in the foreground and rendering e. g. a weather forecast (https://github.com/helgeerbe/picframe-overlay).

This is a little shaky, because it depends on the order you have to start picframe and electron.

@paddywwoof added a text file input to the clock.

I'm wondering now, how performant it would be if electron would render a bitmap (https://www.electronjs.org/docs/latest/tutorial/offscreen-rendering). Picframe would read this file and render it on top of the actual image.

In this way, you can render any website on top of picframe (weather, calendar, analog clock, .....). This would be pretty cool.

[paddywwoof]

I suppose it would be quite easy to make an overlay that would read an image file. The complexity would be how much flexibility to allow. Maybe a starter would be to simply look in /dev/shm/overlay.png but some way of signalling when to reload that and where to position it might also be minimal requirements. I will think about it a little more.

[helgeerbe]

from PIL import Image

background = Image.open("test1.png")
foreground = Image.open("test2.png")

background.paste(foreground, (0, 0), foreground)
background.show()

Could do the job. Maybe with size and offset info for the foreground image. So it doesn't have to be the same size.

Electron saves the bitmap on changes of the website with a frame rate of up to 240. So this should be hopefully fast enough. Even for an analog clock (counting seconds).

[helgeerbe]

I will prepare a small electron clock app, writing the image to an file.

[paddywwoof]

I suppose the simplest signalling would be to check the os last modified time. Having an image reloaded every second would be quite a load but once a minute should be fine. Also the image could be much lower resolution than the display then scaled up by pi3d using the GPU.

[helgeerbe]

For a static overlay this works pretty well.

foreground = Image.open("/Users/herbe/Development/picframe-overlay/ex.png")
            foreground = foreground.resize(im.size)
            im.paste(foreground, (0, 0), foreground)
            tex = pi3d.Texture(im, blend=True, m_repeat=True, free_after_load=True)

Have to find out how to do the blending within pi3d mapping 2 textures.

[paddywwoof]

Hi, using the CPU (pillow) to do all the blending and resizing will create quite a load on the little RPi. Using the GPU (pi3d) will have scarcely any additional load beyond rendering the existing layers of picframe. However loading from disk has to use pillow so that will always take time. For an analogue clock I might use rotating sprites for the hands.

[helgeerbe]

Hi Paddy,

I agree that for a simple clock sprites would be the better idea. But for me, it's more a speed test.

My intention is to use a smart speaker and mqtt to show some dashbords. Like weather forecast, the family calendar, dates for waste collection, gasoline price, energy state of my solar battery, urgent messages of my home automation …..

It’s quite easy to render everything into a nice png images. And updating this images only every minute should be fine. But I’m curious how much image refreshes my pi 3 can take. For my 1920x1200 images it has a load of 0.2. (all logs are on ramdisk). So there is still some capacity left.

[paddywwoof]

Helge, I also think the best route for doing the overlay is rendering to an image file. That way all the variable complexity can be kept out of picframe - there's always a danger of enhancements making it bloated and un-maintainable.

With pi3d, the image texture dimensions can be different from the pixel dimensions of the plane that is being rendered and the scaling happens automatically and very efficiently (i.e. as objects move around a scene in a computer game). The other efficiency is that the default pi3d.Texture loading will set the blend=False which means that if the texture has an alpha channel, the GPU will discard pixels that are sufficiently transparent without wasting any effort rendering them. The flip side of this is that the pixels that are visible will be drawn using depth priority. If you render a foreground before a background then the pixels that would be masked in the background get discarded. This sometimes causes odd edges - you will see what I mean if you change the constructor argument in the example you posted above (is one of the foreground objects in your im.paste the background object?)

Should I add some extra functionality to the viewer_display.py dev branch, to do the overlay or have you already solved it?

[helgeerbe]

It would be great if you can provide a function which is doing the overlay.

Guess this function has to be called in slideshow_is_running() just before self.__slide.draw() not to affect the blending.

How to handle auto scale, or explicit placing? Adding a parameter [x,y,width,height]? I would start with a full resolution overlay, but maybe a quarter sized overlay image with scaling is good enough?

[paddywwoof]

OK I will knock something up. To start it will just be full size. The quad that the texture is drawn onto will be the size of the display but the texture can be any size. If the aspect ratio is wrong there will be distortion and if the resolution is too low then it won't be sharp.

I will try it just in front of the main image - i.e. behind clock and picture info text. See if blend mode makes the fading jerky.

[helgeerbe]

Perfect, Thanks Paddy

[paddywwoof]

I've pushed an addition to the dev branch 05feb15 which simply pastes whatever image is located at /dev/shm/overlay.png over the picture being displayed. It is behind the clock and info text.

I haven't included any config aspects at the moment, it just takes the image and fits it to the screen. Also the blend mode is False which doesn't give any feathering for blurred edges. Possibly many other things that could be improved.

Paddy

EDIT - I see, just checking the code, that I pushed a version with blend=True but it seems to work well with False so I might change it to that to save a bit of work. Most overlay images will be transparent pixels so this might make a difference.

[helgeerbe]

IMG_5894_small.mov

Thanks Paddy, this is so cool! This is on my Mac. Hope I will find time on the weekend to do some Pi testing.

By the way:

 self.__display = pi3d.Display.create(x=self.__display_x, y=self.__display_y,
                                             w=self.__display_w, h=self.__display_h, frames_per_second=self.__fps,
                                             display_config=pi3d.DISPLAY_CONFIG_HIDE_CURSOR,
                                             background=self.__background, use_glx=self.__use_glx, use_sdl2=self.__use_sdl2)

sdl2 is not released yet for pi3d?

[paddywwoof]

Hi, impressive updating every second! Yes sdl2 is in pi3d main branch but I did find that the pip install of pysdl2 hadn't pulled in the libsdl2-dev so had to manually apt install it on this RPi.

PS I tried an overlay 25% of the screen size (1/16th area) and it looked fine across the room, or even close up.

[helgeerbe]

I did some tests on my pi3 with https://github.com/helgeerbe/picframe-overlay. The overall performance is quite good.
I rendered the clock with 1 frame per second in 1920x1200 and picframe was capable to show it with a load below 2.

[paddywwoof]

That is good; kudos to pillow for being able to load and unpack a png into numpy array so efficiently!

I will make a couple of tweaks so that the brightness also works on the clock and overlay, then, provided it works OK, I will pull into main.

[helgeerbe]

I add a small plugin to picframe-overlay to display the image meta info.

[paddywwoof]

Very neat!
Is that running separately from picframe - just getting the exif info, then generating and writing to /dev/shm/overlay.png? In which case should we pull the changes from dev into main? I've been running dev here for the last week on a Pi3 and a Pi5 and it hasn't had any problems.

Paddy

[helgeerbe]

What I want to test:

1. Using Redis Cache instead of File - Interface.
2. Findout how performant picframe would be (framerate) in sending numpy arrays -> redis -> electron.
   In that way, electron could handle all the user interaction while picframe renders the background.

[jn3va]

Hi - This looks awesome. I've been using PicFrame for a while and love it. Recently I saw Magic Mirror and liked the ability to overlay weather, info from Home Assistant etc., but PicFrame is SO much better because of the matting and transitions! Nice Job. If I understood the conversation above; overlay is a separate app, it it creating a png image and saving it to disk, then PicFrame is picking that up and rendering it over top of the displayed image?

Its been a while since I did any dev but I'd like to better understand this and see if I can create something to get info from HomeAssistant to display as an overlay. I'll share back if I make any progress, but did not want to dive into this if this was something you think would be abandoned in favor of another approach.

I was originally going to try and fork the code and see if I could modify the viewer_display.py to show weather icons, weather data, sensors etc. send to PicFrame via MQTT but this seems like a better approach to protect the stability and future changesd you make to PicFrame :-)

@helgeerbe mentioned

This is a little shaky, because it depends on the order you have to start picframe and electron.

I'm assuming what you mean is that you might need to start the electron app first to make sure the overlay png has current info before PicFrame tries to display it, or was there something else?

Thanks!
Joe

[paddywwoof]

Joe, the overlay and clock text functionality is in the existing viewer_display.py code.

picframe/src/picframe/viewer_display.py

Line 458 in 6540065

def __draw_overlay(self):

and

picframe/src/picframe/viewer_display.py

Line 430 in 6540065

if os.path.isfile("/dev/shm/clock.txt"):

they simply look on the ramdisk /dev/shm/ for the overlay.png or clock.txt files and splat them over the picture. The image overlay only reloads the image to create a new pi3d.Texture (which can use a chunk of CPU) if the file has changed. The clock.txt is regenerated each time the clock time string changes - by default once per minute.

I think the advantage of using this, rather simplistic, method is that the image generation can be done by a completely stand alone process. This means you can mess around getting the image generation however you want it and picframe will just carry on rendering the results on top of the pictures.

If you get anything working or have any ideas for improvements we would love to hear.

Paddy

[jn3va]

Thanks for the quick reply Paddy. I saw those sections in the code but was not sure how to use them. Does the electron app create those file? (I'm going to try and look more closely at the PicFrame Overlay js code later tonight.)

[jn3va]

I got this working - this is VERY cool and I'm learning a lot!

I had to reinstall the RPi OS. The instructions I used originally to get set up had me use the Lite version without the desktop so I was getting and XServer error. I'll tinker more with this over the next few days. Have a great night!

[paddywwoof]

I've not used electron, but I think it's handy if you want to make apps for different platforms. And you like JS!!

You can generate your image using whatever you find easiest, there are probably python modules that build on PIL (which is capable of doing everything you need on its own, but it might be hard work making it all look nice).

You shouldn't need the full desktop if you have the latest setup as per Wolfgang's quick install script. I followed those for my last setup on the kitchen RPi3 and I've just made a png with transparency and scp overlay.png [email protected]:/dev/shm/overlay.png and up it popped!

[jn3va]

Every time I ran picframe-overlay (npm start) i’d get and error about missing X server.  I had to change the auto-login to start the desktop instead of the console and then it worked.  I saved the image on a separate SD card so may play more after I understand everything better. Thanks! On Nov 20, 2024, at 5:25 PM, Paddy ***@***.***> wrote: I've not used electron, but I think it's handy if you want to make apps for different platforms. And you like JS!! You can generate your image using whatever you find easiest, there are probably python modules that build on PIL (which is capable of doing everything you need on its own, but it might be hard work making it all look nice). You shouldn't need the full desktop if you have the latest setup as per Wolfgang's quick install script. I followed those for my last setup on the kitchen RPi3 and I've just made a png with transparency and scp overlay.png ***@***.***:/dev/shm/overlay.png and up it popped! —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[paddywwoof]

ah, I see. You were using Helge's electron project picframe-overlay. It makes sense that there are probably dependencies missing in the lite version. The whole browser environment is pretty heavy weight so not surprising.