Touchscreen Problems

[h0d3nt3uf3l]

Hi, yesterday I intalled the picframe and want to use it as "screensaver" for my HomeAssistant panel in the living room.

Hardware:
I'm using a RPI4 with Raspberry Pi OS (64-bit, Released: 2024-11-19). It's using Wayland with labwc as compositor.
Attached is a Iiyama North America PL2453MT in Portrait-Mode (works ot of the box with the rotation made on the Desktop, the touch function was rotated as well)

I was looking at the code today to get the touchscreen running. A few clicks did some things but it was never predictable.
I found out, that the mouse-postition and touch-position of the pointer where different and the touch position implausible as well (e.g. x= -23403, y= -2233)

That's because in pi3d.display Line 351 is delivering other values. I don't know it for sure, but I guess pi3d is broken on this point...
Btw. don't find where _mouse_relative is set for sure..

I'm not deep enough in that code to do some pull requests, but maybe I can help you with my findings.
It works for me with followed changed lines:

pi3d.display Line 350..

        elif self.ev.type == sdl2.SDL_MOUSEMOTION:
          if self._mouse_relative: # default
->          self._mouse_x = self.ev.motion.x
->          self._mouse_y = -self.ev.motion.y
          else:
            self._mouse_x = self.ev.motion.x
            self._mouse_y = -self.ev.motion.y

pifcrame.interface_peripherals Line 291..

    def __update_pointer_position(self) -> None:
        position_x, position_y = self.__mouse.position()
        if self.__input_type == "mouse":
            position_x -= self.__viewer.display_width // 2
 ->         position_y += self.__viewer.display_height // 2
        elif self.__input_type == "touch":
            # Workaround, pi3d seems to always assume screen ratio 4:3 so touch is incorrectly translated
            # to x, y on screens with a different ratio
 ->         position_x -= self.__viewer.display_width // 2
 ->         position_y += self.__viewer.display_height // 2
 ->         # position_y *= self.__viewer.display_height / (self.__viewer.display_width * 3 / 4)
        self.__pointer_position = (position_x, position_y)

Greetings

P.s. Maybe you could tell me why there is always a bisection of the display resolution... I don't get it :)

[paddywwoof]

Hi @h0d3nt3uf3l thanks for feedback and big fixing. The pi3d mouse location is rather messy as it has adopted functionality from lots of different bits of software which has to be specified on start up. The original RPi worked well without desktop but had to get mouse input using fairly low level IO. Later, with an x11 desktop, or tk, sdl2 etc there ended up with more mechanisms.

I'm away from home at the moment but will have a look at this when I get home.

[paddywwoof]

@h0d3nt3uf3l just had a quick look at this. The interface_peripherals was all @glenvorel's work and I've not actually tried it but I will see what I can figure out.

Your question about display resolution bisection: do you mean why does position_x have half the display width subtracted (and the y have half the height added)? The reason is related to the fact that the pi3d coordinates are based on OpenGL 3D vectors where (x,y,z) = (0,0,1) is straight ahead. Leftwards is negative, right is positive, down is negative, up is positive. In screen coordinates (i.e. used by SDL mouse events but also including OpenGL pixel locations) the top left is 0,0, but, as you can see this is further confused by the fact that the mouse.y value is multiplied by -1 within pi3d.Display. This results in the position_y needing half the display height adding rather than subtracting!

[h0d3nt3uf3l]

@paddywwoof thanks for this explanation. Now I have a hint for a misbehaviour which is still present with my touchscreen.
The "os pointer" isn't in synch with the pi3d pointer atm. After reboot I have to "reset" it with touching in the lower right corner (getting the pi3d to the max border)
Not in synch means: If I touch after reboot the upper left corner, the pointer is showing somewhere at x ~ 350 and y ~ -900 on a display resolution of 1920x1080

Do I understand you right that in OpenGL logic, the middle of the display (e.g. 1920 / 2 and 1080 / 2) is point (0, 0, 1)?

[paddywwoof]

@h0d3nt3uf3l TL;DR yes. The longer answer is that there is vector math that takes a projection of the vertices representing polygons in 3D, as if viewed from the point of view of the camera, scales them for perspective and only renders the nearest ones to the camera. Obviously, for rendering a single quad for picframe, that's all a lot more complicated than is really needed. Also, within OpenGL texture lookup (i.e. UV coordinates) the origin is in the top left corner and for fragment sampling (i.e. pixels from a render) the origin is bottom left (I think, from memory).

[h0d3nt3uf3l]

@paddywwoof yeah, just read about the opengl coordinate systems, the different spaces for them and transformation from 3D to the screen.

Will do some changes on my code these days and try to get the touch screen work properly.
Hope that my first thoughts about bugs in pi3d will be wrong because of my lack of understanding openGL, would be hard to work on both projects with missing knowledge if not.

[paddywwoof]

@h0d3nt3uf3l there is almost certainly stuff that needs to be tidied (aka fixed) in the SDL2 mouse and keyboard code for in pi3d. I don't think that your patch of making the _mouse_relative work like an absolute mouse location can be correct as it will break all the relative mouse use (i.e. using the mouse to 'steer' around a 3D landscape you want the camera to rotate as you move the mouse without worrying where the mouse actually is). However, the fact that it worked better after you made the change, makes me think that _mouse_relative should be False.

This is set https://github.com/tipam/pi3d/blob/df0cc84aebc56fb8d62e2517aeb48e59da689681/src/pi3d/Mouse.py#L56 which means that for it to be True the restrict argument to Mouse.__init__() must have been False.

And the mouse is initialised here

picframe/src/picframe/interface_peripherals.py

Line 138 in 6540065

restrict=self.__input_type == "mouse",

i.e. for restrict to be False, __input_type must not be "mouse", and presumably you set it as "touch", so that makes sense... But maybe it needs to operate differently if you had to tweak your code as you did.