Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --scale, --position-x-offset, --position-y-offset and --rotation-offset #4658

Open
wants to merge 4 commits into
base: dev
Choose a base branch
from

Conversation

willykc
Copy link

@willykc willykc commented Feb 9, 2024

This addresses #4135 and #4345 by enabling additional transformations to frame the image within the window boundaries. It also applies the transformations on input control signals. Recording is not supported.

@rom1v
Copy link
Collaborator

rom1v commented Feb 9, 2024

Thank you for your contribution 👍

This feature is indeed often requested for the Meta Quest 3.


I had some compilation issues, that I fixed by replacing float32_t by float and by adding the math library as a dependency:

sed -i 's/float32_t/float/g' app/src/screen.c
diff --git a/app/meson.build b/app/meson.build
index 88e2df9aa..d6f403dbe 100644
--- a/app/meson.build
+++ b/app/meson.build
@@ -99,6 +99,7 @@ endif
 cc = meson.get_compiler('c')
 
 dependencies = [
+    cc.find_library('m'),
     dependency('libavformat', version: '>= 57.33'),
     dependency('libavcodec', version: '>= 57.37'),
     dependency('libavutil'),

Congrats, the input location seems to work correctly 🎉

I did not really review yet (it may take time, I have other changes to review/implement beforehand).

But I have some general remarks. A priori, I am a bit worried by the (intrinsic) complexity added for supporting any rotation/scale for the rendering. This means that every future change will have to deal with this, and may not assume a landcape/portrait view anymore.

I think that in practice, the Meta Quest 3 is the only use case. Also, this has to be used in conjonction with --crop, and since Android 14 capture cropping is broken (#4011). But anyway.

Semantically, maybe --rotation-offset could be merged with --rotation. One problem is that the window size currently does not depend on the rotation offset. (Should it expand to view the whole content? I don't know.)

@rom1v
Copy link
Collaborator

rom1v commented Feb 9, 2024

As an alternative, I think it should be possible to filter the video directly on the device: IIRC we should be able to capture the screen and filter it via an OpenGL ES shader, and provide the result as input for the encoder (see SurfaceTexture).

There would be several benefits:

  • it would impact recording
  • it would provide an alternative for cropping (broken since Android 14)
  • it would be less invasive

@willykc
Copy link
Author

willykc commented Feb 10, 2024

I did look into a server-side implementation, but the client offered the path of least resistance. The company I work for was interested in casting the Quest 3 for public events, and this implementation got us there without too much effort. In the case of the Quest 3, It turned out to be useful to keep window size independent from rotation and scale so the image could be framed without black corners.

Regarding --crop, it can technically be used without it by scaling up and moving the image around with position offsets, but it is always a waste of bandwidth. If possible, a server-side solution would be better in the long run.

@vuisme
Copy link

vuisme commented Feb 20, 2024

I did look into a server-side implementation, but the client offered the path of least resistance. The company I work for was interested in casting the Quest 3 for public events, and this implementation got us there without too much effort. In the case of the Quest 3, It turned out to be useful to keep window size independent from rotation and scale so the image could be framed without black corners.

Regarding --crop, it can technically be used without it by scaling up and moving the image around with position offsets, but it is always a waste of bandwidth. If possible, a server-side solution would be better in the long run.

I built it successfully and it works well. But how to rotate and then crop instead of cropping and then rotate like now?

@willykc
Copy link
Author

willykc commented Feb 20, 2024

I built it successfully and it works well. But how to rotate and then crop instead of cropping and then rotate like now?

Unfortunately, at the moment, cropping will always happen first since it is done on the device. The only alternative is to be more generous with the initial cropping and then frame the image by playing with --scale, --position-x-offset and --position-y-offset.

@rolandwellinger
Copy link

When is this coming to the mainbranch and release?

@vuisme
Copy link

vuisme commented Feb 23, 2024

When is this coming to the mainbranch and release?
Because the applicability is low, I think it won't make it to the main branch. If scrcpy has a plugin mechanism like yt-dlp it would be better

@willykc : Do you have a solution that can control the Quest screen? I can only control via display ID (main Quest windows such as horizon feed, settings, windows store...) and cannot control via the main screen or in the app

@willykc
Copy link
Author

willykc commented Feb 23, 2024

Unfortunately no, I haven't looked at controlling the Quest externally.

@ZaherAlali001
Copy link

I did look into a server-side implementation, but the client offered the path of least resistance. The company I work for was interested in casting the Quest 3 for public events, and this implementation got us there without too much effort. In the case of the Quest 3, It turned out to be useful to keep window size independent from rotation and scale so the image could be framed without black corners.

Regarding --crop, it can technically be used without it by scaling up and moving the image around with position offsets, but it is always a waste of bandwidth. If possible, a server-side solution would be better in the long run.

Can you post the last scrcpy command for casting to the pc with the rotatiom and cropping...etc, that you use for the quest 3?

@willykc
Copy link
Author

willykc commented Feb 27, 2024

Can you post the last scrcpy command for casting to the pc with the rotatiom and cropping...etc, that you use for the quest 3?

This worked well for me on a 16:9 display in fullscreen mode:
scrcpy --crop=2744:1544:20:350 --rotation-offset=22 --scale=159 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f
I think cropping could be more aggressive though. It can still be optimized.

@Varsett
Copy link

Varsett commented Mar 2, 2024

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

@rom1v
Copy link
Collaborator

rom1v commented Mar 2, 2024

As an alternative, I think it should be possible to filter the video directly on the device

Refs #4011 (comment) for information about implementing a server-side filter.

@vuisme
Copy link

vuisme commented Mar 2, 2024

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

@Varsett
Copy link

Varsett commented Mar 2, 2024

This is my build if you need

Thank you so much!

@rolandwellinger
Copy link

When is this coming to the mainbranch and release?
Because the applicability is low, I think it won't make it to the main branch. If scrcpy has a plugin mechanism like yt-dlp it would be better

What? This must come to the mainbranch. i really wonder what sidequest ist doing without that. sorry but quest 3 is sold out!

@rolandwellinger
Copy link

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

Absolute Hero. Thanks für the Build. Will test it right away.

@hiroyamochi
Copy link

hiroyamochi commented Mar 17, 2024

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

Great work! I will use your modified scrcpy to include it in my app which specially made for Meta Quest mirroring.

@qianlishun
Copy link

This worked well for me on a 16:9 display in fullscreen mode:

Hi! I'm develope special software for Meta Quest's headsets and scrcpy included to it. Where i can find this modified version scrcpy, with support rotation for Quest 3?

Just build from @willykc source. This is my build if you need https://github.com/vuisme/escrcpy/tree/main/electron/resources/extra/win/scrcpy

Good work. However, when using -- record, there may be some issues where the recorded video is not rotated or cropped.
Is there any way?

@willykc
Copy link
Author

willykc commented Mar 28, 2024

Unfortunately recording is not supported by this approach. Implementing the rotation on the device as discussed previously would address that limitation though.

@ReadOnlyMAIN
Copy link

ReadOnlyMAIN commented Apr 8, 2024

Can you post the last scrcpy command for casting to the pc with the rotatiom and cropping...etc, that you use for the quest 3?

This worked well for me on a 16:9 display in fullscreen mode: scrcpy --crop=2744:1544:20:350 --rotation-offset=22 --scale=159 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f I think cropping could be more aggressive though. It can still be optimized.

Hello, using this command as a basis, I found out that this crop and scale were absolutly perfect for quest 3.
scrcpy --crop=2064:2208:2064:100 --window-title='Quest3Cast' --rotation-offset=-22 --scale=195 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f

@Xericore
Copy link

Xericore commented Apr 9, 2024

Hello, using this command as a basis, I found out that this crop and scale were absolutly perfect for quest 3. scrcpy --crop=2064:2208:2064:100 --window-title='Quest3Cast' --rotation-offset=-22 --scale=195 --position-x-offset=-520 --position-y-offset=-490 --video-bit-rate=16M -f

This command works well, thank you. The rotation-offset feature is great for the Quest 3 and Quest Pro. Any reason this is not being merged into the main branch?

@rom1v
Copy link
Collaborator

rom1v commented Apr 9, 2024

Any reason this is not being merged into the main branch

Because I would like the transform to be performed on the server side (before encoding): #4658 (comment)

@qianlishun
Copy link

I tried to modify a window size of Quest3, it feels good for me.

--crop=1920:1440:20:350 --rotation-offset=20 --scale=159 --position-x-offset=-800 --position-y-offset=-500

@beaugunderson
Copy link

I applied the chunks individually against master and this does still work as advertised.

@Montana420
Copy link

Sorry to pester any updates on this one? Works fine via the alternative method/branch but my god would it be nice to just use SideQuest..

@demencia89
Copy link

Is there any way to run the modded build that supports rotation on linux? Thx!

@luixo
Copy link

luixo commented Sep 19, 2024

@rom1v Can this be merged with a TODO for server-side transform?
That will improve quality of life for a lot of Quest users.

@rom1v
Copy link
Collaborator

rom1v commented Nov 7, 2024

I just published a prototype to apply a video filter on the device: #5455

You can edit the code to hardcode the AffineTransform you need for the Quest 3 (typically, a "reframe" followed by a rotation).

@rom1v
Copy link
Collaborator

rom1v commented Nov 15, 2024

Please test #5455 and the new --angle option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.