Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added on_gain_focus, on_lose_focus, on_show & on_hide handlers on toga.Window #2096

Open
wants to merge 93 commits into
base: main
Choose a base branch
from

Conversation

proneon267
Copy link
Contributor

@proneon267 proneon267 commented Aug 23, 2023

Implements the APIs described in #2009.

on_gain_focus, on_lose_focus, on_show& on_hide handles are available both as properties and also as initialization parameters in toga.Window.

Only tested on WinForms and gtk. This will take sometime to complete for all backends.

Fixes #2009

PR Checklist:

  • All new features have been tested
  • All new features have been documented
  • I have read the CONTRIBUTING.md file
  • I will abide by the code of conduct

Copy link
Member

@freakboy3742 freakboy3742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution - this is another feature that is going to hit up against #2058 (and #2075); as such, I'm hesitant to merge it without tests.

I'm also hesitant because it doesn't currently have a Cocoa implementation (that should be easy enough, but it's worth flagging); and it's not 100% obvious how this would behave on mobile. My immediate reaction (and the one raised in #2006) is that gain/lose focus might link into application lifecycle hooks (so when the app comes into the foreground, that's the "gain focus" event for both the window and the app), but there's an open question of how those signals interact with tablet platforms that allow split-screen and other multi-app modes. This is an area where some additional design is required.

@proneon267
Copy link
Contributor Author

Yes, I agree with you. This should not be merged currently. I will need some time to write the implementation for other backends. Like #1930, I will wait until the audits are merged, and will write the tests thereafter.

As for additional design for tablet modes, I agree with you. I will research more about it and will discuss with you while implementing for mobile platforms.

@proneon267
Copy link
Contributor Author

proneon267 commented Aug 25, 2023

@mhsmith I need your guidance on android side. According to: https://developer.android.com/guide/components/activities/intro-activities#onpause

The system calls onPause() when the activity loses focus and enters a Paused state. This state occurs when, for example, the user taps the Back or Recents button.

On the Emulator(Android 12):

When starting the app, the following events are triggered: onCreate()->onStart()->onResume() . But, when I press either the Back or Recents button, onPause() event is not triggered. No other events are triggered.

When I select the app by pressing the Recents button, only onStart()->onResume() are triggered. onRestart() is not triggered.

D/MainActivity: onStart() start
I/python.stdout: Toga app: onStart
D/MainActivity: onStart() complete
D/MainActivity: onResume() start
I/python.stdout: Toga app: onResume
D/MainActivity: onResume() complete
I/OpenGLRenderer: Davey! duration=18074ms; Flags=1, FrameTimelineVsyncId=10773, IntendedVsync=2419137327998, Vsync=2419137327998, InputEventId=0, HandleInputStart=2419139817470, AnimationStart=2419139847710, PerformTraversalsStart=2419139888290, DrawStart=2419223580770, FrameDeadline=2419170661330, FrameInterval=2419139788630, FrameStartTime=16666666, SyncQueued=2419225272400, SyncStart=2419226783620, IssueDrawCommandsStart=2419227470230, SwapBuffers=2419249275450, FrameCompleted=2437213784760, DequeueBufferDuration=21300, QueueBufferDuration=360800, GpuCompleted=2437213784760, SwapBuffersCompleted=2419251729810, DisplayPresentTime=0,

When I press the home button, neither onPause() or onStop() events are triggered. No other events are triggered.

On a Physical Device(Android 13):

When starting the app, the following events are triggered: onCreate()->onStart()->onResume() . But, when I press either the Back or Recents button, onPause() event is not triggered. Instead the following events are triggered:

D/VRI[MainActivity]: onFocusEvent false
D/VRI[MainActivity]: dispatchAppVisibility visible:false
D/BufferQueueProducer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:1,p:21956,c:21956) disconnect: api 1
D/BufferQueueConsumer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:0,p:-1,c:21956) disconnect
D/VRI[MainActivity]: setWindowStopped stopped:true
D/ActivityThread: do gfx trim 40 success

When I select the app by pressing the Recents button, only onStart()->onResume() are triggered. onRestart() is not triggered. Instead the following events are triggered:

D/VRI[MainActivity]: dispatchAppVisibility visible:true
D/VRI[MainActivity]: setWindowStopped stopped:false
D/MainActivity: onStart() start
I/python.stdout: Toga app: onStart
D/MainActivity: onStart() complete
D/MainActivity: onResume() start
I/python.stdout: Toga app: onResume
D/MainActivity: onResume() complete
I/Quality : Skipped: false 0 cost 5.253646 refreshRate 8332212 bit true processName com.example.helloworld
D/BufferQueueConsumer: [](id:55c400000001,api:0,p:-1,c:21956) connect: controlledByApp=false
E/IPCThreadState: attemptIncStrongHandle(57): Not supported
D/BufferQueueProducer: [VRI[MainActivity]#1(BLAST Consumer)1](id:55c400000001,api:1,p:21956,c:21956) connect: api=1 producerControlledByApp=true
D/VRI[MainActivity]: registerCallbacksForSync syncBuffer=false
D/VRI[MainActivity]: Received frameCommittedCallback lastAttemptedDrawFrameNum=1 didProduceBuffer=true syncBuffer=false
D/VRI[MainActivity]: draw finished.
D/VRI[MainActivity]: onFocusEvent true

When I press the home button, neither onPause() or onStop() events are triggered. But, the following events are triggered:

D/VRI[MainActivity]: onFocusEvent false
D/VRI[MainActivity]: dispatchAppVisibility visible:false
D/BufferQueueProducer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:1,p:21956,c:21956) disconnect: api 1
D/BufferQueueConsumer: [VRI[MainActivity]#0(BLAST Consumer)0](id:55c400000000,api:0,p:-1,c:21956) disconnect
D/VRI[MainActivity]: setWindowStopped stopped:true
D/ActivityThread: do gfx trim 40 success

As, you can see, onFocusEvent, dispatchAppVisibility visible and setWindowStopped stopped show consistent behaviors.

Why are the documented Activity lifecycle events not being triggered as per the documentation?

@proneon267
Copy link
Contributor Author

proneon267 commented Aug 27, 2023

Regarding the gain/lose focus on mobile platforms like android:

From my testing, the app will lose focus when either the Home or Recent App List buttons are pressed. On pressing the back button, the app also loses focus and sends the user to the home screen.

In split screen mode (like dual app mode), suppose there are two apps A and B. App A will gain focus when the user touches the A app's screen. The focus is lost when the user touches B app's screen or interacts with the system launcher or presses the Home or Recent App List Button.

In floating window mode, the app will gain focus when the user touches the app's screen. The focus is lost when the user touches anything outside the app, like interacting with the system launcher or another app.

In iOS like the cocoa, there exists UIApplicationDelegate, which has methods like: applicationDidBecomeActive and applicationWillResignActive

But, there needs to be another handler to differentiate between the states when (the app is not visible to the user & is not receiving inputs) and (when the app is visible to the user & is receiving inputs).

Hence, I would like to propose other additional handlers, on_background/foreground to call the handler when the app is in background/foreground and not visible/ visible to the user.

What do you guys think?

Also, without confirmation from @mhsmith regarding the Activity life events triggering behavior, I cannot proceed with the android implementation. Hence, I was thinking about working on the iOS implementation first.

@mhsmith
Copy link
Member

mhsmith commented Aug 27, 2023

When I press the home button, neither onPause() or onStop() events are triggered.

That's because those methods aren't included in the Android template, either in IPythonApp or in MainActivity. They should be added to both places, but ideally with an emptydefault implementation in order not to break any third-party implementations of IPythonApp. And all the existing methods should become default as well.

In split screen mode (like dual app mode), suppose there are two apps A and B.

See this page for how this is notified on Android.

But, there needs to be another handler to differentiate between the states when (the app is not visible to the user & is not receiving inputs) and (when the app is visible to the user & is receiving inputs).

Every API has a maintenance and testing cost, so I'd prefer not to add additional events unless there's a clear need for them, especially if they're only applicable to certain platforms.

@proneon267
Copy link
Contributor Author

Thank you for helping. I will add default implementations for the remaining methods in the Android template and will submit a PR there after getting a stable behavior.

I agree with you that additional events will incur more maintenance. I feel that the on_background and on_foreground events are mostly needed for mobile platforms. But there is a need for distinction between [not visible state] and [visible but not receiving inputs state] of an app.

For example, the app should be put to a sleep mode(not updating the layout or text) when it is in background or [not visible state].
But when it is in a [visible but not receiving inputs state](like in a multi app screen mode), the app should update the layout or text, so that the user can get the latest updated information.

What do you think?

@proneon267
Copy link
Contributor Author

I have tested android implementation both on a physical device and on the emulator.

I have submitted a PR at beeware/briefcase-android-gradle-template#69 so that the app focus event can be detected.

@proneon267
Copy link
Contributor Author

proneon267 commented Aug 28, 2023

Completed implementations of all the platforms and also added a test in the window example app.

I will write the tests after the audits are merged. But I think this PR is ready for a review.

Also, the CI android testbed is failing on its own for some reason.

Copy link
Member

@freakboy3742 freakboy3742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What you've done here looks like a good pass at implementing the API as proposed in #2009; however, I think we're hitting an area where we need more design before we proceed.

The detail you've dug up as part of the Android implementation has opened a bunch of design questions about what "focus" even means at the App/Window level. What are we actually trying to achieve with these signal handlers? Is "visibility" a better metaphor than "focus" in this case? Is there any use case for a literal "focus" event on a window? Do we need to differentiate between an app that is "visible", but isn't currently accepting input events, and an app that isn't accepting input events? How does the rest of the app lifecycle map into these events (on all platforms)?

Rather than pressing forward with an implementation, I think we need to step back and come up with a consistent design for these app/window lifecycle events, and work out how they map onto all the platforms we're targeting.

@proneon267
Copy link
Contributor Author

proneon267 commented Sep 1, 2023

Researching some more on the topic, it seems like we need 3 categories of events:

  • Input Focus -> [Receiving Inputs] or [Not Receiving Inputs]
  • Visibility -----> [Visible to User] or [Not Visible to User]
  • Hover

The following are the states associated with the event categories mentioned above and their implications for other event categories:

Input Focus ---> [Receiving Inputs]
  |                               | ->[Visible to User]
  |
  |---------------> [Not Receiving Inputs]
                                 | ->[Visible to User] or
                                 | ->[Not Visible to User]

Visiblity -> [Visible to User]
  |                      | ->[Receiving Inputs] or
  |                      | ->[Not Receiving Inputs]
  |
  |---------> [Not Visible to User]
                         | ->[Not Receiving Inputs]

Hover ->[Interacting with a Pointing Device or Mouse] & [Visible to User] & [Not Receiving Inputs]

Use Cases:

  • The use case for Input Focus might be to show a highlight effect indicating to the user that it is receiving input.

  • The use case for Visibility might be to:

    • Put the app to a sleep mode(not updating the layout or text) when it is [Not Visible to User].
    • Update the layout or text when it is [Visible to User], so that the user can get the latest information.
  • So far, I am not sure what the use case for Hover might be. Maybe showing effects that would imply to the user, clicking on it, will make it to receive input focus.

Who should have which event categories:

  • Toga.Window should have:
    • Input Focus: Since, multiple windows may be present, hence each window should trigger Input Focus event when they receive/lose input focus.
    • Visibility: Same reason as above
    • Hover: Same reason as above
  • Toga.App doesn't really need any of the event categories since it can just iterate through the Toga.App.windows list and add handler methods to the event handlers of the windows. But, we can add them if it would make intuitive sense for the user.

APIs are not much of a problem as the available platform APIs can be properly mapped onto the above described event categories.

@proneon267
Copy link
Contributor Author

I know that this PR is a larger one to review at once, but if possible could you review only the core tests, so that I would have it cleaned up before the final review?

Copy link
Member

@freakboy3742 freakboy3742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As requested, I've done an initial review of the core. The implementation looks fine; there's a couple of issues flagged in the tests.

I've had a quick look at some of the implementations as well; I've flagged a couple of things that stood out, but the biggest issue that occurred to me is the interaction of MINIMIZED and HIDE. What happens if a MINIMIZED window is HIDDEN, then restored to normal? It should still be HIDDEN... but AFAICT it will trigger an on_show event in most of these implementations.

changes/2009.feature.rst Outdated Show resolved Hide resolved
core/src/toga/window.py Outdated Show resolved Hide resolved
core/src/toga/window.py Outdated Show resolved Hide resolved
core/src/toga/window.py Outdated Show resolved Hide resolved
core/src/toga/window.py Outdated Show resolved Hide resolved
# than Q. onResume is the best indicator for the gain input focus event.
# https://developer.android.com/reference/android/app/Activity#onWindowFocusChanged(boolean):~:text=If%20the%20intent,the%20best%20indicator.
if Build.VERSION.SDK_INT < Build.VERSION_CODES.Q:
for window in self._impl.interface.windows:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is iterating over all windows appropriate? Won't it just be current_window that gains focus? Admittedly, this is mostly a moot point as long as Toga doesn't support multiple windows on Android, but in the event that we ever do support multiple windows (e.g., supporting an external display), it's worth getting the logic right here.

Comment on lines 587 to 636
def test_on_gain_focus(window):
assert window._on_gain_focus._raw is None

on_gain_focus_handler = Mock()
window.on_gain_focus = on_gain_focus_handler

assert window.on_gain_focus._raw == on_gain_focus_handler

window._impl.simulate_on_gain_focus()

on_gain_focus_handler.assert_called_once_with(window)


def test_on_lose_focus(window):
assert window.on_lose_focus._raw is None

on_lose_focus_handler = Mock()
window.on_lose_focus = on_lose_focus_handler

assert window.on_lose_focus._raw == on_lose_focus_handler

window._impl.simulate_on_lose_focus()

on_lose_focus_handler.assert_called_once_with(window)


def test_on_show(window):
assert window.on_show._raw is None

on_show_handler = Mock()
window.on_show = on_show_handler

assert window.on_show._raw == on_show_handler

window._impl.simulate_on_show()

on_show_handler.assert_called_once_with(window)


def test_on_hide(window):
assert window.on_hide._raw is None

on_hide_handler = Mock()
window.on_hide = on_hide_handler

assert window.on_hide._raw == on_hide_handler

window._impl.simulate_on_hide()

on_hide_handler.assert_called_once_with(window)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're not testing anything that isn't covered by the "real" focus/show tests, so I'd say they're not needed.

window2.on_lose_focus = window2_on_lose_focus_handler

app.current_window = window1
window1_on_gain_focus_handler.assert_called_once_with(window1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These calls assertions will be potentially misleading, as the mocks haven't been reset between each use. The test also isn't asserting the handlers that aren't called.

Suggested change
window1_on_gain_focus_handler.assert_called_once_with(window1)
window1_on_gain_focus_handler.assert_called_once_with(window1)
window1_on_lose_focus_handler.assert_not_called()
window2_on_gain_focus_handler.assert_not_called()
window2_on_lose_focus_handler.assert_not_called()
window1_on_gain_focus_handler.reset_mock()
window1_on_lose_focus_handler.reset_mock()
window2_on_gain_focus_handler.reset_mock()
window2_on_lose_focus_handler.reset_mock()

Given that this will be a repeated pattern, it might be worth wrapping this sequence of calls in a utility assert methods.

@@ -133,9 +133,15 @@ def get_current_window(self):
return self._get_value("current_window", main_window)

def set_current_window(self, window):
previous_current_window = getattr(self.get_current_window(), "interface", None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIR, get_current_window() can return None; but if the window exists, it will always have an interface attribute. On that basis, this getattr() is protecting against the wrong missing attribute.

Comment on lines 186 to 197
def simulate_on_gain_focus(self):
self.interface.on_gain_focus()

def simulate_on_lose_focus(self):
self.interface.on_lose_focus()

def simulate_on_show(self):
self.interface.on_show()

def simulate_on_hide(self):
self.interface.on_hide()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in the linked thread, I think these aren't needed. simulate_ calls are generally only needed when it's a user interaction that needs to be simulated but there's no corresponding API (or the programmatic API behaves differently to the user-initiated one). In this case, the "user action" is giving a window focus, which is equivalent to the programmatic focus() call (and the same for hide/show)

Co-authored-by: Russell Keith-Magee <[email protected]>
@freakboy3742 freakboy3742 reopened this Jan 9, 2025
@freakboy3742
Copy link
Member

Apologies - the close was a stray mouse click.

@proneon267
Copy link
Contributor Author

proneon267 commented Jan 9, 2025

I've had a quick look at some of the implementations as well; I've flagged a couple of things that stood out, but the biggest issue that occurred to me is the interaction of MINIMIZED and HIDE. What happens if a MINIMIZED window is HIDDEN, then restored to normal? It should still be HIDDEN... but AFAICT it will trigger an on_show event in most of these implementations.

Currently on the main branch, we are ignoring window state requests on hidden windows, as changing state of hidden window causes glitches on some platforms:

@state.setter
def state(self, state: WindowState) -> None:
if not self.visible:
raise RuntimeError("Window state of a hidden window cannot be changed.")

So, the following cannot be done:
WindowState.MINIMIZED -> window.hide() -> WindowState.NORMAL
However, the following is valid, and works correctly:
WindowState.MINIMIZED -> window.hide() -> window.show() -> WindowState.NORMAL

EDIT: To confirm that the valid pathway in the above works correctly, I have added a new test. This has helped me catch the cases where the events were getting double triggered.

@proneon267
Copy link
Contributor Author

On testing, I have found that on gtk, when Gtk.Window.hide() or Gtk.Window.show() is called then gtk decides to trigger window-state-event multiple times, and that too with the wrong window_state_flags, causing the visibility events(on_show and on_hide) to be triggered at wrong times.

Also, when Gtk.Window.hide() is called, then afterwards the window state cannot be determined correctly, so we need to cache the previous state before Gtk.Window.hide() is called.

Furthermore, it turns out that Gtk.Window.hide() and Gtk.Window.show() are non-blocking events, so we need to wait for appropriate request completion callbacks, before asserting cases on the testbed.

Since, we need to accomodate all of these cases, the implementation on gtk will take some time. Therefore, can I do the gtk implementation on a separate PR, and skip it here? As the implementation is working correctly on all platforms except on gtk, so skipping the gtk implementation on this PR should not be an issue.

@freakboy3742
Copy link
Member

Since, we need to accomodate all of these cases, the implementation on gtk will take some time. Therefore, can I do the gtk implementation on a separate PR, and skip it here? As the implementation is working correctly on all platforms except on gtk, so skipping the gtk implementation on this PR should not be an issue.

Sure - an implementation that works on not-all platforms is fine - in fact, if there's an underlying issue with the state change tests, then we should definitely resolve those first.

As long as the tests pass reliably on all platforms (and are skipped on platforms where the implementation doesn't exist/doesn't work reliably), and it's clear in the review request what does (and doesn't) work, I'm fine with reviewing and merging. All the usual guidelines about skips apply (i.e, avoid referencing GTK specifically in the test - try to raise the skip in the probe).

@proneon267
Copy link
Contributor Author

proneon267 commented Jan 13, 2025

Ok, I have removed the gtk's visibility events(i.e., on_show() and on_hide()) triggering implementations from this PR, and have updated the tests accordingly.

So, both the focus events i.e., on_gain_focus & on_lose_focus are implemented and tested on all platforms(includiing gtk). However, both the visibility events i.e., on_show and on_hide are implemented and tested on all the platforms, except on gtk(I will implement them in a separate PR for gtk).

Also, another issue is testing of the events on mobile platforms, where the app cannot be sent to background and brought forward, switch between apps, or switch to the "App switcher" screen. All of these are required to automate the testing of focus and visibility events on the mobile platforms. Since, these automated tests are currently not possible, hence I have manually tested them on a physical Android 14 device, on the Android emulator and on the iOS simulator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide on_gain_focus and on_lose_focus handlers on the Window class
3 participants