Pixel searches

[Perlence]

Hey @t89! You mentioned in #9 that you implemented pixel searches with NumPy and OpenCV. I'm interested in how you generally use them. Could you share some examples, please?

I haven't used pixel searches in the original AHK, and I didn't feel confident about rethinking the API in Python. However, I feel that the old Win32 API for windows and controls has been largely replaced with the modern frameworks like WinUI, Qt, and Electron. Of all the apps I use everyday, only like 2–3 use the classic API.

[t89]

Will gladly do so; might need a few days before I've got time to elaborate, though. Expect an answer by the weekend.

[t89]

General usecases and importance of pixel-functions

Generally speaking I find both functions a bit awkward to use with vanilla Autohotkey: Results are returned via call-by-ref, handling color components is kind of awkward (e.g. splitting a hex-string into components), and PixelSearch literally returns an error, if the color was not found. I would assume most people implement their own wrappers for these two functions. I see their biggest usecases in interactions with programs which do not offer you an API (most likely gaming); and for me they are core functions of AHK. In combination with image search these enable the user to create truly magic macros.

PixelGetColor

I mostly use PixelGetColor to perform a quick check of a state which has a visual representation, but which would be hard to verify otherwise. E.g.: When you start an application and it displays a splash screen and you cannot interact with it, yet. You find the app's pid, you find a window (which is not necessarily named "splash"), but if you start your script you will potentially run into trouble with the first commands not being registered the way you expect... You can get the coordinates of the splash screen and check if certain pixels still have the "splashscreen color", or if the color has changed compared to the last check. Or, if you are automating stubborn business apps, like I used to for the past weeks, I use it to determine the state the user is currently in, by quickly checking a set of uniquely colored pixels on the screen (buttons are grayed-out, etc). I know that this code is likely to break on an update, but that application hasn't seen an update in 8 years, anyway... The most frequent use-case of this function you'll encounter in the wild would certainly be in gaming, though. A so called auto-fire script: You check the color of a dot next to your crosshair and trigger left mouse when the color changes by a certain threshold. You can also check for availability of abilities and check for active elements. (Check if armor is broken, if unit exists, if inventory is full, count elements...; the sky is the limit)

Most of my use-cases for this function are in a wrapper which returns True when the color of a pixel lies within a certain threshold: pixel_has_color_with_threshold(x, y, color_string, threshold) -> BOOL

PixelSearch

PixelSearch scans a rectangle defined by two points and returns the coordinates of the first occurance of a pixel in a certain color; or an error. The search is directional and starts from the first point you provide it with. This way you can adjust the direction from which you approach the search by switching your coordinates. I consider this behaviour a crucial feature, which should be accounted for in a wrapper function. My two main usecases for this function are 1. orientation and 2. data extraction.

1. Orientation: I mostly use this in a gaming context: You can detect the position of certain UI elements like health bars, or icons which pop up on the screen. These UI elements usually stand out quite well due to their popping colors. This orientation process is usually the starting point of a macro.

2. Data extraction: When a loading bar is present you can scan a rectangle with the height of 1px and the width of the bar. The coordinates of the first pixel in non-loading bar color allow you to calculate the current percentage of the bar. You can also check the fill-status of a textfield by looking for the color black and searching from the bottom right. You would receive the exact character position and could draw conclusions about the fill-status. This function is also one of the biggest bottle necks in Autohotkey: Taking a screenshot, iterating over hundreds of pixels takes time and AHK is limited by its single thread. Especially in a gaming context you have to get creative to find the elements you are looking for under a strict time constraint: Even if you run the game at just 60fps (~17ms/frame), the coordinates will certainly have changed once you have run the initial orientation phase and performed multiple pixel operations. Running these operations multiple times in an algorithm is very inefficient. This is where I achieved the significant speed up by using python:

Example

Let me elaborate and give you an example including both:

Disclaimer: I do not use this online; cheating sucks. I just enjoy the challenge of improving the usability of games and squeezing any bit of efficiency out of them. It's more fun than playing to me.

In Age of Empires 2 you can control up to 60 units in one group. The positions of the unit icons in the bottom UI would allow you to count the amount of units by using PixelGetColor on an icon and check if it has the UI's background color. I implemented a binary search pattern which counts the units in worst case log(60)=~6 calls. Knowing the position of the health bars we can then run PixelSearch for each health bar and determine a close estimate of an individual unit's health. Using Autohotkey functionality the script works fine after the binary search optimization, but is not viable during a hectic fight, because every tick counts. Calling a function 60 times sequentially to capture a (tiny) region of the screen and iterating through the pixels is extremely inefficient, obviously. In between fights it still was an awesome feature. With python win32gui and numpy I'm able to just take one screenshot of the area, mask it so that I have a boolean matrix and then I'm able to run well performing vector operations on it. Counting the units and extracting their health at the same time, with up to 2-3hp accuracy pretty much happens instantaneously compared to AHK.

Initial screenshot for comparison

List of health-values and mask after bitwise-and operation with screenshot

I can even run the whole capture and masking process on a second thread and the performance boost is immense. Only limit now is the input limit the UI imposes. Performance is so good, actually, that I can pass live images to a neural net I trained for AoE which detects units on the fly, at the same time. I uploaded an unlisted youtube video here. Please refrain from sharing it.

My thoughts on your implementation

If I install autohotkey.py, I install it as a wrapper for AHK and would expect it to call the exact AHK function. Even though I have used AHK for several years, I might be completely unaware of quirks or a certain behavior of a function other users rely on. Even if there would be a quicker solutions for the problem. What I would appreciate is a more pythonic way of interacting with them. E.g.: automated error handling in the case of a negative search, maybe choosing a color-output format... For highly specific use-cases people will have to find their own solution, anyway. As a wrapper I would also try to keep its dependency footprint as constraint as possible: I haven't checked your projects dependencies, but if you do not include numpy, pillow, and win32gui already, I would not bother with reinventing the wheel. If you do, you could offer a better performing re-implementation at a later point in development and make it opt-in via a parameter at first. Hope this helped you a bit to get a feel for the functions.

[Perlence]

This is an awesome overview, thank you so much!

[t89]

You're welcome. Let me know when you've other questions or feedback requests.

[t89]

Hi Sviatoslav, hope my message finds you well during these troublesome times.
Over the past weeks I implemented a basic check-pixel functionality inside of python using some win dll calls. I am not using AHK to improve latency. If you are interested in some sample code, let me know.

[Perlence]

Hey Thomas, thank you, I would love to see it.